memory increasing all the time when using leveldb with LRUCache in different threads

[赵琦]

I'm using leveldb with large LRUcache in multithread environment.
Db is started in the main thread. Then I started a new thread and scan some
data into LRUCache in the thread using seek() and next() api. Without the
former thread exit, I do repeatelly starting a new thread and loading the
same data to LRUCache. The werid thing is the memory of the process never
stop increasing, and far exceeded the capcity of LRUCache i had set. BTW, I
had closed the mmap by modify src of leveldb. So mmap is not account for
ceaseless incremental of memory usage.

Can any one tell me what happend?
I have been bothered by this problem for weeks...

[赵琦]

test code is as below. Memory usage continue to increase when starting new
thread and loading data.



#include <cassert>
#include <vector>
#include <iostream>
#include <string>
#include <unistd.h>
#include <stdio.h>
#include <pthread.h>
#include "leveldb/db.h"
#include "leveldb/cache.h"

using namespace std;

leveldb::DB* pldb;
leveldb::Cache* pblock_cache = NULL;

void* loadhotThread(void* arg){
cout<<"new thread start"<<endl;
int cnt = 0;

leveldb::Iterator* it = pldb->NewIterator(leveldb::ReadOptions());
for (it->SeekToFirst();
it->Valid();
it->Next()) {
it->value()[0];
cnt++;
}
delete it;
cout<<cnt<<" data load;"<<endl;

//thread not exit
cout<<"thread begin sleep"<<endl;
while(1){
sleep(10);
}

printf("thread exit\n");
fflush(stdout);
pthread_detach(pthread_self());
return NULL;
}

int loadhotMultiThread(int argc, char **argv)
{
int cnt = 0;
while(cnt<10){
pthread_t pid;
pthread_create(&pid, NULL, loadhotThread, NULL);
//wait the thread load data
sleep(60);
cnt++;
}

cout<<"main thread sleep"<<endl;
while(1){
sleep(10);
}
return 0;
}

int main(int argc, char **argv)
{
if (argc != 2){
cout<<"usage:"<<argv[0]<<" /path/to/leveldb/data/"<<endl;
return -1;
}
string dir = argv[1];
leveldb::Options options;
//2G cache size
uint64_t a = 2000000000;
options.block_cache = leveldb::NewLRUCache(a);
pblock_cache = options.block_cache;
options.max_open_files = 65535;
options.create_if_missing = true;
options.block_size = 1024*32;
options.write_buffer_size = 128000000;
leveldb::Status s = leveldb::DB::Open(options, dir, &pldb);

loadhotMultiThread(argc, argv);

delete pldb;
delete pblock_cache;
return 0;
}

//g++ test_levelcache.cpp
/data/pavelliu/extern_source/leveldb-1.14.0/libleveldb.a -o test_levelcache
-I /data/pavelliu/extern_source/leveldb-1.14.0/include/
-L/data/pavelliu/local/lib/ -I/data/pavelliu/local/include/ -g -lsnappy
-lpthread

[Xingxiao Zh]

Hi,

What's the size of your testing db, and how much memory increased after
each full scan of db?

Btw, how about using some memory profiler, e.g. gperftools heap profiler,
to identify which exactly part of code is eating your memory?

Best Regards.

Post by 赵琦
test code is as below. Memory usage continue to increase when starting new
thread and loading data.
#include <cassert>
#include <vector>
#include <iostream>
#include <string>
#include <unistd.h>
#include <stdio.h>
#include <pthread.h>
#include "leveldb/db.h"
#include "leveldb/cache.h"
using namespace std;
leveldb::DB* pldb;
leveldb::Cache* pblock_cache = NULL;
void* loadhotThread(void* arg){
cout<<"new thread start"<<endl;
int cnt = 0;
leveldb::Iterator* it = pldb->NewIterator(leveldb::ReadOptions());
for (it->SeekToFirst();
it->Valid();
it->Next()) {
it->value()[0];
cnt++;
}
delete it;
cout<<cnt<<" data load;"<<endl;
//thread not exit
cout<<"thread begin sleep"<<endl;
while(1){
sleep(10);
}
printf("thread exit\n");
fflush(stdout);
pthread_detach(pthread_self());
return NULL;
}
int loadhotMultiThread(int argc, char **argv)
{
int cnt = 0;
while(cnt<10){
pthread_t pid;
pthread_create(&pid, NULL, loadhotThread, NULL);
//wait the thread load data
sleep(60);
cnt++;
}
cout<<"main thread sleep"<<endl;
while(1){
sleep(10);
}
return 0;
}
int main(int argc, char **argv)
{
if (argc != 2){
cout<<"usage:"<<argv[0]<<" /path/to/leveldb/data/"<<endl;
return -1;
}
string dir = argv[1];
leveldb::Options options;
//2G cache size
uint64_t a = 2000000000;
options.block_cache = leveldb::NewLRUCache(a);
pblock_cache = options.block_cache;
options.max_open_files = 65535;
options.create_if_missing = true;
options.block_size = 1024*32;
options.write_buffer_size = 128000000;
leveldb::Status s = leveldb::DB::Open(options, dir, &pldb);
loadhotMultiThread(argc, argv);
delete pldb;
delete pblock_cache;
return 0;
}
//g++ test_levelcache.cpp
/data/pavelliu/extern_source/leveldb-1.14.0/libleveldb.a -o test_levelcache
-I /data/pavelliu/extern_source/leveldb-1.14.0/include/
-L/data/pavelliu/local/lib/ -I/data/pavelliu/local/include/ -g -lsnappy
-lpthread
--
You received this message because you are subscribed to the Google Groups "leveldb" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/groups/opt_out.

[赵琦]

The size of my testing db is 2.5GB. I set LRUCache size to 2GB.
And each time data was loaded, about 2GB memory increased.

haven't using any memory profiler. i'll give a try.
thanks for your advice.


圚 2014幎2月13日星期四UTC+8䞋午12时05分38秒Xingxiao Zh写道

Post by Xingxiao Zh
Hi,
What's the size of your testing db, and how much memory increased after
each full scan of db?
Btw, how about using some memory profiler, e.g. gperftools heap profiler,
to identify which exactly part of code is eating your memory?
Best Regards.

Post by 赵琦
test code is as below. Memory usage continue to increase when starting
new thread and loading data.
#include <cassert>
#include <vector>
#include <iostream>
#include <string>
#include <unistd.h>
#include <stdio.h>
#include <pthread.h>
#include "leveldb/db.h"
#include "leveldb/cache.h"
using namespace std;
leveldb::DB* pldb;
leveldb::Cache* pblock_cache = NULL;
void* loadhotThread(void* arg){
cout<<"new thread start"<<endl;
int cnt = 0;
leveldb::Iterator* it = pldb->NewIterator(leveldb::ReadOptions());
for (it->SeekToFirst();
it->Valid();
it->Next()) {
it->value()[0];
cnt++;
}
delete it;
cout<<cnt<<" data load;"<<endl;
//thread not exit
cout<<"thread begin sleep"<<endl;
while(1){
sleep(10);
}
printf("thread exit\n");
fflush(stdout);
pthread_detach(pthread_self());
return NULL;
}
int loadhotMultiThread(int argc, char **argv)
{
int cnt = 0;
while(cnt<10){
pthread_t pid;
pthread_create(&pid, NULL, loadhotThread, NULL);
//wait the thread load data
sleep(60);
cnt++;
}
cout<<"main thread sleep"<<endl;
while(1){
sleep(10);
}
return 0;
}
int main(int argc, char **argv)
{
if (argc != 2){
cout<<"usage:"<<argv[0]<<" /path/to/leveldb/data/"<<endl;
return -1;
}
string dir = argv[1];
leveldb::Options options;
//2G cache size
uint64_t a = 2000000000;
options.block_cache = leveldb::NewLRUCache(a);
pblock_cache = options.block_cache;
options.max_open_files = 65535;
options.create_if_missing = true;
options.block_size = 1024*32;
options.write_buffer_size = 128000000;
leveldb::Status s = leveldb::DB::Open(options, dir, &pldb);
loadhotMultiThread(argc, argv);
delete pldb;
delete pblock_cache;
return 0;
}
//g++ test_levelcache.cpp
/data/pavelliu/extern_source/leveldb-1.14.0/libleveldb.a -o test_levelcache
-I /data/pavelliu/extern_source/leveldb-1.14.0/include/
-L/data/pavelliu/local/lib/ -I/data/pavelliu/local/include/ -g -lsnappy
-lpthread
--
You received this message because you are subscribed to the Google Groups
"leveldb" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/groups/opt_out.

[t***@gmail.com]

what you set is the number of LRUHandle and each LRUHandle contains a
pointer to one block, so the LRUCache size should be 2TB at least since the
minum block size is 1KB.

圚 2014幎2月12日星期䞉 UTC+8䞋午2:35:03赵琊写道

Post by 赵琦
test code is as below. Memory usage continue to increase when starting new
thread and loading data.
#include <cassert>
#include <vector>
#include <iostream>
#include <string>
#include <unistd.h>
#include <stdio.h>
#include <pthread.h>
#include "leveldb/db.h"
#include "leveldb/cache.h"
using namespace std;
leveldb::DB* pldb;
leveldb::Cache* pblock_cache = NULL;
void* loadhotThread(void* arg){
cout<<"new thread start"<<endl;
int cnt = 0;
leveldb::Iterator* it = pldb->NewIterator(leveldb::ReadOptions());
for (it->SeekToFirst();
it->Valid();
it->Next()) {
it->value()[0];
cnt++;
}
delete it;
cout<<cnt<<" data load;"<<endl;
//thread not exit
cout<<"thread begin sleep"<<endl;
while(1){
sleep(10);
}
printf("thread exit\n");
fflush(stdout);
pthread_detach(pthread_self());
return NULL;
}
int loadhotMultiThread(int argc, char **argv)
{
int cnt = 0;
while(cnt<10){
pthread_t pid;
pthread_create(&pid, NULL, loadhotThread, NULL);
//wait the thread load data
sleep(60);
cnt++;
}
cout<<"main thread sleep"<<endl;
while(1){
sleep(10);
}
return 0;
}
int main(int argc, char **argv)
{
if (argc != 2){
cout<<"usage:"<<argv[0]<<" /path/to/leveldb/data/"<<endl;
return -1;
}
string dir = argv[1];
leveldb::Options options;
//2G cache size
uint64_t a = 2000000000;
options.block_cache = leveldb::NewLRUCache(a);
pblock_cache = options.block_cache;
options.max_open_files = 65535;
options.create_if_missing = true;
options.block_size = 1024*32;
options.write_buffer_size = 128000000;
leveldb::Status s = leveldb::DB::Open(options, dir, &pldb);
loadhotMultiThread(argc, argv);
delete pldb;
delete pblock_cache;
return 0;
}
//g++ test_levelcache.cpp
/data/pavelliu/extern_source/leveldb-1.14.0/libleveldb.a -o test_levelcache
-I /data/pavelliu/extern_source/leveldb-1.14.0/include/
-L/data/pavelliu/local/lib/ -I/data/pavelliu/local/include/ -g -lsnappy
-lpthread