leveldb data corrupt after power down.

[Chunping Ruan]

2 days ago, my leveldb server restart after power switch tripping.

Unfortunately, a 5GB size db corrupted. all keys (10,000,000+) became
inavaiable.

I googled a python script, and tried to repair the db.

#!/usr/local/bin/python

import leveldb
leveldb.RepairDB('/data/leveldb-db1')


I can see lot's of such info in LOG file:

................
013/07/22-01:16:48.812110 801407400 Table #8527104: 0 entries Corruption:
corrupted compressed block contents
2013/07/22-01:16:48.812123 801407400 Table #8527104: ignoring Corruption:
corrupted compressed block contents
2013/07/22-01:16:48.812170 801407400 Archiving
/data/sleveldb-db1/8527104.sst: OK
............

After "repair" , there's only 2 .sst files left in the directory, and the
other 3000+ .sst files were moved to another subdir called "lost"

Is there some useful tools to repair the corrupted data ,or how to repair
the db?

OS: FreeBSD 8.2 amd64
Leveldb: 1.8.0
snappy: 1.0.5

[Sanjay Ghemawat]

Post by Chunping Ruan
2 days ago, my leveldb server restart after power switch tripping.
Unfortunately, a 5GB size db corrupted. all keys (10,000,000+) became
inavaiable.

What was the error message when you tried to open the database after the
restart?

Post by Chunping Ruan
I googled a python script, and tried to repair the db.
#!/usr/local/bin/python
import leveldb
leveldb.RepairDB('/data/leveldb-db1')
................
corrupted compressed block contents
corrupted compressed block contents

I presume that your normal DB handling code links in the snappy library?
Perhaps your python bindings that you used for the repair above don't link
in snappy? I would suggest adding a "--repair" type option to your normal
application and have it call Repair itself (passing in any special
comparator you may be using for normal operation).

Since you already ran repair, you will have to undo what it did.
(a) Move all of the lost/* files back into the db directory
(b) Save away all files that were created during repair (file system
modification time should help)

Also, before you run repair again, I would strongly suggest just making a
copy of the db directory and running repair on that. You can use
hard-links (the "ln" command) to copy any *.sst files cheaply.

[Chunping Ruan]

Dear sanjay,

Post by Chunping Ruan
2 days ago, my leveldb server restart after power switch tripping.

Post by Chunping Ruan
Unfortunately, a 5GB size db corrupted. all keys (10,000,000+) became
inavaiable.

What was the error message when you tried to open the database after the
restart?

Post by Chunping Ruan
I googled a python script, and tried to repair the db.
#!/usr/local/bin/python
import leveldb
leveldb.RepairDB('/data/leveldb-db1')
................
corrupted compressed block contents
corrupted compressed block contents

I presume that your normal DB handling code links in the snappy library?
Perhaps your python bindings that you used for the repair above don't link
in snappy? I would suggest adding a "--repair" type option to your normal
application and have it call Repair itself (passing in any special
comparator you may be using for normal operation).

You are right!
I checked my libleveldb.so , snappy was not linked :(

It's very strange, I installed the leveldb from freebsd's ports, and did
choose SNAPPY...

Now, I modiffed the build_detect_platform, and install it again.

$ ldd /usr/local/lib/libleveldb.so
/usr/local/lib/libleveldb.so:
libsnappy.so.2 => /usr/local/lib/libsnappy.so.2 (0x80125d000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x801463000)
libm.so.5 => /lib/libm.so.5 (0x801773000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x801994000)
libthr.so.3 => /lib/libthr.so.3 (0x801ba1000)
libc.so.7 => /lib/libc.so.7 (0x80081a000)


And installed py-leveldb again too.

$ldd /usr/local/lib/python2.7/site-packages/leveldb.so
/usr/local/lib/python2.7/site-packages/leveldb.so:
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x80120b000)
libleveldb.so.1 => /usr/local/lib/libleveldb.so.1 (0x80151b000)
libm.so.5 => /lib/libm.so.5 (0x801778000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x801999000)
libc.so.7 => /lib/libc.so.7 (0x80081a000)
libsnappy.so.2 => /usr/local/lib/libsnappy.so.2 (0x801ba6000)
libthr.so.3 => /lib/libthr.so.3 (0x801dac000)

(there is a mistake in my last post, my OS is FreeBSD 9.1-RELEASE amd 64)

The db was generated by redis-storage ( redis API and leveldb as backend ,
https://github.com/rchunping/redis-storage )
The snappy and leveldb were static linked into redis-storage, so the data
was compressed by snappy.


Since you already ran repair, you will have to undo what it did.

Post by Chunping Ruan
(a) Move all of the lost/* files back into the db directory
(b) Save away all files that were created during repair (file system
modification time should help)
Also, before you run repair again, I would strongly suggest just making a
copy of the db directory and running repair on that. You can use
hard-links (the "ln" command) to copy any *.sst files cheaply.

I will try these steps, thank you very much!!!!

[Chunping Ruan]

Dear sanjay,

Thank you very much,follow your steps, db was fixed.everything is ok now.

Post by Chunping Ruan
Dear sanjay,

Post by Chunping Ruan
2 days ago, my leveldb server restart after power switch tripping.

Post by Chunping Ruan
Unfortunately, a 5GB size db corrupted. all keys (10,000,000+) became
inavaiable.

What was the error message when you tried to open the database after the
restart?

Post by Chunping Ruan
I googled a python script, and tried to repair the db.
#!/usr/local/bin/python
import leveldb
leveldb.RepairDB('/data/leveldb-db1')
................
013/07/22-01:16:48.812110 801407400 Table #8527104: 0 entries
Corruption: corrupted compressed block contents
2013/07/22-01:16:48.812123 801407400 Table #8527104: ignoring
Corruption: corrupted compressed block contents

I presume that your normal DB handling code links in the snappy library?
Perhaps your python bindings that you used for the repair above don't link
in snappy? I would suggest adding a "--repair" type option to your normal
application and have it call Repair itself (passing in any special
comparator you may be using for normal operation).

You are right!
I checked my libleveldb.so , snappy was not linked :(
It's very strange, I installed the leveldb from freebsd's ports, and did
choose SNAPPY...
Now, I modiffed the build_detect_platform, and install it again.
$ ldd /usr/local/lib/libleveldb.so
libsnappy.so.2 => /usr/local/lib/libsnappy.so.2 (0x80125d000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x801463000)
libm.so.5 => /lib/libm.so.5 (0x801773000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x801994000)
libthr.so.3 => /lib/libthr.so.3 (0x801ba1000)
libc.so.7 => /lib/libc.so.7 (0x80081a000)
And installed py-leveldb again too.
$ldd /usr/local/lib/python2.7/site-packages/leveldb.so
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x80120b000)
libleveldb.so.1 => /usr/local/lib/libleveldb.so.1 (0x80151b000)
libm.so.5 => /lib/libm.so.5 (0x801778000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x801999000)
libc.so.7 => /lib/libc.so.7 (0x80081a000)
libsnappy.so.2 => /usr/local/lib/libsnappy.so.2 (0x801ba6000)
libthr.so.3 => /lib/libthr.so.3 (0x801dac000)
(there is a mistake in my last post, my OS is FreeBSD 9.1-RELEASE amd 64)
The db was generated by redis-storage ( redis API and leveldb as backend ,
https://github.com/rchunping/redis-storage )
The snappy and leveldb were static linked into redis-storage, so the data
was compressed by snappy.
Since you already ran repair, you will have to undo what it did.

Post by Chunping Ruan
(a) Move all of the lost/* files back into the db directory
(b) Save away all files that were created during repair (file system
modification time should help)
Also, before you run repair again, I would strongly suggest just making a
copy of the db directory and running repair on that. You can use
hard-links (the "ln" command) to copy any *.sst files cheaply.

I will try these steps, thank you very much!!!!

[Junaid Farooq]

I know this is an old post.. but is there any possibility to repair the data but dint recover very old indexs and files?

We used that leveldb.repair and it worked but it not just repaired but recovered so many data which wasnt even needed.. is there any way to clean this up?

Is there any way to repair but the option of dont recover?

We just want to repair the corruption.

2 days ago, my leveldb server restart after power switch tripping.
Unfortunately, a 5GB size db corrupted. all keys (10,000,000+) became inavaiable.   
I googled a python script, and tried to repair the db.
#!/usr/local/bin/python
import leveldb
leveldb.RepairDB('/data/leveldb-db1')
................
013/07/22-01:16:48.812110 801407400 Table #8527104: 0 entries Corruption: corrupted compressed block contents
2013/07/22-01:16:48.812123 801407400 Table #8527104: ignoring Corruption: corrupted compressed block contents
2013/07/22-01:16:48.812170 801407400 Archiving /data/sleveldb-db1/8527104.sst: OK
............
After "repair" , there's only 2 .sst files left in the directory, and the other 3000+ .sst files were moved to another subdir called "lost"
Is there some useful tools to repair the corrupted data ,or how to repair the db?
OS: FreeBSD 8.2 amd64
Leveldb: 1.8.0
snappy: 1.0.5