Thanks, that was very elucidative. I am taking a look at both leveldb and
participate in rocksdb discussion group. Anyway, it is cool to see that you
Post by Dhruba BorthakurThis exact problem has caused us some pain too earlier. We enhanced this
https://github.com/facebook/rocksdb/blob/master/include/rocksdb/options.h#L102
There were some use-cases that were ok with the default leveldb recovery
mode (which skips over corruptions in the transaction log), but there were
other use-cases that needed the database open to fail even if there is a
single corruption in the transaction log.
enum class WALRecoveryMode : char {
// Original levelDB recovery
// We tolerate incomplete record in trailing data on all logs
// Use case : This is legacy behavior (default)
kTolerateCorruptedTailRecords = 0x00,
// Recover from clean shutdown
// We don't expect to find any corruption in the WAL
// Use case : This is ideal for unit tests and rare applications that
// can require high consistency guarantee
kAbsoluteConsistency = 0x01,
// Recover to point-in-time consistency
// We stop the WAL playback on discovering WAL inconsistency
// Use case : Ideal for systems that have disk controller cache like
// hard disk, SSD without super capacitor that store related data
kPointInTimeRecovery = 0x02,
// Recovery after a disaster
// We ignore any corruption in the WAL and try to salvage as much data as
// possible
// Use case : Ideal for last ditch effort to recover data or systems that
// operate with low grade unrelated data
kSkipAnyCorruptedRecords = 0x03,
};
Post by Lucas LerschThanks for the answer. I get it. But in case you have a system failure
and need to rebuild based on the log file, if there is a corruption early
in the file and you just seek forward to the next block, you lose all the
updated in the first block. Putting in other words, why is a corruption in
the log file not treated as something critical? Why can you just ignore it
and keep going?
Post by Robert EscrivaThe block format means that corruption early in a file does not damage
the entire file. You can simply seek forward 32KB at a time until you
find a valid place to resume parsing.
-Robert
Post by Lucas LerschHi,
this is probably a basic question, but the documentation says: "The
log file
Post by Lucas Lerschcontents are a sequence of 32KB blocks. The only exception is that
the tail of
Post by Lucas Lerschthe file may contain a partial block". Why exactly is it organized as
32KB
Post by Lucas Lerschblocks? In other words, why is the block organization useful? Can't I
just
Post by Lucas Lerschappend log entries in the following format?
entry :=
checksum: uint32 // crc32c of type and data[] ; little-endian
sequence: fixed64
count: fixed32
data: record[count]
record := kTypeValue varstring varstring | kTypeDeletion
varstring
Post by Lucas Lerschvarstring :=
len: varint32
data: uint8[len]
Best regards.
--
You received this message because you are subscribed to the Google
Groups
Post by Lucas Lersch"leveldb" group.
To unsubscribe from this group and stop receiving emails from it, send
an email
Post by Lucas LerschFor more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to a topic in the
Google Groups "leveldb" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/leveldb/-5iAL3Fr8i0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
For more options, visit https://groups.google.com/d/optout.
--
Lucas Lersch
--
You received this message because you are subscribed to the Google Groups
"leveldb" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.
--
Subscribe to my posts at http://www.facebook.com/dhruba
--
You received this message because you are subscribed to a topic in the
Google Groups "leveldb" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/leveldb/-5iAL3Fr8i0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
For more options, visit https://groups.google.com/d/optout.