[leveldb] Durability Guarantees

[Lucas Lersch]

Hi,

I have a question regarding durability. As stated in the documentation,
leveldb uses async writes to the log file meaning that it may lose updates
in the event of a system failure. If I want to have full durability
guarantees I have to write every WriteBatch in synchronous mode. If I run a
YCSB benchmark with inserts/updates directly on top of leveldb with 1
thread only, I get really poor performance since there is one IO call for
every key-value pair (or so). My question is regarding the best way to
ensure durability while not losing so much in performance. Should this case
be treated at a higher level or maybe at a lower level (fs with
journaling)? Would multi-threading alleviate the high performance-cost in
this case?

Best regards.

[Robert Escriva]

Hi,
I have a question regarding durability. As stated in the documentation, leveldb
uses async writes to the log file meaning that it may lose updates in the event
of a system failure. If I want to have full durability guarantees I have to
write every WriteBatch in synchronous mode. If I run a YCSB benchmark with
inserts/updates directly on top of leveldb with 1 thread only, I get really
poor performance since there is one IO call for every key-value pair (or so).
My question is regarding the best way to ensure durability while not losing so
much in performance. Should this case be treated at a higher level or maybe at
a lower level (fs with journaling)? Would multi-threading alleviate the high
performance-cost in this case?
Best regards.

For LevelDB, you can either increase the batch size to increase the
number of items written per fsync call, or you can loosen the durability
guarantees you get. Ultimately, whenever you flush to disk durably, you
will wait for the disk to write and performance will be worse than when
you don't have to wait for the disk.

You can improve performance by setting the "relatime,data=writeback"
filesystem options on ext4. This will change the guarantees made by the
filesystem in a way that can impact some applications, but should not
mean anything negative for LevelDB.

-Robert