[leveldb] leveldb db corruption on brownout and shutdown?

[Joel de Guzman]

Hello,

We're using leveldb in an embedded system (linux) that may be powered off
without shutting down the OS.
The system may also restart when some failure is detected. Although it is
not 100% confirmed yet, it seems
we're experiencing db corruption when that happens. Is leveldb resilient on
unexpected power outages/
brownouts? is there any other key-value store library that is resilient on
such in the first place? Or is it not
a good idea to use a database like leveldb in such a use-case?

Thoughts?

Thanks in advance,
--Joel

[Joel de Guzman]

Post by Joel de Guzman
Hello,
We're using leveldb in an embedded system (linux) that may be powered off
without shutting down the OS.
The system may also restart when some failure is detected. Although it is
not 100% confirmed yet, it seems
we're experiencing db corruption when that happens. Is leveldb resilient
on unexpected power outages/
brownouts? is there any other key-value store library that is resilient on
such in the first place? Or is it not
a good idea to use a database like leveldb in such a use-case?
Thoughts?
Thanks in advance,
--Joel

[Florian Weimer]

Post by Joel de Guzman
we're experiencing db corruption when that happens. Is leveldb
resilient on unexpected power outages/ brownouts?

I have forgotten what my review of the leveldb sources for this aspect
yielded a while back, sorry (and fixes may have went in since then).

The larger problem is cheap storage lying about write completion, or
kernels which do not even attempt to implement barriers correctly.

Most key-value stores/databases assume that some amount of data is
written atomically. This is supposed to match spinning disk behavior
(unless you have IBM DTLAs, which may require a sector overwrite to
revive a sector which was written during drive power-down). With
other kinds of storage (such as degraded RAID-5), a one-sector write
can wipe out much more than just a single sector during a power fault.

Berkeley DB assumes that entire (multi-sector) pages are written
atomically, which is probably not true in general. On the other hand,
SQLite and PostgreSQL should cope rather well with a storage subsystem
which implements barriers properly, without requiring anything
further.

(But some versions of SQLite skip a barrier before returning success
to the application, and this is very difficult to change without
recompiling SQLite. As a result, one transaction can be rolled back
after a power outage, even if its successful completion was reported
to the application. SQLite database consistency is not affected,
though.)

[Robert Escriva]

Post by Joel de Guzman
Hello,
We're using leveldb in an embedded system (linux) that may be powered off
without shutting down the OS.
The system may also restart when some failure is detected. Although it is not
100% confirmed yet, it seems 
we're experiencing db corruption when that happens. Is leveldb resilient on
unexpected power outages/
brownouts? is there any other key-value store library that is resilient on such
in the first place? Or is it not
a good idea to use a database like leveldb in such a use-case?
Thoughts?
Thanks in advance,
--Joel

You may find some answers in this paper and its related research:
https://www.usenix.org/node/186195

The authors managed to find real application-level bugs in various data
storage backends on various filesystems.

The answer to your question of what is resilient, it turns out, is very
heavily dependent upon both the storage backend you use, and the
filesystem it accesses.

-Robert

[Joel de Guzman]

Thank you all for your replies!

Post by Florian Weimer

Post by Joel de Guzman
Hello,
We're using leveldb in an embedded system (linux) that may be powered

off

Post by Joel de Guzman
without shutting down the OS.
The system may also restart when some failure is detected. Although it

is not

Post by Joel de Guzman
100% confirmed yet, it seems
we're experiencing db corruption when that happens. Is leveldb resilient

on

Post by Joel de Guzman
unexpected power outages/
brownouts? is there any other key-value store library that is resilient

on such

Post by Joel de Guzman
in the first place? Or is it not
a good idea to use a database like leveldb in such a use-case?
Thoughts?
Thanks in advance,
--Joel

https://www.usenix.org/node/186195
The authors managed to find real application-level bugs in various data
storage backends on various filesystems.
The answer to your question of what is resilient, it turns out, is very
heavily dependent upon both the storage backend you use, and the
filesystem it accesses.
-Robert