[leveldb] Performance question for writes on SSD/external hard drive

[Bradley Neuberg]

I'm writing a fairly large dataset into a LevelDB database (a bit larger
than 1 TB) that I'm then feeding into a library named Caffe to do deep
learning.

I have an SSD drive that is fast but has only about 200 GBs, and an
external large drive that is 3 TBs over a USB 3 connection but slower. I am
currently completely writing out the database to the large external drive
currently; it takes about 5 hours, both due to write time and protobuf
serialization. I'd like to speed up the write portion (I'm investigating
other things for the protobuf serialization).

While writing the LevelDB files I was wondering if its possible to write
out segments of the database to the SSD for much faster write performance
and then move the pieces over to the larger external hard drive during the
whole process? Any advice on doing this? I'm not sure how LevelDB will
respond if I move a file out from under it after its finished writing.

I am currently using the Python LevelDB libraries.

Here's a gist with the method doing the writes for reference:

https://gist.github.com/BradNeuberg/c47a07cce0230cfc2f5d

Best,
Brad Neuberg