You might be thinking that the common prefixes (/1000/10) for the keys are wasting storage, but RocksDB our underlying storage engine, eliminates almost all of the overhead via prefix compression of keys.
But I couldn’t find an in-depth description of this feature in RocksDB (searched for prefix compression and some similar queries). Could you please provide a pointer, or a better term to search for?
Hi @bra! Thank you for your interest in CockroachDB and RocksDB.
The prefix compression for keys is documented here:
// When we store a key, we drop the prefix shared with the previous
// string. This helps reduce the space requirement significantly.
// Furthermore, once every K keys, we do not apply the prefix
// compression and store the entire key. We call this a "restart
// point". The tail end of the block stores the offsets of all of the
// restart points, and can be used to do a binary search when looking
// for a particular key.