rocksdb-native 0.0.0

package/vendor/rocksdb/docs/_posts/2018-08-23-data-block-hash-index.markdown

@@ -0,0 +1,118 @@
+---
+title: Improving Point-Lookup Using Data Block Hash Index
+layout: post
+author: fgwu
+category: blog
+---
+We've designed and implemented a _data block hash index_ in RocksDB that has the benefit of both reducing the CPU util and increasing the throughput for point lookup queries with a reasonable and tunable space overhead. 
+
+Specifially, we append a compact hash table to the end of the data block for efficient indexing. It is backward compatible with the data base created without this feature. After turned on the hash index feature, existing data will be gradually converted to the hash index format.
+
+Benchmarks with `db_bench`  show the CPU utilization of one of the main functions in the point lookup code path, `DataBlockIter::Seek()`, is reduced by 21.8%, and the overall RocksDB throughput is increased by 10% under purely cached workloads, at an overhead of 4.6% more space. Shadow testing with Facebook production traffic shows good CPU improvements too.
+
+
+### How to use it
+Two new options are added as part of this feature: `BlockBasedTableOptions::data_block_index_type` and `BlockBasedTableOptions::data_block_hash_table_util_ratio`.
+
+The hash index is disabled by default unless `BlockBasedTableOptions::data_block_index_type` is set to `data_block_index_type = kDataBlockBinaryAndHash`. The hash table utilization ratio is adjustable using `BlockBasedTableOptions::data_block_hash_table_util_ratio`, which is valid only if `data_block_index_type = kDataBlockBinaryAndHash`.
+
+
+```
+// the definitions can be found in include/rocksdb/table.h
+
+// The index type that will be used for the data block.
+enum DataBlockIndexType : char {
+  kDataBlockBinarySearch = 0,  // traditional block type
+  kDataBlockBinaryAndHash = 1, // additional hash index
+};
+
+// Set to kDataBlockBinaryAndHash to enable hash index
+DataBlockIndexType data_block_index_type = kDataBlockBinarySearch;
+
+// #entries/#buckets. It is valid only when data_block_hash_index_type is
+// kDataBlockBinaryAndHash.
+double data_block_hash_table_util_ratio = 0.75;
+
+```
+
+
+### Data Block Hash Index Design
+
+Current data block format groups adjacent keys together as a restart interval. One block consists of multiple restart intervals. The byte offset of the beginning of each restart interval, i.e. a restart point, is stored in an array called restart interval index or binary seek index. RocksDB does a binary search when performing point lookup for keys in data blocks to find the right restart interval the key may reside. We will use binary seek and binary search interchangeably in this post.
+
+In order to find the right location where the key may reside using binary search, multiple key parsing and comparison are needed. Each binary search branching triggers CPU cache miss, causing much CPU utilization. We have seen that this binary search takes up considerable CPU in production use-cases.
+
+![](/static/images/data-block-hash-index/block-format-binary-seek.png)
+
+We implemented a hash map at the end of the block to index the key to reduce the CPU overhead of the binary search. The hash index is just an array of pointers pointing into the binary seek index.
+
+![](/static/images/data-block-hash-index/block-format-hash-index.png)
+
+
+Each array element is considered as a hash bucket when storing the location of a key (or more precisely, the restart index of the restart interval where the key resides). When multiple keys happen to hash into the same bucket (hash collision), we just mark the bucket as “collision”. So that when later querying on that key, the hash table lookup knows that there was a hash collision happened so it can fall back to the traditional binary search to find the location of the key.
+
+We define hash table utilization ratio as the #keys/#buckets. If a utilization ratio is 0.5 and there are 100 buckets, 50 keys are stored in the bucket. The less the util ratio, the less hash collision, and the less chance for a point lookup falls back to binary seek (fall back ratio) due to the collision. So a small util ratio has more benefit to reduce the CPU time but introduces more space overhead.
+
+Space overhead depends on the util ratio. Each bucket is a `uint8_t`  (i.e. one byte). For a util ratio of 1, the space overhead is 1Byte per key, the fall back ratio observed is ~52%.
+
+![](/static/images/data-block-hash-index/hash-index-data-structure.png)
+
+### Things that Need Attention
+
+**Customized Comparator**
+
+Hash index will hash different keys (keys with different content, or byte sequence) into different hash values. This assumes the comparator will not treat different keys as equal if they have different content. 
+
+The default bytewise comparator orders the keys in alphabetical order and works well with hash index, as different keys will never be regarded as equal. However, some specially crafted comparators will do. For example, say, a `StringToIntComparator` can convert a string into an integer, and use the integer to perform the comparison. Key string “16” and “0x10” is equal to each other as seen by this `StringToIntComparator`, but they probably hash to different value. Later queries to one form of the key will not be able to find the existing key been stored in the other format.
+
+We add a new function member to the comparator interface: 
+
+```
+virtual bool CanKeysWithDifferentByteContentsBeEqual() const { return true; }
+```
+
+
+Every comparator implementation should override this function and specify the behavior of the comparator. If a comparator can regard different keys equal, the function returns true, and as a result the hash index feature will not be enabled, and vice versa.
+
+NOTE: to use the hash index feature, one should 1) have a comparator that can never treat different keys as equal; and 2) override the `CanKeysWithDifferentByteContentsBeEqual()` function to return `false`, so the hash index can be enabled.
+
+
+**Util Ratio's Impact on Data Block Cache**
+
+Adding the hash index to the end of the data block essentially takes up the data block cache space, making the effective data block cache size smaller and increasing the data block cache miss ratio. Therefore, a very small util ratio will result in a large data block cache miss ratio, and the extra I/O may drag down the throughput gain achieved by the hash index lookup. Besides, when compression is enabled, cache miss also incurs data block decompression, which is CPU-consuming. Therefore the CPU may even increase if using a too small util ratio. The best util ratio depends on workloads, cache to data ratio, disk bandwidth/latency etc. In our experiment, we found util ratio = 0.5 ~ 1 is a good range to explore that brings both CPU and throughput gains.
+
+
+### Limitations
+
+As we use `uint8_t` to store binary seek index, i.e. restart interval index, the total number of restart intervals cannot be more than 253 (we reserved  255 and 254 as special flags). For blocks having a larger number of restart intervals, the hash index will not be created and the point lookup will be done by traditional binary seek.
+
+Data block hash index only supports point lookup. We do not support range lookup. Range lookup request will fall back to BinarySeek.
+
+RocksDB supports many types of records, such as `Put`, `Delete`, `Merge`, etc (visit [here](https://github.com/facebook/rocksdb/wiki/rocksdb-basics) for more information). Currently we only support `Put` and `Delete`, but not `Merge`. Internally we have a limited set of supported record types:
+
+
+```
+kPutRecord,          <=== supported
+kDeleteRecord,       <=== supported
+kSingleDeleteRecord, <=== supported
+kTypeBlobIndex,      <=== supported
+```
+
+For records not supported, the searching process will fall back to the traditional binary seek. 
+
+
+
+### Evaluation
+To evaluate the CPU util reduction and isolate other factors such as disk I/O and block decompression, we first evaluate the hash idnex in a purely cached workload. We observe that the CPU utilization of one of the main functions in the point lookup code path, DataBlockIter::Seek(), is reduced by 21.8% and the overall throughput is increased by 10% at an overhead of 4.6% more space.
+
+However, general worload is not always purely cached. So we also evaluate the performance under different cache space pressure. In the following test, we use `db_bench` with RocksDB deployed on SSDs. The total DB size is 5~6GB, and it is about 14GB if decompressed. Different block cache sizes are used, ranging from 14GB down to 2GB, with an increasing cache miss ratio.
+
+Orange bars are representing our hash index performance. We use a hash util ratio of 1.0 in this test. Block size are set to 16KiB with the restart interval as 16.
+
+![](/static/images/data-block-hash-index/perf-throughput.png)
+![](/static/images/data-block-hash-index/perf-cache-miss.png)
+
+We can see that if cache size is greater than 8GB, hash index can bring throughput gain. Cache size greater than 8GB can be translated to a cache miss ratio smaller than 40%. So if the workload has a cache miss ratio smaller than 40%, hash index is able to increase the throughput.
+
+Besides, shadow testing with Facebook production traffic shows good CPU improvements too.
+

package/vendor/rocksdb/docs/_posts/2018-11-21-delete-range.markdown

@@ -0,0 +1,292 @@
+---
+title: "DeleteRange: A New Native RocksDB Operation"
+layout: post
+author:
+- abhimadan
+- ajkr
+category: blog
+---
+## Motivation
+
+### Deletion patterns in LSM
+
+Deleting a range of keys is a common pattern in RocksDB. Most systems built on top of
+RocksDB have multi-component key schemas, where keys sharing a common prefix are
+logically related. Here are some examples.
+
+MyRocks is a MySQL fork using RocksDB as its storage engine. Each key's first
+four bytes identify the table or index to which that key belongs. Thus dropping
+a table or index involves deleting all the keys with that prefix.
+
+Rockssandra is a Cassandra variant that uses RocksDB as its storage engine. One
+of its admin tool commands, `nodetool cleanup`, removes key-ranges that have been migrated
+to other nodes in the cluster.
+
+Marketplace uses RocksDB to store product data. Its key begins with product ID,
+and it stores various data associated with the product in separate keys. When a
+product is removed, all these keys must be deleted.
+
+When we decide what to improve, we try to find a use case that's common across
+users, since we want to build a generally useful system, not one that has many
+one-off features for individual users. The range deletion pattern is common as
+illustrated above, so from this perspective it's a good target for optimization.
+
+### Existing mechanisms: challenges and opportunities
+
+The most common pattern we see is scan-and-delete, i.e., advance an iterator
+through the to-be-deleted range, and issue a `Delete` for each key. This is
+slow (involves read I/O) so cannot be done in any critical path. Additionally,
+it creates many tombstones, which slows down iterators and doesn't offer a deadline
+for space reclamation.
+
+Another common pattern is using a custom compaction filter that drops keys in
+the deleted range(s). This deletes the range asynchronously, so cannot be used
+in cases where readers must not see keys in deleted ranges. Further, it has the
+disadvantage of outputting tombstones to all but the bottom level. That's
+because compaction cannot detect whether dropping a key would cause an older
+version at a lower level to reappear.
+
+If space reclamation time is important, or it is important that the deleted
+range not affect iterators, the user can trigger `CompactRange` on the deleted
+range. This can involve arbitrarily long waits in the compaction queue, and
+increases write-amp. By the time it's finished, however, the range is completely
+gone from the LSM.
+
+`DeleteFilesInRange` can be used prior to compacting the deleted range as long
+as snapshot readers do not need to access them. It drops files that are
+completely contained in the deleted range. That saves write-amp because, in
+`CompactRange`, the file data would have to be rewritten several times before it
+reaches the bottom of the LSM, where tombstones can finally be dropped.
+
+In addition to the above approaches having various drawbacks, they are quite
+complicated to reason about and implement. In an ideal world, deleting a range
+of keys would be (1) simple, i.e., a single API call; (2) synchronous, i.e.,
+when the call finishes, the keys are guaranteed to be wiped from the DB; (3) low
+latency so it can be used in critical paths; and (4) a first-class operation
+with all the guarantees of any other write, like atomicity, crash-recovery, etc.
+
+## v1: Getting it to work
+
+### Where to persist them?
+
+The first place we thought about storing them is inline with the data blocks.
+We could not think of a good way to do it, however, since the start of a range
+tombstone covering a key could be anywhere, making binary search impossible.
+So, we decided to investigate segregated storage.
+
+A second solution we considered is appending to the manifest. This file is
+append-only, periodically compacted, and stores metadata like the level to which
+each SST belongs. This is tempting because it leverages an existing file, which
+is maintained in the background and fully read when the DB is opened. However,
+it conceptually violates the manifest's purpose, which is to store metadata. It
+also has no way to detect when a range tombstone no longer covers anything and
+is droppable. Further, it'd be possible for keys above a range tombstone to disappear
+when they have their seqnums zeroed upon compaction to the bottommost level.
+
+A third candidate is using a separate column family. This has similar problems
+to the manifest approach. That is, we cannot easily detect when a range
+tombstone is obsolete, and seqnum zeroing can cause a key
+to go from above a range tombstone to below, i.e., disappearing. The upside is
+we can reuse logic for memory buffering, consistent reads/writes, etc.
+
+The problems with the second and third solutions indicate a need for range
+tombstones to be aware of flush/compaction. An easy way to achieve this is put
+them in the SST files themselves - but not in the data blocks, as explained for
+the first solution. So, we introduced a separate meta-block for range tombstones.
+This resolved the problem of when to obsolete range tombstones, as it's simple:
+when they're compacted to the bottom level. We also reused the LSM invariants
+that newer versions of a key are always in a higher level to prevent the seqnum
+zeroing problem. This approach has the side benefit of constraining the range
+tombstones seen during reads to ones in a similar key-range.
+
+![](/static/images/delrange/delrange_sst_blocks.png)
+{: style="display: block; margin-left: auto; margin-right: auto; width: 80%"}
+
+*When there are range tombstones in an SST, they are segregated in a separate meta-block*
+{: style="text-align: center"}
+
+![](/static/images/delrange/delrange_key_schema.png)
+{: style="display: block; margin-left: auto; margin-right: auto; width: 80%"}
+
+*Logical range tombstones (left) and their corresponding physical key-value representation (right)*
+{: style="text-align: center"}
+
+### Write path
+
+`WriteBatch` stores range tombstones in its buffer which are logged to the WAL and
+then applied to a dedicated range tombstone memtable during `Write`. Later in
+the background the range tombstone memtable and its corresponding data memtable
+are flushed together into a single SST with a range tombstone meta-block. SSTs
+periodically undergo compaction which rewrites SSTs with point data and range
+tombstones dropped or merged wherever possible.
+
+We chose to use a dedicated memtable for range tombstones. The memtable
+representation is always skiplist in order to minimize overhead in the usual
+case, which is the memtable contains zero or a small number of range tombstones.
+The range tombstones are segregated to a separate memtable for the same reason
+we segregated range tombstones in SSTs. That is, we did not know how to
+interleave the range tombstone with point data in a way that we would be able to
+find it for arbitrary keys that it covers.
+
+![](/static/images/delrange/delrange_write_path.png)
+{: style="display: block; margin-left: auto; margin-right: auto; width: 70%"}
+
+*Lifetime of point keys and range tombstones in RocksDB*
+{: style="text-align: center"}
+
+During flush and compaction, we chose to write out all non-obsolete range
+tombstones unsorted. Sorting by a single dimension is easy to implement, but
+doesn't bring asymptotic improvement to queries over range data. Ideally, we
+want to store skylines (see “Read Path” subsection below) computed over our ranges so we can binary search.
+However, a couple of concerns cause doing this in flush and compaction to feel
+unsatisfactory: (1) we need to store multiple skylines, one for each snapshot,
+which further complicates the range tombstone meta-block encoding; and (2) even
+if we implement this, the range tombstone memtable still needs to be linearly
+scanned. Given these concerns we decided to defer collapsing work to the read
+side, hoping a good caching strategy could optimize this at some future point.
+
+
+### Read path
+
+In point lookups, we aggregate range tombstones in an unordered vector as we
+search through live memtable, immutable memtables, and then SSTs. When a key is
+found that matches the lookup key, we do a scan through the vector, checking
+whether the key is deleted.
+
+In iterators, we aggregate range tombstones into a skyline as we visit live
+memtable, immutable memtables, and SSTs. The skyline is expensive to construct but fast to determine whether a key is covered. The skyline keeps track of the most recent range tombstone found to optimize `Next` and `Prev`.
+
+|![](/static/images/delrange/delrange_uncollapsed.png)	|![](/static/images/delrange/delrange_collapsed.png)	|
+
+*([Image source: Leetcode](https://leetcode.com/problems/the-skyline-problem/description/)) The skyline problem involves taking building location/height data in the
+unsearchable form of A and converting it to the form of B, which is
+binary-searchable. With overlapping range tombstones, to achieve efficient
+searching we need to solve an analogous problem, where the x-axis is the
+key-space and the y-axis is the sequence number.*
+{: style="text-align: center"}
+
+### Performance characteristics
+
+For the v1 implementation, writes are much faster compared to the scan and
+delete (optionally within a transaction) pattern. `DeleteRange` only logs to WAL
+and applies to memtable. Logging to WAL always `fflush`es, and optionally
+`fsync`s or `fdatasync`s. Applying to memtable is always an in-memory operation.
+Since range tombstones have a dedicated skiplist memtable, the complexity of inserting is O(log(T)), where T is the number of existing buffered range tombstones.
+
+Reading in the presence of v1 range tombstones, however, is much slower than reads
+in a database where scan-and-delete has happened, due to the linear scan over
+range tombstone memtables/meta-blocks.
+
+Iterating in a database with v1 range tombstones is usually slower than in a
+scan-and-delete database, although the gap lessens as iterations grow longer.
+When an iterator is first created and seeked, we construct a skyline over its
+tombstones. This operation is O(T\*log(T)) where T is the number of tombstones
+found across live memtable, immutable memtable, L0 files, and one file from each
+of the L1+ levels. However, moving the iterator forwards or backwards is simply
+a constant-time operation (excluding edge cases, e.g., many range tombstones
+between consecutive point keys).
+
+## v2: Making it fast
+
+`DeleteRange`’s negative impact on read perf is a barrier to its adoption. The
+root cause is range tombstones are not stored or cached in a format that can be
+efficiently searched. We needed to design DeleteRange so that we could maintain
+write performance while making read performance competitive with workarounds
+used in production (e.g., scan-and-delete).
+
+### Representations
+
+The key idea of the redesign is that, instead of globally collapsing range tombstones,
+ we can locally “fragment” them for each SST file and memtable to guarantee that:
+
+* no range tombstones overlap; and
+* range tombstones are ordered by start key.
+
+Combined, these properties make range tombstones binary searchable. This
+ fragmentation will happen on the read path, but unlike the previous design, we can
+ easily cache many of these range tombstone fragments on the read path.
+
+### Write path
+
+The write path remains unchanged.
+
+### Read path
+
+When an SST file is opened, its range tombstones are fragmented and cached. For point
+ lookups, we binary search each file's fragmented range tombstones for one that covers
+ the lookup key. Unlike the old design, once we find a tombstone, we no longer need to
+ search for the key in lower levels, since we know that any keys on those levels will be
+ covered (though we do still check the current level since there may be keys written after
+ the range tombstone).
+
+For range scans, we create iterators over all the fragmented range
+ tombstones and store them in a list, seeking each one to cover the start key of the range
+ scan (if possible), and query each encountered key in this structure as in the old design,
+ advancing range tombstone iterators as necessary. In effect, we implicitly create a skyline.
+ This requires significantly less work on iterator creation, but since each memtable/SST has
+its own range tombstone iterator, querying range tombstones requires key comparisons (and
+possibly iterator increments) for several iterators (as opposed to v1, where we had a global
+collapsed representation of all range tombstones). As a result, very long range scans may become
+ slower than before, but short range scans are an order of magnitude faster, which are the
+ more common class of range scan.
+
+## Benchmarks
+
+To understand the performance of this new design, we used `db_bench` to compare point lookup, short range scan,
+ and long range scan performance across:
+
+* the v1 DeleteRange design,
+* the scan-and-delete workaround, and
+* the v2 DeleteRange design.
+
+In these benchmarks, we used a database with 5 million data keys, and 10000 range tombstones (ignoring
+those dropped during compaction) that were written in regular intervals after 4.5 million data keys were written.
+Writing the range tombstones ensures that most of them are not compacted away, and we have more tombstones
+in higher levels that cover keys in lower levels, which allows the benchmarks to exercise more interesting behavior
+when reading deleted keys.
+
+Point lookup benchmarks read 100000 keys from a database using `readwhilewriting`. Range scan benchmarks used
+`seekrandomwhilewriting` and seeked 100000 times, and advanced up to 10 keys away from the seek position for short range scans, and advanced up to 1000 keys away from the seek position for long range scans.
+
+The results are summarized in the tables below, averaged over 10 runs (note the
+different SHAs for v1 benchmarks are due to a new `db_bench` flag that was added in order to compare performance with databases with no tombstones; for brevity, those results are not reported here). Also note that the block cache was large enough to hold the entire db, so the large throughput is due to limited I/Os and little time spent on decompression. The range tombstone blocks are always pinned uncompressed in memory. We believe these setup details should not affect relative performance between versions.
+
+### Point Lookups
+
+|Name	|SHA	|avg micros/op	|avg ops/sec	|
+|v1	|35cd754a6	|1.3179	|759,830.90	|
+|scan-del	|7528130e3	|0.6036	|1,667,237.70	|
+|v2	|7528130e3	|0.6128	|1,634,633.40	|
+
+### Short Range Scans
+
+|Name	|SHA	|avg micros/op	|avg ops/sec	|
+|v1	|0ed738fdd	|6.23	|176,562.00	|
+|scan-del	|PR 4677	|2.6844	|377,313.00	|
+|v2	|PR 4677	|2.8226	|361,249.70	|
+
+### Long Range scans
+
+|Name	|SHA	|avg micros/op	|avg ops/sec	|
+|v1	|0ed738fdd	|52.7066	|19,074.00	|
+|scan-del	|PR 4677	|38.0325	|26,648.60	|
+|v2	|PR 4677	|41.2882	|24,714.70	|
+
+## Future Work
+
+Note that memtable range tombstones are fragmented every read; for now this is acceptable,
+ since we expect there to be relatively few range tombstones in memtables (and users can
+ enforce this by keeping track of the number of memtable range deletions and manually flushing
+ after it passes a threshold). In the future, a specialized data structure can be used for storing
+ range tombstones in memory to avoid this work.
+
+Another future optimization is to create a new format version that requires range tombstones to
+ be stored in a fragmented form. This would save time when opening SST files, and when `max_open_files` 
+is not -1 (i.e., files may be opened several times).
+
+## Acknowledgements
+
+Special thanks to Peter Mattis and Nikhil Benesch from Cockroach Labs, who were early users of
+DeleteRange v1 in production, contributed the cleanest/most efficient v1 aggregation implementation, found and fixed bugs, and provided initial DeleteRange v2 design and continued help.
+
+Thanks to Huachao Huang and Jinpeng Zhang from PingCAP for early DeleteRange v1 adoption, bug reports, and fixes.

package/vendor/rocksdb/docs/_posts/2019-03-08-format-version-4.markdown

@@ -0,0 +1,36 @@
+---
+title: format_version 4
+layout: post
+author: maysamyabandeh
+category: blog
+---
+
+The data blocks in RocksDB consist of a sequence of key/values pairs sorted by key, where the pairs are grouped into _restart intervals_ specified by `block_restart_interval`. Up to RocksDB version 5.14, where the latest and default value of `BlockBasedTableOptions::format_version` is 2, the format of index and data blocks are the same: index blocks use the same key format of <`user_key`,`seq`> and encode pointers to data blocks, <`offset`,`size`>, to a byte string and use them as values. The only difference is that the index blocks use `index_block_restart_interval` for the size of _restart intervals_. `format_version=`3,4 offer more optimized, backward-compatible, yet forward-incompatible format for index blocks. 
+
+### Pros
+
+Using `format_version`=4 significantly reduces the index block size, in some cases around 4-5x. This frees more space in block cache, which would result in higher hit rate for data and filter blocks, or offer the same performance with a smaller block cache size.
+
+### Cons
+
+Being _forward-incompatible_ means that if you enable `format_version=`4 you cannot downgrade to a RocksDB version lower than 5.16.
+
+### How to use it?
+
+- `BlockBasedTableOptions::format_version` = 4
+- `BlockBasedTableOptions::index_block_restart_interval` = 16
+
+### What is format_version 3?
+(Since RocksDB 5.15) In most cases, the sequence number `seq` is not necessary for keys in the index blocks. In such cases, `format_version`=3 skips encoding the sequence number and sets `index_key_is_user_key` in TableProperties, which is used by the reader to know how to decode the index block.
+
+### What is format_version 4?
+(Since RocksDB 5.16) Changes the format of index blocks by delta encoding the index values, which are the block handles. This saves the encoding of `BlockHandle::offset` of the non-head index entries in each restart interval. If used, `TableProperties::index_value_is_delta_encoded` is set, which is used by the reader to know how to decode the index block.  The format of each key is (shared_size, non_shared_size, shared, non_shared). The format of each value, i.e., block handle, is (offset, size) whenever the shared_size is 0, which included the first entry in each restart point. Otherwise the format is delta-size = block handle size - size of last block handle.
+
+The index format in `format_version=4` would be as follows:
+
+    restart_point   0: k, v (off, sz), k, v (delta-sz), ..., k, v (delta-sz)
+    restart_point   1: k, v (off, sz), k, v (delta-sz), ..., k, v (delta-sz)
+    ...
+    restart_point n-1: k, v (off, sz), k, v (delta-sz), ..., k, v (delta-sz)
+    where, k is key, v is value, and its encoding is in parenthesis.
+

package/vendor/rocksdb/docs/_posts/2019-08-15-unordered-write.markdown

@@ -0,0 +1,56 @@
+---
+title: Higher write throughput with `unordered_write` feature
+layout: post
+author: maysamyabandeh
+category: blog
+---
+
+Since RocksDB 6.3, The `unordered_write=`true option together with WritePrepared transactions offers 34-42% higher write throughput compared to vanilla RocksDB. If the application can handle more relaxed ordering guarantees, the gain in throughput would increase to 63-131%.
+
+### Background
+
+Currently RocksDB API delivers the following powerful guarantees:
+- Atomic reads: Either all of a write batch is visible to reads or none of it.
+- Read-your-own writes: When a write thread returns to the user, a subsequent read by the same thread will be able to see its own writes.
+- Immutable Snapshots: The reads visible to the snapshot are immutable in the sense that it will not be affected by any in-flight or future writes.
+
+### `unordered_write`
+
+The `unordered_write` feature, when turned on, relaxes the default guarantees of RocksDB. While it still gives read-your-own-write property, neither atomic reads nor the immutable snapshot properties are provided any longer. However, RocksDB users could still get read-your-own-write and immutable snapshots when using this feature in conjunction with TransactionDB configured with WritePrepared transactions and `two_write_queues`. You can read [here](https://github.com/facebook/rocksdb/wiki/unordered_write) to learn about the design of `unordered_write` and [here](https://github.com/facebook/rocksdb/wiki/WritePrepared-Transactions) to learn more about WritePrepared transactions.
+
+### How to use it?
+
+To get the same guarantees as vanilla RocksdB:
+
+    DBOptions db_options;
+    db_options.unordered_write = true;
+    db_options.two_write_queues = true;
+    DB* db;
+    {
+      TransactionDBOptions txn_db_options;
+      txn_db_options.write_policy = TxnDBWritePolicy::WRITE_PREPARED;
+      txn_db_options.skip_concurrency_control = true;
+      TransactionDB* txn_db;
+      TransactionDB::Open(options, txn_db_options, kDBPath, &txn_db);
+      db = txn_db;
+    }
+    db->Write(...);
+
+To get relaxed guarantees:
+
+    DBOptions db_options;
+    db_options.unordered_write = true;
+    DB* db;
+    DB::Open(db_options, kDBPath, &db);
+    db->Write(...);
+
+# Benchmarks
+
+    TEST_TMPDIR=/dev/shm/ ~/db_bench --benchmarks=fillrandom --threads=32 --num=10000000 -max_write_buffer_number=16 --max_background_jobs=64 --batch_size=8 --writes=3000000 -level0_file_num_compaction_trigger=99999 --level0_slowdown_writes_trigger=99999 --level0_stop_writes_trigger=99999 -enable_pipelined_write=false -disable_auto_compactions --transaction_db=true --unordered_write=1 --disable_wal=0
+
+Throughput with `unordered_write`=true and using WritePrepared transaction:
+- WAL: +42%
+- No-WAL: +34%
+Throughput with `unordered_write`=true
+- WAL: +63%
+- NoWAL: +131%

package/vendor/rocksdb/docs/_posts/2021-04-12-universal-improvements.markdown

@@ -0,0 +1,46 @@
+---
+title: (Call For Contribution) Make Universal Compaction More Incremental
+layout: post
+author: sdong
+category: blog
+---
+
+### Motivation
+
+Universal Compaction is an important compaction style, but few changes were made after we made the structure multi-leveled. Yet the major restriction of always compacting full sorted run is not relaxed. Compared to Leveled Compaction, where we usually only compile several SST files together, in universal compaction, we frequently compact GBs of data. Two issues with this gap: 1. it makes it harder to unify universal and leveled compaction; 2. periodically data is fully compacted, and in the mean time space is doubled. To ease the problem, we can break the restriction and do similar as leveled compaction, and bring it closer to unified compaction.
+
+We call for help for making following improvements.
+
+
+### How Universal Compaction Works
+
+In universal, whole levels are compacted together to satisfy two conditions (See [wiki page](https://github.com/facebook/rocksdb/wiki/Universal-Compaction) for more details):
+
+1. total size / bottommost level size > a threshold, or
+2. total number of sorted runs (non-0 levels + L0 files) is within a threshold
+
+1 is to limit extra space overhead used for dead data and 2 is for read performance.
+
+If 1 is triggered, likely a full compaction will be triggered. If 2 is triggered, RocksDB compact some sorted runs to bring the number down. It does it by using a simple heuristic so that less writes needed for that purpose over time: it starts from compacting smaller files, but if total size to compact is similar to or larger than size of the next level, it will take that level together, as soon on (whether it is the best heuristic is another question and we’ve never seriously looked at it).
+
+### How We Can Improve?
+
+Let’s start from condition 1. Here we do full compaction but is not necessary.  A simple optimization would be to compact so that just enough files are merged into the bottommost level (Lmax) to satisfy condition 1. It would work if we only need to pick some files from Lmax-1, or if it is cheaper over time, we can pick some files from other levels too.
+
+Then condition 2. If we finish condition 1, there might be holes in some ranges in older levels. These holes might make it possible that only by compacting some sub ranges, we can fix the LSM-tree for condition 2. RocksDB can take single files into consideration and apply more sophisticated heuristic.
+
+This new approach makes universal compaction closer to leveled compaction. The operation for 1 is closer to how Leveled compaction triggeres Lmax-1 to Lmax compaction. And 2 can potentially be implemented as something similar to level picking in Leveled Compaction. In fact, all those file picking can co-existing in one single compaction style and there isn’t fundamental conflicts to that.
+
+### Limitation
+
+There are two limitations:
+
+* Periodic automatic full compaction is unpleasant but at the same time is pleasant in another way. Some users might uses it to reason that everything is periodically collapsed so dead data is gone and old data is rewritten. We need to make sure periodic compaction works to continue with that.
+* L0 to the first non-L0 level compaction is the first time data is partitioned in LSM-tree so that incremental compaction by range is possible. We might need to do more of these compactions in order to make incremental possible, which will increase compaction slightly.
+* Compacting subset of a level would introduce some extra overhead for unaligned files, just as in leveled compaction. More SST boundary cutting heuristic can reduce this overhead but it will be there.
+
+But I believe the benefits would outweight the limitations. Reducing temporary space doubling and moving towards to unified compaction would be important achievements.
+
+### Interested in Help?
+
+Compaction is the core of LSM-tree, but its improvements are far overdue. If you are a user of universal compaction and would be able to benefit from those improvements, we will be happy to work with you on speeding up the project and bring them to RocksDB sooner. Feel free to communicate with us in [this issue](https://github.com/facebook/rocksdb/issues/8181).