tigerbeetle-node 0.10.0 → 0.11.0

package/src/tigerbeetle/src/lsm/README.md

@@ -0,0 +1,214 @@
+# Glossary
+
+- _bar_/_measure_: `lsm_batch_multiple` beats; unit of incremental compaction.
+- _beat_: `op % lsm_batch_multiple`; Single step of an incremental compaction.
+- _groove_: A collection of LSM trees, storing objects and their indices.
+- _immutable table_: in-memory table; one per tree. Used to periodically flush the mutable table to
+  disk.
+- _level_: Between `0` and `lsm_levels - 1` (usually `lsm_levels = 7`).
+- _forest_: a collection of grooves.
+- _manifest_: index of table and level metadata; one per tree.
+- _mutable table_: in-memory table; one per tree. All tree updates are applied only to this table.
+- _snapshot_: sequence number which selects the queryable partition of on-disk tables.
+
+# Tree
+## Tables
+
+A tree is a hierarchy of in-memory and on-disk tables. There are three categories of tables:
+
+- The [mutable table](table_mutable.zig) is an in-memory table.
+  - Each tree has a single mutable table.
+  - All tree updates, inserts, and removes are applied to the mutable table.
+  - The mutable table's size is allocated to accommodate a full bar of updates.
+- The [immutable table](table_immutable.zig) is an in-memory table.
+  - Each tree has a single immutable table.
+  - The mutable table's contents are periodically moved to the immutable table,
+    where they are stored while being flushed to level `0`.
+- Level `0` … level `config.lsm_levels - 1` each contain an exponentially increasing number of
+  on-disk tables.
+  - Each tree has as many as `config.lsm_growth_factor ^ (level + 1)` tables per level.
+    (`config.lsm_growth_factor` is typically 8).
+  - Within a given level and snapshot, the tables' key ranges are [disjoint](manifest_level.zig).
+
+## Compaction
+
+Tree compaction runs to the sound of music!
+
+Compacting LSM trees involves merging and moving tables into the next levels as needed.
+To avoid write amplification stalls and bound latency, compaction is done incrementally.
+
+A full compaction phase is denoted as a bar or measure, using terms from music notation.
+Each bar consists of `lsm_batch_multiple` beats or "compaction ticks" of work.
+A compaction tick executes asynchronously immediately after every commit, with
+`beat = commit.op % lsm_batch_multiple`.
+
+A bar is split in half according to the "first" beat and "middle" beat.
+The first half of the bar compacts even levels while the latter compacts odd levels.
+Mutable table changes are sorted and compacted into the immutable table.
+The immutable table is compacted into level 0 during the odd level half of the bar.
+
+At any given point, there are at most `levels/2` compactions running concurrently.
+The source level is denoted as `level_a` and the target level as `level_b`.
+The last level in the LSM tree has no target level so it is never a source level.
+Each compaction compacts a [single table](#table-selection) from `level_a` into all tables in
+`level_b` which intersect the `level_a` table's key range.
+
+Invariants:
+* At the end of every beat, there is space in mutable table for the next beat.
+* The manifest is compacted at the end of every beat.
+* The compactions' output tables are not [visible](#snapshots-and-compaction) until the compaction has finished.
+
+1. First half-bar, first beat ("first beat"):
+    * Assert no compactions are currently running.
+    * Allow the per-level table limits to overflow if needed (for example, if we may compact a table
+      from level `A` to level `B`, where level `B` is already full).
+    * Start compactions from even levels that have reached their table limit.
+
+2. First half-bar, last beat:
+    * Finish ticking any incomplete even-level compactions.
+    * Assert on callback completion that all compactions are complete.
+
+3. Second half-bar, first beat ("middle beat"):
+    * Assert no compactions are currently running.
+    * Start compactions from odd levels that have reached their table limit.
+    * Compact the immutable table if it contains any sorted values (it might be empty).
+
+4. Second half-bar, last beat:
+    * Finish ticking any incomplete odd-level and immutable table compactions.
+    * Assert on callback completion that all compactions are complete.
+    * Assert on callback completion that no level's table count overflows.
+    * Flush, clear, and sort mutable table values into immutable table for next bar.
+    * Remove input tables that are invisible to all current and persisted snapshots.
+
+### Compaction Selection Policy
+
+Compaction targets the table from level `A` which overlaps the fewest tables of level `B`.
+
+For example, in the following table (with `lsm_growth_factor=2`), each table is depicted as the range of keys it includes. The tables with uppercase letters would be chosen for compaction next.
+
+```
+Level 0   A─────────────H       l───────────────────────────z
+Level 1   a───────e             L─M   o───────s   u───────y
+Level 2     b───d e─────h i───k l───n o─p q───s   u─v w─────z
+(Keys)    a b c d e f g h i j k l m n o p q r s t u v w x y z
+```
+
+Links:
+- [`Manifest.compaction_table`](manifest.zig)
+- [Constructing and Analyzing the LSM Compaction Design Space](http://vldb.org/pvldb/vol14/p2216-sarkar.pdf) describes the tradeoffs of various data movement policies. TigerBeetle implements the "least overlapping with parent" policy.
+- [Option of Compaction Priority](https://rocksdb.org/blog/2016/01/29/compaction_pri.html)
+
+## Snapshots
+
+Each table has a minimum and maximum integer snapshot (`snapshot_min` and `snapshot_max`).
+
+Each query targets a particular snapshot. A table `T` is _visible_ to a snapshot `S` when
+
+```
+T.snapshot_min ≤ S ≤ T.snapshot_max
+```
+
+and is _invisible_ to the snapshot otherwise.
+
+Compaction does not modify tables in place — it copies data. Snapshots control and distinguish
+which copies are useful, and which can be deleted. Snapshots can also be persisted, enabling
+queries against past states of the tree (unimplemented; future work).
+
+### Snapshots and Compaction
+
+Consider the half-bar compaction beginning at op=`X` (`12`), with `lsm_batch_multiple=M` (`8`).
+Each half-bar contains `N=M/2` (`4`) beats. The next half-bar begins at `Y=X+N` (`16`).
+
+During the half-bar compaction `X` (op=`X…Y-1`; `12…15`), each commit prefetches from the snapshot
+[equal to its own op](#current-snapshot). As shown, they continue to query the old (input) tables.
+
+During the half-bar compaction `X`:
+- `snapshot_max` of each input table is truncated to `Y-1` (`15`).
+- `snapshot_min` of each output table is initialized to `Y` (`16`).
+
+```
+0   4   8  12  16  20  24  (op, snapshot)
+┼───┬───┼───┬───┼───┬───┼
+            ####
+····────────X────────····  (input  tables, before compaction)
+····────────────           (input  tables,  after compaction)
+                Y────····  (output tables,  after compaction)
+```
+
+Beginning from the next op after the compaction (`Y`; `16`):
+- The output tables of the above compaction `X` are visible.
+- The input tables of the above compaction `X` are invisible.
+- Therefore, it will lookup from the output tables, but ignore the input tables.
+- Callers must not query from the output tables of `X` before the compaction half-bar has finished
+  (i.e. before the end of beat `Y-1` (`15`)), since those tables are incomplete.
+
+At this point the input tables can be removed if they are invisible to all persistent snapshots.
+
+### Snapshot Queries
+
+Each query targets a particular snapshot, either:
+- the [current snapshot](#current-snapshot), or
+- a [persisted snapshot](#persistent-snapshots).
+
+#### Current Snapshot
+
+Each tree tracks the highest snapshot safe to query from (`tree.lookup_snapshot_max`), to ensure that
+an ongoing compaction's incomplete output tables are not visible. Queries targeting
+`tree.lookup_snapshot_max` always read from the mutable and immutable tables — so each commit can
+see all previous commits' updates.)
+
+During typical operation, the `lookup_snapshot_max` when prefetching op `S` is snapshot `S`.
+The following chart depicts:
+- `lookup_snapshot_max` (`$`)
+- for each commit op (the left column)
+- and a compaction that began at op `12` and completed at the end of op `15`.
+
+```
+op  0   4   8  12  16  20  24  (op, snapshot)
+    ┼───┬───┼───┬───┼───┬───┼
+12  ····────────$───
+13  ····─────────$──
+14  ····──────────$─
+15  ····───────────$
+16                  $────····
+17                  ─$───····
+18                  ──$──····
+19                  ───$─····
+```
+
+However, commits in the first measure following recovery from a checkpoint prefetch from a higher
+snapshot to avoid querying tables that were deleted at the checkpoint.
+See [`lookup_snapshot_max_for_checkpoint()`](#tree.zig) for more detail.
+
+#### Persistent Snapshots
+
+TODO(Persistent Snapshots): Expand this section.
+
+### Snapshot Values
+
+- The on-disk tables visible to a snapshot `B` do not contain the updates from the commit with op `B`.
+- Rather, snapshot `B` is first visible to a prefetch from the commit with op `B`.
+
+Consider the following diagram (`lsm_batch_multiple=8`):
+
+```
+0   4   8  12  16  20  24  28  (op, snapshot)
+┼───┬───┼───┬───┼───┬───┼───┬
+        ,,,,,,,,........
+        ↑A      ↑B      ↑C
+```
+
+Compaction is driven by the commits of ops `B→C` (`16…23`). While these ops are being committed:
+- Updates from ops `0→A` (`0…7`) are on-disk.
+- Updates from ops `A→B` (`8…15`) are in the immutable table.
+  - These updates were moved to the immutable table from the immutable table at the end of op `B-1`
+    (`15`).
+  - These updates will exist in the immutable table until it is reset at the end of op `C-1` (`23`).
+- Updates from ops `B→C` (`16…23`) are added to the mutable table (by the respective commit).
+- `tree.lookup_snapshot_max` is `B` when committing op `B`.
+- `tree.lookup_snapshot_max` is `x` when committing op `x` (for `x ∈ {16,17,…,23}`).
+
+At the end of the last beat of the compaction bar (`23`):
+- Updates from ops `0→B` (`0…15`) are on disk.
+- Updates from ops `B→C` (`16…23`) are moved from the mutable table to the immutable table.
+- `tree.lookup_snapshot_max` is `x` when committing op `x` (for `x ∈ {24,25,…}`).

package/src/tigerbeetle/src/lsm/binary_search.zig

@@ -2,14 +2,23 @@ const std = @import("std");
 const assert = std.debug.assert;
 const math = std.math;
 
+pub const Config = struct {
+    verify: bool = false,
+};
+
 // TODO Add prefeching when @prefetch is available: https://github.com/ziglang/zig/issues/3600.
 //
 // TODO The Zig self hosted compiler will implement inlining itself before passing the IR to llvm,
 // which should eliminate the current poor codegen of key_from_value/compare_keys.
 
-/// Returns the index of the key either exactly equal to the target key or, if there is no exact
-/// match, the next greatest key.
-/// Doesn't preform the extra key comparison to determine if the match is exact
+/// Returns either the index of the first value equal to `key`,
+/// or if there is no such value then the index where `key` would be inserted.
+///
+/// In other words, return `i` such that both:
+/// * key_from_value(values[i])  >= key or i == values.len
+/// * key_value_from(values[i-1]) < key or i == 0
+///
+/// Doesn't perform the extra key comparison to determine if the match is exact.
 pub fn binary_search_values_raw(
     comptime Key: type,
     comptime Value: type,
@@ -17,24 +26,57 @@ pub fn binary_search_values_raw(
     comptime compare_keys: fn (Key, Key) callconv(.Inline) math.Order,
     values: []const Value,
     key: Key,
+    comptime config: Config,
 ) u32 {
-    assert(values.len > 0);
+    if (values.len == 0) return 0;
+
+    if (config.verify) {
+        // Input must be sorted by key.
+        for (values) |_, i| {
+            assert(i == 0 or
+                compare_keys(key_from_value(&values[i - 1]), key_from_value(&values[i])) != .gt);
+        }
+    }
 
     var offset: usize = 0;
     var length: usize = values.len;
     while (length > 1) {
+        if (config.verify) {
+            assert(offset == 0 or
+                compare_keys(key_from_value(&values[offset - 1]), key) != .gt);
+            assert(offset + length == values.len or
+                compare_keys(key_from_value(&values[offset + length]), key) != .lt);
+        }
+
         const half = length / 2;
         const mid = offset + half;
 
         // This trick seems to be what's needed to get llvm to emit branchless code for this,
-        // a ternay-style if expression was generated as a jump here for whatever reason.
+        // a ternary-style if expression was generated as a jump here for whatever reason.
         const next_offsets = [_]usize{ offset, mid };
         offset = next_offsets[@boolToInt(compare_keys(key_from_value(&values[mid]), key) == .lt)];
 
         length -= half;
     }
 
-    return @intCast(u32, offset + @boolToInt(compare_keys(key_from_value(&values[offset]), key) == .lt));
+    if (config.verify) {
+        assert(length == 1);
+        assert(offset == 0 or
+            compare_keys(key_from_value(&values[offset - 1]), key) != .gt);
+        assert(offset + length == values.len or
+            compare_keys(key_from_value(&values[offset + length]), key) != .lt);
+    }
+
+    offset += @boolToInt(compare_keys(key_from_value(&values[offset]), key) == .lt);
+
+    if (config.verify) {
+        assert(offset == 0 or
+            compare_keys(key_from_value(&values[offset - 1]), key) == .lt);
+        assert(offset == values.len or
+            compare_keys(key_from_value(&values[offset]), key) != .lt);
+    }
+
+    return @intCast(u32, offset);
 }
 
 pub inline fn binary_search_keys_raw(
@@ -42,6 +84,7 @@ pub inline fn binary_search_keys_raw(
     comptime compare_keys: fn (Key, Key) callconv(.Inline) math.Order,
     keys: []const Key,
     key: Key,
+    comptime config: Config,
 ) u32 {
     return binary_search_values_raw(
         Key,
@@ -54,6 +97,7 @@ pub inline fn binary_search_keys_raw(
         compare_keys,
         keys,
         key,
+        config,
     );
 }
 
@@ -69,8 +113,9 @@ pub inline fn binary_search_values(
     comptime compare_keys: fn (Key, Key) callconv(.Inline) math.Order,
     values: []const Value,
     key: Key,
+    comptime config: Config,
 ) BinarySearchResult {
-    const index = binary_search_values_raw(Key, Value, key_from_value, compare_keys, values, key);
+    const index = binary_search_values_raw(Key, Value, key_from_value, compare_keys, values, key, config);
     return .{
         .index = index,
         .exact = index < values.len and compare_keys(key_from_value(&values[index]), key) == .eq,
@@ -82,8 +127,9 @@ pub inline fn binary_search_keys(
     comptime compare_keys: fn (Key, Key) callconv(.Inline) math.Order,
     keys: []const Key,
     key: Key,
+    comptime config: Config,
 ) BinarySearchResult {
-    const index = binary_search_keys_raw(Key, compare_keys, keys, key);
+    const index = binary_search_keys_raw(Key, compare_keys, keys, key, config);
     return .{
         .index = index,
         .exact = index < keys.len and compare_keys(keys[index], key) == .eq,
@@ -91,6 +137,8 @@ pub inline fn binary_search_keys(
 }
 
 const test_binary_search = struct {
+    const fuzz = @import("../test/fuzz.zig");
+
     const log = false;
 
     const gpa = std.testing.allocator;
@@ -99,6 +147,10 @@ const test_binary_search = struct {
         return math.order(a, b);
     }
 
+    fn less_than_key(_: void, a: u32, b: u32) bool {
+        return a < b;
+    }
+
     fn exhaustive_search(keys_count: u32) !void {
         const keys = try gpa.alloc(u32, keys_count);
         defer gpa.free(keys);
@@ -131,6 +183,7 @@ const test_binary_search = struct {
                 compare_keys,
                 keys,
                 target_key,
+                .{ .verify = true },
             );
 
             if (log) std.debug.print("expected: {}, actual: {}\n", .{ expect, actual });
@@ -159,11 +212,50 @@ const test_binary_search = struct {
                 compare_keys,
                 keys,
                 target_key,
+                .{ .verify = true },
             );
             try std.testing.expectEqual(expect.index, actual.index);
             try std.testing.expectEqual(expect.exact, actual.exact);
         }
     }
+
+    fn random_search(random: std.rand.Random, iter: usize) !void {
+        const keys_count = @minimum(
+            @as(usize, 1E6),
+            fuzz.random_int_exponential(random, usize, iter),
+        );
+
+        const keys = try std.testing.allocator.alloc(u32, keys_count);
+        defer std.testing.allocator.free(keys);
+
+        for (keys) |*key| key.* = fuzz.random_int_exponential(random, u32, 100);
+        std.sort.sort(u32, keys, {}, less_than_key);
+        const target_key = fuzz.random_int_exponential(random, u32, 100);
+
+        var expect: BinarySearchResult = .{ .index = 0, .exact = false };
+        for (keys) |key, i| {
+            switch (compare_keys(key, target_key)) {
+                .lt => expect.index = @intCast(u32, i) + 1,
+                .eq => {
+                    expect.exact = true;
+                    break;
+                },
+                .gt => break,
+            }
+        }
+
+        const actual = binary_search_keys(
+            u32,
+            compare_keys,
+            keys,
+            target_key,
+            .{ .verify = true },
+        );
+
+        if (log) std.debug.print("expected: {}, actual: {}\n", .{ expect, actual });
+        try std.testing.expectEqual(expect.index, actual.index);
+        try std.testing.expectEqual(expect.exact, actual.exact);
+    }
 };
 
 // TODO test search on empty slice
@@ -178,11 +270,38 @@ test "binary search: exhaustive" {
 test "binary search: explicit" {
     if (test_binary_search.log) std.debug.print("\n", .{});
     try test_binary_search.explicit_search(
-        &[_]u32{ 0, 3, 5, 8, 9, 11 },
-        &[_]u32{ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13 },
+        &[_]u32{},
+        &[_]u32{0},
+        &[_]BinarySearchResult{
+            .{ .index = 0, .exact = false },
+        },
+    );
+    try test_binary_search.explicit_search(
+        &[_]u32{1},
+        &[_]u32{ 0, 1, 2 },
         &[_]BinarySearchResult{
+            .{ .index = 0, .exact = false },
             .{ .index = 0, .exact = true },
             .{ .index = 1, .exact = false },
+        },
+    );
+    try test_binary_search.explicit_search(
+        &[_]u32{ 1, 3 },
+        &[_]u32{ 0, 1, 2, 3, 4 },
+        &[_]BinarySearchResult{
+            .{ .index = 0, .exact = false },
+            .{ .index = 0, .exact = true },
+            .{ .index = 1, .exact = false },
+            .{ .index = 1, .exact = true },
+            .{ .index = 2, .exact = false },
+        },
+    );
+    try test_binary_search.explicit_search(
+        &[_]u32{ 1, 3, 5, 8, 9, 11 },
+        &[_]u32{ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13 },
+        &[_]BinarySearchResult{
+            .{ .index = 0, .exact = false },
+            .{ .index = 0, .exact = true },
             .{ .index = 1, .exact = false },
             .{ .index = 1, .exact = true },
             .{ .index = 2, .exact = false },
@@ -212,3 +331,11 @@ test "binary search: duplicates" {
         },
     );
 }
+
+test "binary search: random" {
+    var rng = std.rand.DefaultPrng.init(42);
+    var i: usize = 0;
+    while (i < 2048) : (i += 1) {
+        try test_binary_search.random_search(rng.random(), i);
+    }
+}

package/src/tigerbeetle/src/lsm/bloom_filter.zig

@@ -80,3 +80,46 @@ inline fn block_index(hash: u32, size: usize) u32 {
 test {
     _ = std.testing.refAllDecls(@This());
 }
+
+const test_bloom_filter = struct {
+    const fuzz = @import("../test/fuzz.zig");
+    const block_size = @import("../config.zig").block_size;
+
+    fn random_keys(random: std.rand.Random, iter: usize) !void {
+        const keys_count = @minimum(
+            @as(usize, 1E6),
+            fuzz.random_int_exponential(random, usize, iter),
+        );
+
+        const keys = try std.testing.allocator.alloc(u32, keys_count);
+        defer std.testing.allocator.free(keys);
+
+        for (keys) |*key| key.* = random.int(u32);
+
+        // `block_size` is currently the only size bloom_filter that we use.
+        const filter = try std.testing.allocator.alloc(u8, block_size);
+        std.mem.set(u8, filter, 0);
+        defer std.testing.allocator.free(filter);
+
+        for (keys) |key| {
+            add(Fingerprint.create(std.mem.asBytes(&key)), filter);
+        }
+        for (keys) |key| {
+            try std.testing.expect(may_contain(Fingerprint.create(std.mem.asBytes(&key)), filter));
+        }
+
+        // TODO Test the false positive rate:
+        // * Calculate the expected false positive rate
+        // * Test with a large number of random keys.
+        // * Use Chernoff bound or similar to determine a reasonable test cutoff.
+    }
+};
+
+test "bloom filter: random" {
+    var rng = std.rand.DefaultPrng.init(42);
+    const iterations_max: usize = (1 << 12);
+    var iterations: usize = 0;
+    while (iterations < iterations_max) : (iterations += 1) {
+        try test_bloom_filter.random_keys(rng.random(), iterations);
+    }
+}