flatsql 3.5.1 → 3.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -13,7 +13,9 @@ records the memory-safety and race work: the ps-wasm trap, the sanitizers and
13
13
  the first disk-full emergency. Part VII (§39) closes the query gaps T6 found
14
14
  on the host-02 fixture and accepts records stored with their own size
15
15
  prefix. Part VIII (§40) makes a partition's bookkeeping O(1) or O(log S) per
16
- commit (the terabyte audit's B4) and adds manifest version 3 (M3). The
16
+ commit (the terabyte audit's B4) and adds manifest version 3 (M3). Part IX
17
+ (§41) is TB03: format levels with a ratchet, and paged lane checkpoints that
18
+ lift the 1,170-lane cap (level 3). The
17
19
  design is
18
20
  the stack's `docs/architecture/flatsql-partition-store.md` (§22 overrides
19
21
  §0–§21; §22.4 holds the owner rulings). This file records what was built, how
@@ -47,6 +49,7 @@ The wasm artifact is T4; SDN and browser integration are T6/T10.
47
49
  ```
48
50
  <root>/fsql2/
49
51
  STORE MIGRATED registry.fsl registry.fsh
52
+ STORE.tmp only while a format ratchet runs (§41)
50
53
  j/<writer:02x>-{a,b}.fsj commit journal (only with commitJournal)
51
54
  p/<pid:08x>/
52
55
  h.fsh head, A/B 4 KiB slots
@@ -55,7 +58,8 @@ The wasm artifact is T4; SDN and browser integration are T6/T10.
55
58
  r-<seg>.fsr a-<seg>.fsa dense rows / attributes (merged ranges)
56
59
  x-<seg>-<gen:04x>.fsx L1 runs
57
60
  mf-<gen:06x>.fsm segment manifest
58
- l.fsl lane table
61
+ l.fsl lane table (tuples)
62
+ lk-<gen:06x>.fsl level 3: paged lane checkpoint (§41)
59
63
  t/<fid hex8>/
60
64
  h.fsh m-<seg>.fsl type head and log (A10 batches)
61
65
  g-<seg:06x>.fsg arrivals, 24 B entries (A15 segments)
@@ -103,7 +107,9 @@ mutation.
103
107
 
104
108
  ## 4. Open and recovery
105
109
 
106
- 1. `STORE` and `MIGRATED` (refused without the marker).
110
+ 1. `STORE` and `MIGRATED` (refused without the marker). A level above the
111
+ engine's `kFormatMax` is refused before any file is touched; a lower one
112
+ ratchets once the registry is non-empty (§41).
107
113
  2. Registry: frames scanned past the head, truncated at the first bad CRC;
108
114
  incarnation = max seen + 1, recorded durably before any batch.
109
115
  3. Journal replay (A8): both files of every writer id from 0 until the first
@@ -114,6 +120,8 @@ mutation.
114
120
  if replayed), fence index validation, fsync before adoption.
115
121
  5. Partitions: head slot, tail batches across SEAL continuations (same rules),
116
122
  fsync `d`/`m`, A11 intent cleanup (`UNLINK_IF_UNUSED`), DURABLE_CKPT head.
123
+ At level 3 a head that names a LaneRef folds the lane table from it, and
124
+ open removes a lane checkpoint no durable head names (§41).
117
125
  6. Type heads for adopted type tails, written after the partitions are
118
126
  attached (their inline labels come from them).
119
127
 
@@ -752,8 +760,9 @@ appended.
752
760
  without SWAP, and r/a files a first merge created without MERGE_DONE, then
753
761
  writes a durable head.
754
762
  - **Meta segments (A9).** A sealed `m-<seg>` whose batches are all merged is
755
- retired by a batch that also re-emits the full lane table (LANE_CKPT) and the
756
- RETIRE set, whose header carries the manifest generation, `merged_through`,
763
+ retired by a batch that also re-emits the full lane table (LANE_CKPT; at
764
+ level 3 past 32 lanes, a lane checkpoint must cover the segment first, §41)
765
+ and the RETIRE set, whose header carries the manifest generation, `merged_through`,
757
766
  `first_live_m_seg` and `next_gen`; `first_live_m_seg` advances when that
758
767
  batch commits. A rebuild of a head whose both slots are torn probes for the
759
768
  first existing meta segment and replays from its first batch.
@@ -2078,3 +2087,294 @@ Mutations the checks caught:
2078
2087
  - a SWAP input not touched;
2079
2088
  - a ledger update dropped;
2080
2089
  - the valve off.
2090
+
2091
+ # Part IX: format levels and lane checkpoints
2092
+
2093
+ ## 41. Lanes past 1,170 and format levels (TB03)
2094
+
2095
+ The terabyte design (stack `docs/architecture/flatsql-ps-terabyte.md` §2.9 K0,
2096
+ §3, §10 TB03) found that a partition stops committing past 1,170 live lanes.
2097
+ Past 32, every batch that changed a lane carried the whole live table (56 B a
2098
+ lane) in one LANE_CKPT ctl record. A u16 bounds that record, so a table past
2099
+ (65,535 − 4) / 56 = 1,170 lanes refused the batch, and every retry refused it
2100
+ too. SDN mints a lane per fetch, so a 3-hourly source whose history stays
2101
+ live crosses the cap in about 146 days. Below the cap each such batch still
2102
+ rewrote about 65 KB near 1,170 lanes. The RETIRE encoder also reserved room
2103
+ for every lane ever loaded.
2104
+
2105
+ Measured before the change on 2026-09-30:
2106
+ - host-02's legacy store (read-only, `sdn_record_source_summary`): the
2107
+ largest partition is MPE.fbs, celestrak-gp, at 66 live lanes. That is 5.6%
2108
+ of the cap. It grows about 4.3 lanes a day, so it would reach 1,170 in
2109
+ about 257 days. The next largest are RFB.fbs at 11 and OMM.fbs at 3.
2110
+ - The host-02 fixture migrated with SDN's store-migrate (4,364,873 records):
2111
+ OMM has 53 lanes, MPE 52, CAT 2 and IQC 1.
2112
+
2113
+ ### Format levels
2114
+
2115
+ `cpp/include/flatsql/ps/format_level.h` holds `kFormatMin` 2, `kFormatMax`
2116
+ 3 and `kLevelLaneCkpt` 3. `STORE.format` is the store's level.
2117
+ `HeadPrefix.format` and MIGRATED stay 2, because SDN's Go head reader checks
2118
+ them.
2119
+
2120
+ - **Accepted levels.** An engine opens `kFormatMin ≤ format ≤ kFormatMax`.
2121
+ A valid STORE above that is refused ("STORE format N is newer than this
2122
+ engine") before any file is created or written. So is a torn STORE whose
2123
+ STORE.tmp (below) is above it. An intact STORE the engine opens is never
2124
+ refused because of a higher STORE.tmp: that ratchet never reached step 2,
2125
+ so nothing of its level exists, and the leftover is removed. Readers apply
2126
+ the same rules: an intact STORE decides alone, and only a torn one reads
2127
+ through STORE.tmp.
2128
+ - **writeFormat** (writer TLV 31, `EngineConfig::writeFormat`; 0 is
2129
+ `kFormatMax`). A fresh store is created at it. At open, STORE ratchets up
2130
+ to it, but only once `registry.fsl` is non-empty: an empty store can still
2131
+ be recreated by an older engine. Nothing of a level is written before the
2132
+ ratchet to it is durable. A host can pin a lower level for a staged
2133
+ rollout, and a pinned host keeps its level: a STORE.tmp above the pin is
2134
+ removed, not finished. A store never ratchets down.
2135
+ - **A full device.** When STORE.tmp cannot be created, written or synced
2136
+ (step 1), the ratchet does not start: the leftover is removed, the store
2137
+ opens at its level exactly as the level-2 open would, and the next open
2138
+ tries again. A ratcheting engine on a full disk therefore never keeps a
2139
+ host down that the older engine would start.
2140
+ - **The ratchet.** The I/O ABI has no rename (deviation 39), so the ratchet
2141
+ takes three steps:
2142
+ 1. STORE.tmp (the new STORE) is written whole and synced; its directory
2143
+ entry is durable from creation.
2144
+ 2. STORE is rewritten in place (64 bytes, one sector) and synced.
2145
+ 3. STORE.tmp is unlinked.
2146
+
2147
+ A STORE.tmp is this store's when every field but the format equals
2148
+ STORE's. STORE is written once, and only the ratchet rewrites it, changing
2149
+ only its format, so a STORE.tmp an older engine left beside the store it
2150
+ went on writing still describes that store. An open finishes the ratchet
2151
+ from STORE.tmp when STORE is torn (STORE.tmp is the one valid record), or
2152
+ when STORE is intact and the open writes that level. Otherwise STORE.tmp is
2153
+ removed and STORE stands. Level-3 files exist only after step 2.
2154
+
2155
+ The released 3.5.1 engine reads STORE only, run natively on each crash
2156
+ state:
2157
+
2158
+ | State | 3.5.1 | Files |
2159
+ |---|---|---|
2160
+ | STORE 2, STORE.tmp 3 (step 1 done) | opens, writes level-2 data | changed |
2161
+ | STORE torn, STORE.tmp 3 (inside step 2) | "STORE is corrupt" | unchanged |
2162
+ | STORE 3, STORE.tmp 3 (step 2 done) | "STORE is corrupt" | unchanged |
2163
+ | STORE 3 (finished) | "STORE is corrupt" | unchanged |
2164
+
2165
+ A level-3 open afterwards recovered all four at level 3, with the lanes
2166
+ 3.5.1's own reopen reports. In the first state, 3.5.1 writes level-2 data
2167
+ into a level-2 store, which level 3 reads and converts.
2168
+ - **Rollback.** The ratchet is durable before the open reads a partition,
2169
+ so a first level-3 open that then fails for another reason (a tail scan,
2170
+ a manifest) also leaves the store at level 3, which 3.5.1 refuses. Rolling
2171
+ back after the first start restores a snapshot of the store taken before
2172
+ it, or re-syncs. SDN forwards TLV 31 as `SDN_F2_WRITE_FORMAT` for a host
2173
+ held at level 2.
2174
+ - **Stats** (appended, entries 36–40): the store's level, `kFormatMax`, the
2175
+ level this open raised STORE from (0: none; 1: STORE was torn inside a
2176
+ ratchet this open finished, so its old level is unknown), lane
2177
+ checkpoints cut, and their bytes.
2178
+
2179
+ At level 3, manifest version 3 is the written version. Below level 3, v3 is
2180
+ written only past 65,535 segments (§40, deviation 38).
2181
+
2182
+ ### Lanes at level 3
2183
+
2184
+ - **A batch carries only the lanes it changed.** That is its lane-delta
2185
+ section: one 48 B LaneDelta per changed lane, counted by a u32 (deviation
2186
+ 40: this section is the design's `LANE_PAGE`, so no new ctl kind was
2187
+ needed). The per-batch delta table is now indexed by lane id and has no
2188
+ cap. It was 256, so a kill of a record with more than 256 tag instances
2189
+ failed its batch for good.
2190
+ - **Up to 32 live lanes**, the head holds the table inline, as at level 2.
2191
+ - **The batch that takes a partition past 32** writes the table once as a
2192
+ LANE_CKPT. In inline mode, staging stops before an entry once the batch
2193
+ holds 64 lane deltas. An entry makes at most one lane live (a PUT or a
2194
+ re-tag), so that table holds at most the live lanes plus 65, 97 lanes.
2195
+ - **From that batch on, the head names a LaneRef** (`nLanes` 0xFFFE,
2196
+ 48 bytes after the L0 directory): a base plus the batch position its
2197
+ deltas replay from. The base is `lk-<gen>.fsl`, or the crossing
2198
+ LANE_CKPT. The table is the base, plus the deltas of every batch from the
2199
+ replay offset up to the head's `(mSeg, mEnd)`. Batches are walked by
2200
+ their headers, and a SEAL moves to segment + 1.
2201
+ - **`lk-<gen:06x>.fsl`** is a 4 KiB header page (pid, gen, lane count,
2202
+ `mintedThrough`, the cut position, CRC). Then come pages of up to 73
2203
+ counters sorted by lane id, each page with a CRC. Count-0 lanes are
2204
+ dropped. There is no size cap.
2205
+ - **Cuts.** A checkpoint is cut:
2206
+ - after the crossing;
2207
+ - when A9 wants to retire a meta segment at or above the ref's floor (its
2208
+ replay segment, and a LANE_CKPT base's segment);
2209
+ - once a fold would walk `laneCkptBatches` batches with lane deltas
2210
+ (writer TLV 32, default 128), or four times that many batches in all.
2211
+ A fold reads every batch it walks, and every `flatsql_lanes` statement
2212
+ and every open folds, so this bounds their work. Batches without lane
2213
+ deltas count a quarter, so the RETIRE and UNLINKED batches a cut itself
2214
+ causes never make the next cut due.
2215
+
2216
+ The owner copies its live counters at a batch boundary (O(live lanes),
2217
+ once per cut), and a maintenance helper writes and syncs the file
2218
+ (deviation 41). The encoded file counts in the writer's committed memory
2219
+ until it is written.
2220
+ - **Crash protocol**, as built:
2221
+ 1. The helper writes `lk-<g>` and syncs it. One cut is in flight per
2222
+ partition, so g is always the ref's gen + 1.
2223
+ 2. The owner installs the ref and writes a DURABLE_CKPT head naming it.
2224
+ 3. Only once a durable head names it does a RETIRE list `lk-<g−1>`
2225
+ (letter `L`), and A9 retire a meta segment below the new floor.
2226
+ 4. Open unlinks `lk-<gen + 1>`, a checkpoint no durable head names. An
2227
+ `lk-<gen − 1>` whose RETIRE never committed goes after the open's
2228
+ durable head.
2229
+
2230
+ No cut starts while a RETIRE that lists a replaced checkpoint has not
2231
+ committed. Until it commits only the head names what open cleans up, so
2232
+ the files on disk stay within `lk-<gen − 1>` to `lk-<gen + 1>`. Without
2233
+ that wait, meta writes failing ENOSPC let the next cut install and a crash
2234
+ left `lk-<gen − 2>` named by nothing, for good. Open also probes
2235
+ `lk-<gen − 2>`.
2236
+ - **Torn heads.** Every batch that carries a RETIRE also carries the LaneRef
2237
+ in force (ctl kind 13, LANE_REF). A torn-head rebuild from a later meta
2238
+ segment folds the table from the last one.
2239
+ - **The RETIRE valve** reserves only what the batch's finalize step adds:
2240
+ the LANE_REF in ref mode; in inline mode the table a crossing can write,
2241
+ the live lanes plus 65; and the whole table at level 2, as before. With 32
2242
+ or fewer live lanes the soft cap stays at 2,048 items, as at level 2.
2243
+ - **Readers** fold the same way (`LaneStore::laneCounters`) up to their
2244
+ snapshot's head. A file retired since then answers SNAPSHOT_GONE.
2245
+ - **A level-2 head at a level-3 open.** A head that names a LANE_CKPT
2246
+ becomes a LaneRef with that record as its base and the head position as
2247
+ its replay offset. This holds because level 2 wrote a LANE_CKPT with every
2248
+ batch that changed a lane past 32. A first checkpoint follows.
2249
+ - **In memory**, lanes are found by id through an index. It was a linear
2250
+ search per delta at publish, replay and RECONCILE. Replay, readers and
2251
+ checkpoints apply a delta by one rule (`laneApplyDelta`). A lane at count
2252
+ 0 carries no history: checkpoints and head tables drop it, so a delta that
2253
+ makes it live again starts its counters afresh (`first_seen`, `max_pseq`,
2254
+ `updated`). The writer's memory, a fold from any base and a replay then
2255
+ agree whether or not a checkpoint dropped the lane. Before, a lane revived
2256
+ after a checkpoint dropped it read its old `first_seen` until the next cut
2257
+ and the new one after it, with no write in between.
2258
+ - **A failed batch that minted lanes** rolls `nextLaneId` back to the first
2259
+ id it minted. The table is not in id order when l.fsl lost frames the
2260
+ head still counts, and the old rule, the last lane's id + 1, could fall
2261
+ below a live id.
2262
+ - **A LaneRef head in a store below level 3** (a partial snapshot restore)
2263
+ quarantines that partition. It used to fail the whole open.
2264
+
2265
+ ### Live-only candidate caps (N2)
2266
+
2267
+ Staging gathered at most a fixed number of postings per lookup and then
2268
+ dropped the dead ones. A key with more dead postings than the cap hid its
2269
+ live row:
2270
+
2271
+ | Lookup | Old cap |
2272
+ |---|---|
2273
+ | supersede candidates | 8 |
2274
+ | CID candidates | 64 |
2275
+ | tag instances of one tuple | 64 |
2276
+ | live instances of a PUT | 256 |
2277
+ | licence and control supersedes | 8 |
2278
+
2279
+ Only live rows count now. A key holds at most one live row of each kind:
2280
+ every PUT kills the live PUTs of its supersede keys whose CID differs, every
2281
+ control record every live row of its key, every licence the live licences
2282
+ of its key, and a CID is stored once per partition. A live control record
2283
+ is also older than any live PUT beside it. So a supersede or licence lookup
2284
+ ends at the key's newest live row, and a control record's at the live
2285
+ control record.
2286
+
2287
+ The walk goes newest first through the postings' sources. The unmerged L0
2288
+ blocks and each segment's L1 runs hold disjoint, ascending pseq ranges,
2289
+ with every run below every block, so the sources are read from the newest,
2290
+ each one's postings sorted descending, and the walk stops at the end
2291
+ without reading older sources. Should the ranges not be in that order, all
2292
+ postings are gathered and sorted instead. A PUT of a hot key reads its
2293
+ newest version, not every dead version compaction has not dropped yet.
2294
+ CID and tag-instance lookups gather every posting and stop at the first
2295
+ live one. The lists grow outside the hot-path count.
2296
+
2297
+ ### Tests
2298
+
2299
+ The tests are in `cpp/test/ps/lane_cap_test.cpp`. The Mac is an M3 Ultra
2300
+ with 28 cores, APFS, and the in-memory fault host, shared with other lanes
2301
+ (1-minute load 9–20 during the runs). Every crash sweep and trial checks,
2302
+ after each reopen, the lane table on disk against a recount from rows
2303
+ (count and bytes), and against the engine's own table in every counter
2304
+ field, for every live lane.
2305
+
2306
+ | Test | Result |
2307
+ |---|---|
2308
+ | `lanecap_5000_live_lanes` | 5,000 lanes, then 500 re-tags into new lanes and 200 kills, all acked. The engine, `flatsql_lanes`, the on-disk fold and a recount from rows all agree (5,272 lanes); again after a reopen and 100 more lanes |
2309
+ | `lanecap_crash_trials_full` | 1,000 trials, 2 writers, helper-written checkpoints, file and journal mode, 114 s. Each crash lands after at least 5,000 live lanes: 160 left an unnamed checkpoint and 362 a replaced one pending. After every reopen the table equals a recount from rows and the engine's table, and no `lk` orphan is left. 0 failures |
2310
+ | `lanecap_ckpt_crash_every_op_full` | A crash at each of 2,912 mutating I/O calls of a workload that crosses 32 lanes, cuts 30 checkpoints, seals, and retires meta segments and replaced checkpoints. All five crash modes. Before open: 965 crashes left an unnamed checkpoint and 1,241 a replaced one pending. After open: lanes equal a recount and the engine's table, and the directory holds exactly the checkpoint the head names. 0 failures |
2311
+ | `lanecap_2000_lanes_retire_bearing` | 2,000 lanes with seals, merges, a coalescing SWAP, 97 files retired, 5 meta segments retired, 8 checkpoints: 0 batches refused, every lane correct |
2312
+ | `lanecap_batch_lane_bytes_proportional` | A batch that changes one lane writes 56 B of lane records at level 3, with 66 live lanes (host-02's largest partition) or 5,000. At level 2 the same batch writes 3,760 B at 66 lanes and 61,664 B at 1,100. At level 2, 1,180 lanes never ack |
2313
+ | `lanecap_no_cut_while_lk_retire_pending` | Meta writes fail ENOSPC for 300 steps: no cut starts while a RETIRE naming a replaced checkpoint waits (355 steps waited), and a crash then leaves exactly the named checkpoint after open. With the wait removed, 82 steps had a cut in flight |
2314
+ | `lanecap_install_head_nospace` | The head write naming a newly installed checkpoint fails ENOSPC (40 installs), then a crash in each of the five modes at four points after: every reopen agrees with the recount and the engine and keeps only the named checkpoint. 0 failures of 20 trials. Removing the replaced checkpoint's wait for a durable head is not caught here: this workload's checkpoint heads follow in the same step, and every RETIRE batch also carries the LaneRef in force |
2315
+ | `lanecap_fold_walk_bounded` | 1,500 one-lane batches on 100 lanes at the default cadence: a fold walks at most 128 batches (13 checkpoints); `flatsql_lanes` 0.3–1.0 ms |
2316
+ | `lanecap_revived_lane_counters_stable` | A lane killed to 0, dropped by a checkpoint and written again reads the same `first_seen` (the revival's) before the next cut, after it and after a reopen. Without the restart rule it flipped at the cut |
2317
+ | `lanecap_rollback_keeps_lane_ids_unique` | l.fsl loses 6 frames (the table is out of id order), a batch that minted a lane fails ENOSPC and retries: every lane id unique, the new lane's id past every live one. The old rule fails it |
2318
+ | `tb_supersede_candidate_cap`, `_autocompact` | 1 live version; 381 of 381 live. The old cap gives 12, and 392 of 381 |
2319
+ | `tb_supersede_hot_key_cost` | A CAT key with 400 versions among 19 static objects each, default compaction: its PUT costs 1.8x a new key's (618 vs 340 µs p50); 2.3x at 800 versions. Checking every dead version (the review's measurement: 12.8x at 800) fails it at 7.3x. What remains is the kill's TAG_OF prefix scan across every run, the same as 3.5.1's (TB12's prefix bloom) |
2320
+ | `lanecap_kill_record_with_300_instances` | A kill takes all 300 instances and lanes in one batch |
2321
+ | `lanecap_cid_candidates_past_64` | The 80th copy of a CID dedupes |
2322
+ | `lanecap_torn_heads_rebuild_from_lane_ref` | Both head slots torn: the table is rebuilt from LANE_REF |
2323
+ | `format_older_engine_refuses_ratcheted_store` | An engine built with `kFormatMax` 2 refuses a level-3 store, and this engine refuses a level-4 STORE; neither creates or changes a file |
2324
+ | `format_empty_registry_never_ratchets` | Opened three times, the empty level-2 store stays at 2; with a registration, the next open ratchets to 3 |
2325
+ | `format_ratchet_crash_every_io_call` | A crash at each of the ratcheting open's 13 mutating calls, in all five modes (65 trials). An engine built with `kFormatMax` 2 (this code with `testFormatMax` 2, not the released 3.5.1: see the table above) then opened the store 31 times and refused it 34 times, creating nothing when it refused. This engine then opens at level 3 with every record |
2326
+ | `format_ratchet_leftover_tmp_rules` | A STORE.tmp at 3 beside an intact STORE at 2: removed by a host pinned at 2 and by an engine whose `kFormatMax` is 2, which open at 2; finished by this engine (stats entry 38 says 2). One that differs from STORE in more than the format: removed. Beside a torn STORE: refused by the `kFormatMax`-2 engine with no file changed, finished by this one even pinned (entry 38 says 1) |
2327
+ | `format_ratchet_waits_on_a_full_device` | No room for STORE.tmp: the store opens at 2, as the level-2 open does, and the next open with room raises it |
2328
+ | `format_reader_store_rules` | A reader refuses a STORE at 4 beside a STORE.tmp at 3, reads a torn STORE through a STORE.tmp at 3, and refuses one through a STORE.tmp at 4 |
2329
+ | `format_fresh_store_at_write_format`, `format_manifest_v3_from_level_3`, `format_manifest_v3_on_swap` | Fresh stores at writeFormat. Manifests v3 from level 3 and v2 at level 2, from a merge and from a compaction SWAP |
2330
+
2331
+ Mutations the checks catch: the old supersede cap (12 live, and 392 of
2332
+ 381); checking every dead version (7.3x); no orphan cleanup at open; a cut
2333
+ while a replaced checkpoint's RETIRE waits; no restart of a revived lane;
2334
+ the old rollback rule; the old STORE.tmp rules in the writer and in
2335
+ readers; a ratchet that fails the open on a full device; the old cadence
2336
+ default.
2337
+
2338
+ The native suite (118 fast tests), `crash_faults_T1_1_full` (10,000
2339
+ trials), `orphan_crash_points_during_compaction_T3_4_full` (2,501 trials),
2340
+ `fanout_randomized_vs_bruteforce_T2_2_full` (T2 #2) and
2341
+ `book_crash_every_op_full` (9,389 crash points) pass, 0 failures. In wasm32
2342
+ (wasi-sdk 30, the Node host, `scripts/ps-wasm-suite.mjs`) all 118 fast
2343
+ tests and `crash_faults_T1_1` with 1,000 trials pass. ThreadSanitizer
2344
+ reports nothing on the threaded `lanecap_` and `format_` tests, the
2345
+ supersede autocompaction test and the SWAP manifest test.
2346
+
2347
+ Deviations:
2348
+
2349
+ 39. **The ratchet uses STORE.tmp and an in-place rewrite** (the design says
2350
+ an atomic rename). The host I/O contract has seven imports and no
2351
+ rename, and adding one would change every host. The three steps above
2352
+ leave the old STORE, the new one, or a torn one that STORE.tmp repairs.
2353
+ An engine that predates levels never misreads a ratcheted store. SDN
2354
+ reads a torn STORE through STORE.tmp too (its `Migrated` and the update
2355
+ guard), so neither the daemon's start nor the Apply/Rollback guard
2356
+ stops short of the engine's repair.
2357
+ 40. **No LANE_PAGE ctl kind.** The batch's lane-delta section already holds
2358
+ exactly the changed lanes, and a u32 counts it.
2359
+ 41. **The owner cuts the checkpoint from its in-memory table** (the design:
2360
+ a helper streams the previous checkpoint plus the deltas since). The
2361
+ writer holds every lane's counters until TB15b removes them. The copy
2362
+ costs O(live lanes) once per cut (the review measured 0.055 ms at 5,000
2363
+ lanes, 1.5 ms at 100,000 and 14 ms at 1,000,000), and the helper still
2364
+ writes and syncs the file; the encoded bytes count in the writer's
2365
+ committed memory until then. The copy equals a fold of the checkpoint
2366
+ it replaces plus the deltas since because a lane at count 0 restarts
2367
+ (above). TB15b switches the source when the writer stops holding the
2368
+ table, before a partition can hold 10^5 lanes.
2369
+
2370
+ Not built here:
2371
+ - A reader cache of folded tables. A reader folds at most 128 batches with
2372
+ lane deltas (512 in all) per partition per `flatsql_lanes` statement.
2373
+ - `flatsql_lanes` still matches lanes to tuples in O(lanes²)
2374
+ (`vtab_meta.cpp` `fillLanes`, the fan-out estimate): 15 ms per statement
2375
+ at 5,000 lanes, 0.1 ms at 100. That is TB15a's (pushdown and a hash from
2376
+ lane id to tuple); today's largest partition holds 66 lanes.
2377
+ - The LANE kind, the lane LRU and `flatsql_lanes` pushdown (TB15a, TB15b).
2378
+ - A kill's TAG_OF prefix scan still reads every run of the partition (the
2379
+ cost left in `tb_supersede_hot_key_cost`); TB12's prefix bloom removes
2380
+ it.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "flatsql",
3
- "version": "3.5.1",
3
+ "version": "3.6.0",
4
4
  "description": "SQL queries over raw FlatBuffer storage: the SQLite-backed wasm engine and the multithreaded partition store (flatsql-ps-threads.wasm)",
5
5
  "type": "module",
6
6
  "main": "wasm/index.js",
Binary file
@@ -5,9 +5,9 @@
5
5
  "size": 2246620,
6
6
  "files": {
7
7
  "flatsql-ps-threads.wasm": {
8
- "sha256": "3a215da45a53f3a7016257429c392720e728caf850d3a1213dc501bfb5844359",
9
- "sri": "sha384-HJK5ECR2v2kAKnOHsyvXE+QgSywyuEUa2ipQQMhxR22dg7SZPAYPY985iyy2O7FP",
10
- "size": 2485719
8
+ "sha256": "87ea0c727eb3c0b889a2d3fb41f8dac631ec2af07f521fe9526556d094d2e119",
9
+ "sri": "sha384-RUEUl560TtNBLtx6OauGyCwNdLJNbYfXPDEKVgV1M5au/stPrlxS+r9hlWxNprWI",
10
+ "size": 2629364
11
11
  },
12
12
  "flatsql-wasi-noeh.wasm": {
13
13
  "sha256": "8f11fd49ee2e6961b9c1c22dd891d5a6645d0faf481f815408c0f1f5d3a385ab",