flatsql 3.0.0 → 3.2.0

package/docs/PARTITION-STORE-WASM.md

@@ -109,14 +109,15 @@ FLATSQL_PS_WASMEDGE=1 npm test -- test/ps-parity-vectors.test.ts   # + the C run
 
 Machine: the owner's Mac Studio (Darwin 25.3.0, arm64, 28 hardware threads,
 APFS), Node 25.4.0, shared with other lanes: 1-minute load average 18–25
-during these runs. Native builds RelWithDebInfo; wasm builds as §1.
+during these runs (37–41 for the release rerun, §4.2). Native builds
+RelWithDebInfo; wasm builds as §1.
 
 | # | Acceptance | Result |
 |---|---|---|
 | 1 | Imports exactly WASI p1 + `wasi.thread-spawn` + shared `env.memory` (≤ 32768 pages) + the 7 `env.flatsql_io_*`; exports `wasi_thread_start`; 0 emscripten imports; passes `assertPthreadArtifact` | **Pass.** §1 table; `scripts/check-wasm-imports.mjs`, `test/standalone-wasm-imports.test.ts` (the SDK guard runs in a child process). |
 | 2 | The T1 and T2 suites, including 1,000 fault crashes, pass under the Node host with distinct OS thread ids ≥ writers + lanes | **Pass.** `scripts/ps-wasm-suite.mjs`: all 53 default-suite tests, each in its own guest process, plus `crash_faults_T1_1 --crash-trials=1000` (1,000 trials over the five modes, 298,923 acked records verified, 41.8 s). `readers_under_saturating_writers_T2_1` (8 writers, 16 lanes) ran 54 guest threads at once on 54 distinct worker (OS) threads. `test/ps-node-threads.test.ts`: the shipped artifact as a writer instance (4 writers) and a reader instance (4 lanes), each running thread on its own worker. `writer_backpressure_flood_T1_5` runs with `--bp-samples=40` (deviation 3). `crash_faults_T1_1_full` (10,000 trials, 64 types, 256 partitions): §4.1. |
 | 3 | Parity: canonical dumps (by (pid, pseq)) and query results identical native vs Node; deterministic mode byte-identical across native, Node and the SDK WasmEdge C runner (patched 0.16.4) | **Pass.** Canonical dump, 4 writers, 11,313 lines (rows, counters, lane counters and six queries, partition and type level): sha256 `f3b419a2…` natively and under Node, three runs each. Deterministic mode, 122 files: in memory `e7ab8592…` natively, under Node and under the C runner (18.7 s, interpreted); on the host's files `c7e34c55…` natively and under Node. `test/ps-parity-vectors.test.ts`. |
-| 4 | A published npm version with provenance; the wasm sha256 in `integrity.json`; `npm ls sql.js` empty; the package root exports only the wasm bindings | See §6 and the release record in §4.2. |
+| 4 | A published npm version with provenance; the wasm sha256 in `integrity.json`; `npm ls sql.js` empty; the package root exports only the wasm bindings | **Pass.** `flatsql@3.0.0`, §4.2 and §6. |
 
 Host-level crashes (the overlay, §2): `test/ps-host-fault.test.ts` kills the
 guest mid-ingest at a random 0.4–2.0 s, crashes the store in the three modes
@@ -140,7 +141,27 @@ journal. A negative control (half a data segment cut off) fails the verifier.
 
 ### 4.2 Release
 
-Filled in when the release is published.
+- `flatsql@3.0.0`, published by `npm-publish.yml` from tag `v3.0.0` =
+  `79fd694` (run 36486543061) with a signed provenance statement
+  (`npm audit signatures`: verified attestation); `gitHead` 79fd694, dist-tag
+  `latest`, `dependencies` `{}`.
+- `wasm/flatsql-ps-threads.wasm`: 2,074,707 bytes, sha256
+  `572a4bc38345d07ff91c345fc8d81e28873d72efa0fb8864a2edc5242145f9f5`, the same
+  in `wasm/integrity.json`, from the Linux arm64 Docker build, the x86_64
+  publish runner and the installed tarball.
+- A fresh `npm install flatsql@3.0.0`: `npm ls sql.js` is empty; the root's
+  exports equal `flatsql/wasm`'s; `loadFlatSQLPsThreads()` checks the bytes
+  against `integrity.json`.
+- At the release sources (`dac8d89` engine), under the Node host: the 55
+  default-suite tests and `crash_faults_T1_1 --crash-trials=1000` pass (1,000
+  trials, 403,798 acked records verified, 47.3 s; load 37–41). Main CI green
+  on `79fd694` (run 36482613908).
+- Two engine defects the Node host found before the release, fixed in the
+  sources the artifact builds: the open path dropped the lane checkpoint
+  pointer of a head with more than 32 live lanes (`5c7e769`; the 1,000-trial
+  crash run, trial 317), and the type owner never merged its L0 directory
+  when `mergeL0Blocks` exceeded the directory cap, so labeling, partition
+  merges and acks stopped (`dac8d89`).
 
 ## 5. Design deviations
 

package/docs/PARTITION-STORE.md

@@ -4,13 +4,16 @@ The engine of the partition store program. Part I (§1–§11) is T1: append-onl
 partition logs, one writer per partition on a pinned pool, durable acks,
 per-commit indexes, type owners, and durable-tail open. Part II (§12–§21) is
 T2: reader instances and lanes, the snapshot protocol, the SQL surface, the
-fan-out merge, admission and results. The design is the stack's
-`docs/architecture/flatsql-partition-store.md` (§22 overrides §0–§21; §22.4
-holds the owner rulings). This file records what was built, how to run it, the
-measured acceptance, and every place the build departs from the design.
+fan-out merge, admission and results. Part III (§23–§30) is T3: compaction,
+reclamation, meta-segment retirement, disk accounting and quota. Part IV
+(§31–) is T3b: store-migrate gseqs, the entry-size setting, hot-partition
+splitting and arrivals compaction. The design is
+the stack's `docs/architecture/flatsql-partition-store.md` (§22 overrides
+§0–§21; §22.4 holds the owner rulings). This file records what was built, how
+to run it, the measured acceptance, and every place the build departs from the
+design.
 
-Compaction, splits and quota are T3; the wasm artifact is T4; SDN and browser
-integration are T6/T10.
+The wasm artifact is T4; SDN and browser integration are T6/T10.
 
 ## 1. Code map
 
@@ -633,3 +636,445 @@ engine. If the bound fails there, the next steps are the design's own levers:
 fewer writer threads than cores (design §5.1: `clamp(cores−2, 1, 16)`), interactive
 lanes at a higher priority than writers (design A28 gives nice −5 only on ≤ 2 vCPU),
 and idle-lane sub-cursors (§21 deviation 1).
+
+# Part III: compaction, reclamation and quota (T3)
+
+## 23. Code map
+
+| Path | Contents |
+|---|---|
+| `ps/compaction.h`, `src/ps/compaction.cpp` | Manifest version 2, the compacted rows format, retire items; the compaction pipeline (plan, build, SWAP, apply, abort). |
+| `src/ps/reclaim.cpp` | The per-partition file ledger (`disk_bytes`), the RETIRE and UNLINKED records, reader-gated unlinking, meta-segment retirement (A9), open-time cleanup; the same for the type logs (meta log rotation and retirement, retired catalog runs and manifests, the type ledger). |
+| `ps/quota.h`, `src/ps/quota.cpp` | The quota planner, the space emergency, the ballast. |
+| `src/ps/sources-t3.cmake` | Registers the T3 sources and tests on the targets `cpp/CMakeLists.txt` defines. |
+| `cpp/test/ps/{compaction,orphan,quota}_test.cpp` | T3 tests. `checkPartitionDir` and `checkTypeDir` (fixtures) walk a partition's or a type's directory against the files its head and manifest name. |
+
+## 24. Compaction (§11, A11, minor 1)
+
+A compaction rewrites a run of sealed, fully merged segments `[seg, segEnd]`
+into one new file set, `c-<seg>-<gen>.{fsd,fsr,fsa}` plus
+`x-<seg>-<gen>.fsx` and `mf-<gen>.fsm`:
+
+1. The owner plans. `INTENT_COMPACT{seg, segEnd, gen}` rides the next batch; no
+   output exists before it is durable. The head carries the intent.
+2. A compaction thread builds. It reads the DEAD and TAG_DEAD postings over the
+   inputs' pseq range from every run and L0 block at or after the inputs, keeps
+   a killer only when it is at or below the plan's kill bound (the type owner's
+   `labeled_through`), and decides per row:
+   - PUT, LICENCE and CTL rows with a killer go;
+   - RETAG rows go when their instance is TAG_DEAD, or their PUT is dead or gone;
+   - TOMB, CTL_TOMB and TAG_TOMB rows go only when an earlier SWAP removed their
+     target (minor 1): outside the inputs, or VOID within them.
+
+   It copies the surviving frames, attributes and rows in 4 MiB slices, filters
+   the inputs' L1 runs down to the surviving rows' postings (every partition
+   posting's value is the pseq of the row that emitted it), fsyncs everything,
+   and re-checks the owner word and an abort flag before every mutation (A26).
+3. The owner, once no merge is in flight, writes the manifest (its fsync joins
+   the next round's first sync phase) and queues `SWAP{seg, gen, oldCgen,
+   segEnd}`; the SWAP, the RETIRE set and the counters' adjustment ride one
+   batch. Publish attaches the new files.
+
+Rows keep their pseqs. `c-*.fsr` holds the surviving rows (dense by rank) and a
+presence directory (a bit per pseq, a rank per 512); a pseq the directory
+lacks reads as a `VOID` row (kind 8, never stored, never live). A segment with
+no survivor is `EMPTY` in the manifest and has no files.
+
+Candidates (maintenance, every 50 ms per warm partition, and at once after a
+SWAP): the sealed segment with the largest dead share at or over
+`compactDeadRatio` (15%; dead frame bytes counted as kills commit), joined by
+its adjacent segments past the ratio too (one output per wave, within the
+coalescing limits); adjacent sealed segments under 8 MiB, coalesced up to 16
+at a time and 64 MiB of output into one segment named by the first (`lastSeg`
+in the manifest; rows' `seg` is rewritten); after a restart, when the
+partition's dead share is over the ratio and the per-segment counts are
+unknown, the oldest
+segment not yet surveyed (a build that would save under 10% is abandoned and
+its count remembered). `Engine::swapSegment` requests one (targeted, or the
+oldest-generation compactable segment).
+
+Kill bound and readers. A type-level reader's visibility is `V_p =
+min(pseq_hi, labeled_through[p])`. A compacted segment records `killThrough`
+(the largest killer behind a removed row, the inputs' own, and the bound of
+the SWAP that removed a dropped tombstone's target) and `prevGen` (the manifest
+before its SWAP). A type-level snapshot whose `V_p` is below a segment's
+`killThrough` reads that segment through `prevGen`, repeatedly for a segment
+compacted more than once; the reader gate keeps those files for any statement
+older than the SWAP. Partition-level readers see `pseq_hi`, which is never
+below a SWAP's bound.
+
+Counters: `total_count`, `total_bytes` and `tomb_count` count the rows the
+files hold (a SWAP subtracts what it removed), so they equal a recount;
+`min_epoch`, `max_epoch` and `latest_arrival` stay bounds over everything ever
+appended.
+
+## 25. Reclamation (A12) and meta segments (A9)
+
+- **Retire.** A MERGE_DONE retires the folded runs and the manifest it
+  replaces; a SWAP retires the inputs' files, their runs and the manifest. The
+  whole outstanding set is a `RETIRE` ctl record in the same batch (the head
+  points at it: `retireSeg/retireOff/retireN`); consecutive manifests collapse
+  into one range item.
+- **Unlink.** Maintenance unlinks a retired file with `UNLINK_IF_UNUSED` once
+  the reader gate (the start of the oldest running reader statement) is past
+  its retirement time, checked twice a grace apart (60 s by
+  default). The retirement time is that of the first head write after the
+  batch commits (the head that stops naming the file): a statement that
+  starts earlier may read the previous head. In journal mode also once every
+  journal record older than the
+  retirement is checkpointed; a meta segment once a DURABLE_CKPT head names a
+  later `first_live_m_seg` (a quiet partition writes one; later items do not
+  wait for it, and a partition with retired files never goes cold). Each pass
+  takes the first check of every file past the gate, so a backlog goes about
+  one grace after its statement ends (`reclaim_backlog_goes_in_one_grace`:
+  133 files in 0.31 s at a 0.3 s grace). BUSY (a handle is open) skips to
+  later items and retries after 50 ms; idle
+  reader lanes close cached handles after 5 s. The next batch carries
+  `UNLINKED` and the shrunken set; `disk_bytes` drops then. A set past 2,048
+  items unlinks its oldest regardless of the gate (a statement older than
+  hours of retirements gets the retryable SNAPSHOT_GONE instead of a writer
+  waiting on it).
+- **Open** unlinks every file of the persisted set (a file a surviving reader
+  instance still holds stays retired), the outputs of an INTENT_COMPACT
+  without SWAP, and r/a files a first merge created without MERGE_DONE, then
+  writes a durable head.
+- **Meta segments (A9).** A sealed `m-<seg>` whose batches are all merged is
+  retired by a batch that also re-emits the full lane table (LANE_CKPT) and the
+  RETIRE set, whose header carries the manifest generation, `merged_through`,
+  `first_live_m_seg` and `next_gen`; `first_live_m_seg` advances when that
+  batch commits. A rebuild of a head whose both slots are torn probes for the
+  first existing meta segment and replays from its first batch.
+- **Type logs.** A type's MERGE_DONE retires the catalog runs it folds and
+  the manifest it replaces. The new manifest persists the whole outstanding
+  set (these and any still waiting) in an appendix after its CRC trailer,
+  `[magic FSTR][n][n x RetireItem][crc][pad]`, which readers that predate it
+  never read. Unlinking follows the partition rules (reader gate twice a
+  grace apart, journal safe time, BUSY retried after 50 ms). Open unlinks the
+  appendix's files, the outputs of a merge a crash cut short (it sweeps the
+  generations after the manifest's to `next_gen + 64`, and 16 past the last
+  leftover it finds, since a head's `next_gen` can lag), and anything below
+  `first_live_m_seg`. A failed merge unlinks its outputs at once (durably;
+  one that cannot go yet is retired like the rest) and retries with the same
+  generation. At stop a merge in flight closes its handles; an unfinished
+  build's outputs go, a finished one's are left to open (a MERGE_DONE already
+  in the log may name them).
+- **Type meta segments.** The type meta log starts `m-<seg+1>` once it passes
+  `typeMetaSegBytes` (64 MiB): the sealed segment is cut at the end of its
+  last batch and synced, the new one created, and a durable head names it.
+  Open follows the batch chain from a segment's end into the next one, so a
+  stale head slot loses nothing. With more than 128 partitions the new
+  segment's first batch is a FULL_LABELS checkpoint, and readers read label
+  deltas across segments. The oldest sealed segment retires once no unmerged
+  L0 block, merge input or label checkpoint lives in it: `first_live_m_seg`
+  (a type head field) is written to both head slots and synced, then the file
+  waits for readers like any retired file.
+- **Dead copies leave the catalog (A15, catalog half).** A type merge drops the
+  CID, LABEL and REPEAT entries of every copy killed within its inputs whose
+  first posting (FIRST or REPEAT) is also among them: a copy never leaves
+  DEAD, and no older run holds any of its postings. A full fold drops every
+  dead copy. GONE and REHOME entries stay: they are keyed by gseq for
+  arrivals order, and arrivals are not compacted yet (§30).
+- **Reader gate in production.** The writer is its own wasm instance: the host
+  reads every reader instance's lane announcements
+  (`FlatsqlPsReaderLayout.laneAnnounce`) and passes the minimum through
+  `flatsql_ps_reader_gate` (T6 wires it). Natively, `Engine::setReaderGate`.
+
+## 26. Disk bytes (§13)
+
+`disk_bytes` is the exact size of the files a partition names: a ledger of the
+stable files (sealed segments' d and m, r/a, c-*, runs, manifests, the next
+segment's pre-created files, retired files until UNLINKED) plus the extents of
+the files still growing (h, l, the active d and m, zero-fill included). Open
+rebuilds it from the head, the manifest and file sizes; the owner maintains it
+on every append, merge, seal, SWAP and unlink. After every reopen in the crash
+tests, a walk of each partition directory finds exactly the named files and a
+total equal to `disk_bytes`. Head files are written as whole slots until they
+hold both, so a head write never grows the file again.
+
+A type's disk bytes are kept the same way: its head, meta segments from
+`first_live_m_seg`, arrivals segments and fence, catalog runs, manifest, a
+built merge's outputs until MERGE_DONE, and retired files until they are
+unlinked, maintained on every write, seal, rotation, merge and unlink and
+rebuilt at open from file sizes. Type configs (`s-*.fsc`) are registration
+and not counted. The crash tests check the type directory the same way.
+
+## 27. Quota and the full disk (§13, A13, §22.4-3)
+
+- **Planner** (writer 0's maintenance, every 100 ms): usage = Σ partition
+  `disk_bytes` + Σ type log bytes, less retired files waiting only for
+  readers. Over the cap
+  (`EngineConfig::quotaBytes`, `Engine::setQuota`, `flatsql_ps_set_quota`), it
+  evicts whole sealed segments in arrival order across partitions (their
+  `minArrival` zone) down to 0.85 of the cap: a `TOMB_RANGE{seg, all}` on the
+  owner (bounded 512-row steps; supersede-lane heads and control kinds are
+  spared; each step also stops after 4 ms of CPU), then, once the tombstones
+  are durable, a compaction of the segment. One wave at a time.
+- **ENOSPC.** A commit that fails NOSPACE starts an emergency: record entries
+  stop being consumed (producers see zero credits; nothing is acked), the
+  ballast file (`fsql2/ballast`, `ballastBytes`) is released, and waves evict one
+  segment at a time down to 0.85 of the usage at the failure, and at least the
+  ballast's size, but never more than half the store in one episode. Once
+  usage is under that cap the ballast is recreated and ingest resumes. NOSPACE
+  in a merge, a compaction, a head write or a segment creation is transient;
+  a merge that failed retries after 100 ms (a type merge with the same
+  generation when it left nothing behind). A partition whose L0 directory is
+  full stages no rows at all (entries, mailbox kills, TOMB_RANGE steps) until
+  a merge frees a slot.
+
+## 28. Acceptance (§18 T3 as amended)
+
+Machines: **Mac** = the owner's Mac Studio (Darwin 25.3.0, arm64, 28 hardware
+threads), shared with other lanes (1-minute load recorded per run). **Mac mem**
+runs use the in-memory fault host (no fsync). Numbers come from `MEASURED`
+lines of `flatsql_ps_test` (RelWithDebInfo) on the task-branch build landed on
+2026-09-28, run one after another at load 28–38.
+
+| # | Acceptance | Result | Where |
+|---|---|---|---|
+| 1 | A partition with 50% dead bytes compacts to ≤ 55% of its prior disk bytes; owner commit p99 and interactive lane p99 ≤ 1.5× baseline during compaction; 1,000 pre-SWAP statements complete correctly | **Size and statements pass; the latency ratio is not met on the Mac (needs a quiet Linux-8 box).** Mac mem, load 31–35: 4 partitions × 20,000 OMM records, half killed: 106.4 MB → 51.9 MB (0.488×); 1,003 bulk statements that started before a SWAP returned exactly the live rows with their bytes, 0 errors (2,177 SWAPs). Latency over the whole SWAP period (the compaction plus the re-compactions that produced those statements; 16,093 samples): ack p99 0.133 → 18.5 ms, interactive lane p99 0.339 → 3.75 ms, commit round p99 1.05 ms. | `compaction_half_dead_T3_1_full` |
+| 2 | CAT supersede, 20% changed per cycle, 100 cycles: disk plateaus ≤ 1.3× live bytes; A9: `m` bytes ≤ one active plus one sealed segment | **Pass with the 15% trigger (deviation 24), "live bytes" read as the live set's compacted footprint.** Mac mem, load 31–33: 20,000 CAT objects, 100 cycles (each updates a random 20%), the store measured at rest after each cycle: disk peaked at 1.120× the footprint of the same live set fully compacted (19.5 MB); 346 compactions, 520 meta segments retired, 5,291 files unlinked; at most 1 meta segment on disk (A9). With §11's 25% trigger it peaks at 1.31–1.38× (six runs). Against the records' own bytes (Σ len − 4 of live PUTs, 4.9 MB) the disk is 4.5×: a fully compacted CAT record costs about 4× its frame in rows, attributes and postings, which no compaction removes | `compaction_cat_supersede_plateau_T3_2_full` |
+| 3 | Quota at 80% of current bytes: oldest records evicted first; no supersede-lane head evicted; disk ≤ cap within 3 passes; each eviction step ≤ 10 ms | **Pass (arrival order, §22.4-3).** Mac mem, load 33–35: the store with its type logs 14.1 MB, cap 11.3 MB: under the cap after 1 pass (14 segments), 0 of 256 CAT heads evicted, in each partition the evicted records are exactly its oldest (0 holes); step max 3.4 ms over 15 steps. Across partitions eviction is by whole segment, so a survivor can be older than an evicted record of another partition (288 of 6,367 records here). The 10 ms bound is enforced on a quiet box (the wasm suite at load 41 measured past it). | `quota_arrival_order_heads_spared_T3_3` |
+| 4 | 1,000 crash points during compaction: exactly one of the old or new set is live, 0 orphans, 0 acked records lost; A12: crash points inside the grace window, Σ on-disk sizes = Σ `disk_bytes` after each reopen | **Pass.** Mac mem, load 28–38: 3,224 trials, 1,000 with a compaction planned but not applied at the crash, 1,744 with retired files waiting; all five crash modes (664 drop-all, 637 drop-subset, 658 torn, 623 reordered, 642 kill -9); 41,825 compactions, 291,864 files unlinked, 24,719 meta segments retired (partitions and the type's), 723,969 dead catalog entries folded out; after every reopen each partition directory and the type directory held exactly the named files with sizes equal to the engine's disk bytes, every acked live record live with its bytes, every acked kill dead, counters equal to a recount; once labels caught up, `first_live_count` equal to the live records and 16 catalog lookups right. 507 s. The T1 crash harness (merges, A9, reclamation; compaction off: its oracle keeps dead bytes) checks the partition directories after every reopen too; the wasm build runs it ×1000. | `orphan_crash_points_during_compaction_T3_4_full`, `crash_faults_T1_1` |
+| 5 | Hot split at 4× single-writer capacity with 4 helpers | **Not built** (§30). | – |
+| 6 | Fill the disk to ENOSPC under ingest: ingest resumes with 0 acked losses and no operator action | **Pass.** Mac mem, load 34–35, a 16 MiB device, 1 MiB ballast, two OMM producers for 4 s: 17 episodes, the ballast released 17 times and restored 16, 41,112 records acked after a recovery, 6,318 alive at the end; in each partition the surviving acked records are exactly its newest (0 holes); after a reopen (before the engine starts) the partition and type directories hold exactly the named files at the engine's disk bytes. Before type-log reclamation the type logs filled 16 of the device's 17 MB and the run ended in the emergency with no survivor. | `quota_disk_full_resumes_without_operator_T3_6` |
+
+§22 amendments naming T3:
+
+| Amendment | Status |
+|---|---|
+| A9 meta-segment retirement | Built (§25). |
+| A11 intents | INTENT_COMPACT built; merge and compaction intents coexist in one batch. |
+| A12 reclamation lifecycle | Built (§25): RETIRE/UNLINKED, reader gate twice a grace apart, open replays the set. |
+| A13 ENOSPC, ballast, arrival order | Built with deviations 12–13. |
+| A15 arrivals compaction | **Half built**: dead catalog copies are folded out of the type's runs and the type logs are reclaimed and counted (§25–§27); arrivals entries and GONE postings of dead gseqs stay (§30). |
+| A26 stage-1 prep states | Not built (with #5). |
+
+## 29. Design deviations (T3)
+
+1. **Compaction builds on its own threads** (§11: the owner in 4 MiB / 10 ms
+   slices; helpers only for split partitions). As T1 deviation 1 did for
+   merges: a build on the writer stalls every partition it owns. The builder
+   re-checks the owner word and an abort flag before every mutation, and a
+   HANDOFF waits for it to stop (one slice).
+2. **The owner writes the manifest at SWAP time**, synced in the next round's
+   first phase, instead of the builder: merges of the active segment continue
+   while a build runs; only the SWAP waits for no merge in flight.
+3. **Kill bound and `prevGen` fallback** (not in the design): a row is removed
+   only when its killer is labeled, and a type-level snapshot below a
+   segment's bound reads the previous file set. Without it, a type-level
+   statement whose type snapshot predates a death could miss a live copy
+   (A14).
+4. **Counters are physical** (§24): recounts equal them after compaction.
+5. **Coalescing** adjacent small segments (§11 "adjacent sealed segments under
+   8 MiB each") into one output named by the first segment.
+6. **Minor 1 extended**: a tombstone whose target is VOID in the same inputs
+   (an earlier SWAP of that segment removed it) goes too. Without it the
+   supersede tombstones of a partition compacted in one piece never went
+   (measured: the CAT plateau grew by 77 KB a cycle).
+7. **Tombstone-only segments seal by age** (T1 sealed only segments with
+   frames), so a partition that stops ingesting can drop its tombstones.
+8. **Reader gate by statement start time**, not by announced manifest
+   generations (A12 text): T2 built announcements as start times
+   (`oldestActiveStart`); the host passes the minimum
+   (`flatsql_ps_reader_gate`).
+9. **RETIRE is the whole set** per changing batch, not incremental records, so
+   open reads one record (head pointer) plus the tail.
+10. **UNLINK_IF_UNUSED BUSY skips** to later items (A12: "retry later");
+    reader lanes close idle handles after 5 s.
+11. **Disk bytes include zero-fill and pre-created files** (the file sizes);
+    open stats the files it names.
+12. **ENOSPC evicts with tombstones, one segment per wave** (A13: "first unlink
+    whole eligible sealed segments, oldest first ... needs no new space").
+    Dropping a segment without per-record tombstones would leave the type
+    owner unaware of the deaths (A14: FIRST copies must be labeled DEAD and
+    REPEATs promoted), and those catalog updates need space too. The ballast
+    must hold one segment's tombstones: about 128 B of row plus postings per
+    record, twice until the tombstones' own segment is compacted.
+13. **Ballast is off in the engine default**; hosts set it (servers 256 MiB,
+    browsers one seal plus one commit), so tests and benchmarks are unaffected.
+14. **Quota counts partitions and type logs**, not the registry (a few KB)
+    or type configs (registration).
+15. **A rebalance of a partition already handed off is ignored** (a T1 race
+    the compaction timing exposed: a second release before the target adopted
+    could leave two owners).
+16. **RecordAttr bytes**: `buildRecordAttr` created its strings inside one
+    argument list, whose evaluation order C++ leaves unspecified; GCC on x86_64
+    built different bytes from clang. Each string is now created in field order;
+    `format_record_attr_golden_bytes` checks `vectors/record_attr.hex`
+    (22.3a-6).
+17. **T3 #2 "live bytes" is the live set's compacted footprint**: the same
+    records fully compacted, rows, attributes and postings included. Every
+    record carries about four times its frame in index structure, so a ratio
+    to frame bytes alone would measure the index, not garbage; both ratios are
+    reported (§28).
+18. **Type logs persist their retired set in the manifest** (not in ctl
+    records): a type has no ctl-record log, and the manifest is the one file a
+    MERGE_DONE names. The appendix sits after the CRC trailer, so readers and
+    opens that predate it parse the manifest unchanged.
+19. **A type meta segment retires with two synced head writes** instead of a
+    LANE_CKPT batch and a later DURABLE_CKPT head (A9): a type head has no
+    lane table, and after the second write neither slot names the segment,
+    whichever one a crash leaves torn.
+20. **A compaction plan pins its files**: anything retired after the plan was
+    made stays until the SWAP or the abort (a build reads the runs and meta
+    segments current at planning).
+21. **A failed build never quarantines**: the inputs are untouched, the
+    partition carries on, and requests end with an error (NOSPACE starts the
+    emergency). A quarantined partition ends its compaction requests at once.
+22. **TOMB_RANGE steps are also bounded in time** (4 ms of CPU, besides 512
+    rows): a kill looks up the row's tag instances in every run, so rows cost
+    more as a partition grows, and more under wasm (T3 #3: ≤ 10 ms a step).
+23. **Dead catalog copies are dropped by merges** (A15 says "type-log
+    compaction"): the type merge already reads every entry it could drop, so
+    no separate pass rewrites the runs; a copy goes once its first posting and
+    its DEAD posting meet in one merge (at the latest, the next full fold).
+24. **The dead-share trigger is 15%, not 25%** (§11). At rest a segment can
+    hold up to the trigger's share of dead rows, and a dead row costs as much
+    as a live one in rows, attributes and postings, so a store at rest sits
+    at up to 1/(1 − trigger) of its compacted footprint: 1.33× at 25%, above
+    T3 #2's 1.3. Measured on the T3 #2 run (20,000 CAT objects, 20% updated
+    per cycle, 100 cycles; Mac mem, load 6–26): 25%: 1.31–1.38 over six
+    runs; 20%: 1.35, 1.38; 15%: 1.115, 1.120; 10%: 1.19, with about the
+    same number of compactions (334–361). The price is write
+    amplification: a compaction at 15% rewrites 5.7 bytes per byte it
+    reclaims (3 at 25%). `EngineConfig::compactDeadRatio` sets it.
+25. **A compaction wave is one output**: segments next to the chosen one that
+    are past the trigger too join it, and the next candidate is looked for
+    right after a SWAP (not 50 ms later).
+
+## 30. Not built yet; running the T3 tests
+
+Remaining T3 scope:
+
+- **#5 hot split** with stage-1 helpers and sealed-segment re-homing (§12), and
+  A26's ring prep states (FREE → CLAIMED(epoch) → PREPARED). Sealed segments'
+  merges and compactions already build off the owner (deviation 1 here, T1
+  deviation 1); what is missing is stage-1 (verify, sha256, key extraction
+  claimed in ranges by idle writers) and its trigger. Its acceptance (≥ 2.5×
+  the unsplit rate with 4 helpers) is a throughput ratio that needs a quiet
+  many-core box to mean anything.
+- **A15 arrivals compaction**: arrivals entries (24 B) and the GONE posting
+  of every dead gseq stay, about 40 B per record ever ingested; the quota
+  counts them but nothing shrinks them (the type-log test: 2.4 MB of catalog
+  runs, mostly GONE, and 2.5 MB of arrivals after 104,100 records with 835
+  alive). The type-log half (catalog runs, manifests, meta segments, dead
+  catalog copies) is built (§25). The arrivals half must drop an entry and its
+  GONE posting in one commit, or offset paging (live = entries − GONE, T2)
+  miscounts: the plan is to rewrite sealed arrivals segments inside the type
+  merge that drops their GONE postings, named by the merge's generation in
+  its manifest.
+- **#1 latency ratio** on a quiet Linux-8 box.
+
+```
+cpp/build/flatsql_ps_test --test=compaction_          # basic, T3 #1 and #2 (short forms)
+cpp/build/flatsql_ps_test --test=compaction_half_dead_T3_1_full [--dir=<dir>]
+cpp/build/flatsql_ps_test --test=compaction_cat_supersede_plateau_T3_2_full
+cpp/build/flatsql_ps_test --test=orphan_crash_points_during_compaction_T3_4_full   # 1,000 crash points
+cpp/build/flatsql_ps_test --test=quota_                # T3 #3, #6
+cpp/build/flatsql_ps_test --test=type_logs_reclaimed_under_readers
+```
+
+**T3 #1 on a quiet Linux-8 box** (8 cores, ext4, nothing else running), from a
+checkout at the landed commit; the test enforces the 1.5× bounds on such a box:
+
+```
+cpp/build/flatsql_ps_test --test=compaction_half_dead_T3_1_full --dir=<ext4 dir>
+```
+
+# Part IV: T3b
+
+## 31. store-migrate gseqs, the entry-size setting, tag conditions
+
+store-migrate (T6, design §16.1-5) gives every migrated FIRST copy gseq = its
+legacy `sdn_record_index.rowid`, so per-schema MaxRowID and the FTS rowids
+carry over.
+
+- **Input.** `RecordAttr.migrated_gseq` (`ulong`, field 6, appended to
+  `schemas/flatsql_attr.fbs`; absent = 0, so older attributes are
+  unchanged byte for byte). `buildRecordAttr(..., migratedGseq)` writes it.
+- **Row.** A new PUT whose attribute carries one gets row flag
+  `MIGRATED_GSEQ` (0x80). The gseq stays in the attribute (in the same meta
+  batch, `ATTR_IN_M`): the RecRow has no spare u64 (offsets 108–127 hold
+  `supersedeHash`, `tagHash` and `aux`), and the attribute is as durable as
+  the row. Deviation from the T6 proposal "(b) the RecRow reserved u64 at
+  offset 116", which T1's layout had already given to `tagHash`.
+- **Labeling.** The type owner reads the flagged rows' attributes in windows
+  over the chunk it labels (one read per chunk of up to 4,096 rows when the
+  attributes fit in 1 MiB). A flagged copy labeled FIRST keeps its gseq when it
+  is above the type's committed `gseq_hi` and no arrival of the same type
+  commit holds it; the engine's gseq counter is then raised to at least gseq
+  + 1. Otherwise it gets an allocated gseq and counts a fallback; a flagged
+  copy labeled REPEAT counts a fallback too. A type commit's new arrivals are
+  sorted by gseq before they are appended, so arrivals stay strictly
+  increasing whatever order partitions are labeled in.
+- **Counters.** `EngineStats::migratedGseqs` and `migratedGseqFallbacks`,
+  appended to `flatsql_ps_stats` (entries 24 and 25). store-migrate's
+  verification (§16.1-7) needs fallbacks = 0; they are counted per labeling
+  decision, so a type commit that fails and is relabeled counts again.
+- **Crash.** The gseq is read from durable rows and attributes, so a crash
+  that loses the type commit relabels the same copies with the same gseqs.
+  An allocated fallback gseq that was never published can be reassigned
+  (as for every allocated gseq).
+- **What T6 must do** (its (d)): feed each type's FIRST copies in rowid
+  order and wait for `labeled_through` before crossing partitions, and run
+  with fallbacks = 0. gseqs are unique per type; across types a fallback could
+  reuse a legacy rowid another type migrates later, which is one more reason
+  a fallback fails the migration.
+- **Entry size.** Writer TLV 19 = `maxEntryBytes` (u64; default 1 MiB +
+  4 KiB, clamped to [64 KiB, 1 GiB]): the largest ring entry (header,
+  RecordAttr and frame). Each writer's arena grows to at least 4 ×
+  (maxEntryBytes + 4 KiB), so the frames half holds two entries. Frames
+  larger than a segment (A27 jumbo segments) are still not built.
+
+Tests: `type_migrated_gseq_preserved_fallback_and_stats` (kept, fallback
+below `gseq_hi`, REPEAT copy, ordinary allocation continues above, flags on
+disk, reopen), `type_migrated_gseq_one_commit_sorted_and_crash_reopen` (two
+partitions' interleaved gseqs in one type commit come out sorted; a crash
+before and after the type commit gives the same arrivals),
+`writer_max_entry_bytes_configurable` (a 9 MiB frame refused by default,
+stored and read back at 10 MiB), `writer_capi_tlv19_max_entry_and_stats`
+(tag 19 through `flatsql_ps_init`, the 26-entry stats array).
+
+### 31.1 Tag conditions match any live tag instance (A2)
+
+`_provider`, `_batch`, `_peer_id`, `_source` and `_source_name` conditions
+have the legacy tag table's ANY-row semantics (`indexedRecordWindowSQL`: a
+CID matches when one of its tag rows satisfies every requested condition).
+In format 2 a record's tag instances are its PUT's own tag and its RETAG rows
+(A2). As T2 built it, the vtab scanned tag postings (which name every
+instance, RETAGs included) but left each constraint for SQLite to re-check
+against the projected column, which shows only the PUT's own tag, so every
+record matched through a RETAG was dropped (T6 #7 on a migrated copy: a
+provider window of 200 read as 100; a source + batch window lost the
+re-tagged records).
+
+Now `xBestIndex` consumes every tag condition (`omit = 1`) and the vtab
+evaluates them all on one instance:
+
+- A tag or source posting scan (the plan's key) checks the remaining
+  conditions on the instance the posting names (its RecordAttr), then emits
+  the instance's record. A single condition that is the scanned key needs no
+  attribute read unless the key was capped (over 512 bytes).
+- Any other plan (CID, column index, epoch, arrivals order, full scan)
+  keeps a record when one of its live tag instances matches every condition
+  (`RowFilter::hasLiveTag`, the TAG_OF postings), as the alias table's source
+  filter already did.
+- An untagged record matches no tag condition (as the index-driven plans
+  already behaved, T2 deviation 8), and conditions that contradict each other
+  (two sources) return no rows.
+- `_source = '<TYPE>@<name>'` is compared as the column projects it: the type
+  prefix is case-sensitive (a lower-case prefix returned no rows before
+  either, after SQLite's re-check).
+- `_peer_id` conditions match the tag's producer peer (the TAG_PEER postings
+  the T2 plan scanned); the `_peer_id` column still projects the storing
+  call's `RecordAttr.peer_id`. Before, SQLite compared that BLOB column with
+  the argument, so a text argument never matched.
+- `<TYPE>_current` groups first and filters after, as SQL says: its tag
+  conditions stay SQLite's (the current version's own tag).
+- Projected columns are unchanged: they show the PUT's own tag.
+
+Test: `lane_tag_conditions_match_any_instance_A2` (two partitions, PUT tags,
+RETAGs and REPEAT copies; eleven condition sets at type and partition level
+against an ANY-instance oracle, the full `_source` form, the alias, an
+epoch-ordered window, column and CID plans). With `omit = 0` restored it
+fails 18 checks.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "flatsql",
-  "version": "3.0.0",
+  "version": "3.2.0",
   "description": "SQL queries over raw FlatBuffer storage: the SQLite-backed wasm engine and the multithreaded partition store (flatsql-ps-threads.wasm)",
   "type": "module",
   "main": "wasm/index.js",

package/wasm/integrity.json

@@ -5,9 +5,9 @@
   "size": 2139350,
   "files": {
     "flatsql-ps-threads.wasm": {
-      "sha256": "572a4bc38345d07ff91c345fc8d81e28873d72efa0fb8864a2edc5242145f9f5",
-      "sri": "sha384-ewHpK0cJ1AAFg2xqLIWtGWyAE9W0A+4Vq6THCJ2WJpPcKxHqabdf7bbxt9Ngf7oZ",
-      "size": 2074707
+      "sha256": "a90d9488187e527bccd3f02ce3f68be6443690bc1a2ee6c6786c64c907376e36",
+      "sri": "sha384-zzvS8qbCe33AJCklxaZX4xr1+QajJAaz0G0FuBv9WyEXh2rSKAz9+Rq4o2XksFaF",
+      "size": 2261085
     },
     "flatsql-wasi-noeh.wasm": {
       "sha256": "fd744d4c502310669bc74ab555a36de1692030a0fa6e22fdb1df744809eeacbe",