arrowbase 0.1.1

package/README.md

@@ -0,0 +1,897 @@
+# ArrowBase
+
+[![CI](https://github.com/natanelia/arrowbase/actions/workflows/ci.yml/badge.svg)](https://github.com/natanelia/arrowbase/actions/workflows/ci.yml)
+[![npm](https://img.shields.io/npm/v/arrowbase.svg)](https://www.npmjs.com/package/arrowbase)
+[![license: MIT](https://img.shields.io/badge/license-MIT-171a18.svg)](./LICENSE)
+[![playground](https://img.shields.io/badge/playground-open-df542d.svg)](https://natanelia.github.io/arrowbase/)
+
+> ArrowBase — columnar Arrow store with @tanstack/db-compatible API.
+
+ArrowBase is a local-first TypeScript database built on Arrow-style column
+buffers. It keeps the TanStack DB mental model—collections, transactions,
+live queries, persistence, and React hooks—while adding vectorized filters,
+spatial indexes, Arrow IPC, snapshots, and cross-tab sync.
+
+**v0.1.1:** the compatibility catalog is 294/294 green
+against `@tanstack/db@0.6.16`. Exact public-export and operator-manifest tests
+guard that surface. Pre-1.0 means the API can still evolve between minor
+versions; documented tolerances cover fixed capacity, safe-integer key
+normalization, and floating-point aggregates.
+
+[Open the live playground](https://natanelia.github.io/arrowbase/) ·
+[Read the migration guide](./MIGRATING.md) ·
+[Inspect compatibility](./COMPATIBILITY.md)
+
+**Project docs**:
+
+- [COMPATIBILITY.md](./COMPATIBILITY.md) — all 294 audited upstream exports.
+- [MIGRATING.md](./MIGRATING.md) — side-by-side TanStack DB replacements.
+- [docs/showcase.md](./docs/showcase.md) — playground and benchmark contract.
+- [docs/ivm.md](./docs/ivm.md) — incremental query engine boundaries.
+- [SECURITY.md](./SECURITY.md) and [CONTRIBUTING.md](./CONTRIBUTING.md).
+- [RELEASE_NOTES.md](./RELEASE_NOTES.md) and [CHANGELOG.md](./CHANGELOG.md).
+
+## Requirements
+
+- Node.js 20 or newer.
+- Modern browsers. ArrowBase uses `SharedArrayBuffer` when cross-origin
+  isolation is available and safely falls back to `ArrayBuffer` otherwise.
+
+## Install
+
+```sh
+npm install arrowbase
+```
+
+Optional integrations are installed only when you use their subpaths:
+
+```sh
+npm install @tanstack/db   # arrowbase/tanstack
+npm install apache-arrow   # arrowbase/arrow
+npm install react          # arrowbase/react
+```
+
+## Playground
+
+The hosted [ArrowBase City Ops Console](https://natanelia.github.io/arrowbase/)
+is an interactive ArrowBase vs TanStack DB playground. It runs deterministic
+spatial, reactive, transactional, and IPC workloads in browser workers. A
+correctness hash must match before timing or memory ratios unlock.
+
+See [docs/showcase.md](./docs/showcase.md) for the benchmark contract, URL
+controls, operation registry, and local or tunnel workflows.
+
+## Development
+
+```sh
+pnpm install --frozen-lockfile
+pnpm check                  # typecheck + 700+ behavioral tests
+pnpm playground:check       # UI typecheck + tests + production build
+pnpm playground:e2e         # desktop and mobile Playwright smoke
+pnpm bench:vs-tanstack      # head-to-head benchmark
+pnpm build
+pnpm verify:package-exports
+```
+
+Use these focused playground commands during UI work:
+
+```sh
+pnpm playground:dev
+pnpm playground:typecheck
+pnpm playground:test
+pnpm playground:build
+pnpm playground:e2e
+```
+
+Performance ratios are evidence, not promises: the showcase compares equal
+result hashes before presenting speed or memory measurements.
+
+## What's in the box
+
+### Phase 1 — Core primitives
+
+Fixed-width columns (`int8..int64`, `uint8..uint64`, `float32/64`, `bool`)
+backed by a 64-byte-aligned SAB bump allocator. Dense `uint32` primary key,
+soft-delete, per-column + global version counters, bounded undo ring with
+mark/rollback.
+
+```ts
+import { Collection, defineSchema } from 'arrowbase';
+
+const schema = defineSchema({
+  name: 'features',
+  columns: {
+    __id: { type: 'uint32' },
+    __deleted: { type: 'bool' },
+    speed: { type: 'int32', nullable: true },
+    weight: { type: 'float64' },
+  },
+});
+
+const features = Collection.create(schema, { capacity: 10_000 });
+features.insert({ __id: 1, __deleted: false, speed: 50, weight: 1.5 });
+features.update(1, { speed: 60 });
+features.delete(1); // soft; becomes hard in hard-delete mode
+
+const mark = features.beginTransaction();
+features.update(1, { speed: 999 });
+features.rollback(mark);
+```
+
+### Phase 2 — Reactive queries
+
+Immutable query builder (`where`/`select`/`orderBy`/`limit`/`offset`/`deps`).
+`QueryExecutor` caches snapshots keyed on observed-column versions. A
+passive change listener invalidates the cache on insert/delete/compact/
+rollback; updates that touch only unobserved columns keep the cached
+snapshot identity.
+
+```ts
+import { query, run } from 'arrowbase';
+
+const q = query(features)
+  .where((r) => r.speed !== null && (r.speed as number) > 50)
+  .orderBy('weight', 'desc')
+  .select(['__id', 'speed'])
+  .limit(20);
+
+const exec = run(q);
+const unsub = exec.subscribe((snap) => console.log(snap.rows));
+// unsub() when done; exec.dispose() to detach the passive listener.
+```
+
+Predicate deps are inferred via a tracking `Proxy`. Override with
+`.deps(['surface', 'speed'])` when inference is incomplete.
+
+### Phase 3 — Variable-width + dict columns
+
+`utf8`, `binary`, and dictionary-encoded `dict_utf8`. Utf8/binary use a
+ranged-offsets layout (`Int32Array(capacity * 2)` pairs + append-only
+`ValuesHeap`) so updates are O(1) at the cost of leaked bytes until
+`compact()` runs.
+
+```ts
+const schema = defineSchema({
+  name: 'road',
+  columns: {
+    __id: { type: 'uint32' },
+    name: { type: 'utf8', nullable: true, valuesHeapBytes: 512 * 1024 },
+    surface: { type: 'dict_utf8' },
+    blob: { type: 'binary' },
+  },
+});
+```
+
+### Deferred cleanup — compaction
+
+`Collection.compact()` reclaims leaked utf8/binary heap bytes and hard-
+deleted row slots, rebuilds the row index, clears the undo ring, and fires
+a `compact` `ChangeEvent` so reactive queries recompute.
+
+```ts
+const { reclaimedRows, reclaimedHeapBytes } = features.compact();
+```
+
+### Phase 4 — TanStack DB adapter
+
+`arrowbase/tanstack` exposes an ArrowBase collection as a first-class
+TanStack DB collection. Use `@tanstack/db`'s native live-query builder to
+join ArrowBase with native TanStack collections.
+
+This adapter prioritizes drop-in native interoperability. TanStack's public
+collection boundary exchanges JavaScript row objects, so this route does not
+preserve zero-copy source reads or push ArrowBase spatial indexes into
+TanStack's native query compiler. Use ArrowBase's root
+`createLiveQueryCollection` API when columnar differential execution and
+spatial pushdown matter; it keeps the same builder shape and runs structural
+queries through `@tanstack/db-ivm`. See [docs/ivm.md](./docs/ivm.md) for the
+exact supported/fallback boundary.
+
+```ts
+import { createCollection, createLiveQueryCollection, eq } from '@tanstack/db';
+import { toTanstackCollection } from 'arrowbase/tanstack';
+
+const abFeatures = Collection.create(/* ... */);
+const features = createCollection(toTanstackCollection(abFeatures));
+
+const tags = createCollection({ /* native TS collection with tags */ });
+
+const joined = createLiveQueryCollection({
+  query: (q) =>
+    q
+      .from({ t: tags })
+      .join({ f: features }, ({ t, f }) => eq(t.featureId, f.__id))
+      .select(({ t, f }) => ({ label: t.label, name: f.name })),
+});
+```
+
+`onInsert`/`onUpdate`/`onDelete` forward TanStack mutations into ArrowBase;
+the sync listener mirrors ArrowBase changes back into TanStack. `compact`
+and `rollback` trigger a `truncate()` + replay for full consistency.
+
+Options:
+- `id?: string` — TanStack collection id. Default `'arrowbase:<schemaName>'`.
+- `includeSoftDeleted?: boolean` — mirror tombstones as regular rows.
+
+### Phase 5 — Snapshot + sync
+
+Custom binary snapshot (`ARBS` magic, format v1) preserves schema, data,
+dictionaries, and `globalVersion` across round-trips. Sync adapter
+interface + in-process `MemorySyncAdapter` for tests; `ChangeLog`
+records outgoing deltas; `applyDeltas` handles inbound with upsert
+promotion.
+
+```ts
+import {
+  exportSnapshot,
+  importSnapshot,
+  ChangeLog,
+  MemorySyncAdapter,
+  SharedDeltaLog,
+  applyDeltas,
+} from 'arrowbase';
+
+// Persistence
+const buf = exportSnapshot(features, { tag: 'nightly' });
+const restored = importSnapshot(buf, (schema, capacity) =>
+  Collection.create(schema, { capacity }),
+);
+
+// Sync
+const log = new SharedDeltaLog();
+const outbound = new ChangeLog(features);
+const adapter = new MemorySyncAdapter(features, log, outbound);
+
+const { deltas, upTo } = await adapter.pull(0);
+await adapter.apply(deltas);
+```
+
+### Phase 6 — Devtools + subset push-down
+
+`describeCollection(abColl)` returns a framework-free JSON snapshot of
+schema + per-column metadata (version, nullCount, heap usage, dict sample).
+`pushDown(expr, knownColumns)` translates TanStack DB `BasicExpression`
+ASTs (`eq`, `and`, `or`, `not`, boolean ref) into an ArrowBase
+`Predicate`; unsupported expressions return `null` so the caller falls
+back gracefully.
+
+```ts
+import { describeCollection, pushDown } from 'arrowbase';
+
+console.log(describeCollection(features));
+// → { schemaName, primaryKey, rowCount, liveRowCount, globalVersion,
+//     hasSoftDelete, schemaFingerprint, columns: [...] }
+```
+
+### Phase 7 — Vectorized filter + sort kernels
+
+`.filter(expr)` takes a column-op DSL expression that compiles to a
+row-test closure reading directly from typed-array column buffers. No
+per-row proxy, no row-object allocation during filter, numeric sort via
+a `Float64Array` shadow-key buffer, materialization only of the
+`[offset, offset+limit)` window. The fn-predicate path via `.where(fn)`
+still works and can be combined with `.filter()` — the kernel runs
+first, the fn runs only on survivors.
+
+```ts
+import { query, run, f, fEq, fGt, fAnd } from 'arrowbase';
+
+const q = query(features)
+  .filter(fAnd(fEq(f('surface'), 'asphalt'), fGt(f('speed'), 50)))
+  .orderBy('weight', 'desc')
+  .select(['__id', 'speed', 'weight'])
+  .limit(20);
+
+const snap = run(q).snapshot();
+```
+
+Operators: `fEq`, `fNeq`, `fGt`, `fGte`, `fLt`, `fLte`, `fAnd`, `fOr`,
+`fNot`, `fIsNull`, `fIsNotNull`. Supported types: all fixed-width
+numerics, `bool`, `int64`/`uint64` (with number→bigint literal
+promotion), `utf8` (eq/neq), `dict_utf8` (eq/neq via O(1) dictionary
+code lookup).
+
+**String predicates.** `fContains`, `fStartsWith`, `fEndsWith`, `fLike`,
+`fMatches` work on `utf8` and `dict_utf8` columns. On dict-encoded
+columns the kernel walks the dictionary once up-front to build a
+`Set<code>` of matching entries, then each row is a single `Set.has()`
+— ~4× faster than per-row utf8 substring search on 10k rows.
+
+```ts
+import {
+  query, run, f,
+  fContains, fStartsWith, fEndsWith, fLike, fMatches, fAnd,
+} from 'arrowbase';
+
+// Case-insensitive substring (ASCII-fold).
+query(c).filter(fContains(f('tag'), 'asp', { caseInsensitive: true }));
+
+// SQL LIKE. `%` = any sequence, `_` = any single char, `\` escapes.
+query(c).filter(fLike(f('path'), '/users/%/active'));
+
+// Raw regex (pattern string or RegExp).
+query(c).filter(fMatches(f('url'), /^https?:/));
+
+// Composable, negatable.
+query(c).filter(fAnd(
+  fStartsWith(f('tag'), 'A'),
+  fContains(f('text'), 'road', { caseInsensitive: true, negated: true }),
+));
+```
+
+The fn-compiler auto-lowers the three common JS method forms into the
+equivalent DSL node:
+
+```ts
+// Both run through the vectorized kernel:
+query(c).where(r => r.tag.includes('asphalt'));      // → fContains
+query(c).where(r => r.tag.startsWith('A'));           // → fStartsWith
+query(c).where(r => r.tag.endsWith('vel'));           // → fEndsWith
+```
+
+### Phase 8 — List + Struct column types
+
+Two nested column shapes landed, rounding out the type system:
+
+- `list<T>` — each cell is an array of T. Arrow-style offsets + child
+  buffer; O(1) updates with leak-until-compact semantics. v1 item types:
+  every leaf (all primitives, `bool`, `utf8`, `dict_utf8`, `binary`).
+- `struct<{f1, f2, ...}>` — each cell is a record of fixed named fields.
+  Parent validity bitmap in SAB + one regular child column per field.
+  v1 field types: leaves only.
+
+Nested list-in-list, struct-in-struct, list-in-struct, struct-in-list
+are rejected at schema compile time. Snapshot format bumped to v2 so
+list offsets + child buffers and per-field struct buffers round-trip.
+
+```ts
+const schema = defineSchema({
+  name: 'features',
+  columns: {
+    __id: { type: 'uint32' },
+    tags: { type: 'list', items: { type: 'utf8' } },
+    geo: {
+      type: 'struct',
+      fields: {
+        lat: { type: 'float64' },
+        lng: { type: 'float64' },
+        label: { type: 'utf8', nullable: true },
+      },
+    },
+    attrs: {
+      type: 'list',
+      items: { type: 'int32', nullable: true }, // list items may be null
+    },
+  },
+});
+const c = Collection.create(schema, { capacity: 1_000 });
+c.insert({
+  __id: 1,
+  tags: ['road', 'highway'],
+  geo: { lat: 51.5, lng: -0.12, label: 'London' },
+  attrs: [10, null, 30],
+});
+// update replaces the full list/struct cell
+c.update(1, { tags: ['updated'] });
+// compact reclaims leaked list child bytes + soft-delete slots
+const { reclaimedHeapBytes } = c.compact();
+```
+
+List child storage lives in JS heap (outside the SAB segment) because
+its size is unbounded at schema-compile time. This is a deliberate v1
+tradeoff — zero-copy cross-worker sharing for list columns is deferred.
+Struct field buffers and parent validity stay in the SAB segment.
+
+### Auto-lowering `.where(fn)` to the vectorized kernel
+
+`.where(fn)` attempts to compile the predicate closure into a
+`FilterExpr` tree via a tiny recursive-descent parser over
+`Function.prototype.toString()`. When it succeeds, the fn is discarded
+and the vectorized kernel runs — same fast path as `.filter(expr)`,
+no caller-side rewrite needed:
+
+```ts
+// Both forms take the vectorized path; second one compiles the closure
+// into the equivalent FilterExpr at query-build time.
+query(c).filter(fAnd(fEq(f('active'), true), fGt(f('age'), 18)));
+query(c).where((r) => r.active === true && r.age > 18);
+```
+
+Supported shapes: `===`/`!==`/`==`/`!=`/`>`/`>=`/`<`/`<=`/`&&`/`||`/`!`,
+param property or bracket access (`r.col` or `r['col']`), string/number/
+bool/null literals, `row.col == null` → `isNull`. Anything else
+(captures, method calls, arithmetic, nested field access, ternary, etc.)
+falls back silently to the legacy row-scan path:
+
+```ts
+// Falls back — outer variable is a capture the compiler can't inline.
+const min = 18;
+query(c).where((r) => r.age > min);
+```
+
+The end-to-end speed-up on a 10k row filter+sort+paginate is ~5× (0.5 ms
+vs 2.5 ms); reactive notifications drop from ~300 µs to ~60 µs. See
+[Benchmarks](#benchmarks).
+
+### Hash joins
+
+Inner, left, right, and full outer joins between two collections. The
+right side is hashed; the left side (post-filter) probes into the
+hash. Nested row shape — the right row is attached at `row[as]` — so
+column-name collisions never matter.
+
+```ts
+import { query, join, runJoin, f, fEq } from 'arrowbase';
+
+const plan = join(
+  query(orders).filter(fEq(f('status'), 'paid')),
+  users,
+  { on: { left: 'userId', right: '__id' }, as: 'user', type: 'left' },
+)
+  .where((r) => r.user === null || r.user.region === 'us')
+  .orderBy('user.name')
+  .select(['__id', 'amount', 'user.name']);
+
+const exec = runJoin(plan);
+exec.subscribe((snap) => render(snap.rows));
+// Re-runs on mutations to EITHER collection.
+```
+
+- **Join types** — `'inner'` (default), `'left'`, `'right'`, `'full'`.
+  - `left` / `full` — unmatched left rows emit `{...leftRow, [as]: null}`.
+    Detect via `row[as] === null`.
+  - `right` / `full` — unmatched right rows emit `{[as]: rightRow}`
+    with left columns absent (not null — `undefined`), matching SQL
+    projection semantics.
+- **Key normalization.** Safe-range BigInts are unified with Numbers
+  during hashing, so an `int64` foreign-key column on one side joins
+  cleanly against a `uint32` PK on the other.
+- **Null keys never join** (SQL semantics). For `left` / `full`, a
+  left row with a null key is still emitted as unmatched.
+- **Left-side filters push down.** Any `.filter()` / `.where()` on
+  the left query runs through the vectorized kernel before the probe
+  — the hash only sees survivors.
+- **Chainable builder.** `query(orders).join(users, …)` is equivalent
+  to the standalone `join(…)` function; both return a `JoinQuery`
+  supporting the full post-join surface (`where` / `select` /
+  `orderBy` / `limit` / `offset`).
+- **Paths.** Selection and ordering accept plain column names (left
+  side) or `'<as>.<col>'` dotted paths (right side). Dotted access
+  into a null namespace yields `undefined`, so nulls sort last by
+  default.
+- **Reactivity.** The executor subscribes to both collections;
+  mutations on either trigger a recompute. Cache keyed by
+  `(leftVersion, rightVersion)` — calling `snapshot()` twice with no
+  mutations returns the same object.
+
+**Limitations.** Single-column equi-key per side (no composite keys),
+full recompute on mutation (no incremental update). The kernel is
+scalar JS, not SIMD — 1k × 1k runs in ~0.8 ms on the reference
+machine.
+
+### GeoArrowBase — geo columns, predicates, and spatial indexes
+
+ArrowBase has dedicated geometry columns for local-first map UIs:
+`point`, `linestring`, and `polygon`. Coordinates are `[lng, lat]` in
+EPSG:4326. Bbox predicates use planar lon/lat bboxes; point `dwithin`
+uses Haversine meters. Geo helpers are ArrowBase extensions and do not
+change the TanStack-compatible `operators` / `comparisonFunctions`
+export lists.
+
+```ts
+import {
+  Collection,
+  bboxIntersects,
+  bboxWithin,
+  createEffect,
+  createTransaction,
+  defineSchema,
+  dwithin,
+  queryOnce,
+  within,
+} from 'arrowbase';
+import { exportArrowIPC, importArrowIPC } from 'arrowbase/arrow';
+
+const featureSchema = defineSchema({
+  name: 'geo_features',
+  columns: {
+    __id: { type: 'uint32' },
+    name: { type: 'utf8', valuesHeapBytes: 64 * 1024 },
+    geometry: { type: 'point', nullable: true },
+    route: { type: 'linestring', nullable: true },
+    footprint: { type: 'polygon', nullable: true, autoCloseRings: true },
+  },
+});
+
+const features = Collection.create(featureSchema, {
+  capacity: 100_000,
+  spatialIndex: { column: 'geometry', type: 'rtree', eager: true },
+});
+
+features.insert({
+  __id: 1,
+  name: 'Singapore depot',
+  geometry: [103.8198, 1.3521],
+  route: [
+    [103.72, 1.30],
+    [103.82, 1.35],
+    [103.92, 1.40],
+  ],
+  footprint: [[
+    [103.80, 1.30],
+    [103.90, 1.30],
+    [103.90, 1.40],
+    [103.80, 1.40],
+    // autoCloseRings adds the closing coordinate.
+  ]],
+});
+```
+
+Spatial query helpers work in `queryOnce(...)`, live queries, and effects:
+
+```ts
+const viewport = [103.6, 1.2, 104.0, 1.5] as const;
+const singapore = [103.8198, 1.3521] as const;
+
+const visible = await queryOnce((q) =>
+  q
+    .from({ feature: features })
+    .where(({ feature }) => bboxIntersects(feature.geometry, viewport))
+    .select(({ feature }) => ({ id: feature.__id, name: feature.name }))
+    .orderBy(({ $selected }) => $selected.id, 'asc'),
+);
+
+const fullyInsideBbox = await queryOnce((q) =>
+  q
+    .from({ feature: features })
+    .where(({ feature }) => bboxWithin(feature.geometry, viewport))
+    .select(({ feature }) => ({ id: feature.__id, name: feature.name }))
+    .orderBy(({ $selected }) => $selected.id, 'asc'),
+);
+
+const nearby = await queryOnce((q) =>
+  q
+    .from({ feature: features })
+    .where(({ feature }) => dwithin(feature.geometry, singapore, 30_000))
+    .select(({ feature }) => ({ id: feature.__id, name: feature.name }))
+    .orderBy(({ $selected }) => $selected.id, 'asc'),
+);
+
+const pointsInsideTheirPolygon = await queryOnce((q) =>
+  q
+    .from({ feature: features })
+    .where(({ feature }) => within(feature.geometry, feature.footprint))
+    .select(({ feature }) => ({ id: feature.__id, name: feature.name })),
+);
+```
+
+Use `createEffect` for viewport enter/update/exit deltas:
+
+```ts
+const viewportEffect = createEffect({
+  id: 'visible-features',
+  query: (q) =>
+    q
+      .from({ feature: features })
+      .where(({ feature }) => bboxIntersects(feature.geometry, viewport))
+      .select(({ feature }) => ({
+        id: feature.__id,
+        name: feature.name,
+        geometry: feature.geometry,
+      }))
+      .orderBy(({ $selected }) => $selected.id, 'asc'),
+  onEnter: (event) => console.log('entered viewport', event.key),
+  onUpdate: (event) => console.log('moved in viewport', event.previousValue, event.value),
+  onExit: (event) => console.log('left viewport', event.key),
+  onBatch: (events, ctx) => console.log(ctx.effectId, events.length),
+});
+
+await viewportEffect.dispose();
+```
+
+Optimistic edits use the same transaction shape as non-geo rows.
+Rollback restores geometry values, bbox cache, dirty index overlay state,
+and live-query/effect deltas:
+
+```ts
+const tx = createTransaction({
+  mutationFn: async ({ transaction }) => {
+    const res = await fetch('/api/features/sync', {
+      method: 'POST',
+      headers: { 'content-type': 'application/json' },
+      body: JSON.stringify(transaction.mutations),
+    });
+    if (!res.ok) throw new Error('sync failed');
+  },
+});
+
+tx.mutate(() => {
+  features.update(1, (draft) => {
+    draft.geometry = [103.85, 1.31];
+  });
+});
+
+await tx.isPersisted.promise;
+```
+
+`spatialIndex` supports point, linestring, and polygon envelopes. It maintains
+a compact Flatbush base plus immutable mini-tree segments and a small dirty
+overlay, so sustained inserts/updates do not degrade into a full linear dirty
+scan and do not require `compact()`. Indexed `bboxIntersects`, `bboxWithin`,
+and `dwithin` queries still run the original predicate as a residual exact
+filter after candidate lookup.
+
+Arrow IPC is not Parquet. `exportArrowIPC()` and `importArrowIPC()` read
+and write Arrow IPC streams/files with GeoArrow extension metadata
+(`geoarrow.point`, `geoarrow.linestring`, `geoarrow.polygon`). GeoParquet
+would be a separate export path; do not treat these bytes as Parquet.
+
+```ts
+const bytes = await exportArrowIPC(features);
+const restored = await importArrowIPC(bytes, { name: 'restored_features' });
+```
+
+### Apache Arrow IPC — `arrowbase/arrow`
+
+Optional subpath for interop with the broader Arrow ecosystem
+(arrow-js, DuckDB-WASM, Polars-WASM, pyarrow, Arrow Flight…). Install
+the peer dependency once:
+
+```bash
+pnpm add apache-arrow
+```
+
+Export a collection as an IPC byte stream or file:
+
+```ts
+import {
+  exportArrowIPC,
+  importArrowIPC,
+  exportArrowTable,
+} from 'arrowbase/arrow';
+
+const bytes = await exportArrowIPC(collection);           // stream (default)
+const file  = await exportArrowIPC(collection, { format: 'file' });
+const table = await exportArrowTable(collection);         // arrow.Table directly
+
+// Anywhere downstream — arrow-js, DuckDB-WASM, pyarrow, Arrow Flight:
+// tableFromIPC(bytes) consumes this byte stream.
+
+const rehydrated = await importArrowIPC(bytes);
+```
+
+Supported type mapping (both directions):
+
+| ArrowBase           | arrow-js                |
+|---------------------|-------------------------|
+| `bool`              | `Bool`                  |
+| `int8/16/32/64`     | `Int8/16/32/64`         |
+| `uint8/16/32/64`    | `Uint8/16/32/64`        |
+| `float32/64`        | `Float32/Float64`       |
+| `utf8`              | `Utf8`                  |
+| `dict_utf8`         | `Dictionary<Int32,Utf8>`|
+| `binary`            | `Binary`                |
+| `list<T>`           | `List<map(T)>`          |
+| `struct<…>`         | `Struct<…>`             |
+| `point`             | `geoarrow.point` (`FixedSizeList<Float64>[2]`) |
+| `linestring`        | `geoarrow.linestring` (`List<FixedSizeList<Float64>[2]>`) |
+| `polygon`           | `geoarrow.polygon` (`List<List<FixedSizeList<Float64>[2]>>`) |
+
+Geo columns use GeoArrow extension metadata with `{ crs: 'EPSG:4326',
+edges: 'planar' }`. Arrow IPC is not Parquet: this subpath serializes
+Arrow streams/files only, not GeoParquet files.
+
+**Internal columns.** `__deleted` is stripped from the Arrow schema by
+default and soft-deleted rows are dropped. Pass
+`{ includeInternalColumns: true }` for a loss-less round-trip.
+
+**Import inference.** `importArrowIPC()` synthesizes an ArrowBase
+schema from the Arrow `Schema`. Columns using `Timestamp`, `Decimal`,
+`Date_`, `Map`, etc. throw with a clear error — those map-to-ArrowBase
+semantics are out of scope for v1. Pass `{ capacity, name }` to size
+or label the rehydrated collection.
+
+**Not zero-copy.** Export materializes rows and feeds `vectorFromArray`;
+import walks `table.toArray()`. ArrowBase uses ranged offsets for O(1)
+updates and keeps dict tables in JS heap, so wire compatibility with
+Arrow's dense-offset layout needs a rebuild. This is a serialization
+path, not a hot one.
+
+### IndexedDB persistence — `arrowbase/idb`
+
+Durable storage for long-lived client apps. Loads a snapshot on boot,
+auto-saves with debounce on every mutation, flushes on demand.
+
+```ts
+import { defineSchema } from 'arrowbase';
+import { IdbPersistence } from 'arrowbase/idb';
+
+const schema = defineSchema({ /* ... */ });
+
+const persist = await IdbPersistence.open(schema, {
+  dbName: 'my-app',
+  key: 'features',
+  capacity: 10_000,
+  // format: 'arbs' (default) or 'arrow-ipc' (requires apache-arrow).
+});
+
+const collection = persist.collection; // either restored or fresh
+persist.start({ debounceMs: 500 });
+
+window.addEventListener('beforeunload', () => { persist.flush(); });
+```
+
+Snapshot format is pluggable:
+
+- `'arbs'` (default) — native binary snapshot. Fast, preserves ranged
+  offsets, round-trips soft-deletes byte-for-byte.
+- `'arrow-ipc'` — Apache Arrow IPC file, portable across arrow-js,
+  DuckDB-WASM, pyarrow, etc. Requires the `apache-arrow` peer dep
+  (dynamically imported only when this format is selected).
+
+Storage model: one IDB object store (default `'snapshots'`), caller
+supplies a `key` per collection. Each record holds
+`{ format, bytes, version, savedAt }`, keeping snapshots
+self-describing so `IdbPersistence.open()` picks the right decoder.
+Schema-fingerprint mismatches are rejected with an actionable error.
+
+Lifecycle:
+
+- `start({ debounceMs, saveImmediately })` — subscribe to the
+  collection and auto-save. Bursts of mutations coalesce into a
+  single IDB write per window (default 250 ms).
+- `flush()` — bypass the debounce and await the IDB transaction.
+- `stop()` — cancel any pending write + unsubscribe. Does not close
+  the underlying IDB connection.
+- `close()` — `stop()` + release the IDB connection. Idempotent.
+
+For cache-wipe UX:
+
+```ts
+import { deleteStoredSnapshot } from 'arrowbase/idb';
+await deleteStoredSnapshot('features', { dbName: 'my-app' });
+```
+
+### Cross-tab sync — `arrowbase/broadcast`
+
+Propagate mutations between same-origin tabs (or iframes, or shared
+workers) via the platform `BroadcastChannel` API. Every instance on
+the same channel name shares state; no server round-trip.
+
+```ts
+import { Collection, defineSchema } from 'arrowbase';
+import { BroadcastSync } from 'arrowbase/broadcast';
+
+const schema = defineSchema({ /* ... */ });
+const collection = Collection.create(schema, { capacity: 10_000 });
+
+const sync = new BroadcastSync(collection, { channelName: 'users' });
+sync.start();
+// …mutations here are mirrored to every other tab in real time…
+
+// Teardown:
+sync.stop();
+```
+
+Protocol: four message types (`delta`, `sync-request`, `sync-response`,
+`resync`) tagged with a per-instance `nodeId`. Self-echoes are
+filtered; re-broadcast of inbound deltas is suppressed by a
+reentrancy counter; a per-peer cursor dedupes deltas that arrive both
+via the live stream and a racing `sync-response`. Late joiners
+automatically send a `sync-request(version=0)` on `start()` and peers
+reply with the relevant log slice — pass `{ requestSyncOnStart: false }`
+to opt out.
+
+Compact / rollback events publish a `resync` sentinel (those break
+incremental delta coherence). Peers receive via the `onResync` hook
+and should refetch a snapshot out of band, typically via
+[`IdbPersistence`](#indexeddb-persistence--arrowbaseidb), then rejoin:
+
+```ts
+const sync = new BroadcastSync(collection, {
+  channelName: 'users',
+  onResync: async ({ fromNodeId, version }) => {
+    // Rehydrate from IDB, then rejoin.
+    await persist.restore();
+    sync.requestSync(collection.globalVersion);
+  },
+});
+```
+
+**Non-goals.** Convergence under concurrent edits (last-writer-wins,
+no CRDT merge), delivery guarantees (`BroadcastChannel` is
+best-effort), and security (every same-origin document on the channel
+sees every message — don't broadcast secrets).
+
+### React — `arrowbase/react`
+
+Three hooks, zero boilerplate, concurrent-mode safe (`useSyncExternalStore`):
+
+- `useLiveQuery(query)` — subscribe to a Query, returns the latest
+  `{ rows, version }` snapshot, re-renders only when an observed column
+  moves.
+- `useLiveQueryResult(collection, build, deps)` — builder-closure
+  variant that rebuilds the query when `deps` change. Saves callers
+  from memoizing the Query object themselves.
+- `useCollectionVersion(collection)` — thin subscription to
+  `collection.globalVersion`; useful as a cache-bust key.
+
+React is an optional `peerDependency` (^18 or ^19). Add it to your app
+dependencies and import from the subpath:
+
+```tsx
+import { useLiveQuery, useLiveQueryResult, useCollectionVersion } from 'arrowbase/react';
+import { query, f, fEq } from 'arrowbase';
+
+function ActiveUsers({ users }) {
+  const { rows } = useLiveQueryResult(
+    users,
+    (q) => q.filter(fEq(f('active'), true)).orderBy('name'),
+  );
+  return <ul>{rows.map((r) => <li key={r.__id}>{r.name}</li>)}</ul>;
+}
+```
+
+Re-render economy is inherited from the executor's dep-tracking: a
+mutation to an unobserved column produces zero extra renders.
+
+## Benchmarks
+
+`pnpm bench` runs the regression benches and fails if any exceeds
+its soft ceiling. `pnpm bench:operations` runs a manifest-checked matrix of
+all structural query operators, relational stages, join modes, GeoArrow
+predicates, differential backend shapes, and public CRUD operations.
+`pnpm bench:all` runs every benchmark suite in sequence.
+`pnpm bench:spatial` runs bounded GeoArrowBase scan,
+indexed small-candidate, and dirty-overlay query benches.
+`pnpm bench:vs-tanstack` runs the published head-to-head release bench
+against `@tanstack/db` for aggregate scan and memory-footprint
+comparisons. Latest local validation (Node 24):
+
+| bench                                               | mean     | budget |
+|-----------------------------------------------------|----------|--------|
+| insert 100k int32 rows                              | 60.5 ms  | 100 ms |
+| insert 10k utf8 rows                                | 12.7 ms  | 50 ms  |
+| filter+sort+paginate 10k (fn predicate, auto)       | 0.76 ms  | 1 ms   |
+| filter+sort+paginate 10k (fn predicate, fallback)   | 2.84 ms  | 5 ms   |
+| filter+sort+paginate 10k (vectorized DSL)           | 0.56 ms  | 1 ms   |
+| reactive notify (fn predicate, auto)                | 116 µs   | 150 µs |
+| reactive notify (fn predicate, fallback)            | 328 µs   | 500 µs |
+| reactive notify (vectorized DSL)                    | 94 µs    | 150 µs |
+| string contains 10k (dict_utf8, pre-resolved)       | 0.67 ms  | 1 ms   |
+| string contains 10k (utf8, per-row decode)          | 2.74 ms  | 5 ms   |
+| string LIKE 10k (utf8, simple-pattern specialization) | 3.22 ms | 5 ms   |
+| string startsWith 10k (dict_utf8)                   | 0.58 ms  | 1 ms   |
+| snapshot export + import 10k                        | 101.1 ms | info   |
+
+Insert benches use the raw column-buffer path so the regression guard tracks
+storage-kernel throughput separately from TanStack-compatible optimistic
+transaction overhead, which is covered by the transaction suites.
+
+Vectorized filter+sort and reactive-notify paths beat the PRD v2.1 §10
+aspirational targets (1 ms filter+sort, 100 µs reactive notify), while
+auto-lowered `.where(fn)` remains inside the release budget. Auto-lowering
+keeps the gap between common `.where(fn)` predicates and hand-written DSL
+small enough for ergonomic app code. On dict-encoded columns, the string
+predicates pre-resolve matching codes once and drop to a single `Set.has()` per
+row — ~4× faster than the utf8 per-row decode path.
+
+## Non-goals (see [`PRD.md §2.1`](./PRD.md))
+
+- Cross-tab *shared memory* (SAB is same-document only). Cross-tab
+  *state replication* is available via
+  [`arrowbase/broadcast`](#cross-tab-sync--arrowbasebroadcast)
+  (message-based, last-writer-wins).
+- Server-side durable SQL storage (use IndexedDB/localStorage client persistence, or bring your own server layer).
+- Full SQL (query surface is TanStack-DB-shaped).
+- Server-side use (browser + Node `worker_threads` only).
+- Arrow compute kernels (possible later, out of scope v1).
+
+## Future work
+
+- Nested `list<struct>` / `list<list>` / `struct<struct>`.
+- SAB-resident dictionary + list child for cross-worker visibility.
+
+## License
+
+MIT. See [`LICENSE`](./LICENSE).