@crvouga/sqlite-mem 1.1.1 → 1.2.0

package/AGENTS.md

@@ -124,13 +124,16 @@ SQLITE_MEM_FUZZ_SEED=12345 SQLITE_MEM_FUZZ_PATH='0:1' bun test tests/fuzz
 
 | Command | Role |
 | --- | --- |
-| `bun run test:sqlite-compat` | Requirements + fail-closed gate + contract/fuzz/harness |
+| `bun run test:sqlite-compat` | Requirements + fail-closed gate + construct catalog + 𝔇 + smoke ratchet + contract/fuzz/harness |
 | `bun run inventory` | Oracle `pragma_function_list` / modules vs memory registries |
+| `bun run scenarios` | Construct-level scenario catalog (`compat/scenarios.ts`) + 𝔇 / smoke gates |
 | `bun run requirements` | Refresh sqlite.org requirements → `compat/requirements.json` + `compat/coverage.json` |
 | `bun run fts-surface` | FTS oracle surface → `compat/fts-oracle-surface.json` |
 
 Statuses: **VERIFIED** / **PARTIALLY VERIFIED** / **UNSUPPORTED** / **NOT APPLICABLE**. Do not market PARTIAL as complete. Coverage evidence is directory paths (e.g. `tests/contract/joins/`), not automatic from test filenames.
 
+**Catalog vs proof:** `tests/contract/catalog/` IDs must execute; smoke (`SELECT 1 AS v`) is tracked in `compat/smoke-baseline.json`. Documented divergences bind to `compat/divergences.json`. Generated operator/CAST matrices: `tests/contract/matrices/`. Stateful dump-after-each fuzz: `tests/fuzz/stateful.test.ts`. Oracle `sqlite_version()` must be 3.51.0 or 3.53.0.
+
 Details: [COMPATIBILITY.md](COMPATIBILITY.md), audit: [COMPATIBILITY-AUDIT.md](COMPATIBILITY-AUDIT.md).
 
 ## Local gates

package/COMPATIBILITY-AUDIT.md

@@ -69,34 +69,47 @@ Prepared statements / errors / snapshot:
   VERIFIED — schema invalidation re-prepares; SQLM logical round-trip VERIFIED
 
   Differential tests:
-  Total: 727 under `bun test` (contract + fuzz + harness)
-  Passed: 727
+  Total: 872 under `bun test` (contract + fuzz + harness) after hardening pass
+  Passed: 872
   Failed: 0
 
 Stateful / fuzz:
   Seeds: 0x5a17e0e1 (+ SQLITE_MEM_FUZZ_SEED override)
   Combination fuzz: tests/fuzz/combinations-scope3.test.ts
   FTS fuzz: tests/fuzz/fts.test.ts
+  Collate / window / CTE+DML / JSON subtype fuzz extended
   Mismatches: 0
 
-New incompatibilities found & fixed (this pass):
-  1. Missing ~85 oracle builtins (math/string/date/window/uuid/ieee754/…)
-  2. Missing modules FTS3/4, RTREE, dbstat, bytecode, tables_used, fts5vocab
-  3. ANALYZE/REINDEX/VACUUM not executed
-  4. Row-value comparisons threw misuse instead of SQLite semantics
-  5. Inventory asserted builtins absent (Scope-3 inverted)
-  6. No requirements-matrix ingest / no test:sqlite-compat gate
-
-Remaining known differences:
-  Custom SQLM snapshots; deterministic random()/'now';
-  EXPLAIN/INDEXED BY stubs/no-ops; some PRAGMA storage no-ops;
+Hardening pass (2026-08-19) — incompatibilities found & fixed:
+  1. Comparison affinity (INTEGER/TEXT/NUMERIC column vs literal) missing
+  2. WITH on UPDATE/DELETE/INSERT VALUES dispatched as SELECT only
+  3. Plain INTEGER PRIMARY KEY never reused deleted max rowid
+  4. date() `weekday N` modifier unimplemented
+  5. REGEXP operator not parsed (now calls missing regexp() like oracle)
+  6. last_insert_rowid() stale inside AFTER INSERT triggers
+  7. INSTEAD OF triggers never fired; view DML rejected
+  8. INSERT/UPDATE OR ROLLBACK / OR FAIL not distinguished
+  9. Window GROUPS/RANGE frames, window FILTER, recursive CTE queue edges
+  10. Collation-aware GROUP BY; FTS3 matchinfo format variants; external-content delete
+
+Remaining known differences / intentional:
+  Custom SQLM snapshots; deterministic random()/'now' by default
+  (`random: "os"` / `now: "system"` match SQLite entropy and wall clock);
+  EXPLAIN stubs (shape contracts only); INDEXED BY no-op (documented);
+  ATTACH file path records filename but opens empty in-memory schema;
+  some PRAGMA storage no-ops; FTS shadow-table changes() diverge;
+  MATERIALIZED/NOT MATERIALIZED stored but both materialize today;
+  compile_options / function_list content is sqlite-mem's;
+  generate_series is sqlite-mem extension (not in bun:sqlite default);
   BigInt beyond Number.MAX_SAFE_INTEGER without bun safeIntegers;
-  NOT APPLICABLE C API / on-disk / VFS surfaces.
+  NOT APPLICABLE C API / on-disk / VFS surfaces; uri.html remapped N/A.
 
 Final assessment:
   Verified against SQLite 3.51.0 (bun:sqlite). Oracle function/module
   inventory is closed (0 missing). Requirements matrix ingested with
   zero unknown statuses. Gate: `bun run test:sqlite-compat`.
+  Hardening pass closed construct-level P0 gaps from docs/PARITY-GAPS.md;
+  FTS/EXPLAIN remain PARTIALLY VERIFIED honestly.
 ```
 
 ---
@@ -144,10 +157,11 @@ Special commands verified:
   delete-all / merge / automerge: error parity with oracle where probed
 
 Differential tests:
-  Passed: 656 (contract + fuzz + harness)
+  Passed: 872 (contract + fuzz + harness)
   Failed: 0
   FTS contract: tests/contract/fts/basic.test.ts,
                 tests/contract/fts/comprehensive.test.ts
+                (+ matchinfo formats, external-content, trigger maintenance)
 
 Fuzz cases:
   Generated: fast-check seed 0x5a17e0e1 (override SQLITE_MEM_FUZZ_SEED)

package/COMPATIBILITY.md

@@ -8,7 +8,15 @@ bun run test:sqlite-compat
 
 See [COMPATIBILITY-AUDIT.md](COMPATIBILITY-AUDIT.md) for the latest evidence-based audit report.
 
-Reference oracle: **SQLite 3.51.0** (`bun:sqlite`). Inventory: `bun run inventory`. Requirements matrix: `bun run requirements` → `compat/requirements.json` + `compat/coverage.json`.
+Reference oracle: **SQLite 3.51.0** (`bun:sqlite`; Linux/Windows bun may report **3.53.0** — see 𝔇 `oracle-platform-sqlite-version`). Inventory: `bun run inventory`. Construct catalog: `bun run scenarios` → [`compat/scenarios.ts`](compat/scenarios.ts). Divergences: [`compat/divergences.json`](compat/divergences.json). Requirements matrix: `bun run requirements` → `compat/requirements.json` + `compat/coverage.json`.
+
+## Proof surface (Phase 1)
+
+Differential tests compare a **B-tuple**: rows (plus `typeof` where requested), column names, error category / sqliteCode / message (Tier A exact or Tier B prefix-normalized), `changes`, `total_changes`, `lastInsertRowid`, and autocommit, plus a **logical Dump** (`sqlite_master` names, `table_info`, row payloads with per-column `typeof`, `sqlite_sequence`, selected pragmas).
+
+A catalog ID appearing in a test file is **not** proof by itself. Cases whose SQL is `SELECT 1 AS v` are **smoke**; [`compat/smoke-baseline.json`](compat/smoke-baseline.json) ratchets that list downward. Generated matrices live under [`tests/contract/matrices/`](tests/contract/matrices/). Observed mem≠oracle diffs must be `known-divergence(id)` from 𝔇 or **FAILURE** — unexplained diffs are not allowed.
+
+Intentional differences are finite and machine-readable in `compat/divergences.json` (SQLM snapshots, seeded `random()`/`now`, ATTACH empty schema, EXPLAIN stubs, INDEXED BY discarded, MATERIALIZED hint ignored, FTS shadow counters, compile_options/function_list, `-0`, JS API extras, snapshot exclusions).
 
 ## Status vocabulary
 
@@ -36,9 +44,9 @@ Oracle builtins (math, string extras, uuid, ieee754, …) and modules (FTS3/4/5,
 | Core DML / SELECT / joins / CTE / UPSERT / RETURNING | VERIFIED | Contract + fuzz |
 | STRICT tables / indexes | VERIFIED | STRICT types; partial + expression indexes; leftmost prefix |
 | Expressions / operators / `->` `->>` / row values | VERIFIED | Row-value + precedence contracts |
-| Affinity / NULL / COLLATE | VERIFIED | |
-| Constraints / FK / triggers / views / ATTACH | VERIFIED | Deferred FK, composite FK |
-| Windows (incl. ntile/cume_dist/percent_rank) | VERIFIED | EXCLUDE NO OTHERS/CURRENT ROW/GROUP/TIES |
+| Affinity / NULL / COLLATE | VERIFIED | Comparison affinity + collation on GROUP BY/JOIN |
+| Constraints / FK / triggers / views / ATTACH | VERIFIED | Deferred FK, composite FK, INSTEAD OF, OR ROLLBACK/FAIL |
+| Windows (incl. ntile/cume_dist/percent_rank) | VERIFIED | EXCLUDE; GROUPS/RANGE frames; window FILTER |
 | JSON1 / JSONB / TVFs | VERIFIED | |
 | Math / string / date extras / uuid / ieee754 | VERIFIED | Scope-3 inventory |
 | FTS3 / FTS4 / FTS5 + MATCH | PARTIALLY VERIFIED | Differential FTS suite + fuzz vs 3.51.0; see FTS matrix below. Shadow-table change counters intentionally diverge. |
@@ -62,11 +70,11 @@ Reference: **SQLite 3.51.0** (`bun:sqlite`). Inventory: `bun run scripts/fts-ora
 | Column filters | VERIFIED |
 | Ranking / bm25 / rank | VERIFIED |
 | highlight / snippet | VERIFIED |
-| matchinfo / offsets (FTS3/4) | PARTIALLY VERIFIED |
+| matchinfo / offsets (FTS3/4) | PARTIALLY VERIFIED | Default + common format strings verified; some FTS4-only formats thinner |
 | Contentless tables | VERIFIED |
-| External content | PARTIALLY VERIFIED |
+| External content | PARTIALLY VERIFIED | Canonical delete/sync covered; backfill/projection edges thinner |
 | Content tables | VERIFIED |
-| Triggers + FTS | PARTIALLY VERIFIED |
+| Triggers + FTS | PARTIALLY VERIFIED | Maintenance sequences covered; advanced edges thinner |
 | Special commands (optimize/rebuild/integrity-check) | VERIFIED |
 | Prefix indexes | VERIFIED |
 | Unicode / adversarial corpus | VERIFIED |
@@ -76,7 +84,7 @@ Reference: **SQLite 3.51.0** (`bun:sqlite`). Inventory: `bun run scripts/fts-ora
 | FTS stateful fuzz | VERIFIED |
 | RTREE / dbstat / bytecode / tables_used | VERIFIED | dbstat synthetic pages; bytecode empty cursor |
 | ANALYZE / REINDEX / VACUUM | VERIFIED | `:memory:` observable parity |
-| EXPLAIN / INDEXED BY | PARTIALLY VERIFIED | Stub shapes / no-ops |
+| EXPLAIN / INDEXED BY | PARTIALLY VERIFIED | Stub shapes / no-ops (missing INDEXED BY errors documented) |
 | Prepared stmt schema invalidation | VERIFIED | Re-prepare after ALTER/DROP; `tests/contract/api/schema-invalidation.test.ts` |
 | On-disk file format / C API | NOT APPLICABLE | |
 

package/README.md

@@ -76,8 +76,9 @@ From the repo root after that install: `bun run example`.
 import { Database, SqliteError } from "@crvouga/sqlite-mem";
 
 interface DatabaseOptions {
-  seed?: number | bigint;           // default 1 — PRNG for random() / randomblob()
-  now?: Date | (() => Date);        // default 2000-01-01T00:00:00.000Z
+  seed?: number | bigint;                 // default 1 — ignored when random is "os"
+  random?: "deterministic" | "os";        // default "deterministic"; "os" is CSPRNG like SQLite
+  now?: Date | (() => Date) | "system";   // default 2000-01-01T00:00:00.000Z; "system" is wall clock
 }
 
 interface Database {
@@ -171,7 +172,7 @@ Duplicate column names collapse in row objects (last write wins). Use `stmt.resu
 - Round-trips ordinary tables, views, indexes, change counters, PRNG state, and clock.
 - **Not** encoded: triggers, ATTACH’d schemas, virtual tables (FTS / RTREE / …), `userVersion`.
 - Cannot `restore()` while a transaction is open.
-- `restore()` replaces `now` with a fixed clock from the snapshot (a live `() => Date` is overwritten).
+- `restore()` replaces `now` with a fixed clock from the snapshot (a live `() => Date` is overwritten). `{ now: "system" }` stays live after restore.
 - Equivalent databases produce byte-identical snapshots (schema/rows sorted) **within a single library version**.
 - **Compatibility policy:** newer library versions can always restore older snapshots; older libraries cannot restore newer format versions (`snapshot_version` / `SQLITE_FORMAT`). Corrupt magic yields a distinct error.
 
@@ -181,8 +182,8 @@ The engine is deterministic by default. Invariants:
 
 | Source | Default | Override / notes |
 | --- | --- | --- |
-| `random()` / `randomblob()` | Seeded xorshift64* (`seed: 1`) | `new Database({ seed })` |
-| `date('now')` / friends | Fixed `2000-01-01T00:00:00.000Z` | `new Database({ now: Date \| (() => Date) })` |
+| `random()` / `randomblob()` | Seeded xorshift64* (`seed: 1`) | `new Database({ seed })` or `{ random: "os" }` for CSPRNG (not rolled back / not restored) |
+| `date('now')` / friends | Fixed `2000-01-01T00:00:00.000Z` | `new Database({ now: Date \| (() => Date) \| "system" })` — `"system"` is wall clock and is **not** frozen by `restore()` |
 | Table scans | Rowid order | Same order after `snapshot`/`restore` |
 | Snapshots | Sorted schema/rows + PRNG state + clock | Restored into PRNG and `now` |
 | Transactions | PRNG rolls back with `ROLLBACK`/`SAVEPOINT` | Matches data rollback |
@@ -209,14 +210,19 @@ The exports of the main entry (`@crvouga/sqlite-mem`) are **frozen**:
 
 Goal: drop-in SQL behavior vs SQLite **3.51.0**. Full matrix: [COMPATIBILITY.md](COMPATIBILITY.md).
 
-**Intentional differences:** custom `SQLM` snapshots; seeded `random()` / fixed `'now'`; no C API / on-disk DB / VFS.
+**Intentional differences:** custom `SQLM` snapshots; seeded `random()` / fixed `'now'` by default (`{ random: "os" }` / `{ now: "system" }` match SQLite entropy and wall clock); no C API / on-disk DB / VFS.
 
 **Know these thin or partial areas** (do not assume full oracle fidelity):
 
 - FTS3/4/5 — largely implemented; shadow-table change counters intentionally diverge; some edges partial
 - `EXPLAIN` / `EXPLAIN QUERY PLAN` — stub shapes, not real bytecode
-- `INDEXED BY` / `NOT INDEXED` — parsed and discarded
-- Unknown statement `PRAGMA` succeeds with an empty result (SQLite-like). All oracle-exposed `pragma_*` eponymous TVFs are supported (`SELECT * FROM pragma_table_info('t')`, bare `FROM pragma_database_list`, …), including **correlated** args such as `FROM table_list AS tl, pragma_table_info(tl.name) AS p` (Kysely SQLite introspector). Storage/journal getters return bun `:memory:`-compatible defaults.
+- `INDEXED BY` / `NOT INDEXED` — parsed and discarded (missing indexes do not error)
+- `ATTACH 'file'` — filename is recorded; schema is always a new empty in-memory database
+- `MATERIALIZED` / `NOT MATERIALIZED` — both execute as materialized
+- `PRAGMA compile_options` / `function_list` — sqlite-mem’s set, not Bun’s native build
+- Unknown statement `PRAGMA` succeeds with an empty result (SQLite-like). All oracle-exposed `pragma_*` eponymous TVFs are supported (`SELECT * FROM pragma_table_info('t')`, bare `FROM pragma_database_list`, …), including **correlated** args such as `FROM table_list AS tl, pragma_table_info(tl.name) AS p` (Kysely SQLite introspector). Storage/journal getters return bun `:memory:`-compatible defaults. `PRAGMA case_sensitive_like` is implemented.
+
+**Also supported (oracle-parity):** boolean literals **`TRUE` / `FALSE`** (any case → integers `1` / `0`) and **`IS [NOT] TRUE` / `IS [NOT] FALSE`** (SQLite truthiness, including NULL). A column named `true`/`false` shadows the literal.
 
 ## Common pitfalls
 
@@ -224,15 +230,15 @@ Goal: drop-in SQL behavior vs SQLite **3.51.0**. Full matrix: [COMPATIBILITY.md]
 2. **No named-object binds and no sticky `bind()`** — pass positional rest args / arrays in declaration order to `query` / `run` / `all` / `get` / `result`.
 3. **`query` / `prepare` are single-statement only** — multi-statement scripts belong in `exec()` (which does not take bind parameters).
 4. **`exec` returns `void` and takes no params** — use `db.prepare(…).run(…)` or `db.query(…)` for binds; use `db.changes` / `stmt.run()` for counters.
-5. **`'now'` is not wall-clock** unless you pass `{ now: () => new Date() }`. Default is year 2000.
-6. **`random()` is seeded**, not OS entropy; snapshots restore the PRNG.
+5. **`'now'` is not wall-clock** unless you pass `{ now: "system" }` or `{ now: () => new Date() }`. Default is year 2000. `restore()` freezes a snapshot clock except when constructed with `"system"`.
+6. **`random()` is seeded**, not OS entropy, unless you pass `{ random: "os" }`. Snapshots restore the seeded PRNG; OS entropy is not rewound.
 7. **Snapshots are not `.sqlite` files** and do not round-trip FTS / triggers / ATTACH.
 8. **No better-sqlite3 extras** — no `iterate`, `pluck`/`raw`, `safeIntegers` option, `pragma()` helper, `loadExtension`, or SQLite-file `serialize()`.
 9. **Do not bind `Date` objects** — store unixepoch integers or ISO text. Do not bind `DataView` / non-`Uint8Array` typed arrays.
 10. **Do not use `Number.isInteger` for SQL REAL vs INTEGER** — use SQL `typeof()`.
 11. **Do not import `@crvouga/sqlite-mem/unstable` in application code** unless you accept breakage in any release.
 
-Working examples beyond this README: `examples/react-vite`, `tests/contract/api/`, `tests/contract/parameters/`, `tests/browser/run.ts`.
+Working examples beyond this README: `examples/react-vite`, `tests/contract/api/`, and `tests/contract/parameters/`.
 
 ## Development
 
@@ -250,7 +256,6 @@ bun run typecheck
 bun run test:sqlite-compat   # requirements + inventory gate + differential suite
 bun test                     # contract + fuzz + harness
 bun run build
-bun run test:browser         # Playwright smoke (Chrome/Firefox/Safari; CI uses Chromium)
 ```
 
 See [COMPATIBILITY.md](./COMPATIBILITY.md).