@crvouga/sqlite-mem 0.1.0

package/COMPATIBILITY-AUDIT.md

@@ -0,0 +1,199 @@
+# SQLite3 Compatibility Audit
+
+```text
+SQLite3 Compatibility Audit
+===========================
+
+Reference SQLite version:
+  bun:sqlite (SQLite 3.51.0)
+
+Reference SQLite compile options:
+  ENABLE_MATH_FUNCTIONS, ENABLE_FTS3/4/5, ENABLE_RTREE, ENABLE_DBSTAT_VTAB,
+  ENABLE_BYTECODE_VTAB, JSON builtin (not OMIT_JSON), plus Bun/SEE codec flags.
+  Full list: `bun run inventory`
+
+sqlite-mem version:
+  0.1.0
+
+Scope:
+  Scope 3 — every oracle-exposed SQL builtin/module is in-scope except
+  NOT APPLICABLE (C API, on-disk format, VFS/pager/WAL).
+
+SQLite requirements reviewed:
+  3487 (from sqlite.org requirements matrix → compat/requirements.json)
+
+SQLite requirements classification:
+  NOT APPLICABLE:     see compat/coverage.json counts.notApplicable
+  SQL_BEHAVIOR:       see counts.sqlBehavior
+  unknown:            0 (gate fails if non-zero)
+
+Coverage statuses (SQL_BEHAVIOR):
+  VERIFIED / PARTIALLY_VERIFIED / UNSUPPORTED — see `bun run requirements`
+  Regenerated via scripts/sqlite-requirements.ts + coverage upgrades.
+
+SQL grammar / operators / expressions:
+  VERIFIED — contracts + row-value comparisons + ->/->> precedence
+
+Types / affinity / NULL:
+  VERIFIED
+
+Functions (oracle surface):
+  VERIFIED — inventory missingOracleFunctions = 0 (163 names covered)
+
+JSON / JSONB:
+  VERIFIED
+
+Aggregates / windows:
+  VERIFIED — includes string_agg, ntile, cume_dist, percent_rank
+  (window EXCLUDE still thinner)
+
+CTEs / transactions / savepoints / constraints / FK / triggers:
+  VERIFIED (deferred FK thinner → PARTIAL edges)
+
+Indexes / views / generated / STRICT / WITHOUT ROWID:
+  VERIFIED / PARTIAL per COMPATIBILITY.md
+
+PRAGMAs:
+  PARTIALLY VERIFIED — schema/FK; storage pragmas N/A or :memory: no-op
+
+ATTACH/DETACH:
+  VERIFIED (in-memory schemas; temp.schema = main state)
+
+Virtual tables / extensions:
+  VERIFIED for oracle modules: fts3/4/5, fts5vocab, rtree/rtree_i32,
+  dbstat, bytecode, tables_used (bytecode/tables_used empty cursors)
+
+ANALYZE / REINDEX / VACUUM:
+  VERIFIED (:memory: observable parity)
+
+Prepared statements / errors / snapshot:
+  PARTIAL / VERIFIED — schema invalidation thin; SQLM logical round-trip VERIFIED
+
+Differential tests:
+  Total: 656 under `bun test` (contract + fuzz + harness)
+  Passed: 656
+  Failed: 0
+
+Stateful / fuzz:
+  Seeds: 0x5a17e0e1 (+ SQLITE_MEM_FUZZ_SEED override)
+  Combination fuzz: tests/fuzz/combinations-scope3.test.ts
+  FTS fuzz: tests/fuzz/fts.test.ts
+  Mismatches: 0
+
+New incompatibilities found & fixed (this pass):
+  1. Missing ~85 oracle builtins (math/string/date/window/uuid/ieee754/…)
+  2. Missing modules FTS3/4, RTREE, dbstat, bytecode, tables_used, fts5vocab
+  3. ANALYZE/REINDEX/VACUUM not executed
+  4. Row-value comparisons threw misuse instead of SQLite semantics
+  5. Inventory asserted builtins absent (Scope-3 inverted)
+  6. No requirements-matrix ingest / no test:sqlite-compat gate
+
+Remaining known differences:
+  Custom SQLM snapshots; deterministic random()/'now';
+  EXPLAIN/INDEXED BY stubs/no-ops; some PRAGMA storage no-ops;
+  Window EXCLUDE / deferred FK / STRICT / prepared invalidation thinner;
+  BigInt beyond Number.MAX_SAFE_INTEGER without bun safeIntegers;
+  NOT APPLICABLE C API / on-disk / VFS surfaces.
+
+Final assessment:
+  Verified against SQLite 3.51.0 (bun:sqlite). Oracle function/module
+  inventory is closed (0 missing). Requirements matrix ingested with
+  zero unknown statuses. Gate: `bun run test:sqlite-compat`.
+```
+
+---
+
+## SQLite Full-Text Search Compatibility Audit
+
+```text
+SQLite Full-Text Search Compatibility Audit
+==========================================
+
+Reference SQLite:
+  version: 3.51.0 (bun:sqlite)
+  source_id: 2025-06-12 13:14:41 f0ca7bba1c5e232e5d279fad6338121ab55af0c8c68c84cdfb18ba5114dcaapl
+  compile options: ENABLE_FTS3, ENABLE_FTS3_PARENTHESIS, ENABLE_FTS3_TOKENIZER,
+                   ENABLE_FTS4, ENABLE_FTS5
+  inventory: compat/fts-oracle-surface.json (bun run scripts/fts-oracle-surface.ts)
+
+FTS3: PARTIALLY VERIFIED (MATCH, snippet, offsets; matchinfo formats thinner)
+FTS4: PARTIALLY VERIFIED (same surface as FTS3 for tested paths)
+FTS5: PARTIALLY VERIFIED overall — core MATCH/tokenizers/ranking/aux VERIFIED;
+      external-content + full matchinfo format strings still thinner
+
+Tokenizers verified:
+  unicode61 (incl. remove_diacritics 0/1/2), ascii, porter,
+  porter unicode61, porter ascii, trigram
+
+MATCH grammar verified:
+  terms, AND/OR/NOT, implicit AND, phrases, prefix *, column filters
+  (col : term / {cols} :), NEAR / NEAR(…, N), parentheses, NEAR-as-term
+
+Ranking verified:
+  bm25() / rank column — order + scores vs oracle (1e-15 abs epsilon for ULP)
+
+Auxiliary functions verified:
+  bm25, highlight, snippet (FTS5); snippet, offsets (FTS3/4);
+  matchinfo present (default format PARTIAL)
+
+Content modes verified:
+  normal content tables: VERIFIED
+  contentless (content=''): VERIFIED
+  external content: PARTIALLY VERIFIED (CREATE accepted; sync/triggers thinner)
+
+Special commands verified:
+  optimize, rebuild, integrity-check (success + post-command MATCH)
+  delete-all / merge / automerge: error parity with oracle where probed
+
+Differential tests:
+  Passed: 656 (contract + fuzz + harness)
+  Failed: 0
+  FTS contract: tests/contract/fts/basic.test.ts,
+                tests/contract/fts/comprehensive.test.ts
+
+Fuzz cases:
+  Generated: fast-check seed 0x5a17e0e1 (override SQLITE_MEM_FUZZ_SEED)
+  Files: tests/fuzz/fts.test.ts (MATCH queries, tokenizers, stateful DML)
+  Mismatches: 0 (after trigram phrase expansion + NEAR-as-term fixes)
+
+Stateful cases:
+  Operations: INSERT/UPDATE/DELETE/MATCH sequences + txn/savepoint contracts
+  Mismatches: 0
+
+Compatibility gaps found:
+  1. Toy AND-token matcher (replaced)
+  2. Aux functions always threw (wired MATCH cursor + real bm25/highlight/snippet)
+  3. No CREATE option parsing (tokenize=/content=/prefix=/UNINDEXED)
+  4. No FTS5 query language / NEAR / phrases / column filters
+  5. False VERIFIED status on fts5.html / fts3.html (downgraded then re-evidenced)
+
+Compatibility gaps fixed:
+  Positional inverted index; unicode61/ascii/porter/trigram tokenizers;
+  FTS5 query parser; BM25 with SQLite IDF floor 1e-6; highlight/snippet;
+  contentless mode; special commands; FTS3/4 MATCH + snippet/offsets;
+  comparator realEpsilon for ranking ULP only; FTS fuzz suites
+
+Regression tests added:
+  tests/contract/fts/comprehensive.test.ts
+  tests/fuzz/fts.test.ts
+  tests/harness/fts-compare.ts
+  scripts/fts-oracle-surface.ts
+
+Remaining unsupported / thinner functionality:
+  - SQLite FTS shadow tables not mirrored (change counters diverge; documented)
+  - External-content sync/stale-index edge cases thinner
+  - FTS3 matchinfo format-string variants not exhaustively verified
+  - fts3tokenize / fts4aux still thin CREATE stubs
+  - locale / tokendata / detail=none advanced options accepted but lightly tested
+```
+
+Verification commands:
+
+```bash
+bun run test:sqlite-compat
+bun run scripts/fts-oracle-surface.ts
+bun test tests/contract/fts tests/fuzz/fts.test.ts
+bun run inventory
+bun run requirements
+bun run typecheck
+```

package/COMPATIBILITY.md

@@ -0,0 +1,93 @@
+# Compatibility
+
+Goal: **full SQLite3 SQL dialect behavioral parity** as a drop-in for the same statements against the reference oracle. Compatibility is proven by the differential contract suite and the fail-closed gate:
+
+```bash
+bun run test:sqlite-compat
+```
+
+See [COMPATIBILITY-AUDIT.md](COMPATIBILITY-AUDIT.md) for the latest evidence-based audit report.
+
+Reference oracle: **SQLite 3.51.0** (`bun:sqlite`). Inventory: `bun run inventory`. Requirements matrix: `bun run requirements` → `compat/requirements.json` + `compat/coverage.json`.
+
+## Status vocabulary
+
+| Status | Meaning |
+| --- | --- |
+| **VERIFIED** | Differential contracts (+ fuzz where applicable) cover happy path **and** meaningful edges vs oracle |
+| **PARTIALLY VERIFIED** | Implemented; coverage thin or known edges remain |
+| **UNSUPPORTED** | Missing from SQL surface (must fail loud; gate fails if oracle-exposed) |
+| **NOT APPLICABLE** | Outside the in-memory dialect surface (C API, on-disk `.sqlite`, VFS/pager/WAL) |
+
+## Scope 3 bound
+
+Anything a SQLite application can invoke through SQL against this Bun/SQLite **3.51.0** build must match oracle observable behavior, except:
+
+1. **Snapshot format** — custom binary codec (`SQLM`), not the on-disk SQLite database file format (logical state still round-trips).
+2. **Deterministic `random()` / `'now'`** — seeded PRNG and fixed clock by default (injectable).
+3. **NOT APPLICABLE** rows in `compat/coverage.json` (C API, VFS, pager, file locking, etc.).
+
+Oracle builtins (math, string extras, uuid, ieee754, …) and modules (FTS3/4/5, RTREE, dbstat, bytecode, tables_used) are **in scope**.
+
+## Feature matrix (summary)
+
+| Area | Status | Notes |
+| --- | --- | --- |
+| Core DML / SELECT / joins / CTE / UPSERT / RETURNING | VERIFIED | Contract + fuzz |
+| Expressions / operators / `->` `->>` / row values | VERIFIED | Row-value + precedence contracts |
+| Affinity / NULL / COLLATE | VERIFIED | |
+| Constraints / FK / triggers / views / ATTACH | VERIFIED | Deferred FK thinner → PARTIAL edges |
+| Windows (incl. ntile/cume_dist/percent_rank) | VERIFIED | EXCLUDE still thinner |
+| JSON1 / JSONB / TVFs | VERIFIED | |
+| Math / string / date extras / uuid / ieee754 | VERIFIED | Scope-3 inventory |
+| FTS3 / FTS4 / FTS5 + MATCH | PARTIALLY VERIFIED | Differential FTS suite + fuzz vs 3.51.0; see FTS matrix below. Shadow-table change counters intentionally diverge. |
+
+## FTS compatibility matrix
+
+Reference: **SQLite 3.51.0** (`bun:sqlite`). Inventory: `bun run scripts/fts-oracle-surface.ts` → `compat/fts-oracle-surface.json`.
+
+| Feature | Status |
+| --- | --- |
+| FTS3 | PARTIALLY VERIFIED |
+| FTS4 | PARTIALLY VERIFIED |
+| FTS5 | PARTIALLY VERIFIED |
+| Virtual table creation (options/tokenizers) | VERIFIED |
+| Tokenizers (unicode61/ascii/porter/trigram) | VERIFIED |
+| MATCH grammar (AND/OR/NOT/phrase/prefix/NEAR/columns) | VERIFIED |
+| Boolean operators | VERIFIED |
+| Phrases | VERIFIED |
+| Prefix queries | VERIFIED |
+| NEAR | VERIFIED |
+| Column filters | VERIFIED |
+| Ranking / bm25 / rank | VERIFIED |
+| highlight / snippet | VERIFIED |
+| matchinfo / offsets (FTS3/4) | PARTIALLY VERIFIED |
+| Contentless tables | VERIFIED |
+| External content | PARTIALLY VERIFIED |
+| Content tables | VERIFIED |
+| Triggers + FTS | PARTIALLY VERIFIED |
+| Special commands (optimize/rebuild/integrity-check) | VERIFIED |
+| Prefix indexes | VERIFIED |
+| Unicode / adversarial corpus | VERIFIED |
+| Transactions / savepoints | VERIFIED |
+| Error behavior | VERIFIED |
+| FTS differential fuzz | VERIFIED |
+| FTS stateful fuzz | VERIFIED |
+| RTREE / dbstat / bytecode / tables_used | VERIFIED | dbstat synthetic pages; bytecode empty cursor |
+| ANALYZE / REINDEX / VACUUM | VERIFIED | `:memory:` observable parity |
+| EXPLAIN / INDEXED BY | PARTIALLY VERIFIED | Stub shapes / no-ops |
+| Prepared stmt schema invalidation | PARTIALLY VERIFIED | Thin |
+| On-disk file format / C API | NOT APPLICABLE | |
+
+## How to verify
+
+```bash
+bun run test:sqlite-compat   # requirements + gate + contract/fuzz/harness
+bun run inventory            # oracle function/module inventory
+bun run requirements         # refresh SQLite.org requirements + coverage
+bun run test:browser         # Playwright smoke (not the SQL oracle)
+```
+
+Do not treat isolated unit tests of internal modules as proof of SQLite compatibility. The differential matrix runner is authoritative for SQL behavior; `test:sqlite-compat` is the release gate.
+
+**Parity claim:** Verified against **SQLite 3.51.0** (`bun:sqlite`). Features marked **VERIFIED** are oracle-proven. **PARTIALLY VERIFIED** rows must not be marketed as complete. **NOT APPLICABLE** is the only allowed permanent omission from the SQL drop-in claim.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Chris Vouga
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,171 @@
+# sqlite-mem
+
+Pure TypeScript, completely in-memory SQLite implementation aiming for **full SQLite3 SQL dialect parity** (same statements → same results).
+
+- Runs in modern browsers and Node.js / Bun
+- **Zero** WASM, native bindings, workers, or filesystem dependencies
+- Entire database stored in memory
+- **Verified against SQLite 3.51.0** (`bun:sqlite`) via differential contracts + fail-closed gate
+- Intentional differences: deterministic `random()` / `'now'` by default, and a custom snapshot format (not `.sqlite` files)
+
+See [COMPATIBILITY.md](COMPATIBILITY.md) for the matrix and [COMPATIBILITY-AUDIT.md](COMPATIBILITY-AUDIT.md) for the audit report.
+
+## Install
+
+```bash
+bun add @crvouga/sqlite-mem
+# or
+npm install @crvouga/sqlite-mem
+```
+
+## Usage
+
+```ts
+import { Database } from "@crvouga/sqlite-mem";
+
+const db = new Database();
+
+db.exec(`
+  CREATE TABLE users (
+    id INTEGER PRIMARY KEY,
+    name TEXT NOT NULL
+  )
+`);
+
+db.exec(`INSERT INTO users (name) VALUES (?)`, ["Alice"]);
+
+const users = db.query<{ id: number; name: string }>(`SELECT * FROM users`);
+console.log(users);
+
+const snap = db.snapshot();
+const db2 = new Database();
+db2.restore(snap);
+```
+
+## API
+
+```ts
+interface Database {
+  // seed default 1; now default 2000-01-01T00:00:00.000Z
+  constructor(options?: { seed?: number | bigint; now?: Date | (() => Date) });
+  exec(sql: string, params?: unknown[]): void;
+  query<T = Record<string, unknown>>(sql: string, params?: unknown[]): T[];
+  prepare(sql: string): Statement;
+  transaction<T>(fn: () => T): T;
+  snapshot(): Uint8Array;
+  restore(snapshot: Uint8Array): void;
+  close(): void;
+}
+
+interface Statement {
+  bind(...params: unknown[]): Statement;
+  run(...params: unknown[]): { changes: number; lastInsertRowid: number | bigint };
+  all<T = Record<string, unknown>>(...params: unknown[]): T[];
+  get<T = Record<string, unknown>>(...params: unknown[]): T | undefined;
+}
+```
+
+## Determinism
+
+The engine is deterministic by default. Invariants:
+
+| Source | Default | Override / notes |
+| --- | --- | --- |
+| `random()` / `randomblob()` | Seeded xorshift64* (`seed: 1`) | `new Database({ seed })` or `{ prng }` |
+| `date('now')` / friends | Fixed `2000-01-01T00:00:00.000Z` | `new Database({ now: Date \| (() => Date) })` |
+| Table scans | Rowid order | Same order after `snapshot`/`restore` |
+| Snapshots | Sorted schema/rows + PRNG state + clock | Restored into PRNG and `now` |
+| Transactions | PRNG rolls back with `ROLLBACK`/`SAVEPOINT` | Matches data rollback |
+| Numbers | IEEE `-0` canonicalized to `+0` | Bind, affinity, and arithmetic |
+
+Fuzz / property tests use a fixed seed (`0x5a17e0e1`) and print it on failure:
+
+```bash
+bun test tests/fuzz
+SQLITE_MEM_FUZZ_SEED=12345 bun test tests/fuzz
+SQLITE_MEM_FUZZ_SEED=12345 SQLITE_MEM_FUZZ_PATH='0:1' bun test tests/fuzz  # exact replay
+```
+
+## Development
+
+Requires [Bun](https://bun.sh).
+
+```bash
+bun install
+bun run check                # format + lint + typecheck + sqlite-compat suite
+bun run format               # write Biome formatting
+bun run lint                 # Biome lint
+bun run typecheck
+bun run test:sqlite-compat   # requirements + inventory gate + differential suite
+bun test                     # contract + fuzz + harness
+bun run build
+bun run test:browser         # Playwright smoke (Chrome/Firefox/Safari)
+```
+
+Contract tests compare the pure TypeScript engine against real SQLite (`bun:sqlite`). See [COMPATIBILITY.md](./COMPATIBILITY.md).
+
+## Releasing
+
+Publishing is fully automated. You never bump `version` or run `npm publish` by hand.
+
+### How a release happens
+
+1. Push or merge to `main` with [Conventional Commits](https://www.conventionalcommits.org/).
+2. CI runs commitlint, format/lint/typecheck, build, package verification, tests, browser smoke, and benchmarks.
+3. If every gate is green, [semantic-release](https://semantic-release.gitbook.io/) analyzes commits since the last git tag, bumps semver, publishes to npm, and creates a GitHub Release.
+
+| Commit | Version bump |
+| --- | --- |
+| `fix: …` | patch (`0.1.0` → `0.1.1`) |
+| `feat: …` | minor (`0.1.0` → `0.2.0`) |
+| `feat!: …` or `BREAKING CHANGE:` footer | major (`0.2.0` → `1.0.0`) |
+| `docs:`, `chore:`, `refactor:`, `test:`, … | no release |
+
+Examples:
+
+```text
+feat: add window function support
+fix: handle NULL in UNIQUE constraints
+feat!: rename snapshot() return type
+
+chore: tweak CI timeouts
+docs: clarify determinism table
+```
+
+PR titles must also follow Conventional Commits (enforced in CI). Prefer squash merges with a conventional title.
+
+Local checks:
+
+```bash
+bunx commitlint --last --verbose
+bun run build && bun run verify-package
+# dry-run needs a GitHub token for API calls; CI publish uses Trusted Publishing (no NPM_TOKEN)
+bun run release:dry-run
+```
+
+`package.json` version is `0.0.0-development` on purpose — **git tags** (`v0.1.0`, …) are the source of truth.
+
+### One-time setup (maintainers)
+
+Do this once so CI can publish. Full checklist: **[docs/SECRETS.md](./docs/SECRETS.md)**.
+
+1. **Create the package on npm (once), then Trusted Publishing.** If https://www.npmjs.com/package/@crvouga/sqlite-mem 404s:
+
+   ```bash
+   npm login --auth-type=web
+   bun run npm:seed -- --yes
+   ```
+
+   npm does not email a publish code — complete 2FA in the browser or authenticator app.
+
+   Then on [package Access](https://www.npmjs.com/package/@crvouga/sqlite-mem/access) → Trusted Publisher → GitHub Actions (`crvouga/sqlite-mem`, workflow `ci.yml`). Do **not** create an Automation / granular access token for CI.
+2. Confirm GitHub Actions is enabled and can create releases (default `GITHUB_TOKEN` is enough with this workflow’s permissions). No `NPM_TOKEN` repo secret.
+3. Ensure the baseline tag exists and is pushed: `v0.1.0` (semver continues from there; the next `feat` publishes `0.2.0`).
+
+Validate the checklist anytime with `bun run secrets:doctor`.
+
+After that, every green push to `main` with releasable commits updates npm automatically.
+
+## License
+
+MIT