sqlrite 0.1.25__tar.gz → 0.2.0__tar.gz

sqlrite-0.2.0/CLAUDE.md

@@ -0,0 +1,94 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project
+
+SQLRite is a from-scratch SQLite-style embedded database written in Rust. It's published on crates.io as `sqlrite-engine` (imported as `use sqlrite::…` — the lib target keeps the short name) and ships as: a REPL binary (`sqlrite`), a Tauri 2 + Svelte 5 desktop app, a Model Context Protocol stdio server (`sqlrite-mcp`), a C FFI shim (`sqlrite-ffi`), and language SDKs (Python via PyO3, Node via napi-rs, Go via cgo, WASM via wasm-bindgen). Phases 1–7 are shipped; the current branch `phase-8-plan` drafts inverted-index + BM25 full-text search and hybrid retrieval.
+
+## Workspace layout
+
+`Cargo.toml` is a workspace whose members are: `.` (the engine, package `sqlrite-engine`, lib `sqlrite`), `desktop/src-tauri`, `sqlrite-ffi`, `sqlrite-ask`, `sqlrite-mcp`, `sdk/python`, `sdk/nodejs`. `sdk/wasm` and `sdk/go` are deliberately **not** workspace members (wasm32 target / cgo separation).
+
+- `src/` — engine. Public API is `Connection`/`Statement`/`Rows`/`Row`/`Value` from [src/connection.rs](src/connection.rs), re-exported via [src/lib.rs](src/lib.rs). Any new SDK should bind only to this surface.
+- `sqlrite-ask/` — pure-Rust LLM adapter (Anthropic/OpenAI/Ollama) for natural-language → SQL. The engine's `ask` feature provides the thin `ConnectionAskExt::ask` glue.
+- `sqlrite-mcp/` — MCP stdio server. Seven tools: `list_tables`, `describe_table`, `query`, `execute`, `schema_dump`, `vector_search`, `ask`. `--read-only` opens with a shared lock and hides `execute`.
+- `sqlrite-ffi/` — C ABI cdylib + generated `sqlrite.h` header. Backs the Go SDK and any C consumer.
+- `desktop/` — Tauri 2 + Svelte 5 GUI. Embeds the engine directly (no FFI hop).
+
+Architecture deep-dive: [docs/architecture.md](docs/architecture.md). The full doc index is [docs/_index.md](docs/_index.md).
+
+## Engine data flow
+
+SQL string → [src/sql/mod.rs](src/sql/mod.rs) `process_command` parses with the external `sqlparser` crate (SQLite dialect) → [src/sql/parser/](src/sql/parser/) trims the AST into internal structs (`CreateQuery`, `InsertQuery`, `SelectQuery`) → [src/sql/executor.rs](src/sql/executor.rs) runs the statement against the in-memory `Database` ([src/sql/db/database.rs](src/sql/db/database.rs)). On any write, auto-save serializes changed pages through [src/sql/pager/](src/sql/pager/) — 4 KiB pages, cell-encoded B-trees per table and index, WAL + crash-safe checkpoint, fs2 advisory locks. Vector search (Phase 7d) goes through [src/sql/hnsw.rs](src/sql/hnsw.rs); KNN uses a bounded-heap top-k in the executor. Transactions snapshot the in-memory state; ROLLBACK restores it. There is no query optimizer beyond the KNN/HNSW shortcut, no joins, no aggregates yet.
+
+## Commands
+
+CI is the source of truth — the workspace excludes that follow are required because the desktop crate needs a Svelte build first and the PyO3/napi-rs cdylibs can't link standalone test binaries.
+
+```sh
+# Build / test the Rust workspace (matches CI)
+cargo build --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs --all-targets
+cargo test  --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs
+
+# Single test (exact name; --nocapture to see println!)
+cargo test <test_name> -- --nocapture
+
+# Lint (CI runs all three)
+cargo fmt --all -- --check
+cargo clippy --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs --all-targets
+cargo doc    --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs --no-deps
+
+# Run the REPL (default features include cli + ask + file-locks)
+cargo run                                  # in-memory
+cargo run -- path/to/db.sqlrite            # open/create file
+cargo run -- --readonly path/to/db.sqlrite # shared-lock open
+
+# Crate-specific
+cargo build --release -p sqlrite-ffi       # C cdylib + sqlrite.h
+cd desktop && npm install && npm run tauri dev   # desktop app dev mode
+cargo run -p sqlrite-mcp -- /path/to.sqlrite     # MCP server (stdio)
+
+# Release plumbing
+scripts/bump-version.sh 0.2.0              # bumps version across 11 manifests
+```
+
+`SQLRITE_LLM_API_KEY` is required for the `.ask` REPL command, the engine's `ask` feature, and the MCP `ask` tool. Clippy is **not** `-D warnings` yet (intentional — see top of [.github/workflows/ci.yml](.github/workflows/ci.yml)); deny-by-default lints still fail CI.
+
+## Project-specific conventions
+
+- **Errors.** Single `SQLRiteError` enum (thiserror, six variants) with a project-wide `Result<T>` alias. All public APIs return typed errors; no panics. The enum hand-rolls `PartialEq` because `std::io::Error` doesn't derive it.
+- **Storage isn't bincode.** Tables and indexes share a cell-encoded B-tree format with a 4 KiB page size; the file header carries a format version (currently v4 after the Phase 7a vector column work). The diff-based pager only writes changed pages. See [docs/file-format.md](docs/file-format.md) and [docs/pager.md](docs/pager.md).
+- **B-tree commit strategy.** Bottom-up rebuild on every commit (O(N), correct-by-construction). No in-place splits — deferred design decision.
+- **Feature gates matter.** `default = ["cli", "ask", "file-locks"]`. The REPL `[[bin]]` `required-features = ["cli", "ask"]`. WASM and lean library embeddings build with `default-features = false` to avoid rustyline / clap / fs2 / sqlrite-ask. Don't pull these into the always-on dependency set.
+- **Don't reinvent the SQL parser.** `sqlparser` is the tokenizer and AST source; project code only narrows that AST. New SQL features start by mapping the existing `sqlparser` AST node, not by extending a custom grammar.
+- **Phase numbering is real.** The roadmap is sequenced in [docs/roadmap.md](docs/roadmap.md); design discussions live at github.com/sqlrite/design and feature work generally tracks an open phase plan in `docs/phase-*-plan.md`. Treat in-flight phase plans as load-bearing context.
+- **Concurrency.** Engine mutates state through `Arc<Mutex<_>>` (Tauri-friendly). On-disk concurrency uses fs2 advisory locks: shared for readers, exclusive for the single writer.
+
+## Knowledge Base
+
+### Project-specific — `~/Documents/josh-obsidian-synced/Projects/rust_sqlite/`
+
+- **Code:** `/Users/joaoh82/projects/rust_sqlite`
+- **Context (read first):** `~/Documents/josh-obsidian-synced/Projects/rust_sqlite/context.md`
+- **Notes (running journal):** `~/Documents/josh-obsidian-synced/Projects/rust_sqlite/notes.md`
+- **Project wiki:** `~/Documents/josh-obsidian-synced/Projects/rust_sqlite/wiki/`
+
+**How to use each:**
+
+- `context.md` — stable background (product goals, stakeholders, domain). Read before starting non-trivial work. Update only when underlying facts change.
+- `notes.md` — append-only dated journal. Add entries under `## YYYY-MM-DD` headings for decisions, blockers, TODOs, and incidents — anything worth preserving but not stable enough for `context.md`.
+- `wiki/` — reference sub-docs (e.g. `Architecture.md`, `Local Dev Setup.md`, `Tech Services.md`). Create new files as topics emerge.
+
+**When to save:**
+
+- New stable fact about the product/domain → update `context.md`.
+- A decision, incident, or working note → append a dated entry to `notes.md`.
+- Reusable reference material (setup steps, credential locations, architecture) → new/updated file in `wiki/`.
+
+### Cross-project knowledge — `~/Documents/josh-obsidian-synced/vault/`
+
+- **General wiki:** `~/Documents/josh-obsidian-synced/vault/wiki/` — start at `_master-index.md`, then drill into the relevant topic's `_index.md`.
+- **Raw dumps:** `~/Documents/josh-obsidian-synced/vault/raw/` — drop unprocessed research here as `YYYY-MM-DD-{slug}.md`.
+
+Read the general wiki when the question isn't specific to this project. Drop raw research or imported notes into `vault/raw/` so it's captured even before it's distilled.

{sqlrite-0.1.25 → sqlrite-0.2.0}/Cargo.lock

@@ -3817,7 +3817,7 @@ dependencies = [
 
 [[package]]
 name = "sqlrite-ask"
-version = "0.1.25"
+version = "0.2.0"
 dependencies = [
  "serde",
  "serde_json",
@@ -3828,7 +3828,7 @@ dependencies = [
 
 [[package]]
 name = "sqlrite-desktop"
-version = "0.1.25"
+version = "0.2.0"
 dependencies = [
  "serde",
  "serde_json",
@@ -3840,7 +3840,7 @@ dependencies = [
 
 [[package]]
 name = "sqlrite-engine"
-version = "0.1.25"
+version = "0.2.0"
 dependencies = [
  "clap",
  "env_logger",
@@ -3857,7 +3857,7 @@ dependencies = [
 
 [[package]]
 name = "sqlrite-ffi"
-version = "0.1.25"
+version = "0.2.0"
 dependencies = [
  "cbindgen",
  "serde",
@@ -3867,7 +3867,7 @@ dependencies = [
 
 [[package]]
 name = "sqlrite-mcp"
-version = "0.1.25"
+version = "0.2.0"
 dependencies = [
  "clap",
  "libc",
@@ -3878,7 +3878,7 @@ dependencies = [
 
 [[package]]
 name = "sqlrite-nodejs"
-version = "0.1.25"
+version = "0.2.0"
 dependencies = [
  "napi",
  "napi-build",
@@ -3888,7 +3888,7 @@ dependencies = [
 
 [[package]]
 name = "sqlrite-python"
-version = "0.1.25"
+version = "0.2.0"
 dependencies = [
  "pyo3",
  "sqlrite-engine",

{sqlrite-0.1.25 → sqlrite-0.2.0}/Cargo.toml

@@ -20,14 +20,14 @@ resolver = "3"
 # dependency declaration change:
 #
 #     [dependencies]
-#     sqlrite-engine = "0.1"
+#     sqlrite-engine = "0.2"
 #     # then in code: use sqlrite::{Database, …};
 #
 # Any workspace member here that depends on the engine uses the
 # `package =` key so the import name stays `sqlrite` internally:
 #     sqlrite = { package = "sqlrite-engine", path = "…" }
 name = "sqlrite-engine"
-version = "0.1.25"
+version = "0.2.0"
 authors = ["Joao Henrique Machado Silva <joaoh82@gmail.com>"]
 edition = "2024"
 rust-version = "1.85"
@@ -65,6 +65,14 @@ required-features = ["cli", "ask"]
 name = "quickstart"
 path = "examples/rust/quickstart.rs"
 
+# Phase 8d — hybrid retrieval (BM25 + vector cosine via raw arithmetic).
+# Run with `cargo run --example hybrid-retrieval`. Pre-baked vectors
+# (no embedding model dependency) over a 6-doc tech-blurb corpus,
+# showing where pure-BM25, pure-vector, and 50/50 hybrid each win.
+[[example]]
+name = "hybrid-retrieval"
+path = "examples/hybrid-retrieval/hybrid_retrieval.rs"
+
 [features]
 # Default build includes everything: the REPL binary (cli) and
 # POSIX/Windows advisory file locks on the Pager (file-locks).
@@ -130,4 +138,4 @@ fs2 = { version = "0.4", optional = true }
 # crate publishes to crates.io, and a path-only dep without a
 # version field fails the manifest verification step. See PR #58
 # retrospective in docs/roadmap.md.
-sqlrite-ask = { version = "0.1", path = "sqlrite-ask", optional = true }
+sqlrite-ask = { version = "0.2", path = "sqlrite-ask", optional = true }

{sqlrite-0.1.25 → sqlrite-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sqlrite
-Version: 0.1.25
+Version: 0.2.0
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License

{sqlrite-0.1.25 → sqlrite-0.2.0}/desktop/package.json

@@ -1,7 +1,7 @@
 {
   "name": "sqlrite-desktop-frontend",
   "private": true,
-  "version": "0.1.25",
+  "version": "0.2.0",
   "type": "module",
   "scripts": {
     "dev": "vite",

{sqlrite-0.1.25 → sqlrite-0.2.0}/docs/_index.md

@@ -22,7 +22,11 @@ A small, hand-written guide to the SQLRite codebase — how it's structured, how
 
 - [Ask — natural-language → SQL](ask.md) — the canonical reference for the `ask()` feature across every product surface (REPL, desktop, Rust library, Python / Node / Go / WASM SDKs, MCP server); env vars, defaults, prompt caching, security
 - [Ask backend proxy templates](ask-backend-examples.md) — copy-paste backend examples for the WASM SDK's split design: Cloudflare Workers, Vercel Edge, Deno Deploy, Firebase Functions, AWS Lambda, Express, pure Node
-- [MCP server (`sqlrite-mcp`)](mcp.md) — Phase 7h: SQLRite as a Model Context Protocol stdio server. Wiring into Claude Code / Cursor / `mcp-inspector`; the seven tools (`list_tables`, `describe_table`, `query`, `execute`, `schema_dump`, `vector_search`, `ask`); read-only mode; the JSON-RPC wire format
+- [MCP server (`sqlrite-mcp`)](mcp.md) — Phase 7h + 8e: SQLRite as a Model Context Protocol stdio server. Wiring into Claude Code / Cursor / `mcp-inspector`; the eight tools (`list_tables`, `describe_table`, `query`, `execute`, `schema_dump`, `vector_search`, `bm25_search`, `ask`); read-only mode; the JSON-RPC wire format
+
+## Phase 8 — Full-text search + hybrid retrieval
+
+- [FTS — full-text search + hybrid retrieval](fts.md) — the canonical reference for `CREATE INDEX … USING fts`, the `fts_match` / `bm25_score` scalar functions, the `try_fts_probe` optimizer hook, hybrid retrieval via raw arithmetic with `vec_distance_cosine`, persistence + the on-demand v4 → v5 file-format bump, and the `bm25_search` MCP tool
 
 ## Internals
 
@@ -42,11 +46,10 @@ As of May 2026, SQLRite has:
 - WAL-backed persistence with crash-safe checkpointing, shared/exclusive lock modes, and real `BEGIN` / `COMMIT` / `ROLLBACK` transactions (Phase 4 complete)
 - A stable public Rust embedding API plus C FFI shim and SDKs for Python, Node.js, Go, and WASM (Phase 5 complete except the optional 5f crate-polish task)
 - A Tauri 2.0 + Svelte desktop app (Phase 2.5 complete)
-- AI-era extensions across the product surface (Phase 7 complete except FTS): VECTOR columns + HNSW indexes (7a-7d), JSON columns (7e), the `ask()` natural-language → SQL family across the REPL / desktop / Rust / Python / Node / Go / WASM (7g.1-7g.7), and the [`sqlrite-mcp`](mcp.md) Model Context Protocol server with seven tools including `ask` (7h + 7g.8)
+- AI-era extensions across the product surface (Phase 7 complete): VECTOR columns + HNSW indexes (7a-7d), JSON columns (7e), the `ask()` natural-language → SQL family across the REPL / desktop / Rust / Python / Node / Go / WASM (7g.1-7g.7), and the [`sqlrite-mcp`](mcp.md) Model Context Protocol server (7h + 7g.8)
+- Full-text search + hybrid retrieval (Phase 8 complete): FTS5-style inverted index with BM25 ranking + `fts_match` / `bm25_score` scalar functions + `try_fts_probe` optimizer hook + on-disk persistence with on-demand v4 → v5 file-format bump (8a-8c), a worked hybrid-retrieval example combining BM25 with vector cosine via raw arithmetic (8d), and a `bm25_search` MCP tool symmetric with `vector_search` (8e). See [`docs/fts.md`](fts.md).
 - A fully-automated release pipeline that ships every product to its registry on every release with one human action — Rust engine + `sqlrite-ask` + `sqlrite-mcp` to crates.io, Python wheels to PyPI (`sqlrite`), Node.js + WASM to npm (`@joaoh82/sqlrite` + `@joaoh82/sqlrite-wasm`), Go module via `sdk/go/v*` git tag, plus C FFI tarballs, MCP binary tarballs, and unsigned desktop installers as GitHub Release assets (Phase 6 complete)
 
-The active frontier is **Phase 8 — Full-text search + hybrid retrieval** (the deferred 7f scope).
-
 See the [Roadmap](roadmap.md) for the full phase plan.
 
 ## Release engineering
@@ -58,7 +61,7 @@ See the [Roadmap](roadmap.md) for the full phase plan.
 ## Future work
 
 - [Phase 7 plan](phase-7-plan.md) — AI-era extensions (vector column type + HNSW, JSON, NL→SQL `ask()` API across REPL/library/SDKs/desktop/MCP, MCP server). **Implementation complete except 7f, which deferred to Phase 8.**
-- Phase 8 — Full-text search (FTS5-style BM25) + hybrid retrieval, deferred from Phase 7 per the plan-doc's Q1. **Active frontier as of May 2026.**
+- [Phase 8 plan](phase-8-plan.md) — Full-text search (FTS5-style BM25) + hybrid retrieval. The deferred 7f scope. **All six sub-phases (8a–8f) shipped.** Canonical reference: [`docs/fts.md`](fts.md).
 
 ## Conventions
 

{sqlrite-0.1.25 → sqlrite-0.2.0}/docs/architecture.md

@@ -89,18 +89,19 @@ The engine never depends on the SDK crates; the SDK crates each depend on the en
 | Module | What it owns |
 |---|---|
 | [`src/main.rs`](../src/main.rs) | Binary entry: init env_logger, build rustyline editor, run the REPL loop, route input to either the meta or SQL dispatcher |
-| [`src/lib.rs`](../src/lib.rs) | Library entry: re-exports `Connection`, `Statement`, `Rows`, `Value`, `Database`, `process_command`, the `ask` module (when feature on), etc. — the stable public surface every SDK binds against |
+| [`src/lib.rs`](../src/lib.rs) | Library entry: re-exports `Connection`, `Statement`, `Rows`, `Value`, `Database`, `process_command` / `process_command_with_render` / `CommandOutput`, the `ask` module (when feature on), etc. — the stable public surface every SDK binds against |
 | [`src/connection.rs`](../src/connection.rs) | `Connection` / `Statement` / `Rows` / `Row` / `OwnedRow` / `FromValue` — the Phase 5a public API |
 | [`src/ask/`](../src/ask/) | Engine integration with `sqlrite-ask`: `ConnectionAskExt`, `ask_with_database`, the `schema::dump_schema_for_database` helper. The `schema` submodule is always available; the rest is gated behind the `ask` feature. Phase 7g.2. |
 | [`src/repl/`](../src/repl/) | `REPLHelper` (implements rustyline's `Helper` trait: completer, hinter, highlighter, validator). Also `get_config` and `get_command_type` |
 | [`src/meta_command/`](../src/meta_command/) | `MetaCommand` enum, parsing (`.open FOO.db` → `Open(PathBuf)`, `.ask <Q>` → `Ask(String)`), and dispatch to persistence + ask helpers |
 | [`src/error.rs`](../src/error.rs) | `SQLRiteError` (thiserror-derived), `Result<T>` alias, hand-rolled `PartialEq` that handles `io::Error` |
-| [`src/sql/mod.rs`](../src/sql/mod.rs) | `SQLCommand` classifier, `process_command` — the top-level entry that parses a SQL string and routes to the right executor. Also triggers auto-save. |
+| [`src/sql/mod.rs`](../src/sql/mod.rs) | `SQLCommand` classifier, `process_command` / `process_command_with_render` — the top-level entries that parse a SQL string and route to the right executor. Also triggers auto-save. **Never writes to stdout** — for SELECT statements, the rendered prettytable comes back inside `CommandOutput.rendered` so the REPL can print it (the engine itself doesn't); the SDK / FFI / MCP callers ignore it. |
 | [`src/sql/parser/`](../src/sql/parser/) | Takes a `sqlparser::ast::Statement` and produces internal query structs (`CreateQuery`, `InsertQuery`, `SelectQuery`) with only the fields we actually use |
-| [`src/sql/executor.rs`](../src/sql/executor.rs) | `execute_select`, `execute_delete`, `execute_update`, plus the shared expression evaluator `eval_expr` / `eval_predicate`. Also the bounded-heap top-k optimization (Phase 7c) and the HNSW probe shortcut (Phase 7d.2). |
+| [`src/sql/executor.rs`](../src/sql/executor.rs) | `execute_select`, `execute_delete`, `execute_update`, plus the shared expression evaluator `eval_expr` / `eval_predicate`. Also the bounded-heap top-k optimization (Phase 7c), the HNSW probe shortcut (Phase 7d.2), and the FTS probe shortcut (Phase 8b). |
 | [`src/sql/db/database.rs`](../src/sql/db/database.rs) | `Database`: table map + optional `source_path` + optional long-lived `Pager` + transaction-snapshot state |
 | [`src/sql/db/table.rs`](../src/sql/db/table.rs) | `Table`, `Column`, `Row`, `Value` (in-memory storage incl. VECTOR + JSON columns); helpers for row iteration (`rowids`, `get_value`, `set_value`, `delete_row`, `insert_row`) |
 | [`src/sql/hnsw.rs`](../src/sql/hnsw.rs) | Standalone HNSW algorithm — insert / search / layer assignment / beam search. Phase 7d.1. |
+| [`src/sql/fts/`](../src/sql/fts/) | Full-text search — standalone tokenizer, BM25 scorer, and in-memory `PostingList` inverted index. Wired into the executor via the `fts_match` / `bm25_score` scalar functions and the `try_fts_probe` optimizer hook. Phase 8a-8b; persistence in 8c. See [`docs/fts.md`](fts.md). |
 | [`src/sql/json.rs`](../src/sql/json.rs) | JSON column type + path-extraction functions (`json_extract`, `json_type`, `json_array_length`, `json_object_keys`). Phase 7e. |
 | [`src/sql/pager/`](../src/sql/pager/) | On-disk file format and I/O — see [file-format.md](file-format.md) and [pager.md](pager.md) for details. WAL + checkpointer + shared/exclusive lock modes (Phase 4a-4e) live here. |
 
@@ -127,7 +128,7 @@ Steps 1–7 are purely in-memory; step 8 is the only disk contact, and after the
 - **Planning**: intentionally not a thing yet. Execution is direct — a query plan is implicit in the executor code path.
 - **Execution**: `src/sql/executor.rs` walks the internal structs, drives reads against `Table`, and writes via `Table::set_value` / `insert_row` / `delete_row`.
 - **Storage (in memory)**: `src/sql/db/table.rs` — column-oriented `BTreeMap<rowid, value>` per column; indexes as separate `BTreeMap`s on UNIQUE/PK columns.
-- **Storage (on disk)**: `src/sql/pager/` — 4 KiB pages, real B-Tree per table (Phase 3d), secondary indexes (3e), HNSW indexes as their own page tree (7d.3), WAL + crash-safe checkpointer (4c-4d), shared/exclusive lock modes (4e).
+- **Storage (on disk)**: `src/sql/pager/` — 4 KiB pages, real B-Tree per table (Phase 3d), secondary indexes (3e), HNSW indexes as their own page tree (7d.3), FTS posting lists as their own page tree (8c, on-demand v5 file format), WAL + crash-safe checkpointer (4c-4d), shared/exclusive lock modes (4e).
 - **Persistence policy**: `src/sql/mod.rs::process_command` for when to auto-save; `src/sql/pager/mod.rs::save_database` for how. Inside a `BEGIN`/`COMMIT` block, auto-save is suppressed and changes accumulate against an in-memory snapshot — `COMMIT` flushes the whole batch in one WAL frame; `ROLLBACK` restores the snapshot.
 - **Error handling**: `src/error.rs` defines a single `SQLRiteError` enum used throughout, with `#[from]` conversions from `ParserError` and `io::Error`.
 

{sqlrite-0.1.25 → sqlrite-0.2.0}/docs/file-format.md

@@ -4,7 +4,7 @@ A SQLRite database is a single file, by convention named `*.sqlrite`. The file i
 
 All multi-byte integers in this format are **little-endian**.
 
-The current on-disk format is **version 3** (Phase 3e). Files produced by earlier versions are rejected on open.
+The current on-disk format is **version 4** (Phase 7) by default, with **version 5** written on demand whenever an FTS index is attached to the database (Phase 8c). Decoders accept both v4 and v5; writers preserve the existing version on no-op resaves so a v4 database without FTS stays v4. Files produced by versions 1 – 3 are rejected on open.
 
 ## Page 0 — the database header
 
@@ -15,7 +15,7 @@ The first 4096 bytes of every file are the header page. Only the first 28 bytes
 │ offset │ length │ content                                         │
 ├────────┼────────┼─────────────────────────────────────────────────┤
 │     0  │   16   │ magic:  "SQLRiteFormat\0\0\0"                   │
-│    16  │    2   │ format version (u16 LE) = 3                     │
+│    16  │    2   │ format version (u16 LE) = 4 or 5                │
 │    18  │    2   │ page size      (u16 LE) = 4096                  │
 │    20  │    4   │ total page count (u32 LE), includes page 0      │
 │    24  │    4   │ root page of sqlrite_master (u32 LE)            │
@@ -25,7 +25,7 @@ The first 4096 bytes of every file are the header page. Only the first 28 bytes
 
 The magic string is 14 ASCII bytes (`SQLRiteFormat`) padded with two NUL bytes to fill 16 bytes. It's deliberately different from SQLite's `"SQLite format 3\0"` so the two formats can't be confused on inspection.
 
-`decode_header` in [`src/sql/pager/header.rs`](../src/sql/pager/header.rs) validates all three of (magic, format version, page size) on open. A wrong magic produces `not a SQLRite database`; a wrong version or page size produces `unsupported ...` errors.
+`decode_header` in [`src/sql/pager/header.rs`](../src/sql/pager/header.rs) validates all three of (magic, format version, page size) on open. A wrong magic produces `not a SQLRite database`; a wrong version or page size produces `unsupported ...` errors. The decoder accepts both v4 and v5 (anything else is rejected); the parsed `format_version` is propagated through the in-memory `DbHeader` so the writer can preserve it on resave when no version-bumping feature has been added.
 
 ## Pages 1..page_count — payload pages
 
@@ -126,14 +126,16 @@ A cell is length-prefixed; its body starts with a `kind_tag` byte:
 
 ```
 cell_length    varint          excludes itself; total bytes of kind_tag + body
-kind_tag       u8              0x01 = Local    (full row on a leaf)
-                               0x02 = Overflow (pointer to spilled body)
-                               0x03 = Interior (divider on an interior node)
-                               0x04 = Index    (one entry in an index leaf)
+kind_tag       u8              0x01 = Local         (full row on a leaf)
+                               0x02 = Overflow      (pointer to spilled body)
+                               0x03 = Interior      (divider on an interior node)
+                               0x04 = Index         (entry in a secondary-index leaf)
+                               0x05 = HNSW          (Phase 7d.3 — one HNSW node)
+                               0x06 = FTS Posting   (Phase 8c — one FTS posting list)
 body           variable        depends on kind_tag
 ```
 
-The shared prefix means `Cell::peek_rowid` works uniformly across all three kinds — useful for binary search over a page's slot directory without decoding full bodies.
+The shared prefix means `Cell::peek_rowid` works uniformly across all kinds — useful for binary search over a page's slot directory without decoding full bodies.
 
 ### Local cell body
 
@@ -205,6 +207,25 @@ value_body      variable          encoded per the Local cell's value-block rules
 
 NULL values are never indexed — `SecondaryIndex::insert` skips them — so there's no null bitmap here; a non-null value is always present.
 
+### FTS posting cell body — `KIND_FTS_POSTING` (0x06, Phase 8c)
+
+Used on the leaves of an FTS index B-Tree. Each cell carries either a posting list for one term (`term`-bytes non-empty), or — in a single sidecar cell — the per-doc length map (`term`-bytes empty). The B-Tree key is `cell_id`, a sequential integer assigned at save time; it has no meaning beyond ordering cells within their tree (so `Cell::peek_rowid`'s slot-directory ordering still works without FTS-specific page plumbing).
+
+```
+cell_id         zigzag varint     sequential B-Tree slot key (1, 2, 3, ...)
+term_len        varint            byte length of `term` (0 → sidecar cell)
+term            term_len bytes    ASCII-lowercased term per Phase 8 Q3
+count           varint            number of (rowid, value) pairs
+for each:
+  rowid         zigzag varint     the row this entry refers to
+  value         varint            term frequency for this (term, row),
+                                  or doc length when term_len == 0
+```
+
+One sidecar cell with `term_len == 0` exists per index, holding `(rowid, doc_len)` pairs for every indexed doc — including any with zero-token text — so `total_docs` and `avg_doc_len` round-trip in BM25 even on degenerate corner cases. Posting cells follow, one per unique term in lexicographic order.
+
+A single posting cell that exceeds page capacity (~4 KiB) errors at save time; overflow chaining is a Phase 8.1 stretch goal. In practice — even `'the'` in a million-row English corpus stays under the limit with the varint encoding above.
+
 ## The schema catalog: `sqlrite_master`
 
 The schema catalog is itself a table named `sqlrite_master`, stored in the same `TableLeaf` format as any user table. Its schema is hardcoded into the engine so the open path can bootstrap:
@@ -284,7 +305,8 @@ These are not all enforced on open — we validate the header strictly and rely
 - **v1** (Phases 2 / 3a / 3b) — schema catalog and table data were opaque `bincode` blobs chained across typed payload pages.
 - **v2** (Phases 3c / 3d) — cell-based storage and `sqlrite_master`. Phase 3d added interior pages without a version bump.
 - **v3** (Phase 3e) — `sqlrite_master` gains a `type` column; secondary indexes persist as their own cell-based B-Trees whose leaves carry `KIND_INDEX` cells.
-- **v4** (Phase 7a, current) — value block dispatch gains the `0x04 Vector` tag for the new `VECTOR(N)` column type. Per the [Phase 7 plan's Q8](phase-7-plan.md#q8-file-format-version-bump), later Phase 7 sub-phases (JSON storage, HNSW indexes) will add their own value/cell tags inside this same v4 envelope — no v5 mid-Phase-7. The `CREATE TABLE` SQL stored in `sqlrite_master` carries vector columns as `VECTOR(N)` in the type position; on open, the engine re-parses that SQL and reconstructs `DataType::Vector(N)` from the `Custom` AST node sqlparser produces.
+- **v4** (Phase 7a) — value block dispatch gains the `0x04 Vector` tag for the new `VECTOR(N)` column type. Per the [Phase 7 plan's Q8](phase-7-plan.md#q8-file-format-version-bump), later Phase 7 sub-phases (JSON storage, HNSW indexes) added their own value/cell tags inside this same v4 envelope. The `CREATE TABLE` SQL stored in `sqlrite_master` carries vector columns as `VECTOR(N)` in the type position; on open, the engine re-parses that SQL and reconstructs `DataType::Vector(N)` from the `Custom` AST node sqlparser produces.
+- **v5** (Phase 8c, current for FTS-bearing files) — adds the `KIND_FTS_POSTING` cell tag for persisted FTS posting lists. Bumped **on demand** per the [Phase 8 plan's Q10](phase-8-plan.md#q10-file-format-version-bump-strategy): existing v4 databases without FTS keep writing v4 across non-FTS saves; the first save with at least one FTS index attached promotes the file to v5. Decoders accept both v4 and v5; opening a v4 file with a build that supports v5 is a no-op until the user creates an FTS index.
 
 The page header (7 bytes) and chaining mechanism are stable across future phases. Phase 4's WAL introduces a sibling file (`.sqlrite-wal`) rather than changing the main file format.