@dikolab/kbdb 0.4.3 → 0.6.0

package/docs/goals/database.md

@@ -1,1679 +0,0 @@
-# Database -- Filesystem Storage Engine
-
-*A portable, directory-based database managed by
-fs-database.wasm -- no server, no cloud, just a folder
-on disk.*
-
-## Goal
-
-The knowledge base is stored in a `.kbdb` directory on
-the local filesystem. This directory is the entire
-database. There is no cloud dependency.
-
-`fs-database.wasm` is the sole owner of all read and
-write operations on this directory. It runs inside the
-worker daemon (see [Worker Daemon](worker-daemon.md)),
-which loads it alongside the other WASM modules.
-`src/cli.ts` does not read or write database files
-directly -- it uses the worker client module to send
-requests to the daemon over IPC, and the daemon
-delegates storage operations to `fs-database.wasm`.
-
-## WASM Module: fs-database.wasm
-
-`fs-database.wasm` manages the `.kbdb` directory
-structure, file formats, indexes, and data integrity.
-It is a standalone module -- it does not import
-`query-parser.wasm` or any other WASM module. Text processing
-(tokenization, query parsing, ranking) is handled by
-`query-parser.wasm` separately; `kb-worker.wasm` orchestrates
-both modules.
-
-### Dependency Position
-
-```
-query-parser.wasm          fs-database.wasm
-(standalone)         (this module -- standalone)
-      ↑                    ↑
-      └────────┬───────────┘
-               |
-        kb-worker.wasm
-   (imports both, orchestrates)
-```
-
-### Host Imports
-
-`fs-database.wasm` cannot access the filesystem
-directly. The TypeScript host (inside the daemon)
-provides file I/O functions as WASM imports:
-
-| Import       | Description                        |
-|--------------|------------------------------------|
-| `memory`     | Shared `WebAssembly.Memory`        |
-| `read_file`  | Read file contents by path         |
-| `write_file` | Write data to a file path          |
-| `remove_file`| Delete a file by path              |
-| `list_dir`   | List entries in a directory        |
-| `mkdir`      | Create a directory                 |
-| `file_exists`| Check whether a path exists        |
-
-All paths are relative to the context directory
-(`${target_dir}/.kbdb`). The host resolves them to
-absolute paths and enforces that no operation escapes
-the database directory.
-
-## Storage Directory
-
-`fs-database.wasm` accepts a `target_dir` parameter --
-the absolute path to the directory that will contain the
-`.kbdb` database. The worker client module resolves this
-path before connecting to the daemon (see
-[Worker Client](worker-client.md)):
-
-1. If a path is specified by the caller (e.g. via a CLI
-   flag `--db <path>`), use that path.
-2. Otherwise, use the current working directory (CWD).
-
-The database directory is always `${target_dir}/.kbdb`.
-`fs-database.wasm` never resolves CWD itself -- it
-always receives an absolute path from the host.
-
-If `.kbdb` does not exist at the resolved path, the
-worker client throws. The caller is responsible for
-database initialization before creating a worker
-client -- e.g. the CLI prompts the user to create it.
-
-The directory is portable. It can be copied, moved, or
-version-controlled. Its contents are self-contained -- no
-external state or configuration is required.
-
-The `target_dir` also serves as the identity key for daemon
-discovery. Each unique `target_dir` gets its own independent
-cache daemon. Daemon PID files are stored in
-the OS temporary directory, not inside `.kbdb`. Multiple
-databases on the same machine run independent daemons with
-independent caches. See
-[Worker Daemon](worker-daemon.md) for daemon details.
-
-## Directory Layout
-
-The `.kbdb` directory is structured like a PostgreSQL
-directory-format dump: a human-readable catalog at the
-root, one subdirectory per data table (sections,
-documents), and a separate directory for binary indexes.
-Metadata files use TOML for human inspectability.
-Indexes are binary but derived and rebuildable.
-
-```
-.kbdb/
-├── catalog.toml           # database table of contents:
-│                          # section count, document
-│                          # count, timestamps, integrity
-├── sections/
-│   ├── ab/
-│   │   └── cdef1234...sec # section file (TOML header
-│   │                      # + raw content body)
-│   └── ...
-├── documents/
-│   └── {docid}.toml       # document manifest: title +
-│                          # ordered section list
-├── indexes/
-│   ├── meta.toml          # index metadata: last rebuild
-│   │                      # timestamp, term count, stats
-│   ├── terms.idx          # inverted index (binary,
-│   │                      # B+ tree)
-│   ├── positions.idx      # positional data (binary)
-│   ├── corpus.dat         # corpus-wide statistics
-│   │                      # (binary)
-│   └── references.idx     # bidirectional reference
-│                          # graph (binary)
-├── worker.toml            # cache daemon config
-│                          # (see Worker)
-└── .gitignore             # excludes indexes/ from
-                           # version control
-```
-
-### Catalog (`catalog.toml`)
-
-The catalog is the database's table of contents --
-analogous to `toc.dat` in a PostgreSQL directory dump.
-It is the first file `fs-database.wasm` reads when
-opening a database. It records:
-
-- Database format version.
-- Total section count and document count.
-- Creation and last-modified timestamps.
-- Integrity checksums.
-
-The catalog is updated on every write operation.
-
-Example:
-
-```toml
-[database]
-version = 1
-created_at = "2026-06-15T10:00:00Z"
-last_modified = "2026-06-15T12:00:00Z"
-
-[counts]
-sections = 42
-documents = 5
-
-[integrity]
-checksum = "sha256:abcdef..."
-```
-
-### Sections (`sections/`)
-
-One file per section, stored in hash-sharded
-subdirectories (first two characters of the kb-id) to
-avoid large flat directories.
-
-Every section's kb-id is derived from a SHA-256 hash
-of a **typed header plus preprocessed content**,
-following the same content-addressing pattern used by
-Git objects (`blob <size>\0<content>`). The hash input
-is:
-
-```
-section <mime-type> <byte-size>\0<preprocessed-content>
-```
-
-- `<mime-type>` -- the section's MIME type string
-  (e.g. `text/markdown`, `text/typescript`,
-  `image/png`).
-- `<byte-size>` -- the byte length of the preprocessed
-  content, expressed as a decimal ASCII string (like
-  Git).
-- `\0` -- a literal null byte separator.
-- `<preprocessed-content>` -- the content after the
-  preprocessing pipeline (see **Content Preprocessing**
-  below).
-
-The SHA-256 digest is truncated to the first 16 bytes
-(128 bits) and base32-encoded (no padding, lowercase)
-to produce a 26-character alphanumeric identifier that
-is deterministic, collision-resistant, and URL-safe.
-
-Because the MIME type is part of the hash input,
-identical content stored as different types (e.g.
-`text/markdown` vs `text/typescript`) produces
-different kb-ids.
-
-Each section file uses a TOML frontmatter header
-followed by a `---` separator and the content body.
-The header is human-readable; the content after the
-separator is stored verbatim with no escaping.
-
-The `type` field identifies the content kind:
-
-- **Text sections** -- `type` is `"text/markdown"`.
-  The content body after `---` is raw text (markdown)
-  stored verbatim.
-- **Code sections** -- `type` is a `text/*` MIME type
-  (e.g. `"text/typescript"`, `"text/html"`,
-  `"text/css"`) or `"application/json"`. The content
-  body after `---` is the raw source text stored
-  verbatim. Code sections are indexed -- the
-  tokenization pipeline processes their content,
-  including identifier splitting for camelCase and
-  snake_case terms.
-- **Image sections** -- `type` is an `image/*` MIME
-  type (e.g. `"image/png"`, `"image/jpeg"`,
-  `"image/webp"`). The content body after `---` is the
-  image file's binary data encoded as **base64**. The
-  kb-id is the SHA-256 hash of the base64-encoded
-  string (not the raw binary), so identical base64
-  input always produces the same kb-id.
-
-All section types use proper MIME type strings. The
-bare string `"text"` is no longer used; text sections
-use `"text/markdown"` as their canonical type.
-
-Image section binary content is not indexed directly.
-However, image sections can be searchable via their
-`title` and `description` attributes (see below).
-They are also retrievable by kb-id or through
-document grouping.
-
-Several metadata and pre-computed fields are stored
-in the TOML header:
-
-- **`title`** -- A short descriptive title for the
-  section. **Required** for image sections (`image/*`)
-  since images have no indexable text content and need
-  a human-readable identifier. Optional for text
-  (`text/markdown`) and code (`text/*`,
-  `application/json`) sections. Not included in the
-  kbid hash -- changing the title does not change the
-  section's identity. When present, the title is
-  tokenized and indexed with the **heading** field
-  weight (2.0x) for search ranking. See
-  [Query Result](query-result.md) for field weight
-  details.
-- **`description`** -- An optional longer description
-  of the section's content or purpose. Always optional
-  for all section types. Not included in the kbid hash.
-  When present, the description is tokenized and
-  indexed with the **body** field weight (1.0x) for
-  search ranking.
-- **`token_count`** -- Total number of tokens after
-  tokenization (u32). Sum of all per-field token
-  counts. Used for BM25F length normalization and
-  zone scoring. Computed by `query-parser.wasm` during
-  indexing.
-- **`heading_token_count`** -- Number of tokens from
-  the `title` attribute (u32). Used for per-field
-  BM25 length normalization.
-- **`body_token_count`** -- Number of tokens from the
-  body text content and `description` attribute (u32).
-  Used for per-field BM25 length normalization.
-- **`code_token_count`** -- Number of tokens from
-  code content (u32). Non-zero only for code sections.
-  Used for per-field BM25 length normalization.
-
-Token count fields are computed by `query-parser.wasm`
-during tokenization and passed to `fs-database.wasm`
-for storage. They are pre-computed once at insertion
-time and reused on every search -- trading a small
-write-time cost for significant query-time speed.
-
-Additional metadata fields support truth maintenance
-and provenance tracking:
-
-- **`source_key`** -- Logical identity hash (16
-  lowercase hex characters). Derived from the source
-  file path and heading text:
-  `sha256("${source_path}\0${heading}")[..8]`. Used
-  for supersession -- matching existing sections when
-  content is re-learned. `null` for stdin input and
-  sections without a source path. See **Logical
-  Identity (Source Key)** below.
-- **`source_path`** -- Absolute path of the file that
-  was learned. `null` for stdin input and
-  programmatic API calls. Stored for provenance
-  tracking and source-aware garbage collection.
-- **`source_mtime`** -- ISO 8601 timestamp of the
-  source file's last modification time at learn time.
-  `null` when no source file exists. Allows staleness
-  detection: if the source file's mtime has changed
-  since this timestamp, the section may be outdated.
-- **`updated_at`** -- ISO 8601 timestamp of the last
-  supersession. `null` on first creation. Set when a
-  section supersedes an existing one with the same
-  source key.
-- **`tags`** -- Array of user-supplied string tags for
-  scoping and filtering. Empty array when no tags are
-  provided. Tags are not indexed for search -- they
-  are metadata for query-time filtering.
-- **`supersedes`** -- The kbid of the section this one
-  replaced via supersession. `null` on first creation
-  and for non-superseding additions. Creates a
-  supersession chain that can be traversed to find
-  prior versions.
-
-The `add_section` operation rejects requests with
-`image/*` type that do not include a `title`. Text
-and code sections may omit both fields.
-
-### Embedding Sections in Text Content
-
-Text sections can contain **markers** -- escape
-sequences that embed or link to other sections and
-documents using the `$(type: value)` syntax. Six
-marker types are supported: four for embedding
-content inline (`$(image:)`, `$(code:)`, `$(text:)`,
-`$(document:)`) and two for cross-references
-(`$(link:)`, `$(doc-link:)`).
-
-Markers are stored verbatim in the content body --
-they are not expanded or resolved at storage time.
-They are structural references, not searchable text;
-the tokenization pipeline strips them before
-indexing.
-
-See [Document](document.md) for the full content
-format specification, all marker types, syntax
-rules, rendering behaviour, and the MIME-to-language
-mapping table for code embeddings. See
-[Content Parser](content-parser.md) for how markers
-are extracted from content as structured data.
-
-#### Section File Examples
-
-**Text section** (`sections/ab/cdef1234...sec`):
-
-```
-kbid = "abcdef1234567890abcdef1234"
-docids = ["doc-5678"]
-type = "text/markdown"
-title = "Architecture Overview"
-description = "High-level overview of the system"
-size = 1234
-token_count = 187
-heading_token_count = 2
-body_token_count = 187
-code_token_count = 0
-created_at = "2026-06-15T10:00:00Z"
-checksum = "sha256:abcdef..."
-source_key = "a1b2c3d4e5f67890"
-source_path = "/home/user/docs/architecture.md"
-source_mtime = "2026-06-15T09:45:00Z"
-updated_at = ""
-tags = ["architecture", "overview"]
-supersedes = ""
----
-Section body text here.
-
-$(image: 7fa93b2e4c8d1076a93b2e4c8d)
-
-$(code: c4e8a91d3f7b20564e8a91d3f7)
-
-This content continues after the embedded sections.
-No escaping or encoding is applied.
-```
-
-**Code section** (`sections/c4/e8a91d...sec`):
-
-```
-kbid = "c4e8a91d3f7b20564e8a91d3f7"
-docids = ["doc-5678"]
-type = "text/typescript"
-title = "greet function"
-size = 256
-token_count = 22
-heading_token_count = 2
-body_token_count = 0
-code_token_count = 22
-created_at = "2026-06-15T10:15:00Z"
-checksum = "sha256:1a2b3c..."
-source_key = "f8e7d6c5b4a39281"
-source_path = "/home/user/src/greet.ts"
-source_mtime = "2026-06-15T10:10:00Z"
-updated_at = ""
-tags = ["api", "greeting"]
-supersedes = ""
----
-export function greet(name: string): string {
-   return `Hello, ${name}!`;
-}
-```
-
-**Image section** (`sections/7f/a93b2e...sec`):
-
-```
-kbid = "7fa93b2e4c8d1076a93b2e4c8d"
-docids = ["doc-5678"]
-type = "image/png"
-title = "Architecture diagram"
-description = "Five-layer stack diagram"
-size = 48576
-token_count = 5
-heading_token_count = 2
-body_token_count = 3
-code_token_count = 0
-created_at = "2026-06-15T10:30:00Z"
-checksum = "sha256:fedcba..."
-source_key = "1029384756abcdef"
-source_path = "/home/user/docs/arch-diagram.png"
-source_mtime = "2026-06-15T10:25:00Z"
-updated_at = ""
-tags = ["diagram", "architecture"]
-supersedes = ""
----
-iVBORw0KGgoAAAANSUhEUgAA...base64 data...
-```
-
-The image section's `token_count` is 5 (from title +
-description tokens only). Binary content itself
-produces zero tokens.
-
-#### Content Preprocessing (kb-id computation)
-
-Before computing the kb-id, text and code section
-content passes through a preprocessing pipeline. The
-pipeline normalizes content so that semantically
-identical inputs -- differing only in BOM, line
-endings, Unicode representation, or surrounding
-whitespace -- produce the same kb-id.
-
-The pipeline steps, applied in order:
-
-1. **BOM removal** -- Strip the UTF-8 BOM (U+FEFF) if
-   present at the start of the content. The BOM is an
-   encoding artifact, not meaningful content.
-2. **Line ending normalization** -- Convert all `\r\n`
-   (Windows) and bare `\r` (old Mac) sequences to `\n`
-   (Unix). Ensures content created on different
-   platforms produces the same kb-id.
-3. **Unicode NFC normalization** -- Compose decomposed
-   Unicode sequences into their canonical composed form
-   (NFC). For example, `e` + combining acute accent
-   becomes `é` (U+00E9). This is the same
-   normalization used in the tokenization pipeline (see
-   [Query Parser](query-parser.md)).
-4. **Whitespace trimming** -- Remove all leading and
-   trailing whitespace characters (spaces, tabs,
-   newlines, carriage returns, and other Unicode
-   whitespace and control characters).
-
-This pipeline applies to **text and code sections
-only**. Image sections use the base64-encoded string
-as-is -- no preprocessing is applied.
-
-The preprocessed content is what is stored on disk.
-The pipeline runs once at ingest time; subsequent
-reads return the already-preprocessed content. The
-`<byte-size>` in the hash header is computed from the
-content **after** preprocessing.
-
-Example hash input construction:
-
-```
-Given:
-  type    = "text/markdown"
-  content (after preprocessing) = "Hello, world!"
-                                  (13 bytes)
-
-Hash input (41 bytes):
-  section text/markdown 13\0Hello, world!
-
-  ├─ header: "section text/markdown 13"
-  ├─ null byte: \0
-  └─ preprocessed content: "Hello, world!"
-
-kbid = base32(sha256(hash_input)[..16])
-     → 26-character lowercase alphanumeric string
-```
-
-Sections are **content-addressable** -- identical
-preprocessed content stored with the same type always
-produces the same kbid and is stored once (automatic
-deduplication). Because the MIME type is part of the
-hash input, the same raw content stored as different
-types (e.g. `text/markdown` vs `text/typescript`)
-produces different kbids and different section files.
-When identical content is added with the same type:
-
-- If a different docid is provided, the existing
-  section's `docids` array is updated to include the
-  new docid -- no new section file is created
-  (`attached: true`).
-- If no docid is provided, the section is skipped
-  (`skipped: true`).
-- If the provided docid is already in the `docids`
-  array, the operation is idempotent
-  (`skipped: false`, `attached: false`).
-
-A section can belong to **zero or more** documents.
-The `docids` field is an array of document IDs. The
-relationship between sections and documents is
-**many-to-many**: a section may appear in multiple
-documents, and a document contains multiple sections.
-
-Updating a section's content creates a new file with
-a new kbid; the old file is eligible for garbage
-collection.
-
-#### Logical Identity (Source Key)
-
-Content-addressing identifies *what a section
-currently says*. The **source key** identifies *what
-a section is about* -- it is a stable logical
-identity that persists across content edits.
-
-The source key is derived from the section's origin:
-
-```
-source_key = sha256("${source_path}\0${heading}")[..8]
-           → 16 lowercase hex characters
-```
-
-- `source_path` -- the absolute path of the file that
-  was learned (normalized: resolved symlinks, no
-  trailing separator).
-- `heading` -- the section's title text (or empty
-  string when no title is provided).
-- `\0` -- a null byte separator.
-
-The SHA-256 digest is truncated to 8 bytes (64 bits)
-and hex-encoded to produce a 16-character identifier.
-
-Source keys enable **supersession**: when content is
-re-learned from the same source with the same
-heading, the system can identify the existing section
-and replace it rather than creating a sibling. This
-prevents the brain from silently accumulating
-contradictions when facts are corrected.
-
-Source key is `null` for:
-- Stdin input (`-`) -- no source path exists.
-- Sections created without a source path (e.g.
-  programmatic API calls that omit it).
-
-When `source_key` is `null`, supersession is not
-available and the section behaves as a pure
-content-addressed entry (current behavior).
-
-See **Add section** below for supersession semantics.
-
-### Documents (`documents/`)
-
-One TOML file per document. A document groups sections
-under a title with a defined order. A section may
-appear in multiple documents (many-to-many). The
-document manifest contains:
-
-- Document ID (`docid`).
-- Title.
-- Creation timestamp.
-- An ordered list of section references (kb-ids).
-
-Example (`documents/{docid}.toml`):
-
-```toml
-[document]
-docid = "doc-5678"
-title = "Getting Started Guide"
-created_at = "2026-06-15T10:00:00Z"
-
-[[sections]]
-kbid = "abcdef1234567890abcdef1234"
-position = 0
-
-[[sections]]
-kbid = "fedcba4321098765fedcba4321"
-position = 1
-```
-
-Human-readable -- can be inspected and diffed in git.
-
-### Indexes (`indexes/`)
-
-Binary format, not human-readable. Optimized for fast
-term lookup using B+ tree structures. Rebuilt from
-sections and documents on demand. Can be excluded from
-version control via `.gitignore`.
-
-Three index files support the search and ranking
-pipeline:
-
-- **Inverted index** (`terms.idx`) -- B+ tree mapping
-  normalized terms to posting lists. Each posting
-  entry contains a kbid and **per-field term
-  frequencies**: `heading_tf` (from title), `body_tf`
-  (from body text + description), and `code_tf` (from
-  code content). Storing per-field TFs separately
-  (rather than a single aggregated weight) allows
-  BM25F field weights to be tuned at query time
-  without rebuilding indexes. See
-  [Query Result](query-result.md) for how these
-  values feed into the BM25F scoring formula.
-
-- **Positional index** (`positions.idx`) -- B+ tree
-  mapping `(term, kbid)` pairs to position arrays
-  (zero-based token offsets). Supports phrase queries,
-  proximity scoring, and zone scoring.
-
-- **Corpus statistics** (`corpus.dat`) -- Corpus-wide
-  aggregates: `total_sections` (u32),
-  `total_terms` (u32), `avg_section_length` (f32),
-  `avg_heading_length` (f32), `avg_body_length` (f32),
-  `avg_code_length` (f32), and per-term document
-  frequencies. Per-field average lengths enable
-  accurate per-field BM25 normalization -- a 3-word
-  title is not penalized against a 500-word average
-  body length.
-
-- **Reference graph** (`references.idx`) --
-  Bidirectional index mapping content marker
-  references between sections and documents. Each
-  entry records:
-
-  | Field            | Type   | Description         |
-  |------------------|--------|---------------------|
-  | `source_kbid`    | string | Section containing the marker |
-  | `target_id`      | string | Referenced kbid or docid |
-  | `marker_type`    | enum   | Image, Code, Text, Document, Link, DocLink |
-  | `context_snippet` | string | ~150 chars of surrounding text where the marker appears |
-  | `direction`      | enum   | `forward` or `back` |
-
-  For every content marker `$(type: value)` in a
-  section's content, the index stores two entries:
-  a **forward** entry (`source → target`) and a
-  **back** entry (`target ← source`). This enables
-  the `recall` tool to look up both "what this
-  section references" and "what references this
-  section" in O(1) without re-parsing content.
-
-  **Build process:** When a section is added or
-  updated, `fs-database.wasm` calls
-  `extract_content_markers()` on its content,
-  extracts a ~150-char context snippet around each
-  marker, and writes forward and back-reference
-  entries.
-
-  **Maintenance:** When a section is removed, all
-  its forward-reference entries and all
-  back-reference entries pointing to it are
-  deleted.
-
-  **Rebuild:** `rebuild` regenerates
-  `references.idx` from section content alongside
-  the other index files. The reference graph is a
-  derived artifact -- the content markers in
-  section bodies are the source of truth.
-
-The `meta.toml` file inside `indexes/` provides
-human-readable metadata about the indexes: last rebuild
-timestamp, total term count, and corpus statistics
-summary. This file is informational only -- the binary
-index files are the source of truth for search.
-
-### Storage Design
-
-The directory layout is optimized for **read speed**:
-
-- **O(1) section lookup** -- Hash-sharded directories
-  allow direct file access by kb-id without scanning.
-- **O(log n) term lookup** -- B+ tree indexes support
-  efficient range and equality queries on terms.
-- **Catalog-first discovery** -- `catalog.toml` provides
-  section and document counts without scanning
-  directories.
-- **Separated metadata and content** -- TOML manifests
-  (small, frequently read) are separate from section
-  content files (large, read on demand).
-
-Write operations may be slower but are designed to be
-safe:
-
-- **Atomic writes** -- New files are written to a
-  temporary path and renamed into place, ensuring no
-  partial writes on crash.
-- **Catalog update last** -- The catalog is updated
-  after all data files are written, so a crash mid-write
-  leaves the catalog consistent with the last complete
-  state.
-- **Index rebuilds** -- Index files can be rebuilt
-  entirely from sections and document manifests. Index
-  corruption is recoverable.
-
-## What the Database Stores
-
-The database holds three categories of data:
-
-1. **Sections** -- Individual content units stored as
-   files in `sections/`. Each identified by its content
-   hash (kb-id). The section file contains a TOML
-   header with metadata (including pre-computed token
-   counts for ranking) and the content body (raw text
-   for text and code sections, base64 for image
-   sections). Identical content is deduplicated
-   automatically.
-
-2. **Documents** -- Groupings of sections stored as
-   TOML manifests in `documents/`. Each document has a
-   title and an ordered list of section kb-ids. A
-   document does not contain content directly -- it
-   references sections.
-
-3. **Indexes** -- Binary files in `indexes/` mapping
-   extracted terms to sections (kb-ids) with per-field
-   term frequencies, positional data, corpus-wide
-   statistics (including per-field averages), and a
-   bidirectional reference graph of content marker
-   relationships. Supports term lookup, phrase
-   queries, BM25F ranking with tunable field weights,
-   and O(1) forward/back-reference lookup for the
-   `recall` tool. Derived data -- rebuildable from
-   sections and documents.
-
-## Operations
-
-`fs-database.wasm` exposes the following operations.
-These are called by `kb-worker.wasm` inside the daemon,
-not by the worker client or TypeScript callers
-directly.
-
-### Section Operations
-
-1. **Read sections** -- Given a list of kb-ids, return
-   the section content and metadata for each. Reads
-   from `sections/{prefix}/{kbid}.sec`. Returns the
-   TOML header fields and the raw content body.
-
-2. **Add section** -- Given a type, section content,
-   and optional title and description, create the
-   section file. For text and code sections, runs the
-   content through the preprocessing pipeline (BOM
-   removal, line ending normalization, Unicode NFC,
-   whitespace trimming). Computes the kb-id using the
-   typed header format
-   (`section <mime-type> <byte-size>\0<content>`).
-
-   During indexing, `kb-worker.wasm` calls
-   `query-parser.wasm` to tokenize the content per
-   field (title → heading, body text + description →
-   body, code content → code) and compute per-field
-   term frequencies and token counts. These
-   pre-computed values are passed to
-   `fs-database.wasm`, which writes the TOML header
-   (kbid, docids, type, title, description, size,
-   token_count, heading_token_count, body_token_count,
-   code_token_count, created_at, checksum, source_key,
-   source_path, source_mtime, updated_at, tags,
-   supersedes), the separator, and the content body
-   (preprocessed text for text and code sections,
-   base64 for image sections). Updates `catalog.toml`
-   counts. Updates the inverted index with per-field
-   term frequencies, the positional index with term
-   positions, and corpus statistics (including
-   per-field average lengths).
-
-   Image sections (`image/*` type) have their binary
-   content stored but not indexed. However, their
-   `title` (required) and `description` (optional)
-   attributes are tokenized and indexed, making image
-   sections searchable via their metadata.
-
-   **Supersession:** When the `replace` flag is set
-   and the new section has a non-null `source_key`,
-   `fs-database.wasm` searches for an existing section
-   with the same `source_key`. If found:
-
-   - The new section's `supersedes` field is set to
-     the old section's kbid.
-   - The new section's `updated_at` is set to the
-     current timestamp.
-   - The old kbid is replaced with the new kbid in all
-     document manifests that reference it.
-   - The old section's `docids` array is emptied,
-     making it eligible for garbage collection.
-   - Index entries for the old section are removed and
-     replaced with entries for the new section.
-
-   When `replace` is not set, or `source_key` is
-   `null`, the add operation behaves as before (pure
-   content-addressed addition with deduplication).
-
-   **Near-duplicate detection:** After the kbid is
-   computed and deduplication is checked, the add
-   operation computes a Jaccard similarity coefficient
-   between the new section's stemmed term set and
-   existing sections' term sets. Sections exceeding a
-   similarity threshold of 0.85 are reported as
-   near-duplicates in the response
-   (`near_duplicates: [{kbid, similarity}]`). This is
-   advisory -- the section is still stored. The check
-   uses the inverted index for efficient comparison
-   without reading every section's content.
-
-   **Heading-less import warning:** Text and code
-   sections added without a `title` emit a warning in
-   the response: `heading_missing`. The heading field
-   carries 2.0x ranking weight; omitting it means the
-   section loses its strongest searchability signal.
-   The section is still stored -- this is a warning,
-   not an error.
-
-   **Validation:** requests with `image/*` type that
-   omit `title` are rejected with an error. Text and
-   code sections may omit `title` (with the
-   heading-less warning above).
-
-3. **Update section** -- Given a kb-id and new content,
-   replace the section's content. Since kb-ids are
-   content-addressed, this creates a new section file
-   with a new kb-id (including fresh token counts),
-   updates all document references from the old kb-id
-   to the new one, and marks the old section file for
-   garbage collection. Triggers index update: removes
-   old per-field term frequencies and positions,
-   re-tokenizes the new content per field, and stores
-   updated per-field TFs, positions, and token counts.
-   Updates corpus statistics.
-
-4. **Remove section** -- Given a kb-id, delete the
-   section file. Removes the kb-id from any document
-   that references it. Updates `catalog.toml` counts.
-   Triggers index update to remove the section's terms.
-
-### Document Operations
-
-5. **Group sections into document** -- Given a list of
-   kb-ids, a document title, and a docid, create a
-   document manifest in `documents/{docid}.toml`. The
-   manifest records the title, timestamp, and ordered
-   section list. Adds the docid to each referenced
-   section's `docids` array. Updates `catalog.toml`.
-
-6. **Remove document grouping** -- Given a docid,
-   delete the document manifest. The sections
-   themselves are not deleted -- only the grouping is
-   removed. Removes the docid from each section's
-   `docids` array. Updates `catalog.toml`.
-
-### Index and Storage Management
-
-7. **Manage indexes** -- Maintain the inverted index,
-   positional index, and corpus statistics in
-   `indexes/`. `fs-database.wasm` receives
-   pre-computed per-field term frequencies, term
-   positions, and token counts from the caller
-   (produced by `query-parser.wasm` via
-   `kb-worker.wasm` orchestration -- see
-   [Query Parser](query-parser.md)) and stores them
-   in B+ tree structures optimized for read speed.
-
-   The inverted index stores per-field term
-   frequencies (`heading_tf`, `body_tf`, `code_tf`)
-   per posting entry rather than a single aggregated
-   weight, enabling query-time field weight tuning.
-   Corpus statistics include per-field average lengths
-   (`avg_heading_length`, `avg_body_length`,
-   `avg_code_length`) for per-field BM25
-   normalization. See [Query Result](query-result.md)
-   for how these pre-computed values feed into the
-   ranking pipeline.
-
-   Index updates are triggered by section add, update,
-   and remove operations. For text and code sections,
-   body content is tokenized and indexed. For all
-   section types with `title` and/or `description`
-   attributes, those attributes are tokenized and
-   indexed with their respective field weights. Full
-   index rebuilds can be triggered explicitly and
-   reconstruct all index files, token counts, and
-   corpus statistics from the section and document
-   data on disk.
-
-### Search
-
-8. **Execute query plan** -- Given a pre-parsed query
-   plan (produced by `query-parser.wasm` via
-   `kb-worker.wasm` orchestration), execute the plan
-   against the indexes and return raw matches: kb-ids
-   with per-field term frequencies (`heading_tf`,
-   `body_tf`, `code_tf`), positional data, token
-   counts, and per-field token counts.
-   `fs-database.wasm` does not parse queries or rank
-   results -- it performs index lookups only. Ranking
-   is handled by `query-parser.wasm` after
-   `kb-worker.wasm` retrieves the raw results. See
-   [Query Result](query-result.md) for the full
-   ranking pipeline and score computation.
-
-### Retrieval
-
-9. **Retrieve document** -- Given a docid, return the
-   document manifest and all its sections in position
-   order, including section content, type, title, and
-   description.
-
-10. **Status** -- Return database metadata: target
-    directory, document count, section count, index
-    size, last-modified timestamp, and cache
-    statistics.
-
-### Lifecycle
-
-11. **Initialization** -- Given a target directory
-    path, initialize a new `.kbdb` database. Creates
-    the directory structure, empty catalog, and empty
-    index files. Returns the context ID and initial
-    counts.
-
-### Integrity and Maintenance
-
-12. **Integrity check** -- Verify section file
-    checksums match their content. Detect orphaned
-    sections (not referenced by any document),
-    orphaned documents (referencing missing sections),
-    broken embedding references -- markers whose
-    target does not exist or has the wrong type,
-    circular reference chains in `$(text:)` and
-    `$(document:)` markers, and **divergent content**
-    -- multiple sections sharing the same `source_key`
-    (same origin) but with different content,
-    indicating contradictory knowledge that should
-    have been resolved by supersession.
-13. **Garbage collection** -- Remove section files not
-    referenced by any document manifest. Additionally
-    report `stale_sources` -- sections whose
-    `source_path` points to a file that no longer
-    contains their heading text. Stale sources are
-    advisory (not automatically removed) and surfaced
-    in the GC response for manual review.
-14. **Export** -- Copy sections, documents, and
-    catalog to a self-contained output directory (no
-    indexes -- they are rebuildable). Provides a
-    portable, git-friendly snapshot of the knowledge
-    base for backup and versioning.
-15. **Index rebuild** -- Reconstruct all index files
-    from sections and documents. Re-tokenizes all text
-    and code sections per field, recomputes per-field
-    term frequencies and token counts (updating
-    section headers), and rebuilds corpus statistics
-    including per-field average lengths. Recovers
-    from index corruption.
-
-## Relationship to query-parser.wasm
-
-Tokenization, keyword extraction, query parsing, and
-ranking are provided by `query-parser.wasm` -- a separate
-standalone module. See [Query Parser](query-parser.md) for the
-full specification of the tokenization pipeline, query
-parsing stages, statistical keyword extraction (RAKE,
-TF-IDF), and ranking functions (BM25F, proximity
-scoring, zone scoring).
-
-`fs-database.wasm` does **not** import or call
-`query-parser.wasm` directly. Instead, `kb-worker.wasm`
-orchestrates both modules:
-
-- **Indexing** -- When a section is added or updated,
-  `kb-worker.wasm` calls `query-parser.wasm` to
-  tokenize content per field (title → heading, body
-  text + description → body, code content → code)
-  and compute per-field term frequencies and token
-  counts. It then passes the pre-computed per-field
-  TFs, positions, and token counts to
-  `fs-database.wasm` for storage in the inverted
-  index and section header. Image sections (`image/*`
-  type) have their binary content stored without
-  indexing, but their `title` and `description`
-  attributes are tokenized and indexed.
-- **Searching** -- When a query arrives,
-  `kb-worker.wasm` calls `query-parser.wasm` to parse
-  the query into a retrieval plan, passes the plan to
-  `fs-database.wasm` for index lookups (returning
-  per-field TFs, positions, and token counts), then
-  calls `query-parser.wasm`'s ranking functions to
-  score and order the raw results. See
-  [Query Result](query-result.md) for the full
-  ranking pipeline.
-
-### Result Granularity
-
-Three result modes are available. See
-[Query Result](query-result.md) for ranking
-algorithms, score computation, and result mode
-details.
-
-- **Sections** -- Individual matching sections,
-  identified by kb-id.
-- **Whole documents** -- Full documents containing
-  matches, reassembled from the ordered sections in the
-  document manifest.
-- **Statistics** -- Result counts and ranking summaries
-  without content bodies.
-
-### Result Structure
-
-Each result includes the section's kb-id, the parent
-document ID (docid), a relevance score, a content
-snippet showing the matching context, and the matched
-terms. Results are ordered by descending score.
-
-## Worker Architecture and Caching
-
-See [Worker Daemon](worker-daemon.md) for the full
-specification of the cache daemon, worker
-configuration, cache invalidation, and daemon
-lifecycle.
-
-## API Contracts
-
-The worker client module communicates with the worker
-daemon over IPC using JSON-RPC 2.0 (see
-[Worker Daemon](worker-daemon.md) for the IPC protocol
-and [Worker Client](worker-client.md) for the client
-specification). Inside the daemon,
-`kb-worker.wasm` calls `fs-database.wasm` for database
-operations. All data at the IPC boundary uses JSON
-serialization.
-
-The daemon may use alternative formats (shared
-`WebAssembly.Memory` buffers, pointer/length pairs)
-internally between `kb-worker.wasm` and
-`fs-database.wasm` for performance. The module-internal
-format is an implementation detail that does not affect
-the worker client's IPC API.
-
-### TypeScript Layer Role
-
-`src/cli.ts` is a thin relay that delegates all
-database work to the worker client module.
-For example, the CLI's responsibilities are limited to:
-
-1. **Receive** -- Accept the request from CLI arguments.
-2. **Create worker client** -- Call
-   `createWorkerClient({ targetDir })`, which handles
-   daemon discovery, spawning, and IPC connection (see
-   [Worker Client](worker-client.md)).
-3. **Call** -- Invoke the appropriate worker client
-   method (e.g. `client.search()`,
-   `client.addSection()`).
-4. **Wait** -- Block until the worker client returns
-   the typed response.
-5. **Relay or reformat** -- Return the response based
-   on the requested output format:
-   - If the requested format is JSON (the default), or
-     no format is specified, relay the response directly
-     to stdout -- no transformation needed.
-   - If the requested format is plain text (CLI only,
-     via `--format` flag), reformat the response before
-     outputting.
-6. **Disconnect** -- Call `client.disconnect()`.
-
-The TypeScript layer does not parse, enrich, filter, or
-transform the worker client's response when relaying as
-JSON. The response is the final output. The TypeScript
-layer only performs reformatting when an alternative
-output format is explicitly requested via CLI flags.
-
-### Operations
-
-These are the operations that `fs-database.wasm`
-exposes to `kb-worker.wasm` inside the daemon. The
-worker client does not call them directly -- it uses
-the low-level IPC protocol documented in
-[Worker Daemon](worker-daemon.md) (query/token
-pattern). `kb-worker.wasm` orchestrates the WASM
-calls and returns results through the daemon's IPC
-layer.
-
-**Initialization:**
-
-```json
-// Request
-{
-  "target_dir": "<absolute path>"
-}
-
-// Response
-{
-  "ctx": "<context-id>",
-  "document_count": 0,
-  "section_count": 0,
-  "index_state": "ready"
-}
-```
-
-**Search:**
-
-```json
-// Request
-{
-  "query": "<keyword/phrase query>",
-  "mode": "sections | documents | stats",
-  "limit": 20,
-  "offset": 0
-}
-```
-
-Response per mode:
-
-- **sections**:
-  ```json
-  {
-    "results": [
-      {
-        "kbid": "<26-char hash>",
-        "docids": ["<doc id>"],
-        "type": "text/markdown",
-        "heading": "Section Heading",
-        "score": 8.74,
-        "confidence": 1.0,
-        "snippet": "...matching context...",
-        "matched_terms": ["term1", "term2"],
-        "matched_fields": ["heading", "body"],
-        "created_at": "2026-06-15T10:00:00Z"
-      }
-    ],
-    "total": 42,
-    "query_time_ms": 12,
-    "query": "original query"
-  }
-  ```
-
-- **documents**:
-  ```json
-  {
-    "results": [
-      {
-        "docid": "<doc id>",
-        "title": "Document Title",
-        "score": 12.3,
-        "matching_sections": [
-          {
-            "kbid": "<26-char hash>",
-            "type": "text/markdown",
-            "heading": "Section Heading",
-            "score": 8.74,
-            "snippet": "...matching context..."
-          }
-        ]
-      }
-    ],
-    "total": 5,
-    "query_time_ms": 15,
-    "query": "original query"
-  }
-  ```
-
-- **stats**:
-  ```json
-  {
-    "total_matches": 42,
-    "query_time_ms": 8,
-    "top_terms": [
-      {
-        "term": "keyword",
-        "frequency": 15,
-        "sections_matched": 7
-      }
-    ],
-    "query": "original query"
-  }
-  ```
-
-**Read sections:**
-
-```json
-// Request
-{
-  "kbids": ["<26-char hash>", "<26-char hash>"]
-}
-
-// Response
-{
-  "sections": [
-    {
-      "kbid": "<26-char hash>",
-      "docids": ["<doc id>"],
-      "type": "text/markdown",
-      "title": "Architecture Overview",
-      "description": "High-level overview",
-      "content": "Section body text..."
-    }
-  ]
-}
-```
-
-**Add section (text):**
-
-```json
-// Request
-{
-  "content": "Section body text...",
-  "type": "text/markdown",
-  "title": "Architecture Overview",
-  "description": "High-level overview",
-  "docid": "<doc id>",
-  "source_path": "/home/user/docs/architecture.md",
-  "tags": ["architecture", "overview"],
-  "replace": false
-}
-
-// Response
-{
-  "kbid": "<26-char hash>",
-  "type": "text/markdown",
-  "title": "Architecture Overview",
-  "description": "High-level overview",
-  "indexed_terms_count": 42,
-  "elapsed_ms": 15,
-  "attached": false,
-  "superseded": null,
-  "warnings": [],
-  "near_duplicates": []
-}
-```
-
-**Add section (image):**
-
-```json
-// Request
-{
-  "content": "iVBORw0KGgoAAAANSUhEUgAA...",
-  "type": "image/png",
-  "title": "Architecture diagram",
-  "description": "Five-layer stack diagram",
-  "docid": "<doc id>"
-}
-
-// Response
-{
-  "kbid": "<26-char hash>",
-  "type": "image/png",
-  "title": "Architecture diagram",
-  "description": "Five-layer stack diagram",
-  "indexed_terms_count": 0,
-  "elapsed_ms": 8,
-  "attached": false
-}
-```
-
-The `content` field for image sections is the
-base64-encoded binary image data. The binary content
-itself is not indexed, but the `title` (required) and
-`description` (optional) attributes are tokenized and
-indexed -- so `indexed_terms_count` reflects the terms
-extracted from those attributes. The `title` field is
-**required** for image sections -- requests that omit
-it are rejected with an error.
-
-**Add section (code):**
-
-```json
-// Request
-{
-  "content": "export function greet(name: string)...",
-  "type": "text/typescript",
-  "title": "greet function",
-  "docid": "<doc id>"
-}
-
-// Response
-{
-  "kbid": "<26-char hash>",
-  "type": "text/typescript",
-  "title": "greet function",
-  "indexed_terms_count": 12,
-  "elapsed_ms": 10,
-  "attached": false
-}
-```
-
-Code sections (`text/*` and `application/json`) are
-stored as raw text and indexed through the
-tokenization pipeline. The identifier splitting stage
-handles camelCase and snake_case decomposition of
-code tokens.
-
-**Update section:**
-
-```json
-// Request
-{
-  "kbid": "<26-char hash>",
-  "content": "New section body text..."
-}
-
-// Response
-{
-  "old_kbid": "<26-char hash>",
-  "new_kbid": "<26-char hash>",
-  "reindexed_terms_count": 38,
-  "elapsed_ms": 12
-}
-```
-
-**Remove section:**
-
-```json
-// Request
-{
-  "kbid": "<26-char hash>"
-}
-
-// Response
-{
-  "removed": true,
-  "elapsed_ms": 5
-}
-```
-
-**Group sections (create document):**
-
-```json
-// Request
-{
-  "docid": "<doc id>",
-  "title": "Document Title",
-  "kbids": ["<26-char hash>", "<26-char hash>"]
-}
-
-// Response
-{
-  "docid": "<doc id>",
-  "section_count": 2,
-  "elapsed_ms": 8
-}
-```
-
-**Remove document grouping:**
-
-```json
-// Request
-{
-  "docid": "<doc id>"
-}
-
-// Response
-{
-  "removed": true,
-  "sections_ungrouped": 2,
-  "elapsed_ms": 5
-}
-```
-
-**Retrieve document:**
-
-```json
-// Request
-{
-  "docid": "<doc id>"
-}
-
-// Response
-{
-  "docid": "<doc id>",
-  "title": "Document Title",
-  "created_at": "2026-06-15T10:00:00Z",
-  "sections": [
-    {
-      "kbid": "<26-char hash>",
-      "docids": ["<doc id>"],
-      "type": "text/markdown",
-      "title": "Architecture Overview",
-      "description": "High-level overview",
-      "content": "Section body text...",
-      "position": 0
-    }
-  ]
-}
-```
-
-**Status:**
-
-```json
-// Request (no parameters)
-{}
-
-// Response
-{
-  "target_dir": "/absolute/path",
-  "document_count": 5,
-  "section_count": 42,
-  "index_size_bytes": 2048576,
-  "last_modified": "2026-06-15T12:00:00Z",
-  "cache_stats": {
-    "items_cached": 150,
-    "hit_rate": 0.87
-  }
-}
-```
-
-**Integrity check:**
-
-```json
-// Request (no parameters)
-{}
-
-// Response
-{
-  "ok": false,
-  "errors": [
-    {
-      "type": "checksum_mismatch",
-      "entity": "a1b2c3d4e5f6g7h8j9k0l1m2n3",
-      "detail": "File hash does not match filename"
-    },
-    {
-      "type": "orphan_section",
-      "entity": "x9y8z7w6v5u4t3s2r1q0p9o8n7",
-      "detail": "Section not referenced by any document"
-    },
-    {
-      "type": "broken_reference",
-      "entity": "a1b2c3d4e5f6g7h8j9k0l1m2n3",
-      "detail": "$(image: missing0id) references non-existent section"
-    }
-  ],
-  "sections_checked": 42,
-  "documents_checked": 5,
-  "references_checked": 87,
-  "elapsed_ms": 150
-}
-```
-
-The integrity check verifies:
-
-- **checksum_mismatch** -- section file content hash
-  does not match the filename (kbid). Indicates
-  corruption or manual editing.
-- **orphan_section** -- section file exists but is not
-  referenced by any document manifest. Eligible for
-  garbage collection.
-- **orphan_document** -- document manifest references
-  one or more section kbids that do not exist on disk.
-- **broken_reference** -- a marker (`$(image:)`,
-  `$(code:)`, `$(text:)`, `$(document:)`, `$(link:)`,
-  or `$(doc-link:)`) references an entity that does
-  not exist.
-- **type_mismatch** -- a marker references an entity
-  of the wrong type (e.g. `$(image:)` pointing to a
-  text section, or `$(code:)` pointing to an image
-  section).
-- **circular_reference** -- a `$(text:)` or
-  `$(document:)` marker chain loops back to an entity
-  already in the expansion path.
-- **divergent_content** -- two or more sections share
-  the same `source_key` but have different content
-  (different kbids). Indicates contradictory knowledge
-  that should have been resolved by supersession.
-
-When `ok` is `true`, the `errors` array is empty.
-
-**Garbage collection:**
-
-```json
-// Request (no parameters)
-{}
-
-// Response
-{
-  "removed_sections": 3,
-  "removed_bytes": 15360,
-  "stale_sources": 1,
-  "elapsed_ms": 25
-}
-```
-
-Garbage collection removes section files not listed
-in any document manifest. A section is eligible for
-removal when its `docids` array is empty -- it exists
-on disk but no document references it. GC does not
-remove documents, indexes, or the catalog. After
-removal, all caches are invalidated to prevent stale
-reads.
-
-`stale_sources` reports the number of sections whose
-`source_path` points to a file that no longer
-contains their heading text. These sections may be
-outdated but are not automatically removed -- they
-are reported for manual review.
-
-**Index rebuild:**
-
-```json
-// Request (no parameters)
-{}
-
-// Response
-{
-  "sections_reindexed": 39,
-  "terms_indexed": 1500,
-  "elapsed_ms": 500
-}
-```
-
-Index rebuild reconstructs all index files
-(`indexes/` directory) from section files and
-document manifests on disk. Text and code sections
-are re-tokenized per field through the
-[Query Parser](query-parser.md) pipeline and their
-per-field term frequencies re-indexed. Image section
-binary content is skipped, but their `title` and
-`description` attributes are re-tokenized and
-re-indexed. All token counts and per-field corpus
-averages are recomputed. Content markers in all
-text sections are re-parsed to rebuild
-`references.idx` (forward and back-reference
-entries). Equivalent to deleting `indexes/` and
-re-adding every section. After rebuild, all caches
-are invalidated.
-
-**Export:**
-
-```json
-// Request
-{
-  "path": "/home/user/backup/kbdb-export"
-}
-
-// Response
-{
-  "path": "/home/user/backup/kbdb-export",
-  "sections_exported": 42,
-  "documents_exported": 5,
-  "elapsed_ms": 120
-}
-```
-
-Export copies sections, documents, and `catalog.toml`
-to the specified output directory. Indexes are not
-included -- they are derived data and can be rebuilt
-with `rebuild` after importing. The export directory
-is a self-contained, git-friendly snapshot suitable
-for backup, transfer, or version control.
-
-### Error Responses
-
-Errors are returned as JSON-RPC 2.0 error objects:
-
-```json
-{
-  "error": "Description of the error",
-  "code": "ERROR_CODE"
-}
-```
-
-The worker client throws these as typed errors. The
-caller (CLI or MCP) handles them -- e.g. the CLI maps
-them to stderr output.
-
-## Tech and Algorithms Behind
-
-This section lists the specific algorithms and data
-structures used, organized by module.
-
-### fs-database.wasm -- Storage
-
-| Approach                            | Role                                    |
-|-------------------------------------|-----------------------------------------|
-| Content-Addressable Store (SHA-256, typed header) | Section files named by hash of `section <mime-type> <byte-size>\0<content>` -- type-aware deduplication, integrity |
-| Content Preprocessing Pipeline      | BOM removal, line ending normalization, Unicode NFC, whitespace trimming -- applied before hashing for text and code sections |
-| Hash-Sharded Directories            | 2-char prefix subdirectories for O(1) section lookup |
-| B+ Tree Indexes                     | Ordered key lookups in binary index files for fast term search |
-| Inverted Index                      | Maps terms to kb-ids and positions for search |
-| Atomic Writes (temp + rename)       | Crash-safe file writes                  |
-| TOML Manifests                      | Human-readable metadata for catalog, documents, and index metadata |
-
-### query-parser.wasm -- Processing
-
-See [Query Parser](query-parser.md) for full details.
-
-| Algorithm / Technique               | Role                                    |
-|-------------------------------------|-----------------------------------------|
-| RAKE                                | Keyphrase extraction                    |
-| TF-IDF                              | Term weighting                          |
-| Snowball Stemmer (Porter2)          | Word root form reduction                |
-| Unicode Segmentation                | Word boundary detection                 |
-| N-gram Extraction                   | Bigrams/trigrams for phrase matching     |
-| BM25F                               | Primary ranking function                |
-| Proximity / Zone Scoring            | Supplementary ranking signals           |
-
-### Section Identification
-
-| Approach                    | Details                                  |
-|-----------------------------|------------------------------------------|
-| Git-Style Typed Header      | Hash input is `section <mime-type> <byte-size>\0<preprocessed-content>` -- MIME type is part of the hash, so same content as different types produces different kb-ids |
-| SHA-256 + Truncation + Base32 | SHA-256 of the typed header, truncated to 128 bits (16 bytes), base32 lowercase (no padding) -- 26 alphanumeric characters |
-| Content Preprocessing       | Text and code: BOM removal, `\r\n`/`\r` to `\n`, Unicode NFC, whitespace trimming. Images: base64 string used as-is |
-
-## Concurrent Write Behavior
-
-The daemon serializes all write operations through a
-**single-writer lock**. The lock is an exclusive file
-lock on `kbdb-{ctx}.lock` in the OS temporary
-directory, keyed to the same context ID used for the
-PID file and IPC socket.
-
-- **Reads** proceed without acquiring the lock. The
-  daemon serves concurrent read requests from
-  multiple worker client connections.
-- **Writes** (add section, update section, remove
-  section, group/remove document, garbage collection,
-  index rebuild, export) acquire the lock before
-  modifying any file. If another write is in progress,
-  the request queues behind the lock.
-- The lock is released after each write operation
-  completes, not held across multiple operations.
-- CLI invocations sharing the same daemon are
-  serialized automatically.
-- Direct file access to `.kbdb` outside the daemon is
-  not supported and may cause corruption. All
-  mutations must go through the daemon's IPC API.
-
-This model is equivalent to SQLite's default locking:
-one writer at a time, concurrent readers, no external
-coordination required.
-
-## Design Constraints
-
-- `fs-database.wasm` is the sole owner of `.kbdb` I/O.
-  The worker client module delegates all storage
-  operations to the worker daemon over IPC, and the
-  daemon delegates to `fs-database.wasm` via
-  `kb-worker.wasm`. `src/cli.ts` does not read or
-  write database files directly.
-- Sections are content-addressable and identified by
-  their content hash. `fs-database.wasm` never modifies
-  a section file in place -- updates produce new files
-  with new kb-ids.
-- Source keys are advisory logical identity. The
-  content hash (kbid) remains the primary identifier.
-  Supersession is opt-in via the `replace` flag --
-  default learn behavior is unchanged.
-- Indexes are derived data. They can be rebuilt entirely
-  from sections and document manifests. Loss of index
-  files is recoverable.
-- The `.gitignore` at the database root excludes binary
-  indexes from version control by default. Sections,
-  documents, and the catalog are the version-controlled
-  source of truth.
-- The database should remain human-inspectable and
-  version-controllable where practical. Metadata files
-  use TOML; section files use TOML headers with raw
-  content bodies.
-- No external database process or server dependency.
-- The database must support concurrent read access from
-  multiple CLI invocations via the shared daemon without
-  corruption. Write operations are serialized through
-  the daemon's single-writer lock.
-- Garbage collection of unreferenced sections is
-  `fs-database.wasm`'s responsibility. Sections not
-  listed in any document manifest may be removed.
-- The cache daemon is a short-lived background process,
-  not a persistent server. See
-  [Worker Daemon](worker-daemon.md) for daemon lifecycle
-  and configuration.
-- All IPC API request and response data uses JSON
-  serialization (JSON-RPC 2.0). Inter-module
-  communication (`kb-worker.wasm` to both
-  `fs-database.wasm` and `query-parser.wasm`) uses shared
-  `WebAssembly.Memory` buffers.