@dikolab/kbdb 0.1.5 → 0.2.0

package/docs/goals/database.md

@@ -0,0 +1,1452 @@
+# Database -- Filesystem Storage Engine
+
+*A portable, directory-based database managed by
+fs-database.wasm -- no server, no cloud, just a folder
+on disk.*
+
+## Goal
+
+The knowledge base is stored in a `.kbdb` directory on
+the local filesystem. This directory is the entire
+database. There is no cloud dependency.
+
+`fs-database.wasm` is the sole owner of all read and
+write operations on this directory. It runs inside the
+worker daemon (see [Worker Daemon](worker-daemon.md)),
+which loads it alongside the other WASM modules.
+`src/cli.ts` does not read or write database files
+directly -- it uses the worker client module to send
+requests to the daemon over IPC, and the daemon
+delegates storage operations to `fs-database.wasm`.
+
+## WASM Module: fs-database.wasm
+
+`fs-database.wasm` manages the `.kbdb` directory
+structure, file formats, indexes, and data integrity.
+It is a standalone module -- it does not import
+`query-parser.wasm` or any other WASM module. Text processing
+(tokenization, query parsing, ranking) is handled by
+`query-parser.wasm` separately; `kb-worker.wasm` orchestrates
+both modules.
+
+### Dependency Position
+
+```
+query-parser.wasm          fs-database.wasm
+(standalone)         (this module -- standalone)
+      ↑                    ↑
+      └────────┬───────────┘
+               |
+        kb-worker.wasm
+   (imports both, orchestrates)
+```
+
+### Host Imports
+
+`fs-database.wasm` cannot access the filesystem
+directly. The TypeScript host (inside the daemon)
+provides file I/O functions as WASM imports:
+
+| Import       | Description                        |
+|--------------|------------------------------------|
+| `memory`     | Shared `WebAssembly.Memory`        |
+| `read_file`  | Read file contents by path         |
+| `write_file` | Write data to a file path          |
+| `remove_file`| Delete a file by path              |
+| `list_dir`   | List entries in a directory        |
+| `mkdir`      | Create a directory                 |
+| `file_exists`| Check whether a path exists        |
+
+All paths are relative to the context directory
+(`${target_dir}/.kbdb`). The host resolves them to
+absolute paths and enforces that no operation escapes
+the database directory.
+
+## Storage Directory
+
+`fs-database.wasm` accepts a `target_dir` parameter --
+the absolute path to the directory that will contain the
+`.kbdb` database. The worker client module resolves this
+path before connecting to the daemon (see
+[Worker Client](worker-client.md)):
+
+1. If a path is specified by the caller (e.g. via a CLI
+   flag `--db <path>`), use that path.
+2. Otherwise, use the current working directory (CWD).
+
+The database directory is always `${target_dir}/.kbdb`.
+`fs-database.wasm` never resolves CWD itself -- it
+always receives an absolute path from the host.
+
+If `.kbdb` does not exist at the resolved path, the
+worker client throws. The caller is responsible for
+database initialization before creating a worker
+client -- e.g. the CLI prompts the user to create it.
+
+The directory is portable. It can be copied, moved, or
+version-controlled. Its contents are self-contained -- no
+external state or configuration is required.
+
+The `target_dir` also serves as the identity key for daemon
+discovery. Each unique `target_dir` gets its own independent
+cache daemon. Daemon PID files are stored in
+the OS temporary directory, not inside `.kbdb`. Multiple
+databases on the same machine run independent daemons with
+independent caches. See
+[Worker Daemon](worker-daemon.md) for daemon details.
+
+## Directory Layout
+
+The `.kbdb` directory is structured like a PostgreSQL
+directory-format dump: a human-readable catalog at the
+root, one subdirectory per data table (sections,
+documents), and a separate directory for binary indexes.
+Metadata files use TOML for human inspectability.
+Indexes are binary but derived and rebuildable.
+
+```
+.kbdb/
+├── catalog.toml           # database table of contents:
+│                          # section count, document
+│                          # count, timestamps, integrity
+├── sections/
+│   ├── ab/
+│   │   └── cdef1234...sec # section file (TOML header
+│   │                      # + raw content body)
+│   └── ...
+├── documents/
+│   └── {docid}.toml       # document manifest: title +
+│                          # ordered section list
+├── indexes/
+│   ├── meta.toml          # index metadata: last rebuild
+│   │                      # timestamp, term count, stats
+│   ├── terms.idx          # inverted index (binary,
+│   │                      # B+ tree)
+│   ├── positions.idx      # positional data (binary)
+│   ├── corpus.dat         # corpus-wide statistics
+│   │                      # (binary)
+│   └── references.idx     # bidirectional reference
+│                          # graph (binary)
+├── worker.toml            # cache daemon config
+│                          # (see Worker)
+└── .gitignore             # excludes indexes/ from
+                           # version control
+```
+
+### Catalog (`catalog.toml`)
+
+The catalog is the database's table of contents --
+analogous to `toc.dat` in a PostgreSQL directory dump.
+It is the first file `fs-database.wasm` reads when
+opening a database. It records:
+
+- Database format version.
+- Total section count and document count.
+- Creation and last-modified timestamps.
+- Integrity checksums.
+
+The catalog is updated on every write operation.
+
+Example:
+
+```toml
+[database]
+version = 1
+created_at = "2026-06-15T10:00:00Z"
+last_modified = "2026-06-15T12:00:00Z"
+
+[counts]
+sections = 42
+documents = 5
+
+[integrity]
+checksum = "sha256:abcdef..."
+```
+
+### Sections (`sections/`)
+
+One file per section, stored in hash-sharded
+subdirectories (first two characters of the kb-id) to
+avoid large flat directories.
+
+Every section's kb-id is derived from a SHA-256 hash
+of a **typed header plus preprocessed content**,
+following the same content-addressing pattern used by
+Git objects (`blob <size>\0<content>`). The hash input
+is:
+
+```
+section <mime-type> <byte-size>\0<preprocessed-content>
+```
+
+- `<mime-type>` -- the section's MIME type string
+  (e.g. `text/markdown`, `text/typescript`,
+  `image/png`).
+- `<byte-size>` -- the byte length of the preprocessed
+  content, expressed as a decimal ASCII string (like
+  Git).
+- `\0` -- a literal null byte separator.
+- `<preprocessed-content>` -- the content after the
+  preprocessing pipeline (see **Content Preprocessing**
+  below).
+
+The SHA-256 digest is truncated to the first 16 bytes
+(128 bits) and base32-encoded (no padding, lowercase)
+to produce a 26-character alphanumeric identifier that
+is deterministic, collision-resistant, and URL-safe.
+
+Because the MIME type is part of the hash input,
+identical content stored as different types (e.g.
+`text/markdown` vs `text/typescript`) produces
+different kb-ids.
+
+Each section file uses a TOML frontmatter header
+followed by a `---` separator and the content body.
+The header is human-readable; the content after the
+separator is stored verbatim with no escaping.
+
+The `type` field identifies the content kind:
+
+- **Text sections** -- `type` is `"text/markdown"`.
+  The content body after `---` is raw text (markdown)
+  stored verbatim.
+- **Code sections** -- `type` is a `text/*` MIME type
+  (e.g. `"text/typescript"`, `"text/html"`,
+  `"text/css"`) or `"application/json"`. The content
+  body after `---` is the raw source text stored
+  verbatim. Code sections are indexed -- the
+  tokenization pipeline processes their content,
+  including identifier splitting for camelCase and
+  snake_case terms.
+- **Image sections** -- `type` is an `image/*` MIME
+  type (e.g. `"image/png"`, `"image/jpeg"`,
+  `"image/webp"`). The content body after `---` is the
+  image file's binary data encoded as **base64**. The
+  kb-id is the SHA-256 hash of the base64-encoded
+  string (not the raw binary), so identical base64
+  input always produces the same kb-id.
+
+All section types use proper MIME type strings. The
+bare string `"text"` is no longer used; text sections
+use `"text/markdown"` as their canonical type.
+
+Image section binary content is not indexed directly.
+However, image sections can be searchable via their
+`title` and `description` attributes (see below).
+They are also retrievable by kb-id or through
+document grouping.
+
+Several metadata and pre-computed fields are stored
+in the TOML header:
+
+- **`title`** -- A short descriptive title for the
+  section. **Required** for image sections (`image/*`)
+  since images have no indexable text content and need
+  a human-readable identifier. Optional for text
+  (`text/markdown`) and code (`text/*`,
+  `application/json`) sections. Not included in the
+  kbid hash -- changing the title does not change the
+  section's identity. When present, the title is
+  tokenized and indexed with the **heading** field
+  weight (2.0x) for search ranking. See
+  [Query Result](query-result.md) for field weight
+  details.
+- **`description`** -- An optional longer description
+  of the section's content or purpose. Always optional
+  for all section types. Not included in the kbid hash.
+  When present, the description is tokenized and
+  indexed with the **body** field weight (1.0x) for
+  search ranking.
+- **`token_count`** -- Total number of tokens after
+  tokenization (u32). Sum of all per-field token
+  counts. Used for BM25F length normalization and
+  zone scoring. Computed by `query-parser.wasm` during
+  indexing.
+- **`heading_token_count`** -- Number of tokens from
+  the `title` attribute (u32). Used for per-field
+  BM25 length normalization.
+- **`body_token_count`** -- Number of tokens from the
+  body text content and `description` attribute (u32).
+  Used for per-field BM25 length normalization.
+- **`code_token_count`** -- Number of tokens from
+  code content (u32). Non-zero only for code sections.
+  Used for per-field BM25 length normalization.
+
+Token count fields are computed by `query-parser.wasm`
+during tokenization and passed to `fs-database.wasm`
+for storage. They are pre-computed once at insertion
+time and reused on every search -- trading a small
+write-time cost for significant query-time speed.
+
+The `add_section` operation rejects requests with
+`image/*` type that do not include a `title`. Text
+and code sections may omit both fields.
+
+### Embedding Sections in Text Content
+
+Text sections can contain **markers** -- escape
+sequences that embed or link to other sections and
+documents using the `$(type: value)` syntax. Six
+marker types are supported: four for embedding
+content inline (`$(image:)`, `$(code:)`, `$(text:)`,
+`$(document:)`) and two for cross-references
+(`$(link:)`, `$(doc-link:)`).
+
+Markers are stored verbatim in the content body --
+they are not expanded or resolved at storage time.
+They are structural references, not searchable text;
+the tokenization pipeline strips them before
+indexing.
+
+See [Document](document.md) for the full content
+format specification, all marker types, syntax
+rules, rendering behaviour, and the MIME-to-language
+mapping table for code embeddings. See
+[Content Parser](content-parser.md) for how markers
+are extracted from content as structured data.
+
+#### Section File Examples
+
+**Text section** (`sections/ab/cdef1234...sec`):
+
+```
+kbid = "abcdef1234567890abcdef1234"
+docids = ["doc-5678"]
+type = "text/markdown"
+title = "Architecture Overview"
+description = "High-level overview of the system"
+size = 1234
+token_count = 187
+heading_token_count = 2
+body_token_count = 187
+code_token_count = 0
+created_at = "2026-06-15T10:00:00Z"
+checksum = "sha256:abcdef..."
+---
+Section body text here.
+
+$(image: 7fa93b2e4c8d1076a93b2e4c8d)
+
+$(code: c4e8a91d3f7b20564e8a91d3f7)
+
+This content continues after the embedded sections.
+No escaping or encoding is applied.
+```
+
+**Code section** (`sections/c4/e8a91d...sec`):
+
+```
+kbid = "c4e8a91d3f7b20564e8a91d3f7"
+docids = ["doc-5678"]
+type = "text/typescript"
+title = "greet function"
+size = 256
+token_count = 22
+heading_token_count = 2
+body_token_count = 0
+code_token_count = 22
+created_at = "2026-06-15T10:15:00Z"
+checksum = "sha256:1a2b3c..."
+---
+export function greet(name: string): string {
+   return `Hello, ${name}!`;
+}
+```
+
+**Image section** (`sections/7f/a93b2e...sec`):
+
+```
+kbid = "7fa93b2e4c8d1076a93b2e4c8d"
+docids = ["doc-5678"]
+type = "image/png"
+title = "Architecture diagram"
+description = "Five-layer stack diagram"
+size = 48576
+token_count = 5
+heading_token_count = 2
+body_token_count = 3
+code_token_count = 0
+created_at = "2026-06-15T10:30:00Z"
+checksum = "sha256:fedcba..."
+---
+iVBORw0KGgoAAAANSUhEUgAA...base64 data...
+```
+
+The image section's `token_count` is 5 (from title +
+description tokens only). Binary content itself
+produces zero tokens.
+
+#### Content Preprocessing (kb-id computation)
+
+Before computing the kb-id, text and code section
+content passes through a preprocessing pipeline. The
+pipeline normalizes content so that semantically
+identical inputs -- differing only in BOM, line
+endings, Unicode representation, or surrounding
+whitespace -- produce the same kb-id.
+
+The pipeline steps, applied in order:
+
+1. **BOM removal** -- Strip the UTF-8 BOM (U+FEFF) if
+   present at the start of the content. The BOM is an
+   encoding artifact, not meaningful content.
+2. **Line ending normalization** -- Convert all `\r\n`
+   (Windows) and bare `\r` (old Mac) sequences to `\n`
+   (Unix). Ensures content created on different
+   platforms produces the same kb-id.
+3. **Unicode NFC normalization** -- Compose decomposed
+   Unicode sequences into their canonical composed form
+   (NFC). For example, `e` + combining acute accent
+   becomes `é` (U+00E9). This is the same
+   normalization used in the tokenization pipeline (see
+   [Query Parser](query-parser.md)).
+4. **Whitespace trimming** -- Remove all leading and
+   trailing whitespace characters (spaces, tabs,
+   newlines, carriage returns, and other Unicode
+   whitespace and control characters).
+
+This pipeline applies to **text and code sections
+only**. Image sections use the base64-encoded string
+as-is -- no preprocessing is applied.
+
+The preprocessed content is what is stored on disk.
+The pipeline runs once at ingest time; subsequent
+reads return the already-preprocessed content. The
+`<byte-size>` in the hash header is computed from the
+content **after** preprocessing.
+
+Example hash input construction:
+
+```
+Given:
+  type    = "text/markdown"
+  content (after preprocessing) = "Hello, world!"
+                                  (13 bytes)
+
+Hash input (41 bytes):
+  section text/markdown 13\0Hello, world!
+
+  ├─ header: "section text/markdown 13"
+  ├─ null byte: \0
+  └─ preprocessed content: "Hello, world!"
+
+kbid = base32(sha256(hash_input)[..16])
+     → 26-character lowercase alphanumeric string
+```
+
+Sections are **content-addressable** -- identical
+preprocessed content stored with the same type always
+produces the same kbid and is stored once (automatic
+deduplication). Because the MIME type is part of the
+hash input, the same raw content stored as different
+types (e.g. `text/markdown` vs `text/typescript`)
+produces different kbids and different section files.
+When identical content is added with the same type:
+
+- If a different docid is provided, the existing
+  section's `docids` array is updated to include the
+  new docid -- no new section file is created
+  (`attached: true`).
+- If no docid is provided, the section is skipped
+  (`skipped: true`).
+- If the provided docid is already in the `docids`
+  array, the operation is idempotent
+  (`skipped: false`, `attached: false`).
+
+A section can belong to **zero or more** documents.
+The `docids` field is an array of document IDs. The
+relationship between sections and documents is
+**many-to-many**: a section may appear in multiple
+documents, and a document contains multiple sections.
+
+Updating a section's content creates a new file with
+a new kbid; the old file is eligible for garbage
+collection.
+
+### Documents (`documents/`)
+
+One TOML file per document. A document groups sections
+under a title with a defined order. A section may
+appear in multiple documents (many-to-many). The
+document manifest contains:
+
+- Document ID (`docid`).
+- Title.
+- Creation timestamp.
+- An ordered list of section references (kb-ids).
+
+Example (`documents/{docid}.toml`):
+
+```toml
+[document]
+docid = "doc-5678"
+title = "Getting Started Guide"
+created_at = "2026-06-15T10:00:00Z"
+
+[[sections]]
+kbid = "abcdef1234567890abcdef1234"
+position = 0
+
+[[sections]]
+kbid = "fedcba4321098765fedcba4321"
+position = 1
+```
+
+Human-readable -- can be inspected and diffed in git.
+
+### Indexes (`indexes/`)
+
+Binary format, not human-readable. Optimized for fast
+term lookup using B+ tree structures. Rebuilt from
+sections and documents on demand. Can be excluded from
+version control via `.gitignore`.
+
+Three index files support the search and ranking
+pipeline:
+
+- **Inverted index** (`terms.idx`) -- B+ tree mapping
+  normalized terms to posting lists. Each posting
+  entry contains a kbid and **per-field term
+  frequencies**: `heading_tf` (from title), `body_tf`
+  (from body text + description), and `code_tf` (from
+  code content). Storing per-field TFs separately
+  (rather than a single aggregated weight) allows
+  BM25F field weights to be tuned at query time
+  without rebuilding indexes. See
+  [Query Result](query-result.md) for how these
+  values feed into the BM25F scoring formula.
+
+- **Positional index** (`positions.idx`) -- B+ tree
+  mapping `(term, kbid)` pairs to position arrays
+  (zero-based token offsets). Supports phrase queries,
+  proximity scoring, and zone scoring.
+
+- **Corpus statistics** (`corpus.dat`) -- Corpus-wide
+  aggregates: `total_sections` (u32),
+  `total_terms` (u32), `avg_section_length` (f32),
+  `avg_heading_length` (f32), `avg_body_length` (f32),
+  `avg_code_length` (f32), and per-term document
+  frequencies. Per-field average lengths enable
+  accurate per-field BM25 normalization -- a 3-word
+  title is not penalized against a 500-word average
+  body length.
+
+- **Reference graph** (`references.idx`) --
+  Bidirectional index mapping content marker
+  references between sections and documents. Each
+  entry records:
+
+  | Field            | Type   | Description         |
+  |------------------|--------|---------------------|
+  | `source_kbid`    | string | Section containing the marker |
+  | `target_id`      | string | Referenced kbid or docid |
+  | `marker_type`    | enum   | Image, Code, Text, Document, Link, DocLink |
+  | `context_snippet` | string | ~150 chars of surrounding text where the marker appears |
+  | `direction`      | enum   | `forward` or `back` |
+
+  For every content marker `$(type: value)` in a
+  section's content, the index stores two entries:
+  a **forward** entry (`source → target`) and a
+  **back** entry (`target ← source`). This enables
+  the `recall` tool to look up both "what this
+  section references" and "what references this
+  section" in O(1) without re-parsing content.
+
+  **Build process:** When a section is added or
+  updated, `fs-database.wasm` calls
+  `extract_content_markers()` on its content,
+  extracts a ~150-char context snippet around each
+  marker, and writes forward and back-reference
+  entries.
+
+  **Maintenance:** When a section is removed, all
+  its forward-reference entries and all
+  back-reference entries pointing to it are
+  deleted.
+
+  **Rebuild:** `rebuild` regenerates
+  `references.idx` from section content alongside
+  the other index files. The reference graph is a
+  derived artifact -- the content markers in
+  section bodies are the source of truth.
+
+The `meta.toml` file inside `indexes/` provides
+human-readable metadata about the indexes: last rebuild
+timestamp, total term count, and corpus statistics
+summary. This file is informational only -- the binary
+index files are the source of truth for search.
+
+### Storage Design
+
+The directory layout is optimized for **read speed**:
+
+- **O(1) section lookup** -- Hash-sharded directories
+  allow direct file access by kb-id without scanning.
+- **O(log n) term lookup** -- B+ tree indexes support
+  efficient range and equality queries on terms.
+- **Catalog-first discovery** -- `catalog.toml` provides
+  section and document counts without scanning
+  directories.
+- **Separated metadata and content** -- TOML manifests
+  (small, frequently read) are separate from section
+  content files (large, read on demand).
+
+Write operations may be slower but are designed to be
+safe:
+
+- **Atomic writes** -- New files are written to a
+  temporary path and renamed into place, ensuring no
+  partial writes on crash.
+- **Catalog update last** -- The catalog is updated
+  after all data files are written, so a crash mid-write
+  leaves the catalog consistent with the last complete
+  state.
+- **Index rebuilds** -- Index files can be rebuilt
+  entirely from sections and document manifests. Index
+  corruption is recoverable.
+
+## What the Database Stores
+
+The database holds three categories of data:
+
+1. **Sections** -- Individual content units stored as
+   files in `sections/`. Each identified by its content
+   hash (kb-id). The section file contains a TOML
+   header with metadata (including pre-computed token
+   counts for ranking) and the content body (raw text
+   for text and code sections, base64 for image
+   sections). Identical content is deduplicated
+   automatically.
+
+2. **Documents** -- Groupings of sections stored as
+   TOML manifests in `documents/`. Each document has a
+   title and an ordered list of section kb-ids. A
+   document does not contain content directly -- it
+   references sections.
+
+3. **Indexes** -- Binary files in `indexes/` mapping
+   extracted terms to sections (kb-ids) with per-field
+   term frequencies, positional data, corpus-wide
+   statistics (including per-field averages), and a
+   bidirectional reference graph of content marker
+   relationships. Supports term lookup, phrase
+   queries, BM25F ranking with tunable field weights,
+   and O(1) forward/back-reference lookup for the
+   `recall` tool. Derived data -- rebuildable from
+   sections and documents.
+
+## Operations
+
+`fs-database.wasm` exposes the following operations.
+These are called by `kb-worker.wasm` inside the daemon,
+not by the worker client or TypeScript callers
+directly.
+
+### Section Operations
+
+1. **Read sections** -- Given a list of kb-ids, return
+   the section content and metadata for each. Reads
+   from `sections/{prefix}/{kbid}.sec`. Returns the
+   TOML header fields and the raw content body.
+
+2. **Add section** -- Given a type, section content,
+   and optional title and description, create the
+   section file. For text and code sections, runs the
+   content through the preprocessing pipeline (BOM
+   removal, line ending normalization, Unicode NFC,
+   whitespace trimming). Computes the kb-id using the
+   typed header format
+   (`section <mime-type> <byte-size>\0<content>`).
+
+   During indexing, `kb-worker.wasm` calls
+   `query-parser.wasm` to tokenize the content per
+   field (title → heading, body text + description →
+   body, code content → code) and compute per-field
+   term frequencies and token counts. These
+   pre-computed values are passed to
+   `fs-database.wasm`, which writes the TOML header
+   (kbid, docids, type, title, description, size,
+   token_count, heading_token_count, body_token_count,
+   code_token_count, timestamp, checksum), the
+   separator, and the content body (preprocessed text
+   for text and code sections, base64 for image
+   sections). Updates `catalog.toml` counts. Updates
+   the inverted index with per-field term frequencies,
+   the positional index with term positions, and
+   corpus statistics (including per-field average
+   lengths).
+
+   Image sections (`image/*` type) have their binary
+   content stored but not indexed. However, their
+   `title` (required) and `description` (optional)
+   attributes are tokenized and indexed, making image
+   sections searchable via their metadata.
+
+   **Validation:** requests with `image/*` type that
+   omit `title` are rejected with an error. Text and
+   code sections may omit `title`.
+
+3. **Update section** -- Given a kb-id and new content,
+   replace the section's content. Since kb-ids are
+   content-addressed, this creates a new section file
+   with a new kb-id (including fresh token counts),
+   updates all document references from the old kb-id
+   to the new one, and marks the old section file for
+   garbage collection. Triggers index update: removes
+   old per-field term frequencies and positions,
+   re-tokenizes the new content per field, and stores
+   updated per-field TFs, positions, and token counts.
+   Updates corpus statistics.
+
+4. **Remove section** -- Given a kb-id, delete the
+   section file. Removes the kb-id from any document
+   that references it. Updates `catalog.toml` counts.
+   Triggers index update to remove the section's terms.
+
+### Document Operations
+
+5. **Group sections into document** -- Given a list of
+   kb-ids, a document title, and a docid, create a
+   document manifest in `documents/{docid}.toml`. The
+   manifest records the title, timestamp, and ordered
+   section list. Adds the docid to each referenced
+   section's `docids` array. Updates `catalog.toml`.
+
+6. **Remove document grouping** -- Given a docid,
+   delete the document manifest. The sections
+   themselves are not deleted -- only the grouping is
+   removed. Removes the docid from each section's
+   `docids` array. Updates `catalog.toml`.
+
+### Index and Storage Management
+
+7. **Manage indexes** -- Maintain the inverted index,
+   positional index, and corpus statistics in
+   `indexes/`. `fs-database.wasm` receives
+   pre-computed per-field term frequencies, term
+   positions, and token counts from the caller
+   (produced by `query-parser.wasm` via
+   `kb-worker.wasm` orchestration -- see
+   [Query Parser](query-parser.md)) and stores them
+   in B+ tree structures optimized for read speed.
+
+   The inverted index stores per-field term
+   frequencies (`heading_tf`, `body_tf`, `code_tf`)
+   per posting entry rather than a single aggregated
+   weight, enabling query-time field weight tuning.
+   Corpus statistics include per-field average lengths
+   (`avg_heading_length`, `avg_body_length`,
+   `avg_code_length`) for per-field BM25
+   normalization. See [Query Result](query-result.md)
+   for how these pre-computed values feed into the
+   ranking pipeline.
+
+   Index updates are triggered by section add, update,
+   and remove operations. For text and code sections,
+   body content is tokenized and indexed. For all
+   section types with `title` and/or `description`
+   attributes, those attributes are tokenized and
+   indexed with their respective field weights. Full
+   index rebuilds can be triggered explicitly and
+   reconstruct all index files, token counts, and
+   corpus statistics from the section and document
+   data on disk.
+
+### Search
+
+8. **Execute query plan** -- Given a pre-parsed query
+   plan (produced by `query-parser.wasm` via
+   `kb-worker.wasm` orchestration), execute the plan
+   against the indexes and return raw matches: kb-ids
+   with per-field term frequencies (`heading_tf`,
+   `body_tf`, `code_tf`), positional data, token
+   counts, and per-field token counts.
+   `fs-database.wasm` does not parse queries or rank
+   results -- it performs index lookups only. Ranking
+   is handled by `query-parser.wasm` after
+   `kb-worker.wasm` retrieves the raw results. See
+   [Query Result](query-result.md) for the full
+   ranking pipeline and score computation.
+
+### Retrieval
+
+9. **Retrieve document** -- Given a docid, return the
+   document manifest and all its sections in position
+   order, including section content, type, title, and
+   description.
+
+10. **Status** -- Return database metadata: target
+    directory, document count, section count, index
+    size, last-modified timestamp, and cache
+    statistics.
+
+### Lifecycle
+
+11. **Initialization** -- Given a target directory
+    path, initialize a new `.kbdb` database. Creates
+    the directory structure, empty catalog, and empty
+    index files. Returns the context ID and initial
+    counts.
+
+### Integrity and Maintenance
+
+12. **Integrity check** -- Verify section file
+    checksums match their content. Detect orphaned
+    sections (not referenced by any document),
+    orphaned documents (referencing missing sections),
+    broken embedding references -- markers whose
+    target does not exist or has the wrong type -- and
+    circular reference chains in `$(text:)` and
+    `$(document:)` markers.
+13. **Garbage collection** -- Remove section files not
+    referenced by any document manifest.
+14. **Index rebuild** -- Reconstruct all index files
+    from sections and documents. Re-tokenizes all text
+    and code sections per field, recomputes per-field
+    term frequencies and token counts (updating
+    section headers), and rebuilds corpus statistics
+    including per-field average lengths. Recovers
+    from index corruption.
+
+## Relationship to query-parser.wasm
+
+Tokenization, keyword extraction, query parsing, and
+ranking are provided by `query-parser.wasm` -- a separate
+standalone module. See [Query Parser](query-parser.md) for the
+full specification of the tokenization pipeline, query
+parsing stages, statistical keyword extraction (RAKE,
+TF-IDF), and ranking functions (BM25F, proximity
+scoring, zone scoring).
+
+`fs-database.wasm` does **not** import or call
+`query-parser.wasm` directly. Instead, `kb-worker.wasm`
+orchestrates both modules:
+
+- **Indexing** -- When a section is added or updated,
+  `kb-worker.wasm` calls `query-parser.wasm` to
+  tokenize content per field (title → heading, body
+  text + description → body, code content → code)
+  and compute per-field term frequencies and token
+  counts. It then passes the pre-computed per-field
+  TFs, positions, and token counts to
+  `fs-database.wasm` for storage in the inverted
+  index and section header. Image sections (`image/*`
+  type) have their binary content stored without
+  indexing, but their `title` and `description`
+  attributes are tokenized and indexed.
+- **Searching** -- When a query arrives,
+  `kb-worker.wasm` calls `query-parser.wasm` to parse
+  the query into a retrieval plan, passes the plan to
+  `fs-database.wasm` for index lookups (returning
+  per-field TFs, positions, and token counts), then
+  calls `query-parser.wasm`'s ranking functions to
+  score and order the raw results. See
+  [Query Result](query-result.md) for the full
+  ranking pipeline.
+
+### Result Granularity
+
+Three result modes are available. See
+[Query Result](query-result.md) for ranking
+algorithms, score computation, and result mode
+details.
+
+- **Sections** -- Individual matching sections,
+  identified by kb-id.
+- **Whole documents** -- Full documents containing
+  matches, reassembled from the ordered sections in the
+  document manifest.
+- **Statistics** -- Result counts and ranking summaries
+  without content bodies.
+
+### Result Structure
+
+Each result includes the section's kb-id, the parent
+document ID (docid), a relevance score, a content
+snippet showing the matching context, and the matched
+terms. Results are ordered by descending score.
+
+## Worker Architecture and Caching
+
+See [Worker Daemon](worker-daemon.md) for the full
+specification of the cache daemon, worker
+configuration, cache invalidation, and daemon
+lifecycle.
+
+## API Contracts
+
+The worker client module communicates with the worker
+daemon over IPC using JSON-RPC 2.0 (see
+[Worker Daemon](worker-daemon.md) for the IPC protocol
+and [Worker Client](worker-client.md) for the client
+specification). Inside the daemon,
+`kb-worker.wasm` calls `fs-database.wasm` for database
+operations. All data at the IPC boundary uses JSON
+serialization.
+
+The daemon may use alternative formats (shared
+`WebAssembly.Memory` buffers, pointer/length pairs)
+internally between `kb-worker.wasm` and
+`fs-database.wasm` for performance. The module-internal
+format is an implementation detail that does not affect
+the worker client's IPC API.
+
+### TypeScript Layer Role
+
+`src/cli.ts` is a thin relay that delegates all
+database work to the worker client module.
+For example, the CLI's responsibilities are limited to:
+
+1. **Receive** -- Accept the request from CLI arguments.
+2. **Create worker client** -- Call
+   `createWorkerClient({ targetDir })`, which handles
+   daemon discovery, spawning, and IPC connection (see
+   [Worker Client](worker-client.md)).
+3. **Call** -- Invoke the appropriate worker client
+   method (e.g. `client.search()`,
+   `client.addSection()`).
+4. **Wait** -- Block until the worker client returns
+   the typed response.
+5. **Relay or reformat** -- Return the response based
+   on the requested output format:
+   - If the requested format is JSON (the default), or
+     no format is specified, relay the response directly
+     to stdout -- no transformation needed.
+   - If the requested format is plain text (CLI only,
+     via `--format` flag), reformat the response before
+     outputting.
+6. **Disconnect** -- Call `client.disconnect()`.
+
+The TypeScript layer does not parse, enrich, filter, or
+transform the worker client's response when relaying as
+JSON. The response is the final output. The TypeScript
+layer only performs reformatting when an alternative
+output format is explicitly requested via CLI flags.
+
+### Operations
+
+These are the operations that `fs-database.wasm`
+exposes to `kb-worker.wasm` inside the daemon. The
+worker client does not call them directly -- it uses
+the low-level IPC protocol documented in
+[Worker Daemon](worker-daemon.md) (query/token
+pattern). `kb-worker.wasm` orchestrates the WASM
+calls and returns results through the daemon's IPC
+layer.
+
+**Initialization:**
+
+```json
+// Request
+{
+  "target_dir": "<absolute path>"
+}
+
+// Response
+{
+  "ctx": "<context-id>",
+  "document_count": 0,
+  "section_count": 0,
+  "index_state": "ready"
+}
+```
+
+**Search:**
+
+```json
+// Request
+{
+  "query": "<keyword/phrase query>",
+  "mode": "sections | documents | stats",
+  "limit": 20,
+  "offset": 0
+}
+```
+
+Response per mode:
+
+- **sections**:
+  ```json
+  {
+    "results": [
+      {
+        "kbid": "<26-char hash>",
+        "docids": ["<doc id>"],
+        "type": "text/markdown",
+        "heading": "Section Heading",
+        "score": 8.74,
+        "snippet": "...matching context...",
+        "matched_terms": ["term1", "term2"]
+      }
+    ],
+    "total": 42,
+    "query_time_ms": 12,
+    "query": "original query"
+  }
+  ```
+
+- **documents**:
+  ```json
+  {
+    "results": [
+      {
+        "docid": "<doc id>",
+        "title": "Document Title",
+        "score": 12.3,
+        "matching_sections": [
+          {
+            "kbid": "<26-char hash>",
+            "type": "text/markdown",
+            "heading": "Section Heading",
+            "score": 8.74,
+            "snippet": "...matching context..."
+          }
+        ]
+      }
+    ],
+    "total": 5,
+    "query_time_ms": 15,
+    "query": "original query"
+  }
+  ```
+
+- **stats**:
+  ```json
+  {
+    "total_matches": 42,
+    "query_time_ms": 8,
+    "top_terms": [
+      {
+        "term": "keyword",
+        "frequency": 15,
+        "sections_matched": 7
+      }
+    ],
+    "query": "original query"
+  }
+  ```
+
+**Read sections:**
+
+```json
+// Request
+{
+  "kbids": ["<26-char hash>", "<26-char hash>"]
+}
+
+// Response
+{
+  "sections": [
+    {
+      "kbid": "<26-char hash>",
+      "docids": ["<doc id>"],
+      "type": "text/markdown",
+      "title": "Architecture Overview",
+      "description": "High-level overview",
+      "content": "Section body text..."
+    }
+  ]
+}
+```
+
+**Add section (text):**
+
+```json
+// Request
+{
+  "content": "Section body text...",
+  "type": "text/markdown",
+  "title": "Architecture Overview",
+  "description": "High-level overview",
+  "docid": "<doc id>"
+}
+
+// Response
+{
+  "kbid": "<26-char hash>",
+  "type": "text/markdown",
+  "title": "Architecture Overview",
+  "description": "High-level overview",
+  "indexed_terms_count": 42,
+  "elapsed_ms": 15,
+  "attached": false
+}
+```
+
+**Add section (image):**
+
+```json
+// Request
+{
+  "content": "iVBORw0KGgoAAAANSUhEUgAA...",
+  "type": "image/png",
+  "title": "Architecture diagram",
+  "description": "Five-layer stack diagram",
+  "docid": "<doc id>"
+}
+
+// Response
+{
+  "kbid": "<26-char hash>",
+  "type": "image/png",
+  "title": "Architecture diagram",
+  "description": "Five-layer stack diagram",
+  "indexed_terms_count": 0,
+  "elapsed_ms": 8,
+  "attached": false
+}
+```
+
+The `content` field for image sections is the
+base64-encoded binary image data. The binary content
+itself is not indexed, but the `title` (required) and
+`description` (optional) attributes are tokenized and
+indexed -- so `indexed_terms_count` reflects the terms
+extracted from those attributes. The `title` field is
+**required** for image sections -- requests that omit
+it are rejected with an error.
+
+**Add section (code):**
+
+```json
+// Request
+{
+  "content": "export function greet(name: string)...",
+  "type": "text/typescript",
+  "title": "greet function",
+  "docid": "<doc id>"
+}
+
+// Response
+{
+  "kbid": "<26-char hash>",
+  "type": "text/typescript",
+  "title": "greet function",
+  "indexed_terms_count": 12,
+  "elapsed_ms": 10,
+  "attached": false
+}
+```
+
+Code sections (`text/*` and `application/json`) are
+stored as raw text and indexed through the
+tokenization pipeline. The identifier splitting stage
+handles camelCase and snake_case decomposition of
+code tokens.
+
+**Update section:**
+
+```json
+// Request
+{
+  "kbid": "<26-char hash>",
+  "content": "New section body text..."
+}
+
+// Response
+{
+  "old_kbid": "<26-char hash>",
+  "new_kbid": "<26-char hash>",
+  "reindexed_terms_count": 38,
+  "elapsed_ms": 12
+}
+```
+
+**Remove section:**
+
+```json
+// Request
+{
+  "kbid": "<26-char hash>"
+}
+
+// Response
+{
+  "removed": true,
+  "elapsed_ms": 5
+}
+```
+
+**Group sections (create document):**
+
+```json
+// Request
+{
+  "docid": "<doc id>",
+  "title": "Document Title",
+  "kbids": ["<26-char hash>", "<26-char hash>"]
+}
+
+// Response
+{
+  "docid": "<doc id>",
+  "section_count": 2,
+  "elapsed_ms": 8
+}
+```
+
+**Remove document grouping:**
+
+```json
+// Request
+{
+  "docid": "<doc id>"
+}
+
+// Response
+{
+  "removed": true,
+  "sections_ungrouped": 2,
+  "elapsed_ms": 5
+}
+```
+
+**Retrieve document:**
+
+```json
+// Request
+{
+  "docid": "<doc id>"
+}
+
+// Response
+{
+  "docid": "<doc id>",
+  "title": "Document Title",
+  "created_at": "2026-06-15T10:00:00Z",
+  "sections": [
+    {
+      "kbid": "<26-char hash>",
+      "docids": ["<doc id>"],
+      "type": "text/markdown",
+      "title": "Architecture Overview",
+      "description": "High-level overview",
+      "content": "Section body text...",
+      "position": 0
+    }
+  ]
+}
+```
+
+**Status:**
+
+```json
+// Request (no parameters)
+{}
+
+// Response
+{
+  "target_dir": "/absolute/path",
+  "document_count": 5,
+  "section_count": 42,
+  "index_size_bytes": 2048576,
+  "last_modified": "2026-06-15T12:00:00Z",
+  "cache_stats": {
+    "items_cached": 150,
+    "hit_rate": 0.87
+  }
+}
+```
+
+**Integrity check:**
+
+```json
+// Request (no parameters)
+{}
+
+// Response
+{
+  "ok": false,
+  "errors": [
+    {
+      "type": "checksum_mismatch",
+      "entity": "a1b2c3d4e5f6g7h8j9k0l1m2n3",
+      "detail": "File hash does not match filename"
+    },
+    {
+      "type": "orphan_section",
+      "entity": "x9y8z7w6v5u4t3s2r1q0p9o8n7",
+      "detail": "Section not referenced by any document"
+    },
+    {
+      "type": "broken_reference",
+      "entity": "a1b2c3d4e5f6g7h8j9k0l1m2n3",
+      "detail": "$(image: missing0id) references non-existent section"
+    }
+  ],
+  "sections_checked": 42,
+  "documents_checked": 5,
+  "references_checked": 87,
+  "elapsed_ms": 150
+}
+```
+
+The integrity check verifies:
+
+- **checksum_mismatch** -- section file content hash
+  does not match the filename (kbid). Indicates
+  corruption or manual editing.
+- **orphan_section** -- section file exists but is not
+  referenced by any document manifest. Eligible for
+  garbage collection.
+- **orphan_document** -- document manifest references
+  one or more section kbids that do not exist on disk.
+- **broken_reference** -- a marker (`$(image:)`,
+  `$(code:)`, `$(text:)`, `$(document:)`, `$(link:)`,
+  or `$(doc-link:)`) references an entity that does
+  not exist.
+- **type_mismatch** -- a marker references an entity
+  of the wrong type (e.g. `$(image:)` pointing to a
+  text section, or `$(code:)` pointing to an image
+  section).
+- **circular_reference** -- a `$(text:)` or
+  `$(document:)` marker chain loops back to an entity
+  already in the expansion path.
+
+When `ok` is `true`, the `errors` array is empty.
+
+**Garbage collection:**
+
+```json
+// Request (no parameters)
+{}
+
+// Response
+{
+  "removed_sections": 3,
+  "removed_bytes": 15360,
+  "elapsed_ms": 25
+}
+```
+
+Garbage collection removes section files not listed
+in any document manifest. A section is eligible for
+removal when its `docids` array is empty -- it exists
+on disk but no document references it. GC does not
+remove documents, indexes, or the catalog. After
+removal, all caches are invalidated to prevent stale
+reads.
+
+**Index rebuild:**
+
+```json
+// Request (no parameters)
+{}
+
+// Response
+{
+  "sections_reindexed": 39,
+  "terms_indexed": 1500,
+  "elapsed_ms": 500
+}
+```
+
+Index rebuild reconstructs all index files
+(`indexes/` directory) from section files and
+document manifests on disk. Text and code sections
+are re-tokenized per field through the
+[Query Parser](query-parser.md) pipeline and their
+per-field term frequencies re-indexed. Image section
+binary content is skipped, but their `title` and
+`description` attributes are re-tokenized and
+re-indexed. All token counts and per-field corpus
+averages are recomputed. Content markers in all
+text sections are re-parsed to rebuild
+`references.idx` (forward and back-reference
+entries). Equivalent to deleting `indexes/` and
+re-adding every section. After rebuild, all caches
+are invalidated.
+
+### Error Responses
+
+Errors are returned as JSON-RPC 2.0 error objects:
+
+```json
+{
+  "error": "Description of the error",
+  "code": "ERROR_CODE"
+}
+```
+
+The worker client throws these as typed errors. The
+caller (CLI or MCP) handles them -- e.g. the CLI maps
+them to stderr output.
+
+## Tech and Algorithms Behind
+
+This section lists the specific algorithms and data
+structures used, organized by module.
+
+### fs-database.wasm -- Storage
+
+| Approach                            | Role                                    |
+|-------------------------------------|-----------------------------------------|
+| Content-Addressable Store (SHA-256, typed header) | Section files named by hash of `section <mime-type> <byte-size>\0<content>` -- type-aware deduplication, integrity |
+| Content Preprocessing Pipeline      | BOM removal, line ending normalization, Unicode NFC, whitespace trimming -- applied before hashing for text and code sections |
+| Hash-Sharded Directories            | 2-char prefix subdirectories for O(1) section lookup |
+| B+ Tree Indexes                     | Ordered key lookups in binary index files for fast term search |
+| Inverted Index                      | Maps terms to kb-ids and positions for search |
+| Atomic Writes (temp + rename)       | Crash-safe file writes                  |
+| TOML Manifests                      | Human-readable metadata for catalog, documents, and index metadata |
+
+### query-parser.wasm -- Processing
+
+See [Query Parser](query-parser.md) for full details.
+
+| Algorithm / Technique               | Role                                    |
+|-------------------------------------|-----------------------------------------|
+| RAKE                                | Keyphrase extraction                    |
+| TF-IDF                              | Term weighting                          |
+| Snowball Stemmer (Porter2)          | Word root form reduction                |
+| Unicode Segmentation                | Word boundary detection                 |
+| N-gram Extraction                   | Bigrams/trigrams for phrase matching     |
+| BM25F                               | Primary ranking function                |
+| Proximity / Zone Scoring            | Supplementary ranking signals           |
+
+### Section Identification
+
+| Approach                    | Details                                  |
+|-----------------------------|------------------------------------------|
+| Git-Style Typed Header      | Hash input is `section <mime-type> <byte-size>\0<preprocessed-content>` -- MIME type is part of the hash, so same content as different types produces different kb-ids |
+| SHA-256 + Truncation + Base32 | SHA-256 of the typed header, truncated to 128 bits (16 bytes), base32 lowercase (no padding) -- 26 alphanumeric characters |
+| Content Preprocessing       | Text and code: BOM removal, `\r\n`/`\r` to `\n`, Unicode NFC, whitespace trimming. Images: base64 string used as-is |
+
+## Design Constraints
+
+- `fs-database.wasm` is the sole owner of `.kbdb` I/O.
+  The worker client module delegates all storage
+  operations to the worker daemon over IPC, and the
+  daemon delegates to `fs-database.wasm` via
+  `kb-worker.wasm`. `src/cli.ts` does not read or
+  write database files directly.
+- Sections are content-addressable and identified by
+  their content hash. `fs-database.wasm` never modifies
+  a section file in place -- updates produce new files
+  with new kb-ids.
+- Indexes are derived data. They can be rebuilt entirely
+  from sections and document manifests. Loss of index
+  files is recoverable.
+- The `.gitignore` at the database root excludes binary
+  indexes from version control by default. Sections,
+  documents, and the catalog are the version-controlled
+  source of truth.
+- The database should remain human-inspectable and
+  version-controllable where practical. Metadata files
+  use TOML; section files use TOML headers with raw
+  content bodies.
+- No external database process or server dependency.
+- The database must support concurrent read access from
+  multiple CLI invocations via the shared daemon without
+  corruption.
+- Garbage collection of unreferenced sections is
+  `fs-database.wasm`'s responsibility. Sections not
+  listed in any document manifest may be removed.
+- The cache daemon is a short-lived background process,
+  not a persistent server. See
+  [Worker Daemon](worker-daemon.md) for daemon lifecycle
+  and configuration.
+- All IPC API request and response data uses JSON
+  serialization (JSON-RPC 2.0). Inter-module
+  communication (`kb-worker.wasm` to both
+  `fs-database.wasm` and `query-parser.wasm`) uses shared
+  `WebAssembly.Memory` buffers.