@dikolab/kbdb 0.1.6 → 0.2.1

package/docs/details/storage.md

@@ -0,0 +1,529 @@
+# Storage Architecture
+
+kbdb stores everything in a `.kbdb` folder inside the
+target directory. There is no external database, no
+cloud dependency, and no running server required for
+data persistence. All metadata files use TOML for
+human inspectability; indexes are binary but derived
+and rebuildable from source data. The entire
+directory is portable -- it can be copied, moved, or
+version-controlled.
+
+## Directory Layout
+
+```
+.kbdb/
++-- catalog.toml
++-- sections/
+|   +-- {ab}/
+|   |   +-- {cdef...}.sec
+|   +-- ...
++-- documents/
+|   +-- {docid}.toml
++-- indexes/
+|   +-- meta.toml
+|   +-- terms.idx
+|   +-- positions.idx
+|   +-- corpus.dat
++-- worker.toml
++-- .gitignore
+```
+
+| Path                          | Format | Description                   |
+|-------------------------------|--------|-------------------------------|
+| `catalog.toml`                | TOML   | Database table of contents: section count, document count, timestamps, integrity checksums |
+| `sections/{ab}/{cdef...}.sec` | TOML + raw | Section files: TOML header + content body, sharded by first two characters of kbid |
+| `documents/{docid}.toml`      | TOML   | Document manifests: title + ordered section list |
+| `indexes/meta.toml`           | TOML   | Index metadata: rebuild timestamp, term count, statistics summary |
+| `indexes/terms.idx`           | Binary | Inverted index (B+ tree): maps terms to posting lists with per-field term frequencies |
+| `indexes/positions.idx`       | Binary | Positional index (B+ tree): maps (term, kbid) to token offset arrays |
+| `indexes/corpus.dat`          | Binary | Corpus statistics: section count, average lengths per field, per-term document frequencies |
+| `worker.toml`                 | TOML   | Cache daemon configuration    |
+| `.gitignore`                  | Text   | Excludes `indexes/` from version control |
+
+The layout mirrors a PostgreSQL directory-format
+dump: a human-readable catalog at the root, one
+subdirectory per data table, and a separate directory
+for binary indexes.
+
+## Content-Addressed IDs (kbid)
+
+Every section is identified by a deterministic hash
+of its content. The same content always produces the
+same ID, enabling automatic deduplication.
+
+### Hashing Process
+
+```
+  Input content
+       |
+       v
+  Preprocess (text/code only)
+       |
+       v
+  Construct typed header:
+    "section <mime-type> <byte-size>\0<content>"
+       |
+       v
+  SHA-256 hash
+       |
+       v
+  Truncate to first 16 bytes (128 bits)
+       |
+       v
+  Base32 lowercase encoding (no padding)
+       |
+       v
+  26-character alphanumeric kbid
+```
+
+### Hash Input Format
+
+The hash input follows the same pattern as Git
+objects (`blob <size>\0<content>`):
+
+```
+section <mime-type> <byte-size>\0<preprocessed-content>
+```
+
+- `section` -- literal string prefix (type tag)
+- `<mime-type>` -- the section's MIME type string
+  (e.g. `text/markdown`, `text/typescript`,
+  `image/png`)
+- `<byte-size>` -- byte length of the preprocessed
+  content, as a decimal ASCII string
+- `\0` -- literal null byte separator
+- `<preprocessed-content>` -- the content after
+  preprocessing (text/code) or the base64 string
+  as-is (images)
+
+### Worked Example
+
+```
+Given:
+   type    = "text/markdown"
+   content = "Hello, world!"  (13 bytes after preprocessing)
+
+Hash input (41 bytes):
+   section text/markdown 13\0Hello, world!
+
+   +-- header: "section text/markdown 13"
+   +-- null byte: \0
+   +-- content: "Hello, world!"
+
+kbid = base32(sha256(hash_input)[..16])
+     = 26-character lowercase alphanumeric string
+```
+
+### Properties
+
+- **Deterministic** -- identical content with the
+  same MIME type always produces the same kbid.
+- **Type-aware** -- the MIME type is part of the hash
+  input, so identical raw content stored as different
+  types (e.g. `text/markdown` vs `text/typescript`)
+  produces different kbids.
+- **Collision-resistant** -- SHA-256 truncated to 128
+  bits provides sufficient collision resistance for a
+  local knowledge base.
+- **URL-safe** -- base32 lowercase encoding uses only
+  alphanumeric characters.
+
+## Content Preprocessing
+
+Before computing the kbid, text and code sections
+pass through a preprocessing pipeline that
+normalizes encoding artifacts. This ensures that
+semantically identical content -- differing only in
+BOM, line endings, Unicode representation, or
+surrounding whitespace -- produces the same kbid.
+
+### Pipeline Steps (Applied in Order)
+
+1. **BOM removal** -- Strip the UTF-8 byte order mark
+   (U+FEFF) if present at the start of the content.
+   The BOM is an encoding artifact, not meaningful
+   content.
+
+2. **Line ending normalization** -- Convert all
+   `\r\n` (Windows) and bare `\r` (old Mac) sequences
+   to `\n` (Unix). Content created on different
+   platforms produces the same kbid.
+
+3. **Unicode NFC normalization** -- Compose decomposed
+   Unicode sequences into their canonical form. For
+   example, `e` + combining acute accent becomes `e`
+   (U+00E9). This is the same normalization used in
+   the tokenization pipeline.
+
+4. **Whitespace trimming** -- Remove all leading and
+   trailing whitespace characters (spaces, tabs,
+   newlines, carriage returns, and other Unicode
+   whitespace and control characters).
+
+### Scope
+
+This pipeline applies to **text and code sections
+only**. Image sections use the base64-encoded string
+as-is -- no preprocessing is applied.
+
+The preprocessed content is what gets stored on disk.
+The pipeline runs once at ingest time; subsequent
+reads return the already-preprocessed content. The
+`<byte-size>` in the hash header is computed from the
+content after preprocessing.
+
+## Section File Format
+
+Each section file has a TOML header followed by a
+`---` separator and the raw content body. The header
+is human-readable; the content after the separator is
+stored verbatim.
+
+### Header Fields
+
+| Field                 | Type   | Required | Description           |
+|-----------------------|--------|----------|-----------------------|
+| `kbid`                | string | Yes      | 26-character content hash |
+| `docids`              | array  | Yes      | Document IDs referencing this section |
+| `type`                | string | Yes      | MIME type (`text/markdown`, `text/typescript`, `image/png`, etc.) |
+| `title`               | string | Images: yes, others: no | Short descriptive title. Indexed with heading weight (2.0x) |
+| `description`         | string | No       | Longer description. Indexed with body weight (1.0x) |
+| `size`                | u32    | Yes      | Content byte size     |
+| `token_count`         | u32    | Yes      | Total tokens after tokenization |
+| `heading_token_count` | u32    | Yes      | Tokens from `title`   |
+| `body_token_count`    | u32    | Yes      | Tokens from body text + `description` |
+| `code_token_count`    | u32    | Yes      | Tokens from code content |
+| `created_at`          | string | Yes      | ISO 8601 timestamp    |
+| `checksum`            | string | Yes      | Integrity checksum    |
+
+The `title` and `description` fields are not included
+in the kbid hash -- changing them does not change the
+section's identity.
+
+Token count fields are pre-computed at insertion time
+by `query-parser.wasm` and stored for reuse at query
+time. This trades a small write-time cost for
+significant search speed.
+
+### Example: Text Section
+
+File: `sections/ab/cdef1234...sec`
+
+```
+kbid = "abcdef1234567890abcdef1234"
+docids = ["doc-5678"]
+type = "text/markdown"
+title = "Architecture Overview"
+description = "High-level overview of the system"
+size = 1234
+token_count = 187
+heading_token_count = 2
+body_token_count = 187
+code_token_count = 0
+created_at = "2026-06-15T10:00:00Z"
+checksum = "sha256:abcdef..."
+---
+Section body text here.
+
+$(image: 7fa93b2e4c8d1076a93b2e4c8d)
+
+$(code: c4e8a91d3f7b20564e8a91d3f7)
+
+This content continues after the embedded sections.
+No escaping or encoding is applied.
+```
+
+### Example: Code Section
+
+File: `sections/c4/e8a91d...sec`
+
+```
+kbid = "c4e8a91d3f7b20564e8a91d3f7"
+docids = ["doc-5678"]
+type = "text/typescript"
+title = "greet function"
+size = 256
+token_count = 22
+heading_token_count = 2
+body_token_count = 0
+code_token_count = 22
+created_at = "2026-06-15T10:15:00Z"
+checksum = "sha256:1a2b3c..."
+---
+export function greet(name: string): string {
+   return `Hello, ${name}!`;
+}
+```
+
+### Example: Image Section
+
+File: `sections/7f/a93b2e...sec`
+
+```
+kbid = "7fa93b2e4c8d1076a93b2e4c8d"
+docids = ["doc-5678"]
+type = "image/png"
+title = "Architecture diagram"
+description = "Five-layer stack diagram"
+size = 48576
+token_count = 5
+heading_token_count = 2
+body_token_count = 3
+code_token_count = 0
+created_at = "2026-06-15T10:30:00Z"
+checksum = "sha256:fedcba..."
+---
+iVBORw0KGgoAAAANSUhEUgAA...base64 data...
+```
+
+The image section has `token_count = 5` from title
+and description tokens only. Binary content produces
+zero tokens.
+
+### Section Types
+
+| Type                | Content after `---`    | Indexed? |
+|---------------------|------------------------|----------|
+| `text/markdown`     | Raw markdown text      | Yes (body field) |
+| `text/*` (code)     | Raw source code        | Yes (code field) |
+| `application/json`  | Raw JSON               | Yes (code field) |
+| `image/*`           | Base64-encoded binary  | No (metadata only) |
+
+Code sections are processed through the same
+tokenization pipeline as text, including identifier
+splitting for camelCase and snake_case terms. Image
+binary content is skipped, but `title` (required) and
+`description` (optional) are tokenized and indexed.
+
+## Document Manifests
+
+A document groups sections under a title with a
+defined order. Documents do not contain content
+directly -- they reference sections by kbid. The
+relationship is many-to-many: a section may appear in
+multiple documents, and a document contains multiple
+sections.
+
+### Format
+
+```toml
+[document]
+docid = "doc-5678"
+title = "Getting Started Guide"
+created_at = "2026-06-15T10:00:00Z"
+
+[[sections]]
+kbid = "abcdef1234567890abcdef1234"
+position = 0
+
+[[sections]]
+kbid = "fedcba4321098765fedcba4321"
+position = 1
+```
+
+### Fields
+
+| Field                    | Description              |
+|--------------------------|--------------------------|
+| `document.docid`         | Document identifier      |
+| `document.title`         | Human-readable title     |
+| `document.created_at`    | ISO 8601 timestamp       |
+| `sections[].kbid`        | Section content hash     |
+| `sections[].position`    | Zero-based display order |
+
+Documents are human-readable and diffable in git.
+When a document is retrieved, its sections are
+returned in position order with full content.
+
+## Catalog (`catalog.toml`)
+
+The catalog is the database's table of contents --
+the first file read when opening a database. It
+records high-level state without requiring a
+directory scan.
+
+```toml
+[database]
+version = 1
+created_at = "2026-06-15T10:00:00Z"
+last_modified = "2026-06-15T12:00:00Z"
+
+[counts]
+sections = 42
+documents = 5
+
+[integrity]
+checksum = "sha256:abcdef..."
+```
+
+Updated on every write operation. If a crash occurs
+mid-write, the catalog remains consistent with the
+last complete state (see Write Safety below).
+
+## MIME Types
+
+kbdb uses standard MIME type strings to identify
+section content:
+
+| Category  | MIME types                          | Canonical       |
+|-----------|-------------------------------------|-----------------|
+| Text      | `text/markdown`                     | `text/markdown`  |
+| Code      | `text/typescript`, `text/javascript`, `text/html`, `text/css`, `text/xml`, `text/plain`, `application/json` | Varies |
+| Image     | `image/png`, `image/jpeg`, `image/webp` | Varies       |
+
+The bare string `"text"` is not used. Text sections
+use `text/markdown` as their canonical type. The MIME
+type is part of the kbid hash, so the same raw
+content stored under different types produces
+different section files.
+
+## Write Safety
+
+All write operations are designed to be crash-safe:
+
+- **Atomic writes** -- New files are written to a
+  temporary path and then renamed into place. A crash
+  during the write leaves no partial files -- either
+  the old file remains or the new file is complete.
+
+- **Catalog updated last** -- The catalog is written
+  after all data files. A crash mid-write leaves the
+  catalog consistent with the last fully committed
+  state. The worst case is an unreferenced section
+  file on disk, which garbage collection handles.
+
+- **Indexes rebuildable** -- Index files are derived
+  data. If index files are corrupted or lost, they
+  can be rebuilt entirely from sections and document
+  manifests. The `.gitignore` excludes indexes from
+  version control because they are not source data.
+
+### Write Sequence
+
+```
+  1. Write section file to temp path
+       |
+       v
+  2. Rename temp file into sections/{ab}/{kbid}.sec
+       |
+       v
+  3. Update document manifest (if docid provided)
+       |
+       v
+  4. Update indexes (inverted + positional + corpus)
+       |
+       v
+  5. Update catalog.toml (counts, timestamp, checksum)
+```
+
+Each step is atomic. The catalog update in step 5
+commits the entire operation. Steps 1-4 can fail
+without corrupting the database.
+
+## Content Markers
+
+Text sections can contain markers -- escape sequences
+that embed or link to other sections and documents.
+Markers are stored verbatim in the content body and
+resolved at read time, not at write time.
+
+### Marker Types
+
+| Marker                      | Category  | Target type              | Render behavior        |
+|-----------------------------|-----------|--------------------------|------------------------|
+| `$(image: {kbid})`          | Embedding | `image/*` section        | Inline image           |
+| `$(code: {kbid})`           | Embedding | `text/*` or `application/json` section | Fenced code block |
+| `$(text: {kbid})`           | Embedding | `text/markdown` section  | Inline markdown (recursive) |
+| `$(document: {docid})`      | Embedding | Document manifest        | All sections expanded (recursive) |
+| `$(link: {kbid})`           | Link      | Any section              | Cross-reference        |
+| `$(doc-link: {docid})`      | Link      | Document manifest        | Cross-reference        |
+
+### Syntax
+
+Markers use the format `$(type: value)`:
+
+```
+$(image: 7fa93b2e4c8d1076a93b2e4c8d)
+$(code: c4e8a91d3f7b20564e8a91d3f7)
+$(text: a1b2c3d4e5f6a1b2c3d4e5f6a1)
+$(document: api-reference-v2)
+$(link: d5e6f7a8b9c0d5e6f7a8b9c0d5)
+$(doc-link: getting-started-guide)
+```
+
+Rules:
+- A colon and a space separate type from value.
+- Markers cannot be nested.
+- Unrecognized types pass through unchanged.
+- Unclosed markers are treated as plain text.
+
+### Recursive Resolution
+
+The `$(text:)` and `$(document:)` markers trigger
+recursive expansion -- the embedded content may
+itself contain markers that must be resolved.
+
+Circular references are detected using a visited set:
+
+```
+  Expand marker $(text: A)
+       |
+       v
+  Add A to visited set: {A}
+       |
+       v
+  Section A contains $(text: B)
+       |
+       v
+  Add B to visited set: {A, B}
+       |
+       v
+  Section B contains $(text: A)
+       |
+       v
+  A is already in visited set -> STOP (circular)
+```
+
+Circular references are reported as errors during
+both integrity checks and render-time expansion.
+
+### Tokenization
+
+All six marker types are stripped from content before
+tokenization and indexing (stage 1 of the
+tokenization pipeline). Markers are structural
+references, not searchable text -- they do not appear
+in the search index.
+
+## Deduplication
+
+Sections are content-addressable. When identical
+content (after preprocessing) is added with the same
+MIME type:
+
+| Scenario              | Result                      |
+|-----------------------|-----------------------------|
+| Different `docid`     | Existing section's `docids` array updated to include the new docid. No new file created. |
+| No `docid`            | Section skipped entirely.   |
+| Same `docid` already present | Idempotent -- no change. |
+
+A section can belong to zero or more documents.
+Updating a section's content creates a new file with
+a new kbid; the old file becomes eligible for garbage
+collection.
+
+## Storage Performance
+
+The directory layout is optimized for read speed:
+
+| Operation         | Complexity | Mechanism               |
+|-------------------|------------|-------------------------|
+| Section lookup    | O(1)       | Hash-sharded directories allow direct file access by kbid |
+| Term lookup       | O(log n)   | B+ tree indexes         |
+| Catalog discovery | O(1)       | `catalog.toml` provides counts without scanning |
+| Document listing  | O(1)       | Direct file access by docid |
+
+Write operations prioritize safety over speed. The
+atomic write + catalog-last pattern ensures no
+partial state is visible after a crash.