@dikolab/kbdb 0.4.3 → 0.6.0

package/docs/details/storage.md

@@ -1,531 +0,0 @@
-# Storage Architecture
-
-kbdb stores everything in a `.kbdb` folder inside the
-target directory. There is no external database, no
-cloud dependency, and no running server required for
-data persistence. All metadata files use TOML for
-human inspectability; indexes are binary but derived
-and rebuildable from source data. The entire
-directory is portable -- it can be copied, moved, or
-version-controlled.
-
-## Directory Layout
-
-```
-.kbdb/
-+-- catalog.toml
-+-- sections/
-|   +-- {ab}/
-|   |   +-- {cdef...}.sec
-|   +-- ...
-+-- documents/
-|   +-- {docid}.toml
-+-- indexes/
-|   +-- meta.toml
-|   +-- terms.idx
-|   +-- positions.idx
-|   +-- references.idx
-|   +-- corpus.dat
-+-- worker.toml
-+-- .gitignore
-```
-
-| Path                          | Format | Description                   |
-|-------------------------------|--------|-------------------------------|
-| `catalog.toml`                | TOML   | Database table of contents: section count, document count, timestamps, integrity checksums |
-| `sections/{ab}/{cdef...}.sec` | TOML + raw | Section files: TOML header + content body, sharded by first two characters of kbid |
-| `documents/{docid}.toml`      | TOML   | Document manifests: title + ordered section list |
-| `indexes/meta.toml`           | TOML   | Index metadata: rebuild timestamp, term count, statistics summary |
-| `indexes/terms.idx`           | Binary | Inverted index (B+ tree): maps terms to posting lists with per-field term frequencies |
-| `indexes/positions.idx`       | Binary | Positional index (B+ tree): maps (term, kbid) to token offset arrays |
-| `indexes/references.idx`     | Binary | Reference graph index (B+ tree): bidirectional content marker cross-references between sections |
-| `indexes/corpus.dat`          | Binary | Corpus statistics: section count, average lengths per field, per-term document frequencies |
-| `worker.toml`                 | TOML   | Cache daemon configuration    |
-| `.gitignore`                  | Text   | Excludes `indexes/` from version control |
-
-The layout mirrors a PostgreSQL directory-format
-dump: a human-readable catalog at the root, one
-subdirectory per data table, and a separate directory
-for binary indexes.
-
-## Content-Addressed IDs (kbid)
-
-Every section is identified by a deterministic hash
-of its content. The same content always produces the
-same ID, enabling automatic deduplication.
-
-### Hashing Process
-
-```
-  Input content
-       |
-       v
-  Preprocess (text/code only)
-       |
-       v
-  Construct typed header:
-    "section <mime-type> <byte-size>\0<content>"
-       |
-       v
-  SHA-256 hash
-       |
-       v
-  Truncate to first 16 bytes (128 bits)
-       |
-       v
-  Base32 lowercase encoding (no padding)
-       |
-       v
-  26-character alphanumeric kbid
-```
-
-### Hash Input Format
-
-The hash input follows the same pattern as Git
-objects (`blob <size>\0<content>`):
-
-```
-section <mime-type> <byte-size>\0<preprocessed-content>
-```
-
-- `section` -- literal string prefix (type tag)
-- `<mime-type>` -- the section's MIME type string
-  (e.g. `text/markdown`, `text/typescript`,
-  `image/png`)
-- `<byte-size>` -- byte length of the preprocessed
-  content, as a decimal ASCII string
-- `\0` -- literal null byte separator
-- `<preprocessed-content>` -- the content after
-  preprocessing (text/code) or the base64 string
-  as-is (images)
-
-### Worked Example
-
-```
-Given:
-   type    = "text/markdown"
-   content = "Hello, world!"  (13 bytes after preprocessing)
-
-Hash input (41 bytes):
-   section text/markdown 13\0Hello, world!
-
-   +-- header: "section text/markdown 13"
-   +-- null byte: \0
-   +-- content: "Hello, world!"
-
-kbid = base32(sha256(hash_input)[..16])
-     = 26-character lowercase alphanumeric string
-```
-
-### Properties
-
-- **Deterministic** -- identical content with the
-  same MIME type always produces the same kbid.
-- **Type-aware** -- the MIME type is part of the hash
-  input, so identical raw content stored as different
-  types (e.g. `text/markdown` vs `text/typescript`)
-  produces different kbids.
-- **Collision-resistant** -- SHA-256 truncated to 128
-  bits provides sufficient collision resistance for a
-  local knowledge base.
-- **URL-safe** -- base32 lowercase encoding uses only
-  alphanumeric characters.
-
-## Content Preprocessing
-
-Before computing the kbid, text and code sections
-pass through a preprocessing pipeline that
-normalizes encoding artifacts. This ensures that
-semantically identical content -- differing only in
-BOM, line endings, Unicode representation, or
-surrounding whitespace -- produces the same kbid.
-
-### Pipeline Steps (Applied in Order)
-
-1. **BOM removal** -- Strip the UTF-8 byte order mark
-   (U+FEFF) if present at the start of the content.
-   The BOM is an encoding artifact, not meaningful
-   content.
-
-2. **Line ending normalization** -- Convert all
-   `\r\n` (Windows) and bare `\r` (old Mac) sequences
-   to `\n` (Unix). Content created on different
-   platforms produces the same kbid.
-
-3. **Unicode NFC normalization** -- Compose decomposed
-   Unicode sequences into their canonical form. For
-   example, `e` + combining acute accent becomes `e`
-   (U+00E9). This is the same normalization used in
-   the tokenization pipeline.
-
-4. **Whitespace trimming** -- Remove all leading and
-   trailing whitespace characters (spaces, tabs,
-   newlines, carriage returns, and other Unicode
-   whitespace and control characters).
-
-### Scope
-
-This pipeline applies to **text and code sections
-only**. Image sections use the base64-encoded string
-as-is -- no preprocessing is applied.
-
-The preprocessed content is what gets stored on disk.
-The pipeline runs once at ingest time; subsequent
-reads return the already-preprocessed content. The
-`<byte-size>` in the hash header is computed from the
-content after preprocessing.
-
-## Section File Format
-
-Each section file has a TOML header followed by a
-`---` separator and the raw content body. The header
-is human-readable; the content after the separator is
-stored verbatim.
-
-### Header Fields
-
-| Field                 | Type   | Required | Description           |
-|-----------------------|--------|----------|-----------------------|
-| `kbid`                | string | Yes      | 26-character content hash |
-| `docids`              | array  | Yes      | Document IDs referencing this section |
-| `type`                | string | Yes      | MIME type (`text/markdown`, `text/typescript`, `image/png`, etc.) |
-| `title`               | string | Images: yes, others: no | Short descriptive title. Indexed with heading weight (2.0x) |
-| `description`         | string | No       | Longer description. Indexed with body weight (1.0x) |
-| `size`                | u32    | Yes      | Content byte size     |
-| `token_count`         | u32    | Yes      | Total tokens after tokenization |
-| `heading_token_count` | u32    | Yes      | Tokens from `title`   |
-| `body_token_count`    | u32    | Yes      | Tokens from body text + `description` |
-| `code_token_count`    | u32    | Yes      | Tokens from code content |
-| `created_at`          | string | Yes      | ISO 8601 timestamp    |
-| `checksum`            | string | Yes      | Integrity checksum    |
-
-The `title` and `description` fields are not included
-in the kbid hash -- changing them does not change the
-section's identity.
-
-Token count fields are pre-computed at insertion time
-by `query-parser.wasm` and stored for reuse at query
-time. This trades a small write-time cost for
-significant search speed.
-
-### Example: Text Section
-
-File: `sections/ab/cdef1234...sec`
-
-```
-kbid = "abcdef1234567890abcdef1234"
-docids = ["doc-5678"]
-type = "text/markdown"
-title = "Architecture Overview"
-description = "High-level overview of the system"
-size = 1234
-token_count = 187
-heading_token_count = 2
-body_token_count = 187
-code_token_count = 0
-created_at = "2026-06-15T10:00:00Z"
-checksum = "sha256:abcdef..."
----
-Section body text here.
-
-$(image: 7fa93b2e4c8d1076a93b2e4c8d)
-
-$(code: c4e8a91d3f7b20564e8a91d3f7)
-
-This content continues after the embedded sections.
-No escaping or encoding is applied.
-```
-
-### Example: Code Section
-
-File: `sections/c4/e8a91d...sec`
-
-```
-kbid = "c4e8a91d3f7b20564e8a91d3f7"
-docids = ["doc-5678"]
-type = "text/typescript"
-title = "greet function"
-size = 256
-token_count = 22
-heading_token_count = 2
-body_token_count = 0
-code_token_count = 22
-created_at = "2026-06-15T10:15:00Z"
-checksum = "sha256:1a2b3c..."
----
-export function greet(name: string): string {
-   return `Hello, ${name}!`;
-}
-```
-
-### Example: Image Section
-
-File: `sections/7f/a93b2e...sec`
-
-```
-kbid = "7fa93b2e4c8d1076a93b2e4c8d"
-docids = ["doc-5678"]
-type = "image/png"
-title = "Architecture diagram"
-description = "Five-layer stack diagram"
-size = 48576
-token_count = 5
-heading_token_count = 2
-body_token_count = 3
-code_token_count = 0
-created_at = "2026-06-15T10:30:00Z"
-checksum = "sha256:fedcba..."
----
-iVBORw0KGgoAAAANSUhEUgAA...base64 data...
-```
-
-The image section has `token_count = 5` from title
-and description tokens only. Binary content produces
-zero tokens.
-
-### Section Types
-
-| Type                | Content after `---`    | Indexed? |
-|---------------------|------------------------|----------|
-| `text/markdown`     | Raw markdown text      | Yes (body field) |
-| `text/*` (code)     | Raw source code        | Yes (code field) |
-| `application/json`  | Raw JSON               | Yes (code field) |
-| `image/*`           | Base64-encoded binary  | No (metadata only) |
-
-Code sections are processed through the same
-tokenization pipeline as text, including identifier
-splitting for camelCase and snake_case terms. Image
-binary content is skipped, but `title` (required) and
-`description` (optional) are tokenized and indexed.
-
-## Document Manifests
-
-A document groups sections under a title with a
-defined order. Documents do not contain content
-directly -- they reference sections by kbid. The
-relationship is many-to-many: a section may appear in
-multiple documents, and a document contains multiple
-sections.
-
-### Format
-
-```toml
-[document]
-docid = "doc-5678"
-title = "Getting Started Guide"
-created_at = "2026-06-15T10:00:00Z"
-
-[[sections]]
-kbid = "abcdef1234567890abcdef1234"
-position = 0
-
-[[sections]]
-kbid = "fedcba4321098765fedcba4321"
-position = 1
-```
-
-### Fields
-
-| Field                    | Description              |
-|--------------------------|--------------------------|
-| `document.docid`         | Document identifier      |
-| `document.title`         | Human-readable title     |
-| `document.created_at`    | ISO 8601 timestamp       |
-| `sections[].kbid`        | Section content hash     |
-| `sections[].position`    | Zero-based display order |
-
-Documents are human-readable and diffable in git.
-When a document is retrieved, its sections are
-returned in position order with full content.
-
-## Catalog (`catalog.toml`)
-
-The catalog is the database's table of contents --
-the first file read when opening a database. It
-records high-level state without requiring a
-directory scan.
-
-```toml
-[database]
-version = 1
-created_at = "2026-06-15T10:00:00Z"
-last_modified = "2026-06-15T12:00:00Z"
-
-[counts]
-sections = 42
-documents = 5
-
-[integrity]
-checksum = "sha256:abcdef..."
-```
-
-Updated on every write operation. If a crash occurs
-mid-write, the catalog remains consistent with the
-last complete state (see Write Safety below).
-
-## MIME Types
-
-kbdb uses standard MIME type strings to identify
-section content:
-
-| Category  | MIME types                          | Canonical       |
-|-----------|-------------------------------------|-----------------|
-| Text      | `text/markdown`                     | `text/markdown`  |
-| Code      | `text/typescript`, `text/javascript`, `text/html`, `text/css`, `text/xml`, `text/plain`, `application/json` | Varies |
-| Image     | `image/png`, `image/jpeg`, `image/webp` | Varies       |
-
-The bare string `"text"` is not used. Text sections
-use `text/markdown` as their canonical type. The MIME
-type is part of the kbid hash, so the same raw
-content stored under different types produces
-different section files.
-
-## Write Safety
-
-All write operations are designed to be crash-safe:
-
-- **Atomic writes** -- New files are written to a
-  temporary path and then renamed into place. A crash
-  during the write leaves no partial files -- either
-  the old file remains or the new file is complete.
-
-- **Catalog updated last** -- The catalog is written
-  after all data files. A crash mid-write leaves the
-  catalog consistent with the last fully committed
-  state. The worst case is an unreferenced section
-  file on disk, which garbage collection handles.
-
-- **Indexes rebuildable** -- Index files are derived
-  data. If index files are corrupted or lost, they
-  can be rebuilt entirely from sections and document
-  manifests. The `.gitignore` excludes indexes from
-  version control because they are not source data.
-
-### Write Sequence
-
-```
-  1. Write section file to temp path
-       |
-       v
-  2. Rename temp file into sections/{ab}/{kbid}.sec
-       |
-       v
-  3. Update document manifest (if docid provided)
-       |
-       v
-  4. Update indexes (inverted + positional + corpus)
-       |
-       v
-  5. Update catalog.toml (counts, timestamp, checksum)
-```
-
-Each step is atomic. The catalog update in step 5
-commits the entire operation. Steps 1-4 can fail
-without corrupting the database.
-
-## Content Markers
-
-Text sections can contain markers -- escape sequences
-that embed or link to other sections and documents.
-Markers are stored verbatim in the content body and
-resolved at read time, not at write time.
-
-### Marker Types
-
-| Marker                      | Category  | Target type              | Render behavior        |
-|-----------------------------|-----------|--------------------------|------------------------|
-| `$(image: {kbid})`          | Embedding | `image/*` section        | Inline image           |
-| `$(code: {kbid})`           | Embedding | `text/*` or `application/json` section | Fenced code block |
-| `$(text: {kbid})`           | Embedding | `text/markdown` section  | Inline markdown (recursive) |
-| `$(document: {docid})`      | Embedding | Document manifest        | All sections expanded (recursive) |
-| `$(link: {kbid})`           | Link      | Any section              | Cross-reference        |
-| `$(doc-link: {docid})`      | Link      | Document manifest        | Cross-reference        |
-
-### Syntax
-
-Markers use the format `$(type: value)`:
-
-```
-$(image: 7fa93b2e4c8d1076a93b2e4c8d)
-$(code: c4e8a91d3f7b20564e8a91d3f7)
-$(text: a1b2c3d4e5f6a1b2c3d4e5f6a1)
-$(document: api-reference-v2)
-$(link: d5e6f7a8b9c0d5e6f7a8b9c0d5)
-$(doc-link: getting-started-guide)
-```
-
-Rules:
-- A colon and a space separate type from value.
-- Markers cannot be nested.
-- Unrecognized types pass through unchanged.
-- Unclosed markers are treated as plain text.
-
-### Recursive Resolution
-
-The `$(text:)` and `$(document:)` markers trigger
-recursive expansion -- the embedded content may
-itself contain markers that must be resolved.
-
-Circular references are detected using a visited set:
-
-```
-  Expand marker $(text: A)
-       |
-       v
-  Add A to visited set: {A}
-       |
-       v
-  Section A contains $(text: B)
-       |
-       v
-  Add B to visited set: {A, B}
-       |
-       v
-  Section B contains $(text: A)
-       |
-       v
-  A is already in visited set -> STOP (circular)
-```
-
-Circular references are reported as errors during
-both integrity checks and render-time expansion.
-
-### Tokenization
-
-All six marker types are stripped from content before
-tokenization and indexing (stage 1 of the
-tokenization pipeline). Markers are structural
-references, not searchable text -- they do not appear
-in the search index.
-
-## Deduplication
-
-Sections are content-addressable. When identical
-content (after preprocessing) is added with the same
-MIME type:
-
-| Scenario              | Result                      |
-|-----------------------|-----------------------------|
-| Different `docid`     | Existing section's `docids` array updated to include the new docid. No new file created. |
-| No `docid`            | Section skipped entirely.   |
-| Same `docid` already present | Idempotent -- no change. |
-
-A section can belong to zero or more documents.
-Updating a section's content creates a new file with
-a new kbid; the old file becomes eligible for garbage
-collection.
-
-## Storage Performance
-
-The directory layout is optimized for read speed:
-
-| Operation         | Complexity | Mechanism               |
-|-------------------|------------|-------------------------|
-| Section lookup    | O(1)       | Hash-sharded directories allow direct file access by kbid |
-| Term lookup       | O(log n)   | B+ tree indexes         |
-| Catalog discovery | O(1)       | `catalog.toml` provides counts without scanning |
-| Document listing  | O(1)       | Direct file access by docid |
-
-Write operations prioritize safety over speed. The
-atomic write + catalog-last pattern ensures no
-partial state is visible after a crash.