@dikolab/kbdb 0.1.5 → 0.2.0

package/docs/modules/rust/shared/section/types.md

@@ -0,0 +1,294 @@
+# types::section -- Section Types and Records
+
+*Section type enum, MIME type wrapper, and the data
+structures for section CRUD operations across all
+three WASM modules.*
+
+## File
+
+`engine/crates/shared/src/section/section.rs`
+
+## Overview
+
+Sections are the atomic content units in the
+knowledge base. Each section has a **type** that
+determines how it is stored, whether it is indexed,
+and how it is rendered. The types in this module are
+shared across `fs-database` (storage), `query-parser`
+(indexing decisions), and `kb-worker` (orchestration).
+
+See [Database -- Sections](../../../../../goals/database.md)
+for the behavioral specification of section types,
+embedding markers, and MIME-to-language mapping.
+
+## Type Definitions
+
+### `MimeType`
+
+```rust
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct MimeType(String);
+
+impl MimeType {
+   pub fn new(mime: &str) -> Result<Self, MimeError>;
+   pub fn as_str(&self) -> &str;
+   pub fn subtype(&self) -> &str;
+}
+```
+
+Newtype wrapper for MIME type strings. Validates
+that the string contains a `/` separator (e.g.
+`"text/typescript"`, `"image/png"`,
+`"application/json"`).
+
+`subtype()` returns the part after the `/`:
+`"text/typescript"` -> `"typescript"`,
+`"application/json"` -> `"json"`.
+
+### `SectionType`
+
+```rust
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum SectionType {
+   Text,
+   Code(MimeType),
+   Image(MimeType),
+}
+```
+
+Three variants corresponding to the three content
+kinds in the knowledge base:
+
+| Variant       | Content      | Indexed | Embeddable via    |
+|---------------|--------------|---------|-------------------|
+| `Text`        | Raw text     | Yes     | `$(text: {kbid})` |
+| `Code(mime)`  | Raw source   | Yes     | `$(code: {kbid})` |
+| `Image(mime)` | Base64 bytes | title/description only | `$(image: {kbid})`|
+
+### `SectionType` Methods
+
+```rust
+impl SectionType {
+   pub fn is_indexable(&self) -> bool;
+   pub fn as_mime_str(&self) -> &str;
+   pub fn from_mime(mime: &str) -> Self;
+   pub fn language_tag(&self) -> Option<&str>;
+}
+```
+
+#### `is_indexable`
+
+```rust
+pub fn is_indexable(&self) -> bool {
+   matches!(self, Self::Text | Self::Code(_))
+}
+```
+
+Returns `true` for `Text` and `Code`, `false` for
+`Image`. Used by `kb-worker` to decide whether to
+tokenize the section body content. Note:
+`kb-worker` independently indexes image sections'
+`title` (required, heading field 2.0x) and
+`description` (optional, body field 1.0x)
+regardless of `is_indexable()` returning `false`.
+
+#### `as_mime_str`
+
+Returns the MIME string representation:
+
+- `Text` -> `"text/markdown"`
+- `Code(mime)` -> the inner MIME string
+  (e.g. `"text/typescript"`)
+- `Image(mime)` -> the inner MIME string
+  (e.g. `"image/png"`)
+
+#### `from_mime`
+
+Parses a MIME string into the appropriate variant:
+
+| Input                | Result                              |
+|----------------------|-------------------------------------|
+| `"text/markdown"`    | `SectionType::Text`                 |
+| `"text/typescript"`  | `SectionType::Code("text/typescript")` |
+| `"text/javascript"`  | `SectionType::Code("text/javascript")` |
+| `"text/html"`        | `SectionType::Code("text/html")`    |
+| `"text/css"`         | `SectionType::Code("text/css")`     |
+| `"text/xml"`         | `SectionType::Code("text/xml")`     |
+| `"text/plain"`       | `SectionType::Code("text/plain")`   |
+| `"text/*"` (other)   | `SectionType::Code("text/{subtype}")` |
+| `"application/json"` | `SectionType::Code("application/json")` |
+| `"image/png"`        | `SectionType::Image("image/png")`   |
+| `"image/jpeg"`       | `SectionType::Image("image/jpeg")`  |
+| `"image/*"` (other)  | `SectionType::Image("image/{subtype}")` |
+
+`"text/markdown"` is the canonical type for
+natural-language text sections. Any other `text/*`
+subtype maps to `Code`.
+
+#### `language_tag`
+
+Returns the fenced code block language tag for code
+sections. Returns `None` for `Text` and `Image`.
+
+| MIME Type            | Language Tag |
+|----------------------|--------------|
+| `text/typescript`    | `"ts"`       |
+| `text/javascript`    | `"js"`       |
+| `text/html`          | `"html"`     |
+| `text/css`           | `"css"`      |
+| `text/xml`           | `"xml"`      |
+| `text/plain`         | `None`       |
+| `application/json`   | `"json"`     |
+
+Other `text/*` subtypes return the subtype as the
+language tag (e.g. `"text/python"` -> `"python"`,
+`"text/rust"` -> `"rust"`).
+
+## Section Records
+
+Data structures for section CRUD operations. Used in
+IPC request/response payloads and as cache entries.
+
+### `SectionRecord`
+
+```rust
+pub struct SectionRecord {
+   pub kbid: KbId,
+   pub docids: Vec<String>,
+   pub section_type: SectionType,
+   pub title: Option<String>,
+   pub description: Option<String>,
+   pub content: Vec<u8>,
+   pub size: u32,
+   pub created_at: String,
+   pub checksum: String,
+   pub token_count: u32,
+   pub heading_token_count: u32,
+   pub body_token_count: u32,
+   pub code_token_count: u32,
+}
+```
+
+Full section data as read from disk. `docids` lists
+the document IDs this section belongs to (may be
+empty, one, or many -- many-to-many relationship).
+`content` is raw UTF-8 bytes for text and code
+sections, or base64-encoded bytes for image sections.
+`size` is the byte length of the content body.
+`title` is required for image sections and optional
+for text/code. `description` is always optional.
+Token counts are pre-computed during per-field
+tokenization at indexing time.
+
+### `AddSectionParams`
+
+```rust
+pub struct AddSectionParams {
+   pub content: Vec<u8>,
+   pub section_type: SectionType,
+   pub title: Option<String>,
+   pub description: Option<String>,
+   pub docid: Option<String>,
+}
+```
+
+Input for creating a new section. The caller passes
+raw content; `fs-database.wasm` applies the
+preprocessing pipeline internally to text and code
+sections (BOM removal, line ending normalization,
+Unicode NFC, whitespace trimming) before computing
+the kbid. Image content is stored verbatim. The
+kbid is derived from the preprocessed `content` by
+`fs-database` during creation -- it is not provided
+by the caller.
+
+`title` is required when `section_type` is
+`Image(*)` -- validation fails without it.
+`description` is always optional. Both are
+tokenized during indexing: title into the heading
+field (2.0x weight), description into the body
+field (1.0x weight).
+
+### `AddSectionResult`
+
+```rust
+pub struct AddSectionResult {
+   pub kbid: KbId,
+   pub section_type: SectionType,
+   pub title: Option<String>,
+   pub description: Option<String>,
+   pub indexed_terms_count: u32,
+   pub token_count: u32,
+   pub elapsed_ms: u64,
+   pub attached: bool,
+}
+```
+
+`indexed_terms_count` reflects the number of
+distinct terms indexed across all fields. For
+image sections, this is the count from
+title/description tokenization (zero only when
+both produce no terms). `token_count` is the total
+token count across all fields. `attached` is `true`
+when the content already existed and a new docid
+was added to the section's `docids` list instead of
+creating a new section.
+
+### `UpdateSectionResult`
+
+```rust
+pub struct UpdateSectionResult {
+   pub old_kbid: KbId,
+   pub new_kbid: KbId,
+   pub reindexed_terms_count: u32,
+   pub elapsed_ms: u64,
+}
+```
+
+Updates produce a new section file with a new kb-id
+(content-addressable). The old kb-id is returned so
+the caller and cache can invalidate stale references.
+
+### `RemoveSectionResult`
+
+```rust
+pub struct RemoveSectionResult {
+   pub removed: bool,
+   pub elapsed_ms: u64,
+}
+```
+
+`removed` is `false` if the kb-id was not found.
+
+## Trait Implementations
+
+All structs implement `Clone`, `Debug`, and `Encode`
+(see [encode](../encode/traits.md)).
+`SectionType` also implements `PartialEq` and `Eq`
+for type comparison.
+
+## Error Type
+
+```rust
+#[derive(Debug)]
+pub enum MimeError {
+   MissingSeparator(String),
+   EmptyType(String),
+   EmptySubtype(String),
+}
+```
+
+## Testing
+
+Unit tests:
+
+- `SectionType::from_mime` mapping for all documented
+  MIME types.
+- `is_indexable` returns `true` for `Text` and `Code`,
+  `false` for `Image`.
+- `language_tag` returns correct tags for known MIME
+  types and subtype-as-fallback for unknown ones.
+- `from_mime` with `"text/markdown"` produces `Text`,
+  not `Code`.
+- `MimeType::subtype` extracts correctly.
+- `Encode` round-trip for all record types.

package/docs/modules/rust/shared/term/module.md

@@ -0,0 +1,25 @@
+# term -- Normalized Search Term
+
+*Newtype wrapper for normalized, stemmed search terms
+used as keys in inverted indexes and query plans.*
+
+## Source
+
+`engine/crates/shared/src/term/`
+
+## Responsibility
+
+Defines the `Term` newtype with ordering support for
+B+ tree keys. Terms are the atomic unit of the
+inverted index -- every index lookup, ranking
+computation, and query plan operates on `Term`
+values.
+
+See [Query Parser -- Tokenization Pipeline](
+../../../../goals/query-parser.md) for the pipeline
+that produces terms.
+
+## Exports
+
+- **[types.md](types.md)** -- `Term` newtype,
+  `from_stemmed`, `TermError`.

package/docs/modules/rust/shared/term/types.md

@@ -0,0 +1,139 @@
+# types::term -- Normalized Search Term
+
+*Newtype wrapper around a normalized, stemmed term
+used as the key in inverted indexes and query plans.*
+
+## File
+
+`engine/crates/shared/src/term/term.rs`
+
+## Overview
+
+A `Term` represents a single searchable token after
+the full tokenization pipeline has run: Unicode
+normalization, case folding, stemming, and stop word
+removal. Terms are the atomic unit of the inverted
+index -- every index lookup, every ranking
+computation, and every query plan operates on `Term`
+values.
+
+See [Query Parser -- Tokenization Pipeline](
+../../../../../goals/query-parser.md) for the pipeline
+that produces terms.
+
+## Type Definition
+
+```rust
+#[derive(Clone, Debug, PartialEq, Eq,
+         Hash, Ord, PartialOrd)]
+pub struct Term(String);
+```
+
+The inner `String` is private. Construction goes
+through `Term::new`, which validates the invariants.
+
+## Invariants
+
+A valid `Term`:
+
+- Is non-empty.
+- Contains only lowercase characters (post-case-fold).
+- Contains no whitespace.
+- Is not a stop word (stop words are filtered before
+  `Term` construction).
+- May be a stemmed form (e.g. `"comput"` from
+  `"computing"`).
+
+These invariants are enforced at construction time
+by the tokenization pipeline. Code that receives a
+`Term` can rely on them without re-checking.
+
+## Construction
+
+```rust
+impl Term {
+   pub fn new(s: &str) -> Result<Self, TermError>;
+
+   pub fn from_stemmed(s: String) -> Self;
+}
+```
+
+### `Term::new`
+
+Validates the input string against the term
+invariants. Returns `TermError::Empty` for empty
+strings, `TermError::ContainsWhitespace` if
+whitespace is found.
+
+This is the safe constructor for terms created
+outside the pipeline (e.g. from deserialized index
+data).
+
+### `Term::from_stemmed`
+
+Unchecked constructor for terms produced by the
+stemming stage. The pipeline guarantees the
+invariants, so validation is skipped. This is a
+performance path -- only the tokenization pipeline
+should call it.
+
+## Methods
+
+```rust
+impl Term {
+   pub fn as_str(&self) -> &str;
+
+   pub fn len(&self) -> usize;
+}
+```
+
+### `as_str`
+
+Returns the inner string slice. Used for index
+lookups, display, and serialization.
+
+### `len`
+
+Returns the byte length. Used by TF-IDF weighting
+and B+ tree key sizing.
+
+## Trait Implementations
+
+| Trait        | Purpose                          |
+|--------------|----------------------------------|
+| `Clone`      | Terms are stored in multiple structures (index, cache, results) |
+| `Debug`      | Diagnostic output                |
+| `PartialEq` / `Eq` | Term matching in index lookups |
+| `Hash`       | HashMap keys (inverted index, query cache) |
+| `Ord` / `PartialOrd` | B+ tree key ordering in `fs-database` |
+| `Display`    | Formats as the raw term string   |
+| `Encode`     | Binary serialization for shared memory (see [encode](../encode/traits.md)) |
+
+`Ord` **is** derived for `Term` (unlike `KbId`)
+because the B+ tree index requires ordered keys for
+range queries and efficient lookups.
+
+## Error Type
+
+```rust
+#[derive(Debug)]
+pub enum TermError {
+   Empty,
+   ContainsWhitespace,
+}
+```
+
+## Testing
+
+Unit tests:
+
+- `new` accepts lowercase, non-empty, no-whitespace
+  strings.
+- `new` rejects empty strings and strings with
+  spaces, tabs, or newlines.
+- `from_stemmed` produces a valid term without
+  validation overhead.
+- Ordering: terms sort lexicographically
+  (`"apple" < "banana"`).
+- `Encode` round-trip: encode then decode produces
+  an equal `Term`.

package/docs/modules/typescript/cli.md

@@ -0,0 +1,129 @@
+# src/cli.ts -- CLI and MCP Executable
+
+*Thin entry point that dispatches to CLI mode or MCP
+server mode based on the first argument.*
+
+## File
+
+`src/cli.ts`
+
+## Built To
+
+| Format | Output           | Notes          |
+|--------|------------------|----------------|
+| ESM    | `dist/cli.mjs`   | shebang, npm bin |
+| CJS    | `dist/cli.cjs`   | shebang        |
+| `.d.ts`| `dist/cli.d.ts`  |                |
+
+The npm `bin` entry `kbdb` points to `dist/cli.mjs`.
+
+## Run Modes
+
+```
+kbdb mcp [--db <path>]       → MCP server mode
+kbdb <command> [options]      → CLI mode (default)
+```
+
+The executable inspects its first argument. If `mcp`,
+it starts as a long-lived MCP server. Otherwise, it
+runs a single CLI command and exits.
+
+## Execution Flow
+
+### CLI Mode
+
+```
+src/cli.ts
+  │
+  ├─ import detectRuntime from shared/runtime/
+  ├─ import getArgs from shared/runtime/
+  ├─ import parseArgs from shared/cli/
+  ├─ import createWorkerClient from
+  │    shared/worker-client/
+  │
+  ├─ detect runtime (Deno / Node.js)
+  ├─ argv = getArgs()
+  ├─ options = parseArgs(argv)
+  │
+  ├─ if options.init:
+  │    └─ ensureDb(options.dbPath)
+  │
+  ├─ if options.command ≠ 'mcp':
+  │    ├─ client = await createWorkerClient(options)
+  │    ├─ if options.migrate:
+  │    │    └─ run database migration
+  │    ├─ match command:
+  │    │    ├─ learn    → runLearn(client, options)
+  │    │    ├─ unlearn  → runUnlearn(client, options)
+  │    │    ├─ search   → runSearch(client, options)
+  │    │    ├─ content  → runContent(client, options)
+  │    │    ├─ check    → runCheck(client, options)
+  │    │    ├─ gc       → runGc(client, options)
+  │    │    └─ rebuild  → runRebuild(client, options)
+  │    ├─ formatOutput(result, options.format)
+  │    ├─ print to stdout
+  │    └─ client.disconnect()
+  │
+  └─ exit
+```
+
+### MCP Mode
+
+```
+src/cli.ts
+  │
+  ├─ argv[0] === 'mcp'
+  ├─ resolve --db flag or KBDB_DIR env
+  ├─ client = await createWorkerClient({ targetDir })
+  ├─ server = createMcpServer(client)
+  ├─ server.listen()   ← stdio, blocks until closed
+  └─ client.disconnect()
+```
+
+## Imports
+
+```ts
+import {
+   detectRuntime,
+} from './src/shared/runtime/functions/detect-runtime.function.ts';
+import {
+   getArgs,
+} from './src/shared/runtime/functions/get-args.function.ts';
+import {
+   parseArgs,
+} from './src/shared/cli/functions/parse-args.function.ts';
+import {
+   createWorkerClient,
+} from './src/shared/worker-client/functions/create-worker-client.function.ts';
+import {
+   createMcpServer,
+} from './src/shared/mcp/functions/create-mcp-server.function.ts';
+```
+
+## No Business Logic
+
+This file contains **only** runtime detection, arg
+parsing dispatch, and mode selection. All logic lives
+in the shared sub-modules:
+
+| Concern           | Module                    |
+|-------------------|---------------------------|
+| Arg parsing       | `shared/cli/`             |
+| CLI commands      | `shared/cli/`             |
+| Output formatting | `shared/cli/`             |
+| MCP server        | `shared/mcp/`             |
+| IPC to daemon     | `shared/worker-client/`   |
+| Runtime detection | `shared/runtime/`         |
+
+## Dual Runtime
+
+Works on both Deno and Node.js. Runtime-specific
+code is handled by `shared/runtime/` and
+`shared/platform/`.
+
+- **Deno**: `deno run --allow-all src/cli.ts learn ./docs`
+- **Node**: `node dist/cli.mjs learn ./docs`
+- **npm bin**: `kbdb learn ./docs`
+
+See [CLI](../../goals/cli.md) for the full behavioral
+specification.