@dikolab/kbdb 0.1.6 → 0.2.0

package/docs/modules/rust/shared/content-composer/functions.md

@@ -0,0 +1,114 @@
+# content-composer -- Functions
+
+## `compose_content`
+
+```rust
+pub fn compose_content(
+   request: &ComposeRequest,
+) -> ComposeResult
+```
+
+Takes a fully resolved request (all sections and
+documents pre-fetched, content markers pre-extracted)
+and returns the composed Markdown string. The
+function is pure -- no I/O, no database access.
+Resolution and fetching are the caller's
+responsibility (`kb-worker`).
+
+### Composition Flow
+
+1. Initialize `renderedSections` and
+   `renderedDocuments` tracking sets.
+2. For each `ResolvedId` in `main_ids` (in order):
+   - If `Section(kbid)`: render as a section (anchor
+     + heading + content with marker processing), add
+     to `renderedSections`.
+   - If `Document(docid)`: render as a document
+     section (see Document Rendering below), add to
+     `renderedDocuments`.
+3. Traverse the full references set and render any
+   entities not yet in `renderedSections` or
+   `renderedDocuments` (remaining references, placed
+   at the bottom of the output).
+4. Return `ComposeResult` with the composed Markdown
+   and character count.
+
+### Content Marker Rendering
+
+Each `$(type: value)` content marker in a section's
+body is replaced by its rendered form:
+
+| Marker             | Rendered output                   |
+|--------------------|-----------------------------------|
+| `$(image: {kbid})` | Inline image (base64-decoded content from referenced `image/*` section) |
+| `$(code: {kbid})`  | Fenced code block with language tag derived from MIME type |
+| `$(text: {kbid})`  | Inline expansion of referenced section content (recursive, with cycle detection) |
+| `$(document: {docid})` | Document section rendering (heading + subsections, recursive) |
+| `$(link: {kbid})`  | Markdown link `[{kbid}](#{kbid})` (no inline expansion) |
+| `$(doc-link: {docid})` | Markdown link `[{docid}](#{docid})` (no inline expansion) |
+
+Cyclic `$(text:)` and `$(document:)` embeds (entity
+already in the render stack) are rendered as Markdown
+links to the entity's anchor instead of expanding
+inline.
+
+### MIME-to-Language Mapping
+
+For `$(code:)` markers, the fenced code block
+language tag is derived from the referenced section's
+MIME type:
+
+| MIME type              | Language tag |
+|------------------------|-------------|
+| `text/typescript`      | `ts`        |
+| `text/javascript`      | `js`        |
+| `text/html`            | `html`      |
+| `text/css`             | `css`       |
+| `text/xml`             | `xml`       |
+| `text/plain`           | *(none)*    |
+| `application/json`     | `json`      |
+| other `text/*`         | subtype     |
+
+### Document Rendering
+
+When a document is rendered (from `$(document:)`
+marker, `$(doc-link:)` target, or main docid input):
+
+1. Emit document anchor: `<a id="{docid}"></a>`
+2. Emit heading with document title.
+3. For each section in the document (position order):
+   - Emit section anchor: `<a id="{kbid}"></a>`
+   - Emit sub-heading.
+   - Render section content with marker processing.
+   - Add kbid to `renderedSections`.
+4. Add docid to `renderedDocuments`.
+
+### Tests
+
+Unit tests in
+`engine/crates/shared/src/content-composer/compose.rs`
+and `render.rs`:
+
+- Empty request returns empty markdown.
+- Single text section with no markers.
+- Single section with `$(image:)` marker.
+- Single section with `$(code:)` marker and correct
+  language tag.
+- Recursive `$(text:)` embedding.
+- `$(document:)` renders document heading with
+  section subsections.
+- Cyclic `$(text:)` embed becomes link to anchor.
+- Cyclic `$(document:)` embed becomes link to
+  document anchor.
+- `$(link:)` renders as Markdown link, target
+  rendered at bottom.
+- `$(doc-link:)` renders as Markdown link pointing
+  to document anchor.
+- Mixed kbids and docids in main list.
+- Named anchors assigned correctly to all entities.
+- `renderedSections` prevents duplicate rendering.
+- `renderedDocuments` prevents duplicate document
+  rendering.
+- Remaining references rendered at bottom only for
+  unrendered entities.
+- Unicode content preserved through composition.

package/docs/modules/rust/shared/content-composer/module.md

@@ -0,0 +1,147 @@
+# content-composer -- Content Composition
+
+*Composition engine that resolves sections and
+documents from identifiers, processes content markers
+recursively, and produces a single rendered Markdown
+document.*
+
+## Module
+
+| Field       | Value                                        |
+|-------------|----------------------------------------------|
+| Rust module | `content_composer`                           |
+| Source path | `engine/crates/shared/src/content-composer/` |
+| Crate       | `kbdb-shared` (rlib)                         |
+
+The module is declared in `lib.rs` with a path
+attribute because the source directory uses a
+hyphenated name:
+
+```rust
+#[path = "content-composer/mod.rs"]
+pub mod content_composer;
+```
+
+## Directory Layout
+
+```
+engine/crates/shared/src/content-composer/
+   mod.rs          -- re-exports
+   types.rs        -- ComposeRequest, ComposeResult,
+                      RenderedSection
+   compose.rs      -- compose_content()
+   render.rs       -- section and marker rendering
+```
+
+## Purpose
+
+The content creator takes one or more identifiers
+(kbids or docids, mixed) and produces a composed
+Markdown document. It resolves each identifier to its
+sections, follows embedding and link content markers
+recursively, and renders everything into a single
+UTF-8 Markdown string with named anchors for
+cross-referencing.
+
+The rendering logic is **pure composition** -- no
+I/O, no database access. Resolution and fetching of
+sections from the database is handled by `kb-worker`
+before calling into this module.
+
+## Two-Phase Composition
+
+### Phase 1 -- Resolve and Gather (kb-worker)
+
+1. Resolve identifiers (try kbid, then docid)
+2. Expand documents to section kbids
+3. Deduplicate while preserving order
+4. Fetch main sections from fs-database
+5. Extract content markers recursively using
+   `extract_content_markers()` from content-parser
+6. Fetch reference content
+7. Assign HTML named anchors
+
+### Phase 2 -- Content Composition (this module)
+
+Takes the pre-fetched `ComposeRequest` and produces
+a Markdown string. Processes content markers in
+section bodies, rendering each by type (see
+[functions.md](functions.md) for rendering rules).
+
+## Cycle Detection
+
+Two distinct mechanisms prevent infinite loops:
+
+- **Resolution cycle detection** (Phase 1): a visited
+  set tracks which kbids and docids have been
+  processed during recursive marker extraction.
+- **Render cycle detection** (Phase 2): a render
+  stack tracks which entities are currently being
+  expanded. If a `$(text:)` or `$(document:)` marker
+  references an entity already in the render stack,
+  it is a cyclic embed -- rendered as a Markdown link
+  to the entity's anchor instead of expanding inline.
+
+The render stack is distinct from the
+`renderedSections` / `renderedDocuments` tracking
+sets. An entity in the render stack is currently
+being expanded (not yet complete). An entity in
+`renderedSections` has been fully rendered and will
+not be rendered again.
+
+## Dependencies
+
+Sibling modules within `kbdb-shared`:
+
+- `content_parser::extract_content_markers()` -- for
+  processing markers within referenced content during
+  recursive rendering.
+- `content_parser::ContentMarkerType` -- for
+  dispatching marker rendering by type.
+- `section::SectionType` -- for type-specific
+  rendering (text vs code vs image).
+
+## Integration Points
+
+### Worker Orchestration
+
+The `kb-worker` crate orchestrates the full flow:
+receives a `compose` IPC request, resolves
+identifiers via `fs-database`, fetches content,
+extracts content markers recursively, builds a
+`ComposeRequest`, and calls `compose_content()`.
+See [compose](../../kb-worker/functions/compose.md).
+
+### Worker Client
+
+`WorkerClient.content(ids)` sends the `compose` IPC
+method to the daemon and returns a `ContentResult`.
+
+### CLI
+
+`runContent(client, options)` calls
+`client.content(ids)` and writes the composed
+Markdown to stdout. Command: `kbdb content <id>...`
+
+### MCP
+
+The `content` MCP tool routes to
+`client.content(ids)` and wraps the result in the
+MCP tool response format.
+
+## Sub-documents
+
+- [types.md](types.md) -- `ComposeRequest`,
+  `ComposeResult`, `ResolvedId`, `SectionData`,
+  `DocumentData`, `RenderedSection`
+- [functions.md](functions.md) --
+  `compose_content()`
+
+## Testing
+
+Tests live inline in
+`engine/crates/shared/src/content-composer/compose.rs`
+and
+`engine/crates/shared/src/content-composer/render.rs`
+(`#[cfg(test)]` modules). See
+[functions.md](functions.md) for test descriptions.

package/docs/modules/rust/shared/content-composer/types.md

@@ -0,0 +1,93 @@
+# content-composer -- Types
+
+## `ComposeRequest`
+
+Input to the composition engine. Contains all
+pre-fetched data needed to produce the composed
+Markdown -- no I/O occurs during composition.
+
+```rust
+pub struct ComposeRequest {
+   pub main_ids: Vec<ResolvedId>,
+   pub sections: HashMap<String, SectionData>,
+   pub documents: HashMap<String, DocumentData>,
+}
+```
+
+| Field       | Description                              |
+|-------------|------------------------------------------|
+| `main_ids`  | Ordered list of resolved identifiers forming the primary content |
+| `sections`  | All pre-fetched sections, keyed by kbid  |
+| `documents` | All pre-fetched documents, keyed by docid|
+
+## `ResolvedId`
+
+Discriminated identifier from the resolution phase.
+
+```rust
+pub enum ResolvedId {
+   Section(String),
+   Document(String),
+}
+```
+
+## `SectionData`
+
+Pre-fetched section content with extracted content
+markers.
+
+```rust
+pub struct SectionData {
+   pub kbid: String,
+   pub section_type: SectionType,
+   pub content: String,
+   pub markers: Vec<ExtractedContentMarker>,
+}
+```
+
+| Field          | Description                           |
+|----------------|---------------------------------------|
+| `kbid`         | Content-addressed section identifier  |
+| `section_type` | MIME-based type (`SectionType` from `section` module) |
+| `content`      | Raw section body text                 |
+| `markers`      | Pre-extracted content markers from `extract_content_markers()` |
+
+## `DocumentData`
+
+Pre-fetched document manifest.
+
+```rust
+pub struct DocumentData {
+   pub docid: String,
+   pub title: String,
+   pub section_kbids: Vec<String>,
+}
+```
+
+| Field          | Description                           |
+|----------------|---------------------------------------|
+| `docid`        | Document identifier                   |
+| `title`        | Document title for heading rendering  |
+| `section_kbids`| Ordered section kbids in this document|
+
+## `ComposeResult`
+
+Output of the composition engine.
+
+```rust
+pub struct ComposeResult {
+   pub markdown: String,
+   pub total_chars: usize,
+}
+```
+
+| Field        | Description                             |
+|--------------|-----------------------------------------|
+| `markdown`   | Composed Markdown string                |
+| `total_chars` | Character count of `markdown` (not byte count) |
+
+## `RenderedSection`
+
+Internal rendering type used by `render.rs` during
+Phase 2 composition. Tracks per-section rendering
+state.

package/docs/modules/rust/shared/content-parser/functions.md

@@ -0,0 +1,72 @@
+# content-parser -- Functions
+
+## `extract_content_markers`
+
+```rust
+pub fn extract_content_markers(text: &str)
+   -> Vec<ExtractedContentMarker>
+```
+
+Scans the input text and returns all recognized
+content markers in document order (lowest to highest
+byte offset). Does not modify the input. Returns an
+empty `Vec` if no markers are found.
+
+The function performs a single left-to-right scan.
+It is non-recursive -- it does not follow references
+or read other sections.
+
+### Tests
+
+Unit tests in
+`engine/crates/shared/src/content-parser/extract.rs`:
+
+- Empty input returns empty vec.
+- Single `$(image: {kbid})` returns one marker with
+  correct type, value, offset, and length.
+- All six content marker types recognized
+  individually.
+- Multiple markers in sequence, returned in document
+  order.
+- Adjacent markers with no space between them.
+- Unrecognized `$(formula: ...)` produces no marker.
+- Unclosed marker (no `)`) produces no marker.
+- Unicode text surrounding markers -- offsets are
+  byte-accurate.
+- Marker-only input (no surrounding text).
+- Whitespace variations inside markers (e.g. extra
+  spaces after colon).
+
+## `strip_content_markers`
+
+```rust
+pub fn strip_content_markers(text: &str) -> String
+```
+
+Returns a new string with all recognized content
+markers removed. Whitespace left by marker removal
+is preserved; the caller should normalise whitespace
+if needed.
+
+This function replaces the earlier implementation in
+`query-parser/src/pipeline/markers.rs`, which only
+recognized `$(image:)` and `$(code:)`. The shared
+version recognizes all six content marker types.
+
+Both functions share the same scanning logic
+internally. `strip_content_markers` is equivalent to
+removing every character range identified by
+`extract_content_markers`.
+
+### Tests
+
+Unit tests in
+`engine/crates/shared/src/content-parser/strip.rs`:
+
+- Empty input returns empty string.
+- Plain text with no markers returns unchanged.
+- All six content marker types stripped.
+- Multiple markers stripped in one pass.
+- Unrecognized markers left untouched.
+- Unclosed markers left untouched.
+- Unicode text preserved around stripped markers.

package/docs/modules/rust/shared/content-parser/module.md

@@ -0,0 +1,99 @@
+# content-parser -- Content Marker Extraction
+
+*Format-agnostic parser that extracts `$(type: value)`
+embedding and link markers from section content,
+producing structured reference lists for result set
+resolution.*
+
+## Module
+
+| Field       | Value                                     |
+|-------------|-------------------------------------------|
+| Rust module | `content_parser`                          |
+| Source path | `engine/crates/shared/src/content-parser/`|
+| Crate       | `kbdb-shared` (rlib)                      |
+
+The module is declared in `lib.rs` with a path
+attribute because the source directory uses a
+hyphenated name:
+
+```rust
+#[path = "content-parser/mod.rs"]
+pub mod content_parser;
+```
+
+## Directory Layout
+
+```
+engine/crates/shared/src/content-parser/
+   mod.rs          -- re-exports
+   types.rs        -- ContentMarkerType,
+                      ExtractedContentMarker
+   extract.rs      -- extract_content_markers()
+   strip.rs        -- strip_content_markers()
+```
+
+## Purpose
+
+The content parser scans any UTF-8 text for
+`$(type: value)` markers and returns a structured
+list of extracted references. It is format-agnostic
+-- the marker syntax is independent of the
+surrounding text format (markdown, source code,
+plain text).
+
+The parser performs a **single left-to-right scan**
+of the input. It is **non-recursive** -- it does not
+follow references or read other sections. Recursive
+reference resolution is the caller's responsibility
+(typically `kb-worker`).
+
+## Integration Points
+
+### Query Parser
+
+The `query-parser` crate imports
+`strip_content_markers()` from
+`kbdb_shared::content_parser` for tokenization
+Pipeline Stage 1. This replaces the crate's earlier
+local regex that only recognized `$(image:)` and
+`$(code:)`. The shared version recognizes all six
+content marker types.
+
+### Filesystem Database
+
+The `fs-database` crate imports
+`extract_content_markers()` from
+`kbdb_shared::content_parser` for integrity
+checking. The integrity checker extracts all markers
+and verifies that each referenced kb-id or doc-id
+exists and has the expected type.
+
+### Worker Orchestration
+
+The `kb-worker` crate imports
+`extract_content_markers()` from
+`kbdb_shared::content_parser` to build the
+`references` array in result sets. When a search
+or retrieval request returns sections, `kb-worker`
+scans each section's content, follows `$(text:)` and
+`$(document:)` references recursively using a visited
+set, and returns a flat deduplicated `references`
+array alongside the primary result data.
+
+## Sub-documents
+
+- [types.md](types.md) -- `ContentMarkerType`,
+  `ExtractedContentMarker`
+- [functions.md](functions.md) --
+  `extract_content_markers()`,
+  `strip_content_markers()`
+
+## Testing
+
+Tests live inline in
+`engine/crates/shared/src/content-parser/extract.rs`
+and
+`engine/crates/shared/src/content-parser/strip.rs`
+(`#[cfg(test)]` modules). See
+[functions.md](functions.md) for test descriptions.

package/docs/modules/rust/shared/content-parser/types.md

@@ -0,0 +1,71 @@
+# content-parser -- Types
+
+## `ContentMarkerType`
+
+Enum identifying the six recognized content marker
+types.
+
+```rust
+pub enum ContentMarkerType {
+   /// `$(image: {kbid})` -- image section embedding.
+   Image,
+   /// `$(code: {kbid})` -- code section embedding.
+   Code,
+   /// `$(text: {kbid})` -- text section embedding.
+   Text,
+   /// `$(document: {docid})` -- document embedding.
+   Document,
+   /// `$(link: {kbid})` -- section cross-reference.
+   Link,
+   /// `$(doc-link: {docid})` -- document
+   /// cross-reference.
+   DocLink,
+}
+```
+
+Each variant maps to a `$(type:` prefix:
+
+| Variant    | Prefix         | Value type |
+|------------|----------------|------------|
+| `Image`    | `$(image:`     | kbid       |
+| `Code`     | `$(code:`      | kbid       |
+| `Text`     | `$(text:`      | kbid       |
+| `Document` | `$(document:`  | docid      |
+| `Link`     | `$(link:`      | kbid       |
+| `DocLink`  | `$(doc-link:`  | docid      |
+
+## `ExtractedContentMarker`
+
+A single marker extracted from text content.
+
+```rust
+pub struct ExtractedContentMarker {
+   /// The marker category.
+   pub marker_type: ContentMarkerType,
+   /// The value inside the marker (kb-id or doc-id).
+   pub value: String,
+   /// Byte offset of the marker start (`$`) in the
+   /// original text.
+   pub offset: usize,
+   /// Byte length of the entire marker, including
+   /// the `$(` and `)` delimiters.
+   pub length: usize,
+}
+```
+
+## Syntax Rules
+
+Content markers follow these parsing rules:
+
+1. A marker starts with `$(` and ends with the next
+   `)`.
+2. The type identifier is matched against the six
+   recognized prefixes listed above.
+3. Unrecognized prefixes (e.g. `$(formula:`) are
+   **skipped** -- no `ExtractedContentMarker` is
+   produced.
+4. Unclosed markers (no closing `)` before end of
+   input) are **skipped**.
+5. The value is the text between the colon-space and
+   the closing `)`, trimmed of leading and trailing
+   whitespace.

package/docs/modules/rust/shared/corpus/module.md

@@ -0,0 +1,24 @@
+# corpus -- Corpus Statistics
+
+*Lightweight corpus statistics passed between modules
+for IDF computation and BM25F length normalization.*
+
+## Source
+
+`engine/crates/shared/src/corpus/`
+
+## Responsibility
+
+Defines the lightweight `CorpusStats` view passed
+from `fs-database` to `query-parser` via `kb-worker`
+for ranking computations. Contains only the scalar
+values needed for BM25F -- the full document
+frequency map stays private to `fs-database`.
+
+See [Query Parser -- Ranking](
+../../../../goals/query-parser.md) for how these
+values are used in BM25F scoring.
+
+## Exports
+
+- **[types.md](types.md)** -- `CorpusStats`.