@dikolab/kbdb 0.5.0 → 0.6.1

package/docs/goals/content-composer.md

@@ -1,609 +0,0 @@
-# Content Composer -- Content Composition
-
-*A composition engine that resolves sections and
-documents from identifiers, processes markers
-recursively, and produces a single rendered Markdown
-document.*
-
-## Goal
-
-The content composer takes one or more identifiers
-(kbids or docids, mixed) and produces a composed
-Markdown document. It resolves each identifier to
-its sections, follows embedding and link markers
-recursively, and renders everything into a single
-UTF-8 Markdown string with named anchors for
-cross-referencing.
-
-This is the **rendering engine** -- it performs what
-the marker descriptions in [Document](document.md)
-specify as "at render time" behaviour. It is distinct
-from the [Content Parser](content-parser.md) (which
-extracts markers as structured data) and the
-[Document](document.md) spec (which defines marker
-syntax and semantics).
-
-The rendering logic lives in the `content-composer`
-module in the `kbdb-shared` Rust crate as pure
-composition -- no I/O. Resolution and fetching of
-sections from the database is handled by `kb-worker`.
-
-## CLI Command
-
-```
-kbdb content <id>... [--db <path>]
-```
-
-Compose and output a rendered Markdown document from
-one or more knowledge base identifiers.
-
-**Arguments:**
-- `<id>...` -- one or more kbid or docid identifiers.
-  The two types can be mixed freely. Each identifier
-  is resolved via trial-and-error: the system first
-  attempts to look up the identifier as a kbid
-  (section); if not found, it tries as a docid
-  (document). If neither matches, an error is
-  reported for that identifier.
-
-**Output:**
-- **stdout**: UTF-8 Markdown text. The composed
-  document contains all requested sections and
-  documents with markers rendered, named anchors
-  assigned, and cross-references linked.
-- **stderr**: errors only.
-
-**Exit code:** `0` on success, `1` on error.
-
-**Delegation chain:**
-```
-src/cli.ts  parseArgs()
-  -> ensureDb() / findDb()
-  -> createWorkerClient({ contextPath })
-  -> runContent(client, options)
-       -> client.content(ids)
-            -> IPC -> worker daemon
-                 -> kb-worker.wasm
-                      -> fs-database.wasm (fetch)
-                      -> kbdb-shared content-composer
-                           (compose_content)
-  -> stdout
-```
-
-## MCP Tool
-
-### content
-
-Compose a rendered Markdown document from one or
-more kbid or docid identifiers.
-
-**Parameters:**
-
-| Name  | Type     | Required | Description       |
-|-------|----------|----------|-------------------|
-| `ids` | string[] | yes      | One or more kbid or docid identifiers (mixed) |
-
-**Returns:** Tool result with text content containing
-JSON:
-
-```json
-{
-   "type": "text/markdown",
-   "data": "<composed markdown>",
-   "total": 4567
-}
-```
-
-The `total` field is the character count of `data`
-(not byte count).
-
-Maps to `client.content()`.
-
-## Phase 1 -- Resolve and Gather
-
-The resolution phase collects all sections needed
-for composition. This phase runs in `kb-worker`,
-which has access to `fs-database` for lookups.
-
-### Step 1: Resolve identifiers
-
-For each input identifier:
-
-1. Try to find it as a **kbid** (section lookup via
-   `fs-database`).
-2. If not found, try to find it as a **docid**
-   (document lookup via `fs-database`).
-3. If neither matches, report an error for that
-   identifier and skip it.
-
-Identifiers are processed in the order given. The
-resolved identifiers form the **main list** -- the
-ordered set of sections and documents that become
-the primary content of the composed Markdown.
-
-### Step 2: Expand documents
-
-For each resolved docid in the main list, expand it
-to its ordered section kbids. The docid itself is
-retained as a document reference for rendering (see
-Document Rendering below). Its sections join the
-main kbid list in their document-defined order.
-
-### Step 3: Deduplicate
-
-Remove duplicate kbids from the main list while
-preserving order. If the same kbid appears from
-multiple sources (directly specified and also part
-of a document), the first occurrence wins.
-
-### Step 4: Fetch main sections
-
-Fetch the content of all main kbids from
-`fs-database`. Each section includes its kbid,
-section type, content body, and optional docid.
-
-### Step 5: Extract references recursively
-
-For each main section, call `extract_content_markers()`
-from the `content_parser` module to find all markers in
-its
-content. Build a **references** set of unique kbids
-and docids:
-
-- **All marker types** add their referenced kbid or
-  docid to the references set.
-- **Recursive markers** -- `$(text: {kbid})` and
-  `$(document: {docid})` -- follow into the
-  referenced content and extract markers from it
-  too, recursing until no new references are found.
-- A **visited set** tracks which kbids and docids
-  have already been processed to detect and break
-  circular references.
-
-### Step 6: Fetch reference content
-
-Fetch section content for all kbids in the references
-set that are not already fetched (from step 4). For
-docids in the references set, fetch the document
-manifest and its section content.
-
-### Step 7: Assign named anchors
-
-Assign HTML anchors to every section and document:
-
-- Sections: `<a id="{kbid}"></a>`
-- Documents: `<a id="{docid}"></a>`
-
-Anchors are assigned to all entities -- main sections,
-reference sections, main documents, and reference
-documents.
-
-## Phase 2 -- Content Composition
-
-The composition phase takes the pre-fetched sections
-and documents from Phase 1 and produces a Markdown
-string. This phase runs in the `content-composer`
-module in `kbdb-shared` as pure logic -- no I/O.
-
-### State
-
-The composer maintains two tracking sets:
-
-- **`renderedSections`** -- set of kbids that have
-  been rendered into the output.
-- **`renderedDocuments`** -- set of docids that have
-  been rendered into the output.
-
-### Main sections
-
-For each kbid in the main list (in order):
-
-1. Emit the section's anchor tag:
-   `<a id="{kbid}"></a>`
-2. Emit a heading for the section.
-3. Render the section's content body, processing
-   any markers found (see Marker Rendering below).
-4. Add the kbid to `renderedSections`.
-
-If a main list entry is a docid, render it as a
-document section (see Document Rendering below).
-
-### Marker Rendering
-
-When processing a section's content body, each
-`$(type: value)` marker is replaced by its rendered
-form:
-
-#### `$(image: {kbid})`
-
-Replace with an inline image. The referenced section
-must be of type `image/*`. The image data is the
-base64-decoded content of the referenced section.
-
-#### `$(code: {kbid})`
-
-Replace with a fenced code block. The referenced
-section must be of type `text/*` or
-`application/json`. The language tag is derived from
-the section's MIME type using the mapping in
-[Document -- MIME-to-Language](
-document.md#mime-to-language-mapping).
-
-#### `$(text: {kbid})`
-
-Render the referenced text section's content
-**inline**. The referenced section's content body
-replaces the marker, and any markers within the
-referenced content are processed recursively.
-
-If the kbid is already in the current **render
-stack** (i.e. we are currently in the process of
-rendering it), this is a **cyclic embed**. Instead of
-expanding inline, emit a Markdown link to the
-section's anchor:
-
-```
-[{kbid}](#{kbid})
-```
-
-The referenced section is added to `renderedSections`
-after its content is fully rendered.
-
-#### `$(document: {docid})`
-
-Render the referenced document as a **document
-section** (see Document Rendering below).
-
-If the docid is already in the current render stack,
-this is a cyclic embed. Emit a Markdown link to the
-document's anchor:
-
-```
-[{docid}](#{docid})
-```
-
-#### `$(link: {kbid})`
-
-Emit a Markdown link pointing to the section's named
-anchor:
-
-```
-[{kbid}](#{kbid})
-```
-
-The link marker does **not** expand the referenced
-content inline. If the linked section has not been
-rendered yet, it will be rendered at the bottom of
-the document in the remaining references pass.
-
-#### `$(doc-link: {docid})`
-
-Emit a Markdown link pointing to the document's
-named anchor:
-
-```
-[{docid}](#{docid})
-```
-
-The link points to the document's container heading,
-not to any individual section within the document.
-If the document has not been rendered yet, it will
-be rendered at the bottom in the remaining
-references pass.
-
-## Document Rendering
-
-When a document is rendered -- whether from a
-`$(document: {docid})` marker, as a `$(doc-link:)`
-target, or as a main docid input -- it appears as a
-**structural document section** in the Markdown
-output.
-
-Structure:
-
-1. Emit the document's anchor tag:
-   `<a id="{docid}"></a>`
-2. Emit a heading with the document title.
-3. For each section in the document (in position
-   order):
-   - Emit the section's anchor tag:
-     `<a id="{kbid}"></a>`
-   - Emit a sub-heading for the section.
-   - Render the section's content body with marker
-     processing (same rules as Marker Rendering).
-   - Add the kbid to `renderedSections`.
-4. Add the docid to `renderedDocuments`.
-
-The document heading is the parent; its sections are
-subsections beneath it. `$(doc-link:)` links point to
-the document's anchor (the container heading), giving
-the reader a view of the entire document rather than
-an individual section.
-
-If a section within the document contains markers,
-they are processed recursively using the same
-embed/link rules. Cyclic detection uses the same
-render stack.
-
-## Remaining References
-
-After all main sections and documents have been
-rendered, the composer traverses the references set
-and renders any entities not yet in
-`renderedSections` or `renderedDocuments`.
-
-These are typically sections or documents referenced
-by `$(link:)` or `$(doc-link:)` markers whose targets
-were not directly requested or embedded. They appear
-at the bottom of the composed document, each with its
-own anchor and heading.
-
-For each unrendered reference:
-
-- If it is a kbid: render as a section (anchor +
-  heading + content with marker processing).
-- If it is a docid: render as a document section
-  (see Document Rendering above).
-
-## Cyclic Reference Handling
-
-There are two distinct cycle-detection mechanisms:
-
-### Resolution cycle detection (Phase 1)
-
-During recursive marker extraction (step 5), a
-**visited set** tracks which kbids and docids have
-been processed. If `extract_content_markers()` encounters a
-reference already in the visited set, it stops
-recursing into that reference. This prevents infinite
-loops during the gathering phase.
-
-### Render cycle detection (Phase 2)
-
-During content composition, a **render stack** tracks
-which kbids and docids are currently being rendered
-(i.e. their content is in the process of expanding).
-If a `$(text:)` or `$(document:)` marker references
-an entity already in the render stack, it is a cyclic
-embed. Instead of expanding inline, the composer
-emits a Markdown link to the entity's anchor.
-
-The render stack is distinct from `renderedSections`
-/ `renderedDocuments`. An entity in the render stack
-is currently being expanded (not yet complete). An
-entity in `renderedSections` has been fully rendered
-and will not be rendered again.
-
-## Rust Module
-
-The composition logic lives in the `content-composer`
-module in the `kbdb-shared` rlib crate.
-
-### Module Path
-
-```
-engine/crates/shared/src/content-composer/
-├── mod.rs          -- re-exports
-├── types.rs        -- ComposeRequest, ComposeResult,
-│                      RenderedSection
-├── compose.rs      -- compose_content()
-└── render.rs       -- section and marker rendering
-```
-
-Added to `lib.rs`:
-
-```rust
-#[path = "content-composer/mod.rs"]
-pub mod content_composer;
-```
-
-### Types
-
-```rust
-pub struct ComposeRequest {
-   pub main_ids: Vec<ResolvedId>,
-   pub sections: HashMap<String, SectionData>,
-   pub documents: HashMap<String, DocumentData>,
-}
-
-pub enum ResolvedId {
-   Section(String),
-   Document(String),
-}
-
-pub struct SectionData {
-   pub kbid: String,
-   pub section_type: SectionType,
-   pub content: String,
-   pub markers: Vec<ExtractedContentMarker>,
-}
-
-pub struct DocumentData {
-   pub docid: String,
-   pub title: String,
-   pub section_kbids: Vec<String>,
-}
-
-pub struct ComposeResult {
-   pub markdown: String,
-   pub total_chars: usize,
-}
-```
-
-### API
-
-```rust
-pub fn compose_content(
-   request: &ComposeRequest,
-) -> ComposeResult
-```
-
-Takes a fully resolved request (all sections and
-documents pre-fetched, markers pre-extracted) and
-returns the composed Markdown string. The function
-is pure -- no I/O, no database access. Resolution
-and fetching are the caller's responsibility
-(`kb-worker`).
-
-### Dependencies
-
-The module uses sibling modules within `kbdb-shared`:
-
-- `content_parser::extract_content_markers()` -- for
-  processing markers within referenced content during
-  recursive rendering.
-- `content_parser::ContentMarkerType` -- for
-  dispatching marker rendering by type.
-- `section::SectionType` -- for type-specific
-  rendering (text vs code vs image).
-
-## Integration Points
-
-### Worker Orchestration
-
-The `kb-worker` crate orchestrates the full flow:
-
-1. Receives a `content` request with identifiers.
-2. Resolves identifiers via `fs-database` (Phase 1,
-   steps 1-3).
-3. Fetches section and document content from
-   `fs-database` (steps 4, 6).
-4. Extracts markers recursively using
-   `extract_content_markers()` (step 5).
-5. Builds a `ComposeRequest` with all resolved data.
-6. Calls `compose_content()` from `content-composer`.
-7. Returns the `ComposeResult` to the daemon.
-
-### Worker Client
-
-The `WorkerClient` class exposes a `content()`
-method:
-
-```ts
-content(ids: string[]): Promise<ContentResult>;
-```
-
-See [Worker Client](worker-client.md) for the full
-interface.
-
-### CLI
-
-The `runContent()` function in `src/cli.ts` calls
-`client.content(ids)` and writes the resulting
-Markdown string to stdout.
-
-See [CLI](cli.md) for the full command specification.
-
-### MCP
-
-The `handleToolCall()` function routes the `content`
-tool to `client.content(ids)` and wraps the result
-in the MCP tool response format.
-
-See [MCP](mcp.md) for the full tool specification.
-
-## Example
-
-**Input:** `kbdb content abc12345 my-api-guide`
-
-Where `abc12345` is a kbid (text section) and
-`my-api-guide` is a docid with two sections
-(`def67890` and `ghi11111`).
-
-**Phase 1 resolves:**
-- Main list: `[abc12345, def67890, ghi11111]`
-- `my-api-guide` retained as document reference
-- Markers extracted: `abc12345` contains
-  `$(code: jkl22222)` and
-  `$(link: mno33333)`
-- References: `{jkl22222, mno33333}`
-
-**Phase 2 composes:**
-
-```markdown
-<a id="abc12345"></a>
-
-## abc12345
-
-This section explains the API.
-
-Here is the implementation:
-
-` ` `ts
-function greet(name: string) {
-   return `Hello, ${name}!`;
-}
-` ` `
-
-See also: [mno33333](#mno33333).
-
----
-
-<a id="my-api-guide"></a>
-
-## my-api-guide
-
-<a id="def67890"></a>
-
-### def67890
-
-First section of the API guide...
-
-<a id="ghi11111"></a>
-
-### ghi11111
-
-Second section of the API guide...
-
----
-
-<a id="mno33333"></a>
-
-## mno33333
-
-This section was referenced by a link marker. It
-is rendered here because it was not part of the
-main content or any embedded document.
-```
-
-In this example:
-- `abc12345` is a main section with a code embed
-  (`jkl22222` rendered as a fenced code block) and
-  a link (`mno33333` rendered as a Markdown link).
-- `my-api-guide` is rendered as a document section
-  with `def67890` and `ghi11111` as subsections.
-- `mno33333` appears at the bottom as a remaining
-  reference (linked but not embedded).
-
-## Testing
-
-Unit tests for `compose_content()`:
-
-- Empty request returns empty markdown.
-- Single text section with no markers.
-- Single section with `$(image:)` marker.
-- Single section with `$(code:)` marker and correct
-  language tag.
-- Recursive `$(text:)` embedding.
-- `$(document:)` renders document heading with
-  section subsections.
-- Cyclic `$(text:)` embed becomes link to anchor.
-- Cyclic `$(document:)` embed becomes link to
-  document anchor.
-- `$(link:)` renders as Markdown link, target
-  rendered at bottom.
-- `$(doc-link:)` renders as Markdown link pointing
-  to document anchor.
-- Mixed kbids and docids in main list.
-- Named anchors assigned correctly to all entities.
-- `renderedSections` prevents duplicate rendering.
-- `renderedDocuments` prevents duplicate document
-  rendering.
-- Remaining references rendered at bottom only for
-  unrendered entities.
-- Unicode content preserved through composition.
-
-Tests live in the `kbdb-shared` crate:
-`engine/crates/shared/src/content-composer/compose.rs`
-and `engine/crates/shared/src/content-composer/render.rs`
-(inline `#[cfg(test)]` modules).