@dikolab/kbdb 0.5.0 → 0.6.0

package/docs/goals/content-parser.md

@@ -1,279 +0,0 @@
-# Content Parser -- Marker Extraction
-
-*A format-agnostic parser that extracts embedding
-and link markers from section content, producing
-structured reference lists for result set
-resolution.*
-
-## Goal
-
-The content parser scans any text-based content for
-`$(type: value)` markers and returns a structured
-list of extracted references. It is the engine
-behind the `references` field in retrieval and
-search result sets -- given a section's content, it
-identifies every embedded or linked section and
-document so the caller can resolve them.
-
-The parser is **format-agnostic**. It recognizes the
-`$(type: value)` escape sequence regardless of the
-surrounding text format -- markdown, source code,
-plain text, or any other UTF-8 content. The marker
-syntax is independent of the host format.
-
-See [Document](document.md) for marker syntax and
-semantics (what each marker type means, how it
-renders). This document specifies the **extraction
-mechanism** -- how markers are found in text and
-returned as structured data.
-
-## Input
-
-The parser accepts any **UTF-8 string**. This
-includes:
-
-- **Text section content** -- markdown body text
-  (the primary use case).
-- **Code section content** -- raw source code. Code
-  sections may contain markers in comments or string
-  literals (though uncommon).
-- **Plain text** -- any unstructured text.
-
-Image sections (`image/*`) contain base64-encoded
-binary data and cannot contain markers. The caller
-should skip image sections -- the parser would find
-no markers in base64 content.
-
-## Output
-
-The parser returns an ordered list of extracted
-markers. Each marker is a structured record:
-
-```rust
-pub struct ExtractedContentMarker {
-   /// The marker category (Image, Code, Text, etc).
-   pub marker_type: ContentMarkerType,
-   /// The value inside the marker (kb-id or doc-id).
-   pub value: String,
-   /// Byte offset of the marker start (`$`) in the
-   /// original text.
-   pub offset: usize,
-   /// Byte length of the entire marker, including
-   /// the `$(` and `)` delimiters.
-   pub length: usize,
-}
-
-pub enum ContentMarkerType {
-   /// `$(image: {kbid})` -- image section embedding.
-   Image,
-   /// `$(code: {kbid})` -- code section embedding.
-   Code,
-   /// `$(text: {kbid})` -- text section embedding.
-   Text,
-   /// `$(document: {docid})` -- document embedding.
-   Document,
-   /// `$(link: {kbid})` -- section cross-reference.
-   Link,
-   /// `$(doc-link: {docid})` -- document
-   /// cross-reference.
-   DocLink,
-}
-```
-
-Markers are returned in **document order** -- the
-order they appear in the input text, from lowest to
-highest byte offset.
-
-## Extraction Rules
-
-The parser uses the same syntax rules defined in
-[Document](document.md):
-
-1. A marker starts with `$(` and ends with the next
-   `)`.
-2. The type identifier is matched against the six
-   recognized prefixes:
-   - `$(image:` -> `ContentMarkerType::Image`
-   - `$(code:` -> `ContentMarkerType::Code`
-   - `$(text:` -> `ContentMarkerType::Text`
-   - `$(document:` -> `ContentMarkerType::Document`
-   - `$(link:` -> `ContentMarkerType::Link`
-   - `$(doc-link:` -> `ContentMarkerType::DocLink`
-3. Unrecognized prefixes (e.g. `$(formula:`) are
-   **skipped** -- no `ExtractedContentMarker` is produced.
-4. Unclosed markers (no closing `)` before end of
-   input) are **skipped**.
-5. The value is the text between the colon-space
-   and the closing `)`, trimmed of leading and
-   trailing whitespace.
-
-The parser performs a single left-to-right scan
-of the input. It does not modify the input text.
-
-## API
-
-Two public functions in the `kbdb-shared` crate:
-
-### `extract_content_markers`
-
-```rust
-pub fn extract_content_markers(text: &str)
-   -> Vec<ExtractedContentMarker>
-```
-
-Scans the input text and returns all recognized
-markers in document order. Does not modify the
-input. Returns an empty `Vec` if no markers are
-found.
-
-### `strip_content_markers`
-
-```rust
-pub fn strip_content_markers(text: &str) -> String
-```
-
-Returns a new string with all recognized markers
-removed. Whitespace left by marker removal is
-preserved; the caller should normalise whitespace
-if needed.
-
-This function replaces the current implementation
-in `shared/src/pipeline/markers.rs`. The
-existing `strip_content_markers()` recognizes only
-`$(image:)` and `$(code:)`. The shared version
-recognizes all six marker types.
-
-Both functions share the same scanning logic
-internally. `strip_content_markers` is equivalent to
-removing every character range identified by
-`extract_content_markers`.
-
-## Recursive Resolution
-
-The parser itself is **non-recursive**. It performs
-a single-pass extraction on one text string and
-returns the markers found in that string. It does
-not follow references or read other sections.
-
-**Recursive reference resolution** -- following
-`$(text: {kbid})` and `$(document: {docid})`
-markers into their content to collect transitive
-references -- is the responsibility of the
-**caller**. The typical caller is the `kb-worker`
-orchestration crate, which:
-
-1. Receives a section's content.
-2. Calls `extract_content_markers()` to find direct
-   references.
-3. Fetches each referenced section or document.
-4. For `$(text:)` and `$(document:)` references,
-   calls `extract_content_markers()` on the referenced
-   content (recursing).
-5. Collects all unique references into a flat
-   `references` array.
-6. Uses a **visited set** to detect and break
-   circular references (see
-   [Document -- Recursion](document.md#recursion-and-circular-references)).
-
-The parser provides the building block; the
-orchestration layer provides the recursion.
-
-## Integration Points
-
-### Query Parser
-
-The `query-parser` crate imports `strip_content_markers()`
-from `kbdb-shared` for tokenization Pipeline
-Stage 1. This replaces the crate's earlier
-`strip_content_markers()` in `pipeline/markers.rs`, which
-recognized only `$(image:)` and `$(code:)`. The
-shared version recognizes all six marker types.
-
-See [Query Parser](query-parser.md) for the full
-tokenization pipeline.
-
-### Filesystem Database
-
-The `fs-database` crate imports
-`extract_content_markers()` from `kbdb-shared` for
-**integrity checking**. When validating a section's
-content, the integrity checker extracts all markers
-and verifies that each referenced kb-id or doc-id
-exists and has the expected type.
-
-See [Database -- Integrity](database.md) for the
-full integrity specification.
-
-### Worker Orchestration
-
-The `kb-worker` crate imports `extract_content_markers()`
-from `kbdb-shared` to build the `references` array
-in result sets. When a search or retrieval request
-returns sections, `kb-worker`:
-
-1. Scans each returned section's content with
-   `extract_content_markers()`.
-2. Collects unique kb-ids and doc-ids.
-3. Fetches the referenced sections and documents.
-4. Recurses into `$(text:)` and `$(document:)`
-   references to collect transitive references.
-5. Deduplicates and returns the flat `references`
-   array alongside the primary result data.
-
-## Crate Placement
-
-The parser lives in the `kbdb-shared` rlib crate
-as a `content_parser` module:
-
-```
-engine/crates/shared/src/
-  content-parser/
-    mod.rs          -- re-exports
-    types.rs        -- ContentMarkerType, ExtractedContentMarker
-    extract.rs      -- extract_content_markers()
-    strip.rs        -- strip_content_markers()
-```
-
-Added to `lib.rs` as `pub mod content_parser` alongside the
-existing modules (memory, encode, kbid, term,
-section, document, query, corpus, error).
-
-The `kbdb-shared` crate is an rlib (not a cdylib),
-so it produces no WASM artifact of its own. Its code
-is compiled into whichever cdylib crate imports it
-(`query-parser`, `fs-database`, `kb-worker`), with
-unused code eliminated per-consumer.
-
-## Testing
-
-Unit tests for `extract_content_markers()`:
-
-- Empty input returns empty vec.
-- Single `$(image: {kbid})` returns one marker with
-  correct type, value, offset, and length.
-- All six marker types recognized individually.
-- Multiple markers in sequence, returned in
-  document order.
-- Adjacent markers with no space between them.
-- Unrecognized `$(formula: ...)` produces no marker.
-- Unclosed marker (no `)`) produces no marker.
-- Unicode text surrounding markers -- offsets are
-  byte-accurate.
-- Marker-only input (no surrounding text).
-- Whitespace variations inside markers (e.g.
-  extra spaces after colon).
-
-Unit tests for `strip_content_markers()`:
-
-- Empty input returns empty string.
-- Plain text with no markers returns unchanged.
-- All six marker types stripped.
-- Multiple markers stripped in one pass.
-- Unrecognized markers left untouched.
-- Unclosed markers left untouched.
-- Unicode text preserved around stripped markers.
-
-Tests live in the `kbdb-shared` crate:
-`engine/crates/shared/src/content-parser/extract.rs` and
-`engine/crates/shared/src/content-parser/strip.rs` (inline
-`#[cfg(test)]` modules).