@dikolab/kbdb 0.1.5 → 0.2.0

package/docs/goals/content-parser.md

@@ -0,0 +1,279 @@
+# Content Parser -- Marker Extraction
+
+*A format-agnostic parser that extracts embedding
+and link markers from section content, producing
+structured reference lists for result set
+resolution.*
+
+## Goal
+
+The content parser scans any text-based content for
+`$(type: value)` markers and returns a structured
+list of extracted references. It is the engine
+behind the `references` field in retrieval and
+search result sets -- given a section's content, it
+identifies every embedded or linked section and
+document so the caller can resolve them.
+
+The parser is **format-agnostic**. It recognizes the
+`$(type: value)` escape sequence regardless of the
+surrounding text format -- markdown, source code,
+plain text, or any other UTF-8 content. The marker
+syntax is independent of the host format.
+
+See [Document](document.md) for marker syntax and
+semantics (what each marker type means, how it
+renders). This document specifies the **extraction
+mechanism** -- how markers are found in text and
+returned as structured data.
+
+## Input
+
+The parser accepts any **UTF-8 string**. This
+includes:
+
+- **Text section content** -- markdown body text
+  (the primary use case).
+- **Code section content** -- raw source code. Code
+  sections may contain markers in comments or string
+  literals (though uncommon).
+- **Plain text** -- any unstructured text.
+
+Image sections (`image/*`) contain base64-encoded
+binary data and cannot contain markers. The caller
+should skip image sections -- the parser would find
+no markers in base64 content.
+
+## Output
+
+The parser returns an ordered list of extracted
+markers. Each marker is a structured record:
+
+```rust
+pub struct ExtractedContentMarker {
+   /// The marker category (Image, Code, Text, etc).
+   pub marker_type: ContentMarkerType,
+   /// The value inside the marker (kb-id or doc-id).
+   pub value: String,
+   /// Byte offset of the marker start (`$`) in the
+   /// original text.
+   pub offset: usize,
+   /// Byte length of the entire marker, including
+   /// the `$(` and `)` delimiters.
+   pub length: usize,
+}
+
+pub enum ContentMarkerType {
+   /// `$(image: {kbid})` -- image section embedding.
+   Image,
+   /// `$(code: {kbid})` -- code section embedding.
+   Code,
+   /// `$(text: {kbid})` -- text section embedding.
+   Text,
+   /// `$(document: {docid})` -- document embedding.
+   Document,
+   /// `$(link: {kbid})` -- section cross-reference.
+   Link,
+   /// `$(doc-link: {docid})` -- document
+   /// cross-reference.
+   DocLink,
+}
+```
+
+Markers are returned in **document order** -- the
+order they appear in the input text, from lowest to
+highest byte offset.
+
+## Extraction Rules
+
+The parser uses the same syntax rules defined in
+[Document](document.md):
+
+1. A marker starts with `$(` and ends with the next
+   `)`.
+2. The type identifier is matched against the six
+   recognized prefixes:
+   - `$(image:` -> `ContentMarkerType::Image`
+   - `$(code:` -> `ContentMarkerType::Code`
+   - `$(text:` -> `ContentMarkerType::Text`
+   - `$(document:` -> `ContentMarkerType::Document`
+   - `$(link:` -> `ContentMarkerType::Link`
+   - `$(doc-link:` -> `ContentMarkerType::DocLink`
+3. Unrecognized prefixes (e.g. `$(formula:`) are
+   **skipped** -- no `ExtractedContentMarker` is produced.
+4. Unclosed markers (no closing `)` before end of
+   input) are **skipped**.
+5. The value is the text between the colon-space
+   and the closing `)`, trimmed of leading and
+   trailing whitespace.
+
+The parser performs a single left-to-right scan
+of the input. It does not modify the input text.
+
+## API
+
+Two public functions in the `kbdb-shared` crate:
+
+### `extract_content_markers`
+
+```rust
+pub fn extract_content_markers(text: &str)
+   -> Vec<ExtractedContentMarker>
+```
+
+Scans the input text and returns all recognized
+markers in document order. Does not modify the
+input. Returns an empty `Vec` if no markers are
+found.
+
+### `strip_content_markers`
+
+```rust
+pub fn strip_content_markers(text: &str) -> String
+```
+
+Returns a new string with all recognized markers
+removed. Whitespace left by marker removal is
+preserved; the caller should normalise whitespace
+if needed.
+
+This function replaces the current implementation
+in `query-parser/src/pipeline/markers.rs`. The
+existing `strip_content_markers()` recognizes only
+`$(image:)` and `$(code:)`. The shared version
+recognizes all six marker types.
+
+Both functions share the same scanning logic
+internally. `strip_content_markers` is equivalent to
+removing every character range identified by
+`extract_content_markers`.
+
+## Recursive Resolution
+
+The parser itself is **non-recursive**. It performs
+a single-pass extraction on one text string and
+returns the markers found in that string. It does
+not follow references or read other sections.
+
+**Recursive reference resolution** -- following
+`$(text: {kbid})` and `$(document: {docid})`
+markers into their content to collect transitive
+references -- is the responsibility of the
+**caller**. The typical caller is the `kb-worker`
+orchestration crate, which:
+
+1. Receives a section's content.
+2. Calls `extract_content_markers()` to find direct
+   references.
+3. Fetches each referenced section or document.
+4. For `$(text:)` and `$(document:)` references,
+   calls `extract_content_markers()` on the referenced
+   content (recursing).
+5. Collects all unique references into a flat
+   `references` array.
+6. Uses a **visited set** to detect and break
+   circular references (see
+   [Document -- Recursion](document.md#recursion-and-circular-references)).
+
+The parser provides the building block; the
+orchestration layer provides the recursion.
+
+## Integration Points
+
+### Query Parser
+
+The `query-parser` crate imports `strip_content_markers()`
+from `kbdb-shared` for tokenization Pipeline
+Stage 1. This replaces the crate's earlier
+`strip_content_markers()` in `pipeline/markers.rs`, which
+recognized only `$(image:)` and `$(code:)`. The
+shared version recognizes all six marker types.
+
+See [Query Parser](query-parser.md) for the full
+tokenization pipeline.
+
+### Filesystem Database
+
+The `fs-database` crate imports
+`extract_content_markers()` from `kbdb-shared` for
+**integrity checking**. When validating a section's
+content, the integrity checker extracts all markers
+and verifies that each referenced kb-id or doc-id
+exists and has the expected type.
+
+See [Database -- Integrity](database.md) for the
+full integrity specification.
+
+### Worker Orchestration
+
+The `kb-worker` crate imports `extract_content_markers()`
+from `kbdb-shared` to build the `references` array
+in result sets. When a search or retrieval request
+returns sections, `kb-worker`:
+
+1. Scans each returned section's content with
+   `extract_content_markers()`.
+2. Collects unique kb-ids and doc-ids.
+3. Fetches the referenced sections and documents.
+4. Recurses into `$(text:)` and `$(document:)`
+   references to collect transitive references.
+5. Deduplicates and returns the flat `references`
+   array alongside the primary result data.
+
+## Crate Placement
+
+The parser lives in the `kbdb-shared` rlib crate
+as a `content_parser` module:
+
+```
+engine/crates/shared/src/
+  content-parser/
+    mod.rs          -- re-exports
+    types.rs        -- ContentMarkerType, ExtractedContentMarker
+    extract.rs      -- extract_content_markers()
+    strip.rs        -- strip_content_markers()
+```
+
+Added to `lib.rs` as `pub mod content_parser` alongside the
+existing modules (memory, encode, kbid, term,
+section, document, query, corpus, error).
+
+The `kbdb-shared` crate is an rlib (not a cdylib),
+so it produces no WASM artifact of its own. Its code
+is compiled into whichever cdylib crate imports it
+(`query-parser`, `fs-database`, `kb-worker`), with
+unused code eliminated per-consumer.
+
+## Testing
+
+Unit tests for `extract_content_markers()`:
+
+- Empty input returns empty vec.
+- Single `$(image: {kbid})` returns one marker with
+  correct type, value, offset, and length.
+- All six marker types recognized individually.
+- Multiple markers in sequence, returned in
+  document order.
+- Adjacent markers with no space between them.
+- Unrecognized `$(formula: ...)` produces no marker.
+- Unclosed marker (no `)`) produces no marker.
+- Unicode text surrounding markers -- offsets are
+  byte-accurate.
+- Marker-only input (no surrounding text).
+- Whitespace variations inside markers (e.g.
+  extra spaces after colon).
+
+Unit tests for `strip_content_markers()`:
+
+- Empty input returns empty string.
+- Plain text with no markers returns unchanged.
+- All six marker types stripped.
+- Multiple markers stripped in one pass.
+- Unrecognized markers left untouched.
+- Unclosed markers left untouched.
+- Unicode text preserved around stripped markers.
+
+Tests live in the `kbdb-shared` crate:
+`engine/crates/shared/src/content-parser/extract.rs` and
+`engine/crates/shared/src/content-parser/strip.rs` (inline
+`#[cfg(test)]` modules).