@dikolab/kbdb 0.4.3 → 0.6.0

package/docs/goals/document.md

@@ -1,368 +0,0 @@
-# Document -- Text Content Format
-
-*Content format specification for text sections in
-the knowledge base -- markdown body text with
-embedding markers and cross-reference link markers.*
-
-## Goal
-
-Text sections in the knowledge base store
-**markdown-formatted UTF-8** content. This document
-specifies the content body format: what goes inside
-a section after the TOML header and `---` separator,
-how sections reference other sections and documents,
-and how those references behave at render time.
-
-This document covers content format only. See
-[Database](database.md) for how sections are stored
-on disk (file layout, TOML headers, sharding) and
-[Query Parser](query-parser.md) for how content is
-tokenized and indexed. See
-[Content Parser](content-parser.md) for how markers
-are extracted from content as structured data.
-
-Content is stored **verbatim** -- no processing,
-expansion, or transformation occurs at write time.
-Markers are plain text in the stored content body.
-All rendering and expansion happens at read time.
-
-## Content Format
-
-The content body of a text section (everything after
-the `---` separator in the section file) is
-**markdown-formatted UTF-8 text**. Any valid
-markdown is permitted: headings, paragraphs, lists,
-tables, fenced code blocks, inline formatting, and
-standard markdown links and images.
-
-In addition to standard markdown, text sections may
-contain **markers** -- escape sequences that
-reference other sections or documents in the
-knowledge base.
-
-Code sections (`text/*` and `application/json`)
-store raw source code. Image sections (`image/*`)
-store base64-encoded binary data. Neither code nor
-image sections use markdown formatting, but both
-may appear in text sections via embedding markers.
-
-## Markers
-
-Markers use the escape sequence `$(type: value)`:
-
-- `$` followed by `(`
-- A type identifier (lowercase, may contain hyphen)
-- A colon and a space
-- A value (kb-id or doc-id)
-- A closing `)`
-
-Example: `$(image: 7fa93b2e4c8d1076a93b2e4c8d)`
-
-Markers are stored verbatim in the content body.
-They are **not expanded or resolved at storage
-time**. Rendering happens when the content is read
-and returned to the caller.
-
-There are two categories of markers:
-
-- **Embedding markers** -- the referenced content is
-  expanded inline at render time. The marker is
-  replaced by the referenced content.
-- **Link markers** -- the referenced entity appears
-  as a cross-reference. The marker is replaced by a
-  reference or hyperlink, not by inline content.
-
-### Embedding Markers
-
-Embedding markers expand the referenced content
-inline when the section is rendered.
-
-#### `$(image: {kbid})`
-
-Embeds an image section. The `{kbid}` must reference
-an existing section of type `image/*`.
-
-At render time, the marker is replaced by the image
-content. In markdown output, this renders as an
-inline image. The image data is the base64-decoded
-content of the referenced section.
-
-```
-Here is a diagram of the architecture:
-
-$(image: 7fa93b2e4c8d1076a93b2e4c8d)
-
-The diagram shows the five-layer stack.
-```
-
-#### `$(code: {kbid})`
-
-Embeds a code section. The `{kbid}` must reference
-an existing section of type `text/*` or
-`application/json`.
-
-At render time, the marker is replaced by a fenced
-code block. The language tag is derived from the
-referenced section's MIME type using the mapping
-below.
-
-```
-See the greeting function:
-
-$(code: c4e8a91d3f7b20564e8a91d3f7)
-
-Call it with any name string.
-```
-
-##### MIME-to-Language Mapping
-
-When a `$(code: {kbid})` marker is rendered, the
-code section's MIME type determines the fenced code
-block language tag:
-
-| MIME Type            | Language Tag |
-|----------------------|--------------|
-| `text/typescript`    | `ts`         |
-| `text/javascript`    | `js`         |
-| `text/html`          | `html`       |
-| `text/css`           | `css`        |
-| `text/xml`           | `xml`        |
-| `text/plain`         | *(none)*     |
-| `application/json`   | `json`       |
-
-Other `text/*` subtypes use the subtype as the
-language tag (e.g. `text/python` -> `python`,
-`text/rust` -> `rust`).
-
-#### `$(text: {kbid})`
-
-Embeds a text section's content inline. The `{kbid}`
-must reference an existing section of type
-`text/markdown`.
-
-At render time, the marker is replaced by the
-referenced section's full content body. The
-embedded content is rendered as markdown, inline
-with the surrounding text.
-
-This marker is **recursive** -- the embedded text
-section may itself contain markers, which are
-resolved in turn. See Recursion and Circular
-References below.
-
-```
-## Introduction
-
-$(text: a1b2c3d4e5f6a1b2c3d4e5f6a1)
-
-## Next Steps
-
-Continue with the setup guide.
-```
-
-#### `$(document: {docid})`
-
-Embeds an entire document. The `{docid}` must
-reference an existing document manifest.
-
-At render time, the marker is replaced by all
-sections in the document, expanded in position
-order. Each section is rendered according to its
-type: text sections as markdown, code sections as
-fenced code blocks, image sections as inline images.
-
-This marker is **recursive** -- sections within the
-embedded document may contain their own markers.
-
-```
-For the full API reference, see below:
-
-$(document: api-reference-v2)
-
-End of the combined guide.
-```
-
-### Link Markers
-
-Link markers produce cross-references without
-expanding any content inline.
-
-#### `$(link: {kbid})`
-
-Links to a section by kb-id. The `{kbid}` must
-reference an existing section of **any type** (text,
-code, or image).
-
-At render time, the marker is replaced by a
-cross-reference to the linked section. The section's
-content is not expanded inline.
-
-```
-For the implementation details, see
-$(link: d5e6f7a8b9c0d5e6f7a8b9c0d5).
-```
-
-#### `$(doc-link: {docid})`
-
-Links to a document by doc-id. The `{docid}` must
-reference an existing document manifest.
-
-At render time, the marker is replaced by a
-cross-reference to the linked document. No content
-is expanded inline.
-
-```
-This section is part of
-$(doc-link: getting-started-guide).
-```
-
-### Marker Summary
-
-| Marker | Value | Category | Referenced Entity | Render Behavior |
-|--------|-------|----------|-------------------|-----------------|
-| `$(image: {kbid})` | kb-id | embedding | `image/*` section | Inline image |
-| `$(code: {kbid})` | kb-id | embedding | `text/*` or `application/json` section | Fenced code block |
-| `$(text: {kbid})` | kb-id | embedding | `text/markdown` section | Inline markdown (recursive) |
-| `$(document: {docid})` | doc-id | embedding | Document manifest | All sections expanded (recursive) |
-| `$(link: {kbid})` | kb-id | link | Any section | Cross-reference |
-| `$(doc-link: {docid})` | doc-id | link | Document manifest | Cross-reference |
-
-## Syntax Rules
-
-1. Markers start with `$(` and end with `)`.
-2. The type identifier is followed by a colon and
-   a **space** before the value: `$(type: value)`.
-3. The value is either a **kb-id** (26-character
-   base32-encoded identifier) or a **doc-id**
-   (document manifest identifier string).
-4. Markers cannot be nested. A `$(` inside a marker
-   is treated as literal text.
-5. Unrecognized type identifiers (e.g.
-   `$(formula: x+y)`) pass through unchanged --
-   they are not stripped or processed.
-6. Unclosed markers (no closing `)`) are treated as
-   plain text and left unchanged.
-7. Markers can appear anywhere in the content body:
-   inline with text, on their own line, or adjacent
-   to other markers.
-
-## Recursion and Circular References
-
-The `$(text: {kbid})` and `$(document: {docid})`
-embedding markers cause **recursive expansion**.
-When a text section is embedded, its content may
-itself contain markers that must be resolved. When
-a document is embedded, each of its sections may
-contain markers.
-
-Circular references occur when the expansion chain
-loops back to a section or document already being
-expanded. For example:
-
-- Section A contains `$(text: B)`
-- Section B contains `$(text: A)`
-
-Circular references must be detected and reported
-as errors. The resolution strategy:
-
-1. Maintain a **visited set** of kb-ids and doc-ids
-   during expansion.
-2. Before expanding a marker, check if the target
-   is already in the visited set.
-3. If it is, the reference is **circular** -- stop
-   expansion and report an error for that marker.
-4. If it is not, add the target to the visited set
-   and proceed with expansion.
-
-Integrity checks detect circular references at
-validation time (see Integrity below). The
-expansion engine also guards against them at
-render time.
-
-## Tokenization
-
-All six marker types are stripped from content
-before tokenization and indexing. Markers are
-**structural references**, not searchable text.
-The tokenization pipeline removes them in Stage 1
-(marker stripping) so they do not pollute the
-search index.
-
-See [Query Parser](query-parser.md) for the full
-nine-stage tokenization pipeline and
-[Content Parser](content-parser.md) for the marker
-extraction functions used during stripping.
-
-The `strip_content_markers()` function in `kbdb-shared`
-recognizes all six marker types. See
-[Content Parser](content-parser.md) for the shared
-implementation that replaces the earlier
-query-parser-internal version.
-
-## Integrity
-
-Integrity checks validate that all marker
-references point to existing entities of the
-expected type:
-
-| Marker | Validation |
-|--------|------------|
-| `$(image: {kbid})` | Section exists, type is `image/*` |
-| `$(code: {kbid})` | Section exists, type is `text/*` or `application/json` |
-| `$(text: {kbid})` | Section exists, type is `text/markdown` |
-| `$(document: {docid})` | Document manifest exists |
-| `$(link: {kbid})` | Section exists (any type) |
-| `$(doc-link: {docid})` | Document manifest exists |
-
-Additionally, integrity checks detect **circular
-references** in `$(text:)` and `$(document:)` marker
-chains using the visited-set algorithm described
-above.
-
-See [Database](database.md) for the full integrity
-and maintenance specification.
-
-## Examples
-
-### Text Section with All Marker Types
-
-```
-## API Overview
-
-This module provides the core search interface.
-
-$(image: 7fa93b2e4c8d1076a93b2e4c8d)
-
-The search function accepts a query string and
-returns ranked results. Here is the implementation:
-
-$(code: c4e8a91d3f7b20564e8a91d3f7)
-
-For additional context on the ranking algorithm,
-see the following section:
-
-$(text: a1b2c3d4e5f6a1b2c3d4e5f6a1)
-
-The full API reference is available as a separate
-document:
-
-$(document: api-reference-v2)
-
-Related sections:
-- Ranking details: $(link: d5e6f7a8b9c0d5e6f7a8b9c0d5)
-- See also: $(doc-link: getting-started-guide)
-```
-
-This example shows all six marker types in a single
-text section. At render time:
-- The `$(image:)` marker expands to an inline image.
-- The `$(code:)` marker expands to a fenced code
-  block with the appropriate language tag.
-- The `$(text:)` marker expands to the referenced
-  section's full markdown content.
-- The `$(document:)` marker expands to all sections
-  in the referenced document.
-- The `$(link:)` marker renders as a section
-  cross-reference.
-- The `$(doc-link:)` marker renders as a document
-  cross-reference.