@dikolab/kbdb 0.2.1 → 0.3.0

package/docs/details/cli.md

@@ -30,12 +30,14 @@ kbdb <command> <subcommand> [options] ...[parameters]
 | Knowledge Base | `learn` | Import content from files or stdin |
 | Knowledge Base | `unlearn` | Remove sections by kbid |
 | Knowledge Base | `search` | Query the knowledge base |
+| Knowledge Base | `recall` | Recall full context for sections |
 | Knowledge Base | `content` | Compose rendered content from ids |
 | Database | `db init` | Initialise a new `.kbdb` directory |
 | Database | `db migrate` | Migrate database to current format |
 | Maintenance | `status` | Show database status |
 | Maintenance | `check` | Verify knowledge base integrity |
 | Maintenance | `gc` | Garbage collect unreferenced sections |
+| Maintenance | `export` | Export database snapshot |
 | Maintenance | `rebuild` | Reconstruct all index files |
 | Skills | `skill learn` | Create a new skill definition |
 | Skills | `skill list` | List all skills |
@@ -76,6 +78,7 @@ These options apply to the `search` command.
 |--------|-------|---------|--------|-------------|
 | `--mode <mode>` | `-m` | `sections` | `sections`, `documents`, `stats` | Result granularity |
 | `--limit <n>` | `-l` | `20` | positive integer | Maximum results returned |
+| `--offset <n>` | | `0` | non-negative integer | Zero-based result offset for pagination |
 | `--format <fmt>` | `-f` | `json` | `json`, `text`, `mcp` | Output format |
 | `--content` | `-c` | off | boolean flag | Hydrate results with section content |
 
@@ -115,6 +118,8 @@ or `-` for stdin.
 | `--title <title>` | Section title (applies to all sections) |
 | `--description <d>` | Section description |
 | `--docid <docid>` | Attach sections to this document ID |
+| `--tags <t,...>` | Comma-separated tags for scoping and filtering |
+| `--replace` | Enable supersession by source key |
 
 **Output:** JSON array of `LearnResult` objects:
 
@@ -124,7 +129,10 @@ or `-` for stdin.
       "kbid": "abcdef1234567890abcdef1234",
       "path": "/absolute/path/to/file.md",
       "size": 1234,
-      "skipped": false
+      "skipped": false,
+      "superseded": null,
+      "warnings": [],
+      "near_duplicates": []
    }
 ]
 ```
@@ -162,26 +170,77 @@ are joined into a single query string.
 **Output (json):**
 
 ```json
-[
-   {
-      "kbid": "abcdef1234567890abcdef1234",
-      "score": 8.7432,
-      "preview": "",
-      "matches": [
-         { "term": "machine", "frequency": 3 },
-         { "term": "learning", "frequency": 1 }
-      ]
-   }
-]
+{
+   "items": [
+      {
+         "kbid": "abcdef1234567890abcdef1234",
+         "heading": "Machine Learning Overview",
+         "type": "text/markdown",
+         "docids": ["doc-5678"],
+         "snippet": "...applies machine learning...",
+         "matched_terms": ["machine", "learning"],
+         "matched_fields": ["heading", "body"],
+         "confidence": 1.0,
+         "created_at": "2026-06-15T10:00:00Z"
+      }
+   ],
+   "total": 42,
+   "offset": 0,
+   "limit": 20,
+   "has_more": true,
+   "relaxed": false
+}
 ```
 
 **Output (text):**
 
 ```
-1. [abcdef1234567890abcdef1234] (score: 8.7432)
+1. [abcdef1234567890abcdef1234] (confidence: 1.00)
+   Machine Learning Overview
+   ...applies machine learning...
+   matches: machine, learning (heading, body)
+```
+
+### `recall` -- Recall Context
 
-   matches: machine, learning
 ```
+kbdb recall [options] <kbid>...
+```
+
+Retrieve full content and progressive context for
+one or more sections. Use after `search` to read
+relevant results.
+
+**Options:**
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `--depth <n>` | `0` | Context expansion level (0--3) |
+
+**Expansion levels:**
+
+| Depth | Returns |
+|-------|---------|
+| 0 | Full section content with heading, type, docids |
+| 1 | Depth 0 + parent documents + back-references |
+| 2 | Depth 1 + sibling sections + forward references |
+| 3 | Depth 2 + full text of referenced sections |
+
+**Output:**
+
+```json
+{
+   "kbid": "abcdef1234567890abcdef1234",
+   "heading": "Architecture Overview",
+   "type": "text/markdown",
+   "content": "Full section text...",
+   "docids": ["doc-5678"]
+}
+```
+
+When multiple kbids are provided, output is a JSON
+array. The `--depth` parameter applies to all
+sections.
 
 ### `content` -- Compose Rendered Content
 
@@ -229,6 +288,11 @@ document references, and type consistency.
          "type": "orphan_section",
          "entity": "x9y8z7w6v5u4t3s2r1q0p9o8n7",
          "detail": "Section not referenced by any document"
+      },
+      {
+         "type": "divergent_content",
+         "entity": "a1b2c3d4e5f67890",
+         "detail": "Sections sharing source key have different content"
       }
    ]
 }
@@ -251,7 +315,8 @@ document manifest references).
 ```json
 {
    "removed_sections": 3,
-   "removed_bytes": 15360
+   "removed_bytes": 15360,
+   "stale_sources": 1
 }
 ```
 
@@ -273,6 +338,28 @@ and document manifests on disk.
 }
 ```
 
+### `export` -- Export Database
+
+```
+kbdb export [options] [<path>]
+```
+
+Export the knowledge base as a self-contained
+directory snapshot for backup or versioning. Copies
+sections, documents, and catalog (no indexes -- they
+are rebuildable via `rebuild`). Defaults to
+`./kbdb-export/`.
+
+**Output:**
+
+```json
+{
+   "path": "/absolute/path/to/export",
+   "sections_exported": 39,
+   "documents_exported": 5
+}
+```
+
 ### `db init` -- Initialise Database
 
 ```

package/docs/details/library-api.md

@@ -69,7 +69,9 @@ Queries the knowledge base and returns ranked
 results.
 
 ```typescript
-search(params: SearchParams): Promise<SearchResult[]>
+search(
+   params: SearchParams,
+): Promise<PagedSearchResult>
 ```
 
 | Param | Type | Description |
@@ -188,6 +190,36 @@ listDocumentsByType(
 
 ---
 
+### Recall
+
+#### `recall(params)`
+
+Retrieves full content and progressive context for
+one or more sections. Use after `search()` to read
+relevant results.
+
+```typescript
+recall(
+   params: RecallParams,
+): Promise<RecallResult | RecallResult[]>
+```
+
+| Param | Type | Description |
+|-------|------|-------------|
+| `params.kbid` | `string?` | Single section identifier. |
+| `params.kbids` | `string[]?` | Batch section identifiers. |
+| `params.depth` | `number?` | Expansion depth (0--3). Default `0`. |
+
+At least one of `kbid` or `kbids` is required.
+Returns a single `RecallResult` for `kbid`, or an
+array for `kbids`. Depth controls progressive
+context expansion: 0 = section content, 1 = adds
+parent documents and back-references, 2 = adds
+siblings and forward references, 3 = adds full text
+of referenced sections.
+
+---
+
 ### Content
 
 #### `content(ids)`
@@ -339,6 +371,21 @@ interface SearchParams {
    readonly query: string;
    readonly mode?: 'sections' | 'documents' | 'stats';
    readonly limit?: number;
+   readonly offset?: number;
+   readonly content?: boolean;
+}
+```
+
+### `PagedSearchResult`
+
+```typescript
+interface PagedSearchResult {
+   readonly items: readonly SearchResult[];
+   readonly total: number;
+   readonly offset: number;
+   readonly limit: number;
+   readonly has_more: boolean;
+   readonly relaxed: boolean;
 }
 ```
 
@@ -347,11 +394,13 @@ interface SearchParams {
 ```typescript
 interface SearchResult {
    readonly kbid: string;
-   readonly score: number;
-   readonly matches: readonly {
-      readonly term: string;
-      readonly frequency: number;
-   }[];
+   readonly heading: string | null;
+   readonly type: string;
+   readonly docids: readonly string[];
+   readonly snippet: string;
+   readonly matched_terms: readonly string[];
+   readonly matched_fields: readonly string[];
+   readonly created_at: string;
 }
 ```
 

package/docs/details/mcp-server.md

@@ -188,7 +188,7 @@ own worker client connection.
 
 ## Tools Reference
 
-The MCP server exposes 17 tools grouped by
+The MCP server exposes 19 tools grouped by
 function. All tools communicate with the worker
 daemon via JSON-RPC 2.0 over IPC.
 
@@ -196,15 +196,17 @@ daemon via JSON-RPC 2.0 over IPC.
 
 | Tool | Parameters | Description | Read-only | Destructive |
 |------|-----------|-------------|-----------|-------------|
-| `search` | `query` (string, required), `mode` (string), `limit` (number) | Query the knowledge base by keywords. Returns ranked results with kbid, score, and matched terms. | yes | no |
-| `learn` | `content` (string, required), `type` (string), `title` (string), `description` (string), `docid` (string) | Import content into the knowledge base. Extracts keywords and builds indexes. | no | no |
+| `search` | `query` (string, required), `mode` (string), `limit` (number), `offset` (number), `content` (boolean) | Query the knowledge base by keywords. Returns ranked results with heading, snippet, and matched terms. Use multiple short keyword queries for best coverage. | yes | no |
+| `learn` | `content` (string, required), `type` (string), `title` (string), `description` (string), `docid` (string), `tags` (string[]), `replace` (boolean) | Import content into the knowledge base. Use for durable facts, decisions, and corrections. Use `replace` when updating existing knowledge. Attach `tags` for scoping. | no | no |
 | `unlearn` | `kbid` (string, required) | Remove a section by kb-id. Cascades deletion to index entries and ranking data. | no | yes |
 | `retrieve` | `kbid` (string), `docid` (string) | Fetch a specific section or document by identifier. At least one of `kbid` or `docid` required. | yes | no |
+| `recall` | `kbid` (string), `kbids` (string[]), `depth` (number) | Retrieve full content and progressive context for sections. Use after `search` to read relevant results. Depth 0-3 controls context expansion. | yes | no |
 | `content` | `ids` (string[], required) | Compose a rendered Markdown document from one or more kbid or docid identifiers. | yes | no |
 | `status` | (none) | Return database status and cache statistics. | yes | no |
 | `check` | (none) | Verify knowledge base integrity. Reports checksum mismatches, orphan sections, broken references. | yes | no |
 | `gc` | (none) | Remove unreferenced sections (no document manifest references). | no | yes |
 | `rebuild` | (none) | Reconstruct all index files from section files and document manifests on disk. | no | no |
+| `export` | `path` (string) | Export the knowledge base as a self-contained directory snapshot for backup or versioning. Copies sections, documents, and catalog (no indexes). | no | no |
 
 ### Skills Tools
 
@@ -236,7 +238,9 @@ daemon via JSON-RPC 2.0 over IPC.
 | `status` | yes |
 | `check` | yes |
 | `gc` | no |
+| `recall` | yes |
 | `rebuild` | no |
+| `export` | yes |
 | `skill-learn` | yes |
 | `skill-list` | yes |
 | `skill-get` | yes |
@@ -336,7 +340,7 @@ Protocol version **2025-11-25** and advertises the
 following capabilities:
 
 - **tools** -- `tools/list` and `tools/call` for
-  all 17 tools.
+  all 19 tools.
 - **resources** -- `resources/list`,
   `resources/read`, and
   `resources/templates/list` for URI-based content

package/docs/details/search-and-ranking.md

@@ -43,7 +43,7 @@ sections containing both "auth" and "flow".
 ## Tokenization Pipeline
 
 Both indexing and search queries pass through the
-same eight-stage pipeline. Using identical processing
+same nine-stage pipeline. Using identical processing
 for both sides guarantees that stored terms match
 query terms.
 
@@ -69,10 +69,13 @@ query terms.
   6. Stop word removal
     |
     v
-  7. Stemming
+  7. Synonym expansion
     |
     v
-  8. N-gram extraction
+  8. Stemming
+    |
+    v
+  9. N-gram extraction
     |
     v
   Normalized tokens
@@ -111,13 +114,24 @@ query terms.
    no discriminative value (`the`, `is`, `and`,
    etc.). The stop word list is configurable.
 
-7. **Stemming** -- Reduce words to root forms using
+7. **Synonym expansion** -- Look up each token in a
+   built-in bidirectional synonym map and insert
+   synonym tokens alongside the original. Bridges
+   vocabulary gaps: a document using "sign-in" becomes
+   findable via "login". The map covers common
+   IT/development vocabulary (~30-50 entries:
+   login/sign-in, auth/authentication,
+   remove/delete, config/configuration, etc.).
+   Applied symmetrically during both indexing and
+   querying.
+
+8. **Stemming** -- Reduce words to root forms using
    the Snowball/Porter2 stemmer. `running` and `runs`
    both stem to `run`. Both the stemmed form and the
    original are stored so exact-match queries still
    work.
 
-8. **N-gram extraction** -- Generate bigrams and
+9. **N-gram extraction** -- Generate bigrams and
    trigrams for phrase indexing and multi-word term
    matching. Supports phrase queries without requiring
    a separate phrase index.
@@ -291,6 +305,35 @@ and freshness are additive boosts with configurable
 weights. Results are sorted by `final_score`
 descending; ties broken by kbid lexicographic order.
 
+### Score Normalization
+
+Raw BM25F scores are normalized to a 0.0--1.0
+`confidence` value within each result set. Each
+score is divided by the maximum score in the result
+set, so the top result always has confidence 1.0.
+This lets agents distinguish strong matches from
+weak tail results without interpreting raw score
+magnitudes.
+
+Confidence is exposed in CLI output but omitted from
+MCP results (ordering is the signal for agents).
+
+## Relaxed-Match Fallback
+
+When a multi-term AND query returns zero results,
+the engine automatically retries with OR semantics
+-- any term matches instead of all required. This
+prevents silent misses when the query is too
+specific for the corpus.
+
+The fallback is transparent: no special syntax
+needed. Results from relaxed matching are flagged
+with `relaxed: true` in the response envelope so
+the caller knows the match is weaker. Single-term
+queries are unaffected (AND and OR are equivalent
+for one term). Phrase queries and exclusions are
+preserved in both modes.
+
 ## Keyword Extraction
 
 When content is added to the knowledge base, the
@@ -347,6 +390,7 @@ indexes/
 +-- meta.toml        human-readable metadata
 +-- terms.idx        inverted index (B+ tree)
 +-- positions.idx    positional index (B+ tree)
++-- references.idx   reference graph (B+ tree)
 +-- corpus.dat       corpus statistics (binary)
 ```
 
@@ -378,6 +422,20 @@ zero-based token offsets. Supports three functions:
 - **Zone scoring** -- Determine whether a term falls
   in the heading zone (first 20% of the section).
 
+### `references.idx` -- Reference Graph Index
+
+B+ tree mapping section kbids to their content
+marker cross-references. Stores bidirectional
+edges: for each `$(text:)`, `$(code:)`, `$(image:)`,
+`$(link:)`, `$(document:)`, and `$(doc-link:)`
+marker, both a forward entry (source -> target) and
+a back entry (target -> source) are recorded. This
+enables the `recall` command's progressive context
+expansion -- depth 1+ retrieves back-references
+(sections that point to this one) and depth 2+
+retrieves forward references (sections this one
+points to).
+
 ### `corpus.dat` -- Corpus Statistics
 
 Binary file storing corpus-wide aggregates:

package/docs/details/storage.md

@@ -24,6 +24,7 @@ version-controlled.
 |   +-- meta.toml
 |   +-- terms.idx
 |   +-- positions.idx
+|   +-- references.idx
 |   +-- corpus.dat
 +-- worker.toml
 +-- .gitignore
@@ -37,6 +38,7 @@ version-controlled.
 | `indexes/meta.toml`           | TOML   | Index metadata: rebuild timestamp, term count, statistics summary |
 | `indexes/terms.idx`           | Binary | Inverted index (B+ tree): maps terms to posting lists with per-field term frequencies |
 | `indexes/positions.idx`       | Binary | Positional index (B+ tree): maps (term, kbid) to token offset arrays |
+| `indexes/references.idx`     | Binary | Reference graph index (B+ tree): bidirectional content marker cross-references between sections |
 | `indexes/corpus.dat`          | Binary | Corpus statistics: section count, average lengths per field, per-term document frequencies |
 | `worker.toml`                 | TOML   | Cache daemon configuration    |
 | `.gitignore`                  | Text   | Excludes `indexes/` from version control |