@dikolab/kbdb 0.4.3 → 0.6.0

package/docs/details/cli.md

@@ -1,777 +0,0 @@
-# CLI Reference
-
-The `kbdb` command manages a file-based knowledge
-base database. It imports documents, searches by
-keyword, and manages skills and agents -- all from
-the command line.
-
-## Installation
-
-```sh
-# Node.js
-npm install -g @dikolab/kbdb
-
-# Deno
-deno install -Agf jsr:@dikolab/kbdb/cli
-```
-
-## Synopsis
-
-```
-kbdb [options]
-kbdb <command> [options] ...[parameters]
-kbdb <command> <subcommand> [options] ...[parameters]
-kbdb --version
-kbdb -v <command>          (verbose: warnings)
-kbdb -vv <command>         (verbose: info)
-kbdb -vvv <command>        (verbose: debug)
-```
-
-## Command Groups
-
-| Group | Command | Description |
-|-------|---------|-------------|
-| Knowledge Base | `learn` | Import content from files or stdin |
-| Knowledge Base | `unlearn` | Remove sections by kbid |
-| Knowledge Base | `search` | Query the knowledge base |
-| Knowledge Base | `recall` | Recall full context for sections |
-| Knowledge Base | `content` | Compose rendered content from ids |
-| Database | `db init` | Initialise a new `.kbdb` directory |
-| Database | `db migrate` | Migrate database to current format |
-| Maintenance | `status` | Show database status |
-| Maintenance | `check` | Verify knowledge base integrity |
-| Maintenance | `gc` | Garbage collect unreferenced sections |
-| Maintenance | `export` | Export database snapshot |
-| Maintenance | `rebuild` | Reconstruct all index files |
-| Skills | `skill learn` | Create a new skill definition |
-| Skills | `skill list` | List all skills |
-| Skills | `skill get` | Retrieve a skill by name or kbid |
-| Skills | `skill delete` | Remove a skill |
-| Agents | `agent create` | Create an agent with skill refs |
-| Agents | `agent list` | List all agents |
-| Agents | `agent get` | Retrieve an agent by docid |
-| Agents | `agent delete` | Remove an agent |
-| Server | `mcp` | Start MCP server mode (stdio) |
-| Worker | `worker stop` | Stop the background worker daemon |
-| Worker | `worker restart` | Restart the background worker daemon |
-
-### Reading Data
-
-Three commands retrieve stored content at different
-granularities:
-
-| Command | Purpose |
-|---------|---------|
-| `search` | Find sections by query; returns ranked snippets |
-| `recall` | Expand full context around known kbids (post-search deep-dive) |
-| `content` | Compose rendered Markdown from kbid or docid identifiers |
-
-**When to use which:** Start with `search` to find
-relevant sections by keyword. Use `recall` to expand
-a search result into its full context -- parent
-documents, sibling sections, and references. Use
-`content` when you already know the exact kbid or
-docid and want rendered Markdown output.
-
-## Global Options
-
-| Option | Short | Default | Description |
-|--------|-------|---------|-------------|
-| `--help` | `-h` | | Print help text and exit |
-| `--version` | | | Print version and exit |
-| `--verbose` | `-v` | | Set log level to WARNING (errors + warnings) |
-| `--verbose-info` | `-vv` | | Set log level to INFO |
-| `--verbose-debug` | `-vvv` | | Set log level to DEBUG (all messages) |
-| `--db <path>` | | cwd | Database directory path (optional) |
-| `--format <fmt>` | `-f` | `json` (text when interactive) | Output format (`json`, `text`, `mcp`) |
-| `--timeout-ms <ms>` | | `30000` | Per-request daemon timeout in milliseconds |
-| `--retry <n>` | | `1` | Max retries with daemon respawn on timeout |
-| `--non-interactive` | | | Suppress prompts; use defaults |
-
-### `--db <path>`
-
-Set the path to the directory containing the
-`.kbdb` database folder. The actual database is
-always at `${path}/.kbdb`. This flag is optional.
-
-Resolution order:
-
-1. `--db` flag (highest priority)
-2. `KBDB_DB_DIR` environment variable
-3. Current working directory (CLI mode only)
-
-- **CLI mode**: resolves using the order above.
-- **MCP mode**: falls back to the `KBDB_DB_DIR`
-  environment variable. If neither `--db` nor
-  `KBDB_DB_DIR` is set, the server exits with an
-  error.
-
-## Search Options
-
-These options apply to the `search` command.
-
-| Option | Short | Default | Values | Description |
-|--------|-------|---------|--------|-------------|
-| `--mode <mode>` | `-m` | `sections` | `sections`, `documents`, `stats` | Result granularity |
-| `--limit <n>` | `-l` | `20` | positive integer | Maximum results returned |
-| `--offset <n>` | | `0` | non-negative integer | Zero-based result offset for pagination |
-| `--format <fmt>` | `-f` | `json` | `json`, `text`, `mcp` | Output format |
-| `--algo <algo>` | | `lexical` | `lexical`, `vector`, `hybrid` | Search algorithm |
-| `--content` | `-c` | off | boolean flag | Hydrate results with section content |
-
-### Search Algorithms
-
-| Algorithm | Description |
-|-----------|-------------|
-| `lexical` | Keyword matching with stemming and synonyms |
-| `vector` | Similarity search using embeddings |
-| `hybrid` | Combines keyword and similarity results for best coverage |
-
-### Configuring Hybrid Search
-
-Hybrid and vector search use a built-in TF-IDF
-embedding provider by default -- no configuration
-needed. For higher quality embeddings, configure a
-provider via `worker.toml`:
-
-| Provider | Description |
-|----------|-------------|
-| *(none)* | Built-in TF-IDF (256 dimensions, works offline) |
-| `none` | Explicitly disabled; hybrid and vector return an error |
-| `onnx` | Local ONNX model (no network required) |
-| `api` | Remote embedding API (OpenAI-compatible endpoint) |
-
-### Search Modes
-
-| Mode | Description |
-|------|-------------|
-| `sections` | Individual matching sections (default) |
-| `documents` | Complete documents containing matches |
-| `stats` | Index statistics and result counts |
-
-### Output Formats
-
-| Format | Description |
-|--------|-------------|
-| `json` | Pretty-printed JSON array |
-| `text` | Numbered human-readable list with scores |
-| `mcp` | JSON envelope for MCP protocol consumers |
-
-## Commands
-
-### `learn` -- Import Documents
-
-```
-kbdb learn [options] <path | ->...
-```
-
-Import and index documents into the knowledge base.
-Accepts file paths, directory paths (scanned
-recursively for `.md`, `.txt`, `.rst`, `.adoc`),
-or `-` for stdin.
-
-**Options:**
-
-| Option | Description |
-|--------|-------------|
-| `--title <title>` | Section title (applies to all sections) |
-| `--description <d>` | Section description |
-| `--docid <docid>` | Attach sections to this document ID |
-| `--tags <t,...>` | Comma-separated tags for scoping and filtering |
-| `--replace` | Enable supersession by source key |
-
-**Output:** JSON array of `LearnResult` objects:
-
-```json
-[
-   {
-      "kbid": "abcdef1234567890abcdef1234",
-      "path": "/absolute/path/to/file.md",
-      "size": 1234,
-      "skipped": false,
-      "superseded": null,
-      "warnings": [],
-      "near_duplicates": []
-   }
-]
-```
-
-### `unlearn` -- Remove Sections
-
-```
-kbdb unlearn [options] <kbid>...
-```
-
-Remove sections by their 26-character kb-id.
-Cascades deletion to associated index entries
-and ranking data.
-
-**Output:** JSON array of `UnlearnResult` objects:
-
-```json
-[
-   {
-      "kbid": "abcdef1234567890abcdef1234",
-      "removed": true
-   }
-]
-```
-
-> kbid arguments accept unique prefixes -- kbdb
-> resolves them to full 26-character identifiers.
-
-### `search` -- Query the Knowledge Base
-
-```
-kbdb search [options] <query>...
-```
-
-Search by keywords or phrases. All query terms
-are joined into a single query string.
-
-**Output (json):**
-
-```json
-{
-   "items": [
-      {
-         "kbid": "abcdef1234567890abcdef1234",
-         "heading": "Machine Learning Overview",
-         "type": "text/markdown",
-         "docids": ["doc-5678"],
-         "snippet": "...applies machine learning...",
-         "matched_terms": ["machine", "learning"],
-         "matched_fields": ["heading", "body"],
-         "confidence": 1.0,
-         "created_at": "2026-06-15T10:00:00Z"
-      }
-   ],
-   "total": 42,
-   "offset": 0,
-   "limit": 20,
-   "has_more": true,
-   "relaxed": false
-}
-```
-
-**Output (text):**
-
-```
-1. [abcdef1234567890abcdef1234] (confidence: 1.00)
-   Machine Learning Overview
-   ...applies machine learning...
-   matches: machine, learning (heading, body)
-```
-
-### `recall` -- Recall Context
-
-```
-kbdb recall [options] <kbid>...
-```
-
-Retrieve full content and progressive context for
-one or more sections. Use after `search` to read
-relevant results.
-
-> Identifiers can be shortened to a unique prefix
-> (like git short hashes). kbdb resolves the prefix
-> to the full 26-character kbid.
-
-**Options:**
-
-| Option | Default | Description |
-|--------|---------|-------------|
-| `--depth <n>` | `0` | Context expansion level (0--3) |
-
-**Expansion levels:**
-
-| Depth | Returns |
-|-------|---------|
-| 0 | Full section content with heading, type, docids |
-| 1 | Depth 0 + parent documents + back-references |
-| 2 | Depth 1 + sibling sections + forward references |
-| 3 | Depth 2 + full text of referenced sections |
-
-**Output:**
-
-```json
-{
-   "kbid": "abcdef1234567890abcdef1234",
-   "heading": "Architecture Overview",
-   "type": "text/markdown",
-   "content": "Full section text...",
-   "docids": ["doc-5678"]
-}
-```
-
-When multiple kbids are provided, output is a JSON
-array. The `--depth` parameter applies to all
-sections.
-
-### `content` -- Compose Rendered Content
-
-```
-kbdb content [options] <id>...
-```
-
-Compose a rendered Markdown document from one or
-more kbid or docid identifiers.
-
-> Identifiers can be shortened to a unique prefix
-> (like git short hashes). kbdb resolves the prefix
-> to the full 26-character kbid.
-
-**Output:**
-
-```json
-{
-   "type": "text/markdown",
-   "data": "# Rendered Markdown\n\nContent here...",
-   "total": 42
-}
-```
-
-### `status` -- Database Status
-
-```
-kbdb status [options]
-```
-
-Show database status and cache statistics.
-
-### `check` -- Integrity Check
-
-```
-kbdb check [options]
-```
-
-Verify database integrity. Checks checksums,
-document references, and type consistency.
-
-**Output:**
-
-```json
-{
-   "ok": false,
-   "errors": [
-      {
-         "type": "orphan_section",
-         "entity": "x9y8z7w6v5u4t3s2r1q0p9o8n7",
-         "detail": "Section not referenced by any document"
-      },
-      {
-         "type": "divergent_content",
-         "entity": "a1b2c3d4e5f67890",
-         "detail": "Sections sharing source key have different content"
-      }
-   ]
-}
-```
-
-Exit code is `0` when `ok` is `true`, `1`
-when errors are found.
-
-### `gc` -- Garbage Collection
-
-```
-kbdb gc [options]
-```
-
-Remove unreferenced sections (sections with no
-document manifest references).
-
-**Output:**
-
-```json
-{
-   "removed_sections": 3,
-   "removed_bytes": 15360,
-   "stale_sources": 1
-}
-```
-
-### `rebuild` -- Rebuild Indexes
-
-```
-kbdb rebuild [options]
-```
-
-Reconstruct all index files from section files
-and document manifests on disk.
-
-**Output:**
-
-```json
-{
-   "sections_reindexed": 39,
-   "terms_indexed": 1500
-}
-```
-
-### `export` -- Export Database
-
-```
-kbdb export [options] [<path>]
-```
-
-Export the knowledge base as a self-contained
-directory snapshot for backup or versioning. Copies
-sections, documents, and catalog (no indexes -- they
-are rebuildable via `rebuild`). Defaults to
-`./kbdb-export/`.
-
-**Output:**
-
-```json
-{
-   "path": "/absolute/path/to/export",
-   "sections_exported": 39,
-   "documents_exported": 5
-}
-```
-
-### `db init` -- Initialise Database
-
-```
-kbdb db init [options]
-```
-
-Create a blank `.kbdb` database directory at the
-target path. No daemon required.
-
-**Output:**
-
-```json
-{ "path": "/absolute/path/.kbdb", "created": true }
-```
-
-### `db migrate` -- Migrate Database
-
-```
-kbdb db migrate [options]
-```
-
-Upgrade a `.kbdb` database from an older format
-version to the current version. Prints "already
-up to date" when no migration is needed.
-
-### `skill learn` -- Create a Skill
-
-```
-kbdb skill learn [options] --name <n> --body <file|->
-```
-
-Read a skill body from a file or stdin, extract
-`{{argName}}` placeholders, and store the skill
-as an indexed section.
-
-**Options:**
-
-| Option | Required | Description |
-|--------|----------|-------------|
-| `--name <n>` | yes | Skill name (kebab-case) |
-| `--description <d>` | no | Human-readable description |
-| `--body <file\|->` | yes | Template body file or stdin |
-
-**Output:**
-
-```json
-{
-   "kbid": "abcdef1234567890abcdef1234",
-   "name": "summarize-document",
-   "arguments": ["docid", "max_points"]
-}
-```
-
-### `skill list` -- List Skills
-
-```
-kbdb skill list [options]
-```
-
-**Output:** JSON array of skill summaries:
-
-```json
-[
-   {
-      "kbid": "abcdef1234567890abcdef1234",
-      "name": "summarize-document",
-      "description": "Summarize a document...",
-      "argumentCount": 2
-   }
-]
-```
-
-### `skill get` -- Retrieve a Skill
-
-```
-kbdb skill get [options] <name-or-kbid>
-```
-
-Retrieve a skill by name or 26-character kbid.
-
-**Output:** Full `SkillDefinition` JSON including
-`kbid`, `name`, `description`, `arguments`, and
-`body`.
-
-### `skill delete` -- Remove a Skill
-
-```
-kbdb skill delete [options] <kbid>
-```
-
-Remove a skill section by kbid. The skill body
-and index entries are permanently deleted.
-
-**Output:**
-
-```json
-{
-   "kbid": "abcdef1234567890abcdef1234",
-   "removed": true
-}
-```
-
-### `agent create` -- Create an Agent
-
-```
-kbdb agent create [options] --name <n>
-   --persona <file|-> --skills <kbid,...>
-```
-
-Create an agent document with a persona and an
-ordered list of skill references.
-
-**Options:**
-
-| Option | Required | Description |
-|--------|----------|-------------|
-| `--name <n>` | yes | Agent name (kebab-case) |
-| `--description <d>` | no | Human-readable description |
-| `--persona <file\|->` | yes | Persona file or stdin |
-| `--skills <kbid,...>` | yes | Comma-separated skill kbids |
-
-**Output:**
-
-```json
-{
-   "docid": "agent-code-reviewer",
-   "name": "code-reviewer",
-   "skills": [
-      "abcdef1234567890abcdef1234",
-      "fedcba9876543210fedcba9876"
-   ]
-}
-```
-
-### `agent list` -- List Agents
-
-```
-kbdb agent list [options]
-```
-
-**Output:** JSON array of agent summaries:
-
-```json
-[
-   {
-      "docid": "agent-code-reviewer",
-      "name": "code-reviewer",
-      "description": "Reviews code for best practices",
-      "skillCount": 2
-   }
-]
-```
-
-### `agent get` -- Retrieve an Agent
-
-```
-kbdb agent get [options] <docid>
-```
-
-Retrieve an agent with resolved skill definitions.
-Dangling skill references (deleted skills) appear
-with `name: null`.
-
-**Output:** Full `AgentDefinition` JSON including
-`docid`, `name`, `description`, `persona`, and
-`skills` array.
-
-### `agent delete` -- Remove an Agent
-
-```
-kbdb agent delete [options] <docid>
-```
-
-Remove an agent document manifest. Referenced
-skill sections are not deleted.
-
-**Output:**
-
-```json
-{
-   "docid": "agent-code-reviewer",
-   "removed": true
-}
-```
-
-### `mcp` -- Start MCP Server
-
-```
-kbdb mcp [options]
-```
-
-Start a long-lived MCP server on stdio. Requires
-`--db <path>` or the `KBDB_DB_DIR` environment
-variable. See the
-[MCP Server Guide](mcp-server.md) for setup and
-tool reference.
-
-## Example Workflows
-
-### Import Documents and Search
-
-```sh
-# Initialise a new database
-kbdb db init --db ~/my-project
-
-# Import a single file
-kbdb learn README.md --db ~/my-project
-
-# Import an entire directory of docs
-kbdb learn ./docs/ --db ~/my-project
-
-# Import from stdin
-cat notes.txt | kbdb learn - --db ~/my-project
-
-# Search for content
-kbdb search "API endpoint" --db ~/my-project
-
-# Search with content preview and text output
-kbdb search "authentication" --content \
-   --format text --limit 5
-
-# Search in document mode
-kbdb search "deployment" --mode documents
-
-# Remove a section by kbid
-kbdb unlearn abcdef1234567890abcdef1234
-
-# Compose rendered content from multiple ids
-kbdb content id1 id2 id3
-```
-
-### Create and Use Skills
-
-```sh
-# Write a skill template with placeholders
-cat > summarize.txt << 'EOF'
-Summarize the document {{docid}}.
-Extract at most {{max_points}} key points.
-Return as a bulleted markdown list.
-EOF
-
-# Create the skill
-kbdb skill learn --name summarize-document \
-   --description "Summarize a document" \
-   --body summarize.txt
-
-# List all skills
-kbdb skill list
-
-# Retrieve a skill by name
-kbdb skill get summarize-document
-
-# Retrieve a skill by kbid
-kbdb skill get abcdef1234567890abcdef1234
-
-# Delete a skill
-kbdb skill delete abcdef1234567890abcdef1234
-```
-
-### Create and Manage Agents
-
-```sh
-# Write a persona file
-cat > reviewer-persona.txt << 'EOF'
-You are an expert code reviewer. You focus on
-correctness, readability, and maintainability.
-EOF
-
-# Create an agent referencing existing skills
-kbdb agent create --name code-reviewer \
-   --description "Reviews code for best practices" \
-   --persona reviewer-persona.txt \
-   --skills abc123,def456
-
-# Create an agent with persona from stdin
-cat persona.txt | kbdb agent create \
-   --name my-agent --persona - \
-   --skills abc123
-
-# List all agents
-kbdb agent list
-
-# Retrieve an agent with resolved skills
-kbdb agent get agent-code-reviewer
-
-# Delete an agent (skills are preserved)
-kbdb agent delete agent-code-reviewer
-```
-
-## Environment Variables
-
-| Variable | Used By | Default | Description |
-|----------|---------|---------|-------------|
-| `KBDB_DB_DIR` | CLI + MCP | cwd (CLI) / none (MCP) | Fallback database directory path when `--db` is not specified. |
-| `KBDB_DAEMON_TIMEOUT` | CLI + MCP | `30000` | Per-request daemon timeout in milliseconds. Overridden by `--timeout-ms`. |
-| `KBDB_RETRY` | CLI + MCP | `1` | Max retries with daemon respawn on timeout. Overridden by `--retry`. |
-| `KBDB_LOG_LEVEL` | CLI + MCP + Daemon | `error` | Log verbosity: `debug`, `info`, `warning`, `error`. Overridden by `-v`/`-vv`/`-vvv`. |
-| `KBDB_NON_INTERACTIVE` | CLI + MCP + Daemon | off | Suppress interactive prompts. Accepts `1` or `true`. Overridden by `--non-interactive`. |
-
-## Exit Codes
-
-| Code | Meaning |
-|------|---------|
-| `0` | Success |
-| `1` | Error (invalid arguments, missing database, WASM error, version mismatch, unknown command, integrity check failure) |
-
-## Database Initialisation
-
-Before any data command (`learn`, `unlearn`,
-`search`, etc.), the CLI checks for an existing
-`.kbdb` directory:
-
-1. **No `.kbdb` found + interactive terminal** --
-   prompts to create one.
-2. **No `.kbdb` found + non-interactive** -- prints
-   error and exits.
-
-The `--non-interactive` flag (or
-`KBDB_NON_INTERACTIVE=1`) forces the non-interactive
-path regardless of terminal type. This is useful in
-scripts, CI pipelines, and automated testing.
-
-3. **`.kbdb` found** -- runs a version gate before
-   proceeding. If migration is needed, prints an
-   error directing the user to run `kbdb db migrate`.
-
-Use `kbdb db init` for explicit, non-interactive
-database creation.
-
-## Design Notes
-
-- The CLI is a thin pass-through. All keyword
-  extraction, indexing, ranking, and database I/O
-  is handled by the worker daemon and its WASM
-  modules.
-- **stdout** contains only command output. No
-  banners, progress bars, or decorative text.
-- **stderr** contains log messages. Default level is
-  ERROR (errors only). Use `-v`/`-vv`/`-vvv` or
-  `KBDB_LOG_LEVEL` to increase verbosity. Log lines
-  are prefixed with per-level icons (✖⚠ℹ▸) and ANSI
-  colors.
-- Works on both Deno and Node.js (>= 20).
-- The `--init` and `--migrate` flags are deprecated.
-  Use `kbdb db init` and `kbdb db migrate` instead.