@dikolab/kbdb 0.1.5 → 0.2.0

package/docs/details/README.md

@@ -0,0 +1,26 @@
+# Detailed Documentation
+
+In-depth guides and reference for @dikolab/kbdb.
+
+## Guides
+
+- [CLI Reference](cli.md) -- Full command reference
+  with examples
+- [MCP Server Setup](mcp-server.md) -- Configure
+  kbdb as an AI agent plugin
+- [Library API](library-api.md) -- Use kbdb from
+  Node.js or Deno code
+
+## Technical Reference
+
+- [Search and Ranking](search-and-ranking.md) --
+  How search works under the hood
+- [Storage Architecture](storage.md) -- File formats,
+  directory layout, and design principles
+
+## See Also
+
+- [Architecture Overview](../overview.md)
+- [Design Goals](../goals/)
+- [Module Reference](../modules/)
+- [Release Notes](../release-notes/)

package/docs/details/cli.md

@@ -0,0 +1,608 @@
+# CLI Reference
+
+The `kbdb` command manages a file-based knowledge
+base database. It imports documents, searches by
+keyword, and manages skills and agents -- all from
+the command line.
+
+## Installation
+
+```sh
+# Node.js
+npm install -g @dikolab/kbdb
+
+# Deno
+deno install
+```
+
+## Synopsis
+
+```
+kbdb [options]
+kbdb <command> [options] ...[parameters]
+kbdb <command> <subcommand> [options] ...[parameters]
+```
+
+## Command Groups
+
+| Group | Command | Description |
+|-------|---------|-------------|
+| Knowledge Base | `learn` | Import content from files or stdin |
+| Knowledge Base | `unlearn` | Remove sections by kbid |
+| Knowledge Base | `search` | Query the knowledge base |
+| Knowledge Base | `content` | Compose rendered content from ids |
+| Database | `db init` | Initialise a new `.kbdb` directory |
+| Database | `db migrate` | Migrate database to current format |
+| Maintenance | `status` | Show database status |
+| Maintenance | `check` | Verify knowledge base integrity |
+| Maintenance | `gc` | Garbage collect unreferenced sections |
+| Maintenance | `rebuild` | Reconstruct all index files |
+| Skills | `skill learn` | Create a new skill definition |
+| Skills | `skill list` | List all skills |
+| Skills | `skill get` | Retrieve a skill by name or kbid |
+| Skills | `skill delete` | Remove a skill |
+| Agents | `agent create` | Create an agent with skill refs |
+| Agents | `agent list` | List all agents |
+| Agents | `agent get` | Retrieve an agent by docid |
+| Agents | `agent delete` | Remove an agent |
+| Server | `mcp` | Start MCP server mode (stdio) |
+
+## Global Options
+
+| Option | Short | Default | Description |
+|--------|-------|---------|-------------|
+| `--help` | `-h` | | Print help text and exit |
+| `--version` | `-v` | | Print version and exit |
+| `--db <path>` | | cwd | Database directory path |
+| `--format <fmt>` | `-f` | `json` | Output format (`json`, `text`, `mcp`) |
+
+### `--db <path>`
+
+Set the path to the directory containing the
+`.kbdb` database folder. The actual database is
+always at `${path}/.kbdb`.
+
+- **CLI mode**: defaults to the current working
+  directory.
+- **MCP mode**: falls back to the `KBDB_DIR`
+  environment variable. If neither `--db` nor
+  `KBDB_DIR` is set, the server exits with an error.
+
+## Search Options
+
+These options apply to the `search` command.
+
+| Option | Short | Default | Values | Description |
+|--------|-------|---------|--------|-------------|
+| `--mode <mode>` | `-m` | `sections` | `sections`, `documents`, `stats` | Result granularity |
+| `--limit <n>` | `-l` | `20` | positive integer | Maximum results returned |
+| `--format <fmt>` | `-f` | `json` | `json`, `text`, `mcp` | Output format |
+| `--content` | `-c` | off | boolean flag | Hydrate results with section content |
+
+### Search Modes
+
+| Mode | Description |
+|------|-------------|
+| `sections` | Individual matching sections (default) |
+| `documents` | Complete documents containing matches |
+| `stats` | Index statistics and result counts |
+
+### Output Formats
+
+| Format | Description |
+|--------|-------------|
+| `json` | Pretty-printed JSON array |
+| `text` | Numbered human-readable list with scores |
+| `mcp` | JSON envelope for MCP protocol consumers |
+
+## Commands
+
+### `learn` -- Import Documents
+
+```
+kbdb learn [options] <path | ->...
+```
+
+Import and index documents into the knowledge base.
+Accepts file paths, directory paths (scanned
+recursively for `.md`, `.txt`, `.rst`, `.adoc`),
+or `-` for stdin.
+
+**Options:**
+
+| Option | Description |
+|--------|-------------|
+| `--title <title>` | Section title (applies to all sections) |
+| `--description <d>` | Section description |
+| `--docid <docid>` | Attach sections to this document ID |
+
+**Output:** JSON array of `LearnResult` objects:
+
+```json
+[
+   {
+      "kbid": "abcdef1234567890abcdef1234",
+      "path": "/absolute/path/to/file.md",
+      "size": 1234,
+      "skipped": false
+   }
+]
+```
+
+### `unlearn` -- Remove Sections
+
+```
+kbdb unlearn [options] <kbid>...
+```
+
+Remove sections by their 26-character kb-id.
+Cascades deletion to associated index entries
+and ranking data.
+
+**Output:** JSON array of `UnlearnResult` objects:
+
+```json
+[
+   {
+      "kbid": "abcdef1234567890abcdef1234",
+      "removed": true
+   }
+]
+```
+
+### `search` -- Query the Knowledge Base
+
+```
+kbdb search [options] <query>...
+```
+
+Search by keywords or phrases. All query terms
+are joined into a single query string.
+
+**Output (json):**
+
+```json
+[
+   {
+      "kbid": "abcdef1234567890abcdef1234",
+      "score": 8.7432,
+      "preview": "",
+      "matches": [
+         { "term": "machine", "frequency": 3 },
+         { "term": "learning", "frequency": 1 }
+      ]
+   }
+]
+```
+
+**Output (text):**
+
+```
+1. [abcdef1234567890abcdef1234] (score: 8.7432)
+
+   matches: machine, learning
+```
+
+### `content` -- Compose Rendered Content
+
+```
+kbdb content [options] <id>...
+```
+
+Compose a rendered Markdown document from one or
+more kbid or docid identifiers.
+
+**Output:**
+
+```json
+{
+   "type": "text/markdown",
+   "data": "# Rendered Markdown\n\nContent here...",
+   "total": 42
+}
+```
+
+### `status` -- Database Status
+
+```
+kbdb status [options]
+```
+
+Show database status and cache statistics.
+
+### `check` -- Integrity Check
+
+```
+kbdb check [options]
+```
+
+Verify database integrity. Checks checksums,
+document references, and type consistency.
+
+**Output:**
+
+```json
+{
+   "ok": false,
+   "errors": [
+      {
+         "type": "orphan_section",
+         "entity": "x9y8z7w6v5u4t3s2r1q0p9o8n7",
+         "detail": "Section not referenced by any document"
+      }
+   ]
+}
+```
+
+Exit code is `0` when `ok` is `true`, `1`
+when errors are found.
+
+### `gc` -- Garbage Collection
+
+```
+kbdb gc [options]
+```
+
+Remove unreferenced sections (sections with no
+document manifest references).
+
+**Output:**
+
+```json
+{
+   "removed_sections": 3,
+   "removed_bytes": 15360
+}
+```
+
+### `rebuild` -- Rebuild Indexes
+
+```
+kbdb rebuild [options]
+```
+
+Reconstruct all index files from section files
+and document manifests on disk.
+
+**Output:**
+
+```json
+{
+   "sections_reindexed": 39,
+   "terms_indexed": 1500
+}
+```
+
+### `db init` -- Initialise Database
+
+```
+kbdb db init [options]
+```
+
+Create a blank `.kbdb` database directory at the
+target path. No daemon required.
+
+**Output:**
+
+```json
+{ "path": "/absolute/path/.kbdb", "created": true }
+```
+
+### `db migrate` -- Migrate Database
+
+```
+kbdb db migrate [options]
+```
+
+Upgrade a `.kbdb` database from an older format
+version to the current version. Prints "already
+up to date" when no migration is needed.
+
+### `skill learn` -- Create a Skill
+
+```
+kbdb skill learn [options] --name <n> --body <file|->
+```
+
+Read a skill body from a file or stdin, extract
+`{{argName}}` placeholders, and store the skill
+as an indexed section.
+
+**Options:**
+
+| Option | Required | Description |
+|--------|----------|-------------|
+| `--name <n>` | yes | Skill name (kebab-case) |
+| `--description <d>` | no | Human-readable description |
+| `--body <file\|->` | yes | Template body file or stdin |
+
+**Output:**
+
+```json
+{
+   "kbid": "abcdef1234567890abcdef1234",
+   "name": "summarize-document",
+   "arguments": ["docid", "max_points"]
+}
+```
+
+### `skill list` -- List Skills
+
+```
+kbdb skill list [options]
+```
+
+**Output:** JSON array of skill summaries:
+
+```json
+[
+   {
+      "kbid": "abcdef1234567890abcdef1234",
+      "name": "summarize-document",
+      "description": "Summarize a document...",
+      "argumentCount": 2
+   }
+]
+```
+
+### `skill get` -- Retrieve a Skill
+
+```
+kbdb skill get [options] <name-or-kbid>
+```
+
+Retrieve a skill by name or 26-character kbid.
+
+**Output:** Full `SkillDefinition` JSON including
+`kbid`, `name`, `description`, `arguments`, and
+`body`.
+
+### `skill delete` -- Remove a Skill
+
+```
+kbdb skill delete [options] <kbid>
+```
+
+Remove a skill section by kbid. The skill body
+and index entries are permanently deleted.
+
+**Output:**
+
+```json
+{
+   "kbid": "abcdef1234567890abcdef1234",
+   "removed": true
+}
+```
+
+### `agent create` -- Create an Agent
+
+```
+kbdb agent create [options] --name <n>
+   --persona <file|-> --skills <kbid,...>
+```
+
+Create an agent document with a persona and an
+ordered list of skill references.
+
+**Options:**
+
+| Option | Required | Description |
+|--------|----------|-------------|
+| `--name <n>` | yes | Agent name (kebab-case) |
+| `--description <d>` | no | Human-readable description |
+| `--persona <file\|->` | yes | Persona file or stdin |
+| `--skills <kbid,...>` | yes | Comma-separated skill kbids |
+
+**Output:**
+
+```json
+{
+   "docid": "agent-code-reviewer",
+   "name": "code-reviewer",
+   "skills": [
+      "abcdef1234567890abcdef1234",
+      "fedcba9876543210fedcba9876"
+   ]
+}
+```
+
+### `agent list` -- List Agents
+
+```
+kbdb agent list [options]
+```
+
+**Output:** JSON array of agent summaries:
+
+```json
+[
+   {
+      "docid": "agent-code-reviewer",
+      "name": "code-reviewer",
+      "description": "Reviews code for best practices",
+      "skillCount": 2
+   }
+]
+```
+
+### `agent get` -- Retrieve an Agent
+
+```
+kbdb agent get [options] <docid>
+```
+
+Retrieve an agent with resolved skill definitions.
+Dangling skill references (deleted skills) appear
+with `name: null`.
+
+**Output:** Full `AgentDefinition` JSON including
+`docid`, `name`, `description`, `persona`, and
+`skills` array.
+
+### `agent delete` -- Remove an Agent
+
+```
+kbdb agent delete [options] <docid>
+```
+
+Remove an agent document manifest. Referenced
+skill sections are not deleted.
+
+**Output:**
+
+```json
+{
+   "docid": "agent-code-reviewer",
+   "removed": true
+}
+```
+
+### `mcp` -- Start MCP Server
+
+```
+kbdb mcp [options]
+```
+
+Start a long-lived MCP server on stdio. Requires
+`--db <path>` or the `KBDB_DIR` environment
+variable. See the
+[MCP Server Guide](mcp-server.md) for setup and
+tool reference.
+
+## Example Workflows
+
+### Import Documents and Search
+
+```sh
+# Initialise a new database
+kbdb db init --db ~/my-project
+
+# Import a single file
+kbdb learn README.md --db ~/my-project
+
+# Import an entire directory of docs
+kbdb learn ./docs/ --db ~/my-project
+
+# Import from stdin
+cat notes.txt | kbdb learn - --db ~/my-project
+
+# Search for content
+kbdb search "API endpoint" --db ~/my-project
+
+# Search with content preview and text output
+kbdb search "authentication" --content \
+   --format text --limit 5
+
+# Search in document mode
+kbdb search "deployment" --mode documents
+
+# Remove a section by kbid
+kbdb unlearn abcdef1234567890abcdef1234
+
+# Compose rendered content from multiple ids
+kbdb content id1 id2 id3
+```
+
+### Create and Use Skills
+
+```sh
+# Write a skill template with placeholders
+cat > summarize.txt << 'EOF'
+Summarize the document {{docid}}.
+Extract at most {{max_points}} key points.
+Return as a bulleted markdown list.
+EOF
+
+# Create the skill
+kbdb skill learn --name summarize-document \
+   --description "Summarize a document" \
+   --body summarize.txt
+
+# List all skills
+kbdb skill list
+
+# Retrieve a skill by name
+kbdb skill get summarize-document
+
+# Retrieve a skill by kbid
+kbdb skill get abcdef1234567890abcdef1234
+
+# Delete a skill
+kbdb skill delete abcdef1234567890abcdef1234
+```
+
+### Create and Manage Agents
+
+```sh
+# Write a persona file
+cat > reviewer-persona.txt << 'EOF'
+You are an expert code reviewer. You focus on
+correctness, readability, and maintainability.
+EOF
+
+# Create an agent referencing existing skills
+kbdb agent create --name code-reviewer \
+   --description "Reviews code for best practices" \
+   --persona reviewer-persona.txt \
+   --skills abc123,def456
+
+# Create an agent with persona from stdin
+cat persona.txt | kbdb agent create \
+   --name my-agent --persona - \
+   --skills abc123
+
+# List all agents
+kbdb agent list
+
+# Retrieve an agent with resolved skills
+kbdb agent get agent-code-reviewer
+
+# Delete an agent (skills are preserved)
+kbdb agent delete agent-code-reviewer
+```
+
+## Environment Variables
+
+| Variable | Used By | Description |
+|----------|---------|-------------|
+| `KBDB_DIR` | MCP mode | Fallback database directory path when `--db` is not specified. Not used in CLI mode (CLI defaults to cwd). |
+
+## Exit Codes
+
+| Code | Meaning |
+|------|---------|
+| `0` | Success |
+| `1` | Error (invalid arguments, missing database, WASM error, version mismatch, unknown command, integrity check failure) |
+
+## Database Initialisation
+
+Before any data command (`learn`, `unlearn`,
+`search`, etc.), the CLI checks for an existing
+`.kbdb` directory:
+
+1. **No `.kbdb` found + interactive terminal** --
+   prompts to create one.
+2. **No `.kbdb` found + non-interactive** -- prints
+   error and exits.
+3. **`.kbdb` found** -- runs a version gate before
+   proceeding. If migration is needed, prints an
+   error directing the user to run `kbdb db migrate`.
+
+Use `kbdb db init` for explicit, non-interactive
+database creation.
+
+## Design Notes
+
+- The CLI is a thin pass-through. All keyword
+  extraction, indexing, ranking, and database I/O
+  is handled by the worker daemon and its WASM
+  modules.
+- **stdout** contains only command output. No
+  banners, progress bars, or decorative text.
+- **stderr** contains only errors (and deprecation
+  warnings for the legacy `--init`/`--migrate`
+  flags).
+- Works on both Deno and Node.js (>= 20).
+- The `--init` and `--migrate` flags are deprecated.
+  Use `kbdb db init` and `kbdb db migrate` instead.