@dikolab/kbdb 0.1.5 → 0.2.0

package/docs/modules/typescript/shared/worker-daemon/typings.md

@@ -0,0 +1,126 @@
+# worker-daemon/typings -- Daemon Type Definitions
+
+*Interfaces and type aliases for daemon configuration,
+IPC message shapes, and WASM module references.*
+
+## Source
+
+`src/shared/worker-daemon/typings/`
+
+## Typings
+
+### `DaemonConfig`
+
+**File:** `daemon-config.interface.ts`
+
+Parsed contents of `worker.toml`. Missing keys use
+defaults from
+[daemon-defaults.constant.ts](constants.md).
+
+```ts
+interface DaemonConfig {
+   readonly contextPath: string;
+   readonly itemLimit: number;
+   readonly indexLimit: number;
+   readonly timeoutMs: number;
+   readonly itemTtlMs: number;
+}
+```
+
+| Property      | Type     | Default  | Description       |
+|---------------|----------|----------|-------------------|
+| `contextPath` | `string` | --       | Absolute path to  |
+|               |          |          | the context dir   |
+| `itemLimit`   | `number` | `1000`   | Max content items |
+|               |          |          | in cache          |
+| `indexLimit`  | `number` | `100`    | Max index entries |
+|               |          |          | in cache          |
+| `timeoutMs`   | `number` | `300000` | Daemon idle       |
+|               |          |          | timeout (ms)      |
+| `itemTtlMs`   | `number` | `60000`  | Per-item cache    |
+|               |          |          | TTL (ms)          |
+
+### `IpcRequest`
+
+**File:** `ipc-handler.interface.ts`
+
+A JSON-RPC 2.0 request sent to the daemon over IPC.
+
+```ts
+interface IpcRequest {
+   readonly jsonrpc: '2.0';
+   readonly id: number | string;
+   readonly method: string;
+   readonly params?: unknown;
+}
+```
+
+### `IpcErrorDetail`
+
+**File:** `ipc-handler.interface.ts`
+
+Error detail included in a failed JSON-RPC response.
+
+```ts
+interface IpcErrorDetail {
+   readonly code: number;
+   readonly message: string;
+   readonly data?: unknown;
+}
+```
+
+### `IpcResponse`
+
+**File:** `ipc-handler.interface.ts`
+
+A JSON-RPC 2.0 response returned by the daemon.
+
+```ts
+interface IpcResponse {
+   readonly jsonrpc: '2.0';
+   readonly id: number | string;
+   readonly result?: unknown;
+   readonly error?: IpcErrorDetail;
+}
+```
+
+### `IpcHandler`
+
+**File:** `ipc-handler.interface.ts`
+
+Function type for handling a single IPC request and
+returning a JSON-RPC response.
+
+```ts
+type IpcHandler = (
+   request: IpcRequest,
+) => Promise<IpcResponse>;
+```
+
+The `WorkerDaemon.handleRequest` method satisfies this
+type and is passed to `createIpcServer`.
+
+### `WasmModules`
+
+**File:** `wasm-modules.interface.ts`
+
+References to the compiled and instantiated WASM
+module and its exported memory.
+
+```ts
+interface WasmModules {
+   readonly memory: WebAssembly.Memory;
+   readonly kbWorker: WebAssembly.Instance;
+}
+```
+
+| Property   | Type                   | Description            |
+|------------|------------------------|------------------------|
+| `memory`   | `WebAssembly.Memory`   | Linear memory exported |
+|            |                        | by `kb-worker.wasm`    |
+| `kbWorker` | `WebAssembly.Instance` | Instantiated           |
+|            |                        | `kb-worker.wasm`       |
+
+Returned by `loadWasmModules()`.
+
+No spec -- typings are compile-time only.

package/docs/overview.md

@@ -0,0 +1,188 @@
+# @dikolab/kbdb -- Knowledge Base Database
+
+*A persistent, file-based knowledge store and MCP server purpose-built for AI agents -- your second brain, on disk.*
+
+## What Is @dikolab/kbdb?
+
+AI agents operate in ephemeral sessions. They lose context between conversations, forget prior research, and have no built-in way to persist or retrieve knowledge across runs. When an agent needs to recall documentation, search a codebase's API surface, or query a corpus of notes, there is no lightweight, local solution optimized for how agents actually retrieve information -- keyword-driven, section-level, ranked.
+
+@dikolab/kbdb is a CLI tool and MCP server that imports documents and code documentation, extracts keywords, builds indexes, and enables ranked search -- all backed by a directory-based file database. No database server, no cloud dependency. A folder on disk is your entire knowledge base.
+
+The computationally intensive work -- keyword extraction, indexing, ranking -- is offloaded to a Rust engine compiled to WebAssembly, delivering near-native performance inside a Deno or Node.js runtime. The TypeScript CLI provides the developer-facing interface and MCP server; Rust/WASM does the heavy lifting underneath.
+
+## Who Is This For?
+
+- **AI agent developers** -- Anyone building agentic workflows (with Claude, LangChain, or custom harnesses) who needs persistent, searchable knowledge across sessions.
+- **MCP tool authors** -- Developers creating Model Context Protocol servers who need a local knowledge backend their tools can query.
+- **Knowledge workers using AI assistants** -- Power users who want their AI to have a second brain -- a searchable, structured store of documentation, research, and reference material that survives session boundaries.
+- **CLI-first developers** -- Those who prefer composable command-line tools over GUI applications for managing knowledge.
+
+## Core Features
+
+### CLI
+
+1. **Markdown import and indexing** -- Import `*.md` files, extract keywords and sections, and store them in the file system database.
+2. **Codebase documentation scanning** -- Scan a codebase and import code documentation (TypeDoc, JSDoc, PHPDoc), index it, and store it in the file system database.
+3. **Ranked search** -- Search and display a ranked list of results by section or by whole document, with output in `.md` or `.json` format.
+
+### MCP
+
+1. **MCP service endpoints** -- Expose the knowledge base as MCP service endpoints for Claude and other AI agents to query, search, and retrieve knowledge programmatically.
+
+### Distribution
+
+This tool is available via **NPM** for Node.js or **JSR** for Deno.
+
+- **Node.js**: `npm install -g @dikolab/kbdb`
+- **Deno**: `deno add @dikolab/kbdb` (via [jsr.io](https://jsr.io))
+
+## High-Level Architecture
+
+The system is organized in five layers. Data flows
+downward from user commands or agent requests through
+the worker client and daemon into the WASM engine,
+which performs heavy processing and manages the file
+database.
+
+![System Architecture](goals/architecture.svg)
+
+**The kbdb Command** -- A single executable with two
+run modes. CLI mode (`kbdb learn`, `kbdb search`,
+`kbdb unlearn`) handles human and script invocation --
+each command is a short-lived process. MCP mode
+(`kbdb mcp`) starts a long-lived MCP server that AI
+agents connect to over stdio.
+
+**Worker Client** -- Shared TypeScript module imported
+by both run modes. Discovers or spawns the worker
+daemon and provides a typed async API over IPC.
+
+**Worker Daemon** -- A detached background process
+that loads WASM modules, manages an in-memory cache,
+and serves JSON-RPC 2.0 requests over local IPC (Unix
+sockets or Windows named pipes). The daemon has no
+knowledge of its callers.
+
+**Rust/WASM Engine** -- Three WASM modules:
+`query-parser.wasm` (tokenization, query parsing,
+ranking), `fs-database.wasm` (file database I/O), and
+`kb-worker.wasm` (imports both, orchestrates). Rust
+provides memory safety and performance; WASM provides
+portability across runtimes.
+
+**File Database** -- A structured directory on disk.
+No external database process required. Documents,
+indexes, keywords, and metadata are stored in a
+defined directory layout. The database is portable,
+version-controllable, and human-inspectable.
+
+## Technology Stack
+
+| Layer | Technology | Role |
+|-------|-----------|------|
+| Runtime | Deno (primary), Node.js (compatible) | Executes the CLI, MCP server, and TypeScript orchestration |
+| Language | TypeScript | CLI, command parsing, MCP protocol, I/O coordination |
+| Engine | Rust compiled to WebAssembly | Keyword processing, indexing, ranking, file DB operations |
+| Protocol | MCP (Model Context Protocol) | AI agent integration interface |
+| Distribution | NPM (Node.js), JSR (Deno) | Package distribution for both ecosystems |
+| Storage | Directory-based file database | Persistent, portable, server-less data storage |
+
+## Design Principles
+
+1. **Offline-first, server-less** -- No database server, no cloud dependency. A directory on disk is the entire database. Works anywhere a filesystem exists.
+2. **AI-agent-native** -- Designed from the ground up to be invoked by AI agents, not just humans. Commands, output formats, and the MCP interface prioritize machine consumption.
+3. **MCP-first integration** -- The MCP server is a primary interface, not an afterthought. AI agents connect to the knowledge base as naturally as they connect to any other MCP service.
+4. **Performance through Rust/WASM** -- Computationally intensive work runs in compiled Rust via WebAssembly, keeping the TypeScript layer thin and the heavy processing fast.
+5. **Portability** -- Runs on Deno or Node.js. The file database is a plain directory. No platform lock-in at any layer.
+6. **Section-level precision** -- Documents are decomposed into sections. Search returns the specific passage that matters, not an entire document.
+7. **Incremental design** -- The project ships working slices. Storage format and engine capabilities will evolve; the architecture accommodates change without requiring rewrites.
+
+## Details
+
+The following documents describe the intended behavior and
+design goals for each major subsystem. They serve as
+implementation specifications, not descriptions of current
+functionality.
+
+- **[Database](goals/database.md)** -- Filesystem storage
+  engine (`fs-database.wasm`). The `.kbdb` directory
+  structure, section and document CRUD operations, index
+  management, and storage design.
+- **[Query Parser](goals/query-parser.md)** -- Query
+  processing and tokenization (`query-parser.wasm`).
+  Standalone WASM module for tokenization, query
+  parsing, keyword extraction, and ranking.
+- **[Document](goals/document.md)** -- Text content
+  format specification. Markdown body text with
+  embedding markers and cross-reference link markers.
+- **[Content Parser](goals/content-parser.md)** --
+  Marker extraction engine (`kbdb-shared` rlib).
+  Scans section content for `$(type: value)` markers
+  and returns structured reference lists.
+- **[Content Composer](goals/content-composer.md)** --
+  Content composition engine (`kbdb-shared` rlib).
+  Resolves sections and markers, produces rendered
+  Markdown documents from kbid and docid identifiers.
+- **[CLI](goals/cli.md)** -- The `kbdb` command and its
+  two run modes. CLI mode commands (`learn`, `unlearn`,
+  `search`), MCP mode dispatch, output formats, and the
+  pass-through execution model.
+- **[MCP](goals/mcp.md)** -- MCP server mode
+  (`kbdb mcp`). Invocation, database path
+  configuration, stdio transport, MCP tools and
+  resources, agent wiring examples, and relationship
+  to CLI mode.
+- **[Worker Daemon](goals/worker-daemon.md)** -- Cache
+  daemon (`kb-worker.wasm` + `src/worker.ts`).
+  The detached TypeScript daemon, WASM module loading,
+  caching, IPC protocol, lifecycle, and configuration.
+- **[Worker Client](goals/worker-client.md)** -- Shared
+  TypeScript client module (`src/shared/worker-client/`).
+  Daemon discovery, spawning, IPC connection, typed
+  API, and the sole entry point used by `src/cli.ts`.
+
+## Project Status
+
+@dikolab/kbdb is in active development. The full
+five-layer stack (CLI/MCP -> Worker Client -> Worker
+Daemon -> WASM Engine -> File Database) is wired and
+working end-to-end for all core operations: learn,
+search, recall, content retrieval, skills, agents,
+and maintenance commands.
+
+E2E smoke tests run on both Node.js and Deno via
+Docker, covering 380+ assertions across CLI commands,
+18 MCP tools, MCP protocol features, and worker IPC
+methods.
+
+### Completed
+
+- CLI with 18 commands across 6 groups (tested)
+- MCP server with 18 tools, prompts, resources,
+  and completions (tested)
+- Worker client with typed async IPC API (tested)
+- Worker daemon with WASM module loading (tested)
+- Rust/WASM engine: search, recall, indexing,
+  reference graph, KWIC snippets, pagination
+- Paginated search with rich metadata (heading,
+  type, snippet, matched_terms, docids)
+- Progressive recall at depths 0--3
+- Bidirectional reference graph (references.idx)
+- Skills (template-based prompts) and agents
+  (persona + skills compositions)
+- esbuild bundling pipeline (ESM + CJS + .d.ts)
+- Docker build containers (Deno + Node + Rust +
+  wasm-pack)
+- npm and JSR packaging with documentation
+
+### Remaining
+
+- Complete WASM host I/O imports (some operations
+  still hit stubs: document removal, agent document
+  persistence)
+- Database format versioning and migration
+  infrastructure
+
+## Links
+
+- **Repository**: https://gitlab.com/tech-slaves-internal/knowledge-base-db

package/docs/release-notes/0.1.0.md

@@ -0,0 +1,189 @@
+# 0.1.0
+
+*Initial release -- 2026-06-19*
+
+The first public release of **@dikolab/kbdb**, a
+file-based knowledge base database that stores,
+indexes, and searches documentation without needing
+a database server.
+
+---
+
+## Highlights
+
+- Store documentation in a plain folder on disk --
+  no server, no cloud account, no infrastructure
+- Ranked search powered by a Rust/WASM engine
+- CLI with 17 commands for managing documents,
+  skills, and agents
+- MCP server that gives AI agents a searchable
+  second brain
+- Runs on both Node.js (>= 20) and Deno (>= 2)
+
+---
+
+## Installation
+
+```sh
+# Node.js
+npm install -g @dikolab/kbdb
+
+# Deno
+deno install -Agf jsr:@dikolab/kbdb/cli
+```
+
+---
+
+## CLI
+
+The `kbdb` command organises 17 commands across
+6 groups:
+
+| Group | Commands |
+|-------|----------|
+| Knowledge Base | `learn`, `unlearn`, `search`, `content` |
+| Database | `db init`, `db migrate` |
+| Maintenance | `status`, `check`, `gc`, `rebuild` |
+| Skills | `skill learn`, `skill list`, `skill get`, `skill delete` |
+| Agents | `agent create`, `agent list`, `agent get`, `agent delete` |
+| Server | `mcp` |
+
+See the [CLI Reference](../details/cli.md) for
+options and examples.
+
+---
+
+## Search and Ranking
+
+Search uses **BM25F** field-weighted ranking with
+three indexed fields per section:
+
+| Field | Weight | Description |
+|-------|--------|-------------|
+| Heading | 2.0x | Titles and headings |
+| Body | 1.0x | Main text content |
+| Code | 1.5x | Code blocks and inline code |
+
+Queries support keywords, exact phrases
+(`"error handling"`), boolean operators
+(`auth AND token`, `cache OR redis`), and term
+exclusions (`auth -oauth`).
+
+See [How Search Works](../details/search-and-ranking.md)
+for internals.
+
+---
+
+## MCP Server
+
+The `kbdb mcp` command starts an MCP server over
+stdio using JSON-RPC 2.0 (MCP Protocol 2025-11-25).
+
+**17 tools** available to AI agents:
+
+- **Knowledge Base**: `search`, `learn`, `unlearn`,
+  `retrieve`, `content`
+- **Database**: `status`, `check`, `gc`, `rebuild`
+- **Skills**: `skill-learn`, `skill-list`,
+  `skill-get`, `skill-delete`
+- **Agents**: `agent-create`, `agent-list`,
+  `agent-get`, `agent-delete`
+
+Works with Claude Code, Claude for VS Code,
+Claude Desktop, and any MCP-compatible agent.
+See the [MCP Server Guide](../details/mcp-server.md)
+for setup instructions.
+
+---
+
+## Skills and Agents
+
+**Skills** are reusable templates with placeholder
+arguments. Store common prompts or instructions
+once and reference them by name.
+
+**Agents** are AI agent profiles that combine a
+persona with a set of skills. Create named agent
+configurations that persist across sessions.
+
+---
+
+## Storage
+
+All data lives in a `.kbdb` directory on disk:
+
+```
+.kbdb/
+   catalog.toml       -- version and stats
+   corpus.bin          -- term frequencies
+   sections/           -- indexed content
+   documents/          -- document groupings
+```
+
+Content is identified by **kbid** -- a content-
+addressed ID derived from SHA-256 (128-bit
+truncation, base32-encoded). Identical content
+always produces the same kbid.
+
+See [Storage Architecture](../details/storage.md)
+for file formats and design details.
+
+---
+
+## Rust/WASM Engine
+
+The search engine is written in Rust and compiled
+to WebAssembly. Four crates handle different
+responsibilities:
+
+| Crate | Role |
+|-------|------|
+| `kbdb-shared` | Memory management, encoding, error types |
+| `kbdb-fs-database` | Storage, indexing, search |
+| `kbdb-query-parser` | Query tokenization and parsing |
+| `kbdb-worker` | IPC request dispatch |
+
+WASM modules load at runtime -- no native
+compilation needed on the user's machine.
+
+---
+
+## Build Pipeline
+
+Development uses a Docker-based build pipeline
+that includes Node.js, Deno, Rust, and wasm-pack:
+
+```sh
+make build       # Full pipeline: lint, test, wasm, bundle
+make test-node   # E2E smoke tests (Node.js)
+make test-deno   # E2E smoke tests (Deno)
+make lint        # Rust + TypeScript linting
+```
+
+See the [Makefile](../../Makefile) for all targets.
+
+---
+
+## Platforms
+
+| Runtime | Minimum version |
+|---------|----------------|
+| Node.js | 20.0.0 |
+| Deno | 2.0.0 |
+
+Published to:
+
+- [npm](https://www.npmjs.com/package/@dikolab/kbdb)
+  as `@dikolab/kbdb`
+- [JSR](https://jsr.io/@dikolab/kbdb) as
+  `@dikolab/kbdb`
+
+---
+
+## Documentation
+
+- [CLI Reference](../details/cli.md)
+- [MCP Server Guide](../details/mcp-server.md)
+- [Search and Ranking](../details/search-and-ranking.md)
+- [Storage Architecture](../details/storage.md)
+- [Library API](../details/library-api.md)

package/docs/release-notes/0.1.1.md

@@ -0,0 +1,46 @@
+# 0.1.1
+
+*Patch release -- 2026-06-20*
+
+Fixes a Deno type-checking error that blocked
+`deno publish`, and improves the build pipeline.
+
+---
+
+## Fixes
+
+- **Deno type error in WASM memory buffer** --
+  `WebAssembly.Memory.buffer` returns
+  `ArrayBuffer | SharedArrayBuffer` under Deno's
+  type system but `ArrayBuffer` under Node.js.
+  Added explicit cast to satisfy both type-checkers.
+
+## Improvements
+
+- **WASM directory resolution** -- `resolveBaseDir`
+  now supports three fallback strategies for
+  locating WASM files:
+  1. `__dirname` (Node.js CJS)
+  2. `import.meta.dirname` (Deno local, Node.js
+     ESM v21.2+)
+  3. URL-based derivation (JSR remote modules)
+
+- **Build pipeline cleanup** -- `build-wasm.sh`
+  and `build-ts.sh` now remove stale output
+  directories (`src/wasm/`, `dist/`) before
+  building, ensuring fresh builds every time.
+
+- **Rust test stability** -- Added
+  `--test-threads=1` to Rust integration tests
+  to prevent race conditions on shared WASM
+  return-slot globals.
+
+---
+
+## Upgrade
+
+```sh
+npm install -g @dikolab/kbdb@0.1.1
+```
+
+No breaking changes from 0.1.0.

package/docs/release-notes/0.1.2.md

@@ -0,0 +1,38 @@
+# 0.1.2
+
+*Patch release -- 2026-06-20*
+
+Fixes WASM loading failure when installed from JSR
+(`deno install -Agf jsr:@dikolab/kbdb/cli`).
+
+---
+
+## Fixes
+
+- **WASM loading from JSR remote URLs** -- When
+  installed from JSR, module URLs use `https://`
+  instead of filesystem paths. `path.join` mangled
+  these URLs by stripping the `//` in `https://`,
+  and `readFileSync` cannot open URLs. Added
+  URL-aware path joining (`joinPath`) and async
+  WASM loading via `fetch()` for remote URLs.
+
+- **Worker script resolution for JSR** --
+  `resolveWorkerScript` now preserves URL format
+  when the script directory is a remote URL, so
+  `deno run` receives a valid URL instead of a
+  mangled path.
+
+---
+
+## Upgrade
+
+```sh
+# Node.js
+npm install -g @dikolab/kbdb@0.1.2
+
+# Deno
+deno install -Agf jsr:@dikolab/kbdb/cli
+```
+
+No breaking changes from 0.1.1.

package/docs/release-notes/0.1.3.md

@@ -0,0 +1,42 @@
+# 0.1.3
+
+*Patch release -- 2026-06-20*
+
+Fixes WASM binaries missing from JSR package, which
+caused 404 errors when installed via
+`deno install -Agf jsr:@dikolab/kbdb/cli`.
+
+---
+
+## Fixes
+
+- **WASM binaries excluded from JSR publish** --
+  `deno publish` respects `.gitignore`, which listed
+  `src/wasm/`. Additionally, wasm-pack generates
+  `.gitignore` files (containing `*`) inside each
+  WASM output directory. Both layers prevented WASM
+  binaries from being included in the JSR package.
+  The build script now removes wasm-pack's nested
+  `.gitignore` files, and the publish script
+  temporarily un-ignores `src/wasm/` during
+  `deno publish`.
+
+- **Excluded unnecessary wasm-pack artifacts from
+  JSR** -- Added `publish.exclude` patterns for
+  `.js`, `.d.ts`, and `package.json` files inside
+  `src/wasm/`, so only the `.wasm` binaries are
+  published to JSR.
+
+---
+
+## Upgrade
+
+```sh
+# Node.js
+npm install -g @dikolab/kbdb@0.1.3
+
+# Deno
+deno install -Agf jsr:@dikolab/kbdb/cli
+```
+
+No breaking changes from 0.1.2.