@dikolab/kbdb 0.4.0 → 0.4.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +30 -22
- package/dist/{chunk-BJPSPNGX.mjs → chunk-BJXEVALU.mjs} +115 -1
- package/dist/chunk-BJXEVALU.mjs.map +7 -0
- package/dist/cli.cjs +462 -79
- package/dist/cli.cjs.map +4 -4
- package/dist/cli.mjs +356 -79
- package/dist/cli.mjs.map +4 -4
- package/dist/kbdb-worker.cjs +413 -53
- package/dist/kbdb-worker.cjs.map +4 -4
- package/dist/mod.cjs +230 -54
- package/dist/mod.cjs.map +4 -4
- package/dist/mod.mjs +229 -53
- package/dist/mod.mjs.map +4 -4
- package/dist/src/shared/cli/typings/cli-options.interface.d.ts +5 -0
- package/dist/src/shared/embedding/classes/tfidf-embedding-provider.class.d.ts +28 -0
- package/dist/src/shared/embedding/constants/tfidf-defaults.constant.d.ts +4 -0
- package/dist/src/shared/embedding/functions/create-embedding-provider.function.d.ts +5 -2
- package/dist/src/shared/embedding/index.d.ts +2 -0
- package/dist/src/shared/log/constants/log-colors.constant.d.ts +5 -0
- package/dist/src/shared/log/constants/log-defaults.constant.d.ts +3 -0
- package/dist/src/shared/log/constants/log-icons.constant.d.ts +3 -0
- package/dist/src/shared/log/functions/configure-log.function.d.ts +12 -0
- package/dist/src/shared/log/functions/log.function.d.ts +28 -0
- package/dist/src/shared/log/functions/parse-log-level.function.d.ts +10 -0
- package/dist/src/shared/log/functions/resolve-log-level.function.d.ts +12 -0
- package/dist/src/shared/log/functions/write-stderr.function.d.ts +10 -0
- package/dist/src/shared/log/index.d.ts +10 -0
- package/dist/src/shared/log/typings/log-config.interface.d.ts +7 -0
- package/dist/src/shared/log/typings/log-level.type.d.ts +7 -0
- package/dist/src/shared/log/typings/log-output.interface.d.ts +4 -0
- package/dist/src/shared/version/constants/version.constant.d.ts +1 -1
- package/dist/src/shared/worker-client/classes/worker-client.class.d.ts +7 -0
- package/dist/src/shared/worker-client/functions/retry-on-timeout.function.d.ts +1 -0
- package/dist/src/shared/worker-daemon/classes/worker-daemon.class.d.ts +1 -0
- package/dist/src/shared/worker-daemon/functions/run-worker.function.d.ts +6 -1
- package/dist/src/shared/worker-daemon/typings/wasm-modules.interface.d.ts +5 -0
- package/dist/wasm/default-embedding/kbdb_default_embedding.d.ts +55 -0
- package/dist/wasm/default-embedding/kbdb_default_embedding.js +156 -0
- package/dist/wasm/default-embedding/kbdb_default_embedding_bg.wasm +0 -0
- package/dist/wasm/default-embedding/kbdb_default_embedding_bg.wasm.d.ts +12 -0
- package/dist/wasm/default-embedding/package.json +13 -0
- package/dist/wasm/fs-database/kbdb_fs_database_bg.wasm +0 -0
- package/dist/wasm/fs-migration/kbdb_fs_migration.d.ts +38 -0
- package/dist/wasm/fs-migration/kbdb_fs_migration.js +130 -0
- package/dist/wasm/fs-migration/kbdb_fs_migration_bg.wasm +0 -0
- package/dist/wasm/fs-migration/kbdb_fs_migration_bg.wasm.d.ts +9 -0
- package/dist/wasm/fs-migration/package.json +13 -0
- package/dist/wasm/kb-worker/kbdb_worker_bg.wasm +0 -0
- package/dist/wasm/query-parser/kbdb_query_parser_bg.wasm +0 -0
- package/dist/worker.mjs +306 -54
- package/dist/worker.mjs.map +4 -4
- package/docs/details/agent-tooling.md +10 -0
- package/docs/details/cli.md +29 -13
- package/docs/details/knowledge-base.md +15 -8
- package/docs/details/mcp-server.md +16 -1
- package/docs/details/search-and-ranking.md +6 -2
- package/docs/goals/cli.md +33 -4
- package/docs/goals/content-parser.md +1 -1
- package/docs/goals/hybrid-search.md +71 -42
- package/docs/goals/mcp.md +3 -1
- package/docs/goals/overview.md +1 -1
- package/docs/goals/query-parser.md +1 -1
- package/docs/goals/semantic-dedup.md +7 -3
- package/docs/goals/worker-daemon.md +31 -20
- package/docs/modules/overview.md +43 -22
- package/docs/modules/rust/embedding/module.md +91 -0
- package/docs/modules/rust/overview.md +56 -21
- package/docs/modules/rust/query-parser/functions/module.md +7 -5
- package/docs/modules/rust/query-parser/functions/pipeline.md +6 -138
- package/docs/modules/rust/query-parser/module.md +16 -22
- package/docs/modules/rust/query-parser/types/module.md +5 -4
- package/docs/modules/rust/query-parser/types/token.md +5 -1
- package/docs/modules/rust/shared/content-parser/functions.md +1 -1
- package/docs/modules/rust/shared/memory/module.md +1 -1
- package/docs/modules/rust/shared/module.md +33 -5
- package/docs/modules/rust/shared/pipeline/functions.md +141 -0
- package/docs/modules/rust/shared/pipeline/module.md +62 -0
- package/docs/modules/rust/shared/pipeline/types.md +43 -0
- package/docs/modules/rust/shared/section/module.md +1 -1
- package/docs/modules/rust/shared/section/types.md +1 -1
- package/docs/modules/typescript/cli.md +4 -2
- package/docs/modules/typescript/kbdb-worker.md +20 -5
- package/docs/modules/typescript/overview.md +28 -6
- package/docs/modules/typescript/shared/cli/functions.md +15 -4
- package/docs/modules/typescript/shared/cli/typings.md +21 -0
- package/docs/modules/typescript/shared/embedding/functions.md +30 -3
- package/docs/modules/typescript/shared/embedding/module.md +10 -4
- package/docs/modules/typescript/shared/embedding/typings.md +7 -5
- package/docs/modules/typescript/shared/log/constants.md +82 -0
- package/docs/modules/typescript/shared/log/functions.md +131 -0
- package/docs/modules/typescript/shared/log/module.md +153 -0
- package/docs/modules/typescript/shared/log/typings.md +82 -0
- package/docs/modules/typescript/shared/module.md +22 -6
- package/docs/modules/typescript/shared/worker-daemon/functions.md +19 -6
- package/docs/modules/typescript/shared/worker-daemon/module.md +5 -2
- package/docs/modules/typescript/shared/worker-daemon/typings.md +13 -7
- package/docs/release-notes/0.4.1.md +149 -0
- package/docs/release-notes/0.4.3.md +60 -0
- package/package.json +2 -2
- package/dist/chunk-BJPSPNGX.mjs.map +0 -7
|
@@ -37,6 +37,16 @@ file for your tool.
|
|
|
37
37
|
`@dikolab/kbdb` with `deno` / `run` / `-A` /
|
|
38
38
|
`jsr:@dikolab/kbdb/cli`.
|
|
39
39
|
|
|
40
|
+
Environment variables:
|
|
41
|
+
|
|
42
|
+
| Variable | Description |
|
|
43
|
+
|----------|-------------|
|
|
44
|
+
| `KBDB_DB_DIR` | Database path (replaces `--db`) |
|
|
45
|
+
| `KBDB_LOG_LEVEL` | `debug`, `info`, `warning`, `error` |
|
|
46
|
+
| `KBDB_NON_INTERACTIVE` | Suppress prompts (`1` / `true`) |
|
|
47
|
+
| `KBDB_DAEMON_TIMEOUT` | Request timeout in ms |
|
|
48
|
+
| `KBDB_RETRY` | Max retries on timeout |
|
|
49
|
+
|
|
40
50
|
See the [MCP Server Guide](mcp-server.md) for
|
|
41
51
|
environment variable config, multiple knowledge
|
|
42
52
|
bases, and the full 20-tool reference.
|
package/docs/details/cli.md
CHANGED
|
@@ -21,6 +21,10 @@ deno install -Agf jsr:@dikolab/kbdb/cli
|
|
|
21
21
|
kbdb [options]
|
|
22
22
|
kbdb <command> [options] ...[parameters]
|
|
23
23
|
kbdb <command> <subcommand> [options] ...[parameters]
|
|
24
|
+
kbdb --version
|
|
25
|
+
kbdb -v <command> (verbose: warnings)
|
|
26
|
+
kbdb -vv <command> (verbose: info)
|
|
27
|
+
kbdb -vvv <command> (verbose: debug)
|
|
24
28
|
```
|
|
25
29
|
|
|
26
30
|
## Command Groups
|
|
@@ -74,11 +78,15 @@ docid and want rendered Markdown output.
|
|
|
74
78
|
| Option | Short | Default | Description |
|
|
75
79
|
|--------|-------|---------|-------------|
|
|
76
80
|
| `--help` | `-h` | | Print help text and exit |
|
|
77
|
-
| `--version` |
|
|
81
|
+
| `--version` | | | Print version and exit |
|
|
82
|
+
| `--verbose` | `-v` | | Set log level to WARNING (errors + warnings) |
|
|
83
|
+
| `--verbose-info` | `-vv` | | Set log level to INFO |
|
|
84
|
+
| `--verbose-debug` | `-vvv` | | Set log level to DEBUG (all messages) |
|
|
78
85
|
| `--db <path>` | | cwd | Database directory path (optional) |
|
|
79
86
|
| `--format <fmt>` | `-f` | `json` (text when interactive) | Output format (`json`, `text`, `mcp`) |
|
|
80
87
|
| `--timeout-ms <ms>` | | `30000` | Per-request daemon timeout in milliseconds |
|
|
81
88
|
| `--retry <n>` | | `1` | Max retries with daemon respawn on timeout |
|
|
89
|
+
| `--non-interactive` | | | Suppress prompts; use defaults |
|
|
82
90
|
|
|
83
91
|
### `--db <path>`
|
|
84
92
|
|
|
@@ -116,25 +124,23 @@ These options apply to the `search` command.
|
|
|
116
124
|
| Algorithm | Description |
|
|
117
125
|
|-----------|-------------|
|
|
118
126
|
| `lexical` | Keyword matching with stemming and synonyms |
|
|
119
|
-
| `vector` |
|
|
127
|
+
| `vector` | Similarity search using embeddings |
|
|
120
128
|
| `hybrid` | Combines keyword and similarity results for best coverage |
|
|
121
129
|
|
|
122
130
|
### Configuring Hybrid Search
|
|
123
131
|
|
|
124
|
-
Hybrid and vector search
|
|
125
|
-
|
|
126
|
-
|
|
132
|
+
Hybrid and vector search use a built-in TF-IDF
|
|
133
|
+
embedding provider by default -- no configuration
|
|
134
|
+
needed. For higher quality embeddings, configure a
|
|
135
|
+
provider via `worker.toml`:
|
|
127
136
|
|
|
128
137
|
| Provider | Description |
|
|
129
138
|
|----------|-------------|
|
|
130
|
-
|
|
|
139
|
+
| *(none)* | Built-in TF-IDF (256 dimensions, works offline) |
|
|
140
|
+
| `none` | Explicitly disabled; hybrid and vector return an error |
|
|
131
141
|
| `onnx` | Local ONNX model (no network required) |
|
|
132
142
|
| `api` | Remote embedding API (OpenAI-compatible endpoint) |
|
|
133
143
|
|
|
134
|
-
When no provider is configured, `--algo hybrid` and
|
|
135
|
-
`--algo vector` return an error prompting you to
|
|
136
|
-
configure an embedding provider.
|
|
137
|
-
|
|
138
144
|
### Search Modes
|
|
139
145
|
|
|
140
146
|
| Mode | Description |
|
|
@@ -720,6 +726,8 @@ kbdb agent delete agent-code-reviewer
|
|
|
720
726
|
| `KBDB_DB_DIR` | CLI + MCP | cwd (CLI) / none (MCP) | Fallback database directory path when `--db` is not specified. |
|
|
721
727
|
| `KBDB_DAEMON_TIMEOUT` | CLI + MCP | `30000` | Per-request daemon timeout in milliseconds. Overridden by `--timeout-ms`. |
|
|
722
728
|
| `KBDB_RETRY` | CLI + MCP | `1` | Max retries with daemon respawn on timeout. Overridden by `--retry`. |
|
|
729
|
+
| `KBDB_LOG_LEVEL` | CLI + MCP + Daemon | `error` | Log verbosity: `debug`, `info`, `warning`, `error`. Overridden by `-v`/`-vv`/`-vvv`. |
|
|
730
|
+
| `KBDB_NON_INTERACTIVE` | CLI + MCP + Daemon | off | Suppress interactive prompts. Accepts `1` or `true`. Overridden by `--non-interactive`. |
|
|
723
731
|
|
|
724
732
|
## Exit Codes
|
|
725
733
|
|
|
@@ -738,6 +746,12 @@ Before any data command (`learn`, `unlearn`,
|
|
|
738
746
|
prompts to create one.
|
|
739
747
|
2. **No `.kbdb` found + non-interactive** -- prints
|
|
740
748
|
error and exits.
|
|
749
|
+
|
|
750
|
+
The `--non-interactive` flag (or
|
|
751
|
+
`KBDB_NON_INTERACTIVE=1`) forces the non-interactive
|
|
752
|
+
path regardless of terminal type. This is useful in
|
|
753
|
+
scripts, CI pipelines, and automated testing.
|
|
754
|
+
|
|
741
755
|
3. **`.kbdb` found** -- runs a version gate before
|
|
742
756
|
proceeding. If migration is needed, prints an
|
|
743
757
|
error directing the user to run `kbdb db migrate`.
|
|
@@ -753,9 +767,11 @@ database creation.
|
|
|
753
767
|
modules.
|
|
754
768
|
- **stdout** contains only command output. No
|
|
755
769
|
banners, progress bars, or decorative text.
|
|
756
|
-
- **stderr** contains
|
|
757
|
-
|
|
758
|
-
|
|
770
|
+
- **stderr** contains log messages. Default level is
|
|
771
|
+
ERROR (errors only). Use `-v`/`-vv`/`-vvv` or
|
|
772
|
+
`KBDB_LOG_LEVEL` to increase verbosity. Log lines
|
|
773
|
+
are prefixed with per-level icons (✖⚠ℹ▸) and ANSI
|
|
774
|
+
colors.
|
|
759
775
|
- Works on both Deno and Node.js (>= 20).
|
|
760
776
|
- The `--init` and `--migrate` flags are deprecated.
|
|
761
777
|
Use `kbdb db init` and `kbdb db migrate` instead.
|
|
@@ -74,11 +74,11 @@ of creating duplicates.
|
|
|
74
74
|
| `--algo hybrid` | hybrid | Keyword + AI similarity |
|
|
75
75
|
| `--algo vector` | vector | AI similarity only |
|
|
76
76
|
|
|
77
|
-
Hybrid and vector modes
|
|
78
|
-
|
|
79
|
-
configure an
|
|
80
|
-
See the [CLI Reference](cli.md) for
|
|
81
|
-
configuration.
|
|
77
|
+
Hybrid and vector modes work out of the box using a
|
|
78
|
+
built-in TF-IDF embedding provider. For higher
|
|
79
|
+
quality, configure an ONNX or API provider in
|
|
80
|
+
`worker.toml`. See the [CLI Reference](cli.md) for
|
|
81
|
+
provider configuration.
|
|
82
82
|
|
|
83
83
|
When a query returns no exact matches, kbdb
|
|
84
84
|
automatically loosens the search (OR instead of AND)
|
|
@@ -159,10 +159,17 @@ two global flags:
|
|
|
159
159
|
|------|---------|-------------|
|
|
160
160
|
| `--timeout-ms <ms>` | `30000` | Per-request daemon timeout |
|
|
161
161
|
| `--retry <n>` | `1` | Retries with daemon respawn on timeout |
|
|
162
|
+
| `--non-interactive` | off | Suppress prompts; use defaults |
|
|
162
163
|
|
|
163
|
-
These can also be set via
|
|
164
|
-
|
|
165
|
-
|
|
164
|
+
These can also be set via environment variables.
|
|
165
|
+
The flags take priority over env vars.
|
|
166
|
+
|
|
167
|
+
| Variable | Description |
|
|
168
|
+
|----------|-------------|
|
|
169
|
+
| `KBDB_DAEMON_TIMEOUT` | Per-request daemon timeout (ms) |
|
|
170
|
+
| `KBDB_RETRY` | Max retry count |
|
|
171
|
+
| `KBDB_LOG_LEVEL` | Log verbosity: `debug`, `info`, `warning`, `error` |
|
|
172
|
+
| `KBDB_NON_INTERACTIVE` | Suppress prompts (`1` or `true`) |
|
|
166
173
|
|
|
167
174
|
## Deep Dives
|
|
168
175
|
|
|
@@ -169,6 +169,20 @@ Maximum number of retries with daemon respawn when
|
|
|
169
169
|
a request times out. Defaults to `1`. Overridden
|
|
170
170
|
by the `--retry` flag.
|
|
171
171
|
|
|
172
|
+
### `KBDB_LOG_LEVEL`
|
|
173
|
+
|
|
174
|
+
Log verbosity: `debug`, `info`, `warning`, `error`
|
|
175
|
+
(case-insensitive). Defaults to `error`. Overridden
|
|
176
|
+
by `-v`/`-vv`/`-vvv` flags. The level is forwarded
|
|
177
|
+
to the worker daemon via `--log-level`.
|
|
178
|
+
|
|
179
|
+
### `KBDB_NON_INTERACTIVE`
|
|
180
|
+
|
|
181
|
+
Suppress interactive prompts. Accepts `1` or
|
|
182
|
+
`true`. The MCP server does not prompt interactively
|
|
183
|
+
(it reads JSON-RPC from stdin), but the variable is
|
|
184
|
+
inherited by the worker daemon for consistency.
|
|
185
|
+
|
|
172
186
|
These variables can be set in the `env` block of
|
|
173
187
|
your MCP configuration:
|
|
174
188
|
|
|
@@ -181,7 +195,8 @@ your MCP configuration:
|
|
|
181
195
|
"env": {
|
|
182
196
|
"KBDB_DB_DIR": "/path/to/.kbdb",
|
|
183
197
|
"KBDB_DAEMON_TIMEOUT": "60000",
|
|
184
|
-
"KBDB_RETRY": "2"
|
|
198
|
+
"KBDB_RETRY": "2",
|
|
199
|
+
"KBDB_LOG_LEVEL": "warning"
|
|
185
200
|
}
|
|
186
201
|
}
|
|
187
202
|
}
|
|
@@ -567,5 +567,9 @@ into a score (`1 / (k + rank)` with k=60), then
|
|
|
567
567
|
summing scores across both lists. Results that rank
|
|
568
568
|
well in both lists get the highest combined scores.
|
|
569
569
|
|
|
570
|
-
Hybrid mode
|
|
571
|
-
|
|
570
|
+
Hybrid mode uses a built-in TF-IDF embedding provider
|
|
571
|
+
by default. Configure `provider = "onnx"` or
|
|
572
|
+
`provider = "api"` in `worker.toml` for higher
|
|
573
|
+
quality embeddings. Setting `provider = "none"`
|
|
574
|
+
explicitly disables embeddings and returns an error
|
|
575
|
+
for hybrid and vector searches.
|
package/docs/goals/cli.md
CHANGED
|
@@ -49,7 +49,10 @@ All commands and their canonical forms:
|
|
|
49
49
|
|
|
50
50
|
```
|
|
51
51
|
kbdb --help | -h
|
|
52
|
-
kbdb --version
|
|
52
|
+
kbdb --version
|
|
53
|
+
kbdb -v <command> (verbose: warnings)
|
|
54
|
+
kbdb -vv <command> (verbose: info)
|
|
55
|
+
kbdb -vvv <command> (verbose: debug)
|
|
53
56
|
kbdb learn [options] <path | ->...
|
|
54
57
|
kbdb unlearn [options] <kbid>...
|
|
55
58
|
kbdb search [options] <query>...
|
|
@@ -120,11 +123,15 @@ These options apply to all commands.
|
|
|
120
123
|
| Option | Short | Description |
|
|
121
124
|
|--------------------|-------|--------------------------|
|
|
122
125
|
| `--help` | `-h` | Print help text and exit |
|
|
123
|
-
| `--version` |
|
|
126
|
+
| `--version` | | Print version and exit |
|
|
127
|
+
| `--verbose` | `-v` | Set log level to WARNING (errors + warnings) |
|
|
128
|
+
| `--verbose-info` | `-vv` | Set log level to INFO |
|
|
129
|
+
| `--verbose-debug` | `-vvv`| Set log level to DEBUG (all messages) |
|
|
124
130
|
| `--db <path>` | | Database directory path (optional) |
|
|
125
131
|
| `--format <fmt>` | `-f` | Output format (json default; text when stdout is a TTY) |
|
|
126
132
|
| `--timeout-ms <ms>`| | Request timeout in ms |
|
|
127
133
|
| `--retry <n>` | | Max retries on timeout |
|
|
134
|
+
| `--non-interactive` | | Suppress prompts; use defaults |
|
|
128
135
|
|
|
129
136
|
### `--help`, `-h`
|
|
130
137
|
|
|
@@ -139,16 +146,31 @@ kbdb -h
|
|
|
139
146
|
When no command is given (`kbdb` with no arguments),
|
|
140
147
|
the help text is printed as the default behaviour.
|
|
141
148
|
|
|
142
|
-
### `--version
|
|
149
|
+
### `--version`
|
|
143
150
|
|
|
144
151
|
Print the package version string and exit. No database
|
|
145
152
|
connection is made.
|
|
146
153
|
|
|
147
154
|
```
|
|
148
155
|
kbdb --version
|
|
149
|
-
kbdb -v
|
|
150
156
|
```
|
|
151
157
|
|
|
158
|
+
### `--verbose`, `-v` / `-vv` / `-vvv`
|
|
159
|
+
|
|
160
|
+
Increase log verbosity on stderr. Each additional `v`
|
|
161
|
+
raises the level by one step.
|
|
162
|
+
|
|
163
|
+
```
|
|
164
|
+
kbdb -v <command> # WARNING: errors + warnings
|
|
165
|
+
kbdb -vv <command> # INFO: errors, warnings, info
|
|
166
|
+
kbdb -vvv <command> # DEBUG: all messages
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
The default level is ERROR. The `KBDB_LOG_LEVEL`
|
|
170
|
+
environment variable sets the base level; CLI flags
|
|
171
|
+
override it. Log lines are prefixed with per-level
|
|
172
|
+
icons (✖⚠ℹ▸) and ANSI colors.
|
|
173
|
+
|
|
152
174
|
### `--db <path>`
|
|
153
175
|
|
|
154
176
|
Set the path to the directory that contains (or will
|
|
@@ -1709,6 +1731,13 @@ storage directory via `findDb()`.
|
|
|
1709
1731
|
- If stdin is not a TTY (piped / CI): print error to
|
|
1710
1732
|
stderr, exit 1. No prompt.
|
|
1711
1733
|
|
|
1734
|
+
2b. **Non-interactive override** -- when
|
|
1735
|
+
`--non-interactive` is passed or
|
|
1736
|
+
`KBDB_NON_INTERACTIVE` is set to `1` or `true`,
|
|
1737
|
+
the CLI overrides `isStdinTty()` to return
|
|
1738
|
+
`false`. This forces the non-interactive path
|
|
1739
|
+
(error + exit 1) regardless of terminal type.
|
|
1740
|
+
|
|
1712
1741
|
3. **Version check gate** -- when `.kbdb` exists, the
|
|
1713
1742
|
CLI calls `client.checkVersion(dbDir)` before
|
|
1714
1743
|
dispatching any data command:
|
|
@@ -138,7 +138,7 @@ preserved; the caller should normalise whitespace
|
|
|
138
138
|
if needed.
|
|
139
139
|
|
|
140
140
|
This function replaces the current implementation
|
|
141
|
-
in `
|
|
141
|
+
in `shared/src/pipeline/markers.rs`. The
|
|
142
142
|
existing `strip_content_markers()` recognizes only
|
|
143
143
|
`$(image:)` and `$(code:)`. The shared version
|
|
144
144
|
recognizes all six marker types.
|
|
@@ -34,8 +34,9 @@ both modes.
|
|
|
34
34
|
## Architecture
|
|
35
35
|
|
|
36
36
|
Two retrieval paths run in parallel. The vector path
|
|
37
|
-
runs
|
|
38
|
-
|
|
37
|
+
runs in TypeScript, backed by a lightweight WASM crate
|
|
38
|
+
(`kbdb-default-embedding`) for the built-in TF-IDF
|
|
39
|
+
provider.
|
|
39
40
|
|
|
40
41
|
```
|
|
41
42
|
Query → ┌─ Lexical path (WASM) ─→ BM25F ranked list ─┐
|
|
@@ -43,12 +44,19 @@ Query → ┌─ Lexical path (WASM) ─→ BM25F ranked list ─┐
|
|
|
43
44
|
└─ Vector path (TS) ─→ Cosine ranked list ──┘
|
|
44
45
|
```
|
|
45
46
|
|
|
46
|
-
The WASM
|
|
47
|
-
remains a pure-computation lexical
|
|
48
|
-
continues to orchestrate
|
|
49
|
-
`fs-database.wasm` exactly as
|
|
50
|
-
|
|
51
|
-
|
|
47
|
+
The existing WASM modules are untouched.
|
|
48
|
+
`query-parser.wasm` remains a pure-computation lexical
|
|
49
|
+
module. `kb-worker.wasm` continues to orchestrate
|
|
50
|
+
lexical retrieval via `fs-database.wasm` exactly as
|
|
51
|
+
today. No new exports or host imports are added to
|
|
52
|
+
any existing WASM module.
|
|
53
|
+
|
|
54
|
+
A new standalone WASM crate, `kbdb-default-embedding`,
|
|
55
|
+
provides the built-in TF-IDF embedding computation.
|
|
56
|
+
It depends only on `kbdb-shared` and requires no host
|
|
57
|
+
imports -- pure computation. It is loaded as an
|
|
58
|
+
independent WASM instance by the worker daemon
|
|
59
|
+
alongside the existing three modules.
|
|
52
60
|
|
|
53
61
|
The TypeScript daemon layer adds the vector path as an
|
|
54
62
|
independent retrieval step that runs after the lexical
|
|
@@ -93,15 +101,29 @@ interface EmbeddingProvider {
|
|
|
93
101
|
|
|
94
102
|
### Implementations
|
|
95
103
|
|
|
96
|
-
|
|
104
|
+
Four concrete implementations are specified.
|
|
105
|
+
|
|
106
|
+
**`TfidfEmbeddingProvider`** -- lightweight built-in
|
|
107
|
+
provider backed by the `kbdb-default-embedding` WASM
|
|
108
|
+
crate. Computes feature-hashed sublinear TF vectors
|
|
109
|
+
(256 dimensions, L2-normalized) using kbdb-shared's
|
|
110
|
+
tokenization pipeline (NFC normalization, Unicode
|
|
111
|
+
segmentation, camelCase splitting, stop-word removal,
|
|
112
|
+
synonym expansion, Snowball stemming). This ensures
|
|
113
|
+
token consistency between the lexical index and the
|
|
114
|
+
TF-IDF vectors -- hybrid search fuses two views of
|
|
115
|
+
the same vocabulary. The default when no `[embedding]`
|
|
116
|
+
section is present in `worker.toml`. Enables hybrid
|
|
117
|
+
search out of the box with zero configuration and
|
|
118
|
+
zero external dependencies. Uses FNV-1a feature
|
|
119
|
+
hashing with the sign trick (Weinberger et al., 2009).
|
|
120
|
+
Model name: `tfidf-v1`.
|
|
97
121
|
|
|
98
122
|
**`NullEmbeddingProvider`** -- no-op, returns empty
|
|
99
|
-
arrays.
|
|
100
|
-
|
|
101
|
-
`
|
|
102
|
-
|
|
103
|
-
results. Satisfies POLA: the absence of configuration
|
|
104
|
-
does not silently degrade results.
|
|
123
|
+
arrays. Available for explicit use but no longer the
|
|
124
|
+
default. Returns 0 dimensions, which causes
|
|
125
|
+
`assertProvider` to reject hybrid/vector search
|
|
126
|
+
requests with a structured error.
|
|
105
127
|
|
|
106
128
|
**`OnnxEmbeddingProvider`** -- local offline inference
|
|
107
129
|
using ONNX Runtime. Default model: `all-MiniLM-L6-v2`
|
|
@@ -305,9 +327,12 @@ The default is `lexical`. Existing integrations that do
|
|
|
305
327
|
not supply `search_mode` are unchanged. This is a
|
|
306
328
|
strict backward-compatibility guarantee.
|
|
307
329
|
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
330
|
+
When `[embedding]` is absent from `worker.toml`, the
|
|
331
|
+
`TfidfEmbeddingProvider` is active as the default.
|
|
332
|
+
Hybrid and vector search work out of the box using
|
|
333
|
+
feature-hashed TF vectors. If the user explicitly sets
|
|
334
|
+
`provider = "none"`, `NullEmbeddingProvider` is used
|
|
335
|
+
and hybrid/vector search returns a structured error:
|
|
311
336
|
|
|
312
337
|
```json
|
|
313
338
|
{
|
|
@@ -320,21 +345,21 @@ active) returns a structured error:
|
|
|
320
345
|
|
|
321
346
|
### CLI
|
|
322
347
|
|
|
323
|
-
The `search` subcommand gains an optional `--
|
|
348
|
+
The `search` subcommand gains an optional `--algo` flag:
|
|
324
349
|
|
|
325
350
|
```
|
|
326
351
|
kbdb search "database rationale"
|
|
327
|
-
kbdb search --
|
|
328
|
-
kbdb search --
|
|
352
|
+
kbdb search --algo hybrid "database rationale"
|
|
353
|
+
kbdb search --algo vector "database rationale"
|
|
329
354
|
```
|
|
330
355
|
|
|
331
|
-
Omitting `--
|
|
356
|
+
Omitting `--algo` defaults to `lexical`. The flag is
|
|
332
357
|
not shown in the short help unless an embedding provider
|
|
333
358
|
is configured.
|
|
334
359
|
|
|
335
360
|
### MCP
|
|
336
361
|
|
|
337
|
-
The `search` MCP tool gains an optional `
|
|
362
|
+
The `search` MCP tool gains an optional `algo` parameter
|
|
338
363
|
(string enum: `"lexical" | "vector" | "hybrid"`). The
|
|
339
364
|
parameter is documented as optional in the tool schema
|
|
340
365
|
with a default of `"lexical"`.
|
|
@@ -378,8 +403,10 @@ api_key_env = "KBDB_API_KEY"
|
|
|
378
403
|
| `api_key_env` | `"KBDB_API_KEY"` | Environment variable holding the API key |
|
|
379
404
|
|
|
380
405
|
When `[embedding]` is absent from `worker.toml`,
|
|
381
|
-
`provider` defaults to `"none"` and
|
|
382
|
-
is used
|
|
406
|
+
`provider` defaults to `"none"` and the built-in
|
|
407
|
+
`TfidfEmbeddingProvider` is used (256-dimensional
|
|
408
|
+
feature-hashed TF vectors). To fully disable
|
|
409
|
+
embeddings, set `provider = "none"` explicitly.
|
|
383
410
|
|
|
384
411
|
## Fallback Behaviour
|
|
385
412
|
|
|
@@ -388,11 +415,12 @@ in the search response with a specific warning field
|
|
|
388
415
|
rather than silently degrading.
|
|
389
416
|
|
|
390
417
|
**No embedding provider configured.**
|
|
391
|
-
`
|
|
392
|
-
|
|
393
|
-
|
|
394
|
-
|
|
395
|
-
|
|
418
|
+
The built-in `TfidfEmbeddingProvider` is active by
|
|
419
|
+
default, so hybrid and vector search work without
|
|
420
|
+
configuration. If the user explicitly sets
|
|
421
|
+
`provider = "none"`, requesting `hybrid` or `vector`
|
|
422
|
+
returns a structured error with code
|
|
423
|
+
`EMBEDDING_PROVIDER_NOT_CONFIGURED`.
|
|
396
424
|
|
|
397
425
|
**Stale vector index (model mismatch).**
|
|
398
426
|
When `vectors-meta.toml` model name does not match the
|
|
@@ -413,24 +441,25 @@ results until a rebuild is run.
|
|
|
413
441
|
|
|
414
442
|
## Provider Tradeoffs
|
|
415
443
|
|
|
416
|
-
| Aspect | ONNX (local) | API (cloud) |
|
|
417
|
-
|
|
418
|
-
| Latency | ~10 ms/embed | ~100 ms/embed |
|
|
419
|
-
| Offline | Yes | No |
|
|
420
|
-
| Cost | Free | Per-token |
|
|
421
|
-
| Quality | Good (384 dimensions) | Excellent (1536+ dimensions) |
|
|
422
|
-
| Setup | Model download on first use | API key required |
|
|
423
|
-
|
|
|
424
|
-
|
|
425
|
-
|
|
444
|
+
| Aspect | TF-IDF (built-in) | ONNX (local) | API (cloud) |
|
|
445
|
+
|---------------|---------------------------|---------------------------|------------------------------|
|
|
446
|
+
| Latency | < 1 ms/embed | ~10 ms/embed | ~100 ms/embed |
|
|
447
|
+
| Offline | Yes | Yes | No |
|
|
448
|
+
| Cost | Free | Free | Per-token |
|
|
449
|
+
| Quality | Adequate (256 dimensions) | Good (384 dimensions) | Excellent (1536+ dimensions) |
|
|
450
|
+
| Setup | None (default) | Model download on first use | API key required |
|
|
451
|
+
| Dependencies | None | onnxruntime-node | None |
|
|
452
|
+
| Dual runtime | Deno + Node.js >= 20 | Deno + Node.js >= 20 | Deno + Node.js >= 20 |
|
|
453
|
+
|
|
454
|
+
All providers implement the same `EmbeddingProvider`
|
|
426
455
|
interface. The daemon interacts only with the interface.
|
|
427
456
|
|
|
428
457
|
## Design Constraints
|
|
429
458
|
|
|
430
459
|
| Constraint | Details |
|
|
431
460
|
|-----------------------|------------------------------------------------|
|
|
432
|
-
| WASM purity | `query-parser.wasm` remains pure lexical computation with no I/O.
|
|
461
|
+
| WASM purity | `query-parser.wasm` remains pure lexical computation with no I/O. A new standalone crate, `kbdb-default-embedding`, is added for TF-IDF embedding but has no host imports. Existing WASM modules are unchanged. |
|
|
433
462
|
| Backward compatibility | Default `search_mode` is `lexical`. Callers that do not supply the parameter receive identical behaviour to today. |
|
|
434
463
|
| File-based storage | `vectors.idx` and `vectors-meta.toml` follow the same derived-and-rebuildable pattern as all other index files. |
|
|
435
464
|
| Dual runtime | `EmbeddingProvider` and all embedding logic must work on Deno and Node.js >= 20. No Node-specific or Deno-specific APIs in the implementation. |
|
|
436
|
-
| No new
|
|
465
|
+
| No new deps on existing WASM | The existing WASM modules (`fs-database.wasm`, `query-parser.wasm`, `kb-worker.wasm`) are not modified or recompiled. A new standalone module (`default-embedding.wasm`) is added, bringing the total to four WASM modules. |
|
package/docs/goals/mcp.md
CHANGED
|
@@ -28,7 +28,7 @@ identically.
|
|
|
28
28
|
|
|
29
29
|
```
|
|
30
30
|
kbdb mcp [--db <path>] [--timeout-ms <ms>]
|
|
31
|
-
[--retry <n>]
|
|
31
|
+
[--retry <n>] [--non-interactive]
|
|
32
32
|
```
|
|
33
33
|
|
|
34
34
|
All flags are optional. `--db` specifies the path
|
|
@@ -49,6 +49,8 @@ Resolution order:
|
|
|
49
49
|
| `KBDB_DB_DIR` | Fallback database directory when `--db` is not specified |
|
|
50
50
|
| `KBDB_DAEMON_TIMEOUT` | Request timeout fallback (ms); default 30 000 |
|
|
51
51
|
| `KBDB_RETRY` | Retry count fallback; default 5 |
|
|
52
|
+
| `KBDB_LOG_LEVEL` | Log verbosity: `debug`, `info`, `warning`, `error` |
|
|
53
|
+
| `KBDB_NON_INTERACTIVE`| Suppress prompts (`1` or `true`) |
|
|
52
54
|
|
|
53
55
|
See [CLI -- Run Modes](cli.md#run-modes) for how
|
|
54
56
|
the `kbdb` executable dispatches to MCP mode.
|
package/docs/goals/overview.md
CHANGED
|
@@ -114,7 +114,7 @@ for both the Rust engine and the TypeScript layer.
|
|
|
114
114
|
lands in the TypeScript source tree, and gets
|
|
115
115
|
bundled into `dist/`. Start here.
|
|
116
116
|
- **[Rust/WASM Modules](../modules/rust/overview.md)**
|
|
117
|
-
-- Cargo workspace,
|
|
117
|
+
-- Cargo workspace, 5 crates, module namespaces,
|
|
118
118
|
WASM exports/imports, size optimization, and
|
|
119
119
|
testing.
|
|
120
120
|
- **[TypeScript Modules](../modules/typescript/overview.md)**
|
|
@@ -195,7 +195,7 @@ in the `.kbdb` directory.
|
|
|
195
195
|
|
|
196
196
|
**Implementation status:** Synonym expansion is
|
|
197
197
|
implemented in the Rust pipeline
|
|
198
|
-
(`
|
|
198
|
+
(`shared/src/pipeline/synonyms.rs`). The
|
|
199
199
|
built-in map covers common technical synonyms.
|
|
200
200
|
The `query-guidance` default skill complements this
|
|
201
201
|
by instructing AI agents to rephrase queries with
|
|
@@ -58,9 +58,13 @@ with no further processing.
|
|
|
58
58
|
|
|
59
59
|
### Pass 2: Cosine (when embeddings available)
|
|
60
60
|
|
|
61
|
-
Pass 2 runs
|
|
62
|
-
`
|
|
63
|
-
|
|
61
|
+
Pass 2 runs when the active `EmbeddingProvider` has
|
|
62
|
+
non-zero dimensions and `vectors.idx` is present and
|
|
63
|
+
not stale. With the built-in `TfidfEmbeddingProvider`
|
|
64
|
+
as the default (256 dimensions), Pass 2 runs
|
|
65
|
+
automatically without configuration. It is skipped
|
|
66
|
+
only when embeddings are explicitly disabled
|
|
67
|
+
(`provider = "none"` in `worker.toml`).
|
|
64
68
|
|
|
65
69
|
For each gray-zone candidate from Pass 1, retrieve the
|
|
66
70
|
candidate's embedding from `vectors.idx`. If the
|
|
@@ -14,8 +14,9 @@ retrieval and caching management. It is accessible
|
|
|
14
14
|
|
|
15
15
|
The daemon:
|
|
16
16
|
|
|
17
|
-
1. Loads and compiles the
|
|
18
|
-
|
|
17
|
+
1. Loads and compiles the WASM modules once at startup
|
|
18
|
+
(`kb-worker` for database operations, `embedding`
|
|
19
|
+
for default TF-IDF vector computation).
|
|
19
20
|
2. Holds indexes and content objects in an in-memory
|
|
20
21
|
cache managed by the WASM module.
|
|
21
22
|
3. Listens for IPC requests from worker client
|
|
@@ -299,24 +300,28 @@ the daemon:
|
|
|
299
300
|
exists.
|
|
300
301
|
7. Calls `process.exit(0)`.
|
|
301
302
|
|
|
302
|
-
## WASM
|
|
303
|
+
## WASM Modules
|
|
303
304
|
|
|
304
|
-
The daemon loads
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
305
|
+
The daemon loads four WASM modules at startup. Three
|
|
306
|
+
share a single `WebAssembly.Memory` and are wired
|
|
307
|
+
together via imports; the fourth is a standalone
|
|
308
|
+
pure-computation module.
|
|
308
309
|
|
|
309
310
|
```
|
|
310
|
-
|
|
311
|
-
|
|
311
|
+
query-parser.wasm (standalone)
|
|
312
|
+
fs-database.wasm (standalone)
|
|
313
|
+
default-embedding.wasm (standalone, no host imports)
|
|
314
|
+
↓
|
|
315
|
+
kb-worker.wasm
|
|
316
|
+
(imports query-parser + fs-database)
|
|
312
317
|
↕
|
|
313
318
|
host I/O imports (TypeScript)
|
|
314
319
|
```
|
|
315
320
|
|
|
316
321
|
### WASM Module: kb-worker.wasm
|
|
317
322
|
|
|
318
|
-
`kb-worker.wasm` is the
|
|
319
|
-
|
|
323
|
+
`kb-worker.wasm` is the orchestration module. Its
|
|
324
|
+
responsibilities:
|
|
320
325
|
|
|
321
326
|
- **Query orchestration** -- Parses queries, performs
|
|
322
327
|
index lookups, scores and ranks results.
|
|
@@ -1523,14 +1528,20 @@ The worker daemon (`src/worker.ts` and its compiled
|
|
|
1523
1528
|
`kbdb-worker.js`) ships as part of the npm/JSR package.
|
|
1524
1529
|
No per-platform native binaries are required.
|
|
1525
1530
|
|
|
1526
|
-
The package includes
|
|
1527
|
-
|
|
1528
|
-
- `kb-worker/kbdb_worker_bg.wasm` --
|
|
1529
|
-
that
|
|
1530
|
-
|
|
1531
|
-
|
|
1532
|
-
|
|
1533
|
-
|
|
1534
|
-
|
|
1531
|
+
The package includes four WASM modules:
|
|
1532
|
+
|
|
1533
|
+
- `kb-worker/kbdb_worker_bg.wasm` -- orchestration
|
|
1534
|
+
module that imports query-parser and fs-database
|
|
1535
|
+
- `query-parser/query_parser_bg.wasm` -- tokenization
|
|
1536
|
+
and query parsing
|
|
1537
|
+
- `fs-database/fs_database_bg.wasm` -- filesystem
|
|
1538
|
+
storage engine
|
|
1539
|
+
- `default-embedding/kbdb_default_embedding_bg.wasm`
|
|
1540
|
+
-- built-in TF-IDF embedding (standalone, no host
|
|
1541
|
+
imports)
|
|
1542
|
+
|
|
1543
|
+
All are compiled from Rust via `wasm-pack` and work
|
|
1544
|
+
on all platforms. The daemon loads and compiles them
|
|
1545
|
+
at startup using the runtime's `WebAssembly` API. The
|
|
1535
1546
|
daemon runs on the same runtime (Node.js or Deno) as
|
|
1536
1547
|
the worker client that spawns it.
|