@dikolab/kbdb 0.4.0 → 0.4.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (99) hide show
  1. package/README.md +13 -5
  2. package/dist/{chunk-BJPSPNGX.mjs → chunk-BJXEVALU.mjs} +115 -1
  3. package/dist/chunk-BJXEVALU.mjs.map +7 -0
  4. package/dist/cli.cjs +462 -79
  5. package/dist/cli.cjs.map +4 -4
  6. package/dist/cli.mjs +356 -79
  7. package/dist/cli.mjs.map +4 -4
  8. package/dist/kbdb-worker.cjs +413 -53
  9. package/dist/kbdb-worker.cjs.map +4 -4
  10. package/dist/mod.cjs +230 -54
  11. package/dist/mod.cjs.map +4 -4
  12. package/dist/mod.mjs +229 -53
  13. package/dist/mod.mjs.map +4 -4
  14. package/dist/src/shared/cli/typings/cli-options.interface.d.ts +5 -0
  15. package/dist/src/shared/embedding/classes/tfidf-embedding-provider.class.d.ts +28 -0
  16. package/dist/src/shared/embedding/constants/tfidf-defaults.constant.d.ts +4 -0
  17. package/dist/src/shared/embedding/functions/create-embedding-provider.function.d.ts +5 -2
  18. package/dist/src/shared/embedding/index.d.ts +2 -0
  19. package/dist/src/shared/log/constants/log-colors.constant.d.ts +5 -0
  20. package/dist/src/shared/log/constants/log-defaults.constant.d.ts +3 -0
  21. package/dist/src/shared/log/constants/log-icons.constant.d.ts +3 -0
  22. package/dist/src/shared/log/functions/configure-log.function.d.ts +12 -0
  23. package/dist/src/shared/log/functions/log.function.d.ts +28 -0
  24. package/dist/src/shared/log/functions/parse-log-level.function.d.ts +10 -0
  25. package/dist/src/shared/log/functions/resolve-log-level.function.d.ts +12 -0
  26. package/dist/src/shared/log/functions/write-stderr.function.d.ts +10 -0
  27. package/dist/src/shared/log/index.d.ts +10 -0
  28. package/dist/src/shared/log/typings/log-config.interface.d.ts +7 -0
  29. package/dist/src/shared/log/typings/log-level.type.d.ts +7 -0
  30. package/dist/src/shared/log/typings/log-output.interface.d.ts +4 -0
  31. package/dist/src/shared/version/constants/version.constant.d.ts +1 -1
  32. package/dist/src/shared/worker-client/classes/worker-client.class.d.ts +7 -0
  33. package/dist/src/shared/worker-client/functions/retry-on-timeout.function.d.ts +1 -0
  34. package/dist/src/shared/worker-daemon/classes/worker-daemon.class.d.ts +1 -0
  35. package/dist/src/shared/worker-daemon/functions/run-worker.function.d.ts +6 -1
  36. package/dist/src/shared/worker-daemon/typings/wasm-modules.interface.d.ts +5 -0
  37. package/dist/wasm/default-embedding/kbdb_default_embedding.d.ts +55 -0
  38. package/dist/wasm/default-embedding/kbdb_default_embedding.js +156 -0
  39. package/dist/wasm/default-embedding/kbdb_default_embedding_bg.wasm +0 -0
  40. package/dist/wasm/default-embedding/kbdb_default_embedding_bg.wasm.d.ts +12 -0
  41. package/dist/wasm/default-embedding/package.json +13 -0
  42. package/dist/wasm/fs-database/kbdb_fs_database_bg.wasm +0 -0
  43. package/dist/wasm/fs-migration/kbdb_fs_migration.d.ts +38 -0
  44. package/dist/wasm/fs-migration/kbdb_fs_migration.js +130 -0
  45. package/dist/wasm/fs-migration/kbdb_fs_migration_bg.wasm +0 -0
  46. package/dist/wasm/fs-migration/kbdb_fs_migration_bg.wasm.d.ts +9 -0
  47. package/dist/wasm/fs-migration/package.json +13 -0
  48. package/dist/wasm/kb-worker/kbdb_worker_bg.wasm +0 -0
  49. package/dist/wasm/query-parser/kbdb_query_parser_bg.wasm +0 -0
  50. package/dist/worker.mjs +306 -54
  51. package/dist/worker.mjs.map +4 -4
  52. package/docs/details/agent-tooling.md +10 -0
  53. package/docs/details/cli.md +29 -13
  54. package/docs/details/knowledge-base.md +15 -8
  55. package/docs/details/mcp-server.md +16 -1
  56. package/docs/details/search-and-ranking.md +6 -2
  57. package/docs/goals/cli.md +33 -4
  58. package/docs/goals/content-parser.md +1 -1
  59. package/docs/goals/hybrid-search.md +71 -42
  60. package/docs/goals/mcp.md +3 -1
  61. package/docs/goals/overview.md +1 -1
  62. package/docs/goals/query-parser.md +1 -1
  63. package/docs/goals/semantic-dedup.md +7 -3
  64. package/docs/goals/worker-daemon.md +31 -20
  65. package/docs/modules/overview.md +43 -22
  66. package/docs/modules/rust/embedding/module.md +91 -0
  67. package/docs/modules/rust/overview.md +56 -21
  68. package/docs/modules/rust/query-parser/functions/module.md +7 -5
  69. package/docs/modules/rust/query-parser/functions/pipeline.md +6 -138
  70. package/docs/modules/rust/query-parser/module.md +16 -22
  71. package/docs/modules/rust/query-parser/types/module.md +5 -4
  72. package/docs/modules/rust/query-parser/types/token.md +5 -1
  73. package/docs/modules/rust/shared/content-parser/functions.md +1 -1
  74. package/docs/modules/rust/shared/memory/module.md +1 -1
  75. package/docs/modules/rust/shared/module.md +33 -5
  76. package/docs/modules/rust/shared/pipeline/functions.md +141 -0
  77. package/docs/modules/rust/shared/pipeline/module.md +62 -0
  78. package/docs/modules/rust/shared/pipeline/types.md +43 -0
  79. package/docs/modules/rust/shared/section/module.md +1 -1
  80. package/docs/modules/rust/shared/section/types.md +1 -1
  81. package/docs/modules/typescript/cli.md +4 -2
  82. package/docs/modules/typescript/kbdb-worker.md +20 -5
  83. package/docs/modules/typescript/overview.md +28 -6
  84. package/docs/modules/typescript/shared/cli/functions.md +15 -4
  85. package/docs/modules/typescript/shared/cli/typings.md +21 -0
  86. package/docs/modules/typescript/shared/embedding/functions.md +30 -3
  87. package/docs/modules/typescript/shared/embedding/module.md +10 -4
  88. package/docs/modules/typescript/shared/embedding/typings.md +7 -5
  89. package/docs/modules/typescript/shared/log/constants.md +82 -0
  90. package/docs/modules/typescript/shared/log/functions.md +131 -0
  91. package/docs/modules/typescript/shared/log/module.md +153 -0
  92. package/docs/modules/typescript/shared/log/typings.md +82 -0
  93. package/docs/modules/typescript/shared/module.md +22 -6
  94. package/docs/modules/typescript/shared/worker-daemon/functions.md +19 -6
  95. package/docs/modules/typescript/shared/worker-daemon/module.md +5 -2
  96. package/docs/modules/typescript/shared/worker-daemon/typings.md +13 -7
  97. package/docs/release-notes/0.4.1.md +149 -0
  98. package/package.json +2 -2
  99. package/dist/chunk-BJPSPNGX.mjs.map +0 -7
@@ -37,6 +37,16 @@ file for your tool.
37
37
  `@dikolab/kbdb` with `deno` / `run` / `-A` /
38
38
  `jsr:@dikolab/kbdb/cli`.
39
39
 
40
+ Environment variables:
41
+
42
+ | Variable | Description |
43
+ |----------|-------------|
44
+ | `KBDB_DB_DIR` | Database path (replaces `--db`) |
45
+ | `KBDB_LOG_LEVEL` | `debug`, `info`, `warning`, `error` |
46
+ | `KBDB_NON_INTERACTIVE` | Suppress prompts (`1` / `true`) |
47
+ | `KBDB_DAEMON_TIMEOUT` | Request timeout in ms |
48
+ | `KBDB_RETRY` | Max retries on timeout |
49
+
40
50
  See the [MCP Server Guide](mcp-server.md) for
41
51
  environment variable config, multiple knowledge
42
52
  bases, and the full 20-tool reference.
@@ -21,6 +21,10 @@ deno install -Agf jsr:@dikolab/kbdb/cli
21
21
  kbdb [options]
22
22
  kbdb <command> [options] ...[parameters]
23
23
  kbdb <command> <subcommand> [options] ...[parameters]
24
+ kbdb --version
25
+ kbdb -v <command> (verbose: warnings)
26
+ kbdb -vv <command> (verbose: info)
27
+ kbdb -vvv <command> (verbose: debug)
24
28
  ```
25
29
 
26
30
  ## Command Groups
@@ -74,11 +78,15 @@ docid and want rendered Markdown output.
74
78
  | Option | Short | Default | Description |
75
79
  |--------|-------|---------|-------------|
76
80
  | `--help` | `-h` | | Print help text and exit |
77
- | `--version` | `-v` | | Print version and exit |
81
+ | `--version` | | | Print version and exit |
82
+ | `--verbose` | `-v` | | Set log level to WARNING (errors + warnings) |
83
+ | `--verbose-info` | `-vv` | | Set log level to INFO |
84
+ | `--verbose-debug` | `-vvv` | | Set log level to DEBUG (all messages) |
78
85
  | `--db <path>` | | cwd | Database directory path (optional) |
79
86
  | `--format <fmt>` | `-f` | `json` (text when interactive) | Output format (`json`, `text`, `mcp`) |
80
87
  | `--timeout-ms <ms>` | | `30000` | Per-request daemon timeout in milliseconds |
81
88
  | `--retry <n>` | | `1` | Max retries with daemon respawn on timeout |
89
+ | `--non-interactive` | | | Suppress prompts; use defaults |
82
90
 
83
91
  ### `--db <path>`
84
92
 
@@ -116,25 +124,23 @@ These options apply to the `search` command.
116
124
  | Algorithm | Description |
117
125
  |-----------|-------------|
118
126
  | `lexical` | Keyword matching with stemming and synonyms |
119
- | `vector` | AI-powered similarity search using embeddings (requires a configured embedding provider) |
127
+ | `vector` | Similarity search using embeddings |
120
128
  | `hybrid` | Combines keyword and similarity results for best coverage |
121
129
 
122
130
  ### Configuring Hybrid Search
123
131
 
124
- Hybrid and vector search require an embedding provider.
125
- kbdb supports pluggable providers configured via the
126
- `EmbeddingConfig` interface:
132
+ Hybrid and vector search use a built-in TF-IDF
133
+ embedding provider by default -- no configuration
134
+ needed. For higher quality embeddings, configure a
135
+ provider via `worker.toml`:
127
136
 
128
137
  | Provider | Description |
129
138
  |----------|-------------|
130
- | `none` | No embeddings; hybrid and vector return an error |
139
+ | *(none)* | Built-in TF-IDF (256 dimensions, works offline) |
140
+ | `none` | Explicitly disabled; hybrid and vector return an error |
131
141
  | `onnx` | Local ONNX model (no network required) |
132
142
  | `api` | Remote embedding API (OpenAI-compatible endpoint) |
133
143
 
134
- When no provider is configured, `--algo hybrid` and
135
- `--algo vector` return an error prompting you to
136
- configure an embedding provider.
137
-
138
144
  ### Search Modes
139
145
 
140
146
  | Mode | Description |
@@ -720,6 +726,8 @@ kbdb agent delete agent-code-reviewer
720
726
  | `KBDB_DB_DIR` | CLI + MCP | cwd (CLI) / none (MCP) | Fallback database directory path when `--db` is not specified. |
721
727
  | `KBDB_DAEMON_TIMEOUT` | CLI + MCP | `30000` | Per-request daemon timeout in milliseconds. Overridden by `--timeout-ms`. |
722
728
  | `KBDB_RETRY` | CLI + MCP | `1` | Max retries with daemon respawn on timeout. Overridden by `--retry`. |
729
+ | `KBDB_LOG_LEVEL` | CLI + MCP + Daemon | `error` | Log verbosity: `debug`, `info`, `warning`, `error`. Overridden by `-v`/`-vv`/`-vvv`. |
730
+ | `KBDB_NON_INTERACTIVE` | CLI + MCP + Daemon | off | Suppress interactive prompts. Accepts `1` or `true`. Overridden by `--non-interactive`. |
723
731
 
724
732
  ## Exit Codes
725
733
 
@@ -738,6 +746,12 @@ Before any data command (`learn`, `unlearn`,
738
746
  prompts to create one.
739
747
  2. **No `.kbdb` found + non-interactive** -- prints
740
748
  error and exits.
749
+
750
+ The `--non-interactive` flag (or
751
+ `KBDB_NON_INTERACTIVE=1`) forces the non-interactive
752
+ path regardless of terminal type. This is useful in
753
+ scripts, CI pipelines, and automated testing.
754
+
741
755
  3. **`.kbdb` found** -- runs a version gate before
742
756
  proceeding. If migration is needed, prints an
743
757
  error directing the user to run `kbdb db migrate`.
@@ -753,9 +767,11 @@ database creation.
753
767
  modules.
754
768
  - **stdout** contains only command output. No
755
769
  banners, progress bars, or decorative text.
756
- - **stderr** contains only errors (and deprecation
757
- warnings for the legacy `--init`/`--migrate`
758
- flags).
770
+ - **stderr** contains log messages. Default level is
771
+ ERROR (errors only). Use `-v`/`-vv`/`-vvv` or
772
+ `KBDB_LOG_LEVEL` to increase verbosity. Log lines
773
+ are prefixed with per-level icons (✖⚠ℹ▸) and ANSI
774
+ colors.
759
775
  - Works on both Deno and Node.js (>= 20).
760
776
  - The `--init` and `--migrate` flags are deprecated.
761
777
  Use `kbdb db init` and `kbdb db migrate` instead.
@@ -74,11 +74,11 @@ of creating duplicates.
74
74
  | `--algo hybrid` | hybrid | Keyword + AI similarity |
75
75
  | `--algo vector` | vector | AI similarity only |
76
76
 
77
- Hybrid and vector modes need an embedding provider.
78
- Without one, they return an error prompting you to
79
- configure an embedding provider.
80
- See the [CLI Reference](cli.md) for provider
81
- configuration.
77
+ Hybrid and vector modes work out of the box using a
78
+ built-in TF-IDF embedding provider. For higher
79
+ quality, configure an ONNX or API provider in
80
+ `worker.toml`. See the [CLI Reference](cli.md) for
81
+ provider configuration.
82
82
 
83
83
  When a query returns no exact matches, kbdb
84
84
  automatically loosens the search (OR instead of AND)
@@ -159,10 +159,17 @@ two global flags:
159
159
  |------|---------|-------------|
160
160
  | `--timeout-ms <ms>` | `30000` | Per-request daemon timeout |
161
161
  | `--retry <n>` | `1` | Retries with daemon respawn on timeout |
162
+ | `--non-interactive` | off | Suppress prompts; use defaults |
162
163
 
163
- These can also be set via the `KBDB_DAEMON_TIMEOUT`
164
- and `KBDB_RETRY` environment variables. The flags
165
- take priority over env vars.
164
+ These can also be set via environment variables.
165
+ The flags take priority over env vars.
166
+
167
+ | Variable | Description |
168
+ |----------|-------------|
169
+ | `KBDB_DAEMON_TIMEOUT` | Per-request daemon timeout (ms) |
170
+ | `KBDB_RETRY` | Max retry count |
171
+ | `KBDB_LOG_LEVEL` | Log verbosity: `debug`, `info`, `warning`, `error` |
172
+ | `KBDB_NON_INTERACTIVE` | Suppress prompts (`1` or `true`) |
166
173
 
167
174
  ## Deep Dives
168
175
 
@@ -169,6 +169,20 @@ Maximum number of retries with daemon respawn when
169
169
  a request times out. Defaults to `1`. Overridden
170
170
  by the `--retry` flag.
171
171
 
172
+ ### `KBDB_LOG_LEVEL`
173
+
174
+ Log verbosity: `debug`, `info`, `warning`, `error`
175
+ (case-insensitive). Defaults to `error`. Overridden
176
+ by `-v`/`-vv`/`-vvv` flags. The level is forwarded
177
+ to the worker daemon via `--log-level`.
178
+
179
+ ### `KBDB_NON_INTERACTIVE`
180
+
181
+ Suppress interactive prompts. Accepts `1` or
182
+ `true`. The MCP server does not prompt interactively
183
+ (it reads JSON-RPC from stdin), but the variable is
184
+ inherited by the worker daemon for consistency.
185
+
172
186
  These variables can be set in the `env` block of
173
187
  your MCP configuration:
174
188
 
@@ -181,7 +195,8 @@ your MCP configuration:
181
195
  "env": {
182
196
  "KBDB_DB_DIR": "/path/to/.kbdb",
183
197
  "KBDB_DAEMON_TIMEOUT": "60000",
184
- "KBDB_RETRY": "2"
198
+ "KBDB_RETRY": "2",
199
+ "KBDB_LOG_LEVEL": "warning"
185
200
  }
186
201
  }
187
202
  }
@@ -567,5 +567,9 @@ into a score (`1 / (k + rank)` with k=60), then
567
567
  summing scores across both lists. Results that rank
568
568
  well in both lists get the highest combined scores.
569
569
 
570
- Hybrid mode requires an embedding provider. Without
571
- one, it returns an error.
570
+ Hybrid mode uses a built-in TF-IDF embedding provider
571
+ by default. Configure `provider = "onnx"` or
572
+ `provider = "api"` in `worker.toml` for higher
573
+ quality embeddings. Setting `provider = "none"`
574
+ explicitly disables embeddings and returns an error
575
+ for hybrid and vector searches.
package/docs/goals/cli.md CHANGED
@@ -49,7 +49,10 @@ All commands and their canonical forms:
49
49
 
50
50
  ```
51
51
  kbdb --help | -h
52
- kbdb --version | -v
52
+ kbdb --version
53
+ kbdb -v <command> (verbose: warnings)
54
+ kbdb -vv <command> (verbose: info)
55
+ kbdb -vvv <command> (verbose: debug)
53
56
  kbdb learn [options] <path | ->...
54
57
  kbdb unlearn [options] <kbid>...
55
58
  kbdb search [options] <query>...
@@ -120,11 +123,15 @@ These options apply to all commands.
120
123
  | Option | Short | Description |
121
124
  |--------------------|-------|--------------------------|
122
125
  | `--help` | `-h` | Print help text and exit |
123
- | `--version` | `-v` | Print version and exit |
126
+ | `--version` | | Print version and exit |
127
+ | `--verbose` | `-v` | Set log level to WARNING (errors + warnings) |
128
+ | `--verbose-info` | `-vv` | Set log level to INFO |
129
+ | `--verbose-debug` | `-vvv`| Set log level to DEBUG (all messages) |
124
130
  | `--db <path>` | | Database directory path (optional) |
125
131
  | `--format <fmt>` | `-f` | Output format (json default; text when stdout is a TTY) |
126
132
  | `--timeout-ms <ms>`| | Request timeout in ms |
127
133
  | `--retry <n>` | | Max retries on timeout |
134
+ | `--non-interactive` | | Suppress prompts; use defaults |
128
135
 
129
136
  ### `--help`, `-h`
130
137
 
@@ -139,16 +146,31 @@ kbdb -h
139
146
  When no command is given (`kbdb` with no arguments),
140
147
  the help text is printed as the default behaviour.
141
148
 
142
- ### `--version`, `-v`
149
+ ### `--version`
143
150
 
144
151
  Print the package version string and exit. No database
145
152
  connection is made.
146
153
 
147
154
  ```
148
155
  kbdb --version
149
- kbdb -v
150
156
  ```
151
157
 
158
+ ### `--verbose`, `-v` / `-vv` / `-vvv`
159
+
160
+ Increase log verbosity on stderr. Each additional `v`
161
+ raises the level by one step.
162
+
163
+ ```
164
+ kbdb -v <command> # WARNING: errors + warnings
165
+ kbdb -vv <command> # INFO: errors, warnings, info
166
+ kbdb -vvv <command> # DEBUG: all messages
167
+ ```
168
+
169
+ The default level is ERROR. The `KBDB_LOG_LEVEL`
170
+ environment variable sets the base level; CLI flags
171
+ override it. Log lines are prefixed with per-level
172
+ icons (✖⚠ℹ▸) and ANSI colors.
173
+
152
174
  ### `--db <path>`
153
175
 
154
176
  Set the path to the directory that contains (or will
@@ -1709,6 +1731,13 @@ storage directory via `findDb()`.
1709
1731
  - If stdin is not a TTY (piped / CI): print error to
1710
1732
  stderr, exit 1. No prompt.
1711
1733
 
1734
+ 2b. **Non-interactive override** -- when
1735
+ `--non-interactive` is passed or
1736
+ `KBDB_NON_INTERACTIVE` is set to `1` or `true`,
1737
+ the CLI overrides `isStdinTty()` to return
1738
+ `false`. This forces the non-interactive path
1739
+ (error + exit 1) regardless of terminal type.
1740
+
1712
1741
  3. **Version check gate** -- when `.kbdb` exists, the
1713
1742
  CLI calls `client.checkVersion(dbDir)` before
1714
1743
  dispatching any data command:
@@ -138,7 +138,7 @@ preserved; the caller should normalise whitespace
138
138
  if needed.
139
139
 
140
140
  This function replaces the current implementation
141
- in `query-parser/src/pipeline/markers.rs`. The
141
+ in `shared/src/pipeline/markers.rs`. The
142
142
  existing `strip_content_markers()` recognizes only
143
143
  `$(image:)` and `$(code:)`. The shared version
144
144
  recognizes all six marker types.
@@ -34,8 +34,9 @@ both modes.
34
34
  ## Architecture
35
35
 
36
36
  Two retrieval paths run in parallel. The vector path
37
- runs entirely in TypeScript. No WASM code is added or
38
- modified.
37
+ runs in TypeScript, backed by a lightweight WASM crate
38
+ (`kbdb-default-embedding`) for the built-in TF-IDF
39
+ provider.
39
40
 
40
41
  ```
41
42
  Query → ┌─ Lexical path (WASM) ─→ BM25F ranked list ─┐
@@ -43,12 +44,19 @@ Query → ┌─ Lexical path (WASM) ─→ BM25F ranked list ─┐
43
44
  └─ Vector path (TS) ─→ Cosine ranked list ──┘
44
45
  ```
45
46
 
46
- The WASM boundary is untouched. `query-parser.wasm`
47
- remains a pure-computation lexical module. `kb-worker.wasm`
48
- continues to orchestrate lexical retrieval via
49
- `fs-database.wasm` exactly as today. No new WASM crate,
50
- no new WASM export, and no new host import is needed for
51
- the vector path.
47
+ The existing WASM modules are untouched.
48
+ `query-parser.wasm` remains a pure-computation lexical
49
+ module. `kb-worker.wasm` continues to orchestrate
50
+ lexical retrieval via `fs-database.wasm` exactly as
51
+ today. No new exports or host imports are added to
52
+ any existing WASM module.
53
+
54
+ A new standalone WASM crate, `kbdb-default-embedding`,
55
+ provides the built-in TF-IDF embedding computation.
56
+ It depends only on `kbdb-shared` and requires no host
57
+ imports -- pure computation. It is loaded as an
58
+ independent WASM instance by the worker daemon
59
+ alongside the existing three modules.
52
60
 
53
61
  The TypeScript daemon layer adds the vector path as an
54
62
  independent retrieval step that runs after the lexical
@@ -93,15 +101,29 @@ interface EmbeddingProvider {
93
101
 
94
102
  ### Implementations
95
103
 
96
- Three concrete implementations are specified.
104
+ Four concrete implementations are specified.
105
+
106
+ **`TfidfEmbeddingProvider`** -- lightweight built-in
107
+ provider backed by the `kbdb-default-embedding` WASM
108
+ crate. Computes feature-hashed sublinear TF vectors
109
+ (256 dimensions, L2-normalized) using kbdb-shared's
110
+ tokenization pipeline (NFC normalization, Unicode
111
+ segmentation, camelCase splitting, stop-word removal,
112
+ synonym expansion, Snowball stemming). This ensures
113
+ token consistency between the lexical index and the
114
+ TF-IDF vectors -- hybrid search fuses two views of
115
+ the same vocabulary. The default when no `[embedding]`
116
+ section is present in `worker.toml`. Enables hybrid
117
+ search out of the box with zero configuration and
118
+ zero external dependencies. Uses FNV-1a feature
119
+ hashing with the sign trick (Weinberger et al., 2009).
120
+ Model name: `tfidf-v1`.
97
121
 
98
122
  **`NullEmbeddingProvider`** -- no-op, returns empty
99
- arrays. The default when no `[embedding]` section is
100
- present in `worker.toml`. Requesting `vector` or
101
- `hybrid` search mode with this provider returns a
102
- structured error rather than silently returning empty
103
- results. Satisfies POLA: the absence of configuration
104
- does not silently degrade results.
123
+ arrays. Available for explicit use but no longer the
124
+ default. Returns 0 dimensions, which causes
125
+ `assertProvider` to reject hybrid/vector search
126
+ requests with a structured error.
105
127
 
106
128
  **`OnnxEmbeddingProvider`** -- local offline inference
107
129
  using ONNX Runtime. Default model: `all-MiniLM-L6-v2`
@@ -305,9 +327,12 @@ The default is `lexical`. Existing integrations that do
305
327
  not supply `search_mode` are unchanged. This is a
306
328
  strict backward-compatibility guarantee.
307
329
 
308
- Requesting `vector` or `hybrid` when no embedding
309
- provider is configured (i.e. `NullEmbeddingProvider` is
310
- active) returns a structured error:
330
+ When `[embedding]` is absent from `worker.toml`, the
331
+ `TfidfEmbeddingProvider` is active as the default.
332
+ Hybrid and vector search work out of the box using
333
+ feature-hashed TF vectors. If the user explicitly sets
334
+ `provider = "none"`, `NullEmbeddingProvider` is used
335
+ and hybrid/vector search returns a structured error:
311
336
 
312
337
  ```json
313
338
  {
@@ -320,21 +345,21 @@ active) returns a structured error:
320
345
 
321
346
  ### CLI
322
347
 
323
- The `search` subcommand gains an optional `--mode` flag:
348
+ The `search` subcommand gains an optional `--algo` flag:
324
349
 
325
350
  ```
326
351
  kbdb search "database rationale"
327
- kbdb search --mode hybrid "database rationale"
328
- kbdb search --mode vector "database rationale"
352
+ kbdb search --algo hybrid "database rationale"
353
+ kbdb search --algo vector "database rationale"
329
354
  ```
330
355
 
331
- Omitting `--mode` defaults to `lexical`. The flag is
356
+ Omitting `--algo` defaults to `lexical`. The flag is
332
357
  not shown in the short help unless an embedding provider
333
358
  is configured.
334
359
 
335
360
  ### MCP
336
361
 
337
- The `search` MCP tool gains an optional `mode` parameter
362
+ The `search` MCP tool gains an optional `algo` parameter
338
363
  (string enum: `"lexical" | "vector" | "hybrid"`). The
339
364
  parameter is documented as optional in the tool schema
340
365
  with a default of `"lexical"`.
@@ -378,8 +403,10 @@ api_key_env = "KBDB_API_KEY"
378
403
  | `api_key_env` | `"KBDB_API_KEY"` | Environment variable holding the API key |
379
404
 
380
405
  When `[embedding]` is absent from `worker.toml`,
381
- `provider` defaults to `"none"` and `NullEmbeddingProvider`
382
- is used.
406
+ `provider` defaults to `"none"` and the built-in
407
+ `TfidfEmbeddingProvider` is used (256-dimensional
408
+ feature-hashed TF vectors). To fully disable
409
+ embeddings, set `provider = "none"` explicitly.
383
410
 
384
411
  ## Fallback Behaviour
385
412
 
@@ -388,11 +415,12 @@ in the search response with a specific warning field
388
415
  rather than silently degrading.
389
416
 
390
417
  **No embedding provider configured.**
391
- `search_mode` defaults to `lexical`. Requesting `hybrid`
392
- or `vector` returns a structured error with code
393
- `EMBEDDING_PROVIDER_NOT_CONFIGURED`. No silent fallback
394
- occurs -- the caller receives an actionable error
395
- message.
418
+ The built-in `TfidfEmbeddingProvider` is active by
419
+ default, so hybrid and vector search work without
420
+ configuration. If the user explicitly sets
421
+ `provider = "none"`, requesting `hybrid` or `vector`
422
+ returns a structured error with code
423
+ `EMBEDDING_PROVIDER_NOT_CONFIGURED`.
396
424
 
397
425
  **Stale vector index (model mismatch).**
398
426
  When `vectors-meta.toml` model name does not match the
@@ -413,24 +441,25 @@ results until a rebuild is run.
413
441
 
414
442
  ## Provider Tradeoffs
415
443
 
416
- | Aspect | ONNX (local) | API (cloud) |
417
- |---------------|---------------------------|------------------------------|
418
- | Latency | ~10 ms/embed | ~100 ms/embed |
419
- | Offline | Yes | No |
420
- | Cost | Free | Per-token |
421
- | Quality | Good (384 dimensions) | Excellent (1536+ dimensions) |
422
- | Setup | Model download on first use | API key required |
423
- | Dual runtime | Deno + Node.js >= 20 | Deno + Node.js >= 20 |
424
-
425
- Both providers implement the same `EmbeddingProvider`
444
+ | Aspect | TF-IDF (built-in) | ONNX (local) | API (cloud) |
445
+ |---------------|---------------------------|---------------------------|------------------------------|
446
+ | Latency | < 1 ms/embed | ~10 ms/embed | ~100 ms/embed |
447
+ | Offline | Yes | Yes | No |
448
+ | Cost | Free | Free | Per-token |
449
+ | Quality | Adequate (256 dimensions) | Good (384 dimensions) | Excellent (1536+ dimensions) |
450
+ | Setup | None (default) | Model download on first use | API key required |
451
+ | Dependencies | None | onnxruntime-node | None |
452
+ | Dual runtime | Deno + Node.js >= 20 | Deno + Node.js >= 20 | Deno + Node.js >= 20 |
453
+
454
+ All providers implement the same `EmbeddingProvider`
426
455
  interface. The daemon interacts only with the interface.
427
456
 
428
457
  ## Design Constraints
429
458
 
430
459
  | Constraint | Details |
431
460
  |-----------------------|------------------------------------------------|
432
- | WASM purity | `query-parser.wasm` remains pure lexical computation with no I/O. No new WASM crate or export is added. |
461
+ | WASM purity | `query-parser.wasm` remains pure lexical computation with no I/O. A new standalone crate, `kbdb-default-embedding`, is added for TF-IDF embedding but has no host imports. Existing WASM modules are unchanged. |
433
462
  | Backward compatibility | Default `search_mode` is `lexical`. Callers that do not supply the parameter receive identical behaviour to today. |
434
463
  | File-based storage | `vectors.idx` and `vectors-meta.toml` follow the same derived-and-rebuildable pattern as all other index files. |
435
464
  | Dual runtime | `EmbeddingProvider` and all embedding logic must work on Deno and Node.js >= 20. No Node-specific or Deno-specific APIs in the implementation. |
436
- | No new WASM deps | All new dependencies are TypeScript-side only. The WASM modules (`fs-database.wasm`, `query-parser.wasm`, `kb-worker.wasm`) are not recompiled for this feature. |
465
+ | No new deps on existing WASM | The existing WASM modules (`fs-database.wasm`, `query-parser.wasm`, `kb-worker.wasm`) are not modified or recompiled. A new standalone module (`default-embedding.wasm`) is added, bringing the total to four WASM modules. |
package/docs/goals/mcp.md CHANGED
@@ -28,7 +28,7 @@ identically.
28
28
 
29
29
  ```
30
30
  kbdb mcp [--db <path>] [--timeout-ms <ms>]
31
- [--retry <n>]
31
+ [--retry <n>] [--non-interactive]
32
32
  ```
33
33
 
34
34
  All flags are optional. `--db` specifies the path
@@ -49,6 +49,8 @@ Resolution order:
49
49
  | `KBDB_DB_DIR` | Fallback database directory when `--db` is not specified |
50
50
  | `KBDB_DAEMON_TIMEOUT` | Request timeout fallback (ms); default 30 000 |
51
51
  | `KBDB_RETRY` | Retry count fallback; default 5 |
52
+ | `KBDB_LOG_LEVEL` | Log verbosity: `debug`, `info`, `warning`, `error` |
53
+ | `KBDB_NON_INTERACTIVE`| Suppress prompts (`1` or `true`) |
52
54
 
53
55
  See [CLI -- Run Modes](cli.md#run-modes) for how
54
56
  the `kbdb` executable dispatches to MCP mode.
@@ -114,7 +114,7 @@ for both the Rust engine and the TypeScript layer.
114
114
  lands in the TypeScript source tree, and gets
115
115
  bundled into `dist/`. Start here.
116
116
  - **[Rust/WASM Modules](../modules/rust/overview.md)**
117
- -- Cargo workspace, 4 crates, module namespaces,
117
+ -- Cargo workspace, 5 crates, module namespaces,
118
118
  WASM exports/imports, size optimization, and
119
119
  testing.
120
120
  - **[TypeScript Modules](../modules/typescript/overview.md)**
@@ -195,7 +195,7 @@ in the `.kbdb` directory.
195
195
 
196
196
  **Implementation status:** Synonym expansion is
197
197
  implemented in the Rust pipeline
198
- (`query-parser/src/pipeline/synonyms.rs`). The
198
+ (`shared/src/pipeline/synonyms.rs`). The
199
199
  built-in map covers common technical synonyms.
200
200
  The `query-guidance` default skill complements this
201
201
  by instructing AI agents to rephrase queries with
@@ -58,9 +58,13 @@ with no further processing.
58
58
 
59
59
  ### Pass 2: Cosine (when embeddings available)
60
60
 
61
- Pass 2 runs only when an `EmbeddingProvider` other than
62
- `NullEmbeddingProvider` is active and
63
- `vectors.idx` is present and not stale.
61
+ Pass 2 runs when the active `EmbeddingProvider` has
62
+ non-zero dimensions and `vectors.idx` is present and
63
+ not stale. With the built-in `TfidfEmbeddingProvider`
64
+ as the default (256 dimensions), Pass 2 runs
65
+ automatically without configuration. It is skipped
66
+ only when embeddings are explicitly disabled
67
+ (`provider = "none"` in `worker.toml`).
64
68
 
65
69
  For each gray-zone candidate from Pass 1, retrieve the
66
70
  candidate's embedding from `vectors.idx`. If the
@@ -14,8 +14,9 @@ retrieval and caching management. It is accessible
14
14
 
15
15
  The daemon:
16
16
 
17
- 1. Loads and compiles the `kb-worker` WASM module once
18
- at startup.
17
+ 1. Loads and compiles the WASM modules once at startup
18
+ (`kb-worker` for database operations, `embedding`
19
+ for default TF-IDF vector computation).
19
20
  2. Holds indexes and content objects in an in-memory
20
21
  cache managed by the WASM module.
21
22
  3. Listens for IPC requests from worker client
@@ -299,24 +300,28 @@ the daemon:
299
300
  exists.
300
301
  7. Calls `process.exit(0)`.
301
302
 
302
- ## WASM Module
303
+ ## WASM Modules
303
304
 
304
- The daemon loads a single WASM module: `kb-worker.wasm`
305
- (file: `kbdb_worker_bg.wasm`). This module statically
306
- links the `fs-database` and `query-parser` crates, so
307
- only one module needs to be compiled and instantiated.
305
+ The daemon loads four WASM modules at startup. Three
306
+ share a single `WebAssembly.Memory` and are wired
307
+ together via imports; the fourth is a standalone
308
+ pure-computation module.
308
309
 
309
310
  ```
310
- kbdb_worker_bg.wasm
311
- (statically links fs-database + query-parser)
311
+ query-parser.wasm (standalone)
312
+ fs-database.wasm (standalone)
313
+ default-embedding.wasm (standalone, no host imports)
314
+
315
+ kb-worker.wasm
316
+ (imports query-parser + fs-database)
312
317
 
313
318
  host I/O imports (TypeScript)
314
319
  ```
315
320
 
316
321
  ### WASM Module: kb-worker.wasm
317
322
 
318
- `kb-worker.wasm` is the single module loaded by the
319
- daemon. Its responsibilities:
323
+ `kb-worker.wasm` is the orchestration module. Its
324
+ responsibilities:
320
325
 
321
326
  - **Query orchestration** -- Parses queries, performs
322
327
  index lookups, scores and ranks results.
@@ -1523,14 +1528,20 @@ The worker daemon (`src/worker.ts` and its compiled
1523
1528
  `kbdb-worker.js`) ships as part of the npm/JSR package.
1524
1529
  No per-platform native binaries are required.
1525
1530
 
1526
- The package includes one WASM module:
1527
-
1528
- - `kb-worker/kbdb_worker_bg.wasm` -- single module
1529
- that statically links the query-parser and
1530
- fs-database crates
1531
-
1532
- It is compiled from Rust via `wasm-pack` and works on
1533
- all platforms. The daemon loads and compiles it at
1534
- startup using the runtime's `WebAssembly` API. The
1531
+ The package includes four WASM modules:
1532
+
1533
+ - `kb-worker/kbdb_worker_bg.wasm` -- orchestration
1534
+ module that imports query-parser and fs-database
1535
+ - `query-parser/query_parser_bg.wasm` -- tokenization
1536
+ and query parsing
1537
+ - `fs-database/fs_database_bg.wasm` -- filesystem
1538
+ storage engine
1539
+ - `default-embedding/kbdb_default_embedding_bg.wasm`
1540
+ -- built-in TF-IDF embedding (standalone, no host
1541
+ imports)
1542
+
1543
+ All are compiled from Rust via `wasm-pack` and work
1544
+ on all platforms. The daemon loads and compiles them
1545
+ at startup using the runtime's `WebAssembly` API. The
1535
1546
  daemon runs on the same runtime (Node.js or Deno) as
1536
1547
  the worker client that spawns it.