@dikolab/kbdb 0.5.0 → 0.6.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (224) hide show
  1. package/CLA.md +68 -0
  2. package/LICENSE +676 -12
  3. package/LICENSING.md +58 -0
  4. package/README.md +50 -25
  5. package/README.md.bak +322 -0
  6. package/dist/README.md +322 -0
  7. package/dist/{chunk-BJXEVALU.mjs → chunk-QLJ33C74.mjs} +44 -44
  8. package/dist/chunk-QLJ33C74.mjs.map +7 -0
  9. package/dist/cli.cjs +550 -232
  10. package/dist/cli.cjs.map +4 -4
  11. package/dist/cli.d.ts +0 -8
  12. package/dist/cli.mjs +483 -154
  13. package/dist/cli.mjs.map +4 -4
  14. package/dist/kbdb-worker.cjs +36 -5
  15. package/dist/kbdb-worker.cjs.map +3 -3
  16. package/dist/mod.cjs +89 -30
  17. package/dist/mod.cjs.map +4 -4
  18. package/dist/mod.mjs +88 -30
  19. package/dist/mod.mjs.map +4 -4
  20. package/dist/src/cli.d.ts +0 -8
  21. package/dist/src/shared/cli/constants/defaults.constant.d.ts +1 -1
  22. package/dist/src/shared/cli/constants/recfile-command-field-map.constant.d.ts +8 -0
  23. package/dist/src/shared/cli/constants/recfile-field-maps.constant.d.ts +21 -0
  24. package/dist/src/shared/cli/functions/compute-directory-level.function.d.ts +14 -0
  25. package/dist/src/shared/cli/functions/dispatch-command.function.d.ts +2 -28
  26. package/dist/src/shared/cli/functions/format-command-output.function.d.ts +14 -0
  27. package/dist/src/shared/cli/functions/format-db-not-found.function.d.ts +20 -0
  28. package/dist/src/shared/cli/functions/format-output.function.d.ts +1 -0
  29. package/dist/src/shared/cli/functions/format-recfile-output.function.d.ts +11 -0
  30. package/dist/src/shared/cli/functions/render-recfile-value.function.d.ts +11 -0
  31. package/dist/src/shared/cli/functions/resolve-db-dir.function.d.ts +24 -0
  32. package/dist/src/shared/cli/functions/resolve-recfile-key.function.d.ts +9 -0
  33. package/dist/src/shared/cli/functions/run-db-path.function.d.ts +10 -0
  34. package/dist/src/shared/cli/functions/run-export.function.d.ts +8 -4
  35. package/dist/src/shared/cli/functions/run-import.function.d.ts +4 -0
  36. package/dist/src/shared/cli/functions/run-learn.function.d.ts +12 -0
  37. package/dist/src/shared/cli/index.d.ts +4 -1
  38. package/dist/src/shared/cli/typings/cli-options.interface.d.ts +9 -2
  39. package/dist/src/shared/cli/typings/db-resolution.interface.d.ts +9 -0
  40. package/dist/src/shared/cli/typings/recfile-field-map.type.d.ts +10 -0
  41. package/dist/src/shared/version/constants/version.constant.d.ts +1 -1
  42. package/dist/src/shared/wasm-codec/functions/decode-option-u8.function.d.ts +9 -0
  43. package/dist/src/shared/wasm-codec/functions/decode-section-records.function.d.ts +4 -0
  44. package/dist/src/shared/wasm-codec/functions/encode-add-section-params.function.d.ts +2 -45
  45. package/dist/src/shared/wasm-codec/functions/encode-update-section-params.function.d.ts +10 -1
  46. package/dist/src/shared/wasm-codec/index.d.ts +1 -1
  47. package/dist/src/shared/wasm-codec/typings/add-section-input.model.d.ts +6 -0
  48. package/dist/src/shared/wasm-codec/typings/section-record-trailing.model.d.ts +5 -0
  49. package/dist/src/shared/wasm-codec/typings/section-record.model.d.ts +5 -0
  50. package/dist/src/shared/worker-client/functions/build-deno-permissions.function.d.ts +11 -0
  51. package/dist/src/shared/worker-client/functions/read-crash-reason.function.d.ts +9 -0
  52. package/dist/src/shared/worker-client/functions/resolve-spawn-script.function.d.ts +6 -0
  53. package/dist/src/shared/worker-client/functions/spawn-daemon.function.d.ts +1 -1
  54. package/dist/src/shared/worker-client/typings/section.model.d.ts +5 -0
  55. package/dist/src/shared/worker-client/typings/worker-client-options.interface.d.ts +0 -6
  56. package/dist/src/shared/worker-daemon/functions/count-files.function.d.ts +12 -0
  57. package/dist/src/shared/worker-daemon/functions/handle-import.function.d.ts +2 -0
  58. package/dist/src/shared/worker-daemon/functions/handle-read-ops.function.d.ts +8 -0
  59. package/dist/src/shared/worker-daemon/functions/validate-daemon-path.function.d.ts +15 -0
  60. package/dist/src/worker.d.ts +0 -7
  61. package/dist/wasm/default-embedding/kbdb_default_embedding_bg.wasm +0 -0
  62. package/dist/wasm/default-embedding/package.json +1 -1
  63. package/dist/wasm/fs-database/kbdb_fs_database_bg.wasm +0 -0
  64. package/dist/wasm/fs-database/package.json +1 -1
  65. package/dist/wasm/fs-migration/package.json +1 -1
  66. package/dist/wasm/kb-worker/kbdb_worker_bg.wasm +0 -0
  67. package/dist/wasm/kb-worker/package.json +1 -1
  68. package/dist/wasm/query-parser/kbdb_query_parser_bg.wasm +0 -0
  69. package/dist/wasm/query-parser/package.json +1 -1
  70. package/dist/worker.d.ts +0 -7
  71. package/dist/worker.mjs +37 -6
  72. package/dist/worker.mjs.map +3 -3
  73. package/package.json +11 -10
  74. package/dist/chunk-BJXEVALU.mjs.map +0 -7
  75. package/dist/src/shared/cli/functions/resolve-raw-db-path.function.d.ts +0 -10
  76. package/docs/details/README.md +0 -30
  77. package/docs/details/agent-tooling.md +0 -132
  78. package/docs/details/cli.md +0 -777
  79. package/docs/details/knowledge-base.md +0 -180
  80. package/docs/details/library-api.md +0 -495
  81. package/docs/details/mcp-server.md +0 -482
  82. package/docs/details/search-and-ranking.md +0 -575
  83. package/docs/details/storage.md +0 -531
  84. package/docs/goals/agents.md +0 -751
  85. package/docs/goals/architecture.svg +0 -198
  86. package/docs/goals/auto-capture.md +0 -378
  87. package/docs/goals/benchmarking.md +0 -296
  88. package/docs/goals/cli.md +0 -2654
  89. package/docs/goals/concurrency.md +0 -259
  90. package/docs/goals/content-composer.md +0 -609
  91. package/docs/goals/content-parser.md +0 -279
  92. package/docs/goals/database.md +0 -1679
  93. package/docs/goals/document.md +0 -368
  94. package/docs/goals/hybrid-search.md +0 -465
  95. package/docs/goals/mcp.md +0 -1541
  96. package/docs/goals/overview.md +0 -124
  97. package/docs/goals/query-parser.md +0 -373
  98. package/docs/goals/query-result.md +0 -860
  99. package/docs/goals/semantic-dedup.md +0 -233
  100. package/docs/goals/skills.md +0 -810
  101. package/docs/goals/sync.md +0 -217
  102. package/docs/goals/worker-client.md +0 -1005
  103. package/docs/goals/worker-daemon.md +0 -1547
  104. package/docs/modules/overview.md +0 -319
  105. package/docs/modules/rust/embedding/module.md +0 -91
  106. package/docs/modules/rust/fs-database/functions/document.md +0 -48
  107. package/docs/modules/rust/fs-database/functions/integrity.md +0 -79
  108. package/docs/modules/rust/fs-database/functions/io.md +0 -87
  109. package/docs/modules/rust/fs-database/functions/module.md +0 -22
  110. package/docs/modules/rust/fs-database/functions/query-exec.md +0 -44
  111. package/docs/modules/rust/fs-database/functions/section.md +0 -76
  112. package/docs/modules/rust/fs-database/indexes/btree.md +0 -35
  113. package/docs/modules/rust/fs-database/indexes/corpus.md +0 -61
  114. package/docs/modules/rust/fs-database/indexes/inverted.md +0 -60
  115. package/docs/modules/rust/fs-database/indexes/module.md +0 -64
  116. package/docs/modules/rust/fs-database/indexes/positional.md +0 -39
  117. package/docs/modules/rust/fs-database/indexes/vectors.md +0 -110
  118. package/docs/modules/rust/fs-database/module.md +0 -150
  119. package/docs/modules/rust/fs-database/types/catalog.md +0 -53
  120. package/docs/modules/rust/fs-database/types/module.md +0 -12
  121. package/docs/modules/rust/fs-database/types/section.md +0 -127
  122. package/docs/modules/rust/kb-worker/functions/compose.md +0 -57
  123. package/docs/modules/rust/kb-worker/functions/module.md +0 -20
  124. package/docs/modules/rust/kb-worker/functions/retrieve.md +0 -194
  125. package/docs/modules/rust/kb-worker/functions/search.md +0 -173
  126. package/docs/modules/rust/kb-worker/functions/write.md +0 -172
  127. package/docs/modules/rust/kb-worker/module.md +0 -171
  128. package/docs/modules/rust/kb-worker/types/cache.md +0 -90
  129. package/docs/modules/rust/kb-worker/types/context.md +0 -26
  130. package/docs/modules/rust/kb-worker/types/module.md +0 -16
  131. package/docs/modules/rust/kb-worker/types/status.md +0 -29
  132. package/docs/modules/rust/kb-worker/types/token.md +0 -50
  133. package/docs/modules/rust/overview.md +0 -531
  134. package/docs/modules/rust/query-parser/functions/extract.md +0 -56
  135. package/docs/modules/rust/query-parser/functions/module.md +0 -18
  136. package/docs/modules/rust/query-parser/functions/parse.md +0 -33
  137. package/docs/modules/rust/query-parser/functions/pipeline.md +0 -9
  138. package/docs/modules/rust/query-parser/functions/rank.md +0 -85
  139. package/docs/modules/rust/query-parser/module.md +0 -131
  140. package/docs/modules/rust/query-parser/types/cache.md +0 -39
  141. package/docs/modules/rust/query-parser/types/keyphrase.md +0 -71
  142. package/docs/modules/rust/query-parser/types/module.md +0 -18
  143. package/docs/modules/rust/query-parser/types/token.md +0 -46
  144. package/docs/modules/rust/shared/content-composer/functions.md +0 -114
  145. package/docs/modules/rust/shared/content-composer/module.md +0 -147
  146. package/docs/modules/rust/shared/content-composer/types.md +0 -93
  147. package/docs/modules/rust/shared/content-parser/functions.md +0 -72
  148. package/docs/modules/rust/shared/content-parser/module.md +0 -99
  149. package/docs/modules/rust/shared/content-parser/types.md +0 -71
  150. package/docs/modules/rust/shared/corpus/module.md +0 -24
  151. package/docs/modules/rust/shared/corpus/types.md +0 -141
  152. package/docs/modules/rust/shared/document/module.md +0 -25
  153. package/docs/modules/rust/shared/document/types.md +0 -154
  154. package/docs/modules/rust/shared/encode/module.md +0 -22
  155. package/docs/modules/rust/shared/encode/traits.md +0 -304
  156. package/docs/modules/rust/shared/error/module.md +0 -28
  157. package/docs/modules/rust/shared/error/types.md +0 -302
  158. package/docs/modules/rust/shared/kbid/module.md +0 -24
  159. package/docs/modules/rust/shared/kbid/types.md +0 -147
  160. package/docs/modules/rust/shared/memory/functions.md +0 -209
  161. package/docs/modules/rust/shared/memory/module.md +0 -24
  162. package/docs/modules/rust/shared/module.md +0 -196
  163. package/docs/modules/rust/shared/pipeline/functions.md +0 -141
  164. package/docs/modules/rust/shared/pipeline/module.md +0 -62
  165. package/docs/modules/rust/shared/pipeline/types.md +0 -43
  166. package/docs/modules/rust/shared/query/module.md +0 -27
  167. package/docs/modules/rust/shared/query/types.md +0 -236
  168. package/docs/modules/rust/shared/section/module.md +0 -25
  169. package/docs/modules/rust/shared/section/types.md +0 -294
  170. package/docs/modules/rust/shared/term/module.md +0 -25
  171. package/docs/modules/rust/shared/term/types.md +0 -139
  172. package/docs/modules/typescript/cli.md +0 -131
  173. package/docs/modules/typescript/kbdb-worker.md +0 -150
  174. package/docs/modules/typescript/overview.md +0 -400
  175. package/docs/modules/typescript/shared/auto-capture/module.md +0 -73
  176. package/docs/modules/typescript/shared/cli/constants.md +0 -23
  177. package/docs/modules/typescript/shared/cli/functions.md +0 -809
  178. package/docs/modules/typescript/shared/cli/module.md +0 -138
  179. package/docs/modules/typescript/shared/cli/typings.md +0 -197
  180. package/docs/modules/typescript/shared/embedding/functions.md +0 -200
  181. package/docs/modules/typescript/shared/embedding/module.md +0 -77
  182. package/docs/modules/typescript/shared/embedding/typings.md +0 -124
  183. package/docs/modules/typescript/shared/hash/functions.md +0 -20
  184. package/docs/modules/typescript/shared/hash/module.md +0 -21
  185. package/docs/modules/typescript/shared/log/constants.md +0 -82
  186. package/docs/modules/typescript/shared/log/functions.md +0 -131
  187. package/docs/modules/typescript/shared/log/module.md +0 -153
  188. package/docs/modules/typescript/shared/log/typings.md +0 -82
  189. package/docs/modules/typescript/shared/mcp/classes.md +0 -179
  190. package/docs/modules/typescript/shared/mcp/functions.md +0 -386
  191. package/docs/modules/typescript/shared/mcp/module.md +0 -160
  192. package/docs/modules/typescript/shared/mcp/typings.md +0 -529
  193. package/docs/modules/typescript/shared/module.md +0 -110
  194. package/docs/modules/typescript/shared/platform/functions.md +0 -42
  195. package/docs/modules/typescript/shared/platform/module.md +0 -25
  196. package/docs/modules/typescript/shared/runtime/functions.md +0 -26
  197. package/docs/modules/typescript/shared/runtime/module.md +0 -27
  198. package/docs/modules/typescript/shared/runtime/typings.md +0 -13
  199. package/docs/modules/typescript/shared/version/module.md +0 -27
  200. package/docs/modules/typescript/shared/wasm-codec/functions.md +0 -391
  201. package/docs/modules/typescript/shared/wasm-codec/module.md +0 -98
  202. package/docs/modules/typescript/shared/worker-client/classes.md +0 -264
  203. package/docs/modules/typescript/shared/worker-client/functions.md +0 -175
  204. package/docs/modules/typescript/shared/worker-client/module.md +0 -92
  205. package/docs/modules/typescript/shared/worker-client/typings.md +0 -511
  206. package/docs/modules/typescript/shared/worker-daemon/classes.md +0 -239
  207. package/docs/modules/typescript/shared/worker-daemon/constants.md +0 -37
  208. package/docs/modules/typescript/shared/worker-daemon/functions.md +0 -234
  209. package/docs/modules/typescript/shared/worker-daemon/module.md +0 -118
  210. package/docs/modules/typescript/shared/worker-daemon/typings.md +0 -132
  211. package/docs/overview.md +0 -191
  212. package/docs/release-notes/0.1.0.md +0 -189
  213. package/docs/release-notes/0.1.1.md +0 -46
  214. package/docs/release-notes/0.1.2.md +0 -38
  215. package/docs/release-notes/0.1.3.md +0 -42
  216. package/docs/release-notes/0.2.0.md +0 -147
  217. package/docs/release-notes/0.3.0.md +0 -89
  218. package/docs/release-notes/0.3.1.md +0 -121
  219. package/docs/release-notes/0.3.2.md +0 -64
  220. package/docs/release-notes/0.4.0.md +0 -80
  221. package/docs/release-notes/0.4.1.md +0 -149
  222. package/docs/release-notes/0.4.3.md +0 -60
  223. package/docs/release-notes/0.5.0.md +0 -41
  224. package/docs/release-notes/README.md +0 -16
@@ -1,1005 +0,0 @@
1
- # Worker Client
2
-
3
- *A shared TypeScript module that discovers or spawns
4
- the worker daemon and provides a typed API for all
5
- knowledge base operations over IPC.*
6
-
7
- ## Goal
8
-
9
- `src/cli.ts` needs to communicate with the worker
10
- daemon (`src/worker.ts`) to perform database
11
- operations. The worker client module is the single
12
- shared entry point for that communication. It
13
- handles:
14
-
15
- - Resolving the database directory path.
16
- - Computing the context ID.
17
- - Discovering an existing daemon or spawning a new one.
18
- - Connecting to the daemon's IPC socket.
19
- - Exposing every daemon IPC method as a typed async
20
- function.
21
- - Managing connection lifecycle (connect, reconnect,
22
- disconnect).
23
-
24
- No other module in the TypeScript layer communicates
25
- with the daemon directly. `src/cli.ts` imports the
26
- worker client and calls its methods -- it never
27
- constructs IPC messages, manages PID files, or
28
- spawns processes itself.
29
-
30
- ```
31
- src/cli.ts ──→ worker-client ──→ src/worker.ts
32
- ```
33
-
34
- ## Module Location
35
-
36
- ```
37
- src/shared/worker-client/
38
- ├── mod.ts # public API: createWorkerClient
39
- ├── resolve.ts # database directory resolution
40
- ├── context.ts # context ID computation
41
- ├── discovery.ts # PID file lookup, process check
42
- ├── spawn.ts # daemon spawning (Node + Deno)
43
- ├── connection.ts # IPC socket connect / reconnect
44
- └── types.ts # request/response type defs
45
- ```
46
-
47
- The module is internal to the package. It is not
48
- exported from the library's public API (`mod.ts` at
49
- the project root). Only `src/cli.ts` imports it.
50
-
51
- ## When the Daemon Is Needed
52
-
53
- The worker client is only created when the caller needs
54
- to perform operations that involve the `.kbdb` database
55
- directory:
56
-
57
- - **Search and query** -- Looking up indexes, ranking
58
- results, retrieving sections or documents.
59
- - **Content management** -- `kbdb learn` (import,
60
- index) and `kbdb unlearn` (remove, re-index).
61
- - **Status and retrieval** -- Database metadata,
62
- content retrieval by ID.
63
-
64
- Operations that do not touch the database do not create
65
- a worker client and do not require a daemon:
66
-
67
- - `kbdb --help`, `kbdb --version`
68
- - Invalid commands or argument validation errors
69
- - Any subcommand that exits before reaching the
70
- database
71
-
72
- ## Database Directory Resolution
73
-
74
- The worker client resolves the database directory path
75
- before connecting to the daemon:
76
-
77
- 1. If the caller provides a `targetDir` (e.g. from a
78
- CLI `--db <path>` flag), use it.
79
- 2. Otherwise, use the current working directory.
80
-
81
- The database directory is always
82
- `${targetDir}/.kbdb`. The worker client checks that
83
- this directory exists before proceeding. If `.kbdb`
84
- does not exist, the factory throws -- the caller is
85
- responsible for database initialization (e.g. the CLI
86
- prompts the user to create it).
87
-
88
- ## Context ID
89
-
90
- The worker client computes the context ID during
91
- `createWorkerClient()` and includes it in every IPC
92
- request. The context ID is a **SHA-256 hash of the
93
- database directory's absolute path**, truncated to 16
94
- lowercase hexadecimal characters.
95
-
96
- ```
97
- SHA-256(absolutePath) → truncate to 16 hex chars → ctx
98
- ```
99
-
100
- See [Worker Daemon -- Context ID](
101
- worker-daemon.md#context-id) for the full
102
- specification, including how the daemon validates the
103
- context ID on every request.
104
-
105
- ## Daemon Discovery
106
-
107
- When the worker client is created, it follows this
108
- discovery sequence:
109
-
110
- 1. Compute the context ID from the database directory
111
- path.
112
- 2. Look for `kb-worker-{ctx}.pid` in the OS temporary
113
- directory (e.g., `/tmp` on Linux/macOS, `%TEMP%` on
114
- Windows).
115
- 3. If the PID file exists, read the process ID and
116
- check whether the process is alive (see
117
- Platform-Specific PID Handling below).
118
- 4. If the process is alive, connect to the IPC socket
119
- at `kb-worker-{ctx}.sock` (or the named pipe on
120
- Windows).
121
- 5. If the PID file is missing or the process is dead
122
- (stale PID file), remove the stale file and spawn
123
- a new daemon (see Spawning below).
124
-
125
- The PID file contains only the process ID number as
126
- plain text (e.g., `12345`). No other data.
127
-
128
- ### Platform-Specific PID Handling
129
-
130
- Both Linux/macOS and Windows use integer process IDs.
131
- The PID file format is identical across platforms -- a
132
- single integer as plain text. The differences are in
133
- how the worker client resolves the temporary directory
134
- and checks whether the process is alive.
135
-
136
- **Temporary directory:**
137
-
138
- | Platform | Resolution |
139
- |---------------|---------------------------------------|
140
- | Linux / macOS | `$TMPDIR` if set, otherwise `/tmp` |
141
- | Windows | `%TEMP%` (typically `C:\Users\{user}\AppData\Local\Temp`) |
142
-
143
- In Node.js, `os.tmpdir()` returns the correct path on
144
- all platforms. In Deno, `Deno.env.get("TMPDIR")` (Unix)
145
- or `Deno.env.get("TEMP")` (Windows) provides the same.
146
-
147
- **Checking if a process is alive:**
148
-
149
- The worker client must verify that the PID read from
150
- the file refers to a running daemon, not a dead process
151
- whose PID has been recycled by the OS.
152
-
153
- **Linux / macOS** -- Use a zero signal via `kill()`:
154
-
155
- ```ts
156
- // Node.js
157
- try {
158
- process.kill(pid, 0); // signal 0 = no signal sent
159
- // process exists
160
- } catch {
161
- // process does not exist (ESRCH) or
162
- // not permitted (EPERM -- process exists
163
- // but is owned by another user)
164
- }
165
- ```
166
-
167
- `kill(pid, 0)` sends no actual signal. It returns
168
- successfully if the process exists and the caller has
169
- permission to signal it. It throws `ESRCH` if the
170
- process does not exist. An `EPERM` error means the
171
- process exists but belongs to another user -- this
172
- should be treated as "alive" since the PID is in use.
173
-
174
- **Windows** -- `process.kill(pid, 0)` is supported in
175
- Node.js on Windows and behaves the same way: it checks
176
- process existence without sending a signal. This is the
177
- preferred cross-platform method.
178
-
179
- For Deno on Windows, where `Deno.kill()` does not
180
- support signal 0, use a subprocess check:
181
-
182
- ```ts
183
- // Deno (Windows fallback)
184
- const cmd = new Deno.Command('tasklist', {
185
- args: ['/FI', `PID eq ${pid}`, '/NH', '/FO', 'CSV'],
186
- stdout: 'piped',
187
- stderr: 'null',
188
- });
189
- const { stdout } = await cmd.output();
190
- const output = new TextDecoder().decode(stdout);
191
- const alive = output.includes(`"${pid}"`);
192
- ```
193
-
194
- `tasklist /FI "PID eq {pid}"` filters the process list
195
- by PID. If the output contains the PID, the process is
196
- alive. The `/NH` flag suppresses the header and `/FO
197
- CSV` outputs in CSV format for reliable parsing.
198
-
199
- **Cross-platform wrapper:**
200
-
201
- The worker client implements a single
202
- `isProcessAlive(pid)` function that selects the correct
203
- method based on `process.platform` (Node.js) or
204
- `Deno.build.os` (Deno):
205
-
206
- | Runtime | Linux / macOS | Windows |
207
- |-----------|------------------------|--------------------------|
208
- | Node.js | `process.kill(pid, 0)` | `process.kill(pid, 0)` |
209
- | Deno | `Deno.kill(pid, "SIGCONT")` | `tasklist` subprocess |
210
-
211
- This function returns `true` if the process is alive,
212
- `false` if it is not. The daemon discovery sequence
213
- calls it after reading the PID file to decide whether
214
- to connect or spawn.
215
-
216
- ## Spawning
217
-
218
- The worker client spawns the daemon as a **detached
219
- child process** so it outlives the parent. Unlike a
220
- typical fire-and-forget detach, the worker client
221
- **captures the daemon's stderr** during the spawn
222
- window so that startup failures can be relayed to the
223
- caller.
224
-
225
- **Node.js:**
226
-
227
- ```js
228
- const child = spawn(
229
- 'node',
230
- ['kb-worker.js', contextPath],
231
- {
232
- detached: true,
233
- stdio: ['ignore', 'ignore', 'pipe'],
234
- },
235
- );
236
- ```
237
-
238
- **Deno:**
239
-
240
- ```ts
241
- const cmd = new Deno.Command('deno', {
242
- args: [
243
- 'run', '--allow-all',
244
- 'src/worker.ts', contextPath,
245
- ],
246
- stdin: 'null',
247
- stdout: 'null',
248
- stderr: 'piped',
249
- });
250
- const child = cmd.spawn();
251
- ```
252
-
253
- Note: `stderr` is `'pipe'` / `'piped'`, not
254
- `'ignore'`. The worker client reads from it during
255
- the spawn window. Once the PID file appears
256
- (indicating successful startup), the worker client
257
- closes the stderr pipe and unrefs the child so the
258
- daemon can outlive the parent.
259
-
260
- The spawning sequence:
261
-
262
- 1. Worker client ensures `worker.toml` exists in the
263
- context directory (writes defaults if missing).
264
- 2. Worker client spawns `src/worker.ts` (or its
265
- compiled `.js` equivalent) with the context path as
266
- the sole command-line argument. stderr is piped to
267
- the worker client.
268
- 3. Worker client polls for the PID file to appear (with
269
- a short timeout, default 10 000 ms) to confirm the
270
- daemon started.
271
- 4. **On PID file found (success):** worker client
272
- closes the stderr pipe, detaches the child, unrefs
273
- it, and connects to the IPC socket. Because the
274
- daemon writes the PID file only after the socket
275
- is listening, the client can connect immediately.
276
- 5. **On timeout or child exit (failure):** worker
277
- client reads all buffered stderr from the daemon,
278
- includes it in the thrown error (see Startup
279
- Failure Handling below).
280
-
281
- There is no fallback to in-process WASM loading -- the
282
- daemon is required for all database operations.
283
-
284
- ### Startup Failure Handling
285
-
286
- When the daemon fails to start -- either because a
287
- startup precondition failed (see
288
- [Worker Daemon -- Startup Preconditions](
289
- worker-daemon.md#startup-preconditions)) or the
290
- process exited unexpectedly -- the worker client
291
- detects the failure by one of:
292
-
293
- - The child process exits before the PID file appears.
294
- - The PID file poll times out.
295
-
296
- In either case, the worker client:
297
-
298
- 1. Reads the daemon's buffered stderr output.
299
- 2. Throws a `WorkerClientError` that includes the
300
- daemon's stderr message in the error's `message`
301
- property.
302
-
303
- ```ts
304
- interface WorkerClientError extends Error {
305
- code: number;
306
- /** Daemon stderr output, if available. */
307
- daemonStderr?: string;
308
- }
309
- ```
310
-
311
- The error code is `-32003` (daemon unavailable).
312
-
313
- The caller (CLI or MCP) is responsible for printing
314
- the error. In practice, the CLI prints the error to
315
- its own stderr and exits non-zero. This means the
316
- daemon's error message flows through to the user's
317
- terminal:
318
-
319
- ```
320
- kbdb search "query"
321
- -> worker client spawns daemon
322
- -> daemon checks preconditions
323
- -> daemon prints to its stderr:
324
- "error: database version 3 is newer than kbdb
325
- version 2 -- upgrade kbdb to open this database"
326
- -> daemon exits with code 1
327
- -> worker client reads daemon stderr
328
- -> worker client throws WorkerClientError with
329
- daemonStderr containing the message
330
- -> src/cli.ts catches error, prints to stderr:
331
- "error: daemon failed to start: database version
332
- 3 is newer than kbdb version 2 -- upgrade kbdb
333
- to open this database"
334
- -> src/cli.ts sets process.exitCode = 1
335
- ```
336
-
337
- This relay ensures the user sees the daemon's specific
338
- error reason even though the daemon runs as a separate
339
- detached process. The daemon's stderr is the single
340
- source of truth for startup failure diagnostics.
341
-
342
- ## Connection Retry
343
-
344
- When the worker client calls `connect()`, transient
345
- errors (`ENOENT` -- socket not yet created, or
346
- `ECONNREFUSED` -- server not yet accepting) are
347
- retried with exponential backoff:
348
-
349
- - **Max retries**: 5
350
- - **Initial delay**: 50 ms
351
- - **Backoff**: doubles each attempt (50, 100, 200,
352
- 400, 800 ms = ~1.55 s worst case)
353
- - **Non-transient errors** (`EACCES`, `EINVAL`, etc.)
354
- fail immediately without retry.
355
-
356
- This covers the narrow window where the PID file
357
- exists but the socket has not yet finished binding
358
- (e.g. under heavy load or slow I/O).
359
-
360
- ## Post-Connect Failure Handling
361
-
362
- After a successful `connect()`, if the daemon crashes
363
- or the IPC socket dies:
364
-
365
- 1. All pending RPC calls (in `this.pending`) are
366
- rejected with a descriptive error.
367
- 2. The client marks itself as disconnected
368
- (`this.socket = null`).
369
- 3. The socket is destroyed.
370
-
371
- The `isConnected` getter exposes the connection
372
- state so callers (and the cached client factory) can
373
- check before reusing a client.
374
-
375
- ## Cached Client Factory
376
-
377
- `getOrCreateClient(options?)` returns a cached
378
- `WorkerClient` for the given context path, or creates
379
- and caches a new one if the cached client is
380
- disconnected or absent.
381
-
382
- ```ts
383
- import { getOrCreateClient } from '@dikolab/kbdb';
384
-
385
- const client = await getOrCreateClient({
386
- dbPath: '/path/to/project',
387
- });
388
- ```
389
-
390
- Intended for library consumers and the MCP server.
391
- CLI one-shots should use `createWorkerClient()`
392
- directly (each is a separate process anyway).
393
-
394
- ## Socket Keepalive
395
-
396
- Both the client and server enable `SO_KEEPALIVE`
397
- with a 30-second interval. This ensures both ends
398
- detect a dead peer within ~30 seconds, even if no
399
- application-level traffic is flowing.
400
-
401
- ## `createWorkerClient(options?)`
402
-
403
- Factory function. Returns a connected `WorkerClient`
404
- instance.
405
-
406
- ```ts
407
- interface WorkerClientOptions {
408
- targetDir?: string;
409
- }
410
-
411
- function createWorkerClient(
412
- options?: WorkerClientOptions,
413
- ): Promise<WorkerClient>;
414
- ```
415
-
416
- **Resolution order for `targetDir`:**
417
-
418
- 1. If `options.targetDir` is provided, use it.
419
- 2. Otherwise, use the current working directory.
420
-
421
- The factory then:
422
-
423
- 1. Resolves the absolute path: `${targetDir}/.kbdb`.
424
- 2. Computes the context ID (SHA-256 of the absolute
425
- path, truncated to 16 lowercase hex characters).
426
- 3. Checks whether `.kbdb` exists at the resolved path.
427
- If not, throws -- the caller is responsible for
428
- prompting the user or initializing the database
429
- before creating a client.
430
- 4. Looks for `kb-worker-{ctx}.pid` in the OS temp
431
- directory and checks whether the daemon is alive
432
- (see Daemon Discovery above).
433
- 5. If no daemon is running, spawns one (see Spawning
434
- above).
435
- 6. Connects to the IPC socket
436
- (`kb-worker-{ctx}.sock` on Unix,
437
- `\\.\pipe\kb-worker-{ctx}` on Windows).
438
- 7. Returns the connected `WorkerClient`.
439
-
440
- If the daemon fails to start or the IPC connection
441
- cannot be established, the factory throws.
442
-
443
- ## `WorkerClient`
444
-
445
- All methods are async. High-level methods (`search`,
446
- `readSections`, `addSection`, `updateSection`,
447
- `removeSection`, `groupSections`, `removeGrouping`,
448
- `retrieveDocument`, `content`) are composed from
449
- the low-level IPC primitives (`query` →
450
- `get_result_status` → `get_result`) documented in
451
- [Worker Daemon -- IPC API](worker-daemon.md). The
452
- daemon routes these queries to `kb-worker.wasm`,
453
- which orchestrates `fs-database.wasm` and
454
- `query-parser.wasm`.
455
-
456
- `content()` takes one or more kbid or docid
457
- identifiers (mixed freely) and returns a composed
458
- Markdown document. It delegates to the daemon's
459
- `compose` IPC method, which triggers `kb-worker.wasm`
460
- to resolve identifiers, fetch sections, extract
461
- markers recursively, and call the `content-composer`
462
- module in `kbdb-shared`. See
463
- [Content Composer](content-composer.md) for the full
464
- composition specification.
465
-
466
- Maintenance methods (`integrityCheck`, `gc`,
467
- `rebuildIndexes`) delegate to the corresponding
468
- daemon IPC methods (`integrity_check`, `gc`,
469
- `rebuild_indexes`). These return results directly
470
- (not via the token pattern) because maintenance
471
- operations are infrequent and their results are not
472
- cached.
473
-
474
- `dbStatus()` returns data from the initialization
475
- response cached at connect time -- it does not
476
- issue a separate IPC call.
477
-
478
- ```ts
479
- interface WorkerClient {
480
- readonly ctx: string;
481
- readonly targetDir: string;
482
-
483
- // Lifecycle
484
- disconnect(): Promise<void>;
485
-
486
- // Status
487
- status(): Promise<DaemonStatus>;
488
- idleAt(): Promise<number>;
489
- config(key: ConfigKey): Promise<number>;
490
- dbStatus(): Promise<DbStatus>;
491
-
492
- // Search
493
- search(params: SearchParams):
494
- Promise<PagedSearchResult>;
495
-
496
- // Context expansion
497
- recall(params: {
498
- kbid?: string;
499
- kbids?: string[];
500
- depth?: number;
501
- }): Promise<RecallResult | RecallResult[]>;
502
-
503
- // Sections
504
- readSections(kbIds: string[]):
505
- Promise<KbdbResult<Section[]>>;
506
- addSection(params: AddSectionParams):
507
- Promise<AddSectionResult>;
508
- updateSection(params: UpdateSectionParams):
509
- Promise<UpdateSectionResult>;
510
- removeSection(kbId: string):
511
- Promise<RemoveSectionResult>;
512
-
513
- // Documents
514
- groupSections(params: GroupSectionsParams):
515
- Promise<GroupSectionsResult>;
516
- removeGrouping(docid: string):
517
- Promise<RemoveGroupingResult>;
518
- retrieveDocument(docid: string):
519
- Promise<KbdbResult<DocumentResult>>;
520
-
521
- // Content composition
522
- content(ids: string[]):
523
- Promise<ContentResult>;
524
-
525
- // Maintenance
526
- integrityCheck(): Promise<IntegrityCheckResult>;
527
- gc(): Promise<GcResult>;
528
- rebuildIndexes(): Promise<RebuildResult>;
529
-
530
- // Cache
531
- invalidate(keys: string[]): Promise<number>;
532
- invalidateAll(): Promise<number>;
533
-
534
- // Low-level query (token-based)
535
- query(kbId: string): Promise<string>;
536
- getResultStatus(token: string):
537
- Promise<ResultStatus>;
538
- getResult<T extends ResultType>(
539
- token: string,
540
- type: T,
541
- ): Promise<ResultData<T> | null>;
542
- }
543
- ```
544
-
545
- ## Type Definitions
546
-
547
- **Status types:**
548
-
549
- ```ts
550
- type DaemonStatus = 'busy' | 'idle';
551
-
552
- type ConfigKey =
553
- | 'timeout_ms'
554
- | 'item_ttl_ms'
555
- | 'item_limit'
556
- | 'index_limit';
557
-
558
- interface DbStatus {
559
- target_dir: string;
560
- document_count: number;
561
- section_count: number;
562
- index_size_bytes: number;
563
- last_modified: string;
564
- cache_stats: {
565
- items_cached: number;
566
- hit_rate: number;
567
- };
568
- }
569
- ```
570
-
571
- **Search types:**
572
-
573
- `search()` now returns a `PagedSearchResult` envelope.
574
- Individual `SectionMatch` items include `heading`,
575
- `type`, `docids`, `snippet`, and `matched_terms`
576
- alongside `score`. The MCP layer strips `score`; the
577
- CLI and IPC layers retain it.
578
-
579
- ```ts
580
- interface SearchParams {
581
- query: string;
582
- mode: 'sections' | 'documents' | 'stats';
583
- limit?: number;
584
- offset?: number;
585
- }
586
-
587
- interface PagedSearchResult {
588
- items: SectionMatch[];
589
- total: number;
590
- offset: number;
591
- limit: number;
592
- has_more: boolean;
593
- }
594
-
595
- type SearchResult =
596
- | SectionSearchResult
597
- | DocumentSearchResult
598
- | StatsSearchResult;
599
-
600
- interface SectionSearchResult {
601
- type: 'section';
602
- data: SectionMatch[];
603
- references: (Section | DocumentResult)[];
604
- total: number;
605
- query_time_ms: number;
606
- query: string;
607
- }
608
-
609
- interface DocumentSearchResult {
610
- type: 'document';
611
- data: DocumentMatch[];
612
- references: (Section | DocumentResult)[];
613
- total: number;
614
- query_time_ms: number;
615
- query: string;
616
- }
617
-
618
- interface StatsSearchResult {
619
- total_matches: number;
620
- query_time_ms: number;
621
- top_terms: TermStat[];
622
- query: string;
623
- }
624
- ```
625
-
626
- **Match types:**
627
-
628
- ```ts
629
- interface SectionMatch {
630
- kbid: string;
631
- docids: string[];
632
- type: string;
633
- heading: string | null;
634
- score: number;
635
- snippet: string;
636
- matched_terms: string[];
637
- }
638
-
639
- interface DocumentMatch {
640
- docid: string;
641
- title: string;
642
- score: number;
643
- matching_sections: Array<{
644
- kbid: string;
645
- type: string;
646
- heading: string;
647
- score: number;
648
- snippet: string;
649
- }>;
650
- }
651
-
652
- interface TermStat {
653
- term: string;
654
- frequency: number;
655
- sections_matched: number;
656
- }
657
- ```
658
-
659
- **Recall types:**
660
-
661
- `recall()` returns progressive context expansion for
662
- one or more sections. Returns `RecallResult` for a
663
- single kbid or `RecallResult[]` for a batch. The
664
- `depth` parameter controls how much context is
665
- included (0--3).
666
-
667
- ```ts
668
- interface RecallResult {
669
- kbid: string;
670
- heading: string | null;
671
- type: string;
672
- content: string;
673
- docids: string[];
674
- // Present at depth >= 1:
675
- documents?: RecallDocument[];
676
- back_references?: RecallReference[];
677
- // Present at depth >= 2:
678
- siblings?: RecallResult[];
679
- // Present at depth >= 3:
680
- references?: RecallResult[];
681
- }
682
-
683
- interface RecallDocument {
684
- docid: string;
685
- title: string;
686
- sections: Array<{
687
- kbid: string;
688
- heading: string | null;
689
- type: string;
690
- }>;
691
- }
692
-
693
- interface RecallReference {
694
- kbid: string;
695
- heading: string | null;
696
- type: string;
697
- marker_type: string;
698
- context_snippet: string;
699
- direction: 'back';
700
- }
701
- ```
702
-
703
- Maps to the daemon's `recall` IPC method, which
704
- calls `worker_recall` in the WASM engine.
705
-
706
- **Section types:**
707
-
708
- ```ts
709
- interface Section {
710
- kbid: string;
711
- docids: string[];
712
- type: string;
713
- title?: string;
714
- description?: string;
715
- token_count: number;
716
- heading_token_count: number;
717
- body_token_count: number;
718
- code_token_count: number;
719
- content: string;
720
- }
721
-
722
- interface AddSectionParams {
723
- content: string;
724
- type: string;
725
- title?: string;
726
- description?: string;
727
- docid?: string;
728
- }
729
-
730
- interface AddSectionResult {
731
- kbid: string;
732
- type: string;
733
- title?: string;
734
- description?: string;
735
- token_count: number;
736
- indexed_terms_count: number;
737
- elapsed_ms: number;
738
- attached: boolean;
739
- }
740
-
741
- interface UpdateSectionParams {
742
- kbid: string;
743
- content: string;
744
- }
745
-
746
- interface UpdateSectionResult {
747
- old_kbid: string;
748
- new_kbid: string;
749
- reindexed_terms_count: number;
750
- elapsed_ms: number;
751
- }
752
-
753
- interface RemoveSectionResult {
754
- removed: boolean;
755
- elapsed_ms: number;
756
- }
757
- ```
758
-
759
- **Document types:**
760
-
761
- ```ts
762
- interface GroupSectionsParams {
763
- docid: string;
764
- title: string;
765
- kbids: string[];
766
- }
767
-
768
- interface GroupSectionsResult {
769
- docid: string;
770
- section_count: number;
771
- elapsed_ms: number;
772
- }
773
-
774
- interface RemoveGroupingResult {
775
- removed: boolean;
776
- sections_ungrouped: number;
777
- elapsed_ms: number;
778
- }
779
-
780
- interface DocumentResult {
781
- docid: string;
782
- title: string;
783
- created_at: string;
784
- sections: Array<{
785
- kbid: string;
786
- docids: string[];
787
- type: string;
788
- title?: string;
789
- description?: string;
790
- content: string;
791
- position: number;
792
- }>;
793
- }
794
- ```
795
-
796
- **Unified result shape:**
797
-
798
- ```ts
799
- interface KbdbResult<T> {
800
- type: 'section' | 'document';
801
- data: T;
802
- references: (Section | DocumentResult)[];
803
- total: number;
804
- }
805
- ```
806
-
807
- `KbdbResult<T>` wraps all retrieval and search
808
- responses. The `references` array contains unique,
809
- recursively resolved records for every marker found
810
- in the returned content (see
811
- [Document](document.md) for marker types and
812
- [Content Parser](content-parser.md) for extraction).
813
- `total` is the full count of matching entities --
814
- `data.length` may be smaller when `limit` is applied.
815
-
816
- `SearchResult` (`SectionSearchResult` and
817
- `DocumentSearchResult`) follows the same shape with
818
- additional search-specific fields (`query_time_ms`,
819
- `query`). `StatsSearchResult` does not use the
820
- unified shape since it returns aggregate statistics,
821
- not entities.
822
-
823
- **Content composition result:**
824
-
825
- ```ts
826
- interface ContentResult {
827
- type: 'text/markdown';
828
- data: string;
829
- total: number;
830
- }
831
- ```
832
-
833
- `ContentResult` wraps the output of the content
834
- composition engine. `data` is the composed Markdown
835
- string. `total` is the character count of `data`
836
- (not byte count). See
837
- [Content Composer](content-composer.md) for the full
838
- composition specification.
839
-
840
- **Maintenance types:**
841
-
842
- ```ts
843
- interface IntegrityCheckResult {
844
- ok: boolean;
845
- errors: IntegrityError[];
846
- sections_checked: number;
847
- documents_checked: number;
848
- references_checked: number;
849
- elapsed_ms: number;
850
- }
851
-
852
- interface IntegrityError {
853
- type:
854
- | 'checksum_mismatch'
855
- | 'orphan_section'
856
- | 'orphan_document'
857
- | 'broken_reference'
858
- | 'type_mismatch'
859
- | 'circular_reference';
860
- entity: string;
861
- detail: string;
862
- }
863
-
864
- interface GcResult {
865
- removed_sections: number;
866
- removed_bytes: number;
867
- elapsed_ms: number;
868
- }
869
-
870
- interface RebuildResult {
871
- sections_reindexed: number;
872
- terms_indexed: number;
873
- elapsed_ms: number;
874
- }
875
- ```
876
-
877
- `IntegrityCheckResult` reports all issues found in
878
- the database. Error types match
879
- [Database -- Integrity and Maintenance](database.md).
880
- `GcResult` reports sections removed and bytes freed.
881
- `RebuildResult` reports sections and terms
882
- re-indexed.
883
-
884
- **Low-level query types:**
885
-
886
- ```ts
887
- type ResultType =
888
- | 'kbid'
889
- | 'docid'
890
- | 'section'
891
- | 'content'
892
- | 'kbid[]'
893
- | 'docid[]'
894
- | 'section[]'
895
- | 'content[]';
896
-
897
- interface ResultStatus {
898
- status: 'busy' | 'success' | 'fail';
899
- reason: string;
900
- }
901
- ```
902
-
903
- ## Connection Lifecycle
904
-
905
- **Short-lived (CLI):**
906
-
907
- ```
908
- src/cli.ts
909
-
910
- ├─ parse args, resolve targetDir
911
- ├─ createWorkerClient({ targetDir })
912
- │ ├─ find or spawn daemon
913
- │ └─ connect to IPC socket
914
- ├─ client.search(...) / client.addSection(...)
915
- ├─ print result to stdout
916
- └─ client.disconnect()
917
- ```
918
-
919
- The CLI creates the client, makes one or a few calls,
920
- and disconnects. The daemon stays alive for subsequent
921
- invocations.
922
-
923
- **Long-lived (MCP):**
924
-
925
- ```
926
- src/cli.ts
927
-
928
- ├─ createWorkerClient({ targetDir })
929
- │ ├─ find or spawn daemon
930
- │ └─ connect to IPC socket
931
- ├─ listen for MCP requests
932
- │ ├─ MCP search → client.search(...)
933
- │ ├─ MCP add → client.addSection(...)
934
- │ ├─ MCP remove → client.removeSection(...)
935
- │ └─ ...
936
- └─ on shutdown: client.disconnect()
937
- ```
938
-
939
- The MCP server holds the connection open. All incoming
940
- MCP requests route through the same `WorkerClient`.
941
-
942
- ## Reconnection
943
-
944
- If the IPC connection drops (daemon crashed, idle
945
- timeout, OS killed the process), the client does not
946
- throw immediately. On the next method call, it:
947
-
948
- 1. Detects the broken connection.
949
- 2. Runs the discovery sequence again (PID file check,
950
- spawn if needed).
951
- 3. Reconnects to the new daemon's IPC socket.
952
- 4. Retries the method call once.
953
-
954
- If reconnection fails, the method throws. The caller
955
- handles the error.
956
-
957
- This makes the MCP server resilient to daemon restarts
958
- without manual reconnection logic. The CLI benefits
959
- too -- if a daemon dies between the spawn check and
960
- the first method call, the client recovers.
961
-
962
- ## Error Handling
963
-
964
- All client methods throw on:
965
-
966
- - **Connection failure** -- Cannot connect to the
967
- daemon after discovery and spawn attempts.
968
- - **Spawn failure** -- Daemon did not start within
969
- the timeout.
970
- - **IPC error** -- JSON-RPC 2.0 error response from
971
- the daemon (context mismatch, invalid token, method
972
- not found).
973
- - **Reconnection failure** -- Broken connection and
974
- the retry also failed.
975
-
976
- Errors are thrown as typed error objects with a `code`
977
- property matching the JSON-RPC 2.0 error codes defined
978
- in [Worker Daemon -- Error Responses](
979
- worker-daemon.md#error-responses):
980
-
981
- ```ts
982
- interface WorkerClientError extends Error {
983
- code: number;
984
- }
985
- ```
986
-
987
- ## Design Constraints
988
-
989
- - The worker client is the **sole IPC client** for the
990
- daemon. No other TypeScript module sends messages to
991
- `src/worker.ts`.
992
- - The client does not cache data. Caching is the
993
- daemon's responsibility (via `kb-worker.wasm`).
994
- - The client does not perform database I/O. All
995
- storage operations go through the daemon.
996
- - The client does not perform text processing. That is
997
- `query-parser.wasm`'s job inside the daemon.
998
- - The module works on both Deno and Node.js. Platform
999
- differences (IPC sockets, process spawning, PID
1000
- checks) are abstracted inside internal files.
1001
- - The client's public API mirrors the IPC wire format.
1002
- Types match the JSON shapes defined in the Endpoints
1003
- section of [Database](database.md) and the IPC API
1004
- sections of [Worker Daemon](worker-daemon.md). No
1005
- transformation layer.