@dikolab/kbdb 0.3.2 → 0.4.1

package/docs/modules/rust/query-parser/types/token.md

@@ -4,7 +4,11 @@
 
 ## Source
 
-`engine/crates/query-parser/src/pipeline/mod.rs`
+`engine/crates/shared/src/pipeline/mod.rs`
+
+**Note:** These types are now defined in `kbdb-shared`.
+See [shared/pipeline/types](../../shared/pipeline/types.md)
+for the canonical documentation.
 
 ## Type Definitions
 

package/docs/modules/rust/shared/content-parser/functions.md

@@ -49,7 +49,7 @@ is preserved; the caller should normalise whitespace
 if needed.
 
 This function replaces the earlier implementation in
-`query-parser/src/pipeline/markers.rs`, which only
+`shared/src/pipeline/markers.rs`, which only
 recognized `$(image:)` and `$(code:)`. The shared
 version recognizes all six content marker types.
 

package/docs/modules/rust/shared/memory/module.md

@@ -11,7 +11,7 @@ and WASM modules.*
 ## Responsibility
 
 Provides the shared-memory allocation protocol used
-by all three WASM modules. Every WASM export that
+by all four WASM modules. Every WASM export that
 exchanges variable-length data (strings, byte
 slices, encoded structs) goes through this module's
 allocator and read/write helpers.

package/docs/modules/rust/shared/module.md

@@ -1,8 +1,8 @@
 # shared -- Shared Types and Utilities
 
 *Rust crate that provides types, memory helpers, error
-types, and serialization utilities shared across all
-three WASM modules.*
+types, the tokenization pipeline, and serialization
+utilities shared across all four WASM modules.*
 
 ## Crate
 
@@ -15,10 +15,9 @@ three WASM modules.*
 
 `kbdb-shared` is a pure Rust library. It is **not**
 compiled to WASM on its own -- it is compiled into
-each of the three WASM modules that depend on it
+each of the four WASM modules that depend on it
 (`kbdb-fs-database`, `kbdb-query-parser`,
-`kbdb-worker`). It has no external dependencies beyond
-`wasm-bindgen` for the `alloc`/`free` exports.
+`kbdb-default-embedding`, `kbdb-worker`).
 
 ## Workspace Position
 
@@ -29,6 +28,7 @@ engine/
     ├── shared/                  # this crate (leaf)
     ├── fs-database/             # depends on shared
     ├── query-parser/            # depends on shared
+    ├── default-embedding/       # depends on shared
     └── kb-worker/               # depends on shared
 ```
 
@@ -74,6 +74,18 @@ engine/crates/shared/
     │   ├── types.rs             # ComposeRequest, etc.
     │   ├── compose.rs           # compose_content()
     │   └── render.rs            # marker rendering
+    ├── pipeline/
+    │   ├── mod.rs               # pipeline orchestrator
+    │   ├── markers.rs           # stage 1: strip markers
+    │   ├── unicode.rs           # stage 2: NFC normalize
+    │   ├── segment.rs           # stage 3: word boundaries
+    │   ├── identifiers.rs       # stage 4: camelCase split
+    │   ├── case.rs              # stage 5: case folding
+    │   ├── stopwords.rs         # stage 6: stop words
+    │   ├── synonyms.rs          # stage 7: synonym expand
+    │   ├── stem.rs              # stage 8: Snowball stemmer
+    │   ├── ngram.rs             # stage 9: n-gram extract
+    │   └── token_result.rs      # TokenResult struct
     └── error/
         ├── mod.rs               # error module facade
         └── error.rs             # FsError, IndexError, etc.
@@ -147,6 +159,19 @@ counterparts: `module.md` = `mod.rs`,
 - [functions.md](content-composer/functions.md) --
   `compose_content()`.
 
+### [pipeline/](pipeline/module.md) -- Tokenization Pipeline
+
+- [functions.md](pipeline/functions.md) --
+  `tokenize()`, `bigrams()`, `trigrams()`. 9-stage
+  pipeline: marker stripping, NFC, segmentation,
+  identifier splitting, case folding, stop words,
+  synonym expansion, stemming, n-grams.
+- [types.md](pipeline/types.md) -- `TokenResult`,
+  `TokenStream`, `Ngram`, `NgramType`, `StemResult`.
+
+Consumers: `kbdb-query-parser` (re-exports via WASM
+FFI), `kbdb-default-embedding` (TF-IDF vectors).
+
 ### [error/](error/module.md) -- Error Types
 
 - [types.md](error/types.md) -- `FsError`,
@@ -157,6 +182,9 @@ counterparts: `module.md` = `mod.rs`,
 ```toml
 [dependencies]
 wasm-bindgen = "0.2"
+unicode-segmentation = "1"
+unicode-normalization = "0.1"
+rust-stemmers = "1"
 ```
 
 ## Testing

package/docs/modules/rust/shared/pipeline/functions.md

@@ -0,0 +1,141 @@
+# pipeline -- Tokenization Pipeline Functions
+
+*9-stage pipeline that converts raw text into
+normalized, stemmed tokens with synonym expansion.
+Used for both indexing and query processing.*
+
+## Source
+
+`engine/crates/shared/src/pipeline/`
+
+## Entry Point
+
+```rust
+pub fn tokenize(input: &str) -> TokenStream;
+```
+
+Runs all nine stages in sequence. Each stage is a
+pure function.
+
+## Stages
+
+### 1. Content Marker Stripping
+
+```rust
+use kbdb_shared::content_parser::strip_content_markers;
+```
+
+Removes all six content marker types from the input
+text before tokenization:
+`$(image:)`, `$(code:)`, `$(text:)`, `$(document:)`,
+`$(link:)`, `$(doc-link:)`.
+
+Delegates to `strip_content_markers()` from the
+sibling `content_parser` module (see
+[Content Parser](../content-parser/module.md)).
+These are structural references, not searchable
+content.
+
+### 2. Unicode Normalization (`unicode.rs`)
+
+```rust
+pub fn normalize_nfc(input: &str) -> String;
+```
+
+NFC normalization for consistent codepoints.
+
+### 3. Word Boundary Detection (`segment.rs`)
+
+```rust
+pub fn segment_words(input: &str) -> Vec<&str>;
+```
+
+Unicode Segmentation (UAX #29).
+
+### 4. Identifier Splitting (`identifiers.rs`)
+
+```rust
+pub fn split_identifiers(word: &str) -> Vec<String>;
+```
+
+- `camelCase` → `["camel", "case"]`
+- `snake_case` → `["snake", "case"]`
+- `PascalCase` → `["pascal", "case"]`
+- `HTMLParser` → `["html", "parser"]`
+
+### 5. Case Folding (`case.rs`)
+
+```rust
+pub fn fold_case(word: &str) -> String;
+```
+
+Unicode case folding (lowercase).
+
+### 6. Stop Word Removal (`stopwords.rs`)
+
+```rust
+pub fn is_stop_word(word: &str) -> bool;
+pub fn remove_stop_words(
+   words: &[String],
+) -> Vec<String>;
+```
+
+Filters common function words. Single-character
+tokens also removed.
+
+### 7. Synonym Expansion (`synonyms.rs`)
+
+```rust
+pub fn expand_synonyms(
+   word: &str,
+   map: &SynonymMap,
+) -> Vec<String>;
+```
+
+Looks up the lowercased word in the built-in synonym
+map. Returns a vector containing the original word
+plus any synonyms. When no synonyms exist, returns a
+single-element vector containing only the original.
+
+The synonym map is a bidirectional
+`HashMap<String, Vec<String>>` built at compile time.
+Expansion is applied after stop-word removal and
+before stemming, so both original and synonym tokens
+are stemmed consistently.
+
+The map is small (~30-50 entries) and focused on
+common IT/development vocabulary mismatches:
+login/sign-in, auth/authentication, remove/delete,
+config/configuration, credential/password, etc.
+
+### 8. Stemming (`stem.rs`)
+
+```rust
+pub fn stem_word(word: &str) -> StemResult;
+```
+
+Snowball / Porter2. Preserves both original and
+stemmed forms.
+
+### 9. N-gram Extraction (`ngram.rs`)
+
+```rust
+pub fn extract_ngrams(
+   tokens: &[Token],
+) -> Vec<Ngram>;
+```
+
+Bigrams and trigrams for phrase indexing.
+
+## Relaxed Query Mode
+
+The tokenization pipeline is used for both the
+primary AND-mode query plan and the relaxed OR-mode
+fallback plan. `qp_parse_query` produces a
+`QueryPlan` with a `match_mode` field (`All` | `Any`).
+When the AND plan returns zero matches,
+`kb-worker.wasm` retries with the OR plan using the
+same tokenized terms. No additional pipeline stage or
+WASM export is needed. See
+[Query Parser -- Relaxed Query
+Mode](../../../../goals/query-parser.md).

package/docs/modules/rust/shared/pipeline/module.md

@@ -0,0 +1,62 @@
+# pipeline -- Tokenization Pipeline
+
+*9-stage pipeline that converts raw text into
+normalized, stemmed tokens with synonym expansion.
+Shared across all crates that need text
+tokenization.*
+
+## Source
+
+`engine/crates/shared/src/pipeline/`
+
+## Responsibility
+
+The pipeline module provides the canonical
+tokenization function used for both indexing and
+query processing. All crates that need to tokenize
+text import from this single source of truth.
+
+## Consumers
+
+- **kbdb-query-parser** -- re-exports via the
+  `qp_tokenize` WASM FFI export. Uses internally
+  for query parsing (`parse`), keyword extraction
+  (`extract/rake`), and n-gram generation.
+- **kbdb-default-embedding** -- tokenizes text
+  before feature hashing to produce TF-IDF vectors
+  that are token-consistent with the lexical index.
+
+## Directory Layout
+
+```
+engine/crates/shared/src/pipeline/
+├── mod.rs               # pipeline orchestrator
+├── markers.rs           # stage 1: strip markers
+├── unicode.rs           # stage 2: NFC normalize
+├── segment.rs           # stage 3: word boundaries
+├── identifiers.rs       # stage 4: camelCase split
+├── case.rs              # stage 5: case folding
+├── stopwords.rs         # stage 6: stop words
+├── synonyms.rs          # stage 7: synonym expansion
+├── stem.rs              # stage 8: Snowball stemmer
+├── ngram.rs             # stage 9: n-gram extract
+└── token_result.rs      # TokenResult struct
+```
+
+## Exports
+
+- **[functions.md](functions.md)** -- `tokenize()`,
+  `bigrams()`, `trigrams()`. Full 9-stage pipeline.
+- **[types.md](types.md)** -- `TokenResult`,
+  `TokenStream`, `Ngram`, `NgramType`, `StemResult`.
+
+## Dependencies
+
+The pipeline brings three external crates into
+`kbdb-shared`:
+
+```toml
+unicode-segmentation = "1"     # stage 3
+unicode-normalization = "0.1"  # stage 2
+rust-stemmers = "1"            # stage 8
+```

package/docs/modules/rust/shared/pipeline/types.md

@@ -0,0 +1,43 @@
+# pipeline -- Tokenization Types
+
+*Output types from the tokenization pipeline.*
+
+## Source
+
+`engine/crates/shared/src/pipeline/`
+
+## Type Definitions
+
+```rust
+pub struct TokenResult {
+   pub original: String,
+   pub stemmed: String,
+   pub position: u32,
+}
+
+pub struct TokenStream {
+   pub tokens: Vec<TokenResult>,
+   pub ngrams: Vec<Ngram>,
+}
+
+pub struct Ngram {
+   pub terms: Vec<String>,
+   pub gram_type: NgramType,
+}
+
+pub enum NgramType { Bigram, Trigram }
+
+pub struct StemResult {
+   pub original: String,
+   pub stemmed: String,
+}
+```
+
+`TokenResult` preserves both original and stemmed
+forms so the index can support exact-match and
+stem-match queries. `position` is the zero-based
+token offset.
+
+`TokenStream` is the output of `tokenize()` -- the
+full set of tokens and extracted n-grams from a
+single input text.

package/docs/modules/rust/shared/section/module.md

@@ -12,7 +12,7 @@ structures for section CRUD operations.*
 Defines the content type system (`SectionType`,
 `MimeType`) and the data structures for section CRUD
 operations (`SectionRecord`, `AddSectionParams`,
-etc.) shared across all three WASM modules.
+etc.) shared across all four WASM modules.
 
 See [Database -- Sections](../../../../goals/database.md)
 for the behavioral specification of section types.

package/docs/modules/rust/shared/section/types.md

@@ -2,7 +2,7 @@
 
 *Section type enum, MIME type wrapper, and the data
 structures for section CRUD operations across all
-three WASM modules.*
+four WASM modules.*
 
 ## File
 

package/docs/modules/typescript/cli.md

@@ -20,8 +20,9 @@ The npm `bin` entry `kbdb` points to `dist/cli.mjs`.
 ## Run Modes
 
 ```
-kbdb mcp [--db <path>]       → MCP server mode
-kbdb <command> [options]      → CLI mode (default)
+kbdb mcp [--db <path>]             → MCP server mode
+kbdb <command> [options]            → CLI mode
+kbdb --non-interactive <command>    → non-interactive
 ```
 
 The executable inspects its first argument. If `mcp`,
@@ -73,7 +74,8 @@ src/cli.ts
 src/cli.ts
   │
   ├─ argv[0] === 'mcp'
-  ├─ resolve --db flag or KBDB_DIR env
+  ├─ resolve --db flag or KBDB_DB_DIR env or cwd
+  ├─ resolve --non-interactive or KBDB_NON_INTERACTIVE
   ├─ client = await createWorkerClient({ targetDir })
   ├─ server = createMcpServer(client)
   ├─ server.listen()   ← stdio, blocks until closed

package/docs/modules/typescript/kbdb-worker.md

@@ -29,8 +29,8 @@ no running daemon is found for the target database:
 ```ts
 spawn(
    'node',
-   ['kbdb-worker.js', contextPath],
-   { detached: true, stdio: ['ignore', 'ignore', 'pipe'] },
+   ['kbdb-worker.js', contextPath, '--log-level', logLevel],
+   { detached: true, stdio: 'ignore' },
 ).unref();
 ```
 
@@ -41,6 +41,7 @@ new Deno.Command('deno', {
    args: [
       'run', '--allow-all',
       'src/worker.ts', contextPath,
+      '--log-level', logLevel,
    ],
    stdin: 'null',
    stdout: 'null',
@@ -48,8 +49,16 @@ new Deno.Command('deno', {
 }).spawn().unref();
 ```
 
-The sole command-line argument is the **context path**
+The first command-line argument is the **context path**
 -- the absolute path to the `.kbdb` database directory.
+The optional `--log-level <level>` argument controls
+daemon log verbosity. It is forwarded from the
+`KBDB_LOG_LEVEL` environment variable by the spawner.
+
+The `KBDB_NON_INTERACTIVE` environment variable is
+inherited by the daemon through normal process
+environment inheritance (no explicit forwarding
+needed).
 
 ## Execution Flow
 
@@ -60,6 +69,9 @@ src/worker.ts
   │
   ├─ read contextPath from argv[0]
   ├─ validate contextPath exists
+  ├─ parse --log-level from argv (if present)
+  ├─ configure log level
+  │    (CLI arg → KBDB_LOG_LEVEL env → ERROR default)
   │
   └─ await startDaemon(contextPath)
        │
@@ -70,8 +82,10 @@ src/worker.ts
        │    ├─ allocate shared WebAssembly.Memory
        │    ├─ compile query-parser.wasm (standalone)
        │    ├─ compile fs-database.wasm (standalone)
+       │    ├─ compile default-embedding.wasm
+       │    │    (standalone, no host imports)
        │    └─ compile kb-worker.wasm
-       │         (imports both modules)
+       │         (imports query-parser + fs-database)
        │
        ├─ write PID file
        │    └─ kb-worker-{ctx}.pid in tmpdir
@@ -114,12 +128,13 @@ single function call. All daemon logic lives in
 | Query orchestration| WASM (`kb-worker.wasm`)     |
 | Index operations   | WASM (`fs-database.wasm`)   |
 | Tokenization       | WASM (`query-parser.wasm`)  |
+| TF-IDF embedding   | WASM (`default-embedding.wasm`) |
 | Platform helpers   | `shared/platform/`          |
 
 ## Lifecycle
 
 1. Spawned by worker-client as detached child process
-2. Loads and compiles 3 WASM modules (once)
+2. Loads and compiles 4 WASM modules (once)
 3. Serves IPC requests from any connected client
 4. Shuts down after idle timeout (`timeout_ms`) or
    OS signal (SIGTERM, SIGINT)

package/docs/modules/typescript/overview.md

@@ -58,7 +58,7 @@ contains no business logic -- it dispatches to
 ### `src/worker.ts`
 
 The worker daemon. Spawned as a detached child process
-by the worker-client module. Loads the three WASM
+by the worker-client module. Loads the four WASM
 modules, listens on a local IPC socket, and serves
 JSON-RPC 2.0 requests until idle timeout.
 
@@ -230,10 +230,27 @@ src/shared/
 │       ├── get-tmpdir.function.ts
 │       └── is-process-alive.function.ts
 │
-└── hash/                            ← SHA-256 hashing
+├── hash/                            ← SHA-256 hashing
+│   ├── index.ts
+│   └── functions/
+│       └── compute-sha256.function.ts
+│
+└── log/                             ← Structured logging
     ├── index.ts
-    └── functions/
-        └── compute-sha256.function.ts
+    ├── constants/
+    │   ├── log-colors.constant.ts
+    │   ├── log-defaults.constant.ts
+    │   └── log-icons.constant.ts
+    ├── functions/
+    │   ├── configure-log.function.ts
+    │   ├── log.function.ts
+    │   ├── parse-log-level.function.ts
+    │   ├── resolve-log-level.function.ts
+    │   └── write-stderr.function.ts
+    └── typings/
+        ├── log-config.interface.ts
+        ├── log-level.type.ts
+        └── log-output.interface.ts
 ```
 
 ## Module Conventions
@@ -368,9 +385,14 @@ docs/modules/typescript/
     ├── platform/
     │   ├── module.md
     │   └── functions.md
-    └── hash/
+    ├── hash/
+    │   ├── module.md
+    │   └── functions.md
+    └── log/
         ├── module.md
-        └── functions.md
+        ├── functions.md
+        ├── typings.md
+        └── constants.md
 ```
 
 `module.md` = `index.ts` (module entry point).