@dikolab/kbdb 0.3.2 → 0.4.1

package/docs/modules/overview.md

@@ -12,9 +12,10 @@ final `dist/` output.*
 
  engine/crates/              src/shared/
  ├── shared (rlib)           ├── worker-daemon/
- ├── fs-database (cdylib)    │   ├── wasm/          ← 3 WASM modules
+ ├── fs-database (cdylib)    │   ├── wasm/          ← 4 WASM modules
  ├── query-parser (cdylib)   │   │   ├── fs-database.*
- └── kb-worker (cdylib)      │   │   ├── query-parser.*
+ ├── default-embedding (cdylib)│  │   ├── query-parser.*
+ └── kb-worker (cdylib)      │   │   ├── default-embedding.*
        │                     │   │   └── kb-worker.*
        │ wasm-pack build     │   └── functions/
        └─────────────────────┘       └── load-wasm-modules
@@ -31,7 +32,7 @@ final `dist/` output.*
 
 ## Phase 1 -- Build Rust to WASM
 
-The Rust engine compiles to three standalone `.wasm`
+The Rust engine compiles to four standalone `.wasm`
 files. The `kbdb-shared` crate is an `rlib` -- it
 produces no artifact, compiled **into** each consumer
 via dead code elimination.
@@ -43,15 +44,17 @@ via dead code elimination.
 | `kbdb-shared`      | `rlib`   | none (compiled into consumers) |
 | `kbdb-fs-database` | `cdylib` | `fs-database.wasm`   |
 | `kbdb-query-parser`| `cdylib` | `query-parser.wasm`  |
+| `kbdb-default-embedding` | `cdylib` | `default-embedding.wasm` |
 | `kbdb-worker`      | `cdylib` | `kb-worker.wasm`     |
 
 ### Build Order
 
 ```
-1.  kbdb-shared           (leaf, no deps)
-2.  kbdb-fs-database  ┐   (both depend on shared,
-    kbdb-query-parser ┘    built in parallel)
-3.  kbdb-worker           (depends on all three)
+1.  kbdb-shared                (leaf, no deps)
+2.  kbdb-fs-database       ┐   (all depend on shared,
+    kbdb-query-parser      │    built in parallel)
+    kbdb-default-embedding ┘
+3.  kbdb-worker                (depends on all above)
 ```
 
 ### Build Command
@@ -62,20 +65,24 @@ npm run build:wasm
 
 This runs `wasm-pack build` for each cdylib crate.
 Output lands in the TypeScript source tree at
-`src/shared/worker-daemon/wasm/`:
+`src/wasm/`:
 
 ```sh
 wasm-pack build crates/fs-database \
    --target no-modules \
-   --out-dir ../../../../src/shared/worker-daemon/wasm/fs-database
+   --out-dir ../../../src/wasm/fs-database
 
 wasm-pack build crates/query-parser \
    --target no-modules \
-   --out-dir ../../../../src/shared/worker-daemon/wasm/query-parser
+   --out-dir ../../../src/wasm/query-parser
+
+wasm-pack build crates/default-embedding \
+   --target no-modules \
+   --out-dir ../../../src/wasm/default-embedding
 
 wasm-pack build crates/kb-worker \
    --target no-modules \
-   --out-dir ../../../../src/shared/worker-daemon/wasm/kb-worker
+   --out-dir ../../../src/wasm/kb-worker
 ```
 
 ### Size Optimization
@@ -115,6 +122,10 @@ src/shared/worker-daemon/
 │   │   ├── query-parser.wasm
 │   │   ├── query-parser.js
 │   │   └── query-parser.d.ts
+│   ├── default-embedding/
+│   │   ├── default-embedding.wasm
+│   │   ├── default-embedding.js
+│   │   └── default-embedding.d.ts
 │   └── kb-worker/
 │       ├── kb-worker.wasm
 │       ├── kb-worker.js
@@ -145,11 +156,14 @@ loadWasmModules(config)
   │      (standalone, shared memory + host I/O)
   │      ↑ steps 2 and 3 run in parallel
   │
-  └─ 4. Compile + instantiate kb-worker.wasm
+  ├─ 4. Compile + instantiate default-embedding.wasm
+  │      (standalone, no host imports -- pure computation)
+  │
+  └─ 5. Compile + instantiate kb-worker.wasm
          (imports query-parser + fs-database exports)
 ```
 
-All three modules share one `WebAssembly.Memory`
+All four WASM modules share one `WebAssembly.Memory`
 instance. Data crosses module boundaries via
 `(pointer, length)` pairs encoded with the `Encode`
 trait from `kbdb-shared`.
@@ -163,7 +177,7 @@ TypeScript module directly accesses WASM files or the
 ## Phase 3 -- TypeScript Modules
 
 All TypeScript logic lives under `src/shared/` in
-8 sub-modules. Two thin entry points at the project
+9 sub-modules. Two thin entry points at the project
 root dispatch into them.
 
 ### Entry Points
@@ -186,24 +200,31 @@ root dispatch into them.
 | `runtime/`       | Deno / Node.js detection   |
 | `platform/`      | OS abstractions (tmpdir, PID, IPC paths) |
 | `hash/`          | SHA-256 via Web Crypto API |
+| `log/`           | Structured logging with configurable levels and ANSI formatting |
 
 ### Module Dependencies
 
 ```
 src/cli.ts
   ├── shared/cli/ ──→ shared/worker-client/
+  │     └── shared/log/
   └── shared/mcp/ ──→ shared/worker-client/
-                          ├── shared/platform/
-                          └── shared/hash/
+        │                   ├── shared/platform/
+        │                   ├── shared/hash/
+        │                   └── shared/log/
+        └── shared/log/
 
 src/worker.ts
   └── shared/worker-daemon/
          ├── shared/platform/
+         ├── shared/log/
          └── wasm/ (fs-database, query-parser,
-                    kb-worker)
+                    default-embedding, kb-worker)
 
 mod.ts
   └── shared/version/
+
+shared/log/          (no dependencies -- standalone)
 ```
 
 **Full TypeScript architecture:**
@@ -266,7 +287,7 @@ are loaded at runtime by the daemon via
 ## Full Build Sequence
 
 ```sh
-# 1. Build WASM (Rust → .wasm → src/shared/worker-daemon/wasm/)
+# 1. Build WASM (Rust → .wasm → src/wasm/)
 npm run build:wasm
 
 # 2. Build TypeScript (src/ → dist/)
@@ -282,8 +303,8 @@ npm run build:all
 build:all
   │
   ├─ build:wasm
-  │    ├─ cargo build (4 crates)
-  │    └─ wasm-pack build (3 cdylib → wasm/)
+  │    ├─ cargo build (5 crates)
+  │    └─ wasm-pack build (4 cdylib → wasm/)
   │
   └─ build
        ├─ esbuild (2 entry points → dist/)
@@ -294,5 +315,5 @@ build:all
 
 | Layer      | Overview                          | Modules |
 |------------|-----------------------------------|---------|
-| Rust/WASM  | [rust/overview.md](rust/overview.md) | 4 crates, 9 shared namespaces |
-| TypeScript | [typescript/overview.md](typescript/overview.md) | 2 entry points, 8 shared modules |
+| Rust/WASM  | [rust/overview.md](rust/overview.md) | 5 crates, 9 shared namespaces |
+| TypeScript | [typescript/overview.md](typescript/overview.md) | 2 entry points, 9 shared modules |

package/docs/modules/rust/embedding/module.md

@@ -0,0 +1,91 @@
+# default-embedding -- TF-IDF Vector Computation
+
+**Source:** `engine/crates/default-embedding/`
+**Crate type:** `cdylib` + `rlib`
+**Output:** `default-embedding.wasm`
+
+## Responsibility
+
+Computes feature-hashed sublinear TF vectors for the
+default embedding provider. Reuses kbdb-shared's
+9-stage tokenization pipeline (NFC normalization,
+Unicode segmentation, camelCase splitting, stop-word
+removal, synonym expansion, Snowball stemming) so
+that TF-IDF vectors are token-consistent with the
+lexical index.
+
+No host imports -- pure computation with no file I/O.
+The WASM module is instantiated without an import
+object.
+
+## Algorithm
+
+1. Tokenize text via `kbdb_shared::pipeline::tokenize`
+2. Count term frequencies on stemmed tokens
+3. Hash each stemmed token to a bucket using FNV-1a:
+   `bucket = fnv1a(token) % 256`
+4. Apply sign trick to reduce collision bias:
+   `sign = if fnv1a(token + "_s") % 2 == 0 { 1 } else { -1 }`
+5. Accumulate `sign * (1.0 + tf.ln())` per bucket
+   (sublinear TF provides diminishing returns for
+   repeated terms)
+6. L2-normalize the vector
+
+Output: 256-dimensional `Float32Array`, suitable for
+cosine similarity via `cosineSimilarity()` in the
+TypeScript embedding module.
+
+## Directory Layout
+
+```
+engine/crates/default-embedding/
+├── Cargo.toml
+└── src/
+    ├── lib.rs               ← WASM exports
+    ├── mem.rs               ← shared-memory helpers
+    ├── fnv1a.rs             ← FNV-1a hash function
+    └── feature_hash.rs      ← feature hashing + L2 norm
+```
+
+## WASM Exports
+
+| Export                        | Signature              | Description |
+|-------------------------------|------------------------|-------------|
+| `embedding_tfidf`             | `(ptr, len) -> u32`    | Single text → 256×f32 vector |
+| `embedding_tfidf_batch`       | `(ptr, len) -> u32`    | Newline-delimited texts → count + count×256 f32s |
+| `embedding_dimensions`        | `() -> u32`            | Returns 256 |
+| `embedding_model_name`        | `(ptr, len) -> u32`    | Writes `tfidf-v1` to return slot |
+
+All exports use the standard return-slot protocol:
+result data is written via `mem::set_return(ptr, len)`
+and read by TypeScript via `get_return_ptr` /
+`get_return_len`.
+
+## Dependencies
+
+```toml
+[dependencies]
+kbdb-shared = { path = "../shared" }
+wasm-bindgen = "0.2"
+```
+
+- **kbdb-shared** -- memory management (`kbdb_alloc`,
+  `kbdb_free`, `set_return`, `read_str`) and
+  tokenization pipeline (`pipeline::tokenize`,
+  `pipeline::TokenResult`)
+
+## Dependency Direction
+
+```
+kbdb-shared (rlib)
+    │   provides: memory, pipeline::tokenize
+    │
+    └──→ kbdb-default-embedding (cdylib)
+             uses: shared memory + shared tokenization
+             consumed by: TypeScript TfidfEmbeddingProvider
+```
+
+The embedding crate is a leaf in the dependency graph.
+No other Rust crate depends on it. It is loaded as a
+standalone WASM instance by the TypeScript worker
+daemon.

package/docs/modules/rust/overview.md

@@ -27,7 +27,7 @@ other file. Code sharing requires a crate dependency.
 ## Workspace
 
 All Rust code lives in `engine/`. The workspace
-contains four crates under `engine/crates/`:
+contains six crates under `engine/crates/`:
 
 ```
 engine/
@@ -38,6 +38,8 @@ engine/
 └── crates/
     ├── shared/                ← rlib (library)
     ├── fs-database/           ← cdylib (→ .wasm)
+    ├── fs-migration/          ← cdylib (→ .wasm)
+    ├── default-embedding/     ← cdylib (→ .wasm)
     ├── query-parser/          ← cdylib (→ .wasm)
     └── kb-worker/             ← cdylib (→ .wasm)
 ```
@@ -61,6 +63,7 @@ WASM binary.
 
 - **`kbdb-fs-database`** → `fs-database.wasm`
 - **`kbdb-query-parser`** → `query-parser.wasm`
+- **`kbdb-default-embedding`** → `default-embedding.wasm`
 - **`kbdb-worker`** → `kb-worker.wasm`
 
 ## Crate Dependency Graph
@@ -74,6 +77,9 @@ kbdb-shared (rlib)
     ├──→ kbdb-query-parser (cdylib)
     │        produces: query-parser.wasm
     │
+    ├──→ kbdb-default-embedding (cdylib)
+    │        produces: default-embedding.wasm
+    │
     └──→ kbdb-worker (cdylib)
              produces: kb-worker.wasm
              also depends on:
@@ -102,11 +108,11 @@ have **no dependency** on each other or on `kb-worker`.
 ## Build Order
 
 ```
-1.  kbdb-shared         (no deps, builds first)
-2.  kbdb-fs-database    (depends on shared)
-    kbdb-query-parser   (depends on shared)
-    ↑ these two build in parallel
-3.  kbdb-worker         (depends on all three)
+1.  kbdb-shared              (no deps, builds first)
+2.  kbdb-fs-database     ┐   (all depend on shared,
+    kbdb-query-parser    │    built in parallel)
+    kbdb-default-embedding┘
+3.  kbdb-worker              (depends on all above)
 ```
 
 ## WASM Output
@@ -119,12 +125,15 @@ dist/wasm/
 ├── query-parser/
 │   ├── query-parser.wasm
 │   └── query-parser.js
+├── embedding/
+│   ├── default-embedding.wasm
+│   └── default-embedding.js
 └── kb-worker/
     ├── kb-worker.wasm
     └── kb-worker.js
 ```
 
-All three modules share one `WebAssembly.Memory`
+All modules share one `WebAssembly.Memory`
 instance at runtime. Data crosses module boundaries
 via `(pointer, length)` pairs encoded with the
 `Encode` trait from `kbdb-shared`.
@@ -209,6 +218,19 @@ engine/crates/shared/
     │   ├── compose.rs
     │   └── render.rs
     │
+    ├── pipeline/            ← 9-stage tokenizer
+    │   ├── mod.rs
+    │   ├── markers.rs
+    │   ├── unicode.rs
+    │   ├── segment.rs
+    │   ├── identifiers.rs
+    │   ├── case.rs
+    │   ├── stopwords.rs
+    │   ├── synonyms.rs
+    │   ├── stem.rs
+    │   ├── ngram.rs
+    │   └── token_result.rs
+    │
     └── error/               ← error hierarchy
         ├── mod.rs
         └── error.rs
@@ -256,8 +278,9 @@ Detailed docs:
 
 ### kbdb-query-parser (cdylib → .wasm)
 
-Tokenization pipeline, query parsing, keyword
-extraction, and ranking.
+Query parsing, keyword extraction, and ranking.
+Tokenization is imported from `kbdb-shared`'s
+`pipeline` module.
 
 ```
 engine/crates/query-parser/
@@ -266,18 +289,6 @@ engine/crates/query-parser/
     ├── lib.rs               ← WASM exports
     ├── mem.rs               ← shared-memory helpers
     │
-    ├── pipeline/            ← 9-stage tokenizer
-    │   ├── mod.rs
-    │   ├── markers.rs
-    │   ├── unicode.rs
-    │   ├── segment.rs
-    │   ├── identifiers.rs
-    │   ├── case.rs
-    │   ├── stopwords.rs
-    │   ├── synonyms.rs
-    │   ├── stem.rs
-    │   └── ngram.rs
-    │
     ├── parse.rs             ← query → QueryPlan
     │
     ├── extract/             ← keyword extraction
@@ -298,6 +309,27 @@ engine/crates/query-parser/
 Detailed docs:
 [query-parser/module.md](query-parser/module.md)
 
+### kbdb-default-embedding (cdylib → .wasm)
+
+TF-IDF embedding computation. Reuses kbdb-shared's
+tokenization pipeline to produce feature-hashed
+sublinear TF vectors (256 dimensions, L2-normalized).
+Depends only on `kbdb-shared`. No host imports --
+pure computation with no file I/O.
+
+```
+engine/crates/default-embedding/
+├── Cargo.toml
+└── src/
+    ├── lib.rs               ← WASM exports
+    ├── mem.rs               ← shared-memory helpers
+    ├── fnv1a.rs             ← FNV-1a hash function
+    └── feature_hash.rs      ← feature hashing + L2 norm
+```
+
+Detailed docs:
+[embedding/module.md](embedding/module.md)
+
 ### kbdb-worker (cdylib → .wasm)
 
 Orchestration layer. Imports the other two WASM
@@ -483,6 +515,9 @@ docs/modules/rust/
 │   ├── functions/
 │   └── ...
 │
+├── embedding/               ← kbdb-default-embedding crate
+│   └── module.md
+│
 └── kb-worker/               ← kbdb-worker crate
     ├── module.md
     ├── types/

package/docs/modules/rust/query-parser/functions/module.md

@@ -1,16 +1,18 @@
 # query-parser/functions -- Operations
 
-*Functions for tokenization, query parsing, keyword
-extraction, and ranking.*
+*Functions for query parsing, keyword extraction,
+and ranking.*
 
 ## Modules
 
-- **[pipeline](pipeline.md)** -- 9-stage tokenization
-  pipeline. Converts raw text into normalized, stemmed
-  tokens with synonym expansion and n-grams.
 - **[parse](parse.md)** -- Query string parsing into
   structured `QueryPlan`.
 - **[extract](extract.md)** -- RAKE keyphrase
   extraction and TF-IDF term weighting.
 - **[rank](rank.md)** -- BM25F, proximity, zone, and
   freshness scoring functions.
+
+Tokenization is provided by
+[kbdb-shared pipeline](../../shared/pipeline/functions.md).
+This crate re-exports it via the `qp_tokenize`
+WASM export.

package/docs/modules/rust/query-parser/functions/pipeline.md

@@ -1,141 +1,9 @@
-# functions::pipeline -- Tokenization Pipeline
+# functions::pipeline -- Moved to kbdb-shared
 
-*9-stage pipeline that converts raw text into
-normalized, stemmed tokens with synonym expansion.
-Used for both indexing and query processing.*
+The tokenization pipeline has moved to `kbdb-shared`.
 
-## Source
+See [shared/pipeline/functions.md](../../shared/pipeline/functions.md)
+for the full documentation.
 
-`engine/crates/query-parser/src/pipeline/`
-
-## Entry Point
-
-```rust
-pub fn tokenize(input: &str) -> TokenStream;
-```
-
-Runs all nine stages in sequence. Each stage is a
-pure function.
-
-## Stages
-
-### 1. Content Marker Stripping
-
-```rust
-use kbdb_shared::content_parser::strip_content_markers;
-```
-
-Removes all six content marker types from the input
-text before tokenization:
-`$(image:)`, `$(code:)`, `$(text:)`, `$(document:)`,
-`$(link:)`, `$(doc-link:)`.
-
-Delegates to `strip_content_markers()` imported from
-`kbdb_shared::content_parser` (see
-[Content Parser](../../shared/content-parser/module.md)).
-These are structural references, not searchable
-content.
-
-### 2. Unicode Normalization (`unicode.rs`)
-
-```rust
-pub fn normalize_nfc(input: &str) -> String;
-```
-
-NFC normalization for consistent codepoints.
-
-### 3. Word Boundary Detection (`segment.rs`)
-
-```rust
-pub fn segment_words(input: &str) -> Vec<&str>;
-```
-
-Unicode Segmentation (UAX #29).
-
-### 4. Identifier Splitting (`identifiers.rs`)
-
-```rust
-pub fn split_identifiers(word: &str) -> Vec<String>;
-```
-
-- `camelCase` → `["camel", "case"]`
-- `snake_case` → `["snake", "case"]`
-- `PascalCase` → `["pascal", "case"]`
-- `HTMLParser` → `["html", "parser"]`
-
-### 5. Case Folding (`case.rs`)
-
-```rust
-pub fn fold_case(word: &str) -> String;
-```
-
-Unicode case folding (lowercase).
-
-### 6. Stop Word Removal (`stopwords.rs`)
-
-```rust
-pub fn is_stop_word(word: &str) -> bool;
-pub fn remove_stop_words(
-   words: &[String],
-) -> Vec<String>;
-```
-
-Filters common function words. Single-character
-tokens also removed.
-
-### 7. Synonym Expansion (`synonyms.rs`)
-
-```rust
-pub fn expand_synonyms(
-   word: &str,
-   map: &SynonymMap,
-) -> Vec<String>;
-```
-
-Looks up the lowercased word in the built-in synonym
-map. Returns a vector containing the original word
-plus any synonyms. When no synonyms exist, returns a
-single-element vector containing only the original.
-
-The synonym map is a bidirectional
-`HashMap<String, Vec<String>>` built at compile time.
-Expansion is applied after stop-word removal and
-before stemming, so both original and synonym tokens
-are stemmed consistently.
-
-The map is small (~30-50 entries) and focused on
-common IT/development vocabulary mismatches:
-login/sign-in, auth/authentication, remove/delete,
-config/configuration, credential/password, etc.
-
-### 8. Stemming (`stem.rs`)
-
-```rust
-pub fn stem_word(word: &str) -> StemResult;
-```
-
-Snowball / Porter2. Preserves both original and
-stemmed forms.
-
-### 9. N-gram Extraction (`ngram.rs`)
-
-```rust
-pub fn extract_ngrams(
-   tokens: &[Token],
-) -> Vec<Ngram>;
-```
-
-Bigrams and trigrams for phrase indexing.
-
-## Relaxed Query Mode
-
-The tokenization pipeline is used for both the
-primary AND-mode query plan and the relaxed OR-mode
-fallback plan. `qp_parse_query` produces a
-`QueryPlan` with a `match_mode` field (`All` | `Any`).
-When the AND plan returns zero matches,
-`kb-worker.wasm` retries with the OR plan using the
-same tokenized terms. No additional pipeline stage or
-WASM export is needed. See
-[Query Parser -- Relaxed Query
-Mode](../../../goals/query-parser.md).
+This crate re-exports the pipeline via the
+`qp_tokenize` WASM FFI export.

package/docs/modules/rust/query-parser/module.md

@@ -1,8 +1,9 @@
 # query-parser -- Query Processing and Tokenization
 
-*Rust crate that provides the tokenization pipeline,
-query parsing, keyword extraction, and ranking
-functions. Compiles to `query-parser.wasm`.*
+*Rust crate that provides query parsing, keyword
+extraction, and ranking functions. Tokenization is
+imported from `kbdb-shared`'s `pipeline` module.
+Compiles to `query-parser.wasm`.*
 
 ## Crate
 
@@ -41,17 +42,6 @@ engine/crates/query-parser/
 └── src/
     ├── lib.rs                   # crate root, WASM exports
     ├── mem.rs                   # shared-memory helpers
-    ├── pipeline/
-    │   ├── mod.rs               # pipeline orchestrator
-    │   ├── markers.rs           # stage 1: strip markers
-    │   ├── unicode.rs           # stage 2: NFC normalize
-    │   ├── segment.rs           # stage 3: word boundaries
-    │   ├── identifiers.rs       # stage 4: camelCase split
-    │   ├── case.rs              # stage 5: case folding
-    │   ├── stopwords.rs         # stage 6: stop words
-    │   ├── synonyms.rs          # stage 7: synonym expansion
-    │   ├── stem.rs              # stage 8: Snowball stemmer
-    │   └── ngram.rs             # stage 9: n-gram extract
     ├── parse.rs                 # query string → QueryPlan
     ├── extract/
     │   ├── mod.rs               # extraction facade
@@ -66,6 +56,11 @@ engine/crates/query-parser/
     └── cache.rs                 # query-scoped cache
 ```
 
+The tokenization pipeline lives in `kbdb-shared`'s
+[pipeline](../shared/pipeline/module.md) module.
+This crate re-exports it via WASM FFI
+(`qp_tokenize`).
+
 ## Modules
 
 ### [types/](types/module.md) -- Structs
@@ -79,10 +74,6 @@ engine/crates/query-parser/
 
 ### [functions/](functions/module.md) -- Operations
 
-- **[pipeline](functions/pipeline.md)** -- 9-stage
-  tokenization pipeline. Marker stripping, NFC,
-  segmentation, identifier splitting, case folding,
-  stop words, synonym expansion, stemming, n-grams.
 - **[parse](functions/parse.md)** -- Query string
   parsing into `QueryPlan`.
 - **[extract](functions/extract.md)** -- RAKE
@@ -90,6 +81,9 @@ engine/crates/query-parser/
 - **[rank](functions/rank.md)** -- BM25F, proximity,
   zone, and freshness scoring functions.
 
+Tokenization is provided by
+[kbdb-shared pipeline](../shared/pipeline/module.md).
+
 ## WASM Exports
 
 | Export                   | Description             |
@@ -114,12 +108,12 @@ engine/crates/query-parser/
 ```toml
 [dependencies]
 kbdb-shared = { path = "../shared" }
-unicode-segmentation = "1"
-unicode-normalization = "0.1"
-rust-stemmers = "1"
-regex = { version = "1", default-features = false }
 ```
 
+Tokenization dependencies (`unicode-segmentation`,
+`unicode-normalization`, `rust-stemmers`) are now in
+`kbdb-shared` alongside the `pipeline` module.
+
 ## Build
 
 ```sh

package/docs/modules/rust/query-parser/types/module.md

@@ -1,13 +1,10 @@
 # query-parser/types -- Type Definitions
 
 *Structs defined in the query-parser crate for
-tokenization, keyword extraction, and ranking.*
+keyword extraction and ranking.*
 
 ## Modules
 
-- **[token](token.md)** -- `Token`, `TokenStream`,
-  `Ngram`, `NgramType`, `StemResult`. Output types
-  from the tokenization pipeline.
 - **[keyphrase](keyphrase.md)** -- `Keyphrase`,
   `WeightedTerm`, `BM25FParams`, `FieldWeights`,
   `SynonymMap`. Types for keyword extraction, ranking
@@ -15,3 +12,7 @@ tokenization, keyword extraction, and ranking.*
 - **[cache](cache.md)** -- `QueryCache`. Short-lived
   cache scoped to a single query or indexing
   operation.
+
+Token types (`Token`, `TokenStream`, `Ngram`,
+`NgramType`, `StemResult`) are defined in
+[kbdb-shared pipeline](../../shared/pipeline/types.md).