@kedem/okdb 1.6.1 → 1.6.2

package/docs/OKDB-ARCHITECTURE.md

@@ -63,7 +63,9 @@ independent compaction, encryption, and sync control.
   queue/             ← Queue jobs/payloads/stats          (sync: on)
   embeddings/        ← Embedding doc_status + models      (sync: on)
   vectors/           ← HNSW graph files + LMDB snapshots  (sync: off)
-  fts_<type>_<name>/ ← Per-FTS-index LMDB env             (sync: off, rebuildable)
+  ~fts/
+    <env>/           ← Shared FTS env: posting lists + dicts  (sync: off, rebuildable)
+    <env>_docs/      ← Shared FTS env: LZ4-compressed forward index
   blobs/             ← File attachment BLOBs
 ```
 
@@ -309,19 +311,27 @@ GET  /api/env/:env/ref-violations/:type       → violations for a specific type
 
 ### 3.4 FTS Environments
 
-Each FTS index gets its own LMDB environment:
+All FTS data for one OKDB env lives in a pair of shared LMDB environments. Individual indexes are distinguished by an auto-increment `ftsId` used as a key prefix — no new files per index.
 
 ```
-fts_<type>_<name>/
-  ├── post  dupSort=true   token → docKey  (inverted posting list)
-  └── docs               docKey → [tokens] (forward index for update/delete)
+~fts/<env>/                                      ← uncompressed env
+  ├── post       dupSort  [ftsId, token] → docId         (live "add" tier)
+  ├── postDel    dupSort  [ftsId, token] → docId         (live "delete" tombstones)
+  ├── frozen              [ftsId, token] → Roaring blob  (compacted tier)
+  ├── dict                [type, 'k', docKey] → docId    (docKey ↔ docId interning)
+  └── tokenDict           ['k', token] → tokenId         (token ↔ tokenId interning)
+
+~fts/<env>_docs/                                 ← LZ4-compressed env
+  └── docs                [ftsId, docId] → tokenId[]     (forward index)
 ```
 
-- Always `sync: false` — fully rebuildable from the default env's type data.
-- FTS metadata (field config, status, created/updated timestamps) is stored in
-  `default/types` under `ftsIndexes[name]` — this IS synced.
-- On inline writes, `putSync`/`removeSync` are used so results are immediately
-  visible within the same event loop tick.
+- **Two-tier writes.** Hot path lands in the live tier (`post` / `postDel`) — O(1) dupSort puts. The compactor periodically merges live entries into per-token Roaring bitmaps in `frozen` and clears the live tier.
+- **Reads** merge all three tiers: `frozen ∪ liveAdds − liveDeletes`. Roaring's set ops keep this essentially free.
+- **ID interning.** `dict` maps (type, docKey) → uint32 docIds so posting-list values and forward-index keys are 4 bytes instead of full strings. `tokenDict` maps tokens → uint32 tokenIds so forward-index values are integer arrays instead of token strings.
+- Always `sync: false` — fully rebuildable from the env's type data.
+- FTS metadata (field config, status, created/updated timestamps, storage `format` version) is stored in `<env>/types` under `ftsIndexes[name]` — this IS synced. The storage format auto-migrates on open by triggering a rebuild for any index whose format is older than the current code.
+- Writes are async via a background processor (one per type). `fts.flush(type)` waits for the processor and runs the compactor.
+- `fts.compactStorage(env)` reclaims slack inside the FTS data.mdb files (LMDB never shrinks files in place). `env.compact()` runs both the main-data and FTS compactions in one operation.
 
 ### 3.5 Vectors Environment
 

package/docs/data-model.md

@@ -177,14 +177,27 @@ const keys = okdb.fts.search('articles', 'main', 'hello');
 
 ### `flush(type)`
 
-`fts.flush(type)` awaits the background processor catching up to the current write position. Returns immediately if no processor is running or if already caught up. Use this instead of `setTimeout` workarounds in tests or after bulk imports.
+`fts.flush(type)` awaits the background processor catching up to the current write position, then runs the FTS compactor to roll live-tier entries into the compacted Roaring bitmap tier. Returns immediately if there's nothing to do.
 
 ### list() response
 
-`fts.list(type)` returns an entry per registered index. Each entry includes `processorState` and `lag` — per-type fields reflecting the single processor that drives all indexes on that type:
+`fts.list(type)` returns an entry per registered index. Each entry includes:
 
-- `processorState`: `'building' | 'online' | 'error' | 'waiting' | null` — `null` if no processor is running (no writes have occurred yet).
-- `lag`: number of write operations the processor has not yet indexed. `0` means fully caught up. `null` if no processor is running.
+- `processorState`: `'building' | 'online' | 'error' | 'waiting' | null` — `null` if no processor is running (no writes have occurred yet). Same value for every entry of the same type.
+- `lag`: number of write operations the processor has not yet indexed. `0` means fully caught up. `null` if no processor.
+- `sizeBytes`: approximate per-index payload (frozen blobs + live entries + forward index scoped by this index's `ftsId`). Excludes env-shared dictionaries.
+
+### Storage architecture
+
+FTS uses a two-tier inverted index. New writes land in a small **live** tier (LMDB dupSort) on the hot write path. A background **compactor** periodically merges live entries into the **frozen** tier — one Roaring bitmap per token. Reads merge all tiers transparently.
+
+Documents and tokens are interned to uint32 IDs (`dict` and `tokenDict` sub-DBs) so posting lists and the forward index store compact integer arrays rather than strings.
+
+Each registered index carries a storage `format` integer; older formats auto-rebuild on open. See [FTS](./fts.md) for the full schema and the storage-size APIs (`fts.list(type)[].sizeBytes`, `fts.envSize()`).
+
+### Reclaiming FTS slack
+
+LMDB never shrinks `data.mdb` in place — freed pages are reused by future writes but the file stays at its peak high-water mark. `env.compact()` reclaims that slack for both the primary env and the FTS envs in one pass. `fts.compactStorage(env)` is the FTS-only equivalent if you want to reclaim just FTS without disturbing primary storage.
 
 ---
 

package/docs/fts.md

@@ -1,6 +1,6 @@
 # Full-Text Search
 
-OKDB includes a built-in full-text search engine backed by an LMDB inverted index. Each FTS index lives in its own LMDB environment (separate directory) so it can be independently compacted or rebuilt without touching user data.
+OKDB includes a built-in full-text search engine. All FTS data for an environment lives in two shared LMDB environments — one inverted/uncompressed (posting lists + dictionaries), one compressed (forward index). Indexes are differentiated by an auto-increment `ftsId` used as a key prefix inside the shared envs, so adding indexes adds no new files and no per-index LMDB overhead.
 
 ---
 
@@ -21,16 +21,19 @@ await okdb.fts.register('articles', 'content', {
 The index builds from existing data immediately after registration. Wait for it:
 
 ```javascript
-await okdb.fts.ready('articles', 'content');
+await okdb.fts.ready('articles');
 ```
 
 ### Multiple FTS indexes on one type
 
-You can have several FTS indexes on the same type, each covering different fields or using different tokenizer settings:
+You can have several FTS indexes on the same type, each covering different fields or using different tokenizer settings. All of them share the env-level storage — adding a second index does not double your disk usage.
 
 ```javascript
 await okdb.fts.register('products', 'name', { fields: ['name'] });
-await okdb.fts.register('products', 'description', { fields: ['description'], tokenizer: { minTokenLength: 3 } });
+await okdb.fts.register('products', 'description', {
+    fields: ['description'],
+    tokenizer: { minTokenLength: 3 },
+});
 ```
 
 ---
@@ -44,14 +47,13 @@ for (const { key, value } of results) {
 }
 ```
 
-`ftsQuery` returns records that contain **any** of the query tokens (OR semantics). Results are not ranked by relevance — they are returned in primary-key order.
+`ftsQuery` returns records that contain all of the query tokens by default (AND). Use `options.mode: 'or'` for OR semantics, and `options.prefix: true` for prefix matching.
 
 ### Combining FTS with field filters
 
 Pass a sift-compatible filter as the fourth argument to narrow results:
 
 ```javascript
-// Articles matching the query that are also published
 const results = okdb.ftsQuery('articles', 'content', 'database performance', {
     status: 'published',
 });
@@ -62,7 +64,8 @@ const results = okdb.ftsQuery('articles', 'content', 'database performance', {
 ```javascript
 const results = okdb.ftsQuery('articles', 'content', 'hello world', null, {
     limit: 20,
-    offset: 0,
+    mode: 'and', // 'and' (default) | 'or'
+    prefix: false, // enable prefix matching on tokens
 });
 ```
 
@@ -77,9 +80,28 @@ okdb.fts.has('articles', 'content'); // → boolean
 // Get status
 okdb.fts.status('articles', 'content'); // → 'creating' | 'ready' | null
 
-// List all FTS indexes on a type
+// List all FTS indexes on a type — includes runtime state
 okdb.fts.list('articles');
-// → [{ name, status, config, created, updated, sizeBytes }]
+// → [{
+//     name, status, config, created, updated, error,
+//     processorState,  // per-type async processor state
+//     lag,             // writes not yet indexed (0 = caught up)
+//     sizeBytes,       // approximate bytes for this index (excludes shared dicts)
+//   }]
+
+// Env-level storage summary across all FTS indexes
+okdb.fts.envSize();
+// → {
+//     diskBytes,            // ~fts/<env>/ + ~fts/<env>_docs/ data.mdb file sizes
+//     payloadBytes,         // actual stored bytes (indexes + shared dicts)
+//     indexPayloadBytes,    // payload attributable to indexes only
+//     sharedDictBytes,      // shared docKey↔docId and token↔tokenId mappings
+//     slackBytes,           // LMDB free pages (reclaimable via env.compact)
+//     indexes: [{ type, name, sizeBytes }],
+//   }
+
+// Reclaim FTS slack — usually called via env.compact() which does both data and FTS
+await okdb.fts.compactStorage();
 
 // Drop an index
 await okdb.fts.drop('articles', 'content');
@@ -87,31 +109,102 @@ await okdb.fts.drop('articles', 'content');
 
 ---
 
-## How it works
+## Storage architecture
 
-OKDB's FTS engine is a simple inverted posting list:
+OKDB's FTS uses a two-tier inverted index with Roaring bitmaps and ID interning. The names below are the actual LMDB sub-DB names in `~fts/<envName>/data.mdb`:
 
-1. On every write, the changed record's indexed fields are tokenized
-2. Removed tokens are deleted from the posting list (`post` sub-db)
-3. Added tokens are inserted (`token → docKey` pairs in a `dupSort` LMDB sub-db)
-4. A forward index (`docs` sub-db: `docKey → [tokens]`) tracks which tokens each doc holds, enabling correct updates
+```
+~fts/<envName>/data.mdb                                (uncompressed env)
+  post       dupSort   [ftsId, token] → docId         (live "add" tier)
+  postDel    dupSort   [ftsId, token] → docId         (live "delete" tombstones)
+  frozen               [ftsId, token] → Roaring blob  (compacted tier)
+  dict                 [type, 'k', docKey] → docId    (docKey ↔ docId mapping)
+                       [type, 'i', docId] → docKey
+                       [type, 'n'] → counter
+  tokenDict            ['k', token] → tokenId         (token ↔ tokenId mapping)
+                       ['i', tokenId] → token
+                       ['n'] → counter
+
+~fts/<envName>_docs/data.mdb                           (LZ4-compressed env)
+  docs                 [ftsId, docId] → tokenId[]     (forward index)
+```
 
-Token processing (default):
+**Why these pieces:**
 
-- Split on whitespace and punctuation
-- Lowercase
-- Discard tokens shorter than `minTokenLength` (default 2) or longer than `maxTokenLength` (default 64)
-- Numbers kept by default (`keepNumbers: true`)
+| Sub-DB             | Purpose                                                                                                                          | Notes                                                                   |
+| ------------------ | -------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
+| `post` / `postDel` | Live tier — every write goes here first. dupSort makes inserts O(1).                                                             | Compactor periodically merges these into `frozen`.                      |
+| `frozen`           | Compacted tier — one Roaring bitmap per (ftsId, token) holds all matching docIds.                                                | 10–50× smaller than the equivalent dupSort entries for dense postings.  |
+| `dict`             | Per-type docKey ↔ docId interning. uint32 docIds replace string keys in posting lists.                                           | One mapping per type, shared across all that type's indexes.            |
+| `tokenDict`        | Env-wide token ↔ tokenId interning. uint32 tokenIds replace token strings in `docs`.                                             | "the" is the same word in every collection — one map for the whole env. |
+| `docs`             | Forward index — used by the indexer to diff old vs new tokens on every write. LZ4-compressed because token arrays compress well. | Read on every doc update, not on search.                                |
 
-### Storage layout
+**Reads** merge all three tiers: `frozen ∪ liveAdds − liveDeletes`. Roaring's set operations make this essentially free.
 
+**Writes** only touch the live tier — never the frozen tier. The hot path is one dupSort `putSync` per added token.
+
+### Compaction
+
+The compactor periodically rolls live entries into the frozen tier. It runs in two situations:
+
+1. **Automatically after `fts.flush(type)`** — keeps the live tier small after a known burst of writes.
+2. **Explicitly via `fts.compactStorage(env)` or `env.compact()`** — also reclaims `data.mdb` slack by rewriting the file (LMDB never shrinks files in place; freed pages are reused but the high-water mark stays).
+
+After a stress run of 13k docs across 5 indexes, `env.compact()` typically reclaims ~75% of the FTS file size in a few seconds, with no impact on search.
+
+### Storage format versioning
+
+Each registered index carries a `format` integer in its metadata. The schema has evolved (legacy → docId interning → two-tier Roaring → token interning); on every open, OKDB checks each index's format and auto-rebuilds any whose format is older than the current code. This is transparent — no manual migration needed. The rebuild reads from the type's primary data, which is the source of truth.
+
+---
+
+## Async indexing
+
+FTS writes are **always async** — the FTS index is updated by a background processor after each write commits. Secondary indexes (field indexes, geo indexes) update synchronously inside the same LMDB transaction as the write, so they are immediately consistent. FTS isn't.
+
+```javascript
+await okdb.put('articles', 'a1', { title: 'Hello world' });
+// At this point a1 is in the primary data but NOT YET in the FTS index.
+
+await okdb.fts.flush('articles');
+// Now it is — and the live tier has been compacted into the frozen tier.
+
+const results = okdb.fts.search('articles', 'main', 'hello'); // includes a1
 ```
-~fts/<envName>/<type>/<indexName>/
-  post    dupSort=true   token → docKey   (inverted list)
-  docs                   docKey → tokens  (forward index)
-```
 
-FTS environments are `sync: false` — they are never replicated. On a new node they rebuild from synced type data automatically.
+### `ready(type)`
+
+`fts.ready(type)` resolves when all FTS indexes registered on `type` have completed their initial build. The optional second argument is accepted for backward compatibility but ignored — readiness is per-type, not per-index.
+
+### `flush(type)`
+
+`fts.flush(type)` awaits the background processor catching up to the current write position, then runs the compactor. Returns immediately if there's nothing to do.
+
+### Processor state in `list()`
+
+`processorState` and `lag` are per-type fields reflecting the single background processor that drives all of a type's indexes. Both have the same value on every entry of the same type.
+
+- `processorState`: `'building' | 'online' | 'error' | 'waiting' | null` — `null` if no writes have happened yet.
+- `lag`: count of writes the processor hasn't indexed yet. `0` means caught up.
+
+---
+
+## Tokenizer
+
+Default behaviour:
+
+- Splits on whitespace and punctuation
+- Lowercases (when `toLower: true`, default)
+- Discards tokens shorter than `minTokenLength` (default 2) or longer than `maxTokenLength` (default 64)
+- Keeps numbers by default (`keepNumbers: true`)
+
+You can override per-index in the `register()` config.
+
+---
+
+## Sync behaviour
+
+FTS environments are not replicated. On a new node they rebuild from synced type data automatically — the source of truth is the primary data plus the change log, both of which are sync'd.
 
 ---
 
@@ -122,7 +215,7 @@ await okdb.ensureType('articles');
 await okdb.fts.register('articles', 'search', {
     fields: ['title', 'body', 'tags'],
 });
-await okdb.fts.ready('articles', 'search');
+await okdb.fts.ready('articles');
 
 await okdb.put('articles', 'a1', {
     title: 'Getting started with LMDB',
@@ -130,7 +223,8 @@ await okdb.put('articles', 'a1', {
     tags: ['database', 'lmdb', 'storage'],
 });
 
-const results = okdb.ftsQuery('articles', 'search', 'LMDB key value');
+await okdb.fts.flush('articles');
+const results = okdb.fts.search('articles', 'search', 'LMDB key value');
 // → includes 'a1'
 ```
 

package/docs/http-api.md

@@ -461,6 +461,7 @@ GET  /api/:env/type/:type/index/:idx/count?start=<json>&end=<json>  → count in
 ## Full-text search endpoints
 
 ```http
+GET    /api/:env/fts/size                          → FTS storage summary for the env
 GET    /api/:env/type/:type/fts                    → list FTS indexes
 POST   /api/:env/type/:type/fts/:name              → register a new FTS index
 DELETE /api/:env/type/:type/fts/:name              → drop an FTS index
@@ -469,26 +470,40 @@ POST   /api/:env/type/:type/fts/:name/search       → search an FTS index
 POST   /api/:env/type/:type/fts/flush              → flush FTS processor (await catch-up)
 ```
 
+### GET /api/:env/fts/size — env-level storage summary
+
+Returns a single object summarizing FTS storage for the environment. Useful for "how much disk is FTS costing me" overviews and for spotting reclaimable slack.
+
+| Field               | Type     | Description                                                                                  |
+| ------------------- | -------- | -------------------------------------------------------------------------------------------- |
+| `diskBytes`         | `number` | Sum of `~fts/<env>/` + `~fts/<env>_docs/` data.mdb file sizes. LMDB high-water mark.         |
+| `payloadBytes`      | `number` | Actual stored bytes (sum of all index payloads + shared dictionaries).                       |
+| `indexPayloadBytes` | `number` | Payload attributable to indexes only (excludes shared dictionaries).                         |
+| `sharedDictBytes`   | `number` | Bytes used by the env-shared docKey↔docId and token↔tokenId dictionaries.                    |
+| `slackBytes`        | `number` | `diskBytes - payloadBytes`. Free LMDB pages — reclaimable by future writes or `env.compact`. |
+| `indexes`           | `array`  | Per-index breakdown: `[{ type, name, sizeBytes }, ...]`.                                     |
+
 ### GET /api/:env/type/:type/fts — list response
 
 Each entry in the returned array includes:
 
-| Field            | Type                                                        | Description                                                                                        |
-| ---------------- | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
-| `name`           | `string`                                                    | Index name                                                                                         |
-| `status`         | `'creating' \| 'resetting' \| 'ready' \| 'waiting' \| null` | Per-index lifecycle status                                                                         |
-| `config`         | `object \| null`                                            | Index configuration as registered                                                                  |
-| `created`        | `number \| null`                                            | HLC clock at registration time                                                                     |
-| `updated`        | `number \| null`                                            | HLC clock at last config update                                                                    |
-| `error`          | `object \| null`                                            | Last per-index error, if any                                                                       |
-| `processorState` | `'building' \| 'online' \| 'error' \| 'waiting' \| null`    | State of the background FTS processor for this type. `null` if no processor is running.            |
-| `lag`            | `number \| null`                                            | Number of writes the processor has not yet indexed. `0` = fully caught up. `null` if no processor. |
+| Field            | Type                                                        | Description                                                                                                                           |
+| ---------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
+| `name`           | `string`                                                    | Index name                                                                                                                            |
+| `status`         | `'creating' \| 'resetting' \| 'ready' \| 'waiting' \| null` | Per-index lifecycle status                                                                                                            |
+| `config`         | `object \| null`                                            | Index configuration as registered                                                                                                     |
+| `created`        | `number \| null`                                            | HLC clock at registration time                                                                                                        |
+| `updated`        | `number \| null`                                            | HLC clock at last config update                                                                                                       |
+| `error`          | `object \| null`                                            | Last per-index error, if any                                                                                                          |
+| `processorState` | `'building' \| 'online' \| 'error' \| 'waiting' \| null`    | State of the background FTS processor for this type. `null` if no processor is running.                                               |
+| `lag`            | `number \| null`                                            | Number of writes the processor has not yet indexed. `0` = fully caught up. `null` if no processor.                                    |
+| `sizeBytes`      | `number`                                                    | Approximate per-index payload (frozen blobs + live entries + forward index scoped by this `ftsId`). Excludes env-shared dictionaries. |
 
 `processorState` and `lag` are per-type fields — they reflect the single background processor that drives all FTS indexes on a type and are identical on every entry of the same type.
 
 ### POST /api/:env/type/:type/fts/flush — flush processor
 
-Waits for the background FTS processor to catch up to the current write position for this type. Responds `204 No Content` when fully caught up. This is a no-op if no processor is running or if the processor is already caught up.
+Waits for the background FTS processor to catch up to the current write position for this type, then runs the FTS compactor to roll live-tier entries into the compacted Roaring bitmap tier. Responds `204 No Content` when fully caught up. This is a no-op if no processor is running or if the processor is already caught up.
 
 - **Permission:** `schema:read`
 - **Idempotent:** yes

package/docs/transactions.md

@@ -195,4 +195,4 @@ await okdb.fts.flush('articles'); // wait for processor to index the write
 const results = okdb.fts.search('articles', 'main', 'hello'); // guaranteed to include a1
 ```
 
-`fts.flush()` is a no-op if the processor is already caught up. It is safe to call unconditionally.
+`fts.flush()` is a no-op if the processor is already caught up. It is safe to call unconditionally. In addition to waiting for the processor, `flush()` runs the FTS compactor — rolling live-tier entries into the compacted Roaring bitmap tier — so search performance and storage stay stable under bursty writes. See [Full-Text Search](./fts.md) for the storage architecture.