@kedem/okdb 1.5.3 → 1.6.1

package/docs/data-model.md

@@ -8,7 +8,7 @@ A **type** is a named collection of records — similar to a table in SQL or a c
 await okdb.registerType('articles');
 ```
 
-Each type gets its own set of LMDB sub-databases: one for data, one for each secondary index, and one for FTS index metadata. All sub-databases live inside the same LMDB environment (directory) and participate in the same transactions.
+Each type gets its own set of LMDB sub-databases: one for data and one for each secondary index. All sub-databases live inside the same LMDB environment (directory) and participate in the same transactions. FTS data is stored in a separate shared LMDB environment pair (see [Storage layout](#storage-layout)).
 
 ### Type operations
 
@@ -118,10 +118,14 @@ mydb/
   ~system/          ← node identity + env registry
   default/          ← your types and records
   ~<env>:emb:<type>/← per-type vector store
-  ~fts/             ← FTS indexes (one sub-dir per index)
+  ~fts/
+    <envName>/      ← shared FTS inverted index (all types + indexes in this env)
+    <envName>_docs/ ← shared FTS forward index (docKey → tokens, compressed)
   blobs/            ← file attachment blobs (SHA-256 content-addressable)
 ```
 
+All FTS indexes across all types within the same OKDB env share a single pair of LMDB environments. Each index is differentiated by an auto-increment `ftsId` stored as a compound key prefix inside those shared environments.
+
 Inside each LMDB environment, sub-databases are named by convention:
 
 ```
@@ -141,6 +145,49 @@ per type T:
 
 ---
 
+## Full-text search
+
+FTS indexes are **async by default**. When a document is written, the change is recorded in the change log. A background processor picks up changes and updates the FTS index. This means:
+
+- Writes return immediately — no FTS I/O on the write path.
+- There is a short lag between a write and when the document becomes searchable.
+- Use `fts.flush(type)` to wait for the processor to catch up before querying. This is useful in tests and after bulk imports.
+- Use `fts.ready(type)` to wait for the initial index build to complete after registering a new index.
+
+```javascript
+// Register an FTS index
+await okdb.fts.register('articles', 'main', { fields: ['title', 'body'] });
+
+// Wait for the initial build
+await okdb.fts.ready('articles');
+
+// Write documents
+await okdb.put('articles', 'a1', { title: 'Hello world', body: 'Some content' });
+
+// Flush processor before querying (ensures doc is indexed)
+await okdb.fts.flush('articles');
+
+// Search
+const keys = okdb.fts.search('articles', 'main', 'hello');
+```
+
+### `ready(type)`
+
+`fts.ready(type)` resolves when all FTS indexes registered on `type` have completed their initial build. The `name` parameter is accepted for backward compatibility but is ignored — readiness is per-type, not per-index.
+
+### `flush(type)`
+
+`fts.flush(type)` awaits the background processor catching up to the current write position. Returns immediately if no processor is running or if already caught up. Use this instead of `setTimeout` workarounds in tests or after bulk imports.
+
+### list() response
+
+`fts.list(type)` returns an entry per registered index. Each entry includes `processorState` and `lag` — per-type fields reflecting the single processor that drives all indexes on that type:
+
+- `processorState`: `'building' | 'online' | 'error' | 'waiting' | null` — `null` if no processor is running (no writes have occurred yet).
+- `lag`: number of write operations the processor has not yet indexed. `0` means fully caught up. `null` if no processor is running.
+
+---
+
 ## Field schema
 
 OKDB observes field types as you write data and maintains a coarse schema (`string | number | boolean | null | array | object`) per field. You can inspect it:
@@ -163,12 +210,31 @@ Schema is **observed-only** — no validation on write. If you need strict valid
 
 Set at construction time via the `durability` option:
 
-| Mode       | Behaviour                                                | Use when                       |
-| ---------- | -------------------------------------------------------- | ------------------------------ |
-| `strict`   | Full fsync on every write                                | Mission-critical, money        |
-| `balanced` | `overlappingSync` — background sync, still safe on crash | Default; most use cases        |
-| `fast`     | `noSync` — OS controls flush timing                      | Dev, caches, re-generable data |
+| Mode       | Underlying LMDB flag | Behaviour                            | Use when                          |
+| ---------- | -------------------- | ------------------------------------ | --------------------------------- |
+| `strict`   | _(none)_             | Full fsync on every write            | Mission-critical, money           |
+| `balanced` | `overlappingSync`    | Background sync, still safe on crash | Default; most use cases           |
+| `fast`     | `noSync`             | OS controls flush timing             | Dev, caches, re-generable data    |
+| `custom`   | _(see lmdb below)_   | No preset — raw `lmdb` options apply | Append-only, replicated followers |
 
 ```javascript
 const okdb = new OKDB('./db', { durability: 'fast' });
 ```
+
+### Raw LMDB passthrough
+
+For workloads like replication followers or append-only archives where you need precise control, pass raw LMDB flags via the `lmdb` option. Use `durability: 'custom'` to disable preset interference:
+
+```javascript
+const okdb = new OKDB('./db', {
+    durability: 'custom',
+    lmdb: {
+        noSync: true, // skip fsync — fastest, survives only clean shutdown
+        noMetaSync: true, // skip metadata page fsync — partial durability improvement
+    },
+});
+```
+
+`noSync: true` is equivalent to `durability: 'fast'` but lets you combine it with other raw flags. `noMetaSync: true` without `noSync` is the middle ground: data pages are synced, metadata pages are not — faster than `strict`, safer than `noSync`.
+
+OKDB enforces guardrails: specifying `noSync` or `noMetaSync` together with `durability: 'strict'` is a hard error. Use `durability: 'custom'` when passing these flags explicitly.

package/docs/http-api.md

@@ -349,6 +349,8 @@ POST   /api/:env/type/:type/query        → query records
 POST   /api/:env/transaction             → atomic batch writes
 ```
 
+**Atomic bulk writes:** `POST /api/:env/transaction` accepts an array of `put`, `update`, `patch`, and `remove` operations and commits them all in one LMDB transaction — the right primitive for bulk ingest. See [Transactions](./transactions.md#http-api--bulk-writes) for the full body format and examples.
+
 **Hybrid search:** The `options` body field now accepts `fts` and `vector` sub-objects alongside `index`. See the querying guide for full documentation. When `fts` or `vector` is provided, the route automatically awaits the async result.
 
 ```bash
@@ -456,6 +458,61 @@ GET  /api/:env/type/:type/index/:idx/count?start=<json>&end=<json>  → count in
 
 ---
 
+## Full-text search endpoints
+
+```http
+GET    /api/:env/type/:type/fts                    → list FTS indexes
+POST   /api/:env/type/:type/fts/:name              → register a new FTS index
+DELETE /api/:env/type/:type/fts/:name              → drop an FTS index
+POST   /api/:env/type/:type/fts/:name/reset        → rebuild an FTS index from scratch
+POST   /api/:env/type/:type/fts/:name/search       → search an FTS index
+POST   /api/:env/type/:type/fts/flush              → flush FTS processor (await catch-up)
+```
+
+### GET /api/:env/type/:type/fts — list response
+
+Each entry in the returned array includes:
+
+| Field            | Type                                                        | Description                                                                                        |
+| ---------------- | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
+| `name`           | `string`                                                    | Index name                                                                                         |
+| `status`         | `'creating' \| 'resetting' \| 'ready' \| 'waiting' \| null` | Per-index lifecycle status                                                                         |
+| `config`         | `object \| null`                                            | Index configuration as registered                                                                  |
+| `created`        | `number \| null`                                            | HLC clock at registration time                                                                     |
+| `updated`        | `number \| null`                                            | HLC clock at last config update                                                                    |
+| `error`          | `object \| null`                                            | Last per-index error, if any                                                                       |
+| `processorState` | `'building' \| 'online' \| 'error' \| 'waiting' \| null`    | State of the background FTS processor for this type. `null` if no processor is running.            |
+| `lag`            | `number \| null`                                            | Number of writes the processor has not yet indexed. `0` = fully caught up. `null` if no processor. |
+
+`processorState` and `lag` are per-type fields — they reflect the single background processor that drives all FTS indexes on a type and are identical on every entry of the same type.
+
+### POST /api/:env/type/:type/fts/flush — flush processor
+
+Waits for the background FTS processor to catch up to the current write position for this type. Responds `204 No Content` when fully caught up. This is a no-op if no processor is running or if the processor is already caught up.
+
+- **Permission:** `schema:read`
+- **Idempotent:** yes
+- **Long-running:** yes (blocks until processor catches up)
+
+Use this after bulk writes when immediate search consistency is required:
+
+```bash
+# Bulk-import docs, then flush before querying
+curl -X POST http://localhost:8080/api/default/transaction \
+  -H "Authorization: Bearer token" \
+  -d '[{ "action": "put", "type": "articles", "key": "a1", "value": { "body": "hello" } }]'
+
+curl -X POST http://localhost:8080/api/default/type/articles/fts/flush \
+  -H "Authorization: Bearer token"
+
+# Now search is guaranteed up-to-date
+curl -X POST http://localhost:8080/api/default/type/articles/fts/main/search \
+  -H "Authorization: Bearer token" \
+  -d '{ "query": "hello" }'
+```
+
+---
+
 ## System info
 
 ```http

package/docs/transactions.md

@@ -112,6 +112,48 @@ try {
 
 ---
 
+## HTTP API — bulk writes
+
+The `/transaction` endpoint is the right tool for bulk ingest over HTTP. It accepts an array of operations and commits them all in a single LMDB transaction.
+
+```bash
+curl -X POST http://localhost:8080/api/default/transaction \
+  -H "Authorization: Bearer my-token" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "operations": [
+      { "action": "put", "type": "orders", "key": "o1", "value": { "item": "widget", "qty": 3 } },
+      { "action": "put", "type": "orders", "key": "o2", "value": { "item": "gadget", "qty": 1 } },
+      { "action": "put", "type": "orders", "key": "o3", "value": { "item": "doohickey", "qty": 5 } }
+    ]
+  }'
+```
+
+Each operation is `{ action, type, key, … }`:
+
+| `action` | Extra fields                    | Notes                                   |
+| -------- | ------------------------------- | --------------------------------------- |
+| `put`    | `value` (required), `ifVersion` | Upsert — create or overwrite            |
+| `update` | `value` (required), `ifVersion` | Throws if key absent                    |
+| `patch`  | `patch` (required), `ifVersion` | Same patch format as `PATCH /item/:key` |
+| `remove` | `ifVersion`                     | Throws if key absent                    |
+
+The body can also be sent as a bare array (omitting the `operations` wrapper):
+
+```bash
+curl -X POST http://localhost:8080/api/default/transaction \
+  -H "Authorization: Bearer my-token" \
+  -H "Content-Type: application/json" \
+  -d '[
+    { "action": "put", "type": "prices", "key": "sku-001", "value": { "price": 9.99 } },
+    { "action": "put", "type": "prices", "key": "sku-002", "value": { "price": 19.99 } }
+  ]'
+```
+
+Responds `204 No Content` on success. If any operation fails validation, the entire transaction is aborted and nothing is written.
+
+---
+
 ## Timestamp and origin
 
 Every write accepts a `timestamp` and `origin` option used by the sync system:
@@ -138,3 +180,19 @@ Everything that happens inside a single `db.transaction()` call is atomic:
 No partial state is ever visible. If the process crashes mid-write, LMDB rolls back the transaction on the next open.
 
 Post-commit events (`item:create`, `system:clock_change`, etc.) fire **after** the LMDB transaction has been durably committed — never speculatively.
+
+---
+
+## FTS and read-your-writes consistency
+
+FTS writes are **always async** — the FTS index is updated by a background processor after each write commits. Secondary indexes (field indexes, geo indexes) are updated synchronously inside the same LMDB transaction as the write, so they are immediately consistent.
+
+For applications that require read-your-writes consistency after a write, use `fts.flush(type)`:
+
+```javascript
+await okdb.put('articles', 'a1', { title: 'Hello world' });
+await okdb.fts.flush('articles'); // wait for processor to index the write
+const results = okdb.fts.search('articles', 'main', 'hello'); // guaranteed to include a1
+```
+
+`fts.flush()` is a no-op if the processor is already caught up. It is safe to call unconditionally.