@kedem/okdb 1.3.0 → 1.5.0

package/docs/http-api.md

@@ -349,6 +349,8 @@ POST   /api/:env/type/:type/query        → query records
 POST   /api/:env/transaction             → atomic batch writes
 ```
 
+**Hybrid search:** The `options` body field now accepts `fts` and `vector` sub-objects alongside `index`. See the querying guide for full documentation. When `fts` or `vector` is provided, the route automatically awaits the async result.
+
 ```bash
 # Get a record
 curl http://localhost:8080/api/default/type/users/item/alice \

package/docs/querying.md

@@ -192,6 +192,116 @@ const open = okdb.countByIndex('orders', ['status', 'createdAt'], {
 
 ---
 
+## Hybrid queries
+
+`query()` accepts `fts` and `vector` options to combine full-text search, semantic similarity, and index constraints in a single call. All signals compose — the result is a unified ranked list with sift post-filtering applied last.
+
+### FTS signal
+
+```javascript
+// Full-text search — returns results ordered by relevance score
+const results = await okdb.query(
+    'articles',
+    {},
+    {
+        fts: { name: 'articles_fts', query: 'database indexing' },
+        limit: 20,
+    },
+);
+// Each result has: { key, value, ftsScore, numTerms, maxScore }
+```
+
+### FTS + index constraint
+
+The index acts as a **hard constraint** — only documents present in the index range are returned. It does not affect ranking.
+
+```javascript
+// FTS results, but only from published articles
+const results = await okdb.query(
+    'articles',
+    {},
+    {
+        fts: { name: 'articles_fts', query: 'database indexing' },
+        index: { fields: ['status'], prefix: ['published'] },
+        limit: 20,
+    },
+);
+```
+
+### FTS + sift post-filter
+
+Sift filter is always applied last, after FTS and index narrowing:
+
+```javascript
+const results = await okdb.query(
+    'articles',
+    { category: 'tech' },
+    {
+        fts: { name: 'articles_fts', query: 'indexing' },
+        limit: 20,
+    },
+);
+```
+
+### FTS + vector (hybrid semantic + keyword)
+
+When both `fts` and `vector` are provided, candidates from each signal are combined and re-ranked using **Reciprocal Rank Fusion (RRF)**. RRF works across incomparable score scales — no normalization needed.
+
+```javascript
+const results = await okdb.query(
+    'articles',
+    {},
+    {
+        fts: { name: 'articles_fts', query: 'database' },
+        vector: { engine: 'articles_vec', query: 'fast persistent storage', limit: 50 },
+        limit: 20,
+    },
+);
+// Each result has: { key, value, score (RRF), ftsScore?, vectorScore? }
+```
+
+### Signal semantics
+
+| Signal   | Ordering         | Can enumerate all matches | Notes                                 |
+| -------- | ---------------- | ------------------------- | ------------------------------------- |
+| `index`  | Field value      | Yes                       | Hard constraint only — never reorders |
+| `fts`    | Relevance score  | Yes                       | Posting list intersection             |
+| `vector` | Similarity score | No — K-bounded            | ANN search, approximate               |
+
+**Important:** `query()` returns a **Promise** when `fts` or `vector` is present (vector search requires async embedding inference). Without these options, `query()` remains synchronous.
+
+### FTS options
+
+| Option   | Type            | Default              | Description                                   |
+| -------- | --------------- | -------------------- | --------------------------------------------- |
+| `name`   | string          | required             | Registered FTS index name                     |
+| `query`  | string          | `''`                 | Text to search                                |
+| `mode`   | `'and'`\|`'or'` | `'and'`              | AND: all terms required. OR: any term matches |
+| `prefix` | boolean         | false                | Enable prefix matching on last token          |
+| `limit`  | number          | `max(limit×10, 200)` | FTS candidate pool size                       |
+
+### Vector options
+
+| Option   | Type   | Default             | Description               |
+| -------- | ------ | ------------------- | ------------------------- |
+| `engine` | string | required            | Vector search engine name |
+| `query`  | string | required            | Semantic query text       |
+| `limit`  | number | `max(limit×4, 100)` | ANN candidate pool size   |
+
+### Nested index form
+
+The `index` option accepts a nested object for clarity:
+
+```javascript
+// Nested form (preferred for hybrid queries)
+{ index: { fields: ['status'], prefix: ['active'] } }
+
+// Flat form (still works — backward compatible)
+{ index: ['status'], prefix: ['active'] }
+```
+
+---
+
 ## Geo query
 
 OKDB has a built-in geospatial index based on geohash. Register a geo index on a field that stores `{ lat, lon }` (or an `[lat, lon]` array):

package/docs/views.md

@@ -131,6 +131,201 @@ const allOrders = result.total.items();       // → Order[]
 
 ---
 
+## `$group` reducer
+
+`$group` partitions source documents by a field (or compound key) and runs named sub-reducers within each partition. The result is an object keyed by the group value, with each group holding a nested reduced document.
+
+### Syntax
+
+```js
+reduce: {
+    outputField: {
+        $group: {
+            by: 'field' | ['field1', 'field2'],  // group key — single field or compound
+            filter: object,                       // optional — sift-style, applied within $group
+            reduce: {
+                subReducerName: { $count: true },
+                revenue:        { $sum: 'amount' },
+                avg:            { $avg: 'price' },
+                // $countBy also works inside $group
+            },
+        },
+    },
+}
+```
+
+### Output shape
+
+Each populated group key maps to a nested object with one entry per sub-reducer:
+
+```js
+{
+    groupKey1: {
+        revenue: { value: 1200 },
+        count:   { value: 4 },
+    },
+    groupKey2: {
+        revenue: { value: 800 },
+        count:   { value: 3 },
+    },
+}
+```
+
+Empty groups (count/sum reaching 0) are removed automatically — they are never present as `{ value: 0 }`.
+
+```js
+const stats = env.views.get('orderStats');
+stats.byProduct.p1.revenue.value; // → 1200
+stats.byProduct.p1.count.value; // → 4
+```
+
+### Compound key
+
+Pass an array to `by` to build a key from multiple fields. The fields are joined with a `\x00` separator:
+
+```js
+$group: {
+    by: ['product_id', 'region'],
+    reduce: { count: { $count: true } },
+}
+
+// Key for product_id='p1', region='EU': 'p1\x00EU'
+stats.byProductRegion['p1\x00EU'].count.value;
+```
+
+### Filter semantics
+
+Filters are evaluated at three levels:
+
+| Level              | Where it lives                | What it filters                                                                  |
+| ------------------ | ----------------------------- | -------------------------------------------------------------------------------- |
+| View filter        | `definition.filter`           | Applied before map; docs failing it never reach any reducer                      |
+| `$group.filter`    | `$group: { filter: {...} }`   | Applied within `$group`, post-map; docs failing it don't contribute to any group |
+| Sub-reducer filter | Not supported inside `$group` | Sub-reducers within a group share the group's filter                             |
+
+### Limitations
+
+- `$min` and `$max` are **not supported** inside `$group`. They require an index-backed sub-DB per group key. Attempting to use them throws at view-creation time: `$min and $max are not supported inside $group reducers`.
+- Nested `$group` (a `$group` whose sub-reducer is also a `$group`) is not supported. Use a compound `by` array instead.
+
+### Example
+
+```js
+await env.views.create('lineItemStats', {
+    type: 'line_items',
+    reduce: {
+        byProduct: {
+            $group: {
+                by: 'product_id',
+                reduce: {
+                    revenue: { $sum: 'amount' },
+                    count: { $count: true },
+                },
+            },
+        },
+    },
+});
+
+const stats = env.views.get('lineItemStats');
+// stats.byProduct = {
+//     'p1': { revenue: { value: 1500 }, count: { value: 3 } },
+//     'p2': { revenue: { value:  800 }, count: { value: 2 } },
+// }
+```
+
+`$group` also works inside `$ref.reduce`:
+
+```js
+reduce: {
+    lineItems: {
+        $ref: {
+            type: 'line_items',
+            key: 'order_id',
+            reduce: {
+                byProduct: {
+                    $group: {
+                        by: 'product_id',
+                        reduce: { total: { $sum: 'amount' } },
+                    },
+                },
+            },
+        },
+    },
+}
+
+stats.lineItems.byProduct.p1.total.value; // → aggregate revenue for product p1
+```
+
+---
+
+## Live map `$ref`
+
+A map field with `live: true` on its `$ref` spec is a **live enrichment**: when the referenced document changes, all source documents pointing to it are automatically re-evaluated through the view.
+
+### Syntax
+
+```js
+map: {
+    vendorName: { $ref: ['vendors', '$vendor_id', 'name'], live: true },
+}
+```
+
+The array form is identical to the snapshot `$ref` — the only difference is the `live: true` flag.
+
+### Snapshot vs live semantics
+
+| Mode                                | Behavior                                                                                                                                                                                                                          |
+| ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Snapshot (default, no `live: true`) | The referenced field is resolved once when the source doc is written. Later changes to the referenced doc are not reflected in the view.                                                                                          |
+| Live (`live: true`)                 | OKDB maintains an inverse index mapping each referenced doc key to all source docs that point to it. When the referenced doc changes, every source doc in that fan-out set is re-processed through the filter, map, and reducers. |
+
+```js
+// Snapshot — p1's vendor change does NOT update the view
+map: { vendor: { $ref: ['products', '$product_id', 'vendor'] } }
+
+// Live — p1's vendor change re-evaluates all line_items with product_id='p1'
+map: { vendor: { $ref: ['products', '$product_id', 'vendor'], live: true } }
+```
+
+### Cost considerations
+
+Each live map field writes two extra sub-DBs per view:
+
+- An **inverse index** (`liveMapInverse:<viewName>:<field>`) — maps each referenced doc key to the set of source doc keys that point to it.
+- A **values cache** (`liveMapValues:<viewName>:<field>`) — caches the last resolved value per source doc to produce a correct `before` snapshot when the referenced doc changes.
+
+When a referenced doc is updated, OKDB re-processes every source doc in the inverse index for that field. For a referenced type with many source documents per entry (high fan-out), this multiplies write cost:
+
+- **Low fan-out** (e.g., a `products` table where each product has a few line items): `live: true` is fine.
+- **High fan-out** (e.g., a `users` table where each user has thousands of events): prefer snapshot mode or move the enrichment to a materializer pipeline.
+
+### Example
+
+```js
+await db.registerType('products');
+await db.registerType('line_items');
+
+await env.views.create('salesByVendor', {
+    type: 'line_items',
+    map: { vendor: { $ref: ['products', '$product_id', 'vendor'], live: true } },
+    reduce: { byVendor: { $countBy: 'vendor' } },
+});
+
+await db.put('products', 'p1', { vendor: 'ACME' });
+await db.put('line_items', 'li1', { product_id: 'p1', amount: 100 });
+await db.put('line_items', 'li2', { product_id: 'p1', amount: 200 });
+
+// salesByVendor.byVendor = { ACME: { value: 2 } }
+
+await db.put('products', 'p1', { vendor: 'NewCo' }); // rename the vendor
+
+// After propagation:
+// salesByVendor.byVendor = { NewCo: { value: 2 } }
+// ACME bucket is removed; both li1 and li2 moved to NewCo automatically.
+```
+
+---
+
 ## `$ref` aggregations
 
 A `$ref` reducer aggregates documents from a _related_ type, scoped to the filtered parent documents. The child type must have a foreign-key field pointing back to the parent type's key.