@kedem/okdb 1.0.4 → 1.1.0

package/docs/OKDB-ARCHITECTURE.md

@@ -639,7 +639,8 @@ env (not the default env):
   cron scheduling, token-bucket rate limiting (buckets), and per-job result storage
 - Workers claim jobs (`STATUS.PENDING → CLAIMED`) and mark them done/failed
 - Reconciler handles stale claims, retry scheduling, cron advancement
-- Stats (`OKDBQueueStats`) tracked per-bucket in a separate LMDB sub-db
+- Stats are maintained by a `~queue_stats` view (`$count` + `$countBy`
+  reductions over `~queue_jobs`) — auto-updated by the views processor
 
 The queue is OKDB-native: jobs survive crashes, job history is queryable,
 all mutations go through the queue env's change log, and queue state replicates

package/docs/manifest.json

@@ -17,6 +17,7 @@
             { "file": "sync.md", "label": "Sync" },
             { "file": "fts.md", "label": "Full-Text Search" },
             { "file": "embeddings.md", "label": "Embeddings" },
+            { "file": "views.md", "label": "Materialized Views" },
             { "file": "pipelines.md", "label": "Pipelines" },
             { "file": "queue.md", "label": "Queue" },
             { "file": "files.md", "label": "Files" },

package/docs/pipelines.md

@@ -360,6 +360,8 @@ Step 10 adds a `materializer` engine type for deterministic source→target coll
 
 Unlike the generic `processor`, a materializer has an **explicit target type** and a **return-ops contract** that enables safe rebuild.
 
+> **Views vs materializers** — if you need live aggregates (counts, sums, averages, grouped statistics) choose [views](./views.md): define a `reduce` spec and read the result synchronously via `env.views.get()`. If you need a _derived collection_ — one document per source document, with custom field mapping — use a materializer: the function returns ops that the engine applies to the target type, which is then queryable like any other type.
+
 Key differences from `processor`:
 
 | Concern           | `processor`             | `materializer`                        |

package/docs/views.md

@@ -0,0 +1,351 @@
+# Materialized Views
+
+Views are reactive, incrementally-maintained aggregations over a registered type. Every write to the source type is processed immediately and the result is always available synchronously via `.get()`. No polling, no background jobs, no query-time computation.
+
+---
+
+## Quick start
+
+```js
+const db = new OKDB('./mydb');
+await db.open();
+await db.registerType('orders');
+
+const env = db.env('default');
+
+// Create a view
+await env.views.create('orderStats', {
+    type: 'orders',
+    filter: { status: 'completed' },
+    reduce: {
+        total: { $sum: 'amount' },
+        count: { $count: true },
+        avgValue: { $avg: 'amount' },
+        byRegion: { $countBy: 'region' },
+    },
+});
+
+// Read the live result — synchronous, O(1)
+const stats = env.views.get('orderStats');
+// → {
+//     total:    { value: 142500 },
+//     count:    { value: 312 },
+//     avgValue: { value: 456.7 },
+//     byRegion: { EU: { value: 140 }, US: { value: 172 } }
+//   }
+```
+
+---
+
+## Definition shape
+
+```js
+{
+    type:   string,      // required — source type (must be registered)
+    filter: object,      // optional — sift-style filter applied before reducers
+    map:    object,      // optional — declarative per-doc transform applied before reducers
+    reduce: object,      // required, non-empty — output-field → reducer spec
+}
+```
+
+### `filter`
+
+Uses [sift](https://github.com/crcn/sift.js) query operators (MongoDB-style). Only documents that pass the filter contribute to the view.
+
+```js
+filter: { status: 'active', amount: { $gte: 100 } }
+```
+
+### `map`
+
+A declarative transform merged onto each source document before reducers run. The original document fields are preserved — map fields are added/overwritten on top.
+
+| Operator     | Syntax                                                          | Description                                                                   |
+| ------------ | --------------------------------------------------------------- | ----------------------------------------------------------------------------- |
+| Field rename | `{ storeName: 'name' }`                                         | Copy field to output key; dot paths supported: `'address.city'`               |
+| FK lookup    | `{ region: { $ref: ['regions', '$regionId', 'label'] } }`       | FK ref and target field both support dot paths                                |
+| Concatenate  | `{ label: { $concat: ['$city', ' – ', '$name'] } }`             | `$`-prefixed tokens are field refs (dot paths ok); others are literals        |
+| Coalesce     | `{ display: { $coalesce: ['$nickname', '$name', 'Unknown'] } }` | First non-null wins; `$`-prefixed tokens are field refs (dot paths supported) |
+
+```js
+// Example: enrich each store with its region name via a FK lookup,
+// then group-count by that enriched field
+map: {
+    regionName: { $ref: ['regions', '$regionId', 'name'] },
+},
+reduce: {
+    byRegion: { $countBy: 'regionName' },
+}
+```
+
+### `reduce`
+
+Each key in `reduce` becomes an output field. The value is a reducer spec:
+
+| Reducer    | Spec                    | Output                         | Notes                                                |
+| ---------- | ----------------------- | ------------------------------ | ---------------------------------------------------- |
+| `$count`   | `{ $count: true }`      | `{ value: N }`                 | Counts matching docs                                 |
+| `$sum`     | `{ $sum: 'field' }`     | `{ value: N }`                 | Sums a numeric field; non-numeric values are ignored |
+| `$avg`     | `{ $avg: 'field' }`     | `{ value: N }`                 | Running average; `value` is `0` when no docs match   |
+| `$min`     | `{ $min: 'field' }`     | `{ value: X }`                 | Index-backed: auto-creates an index on the field     |
+| `$max`     | `{ $max: 'field' }`     | `{ value: X }`                 | Index-backed: auto-creates an index on the field     |
+| `$countBy` | `{ $countBy: 'field' }` | `{ [groupKey]: { value: N } }` | Groups docs by a field value                         |
+
+Field paths in all reducers support dot notation for nested access: `{ $sum: 'metrics.revenue' }`, `{ $countBy: 'address.country' }`. This is consistent with how indexes handle nested fields.
+
+`$min` and `$max` require an index on the named field. The view engine creates one automatically if none exists; it is removed when the view is removed (unless another consumer owns it).
+
+---
+
+## Output shape
+
+`views.get(name)` returns an object with one key per entry in `reduce`. The shape depends on the reducer:
+
+- **Scalar reducers** (`$count`, `$sum`, `$avg`, `$min`, `$max`): `{ value: N }`
+- **Grouped reducer** (`$countBy`): `{ [groupKey]: { value: N } }`
+- **`$ref` slots**: a nested object with the same shape as a top-level `reduce` result (see [$ref aggregations](#ref-aggregations))
+
+```js
+const result = env.views.get('orderStats');
+result.total.value; // → 142500 (number)
+result.avgValue.value; // → 456.7  (number)
+result.byRegion.EU.value; // → 140    (number)
+```
+
+### `items: true`
+
+Add `items: true` to any reducer spec to enable a `.items()` method on the result group. Calling it returns the live source documents that contribute to that aggregation bucket.
+
+```js
+reduce: {
+    byRegion: { $countBy: 'region', items: true },
+    total:    { $sum: 'amount', items: true },
+}
+
+// Access source docs for a specific group
+const euOrders = result.byRegion.EU.items();  // → Order[]
+
+// Scalar items() returns all matching docs
+const allOrders = result.total.items();       // → Order[]
+```
+
+---
+
+## `$ref` aggregations
+
+A `$ref` reducer aggregates documents from a _related_ type, scoped to the filtered parent documents. The child type must have a foreign-key field pointing back to the parent type's key.
+
+```js
+await env.views.create('storeStats', {
+    type: 'stores',
+    filter: { active: true },
+    reduce: {
+        storeCount: { $count: true },
+
+        // Aggregate the related 'orders' type
+        orders: {
+            $ref: {
+                type: 'orders', // child type
+                key: 'storeId', // FK field on the child pointing to the parent key
+                filter: { status: 'completed' },
+                reduce: {
+                    count: { $count: true },
+                    revenue: { $sum: 'amount' },
+                },
+            },
+        },
+    },
+});
+
+const stats = env.views.get('storeStats');
+stats.orders.count.value; // → total completed orders across all active stores
+stats.orders.revenue.value; // → total revenue
+```
+
+`$ref` reducers support the same built-in reducers as top-level `reduce`, except `$min` and `$max` (index-backed reducers are not supported inside `$ref`).
+
+---
+
+## Lifecycle
+
+### States
+
+A view's meta state is available via `views.getMeta(name)`:
+
+| State       | Meaning                                                                       |
+| ----------- | ----------------------------------------------------------------------------- |
+| `creating`  | Initial bootstrap scan in progress                                            |
+| `ready`     | Fully up-to-date; all writes are being processed                              |
+| `halted`    | A reducer threw; new writes are queued until the next write recovers the view |
+| `stopped`   | Manually paused; writes that arrive while stopped are replayed on start       |
+| `resetting` | Clock regression detected; full rebuild triggered automatically               |
+
+### Rebuild
+
+A full rebuild clears all accumulated state and re-scans the source type from scratch. Triggered automatically on clock regression; also available manually:
+
+```js
+await env.views.rebuild('orderStats');
+```
+
+### Stop / start
+
+```js
+await env.views.stop('orderStats'); // pause processing; definition is preserved
+await env.views.start('orderStats'); // resume; catches up optimistically or rebuilds
+```
+
+When a view is stopped and later started:
+
+- If no writes arrived while stopped, the view resumes instantly.
+- If only _new_ documents were added (no modifications or removals), an optimistic catch-up is applied.
+- Otherwise a full rebuild is performed.
+
+### Remove
+
+```js
+await env.views.remove('orderStats');
+
+// If the view owns an auto-created index, you must decide what happens to it:
+await env.views.remove('orderStats', { managedIndexes: 'drop' }); // drop the index
+await env.views.remove('orderStats', { managedIndexes: 'keep' }); // retain the index
+```
+
+---
+
+## Custom reducers
+
+Register a custom reducer before creating a view that uses it. Custom reducer names must start with `$`.
+
+```js
+env.views.registerReducer('$product', {
+    apply(state, before, after, opts) {
+        // state  — current accumulated value (null on first call)
+        // before — previous doc (null for inserts), post-filter, post-map
+        // after  — new doc (null for deletes), post-filter, post-map
+        // opts   — the value from the spec: { $product: opts }
+        state = state ?? 1;
+        if (before !== null && typeof before[opts] === 'number') state /= before[opts];
+        if (after !== null && typeof after[opts] === 'number') state *= after[opts];
+        return state;
+    },
+});
+
+await env.views.create('priceProduct', {
+    type: 'items',
+    reduce: { product: { $product: 'multiplier' } },
+});
+```
+
+---
+
+## JavaScript API
+
+All methods are on `env.views` (`OKDBEnv#views`):
+
+```js
+const env = db.env('myEnv');
+
+// Create
+await env.views.create(name, definition)      // → { name }
+
+// Read
+env.views.get(name)                           // → output object | null  (synchronous)
+env.views.getMeta(name)                       // → { state, clock, error, refs } | null
+await env.views.getDefinition(name)           // → stored definition | null
+await env.views.list()                        // → string[]
+
+// Lifecycle
+await env.views.rebuild(name)
+await env.views.stop(name)
+await env.views.start(name)
+await env.views.remove(name, options?)
+
+// Extension
+env.views.registerReducer(name, { apply })
+```
+
+---
+
+## HTTP API
+
+Views are scoped per environment. The `{env}` segment is the environment name (e.g. `default`).
+
+| Method   | Path                                     | Description             |
+| -------- | ---------------------------------------- | ----------------------- |
+| `GET`    | `/api/env/{env}/views`                   | List view names         |
+| `POST`   | `/api/env/{env}/views`                   | Create a view           |
+| `GET`    | `/api/env/{env}/views/{name}`            | Get current view output |
+| `GET`    | `/api/env/{env}/views/{name}/meta`       | Get view state/meta     |
+| `GET`    | `/api/env/{env}/views/{name}/definition` | Get stored definition   |
+| `DELETE` | `/api/env/{env}/views/{name}`            | Remove a view           |
+| `POST`   | `/api/env/{env}/views/{name}/rebuild`    | Rebuild from scratch    |
+| `POST`   | `/api/env/{env}/views/{name}/stop`       | Stop processing         |
+| `POST`   | `/api/env/{env}/views/{name}/start`      | Start / resume          |
+
+**Create body:**
+
+```json
+{
+    "name": "orderStats",
+    "type": "orders",
+    "filter": { "status": "completed" },
+    "reduce": {
+        "total": { "$sum": "amount" },
+        "count": { "$count": true }
+    }
+}
+```
+
+**Delete with managed indexes:**
+
+```json
+{ "managedIndexes": "drop" }
+```
+
+---
+
+## MCP
+
+Views are available as the `views` capability. Actions: `list_views`, `create_view`, `get_view`, `get_view_meta`, `get_view_definition`, `remove_view`, `rebuild_view`, `stop_view`, `start_view`.
+
+---
+
+## Views vs materializers
+
+|                      | Views                                                       | Materializer engine                                                 |
+| -------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------- |
+| **What it produces** | A single reduced document (aggregates)                      | A populated target collection (projections)                         |
+| **Definition style** | Declarative (`reduce`, `filter`, `map`)                     | Imperative function returning ops                                   |
+| **Rebuild**          | `views.rebuild(name)`                                       | `engine.api.rebuild()`                                              |
+| **Access**           | `env.views.get(name)` — always synchronous                  | Read the target type via `env.get(...)`                             |
+| **Use when**         | You need live counters, sums, averages, or group statistics | You need a derived collection with one document per source document |
+
+Use views for aggregations (how many, how much, grouped by). Use a materializer engine when you need a structurally transformed copy of a collection that other queries or features can read from directly.
+
+See [pipelines.md](./pipelines.md) for materializer documentation.
+
+---
+
+## Sync behavior
+
+View definitions are stored in the `~views` system type within their environment and sync to connected peers like any other document. On startup and after each sync reconcile pass, OKDB activates any view definitions that arrived from remote peers but are not yet live locally.
+
+---
+
+## Error codes
+
+| Code                           | Cause                                                                                          |
+| ------------------------------ | ---------------------------------------------------------------------------------------------- |
+| `VIEW_INVALID_NAME`            | Name is empty, contains whitespace, or is not a string                                         |
+| `VIEW_INVALID_DEFINITION`      | Definition is not an object                                                                    |
+| `VIEW_MISSING_TYPE`            | `type` field missing or not a string                                                           |
+| `VIEW_MISSING_REDUCE`          | `reduce` field missing or empty                                                                |
+| `VIEW_TYPE_NOT_REGISTERED`     | The source type does not exist in this environment                                             |
+| `VIEW_ALREADY_EXISTS`          | A view with that name already exists                                                           |
+| `VIEW_NOT_FOUND`               | No view with that name                                                                         |
+| `VIEW_HAS_ORPHANED_INDEXES`    | Removing the view would leave auto-created indexes with no owner; pass `managedIndexes` option |
+| `VIEW_REF_UNSUPPORTED_REDUCER` | `$min`/`$max` used inside a `$ref` sub-view                                                    |
+| `VIEW_INVALID_REDUCER_NAME`    | Custom reducer name does not start with `$`                                                    |
+| `VIEW_REDUCER_CONFLICT`        | Custom reducer name conflicts with a built-in                                                  |
+| `VIEW_INVALID_REDUCER`         | Custom reducer missing an `apply` function                                                     |