@kedem/okdb 1.0.0

package/docs/pipelines.md

@@ -0,0 +1,598 @@
+# Pipelines
+
+OKDB now has a base `pipelines` feature for storing environment-defined engine topologies in a synced `~pipelines` registry.
+
+This is the generic layer above `engines`.
+
+- engines remain standalone runtime units
+- pipelines group engines structurally
+- pipeline records live in the environment that defines them
+- pipeline records sync like other env data
+
+---
+
+## V1 record shape
+
+```javascript
+{
+    name: 'articles-ingest',
+    template: null,
+    status: 'active',   // 'active' | 'stopped'
+    meta: { label: 'Articles ingest' },
+    created: 1710000000000,
+    updated: 1710000000000,
+    engines: [
+        { type: 'embedder',      name: 'shared-embedder',           role: 'embedder' },
+        { type: 'indexer',       name: 'analytics:articles-ingest', role: 'indexer' },
+        { type: 'vector-search', name: 'analytics:articles-ingest', role: 'search' },
+    ],
+}
+```
+
+Rules in V1:
+
+- `name` is required and must be a non-empty string
+- `template` must be `null`
+- `status` must be `active` or `stopped`
+- `engines` must be a non-empty ordered array
+- every engine entry must have non-empty `type`, `name`, and `role`
+- engine order is meaningful and preserved
+
+---
+
+## Where records live
+
+Pipelines are environment-defined.
+
+That means:
+
+- `okdb.env('analytics').pipelines.create(...)` stores in `analytics` env `~pipelines`
+- synced peers receive the record in that same env
+- `okdb.pipelines.list()` aggregates pipeline records across opened envs
+
+Unlike env-scoped engines, pipelines do **not** use `~system` by default.
+
+---
+
+## Current SDK surface
+
+### Env-local facade
+
+```javascript
+const analytics = okdb.env('analytics');
+
+// Engines must already exist before a V1 pipeline record can be created.
+await analytics.pipelines.create({
+    name: 'articles-ingest',
+    template: null,
+    status: 'active',
+    meta: {},
+    engines: [
+        { type: 'embedder', name: 'shared-embedder', role: 'embedder' },
+        { type: 'indexer', name: 'analytics:articles-ingest', role: 'indexer' },
+    ],
+});
+
+const record = await analytics.pipelines.get('articles-ingest');
+const all = await analytics.pipelines.list();
+
+await analytics.pipelines.stop('articles-ingest');
+await analytics.pipelines.start('articles-ingest');
+await analytics.pipelines.remove('articles-ingest');
+```
+
+### HTTP API surface
+
+The generic pipeline registry is also exposed over HTTP:
+
+```text
+GET    /api/pipelines/templates
+GET    /api/pipelines/templates/:template
+POST   /api/pipelines/templates/:template/preview
+GET    /api/pipelines
+GET    /api/env/:env/pipelines
+GET    /api/env/:env/pipelines/:pipeline
+POST   /api/env/:env/pipelines
+POST   /api/env/:env/pipelines/templates/:template/create
+POST   /api/env/:env/pipelines/scaffold
+POST   /api/env/:env/pipelines/:pipeline/start
+POST   /api/env/:env/pipelines/:pipeline/stop
+DELETE /api/env/:env/pipelines/:pipeline
+```
+
+- `GET /api/pipelines/templates` lists starter recommendations for both generic and embeddings pipeline families
+- `GET /api/pipelines/templates/:template` returns one pipeline starter definition, including its input schema, defaults, `family`, and `recommended` flag
+- `POST /api/pipelines/templates/:template/preview` expands a starter into the exact plan it would execute without persisting anything
+- `POST /api/env/:env/pipelines/templates/:template/create` expands a starter template and executes the underlying scaffold/create operation
+- `POST /api/env/:env/pipelines` expects referenced engines to already exist
+- `POST /api/env/:env/pipelines/scaffold` is the convenience path that creates env-owned engines and then creates the pipeline record atomically
+- `GET /api/pipelines` aggregates generic pipeline records across all currently opened environments
+
+The pipeline template routes are a filtered view over the broader shared template registry exposed at `GET /api/engines/templates`:
+
+- `/api/pipelines/templates` only returns entries with `kind: "pipeline"`
+- generic starters currently include `generic-processor`, `generic-materializer`, and `generic-queue-worker`
+- embeddings starters currently include `embeddings-inline-fake`, `embeddings-inline-ollama`, `embeddings-queue-fake`, and `embeddings-queue-ollama`
+
+Template execution depends on the selected family:
+
+- generic pipeline templates use the same atomic scaffold flow as `POST /api/env/:env/pipelines/scaffold`
+- embeddings pipeline templates delegate to `okdb.embeddings.createPipeline(...)` and then return the resulting generic `~pipelines` record
+- both flows stamp `meta.templateId` so the resulting record keeps template provenance
+
+For embeddings-backed templates, remember that the persisted generic pipeline record name is the resolved `storage_key` (defaulting to the supplied pipeline `name`). The template response still returns the user-facing label through `meta.label`.
+
+### MCP surface
+
+The grouped `okdb_pipeline` tool intentionally separates the two pipeline families:
+
+- generic pipeline registry actions are exposed as `generic_list_all`, `generic_list`, `generic_get`, `generic_create`, `generic_start`, `generic_stop`, `generic_delete`, and `generic_scaffold`
+- embeddings env/type pipeline actions keep concise names such as `list`, `stats`, `query`, `start`, `stop`, and `delete`
+- starter template actions are exposed as `template_list`, `template_get`, `template_preview`, and `template_create`
+
+For MCP ergonomics, bare `list` also has a convenience fallback:
+
+- `action: "list", env: "analytics", type: "articles"` → embeddings env/type pipeline summaries
+- `action: "list", env: "analytics"` → the same env-scoped generic registry view as `generic_list`
+
+In practice that means an MCP client can:
+
+- call `template_list` to discover recommended starters
+- call `template_preview` to inspect the concrete scaffold or embeddings-create plan
+- call `template_create` to execute that plan in a target environment
+
+Direct engine discovery/creation now lives in the dedicated `okdb_engine` tool instead:
+
+- `action: "types"` lists registered engine kinds and their self-declared config metadata
+- `action: "create"` creates a direct env-scoped engine without a pipeline record
+- `action: "list" | "get" | "patch" | "delete" | "restart" | "pause" | "resume" | "retry" | "reset_cursor" | "rebuild"` manages existing direct engines
+
+### Generic `queue-worker` engine
+
+Step 04 adds a generic `queue-worker` engine type that can run standalone or participate as a reusable pipeline member.
+
+In V1 it is:
+
+- standalone-capable
+- pipeline-compatible
+- queue-backed
+- function-driven
+
+Typical flow:
+
+```javascript
+const analytics = okdb.env('analytics');
+
+await analytics.functions.create({
+    name: 'processArticleJob',
+    source: `async (ctx) => {
+        await ctx.env.ensureType('worker_results');
+        await ctx.env.put('worker_results', String(ctx.payload.id), { ok: true });
+        return { ok: true };
+    }`,
+});
+
+await analytics.engines.createAndStartEngine('queue-worker', 'articles-worker', {
+    job_type: 'article-ingest',
+    handler: { kind: 'function', name: 'processArticleJob' },
+    pollInterval: 100,
+    concurrency: 2,
+});
+
+await analytics.pipelines.create({
+    name: 'articles-ingest',
+    template: null,
+    status: 'active',
+    meta: { label: 'Articles ingest' },
+    engines: [{ type: 'queue-worker', name: 'articles-worker', role: 'worker' }],
+});
+
+await analytics.queue.enqueue('article-ingest', { id: 'job-1' });
+```
+
+`queue-worker` config shape in V1:
+
+```javascript
+{
+    job_type: 'article-ingest',
+    queue_env: 'analytics', // optional, defaults to the engine owning env
+    handler: {
+        kind: 'function',
+        name: 'processArticleJob',
+        env: 'analytics',   // optional, defaults to the engine owning env
+    },
+    concurrency: 2,
+    pollInterval: 100,
+    ttl: 30000,
+}
+```
+
+Config notes:
+
+- `job_type` — queue job type to consume
+- `handler.kind` — currently must be `"function"`
+- `handler.name` — function to invoke for each claimed payload
+- `queue_env` — optional env name for the queue source; defaults to the engine owning env
+- `handler.env` — optional env name for the function registry; defaults to the engine owning env
+- `concurrency`, `pollInterval`, `ttl` — forwarded to the underlying queue worker runtime
+
+The function runner pool scales dynamically to match `concurrency` — no pool configuration is needed.
+
+Behavior:
+
+- claims jobs through the normal queue feature
+- invokes the configured env function once per claimed payload
+- completes or fails jobs using the same queue semantics as other queue workers
+- can read from one env's queue while resolving the handler function from another env
+
+Runtime stats exposed through `engine.api.stats()` include:
+
+- `processed`
+- `failed`
+- `running`
+- resolved `queue_env`
+- resolved handler metadata
+
+Minimal pipeline membership example:
+
+```javascript
+await analytics.pipelines.create({
+    name: 'articles-ingest',
+    template: null,
+    status: 'active',
+    meta: { label: 'Articles ingest' },
+    engines: [{ type: 'queue-worker', name: 'articles-worker', role: 'worker' }],
+});
+```
+
+### Generic `processor` engine
+
+Step 05 adds a generic `processor` engine type for change-log driven pipeline members.
+
+In V1 it is:
+
+- standalone-capable
+- pipeline-compatible
+- built on the existing processor feature
+- function-driven
+- suitable for projection, transformation, and queue fan-out workflows
+
+Typical flow:
+
+```javascript
+const analytics = okdb.env('analytics');
+
+await analytics.registerType('articles');
+
+await analytics.functions.create({
+    name: 'projectArticleChanges',
+    source: `async (ctx) => {
+        await ctx.env.ensureType('article_projection');
+        for (const change of ctx.payload) {
+            if (change.action === 'put') {
+                await ctx.env.put('article_projection', change.key, {
+                    title: change.value.title,
+                    source_env: ctx.processor.sourceEnv,
+                });
+            }
+        }
+        return { processed: ctx.payload.length };
+    }`,
+});
+
+await analytics.engines.createAndStartEngine('processor', 'article-projector', {
+    source_type: 'articles',
+    handler: { kind: 'function', name: 'projectArticleChanges' },
+    bootstrap: 'log',
+    batchSize: 64,
+});
+
+await analytics.put('articles', 'a1', { title: 'Hello' });
+```
+
+`processor` config shape in V1:
+
+```javascript
+{
+    source_type: 'articles',
+    source_env: 'analytics', // optional, defaults to the engine owning env
+    handler: {
+        kind: 'function',
+        name: 'projectArticleChanges',
+        env: 'analytics',   // optional, defaults to the engine owning env
+    },
+    mode: 'async',
+    bootstrap: 'log',
+    originMode: 'all',
+    batchSize: 64,
+    hydrateValues: true,
+    cursorKey: 'processor@article-projector',
+    lockMode: 'exclusive',
+    failOnHandlerError: true,
+}
+```
+
+Config notes:
+
+- `source_type` — type whose change log the processor consumes
+- `handler.kind` — currently must be `"function"`
+- `handler.name` — function to invoke for each delivered batch of changes
+- `source_env` — optional env name for the source type; defaults to the engine owning env
+- `handler.env` — optional env name for the function registry; defaults to the engine owning env
+- `mode`, `bootstrap`, `originMode`, `batchSize`, `hydrateValues`, `cursorKey`, `lockMode` — forwarded to the underlying processor runtime
+- `failOnHandlerError` — included in runtime metadata; V1 engine behavior treats handler failures as engine errors
+
+Behavior:
+
+- consumes change-log batches through the normal processor feature
+- invokes the configured env function with `ctx.payload` = `changes[]` (the batch array) and `ctx.processor` containing trigger metadata (`mode`, `sourceType`, `sourceEnv`, `processor`, `engineKey`, `cursorKey`)
+- exposes processor cursor / progress / lag / error state through the engine runtime API
+- can read from one env's type log while resolving the handler function from another env
+
+Runtime stats exposed through `engine.api.stats()` / `engine.api.status()` include:
+
+- `source_type`
+- resolved `source_env`
+- resolved handler metadata
+- `state`
+- `lastClock`
+- `headClock`
+- `lag`
+- `progress`
+- `processedChanges`
+- `failedBatches`
+
+Minimal pipeline membership example:
+
+```javascript
+await analytics.pipelines.create({
+    name: 'projection-pipeline',
+    template: null,
+    status: 'active',
+    meta: { label: 'Projection pipeline' },
+    engines: [{ type: 'processor', name: 'article-projector', role: 'processor' }],
+});
+```
+
+### `materializer` engine
+
+Step 10 adds a `materializer` engine type for deterministic source→target collection projection.
+
+Unlike the generic `processor`, a materializer has an **explicit target type** and a **return-ops contract** that enables safe rebuild.
+
+Key differences from `processor`:
+
+| Concern           | `processor`             | `materializer`                        |
+| ----------------- | ----------------------- | ------------------------------------- |
+| Target            | implicit (side effects) | explicit (declared in config)         |
+| Rebuild           | not safely possible     | first-class `rebuild()` operation     |
+| Function contract | writes via `ctx.env`    | returns ops array for engine to apply |
+| Target ownership  | unknown                 | engine owns the target lifecycle      |
+
+Typical flow:
+
+```javascript
+const analytics = await db.createEnvironment('analytics');
+await analytics.registerType('orders');
+
+await analytics.functions.create({
+    name: 'projectOrders',
+    source: `async (ctx) => {
+        // ctx.payload       = changes[]
+        // ctx.materializer  = { sourceType, targetType, sourceEnv, targetEnv, mode, ... }
+        const ops = [];
+        for (const change of ctx.payload) {
+            if (change.action === 'put') {
+                ops.push({
+                    action: 'put',
+                    key:    change.key,
+                    value:  { total: change.value.total, customerId: change.value.customerId },
+                });
+            }
+            if (change.action === 'remove') {
+                ops.push({ action: 'remove', key: change.key });
+            }
+        }
+        return ops;   // engine applies these to target_type
+    }`,
+});
+
+const engine = await analytics.engines.createAndStartEngine('materializer', 'order-summary', {
+    source_type: 'orders',
+    target_type: 'order_summaries',
+    handler: { kind: 'function', name: 'projectOrders' },
+    bootstrap: 'snapshot',
+    batchSize: 128,
+});
+
+await analytics.put('orders', 'o1', { total: 99.5, customerId: 'c1' });
+// → order_summaries/o1 is written automatically by the engine
+
+// Re-materialize from scratch:
+await engine.api.rebuild();
+```
+
+`materializer` config shape:
+
+```javascript
+{
+    source_type:          'orders',
+    target_type:          'order_summaries',
+    source_env:           'analytics',  // optional, defaults to engine env
+    target_env:           'analytics',  // optional, defaults to engine env
+    bootstrap:            'snapshot',   // 'snapshot' | 'log'
+    originMode:           'all',
+    batchSize:            128,
+    clearTargetOnRebuild: true,         // whether rebuild truncates target first
+    handler: {
+        kind:        'function',
+        name:        'projectOrders',
+        env:         'analytics',       // optional, defaults to engine env
+        recordRuns:  'errors',
+    },
+}
+```
+
+Config notes:
+
+- `source_type` — type whose change log the materializer consumes
+- `target_type` — type the engine writes to (created automatically if absent)
+- `handler.kind` — currently must be `"function"`
+- `handler.name` — function invoked for each batch; must return an ops array
+- `source_env` / `target_env` — optional env overrides
+- `clearTargetOnRebuild` — whether `rebuild()` truncates the target before replaying (default: `true`)
+- `bootstrap`, `originMode`, `batchSize` — forwarded to the underlying processor runtime
+
+Function contract:
+
+- **`ctx.payload`** = `changes[]` — batch of change records
+    - `change.action` = `'put' | 'remove'`
+    - `change.key` = source key
+    - `change.value` = hydrated source document for puts
+    - `change.clock` = live log clock, or `null` during snapshot bootstrap
+- **`ctx.materializer`** = trigger metadata (`sourceType`, `targetType`, `sourceEnv`, `targetEnv`, `mode`, `engineKey`, `cursorKey`)
+- **Return value** = `ops[]` — array of `{ action: 'put'|'remove', key, value? }` objects
+- The function must **not** write directly via `ctx.env` — the engine applies the returned ops
+
+`ctx.materializer.mode` is:
+
+- `'build'` during bootstrap / replay
+- `'live'` during normal ongoing consumption
+
+Engine API:
+
+- `engine.api.stats()` / `engine.api.status()` — state, lag, processedChanges, appliedOps, targetCount
+- `engine.api.rebuild()` — stop, clear target (if configured), reset cursor, replay from origin
+- `engine.api.resetCursor()` — reset cursor only without clearing target
+- `engine.api.pause()` / `engine.api.resume()` — pause/resume consumption
+- `engine.api.retry()` — retry after error state
+
+```javascript
+const all = await okdb.pipelines.list();
+const analyticsOnly = await okdb.pipelines.list({ envName: 'analytics' });
+const record = await okdb.pipelines.get('analytics', 'articles-ingest');
+```
+
+---
+
+## Lifecycle behavior in V1
+
+The base feature is intentionally narrow.
+
+Allowed updates:
+
+- `status`
+- `meta`
+
+Supported lifecycle operations:
+
+- `create`
+- `get`
+- `list`
+- `update` (`meta` / `status` only)
+- `start`
+- `stop`
+- `remove`
+
+Topology is structurally immutable in V1:
+
+- changing engine membership is rejected
+- changing engine roles is rejected
+- changing engine order is rejected
+
+### Aggregate health
+
+Aggregate health is derived at runtime from persisted pipeline `status` and current member engine state:
+
+- pipeline `status === 'stopped'` → health `stopped`
+- all members online → health `online`
+- no members online and all are `error` / `stopped` / `missing` → health `error`
+- otherwise → health `degraded`
+
+### Safe delete behavior
+
+`remove()` deletes the pipeline record and uninstalls engines owned by the same env.
+
+Shared/global engines are preserved.
+
+### HTTP routes
+
+Generic pipelines now expose env-scoped routes:
+
+- `GET /api/env/:env/pipelines`
+- `GET /api/env/:env/pipelines/:pipeline`
+- `POST /api/env/:env/pipelines`
+- `POST /api/env/:env/pipelines/:pipeline/start`
+- `POST /api/env/:env/pipelines/:pipeline/stop`
+- `DELETE /api/env/:env/pipelines/:pipeline`
+
+Read responses include derived pipeline/member inspection data such as:
+
+- pipeline `health`
+- ordered member `engines`
+- per-member `role`, `type`, `name`
+- per-member `state`, `status`, `isRunning`, `owned`, `storeEnv`
+- engine `key`, `config`, and `meta` when the referenced engine exists
+- runtime cursor / lag / progress / error fields when the member exposes them
+
+### Admin visibility
+
+The admin UI now exposes pipelines and engines through three complementary views:
+
+- **Queue → Workers** — create and manage `queue-worker` engines directly for the selected job type
+- **DB → Type detail** — open a type-scoped engines view for embeddings/vector pipelines, including shortcuts from the vector/index drill-ins back to the owning pipeline
+- **Engines** — the primary env-scoped workspace for cross-type inspection, with filters for mode, pipeline family, engine type, status, search text, and source type
+
+Current behavior:
+
+- all engine CRUD and inspection still resolves through env-scoped APIs such as `/api/env/:env/engines`
+- type-local pipeline views stay constrained to the selected env + source type
+- embedding indexer/search panels can jump back to the owning pipeline record
+- queue-worker creation is already direct from the Queue section
+- type-scoped DB views now open the same shared create wizard in a modal, with the env + source type pre-scoped and the source type locked to the current DB context
+- that type-scoped modal supports `processor`, `materializer`, and `embedding`; `queue-worker` creation stays direct from the Queue section
+- the global Engines create flow still supports `processor`, `materializer`, `queue-worker`, and `embedding` from one place
+
+### Sync / runtime activation
+
+Synced pipeline declarations now reconcile through the same higher-level runtime activation seam used for synced engines.
+
+That shared activation path runs:
+
+- after `OKDB.open()` startup restore
+- after each sync reconcile pass completes
+
+Behavior in V1:
+
+- env-scoped engine records still restore/start through the engine feature
+- pipeline records add a second reconciliation pass based on persisted pipeline `status`
+- `status: 'active'` attempts to bring all present members online in declaration order
+- `status: 'stopped'` ensures running members are stopped in reverse order
+- if a synced pipeline arrives before all referenced engine records exist locally, activation is skipped for that pass and retried on the next shared runtime activation cycle
+- per-pipeline activation failures are logged and retried later; they do not abort the broader reconcile pass
+
+Cluster expectation:
+
+- sync replicates declarations (`~engines`, `~pipelines`, env records)
+- local nodes then reconcile those declarations into runtime state
+- structural completeness is still required for a pipeline to become fully active
+
+---
+
+## Relation to embeddings
+
+Embeddings now use this generic pipeline model as their persisted grouping layer while keeping the richer `okdb.embeddings` facade.
+
+Current behavior:
+
+- `okdb.embeddings.createPipeline(...)` provisions the underlying engines **and** persists an env-local `~pipelines` record
+- that record uses the pipeline `storage_key` as its generic name
+- member roles are explicit and ordered: `embedder`, `indexer`, optional `worker`, `search`
+- the record is tagged with `meta.family = 'embeddings'` so specialized embeddings views can recognize it cleanly
+- embeddings lifecycle routes now delegate to the generic pipeline feature when a generic record exists
+
+Embeddings-backed records also use `meta.lifecycleSkipRoles = ['embedder']`.
+
+That keeps shared embedders out of per-pipeline stop/start behavior while still allowing generic pipeline operations to manage env-owned members such as the indexer, search engine, and optional queue worker.
+
+This makes embeddings the first built-in feature validated on top of the generic pipeline record model rather than only name-based engine grouping.