@kedem/okdb 1.0.0

package/docs/http-api.md

@@ -0,0 +1,705 @@
+# HTTP API
+
+OKDB includes a built-in HTTP server with a REST API, sync endpoint, and admin UI. It can run as a standalone server **or** be integrated into an existing Express, Fastify, or raw Node.js server — your choice.
+
+## Generic pipelines
+
+The generic `pipelines` feature exposes env-scoped CRUD and inspection routes:
+
+```text
+GET    /api/pipelines/templates
+GET    /api/pipelines/templates/:template
+POST   /api/pipelines/templates/:template/preview
+GET    /api/pipelines
+GET    /api/env/:env/pipelines
+GET    /api/env/:env/pipelines/:pipeline
+POST   /api/env/:env/pipelines
+POST   /api/env/:env/pipelines/templates/:template/create
+POST   /api/env/:env/pipelines/scaffold
+POST   /api/env/:env/pipelines/:pipeline/start
+POST   /api/env/:env/pipelines/:pipeline/stop
+DELETE /api/env/:env/pipelines/:pipeline
+```
+
+Pipeline create requests must reference engines that already exist.
+`POST /api/env/:env/pipelines/scaffold` is the atomic convenience variant: it creates one or more env-owned engines and then persists the pipeline record, cleaning up already-created engines if later validation fails.
+`GET /api/pipelines/templates` is the pipeline-only filtered view of the shared template registry; it returns starter plans whose `kind` is `pipeline`.
+`POST /api/pipelines/templates/:template/preview` expands one of those starters into the concrete create payload it would execute.
+`POST /api/env/:env/pipelines/templates/:template/create` executes that plan in the target environment.
+`GET /api/pipelines` aggregates generic pipeline records across all opened environments.
+Pipeline reads include derived aggregate health and member engine runtime state.
+In the admin API explorer these routes are grouped under the `pipelines` tag.
+
+Typical pipeline read responses include:
+
+- pipeline `status` and derived `health`
+- ordered member `engines`
+- per-member `role`, `type`, `name`, `state`, `status`, `isRunning`
+- ownership/runtime hints such as `owned`, `storeEnv`, `lag`, `progress`, `cursorKey`, and `error`
+
+Embeddings-specific env/type pipeline routes are now layered on top of this same generic model when a matching generic pipeline record exists.
+
+In practice that means:
+
+- `POST /api/env/:env/type/:type/pipelines` creates the embeddings engines and the env-local generic pipeline record
+- `GET /api/env/:env/type/:type/pipelines` prefers generic pipeline-backed summaries and only falls back to legacy engine grouping for older pipelines
+- specialized lifecycle routes such as `.../start`, `.../stop`, and `DELETE .../:pipeline` delegate to the generic pipeline feature when possible
+- shared embedder members remain preserved during embeddings pipeline stop/delete behavior
+
+---
+
+## Starting the server
+
+```javascript
+okdb.http.listen(8080);
+// or with a host:
+okdb.http.listen(8080, '0.0.0.0');
+```
+
+The HTTP server is optional. You can use OKDB entirely in-process without it.
+
+---
+
+## Authentication
+
+### Bearer token
+
+```javascript
+const okdb = new OKDB('./db', {
+    api: { tokens: ['my-secret-token'] },
+});
+```
+
+```bash
+curl http://localhost:8080/api/type/users \
+  -H "Authorization: Bearer my-secret-token"
+```
+
+### Basic auth (admin UI)
+
+```javascript
+admin: {
+    auth: {
+        user:         'admin',
+        pass:         'secret',
+        cookieSecret: 'change-me-in-production',
+        cookieTtlMs:  3_600_000,
+    },
+}
+```
+
+---
+
+## Operations metadata model
+
+OKDB's HTTP layer is also the current **operations registry** for the project.
+
+Routes can now carry canonical operation metadata that powers:
+
+- `/api/docs`
+- shell help and route listing
+- MCP `tools/list` and `tools/call`
+- future admin/docs consumers
+
+### Canonical metadata fields
+
+Representative fields include:
+
+- `id` — stable operation identifier
+- `summary`
+- `description`
+- `tags`
+- `inputSchema`
+- `outputSchema`
+- `safety`
+- `access`
+- `http`
+- `cli`
+- `mcp`
+
+### Example
+
+```javascript
+http.add('GET', '/api/info', () => ({ result: okdb.info }), {
+    id: 'system_info',
+    summary: 'Database info',
+    tags: ['system'],
+    inputSchema: {
+        type: 'object',
+        properties: {},
+        additionalProperties: false,
+    },
+    outputSchema: {
+        type: 'object',
+        additionalProperties: true,
+    },
+    safety: {
+        access: 'read',
+        destructive: false,
+        idempotent: true,
+        longRunning: false,
+        streaming: false,
+    },
+    access: {
+        audiences: ['http', 'shell', 'mcp'],
+        authRequired: true,
+        internalOnly: false,
+    },
+    http: {
+        bodyMode: 'none',
+    },
+    mcp: {
+        include: true,
+        tool: 'system_info',
+        readOnlyHint: true,
+    },
+});
+```
+
+### Path/query/body projection rules
+
+OKDB uses a canonical logical input model and projects it into HTTP as follows:
+
+1. **Path params are inferred** from the route path (including optional env defaults)
+2. **Query params are declared explicitly** via `http.querySchema`
+3. **Remaining logical input fields** go to the body by default
+4. **Explicit `http.bindings`** are only needed for ambiguous or nonstandard cases
+
+### `http.bodyMode`
+
+Supported body projection modes include:
+
+- `none`
+- `remaining-fields`
+- `object`
+- `payload-or-self`
+- `array`
+- `raw`
+
+### Validation rules
+
+Canonical operation metadata is validated for:
+
+- missing bindings
+- ambiguous bindings
+- illegal bindings
+- raw/MCP transport incompatibilities
+
+During migration, strict validation applies to canonical routes while legacy routes still pass through until converted.
+
+---
+
+## `/api/docs`
+
+`GET /api/docs` still returns the route registry shape used by existing consumers, but entries now also include normalized operation metadata where available.
+
+Each entry contains at least:
+
+- `method`
+- `path`
+- `defaults`
+- `meta`
+- `operation`
+
+The legacy `meta` shape remains present during migration for compatibility.
+
+---
+
+## MCP generation
+
+OKDB's MCP surface now derives tools primarily from route operation metadata.
+
+Routes may opt into MCP with metadata like:
+
+```javascript
+mcp: {
+  capability: 'queue',
+  action: 'queue_list_jobs',
+  readOnlyHint: true,
+}
+```
+
+Most routes are exposed through grouped capability tools such as `okdb_queue`, `okdb_function_run`, and `okdb_index`.
+Those tools do **not** use dotted method names like `okdb_function_run.preview`.
+Instead, the tool name stays fixed and the requested operation is supplied as an `action` argument.
+
+Examples:
+
+```json
+{
+    "name": "okdb_function_run",
+    "arguments": { "action": "preview", "env": "default", "source": "(ctx) => ({ ok: true })" }
+}
+```
+
+```json
+{
+    "name": "okdb_index",
+    "arguments": {
+        "action": "create",
+        "env": "default",
+        "type": "places",
+        "fields": ["location"],
+        "indexType": "geo",
+        "precision": 7
+    }
+}
+```
+
+Convenience MCP aliases:
+
+- `fields: [...]` is joined into the canonical `index` id using `~`
+- `indexType: "geo"` maps to the underlying index type hint
+- env-scoped MCP calls must always send an explicit `env`
+
+The MCP layer excludes non-tool transports by default, including:
+
+- SSE/event stream routes
+- raw upload routes
+- streaming download/export routes
+- hidden/internal transport routes
+
+Generated MCP calls route through `okdb.http.handle(...)` rather than duplicating feature logic.
+
+For the grouped `okdb_pipeline` tool:
+
+- embeddings env/type pipeline actions keep concise names such as `list`, `stats`, `query`, `start`, and `delete`
+- generic pipeline registry actions use the `generic_*` prefix (`generic_list`, `generic_get`, `generic_create`, `generic_start`, `generic_stop`, `generic_delete`, `generic_scaffold`) so they do not collide with embeddings actions
+- as an MCP convenience, `action: "list"` falls back to env-scoped `generic_list` when `type` is omitted; include `type` to get the specialized embeddings env/type listing
+- template discovery/execution is also available as `template_list`, `template_get`, `template_preview`, and `template_create`
+- those template actions are pipeline-only and correspond to `/api/pipelines/templates...`, not the broader shared engine registry
+
+For the grouped `okdb_engine` tool:
+
+- `action: "types"` lists registered engine kinds and any self-declared config schemas/examples
+- `action: "templates" | "template_get" | "template_preview"` exposes the shared starter registry, including both direct-engine templates and pipeline templates
+- `action: "template_create"` creates a direct env-scoped engine from a starter template
+- `action: "list" | "get" | "create" | "patch" | "delete" | "restart" | "pause" | "resume" | "retry" | "reset_cursor" | "rebuild"` targets direct env-scoped engine admin routes
+
+For the grouped `okdb_embeddings` tool:
+
+- create actions remain available (`embedder_create`, `indexer_create`, `search_create`, `worker_create`)
+- discovery actions are also available (`providers`, `models`, `models_by_provider`, `algorithms`, `probe_ollama_models`, `embedder_models`) so MCP clients can inspect provider schemas and model catalogs before creating engines or pipelines
+
+---
+
+## Engine management endpoints
+
+The direct `engines` feature exposes env-scoped inspection and admin routes:
+
+```text
+GET    /api/engines/types
+GET    /api/engines/templates
+GET    /api/engines/templates/:template
+POST   /api/engines/templates/:template/preview
+POST   /api/env/:env/engines/templates/:template/create
+GET    /api/env/:env/engines?type=<engine-type>
+POST   /api/env/:env/engines
+GET    /api/env/:env/engines/:type/:name
+PATCH  /api/env/:env/engines/:type/:name
+DELETE /api/env/:env/engines/:type/:name
+POST   /api/env/:env/engines/:type/:name/restart
+POST   /api/env/:env/engines/:type/:name/pause
+POST   /api/env/:env/engines/:type/:name/resume
+POST   /api/env/:env/engines/:type/:name/retry
+POST   /api/env/:env/engines/:type/:name/reset-cursor
+POST   /api/env/:env/engines/:type/:name/rebuild
+```
+
+`GET /api/engines/types` returns the registered engine kinds plus any backend-supplied config schema, example config, patchability, creation notes, and matching starter `templateIds` / `recommendedTemplateIds`.
+`GET /api/engines/templates` returns the full starter registry shared by direct engines and pipeline builders.
+That means it includes both `kind: "engine"` and `kind: "pipeline"` entries, whereas `GET /api/pipelines/templates` exposes only the pipeline subset.
+`POST /api/engines/templates/:template/preview` expands a starter template into the concrete create payload it would execute, without persisting anything.
+`POST /api/env/:env/engines/templates/:template/create` only accepts templates whose plan kind is `engine`; pipeline templates must go through `/api/env/:env/pipelines/templates/:template/create`.
+`GET /api/env/:env/engines` returns runtime summaries for engines stored in that environment. The optional `type` query parameter narrows results to one engine type.
+
+MCP exposes these routes through `okdb_engine`, so clients can discover engine types and create/manage direct engines without going through a pipeline.
+
+---
+
+## Contributor guidance for new routes
+
+When adding a new route in any `*-http.js` file:
+
+1. provide a stable `id`
+2. add `summary`, `description`, and `tags`
+3. define `inputSchema`
+4. define `outputSchema` where practical
+5. classify `safety` and `access`
+6. set `http.bodyMode` and `http.querySchema` as needed
+7. decide whether the route should participate in MCP via `mcp.include`
+8. avoid exposing raw/SSE/stream routes to MCP unless you add a deliberate adapter
+
+Feature authors should keep the route's owning feature responsible for its metadata. Do not create a second disconnected metadata source for MCP or CLI.
+
+---
+
+## CRUD endpoints
+
+All routes are under `/api`. The `:env` prefix is `default` unless you specify another environment.
+
+### Records
+
+```http
+GET    /api/:env/type/:type/item/:key    → single record
+PUT    /api/:env/type/:type/item/:key    → upsert (body.value = value)
+PATCH  /api/:env/type/:type/item/:key    → partial update (body.patch = patch doc)
+DELETE /api/:env/type/:type/item/:key    → remove
+POST   /api/:env/type/:type/query        → query records
+POST   /api/:env/transaction             → atomic batch writes
+```
+
+```bash
+# Get a record
+curl http://localhost:8080/api/default/type/users/item/alice \
+  -H "Authorization: Bearer token"
+
+# Upsert by key
+curl -X PUT http://localhost:8080/api/default/type/users/item/alice \
+  -H "Authorization: Bearer token" \
+  -H "Content-Type: application/json" \
+  -d '{"value": {"name": "Alice", "role": "superadmin"}}'
+
+# Patch an existing record
+curl -X PATCH http://localhost:8080/api/default/type/users/item/alice \
+  -H "Authorization: Bearer token" \
+  -H "Content-Type: application/json" \
+  -d '{"patch":{"set":{"role":"admin"},"inc":{"loginCount":1},"append":{"tags":["vip"]},"pull":{"tags":["trial"]}},"ifVersion":7}'
+
+# Delete
+curl -X DELETE http://localhost:8080/api/default/type/users/item/alice \
+  -H "Authorization: Bearer token"
+```
+
+### Patch document format
+
+`PATCH` uses a structured patch document.
+
+- `set` — assign or replace fields/paths
+- `unset` — remove fields/paths
+- `inc` — increment numeric fields
+- `merge` — deep-merge object subtrees
+- `append` — append one or more values to arrays
+- `pull` — remove matching values from arrays
+
+```json
+{
+    "patch": {
+        "set": { "status": "active" },
+        "inc": { "metrics.views": 1 },
+        "merge": { "profile": { "theme": "dark" } },
+        "append": { "tags": ["featured"] },
+        "pull": { "tags": ["draft"] }
+    }
+}
+```
+
+Notes:
+
+- patch applies to existing object records only
+- arrays are incrementally mutated via `append` and `pull`
+- `ifVersion` enables compare-and-swap writes
+- successful patch calls return `204 No Content`
+
+### List records
+
+```http
+GET /api/:env/type/:type
+  ?limit=50
+  &offset=0
+  &start=<key>
+  &end=<key>
+  &reverse=false
+```
+
+```bash
+curl "http://localhost:8080/api/default/type/users?limit=10&reverse=true" \
+  -H "Authorization: Bearer token"
+```
+
+---
+
+## Change log endpoints
+
+```http
+GET /api/:env/changelog
+  ?before=<clock>    → changes up to this clock (inclusive)
+  ?after=<clock>     → changes from this clock onwards
+  ?type=<typename>   → filter by type
+
+GET /api/:env/type/:type/changelog
+  ?before=<clock>
+  ?after=<clock>
+```
+
+```bash
+# Get changes since clock 100
+curl "http://localhost:8080/api/default/changelog?after=100" \
+  -H "Authorization: Bearer token"
+```
+
+---
+
+## Schema endpoints
+
+```http
+GET  /api/:env/types                    → list all registered types
+POST /api/:env/types                    → register a type
+DELETE /api/:env/type/:type             → drop a type
+
+POST /api/:env/type/:type/indexes       → register an index
+DELETE /api/:env/type/:type/index/:idx  → drop an index
+```
+
+---
+
+## System info
+
+```http
+GET /api/info
+```
+
+Returns node ID, version, clock, type stats, plugin list, and resolved LMDB options.
+
+---
+
+## Sync endpoint
+
+```http
+GET /api/sync/delta?from_clock=<N>
+Authorization: Bearer <sync-token>
+```
+
+Used by peer nodes during reconciliation. Returns a JSON stream of change objects.
+
+```http
+POST /api/sync/join
+Authorization: Bearer <sync-token>
+Content-Type: application/json
+
+{ "address": "http://peer.example.com:8080" }
+```
+
+---
+
+## File endpoints
+
+```http
+POST   /api/files                → upload file (multipart or raw)
+GET    /api/files/:id            → download file (streams blob bytes)
+GET    /api/files/:id/meta       → metadata only (no body stream)
+DELETE /api/files/:id            → remove file
+GET    /api/files                → list file metadata
+```
+
+---
+
+## Custom middleware
+
+Add your own middleware before route handlers:
+
+```javascript
+okdb.http.use(async (req, res, next) => {
+    console.log(`${req.method} ${req.url}`);
+    await next();
+});
+```
+
+Register custom routes:
+
+```javascript
+okdb.http.add('GET', '/api/custom/health', () => ({
+    result: { ok: true, clock: okdb.getClock() },
+}));
+```
+
+---
+
+## Integrating with an existing framework
+
+OKDB's HTTP layer is **framework-agnostic**. `okdb.http.listen()` is entirely optional — if you already have an Express, Fastify, or raw Node.js server, you can hand requests to OKDB directly via `okdb.http.handle()`.
+
+`handle(method, path, { body, query, headers }, context)` returns a promise that resolves to `{ status, headers, body }`. Wire that into your framework's response however you like.
+
+### Express
+
+```javascript
+const express = require('express');
+const OKDB = require('src/okdb');
+
+const app = express();
+const okdb = new OKDB('./mydb');
+await okdb.open();
+
+// Mount all /api/* and /admin/* routes on the existing Express server
+app.use(express.json());
+app.use(async (req, res, next) => {
+    const path = req.path;
+    if (!path.startsWith('/api') && !path.startsWith('/admin') && !path.startsWith('/docs')) {
+        return next();
+    }
+
+    const response = await okdb.http.handle(
+        req.method,
+        path,
+        {
+            body: req.body,
+            query: req.query,
+            headers: req.headers,
+        },
+        { req, res },
+    );
+
+    const { status = 200, headers = {}, body } = response;
+    res.status(status);
+    for (const [k, v] of Object.entries(headers)) res.set(k, v);
+
+    if (body == null) return res.end();
+    if (typeof body?.pipe === 'function') return body.pipe(res);
+    if (Buffer.isBuffer(body)) return res.end(body);
+    if (typeof body === 'object') return res.json(body);
+    res.send(String(body));
+});
+
+app.listen(3000);
+```
+
+### Fastify
+
+```javascript
+const fastify = require('fastify')({ logger: false });
+const OKDB = require('src/okdb');
+
+const okdb = new OKDB('./mydb');
+await okdb.open();
+
+fastify.addContentTypeParser('application/json', { parseAs: 'string' }, (req, body, done) => {
+    try {
+        done(null, JSON.parse(body));
+    } catch (e) {
+        done(e);
+    }
+});
+
+fastify.all('/api/*', async (request, reply) => {
+    const response = await okdb.http.handle(
+        request.method,
+        request.url.split('?')[0],
+        {
+            body: request.body,
+            query: request.query,
+            headers: request.headers,
+        },
+        { req: request.raw, res: reply.raw },
+    );
+
+    const { status = 200, headers = {}, body } = response;
+    reply.status(status);
+    for (const [k, v] of Object.entries(headers)) reply.header(k, v);
+
+    if (body == null) return reply.send();
+    if (typeof body?.pipe === 'function') return reply.send(body);
+    return reply.send(body);
+});
+
+await fastify.listen({ port: 3000 });
+```
+
+### Raw Node.js `http`
+
+```javascript
+const http = require('http');
+const OKDB = require('src/okdb');
+
+const okdb = new OKDB('./mydb');
+await okdb.open();
+
+const server = http.createServer(async (req, res) => {
+    // Your own routes first
+    if (req.url === '/health') {
+        res.end('ok');
+        return;
+    }
+
+    // Delegate everything else to OKDB
+    let body = null;
+    if (req.method !== 'GET' && req.method !== 'HEAD') {
+        body = await new Promise((resolve, reject) => {
+            let data = '';
+            req.on('data', (c) => (data += c));
+            req.on('end', () => {
+                try {
+                    resolve(data ? JSON.parse(data) : null);
+                } catch (e) {
+                    reject(e);
+                }
+            });
+            req.on('error', reject);
+        });
+    }
+
+    const url = new URL(req.url, 'http://localhost');
+    const response = await okdb.http.handle(
+        req.method,
+        url.pathname,
+        {
+            body,
+            query: Object.fromEntries(url.searchParams),
+            headers: req.headers,
+        },
+        { req, res },
+    );
+
+    const { status = 200, headers = {}, body: resBody } = response;
+    res.statusCode = status;
+    for (const [k, v] of Object.entries(headers)) res.setHeader(k, v);
+
+    if (resBody == null) {
+        res.end();
+        return;
+    }
+    if (typeof resBody?.pipe === 'function') {
+        resBody.pipe(res);
+        return;
+    }
+    if (Buffer.isBuffer(resBody)) {
+        res.end(resBody);
+        return;
+    }
+    if (typeof resBody === 'object') {
+        res.setHeader('content-type', 'application/json');
+        res.end(JSON.stringify(resBody));
+        return;
+    }
+    res.end(String(resBody));
+});
+
+server.listen(3000);
+```
+
+:::tip
+When you bring your own server, don't call `okdb.http.listen()`. OKDB's routes and middleware are still fully functional — you're just providing the transport layer yourself.
+:::
+
+:::note
+`okdb.http.handle()` still runs all registered middleware and guards. If you configure admin auth (`options.admin.auth`), the `/admin/*` guard still applies — the same cookie/Basic-Auth logic works regardless of which HTTP framework is in front.
+:::
+
+---
+
+## Admin UI
+
+When `admin.auth` is configured, the admin UI is available at:
+
+```
+http://localhost:8080/admin/index.html
+```
+
+It provides:
+
+- Type and record browser
+- Index management (create, drop, view status)
+- Live changelog viewer
+- Embeddings pipeline status
+- Sync peer status
+- System info