@cef-ai/cli 0.2.1

package/README.md

@@ -0,0 +1,1273 @@
+# CEF AI CLI
+
+Build, test, and deploy CEF AI agent topologies from the command line.
+
+Write agent handlers locally, test them with a real dev server (SQLite cubbies, streams, execution tracing), then deploy to the platform from a single YAML config.
+
+**Package:** `@cef-ai/cli` v0.2.0
+**Binaries:** `cef`, `cef-deploy`
+**Requirements:** Node.js 18+
+
+---
+
+## Table of Contents
+
+- [Quick Start](#quick-start)
+- [Developer Workflow](#developer-workflow)
+- [Commands](#commands)
+  - [cef dev - Local Development Server](#cef-dev---local-development-server)
+    - [Dev UI](#dev-ui)
+    - [HTTP API Endpoints](#http-api-endpoints)
+    - [WebSocket API](#websocket-api)
+    - [Data Directory (.cef-dev/)](#data-directory-cef-dev)
+    - [Hot Reload](#hot-reload)
+  - [cef deploy - Deploy Topology](#cef-deploy---deploy-topology)
+    - [Deploy Order](#deploy-order)
+    - [ID Writeback](#id-writeback)
+    - [Stale ID Detection](#stale-id-detection)
+    - [Dry Run](#dry-run)
+    - [Partial Deploy](#partial-deploy)
+  - [cef deploy status - Compare Local vs Deployed](#cef-deploy-status---compare-local-vs-deployed)
+  - [cef deploy clone - Download Existing Service](#cef-deploy-clone---download-existing-service)
+  - [cef deploy delete - Delete Deployed Entities](#cef-deploy-delete---delete-deployed-entities)
+  - [cef service list - List Agent Services](#cef-service-list---list-agent-services)
+  - [cef service create - Create New Agent Service](#cef-service-create---create-new-agent-service)
+  - [cef playground test - Test Agent Tools via MCP](#cef-playground-test---test-agent-tools-via-mcp)
+    - [Single Test](#single-test)
+    - [List Tools](#list-tools)
+    - [Batch Tests](#batch-tests)
+    - [Test File Format (cef.tests.yaml)](#test-file-format-ceftestsyaml)
+- [Configuration (cef.config.yaml)](#configuration-cefconfigyaml)
+  - [Top-Level Fields](#top-level-fields)
+  - [Engagements](#engagements)
+  - [Agents](#agents)
+  - [Cubbies](#cubbies)
+  - [Workspaces](#workspaces)
+  - [Full Example Config](#full-example-config)
+- [Agent Sandbox](#agent-sandbox)
+  - [Handler Signature](#handler-signature)
+  - [Available Globals](#available-globals)
+  - [Restrictions](#restrictions)
+- [Context API Reference](#context-api-reference)
+  - [Logging](#logging)
+  - [Event Emission](#event-emission)
+  - [HTTP Fetch](#http-fetch)
+  - [Agent-to-Agent Calls](#agent-to-agent-calls)
+  - [Cubby Operations](#cubby-operations)
+  - [Stream Subscriptions](#stream-subscriptions)
+  - [Raft Queries](#raft-queries)
+- [Environment Variables](#environment-variables)
+- [Options Reference](#options-reference)
+- [Examples](#examples)
+  - [Hello World](#hello-world)
+  - [Drone Telemetry (Workspaces + Streams)](#drone-telemetry-workspaces--streams)
+  - [Wiki Knowledge Graph (Graph Viz + Batch)](#wiki-knowledge-graph-graph-viz--batch)
+- [Testing](#testing)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Quick Start
+
+```bash
+# Install globally
+npm install -g @cef-ai/cli
+
+# Or use npx
+npx @cef-ai/cli dev
+
+# Start local dev server (no auth needed)
+cef dev
+
+# Start with a specific config
+cef dev --config ./my-agent/cef.config.yaml
+
+# Deploy to platform (requires .env with auth tokens)
+cef deploy --config ./cef.config.yaml
+```
+
+---
+
+## Developer Workflow
+
+The recommended flow from idea to production:
+
+```
+1. Scaffold       cef service create --name my-agent
+                  Set up cef.config.yaml with agents, cubbies, engagements
+
+2. Write agents   Create handler .ts files (fully inline, no imports)
+                  Define cubby migrations in config for any SQL state
+
+3. Dev server     cef dev
+                  Opens http://localhost:8787 with:
+                  - Topology sidebar, agent/engagement panels
+                  - Stream push (single, bulk, file upload)
+                  - Live execution logs and call chain tracing
+
+4. Test agents    POST /api/agents/:alias/:task with payloads
+                  Trigger full engagements via POST /api/trigger
+                  Push stream packets via POST /api/streams/:id/push
+
+5. Inspect data   Use the SQL editor in the dev UI
+                  Or: POST /api/cubbies/:alias/query with {"sql": "SELECT ..."}
+                  Cubbies are real SQLite with vector support (sqlite-vec)
+
+6. Iterate        Edit agent code; changes hot-reload automatically
+                  .cef-dev/ is cleared on shutdown (use --persist to keep)
+
+7. Deploy         Create .env with CEF_AUTH_TOKEN, CEF_ORCHESTRATOR_URL
+                  cef deploy
+                  IDs are written back into cef.config.yaml
+
+8. Verify         cef deploy status (compares local vs deployed)
+                  Redeploy after changes: cef deploy
+```
+
+---
+
+## Commands
+
+| Command | Description | Auth Required |
+|-|-|-|
+| `cef dev` | Start local dev server with UI, hot-reload, real SQLite cubbies | No |
+| `cef deploy` | Deploy topology to orchestrator from cef.config.yaml | Yes |
+| `cef deploy status` | Compare local config vs deployed state | Yes |
+| `cef deploy clone` | Download an existing service into a directory | Yes |
+| `cef deploy delete` | Delete deployed entities (reverse dependency order) | Yes |
+| `cef service list` | List agent services from ROB | Yes |
+| `cef service create` | Create a new agent service on ROB + orchestrator | Yes |
+| `cef playground test` | Run playground tests against deployed agent tools | Yes |
+
+---
+
+### cef dev - Local Development Server
+
+Starts a local dev server that emulates the CEF platform. Loads `cef.config.yaml`, runs agent code in sandboxed V8 contexts, and provides an HTTP API and web UI.
+
+No auth token or orchestrator connection required. Runs as a persistent process until Ctrl+C.
+
+```bash
+cef dev                                    # Default: localhost:8787
+cef dev --config ./my-agent/cef.config.yaml
+cef dev --port 3000
+cef dev --host 0.0.0.0
+cef dev --persist                          # Keep .cef-dev/ data after shutdown
+```
+
+| Option | Default | Description |
+|-|-|-|
+| `--config <path>` | `./cef.config.yaml` | Path to config file |
+| `--port <port>` | `8787` | Server port |
+| `--host <host>` | `localhost` | Server host |
+| `--persist` | `false` | Keep `.cef-dev/` data after shutdown |
+
+#### Dev UI
+
+Open `http://localhost:<port>` to access the built-in web UI:
+
+- **Topology sidebar** listing all engagements, agents, cubbies, and workspaces
+- **Engagement trigger panel** with payload editor and stream push (single, bulk JSON Lines, file upload for .json/.jsonl)
+- **Engagement blast mode** with repeat count, delay, and JSON file upload for burst testing
+- **Agent task panels** with pre-filled payloads based on `parameters` JSON Schema from config
+- **Run Batch** button on agent panels; queries a cubby for input rows via `batch_file` SQL, then calls the agent once per row
+- **Cubby table browser** with SQL editor for ad-hoc queries
+- **Saved queries** for the SQL editor; save, load, and delete named queries from a dropdown
+- **Graph visualization** tab on cubby panels using vis.js; color-coded nodes by type (artifact, work, operator, eval), click-to-inspect detail panel, legend, and stats
+- **Live execution logs** with filtering by execution ID
+- **Call chain tracing** visualizing engagement -> agent -> cubby execution flows
+- **Real-time updates** via WebSocket
+
+**Config hint:** Add `parameters` and `returns` to agent tasks in cef.config.yaml (JSON Schema format). The dev UI uses these to pre-fill payloads and show schema hints. These fields are local-only and not sent to the orchestrator.
+
+#### HTTP API Endpoints
+
+All endpoints are available for scripting with curl or any HTTP client.
+
+| Method | Path | Body | Description |
+|-|-|-|-|
+| `GET` | `/` | - | Dev UI |
+| `GET` | `/api/topology` | - | Returns loaded agents, engagements, cubbies, workspaces |
+| `GET` | `/api/executions` | - | Returns execution logs (optional `?executionId=` filter) |
+| `POST` | `/api/trigger` | `{"engagement": "Name", "payload": {...}}` | Start an engagement |
+| `POST` | `/api/agents/:alias/:task` | `{"payload": {...}}` | Call an agent task directly |
+| `POST` | `/api/streams/:id/push` | `{"data": {"sequenceNum": 0, "payload": {...}}}` | Push a packet to a running stream |
+| `POST` | `/api/cubbies/:alias/query` | `{"sql": "SELECT ...", "params": [], "instanceId": "default"}` | SQL read query |
+| `POST` | `/api/cubbies/:alias/exec` | `{"sql": "INSERT ...", "params": []}` | SQL write operation |
+| `GET` | `/api/cubbies/:alias/instances` | - | List all cubby instances |
+
+Example: call an agent task directly:
+
+```bash
+curl -X POST http://localhost:8787/api/agents/greeter/greet \
+  -H "Content-Type: application/json" \
+  -d '{"payload": {"name": "Alice"}}'
+```
+
+Example: query a cubby:
+
+```bash
+curl -X POST http://localhost:8787/api/cubbies/greetings/query \
+  -H "Content-Type: application/json" \
+  -d '{"sql": "SELECT * FROM greetings"}'
+```
+
+Example: trigger an engagement:
+
+```bash
+curl -X POST http://localhost:8787/api/trigger \
+  -H "Content-Type: application/json" \
+  -d '{"engagement": "Hello World", "payload": {"streamId": "test-stream-1"}}'
+```
+
+#### WebSocket API
+
+Connect to `ws://localhost:<port>/ws` for real-time streaming of logs and trace events.
+
+Messages are JSON with two types:
+
+**Log entries:**
+
+```json
+{
+  "type": "log",
+  "data": {
+    "timestamp": 1712345678000,
+    "executionId": "exec-abc123",
+    "agentAlias": "greeter",
+    "taskAlias": "greet",
+    "level": "log",
+    "args": ["Greeter agent called with name: Alice"]
+  }
+}
+```
+
+**Trace events:**
+
+```json
+{
+  "type": "trace",
+  "data": {
+    "timestamp": 1712345678000,
+    "executionId": "exec-abc123",
+    "type": "agent:call",
+    "caller": "engagement:Hello World",
+    "callee": "greeter.greet",
+    "durationMs": 12,
+    "success": true
+  }
+}
+```
+
+Trace event types: `engagement:start`, `agent:call`, `agent:return`, `cubby:query`, `cubby:exec`
+
+#### Data Directory (.cef-dev/)
+
+The dev server stores SQLite databases in `.cef-dev/cubbies/{alias}/{instanceId}.db` relative to the config directory.
+
+```
+.cef-dev/
+  cubbies/
+    greetings/
+      default.db
+    syncMission/
+      default.db
+      drone-001.db
+```
+
+By default, `.cef-dev/` is deleted on shutdown. Use `--persist` to keep data between runs.
+
+#### Hot Reload
+
+The dev server watches all agent task and engagement handler files. On save:
+
+1. New TypeScript code is read from disk
+2. Code is transpiled (ES2022 target) and cached
+3. Next call to that agent/task uses the updated code
+4. A `[reload] path/to/file` message appears in the console
+
+In-flight executions continue with the old code; only new calls use the updated version. No server restart needed.
+
+**What runs locally:**
+
+- **Agents** execute in sandboxed V8 contexts (see [Agent Sandbox](#agent-sandbox))
+- **Cubbies** are real SQLite databases (better-sqlite3 + sqlite-vec for vector ops)
+- **Streams** are in-memory async iterators; push packets via UI or API
+- **Execution tracing** tracks engagement -> agent -> cubby call chains in real-time
+
+---
+
+### cef deploy - Deploy Topology
+
+Reads `cef.config.yaml`, creates or updates each entity on the orchestrator, and writes returned IDs back into the config.
+
+```bash
+cef deploy
+cef deploy --config ./app/cef.config.yaml
+cef deploy --dry-run
+cef deploy --only agent
+```
+
+| Option | Default | Description |
+|-|-|-|
+| `--config <path>` | `./cef.config.yaml` | Path to config file |
+| `--dry-run` | `false` | Show planned operations without calling the API |
+| `--only <type>` | (all) | Deploy only one entity type |
+
+Valid `--only` values: `engagement`, `agent`, `cubby`, `workspace`, `stream`, `deployment`, `raft`
+
+#### Deploy Order
+
+Entities are created in dependency order:
+
+```
+Engagements -> Agents -> Cubbies -> Workspaces -> Streams -> Deployments -> Rafts
+```
+
+Each entity type completes before the next begins. If an entity already has an ID in config, the CLI issues an update instead of a create.
+
+#### ID Writeback
+
+After each entity type is created, the returned IDs are written back into `cef.config.yaml`. For example, after creating an engagement:
+
+```yaml
+engagements:
+  - name: Hello World
+    file: ./engagements/hello-engagement.ts
+    engagementId: eng-2f8b9517    # written by CLI
+```
+
+If the deploy fails partway through, already-created IDs are preserved in config. Subsequent runs detect existing IDs and issue updates instead of creates. The final `agentServiceId` resolved from ROB is also written back.
+
+Commit the updated config after a successful deploy.
+
+#### Stale ID Detection
+
+If `agentServiceId` in config does not match the resolved service from ROB (e.g. you copied a config from another service and changed `agentServicePubKey`), the CLI warns that IDs may belong to a different service. It prompts to clear all entity IDs and deploy fresh.
+
+#### Dry Run
+
+```bash
+cef deploy --dry-run
+```
+
+Logs what would be created or updated without making any API calls.
+
+#### Partial Deploy
+
+Deploy a single entity type:
+
+```bash
+cef deploy --only agent
+cef deploy --only cubby
+cef deploy --only workspace
+```
+
+---
+
+### cef deploy status - Compare Local vs Deployed
+
+Compares local handler code against the deployed version on the orchestrator. Checks engagements and agents (task code).
+
+```bash
+cef deploy status
+cef deploy status --config ./app/cef.config.yaml
+```
+
+| Option | Default | Description |
+|-|-|-|
+| `--config <path>` | `./cef.config.yaml` | Path to config file |
+
+Status values:
+
+| Status | Meaning |
+|-|-|
+| `up-to-date` | Local code matches deployed version byte-for-byte |
+| `outdated` | Deployed code differs from local |
+| `not-deployed` | Entity has no ID in config (never deployed) |
+| `not-found` | Entity ID exists in config but not found on orchestrator |
+
+Example output:
+
+```
+  [OK] engagement: Hello World (eng-2f8b9517)
+  [OUTDATED] agent: Object Detection (agent-628677bd)
+         task: Yolo [OUTDATED]
+         task: Detect [OK]
+
+1 outdated, 0 not deployed.
+```
+
+For agents, the status is aggregated from all tasks. If any task is outdated, the agent is reported as outdated.
+
+---
+
+### cef deploy clone - Download Existing Service
+
+Downloads all code and config from a deployed agent service and writes local files.
+
+```bash
+cef deploy clone --pubkey 0x... --output-dir ./my-agent
+cef deploy clone --pubkey 0x... --output-dir ./my-agent --force
+cef deploy clone --config ./other/cef.config.yaml
+```
+
+| Option | Default | Description |
+|-|-|-|
+| `--config <path>` | `./cef.config.yaml` | Path to config file |
+| `--pubkey <key>` | (from config) | Agent service public key (`0x...`) |
+| `--output-dir <dir>` | (config directory) | Directory to write files into |
+| `--force` | `false` | Overwrite existing files |
+
+Downloads in parallel:
+- Engagement handler code -> `./engagements/{name}.ts`
+- Agent task code -> `./agents/{alias}/tasks/{taskAlias}.ts`
+- Full ontology (workspaces, streams, deployments, rafts)
+- Cubby configs with migrations
+
+Produces a complete `cef.config.yaml` with all IDs populated, ready for `cef deploy status` or further edits. Without `--force`, existing files are not overwritten.
+
+---
+
+### cef deploy delete - Delete Deployed Entities
+
+Deletes deployed entities in reverse dependency order and removes the corresponding IDs from config.
+
+```bash
+cef deploy delete
+cef deploy delete --only agent
+cef deploy delete --force          # Skip confirmation prompt
+```
+
+| Option | Default | Description |
+|-|-|-|
+| `--config <path>` | `./cef.config.yaml` | Path to config file |
+| `--only <type>` | (all) | Delete only one entity type |
+| `--force` | `false` | Skip confirmation prompt |
+
+Deletion order (reverse of deploy):
+
+```
+Deployments -> Rafts -> Streams -> Workspaces -> Agents -> Engagements
+```
+
+After successful deletion, the corresponding IDs are cleared from `cef.config.yaml`.
+
+**Note:** Cubbies cannot be deleted via the CLI. A warning is shown for cubbies with IDs.
+
+Without `--force`, the CLI lists all entities to be deleted and asks for confirmation. In non-interactive terminals, `--force` is required.
+
+---
+
+### cef service list - List Agent Services
+
+Lists all active (non-archived) agent services from ROB.
+
+```bash
+cef service list
+cef service list --config ./app/cef.config.yaml
+```
+
+| Option | Default | Description |
+|-|-|-|
+| `--config <path>` | `./cef.config.yaml` | Path to config file (finds `.env` in same dir) |
+
+Requires `CEF_ROB_API_URL` in `.env`.
+
+Output:
+
+```
+Agent Services (3):
+  Parking Detection
+    agentServiceId:     2620
+    agentServicePubKey: 0x788b3863...
+    created:            2024-01-15T10:30:00Z
+
+  Hello World Agent
+    agentServiceId:     2621
+    agentServicePubKey: 0xbbccddee...
+    created:            2024-02-01T08:00:00Z
+```
+
+---
+
+### cef service create - Create New Agent Service
+
+Creates a new agent service on both ROB and the orchestrator.
+
+```bash
+cef service create --name my-agent
+cef service create --name my-agent --bucket-id 1505
+```
+
+| Option | Default | Required | Description |
+|-|-|-|-|
+| `--name <name>` | - | Yes | Name of the new agent service |
+| `--config <path>` | `./cef.config.yaml` | No | Path to config file |
+| `--bucket-id <id>` | `0` | No | DDC bucket ID |
+
+Requires `CEF_ROB_API_URL` in `.env`.
+
+Returns the service name, `agentServiceId`, and `agentServicePubKey`. Use the pubkey in your `cef.config.yaml`.
+
+---
+
+### cef playground test - Test Agent Tools via MCP
+
+Tests deployed agent tools using the MCP (Model Context Protocol) playground API. Agents must have an `agentId` (deployed via `cef deploy`).
+
+```bash
+cef playground test [options]
+```
+
+| Option | Default | Description |
+|-|-|-|
+| `--config <path>` | `./cef.config.yaml` | Path to config file |
+| `--agent <name>` | - | Agent name, alias, or agentId |
+| `--tool <alias>` | - | Task alias (tool name) to call |
+| `--args <json>` | - | JSON arguments to pass to the tool |
+| `--test-file <path>` | - | Path to `cef.tests.yaml` for batch testing |
+| `--list-tools` | `false` | List available tools on an agent |
+
+Agent resolution: matches by name, alias, or agentId from config.
+
+#### Single Test
+
+```bash
+cef playground test \
+  --agent objectDetection \
+  --tool yolo \
+  --args '{"image": "https://example.com/photo.jpg"}'
+```
+
+#### List Tools
+
+```bash
+cef playground test --agent objectDetection --list-tools
+```
+
+Lists all available task aliases (tools) on the specified agent.
+
+#### Batch Tests
+
+```bash
+cef playground test --test-file ./cef.tests.yaml
+```
+
+Runs all tests defined in the file. Exits with code 1 if any test fails.
+
+#### Test File Format (cef.tests.yaml)
+
+```yaml
+tests:
+  - agent: objectDetection
+    tool: yolo
+    arguments:
+      image: "https://example.com/photo.jpg"
+    expect:
+      success: true
+      result:
+        totalDetections: 3
+
+  - agent: parkingViolationDetector
+    tool: detect
+    arguments:
+      timestamp: 1712345678
+      detections: []
+      telemetry: {}
+    expect:
+      success: true
+```
+
+Expectations use partial matching; the response only needs to contain the specified fields. Omit `expect` to just verify the call succeeds without error.
+
+Output:
+
+```
+  [PASS] objectDetection > yolo (123ms)
+    {"totalDetections": 3, "detections": [...]}
+  [FAIL] parkingViolationDetector > detect (456ms)
+    - expected totalDetections: 3, got: 0
+  [ERROR] someAgent > someTask (789ms)
+    Agent not found
+
+Results: 1 passed, 1 failed, 1 error
+```
+
+---
+
+## Configuration (cef.config.yaml)
+
+The config file defines your entire agent topology. All file paths are resolved relative to the config file location. By default the CLI looks for `./cef.config.yaml` in the current directory.
+
+### Top-Level Fields
+
+```yaml
+agentServicePubKey: "0x..."       # Required. 64-char hex string with 0x prefix.
+agentServiceId: "2620"            # Optional. Set by CLI on first deploy.
+
+engagements: [...]                # Optional. Engagement definitions.
+agents: [...]                     # Optional. Agent definitions.
+cubbies: [...]                    # Optional. Cubby (database) definitions.
+workspaces: [...]                 # Optional. Workspace/stream/deployment definitions.
+```
+
+Get `agentServicePubKey` from `cef service list` or `cef service create`.
+
+### Engagements
+
+Engagements are the entry points that subscribe to streams and orchestrate agent calls.
+
+```yaml
+engagements:
+  - name: Hello World                    # Required. Display name.
+    description: Greets incoming events  # Optional.
+    file: ./engagements/hello.ts         # Required. Path to handler file.
+    version: 1.0.0                       # Optional. Semantic version.
+    engagementId: eng-2f8b9517           # Set by CLI on deploy. Do not set manually.
+    parameters:                          # Optional. JSON Schema for dev UI hints.
+      type: object
+      properties:
+        streamId:
+          type: string
+      required: [streamId]
+```
+
+The `parameters` field is local-only (not sent to orchestrator). It drives input hints in the dev UI.
+
+### Agents
+
+Agents contain one or more tasks (tools). Each task has its own handler file.
+
+```yaml
+agents:
+  - name: Object Detection               # Required. Display name.
+    alias: objectDetection               # Required. Used in context.agents.{alias}.
+    description: Runs YOLO detection     # Optional.
+    version: 1.0.0                       # Optional.
+    agentId: agent-628677bd              # Set by CLI on deploy. Do not set manually.
+    tasks:
+      - name: Yolo                       # Required. Display name.
+        alias: yolo                      # Required. Used in context.agents.{alias}.{taskAlias}().
+        file: ./agents/od/tasks/yolo.ts  # Required. Path to handler file.
+        parameters:                      # Optional. JSON Schema for input.
+          type: object
+          properties:
+            image:
+              type: string
+              description: URL of image to analyze
+          required: []
+        returns:                         # Optional. JSON Schema for output.
+          type: object
+          properties:
+            totalDetections:
+              type: number
+            detections:
+              type: array
+        batch_file: ./agents/od/eval.sql   # Optional. SQL file for Run Batch (dev only).
+        batch_cubby: detection_store       # Optional. Cubby alias to run batch_file against.
+```
+
+`parameters` and `returns` are local-only. They drive dev UI input hints and output schema display. The MCP playground also uses `parameters` for tool argument schema.
+
+`batch_file` and `batch_cubby` are local dev only. When both are set, the dev UI shows a "Run Batch" button on the agent task panel. Clicking it runs the SQL from `batch_file` against `batch_cubby`, then calls the agent once per result row.
+
+### Cubbies
+
+Cubbies are per-agent SQLite databases with versioned schema migrations.
+
+```yaml
+cubbies:
+  - alias: greetings                     # Required. Used in context.cubbies.{alias}.
+    name: Greetings Store                # Required. Display name.
+    description: Stores greeting records # Optional.
+    cubbyId: 53d8630e-...               # Set by CLI on deploy. Do not set manually.
+    maxSizeBytes: 10485760              # Optional. Max DB size in bytes.
+    idleTimeout: 24h                    # Optional. Instance idle timeout.
+    migrations:
+      - version: 1                       # Required. Integer, applied in order.
+        up: |                            # Required. SQL DDL statements.
+          CREATE TABLE greetings (
+            id INTEGER PRIMARY KEY,
+            name TEXT,
+            message TEXT,
+            created_at TEXT DEFAULT (datetime('now'))
+          );
+          CREATE INDEX idx_greetings_name ON greetings(name);
+      - version: 2
+        up: |
+          ALTER TABLE greetings ADD COLUMN source TEXT DEFAULT 'manual';
+```
+
+Migrations run in version order. Each `up` field can contain multiple SQL statements separated by semicolons. A `schema_migrations` table tracks applied versions.
+
+In the local dev server, cubbies are backed by real SQLite with `better-sqlite3` and `sqlite-vec` (vector operations). Pragmas applied: WAL journal mode, 64MB cache, 5s busy timeout, NORMAL synchronous mode.
+
+### Workspaces
+
+Workspaces group related streams. Streams contain deployments (which bind engagements to event flows) and rafts (query interfaces).
+
+```yaml
+workspaces:
+  - name: Freemont                       # Required.
+    description: Freemont zone           # Optional.
+    workspaceId: "2198"                  # Set by CLI on deploy.
+    streams:
+      - name: DSC 142                    # Required.
+        description: Camera feed DSC 142 # Optional.
+        streamId: stream-f5213550        # Set by CLI on deploy.
+        selectors:                       # Optional. Event type filters.
+          - name: illegalParkingDetection
+            conditions:
+              - event_type:illegalParking
+        deployments:
+          - name: Illegal Parking Detection  # Required.
+            description: Detects violations  # Optional.
+            engagement: Illegal Parking detection  # Required. References engagement by name.
+            isActive: true                   # Optional. Default true.
+            deploymentId: dep-21e72017       # Set by CLI on deploy.
+            triggers:                        # Optional. Event matching rules.
+              - name: all-events
+                conditions:
+                  - "*"
+            engagementRules: []              # Optional.
+        rafts:
+          - name: Query Interface            # Required.
+            description: Raft for queries    # Optional.
+            file: ./queries/raft.ts          # Optional. Path to raft handler.
+            raftId: raft-abc123              # Set by CLI on deploy.
+            dataSourcesIds: []               # Optional.
+            queryOperations: []              # Optional.
+```
+
+Raft handlers export `onInit()`, `onData()`, `query()`, `onCleanup()` functions.
+
+### Full Example Config
+
+A minimal working config (from `example/hello-world/cef.config.yaml`):
+
+```yaml
+agentServicePubKey: "0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+
+workspaces:
+  - name: Local Dev
+    workspaceId: "local-ws-1"
+
+engagements:
+  - name: Hello World
+    file: ./engagements/hello-engagement.ts
+    parameters:
+      type: object
+      properties:
+        streamId:
+          type: string
+          description: Stream to subscribe to
+      required: [streamId]
+
+agents:
+  - name: Greeter
+    alias: greeter
+    tasks:
+      - name: Greet
+        alias: greet
+        file: ./agents/greeter.ts
+        parameters:
+          type: object
+          properties:
+            name:
+              type: string
+              description: Name of the person to greet
+          required: [name]
+        returns:
+          type: object
+          properties:
+            greeting:
+              type: string
+            timestamp:
+              type: number
+
+  - name: Summarizer
+    alias: summarizer
+    tasks:
+      - name: Summarize
+        alias: summarize
+        file: ./agents/summarizer.ts
+        parameters:
+          type: object
+          properties: {}
+        returns:
+          type: object
+          properties:
+            summary:
+              type: string
+            totalGreetings:
+              type: number
+
+cubbies:
+  - alias: greetings
+    name: Greetings Store
+    migrations:
+      - version: 1
+        up: "CREATE TABLE greetings (id INTEGER PRIMARY KEY, name TEXT, message TEXT, created_at TEXT DEFAULT (datetime('now')))"
+```
+
+A full reference config with workspaces, streams, deployments, and rafts is in the `example/` directory.
+
+---
+
+## Agent Sandbox
+
+Agent and engagement handlers run in isolated V8 sandboxes using Node.js `vm` module. Each execution gets its own context with no access to the host environment.
+
+### Handler Signature
+
+Every handler file must define an `async function handle`:
+
+```typescript
+async function handle(event: any, context: any) {
+  // event.payload contains the input data
+  const name = event.payload.name;
+
+  // Use context to interact with the platform
+  context.log('Processing: ' + name);
+
+  // Return a result
+  return { greeting: `Hello, ${name}!` };
+}
+```
+
+The function receives:
+- `event` with `event.payload` containing the input data
+- `context` providing platform APIs (logging, agents, cubbies, streams, fetch)
+
+The return value is sent back to the caller (another agent, the playground, or the dev UI).
+
+### Available Globals
+
+The sandbox provides these globals and nothing else:
+
+| Global | Description |
+|-|-|
+| `event` | Input payload object (`event.payload` has the data) |
+| `context` | Platform API (see [Context API Reference](#context-api-reference)) |
+| `console` | `log`, `warn`, `error` (proxied to context logging) |
+| `Promise` | Standard Promise |
+| `JSON` | Standard JSON |
+| `TextDecoder` | Standard TextDecoder |
+| `TextEncoder` | Standard TextEncoder |
+| `Uint8Array` | Standard Uint8Array |
+
+### Restrictions
+
+The following are **not available** in the sandbox:
+
+- `process` - no access to Node.js process
+- `require` / `import` - no module loading
+- `Buffer` - use `Uint8Array` and `TextEncoder`/`TextDecoder` instead
+- `fs` - no filesystem access
+- `global` / `globalThis` - isolated scope
+- Any Node.js built-in modules (`http`, `path`, `crypto`, etc.)
+
+All handler code must be fully self-contained. No `import` or `require` statements. This is a V8 isolate constraint; the same constraint applies to deployed handlers on the platform.
+
+---
+
+## Context API Reference
+
+The `context` object is injected into every handler and provides all platform interactions.
+
+### Logging
+
+```typescript
+context.log('info message', extraData);
+context.warn('warning message');
+context.error('error message');
+```
+
+Logs appear in the dev server console, the dev UI live log panel, and are broadcast via WebSocket. Each log entry captures the execution ID, agent/task alias, timestamp, and level.
+
+`console.log`, `console.warn`, and `console.error` are proxied to the corresponding context methods.
+
+### Event Emission
+
+```typescript
+context.emit(eventType: string, payload: any, targetId?: string);
+```
+
+Emits a custom event that appears in the trace log. Use for signaling between components or for custom instrumentation.
+
+### HTTP Fetch
+
+```typescript
+const response = await context.fetch('https://api.example.com/data', {
+  method: 'POST',
+  headers: { 'Content-Type': 'application/json' },
+  body: JSON.stringify({ key: 'value' })
+});
+const data = await response.json();
+```
+
+Standard fetch API. Available in both local dev and deployed environments. Use this for calling external APIs from within agent handlers.
+
+### Agent-to-Agent Calls
+
+```typescript
+const result = await context.agents.greeter.greet({ name: 'Alice' });
+const summary = await context.agents.summarizer.summarize({});
+```
+
+Call any agent task by alias. The pattern is `context.agents.{agentAlias}.{taskAlias}(payload)`.
+
+Returns the handler's return value directly. Each call emits `agent:call` and `agent:return` trace events with timing information.
+
+### Cubby Operations
+
+**Read query:**
+
+```typescript
+// Default instance
+const result = await context.cubbies.greetings.query(
+  'SELECT name, message FROM greetings WHERE name = ?',
+  ['Alice']
+);
+// result.columns = ['name', 'message']
+// result.rows = [['Alice', 'Hello, Alice!']]
+// result.meta = { duration: 2, rowsRead: 1 }
+```
+
+**Write operation:**
+
+```typescript
+const result = await context.cubbies.greetings.exec(
+  'INSERT INTO greetings (name, message) VALUES (?, ?)',
+  ['Alice', 'Hello, Alice!']
+);
+// result.rowsAffected = 1
+// result.lastInsertId = 1
+// result.meta = { duration: 1 }
+```
+
+**Named instances:**
+
+```typescript
+// Query a specific instance
+const result = await context.cubbies.greetings.query(
+  'user-123',
+  'SELECT * FROM greetings',
+  []
+);
+
+// Write to a specific instance
+await context.cubbies.greetings.exec(
+  'user-123',
+  'INSERT INTO greetings (name, message) VALUES (?, ?)',
+  ['Bob', 'Hello, Bob!']
+);
+```
+
+Both `query` and `exec` support an optional first argument for the instance ID. If the first argument contains a space, it is treated as SQL (using the `"default"` instance). Otherwise it is treated as an instance ID.
+
+Each operation emits `cubby:query` or `cubby:exec` trace events with timing and row counts.
+
+### Stream Subscriptions
+
+Used in engagements to consume stream events:
+
+```typescript
+async function handle(event: any, context: any) {
+  const stream = await context.streams.subscribe(event.payload.streamId);
+
+  for await (const packet of stream) {
+    const payload = JSON.parse(new TextDecoder().decode(packet.payload));
+    context.log('Received: ' + payload.event_type);
+
+    // Process packet and call agents...
+  }
+}
+```
+
+Each packet has:
+- `sequenceNum` (number) - packet sequence number
+- `payload` (Uint8Array) - packet data, decode with `TextDecoder`
+- `metadata` (object, optional) - additional metadata
+
+### Raft Queries
+
+```typescript
+const data = await context.rafts.myRaft.query();
+```
+
+In local dev mode, raft queries throw `"not available in local dev mode"`. In deployed environments, rafts execute against their configured data sources and query operations.
+
+---
+
+## Environment Variables
+
+Create a `.env` file in the same directory as your `cef.config.yaml`:
+
+```bash
+CEF_AUTH_TOKEN=your-jwt-token-here
+CEF_ORCHESTRATOR_URL=https://compute-1.devnet.ddc-dragon.com/orchestrator
+CEF_ROB_API_URL=https://api.rob.dev.cere.io/rms-node-backend
+```
+
+| Variable | Required For | Description |
+|-|-|-|
+| `CEF_AUTH_TOKEN` | deploy, status, clone, delete, service | JWT bearer token (same as ROB UI) |
+| `CEF_ORCHESTRATOR_URL` | deploy, status, clone, delete | Orchestrator base URL |
+| `CEF_ROB_API_URL` | service list/create, workspace registration | ROB API base URL |
+| `CEF_ROB_ORIGIN` | (optional) | Override Origin/Referer header for ROB requests |
+
+`cef dev` does **not** require any environment variables.
+
+Copy from `example/.env.example` and fill in your credentials.
+
+---
+
+## Options Reference
+
+| Option | Commands | Default | Description |
+|-|-|-|-|
+| `--config <path>` | all | `./cef.config.yaml` | Config file path |
+| `--dry-run` | deploy | `false` | Show planned operations without calling the API |
+| `--only <type>` | deploy, delete | (all) | Act on one entity type only |
+| `--pubkey <key>` | clone | (from config) | Agent service public key (`0x...`) |
+| `--output-dir <dir>` | clone | (config dir) | Directory to write cloned files into |
+| `--force` | clone, delete | `false` | Skip confirmation / overwrite existing files |
+| `--name <name>` | service create | - | Name of the new agent service (required) |
+| `--bucket-id <id>` | service create | `0` | DDC bucket ID |
+| `--port <port>` | dev | `8787` | Server port |
+| `--host <host>` | dev | `localhost` | Server host |
+| `--persist` | dev | `false` | Keep `.cef-dev/` data after shutdown |
+| `--agent <name>` | playground test | - | Agent name, alias, or agentId |
+| `--tool <alias>` | playground test | - | Task alias (tool name) to call |
+| `--args <json>` | playground test | - | JSON arguments for tool |
+| `--test-file <path>` | playground test | - | Path to cef.tests.yaml |
+| `--list-tools` | playground test | `false` | List available tools on an agent |
+| `-V, --version` | (global) | - | Show CLI version |
+| `--help` | (global) | - | Show help |
+
+---
+
+## Examples
+
+### Hello World
+
+A minimal topology with two agents sharing a cubby, driven by a stream-based engagement.
+
+**Engagement handler** (`engagements/hello-engagement.ts`):
+
+```typescript
+async function handle(event: any, context: any) {
+  const streamId = event.payload.streamId;
+  context.log('Hello engagement started, subscribing to stream: ' + streamId);
+
+  const stream = await context.streams.subscribe(streamId);
+
+  for await (const packet of stream) {
+    const data = JSON.parse(new TextDecoder().decode(packet.payload));
+    context.log('Received event: ' + data.event_type);
+
+    if (data.event_type === 'GREET') {
+      const greetResult = await context.agents.greeter.greet({
+        name: data.name,
+        timestamp: data.timestamp,
+      });
+      context.log('Greet result: ' + JSON.stringify(greetResult));
+
+    } else if (data.event_type === 'SUMMARIZE') {
+      const summaryResult = await context.agents.summarizer.summarize({});
+      context.log('Summary result: ' + JSON.stringify(summaryResult));
+    }
+  }
+}
+```
+
+**Greeter agent** (`agents/greeter.ts`):
+
+```typescript
+async function handle(event: any, context: any) {
+  const name = event.payload.name || 'World';
+  context.log(`Greeter agent called with name: ${name}`);
+
+  // Write to cubby
+  await context.cubbies.greetings.exec(
+    'INSERT INTO greetings (name, message) VALUES (?, ?)',
+    [name, `Hello, ${name}!`]
+  );
+
+  return { greeting: `Hello, ${name}!`, timestamp: Date.now() };
+}
+```
+
+**Summarizer agent** (`agents/summarizer.ts`):
+
+```typescript
+async function handle(event: any, context: any) {
+  context.log('Summarizer agent called');
+
+  // Read all greetings from cubby (cross-agent shared state)
+  const result = await context.cubbies.greetings.query(
+    'SELECT name, message FROM greetings ORDER BY id'
+  );
+
+  const names = result.rows.map((r: any) => r[0]);
+  const summary = `Greeted ${result.rows.length} people: ${names.join(', ')}`;
+
+  context.log(`Summary: ${summary}`);
+  return { summary, totalGreetings: result.rows.length };
+}
+```
+
+**Run it:**
+
+```bash
+# Start dev server
+cef dev --config ./example/hello-world/cef.config.yaml
+
+# Call the greeter directly
+curl -s -X POST http://localhost:8787/api/agents/greeter/greet \
+  -H "Content-Type: application/json" \
+  -d '{"payload": {"name": "Alice"}}'
+# -> {"greeting":"Hello, Alice!","timestamp":1712345678000}
+
+# Greet again
+curl -s -X POST http://localhost:8787/api/agents/greeter/greet \
+  -H "Content-Type: application/json" \
+  -d '{"payload": {"name": "Bob"}}'
+
+# Get summary (reads from same cubby)
+curl -s -X POST http://localhost:8787/api/agents/summarizer/summarize \
+  -H "Content-Type: application/json" \
+  -d '{"payload": {}}'
+# -> {"summary":"Greeted 2 people: Alice, Bob","totalGreetings":2}
+
+# Query the cubby directly
+curl -s -X POST http://localhost:8787/api/cubbies/greetings/query \
+  -H "Content-Type: application/json" \
+  -d '{"sql": "SELECT * FROM greetings"}'
+```
+
+See `example/hello-world/` for the complete project.
+
+### Drone Telemetry (Workspaces + Streams)
+
+A more complex topology with workspaces, streams, selectors, deployments, and triggers. This pattern is used when you need:
+
+- **Workspaces** to group related streams by location or domain
+- **Streams** with **selectors** to filter incoming events by type
+- **Deployments** that bind engagements to streams with trigger conditions
+- **Rafts** for query interfaces over collected data
+
+```yaml
+workspaces:
+  - name: Freemont
+    streams:
+      - name: DSC 142
+        selectors:
+          - name: illegalParkingDetection
+            conditions:
+              - event_type:illegalParking
+        deployments:
+          - name: Illegal Parking Detection
+            engagement: Illegal Parking detection    # references engagement by name
+            isActive: true
+            triggers:
+              - name: all-events
+                conditions:
+                  - "*"
+
+agents:
+  - name: Object Detection
+    alias: objectDetection
+    tasks:
+      - name: Yolo
+        alias: yolo
+        file: ./agents/object-detection/tasks/yolo.ts
+      - name: Detect
+        alias: detect
+        file: ./agents/parking-violation-detector/tasks/detect.ts
+
+cubbies:
+  - alias: syncMission
+    name: Sync Mission
+    migrations:
+      - version: 1
+        up: |
+          CREATE TABLE missions (id TEXT PRIMARY KEY, status TEXT NOT NULL, drone_id TEXT, started_at TEXT, ended_at TEXT, created_at TEXT DEFAULT (datetime('now')));
+          CREATE TABLE violations (id TEXT PRIMARY KEY, mission_id TEXT NOT NULL, type TEXT NOT NULL, latitude REAL, longitude REAL, confidence REAL, detected_at TEXT NOT NULL, FOREIGN KEY (mission_id) REFERENCES missions(id));
+          CREATE INDEX idx_violations_mission ON violations(mission_id);
+```
+
+See `example/cef.config.yaml` for the complete drone telemetry config.
+
+### Wiki Knowledge Graph (Graph Viz + Batch)
+
+A knowledge graph pipeline that ingests Notion pages as versioned artifact nodes, evaluates changes with an LLM, and visualizes the result as a graph. Demonstrates graph visualization, saved queries, and batch execution.
+
+```bash
+cd example/wiki-graph
+cef dev --config cef.config.yaml
+
+# Seed saved queries (first time)
+node connector/seed-queries.js
+
+# Ingest test data via the UI (Graph Ingest Handler -> blast mode with test-events.json)
+# Or ingest a real Notion page:
+NOTION_API_KEY=... node connector/ingest-graph.js --page-id <notion_id>
+
+# Run evals via UI (Run Batch on graph-eval-agent.eval-page) or CLI:
+node connector/run-evals.js
+```
+
+Graph schema: `artifact`, `work`, `operator`, `eval` nodes with relationship edges (`ADDED`, `CHANGED`, `OWNS`, `HAS_EVAL`, etc.). See `example/wiki-graph/README.md` for full details.
+
+---
+
+## Testing
+
+```bash
+npm test                    # Unit tests (Vitest)
+npm run test:watch          # Watch mode
+npm run test:coverage       # Coverage report
+npm run test:integration    # Integration tests (real API calls)
+npm run test:e2e            # E2E tests (full deploy cycle, slow)
+npm run lint                # ESLint
+npm run build               # Build TypeScript
+```
+
+| Suite | When to use | Auth needed | Speed |
+|-|-|-|-|
+| Unit (`npm test`) | Always; validates logic without external deps | No | Fast |
+| Integration | Verify API client against dev environment | Yes (`.env`) | Medium |
+| E2E | Full lifecycle: deploy, test, delete | Yes (`.env` + `CEF_E2E_AGENT_SERVICE_PUB_KEY`) | Slow (120s timeout) |
+
+For integration/E2E: copy `example/.env.example` to `.env` and set credentials.
+
+---
+
+## Troubleshooting
+
+**"Missing CEF_AUTH_TOKEN" on deploy**
+Create a `.env` file next to your `cef.config.yaml` with auth tokens. See [Environment Variables](#environment-variables).
+
+**Stale IDs after copying config between services**
+If you copied a `cef.config.yaml` from another service, the embedded IDs point to the wrong service. Run `cef deploy` and accept the prompt to clear all IDs, or manually remove all `*Id` fields from the config.
+
+**"require is not defined" or "import is not supported" in agent code**
+Agent handlers run in a V8 sandbox with no module system. All code must be self-contained in a single file with no imports. See [Agent Sandbox](#agent-sandbox).
+
+**Cubby data disappears after restarting dev server**
+By default, `.cef-dev/` is deleted on shutdown. Use `cef dev --persist` to keep data between runs.
+
+**"Not available in local dev mode" for raft queries**
+Raft queries are not emulated locally. Test rafts by deploying to the platform and using `cef playground test`.
+
+**Agent code changes not taking effect**
+The dev server hot-reloads on file save. Check that the file path in `cef.config.yaml` matches the file you are editing. In-flight executions continue with old code; only new calls use the updated version.
+
+**Deploy fails with 404 on update**
+The entity ID in config points to something that no longer exists on the orchestrator. Clear the ID field for that entity and re-run `cef deploy` to create it fresh.
+
+**Non-interactive terminal and delete/deploy prompts**
+Some operations prompt for confirmation. In CI or non-interactive environments, use `--force` to skip prompts.