@exellix/exellix-jobs 1.6.1

package/docs/specs.md

@@ -0,0 +1,305 @@
+# `@exellix/exellix-jobs` — detailed specification (draft)
+
+**Status:** specification only — package **does not exist** yet.  
+**Audience:** implementers of the new worker package and maintainers of **`@exellix/exellix-runtime`**, **`@exellix/graph-engine`**, **`@x12i/catalox`**.  
+**Supersedes as normative worker doc:** the narrative in [`docs/execution-matrix-worker-service.md`](../execution-matrix-worker-service.md) once this package ships (that file should become a short pointer).
+
+**Hard rule:** **`@exellix/exellix-jobs`** must **not** import **`mongodb`** or **`@x12i/xronox-store`**. All matrix **data** access goes through **`createExellixMatrixDataTier`** on **`@exellix/exellix-runtime` ≥ 3.1**. **Metadata** (catalog listing, optional graph JSON from Catalox) uses **`@x12i/catalox`** where applicable.
+
+---
+
+## 1. Package identity
+
+| Item | Recommendation |
+|------|----------------|
+| **npm name** | `@exellix/exellix-jobs` (or `@exellix/matrix-worker` — pick one and keep it stable). |
+| **Type** | `type: "module"` Node package; **no** browser bundle required for v1. |
+| **Entrypoints** | `dist/index.js` — programmatic API (`createMatrixWorker`, `startMatrixWorkerHttp`, …); optional `dist/cli.js` for `exellix-jobs serve`. |
+| **Node** | Align with **`@exellix/exellix-runtime`** (e.g. Node ≥ 20). |
+
+---
+
+## 2. Goals
+
+1. Run a **long-lived process** (or Kubernetes Deployment) that **claims and executes** execution-matrix graph work using **`@exellix/exellix-runtime`** orchestrator + run-loop APIs.
+2. Enforce **graph-level** operational policy: only **Published** graphs in **Ready** or **Running** (per [`assertGraphSchedulable`](../src/execution-matrix/graph-operational-lifecycle.ts)) participate in claims.
+3. Support **configurable parallelism** (`maxConcurrentClaims`, optional `perGraphConcurrency`).
+4. Support **graceful drain**: stop new claims without aborting in-flight **`executeGraph`**.
+5. Expose **operator HTTP** (health, graph PATCH, worker status, Prometheus metrics) with **auth left to host middleware** or built-in minimal API key (product choice).
+6. Emit **structured metrics** for SRE dashboards (see §10).
+
+---
+
+## 3. Non-goals (v1)
+
+- **Not** re-implementing **`ExecutionMatrixRuntime`**, **`ExellixConfigStore`**, or **`runMatrixCycle`** — **import** from **`@exellix/exellix-runtime`**.
+- **Not** defining new row/config **document shapes** — types live in **`@exellix/exellix-runtime`** `contracts.ts`.
+- **Not** replacing **BFF** read paths — read-only dashboards can keep using **`fetchGraphOperationalDashboardSlice`** from a BFF that holds read-only credentials; **jobs** focuses on **write/claim/execute** and **operator control plane**.
+- **Not** implementing **Catalox** itself — only **clients** of **`@x12i/catalox`** if graph models are catalog-backed.
+
+---
+
+## 4. Architectural boundaries
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│  @exellix/exellix-jobs                                       │
+│  • process supervision, HTTP, metrics, concurrency limits    │
+│  • builds ProcessMatrixClaimDeps + RunMatrixCycleDeps        │
+└───────────────┬─────────────────────────────┬───────────────┘
+                │                             │
+                ▼                             ▼
+┌────────────────────────────────────────────────────────────┐
+│ @exellix/exellix-runtime (≥ 3.1)                            │
+│ • createExellixMatrixDataTier • ExecutionMatrixRuntime      │
+│ • ExellixConfigStore • runMatrixCycle / runMatrixContinuously│
+│ • orchestrator, claim pause                                  │
+└───────────────┬────────────────────────────────────────────┘
+                │
+                ▼
+┌───────────────────────────┐     ┌──────────────────────────┐
+│ @exellix/graph-engine      │     │ @x12i/catalox (optional) │
+│ • executeGraph             │     │ • graph / matrix catalog │
+└────────────────────────────┘     └──────────────────────────┘
+```
+
+**Data flow:** Jobs opens **`createExellixMatrixDataTier`** on runtime → obtains collection handles → passes them into worker **`createMatrixWorker`** (which builds **`ExecutionMatrixRuntime`** + **`ExellixConfigStore`** internally). Jobs then drives **`runMatrixContinuously`** via **`ProcessMatrixClaimDeps`**.
+
+---
+
+## 5. Dependencies (required)
+
+| Package | Semver policy | Usage |
+|---------|----------------|--------|
+| **`@exellix/exellix-runtime`** | **`^3.1.0`** | `createExellixMatrixDataTier`, `ExecutionMatrixRuntime`, `createExecutionMatrixRuntime`, `ExellixConfigStore`, `createExellixConfigStore`, `runMatrixCycle`, `runMatrixContinuously`, `processMatrixGraphBatch`, `processNextMatrixClaim`, `createExecutionMatrixClaimPauseController`, `buildExellixRuntimeRealLivenessPayload`, `runExellixRuntimeConnectivityProbe`, types from `contracts` |
+| **`@exellix/graph-engine`** | **≥ 5.16** (via runtime) | **`executeGraph`** — host supplies executor; record on **`runtime.input`** only (see §6.1) |
+| **`@x12i/catalox`** | Optional | Only if **`resolveGraphModel`** loads graph JSON from Firestore catalogs |
+| **`@x12i/env`** | Optional | `initConfig` for `.env` in dev |
+
+**Forbidden direct deps:** `mongodb`, `@x12i/xronox-store`.
+
+---
+
+## 6. Bootstrap sequence (normative)
+
+Normative bootstrap (runtime **3.1+**):
+
+1. **`initConfig`** (optional) — load `.env` for local dev.
+2. **`const matrixTier = await createExellixMatrixDataTier({ … })`** from **`@exellix/exellix-runtime`** — `await matrixTier.init()` when defined.
+3. **Extract handles:** `rows`, `failures`, `snapshots`, `matrices`, `graphs` (or facades that implement the same method contract **`@exellix/exellix-runtime`** expects — see `runtime-store.ts` / `config-store.ts`).
+4. **`const runtime = createExecutionMatrixRuntime(rows, failures, { snapshots, serializeClaims: true })`**.
+5. **`const configStore = createExellixConfigStore({ matrices, graphs })`**.
+6. **Build graph executor** — host function wrapping **`executeGraph`** with any adapter (**`createMatrixExecuteGraphAdapter`** from runtime).
+7. **Build `ProcessMatrixClaimDeps`:**  
+   - `runtime`, `executeGraph`, `resolveGraphModel` (and/or `graphModel`),  
+   - `resolveGraphSchedulability: (graphId) => configStore.graphs.get(graphId)`,  
+   - `graphSchedulabilityRequireConfig` per deploy policy,  
+   - `shouldClaimNext` from **`createExecutionMatrixClaimPauseController`** or custom,  
+   - optional `matrixActivix`, `matrixLogxer`, `executeOverrides`, `runtimeScope`, `sourceResolver` for **`runMatrixCycle`**.
+8. **Start supervisor loop** (§8) or **HTTP server** (§9) that drives **`runMatrixContinuously`** / **`runMatrixCycle`** per assignment (§7).
+
+**Shutdown:** `signal` abort → await in-flight graph tasks → `await matrixTier.close()` → `process.exit(0)`.
+
+### 6.1 Graph execution input (record placement)
+
+Normative peer contract (flat `runtime.input`, no top-level `input`, no `input.raw`): [`../../exellix-runtime/docs/graph-execution-record-placement.md`](../../exellix-runtime/docs/graph-execution-record-placement.md).
+
+Worker-specific rules:
+
+1. **`@exellix/exellix-jobs` never constructs `ExecuteGraphInput`** — it imports `runMatrixCycle` / `processMatrixGraphBatch` from **`@exellix/exellix-runtime`** and wraps the host `executeGraph` only for concurrency and metrics ([`wrap-execute-graph.ts`](../src/worker/wrap-execute-graph.ts)).
+2. **Materialize** — runtime persists the work-unit on **`ExecutionMatrixRuntimeRecord.input`**. The host must ensure **`sourceResolver`** / **`inputRows`** produce **flat** field names aligned with graph `graphEntry.inputs` (e.g. `subnetId`, `question` on the row object — not nested `raw` wrappers).
+3. **Execute** — orchestrator **`executeMatrixGraphForClaim`** sets **`runtime.input = record.input`**, seeds **`executionMemory`** via `buildExecutionSeedFromGraphEntry`, then calls **`executeGraph({ model, runtime })`**.
+4. **Host `executeGraph`** — must accept `{ model, runtime }` only; use **`createMatrixExecuteGraphAdapter`** when layering defaults without overriding orchestrator-owned `jobId` / `job` / `input` / `executionMemory`.
+5. **Dependencies** — **`@exellix/graph-engine` ≥ 5.16** (no implicit `job.raw` → `execution.input.raw` promotion). Align versions with **`@exellix/exellix-runtime`**.
+
+---
+
+## 7. Assignment set — which work to run (algorithm)
+
+**Inputs:** `configStore`, optional cache TTL.
+
+**Steps (each supervisor tick or on a timer):**
+
+1. **`listGraphConfigs`** (or `graphs.list({ limit })`) — filter in memory or via future server-side filter: **`publicationStatus === 'Published'`** and **`operationalState` in `('Ready','Running')`** (exact set is a **product flag** on the worker).
+2. For each **candidate `graphId`**:  
+   - **`configStore.graphs.get(graphId)`** — re-fetch if using cache; ensure version freshness for operational flips.
+3. **Discover matrices:** **`configStore.matrices.list({ limit })`** (paginate until done) — for each matrix, parse **`payload.graphs`**; if array contains **`graphId`**, enqueue tuple **`(matrixCatalogId, graphId)`**.
+4. For each tuple (and optional **`matrixRunId`** scope from env or config):  
+   - **`configStore.matrices.getEffectiveConfig(matrixCatalogId)`** or equivalent **`getEffectiveConfig`** on the surface **`@exellix/exellix-runtime`** exposes (`config-store.ts`).  
+   - Load **graph records** for all **`payload.graphs`** via **`configStore.graphs.get`** in parallel (bounded concurrency).
+5. **Invoke** **`runMatrixCycle(deps, { matrixCatalogId, matrixItemId, matrixRunId, payload, graphRecords, … })`**  
+   - **`matrixItemId`**: worker config (per-deployment catalog item id) or derived convention documented in worker README.  
+   - For **`event`** matrices without pushed **`inputRows`**, cycle may be **`emptyPass`** — worker should not spin hot; use **`pollWhenEmptySeconds`** from payload / `runMatrixContinuously`.
+
+**Re-fetch rule:** Before **`processMatrixGraphBatch`** or **`processNextMatrixClaim`**, re-run **`resolveGraphSchedulability(graphId)`** so **`operationalState`** changes stop **new** claims quickly; in-flight **`executeGraph`** completes normally.
+
+---
+
+## 8. Concurrency model (normative)
+
+| Mechanism | Specification |
+|-----------|----------------|
+| **Global cap** | **`maxConcurrentClaims`** (integer ≥ 1). Implement with **`p-limit`** (or equivalent) around **`executeMatrixGraphForClaim`** entry points invoked from **`processMatrixGraphBatch`** / **`processNextMatrixClaim`**, **or** run batch with **`limit: 1`** and outer parallelism — document chosen pattern. |
+| **Per-graph cap** | Optional **`Record<graphId, number>`** — secondary limiter inside the global pool to avoid one graph starving others. |
+| **Pause / drain** | **`shouldClaimNext`** returns **`false`** → orchestrator stops scheduling **new** claims; **never** abort **`executeGraph`** mid-flight. Prefer **`createExecutionMatrixClaimPauseController`** from runtime for **`pause()` / `resume()`** wired to HTTP or file signal. |
+| **Continuous loop** | **`runMatrixContinuously`** — one loop per **`(matrixCatalogId, matrixItemId, matrixRunId?)`** worker instance; run **N** parallel worker tasks for **N** active matrices if needed. |
+| **Abort** | **`AbortSignal`** on **`runMatrixContinuously`** — cooperative shutdown between cycles and during sleep. |
+
+---
+
+## 9. Matrix kinds vs worker behavior
+
+| `dataSource.kind` | Worker responsibility |
+|-------------------|-------------------------|
+| **`record`** / **`storage`** | **`runMatrixCycle`** pulls source ids via **`MatrixSourceResolver`**, diffs, materializes — worker must supply **`sourceResolver`** in **`RunMatrixCycleDeps`**. |
+| **`query-snapshot`** | Cycle runs snapshot transform + persist snapshot + materialize — ensure **VM / sandbox** policy matches host security baseline. |
+| **`event`** | **No** self-scan — **`emptyPass`** unless host passes **`inputRows`** into the cycle options. Worker either **skips** these matrices in autonomous mode or subscribes to an **ingress** that calls **`runMatrixCycle`** with **`inputRows`** (out of scope for minimal v1 — document “event requires ingress”). |
+
+---
+
+## 10. HTTP API (suggested — OpenAPI-friendly)
+
+Base path prefix configurable (e.g. **`JOBS_HTTP_PREFIX=/v1`**).
+
+### 10.1 Health
+
+| Method | Path | Behavior |
+|--------|------|----------|
+| **GET** | `/health` | **Liveness:** process up; body from **`buildExellixRuntimeRealLivenessPayload()`** — **no** dependency I/O required. |
+| **GET** | `/health?probe=dependencies` | **Readiness-style:** call **`runExellixRuntimeConnectivityProbe`** with options allowed **after** [`missing.md`](./missing.md) §1.4 (XMemory-only execution/config probes). Until then, probe only **`includeXmemoryEnv`**, **`includeActivixSchedulerProbe`**, etc., without xronox-shaped stores. |
+
+### 10.2 Graph operational control
+
+| Method | Path | Body | Behavior |
+|--------|------|------|----------|
+| **PATCH** | `/graphs/:graphId/operational` | `{ operationalState, expectedVersion }` | **`configStore.graphs.patchOperational`** — **401/403** without auth. |
+| **PATCH** | `/graphs/:graphId/publication` | `{ publicationStatus, expectedVersion }` | Same pattern if split from operational. |
+
+### 10.3 Discovery
+
+| Method | Path | Response (informative) |
+|--------|------|------------------------|
+| **GET** | `/graphs` | List graphs with **`publicationStatus`**, **`operationalState`**, **`version`**, derived **`matrixCatalogIds`** referencing each **`graphId`**. |
+
+### 10.4 Worker introspection
+
+| Method | Path | Response (informative) |
+|--------|------|------------------------|
+| **GET** | `/workers/status` | **`assignedGraphIds`**, **`activeJobs`**, **`maxConcurrency`**, per-graph in-flight counts, last cycle timestamps, optional **`matrixCatalogId`** filter. |
+
+### 10.5 Metrics
+
+| Method | Path | Content-Type |
+|--------|------|----------------|
+| **GET** | `/metrics` | `text/plain; version=0.0.4` — Prometheus exposition format (§11). |
+
+**Auth:** All mutating routes **must** require authentication (API key header, mTLS, or reverse-proxy auth — specify in deploy doc).
+
+---
+
+## 11. Metrics (minimum set)
+
+Counters / histograms (names are **suggestions** — keep stable after v1):
+
+| Metric | Type | Labels | When incremented |
+|--------|------|--------|------------------|
+| `matrix_claims_started_total` | counter | `graph_id`, `matrix_catalog_id` | After successful **claim** before **`executeGraph`**. |
+| `matrix_claims_completed_total` | counter | `graph_id`, `status` | After **`executeGraph`** terminal success (graph-engine status). |
+| `matrix_claims_failed_total` | counter | `graph_id` | Executor throw or persisted failure path. |
+| `matrix_rows_materialized_total` | counter | `matrix_catalog_id` | After **`insertRuntimeRow`** from materializer in a cycle. |
+| `matrix_worker_idle_seconds_total` | counter | `matrix_catalog_id` | Time spent in **`emptyPass`** cycles (or histogram of idle duration per cycle). |
+| `matrix_worker_cycles_total` | counter | `matrix_catalog_id`, `empty` (`true`/`false`)` | Each **`runMatrixCycle`** completion. |
+
+Optional: **`histogram`** for **`executeGraph`** duration per **`graph_id`**.
+
+---
+
+## 12. Configuration (environment variables)
+
+| Variable | Purpose | Default |
+|----------|---------|---------|
+| **`MONGO_URI`** | Passed only into **XMemory** factories — jobs does not parse for its own driver. | — |
+| **`execution_db`** | Operational DB name | `exellix-runtime` |
+| **`config_db`** | Config DB name | `exellix` |
+| **`PORT`** | HTTP listen port | e.g. `8080` |
+| **`MATRIX_WORKER_ID`** | Identity in health / scheduler lane | optional |
+| **`EXELLIX_JOBS_MAX_CONCURRENT_CLAIMS`** | Global parallelism | `4` |
+| **`EXELLIX_JOBS_GRAPH_SCHEDULABILITY_STRICT`** | Maps to **`graphSchedulabilityRequireConfig`** | `false` |
+| **`EXELLIX_JOBS_HTTP_ENABLED`** | Start HTTP server | `true` |
+| **`EXELLIX_JOBS_ACTIVIX_SCHEDULER_PROBE`** | Include Activix scheduler lane on **`/health?probe=dependencies`** | `false` |
+
+(Extend as needed; document every env key in **jobs** README.)
+
+---
+
+## 13. Activix (optional)
+
+If the product keeps **Activix** boundary activities:
+
+- Pass **`matrixActivix`** on **`ProcessMatrixClaimDeps`** per **`orchestrator.ts`**.
+- Health lane: reuse **`testActivixMongoConnection`** / **`getActivixActivityPersistenceSnapshotInMongo`** from **`@x12i/activix`** as **`@exellix/exellix-runtime`** already does in **`runtime-connectivity-health.ts`** — **jobs** calls the same probe builder with **`includeActivixSchedulerProbe: true`** when configured.
+
+**Jobs** does **not** fork Activix behavior into runtime.
+
+---
+
+## 14. CLI split (decision framework)
+
+| Option | Pros | Cons |
+|--------|------|------|
+| **Keep `catalox:inventory` in `@exellix/exellix-runtime`** | Single devDependency for library consumers | Mixes library + operator CLI |
+| **Move to `@exellix/exellix-jobs`** | Operator tooling colocated with worker | Runtime tarball has no inventory script |
+| **`@exellix/cli` meta-package** | Clear “tools” boundary | Extra package to release |
+
+**Spec requirement:** pick one in **jobs** v1 README and link from **`@exellix/exellix-runtime`** `package.json` `scripts` if moved.
+
+---
+
+## 15. High availability
+
+- **Multiple replicas** claiming the **same** **`(matrixCatalogId, graphId)`** without coordination **will** double-execute unless an external leader elector or partition strategy exists — **document:** default deployment is **one active worker per matrix scope** OR use **`serializeClaims: true`** (runtime default) + external single-claimer guarantee.
+- **Recommended:** one Deployment per **matrix family** or global worker with **fair** assignment — product choice.
+
+---
+
+## 16. Testing strategy
+
+| Layer | Tests |
+|-------|--------|
+| **Unit** | Mock **`ExecutionMatrixRuntime`** / **`ExellixConfigStore`** with in-memory fakes from **`@exellix/exellix-runtime`** `memory-store` patterns (if exported) or local test doubles. |
+| **Integration** | Testcontainers or shared dev cluster — **only** through **XMemory** matrix tier (once §1.1 exists). |
+| **Contract** | HTTP golden files for **`/health`** JSON shape. |
+
+---
+
+## 17. Versioning
+
+- **`@exellix/exellix-jobs` major** tracks **`@exellix/exellix-runtime`** breaking changes to **`ProcessMatrixClaimDeps`**, **`RunMatrixCycleDeps`**, or factory signatures.
+- **Changelog** must call out required **`@exellix/exellix-runtime`** minimum (e.g. **`^3.1.0`** for **`createExellixMatrixDataTier`**).
+
+---
+
+## 18. Acceptance checklist (jobs package)
+
+- [x] No **`mongodb`** / **`@x12i/xronox-store`** in **jobs** `package.json` or source.
+- [x] Matrix persistence opened via **`createExellixMatrixDataTier`** on **`@exellix/exellix-runtime` ≥ 3.1**; **`openExellixMatrixPersistence()`** may fall back to internal xronox factories.
+- [x] **`ProcessMatrixClaimDeps.resolveGraphSchedulability`** wired to **`configStore.graphs.get`**.
+- [x] **`shouldClaimNext`** + graceful shutdown verified.
+- [x] **`/health`** + **`/metrics`** implemented; **`PATCH`** routes auth-protected.
+- [x] Prometheus metrics from §11 exposed.
+- [x] README links **`execution-matrix-worker-service.md`** as **historical** and points here as normative.
+
+---
+
+## 19. Reference — runtime source files
+
+| Concern | Path |
+|---------|------|
+| Claim deps | `src/execution-matrix/orchestrator.ts` — **`ProcessMatrixClaimDeps`** |
+| Loops | `src/execution-matrix/run-loop.ts` — **`runMatrixCycle`**, **`runMatrixContinuously`** |
+| Pause | `src/execution-matrix/execution-matrix-claim-pause.ts` |
+| Graph schedulability | `src/execution-matrix/graph-operational-lifecycle.ts` |
+| Health | `src/execution-matrix/runtime-connectivity-health.ts` |
+| Prior worker narrative | `docs/execution-matrix-worker-service.md` |
+
+When **`missing.md`** §1.1 is implemented, replace any remaining references in this spec to **`createExecutionMatrixRuntimeFromXronox`** with the **XMemory matrix tier** bootstrap (§6).

package/docs/troubleshooting.md

@@ -0,0 +1,252 @@
+# Troubleshooting — `@exellix/exellix-jobs`
+
+Quick fixes for local `exellix-jobs serve`, operator HTTP, and “why isn’t my graph running?”
+
+**Related:** [openapi.yaml](./openapi.yaml) (routes), [README](../README.md), [`@exellix/exellix-runtime` client-integration](https://github.com/exellix/exellix-runtime/blob/main/docs/client-integration.md) (materialize + matrix domain).
+
+---
+
+## 1. Five-minute checklist
+
+| Check | Command / where | Healthy |
+|-------|------------------|---------|
+| Server up | `curl -s http://127.0.0.1:$PORT/v1/health` | `"ok": true` |
+| Mongo wired | Logs: `Successfully connected` to `exellix-runtime` + `exellix` | Both DBs |
+| Graph listed | `curl -s http://127.0.0.1:$PORT/v1/graphs` | Your `graphId` appears |
+| On a matrix | Same response: `matrixCatalogIds` | **Non-empty** array |
+| Runnable state | Graph record in Compass | `Published` + `Ready` or `Running`, **no** `deletedAt` |
+| Work exists | DB `exellix-runtime` → `exellix_execution_matrix_rows` | Rows for your matrix/graph |
+| Real executor | Logs | **Not** `graph-engine-stub.mjs` for production |
+
+`exellix-jobs` **claims and executes** rows; it does **not** create matrix configs or materialize rows (Studio / BFF / script does that).
+
+---
+
+## 2. Operator HTTP
+
+Base URL: `http://127.0.0.1:<PORT><JOBS_HTTP_PREFIX>` (default prefix `/v1`).
+
+### `404` — `Route PATCH:/v1/graphs/{graphId} not found`
+
+**Symptom:** Fastify message includes `Route PATCH:/v1/graphs/... not found`.
+
+**Cause:** Wrong path. There is **no** `PATCH /v1/graphs/:graphId`.
+
+**Fix:** Use the suffix:
+
+| Intent | Method | Path |
+|--------|--------|------|
+| Operational state | `PATCH` | `/v1/graphs/{graphId}/operational` |
+| Publication status | `PATCH` | `/v1/graphs/{graphId}/publication` |
+| Pause claims | `POST` | `/v1/workers/pause` |
+| Resume claims | `POST` | `/v1/workers/resume` |
+
+```bash
+curl -X PATCH "http://127.0.0.1:5173/v1/graphs/graph-qcrbz6t/operational" \
+  -H "x-exellix-jobs-api-key: $EXELLIX_JOBS_API_KEY" \
+  -H "content-type: application/json" \
+  -d '{"expectedVersion":1,"operationalState":"Running"}'
+```
+
+Reads (`GET /health`, `/graphs`, `/workers/status`, `/metrics`) need **no** API key.
+
+### `501` — mutations disabled
+
+**Symptom:** `Mutations are disabled until EXELLIX_JOBS_API_KEY is configured`.
+
+**Fix:** Add to `.env` and restart serve:
+
+```bash
+EXELLIX_JOBS_API_KEY=<your-secret>
+```
+
+Send the same value as `x-exellix-jobs-api-key: …` or `Authorization: Bearer …`.
+
+### `401` Unauthorized
+
+**Cause:** Header missing or does not match `EXELLIX_JOBS_API_KEY`.
+
+**Fix:** Match `.env` exactly; restart after changing `.env`.
+
+### `409` / concurrency errors on PATCH
+
+**Cause:** `expectedVersion` in the body does not match the document’s `version` in Mongo.
+
+**Fix:**
+
+```bash
+curl -s "http://127.0.0.1:5173/v1/graphs"
+```
+
+Use the `version` for your `graphId` in the next PATCH (increment after each successful patch).
+
+### `ERR_MODULE_NOT_FOUND` for graph bootstrap
+
+**Symptom:** Cannot find `C:/path/to/your/graph-engine-bootstrap.mjs`.
+
+**Cause:** Placeholder path from docs was exported literally.
+
+**Fix:** Unset `EXELLIX_JOBS_GRAPH_ENGINE_BOOTSTRAP` (uses stub), or point at a real file:
+
+```bash
+export EXELLIX_JOBS_GRAPH_ENGINE_BOOTSTRAP="C:/prometheus/exellix/exellix-jobs/examples/graph-engine-stub.mjs"
+```
+
+---
+
+## 3. Mongo — two databases
+
+| DB (`env`) | Default name | Collections | Role |
+|------------|--------------|-------------|------|
+| `config_db` | `exellix` | `exellix_graph_configs`, `exellix_matrix_configs` | **What** to run (graphs + matrix definition) |
+| `execution_db` | `exellix-runtime` | `exellix_execution_matrix_rows`, failures, snapshots | **Work units** to claim |
+
+BFF, Studio, jobs, and runtime smoke tests must share the same `MONGO_URI`, `execution_db`, and `config_db`.
+
+### Which id to use in Compass vs HTTP
+
+| Field | Meaning | Example |
+|-------|---------|---------|
+| `_id` | Mongo ObjectId | Ignore for APIs |
+| `graphId` | Graph config primary key | `graph-qcrbz6t` — use in PATCH paths |
+| `matrixCatalogId` | Matrix config primary key | `live_matrix_…` — use in materialize / worker status |
+| `payload.graphId` | Must equal top-level `graphId` | Same as `graphId` |
+
+**`matrixCatalogId`** is the matrix’s catalog id (one batch definition listing `payload.graphs`). It is **not** the graph id. Find it in `exellix_matrix_configs.matrixCatalogId`.
+
+---
+
+## 4. Graph config (`exellix_graph_configs`)
+
+### Schedulable state (worker will consider the graph)
+
+| Field | Required value |
+|-------|----------------|
+| `publicationStatus` | `Published` |
+| `operationalState` | `Ready` or `Running` |
+| `deletedAt` | **absent** |
+| `tombstoneId` | **absent** |
+
+Defaults if omitted: `Draft` + `NotOperational` → **not** schedulable.
+
+### Healthy example (`graph-qcrbz6t`)
+
+```json
+{
+  "graphId": "graph-qcrbz6t",
+  "version": 1,
+  "payload": {
+    "graphId": "graph-qcrbz6t",
+    "mappedInput": [{ "tag": "two" }],
+    "execution": { "priority": 20 }
+  },
+  "publicationStatus": "Published",
+  "operationalState": "Running",
+  "createdAt": "2026-05-17T06:11:26.354Z",
+  "updatedAt": "2026-05-17T06:11:28.850Z"
+}
+```
+
+### Common bad records
+
+| Symptom | What’s wrong | Fix |
+|---------|--------------|-----|
+| `Published` + `Running` but worker ignores | `deletedAt` / `tombstoneId` set | Create a **new** graph config from Studio; don’t reuse tombstone |
+| Validation / weird PATCH | `graphId` ≠ `payload.graphId` | Align both to the same string |
+| PATCH 404 (correct URL) | Graph row missing | Create config or fix `graphId` key |
+| `matrixCatalogIds: []` in `/v1/graphs` | Graph not on any matrix | Add `graphId` to `exellix_matrix_configs.payload.graphs` |
+
+---
+
+## 5. Matrix config (`exellix_matrix_configs`)
+
+Minimal shape:
+
+```json
+{
+  "matrixCatalogId": "my-matrix-1",
+  "version": 1,
+  "payload": {
+    "input": {},
+    "graphs": ["graph-qcrbz6t"]
+  }
+}
+```
+
+`payload.graphs` must be a **non-empty** list of existing `graphId` values.
+
+**Linking an existing graph:** `matrices.update` with `expectedVersion` and `graphs` including your id — via Studio, BFF, or a small runtime script (see [client-integration §6](https://github.com/exellix/exellix-runtime/blob/main/docs/client-integration.md) in exellix-runtime).
+
+---
+
+## 6. Worker idle / nothing executes
+
+| Cause | How to confirm | Fix |
+|-------|----------------|-----|
+| No rows | `exellix_execution_matrix_rows` empty | **Materialize** + `insertRuntimeRow` (not jobs) |
+| Graph not schedulable | Draft / NotOperational / deleted | PATCH publication + operational, or Studio |
+| Graph not on matrix | `/v1/graphs` → `matrixCatalogIds: []` | Update matrix `payload.graphs` |
+| Stub executor | Log: `graph-engine-stub.mjs` | Set real `EXELLIX_JOBS_GRAPH_ENGINE_BOOTSTRAP` |
+| Worker paused | POST pause was called | `POST /v1/workers/resume` with API key |
+| Wrong Mongo | Different `MONGO_URI` / db names | Align `.env` across Studio, jobs, BFF |
+
+```bash
+curl -s "http://127.0.0.1:5173/v1/workers/status"
+curl -s "http://127.0.0.1:5173/v1/health?probe=dependencies"
+```
+
+---
+
+## 7. “Server runs” vs “runs for real”
+
+| Layer | Real? | Signal |
+|-------|-------|--------|
+| HTTP + Mongo | Yes | `listening on port …`, Mongo connect logs |
+| Config PATCH | Yes | Correct `/operational` / `/publication` paths |
+| Matrix execution | Only if rows + schedulable graph + real `executeGraph` | Rows in `exellix_execution_matrix_rows`; not stub bootstrap |
+
+Default bootstrap uses **`examples/graph-engine-stub.mjs`** — always returns fake `completed`. That is intentional for smoke tests.
+
+---
+
+## 8. Useful commands (no `jq`)
+
+```bash
+# Health
+curl -s "http://127.0.0.1:5173/v1/health"
+
+# One graph from list
+curl -s "http://127.0.0.1:5173/v1/graphs" | node -e "
+const g=JSON.parse(require('fs').readFileSync(0,'utf8'));
+console.log(g.find(x=>x.graphId==='graph-qcrbz6t') ?? 'not found');
+"
+
+# Worker status (optional matrix filter)
+curl -s "http://127.0.0.1:5173/v1/workers/status?matrixCatalogId=my-matrix-1"
+```
+
+---
+
+## 9. Error message → meaning
+
+| Message | Meaning |
+|---------|---------|
+| `Route PATCH:/v1/graphs/... not found` | Missing `/operational` or `/publication` |
+| `Route … not found` (other) | Wrong prefix/port; check `PORT` and `JOBS_HTTP_PREFIX` |
+| `Mutations are disabled until EXELLIX_JOBS_API_KEY` | Set key in `.env`, restart |
+| `Unauthorized` | Wrong or missing API key header |
+| `expectedVersion (number) is required` | PATCH body needs numeric `expectedVersion` |
+| `ConfigStoreConcurrencyError` / version mismatch | Stale `expectedVersion` |
+| `ConfigStoreNotFoundError` | No graph row for that `graphId` |
+| `GraphNotSchedulableError` (logs) | Not Published or not Ready/Running |
+| `using default graph-engine bootstrap … stub` | Not running real graphs |
+
+---
+
+## 10. Still stuck?
+
+1. Capture: `curl /v1/health`, `/v1/graphs`, `/v1/workers/status`.
+2. Compass screenshots: one `exellix_graph_configs` doc, one `exellix_matrix_configs` doc, count of `exellix_execution_matrix_rows`.
+3. Confirm peer: `@exellix/exellix-runtime` **^3.1.0** (`buildId` in `/health`).
+
+For materialize and Catalox wiring, use **exellix-runtime** docs — jobs only operates on data that already exists in Mongo.

package/examples/graph-engine-stub.mjs

@@ -0,0 +1,40 @@
+/**
+ * Minimal graph-engine bootstrap for local smoke tests.
+ * Replace with a real module in production (Catalox + graph-engine wiring).
+ *
+ * Contract: executeGraph({ model, runtime }) — record on runtime.input (flat keys).
+ * See ../exellix-runtime/docs/graph-execution-record-placement.md
+ */
+export default async function graphEngineBootstrap() {
+  return {
+    executeGraph: async (input) => {
+      if ('input' in input && input.input !== undefined) {
+        throw new Error(
+          'graph-engine-stub: invalid request — use { model, runtime } only; put the record on runtime.input',
+        );
+      }
+      const record = input.runtime?.input;
+      const matrixRowId = input.runtime?.job?.matrixRowId;
+      if (matrixRowId && record && typeof record === 'object') {
+        // Smoke: matrix orchestrator forwarded row.input → runtime.input (flat fields).
+        const flat = record;
+        if (flat.subnetId !== undefined || flat.question !== undefined) {
+          console.error('@exellix/exellix-jobs stub: runtime.input keys', Object.keys(flat));
+        }
+      }
+      return {
+        status: 'completed',
+        jobId: typeof input.runtime?.jobId === 'string' ? input.runtime.jobId : 'stub-job',
+        taskId: 'stub-task',
+        graphId: typeof input.model?.id === 'string' ? input.model.id : 'stub-graph',
+        outputsByNodeId: {},
+        stepsResponses: [],
+        engineSnapshot: {},
+      };
+    },
+    resolveGraphModel: async ({ graphId }) => ({
+      id: graphId,
+      nodes: [],
+    }),
+  };
+}

package/examples/host-bootstrap.mjs

@@ -0,0 +1,74 @@
+/**
+ * Example host bootstrap for `exellix-jobs serve`.
+ *
+ * Copy or symlink this file and set:
+ *   EXELLIX_JOBS_HOST_BOOTSTRAP=/absolute/path/to/host-bootstrap.mjs
+ *
+ * Requires:
+ *   MONGO_URI
+ *   EXELLIX_JOBS_GRAPH_ENGINE_BOOTSTRAP — ESM module default-exporting async () => ({ executeGraph, resolveGraphModel?, sourceResolver? })
+ *
+ * Record placement: ../exellix-runtime/docs/graph-execution-record-placement.md
+ * Matrix rows store flat fields on row.input; orchestrator sets runtime.input = record.input at claim.
+ */
+import { fileURLToPath, pathToFileURL } from 'node:url';
+import { dirname, join, resolve } from 'node:path';
+import { createExellixMatrixDataTier } from '@exellix/exellix-runtime';
+import { loadJobsEnv } from '@exellix/exellix-jobs';
+
+const examplesDir = dirname(fileURLToPath(import.meta.url));
+const defaultGraphBootstrap = join(examplesDir, 'graph-engine-stub.mjs');
+
+async function loadGraphEngineBootstrap() {
+  const path = process.env.EXELLIX_JOBS_GRAPH_ENGINE_BOOTSTRAP ?? defaultGraphBootstrap;
+  if (!process.env.EXELLIX_JOBS_GRAPH_ENGINE_BOOTSTRAP) {
+    console.error(`@exellix/exellix-jobs: using default graph-engine bootstrap ${path}`);
+  }
+  const href = path.startsWith('file:') ? path : pathToFileURL(resolve(path)).href;
+  const mod = await import(href);
+  if (typeof mod.default !== 'function') {
+    throw new Error('Graph-engine bootstrap must default-export an async function.');
+  }
+  return mod.default();
+}
+
+export default async function bootstrap() {
+  const env = loadJobsEnv();
+  const tier = await createExellixMatrixDataTier({
+    mongoUri: process.env.MONGO_URI,
+    executionDb: env.executionDb,
+    configDb: env.configDb,
+  });
+  if (tier.init) {
+    await tier.init();
+  }
+  const persistence = {
+    rows: tier.rows,
+    failures: tier.failures,
+    snapshots: tier.snapshots,
+    matrices: tier.matrices,
+    graphs: tier.graphs,
+    close: () => tier.close(),
+    ...(tier.probe ? { probe: tier.probe } : {}),
+  };
+  const graph = await loadGraphEngineBootstrap();
+  // Production: wrap graph.executeGraph with createMatrixExecuteGraphAdapter from
+  // @exellix/exellix-runtime (see docs/execution-matrix-runtime.md) so modelConfig defaults
+  // merge under orchestrator-owned jobId / job / input / executionMemory.
+  return {
+    workerOptions: {
+      persistence,
+      executeGraph: graph.executeGraph,
+      ...(graph.resolveGraphModel ? { resolveGraphModel: graph.resolveGraphModel } : {}),
+      ...(graph.sourceResolver ? { sourceResolver: graph.sourceResolver } : {}),
+      matrixWorkerId: env.matrixWorkerId,
+      maxConcurrentClaims: env.maxConcurrentClaims,
+      graphSchedulabilityRequireConfig: env.graphSchedulabilityStrict,
+    },
+  };
+}
+
+// When run directly: `node examples/host-bootstrap.mjs` prints resolved paths for local dev.
+if (process.argv[1] && fileURLToPath(import.meta.url) === resolve(process.argv[1])) {
+  console.error('host-bootstrap module path:', fileURLToPath(import.meta.url));
+}