@terreno/mcp 57.10.1 → 57.11.0

package/dist/docs/upgrades/57.11.0.md

@@ -0,0 +1,80 @@
+# Upgrading to 57.11.0
+
+**Action required:** Yes
+**Affected packages:** @terreno/ai, @terreno/api
+
+## Summary
+
+`InProcessRunner` now runs up to eight harness tasks at once. List filters on a field that is not in `queryFields` return 400 before request validation, so that filter can no longer be stripped and ignored. Slack channel helpers and syncdb query windows are opt-in.
+
+## Breaking changes
+
+### InProcessRunner runs tasks in parallel
+
+**What changed**
+
+`new InProcessRunner()` runs up to 8 claimed tasks at once. Each task has its own lease. `stop()` waits for every in-flight task. Turns inside one conversation stay serial. Model calls run in parallel, so provider rate limits, cost, and stream-counter transaction retries all go up.
+
+**Why**
+
+A single slow model call was blocking every other task on that process.
+
+**Migration**
+
+```typescript
+// Before: one claimed task at a time
+const runner = new InProcessRunner();
+
+// After: keep that behavior
+const runner = new InProcessRunner({concurrency: 1});
+
+// Or accept the new default (8)
+const runner = new InProcessRunner();
+```
+
+### List filters outside queryFields are no longer ignored
+
+**What changed**
+
+`GET` list on `modelRouter` checks every filter against `queryFields` before OpenAPI query validation. A field that is not listed returns 400 `query-param-not-allowed`. The body names the field, the model, and `meta.allowedQueryFields`. `@terreno/syncdb` raises `QueryFieldNotAllowedError` for the same response. Pagination params `limit`, `page`, and `sort` are unchanged.
+
+**Why**
+
+Query validation could strip an unknown filter and still return 200, so the client thought the filter applied.
+
+**Migration**
+
+```typescript
+// Before: ?title=ship could be stripped and the list still returned 200
+modelRouter("/todos", Todo, {queryFields: ["completed", "ownerId"]});
+
+// After: add the field, or stop sending it
+modelRouter("/todos", Todo, {queryFields: ["completed", "ownerId", "title"]});
+```
+
+A 400 looks like:
+
+```json
+{
+  "status": 400,
+  "code": "query-param-not-allowed",
+  "title": "Query parameter not allowed",
+  "meta": {"queryParam": "title", "model": "Todo", "allowedQueryFields": ["completed", "ownerId"]}
+}
+```
+
+## Deprecations
+
+None
+
+## New capabilities
+
+- `JobsRunner` from `@terreno/ai/harness/jobsRunner` runs each harness phase as a `@terreno/jobs` job (`terreno.harness.phase`) so more than one instance can share the work. `@terreno/jobs` is an optional peer. `HarnessTask.claims` defaults to 0 and needs no migration. See `docs/how-to/run-harness-on-multiple-instances.md`.
+- `@terreno/api` Slack helpers use the existing `SLACK_BOT_TOKEN`: `createSlackPrivateChannel`, `inviteSlackUsersToChannel`, `postSlackMessage`, and `findSlackChannelByName`. The bot needs `groups:write`, `groups:write.invites`, `channels:write.invites`, `chat:write`, `channels:read`, and `groups:read`. A taken channel name returns 409.
+- `@terreno/syncdb` query windows: set `queryCollections`, then use `client.queryWindows` or `useWindowQuery` (and generated `use{Collection}Window` hooks) for server-filtered, paged lists. `useQuery` and `useEntityIds` accept the same Mongo-style `where`, string sort, and `limit` for local queries. `fullSync: true` adds windows without turning off snapshot sync.
+
+## Verification
+
+- Apps that construct `InProcessRunner` with no options: confirm parallel model calls are acceptable, or pass `{concurrency: 1}`.
+- Hit one list route with a filter field that is not in `queryFields` and confirm 400 `query-param-not-allowed` instead of a 200 that ignored the filter.
+- `bun run compile` and `bun run lint` at the app root.

package/dist/docs/versioned/next/explanation/durable-agent-harness.md

@@ -47,8 +47,11 @@ agents, tools, and conversations (see [The agent loop](#the-agent-loop)), subage
 extensions, hooks, and memos (see [Extensions, hooks, and memos](#extensions-hooks-and-memos)),
 events, waits, and approvals with the admin inbox, the SSE event stream, and version
 pinning. The example backend's `clinic.intakeSummary` runs all of it end to end, including
-a crash drill: [Build a durable workflow](../how-to/build-a-durable-workflow.md). Other
-runners (jobs-backed, concurrent) come in Phase 2.
+a crash drill: [Build a durable workflow](../how-to/build-a-durable-workflow.md).
+
+Phase 2 so far: `InProcessRunner` concurrency (up to 8 tasks at once by
+default) and the `JobsRunner`, which runs each phase as a `@terreno/jobs` job across many
+instances. The Runs admin screen is still to come.
 
 ## Why the audit span shares the checkpoint transaction
 
@@ -237,8 +240,18 @@ tasks. Switching runners needs no data migration.
 
 | Runner | Status | Ownership |
 | --- | --- | --- |
-| `InProcessRunner` | Shipped | One process holds the `HarnessOwner` lease and drains `pending` tasks one at a time; others wait on standby and take over on lease expiry. |
-| `JobsRunner` | Planned (Phase 2) | Each runnable phase becomes a `@terreno/jobs` job. The task lease is the authority. |
+| `InProcessRunner` | Shipped | One process holds the `HarnessOwner` lease and runs up to `concurrency` (default 8) claimed tasks at once, each under its own task lease; others wait on standby and take over on lease expiry. |
+| `JobsRunner` | Shipped | Every instance dispatches runnable tasks as `terreno.harness.phase` jobs; a job claims its task, runs one phase, and hands the task back. The task lease is the authority. |
+
+Why the jobs runner runs **one** phase per job: a phase is the unit that commits, so it is
+the natural unit of work to spread across instances and to bound by a request or job
+timeout. Why the task lease and not the job lock decides who commits: a jobs lock can
+expire while a long phase still runs, and a queue can deliver a job twice. Both are
+harmless here, because the claim is an atomic fenced update and every commit checks the
+lease token. Why the job key carries `claims`: `taskId:phase:attempt` alone repeats when a
+task returns to the same phase without a failure (an event wake or a crash replay), and a
+permanent idempotency key would then swallow the new visit. `claims` changes on every
+claim, so each visit gets one job.
 
 ## Version pinning
 

package/dist/docs/versioned/next/explanation/local-first-data.md

@@ -39,6 +39,7 @@ Terreno 57.
 3. **Encryption at rest (web)** — Web persistence uses AES-GCM via the default `createServerKeyProvider`; understand key fetch and `onDecryptFailure` for support flows.
 4. **Key lifecycle on logout** — Call `syncDb.stop()` and consider `wipeLocalData` when the authenticated user changes so one account does not read another's local store.
 5. **Backend sync registration** — Every synced collection needs `syncPlugin`, `isDeletedPlugin`, and a `sync` block on `modelRouter`, plus `SyncApp` and `RealtimeApp` on the server. Optional `sync.adminBroadcast: true` also emits `sync:delta` to `{collection}|admin`. Admin clients subscribe with `mode: "window"` (`createSyncDb({windowCollections})`) so they do not snapshot or reconcile that collection; they call `hydrateWindow` (REST membership ids + `GET /sync/entities`). Pass the same `accessControl` into `SyncApp` that `AdminApp` uses so `{collection}|admin` requires `admin:access`. `AdminApp.register` also installs that model's admin list/read/`queryFilter` on the admin window so hydrate and live `|admin` deltas match `/admin` REST. Without RBAC, `user.admin` is enough for the shell gate. Live deltas on that stream apply only to ids already in the window; Refresh is a REST re-query plus `hydrateWindow`.
+6. **Collections too large to sync whole** — List them in `createSyncDb({queryCollections})`. They skip snapshot bootstrap; screens load server-filtered *query windows* (`useWindowQuery`) from the `modelRouter` list endpoint. A window stores only membership (ordered ids); rows live once in the shared entity table. Overlapping windows therefore share rows — one delta updates both views — while each view lists only its own query's members, plus cached rows that newly match its `where` inside the loaded range. Live deltas still arrive on the user's normal streams; windows on screen refetch on reconnect to cover deltas missed offline.
 
 ## Tradeoffs
 

package/dist/docs/versioned/next/how-to/README.md

@@ -37,6 +37,7 @@ Problem-oriented, practical steps. Use these when you know what you want to do.
 - [Develop an AI feature](ai-feature-development.md) — Dataset → prompt → evaluators → experiment → production label → live traces (SOP)
 - [Observe LLM calls](observe-llm-calls.md) — Register ObservabilityApp, primaries, price map, sampleRate
 - [Build a durable workflow](build-a-durable-workflow.md) — Phased harness task with an agent, a human sign-off, and a once-only side effect (`clinic.intakeSummary`)
+- [Run the harness on multiple instances](run-harness-on-multiple-instances.md) — `JobsRunner`: harness phases as `@terreno/jobs` jobs on every instance
 - [Ship a new task version](ship-a-new-task-version.md) — Change a durable harness task without stranding in-flight runs
 - [GitHub issue lifecycle](github-issue-lifecycle.md) — Create pick-ready issues, Pick ⇄ Roast, or unattended `status:ready-for-dev` pickup
 - [Call external APIs](call-external-apis.md) — Authenticated HTTP client, retries, and error normalization for third-party integrations

package/dist/docs/versioned/next/how-to/call-external-apis.md

@@ -94,7 +94,7 @@ Note the hook's output classification also feeds the client's retry decision —
 
 - **Your own backend's routes** — use the generated `@terreno/rtk` SDK (frontend) or call the function directly (backend).
 - **SDK-managed transport** (Twilio, googleapis) — don't fight the SDK's HTTP layer; adopt only `normalizeApiError`/`withApiErrorHandling` for consistent failure handling.
-- **Webhook notifiers** — `sendToSlack`/`sendToGoogleChat`/`sendToZoom` already exist; see [Webhooks & Notifications](../reference/api.md#webhooks--notifications). Slack @-mentions require a stored Slack member ID (`mentionUserIds` or `formatSlackUserMention`); names and emails in the text do not notify anyone.
+- **Webhook notifiers** — `sendToSlack`/`sendToGoogleChat`/`sendToZoom` post to a fixed webhook. Slack channel creation, invites, and bot posts use `createSlackPrivateChannel`, `inviteSlackUsersToChannel`, and `postSlackMessage` with `SLACK_BOT_TOKEN`. See [Webhooks & Notifications](../reference/api.md#webhooks--notifications).
 
 ## Related
 

package/dist/docs/versioned/next/how-to/migrate-rtk-to-syncdb.md

@@ -152,7 +152,37 @@ import {useQuery} from "@terreno/syncdb/react";
 const todos = useQuery<Todo>("todos", {filter: (t) => !t.completed});
 ```
 
-`filter` / `sort` run in JS — there are no server query params. Tombstones are excluded unless `{includeDeleted: true}`.
+`where` / `filter` / `sort` / `limit` run in JS over the local store. Tombstones are excluded unless `{includeDeleted: true}`.
+
+### Filtered, sorted, and paged list args
+
+RTK list args map to the same syntax. Pick the hook by collection size:
+
+| RTK | Whole collection synced (`collections`) | Too large to sync whole (`queryCollections`) |
+|-----|------------------------------------------|----------------------------------------------|
+| `useGetMessagesQuery({threadId})` | `useQuery("messages", {where: {threadId}})` | `useWindowQuery("messages", {where: {threadId}})` |
+| `{status: {$in: ["open", "pending"]}}` | `where: {status: {$in: [...]}}` | same |
+| `{sort: "-created"}` | `sort: "-created"` | same |
+| `{limit: 20}` | `limit: 20` | `pageSize: 20` |
+| `{page: 2}` / load more | grow `limit` | `fetchNextPage()` |
+| `data.more` / `data.total` | — | `hasMore` / `total` |
+| `isLoading` / `isFetching` / `refetch` | not needed | `isLoading` / `isFetching` / `refetch` |
+| `{skip: !threadId}` | not needed | `skip: !threadId` |
+
+```typescript
+import {useWindowQuery} from "@terreno/syncdb/react";
+
+const {data: messages, hasMore, fetchNextPage, isLoading} = useWindowQuery<Message>("messages", {
+  where: {threadId},
+  sort: "-created",
+  pageSize: 30,
+  skip: !threadId,
+});
+```
+
+Windows with different filters keep separate result lists but share one local row per
+document, so an edit to a message shown in two windows updates once. See
+[Query windows](../reference/syncdb.md#query-windows).
 
 ### Delete this RTK read code
 

package/dist/docs/versioned/next/how-to/preview-demo.md

@@ -1,7 +1,11 @@
 # Preview a component demo
 
-Open a component demo and change the shareable query. The preview bar is on demo and
-dev component routes. `?embed=1` applies the same query and hides the bar.
+Open a component demo and change the shareable query. The preview controls sit in the
+navigation header on demo and dev component routes: dropdowns for theme, viewport, and
+background, plus a settings button (sliders icon) that opens a modal for right-to-left,
+reduced motion, locale, and the share query. In windows narrower than 768px every
+preview control moves into that modal; the Dev/Demo mode link stays in the header. `?embed=1` applies the same query and hides
+the controls.
 
 ```text
 /demo/Button?theme=dark&viewport=375&background=inverse&locale=en-US&rtl=1&reducedMotion=1
@@ -16,4 +20,4 @@ dev component routes. `?embed=1` applies the same query and hides the bar.
 | `rtl` | `1` sets document direction to `rtl` on web |
 | `reducedMotion` | `1` sets `data-reduced-motion="1"` on the document element |
 
-Reload keeps the query. The bar shows the current query when any value is not the default. Choosing a control's default removes that key from the URL. Leaving the component route restores the app theme and, on web, document direction and reduced motion. Custom palettes stay on `/palette`.
+Reload keeps the query. The settings modal shows the current query when any value is not the default. Choosing a control's default removes that key from the URL. Leaving the component route restores the app theme and, on web, document direction and reduced motion. Custom palettes stay on `/palette`.

package/dist/docs/versioned/next/how-to/run-harness-on-multiple-instances.md

@@ -0,0 +1,93 @@
+# Run the harness on multiple instances
+
+Run durable harness tasks on every instance of a scaled-out backend (for example several
+Cloud Run instances). Swap the default `InProcessRunner`, where one process drains every
+task, for `JobsRunner`, which runs each phase as a `@terreno/jobs` job. API:
+[AI harness reference → JobsRunner](../reference/ai-harness.md#jobsrunner). Why it is safe:
+[Durable agent harness → Runners](../explanation/durable-agent-harness.md#runners).
+
+You need a working harness (see [Build a durable workflow](build-a-durable-workflow.md)) and
+a MongoDB replica set. Task data does not change, so you can switch runners without a
+migration.
+
+## 1. Install the jobs package (1 minute)
+
+`@terreno/jobs` is an optional peer of `@terreno/ai`. Add it to the app:
+
+```bash
+bun add @terreno/jobs
+```
+
+## 2. Open the harness with a JobsRunner (5 minutes)
+
+Create the `JobsApp` first, hand it to the runner, and register it on the app. Build the
+app before the harness starts: `harness.start()` throws when no `JobsApp` is registered.
+The runner defines the `terreno.harness.phase` job on it.
+
+```typescript
+import {Harness} from "@terreno/ai/harness";
+import {JobsRunner} from "@terreno/ai/harness/jobsRunner";
+import {TerrenoApp} from "@terreno/api";
+import {JobsApp} from "@terreno/jobs";
+
+const jobs = new JobsApp();
+const harness = await Harness.open({
+  registry: [intakeSummary, summarizer],
+  runner: new JobsRunner({jobs}),
+});
+
+const app = new TerrenoApp({userModel: User}).register(jobs);
+// ...register HarnessApp and your routes, then build or start the server.
+app.start();
+
+await harness.start();
+await jobs.startWorker();
+```
+
+Shut down in the reverse order so phases in flight finish:
+
+```typescript
+await harness.stop(); // stops dispatching, waits for this instance's phases
+await jobs.stopWorker();
+```
+
+## 3. Run the same code on every instance
+
+Every instance runs the same startup. Each one:
+
+- scans for runnable tasks and enqueues one job per runnable phase (duplicates collapse on
+  the job's idempotency key),
+- runs phase jobs from the shared jobs queue, and
+- sweeps task leases that expired because an instance died.
+
+No instance is special, and none waits on standby.
+
+## 4. Check it works (2 minutes)
+
+1. Start two backend processes against the same database (different `PORT`s).
+2. Create a task (for the example backend, run `startClinicalIntake`).
+3. Open the admin **Jobs** screen. Each phase shows as its own `terreno.harness.phase` job
+   with payload `{taskId}`.
+4. Kill one process mid-phase. Within one lease (`leaseDuration`, default 30 s) the other
+   process replays a `replay: "safe"` phase, or parks any other phase as `interrupted`.
+
+## Tune it
+
+| Want | Set |
+| --- | --- |
+| Faster pickup of new work | `new JobsRunner({jobs, pollInterval: {milliseconds: 200}})` and `new JobsApp({pollIntervalMs: 200})` |
+| Faster crash recovery | A shorter `leaseDuration` with `heartbeatInterval` about a third of it |
+| Phases longer than the jobs lock TTL | Nothing: the task lease, renewed by heartbeats, decides who commits |
+
+## Troubleshoot
+
+- **Jobs fail with `harness-runner-stopped`.** A jobs worker runs on an instance whose
+  harness is not started. Start the harness before `jobs.startWorker()`. The jobs worker
+  retries the job, so no task is lost.
+- **`harness.start()` throws "JobsRunner needs a registered JobsApp".** The dispatcher
+  enqueues through the registered jobs service, and `TerrenoApp` registers plugins when it
+  builds. Call `app.start()` (or `build()`) before `harness.start()`.
+- **A task stays `pending`.** Check that its `name@version` is in the registry of the
+  running instances. A runner only dispatches registered versions.
+- **Pick one runner per database.** Mixing them is safe (every claim and commit is fenced),
+  but an `InProcessRunner` owner then claims tasks itself and runs them to a stop.

package/dist/docs/versioned/next/implementationPlans/durable-agent-harness.md

@@ -140,7 +140,7 @@ On resume, a task whose status is `running` with an expired lease was interrupte
 | Runner | Ownership | Use |
 |---|---|---|
 | `InProcessRunner` (default) | One process holds the `HarnessOwner` lease (heartbeat). A second process gets `standby` and takes over on lease expiry. Polls runnable tasks; wakes on `runAt` and events. | Local agents, single-instance apps, tests |
-| `JobsRunner` (`@terreno/ai/harness/jobsRunner`, optional peer `@terreno/jobs`) | Each runnable phase → job `terreno.harness.phase`, idempotency key `taskId:phase:attempt`. Handler acquires the task lease (fenced, heartbeat) and runs one phase. The task lease, not the job lock, is the authority. | Multi-instance Cloud Run |
+| `JobsRunner` (`@terreno/ai/harness/jobsRunner`, optional peer `@terreno/jobs`) | Each runnable phase → job `terreno.harness.phase`, idempotency key `taskId:phase:attempt:claims` (`claims` counts claims, so a phase revisited by an event wake or crash replay gets a new job; jobs idempotency keys are permanent). Handler acquires the task lease (fenced, heartbeat) and runs one phase. The task lease, not the job lock, is the authority. | Multi-instance Cloud Run |
 
 ### Ownership tree
 

package/dist/docs/versioned/next/reference/ai-harness.md

@@ -141,7 +141,7 @@ The `task` view passed to `run`: `{id, name, version, input, state, phase, attem
 | --- | --- |
 | `Harness.open({registry, runner?, models?, env?, priceMap?, streaming?, testHooks?})` | Checks requirements, rejects a duplicate `name@version` or agent name, ensures collections and indexes exist. `runner` defaults to `new InProcessRunner()`. See [open options](#open-options). |
 | `start()` | Throws, claiming nothing, when any non-terminal task uses a `name@version` missing from the registry (see [Versioning](#versioning)). Then starts the runner. Throws when already started. The runner recovers expired tasks once it owns execution. |
-| `stop()` | Stops claiming work, waits (without a time limit) for the phase in flight while still renewing the owner lease, then releases it. No-op when not started. |
+| `stop()` | Stops claiming work, waits (without a time limit) for every phase in flight while still renewing the owner lease, then releases it. No-op when not started. |
 | `createTask(definition, input, {requestId?, userId?})` | Inserts a `pending` task, its `ObsTrace`, and its root span in one transaction. Wakes the runner. The definition must be in the registry. |
 | `resolveInterrupted(id, {action, reason, result?, userId?})` | Resolve an `interrupted` task. See [resolveInterrupted](#resolveinterrupted). |
 | `abort(id, {reason, userId?})` | Abort a task and every non-terminal task it owns, bottom-up. See [abort](#abort). |
@@ -166,7 +166,7 @@ some), throws. A soft-deleted task still owns its `requestId`.
 | `models` | `({provider, modelId}) => LanguageModel` | Resolves an agent's model refs to Vercel AI SDK models. Required when `registry` lists an agent. |
 | `env` | `ExecutionEnv` | Handed to phases as `rt.env` and to tools as `api.env`. Optional. |
 | `priceMap` | `Record<modelId, {inputPerMTok, outputPerMTok}>` | Prices LLM spans (`usage.costUsd`). Defaults to the registered `ObservabilityApp`'s `priceMap`, read at call time. |
-| `runner` | `HarnessRunner` | Default `new InProcessRunner()`. |
+| `runner` | `HarnessRunner` | Default `new InProcessRunner()`. For many instances, `JobsRunner` (see [JobsRunner](#jobsrunner)). |
 | `streaming` | `{deltaFlushChars?, deltaFlushInterval?, deltaTtl?}` | How streamed model text becomes `delta` events. Defaults: 200 characters, `{milliseconds: 250}`, `{hours: 1}`. Non-positive values throw. See [Event stream (SSE)](#event-stream-sse). |
 | `testHooks` | `HarnessTestHooks` | Test-only. See [Testing](#testing). |
 
@@ -183,7 +183,8 @@ const harness = await Harness.open({
 
 | Option | Default | Description |
 | --- | --- | --- |
-| `pollInterval` | `{milliseconds: 250}` | Idle sleep between claim attempts. `createTask` wakes the runner early. |
+| `concurrency` | `8` | Most claimed tasks run at once. Must be a positive integer, or the constructor throws. `1` runs one task at a time. |
+| `pollInterval` | `{milliseconds: 250}` | Idle sleep between claim attempts. `createTask` and a task that frees a slot wake the runner early. |
 | `leaseDuration` | `{seconds: 30}` | Lifetime of the owner lease and of each task lease without a renewal. Luxon `DurationLike`. |
 | `heartbeatInterval` | `{seconds: 10}` | How often both leases are renewed. Must be positive and shorter than `leaseDuration`, or the constructor throws. |
 | `ownerId` | `hostname:pid:uuid` | Id written into every lease this runner holds. |
@@ -197,13 +198,101 @@ Only the owner claims work. It claims the oldest `pending` task whose `name@vers
 registered, whose `runAt` is empty or past (by Luxon's `DateTime.now()`), and that has no
 `abortRequested`. It sets the task `running` with a fresh task lease and runs phases one
 after another until the task stops: terminal, `pending` for a retry, `waiting`, or
-aborted. One task runs at a time. An idle owner re-polls every `pollInterval`, so a retry
-starts within one poll of its `runAt`.
+aborted. An idle owner re-polls every `pollInterval`, so a retry starts within one poll of
+its `runAt`.
+
+Up to `concurrency` tasks run at once, each under its own task lease and heartbeat, so a
+slow phase (a long model call, a phase blocked on I/O) does not delay other tasks. Ordering
+guarantees:
+
+- Tasks are **claimed** oldest first (`created`), one claim at a time, while a slot is free.
+- With `concurrency` above 1, claimed tasks run in parallel, so a later task can finish
+  first. Use `concurrency: 1` for strict one-at-a-time order.
+- Phases of one task always run in order, and a phase never commits twice: the claim is an
+  atomic update, and every commit is fenced on the task lease. (A runner that froze past
+  its lease may still be executing phase code while recovery replays it; only its commit
+  is rejected.)
+- Turns of one conversation stay serial regardless of `concurrency`; turns of different
+  conversations, and sibling child tasks, run in parallel.
+- `stop()` stops claiming and waits for every task in flight.
+
+Before Phase 2 the runner ran one task at a time. Upgrading to the default of 8 means
+parallel model calls (watch provider rate limits and cost) and more stream-counter retries
+(see [Implementation notes](#implementation-notes)). Pass `concurrency: 1` to keep the old
+behavior.
+
+### JobsRunner
+
+Runs phases as `@terreno/jobs` jobs so any number of instances share the work. Import it
+from its own entry point; `@terreno/jobs` is an optional peer dependency, and importing
+`@terreno/ai` or `@terreno/ai/harness` never loads it. How-to:
+[Run the harness on multiple instances](../how-to/run-harness-on-multiple-instances.md).
+
+```typescript
+import {JobsApp} from "@terreno/jobs";
+import {Harness} from "@terreno/ai/harness";
+import {JobsRunner} from "@terreno/ai/harness/jobsRunner";
+
+const jobs = new JobsApp();
+const harness = await Harness.open({registry, runner: new JobsRunner({jobs})});
+app.register(jobs).build(); // the runner enqueues through the registered jobs service
+await harness.start();
+await jobs.startWorker();
+```
+
+| Option | Default | Description |
+| --- | --- | --- |
+| `jobs` | required | The `JobsApp`. The constructor defines `terreno.harness.phase` on it (`HARNESS_PHASE_JOB_NAME`). Register it and build the app before `harness.start()`; otherwise `start()` throws `harness-config-invalid`. |
+| `pollInterval` | `{milliseconds: 500}` | Sleep between dispatch scans. `createTask`, `sendEvent`, and each finished phase wake the dispatcher early. |
+| `leaseDuration` | `{seconds: 30}` | Lifetime of each task lease without a renewal. |
+| `heartbeatInterval` | `{seconds: 10}` | How often a running phase renews its task lease, and how often this instance sweeps expired leases. Must be positive and shorter than `leaseDuration`, or the constructor throws. |
+| `ownerId` | `hostname:pid:uuid` | Id written into every task lease this runner holds. |
+| `dispatchBatchSize` | `100` | Most runnable tasks one scan enqueues. Must be a positive integer, or the constructor throws. |
+
+Every instance runs a dispatcher and a jobs worker. There is no owner lease.
+
+1. **Dispatch.** Each scan lists runnable tasks (the same rule `InProcessRunner` claims by)
+   and enqueues one `terreno.harness.phase` job per task with payload `{taskId}` and
+   idempotency key `taskId:phase:attempt:claims`. `claims` counts how often the task was
+   claimed, so each runnable visit of a phase (first run, retry, event wake, replay) gets
+   its own job, and repeated scans or instances enqueue it once. When the job with that key
+   already ended without claiming the task (for example it died after its retries, or ran
+   on an instance mid-deploy that did not register the task), the dispatcher follows or
+   extends the chain `<key>:<endedJobId>` until a job is in flight, so the task is not
+   stranded.
+2. **Run one phase.** The job handler claims that task under a fresh fenced task lease,
+   runs **one** phase with heartbeats, and hands a task that is still `running` back as
+   `pending` (lease cleared) for the next job. It then wakes the dispatcher. A handler
+   whose task is no longer runnable (another job claimed it, or it is aborting) does
+   nothing and succeeds.
+3. **Recover.** Each instance sweeps expired task leases once per `heartbeatInterval`, with
+   the same replay and interruption rules as `InProcessRunner`. A replayed task becomes
+   runnable with a new `claims` value, so it is dispatched again.
+
+Load: with N instances, the recovery sweep (expired leases, settled child waits, stranded
+queued conversations) runs N times per `heartbeatInterval`, where `InProcessRunner` runs it
+once, and each scan does one job lookup per runnable task (up to `dispatchBatchSize`). The
+sweeps are fenced, so this costs queries, not correctness. Raise `heartbeatInterval` and
+`pollInterval` for large fleets.
+
+The task lease, not the job lock, is the authority. Two jobs for one phase race on the
+atomic claim, and only one runs it. A worker that froze past its lease cannot commit: its
+`rt.commit` throws `HarnessCommitConflictError`. A phase that runs longer than the jobs
+lock TTL is safe for the same reason. A job that runs on an instance whose harness is not
+started (or is stopping) throws `harness-runner-stopped`, so the jobs worker retries it.
+
+`stop()` stops dispatching and waits for every phase this instance is running. Stop the
+jobs worker (`jobs.stopWorker()`) after `harness.stop()`.
+
+### Custom runners
 
 A custom runner implements `HarnessRunner` (`start(context)`, `stop()`, `wake()`). The
 `context` provides `acquireOwnerLease(lease)`, `releaseOwnerLease(lease)`,
-`recoverExpired()`, `claimNext(lease)`, and `runTask(task, lease)`, where `lease` is
-`{owner, duration, heartbeat}` (Luxon `Duration`s).
+`recoverExpired()`, `claimNext(lease)`, `claimTask(taskId, lease)` (claim one task by id
+when it is runnable now, else `null`), `listRunnable(limit)` (oldest runnable tasks as
+`{taskId, phase, attempt, claims}`, unclaimed), and `runTask(task, lease, {maxPhases?})`,
+where `lease` is `{owner, duration, heartbeat}` (Luxon `Duration`s). With `maxPhases`,
+`runTask` hands a task that is still `running` after that many phases back as `pending`.
 
 ## Runtime (rt)
 
@@ -261,7 +350,9 @@ Assumptions:
   recovery treats them as expired.
 
 Recovery runs when a runner becomes owner (on `start()` or on takeover) and on every owner
-heartbeat. It scans up to 100 `running` tasks whose lease expired (or that have no lease),
+heartbeat. `JobsRunner` has no owner lease: every instance sweeps once per
+`heartbeatInterval`, and concurrent sweeps are fenced, so each expired task is recovered
+once. It scans up to 100 `running` tasks whose lease expired (or that have no lease),
 oldest first, for registered `name@version`s only. A stopping owner skips the sweep. A task
 whose interruption commit keeps failing for a non-conflict reason is logged and retried on
 the next sweep.
@@ -1503,6 +1594,7 @@ empty objects are kept (`minimize: false`), so an initial state `{}` is stored a
 | `outcome` | `{status, result, error}` | Set on terminal commit. |
 | `attempt` | Number | Failed attempts of the current phase. Default 0. |
 | `step` | Number | Phase commits so far; names the current phase visit for idempotent child creation. Default 0. |
+| `claims` | Number | Times a runner claimed the task. Numbers each runnable visit; `JobsRunner` puts it in the job idempotency key. Default 0; a missing value reads as 0. |
 | `retry` | `{maxAttempts, backoffMs, maxBackoffMs}` | Copied from the definition. |
 | `ownership` | `{kind: root \| task \| conversation, id}` | Default `root`. `rt.createTask` children are `{kind: "task", id: <parent>}`; agent turns are `{kind: "conversation", id}`, except subagent turns, which are `{kind: "task", id: <caller>}`. |
 | `rootTaskId` | ObjectId | Top of the ownership tree; equals `_id` for root tasks. |
@@ -1686,12 +1778,13 @@ lists every kind:
 | `code` | `status` | `title` | Kinds of failure |
 | --- | --- | --- | --- |
 | `harness-definition-invalid` | 500 | Invalid harness definition | `defineTask` / `defineTool` / `defineAgent` / `defineExtension` / `approvalGate` validation, and `rt.*` misuse inside a phase. Thrown as `HarnessDefinitionError`, which fails the task at once (no retry). |
-| `harness-config-invalid` | 500 | Invalid harness configuration | `Harness.open`, `HarnessApp`, `InProcessRunner`, `streaming`, and registry problems. |
+| `harness-config-invalid` | 500 | Invalid harness configuration | `Harness.open`, `HarnessApp`, `InProcessRunner` (lease timing, `concurrency`), `JobsRunner`, `streaming`, and registry problems. |
 | `harness-replica-set-required` | 500 | MongoDB replica set required | `Harness.open` on a deployment without transactions. |
-| `harness-invalid-request` | 400 | Invalid harness request | A caller passed a bad argument (`abort`, `sendEvent`, `decideApproval`, `resolveInterrupted`, `submit` / `send`, `createConversation`, memos). |
+| `harness-invalid-request` | 400 | Invalid harness request | A caller passed a bad argument (`abort`, `sendEvent`, `decideApproval`, `resolveInterrupted`, `submit` / `send`, `createConversation`, memos, a `terreno.harness.phase` payload without `taskId`). |
 | `harness-not-registered` | 404 | Not registered in this harness | A task definition, agent, or extension the registry lacks. |
 | `harness-not-found` | 404 | Harness record not found | A conversation or child task disappeared. |
-| `harness-already-started` | 409 | Already started | `Harness.start` / `InProcessRunner.start` called twice. |
+| `harness-already-started` | 409 | Already started | `Harness.start` / `InProcessRunner.start` / `JobsRunner.start` called twice. |
+| `harness-runner-stopped` | 503 | Harness runner is not running | A `terreno.harness.phase` job ran on an instance whose harness is not started; the jobs worker retries it. |
 | `harness-commit-conflict` | 409 | Harness commit lost its checkpoint fence | `HarnessCommitConflictError`. |
 | `harness-approval-not-pending` | 409 | Approval can no longer be decided | `HarnessApprovalConflictError`. |
 | `harness-conversation-busy` | 409 | Conversation is busy | `HarnessConversationBusyError`. |
@@ -1809,7 +1902,7 @@ Low-risk choices made in the first slice:
 | The SSE tail starts at a cluster time read before any subscriber's replay query | No gap between replay and tail; overlap is removed by `seq`. |
 | One shared change stream per `HarnessApp`, not one per connection | A change stream holds a pooled connection while it waits; per-connection streams let a few dozen viewers starve commits and lease renewals. |
 | Queue recovery is a sweep by the execution owner, not part of the turn's terminal commit | Starting a turn is its own transaction; the sweep (fenced, idempotent) covers the crash window between the two. |
-| Writers to one stream conflict on its counter document | The price of commit-ordered `seq`s. `withTransaction` retries the loser. `InProcessRunner` runs one task at a time, so only HTTP submits, approvals, and deltas contend; a parallel runner fanning out many children of one tree will see retries. |
+| Writers to one stream conflict on its counter document | The price of commit-ordered `seq`s. `withTransaction` retries the loser. `InProcessRunner` (default `concurrency: 8`) and `JobsRunner` both run sibling tasks of one tree in parallel, so their writes to the shared root stream retry alongside HTTP submits, approvals, and deltas. `InProcessRunner({concurrency: 1})` avoids task-on-task contention. |
 | Interruption spans start at `lease.acquiredAt` | Records when the cut-off phase began. |
 | `resolveInterrupted({action: "abort"})` closes the trace as `error` | `ObsTrace.status` is `ok` or `error`; an abort is not a success. |
 | `requestId` replays are scoped to the same `userId` | Stops one caller from reading another user's task through a shared key. |

package/dist/docs/versioned/next/reference/api.md

@@ -41,7 +41,7 @@ REST API framework built on Express and Mongoose. Provides modelRouter (CRUD end
 - Validation: `configureOpenApiValidator`, `validateRequestBody`, `validateQueryParams`, `createValidator`
 - Middleware: `openApiEtagMiddleware`, `sentryAppVersionMiddleware`
 - Extensibility: `TerrenoPlugin` interface
-- Notifiers: `sendToSlack`, `formatSlackUserMention`, `lookupSlackUserIdByEmail`, `sendToGoogleChat`, `sendToZoom`
+- Notifiers: `sendToSlack`, `formatSlackUserMention`, `lookupSlackUserIdByEmail`, `createSlackPrivateChannel`, `inviteSlackUsersToChannel`, `postSlackMessage`, `findSlackChannelByName`, `sendToGoogleChat`, `sendToZoom`
 - HTTP client: `createAuthenticatedClient`, `withApiErrorHandling`, `normalizeApiError`, `markRetryUnsafe`
 
 ## Server Setup
@@ -224,6 +224,30 @@ export const todoRouter = modelRouter("/todos", Todo, {
 Do not add `endpoints: (router) => { router.get(...) }` when an action fits.
 Action config supports `allowAnonymous: true` as a per-action override; it does not change authentication for CRUD or sibling actions.
 
+## modelRouter list queries
+
+`GET /{path}` accepts filters on fields listed in `queryFields` (qs syntax, so operators work:
+`?status[$in][0]=open&created[$gte]=2026-01-01`), plus the reserved `limit`, `page`, and `sort`.
+`limit` is clamped to `maxLimit` (default 500); the response echoes the applied `limit` with
+`{data, limit, more, page, total}`.
+
+A filter on a field missing from `queryFields` is rejected before query validation runs, so
+it can never be stripped and silently ignored:
+
+```json
+{
+  "status": 400,
+  "code": "query-param-not-allowed",
+  "title": "Query parameter not allowed",
+  "detail": "title is not allowed as a query param. Add \"title\" to queryFields on the Todo modelRouter to filter on it. Allowed: completed, created, ownerId.",
+  "source": {"parameter": "title"},
+  "meta": {"queryParam": "title", "model": "Todo", "allowedQueryFields": ["completed", "created", "ownerId"]}
+}
+```
+
+`@terreno/syncdb` raises this as `QueryFieldNotAllowedError` (see
+[Query windows](syncdb.md#query-windows)).
+
 ## modelRouter array operations
 
 When a model has at least one array field, `modelRouter` provides subroutes for adding,
@@ -1581,6 +1605,44 @@ await sendToSlack(`${formatSlackUserMention("U012ABCDEF")} Deployment complete`,
 });
 ``````
 
+Incoming webhooks are fixed to the channel chosen when the webhook was created.
+Private channels, invites, and messages to a channel id use the bot token for
+one Slack workspace (`SLACK_BOT_TOKEN`):
+
+| Function | Slack method | Bot scopes |
+| --- | --- | --- |
+| `createSlackPrivateChannel` | `conversations.create`, `conversations.invite`, `chat.postMessage` | `groups:write`, `groups:write.invites`, `chat:write` |
+| `inviteSlackUsersToChannel` | `conversations.invite` | `groups:write.invites` for private channels, `channels:write.invites` for public |
+| `postSlackMessage` | `chat.postMessage` | `chat:write` |
+| `findSlackChannelByName` | `conversations.list` | `channels:read`, `groups:read` |
+| `lookupSlackUserIdByEmail` | `users.lookupByEmail` | `users:read.email` |
+
+The bot can post and invite only in channels it belongs to. Creating a private
+channel makes the bot a member. An existing private channel must already
+include the app; a bot cannot join a private channel by itself. Store the
+returned channel id. Names are unique, mutable, and limited to lowercase
+letters, numbers, hyphens, and underscores.
+
+``````typescript
+import {createSlackPrivateChannel, inviteSlackUsersToChannel} from "@terreno/api";
+
+const channel = await createSlackPrivateChannel({
+  name: "case-12345",
+  userIds: ["U012ABCDEF"],
+  initialMessage: "Case opened.",
+  mentionUserIds: ["U012ABCDEF"],
+});
+
+await inviteSlackUsersToChannel({
+  channelId: channel.channelId,
+  userIds: ["U098765432"],
+});
+``````
+
+A taken name throws status 409. `findSlackChannelByName({name})` returns a
+public channel, or a private channel the bot is already in. It returns
+`undefined` when that channel is not visible to the bot.
+
 ### Google Chat Notifications
 
 ``````typescript
@@ -1913,7 +1975,7 @@ Complete reference of environment variables used by @terreno/api:
 | Variable | Required | Default | Description |
 |----------|----------|---------|-------------|
 | `SLACK_WEBHOOKS` | No | — | JSON object mapping names to Slack webhook URLs: `{"default":"https://..."}` |
-| `SLACK_BOT_TOKEN` | No | — | Bot token (`xoxb-…`) with `users:read.email` for `lookupSlackUserIdByEmail` |
+| `SLACK_BOT_TOKEN` | No | — | Bot token (`xoxb-…`) for one workspace. Scopes: `users:read.email`, `groups:write`, `groups:write.invites`, `channels:write.invites`, `chat:write`, `channels:read`, `groups:read` |
 | `GOOGLE_CHAT_WEBHOOKS` | No | — | JSON object mapping names to Google Chat webhook URLs |
 | `ZOOM_CHAT_WEBHOOKS` | No | — | JSON object mapping names to Zoom webhook URLs |
 | `WEBHOOK_SECRET` | No | — | Secret for validating incoming webhook signatures |

package/dist/docs/versioned/next/reference/environment-variables.md

@@ -223,7 +223,7 @@ There is **no** `RATE_LIMIT_ENABLED` (or similar) read by `@terreno/api`. Apps t
 |----------|---------|----------|---------|--------|-------|
 | `SLACK_WEBHOOKS` | `@terreno/api` | ❌ | — | Yes | server |
 | `SLACK_WEBHOOK_URL` | scripts | ❌ | — | Yes | server |
-| `SLACK_BOT_TOKEN` | `@terreno/api` `lookupSlackUserIdByEmail` | ❌ | — | Yes | server |
+| `SLACK_BOT_TOKEN` | `@terreno/api` Slack Web API (`lookupSlackUserIdByEmail`, private channels, invites, `chat.postMessage`) | ❌ | — | Yes | server |
 | `GOOGLE_CHAT_WEBHOOKS` | `@terreno/api` | ❌ | — | Yes | server |
 | `GOOGLE_CHAT_WEBHOOK_URL` | scripts | ❌ | — | Yes | server |
 | `ZOOM_CHAT_WEBHOOKS` | `@terreno/api` | ❌ | — | Yes | server |

package/dist/docs/versioned/next/reference/lifecycle-plugin.md

@@ -16,6 +16,17 @@ loops that invoke those stages; they are not stages and must not appear as `stag
 | `terreno-4-brew` | Roast PASS for every in-scope task + branch/evidence | pushed head + PR + product-CI trigger check + review-bot wait + attached evidence | Taste |
 | `terreno-5-taste` | PR + current state | one current-head reaction; before push: fetch latest `master` (merge only when needed), re-verify last-run failed tests locally, run root `prepush` when present (otherwise affected-package checks) in a no-context subagent, then watch CI; invoked directly, repeats bounded reactions until terminal | null or fresh Taste |
 
+Pre-Grow skill (not a stage):
+
+| Skill | Input | Primary output | Next |
+| --- | --- | --- | --- |
+| `terreno-prd` | app owner + conversation, prototype, repository | interview, then a published full product PRD (problem, why now, solution, user stories, success metrics and guardrails, constraints, rollout, out of scope); every metric measurable or `TBD` with its decider | Grow, with the PRD as its spec; metrics become acceptance criteria, `TBD`s become grilling questions, settled product decisions are not re-asked |
+
+It is explicit-invoke only (`/terreno-prd`, `$terreno-prd`, `/terreno:prd`). It
+interviews with Grow's grilling procedure over a product tree, is sized for one app owner
+and one or two developers, and carries no implementation or testing decisions; those stay
+in Grow.
+
 Outer loops (not stages):
 
 | Skill | What it walks | Default |