@hotmeshio/long-tail 0.23.2 → 0.24.1

package/docs/events.md

@@ -45,6 +45,12 @@ services:
       - NATS_TOKEN=your-token
 ```
 
+**Browser access.** The dashboard only subscribes. With the WebSocket proxy (`wsProxy`, so browsers reach NATS through Long Tail at `<basePath>/nats-ws`), `GET /api/nats-credentials` returns a WebSocket URL carrying a short-lived ticket for the signed-in person (`NATS_WS_TICKET_TTL_SECONDS`, default 300) and no NATS credential. The proxy admits only a ticket holder, writes the server's `NATS_TOKEN` into the connection itself, and drops anything the browser tries to publish. When browsers connect to NATS directly, set `NATS_DASHBOARD_TOKEN` to a subscribe-only NATS credential and the route hands that out instead of `NATS_TOKEN`.
+
+**What a browser receives.** Builders (superadmin, or the `engineer` role) receive events whole. Everyone else receives the browser view: escalation events keep their routing fields (`id`, `type`, `subtype`, `status`, `priority`, `role`, `assigned_to`, `assigned_until`, `workflow_id`, `workflow_type`, `origin_id`, `task_id`, `parent_id`, `resolved_at`, `claimed_at`, and the bulk and reassignment fields), and every other event arrives without `data` and `milestones`. Records are read through the scoped API by id, so pages refresh as before; the invoke page reads its own run's result from `GET /api/workflows/:id/result`. The WebSocket proxy applies the view per connection from the ticket's person, and the Socket.IO adapter per socket from the person its authenticator returns (`{ userId }`; an authenticator that returns `true` admits the socket with the browser view). Server-side consumers, such as agent triggers and callbacks, receive events whole.
+
+**Reconnecting.** The dashboard fetches a fresh ticket from `GET /api/nats-credentials` for every connection attempt, through the same client as every API call, so an expired session is refreshed first. After a drop the first retry lands at a random point within `NATS_RECONNECT_SPREAD_MS` (default 5000), so tabs that dropped together after a deploy return gradually; later retries back off with jitter from `NATS_RECONNECT_INITIAL_MS` (default 1000) up to `NATS_RECONNECT_MAX_MS` (default 60000), one attempt at a time. It waits while the tab is hidden, and stops only when the person is signed out; a new sign-in starts it again. On reconnect it reopens every active subscription and refetches the open pages once, spread across the same window. When live updates stay down longer than `NATS_LIVE_NOTICE_AFTER_MS` (default 30000), a "Live updates unavailable" notice appears in the announcement banner row. `GET /api/settings` carries these values as `events.reconnect` (`initialDelayMs`, `maxDelayMs`, `noticeAfterMs`, `spreadMs`).
+
 Both adapters still publish events regardless of `EVENT_TRANSPORT` — the setting only controls what the dashboard listens on. This means server-side event consumers (callbacks, NATS subscribers) work independently of the dashboard transport.
 
 ## The event envelope

package/docs/iam.md

@@ -123,14 +123,14 @@ Use `execute_as` when a workflow needs permissions or credentials that differ fr
 Two ways to set `execute_as`:
 
 1. **Workflow config** — set `execute_as` in `lt_config_workflows`. Every invocation of that workflow runs as the specified service account.
-2. **Per-request override** — pass `executeAs` in the invocation payload. Requires admin or superadmin role.
+2. **Per-request override** — pass `execute_as` in the invocation body. A superadmin may act as any account. A caller holding an admin-type grant may act as an account that holds no superadmin grant and whose every role the caller also holds, at the same or a higher type. No other caller may set it.
 
 In both cases, the original invoker is preserved in `initiated_by`. The audit trail always shows both who asked and who executed.
 
 ### Lifecycle
 
 ```bash
-# Create a service account (admin-only)
+# Create a service account (builder: superadmin or the engineer role)
 POST /api/bot-accounts
 { "name": "data-bot", "description": "Nightly analytics" }
 
@@ -287,13 +287,15 @@ The MCP client also emits debug-level audit logs for every tool invocation:
 | Credential | Scope | Lifetime | Revocation |
 |---|---|---|---|
 | User JWT | Full API (RBAC-scoped) | 24 hours | Logout / expiry |
-| Bot API key | Full API (RBAC-scoped) | Until revoked | Admin deletes key |
+| Bot API key | Full API (RBAC-scoped) | Until revoked | A builder revokes the key |
 | Delegation token | Specific scopes | 5 min (max 1 hr) | Expires naturally |
 | OAuth token | Provider scopes | Provider-set | User revokes in Connections |
 
-Service accounts inherit the same RBAC constraints as human users. A service account with the `member` role cannot access admin endpoints. A service account without `engineer` cannot invoke workflows gated by `invocation_roles`.
+Service accounts inherit the same RBAC constraints as human users. A service account with the `member` role cannot access admin endpoints. A service account that holds none of a workflow's `invocation_roles` cannot invoke it.
 
-The `execute_as` override requires admin role. There is no way for a non-admin to impersonate a service account through the API.
+Invocation roles are checked against the caller's current grants in the database, never the role claims a token carries. A workflow with an empty `invocation_roles` list is open to every authenticated caller. A superadmin, or a holder of the `admin` role at admin type, invokes any invocable workflow. `capabilityInvoke`, which calls tools with system authority, is invocable by superadmins only. Identity context for a compiled workflow comes from the server: a caller-supplied `_scope` in the input data is ignored.
+
+The `execute_as` override never exceeds the caller's own authority. A superadmin may act as any account; a caller holding an admin-type grant may act as a non-superadmin account all of whose roles the caller holds at the same or a higher type. Anyone else is refused with `403`.
 
 ## Key Files
 

package/docs/maintenance.md

@@ -2,19 +2,26 @@
 
 HotMesh stores execution artifacts in PostgreSQL: stream messages that carry internal signals between activities, transient job rows that track bookkeeping for individual activity invocations, and entity job rows that represent durable workflow instances. Left unmanaged, these tables grow without bound. The maintenance system provides scheduled, rule-based cleanup that runs as a durable cron workflow inside HotMesh itself.
 
-A default configuration ships with Long Tail. It runs nightly at 2 AM, applies four rules in sequence, and requires no setup. The schedule and rules can be replaced at startup or at runtime through the REST API.
+Maintenance is opt-in. A default configuration ships with Long Tail: pass `maintenance: true` to `start()` and it runs weekly, Sundays at 2 AM, applying five rules in sequence. A config object of your own replaces it, and the rules can be replaced at runtime through the REST API.
 
 ## Configuration via start()
 
-Maintenance is enabled by default when you call `start()`. To customize or disable it:
+`start()` registers maintenance only when its `maintenance` option asks for it:
+
+| `maintenance` | Effect |
+|---|---|
+| omitted or `false` | No maintenance runs. |
+| `true` | The default configuration (see [Default schedule](#default-schedule)). |
+| an `LTMaintenanceConfig` | That configuration, in place of the default. |
 
 ```typescript
 import { start } from '@hotmeshio/long-tail';
 
-// Default: nightly 2 AM cleanup (no config needed)
+// Default: Sundays at 2 AM
 await start({
   database: { connectionString: process.env.DATABASE_URL },
   workers: [ ... ],
+  maintenance: true,
 });
 
 // Custom schedule and rules
@@ -22,6 +29,7 @@ await start({
   database: { connectionString: process.env.DATABASE_URL },
   workers: [ ... ],
   maintenance: {
+    appId: 'durable',
     schedule: '0 3 * * *',
     rules: [
       { target: 'streams', olderThan: '24 hours', action: 'delete' },
@@ -29,21 +37,23 @@ await start({
       { target: 'jobs',    olderThan: '14 days',  action: 'prune',  hasEntity: true },
       { target: 'jobs',    olderThan: '180 days', action: 'delete', pruned: true },
       { target: 'escalations', olderThan: '180 days', action: 'delete' },
+      { target: 'oauth',       olderThan: '7 days',   action: 'delete' },
     ],
   },
 });
 
-// Disabled entirely
+// No maintenance: omit the option (or pass false)
 await start({
   database: { connectionString: process.env.DATABASE_URL },
   workers: [ ... ],
-  maintenance: false,
 });
 ```
 
+Maintenance starts once workers start, and not on a readonly instance.
+
 ## What gets cleaned
 
-Five categories of data are subject to maintenance:
+Six categories of data are subject to maintenance:
 
 | Category | Description |
 |---|---|
@@ -52,6 +62,7 @@ Five categories of data are subject to maintenance:
 | **Entity jobs** | Job rows where `entity` is set. These represent actual workflow instances -- the rows that back `Durable.Client.workflow.search()` calls and execution exports. Deleting them removes the workflow record entirely. |
 | **Pruned jobs** | Entity jobs that have already had their execution artifacts stripped (the `pruned_at` column is not `NULL`). These retain core data but no longer carry execution scaffolding. They can be hard-deleted after a longer retention window. |
 | **Terminal escalations** | Escalation rows in a terminal status (`resolved`, `cancelled`, `expired`). Terminal rows are inert audit records -- every engine state transition guards on `status = 'pending'`, so aging them out is safe for live waiters, claims, and signal delivery. The engine-owned prune deletes in atomic batches until the retention horizon is drained. Pending rows are never touched. |
+| **OAuth server rows** | The `/mcp` authorization server's records. Refresh tokens and authorization codes past expiry, or rotated out, longer ago than the window; then grants left with neither, or revoked longer ago than the window; then clients registered longer ago than the window with no live grant. A client still holding a live refresh token keeps its grant and its registration. |
 
 ## Prune vs. delete
 
@@ -64,26 +75,31 @@ The intended lifecycle for an entity job is: execute, then prune after a short r
 
 ## Default schedule
 
-The built-in configuration (`modules/maintenance.ts`) runs at 2 AM daily and applies four rules in order:
+The default configuration (`modules/maintenance.ts`, enabled with `maintenance: true`) runs Sundays at 2 AM and applies five rules in order:
 
 ```typescript
 {
-  schedule: '0 2 * * *',
+  appId: 'durable',
+  schedule: '0 2 * * 0',
   rules: [
-    { target: 'streams', action: 'delete', olderThan: '7 days' },
-    { target: 'jobs', action: 'delete', olderThan: '7 days', hasEntity: false },
-    { target: 'jobs', action: 'prune',  olderThan: '7 days', hasEntity: true },
-    { target: 'jobs', action: 'delete', olderThan: '90 days', pruned: true },
+    { target: 'streams', action: 'delete', olderThan: '30 days' },
+    { target: 'jobs', action: 'delete', olderThan: '30 days', hasEntity: false },
+    { target: 'jobs', action: 'prune',  olderThan: '30 days', hasEntity: true },
+    { target: 'jobs', action: 'delete', olderThan: '180 days', pruned: true },
+    { target: 'oauth', action: 'delete', olderThan: '7 days' },
   ],
 }
 ```
 
 | Rule | Effect |
 |---|---|
-| 1. Delete streams older than 7 days | Removes internal message data that is no longer needed for replay or debugging. |
-| 2. Delete transient jobs older than 7 days | Removes activity-level bookkeeping rows that have no associated workflow entity. |
-| 3. Prune entity jobs older than 7 days | Strips execution artifacts from workflow instances while preserving core data. Exports and search remain functional. |
-| 4. Delete pruned jobs older than 90 days | Hard-deletes workflow instances that were pruned at least 90 days ago. This is the final removal. |
+| 1. Delete streams older than 30 days | Removes internal message data that is no longer needed for replay or debugging. |
+| 2. Delete transient jobs older than 30 days | Removes activity-level bookkeeping rows that have no associated workflow entity. |
+| 3. Prune entity jobs older than 30 days | Strips execution artifacts from workflow instances while preserving core data. Exports and search remain functional. |
+| 4. Delete pruned jobs older than 180 days | Hard-deletes workflow instances that were pruned at least 180 days ago. This is the final removal. |
+| 5. Delete OAuth server rows older than 7 days | Removes expired codes and tokens, unused or revoked grants, and clients with no live grant. |
+
+A custom `rules` list replaces the defaults, so include the `oauth` rule when the OAuth server is enabled.
 
 Rules execute sequentially. If one fails, the remaining rules still run; failures are logged and do not halt the cycle.
 
@@ -95,7 +111,7 @@ Defined in `types/maintenance.ts`. Each rule describes a single cleanup operatio
 
 | Field | Type | Required | Description |
 |---|---|---|---|
-| `target` | `'streams' \| 'jobs' \| 'escalations'` | Yes | The resource type to act on. |
+| `target` | `'streams' \| 'jobs' \| 'escalations' \| 'oauth'` | Yes | The resource type to act on. |
 | `action` | `'delete' \| 'prune'` | Yes | Whether to remove entirely or strip execution artifacts. |
 | `olderThan` | `string` | Yes | A PostgreSQL interval expression: `'7 days'`, `'24 hours'`, `'90 days'`. |
 | `hasEntity` | `boolean` | No | When `target` is `'jobs'`: `true` selects entity jobs, `false` selects transient jobs (where `entity IS NULL`). |
@@ -106,6 +122,7 @@ Defined in `types/maintenance.ts`. Each rule describes a single cleanup operatio
 
 | Field | Type | Description |
 |---|---|---|
+| `appId` | `string` | The HotMesh application namespace to prune. The default configuration uses `'durable'`. |
 | `schedule` | `string` | A cron expression (`'0 2 * * *'`) or an interval string (`'1 day'`). Passed directly to `Virtual.cron` as the `interval` option. |
 | `rules` | `LTMaintenanceRule[]` | An ordered array of rules. Executed sequentially on each cron tick. |
 
@@ -117,14 +134,14 @@ The `maintenanceRegistry` singleton (`services/maintenance/index.ts`) follows th
 |---|---|
 | `register(config)` | Store a maintenance configuration. Replaces any previously registered config. Call before `connect()`. |
 | `connect()` | Start the `Virtual.cron` workflow that fires on the configured schedule. Idempotent -- given the same internal cron ID, duplicate calls do not create duplicate schedules. |
-| `disconnect()` | Cancel the running cron by calling `Virtual.interrupt`. Safe to call if no cron is running. |
+| `disconnect()` | Mark this registry disconnected, on graceful shutdown or before a reconfiguration. The durable cron job is left in place: it is shared by every container, and another container (or this one after a restart) keeps servicing it. |
 | `clear()` | Remove the config and reset internal state. Used in tests. |
 | `hasConfig` | Boolean property. `true` when a config has been registered. |
 | `config` | Returns the current `LTMaintenanceConfig` or `null`. |
 
 ## Programmatic registration
 
-For advanced use cases, you can register maintenance programmatically instead of through `start()`. The `start()` function handles this automatically when a `maintenance` config is provided (or uses the default when omitted):
+For advanced use cases, you can register maintenance programmatically instead of through `start()`. `start()` does this itself when `maintenance` is `true` (the default configuration) or a config object:
 
 ```typescript
 import { maintenanceRegistry, defaultMaintenanceConfig } from '@hotmeshio/long-tail';
@@ -138,6 +155,7 @@ To use a custom config instead, register it in place of the default:
 import { maintenanceRegistry } from '@hotmeshio/long-tail';
 
 maintenanceRegistry.register({
+  appId: 'durable',
   schedule: '0 3 * * 0',  // Sundays at 3 AM
   rules: [
     { target: 'streams', action: 'delete', olderThan: '14 days' },
@@ -148,7 +166,7 @@ maintenanceRegistry.register({
 });
 ```
 
-The registry accepts exactly one config at a time. Calling `register()` again overwrites the previous config. The cron does not start until `connect()` is called — which happens inside `start()` when `hasConfig` is `true`.
+The registry accepts exactly one config at a time. Calling `register()` again overwrites the previous config. The cron does not start until `connect()` is called, which `start()` does when `hasConfig` is `true`.
 
 ## Runtime API
 
@@ -156,12 +174,13 @@ The REST API allows administrators to replace the maintenance config while the s
 
 ### `GET /api/config/maintenance`
 
-Returns the current configuration and whether a cron is active.
+Returns the current configuration. `active` is `true` when a configuration is registered; `config` is `null` and `active` is `false` when none is.
 
 ```json
 {
   "config": {
-    "schedule": "0 2 * * *",
+    "appId": "durable",
+    "schedule": "0 2 * * 0",
     "rules": [ ... ]
   },
   "active": true
@@ -174,6 +193,7 @@ Admin-only (requires `requireAdmin` middleware). Accepts a full replacement conf
 
 ```json
 {
+  "appId": "durable",
   "schedule": "0 4 * * *",
   "rules": [
     { "target": "streams", "action": "delete", "olderThan": "3 days" },
@@ -186,11 +206,13 @@ Admin-only (requires `requireAdmin` middleware). Accepts a full replacement conf
 
 The endpoint performs three operations in sequence:
 
-1. **Disconnect** -- calls `maintenanceRegistry.disconnect()` to cancel the running cron via `Virtual.interrupt`.
+1. **Disconnect** -- calls `maintenanceRegistry.disconnect()`.
 2. **Register** -- calls `maintenanceRegistry.register()` with the new config.
-3. **Connect** -- calls `maintenanceRegistry.connect()` to start a new cron with the updated schedule and rules.
+3. **Connect** -- calls `maintenanceRegistry.connect()`, which registers this container's cron worker with the new rules.
+
+The new rules apply to the ticks this container runs. The configuration is held in memory: other containers keep the rules they started with, and a restart returns this one to its `start()` configuration. The schedule is fixed when the cron job is first created: an existing job keeps its schedule (see [How it works internally](#how-it-works-internally)).
 
-Returns the new config and `{ "restarted": true }` on success. Returns `400` if `schedule` or `rules` is missing, and `500` if any step fails.
+Returns the new config and `{ "restarted": true }` on success. Returns `400` if `appId`, `schedule` or `rules` is missing, and `500` if any step fails.
 
 ## Custom schedule example
 
@@ -200,6 +222,7 @@ A production deployment that processes high volumes might want aggressive short-
 import { maintenanceRegistry } from '@hotmeshio/long-tail';
 
 maintenanceRegistry.register({
+  appId: 'durable',
   schedule: '0 */6 * * *',  // every 6 hours
   rules: [
     { target: 'streams', action: 'delete', olderThan: '24 hours' },
@@ -217,8 +240,8 @@ This cleans streams and transient jobs within two days, prunes entity jobs after
 The maintenance system runs on top of HotMesh's `Virtual.cron`, the same primitive that powers durable recurring workflows throughout the platform.
 
 1. **Registration.** `maintenanceRegistry.register(config)` stores the config in memory. No I/O occurs.
-2. **Connection.** `maintenanceRegistry.connect()` calls `Virtual.cron()` with a fixed topic (`lt.maintenance.prune`) and a fixed cron ID (`lt-maintenance-nightly`). HotMesh persists the cron schedule in PostgreSQL, making it durable across restarts. If a cron with that ID already exists, it is replaced.
-3. **Execution.** On each tick of the schedule, the cron callback iterates the rules array. For each rule, it translates the rule's fields into the appropriate `dbaService.prune()` parameters and executes the call. Rules run sequentially; a failure in one rule is logged and does not prevent subsequent rules from running.
-4. **Disconnection.** `maintenanceRegistry.disconnect()` calls `Virtual.interrupt()` with the same topic and cron ID, which cancels the recurring schedule. This is called during graceful shutdown and before reconfiguration via the REST API.
+2. **Connection.** `maintenanceRegistry.connect()` calls `Virtual.cron()` with a fixed topic (`lt.maintenance.prune`) and a fixed cron ID (`lt-maintenance-nightly`). This registers the container's worker for the topic and starts the cron job. HotMesh persists the job in PostgreSQL, making it durable across restarts. If a job with that ID already exists, it keeps running on the schedule it was created with, and no second job starts.
+3. **Execution.** On each tick of the schedule, the worker that takes the tick runs its rules in order. Stream and job rules call `dbaService.prune()`; escalation rules call the escalation client's batched prune until the window is drained; the `oauth` rule calls the OAuth server prune. A failure in one rule is logged and does not prevent subsequent rules from running.
+4. **Disconnection.** `maintenanceRegistry.disconnect()` marks the registry disconnected, during graceful shutdown and before reconfiguration via the REST API. It does not interrupt the cron job, which other containers keep servicing.
 
 Because the cron is itself a durable workflow, the schedule survives process restarts. If the server goes down at 2 AM and comes back at 2:05 AM, the missed tick executes on reconnection.