pipework 0.176.0 → 0.177.0

package/CHANGELOG.md

@@ -1,5 +1,13 @@
 # Changelog
 
+## 0.177.0
+
+- **Added: alert sources for schedules, step graphs, pipelines and traces** (failure-alert class, part 3 of 4):
+  - `alerts.source.schedules({ graceMs? })` raises `schedule-overdue:<name>` while a schedule's next run is more than 5 min past, which means no scheduler is firing it.
+  - `alerts.source.steps({ notifyAfterMs?, waitingAfterMs?, runningAfterMs? })` raises `step-run-failed:<graph>` for runs that failed since the last tick, `step-run-unnotified:<graph>` while finished runs were never notified, `step-task-waiting:<graph>` while a task waits on a person too long, and `step-run-stalled:<graph>` while a run stays running with nothing waiting on a person.
+  - `alerts.source.pipelines({ waitingAfterMs?, runningAfterMs? })` raises `pipeline-failed:<pipeline>`, `pipeline-waiting:<pipeline>` and `pipeline-stalled:<pipeline>`.
+  - `alerts.source.traces({ graceMs? })` raises `trace-expired:<entryKind>` for traces the expiry sweep force-closed, and `trace-expiry-undrained:all` while open traces are more than 26 h past expiry (the expiry job is not running).
+
 ## 0.176.0
 
 - **Added: alert sources for mail, storage, access links and sign-in lockouts** (failure-alert class, part 2 of 3):

package/REFERENCE.md

@@ -1754,6 +1754,10 @@ Pass in `sources`; each raises with a fixed key on a tick and, where it can tell
 - `alerts.source.storage({ sweepGraceMs? = 3_600_000, severity?, owner?, runbook?, database? })` — needs `storage: {}`. `storage-expiry-undrained:all` while `storage_file` rows are `sweepGraceMs` past `expires_at` and not swept.
 - `alerts.source.accessLinks({ graceMs? = 900_000, severity?, owner?, runbook?, database? })` — needs `accessLinks: {}`. `access-link-abandoned:<tenantId>` while uses stay `pending` `graceMs` past `reserved_until`.
 - `alerts.source.lockouts({ tables? = the ones that exist, severity? = 'warning', owner?, runbook?, database? })` — `auth-lockout:<table>` while rows have `locked_until` in the future (count only; no account is named). `pipework_signin_attempts` needs `passwords: {}`, `pipework_guard_attempts` needs `guards: {}`; with `tables` omitted, the tables that exist are resolved once on the first evaluate (it throws, naming both keys, if neither does); pass `tables` to override.
+- `alerts.source.schedules({ graceMs? = 300_000, severity?, owner?, runbook?, database? })` — `schedule-overdue:<name>` while a schedule's `next_run_at` is more than `graceMs` past (every scheduler pass moves it forward, firing or skipping, so overdue means no scheduler is running); resolves when it catches up. Needs a worker surface with schedules.
+- `alerts.source.steps({ notifyAfterMs? = 900_000, waitingAfterMs? = 3_600_000, runningAfterMs? = 3_600_000, settleMs?, severity?, owner?, runbook?, database? })` — `step-run-failed:<graph>` for runs that failed since the last tick (context `count`, `lastError`, `lastRunId`, `lastTenantId`; clears by the quiet period); `step-run-unnotified:<graph>` while runs finished more than `notifyAfterMs` ago still have `notified = false`; `step-task-waiting:<graph>` while a task has waited for a person's decision more than `waitingAfterMs` (measured from its run's `created_at`); `step-run-stalled:<graph>` while runs are still `running` more than `runningAfterMs` after `created_at` with no task `waiting` (`blocked` and `queued` tasks are work in flight). Needs a step graph.
+- `alerts.source.pipelines({ waitingAfterMs? = 3_600_000, runningAfterMs? = 3_600_000, settleMs?, severity?, owner?, runbook?, database? })` — `pipeline-failed:<pipeline>` for runs that failed since the last tick (context `count`, `lastError`, `lastRunId`, `lastTenantId`); `pipeline-waiting:<pipeline>` while runs have been `waiting` more than `waitingAfterMs` since they started (`started_at`; no wait time is recorded); `pipeline-stalled:<pipeline>` while runs are `running` more than `runningAfterMs` after `started_at`. Needs the app's `pipework_pipeline_runs` table.
+- `alerts.source.traces({ graceMs? = 93_600_000, settleMs?, severity?, owner?, runbook?, database? })` — `trace-expired:<entryKind>` for traces the expiry sweep force-closed since the last tick (read from `closed_at` where `status = 'expired'`); `trace-expiry-undrained:all` while open traces are more than `graceMs` (default 26 h, as the maintenance job is daily) past `expires_at`, meaning the expiry job is not running. Needs the `trace` block in the blueprint.
 - `queue.totals(db)` — `{ overall, byType }`, each `{ ready, running, oldestReadyWaitMs }` (ready: pending and not waiting on a retry time; running: claimed or running; wait from `queued_at`, null when none is ready); `byType` omits a type with neither.
 - `alerts.source.health(checks, { severity? = 'error' })` — `health:<name>` while a `HealthCheck` is unhealthy or throws; resolves when it recovers. Degraded is not a fault.
 

package/dist/REFERENCE.md

@@ -1,7 +1,7 @@
 # Pipework API Reference
 
 > Auto-generated from source JSDoc. Do not edit manually.
-> Generated: 2026-10-20
+> Generated: 2026-10-22
 
 ## Contents
 
@@ -816,6 +816,10 @@ Acquires a transaction-scoped advisory lock. Released at commit/rollback.
 
 Runs a function in a SERIALIZABLE transaction with automatic retry on serialization failures and deadlocks.
 
+### `pipe.withTimeouts`
+
+Sets statement_timeout and/or lock_timeout for the current transaction in one statement, then runs fn; a statement cancelled for them rejects with StatementTimeoutError or LockTimeoutError.
+
 ### `pipe.bulkSet`
 
 Bulk UPDATE ... FROM (VALUES ...) — updates multiple rows with per-row values in a single statement.
@@ -824,6 +828,10 @@ Bulk UPDATE ... FROM (VALUES ...) — updates multiple rows with per-row values
 
 Produces a typed VALUES clause for bulk operations — the base primitive for bulkSet and bulkUpsert.
 
+### `pipe.fn`
+
+A set-returning SQL function as a typed source for `.from()` and joins — `pipe.fn('schema.f', { args: [field.uuid()], columns: { path: field.text() } })`. Arguments are bound and cast to their declared types.
+
 ### `pipe.field`
 
 Field builders for domain definitions — uuid(), text(), integer(), etc.
@@ -836,6 +844,10 @@ Registers a domain definition with table, validators, and factory projections. T
 
 Returns the global registry of all domain definitions.
 
+### `pipe.external`
+
+Declares a table or view another service owns (`pipe.external(name, fields, { schema, table? })`): typed columns and every query operator, read-only, never migrated, not tenant-scoped.
+
 ### `pipe.update`
 
 Pipework-owned UPDATE builder — the only consumer-facing way to update pipework-managed tables (Q8).
@@ -866,12 +878,54 @@ interface WindowSpec
 interface ValueColumnDef
 ```
 
+**`JsonbElements`** — A jsonb array read as its elements' text. Produced by {@link jsonbElements}.
+
+```typescript
+interface JsonbElements
+```
+
+**`ArraySource`** — Something whose elements `arrayAny` / `arrayAll` can test.
+
+```typescript
+type ArraySource =
+```
+
+**`ElementOf`** — The element type of an {@link ArraySource}.
+
+```typescript
+type ElementOf<S> =
+```
+
+**`FunctionSpec`** — Declares a SQL function's argument and result-column shapes. See {@link fn}.
+
+```typescript
+interface FunctionSpec<TArgs extends readonly AnyField[], TColumns extends Record<string, AnyField>>
+```
+
+**`FunctionSourceFields`** — The selected fields of a function source: each result column as a typed SQL expression.
+
+```typescript
+type FunctionSourceFields<TColumns extends Record<string, AnyField>> =
+```
+
+**`FunctionSource`** — A function call usable in `.from()` and joins; its result columns are properties of it.
+
+```typescript
+type FunctionSource<TAlias extends string, TColumns extends Record<string, AnyField>> =
+```
+
 **`SerializableOptions`** — Retry policy for {@link serializable}: how many attempts on a serialization failure / deadlock, and the base backoff between them.
 
 ```typescript
 interface SerializableOptions
 ```
 
+**`TimeoutOptions`** — Which timeouts {@link withTimeouts} sets; each a positive integer number of milliseconds.
+
+```typescript
+interface TimeoutOptions
+```
+
 **`ManagedConnection`** — A standalone pooled connection: the raw postgres.js client, its drizzle handle, and a `close()` that ends the pool.
 
 ```typescript
@@ -892,6 +946,10 @@ JSONB query operators — typed helpers for PostgreSQL JSONB containment, key ch
 
 The @> containment operator — tests whether a JSONB column contains the given value.
 
+### `jsonb.merge`
+
+Merges an object into a JSONB column in `.set()`: `coalesce(col, '{}') || value`. Top-level keys overwrite; others are kept.
+
 ### `jsonb.containedBy`
 
 The <@ contained-by operator — tests whether a JSONB column is contained by the given value.
@@ -958,6 +1016,60 @@ type Schema<T = unknown> = z.ZodType<T>
 type Infer<T extends Schema> = z.infer<T>
 ```
 
+**`StringSchema`** — A string validator — what check.string() returns, nameable without importing zod.
+
+```typescript
+interface StringSchema extends z.ZodString
+```
+
+**`NumberSchema`** — A number validator — what check.number() returns.
+
+```typescript
+interface NumberSchema extends z.ZodNumber
+```
+
+**`BooleanSchema`** — A boolean validator — what check.boolean() returns.
+
+```typescript
+interface BooleanSchema extends z.ZodBoolean
+```
+
+**`EnumSchema`** — An enum validator over string literals — what check.enum([...]) returns.
+
+```typescript
+interface EnumSchema<V extends string = string> extends z.ZodEnum<
+```
+
+**`LiteralSchema`** — A literal validator — what check.literal(v) returns.
+
+```typescript
+interface LiteralSchema<V extends string | number | boolean | null = string> extends z.ZodLiteral<V>
+```
+
+**`ArraySchema`** — An array validator — what check.array(item) returns.
+
+```typescript
+interface ArraySchema<Item extends z.core.SomeType = z.core.SomeType> extends z.ZodArray<Item>
+```
+
+**`OptionalSchema`** — An optional validator — what check.optional(inner) returns.
+
+```typescript
+interface OptionalSchema<Inner extends z.core.SomeType = z.core.SomeType> extends z.ZodOptional<Inner>
+```
+
+**`NullableSchema`** — A nullable validator — what check.nullable(inner) returns.
+
+```typescript
+interface NullableSchema<Inner extends z.core.SomeType = z.core.SomeType> extends z.ZodNullable<Inner>
+```
+
+**`ObjectSchema`** — An object validator whose .extend, .pick, .omit and .partial results stay ObjectSchema — what check.object({...}) returns.
+
+```typescript
+type ObjectSchema<S extends Shape = Shape> =
+```
+
 ## temporal — Bitemporal Data
 
 Temporal (bitemporal) data namespace — version history with effective date ranges.
@@ -1474,7 +1586,11 @@ interface RateLimitConfig
 interface RateLimitWindow
 ```
 
-**`RateLimitClass`** — A named limit class: any of a per-user, per-tenant and per-address window, each counted on its own.
+**`RateLimitClass`**
+
+A named limit class: any of a per-user, per-tenant, per-address and total window, each counted on its own.
+Each key takes one window or an array of windows (distinct `windowSeconds`); a request is counted against
+all of a key's windows and refused if any is over.
 
 ```typescript
 interface RateLimitClass
@@ -1490,8 +1606,8 @@ interface RateLimitRefusal
 
 Named limit classes, checked after auth and counted in Postgres (`pipework_rate_limit_counts`, opted in
 with the blueprint's `rateLimits: {}`), so every server process on the same database shares one count.
-Windows are counted perUser, perTenant, perAddress in turn, up to the first one over its max, so one user's
-refused requests don't spend the tenant's budget. A route is in `signedIn` (auth required, or optional with
+Keys are counted perUser, perTenant, perAddress, total in turn, up to the first key with a window over its max, so one user's
+refused requests don't spend the tenant's budget. The 429's Retry-After is the longest wait of the windows that refused. A route is in `signedIn` (auth required, or optional with
 a user), `public` (no auth, or optional without a user) or the class its fitting names with `.rateLimit(name)`.
 
 ```typescript
@@ -1643,15 +1759,20 @@ interface TraceTree
 
 ## data
 
-### `tap`
+### `StatementTimeoutError`
+
+A statement was cancelled by Postgres for running past the statement timeout a {@link withTimeouts} call set. `cause` is the driver error.
+
+```typescript
+class StatementTimeoutError extends Error constructor(readonly statementTimeoutMs: number, cause: unknown)
+```
+
+### `LockTimeoutError`
 
-Opens a standalone managed connection outside the manifold's pools — the
-sanctioned door for contexts where `pipe()` structurally cannot resolve:
-`worker_threads`, detached monitors, one-off scripts. Callers own the
-lifecycle: always `await conn.close()`.
+A statement gave up waiting for a lock past the lock timeout a {@link withTimeouts} call set. `cause` is the driver error.
 
 ```typescript
-function tap(url: string, options?: ConnectOptions): ManagedConnection
+class LockTimeoutError extends Error constructor(readonly lockTimeoutMs: number, cause: unknown)
 ```
 
 ## request

package/dist/infra/alerts/async-sources.d.ts

@@ -0,0 +1,54 @@
+import { type AlertSource } from './sources.js';
+import { type Basics } from './source-options.js';
+/**
+ * Raises `schedule-overdue:<schedule name>` while a cron schedule's `next_run_at` is more than `graceMs` (default 5 min)
+ * in the past. Every scheduler pass moves `next_run_at` into the future, whether it fires the job or skips it because
+ * the previous run is still pending, so an overdue row means no scheduler is running (or it cannot reach the row).
+ * Resolves when the scheduler catches up. Needs a worker surface that declares schedules (the `pipework_schedules` table; no blueprint key).
+ */
+export declare function schedulesSource(options?: Basics & {
+    graceMs?: number;
+}): AlertSource;
+/**
+ * Raises `step-run-failed:<graph>` for step runs that failed since the last tick (message with the count and the last
+ * error; context `count`, `lastError`, `lastRunId`, `lastTenantId`), `step-run-unnotified:<graph>` while runs finished
+ * more than `notifyAfterMs` (default 15 min) ago still have `notified = false` (the graph's `onDone` has not run), and
+ * `step-task-waiting:<graph>` while a task has waited for a person's decision more than `waitingAfterMs` (default 1 h);
+ * the wait is measured from its run's `created_at`, as tasks carry no timestamp, and `step-run-stalled:<graph>` while runs
+ * are still `running` more than `runningAfterMs` (default 1 h) after `created_at` with no task `waiting` (a `waiting` task is a
+ * person's decision and is covered above; `blocked` and `queued` tasks are work in flight, so a run stuck on them is stalled). A failed task always fails its run in the
+ * same transaction, so failed runs cover failed tasks. The failed condition clears by the handle's quiet period; the other two
+ * resolve when the rows drain. Needs a step graph defined in the app (the `pipework_step_run` and `pipework_step_task` tables; no blueprint key).
+ */
+export declare function stepsSource(options?: Basics & {
+    notifyAfterMs?: number;
+    waitingAfterMs?: number;
+    runningAfterMs?: number;
+    settleMs?: number;
+}): AlertSource;
+/**
+ * Raises `pipeline-failed:<pipeline name>` for pipeline runs that failed since the last tick (context `count`,
+ * `lastError`, `lastRunId`, `lastTenantId`), and `pipeline-waiting:<pipeline name>` while runs sit in `waiting` more than
+ * `waitingAfterMs` (default 1 h) after they started (the table keeps no time for when a run began to wait, so the age is
+ * `started_at`), and `pipeline-stalled:<pipeline name>` while runs are `running` more than `runningAfterMs` (default 1 h)
+ * after `started_at` (the process died mid-run). The failed condition clears by the handle's quiet period; the others
+ * resolve when the runs resume or finish. Needs the app's own `pipework_pipeline_runs` table (pipelines have no blueprint key; the app creates it).
+ */
+export declare function pipelinesSource(options?: Basics & {
+    waitingAfterMs?: number;
+    runningAfterMs?: number;
+    settleMs?: number;
+}): AlertSource;
+/**
+ * Raises `trace-expired:<entry kind>` (http, job, cron, test) for traces the expiry sweep force-closed since the last
+ * tick: the sweep stamps `status = 'expired'` and `closed_at`, so `closed_at` is the cursor. Also raises
+ * `trace-expiry-undrained:all` while traces still `open` have an `expires_at` more than `graceMs` (default 26 h: the maintenance job that
+ * runs the sweep is daily, so a trace legitimately stays open up to a day past its deadline) in the past, which means
+ * nothing is running the expiry job. The expired condition clears by the handle's quiet period; the
+ * undrained one resolves once the sweep runs. Needs the blueprint's `trace` block (the `trace` table).
+ */
+export declare function tracesSource(options?: Basics & {
+    graceMs?: number;
+    settleMs?: number;
+}): AlertSource;
+//# sourceMappingURL=async-sources.d.ts.map

package/dist/infra/alerts/async-sources.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"async-sources.d.ts","sourceRoot":"","sources":["../../../src/infra/alerts/async-sources.ts"],"names":[],"mappings":"AAIA,OAAO,EAA8C,KAAK,WAAW,EAAE,MAAM,cAAc,CAAA;AAC3F,OAAO,EAA6C,KAAK,MAAM,EAAE,MAAM,qBAAqB,CAAA;AAuB5F;;;;;GAKG;AACH,wBAAgB,eAAe,CAAC,OAAO,GAAE,MAAM,GAAG;IAAE,OAAO,CAAC,EAAE,MAAM,CAAA;CAAO,GAAG,WAAW,CAoBxF;AAED;;;;;;;;;;GAUG;AACH,wBAAgB,WAAW,CAAC,OAAO,GAAE,MAAM,GAAG;IAAE,aAAa,CAAC,EAAE,MAAM,CAAC;IAAC,cAAc,CAAC,EAAE,MAAM,CAAC;IAAC,cAAc,CAAC,EAAE,MAAM,CAAC;IAAC,QAAQ,CAAC,EAAE,MAAM,CAAA;CAAO,GAAG,WAAW,CAiE/J;AAED;;;;;;;GAOG;AACH,wBAAgB,eAAe,CAAC,OAAO,GAAE,MAAM,GAAG;IAAE,cAAc,CAAC,EAAE,MAAM,CAAC;IAAC,cAAc,CAAC,EAAE,MAAM,CAAC;IAAC,QAAQ,CAAC,EAAE,MAAM,CAAA;CAAO,GAAG,WAAW,CA+C3I;AAED;;;;;;;GAOG;AACH,wBAAgB,YAAY,CAAC,OAAO,GAAE,MAAM,GAAG;IAAE,OAAO,CAAC,EAAE,MAAM,CAAC;IAAC,QAAQ,CAAC,EAAE,MAAM,CAAA;CAAO,GAAG,WAAW,CA+BxG"}

package/dist/infra/alerts/async-sources.js

@@ -0,0 +1,233 @@
+// Sources over the failures the async subsystems already record: cron schedules, step graphs, pipelines and
+// trace expiry. Each is a composition of `querySource` and `newRowsSource`.
+import { sql } from '../../data/db/sql.js';
+import { composeSources, newRowsSource, querySource } from './sources.js';
+import { common, positive, text, FIFTEEN_MIN, HOUR } from './source-options.js';
+const FIVE_MIN = 5 * 60_000;
+const TRACE_SWEEP_GRACE = 26 * HOUR;
+/** The `error` jsonb of a recorded failure as one line: its `message`, `error` or `step`/`error` pair, else the JSON. */
+function errorText(v) {
+    if (v === null || v === undefined)
+        return null;
+    const parsed = typeof v === 'string' ? safeParse(v) : v;
+    if (typeof parsed === 'string')
+        return parsed;
+    if (typeof parsed === 'object' && parsed !== null) {
+        const o = parsed;
+        const inner = o['message'] ?? o['error'];
+        if (typeof inner === 'string')
+            return typeof o['step'] === 'string' ? `${o['step']}: ${inner}` : inner;
+        if (typeof inner === 'object' && inner !== null)
+            return errorText(inner);
+    }
+    return JSON.stringify(parsed);
+}
+function safeParse(s) {
+    try {
+        return JSON.parse(s);
+    }
+    catch {
+        return s;
+    }
+}
+/**
+ * Raises `schedule-overdue:<schedule name>` while a cron schedule's `next_run_at` is more than `graceMs` (default 5 min)
+ * in the past. Every scheduler pass moves `next_run_at` into the future, whether it fires the job or skips it because
+ * the previous run is still pending, so an overdue row means no scheduler is running (or it cannot reach the row).
+ * Resolves when the scheduler catches up. Needs a worker surface that declares schedules (the `pipework_schedules` table; no blueprint key).
+ */
+export function schedulesSource(options = {}) {
+    const grace = positive(options.graceMs ?? FIVE_MIN, 'graceMs', 'alerts.source.schedules');
+    return querySource({
+        ...common(options),
+        name: 'schedule-overdue',
+        query: sql `
+      SELECT name, next_run_at, last_run_at, (extract(epoch FROM clock_timestamp() - next_run_at) * 1000)::bigint AS overdue_ms
+      FROM pipework_schedules
+      WHERE next_run_at < clock_timestamp() - ${grace} * interval '1 millisecond'
+      ORDER BY name`,
+        id: r => String(r['name']),
+        message: r => `schedule ${String(r['name'])} is ${String(r['overdue_ms'])} ms past its next run; is the scheduler running?`,
+        subject: r => ({ entity: 'schedule', entityId: String(r['name']) }),
+        context: r => ({
+            schedule: String(r['name']),
+            overdueMs: Number(r['overdue_ms']),
+            nextRunAt: new Date(String(r['next_run_at'])).toISOString(),
+            lastRunAt: r['last_run_at'] === null ? null : new Date(String(r['last_run_at'])).toISOString(),
+        }),
+    });
+}
+/**
+ * Raises `step-run-failed:<graph>` for step runs that failed since the last tick (message with the count and the last
+ * error; context `count`, `lastError`, `lastRunId`, `lastTenantId`), `step-run-unnotified:<graph>` while runs finished
+ * more than `notifyAfterMs` (default 15 min) ago still have `notified = false` (the graph's `onDone` has not run), and
+ * `step-task-waiting:<graph>` while a task has waited for a person's decision more than `waitingAfterMs` (default 1 h);
+ * the wait is measured from its run's `created_at`, as tasks carry no timestamp, and `step-run-stalled:<graph>` while runs
+ * are still `running` more than `runningAfterMs` (default 1 h) after `created_at` with no task `waiting` (a `waiting` task is a
+ * person's decision and is covered above; `blocked` and `queued` tasks are work in flight, so a run stuck on them is stalled). A failed task always fails its run in the
+ * same transaction, so failed runs cover failed tasks. The failed condition clears by the handle's quiet period; the other two
+ * resolve when the rows drain. Needs a step graph defined in the app (the `pipework_step_run` and `pipework_step_task` tables; no blueprint key).
+ */
+export function stepsSource(options = {}) {
+    const notifyAfter = positive(options.notifyAfterMs ?? FIFTEEN_MIN, 'notifyAfterMs', 'alerts.source.steps');
+    const waitingAfter = positive(options.waitingAfterMs ?? HOUR, 'waitingAfterMs', 'alerts.source.steps');
+    const runningAfter = positive(options.runningAfterMs ?? HOUR, 'runningAfterMs', 'alerts.source.steps');
+    const failed = newRowsSource({
+        ...common(options),
+        name: 'step-run-failed',
+        table: 'pipework_step_run',
+        at: 'finished_at',
+        where: sql `t.status = 'failed'`,
+        groupBy: 'graph',
+        ...(options.settleMs !== undefined ? { settleMs: options.settleMs } : {}),
+        message: g => `${String(g.count)} run(s) of step graph ${g.id} failed; last error: ${text(errorText(g.last['error']))}`,
+        context: g => ({
+            graph: g.id,
+            count: g.count,
+            lastError: errorText(g.last['error']),
+            lastRunId: String(g.last['id']),
+            lastTenantId: String(g.last['tenant_id']),
+        }),
+    });
+    const unnotified = querySource({
+        ...common(options),
+        name: 'step-run-unnotified',
+        query: sql `
+      SELECT graph, count(*)::int AS n, (extract(epoch FROM clock_timestamp() - min(finished_at)) * 1000)::bigint AS oldest_ms
+      FROM pipework_step_run
+      WHERE status = 'done' AND notified = false
+        AND finished_at < clock_timestamp() - ${notifyAfter} * interval '1 millisecond'
+      GROUP BY graph ORDER BY graph`,
+        id: r => String(r['graph']),
+        message: r => `${String(r['n'])} finished run(s) of step graph ${String(r['graph'])} not notified for more than ${String(notifyAfter)} ms (oldest ${String(r['oldest_ms'])} ms)`,
+        subject: r => ({ entity: 'step-graph', entityId: String(r['graph']) }),
+        context: r => ({ graph: String(r['graph']), count: Number(r['n']), oldestMs: Number(r['oldest_ms']) }),
+    });
+    const waiting = querySource({
+        ...common(options),
+        name: 'step-task-waiting',
+        query: sql `
+      SELECT r.graph, count(*)::int AS n, (extract(epoch FROM clock_timestamp() - min(r.created_at)) * 1000)::bigint AS oldest_ms
+      FROM pipework_step_task k JOIN pipework_step_run r ON r.id = k.run_id
+      WHERE k.status = 'waiting' AND r.status = 'running'
+        AND r.created_at < clock_timestamp() - ${waitingAfter} * interval '1 millisecond'
+      GROUP BY r.graph ORDER BY r.graph`,
+        id: r => String(r['graph']),
+        message: r => `${String(r['n'])} task(s) of step graph ${String(r['graph'])} waiting more than ${String(waitingAfter)} ms (oldest run ${String(r['oldest_ms'])} ms old)`,
+        subject: r => ({ entity: 'step-graph', entityId: String(r['graph']) }),
+        context: r => ({ graph: String(r['graph']), count: Number(r['n']), oldestMs: Number(r['oldest_ms']) }),
+    });
+    const stalled = querySource({
+        ...common(options),
+        name: 'step-run-stalled',
+        query: sql `
+      SELECT r.graph, count(*)::int AS n, (extract(epoch FROM clock_timestamp() - min(r.created_at)) * 1000)::bigint AS oldest_ms
+      FROM pipework_step_run r
+      WHERE r.status = 'running'
+        AND r.created_at < clock_timestamp() - ${runningAfter} * interval '1 millisecond'
+        AND NOT EXISTS (SELECT 1 FROM pipework_step_task k WHERE k.run_id = r.id AND k.status = 'waiting')
+      GROUP BY r.graph ORDER BY r.graph`,
+        id: r => String(r['graph']),
+        message: r => `${String(r['n'])} run(s) of step graph ${String(r['graph'])} still running after more than ${String(runningAfter)} ms with no person step waiting (oldest ${String(r['oldest_ms'])} ms)`,
+        subject: r => ({ entity: 'step-graph', entityId: String(r['graph']) }),
+        context: r => ({ graph: String(r['graph']), count: Number(r['n']), oldestMs: Number(r['oldest_ms']) }),
+    });
+    return composeSources('steps', [failed, unnotified, waiting, stalled]);
+}
+/**
+ * Raises `pipeline-failed:<pipeline name>` for pipeline runs that failed since the last tick (context `count`,
+ * `lastError`, `lastRunId`, `lastTenantId`), and `pipeline-waiting:<pipeline name>` while runs sit in `waiting` more than
+ * `waitingAfterMs` (default 1 h) after they started (the table keeps no time for when a run began to wait, so the age is
+ * `started_at`), and `pipeline-stalled:<pipeline name>` while runs are `running` more than `runningAfterMs` (default 1 h)
+ * after `started_at` (the process died mid-run). The failed condition clears by the handle's quiet period; the others
+ * resolve when the runs resume or finish. Needs the app's own `pipework_pipeline_runs` table (pipelines have no blueprint key; the app creates it).
+ */
+export function pipelinesSource(options = {}) {
+    const waitingAfter = positive(options.waitingAfterMs ?? HOUR, 'waitingAfterMs', 'alerts.source.pipelines');
+    const runningAfter = positive(options.runningAfterMs ?? HOUR, 'runningAfterMs', 'alerts.source.pipelines');
+    const failed = newRowsSource({
+        ...common(options),
+        name: 'pipeline-failed',
+        table: 'pipework_pipeline_runs',
+        at: 'completed_at',
+        where: sql `t.status = 'failed'`,
+        groupBy: 'pipeline_name',
+        ...(options.settleMs !== undefined ? { settleMs: options.settleMs } : {}),
+        message: g => `${String(g.count)} run(s) of pipeline ${g.id} failed; last error: ${text(errorText(g.last['error']))}`,
+        context: g => ({
+            pipeline: g.id,
+            count: g.count,
+            lastError: errorText(g.last['error']),
+            lastRunId: String(g.last['id']),
+            lastTenantId: String(g.last['tenant_id']),
+        }),
+    });
+    const waiting = querySource({
+        ...common(options),
+        name: 'pipeline-waiting',
+        query: sql `
+      SELECT pipeline_name, count(*)::int AS n, (extract(epoch FROM clock_timestamp() - min(started_at)) * 1000)::bigint AS oldest_ms
+      FROM pipework_pipeline_runs
+      WHERE status = 'waiting' AND started_at < clock_timestamp() - ${waitingAfter} * interval '1 millisecond'
+      GROUP BY pipeline_name ORDER BY pipeline_name`,
+        id: r => String(r['pipeline_name']),
+        message: r => `${String(r['n'])} run(s) of pipeline ${String(r['pipeline_name'])} waiting more than ${String(waitingAfter)} ms (oldest ${String(r['oldest_ms'])} ms)`,
+        subject: r => ({ entity: 'pipeline', entityId: String(r['pipeline_name']) }),
+        context: r => ({ pipeline: String(r['pipeline_name']), count: Number(r['n']), oldestMs: Number(r['oldest_ms']) }),
+    });
+    const stalled = querySource({
+        ...common(options),
+        name: 'pipeline-stalled',
+        query: sql `
+      SELECT pipeline_name, count(*)::int AS n, (extract(epoch FROM clock_timestamp() - min(started_at)) * 1000)::bigint AS oldest_ms
+      FROM pipework_pipeline_runs
+      WHERE status = 'running' AND started_at < clock_timestamp() - ${runningAfter} * interval '1 millisecond'
+      GROUP BY pipeline_name ORDER BY pipeline_name`,
+        id: r => String(r['pipeline_name']),
+        message: r => `${String(r['n'])} run(s) of pipeline ${String(r['pipeline_name'])} still running after more than ${String(runningAfter)} ms (oldest ${String(r['oldest_ms'])} ms)`,
+        subject: r => ({ entity: 'pipeline', entityId: String(r['pipeline_name']) }),
+        context: r => ({ pipeline: String(r['pipeline_name']), count: Number(r['n']), oldestMs: Number(r['oldest_ms']) }),
+    });
+    return composeSources('pipelines', [failed, waiting, stalled]);
+}
+/**
+ * Raises `trace-expired:<entry kind>` (http, job, cron, test) for traces the expiry sweep force-closed since the last
+ * tick: the sweep stamps `status = 'expired'` and `closed_at`, so `closed_at` is the cursor. Also raises
+ * `trace-expiry-undrained:all` while traces still `open` have an `expires_at` more than `graceMs` (default 26 h: the maintenance job that
+ * runs the sweep is daily, so a trace legitimately stays open up to a day past its deadline) in the past, which means
+ * nothing is running the expiry job. The expired condition clears by the handle's quiet period; the
+ * undrained one resolves once the sweep runs. Needs the blueprint's `trace` block (the `trace` table).
+ */
+export function tracesSource(options = {}) {
+    const grace = positive(options.graceMs ?? TRACE_SWEEP_GRACE, 'graceMs', 'alerts.source.traces');
+    const expired = newRowsSource({
+        ...common(options),
+        name: 'trace-expired',
+        table: 'trace',
+        at: 'closed_at',
+        where: sql `t.status = 'expired'`,
+        groupBy: 'entry_kind',
+        ...(options.settleMs !== undefined ? { settleMs: options.settleMs } : {}),
+        message: g => `${String(g.count)} ${g.id} trace(s) force-closed after expiring; last trace ${String(g.last['id'])}`,
+        context: g => ({
+            entryKind: g.id,
+            count: g.count,
+            lastTraceId: String(g.last['id']),
+            lastTenantId: g.last['tenant_id'] === null || g.last['tenant_id'] === undefined ? null : String(g.last['tenant_id']),
+        }),
+    });
+    const undrained = querySource({
+        ...common(options),
+        name: 'trace-expiry-undrained',
+        query: sql `
+      SELECT count(*)::int AS n, (extract(epoch FROM clock_timestamp() - min(expires_at)) * 1000)::bigint AS oldest_ms
+      FROM "trace"
+      WHERE status = 'open' AND expires_at <= clock_timestamp() - ${grace} * interval '1 millisecond'
+      HAVING count(*) > 0`,
+        id: () => 'all',
+        message: r => `${String(r['n'])} open trace(s) expired more than ${String(grace)} ms ago and not closed; is the trace expiry job scheduled?`,
+        context: r => ({ count: Number(r['n']), oldestMs: Number(r['oldest_ms']) }),
+    });
+    return composeSources('traces', [expired, undrained]);
+}
+//# sourceMappingURL=async-sources.js.map

package/dist/infra/alerts/async-sources.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"async-sources.js","sourceRoot":"","sources":["../../../src/infra/alerts/async-sources.ts"],"names":[],"mappings":"AAAA,4GAA4G;AAC5G,4EAA4E;AAE5E,OAAO,EAAE,GAAG,EAAE,MAAM,sBAAsB,CAAA;AAC1C,OAAO,EAAE,cAAc,EAAE,aAAa,EAAE,WAAW,EAAoB,MAAM,cAAc,CAAA;AAC3F,OAAO,EAAE,MAAM,EAAE,QAAQ,EAAE,IAAI,EAAE,WAAW,EAAE,IAAI,EAAe,MAAM,qBAAqB,CAAA;AAE5F,MAAM,QAAQ,GAAG,CAAC,GAAG,MAAM,CAAA;AAC3B,MAAM,iBAAiB,GAAG,EAAE,GAAG,IAAI,CAAA;AAEnC,yHAAyH;AACzH,SAAS,SAAS,CAAC,CAAU;IAC3B,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,KAAK,SAAS;QAAE,OAAO,IAAI,CAAA;IAC9C,MAAM,MAAM,GAAG,OAAO,CAAC,KAAK,QAAQ,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAA;IACvD,IAAI,OAAO,MAAM,KAAK,QAAQ;QAAE,OAAO,MAAM,CAAA;IAC7C,IAAI,OAAO,MAAM,KAAK,QAAQ,IAAI,MAAM,KAAK,IAAI,EAAE,CAAC;QAClD,MAAM,CAAC,GAAG,MAAiC,CAAA;QAC3C,MAAM,KAAK,GAAG,CAAC,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,OAAO,CAAC,CAAA;QACxC,IAAI,OAAO,KAAK,KAAK,QAAQ;YAAE,OAAO,OAAO,CAAC,CAAC,MAAM,CAAC,KAAK,QAAQ,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,MAAM,CAAC,KAAK,KAAK,EAAE,CAAC,CAAC,CAAC,KAAK,CAAA;QACtG,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,KAAK,IAAI;YAAE,OAAO,SAAS,CAAC,KAAK,CAAC,CAAA;IAC1E,CAAC;IACD,OAAO,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,CAAA;AAC/B,CAAC;AAED,SAAS,SAAS,CAAC,CAAS;IAC1B,IAAI,CAAC;QAAC,OAAO,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAA;IAAC,CAAC;IAAC,MAAM,CAAC;QAAC,OAAO,CAAC,CAAA;IAAC,CAAC;AACjD,CAAC;AAED;;;;;GAKG;AACH,MAAM,UAAU,eAAe,CAAC,UAAyC,EAAE;IACzE,MAAM,KAAK,GAAG,QAAQ,CAAC,OAAO,CAAC,OAAO,IAAI,QAAQ,EAAE,SAAS,EAAE,yBAAyB,CAAC,CAAA;IACzF,OAAO,WAAW,CAAC;QACjB,GAAG,MAAM,CAAC,OAAO,CAAC;QAClB,IAAI,EAAE,kBAAkB;QACxB,KAAK,EAAE,GAAG,CAAA;;;gDAGkC,KAAK;oBACjC;QAChB,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC;QAC1B,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,YAAY,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,OAAO,MAAM,CAAC,CAAC,CAAC,YAAY,CAAC,CAAC,kDAAkD;QAC3H,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,MAAM,EAAE,UAAU,EAAE,QAAQ,EAAE,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,EAAE,CAAC;QACnE,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC;YACb,QAAQ,EAAE,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC;YAC3B,SAAS,EAAE,MAAM,CAAC,CAAC,CAAC,YAAY,CAAC,CAAC;YAClC,SAAS,EAAE,IAAI,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,aAAa,CAAC,CAAC,CAAC,CAAC,WAAW,EAAE;YAC3D,SAAS,EAAE,CAAC,CAAC,aAAa,CAAC,KAAK,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,aAAa,CAAC,CAAC,CAAC,CAAC,WAAW,EAAE;SAC/F,CAAC;KACH,CAAC,CAAA;AACJ,CAAC;AAED;;;;;;;;;;GAUG;AACH,MAAM,UAAU,WAAW,CAAC,UAAoH,EAAE;IAChJ,MAAM,WAAW,GAAG,QAAQ,CAAC,OAAO,CAAC,aAAa,IAAI,WAAW,EAAE,eAAe,EAAE,qBAAqB,CAAC,CAAA;IAC1G,MAAM,YAAY,GAAG,QAAQ,CAAC,OAAO,CAAC,cAAc,IAAI,IAAI,EAAE,gBAAgB,EAAE,qBAAqB,CAAC,CAAA;IACtG,MAAM,YAAY,GAAG,QAAQ,CAAC,OAAO,CAAC,cAAc,IAAI,IAAI,EAAE,gBAAgB,EAAE,qBAAqB,CAAC,CAAA;IACtG,MAAM,MAAM,GAAG,aAAa,CAAC;QAC3B,GAAG,MAAM,CAAC,OAAO,CAAC;QAClB,IAAI,EAAE,iBAAiB;QACvB,KAAK,EAAE,mBAAmB;QAC1B,EAAE,EAAE,aAAa;QACjB,KAAK,EAAE,GAAG,CAAA,qBAAqB;QAC/B,OAAO,EAAE,OAAO;QAChB,GAAG,CAAC,OAAO,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,OAAO,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACzE,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,GAAG,MAAM,CAAC,CAAC,CAAC,KAAK,CAAC,yBAAyB,CAAC,CAAC,EAAE,wBAAwB,IAAI,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE;QACvH,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC;YACb,KAAK,EAAE,CAAC,CAAC,EAAE;YACX,KAAK,EAAE,CAAC,CAAC,KAAK;YACd,SAAS,EAAE,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC;YACrC,SAAS,EAAE,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;YAC/B,YAAY,EAAE,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,WAAW,CAAC,CAAC;SAC1C,CAAC;KACH,CAAC,CAAA;IACF,MAAM,UAAU,GAAG,WAAW,CAAC;QAC7B,GAAG,MAAM,CAAC,OAAO,CAAC;QAClB,IAAI,EAAE,qBAAqB;QAC3B,KAAK,EAAE,GAAG,CAAA;;;;gDAIkC,WAAW;oCACvB;QAChC,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,MAAM,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC;QAC3B,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,GAAG,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,kCAAkC,MAAM,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,+BAA+B,MAAM,CAAC,WAAW,CAAC,eAAe,MAAM,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,MAAM;QAChL,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,MAAM,EAAE,YAAY,EAAE,QAAQ,EAAE,MAAM,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,EAAE,CAAC;QACtE,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,KAAK,EAAE,MAAM,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,EAAE,KAAK,EAAE,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,EAAE,QAAQ,EAAE,MAAM,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,EAAE,CAAC;KACvG,CAAC,CAAA;IACF,MAAM,OAAO,GAAG,WAAW,CAAC;QAC1B,GAAG,MAAM,CAAC,OAAO,CAAC;QAClB,IAAI,EAAE,mBAAmB;QACzB,KAAK,EAAE,GAAG,CAAA;;;;iDAImC,YAAY;wCACrB;QACpC,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,MAAM,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC;QAC3B,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,GAAG,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,0BAA0B,MAAM,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,sBAAsB,MAAM,CAAC,YAAY,CAAC,mBAAmB,MAAM,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,UAAU;QACxK,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,MAAM,EAAE,YAAY,EAAE,QAAQ,EAAE,MAAM,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,EAAE,CAAC;QACtE,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,KAAK,EAAE,MAAM,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,EAAE,KAAK,EAAE,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,EAAE,QAAQ,EAAE,MAAM,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,EAAE,CAAC;KACvG,CAAC,CAAA;IACF,MAAM,OAAO,GAAG,WAAW,CAAC;QAC1B,GAAG,MAAM,CAAC,OAAO,CAAC;QAClB,IAAI,EAAE,kBAAkB;QACxB,KAAK,EAAE,GAAG,CAAA;;;;iDAImC,YAAY;;wCAErB;QACpC,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,MAAM,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC;QAC3B,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,GAAG,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,yBAAyB,MAAM,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,kCAAkC,MAAM,CAAC,YAAY,CAAC,2CAA2C,MAAM,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,MAAM;QACvM,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,MAAM,EAAE,YAAY,EAAE,QAAQ,EAAE,MAAM,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,EAAE,CAAC;QACtE,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,KAAK,EAAE,MAAM,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,EAAE,KAAK,EAAE,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,EAAE,QAAQ,EAAE,MAAM,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,EAAE,CAAC;KACvG,CAAC,CAAA;IACF,OAAO,cAAc,CAAC,OAAO,EAAE,CAAC,MAAM,EAAE,UAAU,EAAE,OAAO,EAAE,OAAO,CAAC,CAAC,CAAA;AACxE,CAAC;AAED;;;;;;;GAOG;AACH,MAAM,UAAU,eAAe,CAAC,UAA4F,EAAE;IAC5H,MAAM,YAAY,GAAG,QAAQ,CAAC,OAAO,CAAC,cAAc,IAAI,IAAI,EAAE,gBAAgB,EAAE,yBAAyB,CAAC,CAAA;IAC1G,MAAM,YAAY,GAAG,QAAQ,CAAC,OAAO,CAAC,cAAc,IAAI,IAAI,EAAE,gBAAgB,EAAE,yBAAyB,CAAC,CAAA;IAC1G,MAAM,MAAM,GAAG,aAAa,CAAC;QAC3B,GAAG,MAAM,CAAC,OAAO,CAAC;QAClB,IAAI,EAAE,iBAAiB;QACvB,KAAK,EAAE,wBAAwB;QAC/B,EAAE,EAAE,cAAc;QAClB,KAAK,EAAE,GAAG,CAAA,qBAAqB;QAC/B,OAAO,EAAE,eAAe;QACxB,GAAG,CAAC,OAAO,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,OAAO,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACzE,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,GAAG,MAAM,CAAC,CAAC,CAAC,KAAK,CAAC,uBAAuB,CAAC,CAAC,EAAE,wBAAwB,IAAI,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE;QACrH,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC;YACb,QAAQ,EAAE,CAAC,CAAC,EAAE;YACd,KAAK,EAAE,CAAC,CAAC,KAAK;YACd,SAAS,EAAE,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC;YACrC,SAAS,EAAE,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;YAC/B,YAAY,EAAE,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,WAAW,CAAC,CAAC;SAC1C,CAAC;KACH,CAAC,CAAA;IACF,MAAM,OAAO,GAAG,WAAW,CAAC;QAC1B,GAAG,MAAM,CAAC,OAAO,CAAC;QAClB,IAAI,EAAE,kBAAkB;QACxB,KAAK,EAAE,GAAG,CAAA;;;sEAGwD,YAAY;oDAC9B;QAChD,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,MAAM,CAAC,CAAC,CAAC,eAAe,CAAC,CAAC;QACnC,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,GAAG,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,uBAAuB,MAAM,CAAC,CAAC,CAAC,eAAe,CAAC,CAAC,sBAAsB,MAAM,CAAC,YAAY,CAAC,eAAe,MAAM,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,MAAM;QACrK,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,MAAM,EAAE,UAAU,EAAE,QAAQ,EAAE,MAAM,CAAC,CAAC,CAAC,eAAe,CAAC,CAAC,EAAE,CAAC;QAC5E,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,QAAQ,EAAE,MAAM,CAAC,CAAC,CAAC,eAAe,CAAC,CAAC,EAAE,KAAK,EAAE,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,EAAE,QAAQ,EAAE,MAAM,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,EAAE,CAAC;KAClH,CAAC,CAAA;IACF,MAAM,OAAO,GAAG,WAAW,CAAC;QAC1B,GAAG,MAAM,CAAC,OAAO,CAAC;QAClB,IAAI,EAAE,kBAAkB;QACxB,KAAK,EAAE,GAAG,CAAA;;;sEAGwD,YAAY;oDAC9B;QAChD,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,MAAM,CAAC,CAAC,CAAC,eAAe,CAAC,CAAC;QACnC,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,GAAG,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,uBAAuB,MAAM,CAAC,CAAC,CAAC,eAAe,CAAC,CAAC,kCAAkC,MAAM,CAAC,YAAY,CAAC,eAAe,MAAM,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,MAAM;QACjL,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,MAAM,EAAE,UAAU,EAAE,QAAQ,EAAE,MAAM,CAAC,CAAC,CAAC,eAAe,CAAC,CAAC,EAAE,CAAC;QAC5E,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,QAAQ,EAAE,MAAM,CAAC,CAAC,CAAC,eAAe,CAAC,CAAC,EAAE,KAAK,EAAE,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,EAAE,QAAQ,EAAE,MAAM,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,EAAE,CAAC;KAClH,CAAC,CAAA;IACF,OAAO,cAAc,CAAC,WAAW,EAAE,CAAC,MAAM,EAAE,OAAO,EAAE,OAAO,CAAC,CAAC,CAAA;AAChE,CAAC;AAED;;;;;;;GAOG;AACH,MAAM,UAAU,YAAY,CAAC,UAA4D,EAAE;IACzF,MAAM,KAAK,GAAG,QAAQ,CAAC,OAAO,CAAC,OAAO,IAAI,iBAAiB,EAAE,SAAS,EAAE,sBAAsB,CAAC,CAAA;IAC/F,MAAM,OAAO,GAAG,aAAa,CAAC;QAC5B,GAAG,MAAM,CAAC,OAAO,CAAC;QAClB,IAAI,EAAE,eAAe;QACrB,KAAK,EAAE,OAAO;QACd,EAAE,EAAE,WAAW;QACf,KAAK,EAAE,GAAG,CAAA,sBAAsB;QAChC,OAAO,EAAE,YAAY;QACrB,GAAG,CAAC,OAAO,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,OAAO,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACzE,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,GAAG,MAAM,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,EAAE,qDAAqD,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,EAAE;QACnH,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC;YACb,SAAS,EAAE,CAAC,CAAC,EAAE;YACf,KAAK,EAAE,CAAC,CAAC,KAAK;YACd,WAAW,EAAE,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;YACjC,YAAY,EAAE,CAAC,CAAC,IAAI,CAAC,WAAW,CAAC,KAAK,IAAI,IAAI,CAAC,CAAC,IAAI,CAAC,WAAW,CAAC,KAAK,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,WAAW,CAAC,CAAC;SACrH,CAAC;KACH,CAAC,CAAA;IACF,MAAM,SAAS,GAAG,WAAW,CAAC;QAC5B,GAAG,MAAM,CAAC,OAAO,CAAC;QAClB,IAAI,EAAE,wBAAwB;QAC9B,KAAK,EAAE,GAAG,CAAA;;;oEAGsD,KAAK;0BAC/C;QACtB,EAAE,EAAE,GAAG,EAAE,CAAC,KAAK;QACf,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,GAAG,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,oCAAoC,MAAM,CAAC,KAAK,CAAC,4DAA4D;QAC5I,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,KAAK,EAAE,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,EAAE,QAAQ,EAAE,MAAM,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,EAAE,CAAC;KAC5E,CAAC,CAAA;IACF,OAAO,cAAc,CAAC,QAAQ,EAAE,CAAC,OAAO,EAAE,SAAS,CAAC,CAAC,CAAA;AACvD,CAAC"}

package/dist/infra/alerts/namespace.d.ts

@@ -6,6 +6,7 @@ import { requestFaultsHook } from './request-faults.js';
 import { deadLetterSource, longRunningSource, backlogSource, healthSource, quotaExhaustedSource, unitFaultSource, loopStalledSource, loopCheck, querySource, newRowsSource } from './sources.js';
 import { deliveryFailedSource, webhookFailedSource, creditsSource } from './subsystem-sources.js';
 import { emailSource, storageSource, accessLinksSource, lockoutsSource } from './infra-sources.js';
+import { schedulesSource, stepsSource, pipelinesSource, tracesSource } from './async-sources.js';
 /** Alerts namespace — raise and resolve keyed conditions, grouped, delivered to sinks through an outbox, traced. */
 export declare const alerts: {
     /** Creates an alerts handle. */
@@ -53,6 +54,14 @@ export declare const alerts: {
         readonly accessLinks: typeof accessLinksSource;
         /** Raises `auth-lockout:<table>` (warning) while sign-in or guard lockouts are active; counts only, no accounts. */
         readonly lockouts: typeof lockoutsSource;
+        /** Raises `schedule-overdue:<name>` while a cron schedule's `next_run_at` is long past, which means no scheduler is firing it. */
+        readonly schedules: typeof schedulesSource;
+        /** Raises `step-run-failed:<graph>`, `step-run-unnotified:<graph>` and `step-task-waiting:<graph>` and `step-run-stalled:<graph>` over the step graph tables. */
+        readonly steps: typeof stepsSource;
+        /** Raises `pipeline-failed:<pipeline>` for failed runs `pipeline-waiting:<pipeline>` for runs waiting too long and `pipeline-stalled:<pipeline>` for runs still running too long. */
+        readonly pipelines: typeof pipelinesSource;
+        /** Raises `trace-expired:<entryKind>` for traces force-closed by the expiry sweep and `trace-expiry-undrained:all` while expired open traces are not being swept. */
+        readonly traces: typeof tracesSource;
         /** Raises `loop-stalled:<name>` while a background loop (scheduler, reaper, claim loop, support drain) has not run or succeeded within its allowance; the tick itself is covered by `pipework alerts stale`. */
         readonly loopStalled: typeof loopStalledSource;
     };

package/dist/infra/alerts/namespace.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"namespace.d.ts","sourceRoot":"","sources":["../../../src/infra/alerts/namespace.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAA;AAC1C,OAAO,EAAE,UAAU,EAAE,MAAM,WAAW,CAAA;AACtC,OAAO,EAAE,YAAY,EAAE,MAAM,oBAAoB,CAAA;AACjD,OAAO,EAAE,WAAW,EAAE,UAAU,EAAE,MAAM,YAAY,CAAA;AACpD,OAAO,EAAE,iBAAiB,EAAE,MAAM,qBAAqB,CAAA;AACvD,OAAO,EAAE,gBAAgB,EAAE,iBAAiB,EAAE,aAAa,EAAE,YAAY,EAAE,oBAAoB,EAAE,eAAe,EAAE,iBAAiB,EAAE,SAAS,EAAE,WAAW,EAAE,aAAa,EAAE,MAAM,cAAc,CAAA;AAChM,OAAO,EAAE,oBAAoB,EAAE,mBAAmB,EAAE,aAAa,EAAE,MAAM,wBAAwB,CAAA;AACjG,OAAO,EAAE,WAAW,EAAE,aAAa,EAAE,iBAAiB,EAAE,cAAc,EAAE,MAAM,oBAAoB,CAAA;AAElG,oHAAoH;AACpH,eAAO,MAAM,MAAM;IACjB,gCAAgC;;IAEhC,0HAA0H;;IAE1H,wLAAwL;;IAExL,oHAAoH;;IAEpH,2DAA2D;;IAE3D,+DAA+D;;IAE/D,kJAAkJ;;;QAGhJ,4EAA4E;;QAE5E,iFAAiF;;QAEjF,mGAAmG;;QAEnG,yGAAyG;;QAEzG,yDAAyD;;QAEzD,6IAA6I;;QAE7I,2HAA2H;;QAE3H,iIAAiI;;QAEjI,0IAA0I;;QAE1I,yJAAyJ;;QAEzJ,gIAAgI;;QAEhI,4JAA4J;;QAE5J,+EAA+E;;QAE/E,gGAAgG;;QAEhG,oHAAoH;;QAEpH,gNAAgN;;;CAG1M,CAAA"}
+{"version":3,"file":"namespace.d.ts","sourceRoot":"","sources":["../../../src/infra/alerts/namespace.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAA;AAC1C,OAAO,EAAE,UAAU,EAAE,MAAM,WAAW,CAAA;AACtC,OAAO,EAAE,YAAY,EAAE,MAAM,oBAAoB,CAAA;AACjD,OAAO,EAAE,WAAW,EAAE,UAAU,EAAE,MAAM,YAAY,CAAA;AACpD,OAAO,EAAE,iBAAiB,EAAE,MAAM,qBAAqB,CAAA;AACvD,OAAO,EAAE,gBAAgB,EAAE,iBAAiB,EAAE,aAAa,EAAE,YAAY,EAAE,oBAAoB,EAAE,eAAe,EAAE,iBAAiB,EAAE,SAAS,EAAE,WAAW,EAAE,aAAa,EAAE,MAAM,cAAc,CAAA;AAChM,OAAO,EAAE,oBAAoB,EAAE,mBAAmB,EAAE,aAAa,EAAE,MAAM,wBAAwB,CAAA;AACjG,OAAO,EAAE,WAAW,EAAE,aAAa,EAAE,iBAAiB,EAAE,cAAc,EAAE,MAAM,oBAAoB,CAAA;AAClG,OAAO,EAAE,eAAe,EAAE,WAAW,EAAE,eAAe,EAAE,YAAY,EAAE,MAAM,oBAAoB,CAAA;AAEhG,oHAAoH;AACpH,eAAO,MAAM,MAAM;IACjB,gCAAgC;;IAEhC,0HAA0H;;IAE1H,wLAAwL;;IAExL,oHAAoH;;IAEpH,2DAA2D;;IAE3D,+DAA+D;;IAE/D,kJAAkJ;;;QAGhJ,4EAA4E;;QAE5E,iFAAiF;;QAEjF,mGAAmG;;QAEnG,yGAAyG;;QAEzG,yDAAyD;;QAEzD,6IAA6I;;QAE7I,2HAA2H;;QAE3H,iIAAiI;;QAEjI,0IAA0I;;QAE1I,yJAAyJ;;QAEzJ,gIAAgI;;QAEhI,4JAA4J;;QAE5J,+EAA+E;;QAE/E,gGAAgG;;QAEhG,oHAAoH;;QAEpH,kIAAkI;;QAElI,iKAAiK;;QAEjK,qLAAqL;;QAErL,qKAAqK;;QAErK,gNAAgN;;;CAG1M,CAAA"}

package/dist/infra/alerts/namespace.js

@@ -6,6 +6,7 @@ import { requestFaultsHook } from './request-faults.js';
 import { deadLetterSource, longRunningSource, backlogSource, healthSource, quotaExhaustedSource, unitFaultSource, loopStalledSource, loopCheck, querySource, newRowsSource } from './sources.js';
 import { deliveryFailedSource, webhookFailedSource, creditsSource } from './subsystem-sources.js';
 import { emailSource, storageSource, accessLinksSource, lockoutsSource } from './infra-sources.js';
+import { schedulesSource, stepsSource, pipelinesSource, tracesSource } from './async-sources.js';
 /** Alerts namespace — raise and resolve keyed conditions, grouped, delivered to sinks through an outbox, traced. */
 export const alerts = {
     /** Creates an alerts handle. */
@@ -53,6 +54,14 @@ export const alerts = {
         accessLinks: accessLinksSource,
         /** Raises `auth-lockout:<table>` (warning) while sign-in or guard lockouts are active; counts only, no accounts. */
         lockouts: lockoutsSource,
+        /** Raises `schedule-overdue:<name>` while a cron schedule's `next_run_at` is long past, which means no scheduler is firing it. */
+        schedules: schedulesSource,
+        /** Raises `step-run-failed:<graph>`, `step-run-unnotified:<graph>` and `step-task-waiting:<graph>` and `step-run-stalled:<graph>` over the step graph tables. */
+        steps: stepsSource,
+        /** Raises `pipeline-failed:<pipeline>` for failed runs `pipeline-waiting:<pipeline>` for runs waiting too long and `pipeline-stalled:<pipeline>` for runs still running too long. */
+        pipelines: pipelinesSource,
+        /** Raises `trace-expired:<entryKind>` for traces force-closed by the expiry sweep and `trace-expiry-undrained:all` while expired open traces are not being swept. */
+        traces: tracesSource,
         /** Raises `loop-stalled:<name>` while a background loop (scheduler, reaper, claim loop, support drain) has not run or succeeded within its allowance; the tick itself is covered by `pipework alerts stale`. */
         loopStalled: loopStalledSource,
     },

package/dist/infra/alerts/namespace.js.map

@@ -1 +1 @@
-{"version":3,"file":"namespace.js","sourceRoot":"","sources":["../../../src/infra/alerts/namespace.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAA;AAC1C,OAAO,EAAE,UAAU,EAAE,MAAM,WAAW,CAAA;AACtC,OAAO,EAAE,YAAY,EAAE,MAAM,oBAAoB,CAAA;AACjD,OAAO,EAAE,WAAW,EAAE,UAAU,EAAE,MAAM,YAAY,CAAA;AACpD,OAAO,EAAE,iBAAiB,EAAE,MAAM,qBAAqB,CAAA;AACvD,OAAO,EAAE,gBAAgB,EAAE,iBAAiB,EAAE,aAAa,EAAE,YAAY,EAAE,oBAAoB,EAAE,eAAe,EAAE,iBAAiB,EAAE,SAAS,EAAE,WAAW,EAAE,aAAa,EAAE,MAAM,cAAc,CAAA;AAChM,OAAO,EAAE,oBAAoB,EAAE,mBAAmB,EAAE,aAAa,EAAE,MAAM,wBAAwB,CAAA;AACjG,OAAO,EAAE,WAAW,EAAE,aAAa,EAAE,iBAAiB,EAAE,cAAc,EAAE,MAAM,oBAAoB,CAAA;AAElG,oHAAoH;AACpH,MAAM,CAAC,MAAM,MAAM,GAAG;IACpB,gCAAgC;IAChC,MAAM,EAAE,YAAY;IACpB,0HAA0H;IAC1H,IAAI,EAAE,UAAU;IAChB,wLAAwL;IACxL,YAAY;IACZ,oHAAoH;IACpH,aAAa,EAAE,iBAAiB;IAChC,2DAA2D;IAC3D,OAAO,EAAE,WAAW;IACpB,+DAA+D;IAC/D,MAAM,EAAE,UAAU;IAClB,kJAAkJ;IAClJ,SAAS;IACT,MAAM,EAAE;QACN,4EAA4E;QAC5E,UAAU,EAAE,gBAAgB;QAC5B,iFAAiF;QACjF,WAAW,EAAE,iBAAiB;QAC9B,mGAAmG;QACnG,OAAO,EAAE,aAAa;QACtB,yGAAyG;QACzG,cAAc,EAAE,oBAAoB;QACpC,yDAAyD;QACzD,MAAM,EAAE,YAAY;QACpB,6IAA6I;QAC7I,SAAS,EAAE,eAAe;QAC1B,2HAA2H;QAC3H,KAAK,EAAE,WAAW;QAClB,iIAAiI;QACjI,OAAO,EAAE,aAAa;QACtB,0IAA0I;QAC1I,cAAc,EAAE,oBAAoB;QACpC,yJAAyJ;QACzJ,aAAa,EAAE,mBAAmB;QAClC,gIAAgI;QAChI,OAAO,EAAE,aAAa;QACtB,4JAA4J;QAC5J,KAAK,EAAE,WAAW;QAClB,+EAA+E;QAC/E,OAAO,EAAE,aAAa;QACtB,gGAAgG;QAChG,WAAW,EAAE,iBAAiB;QAC9B,oHAAoH;QACpH,QAAQ,EAAE,cAAc;QACxB,gNAAgN;QAChN,WAAW,EAAE,iBAAiB;KAC/B;CACO,CAAA"}
+{"version":3,"file":"namespace.js","sourceRoot":"","sources":["../../../src/infra/alerts/namespace.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAA;AAC1C,OAAO,EAAE,UAAU,EAAE,MAAM,WAAW,CAAA;AACtC,OAAO,EAAE,YAAY,EAAE,MAAM,oBAAoB,CAAA;AACjD,OAAO,EAAE,WAAW,EAAE,UAAU,EAAE,MAAM,YAAY,CAAA;AACpD,OAAO,EAAE,iBAAiB,EAAE,MAAM,qBAAqB,CAAA;AACvD,OAAO,EAAE,gBAAgB,EAAE,iBAAiB,EAAE,aAAa,EAAE,YAAY,EAAE,oBAAoB,EAAE,eAAe,EAAE,iBAAiB,EAAE,SAAS,EAAE,WAAW,EAAE,aAAa,EAAE,MAAM,cAAc,CAAA;AAChM,OAAO,EAAE,oBAAoB,EAAE,mBAAmB,EAAE,aAAa,EAAE,MAAM,wBAAwB,CAAA;AACjG,OAAO,EAAE,WAAW,EAAE,aAAa,EAAE,iBAAiB,EAAE,cAAc,EAAE,MAAM,oBAAoB,CAAA;AAClG,OAAO,EAAE,eAAe,EAAE,WAAW,EAAE,eAAe,EAAE,YAAY,EAAE,MAAM,oBAAoB,CAAA;AAEhG,oHAAoH;AACpH,MAAM,CAAC,MAAM,MAAM,GAAG;IACpB,gCAAgC;IAChC,MAAM,EAAE,YAAY;IACpB,0HAA0H;IAC1H,IAAI,EAAE,UAAU;IAChB,wLAAwL;IACxL,YAAY;IACZ,oHAAoH;IACpH,aAAa,EAAE,iBAAiB;IAChC,2DAA2D;IAC3D,OAAO,EAAE,WAAW;IACpB,+DAA+D;IAC/D,MAAM,EAAE,UAAU;IAClB,kJAAkJ;IAClJ,SAAS;IACT,MAAM,EAAE;QACN,4EAA4E;QAC5E,UAAU,EAAE,gBAAgB;QAC5B,iFAAiF;QACjF,WAAW,EAAE,iBAAiB;QAC9B,mGAAmG;QACnG,OAAO,EAAE,aAAa;QACtB,yGAAyG;QACzG,cAAc,EAAE,oBAAoB;QACpC,yDAAyD;QACzD,MAAM,EAAE,YAAY;QACpB,6IAA6I;QAC7I,SAAS,EAAE,eAAe;QAC1B,2HAA2H;QAC3H,KAAK,EAAE,WAAW;QAClB,iIAAiI;QACjI,OAAO,EAAE,aAAa;QACtB,0IAA0I;QAC1I,cAAc,EAAE,oBAAoB;QACpC,yJAAyJ;QACzJ,aAAa,EAAE,mBAAmB;QAClC,gIAAgI;QAChI,OAAO,EAAE,aAAa;QACtB,4JAA4J;QAC5J,KAAK,EAAE,WAAW;QAClB,+EAA+E;QAC/E,OAAO,EAAE,aAAa;QACtB,gGAAgG;QAChG,WAAW,EAAE,iBAAiB;QAC9B,oHAAoH;QACpH,QAAQ,EAAE,cAAc;QACxB,kIAAkI;QAClI,SAAS,EAAE,eAAe;QAC1B,iKAAiK;QACjK,KAAK,EAAE,WAAW;QAClB,qLAAqL;QACrL,SAAS,EAAE,eAAe;QAC1B,qKAAqK;QACrK,MAAM,EAAE,YAAY;QACpB,gNAAgN;QAChN,WAAW,EAAE,iBAAiB;KAC/B;CACO,CAAA"}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pipework",
-  "version": "0.176.0",
+  "version": "0.177.0",
   "description": "TypeScript framework for multi-tenant SaaS applications. PostgreSQL-only.",
   "type": "module",
   "license": "MIT",