spine-framework 0.4.4 → 0.5.0-rc.4

package/.framework/src/instructions/reference/functions/system-cron.md

@@ -0,0 +1,136 @@
+---
+title: system-cron
+audience: developers, agents
+kind: reference
+verified_against: f69c37d
+verified_on: 2026-10-30
+source_files:
+  - .framework/functions/system-cron.ts
+  - .framework/functions/trigger-scheduler.ts
+  - .framework/functions/_shared/pipeline-runner.ts
+  - .framework/functions/_shared/retry-deliveries.ts
+  - .framework/functions/_shared/principal.ts
+  - .framework/migrations/068_cron_schedule_timer_rpcs.sql
+  - netlify.toml
+---
+
+# system-cron
+
+## What it does
+
+The worker behind every `trigger-scheduler` tick. Each authenticated run
+performs, in order: a stale-execution sweep, due-schedule execution, due-
+timer pipelines, threshold-alert evaluation, webhook-delivery retries, and
+— on the 00:00 UTC tick — log cleanup. It is also directly invocable over
+HTTP for manual runs.
+
+## Route and methods
+
+`/.netlify/functions/system-cron`. Normally reached internally via
+`trigger-scheduler` (Netlify Clockwork, `* * * * *` in `netlify.toml`),
+which calls this handler either with a fabricated `SYSTEM_PRINCIPAL`
+context (scheduled ticks) or through the normal middleware path (manual
+calls). Any HTTP method reaches the handler — there is no method dispatch.
+
+## Authentication
+
+Inside the handler, after `createHandler` has resolved a non-anonymous
+principal:
+
+- `?api_key=` or `?scheduler_key=` equal to the `SCHEDULER_API_KEY`
+  environment variable, **or**
+- a machine principal with `machineType === 'internal'` — which is what a
+  valid `x-internal-secret` header resolves to (`SYSTEM_PRINCIPAL`).
+
+Anything else returns 403 and emits a `system_cron.unauthorized_access`
+audit event. When `SCHEDULER_API_KEY` is unset the key comparison can
+never match, so only internal machine principals pass — fail closed.
+
+## Request contract
+
+- Query: `api_key` or `scheduler_key` (scheduler key auth path only).
+- Headers: `x-internal-secret` (internal machine principal path).
+- No body is read.
+
+## What runs per invocation
+
+1. **Stale-execution sweep** — `sweepStaleExecutions()` marks
+   `pipeline_executions` rows in `running`/`pending`/`queued` untouched
+   for over 10 minutes as `failed` with a `stale execution` error message.
+   Runs on every invocation, before anything else.
+2. **Due schedules** — `get_due_schedules` RPC returns due
+   `public.schedules` rows. For each: `validate_schedule_creator` RPC →
+   load the `public.actions` row → load the schedule's machine principal
+   from `api_keys` → verify scope (`action.required_scopes[0]` or
+   `<handler>:execute`, satisfied also by `*:*`) → `executeAction` →
+   insert a `schedule_executions` row → `update_schedule_after_run` RPC →
+   `schedule.execute` audit.
+3. **Due timers** — `get_due_timers` RPC; each due timer runs
+   `runPipeline(timer.pipeline_id, …)` under a fabricated machine
+   principal `timer:<timer_id>` with scope `pipelines:execute`, then
+   `update_timer_after_run` reschedules it. Timers without `pipeline_id`
+   fail that item only.
+4. **Threshold alerts** — active `threshold_alerts` rows are evaluated
+   against `error_rate`, `latency_p95`, or `pipeline_failure_rate` metric
+   RPCs over `window_minutes`. `operator: 'gt'` compares `>`; anything
+   else compares `<`. A breached threshold with a `pipeline_id` fires
+   `runPipeline` with the metric details as trigger data, and a
+   `threshold.breached` audit event is emitted either way on breach.
+5. **Log cleanup** — only when the tick lands on UTC 00:00:
+   `cleanup_old_logs` RPC with `p_retention_days: 30`.
+6. **Delivery retries** — `retryPendingDeliveries(ctx)` sweeps due
+   outbound webhook deliveries.
+
+`executeAction` routes `action.handler_module`:
+
+- `functions` — `send_email`, `generate_report`, `notify_watchers` are
+  console-log stubs returning success-shaped objects; `run_pipeline`
+  invokes `runPipeline` for real.
+- `integrations` — stub returning `{ executed: true, handler }`.
+- `custom` — throws `Custom handlers not yet implemented`.
+- anything else — throws `Unknown handler module`.
+
+## Response contract
+
+- 200 — `{ message, executed, success, failed, skipped,
+  thresholds_evaluated, thresholds_breached, logs_cleaned,
+  deliveries_retried, results }` where `results[]` carries
+  `{ scheduleId, actionId, status: success|failed|skipped, error?,
+  durationMs }` per schedule/timer item.
+- 200 with `{ message: 'No schedules due for execution', executed: 0,
+  results: [] }` when `get_due_schedules` returns nothing — see the early
+  return note below.
+- 403 — invalid scheduler authentication.
+- 500 — top-level failure, with a `system_cron.error` audit event.
+
+## Side effects
+
+Writes `pipeline_executions` (via `runPipeline` and the sweep),
+`schedule_executions`, schedule/timer bookkeeping via RPC, `logs` audit
+events (`system_cron.unauthorized_access`, `system_cron.error`,
+`schedule.execute`, `threshold.breached`), deletes old `logs` rows at
+00:00 UTC, and mutates `webhook_deliveries` through the retry sweep.
+
+## Errors
+
+Per-item failures are contained: a failing schedule/timer records a
+`failed` result and a `schedule_executions`/`update_*_after_run` failure
+record rather than aborting the tick. A failure in the outer block —
+e.g. `get_due_schedules` itself erroring — short-circuits to 500 with a
+`system_cron.error` audit. Threshold, timer-loop, cleanup, and sweep
+failures are caught and logged without failing the response.
+
+## Agent notes
+
+- **Early return:** when `get_due_schedules` yields no due schedules the
+  handler returns immediately — due timers, threshold evaluation,
+  delivery retries, and log cleanup all sit *after* that return and only
+  run on ticks where at least one schedule is due. Installations relying
+  on timers need at least one due schedule per tick for those steps to
+  execute; verify this is intended before depending on timer cadence.
+- `send_email`, `generate_report`, `notify_watchers`, and the
+  `integrations` module are stubs — a schedule pointing at them "succeeds"
+  while doing nothing but a console log. Only `run_pipeline` performs real
+  work today.
+- `SCHEDULER_API_KEY` unset does not open the endpoint: the key path
+  simply never matches and only internal machine principals pass.

package/.framework/src/instructions/reference/functions/system.md

@@ -0,0 +1,106 @@
+---
+title: system
+audience: developers, agents
+kind: reference
+verified_against: f69c37d
+verified_on: 2026-10-30
+source_files:
+  - .framework/functions/system.ts
+  - .framework/functions/_shared/middleware.ts
+  - .framework/functions/_shared/db.ts
+---
+
+# system
+
+## What it does
+
+Deployment introspection: a `manifest` action describing version,
+migrations, and the function surface; a `health` action reporting
+database/migration/integrity status; and an `openapi` action generating an
+OpenAPI 3.1.0 document from discovered functions. Read-only — nothing here
+mutates the database.
+
+## Route and methods
+
+`/.netlify/functions/system?action=<manifest|health|openapi>`.
+
+`action` defaults to `manifest` when omitted. There is no method check —
+any HTTP method dispatches the same way.
+
+## Authentication
+
+Wrapped in `createHandler`: any non-anonymous principal passes — human
+Bearer JWT, machine `x-api-key`, or `x-internal-secret`. Anonymous → 401.
+There is no role gate; every authenticated caller can read the manifest,
+health, and OpenAPI output. All queries run on `adminDb`.
+
+## Request contract
+
+Query parameters only: `action` (optional, default `manifest`). No body is
+read.
+
+## Response contract
+
+All responses arrive inside the middleware envelope
+`{ data: { data: <payload> }, error: null, meta }` — the action functions
+return `{ data: … }` objects which are wrapped again by `createHandler`.
+
+- `manifest` → `{ version, schema: 'public', migrations: { applied,
+  latest, pending }, functions: [{ name, methods, path, description }],
+  integrity: { core_hash, verified } }`.
+  - `version` comes from `.spine-manifest.json` in the process working
+    directory (default `'2.0.0'` when missing or unparseable).
+  - `integrity.core_hash` is `manifest.integrity.src`; `verified` is true
+    only when the file exists and parses.
+  - `functions` is built by scanning a `v2-core/functions` directory for
+    `.ts` files and assigning methods from blocklists (see Agent notes).
+- `health` → `{ status: 'healthy' | 'degraded' | 'unhealthy', checks:
+  { database: { connected, latency_ms }, migrations: { current, pending,
+  applied }, integrity: { verified }, test_runs? } }`.
+  - `unhealthy` when the `accounts` probe query fails; `degraded` when
+    migrations are pending or integrity is unverified; else `healthy`.
+  - `test_runs` is the latest `test_runs` row (`suite`, `status`,
+    `started_at`) when present.
+- `openapi` → an OpenAPI `3.1.0` document (`info.title` = "Spine API",
+  `info.version` = manifest version, `servers: ['/ .netlify/functions']`
+  — see Agent notes) whose `paths` are generated from the discovered
+  function list, each operation exposing an optional `action` query
+  parameter and a placeholder object response schema.
+- Unknown action → the handler returns `{ error: 'Unknown action: ….
+  Valid actions: manifest, health, openapi', status: 400 }` **as data** —
+  the HTTP response is still 200 because the value is returned, not
+  thrown.
+
+## Side effects
+
+None on the database. The filesystem reads are: `.spine-manifest.json`,
+and the `v2-core/functions` + `v2-core/migrations_dayzero` directories
+under `process.cwd()`. Database reads: `accounts` (health probe),
+`public.schema_migrations`, `test_runs` — all via `adminDb`.
+
+## Errors
+
+The handler never throws for its own logic — database failures inside
+`getManifest`/`getHealth` are swallowed into `applied: 0` / `pending`
+counts and status flags. `createHandler` still surfaces middleware-level
+errors (401 unauthenticated, 400 invalid JSON body).
+
+## Agent notes
+
+- **Stale paths:** `discoverFunctions` scans `v2-core/functions` and the
+  migration checks scan `v2-core/migrations_dayzero` — `v2-core` is the
+  pre-monorepo layout name; the live code lives under
+  `.framework/functions` and `.framework/migrations`. In the current
+  layout those directories do not exist, so `functions` returns `[]` and
+  `migrations.pending` is always `0` (making `migrations.current` always
+  true and `health` never degraded on migrations). Treat the manifest's
+  `functions` list and pending count as unpopulated until this is fixed.
+- The migration table queried is `schema_migrations`; the CLI migrator
+  records to `_spine_migrations`, so `applied`/`latest` may also read
+  empty — the query failure is swallowed silently.
+- Because `verified` requires a readable `.spine-manifest.json` at the
+  function's working directory, `health` reports `degraded` on deploys
+  where the file is absent from the bundle.
+- This endpoint answers "what is deployed" questions for agents: prefer
+  `health` for a liveness signal and `manifest` for version/integrity; do
+  not rely on `functions` or `openapi` path lists being complete.

package/.framework/src/instructions/reference/functions/tests.md

@@ -0,0 +1,97 @@
+---
+title: tests
+audience: both
+kind: reference
+verified_against: f69c37d
+verified_on: 2026-10-30
+source_files:
+  - .framework/functions/tests.ts
+  - .framework/functions/_shared/middleware.ts
+  - .framework/public/_redirects
+---
+
+# tests
+
+## What it does
+
+Read-only reporting over the `test_runs` and `test_results` tables —
+the suites a Spine deployment records about itself. Lists recent runs,
+returns one run with its results, or summarizes the last 30 days per
+suite.
+
+## Route and methods
+
+`/.netlify/functions/tests`, reachable as `/api/tests` (`/api/*` →
+`/.netlify/functions/:splat`). No method check — the action comes from
+query parameters:
+
+| Call | Result |
+|---|---|
+| `GET /api/tests` or `?action=list` | recent `test_runs` (default limit 20, max 100; `?suite=` filters) |
+| `GET /api/tests?action=get&id=<uuid>` or any `?id=`/`?run_id=` | one run + its `test_results` |
+| `GET /api/tests?action=stats` | per-suite rollup for `unit`, `integration`, `api`, `ui` over the last 30 days |
+
+Note that `?id=` (or `?run_id=`) forces the `get` path regardless of
+`action`.
+
+## Authentication
+
+Standard `createHandler` plus `requireSystemContextWithAudit` — the
+caller must resolve to a `system_admin` principal (human with the
+`system_admin` role, a `*:*` machine key, or the internal secret).
+Anonymous is rejected with 401 by the middleware.
+
+All queries use `adminDb` (service role).
+
+## Request contract
+
+Query params only: `action` (`list` default), `id`/`run_id`, `suite`,
+`limit` (capped at 100).
+
+## Response contract
+
+This handler returns raw `{ data, error }` objects as its result, so the
+middleware wraps them again: the body is
+`{ "data": { "data": <payload>, "error": null }, "error": null, "meta": {...} }`.
+
+- `list` → `data.data` = array of run summaries (`id, suite, status,
+  started_at, finished_at, duration_ms, total, passed, failed, skipped,
+  triggered_by`).
+- `get` → `data.data` = the run row plus `results` (`test_results` ordered
+  by `status`, `file`, `name`).
+- `stats` → `data.data` = one object per suite: `{ suite, total_runs,
+  last_status, last_run_at, last_passed, last_failed, last_total }`.
+
+**Errors are also embedded in the payload**: failures return
+`{ error: "<msg>", status: <code> }` inside `data` — the outer HTTP status
+is still 200.
+
+## Side effects
+
+None — read-only.
+
+## Errors
+
+Embedded under `data` (HTTP 200):
+
+| Payload | Meaning |
+|---|---|
+| `{ error: "id is required for action=get", status: 400 }` | `action=get` without `id`/`run_id` |
+| `{ error: "Test run not found: …", status: 404 }` | unknown run id |
+| `{ error: "<postgrest msg>", status: 500 }` | query failure |
+
+Non-system-admin callers get the `requireSystemContextWithAudit` denial,
+which — like the other responses — is returned as the handler result and
+lands inside `data` (a `{ statusCode, headers, body }` response object)
+with HTTP 200, not as a real 403.
+
+## Agent notes
+
+- Always unwrap twice: `body.data.data` is the payload, `body.data.error`
+  is where this endpoint reports failures. Do not rely on HTTP status
+  codes here.
+- `stats` only covers the four hard-coded suites (`unit`, `integration`,
+  `api`, `ui`); other suite names appear in `list` but never in `stats`.
+- To record runs, write `test_runs`/`test_results` through
+  `/api/admin-data` (raw entity path) or service-role code — this endpoint
+  does not create them.

package/.framework/src/instructions/reference/functions/timers.md

@@ -0,0 +1,117 @@
+---
+title: timers
+audience: both
+kind: reference
+verified_against: f69c37d
+verified_on: 2026-10-30
+source_files:
+  - .framework/functions/timers.ts
+  - .framework/functions/_shared/middleware.ts
+  - .framework/functions/system-cron.ts
+  - .framework/migrations/020_schema_reconciliation.sql
+  - .framework/migrations/068_cron_schedule_timer_rpcs.sql
+  - .framework/public/_redirects
+---
+
+# timers
+
+## What it does
+
+CRUD plus `toggle` over `public.timers` — the config rows that bind a
+`pipeline_id` to a schedule (`timer_type` + `config`, with
+`next_execution`/`execution_count` bookkeeping columns). **This function
+never executes anything**: firing is done by `system-cron`, which polls
+the `get_due_timers` RPC each tick and calls `runPipeline` as a
+`timer:<id>` machine principal, then reschedules via
+`update_timer_after_run` (`.framework/migrations/068_cron_schedule_timer_rpcs.sql`).
+
+## Route and methods
+
+`/.netlify/functions/timers`, reachable as `/api/timers` (`/api/*` →
+`/.netlify/functions/:splat`, `.framework/public/_redirects`). Dispatch
+is by **`?action=` only** — the HTTP method is ignored:
+
+| Call | Handler |
+|---|---|
+| `/api/timers?action=list` | `list` |
+| `/api/timers?action=get&id=<uuid>` | `get` |
+| `/api/timers?action=create` | `create` |
+| `/api/timers?action=update` (`id` in body or query) | `update` |
+| `/api/timers?action=remove` (`id` in body or query) | `remove` |
+| `/api/timers?action=toggle` (`id` in body or query) | `toggle` |
+
+A bare `GET /api/timers` with no action throws
+`Unsupported action: undefined`. Conversely `GET ?action=create` reaches
+`create` (with a null body → required-field error).
+
+## Authentication
+
+Standard `createHandler` auth: `x-api-key` → machine principal,
+`x-internal-secret` → internal principal, `Bearer <jwt>` → human;
+anonymous → 401.
+
+- `list`/`create` require `ctx.accountId`; `create`/`update`/`remove`/
+  `toggle` additionally require a person+account context.
+- No role gate — system admins use `adminDb`, other humans use
+  JWT-scoped `ctx.db` (RLS); machines run service-role.
+- Reads are sanitized: non-admins get `{ id }` per row (timer rows carry
+  no `design_schema`).
+
+## Request contract
+
+**list** (query): `app_id`, `timer_type`, `is_active`
+(`'true'`/`'false'`). Ordered by `name`; no pagination parameters.
+
+**get** (query): `id`.
+
+**create** (body): `name`, `timer_type` required. Optional: `app_id`,
+`description`, `config` `{}`, `pipeline_id`, `metadata` `{}`.
+`account_id` and `created_by` stamped from context. (Columns
+`timer_type`, `pipeline_id`, `next_execution`, `execution_count` were
+added by `.framework/migrations/020_schema_reconciliation.sql`.)
+
+**update** (`id` in body or query + body): allowlist — `name`,
+`description`, `config`, `pipeline_id`, `metadata`, `is_active`,
+`timer_type`, `app_id`.
+
+**remove** (`id` in body or query): **hard delete**.
+
+**toggle** (`id` in body or query): flips `is_active` to its negation —
+no target-state argument.
+
+## Response contract
+
+Standard `{ data, error, meta }` envelope. `list`/`get`/`create`/
+`update`/`toggle` → sanitized row(s); `remove` → `{ success: true }`.
+
+## Side effects
+
+- `toggle` and `update` stamp `updated_at`; `is_active` is what
+  `get_due_timers` gates on (plus `next_execution <= now`).
+- `emitLog` writes `timer.created` / `.updated` / `.deleted` /
+  `.toggled` audit rows to `public.logs`.
+- No schedule recomputation happens here — `next_execution` is only
+  advanced by `update_timer_after_run` during `system-cron` ticks.
+
+## Errors
+
+| Condition | HTTP | Message |
+|---|---|---|
+| No account context (list, create) | 500 | `Account context required` |
+| Missing create fields | 500 | `name and timer_type are required` |
+| No person+account context | 500 | `User context (person and account) required` |
+| Missing `id` | 500 | `Timer ID is required` |
+| Missing row (remove/toggle) | 500 | `Timer not found` |
+| Missing/unknown action | 500 | `Unsupported action: <action>` |
+
+## Agent notes
+
+- `next_execution` is a real column (migration 020) but is not in the
+  create/update allowlists — it is only read by `get_due_timers` and
+  advanced by `update_timer_after_run` on `system-cron` ticks.
+- `toggle` flips blindly — read the row first if you need a target
+  state; concurrent toggles can cancel each other.
+- `remove` is a hard delete (unlike `triggers`, which soft-deletes).
+- Distinct roles: `timers` = config CRUD; `system-cron` = firing;
+  `pipelines` = what runs; `pipeline-executions` = run records. See
+  [Automation](../../concepts/automation.md).

package/.framework/src/instructions/reference/functions/trigger-scheduler.md

@@ -0,0 +1,102 @@
+---
+title: trigger-scheduler
+audience: developers, agents
+kind: reference
+verified_against: f69c37d
+verified_on: 2026-10-30
+source_files:
+  - .framework/functions/trigger-scheduler.ts
+  - .framework/functions/system-cron.ts
+  - .framework/functions/_shared/principal.ts
+  - netlify.toml
+---
+
+# trigger-scheduler
+
+## What it does
+
+The clock for Spine automation. `netlify.toml` declares it a Netlify
+scheduled function at `* * * * *` — Netlify Clockwork invokes it every
+minute. The file itself is a thin shim: it detects whether the invocation
+is a genuine scheduled tick (which arrives with no credentials) and runs
+the shared `system-cron` handler under a fabricated system context, or
+falls through to `system-cron`'s normal middleware auth for everything
+else. All actual work — due schedules, due timers, threshold alerts,
+delivery retries, log cleanup — lives in `system-cron`.
+
+## Route and methods
+
+Two invocation shapes:
+
+1. **Netlify scheduled invocation** — the platform calls
+   `/.netlify/functions/trigger-scheduler` once per minute with a schedule
+   indicator header and a `{"next_run": "<ISO timestamp>"}` body.
+2. **Manual HTTP call** — any authenticated request to the same path
+   (useful for forcing a tick in dev). This falls through to
+   `system-cron`'s `createHandler` auth path.
+
+## Authentication
+
+Clockwork detection (`isNetlifyScheduledInvocation`) requires **both**:
+
+- a schedule indicator — `x-netlify-event: schedule` (production canonical
+  header), `x-nf-event: schedule` (Netlify CLI emulation), or a
+  `Netlify Clockwork` user-agent when neither header is present — **and**
+- a JSON body whose `next_run` field is a string.
+
+Requiring the body shape means a bare spoofed header or UA alone cannot
+authenticate. A detected tick fabricates a `RequestContext` carrying
+`SYSTEM_PRINCIPAL` (`machine`, `machineType: 'internal'`, `*:*` scope) and
+`adminDb`, then calls the cron handler directly — `createHandler`
+short-circuits when `requestId` + `principal` are already present, so no
+second auth resolution happens.
+
+Anything that fails detection goes through `system-cron`'s own checks:
+
+- `x-internal-secret` header → resolves to `SYSTEM_PRINCIPAL` via
+  `resolvePrincipal`, which passes the internal-machine gate.
+- `?api_key=` or `?scheduler_key=` query param equal to
+  `SCHEDULER_API_KEY`.
+- any other internal machine principal.
+- anonymous requests → 401 from `createHandler`; authenticated but
+  unauthorized → 403 plus a `system_cron.unauthorized_access` audit event.
+
+## Request contract
+
+- Scheduled tick: schedule indicator header + body
+  `{"next_run": "<ISO timestamp>"}`.
+- Manual call: `?api_key=$SCHEDULER_API_KEY` (or `?scheduler_key=`), or an
+  `x-internal-secret` header. No request body is read.
+
+## Response contract
+
+Whatever `system-cron`'s handler returns — a JSON summary of the tick
+(`executed`, `success`, `failed`, `skipped`, `thresholds_evaluated`,
+`logs_cleaned`, `deliveries_retried`, `results[]`), or an auth/500 error.
+See the `system-cron` reference page for the full shape.
+
+## Side effects
+
+None of its own — every effect belongs to `system-cron`. The only
+local decision is which auth path the invocation takes.
+
+## Errors
+
+- 401 — unauthenticated non-scheduled request (from `createHandler`).
+- 403 — authenticated but not an internal machine principal and no valid
+  scheduler key (from `system-cron`, with audit event).
+- 500 — bubbled from `system-cron` failures (`system_cron.error` audit).
+
+## Agent notes
+
+- The function is deliberately a shim: scheduled-invocation plumbing has
+  changed across Netlify environments before (CLI vs production header
+  names), so detection is centralized here rather than inside the worker.
+  If a future Netlify rename breaks ticks, this file is the fix point —
+  the Clockwork user-agent check is already the third fallback indicator.
+- A "401 every minute" symptom in production usually means the scheduler
+  hit the non-Clockwork path — check that the invocation carries a
+  recognized schedule header **and** the `next_run` body.
+- There is no per-tick idempotency key: two overlapping ticks would both
+  run `system-cron`. Due-item claiming relies on the RPCs and row state,
+  not on tick uniqueness.