spine-framework 0.4.4 → 0.5.0-rc.4

package/.framework/src/instructions/operations/migrations.md

@@ -0,0 +1,125 @@
+---
+title: Database Migrations
+audience: both
+kind: reference
+verified_against: e13127d
+verified_on: 2026-10-30
+source_files:
+  - .framework/cli/commands/migrate.ts
+  - .framework/cli/commands/migrations.ts
+  - .framework/cli/commands/install-app.ts
+  - .framework/cli/commands/upgrade.ts
+  - .framework/migrations/001_core_schema.sql
+  - .framework/migrations/003_core_system_data.sql
+  - .framework/migrations/069_log_stats_window.sql
+---
+
+# Database Migrations
+
+Spine's migration system is deliberately simple: numbered `.sql` files applied
+once, in filename order, each inside its own transaction, recorded in a
+`_spine_migrations` ledger table. Everything runs through
+`npx spine-framework migrate` (`.framework/cli/commands/migrate.ts`).
+
+## File layout and numbering
+
+- **Framework migrations** live in `.framework/migrations/` — sealed,
+  upgrade-synced. Naming is `NNN_snake_case_description.sql` with a
+  three-digit prefix; plain lexicographic sort is the apply order.
+- **Custom migrations** live in `custom/migrations/` — project-owned, applied
+  **after** all framework files in the same run (skip them with
+  `--framework-only`).
+- Files are filtered to `*.sql` not starting with `.`
+  (`collectMigrations` in `.framework/cli/commands/migrate.ts`).
+- **Numbering gaps are expected.** `22–31`, `39–41`, and `63` are reserved or
+  consolidated numbers — a gap is not a missing migration (see
+  [Database Reference](../reference/database.md)). The same number can also
+  appear twice across the framework and custom dirs: the ledger key is the
+  **filename**, not the numeric prefix, so `062_send_email_action.sql` and a
+  custom `062_*.sql` do not collide.
+
+## The `_spine_migrations` ledger
+
+`migrate` creates and reads `public._spine_migrations`
+(`id SERIAL`, `filename TEXT UNIQUE NOT NULL`, `applied_at TIMESTAMPTZ`).
+
+Per file, inside one transaction: split the SQL into statements, run each,
+insert the ledger row, `COMMIT`. On the **first** statement error the file
+rolls back, the error is printed, and the run stops — later files are never
+attempted. Files already in the ledger are skipped, so reruns are safe.
+
+Note there is a **second, unrelated ledger**: `migrations list` /
+`migrations status` read `supabase_migrations.schema_migrations` — the table
+Supabase's own tooling maintains, which `spine-framework migrate` never
+writes. `migrations status` therefore reports local files as `pending` even
+when `_spine_migrations` shows them applied. Use the `_spine_migrations`
+table itself as the authoritative "what ran" record.
+
+## Running migrations
+
+```bash
+npx spine-framework migrate                # apply pending framework + custom
+npx spine-framework migrate --framework-only
+npx spine-framework migrate --dry-run
+```
+
+- Connects **directly to Postgres** — `db.<project-ref>.supabase.co:5432`,
+  user `postgres`, database `postgres`. `SUPABASE_URL` must be a hosted
+  `https://<ref>.supabase.co` URL and `SUPABASE_DB_PASSWORD` must be set
+  (`--db-password` overrides; `.env` is preloaded).
+- `--dry-run` prints **every collected file**, not just pending ones — the
+  pending filter lives inside `runMigrations`, which dry-run never reaches.
+  To see real pending state, query `_spine_migrations` or run it for real
+  (applied files skip cleanly).
+- `npx spine-framework install-app` also calls `runMigrations` before
+  seeding; a migration failure aborts the install (`--no-seed` installs
+  files only and tolerates missing DB credentials).
+
+## Custom migration rules
+
+1. One file per change, `NNN_name.sql`, in `custom/migrations/`.
+2. **Idempotent by convention.** The ledger makes files apply-once, but
+   drift happens (rows applied by hand, shared dev DBs, ledger restores).
+   Write `IF NOT EXISTS`, `ON CONFLICT DO NOTHING` / `DO UPDATE`, and
+   `CREATE OR REPLACE` so a re-run is survivable — framework files follow
+   the same convention.
+3. Never edit a file after it has been applied anywhere — the ledger keys on
+   filename and will not re-run it. Add a new file instead.
+4. Never put runtime-record seed data (types, roles, pipelines) in a
+   migration when an app manifest `seed` block can express it — seeds get
+   conflict handling, validation, and reconciliation that raw SQL does not.
+
+## Rolling out schema changes with app updates
+
+Two different channels, don't confuse them:
+
+- **Framework schema changes** ship inside `.framework/migrations/` in the
+  npm package. `npx spine-framework upgrade` syncs the files and *reports*
+  new migrations — it never runs them. `npx spine-framework migrate` applies
+  them (see [Upgrade](upgrade.md)).
+- **App-level schema** is data, not DDL: `public.types` `design_schema`
+  rows, roles, pipelines ship in the app package's `manifest.json` `seed`
+  block, applied by `install-app`/`seed-app`. `install-app` copies only
+  `index.tsx`, `manifest.json`, `package.json`, `pages`, `components`,
+  `hooks`, `utils`, `functions`, `seed`, `public` into `custom/apps/<slug>/`
+  — there is no per-app migrations directory. If an app truly needs DDL,
+  ship it as a `custom/migrations/` file in the project repo.
+
+Production rollout order: deploy code first or migrate first — both work
+because migrations are additive-by-convention, but the safe default is
+**migrate, then deploy**, so new code never meets a schema older than it
+expects.
+
+## Checklist
+
+- [ ] `SUPABASE_URL` + `SUPABASE_DB_PASSWORD` set (`.env` or `--db-password`).
+- [ ] `SELECT filename FROM _spine_migrations ORDER BY filename` reviewed for drift before major upgrades.
+- [ ] New custom migration is idempotent and numbered after the highest local prefix.
+- [ ] After `spine-framework upgrade`: `npx spine-framework migrate` run — upgrade only syncs files.
+- [ ] `migrations status` not trusted as the source of truth (it reads `supabase_migrations`).
+
+## Related
+
+- [Database Reference](../reference/database.md) — table catalog and numbering gaps
+- [Upgrade](upgrade.md) · [Deployment](deployment.md) · [Troubleshooting](troubleshooting.md)
+- [CLI Reference](../reference/cli.md) — `migrate`, `migrations`, `install-app` flags

package/.framework/src/instructions/operations/security-checklist.md

@@ -0,0 +1,148 @@
+---
+title: Security Checklist
+audience: both
+kind: guide
+verified_against: e13127d
+verified_on: 2026-10-30
+source_files:
+  - .framework/functions/api-keys.ts
+  - .framework/functions/_shared/hash-api-key.ts
+  - .framework/functions/_shared/principal.ts
+  - .framework/functions/_shared/middleware.ts
+  - .framework/functions/_shared/permissions.ts
+  - .framework/functions/_shared/resolve-credentials.ts
+  - .framework/functions/_shared/security.ts
+  - .framework/functions/security.ts
+  - .framework/functions/logs.ts
+  - .framework/functions/debug-auth.ts
+  - .framework/migrations/049_security_blocklist.sql
+  - netlify.toml
+  - .env.example
+---
+
+# Security Checklist
+
+Operational security posture for a Spine deployment — what the platform
+already enforces, and the surfaces you must keep an eye on yourself.
+Related reading: [Auth & API Keys](../concepts/auth-and-api-keys.md),
+[Permissions](../concepts/permissions.md),
+[Environment Variables](../reference/env-vars.md).
+
+## Secrets that must stay server-side
+
+- **`SUPABASE_SERVICE_ROLE_KEY` bypasses RLS entirely.** It is read only by
+  functions and the CLI (`adminDb` in `.framework/functions/_shared/db.ts`)
+  and must never appear under a `VITE_*` name — `VITE_*` vars are compiled
+  into the public browser bundle. Older docs recommended
+  `VITE_SUPABASE_SERVICE_ROLE_KEY`; that is an anti-pattern the audit
+  flagged (S-004). If it ever shipped, rotate the key.
+- `netlify.toml` `SECRETS_SCAN_OMIT_KEYS` whitelists only vars that are
+  **meant** to be public (`VITE_SUPABASE_URL`, `VITE_SUPABASE_ANON_KEY`,
+  `APP_BASE_URL`, `SITE_URL`, …). Do not add secrets to it.
+- `INTERNAL_INVOKE_SECRET`, `SCHEDULER_API_KEY`, `RESEND_API_KEY`,
+  `OPENAI_API_KEY`, `NETLIFY_ACCESS_TOKEN` — server-side Netlify env only.
+- `.env` is gitignored and excluded from the shipped file set; no framework
+  command writes it (see [Project Map](../guide/02-project-map.md)).
+
+## API key hygiene
+
+- Keys mint as `sk_<uuid>`; only `key_prefix` (first 8 chars) and a
+  `pbkdf2$<iters>$<salt>$<hash>` value (100k iterations, SHA-256,
+  `timingSafeEqual` verify) are stored
+  (`.framework/functions/api-keys.ts`,
+  `.framework/functions/_shared/hash-api-key.ts`). The raw key is returned
+  once as `raw_key`; `omitSecretFields` strips `key_value`/`key_hash` from
+  `list`, `get`, and `revoke` responses.
+- Grant **scopes**, not `*:*`. A machine key with `*:*` resolves to the
+  `system_admin` role and effectively becomes an RLS-immune superuser
+  (`.framework/functions/_shared/principal.ts`). `is_internal` keys are for platform machinery.
+- Every key has a rate limit (default `100`/60s) plus a per-IP ceiling
+  (`SPINE_IP_RATE_LIMIT`, default `300`/60s); usage lands in
+  `api_key_usage_logs` — read via `/api/api-keys?action=usage-logs` before
+  revoking so you see what the key touched.
+- Revoke, don't delete: `POST /api/api-keys?action=revoke` sets
+  `is_active=false`. Audit `api_keys` rows for a non-null `key_value`
+  column — a legacy column that can hold plaintext (the S-001 incident
+  class); nothing in `_shared` reads it, but `select('*')` surfaces used to
+  return it.
+
+## Credential sanitization
+
+- Integration credentials resolve server-side only:
+  `credentials.<field>.source = 'env_var'` reads `process.env` inside the
+  function (`.framework/functions/_shared/resolve-credentials.ts`); literal `value` entries are
+  stored in `integrations.credentials` — treat that column as secret.
+- `integrations` is a second-surface table: non-system-admin and machine
+  reads are sanitized to minimal fields by `.framework/functions/_shared/permissions.ts`.
+- Outbound stage configs (`http_request`) reference the `generic-http`
+  integration — credentials live on the integration row, never in the
+  pipeline stage JSON.
+
+## Ban / blocklist surface
+
+`/api/security` (`.framework/functions/security.ts`) is gated by
+`requireSystemContextWithAudit` — system-admin only. Actions:
+`flagged_accounts`, `ban_user`, `unban_user`, `blocklist`,
+`blocklist_add`, `blocklist_revoke`. A ban is three layers:
+
+1. `security_blocklist` rows (person/email/account/IP) — the authoritative
+   layer; enforced per-request in `createHandler` (`isBlockedRequest`) and
+   on public endpoints (`isBlockedIp`).
+2. Supabase Auth ban (`auth.admin.updateUserById`, ~100y `ban_duration`).
+3. Netlify edge deny rule `spine-security:blocklist` — only when
+   `NETLIFY_ACCESS_TOKEN` + `NETLIFY_SITE_ID` are set; skipped silently
+   otherwise (auth/edge failures are audited, not thrown).
+
+Caveat: blocklist enforcement **fails open** when
+`security_blocklist` is absent (pre-049 installs) — a missing table means
+no blocking, not an error.
+
+## Endpoint exposure notes
+
+- `debug-auth` (`/api/debug-auth`) is gated by *any authenticated
+  principal*, not system-admin — it reports principal type and
+  `SET`/`NOT_SET` env flags. Diagnostic, not product surface; consider
+  whether it belongs in a hardened production deploy
+  (see [debug-auth](../reference/functions/debug-auth.md)).
+- `/api/system` (`manifest`, `health`, `openapi`) likewise answers any
+  authenticated principal — the OpenAPI spec and manifest are not
+  admin-only information.
+- Public endpoints (`invites`, `integration-routes`, registration flows)
+  run through `createPublicHandler` — anonymous by design; their abuse
+  controls are IP blocklist, rate limits, and Turnstile, not auth.
+
+## Known ungated surfaces
+
+Honest list — the audit found these; mitigate by key discipline until
+they are tightened:
+
+- **`logs?action=cleanup`** uses `ctx.db` with no `isSystemAdmin` check and
+  no account scoping (`.framework/functions/logs.ts`). Machine principals
+  get `adminDb` from `getPrincipalDb` — so **any valid API key can delete
+  log rows across all accounts**. Mitigation: issue machine keys narrowly,
+  watch `api_key_usage_logs`, rotate on suspicion.
+- **`api-keys` `list`/`get`** select `*` via `adminDb` for system admins —
+  and `ctx.db` is *also* `adminDb` for machine principals, so a scoped
+  machine key reads `api_keys` metadata cross-account
+  (`api_keys` rows are stripped of `key_value`/`key_hash`, but slugs,
+  integration links, rate limits, and scopes are visible).
+- **Function-level RLS is uneven** — mutating handlers branch
+  `isSystemAdmin ? adminDb : ctx.db`; a scoped machine key still gets
+  `adminDb`, so "machine keys are always RLS-bounded" is not a safe
+  assumption anywhere `ctx.db` is used without an explicit account filter.
+
+## Checklist
+
+- [ ] No `VITE_`-prefixed secret anywhere; service-role key server-only.
+- [ ] `INTERNAL_INVOKE_SECRET`, `SCHEDULER_API_KEY`, `APP_BASE_URL` set; `SITE_URL` pinned to the real origin for CORS.
+- [ ] Machine keys carry least-privilege scopes (never `*:*` for third parties), rate limits, and expiry where possible.
+- [ ] `api_keys` rows audited for legacy `key_value` plaintext.
+- [ ] Blocklist tested end-to-end (a banned IP gets 403); Netlify edge sync env vars set if edge bans are wanted.
+- [ ] `debug-auth` and `/api/system` exposure reviewed for the environment.
+- [ ] Machine-principal surfaces above understood — `ctx.db` = service role for machines.
+
+## Related
+
+- [Auth & API Keys](../concepts/auth-and-api-keys.md) · [Permissions](../concepts/permissions.md)
+- [Errors and Envelopes](../reference/api/errors-and-envelopes.md) · [api-keys](../reference/functions/api-keys.md) · [security](../reference/functions/security.md)
+- [Troubleshooting](troubleshooting.md) · [Deployment](deployment.md)

package/.framework/src/instructions/operations/troubleshooting.md

@@ -0,0 +1,170 @@
+---
+title: Troubleshooting Playbook
+audience: both
+kind: guide
+verified_against: d0c49d4
+verified_on: 2026-10-30
+source_files:
+  - .framework/functions/_shared/trigger-engine.ts
+  - .framework/functions/_shared/pipeline-runner.ts
+  - .framework/functions/_shared/resolve-credentials.ts
+  - .framework/functions/_shared/permissions.ts
+  - .framework/functions/_shared/middleware.ts
+  - .framework/functions/_shared/principal.ts
+  - .framework/functions/logs.ts
+  - .framework/functions/pipeline-executions.ts
+  - .framework/functions/deliveries.ts
+  - .framework/functions/api-keys.ts
+  - .framework/cli/commands/migrate.ts
+  - .framework/cli/commands/migrations.ts
+  - .framework/migrations/003_core_system_data.sql
+  - scripts/assemble-functions.sh
+---
+
+# Troubleshooting Playbook
+
+## Where to look first
+
+Everything operational lands in three places (see
+[Observability](../concepts/observability.md) for the model):
+
+1. **`pipeline_executions`** — did the run exist, and where did it stop?
+   Admin: **Observability → Executions**; API: `/api/pipeline-executions`.
+2. **`logs`** — search by `trace_id`/`span_name`, or `action=search` on the
+   error text (**Observability → Logs** / `/api/logs`). `trigger.fired` /
+   `pipeline.failed` audit rows bracket the run. Netlify function logs
+   carry what never reached the DB (missing env vars, dispatch failures).
+3. **`webhook_deliveries`** — outbound HTTP stages and integrations
+   (**Configs → Deliveries** / `/api/deliveries`): `last_error`,
+   `response_status`, `next_attempt_at`; `POST /api/deliveries?action=retry`
+   re-fires an outbound delivery.
+
+## Common failure signatures
+
+### Trigger-dispatched pipeline never runs
+
+Symptom: `pipeline_executions` row sits at `queued`, no spans — until the
+sweeper marks it `failed` (~10 min) with `stale execution: runner died
+before terminal write (swept)`
+(`.framework/functions/_shared/pipeline-runner.ts`). Causes, in order of
+likelihood:
+
+1. **`INTERNAL_INVOKE_SECRET` unset.** `dispatchTriggerPipeline` logs
+   `INTERNAL_INVOKE_SECRET not set — cannot dispatch trigger pipeline` and
+   returns — the queued row is the only artifact
+   (`.framework/functions/_shared/trigger-engine.ts`). Fix: set it
+   (`openssl rand -hex 32`), redeploy.
+2. Wrong `SITE_URL`/`URL` — dispatch POSTs to
+   `{SITE_URL}/.netlify/functions/pipeline-runner-background`; a mis-pointed
+   base URL sends the run elsewhere (or times out after 5s).
+
+### `integration.credential_*` on `generic-http`
+
+These stages run through `executeOutbound` with
+`integration_slug: 'generic-http'`
+(`.framework/functions/_shared/pipeline-runner.ts`). Read the audit event
+first — only one of them is fatal:
+
+- `integration.credential_env_missing` **fails the call** — an `env_var`
+  entry names a variable that isn't set in the function env
+  (`.framework/functions/_shared/resolve-credentials.ts`). Fix the
+  integration row (Configs → Integrations) or deploy the env var.
+- `integration.credential_missing` / `credential_empty` with
+  `skipped: true` are **informational** — no credential is configured, so
+  the request went out with no `Authorization` header
+  (`.framework/functions/integration-outbound.ts`). If the *target* then
+  401s, the endpoint actually needs auth: set a real `api_token`
+  (`source: "env_var"` preferred) or pass `action_config.headers`.
+
+### `403` for a system admin: the `system-admin` dead role key
+
+`record_permissions` keys match `roles.slug` **verbatim** — nothing
+normalizes hyphens. A design schema granting `"system-admin"` silently
+denies the real `system_admin` role
+(`.framework/functions/_shared/permissions.ts`, the `DOC-LEARNING` note on
+`resolveFirstSurfacePermissions`). Valid keys are real slugs
+(`system_admin`, `member`, `support`, …) or the `all` wildcard. Same class
+of bug: a `people.role_id` that resolves to no role gets the default-deny
+result — check the `people` → `roles` join, not the UI.
+
+### `member` grants do nothing
+
+`member` and `member_admin` are seeded by
+`.framework/migrations/073_seed_member_roles.sql`. On a database that
+predates that migration the slugs exist only when an app seeds them
+(`install-app` applies `seed/roles.json`) or someone creates them via
+`/api/roles`. With no `member` row in `public.roles`,
+`record_permissions.member` grants nobody anything — a guaranteed-deny
+that looks like a permissions bug.
+
+### Trigger never fires at all
+
+- Event names are **singular-kind + `_created|_updated|_deleted`** —
+  `item_created`, `message_updated`, `person_created`. `ENTITY_EVENT_NAME`
+  maps `items`→`item`, `people`→`person`, etc.
+  (`.framework/functions/_shared/trigger-engine.ts`), so a trigger
+  configured for `items_created` or `record_created` matches nothing.
+- `triggers.is_active` must be true; `config.entity_type`, `config.type_slug`,
+  and `config.filters` further narrow matches — check `trigger_executions`
+  and `trigger.fired`/`trigger.failed` audit rows.
+- The admin-data delete path does not fire `item_deleted` triggers.
+
+### HTTP 200 that is actually a denial
+
+The envelope is always `{data, error, meta}`; a handler that *returns* (not
+throws) a refusal produces HTTP 200 with the denial inside `data`
+([Errors and Envelopes](../reference/api/errors-and-envelopes.md)) —
+e.g. `POST /api/api-keys?action=validate` returns
+`{is_valid: false, error: 'Insufficient permissions'}` at 200
+(`.framework/functions/api-keys.ts`). Inspect `data`/`error`, not just the
+status. Conversely, a thrown `Error` without `statusCode` maps to **500**,
+so many 500s are intentional refusals (`Invalid action or method`,
+`Record not found or access denied`), not crashes — read the `error` text.
+
+### Edits to functions "do nothing"
+
+`netlify dev` serves the assembled snapshot in
+`.assembled/netlify/functions`, not `.framework/functions`. Re-run
+`npm run assemble` (or restart `dev:netlify`) after editing function code —
+see [Deployment](deployment.md).
+
+### `dev:netlify` fails with "Could not acquire required port"
+
+Ports 8888 (`netlify dev`) and 3001 (Vite) are fixed in manifest-protected
+files — there is no supported alternate-port path, and moving only one
+side cross-connects the `/api` proxy to whatever still listens on 8888
+(often a second Spine project's dev stack). `scripts/dev-netlify.sh` only
+kills processes it owns; find the holder (`lsof -i :8888 -i :3001`) and
+stop it yourself. `Failed to fetch AI Gateway token … Unauthorized` in the
+same output is unrelated Netlify CLI noise, not a Spine fault.
+
+### Migration state looks wrong
+
+`migrations list`/`status` read the `_spine_migrations` ledger (`filename`,
+`applied_at`) that `migrate` writes and compare it against the collected
+files (`.framework/cli/commands/migrations.ts`). `migrate --dry-run` lists
+every collected file, not only pending ones, and file numbering is
+non-contiguous — retired migrations are removed rather than renumbered, so
+gaps are expected (see [Migrations](migrations.md)).
+
+## Reading `pipeline_executions`
+
+| Status | Meaning |
+|---|---|
+| `pending` | created via `/api/pipeline-executions` POST, not yet claimed |
+| `queued` | trigger reserved the row; dispatch may or may not have landed |
+| `running` | `pipeline-runner-background` claimed it |
+| `completed` / `failed` | terminal; `error_message` + per-stage results name the failure |
+| `cancelled` | manually cancelled via PATCH |
+
+Any of `pending`/`queued`/`running` untouched for >10 minutes is swept to
+`failed` by `sweepStaleExecutions` — a wedged row is evidence, not the bug;
+find why it never reached its terminal write. Useful queries:
+`/api/pipeline-executions?status=failed&pipeline_id=…`,
+`?action=running`, `?action=stats`.
+
+## Related
+
+- [Observability](../concepts/observability.md) — logs/executions/deliveries model
+- [Errors and Envelopes](../reference/api/errors-and-envelopes.md) — the 200-wrapped-denial contract
+- [Security Checklist](security-checklist.md) · [Migrations](migrations.md) · [Deployment](deployment.md)

package/.framework/src/instructions/operations/upgrade.md

@@ -0,0 +1,116 @@
+---
+title: Framework Upgrades
+audience: both
+kind: reference
+verified_against: e13127d
+verified_on: 2026-10-30
+source_files:
+  - .framework/cli/commands/upgrade.ts
+  - .framework/cli/lib/project-files.ts
+  - scripts/checksum-manifest.mts
+  - scripts/verify-integrity.sh
+  - .framework/templates/project.json
+  - netlify.toml
+---
+
+# Framework Upgrades
+
+`npx spine-framework upgrade` syncs a project's framework-owned files to the
+`spine-framework` version installed in `node_modules`
+(`.framework/cli/commands/upgrade.ts`). It is a file sync with an integrity
+gate, not a git operation — and the executing copy is always the **new**
+package's code, since `npx` resolves `node_modules`.
+
+## What is sealed
+
+The frozen manifest `.spine-manifest.json` (written by `init`, regenerated
+by `upgrade`) records a sha256 for every file under the protected trees —
+`.framework/`, `config/`, `scripts/`, plus `bin/`
+(`scripts/checksum-manifest.mts`, `PROTECTED_DIRS` + `--include-dir bin`).
+The file set itself is declared by `.framework/templates/project.json`
+(`framework_dirs`, `framework_files`, `config_files`, `root_scripts`,
+`bin`).
+
+Also recorded in the manifest: `framework_version`,
+`framework_dependencies` (so the package.json merge knows which deps the
+framework owns), a `tree_digest`, and a separate `agent_files` hash map
+(more below). `npm run verify` re-hashes the tree and fails on drift;
+`SPINE_INTEGRITY=warn` downgrades that to warnings.
+
+## The integrity gate
+
+Before writing anything, upgrade re-hashes every manifest entry:
+
+- Any **modified or missing** protected file → abort with a per-file list.
+  The only way through is `--force`, which overwrites modified files and
+  restores missing ones (each is logged).
+- This is the safety rail: overwriting `.framework/` loses nothing the user
+  owned, because the gate proves they never changed it. **Never edit sealed
+  trees** — the change is either blocked or silently reverted later.
+- `--dry-run` computes the full report (added / changed / removed / agent
+  assets / new migrations) and writes nothing.
+
+## What an upgrade does, in order
+
+1. Preconditions: refuses to run from the vendored copy (`package.json`
+   `name` must be `spine-framework`), requires `.spine-manifest.json`,
+   and refuses to downgrade (`cmpSemver` on `framework_version`).
+2. Integrity gate (above).
+3. Collects the new file set, then applies copies; **deletions are confined
+   to `.framework/`** — files outside it that upstream dropped are warned
+   about and left in place.
+4. Syncs agent assets (below).
+5. Merges `package.json`: framework scripts and the dependency set are
+   rewritten, `spine-framework` pinned exactly; a dependency is removed
+   **only** if the previous framework version owned it and the new one
+   doesn't. Customer scripts/deps survive.
+6. Re-renders `netlify.toml`: framework blocks come from the package;
+   `[functions."custom_*"]` sections in the existing file (with their
+   attached comments) are re-appended verbatim — that's where app-owned
+   scheduled functions live. A changed render leaves `netlify.toml.bak`.
+7. Regenerates `.spine-manifest.json` at the new version.
+8. Lists new `.framework/migrations/*.sql` files — reported, **never run**.
+   `npx spine-framework migrate` applies them.
+
+## Agent files merge behavior
+
+`profile.agent_files` (`.framework/templates/project.json`) declares two
+merge modes, applied by `syncAgentFiles` in
+`.framework/cli/lib/project-files.ts`:
+
+- **`framework-block` → `AGENTS.md`.** The template renders
+  `{{SPINE_VERSION}}`, then the marker-delimited block —
+  `<!-- spine-framework:begin version=X.Y.Z -->` …
+  `<!-- spine-framework:end -->` — is replaced in place. Everything outside
+  the markers is customer content and is preserved. A file with no markers
+  gets the block **prepended** (never overwritten). Hashes go into the
+  manifest's `agent_files` map — deliberately not `files`, so edits outside
+  the markers don't count as drift.
+- **`dir-replace` → `.devin/skills`.** Only `spine-*` subdirectories are
+  removed and re-copied from the package. Sibling dirs with any other name
+  are customer-owned and untouched.
+
+## What upgrade never touches
+
+`custom/`, `spine.config.json`, `.env`, `.spine/`, `netlify.toml`
+`[functions."custom_*"]` sections, and the customer-owned parts of
+`AGENTS.md` / `.devin/skills`. If a run would destroy your work, the design
+has failed — that is the promise the sealed layer exists to keep (see
+[Core vs Custom Boundary](../concepts/core-vs-custom.md)).
+
+## Upgrade checklist
+
+- [ ] `npm run verify` clean *before* starting — a dirty tree only aborts later anyway.
+- [ ] `npm install spine-framework@<version>` first; upgrade syncs to whatever `node_modules` has.
+- [ ] `npx spine-framework upgrade --dry-run`, read the report, then run for real.
+- [ ] Never pass `--force` to make local `.framework/` edits stick — it does the opposite. Move the change to `custom/` first.
+- [ ] `npm install` after (dependency set changed), then `npm run verify`.
+- [ ] `npx spine-framework migrate` for the reported new migrations ([Migrations](migrations.md)).
+- [ ] Re-run `npm run assemble` / restart `netlify dev` so `.assembled/` picks up the new files ([Deployment](deployment.md)).
+- [ ] Commit the result — manifest, `netlify.toml`, `package.json`, `.devin/skills`, `AGENTS.md` are all part of the upgrade diff.
+
+## Related
+
+- [Anatomy of an installed project](../guide/02-project-map.md) — sealed vs customer trees
+- [Migrations](migrations.md) · [Deployment](deployment.md) · [Troubleshooting](troubleshooting.md)
+- [CLI Reference](../reference/cli.md) — `upgrade`, `verify`, `doctor`

package/.framework/src/instructions/pipeline-executions.md

@@ -0,0 +1,62 @@
+---
+title: Pipeline Executions
+audience: both
+kind: entity
+verified_against: e13127d
+verified_on: 2026-10-30
+source_files:
+  - .framework/src/pages/admin/PipelineExecutionsPage.tsx
+  - .framework/functions/pipeline-executions.ts
+  - .framework/functions/_shared/pipeline-runner.ts
+  - .framework/functions/_shared/trigger-engine.ts
+---
+
+## What this is for
+
+**Pipeline Executions** (`/spine-framework/admin/observability/executions`) is the run history for every pipeline — manual runs, trigger fires, timer runs, and webhook calls. Each row is a `pipeline_executions` record: who ran it, when, how long, and whether it finished. Use it to confirm a pipeline fired, find the run that failed, and measure cadence and duration.
+
+## Fields and settings
+
+- **Stats cards** — Completed, Failed, Running counts (`/api/pipeline-executions?action=stats`; the Pipelines list reuses `action=stats&date_from=` for "runs today").
+- **Search** — matches pipeline id/name fragments and status text client-side.
+- **Status filter** — `pending`, `running`, `completed`, `failed`, `cancelled`.
+- **Sortable columns** — Pipeline (joined name + execution id prefix), Status, Started (`started_at`), Duration (`duration_ms`), Triggered By (`triggered_by_person` full name/email when known).
+- **Row click** — navigates to `observability/executions/:id`. Note: the same list component renders at that URL — there is currently no separate detail view.
+- The record itself carries `trigger_data` (the payload that fired it), `result.stages` (per-stage output/duration), and `error_message`.
+
+**Downstream effects**
+
+- The trigger engine inserts a `queued` row before dispatching to `pipeline-runner-background`; `runPipeline` claims it `running`, then writes `completed`/`failed` with per-stage results. A row stuck `running`/`queued` >10 min is swept to `failed` by the cron sweeper.
+- `action=cleanup` (server-side) deletes rows older than `days_to_keep` (default 30) — this page is a window, not an archive.
+- `trigger_count`/`last_triggered` on the Triggers page and "Last Run"/"runs today" on the Pipelines page all read from this table.
+
+## Minimal configuration
+
+Nothing to configure — executions are produced by running pipelines. To produce one: run a pipeline manually, fire its trigger, or let a timer come due, then filter Status = `failed` to check the result.
+
+## Full configuration
+
+The API supports more than the UI exposes:
+
+- `GET /api/pipeline-executions?action=list|running|stats|get&id=` — reads
+- `POST action=cleanup` — `days_to_keep` (default 30), optional `status_filter`
+- `PATCH action=start|complete|cancel` — lifecycle writes used by the runner; `complete` marks the row `failed` when `error_message` is present
+
+## Example
+
+A ticket triage trigger fired at 09:12 but no assignment happened. Filter Status = `failed`, find the 09:12 row for the triage pipeline — `error_message` names the stage ("Action not found for stage type"), and `result.stages` shows stages 0–1 succeeded before stage 2 threw.
+
+## Core files
+
+- `.framework/src/pages/admin/PipelineExecutionsPage.tsx` — this page
+- `.framework/functions/pipeline-executions.ts` — API (list/get/running/stats/cleanup + start/complete/cancel)
+- `.framework/functions/_shared/pipeline-runner.ts` — writes execution rows, stage results, and the >10min stale sweeper
+- `.framework/functions/_shared/trigger-engine.ts` — pre-creates the `queued` row before dispatch
+- Table: `public.pipeline_executions`
+
+## Agent notes
+
+- "Stuck at running/queued" is the sweeper's job — a row still `running` an hour later means the background function died mid-run; it will be marked `failed` on the next sweep.
+- `duration_ms` is wall time of the whole run; per-stage timings live in `result.stages[*].durationMs`.
+- The page has no cancel/retry buttons — lifecycle actions exist on the API but the UI is read-only.
+- `executions/:id` renders the same list — deep links to a single execution do not open a detail view today.