spine-framework 0.4.3 → 0.5.0-rc.4

package/.framework/src/instructions/concepts/concurrency-and-idempotency.md

@@ -0,0 +1,125 @@
+---
+title: "Concurrency and idempotency"
+audience: developers, agents
+kind: concept
+verified_against: f69c37d
+verified_on: 2026-10-29
+source_files:
+  - .framework/functions/_shared/message-sequence.ts
+  - .framework/functions/_shared/agent-runner.ts
+  - .framework/functions/_shared/pipeline-runner.ts
+  - .framework/functions/_shared/trigger-engine.ts
+  - .framework/functions/_shared/retry-deliveries.ts
+  - .framework/functions/integration-outbound.ts
+  - .framework/functions/_shared/kb.ts
+  - .framework/cli/commands/migrate.ts
+  - .framework/migrations/001_core_schema.sql
+  - .framework/migrations/035_rate_limits.sql
+  - .framework/migrations/054_kb_ingest_lease.sql
+  - .framework/migrations/064_episode_leases.sql
+  - .framework/migrations/066_episode_steal_thread_merge.sql
+  - .framework/migrations/067_episode_lease_stale_steal.sql
+---
+
+# Concurrency and idempotency
+
+Spine runs on serverless functions: an invocation can die at the response
+flush, a client can retry a request that already succeeded, and two instances
+can race the same resource. Spine's answer is consistent: **the database
+decides**. Ordering, single-flight, and deduplication are enforced by
+constraints and atomic RPCs — never by reading state and deciding in memory.
+
+## Ordered writes: message sequencing
+
+`public.messages` has `UNIQUE(thread_id, sequence)`
+(migration `001_core_schema.sql`) — the hard invariant for conversation order.
+Two writers honor it differently:
+
+- `_shared/message-sequence.ts` (`insertMessageSequenced`) is the simple path:
+  read the max `sequence` for the thread, insert `max + 1`, retry up to 3
+  times on a `23505` duplicate-key error. The retry exists because a racing
+  writer may take the same number; the constraint makes the conflict visible.
+- The agent runner keeps a per-instance counter (`threadSeqCounter`) seeded by
+  one `COUNT(*)` per thread, then allocates locally — a batch of tool-result
+  messages gets consecutive numbers up front so `Promise.all` inserts stay in
+  order. A `23505` triggers a reseed + retry. The counter is an optimization;
+  the unique constraint remains the backstop.
+
+## Pipeline executions: claim, don't create-twice
+
+Trigger dispatches reserve a `pipeline_executions` row with
+`status: 'queued'` *before* calling the background runner. `runPipeline` then
+**claims** that row (`queued → running` on `id = executionId AND status =
+'queued'`) rather than inserting a second execution — a retried or duplicated
+dispatch converges on the same row. If the claim finds nothing, the runner
+falls back to inserting, so an execution record always exists. Rows stuck in
+`queued`/`running`/`pending` for over 10 minutes are swept to `failed` — a
+runner that died mid-flight can never write its own terminal status.
+
+Individual stages can also be idempotent: `create_record` accepts
+`config.dedupe { field, value, account_scoped? }` — when a record matching
+`field = value` already exists, the stage returns the existing row
+(`deduped: true`) instead of inserting a duplicate.
+
+## Webhook deliveries: retry in place
+
+An outbound delivery is one `webhook_deliveries` row for its whole life:
+`pending → processing → delivered | pending | failed`. Retries — automatic
+(`retryPendingDeliveries` each cron tick) or manual
+(`deliveries?action=retry`) — call `integration-outbound` with the existing
+`delivery_id`, which reloads the row, reuses its payload, and increments
+`attempt_count` in place; they never create a second delivery. Backoff is
+exponential (`30s × 2^attempt`, 1h cap, ±20% jitter) until `max_attempts`
+turns the row `failed`; the sweep is bounded (`attempt_count < 10`, 100 rows
+per pass) so one poisoned integration can't monopolize the tick. Inbound
+deliveries get the same row lifecycle, but inbound retry is not implemented —
+a failed inbound delivery stays visible for manual handling.
+
+## Leases: single-flight with fencing
+
+Long-running agent episodes coordinate through `public.episode_leases`
+(migration `064`; service-role only). `acquire_episode_lease` is atomic: an
+`INSERT … ON CONFLICT (thread_id) DO NOTHING` fast path, then a
+`FOR UPDATE` conflict path returning `claimed` + a fresh `lease_token`, or
+`held` + the live holder's `driver_id`. The same driver re-acquiring renews;
+an expired lease is taken over with a new token.
+
+The token is a **fence**, not just a lock:
+
+- `heartbeat_episode_lease(thread_id, lease_token)` extends the TTL only when
+  the token matches — `false` means "you were displaced; stop writing now."
+- `release_episode_lease` is token-gated for the same reason.
+- `merge_thread_data(thread_id, patch, lease_token)` (migration `066`) is the
+  only way episode checkpoints are written: an atomic jsonb merge whose
+  `WHERE` clause refuses the write when a different token holds the lease —
+  so a fenced driver's late write is dropped instead of resurrecting stale
+  state.
+
+Migration `067` adds `p_steal_detached` / `p_stale_seconds`: request-bound
+drivers may steal a lease from a `detached:`-prefixed holder **only** when its
+heartbeat is older than `p_stale_seconds` (default 30). Detached drivers are a
+liveness fallback, never an owner that blocks live work. The same lease shape
+guards knowledge ingest: `document_leases` with
+`acquire_document_lease`/`release_document_lease` (migration `054`), claimed
+by content hash in `_shared/kb.ts`.
+
+## Atomic counters and bookkeeping
+
+- `rate_limit_hit` (migration `035`) upsert-increments the `rate_limits`
+  window counter in one statement — concurrent invocations can't double-spend
+  a window.
+- `record_trigger_fire` (migration `064`) bumps trigger bookkeeping atomically
+  on each fire.
+
+## Migrations: ordered, transactional, ledgered
+
+`spine-framework migrate` (`.framework/cli/commands/migrate.ts`) applies
+`.framework/migrations/*.sql` (plus `custom/migrations/` unless
+`--framework-only`) in **filename order** — the numeric prefix is the order.
+Each file runs inside one transaction together with its ledger insert into
+`_spine_migrations` (`filename UNIQUE`, `applied_at`): a file either lands
+complete with its ledger row or rolls back, and the run stops at the first
+failure. Re-runs skip anything already in the ledger (`--dry-run` lists
+pending without touching the DB). The connection is derived from
+`SUPABASE_URL` (a hosted `*.supabase.co` URL only — no credentials, ports, or
+paths) plus `SUPABASE_DB_PASSWORD`.

package/.framework/src/instructions/concepts/core-vs-custom.md

@@ -0,0 +1,124 @@
+---
+title: Core vs Custom Boundary
+audience: developers, agents
+kind: concept
+verified_against: f69c37d
+verified_on: 2026-10-29
+source_files:
+  - .framework/templates/project.json
+  - .framework/cli/commands/upgrade.ts
+  - .framework/cli/lib/project-files.ts
+  - .framework/cli/commands/migrate.ts
+  - scripts/boundary-check.sh
+  - scripts/assemble-frontend.sh
+  - scripts/assemble-functions.sh
+  - config/vite.config.ts
+  - config/tsconfig.json
+  - config/tsconfig.functions.json
+  - netlify.toml
+---
+
+# Core vs Custom Boundary
+
+Spine projects are two trees with different owners: `.framework/` is the
+sealed, upgradeable package; `custom/` is project-owned code and data the
+package never writes. Everything about the boundary — the alias config, the
+assembly, the upgrade command, the lint script — exists to keep one promise:
+**upgrading the framework can never destroy your work.**
+
+## What lives where
+
+**`.framework/` (core, framework-owned)** — the React app shell and admin UI
+(`.framework/src/`), Netlify functions and shared middleware
+(`.framework/functions/`), ordered SQL migrations (`.framework/migrations/`),
+the CLI (`.framework/cli/`), this instruction book
+(`.framework/src/instructions/`), schemas, seeds, tests, and templates
+(`.framework/templates/`). The shipped file set is declared in
+`.framework/templates/project.json` under `framework_dirs`, `framework_files`,
+`config_files`, `root_scripts`, and `bin`.
+
+**`custom/` (project-owned)** — apps under `custom/apps/<slug>/` (each with a
+`manifest.json` and `index.tsx`), migrations under `custom/migrations/`,
+function handlers under `custom/functions/` and `custom/apps/<slug>/functions`,
+plus tests and vendored packages. This tree is created by `init` / `create-app`
+/ `install-app`; it is not part of the npm package and `upgrade` never writes
+into it.
+
+**Shared but owned:** `config/` (Vite + tsconfig), `netlify.toml`, and the
+`scripts/` assembly-and-integrity scripts ship with the framework and are
+synced on upgrade, with one carve-out — `netlify.toml`
+`[functions."custom_*"]` sections belong to apps and are re-appended verbatim
+by `renderNetlifyToml` in `.framework/cli/lib/project-files.ts`.
+
+## Why the boundary exists: upgrade safety
+
+`npx spine-framework upgrade` is a file sync, not a git operation:
+
+1. It reads `.spine-manifest.json` — frozen per-file hashes written at install
+   — and **aborts if any framework file was locally modified or deleted**
+   unless `--force` is passed (`.framework/cli/commands/upgrade.ts`,
+   `runUpgrade`).
+2. It copies the new package's framework files over yours and deletes only
+   `.framework/` entries that disappeared upstream. Files outside
+   `.framework/` that upstream dropped are warned about, never removed.
+3. It re-merges `package.json` (framework scripts + dependency pin), re-renders
+   `netlify.toml` preserving `custom_*` sections, and regenerates the manifest.
+   New migrations are reported, never run — `npx spine-framework migrate`
+   applies them.
+
+Because overwrite-on-upgrade is guaranteed, the only way a local change inside
+`.framework/` survives is `verify`/`upgrade` treating the tree as sealed. The
+frozen manifest is the integrity contract: `scripts/verify-integrity.sh`
+re-hashes the tree and compares.
+
+## How the boundary is enforced
+
+- **`scripts/boundary-check.sh`** — fails the build if `.framework/` code
+  imports `custom/` or `custom_*` modules, if core source references
+  known app slugs, or if `custom/` code reaches `.framework/` via relative
+  imports instead of the `@core` alias.
+- **Aliases** — `config/tsconfig.json` maps `@core/*` → `.framework/src/*` and
+  `@custom/*` → `custom/*`; `config/vite.config.ts` maps `@core` to the
+  assembled `src` tree. `config/tsconfig.functions.json` maps `@shared/*` to
+  the assembled `_shared/` directory — custom functions import `./_shared/...`
+  at runtime.
+- **Assembly** — `scripts/assemble-frontend.sh` copies `.framework/src` as the
+  base layer and each `custom/apps/<slug>` verbatim into `.assembled/src/<slug>/`;
+  `scripts/assemble-functions.sh` flattens framework + custom handlers into
+  `.assembled/netlify/functions/` and treats duplicate function basenames as a
+  hard error, while letting `custom/` overlay `_shared/` helpers.
+- **Manifest** — `.spine-manifest.json` + `scripts/verify-integrity.sh` seal
+  the `.framework/` tree between upgrades.
+
+## How to extend core without crossing the boundary
+
+Custom code reaches core through three sanctioned channels, in order of
+preference:
+
+1. **Data** — `public.apps` rows, `public.types` `design_schema`, roles,
+   link types, triggers, pipelines. Most "customization" is data.
+2. **Custom apps** — `custom/apps/<slug>/` with a `manifest.json`, mounted by
+   `route_prefix`/`renderer` and loaded dynamically by
+   `.framework/src/components/CustomAppLoader.tsx`.
+3. **Custom functions** — new handler files alongside the assembled
+   `_shared/` layer, for behavior admin-data can't express. Never wrappers
+   for runtime CRUD — that is `/api/admin-data`'s job.
+
+## What breaks if you cross it
+
+- **Core → custom import:** `boundary-check.sh` fails; even if merged, the next
+  `upgrade` overwrites or refuses, and the core now fails in projects lacking
+  that app.
+- **Custom → `.framework/` relative import:** `boundary-check.sh` check 2b
+  fails; the path also breaks under the assembled layout Vite actually builds.
+- **Edit inside `.framework/`:** `verify` reports drift and `upgrade` aborts —
+  or `--force` silently reverts your change. Either way the change was never
+  yours to keep.
+- **Custom runtime-CRUD function:** compiles fine and quietly bypasses
+  permissions, validation, audit, and triggers — the worst kind of violation
+  because nothing fails.
+
+## See also
+
+- [The Spine Way](spine-way.md) — the ten rules this boundary is part of
+- [Apps and Renderers](apps-and-renderers.md) — how `custom/` apps register and mount

package/.framework/src/instructions/concepts/design-schema.md

@@ -0,0 +1,152 @@
+---
+title: Design Schema
+audience: developers, agents
+kind: concept
+verified_against: f69c37d
+verified_on: 2026-10-29
+source_files:
+  - .framework/src/types/types.ts
+  - .framework/functions/types.ts
+  - .framework/functions/admin-data.ts
+  - .framework/functions/_shared/schema-utils.ts
+  - .framework/functions/_shared/permissions.ts
+  - .framework/functions/_shared/resolve-ids.ts
+  - .framework/src/components/app-shell/GenericListPage.tsx
+  - .framework/src/components/app-shell/GenericDetailPage.tsx
+  - .framework/src/hooks/useListSchema.ts
+  - .framework/migrations/058_account_data_field_declarations.sql
+  - .framework/migrations/060_account_person_record_permissions.sql
+---
+
+# Design Schema
+
+`design_schema` is the JSON contract on a `public.types` row that declares
+everything the platform needs to store, validate, permission, and render one
+kind of runtime record — without code. It is the single most important
+structure in Spine: get it right and permissions, validation, and the generic
+UI all work for free; get it wrong or leave it out and records are created
+unpermissionable and unrenderable.
+
+The TypeScript shape is `DesignSchema` in `.framework/src/types/types.ts`:
+`fields`, `views`, `record_permissions`, plus optional `scope` and
+`functionality`.
+
+## Where it lives and how it propagates
+
+- The **type definition** holds the canonical copy in
+  `public.types.design_schema` (created/updated only via
+  `.framework/functions/types.ts`, which requires `system_admin`).
+- **Every record created through `admin-data` is stamped** with the type's
+  `design_schema` and `validation_schema` at insert time
+  (`.framework/functions/admin-data.ts`, `create`). The stamped copy on the
+  record — not the type row — is what the permission engine reads at request
+  time (`.framework/functions/_shared/permissions.ts`,
+  `sanitizeFirstSurfaceRecordData` / `validateFirstSurfaceUpdatePermissions`
+  both refuse when the record carries no `record_permissions`).
+- **Editing a type does not restamp existing records.** Migrations that change
+  a type schema re-stamp rows explicitly — see
+  `.framework/migrations/058_account_data_field_declarations.sql` and
+  `.framework/migrations/060_account_person_record_permissions.sql` for the
+  pattern.
+- `validation_schema` is a derived artifact: `generateValidationSchema` in
+  `.framework/functions/_shared/schema-utils.ts` extracts `data_type`,
+  `required`, `validation`, `options`, and `reference_*` from `fields` and runs
+  on every type create and on every update that touches `design_schema`
+  (`.framework/functions/types.ts`). Never hand-edit it — change
+  `design_schema` and let the API regenerate.
+
+## The sections
+
+### `scope`
+
+`"account"` (default when omitted) or `"platform"`. At create time `admin-data`
+reads `designSchema.scope ?? 'account'`; `platform` records are stamped with
+the `spine-system` account's id instead of the caller's account. Child records
+created against a parent (via `thread_id`, `target_id`, `parent_id`) also
+inherit the parent's stamped scope. See
+[Scope and Accounts](scope-and-accounts.md).
+
+### `fields`
+
+A flat map of `snake_case` field names to `FieldDefinition`
+(`.framework/src/types/types.ts`): `data_type`, `label`, `required`, plus
+optional `system`, `placeholder`, `validation` (pattern, min/max, lengths,
+`reference_kind`/`reference_type`, `currency_code`), `options` (required for
+`select`/`multiselect`/`radio`), and `permissions` (per-role `read`/`write`
+overrides — see [Permissions](permissions.md)).
+
+The `system` flag controls storage: `system: true` fields are top-level record
+columns; all others live under `record.data.<field>` — see
+[Runtime Records](runtime-records.md).
+
+Known `data_type` values (the authoritative switch is `sanitizeFieldData` in
+`.framework/functions/_shared/schema-utils.ts`): `text`, `textarea`,
+`rich_text`, `email`, `phone`, `url`, `number`, `currency`, `range`, `date`,
+`datetime`, `boolean`, `checkbox`, `select`, `multiselect`, `radio`, `color`,
+`file`, `image`, `json`, `reference`, `address`. Unknown types pass through
+unsanitized — they are not rejected, they are unvalidated.
+
+### `views`
+
+A map of view names to view configs. `default_list` and `default_detail` are
+what the generic renderer consumes:
+
+- `default_list` (`type: "list"`, `display: "table" | "card" | "board"`,
+  `fields` map with `sortable`/`display_type`, `default_sort`, optional
+  `filters`/`stats`) becomes columns in `GenericListPage`
+  (`.framework/src/components/app-shell/GenericListPage.tsx`) — the schema
+  itself resolves via `useListSchema`, which prefers a stamped record's
+  `design_schema` and falls back to the types API.
+- `default_detail` (`type: "detail"`, `sections[]` of `{title, fields}`) drives
+  `GenericDetailPage`'s `SchemaDetailForm`; each section field resolves its
+  `FieldDefinition` and reads `record[field]` or `record.data[field]` by
+  `system` (`.framework/src/components/app-shell/GenericDetailPage.tsx`).
+
+`display_type` values for view fields come from `ViewFieldConfig` in
+`.framework/src/types/types.ts` (`input`, `text`, `badge`, `timestamp`,
+`select`, `date_picker`, `file_upload`, …). An app with `renderer = "generic"`
+is broken without these views; a `custom` app may omit them but loses the
+generic admin/runtime fallback.
+
+### `record_permissions`
+
+`{ "<role_slug>": ["create" | "read" | "update" | "delete", ...] }` — the base
+grant for the record, evaluated per request against the caller's role plus the
+`'all'` wildcard role. This is the contract evaluated in
+`.framework/functions/_shared/permissions.ts`; the full model is in
+[Permissions](permissions.md).
+
+If a type has no `record_permissions`, `admin-data` create merges a permissive
+default (`{ all: ["create","read","update","delete"], fields: {} }`) before
+stamping — an explicit empty object would instead deny everything downstream,
+so always author them deliberately.
+
+### `functionality`
+
+Optional automation bindings the runtime interprets — e.g.
+`functionality.lineage` (parent/child resolution for items, read in
+`admin-data` list/get) and the integration/constraint bindings declared in
+`FunctionalityBindings` (`field_map` sync, `unique`/`conditional_required`/
+`immutable` constraints) in `.framework/src/types/types.ts`. Set it to `null`
+until the automation binding you need is confirmed live.
+
+## What each consumer reads
+
+| Consumer | Reads |
+|---|---|
+| `admin-data` create | `fields`/`record_permissions`/`scope` for stamping, gating, and account resolution |
+| `permissions.ts` | `record_permissions`, `fields.*.permissions` — on the **record's** stamped schema |
+| `schema-utils.ts` | generated `validation_schema` for `sanitizeFieldData` / `formatFieldData` |
+| `GenericListPage` | `views.default_list` (columns, sort, filters, stats) |
+| `GenericDetailPage` | `views.default_detail`, `fields`, `display_field` |
+| `useEntityRecord` | `fields.*.system` to split system vs `data` writes |
+
+## Authoring minimum
+
+Every type needs, at minimum: `fields` (each with `data_type`, `label`,
+`required`), `record_permissions` (at least `system_admin`), and — for generic
+apps — `default_list` + `default_detail`. `slug` must be unique per `kind`
+(enforced by the `types_kind_slug_unique` index in
+`.framework/migrations/015_types_kind_slug_unique.sql`), and `kind` must match
+the runtime table the type governs. The entity page
+[../types.md](../types.md) has minimal and full examples.

package/.framework/src/instructions/concepts/integrations-and-webhooks.md

@@ -0,0 +1,127 @@
+---
+title: "Integrations and webhooks"
+audience: developers, agents
+kind: concept
+verified_against: f69c37d
+verified_on: 2026-10-29
+source_files:
+  - .framework/functions/integrations.ts
+  - .framework/functions/integration-outbound.ts
+  - .framework/functions/integration-routes.ts
+  - .framework/functions/deliveries.ts
+  - .framework/functions/_shared/webhook-registration.ts
+  - .framework/functions/_shared/webhook-registry.ts
+  - .framework/functions/_shared/resolve-credentials.ts
+  - .framework/functions/_shared/retry-deliveries.ts
+  - .framework/functions/_shared/notify.ts
+  - .framework/functions/adapters/generic-webhook-outbound.ts
+  - .framework/migrations/004_core_rpc_and_broker.sql
+---
+
+# Integrations and webhooks
+
+An **integration** (`public.integrations`) is a named bundle of `config` +
+`credentials` that outbound calls and inbound webhooks share. Spine routes both
+directions through `public.webhook_deliveries`, so every attempt is a durable
+row you can list, inspect, and retry — there is no fire-and-forget.
+
+## The integration row
+
+`integrations` (`.framework/functions/integrations.ts`) requires `name`,
+`integration_type`, and `provider` on create; it also carries `app_id`,
+`version`, `config`, `credentials`, `metadata`, and a computed
+`is_configured` flag.
+
+Credentials live in a dedicated `credentials` jsonb column (added by migration
+`004`) shaped `{ <field>: { source: "value" | "env_var", value: string } }`.
+`resolve-credentials` resolves them **server-side only**, inside the functions
+runtime: `source: "env_var"` reads `process.env[value]`; `source: "value"`
+returns the literal string. Every resolution (or failure) emits an
+`integration.credential_*` audit event. Non-system-admin, non-machine API reads
+of an integration row are sanitized to minimal fields — credential values are
+never returned to ordinary users, though they are stored in the database (use
+`env_var` sources for the tightest posture).
+
+## Outbound calls
+
+`integration-outbound` (`.framework/functions/integration-outbound.ts`) is the
+single egress path — `callOpenAI` in `_shared/llm.ts`, the `http_request`,
+`webhook`, `send_email`, and `agent_inference` pipeline stages, and manual
+calls all go through its `execute`:
+
+1. **Resolve the integration**: account-scoped row matching
+   `integration_slug` first, then a system integration (`account_id IS NULL`).
+   Both must be `is_active`.
+2. **Open a delivery row**: inserts `webhook_deliveries` with
+   `direction: 'outbound'`, `status: 'pending'`, `attempt_count: 0`, and
+   `max_attempts` from `action_config.max_attempts` →
+   `integration.config.retry_policy.max_attempts` → default 3.
+3. **Resolve the credential** (`action_config.credential_field`, default
+   `api_token`) — skipped when the entry is absent or an empty
+   `source: 'value'` literal, which means "call without auth"; the adapter
+   then decides whether a key is required — and **pick an adapter**:
+   `action_config.adapter || integration.config.outbound_adapter`. Built-ins:
+   `adapters/openai-outbound`, `adapters/generic-webhook-outbound`,
+   `adapters/resend-outbound`; anything else is dynamically imported as a
+   sibling module exposing `buildRequest`.
+4. **Send**: the adapter builds `{ url, method, headers, body }`; the function
+   fetches it, then updates the delivery to `delivered` (with
+   `response_status`, `response_body`, `duration_ms`) or `pending`/`failed`.
+5. Optionally, `action_config.update_record` stamps
+   `last_outbound_status` / `last_outbound_at` back onto the source item.
+
+## Delivery and retry semantics
+
+Both directions share the same retry math: `computeNextAttempt` is exponential
+backoff — `30s * 2^attempt`, capped at 1 hour, with up to 20% jitter. A failed
+attempt stays `pending` with `next_attempt_at` set until `attempt_count`
+reaches `max_attempts`, when it becomes `failed`.
+
+Retries are driven by `retryPendingDeliveries`
+(`.framework/functions/_shared/retry-deliveries.ts`), which `system-cron` runs
+every tick (see [automation](./automation.md)). It sweeps `webhook_deliveries`
+rows that are `pending`, `attempt_count < 10`, and `next_attempt_at` due (or
+null), up to 100 per pass, and re-invokes `integration-outbound` with
+`delivery_id` under the system principal — the `delivery_id` path reuses the
+existing row instead of inserting a duplicate. You can also retry one delivery
+manually via `POST /api/deliveries?action=retry { id }`
+(`.framework/functions/deliveries.ts`); manual retry is outbound-only. Inbound
+retry is not implemented — inbound failures are counted and logged.
+
+`deliveries` list/get are account-scoped reads over the same table, filterable
+by `direction`, `status`, `integration_id`, `api_key_id`.
+
+## Inbound webhooks
+
+`integration-routes` (`.framework/functions/integration-routes.ts`) is a
+`createPublicHandler` endpoint — it resolves the caller as an **API-key
+machine principal** (see [auth](./auth-and-api-keys.md)):
+
+1. `?slug=` names the integration; the caller's API key must belong to the
+   same account, and keys pinned to `integration_id` or `app_id` can only hit
+   those integrations.
+2. The event name comes from the `x-event-type` header or
+   `integration.config.event_path` (a dot-path into the payload).
+3. `config.handler_name || provider || 'generic-webhook'` is resolved through
+   the **webhook registry** (`_shared/webhook-registry.ts`): the
+   `webhook_handlers` table maps a handler name to a `function_name`, scoped
+   app → account → global fallback. The named module is dynamically imported
+   (`custom/…` paths resolve into `custom/`) and must export `adapt`,
+   `handler`, or a default function.
+4. The adapter returns `{ type_slug, data, title?, status? }` — the route then
+   creates an `items` record through `admin-data` (full permission/type/audit
+   pipeline) and marks the inbound delivery `delivered`. Adapter errors marked
+   `retryable` leave it `pending`.
+
+Handlers are registered with `registerWebhookHandler`
+(`_shared/webhook-registration.ts`), an upsert on `(app_id, account_id,
+name)`; `deregisterWebhookHandler` soft-deletes (`is_active=false`,
+`is_deleted=true`).
+
+## Notifications
+
+`_shared/notify.ts` resolves pipeline `send_notification` recipients: each
+entry is either a person UUID or `role:<slug>`, resolved to active `people`
+rows in the caller's account via the `role_id → roles.slug` join. Anything
+else is skipped with a warning. The resulting ids feed the `notifications`
+insert in the pipeline runner.

package/.framework/src/instructions/concepts/observability.md

@@ -0,0 +1,116 @@
+---
+title: "Observability: logs, deliveries, executions"
+audience: developers, agents
+kind: concept
+verified_against: f69c37d
+verified_on: 2026-10-29
+source_files:
+  - .framework/functions/logs.ts
+  - .framework/functions/observability.ts
+  - .framework/functions/deliveries.ts
+  - .framework/functions/pipeline-executions.ts
+  - .framework/functions/item-progress.ts
+  - .framework/functions/_shared/logging.ts
+  - .framework/functions/_shared/tracing.ts
+  - .framework/functions/_shared/audit.ts
+  - .framework/migrations/069_log_stats_window.sql
+---
+
+# Observability: logs, deliveries, executions
+
+Everything an operator needs lands in a small set of queryable tables. There is
+no external telemetry service: `public.logs` is the unified sink, and the
+`logs`, `observability`, `deliveries`, and `pipeline-executions` functions are
+the read APIs over it.
+
+## What gets written, and where
+
+`public.logs` holds three overlapping kinds of rows:
+
+- **Audit/log events** — `emitLog`, `emitAudit`, and `emitActivity` in
+  `_shared/audit.ts` insert business events (`trigger.fired`,
+  `pipeline.completed`, `integration.outbound.failed_http`,
+  `api_key.created`, `registration.completed`, …) with actor, target, and
+  details.
+- **Spans** — `emitSpan` writes trace spans (`span_name`, `trace_id`,
+  `span_id`, `parent_span_id`, `duration_ms`, `outcome`). `_shared/tracing.ts`
+  implements W3C `traceparent` parsing/formatting, and `traceDb` wraps the
+  request's DB client so every query emits a `db.<operation>` span — a single
+  `trace_id` stitches the request, its queries, and the pipeline stages it
+  caused.
+- **Operation logs** — `logAdminDataOperation` in `_shared/logging.ts` writes
+  admin-data CRUD records, gated by resolved log levels.
+
+Log **levels** are `'off' | 'timing' | 'warnings' | 'errors' | 'all'`, resolved
+per request from `system_settings` rows (`key='logging'`, global plus scoped
+overrides); the default is `['timing', 'errors']`. Under the default, errors
+and timing rows are written; `warnings`-level and `all`-level verbosity require
+opt-in settings. A request's `x-log-level` header can raise verbosity for that
+call.
+
+## Reading logs
+
+`logs` (`.framework/functions/logs.ts`) is system-admin oriented (system
+admins query with the service client across accounts; account members get
+their RLS-scoped view):
+
+- `action=account` / `action=target` / `action=person` — filtered lists.
+- `action=search` — `ILIKE` message search.
+- `action=stats` — calls the `get_log_stats` RPC (totals, error count,
+  distinct levels/sources). **Note:** since migration `069`, when no
+  `p_date_from` is passed the window defaults to the **last 7 days** — a call
+  that omits the date range returns 7-day stats, not all-time stats.
+- `action=cleanup` — deletes rows older than `days_to_keep` (default 30).
+  `system-cron` also runs `cleanup_old_logs` on the 00:00 UTC tick, so
+  retention is enforced continuously.
+
+`observability` (`.framework/functions/observability.ts`) aggregates the same
+rows into dashboard metrics — actions `event_volume`, `error_rate`,
+`latency_percentiles`, `pipeline_stats`, `top_actors`, `ai_support_metrics`,
+`cleanup` — scoped to the caller's account unless the principal is
+system-admin.
+
+## Executions and deliveries
+
+Two sibling tables answer "did it run / did it arrive":
+
+- `public.pipeline_executions` — one row per pipeline run with `status`
+  (`queued` → `running` → `completed`/`failed`/`cancelled`), per-stage results,
+  `error_message`, `duration_ms`, and the `trigger_data` that caused it
+  (`.framework/functions/pipeline-executions.ts`; `action=running`,
+  `action=stats`, `action=cleanup`). Rows that never reach a terminal state
+  are swept to `failed` after 10 minutes.
+- `public.webhook_deliveries` — one row per inbound/outbound webhook attempt
+  with `direction`, `status` (`pending`/`processing`/`delivered`/`failed`),
+  `attempt_count`, `next_attempt_at`, `last_error`, `response_status`, and
+  `response_body` (`.framework/functions/deliveries.ts`; `action=retry`).
+
+`item-progress` (`.framework/functions/item-progress.ts`) is a different lens:
+per-person, per-item progress rows (`status`, `score`, `attempts`) that follow
+the type's `design_schema.pipeline` stage list (default
+`not_started → in_progress → completed`). Transitions must move forward
+unless `force: true` is passed; `started_at`/`completed_at` are stamped on
+first entry. Use it when you need per-user completion state rather than
+execution forensics.
+
+## Debugging a failed automation
+
+Work backwards through the tables:
+
+1. **`pipeline_executions`** — find the run. `status: 'failed'` +
+   `error_message` and per-stage results name the failing stage.
+   `status: 'queued'` past 10 minutes = dispatch never arrived (sweeper will
+   mark it failed); `status: 'running'` stuck = the runner died (same
+   sweeper).
+2. **`logs`** — search by `trace_id` (from the request) or `span_name`
+   (`pipeline.run`, `pipeline.stage.<handler>`, `db.<operation>`); audit rows
+   `trigger.fired` / `trigger.failed` / `pipeline.completed` /
+   `pipeline.failed` bracket the run. Check log levels if expected `timing`
+   rows are absent.
+3. **`webhook_deliveries`** — for outbound stages/integrations, the delivery
+   row carries `response_status`, `response_body`, `last_error`, and
+   `next_attempt_at`; retry with `action=retry`.
+4. **`api_key_usage_logs`** (`api-keys?action=usage-logs`) — whether the
+   machine caller ever arrived, and from where.
+5. **`observability` actions** — confirm the blast radius (`error_rate`,
+   `pipeline_stats`, `top_actors`) rather than chasing a single event.