@intentius/behold 0.25.0 → 0.27.0

package/AGENTS.md

@@ -9,7 +9,8 @@ apply creds.
 
 - **behold** serves the live, mixed-substrate graph (and, later, the deployment-lanes
   timeline). It reads; every write it offers is a command it starts (an Op, a
-  `chant run`, a forge dispatch), and on a terragucci repo it starts none.
+  `chant run`, a forge dispatch), and on a terragucci or SQL Yodeler repo it
+  starts none.
 - **chant's MCP** is where the real capabilities live. Prefer it over shelling.
   - Reads: `lifecycle-diff`, `lifecycle-snapshot`, `build`, `lint`.
   - Actions (delegated writes): `op-run` (start an `ApplyOp`/`ReconcileOp`),
@@ -88,6 +89,12 @@ and none is the checkout's git remote) adds two reads, in
   report of its wave's other entries (same gate and digest). No
   `audit.jsonl` is `present: false` plus an `absent` sentence, never an
   empty history; a bad line is the `422 terragucci-report` refusal naming it.
+  #523: an S3 403 on an optional read (`audit.jsonl` here, `run.json` for
+  `/api/terragucci/progress` and the estate's run views) is not a refusal:
+  the source throws `TerragucciDenied` (`src/terragucci-source.ts`), and the
+  answer carries `present: false`, `unreadable: true` and an `absent`
+  sentence naming the key and `s3:GetObject` (or `s3:ListBucket`). A 403 on
+  `index.json`, a report or `estate.json` still fails.
   `src/terragucci-timeline.ts` holds the read and the check (terragucci's
   `dist/audit.schema.json` when it resolves, structural otherwise). The SPA
   draws it in the Terragucci tab and, filtered to the card's root, in the
@@ -95,11 +102,79 @@ and none is the checkout's git remote) adds two reads, in
   snapshot and does not fail on it.
 - `GET /api/terragucci/lifecycle` (#505): the repo's `chant/lifecycle` lane,
   `{branch, found, via, origin?, fetch: {ok, error?}, commit?: {sha,
-  committed}, read: {at}, gates[], locks[], note}`. `gates[]` is one entry per
-  `wave-<k>` in `_gates/tf-apply.jsonl` and `_gates/tf-apply/applied.jsonl`:
-  `state` (`waiting | expired | approved | applied | failed`), `digest`,
-  `pending`, `approval` (by, at, signed), `applied`, the kept reports under
-  `_gates/tf-apply/wave-<k>/`, and `command` when it waits. `locks[]` is
+  committed}, read: {at}, approval?, gates[], locks[], note}`. `gates[]` is one
+  entry per wave gate (`wave-<k>`, chant op-waves' `<gate>-wave-<k>` or a
+  product's `<name>-wave-<k>`: `WAVE_GATE`, #521) in `_gates/tf-apply.jsonl` and
+  `_gates/tf-apply/applied.jsonl`, decided by chant's `decideGateApproval`
+  (`@intentius/chant/gate-decision`, #520) with spent approvals left out, on
+  lines parsed by chant's `parseGateLedger` and `parseAppliedGateLines`. The
+  states:
+  - `waiting`: no approval of the wave's digest counts; `command` is the
+    approve line (`--sign` under `sealed`), or `chant workspace points answer
+    <id>` when the pending fact carries `answer: {point, id}` (#521).
+  - `expired`: the pending fact's `expiresAt` passed with none.
+  - `refused`: an unspent approval stands for another digest
+    (`refused: {digest, by, at}`); no approve line.
+  - `approved`: an unspent approval of the digest counts, not applied yet.
+  - `applied` / `failed`: a wave applied under that approval, and how the
+    apply finished when terragucci recorded it. chant's op waves write no
+    `finished` line: then `applied.result` is absent and the page says
+    "applied (outcome not recorded)".
+  - `refused-by-policy` (#521): a denial on `_gates/policy-override.jsonl`
+    (chant's `recordedDenials`) newer than anything on the wave's gate, that
+    no override counts for (chant's `decideOverride`, with the overriders at
+    base: `.chant/overriders`, or terragucci's `policy.override`) and no run
+    applied an override of since. `denials[]` carries each run, its plan and
+    rules, and chant's `overrideCommand` line; no approve line. The wave is
+    the one the denial's pending fact names (`wave: {number, name, op}`,
+    typed on chant's `PendingGateRecord`, whose parser drops a wave of the
+    wrong shape, chant#3887), else, for a fact an older chant wrote, the one
+    whose pending fact lists the run.
+  - `held` (#521): a pending fact on another gate (any `_gates/<op>.jsonl`)
+    that names this op's wave (`wave: {number, name, op}`), with no approval
+    of its digest that counts under the mode the fact records. `heldOn: {op,
+    gate, digest, since, heldBy, description}`; `heldBy: {hook, moment,
+    target?}` is the hook that holds the wave (chant#3886), and the card
+    (`holderWords` in `web/terragucci-lifecycle.js`) names it from there,
+    or from `description` for a fact an older chant wrote. `command`
+    approves that gate (`chant approve <op> <gate> --plan`) or answers its
+    decision point. A hold fact that names no wave is not read as a hold.
+  Each entry also carries `digest`, `pending` (with `members` and
+  `neverOverMcp` when recorded, `description`, `url`, `answer`), `approval`
+  (by, at, `signed` when the seal has a signature, via (`review` for chant's
+  `review: {pr, head, approvers}` line), pr, head, reviewers, relayedBy, note), `uncounted` (an
+  approval of this digest the mode did not count, with chant's reason, such
+  as an unsealed one under `sealed`), `spent` (an approval of another digest
+  an apply used), `applied` (with `wave` and `review` when present), the kept
+  reports under `_gates/tf-apply/wave-<k>/`, and `command`. `approval` on the
+  answer is the rule read at base, the tip of origin's default branch fetched
+  depth 1 into the same cache (`refs/behold/base`): `mode` from
+  terragucci.yml's `approval:` key, else `sealed` when chant.workspace.json
+  lists `identity.gates`, else `ledger`, and the signers read with chant's
+  `gateSignersAt`. Each wave is decided under the approval mode its newest
+  pending fact records (`approvalMode`, `withRecordedModes`), else the mode at
+  base; the entry carries `approvalMode: {mode, from: "pending" | "base"}`,
+  and `approval.inferred` is set only when some wave's fact records none. A
+  gate `identity.gates` names at base is decided with `refuses` from chant's
+  `identityGateAdmissionAt`, which checks the seal against the signers at
+  base and, when the gate names a class, the role grants in
+  `.chant/trust.json` at base, whatever the mode; the entry carries
+  `identity: {class}`. A class a pinned package supplies is read by chant
+  from `./workspace-principals` (chant#3888): a path pin's file at base, a
+  package pin's from the package installed in the served checkout's
+  node_modules (behold passes the checkout as `packages`, since the bare
+  cache has none), at the pinned version. When that package is missing or
+  at another version, a gate naming a domain class or `human` refuses every
+  approval and chant's reason names the pin. Under `pr-review`, a spec's `headPlans` (the op-waves spec name)
+  has the newest 50 `_wave-plans/<name>/<head>.json` records read with
+  chant's `readHeadPlans` (`@intentius/chant/lifecycle/wave-plans`) through
+  `ledgerPort`, a `LedgerReadPort` over the fetched commit. The op, ledger paths, gate names, approve line and
+  mode reader are a `GateLaneSpec` (`TERRAGUCCI_LANE` is terragucci's), for
+  another product's lane: `opWavesLane(op, over?)` gives one with chant's
+  approve line, mode rule and overriders, and `readLifecycle(checkout, {spec})`
+  reads it. `terragucciApproveCommand` is the one approve line the reports,
+  the estate and the lane share, and `gatePolicyName` reads `on-destroy` as
+  `on-destructive` (terragucci#863). `locks[]` is
   `_locks/tf-apply.json`, one per root (`pr, by, at, head, via?, stage?`).
   Every gate and lock carries `source: {commit, committed, path}`, the newest
   commit of the branch that changed its file. The branch is fetched from the
@@ -111,6 +186,27 @@ and none is the checkout's git remote) adds two reads, in
   `fetch.error`); a remote without the branch is `found: false, via:
   "fetch"`, not an error. 30 s cache, `?fresh=1` skips it. Reader:
   `src/terragucci-lifecycle.ts`.
+- `GET /api/lifecycle/history?op=<op>&gate=<gate>&limit=<n>` (#522): the
+  gate ledgers' history on `chant/lifecycle`, for any product, registered
+  whether or not `--terragucci` is given. Same bare cache and fallback as
+  the lifecycle lane; per `_gates/<op>.jsonl` it runs `git log -p
+  --unified=0 --no-renames` and hands the text to chant's
+  `gateLedgerLinesFromLog`, then all ledgers to `deriveGateAudit`
+  (`@intentius/chant/lifecycle/gate-audit`). behold parses no ledger line
+  itself. Answer: `{branch, source: "ledger", found, via, origin?, fetch,
+  commit?, read, ledgers[], op?, gate?, total, limit, entries[], gaps[],
+  note}`, newest first, 200 by default and 1000 at most. An entry is `{id,
+  kind, at, op, gate, wave?, digest?, actor?, seal?, note?, rules?, subject?,
+  source: {path, line?, commit, committer, committed, change}, audit}`;
+  `kind` is chant's (`approval-requested | approved | override-requested |
+  override-recorded | revoked`), `audit` is chant's whole entry. A revocation
+  is a line a commit removed, and its `actor` is that commit's author. The
+  ledger holds no applies, so `applied` and `override-spent` appear only in
+  a product's own record (terragucci's `audit.jsonl`). The Terragucci tab's
+  Timeline draws it under the audit record, labelled "ledger"; when the
+  source has no `audit.jsonl` the record's lane says so and the ledger is
+  the history. An export carries it as one snapshot. Reader:
+  `src/lifecycle-history.ts`.
 - `GET /api/terragucci/events` (#505): SSE. While at least one page holds it
   open, a timer (`--terragucci-poll <secs>`, default 30, 0 off) asks
   whether `index.json` changed (`If-None-Match` with the last ETag for S3 and
@@ -173,6 +269,112 @@ checkout declares no Terraform root (`controlRepo` in
   structural check otherwise. The page (`web/terragucci-estate.js`) draws it
   in a layer over the graph area.
 
+## Releases per environment, from the release ledger (#525)
+
+`GET /api/lifecycle/releases[?fresh=1]` (src/lifecycle-releases.ts) reads
+`<env>/releases.jsonl` on the served checkout's `chant/lifecycle` with chant's
+own reader (`listReleaseEnvironments`, `readReleaseLedger`,
+`latestPerComponent`, `recordsForDigest`). The branch is fetched into the
+same bare cache the terragucci lane uses (`fetchIntoCache` in
+src/terragucci-lifecycle.ts). When that fetch fails, the checkout's
+`refs/remotes/origin/chant/lifecycle` is fetched into a second cache of
+behold's and read there (`via: "checkout-ref"`); the checkout is never
+written. Reads are reused for 30s.
+
+The answer: `environments` (wave order from `yodel-waves.json` when present,
+`order: "waves"`, else by name), `components` (a tenant is
+`<component>/<tenant>`, as SQL Yodeler records it), and `cells[]`, one per
+component and environment that has a release: `digest`, `timestamp`,
+`actor`, `approver?`, `gitSha`, `runId`, `runUrl?`, `source` (the newest
+commit that changed that environment's ledger), and `differs: {from,
+digest?, everHad}` when the environment before it holds another digest or
+none. `/api/project` says `releaseLedger: true` for a checkout with a
+`yodel.config.ts` or `yodel-waves.json`; the page then opens a Releases tab
+(`web/releases.js`), and `behold export` captures the route as a snapshot.
+
+## SQL Yodeler's report and drift, from files (#526)
+
+`behold serve <dir> --yodel-report <file|dir>` adds `GET /api/yodel-report`
+(`src/yodel-report.ts`). It reads one `yodel report --json` output and any
+`yodel drift <env> --json` output in the same directory, written by the
+person or their pipeline. behold never runs `yodel` and never connects to
+the database.
+
+- The answer is `{source, validation, report, environments[], drift[],
+  marks[], ignored[]}`. Each environment carries its applied migrations, the
+  rest with their state, its gaps and notes, dated by the report's
+  `generatedAt`. Each mark is one object a drift file names (`changed` or
+  `missing`), dated by the file's mtime (drift JSON has no time of its own)
+  and named by its file.
+- Every file is checked against SQL Yodeler's schemas: from
+  `@intentius/sql-yodeler/schemas/` when the package resolves beside the
+  served project or behold, otherwise from the copy pinned in
+  `src/sql-yodeler-schemas/` (sql-yodeler 19ebbff). A file that fails is a
+  422 with `code: "yodel-report"` naming the file, and nothing is drawn.
+  Update the pinned copy by copying the three files from sql-yodeler's
+  `schemas/` and changing `PINNED_COMMIT`.
+- The page (`web/yodel-report.js`) joins a mark to the card whose id is the
+  object's name, or ends in `/<name>` in a composed estate, and draws a
+  corner tag with its age; never a fill. Objects no card is are listed in
+  the SQL Yodeler tab, which also holds the environments and gaps.
+- `audit[]` (#537) carries the report's audit entries with the report file
+  they came from. The tab lists them under their environment, each dated by
+  its own `at` and named by its source (a ledger's op and gate, or a history
+  table's row and run).
+- `behold export --yodel-report <file|dir>` (#537) captures `/api/yodel-report`
+  as a snapshot, and a refusal fails the export, as `--terragucci` does. A
+  SQL Yodeler repo's export also captures `/api/sql-yodeler/lifecycle` and
+  `/api/lifecycle/history`, so the Pipeline lane draws in static mode,
+  without its Read again button.
+
+## A SQL Yodeler project: kinds and a lifecycle lane (#524)
+
+A SQL Yodeler project is a chant project with `lexicons: ["sql"]` and a
+`yodel.config.ts` (#517's detection, `productRepo` in src/terragucci-repo.ts).
+Its graph draws from `chant graph` as any project's does. src/sql-kinds.ts
+registers a pinhole pack for the `sql` lexicon: a line glyph per
+`Postgres::*` and `ClickHouse::*` kind, and cards that lead with the kind in
+words and the qualified name. A `REFERENCES ${other}` foreign key is a `ref`
+edge; e2e/fixtures/sql-yodeler-fk holds a project with one, and
+src/sql-yodeler-lane.test.ts graphs it. `@intentius/chant-lexicon-sql` is a
+devDependency for that test; from 0.124.13 its lint rules use chant's own
+TypeScript (chant#3864), so they run beside behold's TypeScript 7.
+
+`GET /api/sql-yodeler/lifecycle[?fresh=1]` (src/sql-yodeler-lane.ts) reads
+`chant/lifecycle` with `readLifecycle` and a spec per op, sharing one fetch:
+
+- `opWavesLane("yodel-apply")` for the apply pipeline's
+  `yodel-apply-wave-<k>` gates, with the mode at base from `yodel-waves.json`
+  when every wave names the same `approval`. When they differ (#537) the
+  rule carries `waves[]` and each wave is decided under its own mode
+  (`waveRule`; a gate decided so carries `mode`). Under `pr-review` a gate not
+  yet approved carries `headPlans[]`: the heads whose
+  `_wave-plans/yodel-apply/<head>.json` (chant's HeadPlans, newest 50 read)
+  name its digest.
+- `migrateOpLane(env)` for each `migrate-<env>` op that `ops/` or
+  `yodel-waves.json` names: its one gate `approve-migrate-<env>`, drawn as
+  wave 1 of that op.
+
+The answer is the lane's header (branch, via, commit, rule, note) with
+`environments[]` in the order of applies (`yodel-waves.json`, then any
+environment only `ops/` names), each with its wave (`k`, targets, gate
+policy, approval) and `gates[]` (`kind: "wave" | "migrate"`, the GateEntry
+of #520/#521). A waiting gate carries `npx yodel approve <env> --plan
+<digest>` (`--sign` under sealed). Waves the spec does not name go in
+`unplaced`. A served directory that is not SQL Yodeler gets 404 with
+`code: "sql-yodeler"`. The page's Pipeline tab (`web/sql-yodeler-lane.js`)
+draws it, with `/api/lifecycle/history` (#522) below. Reads are reused 30s.
+
+A SQL Yodeler project that is a chant workspace member (#537) keeps its
+records under `_members/<member>/` on `chant/lifecycle`. src/member-ledger.ts
+asks chant's `resolveMemberLedger` (`@intentius/chant/lifecycle/member-ledger`)
+for the prefix; the lane passes it as `GateLaneSpec.ledgerPrefix`, which
+`readLifecycle` puts before every branch path, and `/api/lifecycle/releases`
+hands it to chant's `listReleaseEnvironments` and `readReleaseLedger` as
+`ledgerPrefix` (chant#3865), which list and read the environments under it. Both answers then carry
+`member: { members, prefix, error? }`, and `yodel-waves.json` at base is read
+in the project's own directory.
+
 ## Getting a server
 
 ```sh
@@ -726,8 +928,42 @@ entry is a registered route. `TERRAGUCCI_REFUSED_WRITES` only words the
 refusal for known routes. The remedy names `npx terragucci approve wave-<k>
 --plan <digest>`, the digest from the wave's gate card.
 
+A SQL Yodeler repo (a `yodel.config.ts` in the served directory or above it,
+up to the git root) is guarded the same way (#517): the same allowlist, the
+same refused routes, auto-sync declined, and `/api/project` carries
+`sqlYodeler` so the page offers no write. The refusal is `409` with `code:
+"sql-yodeler"`, and its remedy names `npx yodel approve <env> --plan <digest>`
+(with `--sign` when the environment's gate is sealed) and `yodel apply <env>`
+in the project's pipeline. `productRepo()` in `src/terragucci-repo.ts` looks
+for terragucci's config along the whole walk first, so a yodel project nested
+inside a terragucci repo is refused as terragucci.
+
+Op discovery (`src/ops.ts`) reads a gate from `signalName: "<name>"` or from
+`gate("<name>", ...)`, the form chant's op builder and SQL Yodeler write, so
+a gated Op is never listed or run as ungated on any repo (#517).
+
+A policy denial is never approved from behold (#519, `src/policy-override.ts`).
+`GET /api/workspace/gates` gives a gate of op `policy-override` an `override`
+field `{subject, rules, plan, digest, state, spentAt, product, command,
+source}`: the rules and denied plan come from `chant/lifecycle:_gates/
+policy-override.jsonl` (read with `git show`, the local branch then
+`origin/chant/lifecycle`), `state` is `waiting`, `overridden` (status says an
+override counts) or `spent` (a line in `_gates/policy-override/applied.jsonl`
+for its digest), and `command` is the product's (`npx terragucci override`,
+`npx yodel override`) or chant's `chant approve policy-override <subject>
+--rule <id> --reason "<why>" --plan <digest>`. The card shows that command and
+no button, and `POST /api/workspace/gates/approve` answers such a gate `422`
+with `code: "override"`. The ledger lines are parsed by chant's
+`@intentius/chant/override-gate` (`recordedDenials`, `parseAppliedOverrides`,
+`overrideCommand`) and `parseGateLedger`, and the product is `productRepo()`
+from `src/terragucci-repo.ts` (#537).
+
 ### The exceptions, and their exact size
 
+These are also the only writes a guarded product repo answers: terragucci
+(`terragucci.yml` and its siblings) and SQL Yodeler (`yodel.config.ts`), both
+listed in `TERRAGUCCI_ALLOWED_WRITES`.
+
 `POST /api/layout` (#228) writes **one** file in the served project:
 `.behold/layout.json`, the hand-layout sidecar, `{version, lenses: {<lens>:
 {<node id>: {dx,dy,dw,dh}}}}`. That is the whole of behold's write surface
@@ -1158,7 +1394,10 @@ water park's `access/` before it was taken:
 The scratch project is `<tmpdir>/behold-tf-<hash of the estate path>`:
 `behold-*` and cleared through `assertScratch` (src/scratch.ts), one directory
 per estate reused across runs, asserted to be outside the estate before a byte
-is written. It holds the generated `chant.config.ts` and two symlinks —
+is written. It holds the generated `chant.config.ts`, a `package.json` that
+declares it an ES module (chant's compiled CLI, 0.124.14 on, otherwise reads a
+config tsx compiled as CommonJS as `{default: config}` and finds no lexicon:
+INTENTIUS/chant#3897), and two symlinks —
 `node_modules` to behold's own, which is how the config resolves the lexicon,
 and `estate` to the served directory, which is how each root's `dir` is spelled.
 The second is not decoration: the lexicon sets a root's module boundary to the

package/demos.json

@@ -65,7 +65,7 @@
     },
     {
       "name": "terraform-estate",
-      "description": "A multi-root Terraform estate, declared only: two roots applied separately (envs/prod and platform) plus the shared module both are built on — 37 cards boxed by root, module calls expanded into their own contents, and the two look-alike versions.tf files told apart. Shows what behold does NOT infer as clearly as what it draws: platform declares the KMS key and prod reads it back by alias, and no edge is drawn between them, because both ends carry the same unresolved interpolation (#381). Needs chant's terraform lexicon beside behold, which behold does not install: npm install --no-save @intentius/chant-lexicon-terraform@^0.121.0 @cdktn/hcl2json@^0.24.0 (without them the demo refuses with that line); the copy is linked to the lexicon behold resolved, since chant reads a workspace's pinned kinds from beside the workspace. No cloud, no Docker, no terraform binary.",
+      "description": "A multi-root Terraform estate, declared only: two roots applied separately (envs/prod and platform) plus the shared module both are built on — 37 cards boxed by root, module calls expanded into their own contents, and the two look-alike versions.tf files told apart. Shows what behold does NOT infer as clearly as what it draws: platform declares the KMS key and prod reads it back by alias, and no edge is drawn between them, because both ends carry the same unresolved interpolation (#381). Needs chant's terraform lexicon beside behold, which behold does not install: npm install --no-save @intentius/chant-lexicon-terraform@^0.124.0 @cdktn/hcl2json@^0.24.0 (without them the demo refuses with that line); the copy is linked to the lexicon behold resolved, since chant reads a workspace's pinned kinds from beside the workspace. No cloud, no Docker, no terraform binary.",
       "source": "bundled",
       "dir": "example-terraform-estate",
       "requires": [],