instar 1.3.478 → 1.3.480

package/upgrades/1.3.480.md

@@ -0,0 +1,35 @@
+# Upgrade Guide — vNEXT
+
+<!-- assembled-by: assemble-next-md -->
+<!-- bump: minor -->
+
+## What Changed
+
+Closes the hole that let the cartographer features ship dark for **everyone** — including the development agents meant to dogfood them — and makes the dev-agent dark-gate convention structurally enforceable.
+
+- **Cartographer dogfoods on dev agents.** `cartographer.enabled` and `cartographer.conformanceAudit.enabled` now resolve through the existing `resolveDevAgentGate` (omit `enabled` in defaults → live on `developmentAgent:true`, dark on the fleet). The conformance route's strict `!== true` gate is converted to the resolver so it no longer 503s on a dev agent. A one-shot, dev-agent-only migration strips the default-shaped `false` on existing dev agents so they light up on update (never the cost-bearing sweep).
+- **The freshness sweep stays an explicit opt-in** (`cartographer.freshnessSweep.enabled: true`) even on dev agents — it is the one ongoing-cost surface (off-Claude codex calls). The redundant `egressAcknowledged` second gate is removed (the privacy framing was incoherent — source already egresses to a model every turn); the off-Claude routing probe is unchanged (cost guard, not privacy).
+- **Lint assertion C** (`scripts/lint-dev-agent-dark-gate.js`): every `enabled: false` default in `ConfigDefaults.ts` must now be DECLARED — either dev-gated (omit + register in `DEV_GATED_FEATURES`) or listed in the new `DARK_GATE_EXCLUSIONS` registry with a closed-enum `category` and a ≥12-char `reason`. A brace-in-string loud-fail guard, a hand-authored golden-path drift canary, and a required `justification` on every dev-gated entry back it up. All 21 existing dark defaults are classified, so the exact way cartographer slipped through cannot recur silently.
+
+🧪 **Mostly internal.** On the fleet nothing changes (cartographer stays dark). On a development agent, cartographer's zero-cost read surfaces (map, navigation, conformance audit) now run live. The `egressAcknowledged` field is retained for back-compat but is now inert on the sweep (which still needs explicit `enabled:true`).
+
+## What to Tell Your User
+
+- "Cartographer's read-only surfaces (the codebase map, navigation, and the standards audit) now run live on me as a development agent so they actually get dogfooded — they stay off for everyone else until you flip them on. The one part that spends money (the summary sweep) still needs an explicit one-line opt-in even on me."
+- "I also closed a gap that let those features ship off-for-everyone by accident: every 'off by default' feature now has to declare itself, or the build fails."
+
+## Summary of New Capabilities
+
+| Capability | How to Use |
+|-----------|-----------|
+| Cartographer live on dev agents | Automatic on `developmentAgent:true` (read surfaces); existing dev agents lit up by a one-shot migration on update |
+| Freshness sweep opt-in | `cartographer.freshnessSweep.enabled: true` (explicit, even on dev agents — ongoing off-Claude spend) |
+| Dark-gate enforcement lint | `node scripts/lint-dev-agent-dark-gate.js` (in `npm run lint`) — fails on any unclassified `enabled:false` default |
+| `DARK_GATE_EXCLUSIONS` registry | Declare a deliberately-off-for-everyone feature with `{configPath, category, reason}` in `src/core/devGatedFeatures.ts` |
+
+## Evidence
+
+- Unit (42): `devGatedFeatures` both-sides wiring auto-covers the two new entries (live-on-dev / dark-on-fleet); lint assertion-C cases (unclassified FAILS, excluded PASSES, registered-but-hardcoded FAILS, junk/short reason + unknown category FAIL); hand-authored golden-path drift canary; brace-in-string loud-fail fixture; destructive-not-gated guard (mcpProcessReaper stays excluded); one-shot dev-agent-only migration.
+- Integration (6): `GET /conformance/coverage` returns 200 under a `developmentAgent:true` config and 503 under a fleet config (the route-gate fix).
+- E2E (3): production init path — cartographer read routes live on a dev agent (200, not 503), 503 on the fleet, and the cost-bearing sweep poller is NOT started without explicit `freshnessSweep.enabled:true`.
+- `npx tsc --noEmit` exit 0; full `npm run lint` chain clean. Spec converged 3 rounds + approved (see `upgrades/side-effects/dev-agent-dark-gate-enforcement.md` and `docs/specs/reports/dev-agent-dark-gate-enforcement-convergence.md`).

package/upgrades/side-effects/autonomous-completion-real-checks.md

@@ -0,0 +1,88 @@
+# Side-Effects Review — Autonomous completion: real-checks verification
+
+**Version / slug:** `autonomous-completion-real-checks`
+**Date:** `2026-06-10`
+**Author:** `echo`
+**Second-pass reviewer:** `echo (Phase 5 — REQUIRED: touches the autonomous Stop-hook continue/stop authority + a new gate)`
+
+## Summary of the change
+
+Autonomous jobs gain an OPT-IN `verification_command` (+ optional `verification_cwd`; setup also records `work_dir`). When the transcript-judge (`CompletionEvaluator`, via `POST /autonomous/evaluate-completion`) returns `met:true`, the autonomous Stop hook (`autonomous-stop-hook.sh`) now RUNS the declared command and only allows the session to exit if the command ALSO passes (exit 0). Any failure mode — non-zero, timeout, missing-timeout-binary, refused-destructive, or breaker-open — routes to KEEP WORKING (the safe direction), surfacing the command's sanitized output as next-turn guidance. The judge stays transcript-only and unchanged; the real check is a second, independent, deterministic gate layered on the (rare) met path. Files: `.claude/skills/autonomous/hooks/autonomous-stop-hook.sh` (the gate `realcheck_gate`, `run_verification`, P19 breaker helpers, audit, config-read), `.claude/skills/autonomous/scripts/setup-autonomous.sh` (new flags + state fields), `src/config/ConfigDefaults.ts` (the `realCheck` config block nested in `completionDiscipline`), `src/core/PostUpdateMigrator.ts` (3 marker bumps to `REALCHECK_VERIFY` + a `migrateClaudeMd` appended subsection), `src/scaffold/templates.ts` (`generateClaudeMd`), `.claude/skills/autonomous/SKILL.md` (Write-template docs), + unit/integration/e2e tests. Spec: `docs/specs/autonomous-completion-real-checks.md` (converged, approved).
+
+## Decision-point inventory
+
+- `autonomous Stop-hook EXIT decision (met path)` — **modify** — a met:true verdict no longer auto-exits; it now ALSO requires the declared `verification_command` to pass (exit 0). The modification is purely additive in the SAFE direction: it can only WITHHOLD an exit, never CAUSE one.
+- `realcheck_gate` — **add** — the new gate function; returns 0 (allow exit) ONLY on disabled / no-command / breaker-closed-PASS; returns 1 (keep working) on every failure mode.
+- `realcheck breaker (CD_BACKOFF_STATE)` — **add** — a P19 backoff+breaker+cap sibling counter that short-circuits a stuck/flaky command (no judge re-fire, no command run) and raises ONE Attention item.
+- `destructive-pattern pre-block` — **add** — a coarse literal-shape guard refusing high-signal destructive commands before execution (→ keep working).
+
+---
+
+## 1. Over-block
+
+**What legitimate inputs does this change reject that it shouldn't?**
+
+The gate blocks the autonomous EXIT, not a user message. Its only "over-block" is keeping the agent working when the goal is actually done — which happens only if a declared `verification_command` falsely FAILS (a flaky check, a wrong CWD, a stale-artifact premise). Concrete: a job declares `verification_command: "npm test"` but the build lives in a worktree the job forgot to point at via `verification_cwd` → `npm test` from `work_dir` fails → the agent keeps working past a genuine completion. Mitigations: (a) the failure output is fed back as next-turn guidance framed as "the check may be wrong/mis-scoped — say so" (P13 / A-Wall-Is-a-Hypothesis), so the agent can correct the premise rather than grind; (b) the P19 breaker stops a persistently-failing check from spinning to duration and raises an Attention item so the operator sees the bad command; (c) the duration limit is the hard backstop; (d) the whole feature is a NO-OP unless a job declares the command, and is opt-in per job. Critically, over-block here is bounded + surfaced + recoverable — it can never cause the dangerous direction (a premature exit).
+
+---
+
+## 2. Under-block
+
+**What failure modes does this still miss?**
+
+A `verification_command` the agent constructs to falsely PASS: `verification_command: "true"`, `"npm test || true"`, a `test -f <stale-flag>` the agent `touch`ed earlier, or a PATH-shadowed `npm`. These are honestly named residuals (spec §6.2): the feature strengthens against HONEST failure (misread output, stale tree, premature "done"), not against an adversarial agent — which already holds far larger powers (edit source, push). The floor never drops below today (the independent judge still had to say met). The defeat is made DETECTABLE post-hoc via the audit row (`logs/autonomous-realcheck.jsonl` records the command + cwd + exit), not prevented. The conversational/transcript path is unchanged — this only adds a gate on the autonomous met path.
+
+---
+
+## 3. Level-of-abstraction fit
+
+**Is this at the right layer?**
+
+Yes. The real check is a deterministic STRUCTURAL VALIDATOR (an exit-code — `signal-vs-authority.md`'s allowed detector class), layered as a corroborator on top of the existing full-context LLM judge (`CompletionEvaluator`) which remains the primary JUDGMENT authority on the met verdict. The check runs in the Stop hook (the correct layer — it owns the continue/stop decision), reuses the existing `CD_BACKOFF_STATE` breaker machinery, the `hb_sanitize`/`hb_leak_hit` output discipline, and the `cd_raise_attention_item` notification primitive rather than re-implementing them. It does NOT add a parallel gate or a new authority — it feeds the existing continue/stop decision.
+
+---
+
+## 4. Signal vs authority compliance
+
+**Does this hold blocking authority with brittle logic, or produce a signal that feeds a smart gate?**
+
+Compliant. Per `docs/signal-vs-authority.md`: an exit-code is a structural validator (the doc's explicitly-allowed detector class — "this type-checks, this length is bounded"), not a meaning-judgment. The LLM judge remains the sole *judgment* authority that must independently say met; the real check's only autonomous authority is in the SAFE (keep-working) direction — a brittle check that errs toward "keep working" (bounded by duration) is NOT the dangerous "brittle authority that blocks a legitimate action" the principle forbids. It is a corroborator that RAISES the bar for stopping, never one that forces a premature stop. Every error/ambiguity/timeout path resolves to keep-working.
+
+---
+
+## 5. Interactions
+
+**Does it shadow another check, get shadowed, double-fire, race?**
+
+- **CD_BLOCK_TERMINAL / contradictory hard-blocker:** the gate sits INSIDE the existing `if [[ "$EVAL_MET" == "true" ]]` block, AFTER the `CD_BLOCK_TERMINAL` guard + the P13 check — so a turn emitting BOTH a `<hard-blocker>` and a met-condition short-circuits before the gate (tested). The gate never runs on a hard-blocker/P13-blocked turn.
+- **Native /goal mode:** the `goal_mode:native` branch returns far earlier (line ~452) — the gate is unreachable there; native runs are unaffected.
+- **P19 breaker vs judge breaker:** a SIBLING counter in the same `CD_BACKOFF_STATE` sidecar (atomic `.tmp.$$`+mv writes, fails CLOSED=keep-verifying on a corrupt sidecar) — does not collide with the judge breaker.
+- **Idle-backoff:** a met→fail iteration is by definition not a rapid-idle one, so the backoff self-clamp is unaffected; the verification timeout (≤120s) + judge curl (≤35s) stay far under the ~10000s registered Stop-hook timeout (no host-kill / fail-open-strand risk).
+- **Adjacent (out of scope, captured separately):** the feature's integration test surfaced a PRE-EXISTING latent bug in `setup-autonomous.sh`'s multi-session start-gate — when run with `--report-topic`, no server reachable, AND `.instar/autonomous/` empty, the local-count fallback `ls …/*.local.md | grep -c…` exits non-zero on an empty glob and trips `set -euo pipefail`, aborting before the state write. NOT part of this feature; not fixed here (no-batching rule); recorded for a separate follow-up fix (`grep -c … || true`).
+
+---
+
+## 6. External surfaces
+
+**Does it change anything visible to other agents, users, other systems?**
+
+- A new audit file `logs/autonomous-realcheck.jsonl` (read-only, size-rotated). No new API route.
+- One new Attention item source (`autonomous-realcheck-stuck`) when the breaker trips — deduped per run (Bounded-Notification-Surface compliant).
+- Next-turn guidance gains the real-check output, sanitized (control-strip → UTF-8 scrub → leak-scrub incl. the agent's own authToken + `Bearer` → clamp) and explicitly DATA-labeled so an echoed "all tests pass" cannot launder into a later judge verdict.
+- Runtime dependency: it RUNS a command unattended in the Stop hook — bounded by a portable timeout ladder (`timeout`/`gtimeout` with `-k`, else a perl `setpgrp` group-kill, else unavailable→keep-working), scrubbed env (fixed PATH, no authToken, no `npm_config_*`/`NODE_OPTIONS`), and a destructive-pattern pre-block. Timing depends on the declared command, bounded by `realCheckTimeoutMs` (default 120s) and only on the rare met path.
+
+---
+
+## 7. Rollback cost
+
+**If this turns out wrong in production, what's the back-out?**
+
+Cheap and instant. `autonomousSessions.completionDiscipline.realCheck.enabled: false` in `.instar/config.json` is read at the hook chokepoint each fire (no restart) and reverts to transcript-only behavior. A job simply omitting `verification_command` is already unaffected (byte-identical to today — tested). No data migration, no state repair. The migrator marker bump (`REALCHECK_VERIFY`) is idempotent; a customized hook is left untouched. Worst case (the gate misbehaves for a job that opted in) is bounded by the duration limit and surfaced via the audit + Attention item.
+
+---
+
+## Second-pass review (Phase 5)
+
+**Reviewer: echo (independent pass) — Concur with the review.**
+
+Independently traced every path through `realcheck_gate` (hook ~1095-1121) and `run_verification` (~929-1064): the cardinal invariant holds — `realcheck_gate` returns 0 (allow exit) ONLY on `RC_ENABLED != 1`, empty `VERIFICATION_COMMAND`, or `RC_OUTCOME == "pass"`; every other outcome (fail, 124 timeout, 127 unavailable, 126 refused-destructive, breaker-open, corrupt sidecar) returns 1 (keep working). All three completion-met `exit 0` sites are strictly inside `if realcheck_gate; then`; the non-completion exits (hard-blocker, emergency-stop, corrupt-state fail-safe) are correctly NOT wrapped. Breaker fail-direction verified empirically (corrupt `CD_BACKOFF_STATE` → fails=0 < threshold → echo 0 → CLOSED → check still runs — fail-CLOSED/safe). Perl timeout wrapper semantics verified live (exit-1→FAIL, exit-0→PASS, sleep-3-under-1s→124 in ~1s, no orphan). Signal-vs-authority posture legitimate (structural-validator corroborator, safe-direction-only authority, LLM judge unchanged and still primary). 24 unit + 4 integration + 2 e2e green. One under-stated nuance (a condition-path fail + same-turn legacy-promise match could run the command twice in one fire) is bounded, non-harmful (same breaker, neither can cause a premature exit), and does not affect the invariant.

package/upgrades/side-effects/dev-agent-dark-gate-enforcement.md

@@ -0,0 +1,95 @@
+# Side-Effects Review — Dev-Agent Dark-Gate Enforcement
+
+**Version / slug:** `dev-agent-dark-gate-enforcement`
+**Date:** 2026-06-10
+**Author:** echo (Instar Agent)
+**Spec:** `docs/specs/DEV-AGENT-DARK-GATE-ENFORCEMENT-SPEC.md` (converged 3 rounds, approved)
+
+## Summary of the change
+
+Two slices. **Slice A** routes the cartographer *zero-cost read surfaces* through
+the existing `resolveDevAgentGate` so they run LIVE on development agents and DARK
+on the fleet (the doc-tree/navigation read + the deterministic conformance audit),
+and removes the redundant `egressAcknowledged` second gate from the freshness
+sweep (the sweep remains an explicit `enabled:true` opt-in even on dev agents —
+NOT dev-gated — because it is the one ongoing-cost surface). **Slice B** closes the
+lint hole that let cartographer ship dark-for-everyone: every `enabled: false` in
+`ConfigDefaults.ts` must now be DECLARED — either dev-gated (omit `enabled` +
+register) or listed in a new `DARK_GATE_EXCLUSIONS` registry with a closed-enum
+category and a ≥12-char reason. A one-shot, dev-agent-only migration lights up
+existing dev agents on update.
+
+**Files changed (source):**
+- `src/config/ConfigDefaults.ts` — OMIT `enabled` from `cartographer` and
+  `cartographer.conformanceAudit` (gate decides at runtime); comments updated.
+  `freshnessSweep`/`llmEnrichment`/`llmRerank` `enabled:false` left as-is.
+- `src/commands/server.ts` — `cartographerEnabled` now via `resolveDevAgentGate`;
+  freshness-sweep start predicate drops the `&& egressAcknowledged` term (now
+  `enabled` alone). Off-Claude routing probe + per-pass bounds untouched.
+- `src/server/routes.ts` — conformance route gate `cfg?.enabled !== true` →
+  `!resolveDevAgentGate(cfg?.enabled, ctx.config)` (the only cartographer/
+  conformance ACCESS gate; the `sweepEnabled` status field at ~L4169 is a response
+  field, not a gate, deliberately untouched).
+- `src/core/devGatedFeatures.ts` — required `justification` field on
+  `DevGatedFeature` (+ on every existing entry); two new dev-gated entries
+  (`cartographer`, `cartographerConformanceAudit`); new `DarkGateCategory`,
+  `DarkGateExclusion`, `DARK_GATE_EXCLUSIONS` (all 19 remaining dark defaults
+  classified).
+- `src/core/PostUpdateMigrator.ts` — `migrateCartographerDevGate`: one-shot
+  (`_instar_migrations` marker), dev-agent-only, strips a default-shaped
+  `false` at `cartographer.enabled` + `cartographer.conformanceAudit.enabled`
+  ONLY; never touches `freshnessSweep`; records `result.upgraded`.
+
+**Files changed (lint/infra):**
+- `scripts/lint-dev-agent-dark-gate.js` — Assertion C (unclassified-dark-default +
+  registered-but-hardcoded-false), exclusion quality validation (closed category
+  enum + ≥12-char reason), brace-in-string loud-fail guard, literal-only limit
+  printed in failure header.
+- `scripts/lib/dark-gate-attribution.js` (NEW) — single path-attribution module
+  shared by the lint and the golden-path test (so the canary checks the SAME
+  attributor the lint uses).
+
+**Files changed (tests):** `tests/unit/lint-dev-agent-dark-gate.test.ts`
+(extended: assertion-C cases, hand-authored golden-path map, brace-in-string,
+destructive-not-gated, exclusion-quality), `tests/unit/PostUpdateMigrator-cartographerDevGate.test.ts`
+(NEW), `tests/integration/conformance-dev-gate-route.test.ts` (NEW),
+`tests/e2e/cartographer-dev-gate-lifecycle.test.ts` (NEW).
+
+**Files added (docs):** the spec, its `.eli16.md` companion, and the convergence report.
+
+## Decision-point inventory
+
+- **`cartographer.enabled`, `cartographer.conformanceAudit.enabled`** → dev-gate
+  (omit + register). Zero egress, zero spend — safe to dogfood live on dev agents.
+- **`cartographer.freshnessSweep.enabled`** → NOT dev-gated; explicit opt-in
+  everywhere (DARK_GATE_EXCLUSIONS, category `cost-bearing`). Auto-arming recurring
+  third-party spend across the dev fleet is a P19 blast-radius risk.
+- **`egressAcknowledged`** → neutralized as a second gate (privacy framing was
+  incoherent: source already egresses to a model every turn). Field retained for
+  back-compat; now inert on the sweep. Noted in the upgrade guide.
+- **The other 18 existing `enabled:false` blocks** → classified into
+  `DARK_GATE_EXCLUSIONS` (destructive / cost-bearing / structural-stub /
+  optional-integration / deliberate-fleet-default). NONE retroactively dev-gated
+  in this PR — that is a deliberate, bounded scope. Each observe-only sentinel
+  would need its own safety judgment before going live on dev agents; this change
+  makes that an explicit, lint-visible decision rather than a silent default.
+
+## 1–7. Analysis
+
+- **Behavioral change:** on a `developmentAgent:true` agent, cartographer's
+  read-only surfaces resolve LIVE (previously dark for everyone). On the fleet,
+  unchanged (still dark). The freshness sweep's activation is UNCHANGED for
+  everyone except that it no longer requires the second `egressAcknowledged` flag.
+- **Migration:** existing dev agents have `cartographer.enabled:false` on disk; the
+  one-shot migration strips the default-shaped value so the gate decides. Run-once
+  marker prevents ever re-stripping an operator's later deliberate `false`. Never
+  touches the cost-bearing sweep — no surprise spend on update.
+- **Security/cost:** no new egress path (the sweep was already the only egress, and
+  its activation conditions only got simpler, not broader — it still needs explicit
+  `enabled:true`). No new spend can be auto-armed. The new lint is deterministic,
+  reads one file, no network.
+- **Reversibility:** fully reversible by reverting the commit. The migration is
+  additive (deletes a default-shaped key); reverting restores prior behavior on the
+  next update for agents that hadn't yet run it.
+- **New failure modes:** none introduced. The lint's brace-in-string guard fails
+  LOUD (not silent) if an unhandled case is introduced — a safety improvement.