agent-bios 0.13.0 → 0.15.0

package/codex/guides/verification-discipline.md

@@ -34,11 +34,21 @@ the depth first:
   over persisted real artifacts rather than re-running the whole pipeline to observe it.
 - Probe at N=1 with the inputs precondition-checked. A single well-chosen case that reaches the
   real path outranks a hundred that stop short of it.
+- Bound the N=1 probe to reachability. One case settles whether a path works — never how often a
+  stochastic behavior holds. Before quoting a rate-shaped property (determinism, flake rate, an
+  A/B effect), confirm the mechanism honors your controls, establish the noise floor from runs on
+  hand, then sample enough to bound the rate against a no-treatment control on the real path. A
+  floor larger than the effect breaks the premise — record it before redesigning.
 - Reserve the full design-review-plus-live-verification treatment for first-of-kind work and for
   changes that move authority — who may decide, who may write, what is irreversible.
 - Proportion assurance to the deployment context. A single-user tool operating on its owner's own
   data does not warrant production-grade assurance, and treating it as if it did buys nothing
   while delaying delivery. Prefer shipping.
+- Census the population before a costed or irreversible batch. Read its current state cheaply and
+  deterministically — status distribution, version, presence of the artifacts the plan expects —
+  and compare that against the premise the batch rests on, halting to re-diagnose when the
+  distribution contradicts it. A probe validates the path; only the census validates that the
+  population is what the plan assumes. A cheap idempotent batch over a few items needs none.
 
 The failure this prevents is not under-testing. It is spending the verification budget on the
 cheap half of the risk and having nothing left for the part that could actually hurt.
@@ -62,10 +72,14 @@ cheapest one to write.
 - Config or data: real parsers, schema checks, fixture validation, and sample transformations.
 - Spreadsheets: static workbook checks, fixture-based output checks, cross-sheet flow checks, visual/layout checks, and real Microsoft Excel engine recalculation for formula-dependent results.
 - Docs: links, terminology, current behavior alignment, and references to isolated historical notes.
+- Multi-subject prose: when deliverables draw on material about different people, companies, or cases, bind every captured item to its subject and confirmation status at capture, and narrate nothing under a subject until that binding is confirmed. Cross-check subject-specific proper nouns and figures across all outputs before delivery — one subject's term under another is the signature of context bleed. Keep a figure in the same sentence as its composition, since a detached number is read at its worst.
 - Release or distribution: after publishing to multiple independently writable channels (signed manifest, object storage, release host, embedded updater), digest-verify every referenced object against the staging original per channel — publish success and upload order are not evidence — and run the real installer/updater through its default path.
 - A/B or on/off measurements: before accepting a null result, verify the arms actually received different treatment in the mechanism under test — a shared default or unconditional upstream step can silently apply the treatment to both arms.
+- Multi-stage pipelines with nondeterministic stages: a final-output diff cannot attribute an effect or a regression to a stage — it conflates the change with run-to-run variance. Persist every stage's output, tabulate what each creates, may edit, and only guards, restrict the suspects to the stages that edit the content in question, and find the first stage where the intended effect disappears or the defect appears. Fix there, preferring a structural recheck over another prompt-level instruction that already failed.
+- Before/after comparisons: pin the input to an immutable copy — a snapshot or versioned artifact — and run both arms against it, because a live artifact (a growing log, a regenerated upstream stage) drifts between runs and any diff over it, a matching one included, is evidence of nothing; when the arms are metered, restore the baseline's exact upstream inputs and re-run only the changed stage. This is input identity, not the separate unit-and-denominator basis rule.
 - Model-behavior guardrails: verify by changed behavior, not recitation — a staged battery from named-trigger cases through disguised, deconfounded, category-wide, and single-variable framings; a clean pass means "no known defect", so re-run the battery when the model changes.
 - Branch/version test builds against real data: explicitly separate every state sink the app touches (files, DB, OS-level stores that ignore env overrides), confirm the launch path propagates the isolation to child processes, and back up live data before the first run — a mismatched schema that drops unknown fields on write is data loss, not a no-op.
+- Sandbox, replay, or re-adjudication runs on production-derived config: enumerate every outbound channel the stage can reach — publish, upload, notify, external write — and disable or redirect each one before the run, proving each disarm fires as you would prove a path guard; a guard on the input or target path alone leaves egress armed. Fingerprint every external destination before the run and diff it after, so an escaped write is caught by the run rather than by a recipient.
 - Irreversible capture switches: when activation itself has unreproducible cost (a capture window that cannot be replayed), prove the downstream consumption path against existing samples before enabling — reversibility of the code path alone is not enough.
 
 ## Deriving the case space
@@ -108,6 +122,12 @@ someone typed once.
   interrupted, a restore sitting after it never runs and the plant survives into a
   commit. Plant in a copy where the shape allows it, and when it must be in place, snapshot
   first and restore from the snapshot as its own step rather than trusting the probe to finish.
+- Attribute a fault by a controlled contrast before choosing a remedy. Hold everything constant —
+  principal, path, input — vary only the candidate variable, and read a countable difference
+  (element dimensions, request count, status-code family). A missing error report is not
+  exoneration: channels can be suppressed, unobserved, or unwired from the mechanism at fault, so
+  "no violations logged" rules a cause out only when the contrast shows none either. Read the
+  contrast the evidence already holds before proposing a policy loosening.
 
 ## When a green means nothing
 
@@ -121,17 +141,65 @@ produce a green with no evidence behind it:
   confirm its inputs satisfy the live branch's entry guard. A copied fixture that fails the new
   guard routes silently into the about-to-be-deleted dead branch and stays green after the real
   behavior breaks.
+- **The fixture the producer never emits.** A test's input is evidence only when the thing that
+  produces it in production produced it. A wire fixture must be a raw response captured through
+  the same client that will parse it, never a rendered listing or a hand-written payload; a gated
+  feature's E2E must run with the gate in its production setting, on the state its real upstream
+  leaves. Replay against the live producer once, then probe every sibling built the same way.
 - **The permissive fallback in the checker.** A `a || b` inside a gate absorbs a wrong assumption
   and keeps passing. Checker code must assert the shape it expects and fail loud.
 - **The suspiciously fast or empty run.** When a check goes green unexpectedly quickly, or reports
   nothing at all, dump what it actually ran over before believing it. A harness that crashed early
   and one that found nothing produce the same exit code.
+- **The control that failed by crashing.** A negative control is evidence only when it fails
+  through the assertion it names: a traceback and a caught violation share an exit code, and an
+  early crash can pre-empt every control after it. Treat each traceback in a control run as a
+  defect until the run reports one named failure per planted violation. Run any unattended gate
+  with stdin closed, so a path reaching a prompt fails at once instead of hanging.
 - **The control that went quiet.** A negative control indexing a live list stops testing when that
   list empties, and says nothing about it. New controls build their own subject; resolving an item
   means re-reading the controls for ones that have gone silent.
-
-The discipline that covers all five: after adding a check, revert the fix it guards and watch the
-check fail. A control that survives a faithful revert was never testing the thing it names.
+- **The silent log.** Absence of activity is not evidence of non-use. Before retiring an
+  identity, key, or endpoint on a quiet log, show that the queried field is the one recording the
+  subject — delegated actions are attributed to the caller, with the target in another field, so
+  the wrong field returns clean silence — and that no live binding still references it, since a
+  reference proves use while emitting no traffic. Only a demonstrated "this use would have been
+  logged" evidences non-use.
+- **The population that shrank.** A gate scanning an explicit list of files sees only "listed but
+  empty": a subject that migrates to an unlisted surface leaves the scanned set smaller yet
+  non-empty, so the empty-subject guard never fires while coverage erodes. Retarget the gate in
+  the same change that moves the code, and pin a floor so any decrease fails loudly — proved by
+  moving one subject out. A floor is a ratchet: never lower it to pass a run.
+- **The mutant that never ran.** A mutation verdict counts only if the mutant is valid: it
+  compiled, sits on a path the exercised test traverses, and changes the guarded behavior, not
+  healed downstream or coinciding with a default. The runner must report build failure,
+  unreachable, and equivalent distinctly from KILLED and SURVIVED, and halt on a moved anchor.
+  More tests red than the mutation should touch indicts it; classify a survivor (rebuild,
+  discard, genuine gap) before writing a test.
+- **The probe that measured the original.** A copied script that derives its root or targets from
+  its own location (`$0`, `BASH_SOURCE`, a `cd` to its parent) scans the original tree, not the
+  copy, so its verdict says nothing about the mutation you planted. Pin the subject in the copy,
+  copy the whole tree, or plant in place with a snapshot-restore. The tell is a result identical
+  to the unmutated run; a script taking its subject as an argument is safe to copy.
+
+The discipline that covers every shape above: after adding a check, revert the fix it guards and
+watch the check fail. A control that survives a faithful revert was never testing the thing it
+names. Prove the plant landed: assert the altered input differs from the original and that the
+control's cases reach the mutated branch before asserting rejection. Construct corruption
+deterministically: scanning for a flip site yields no-op mutations reporting a rejection nothing
+exercised.
+
+## Before blaming code for a metric change
+
+When a live metric collapses or crosses a pre-declared threshold, localize the change in time
+before diagnosing the feature: confirm every inbound source is alive, compare only
+contemporaneous cohorts, never re-processed rows, then bracket the transition to the finest unit
+available and read the deploy log around that instant.
+
+A change landing seconds from the last-good point is the prime suspect; a step in a window with
+no deploys is an input-population shift, fixed by scoping the measured population, not the model
+or the code. Declare the threshold before looking, and confirm a failure from the fleet's vantage
+rather than your own — this does not replace fixing the comparison basis, which comes first.
 
 ## Keeping E2E honest