agent-bios 0.9.9 → 0.10.0

package/codex/guides/verification-discipline.md

@@ -0,0 +1,166 @@
+---
+guide_id: verification-discipline
+language: en
+status: active
+use_when:
+  - deciding how much verification a change deserves, before spending on a slow or expensive run
+  - choosing what to run for a domain — code, ontology, config/data, spreadsheets, docs, a release
+  - building the case space for a check, or deciding what its expected answer should be
+  - a check came back green, empty, or fast, and you are about to believe it
+  - running independent or adversarial review, and judging what its agreement is worth
+core_rules:
+  - a check is only evidence if it could have failed — assert a non-empty subject before any "no bad X" claim
+  - enumerate the case space from the artifact that defines it, and record real output instead of typing an expectation
+  - proportion depth to cost, risk, and information gain; a single-user tool does not warrant production assurance
+  - same-kind reviewers share blind spots, so their shared "clean" is an absence of objection, not verification
+---
+
+# Verification Discipline
+
+A scoped extension of the global Verification Discipline section. Its subject is not "did you
+test it" but the harder question underneath: **could this check have failed?** Everything below
+is a way of answering that before the result is believed rather than after it is quoted.
+
+The global rules that stay always-loaded are the ones whose moment does not announce itself — you
+believe you are finished, and that belief is the failure. This guide is what you open once you
+know you are verifying.
+
+## Proportion the depth before you spend
+
+Verification has a cost and an information yield, and they are not correlated by default. Decide
+the depth first:
+
+- Diagnose in code before running anything expensive, and replay the changed deterministic logic
+  over persisted real artifacts rather than re-running the whole pipeline to observe it.
+- Probe at N=1 with the inputs precondition-checked. A single well-chosen case that reaches the
+  real path outranks a hundred that stop short of it.
+- Reserve the full design-review-plus-live-verification treatment for first-of-kind work and for
+  changes that move authority — who may decide, who may write, what is irreversible.
+- Proportion assurance to the deployment context. A single-user tool operating on its owner's own
+  data does not warrant production-grade assurance, and treating it as if it did buys nothing
+  while delaying delivery. Prefer shipping.
+
+The failure this prevents is not under-testing. It is spending the verification budget on the
+cheap half of the risk and having nothing left for the part that could actually hurt.
+
+## The static floor
+
+Run the broad, cheap checks first and let them fail before anything slower starts: typecheck,
+lint, build, format, schema and config validation, graph validation, workbook structure checks,
+import boundaries, and security checks where they exist. These are a floor, not a verdict — they
+prove the artifact is well-formed, never that it behaves.
+
+## Verification Menus
+
+Pick the narrowest reliable mix that proves the changed behavior, meaning, or contract. Inside the
+mix, the unit to add is the narrowest reliable runtime or semantic test that proves it — narrowest
+meaning the smallest test that would fail if the change were wrong, which is not the same as the
+cheapest one to write.
+
+- Code: a layered mix of unit tests, integration tests for E2E segments, targeted E2E for changed flows, and full E2E for release or high-risk changes.
+- Ontology: static graph checks, concept economy gates, changed-path integration checks, and competency-question E2E checks.
+- Config or data: real parsers, schema checks, fixture validation, and sample transformations.
+- Spreadsheets: static workbook checks, fixture-based output checks, cross-sheet flow checks, visual/layout checks, and real Microsoft Excel engine recalculation for formula-dependent results.
+- Docs: links, terminology, current behavior alignment, and references to isolated historical notes.
+- Release or distribution: after publishing to multiple independently writable channels (signed manifest, object storage, release host, embedded updater), digest-verify every referenced object against the staging original per channel — publish success and upload order are not evidence — and run the real installer/updater through its default path.
+- A/B or on/off measurements: before accepting a null result, verify the arms actually received different treatment in the mechanism under test — a shared default or unconditional upstream step can silently apply the treatment to both arms.
+- Model-behavior guardrails: verify by changed behavior, not recitation — a staged battery from named-trigger cases through disguised, deconfounded, category-wide, and single-variable framings; a clean pass means "no known defect", so re-run the battery when the model changes.
+- Branch/version test builds against real data: explicitly separate every state sink the app touches (files, DB, OS-level stores that ignore env overrides), confirm the launch path propagates the isolation to child processes, and back up live data before the first run — a mismatched schema that drops unknown fields on write is data loss, not a no-op.
+- Irreversible capture switches: when activation itself has unreproducible cost (a capture window that cannot be replayed), prove the downstream consumption path against existing samples before enabling — reversibility of the code path alone is not enough.
+
+## Deriving the case space
+
+Which scenarios exist is semantic work: derive them from the diff, the user impact, the
+concept impact, and the failure modes. Running them is not — tools and code execute the
+cases and report the evidence. Keeping that split is what stops a suite from being a
+record of what someone imagined.
+
+A check has two authored halves, and they rot differently. The **verdict** — what the
+answer should be — rots by encoding a belief that was wrong from the start. The
+**space** — which cases exist — rots by staying still while the thing it covers grows.
+Recording the verdict is common practice; deriving the space is the half usually left
+hand-written, and a suite can have every expectation derived and still cover a set
+someone typed once.
+
+- Make the criterion falsifiable before you make it green. Prefer a signal that fails when
+  the mechanism is wrong — a negative or contrast control. Where no existing gate can judge
+  a criterion, build the executable judge or do not claim the criterion met: a criterion
+  nothing can fail is a description of the work, not a check on it.
+- Record the verdict, do not type it. Run the real path and store what came back;
+  drift then shows as a diff instead of as a belief someone has to re-justify.
+- Enumerate the space from the artifact that defines it — the config's entries, the
+  schema's fields, the router's routes, the installer's call sites. Adding one there
+  should widen coverage with no edit here.
+- Derive the exemption rule too. If some cases legitimately have no answer, decide that
+  from a property the artifact carries, never from a list of names: the list is the
+  authored space coming back through a side door, and it absorbs the regression where
+  a case that should have an answer stops having one.
+- Dedupe on the tuple that actually determines the outcome, and report how many
+  collapsed. A coverage count that hides its own truncation reads as more than it is.
+- Split by cost, not by space. When the real path needs money, credentials, or a
+  network, run a cheap stand-in on every commit and the real one on demand — both from
+  the **same enumeration**, so the two can never disagree about which cases exist.
+- Derivation moves authorship rather than removing it: the extractor and the invariants
+  are still written by hand. Give them a negative control, or the derived suite is just
+  a larger unfalsifiable one.
+- Planting a violation to prove a control fires is a write into the working tree, and
+  the restore is not atomic with it: if the probe can time out, abort, or be
+  interrupted, a restore sitting after it never runs and the plant survives into a
+  commit. Plant in a copy where the shape allows it, and when it must be in place, snapshot
+  first and restore from the snapshot as its own step rather than trusting the probe to finish.
+
+## When a green means nothing
+
+A passing check and a check that never ran look identical from outside. These are the shapes that
+produce a green with no evidence behind it:
+
+- **The empty subject.** Any "no bad X" or "all X satisfy P" claim over an empty set is
+  vacuously true. Assert the entity-under-test set has cardinality greater than zero **before**
+  the claim, and make the gate itself refuse to report clean when it judged nothing.
+- **The fixture that misses the guard.** For a test touching a branch you are adding or deleting,
+  confirm its inputs satisfy the live branch's entry guard. A copied fixture that fails the new
+  guard routes silently into the about-to-be-deleted dead branch and stays green after the real
+  behavior breaks.
+- **The permissive fallback in the checker.** A `a || b` inside a gate absorbs a wrong assumption
+  and keeps passing. Checker code must assert the shape it expects and fail loud.
+- **The suspiciously fast or empty run.** When a check goes green unexpectedly quickly, or reports
+  nothing at all, dump what it actually ran over before believing it. A harness that crashed early
+  and one that found nothing produce the same exit code.
+- **The control that went quiet.** A negative control indexing a live list stops testing when that
+  list empties, and says nothing about it. New controls build their own subject; resolving an item
+  means re-reading the controls for ones that have gone silent.
+
+The discipline that covers all five: after adding a check, revert the fix it guards and watch the
+check fail. A control that survives a faithful revert was never testing the thing it names.
+
+## Keeping E2E honest
+
+E2E is where flakiness is mistaken for environment noise and then ignored. Keep it deterministic
+with fixed data, resilient selectors, isolated external dependencies, and explicit waits rather
+than sleeps. A flaky E2E is not a weaker test; it is a test whose result carries no information,
+and a suite that people re-run until it passes has been switched off without anyone deciding to.
+
+## Independent review, and what agreement is worth
+
+For non-trivial designs and high-risk changes, run independent adversarial review across distinct
+lenses — ideally on the design, before implementation, when a finding is still cheap to act on.
+Then re-verify each finding against real code before acting on it: a reviewer reasons from what it
+was shown, and what it was shown may be wrong.
+
+Apply the **convergence heuristic by reviewer kind** — judge the result by reviewer kind, not by count:
+
+- Same-kind convergence is high confidence but blind-spot-sharing. Two reviewers of the same kind
+  agreeing that something is clean is an absence of objection, not verification.
+- Different-kind divergence is the expected signal, not a problem to resolve. Act on the union of
+  what they found rather than the intersection.
+- An orchestrated workflow's self-reported all-green is never sufficient on its own. Re-run the
+  diff inspection and the verification suite yourself.
+
+The cheapest way to buy real independence is a different provider; after that a different model;
+after that strictly higher effort. A reviewer run at lower effort than the work it checks buys
+nothing — cheaper is not another perspective.
+
+## Reporting
+
+Before calling the work done, state the checks that ran, their results, and any risk left
+unverified. "Unverified" is a legitimate outcome and a useful one; silence about it is not.

package/compose/assemble.py

@@ -226,6 +226,14 @@ def copy_filtered(src_dir, names, dest, rewrite=None, dry=False, backup=None):
         target.write_text(body, encoding="utf-8")
 
 
+def hook_command_matches(command, name):
+    """A complete `/hooks/<name>` token: the command ends with it, or it is followed by an
+    argument. One matcher for the merge below AND the domain gate's settings leg — the gate
+    used a substring test, so `central/hooks/<name>.disabled` passed the gate while this
+    merge skipped it, and the install carried no hook under a green gate."""
+    return command.endswith("/hooks/" + name) or f"/hooks/{name} " in command + " "
+
+
 def merge_settings(claude_dir, hook_names, template_path, dry=False, owned_names=None):
     """Merge our hook registrations into the user's settings, owning by NAME.
 
@@ -254,7 +262,8 @@ def merge_settings(claude_dir, hook_names, template_path, dry=False, owned_names
     for event, entries in template.get("hooks", {}).items():  # re-add per selection
         for en in entries:
             cmds = [h.get("command", "") for h in en.get("hooks", [])]
-            owner = next((n for n in hook_names if any(c.endswith("/hooks/" + n) or f"/hooks/{n} " in c + " " for c in cmds)), None)
+            owner = next((n for n in hook_names
+                          if any(hook_command_matches(c, n) for c in cmds)), None)
             if owner is None:
                 continue
             clone = json.loads(json.dumps(en))