agent-bios 0.13.0 → 0.14.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -2
- package/claude/CLAUDE.md +8 -8
- package/claude/guides/cli-multi-model-workflow.md +6 -1
- package/claude/guides/coding-staged-workflow.md +12 -0
- package/claude/guides/learning-flow.md +4 -1
- package/claude/guides/llm-capability-boundary-patterns.md +8 -0
- package/claude/guides/review-request.md +13 -0
- package/claude/guides/session-distill-workflow.md +21 -9
- package/claude/guides/tooling-gotchas.md +181 -14
- package/claude/guides/verification-discipline.md +71 -3
- package/claude/hooks/tooling-gotchas-hook.py +50 -0
- package/codex/AGENTS.md +8 -8
- package/codex/guides/cli-multi-model-workflow.md +6 -1
- package/codex/guides/coding-staged-workflow.md +12 -0
- package/codex/guides/learning-flow.md +4 -1
- package/codex/guides/llm-capability-boundary-patterns.md +8 -0
- package/codex/guides/review-request.md +13 -0
- package/codex/guides/session-distill-workflow.md +21 -9
- package/codex/guides/tooling-gotchas.md +181 -14
- package/codex/guides/verification-discipline.md +71 -3
- package/install.sh +87 -5
- package/launch/agent-launch.zsh +109 -6
- package/learn/collect-learning.py +593 -62
- package/learn/redact.py +2 -1
- package/package.json +2 -2
- package/provenance.json +1 -1
|
@@ -40,28 +40,47 @@ depends on it, pin it explicitly instead of trusting the environment.
|
|
|
40
40
|
shells resolve functions/aliases first, programmatic spawns resolve raw
|
|
41
41
|
PATH, and a same-named package can shadow a system tool with silent empty
|
|
42
42
|
output. Before trusting a result across execution contexts, confirm the
|
|
43
|
-
resolved target (`type -a`, absolute path).
|
|
43
|
+
resolved target (`type -a`, absolute path). A missing prefix wrapper fails
|
|
44
|
+
the same silent way — GNU `timeout` is routinely absent on BSD-derived
|
|
45
|
+
systems — so confirm the wrapper too.
|
|
44
46
|
- **Cloud CLI context**: gcloud/aws/kubectl/terraform carry mutable ambient
|
|
45
47
|
context (active project, profile, cluster) that drifts between sessions.
|
|
46
48
|
Before the first environment-affecting command — or right after a resume —
|
|
47
49
|
verify it against intent, then pin the target explicitly on every command
|
|
48
50
|
(`--project`, `--profile`, `--context`) rather than fixing the global
|
|
49
|
-
default once.
|
|
51
|
+
default once. A forge CLI (`gh`/`glab`) reads its repository from the
|
|
52
|
+
checkout's remotes too: with a fork plus an upstream it can answer for the
|
|
53
|
+
wrong repo, so pass `--repo` wherever the answer feeds a decision.
|
|
50
54
|
- **Installed is not running**: a live process keeps its old code until
|
|
51
55
|
restarted or reloaded. When confirming an update, config change, or
|
|
52
56
|
dependency bump took effect, don't stop at the on-disk artifact — confirm
|
|
53
57
|
the running process's actual version/behavior or force a restart.
|
|
58
|
+
- **Producer newer than consumer**: when a deployed binary validates an
|
|
59
|
+
artifact you produce (a signed config or manifest), produce and verify it
|
|
60
|
+
with the producer tooling checked out at the exact commit that binary was
|
|
61
|
+
built from — a local verify with current-tree tooling proves only that
|
|
62
|
+
newer tooling accepts it. Confirm the consumer's build commit from image
|
|
63
|
+
provenance, not the current branch; any version mismatch is a release
|
|
64
|
+
blocker. A consumer rebuilt from the same commit needs only ordinary
|
|
65
|
+
verification.
|
|
66
|
+
- **A dev-labelled datastore target is a claim**: a localhost URL or
|
|
67
|
+
exported override is no evidence of a non-production target — a local port
|
|
68
|
+
can proxy into the only real instance, and a tool's config loader can
|
|
69
|
+
re-load a dotenv over your exported value. Before the first writing
|
|
70
|
+
command (migration, seeder), print what the connection reaches from inside
|
|
71
|
+
the tool's own path and assert it is the intended target; where the loader
|
|
72
|
+
is untrustworthy, extract the DDL and apply it yourself.
|
|
54
73
|
|
|
55
74
|
## Shell execution traps
|
|
56
75
|
|
|
57
76
|
- **Pipe exit masking**: `$?` after a pipeline reflects only the last stage;
|
|
58
|
-
a real failure
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
77
|
+
a real failure upstream is masked by a successful `tail`/`grep`/`jq`,
|
|
78
|
+
reading green. Capture the tested stage's own status: run it unpiped,
|
|
79
|
+
store `$?` immediately, or use `set -o pipefail`/`PIPESTATUS` — a
|
|
80
|
+
per-command choice, since pipefail breaks early-exit consumers (`cmd |
|
|
81
|
+
head -1`, `grep -q` on a long producer → SIGPIPE 141). The
|
|
82
|
+
final-stage-assertion exemption (`cmd | grep -q pattern`) holds only
|
|
83
|
+
without pipefail; under it, capture output and check its status first.
|
|
65
84
|
- **Passthrough arguments in a CLI you author**: an option meant to carry
|
|
66
85
|
another command's own flags cannot use a greedy-but-dash-stopping arity —
|
|
67
86
|
Python's `nargs="+"` ends at the first token starting with `-`, so the
|
|
@@ -71,6 +90,13 @@ depends on it, pin it explicitly instead of trusting the environment.
|
|
|
71
90
|
second, separate trap: argparse consumes it as its own positional marker
|
|
72
91
|
before the remainder sees it, so the form every caller reaches for first is
|
|
73
92
|
the one that breaks — normalize it out of `argv` before parsing.
|
|
93
|
+
- **CLI flag probes that execute**: probe a CLI only with invocations that
|
|
94
|
+
cannot do real work — a help form, or the candidate flag paired with a
|
|
95
|
+
control flag that forbids execution (dry-run, an invalid required
|
|
96
|
+
argument). Never run a subcommand bare, and assume a value after a flag
|
|
97
|
+
may be read as positional input: a boolean flag does not consume it, so it
|
|
98
|
+
falls through and runs. Tell a boolean from an unregistered flag by the
|
|
99
|
+
parser's error, not by the run succeeding.
|
|
74
100
|
- **Reserved parameter names**: assigning to reserved shell names (`UID`,
|
|
75
101
|
`EUID`, `GID`, `PPID`) can invoke the bound system behavior instead of
|
|
76
102
|
storing a value — silently changing process credentials mid-script. Use
|
|
@@ -105,6 +131,14 @@ depends on it, pin it explicitly instead of trusting the environment.
|
|
|
105
131
|
timestamp resolution (mutation testing). Clear the cache or run no-cache
|
|
106
132
|
per iteration, and re-confirm the unmutated baseline still passes after a
|
|
107
133
|
cache clear.
|
|
134
|
+
- **Transport limits are measured on the wire payload, in the provider's
|
|
135
|
+
unit**: take the unit and value from the provider's rejection or a live
|
|
136
|
+
probe, never from docs or a variable's name, and measure the serialized
|
|
137
|
+
payload the consumer receives — after encoding and wrappers — not the
|
|
138
|
+
object you assembled. A character count against a byte limit undercounts
|
|
139
|
+
multibyte text, hiding while inputs are ASCII; an item count bounds no
|
|
140
|
+
size. Enforce at one dispatch chokepoint, deriving every budget from that
|
|
141
|
+
constant.
|
|
108
142
|
|
|
109
143
|
## Git operations
|
|
110
144
|
|
|
@@ -131,12 +165,31 @@ depends on it, pin it explicitly instead of trusting the environment.
|
|
|
131
165
|
restore from that. The same asymmetry makes the restore step fragile: if the
|
|
132
166
|
probe can time out or abort, the restore must not be the next command in the
|
|
133
167
|
same invocation — put it where a failure cannot skip it.
|
|
168
|
+
- **An ignore rule can swallow a durable record**: before treating a path as
|
|
169
|
+
durable — a new ledger, a cited authority — run `git check-ignore -v
|
|
170
|
+
<path>` and `git ls-files --error-unmatch <path>`. Broad runtime-state
|
|
171
|
+
patterns (`*.jsonl`, `runs/`, `out/`) absorb a new file, and a tracked
|
|
172
|
+
file pointing at an ignored path is an authority that exists in one
|
|
173
|
+
checkout only. Fix with a negation rule proven by a sibling that stays
|
|
174
|
+
ignored; genuinely ephemeral output stays ignored.
|
|
134
175
|
- **Dirty-worktree pulls**: before pulling into a worktree with
|
|
135
176
|
staged/unstaged/untracked changes, fetch first and compare incoming paths
|
|
136
177
|
against every dirty path; on overlap or a non-fast-forward, stop and clear
|
|
137
178
|
the conflict risk (stash, commit, ask). Otherwise pull `--ff-only`, confirm
|
|
138
179
|
dirty changes survived, and regenerate any local derived artifacts whose
|
|
139
180
|
inputs were updated.
|
|
181
|
+
- **A split series is proven commit by commit**: order them by dependency
|
|
182
|
+
and check each out into a throwaway worktree to run the build, tests, and
|
|
183
|
+
gates before pushing. Green only at the tip hides a broken bisect point
|
|
184
|
+
and a commit that cannot be reverted alone — usually a rename or shared
|
|
185
|
+
hunk in the wrong commit. If a handoff cites the branch's hashes, merge
|
|
186
|
+
with a merge commit: squash and rebase rewrite every hash.
|
|
187
|
+
- **A shared tree holds other operators' work**: a commit you did not make, a file the
|
|
188
|
+
editor reports changed on disk, a staged path you never added, one more field than
|
|
189
|
+
your predicted post-state — treat an unexplained delta as someone else's work, not
|
|
190
|
+
noise. Before a sweeping write (`git add -A`/`.`, `commit -a`, `stash`, `clean`,
|
|
191
|
+
`reset --hard`), attribute it (`git status`, reflog timestamps, other live sessions)
|
|
192
|
+
and then act only on what you can prove is yours — add by name.
|
|
140
193
|
|
|
141
194
|
## Config, secrets, and managed services
|
|
142
195
|
|
|
@@ -151,6 +204,13 @@ depends on it, pin it explicitly instead of trusting the environment.
|
|
|
151
204
|
leaving stale, unreferenced definitions live. After updating, re-read the
|
|
152
205
|
resource, check definitions and active references separately, and remove
|
|
153
206
|
the orphans explicitly.
|
|
207
|
+
- **Derive a new revision from the live one, not from a template**: a
|
|
208
|
+
replace-semantics update drops every field the command does not restate,
|
|
209
|
+
and wrappers commonly default mounted secrets to off. Render the exact set
|
|
210
|
+
the command will send, derived from the live resource, and diff it field
|
|
211
|
+
by field; a field that disappears, or an operational value that moves
|
|
212
|
+
backward, is a blocker to explain, not a default to accept. Re-read the
|
|
213
|
+
resource afterwards, since a command rarely labels its semantics.
|
|
154
214
|
- **A new revision is not live traffic**: on a runtime that pins traffic to a
|
|
155
215
|
named revision (e.g. Cloud Run with a fixed split), `gcloud run deploy` (or
|
|
156
216
|
the equivalent) creates the new revision but shifts no traffic to it — the
|
|
@@ -158,23 +218,101 @@ depends on it, pin it explicitly instead of trusting the environment.
|
|
|
158
218
|
update-traffic`. Read the "deploy succeeded" message as "a revision exists",
|
|
159
219
|
not "the new code is serving"; verify the live traffic split before
|
|
160
220
|
concluding the deploy took effect.
|
|
161
|
-
- **
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
221
|
+
- **Job logs outlive the execution**: a managed job that has run more than
|
|
222
|
+
once under one name — including one deleted and recreated with the same
|
|
223
|
+
name — returns the earlier incarnations' output when its logs are read by
|
|
224
|
+
job name. Scope every read to the execution id you received at launch and
|
|
225
|
+
confirm the timestamp window covers that run. An unscoped read merges
|
|
226
|
+
prior runs into the present and yields confident false diagnoses that a
|
|
227
|
+
scoped read reverses.
|
|
228
|
+
- **Dispatch status is not execution**: a CLI `--wait` returning or timing
|
|
229
|
+
out, a scheduler reporting success, a trigger accepted without error —
|
|
230
|
+
each reports what the dispatcher saw, not whether the target ran or what
|
|
231
|
+
state it reached. Before retrying or declaring done, re-derive the state
|
|
232
|
+
from the target's own record (its execution describe, the handler's logs),
|
|
233
|
+
matched to the run by id, and keep a manual probe distinguishable from the
|
|
234
|
+
scheduled one. A blind re-launch is a duplicate execution with side
|
|
235
|
+
effects.
|
|
236
|
+
- **An apply cut off before confirmation is unconfirmed** — neither done nor
|
|
237
|
+
un-run: when a multi-statement side-effecting apply (migration, batch
|
|
238
|
+
write) loses its confirmation channel, enumerate which target objects
|
|
239
|
+
already exist in the store, and plan the rerun from that partial state; a
|
|
240
|
+
naive rerun half-fails on "already exists" and leaves a second partial
|
|
241
|
+
state. Where the runner surfaces only exit status, route the object list
|
|
242
|
+
out through a deliberate failure. A transactional or provably idempotent
|
|
243
|
+
apply needs only the confirmation.
|
|
244
|
+
- **A traffic rollback is not a config rollback**: on a revision-pinned
|
|
245
|
+
runtime, sending traffic back to the previous revision restores behavior
|
|
246
|
+
but leaves the added env var or secret binding in the service template,
|
|
247
|
+
where the next deploy re-enables it silently. Count a rollback complete
|
|
248
|
+
only when the traffic split and the service spec are both back to the
|
|
249
|
+
prior state — re-read the spec and remove the change explicitly. Runtimes
|
|
250
|
+
that redeploy the prior spec itself (immutable-artifact, GitOps) have no
|
|
251
|
+
such gap.
|
|
252
|
+
- **Perimeter controls need the enforcement point's own logs**: an
|
|
253
|
+
agent-side fetch is not an independent observer — its egress IP and
|
|
254
|
+
caching path are opaque, and it may share the protected network or serve a
|
|
255
|
+
stale cached response. Verify allow AND deny directions from those logs,
|
|
256
|
+
and check for a front-side cache/CDN separately. Aim the probe at an
|
|
257
|
+
in-unit sentinel the app answers without credentials: a denial the app
|
|
258
|
+
produces anyway passes with the control off, and a redirect into it is a
|
|
259
|
+
bypass.
|
|
260
|
+
- **Tightening exposure is a behavior change for external clients**: switching
|
|
261
|
+
ingress mode, adding an allowlist, or requiring auth is not safe when
|
|
262
|
+
callers live outside your redeploy. Enumerate which clients reach the
|
|
263
|
+
endpoint and by which hostname, verify from a client's vantage, and
|
|
264
|
+
confirm inbound volume did not fall to zero — clients you cut off raise no
|
|
265
|
+
error on your side, so enforcement-point logs alone can sever ingestion
|
|
266
|
+
silently. Callers you redeploy in the same change need only the ordinary
|
|
267
|
+
deploy check.
|
|
166
268
|
- **Smoke limits outlive the smoke test**: item caps, sample sizes, and row
|
|
167
269
|
limits left in env vars/flags/config make a later "full-scale" run silently
|
|
168
270
|
succeed on a slice. Clearing or explicitly verifying their absence is a
|
|
169
271
|
precondition of declaring a full run.
|
|
272
|
+
- **A remote handle is valid only when it was read**: a row index in a sheet
|
|
273
|
+
that auto-sorts, a downloaded copy of a hosted file — each moves between
|
|
274
|
+
your read and your write. Before writing, re-establish the target from the
|
|
275
|
+
live source: re-locate the row by its key column, not a remembered
|
|
276
|
+
position, and assert the key matches after the write; compare the remote's
|
|
277
|
+
version against the copy you edited, re-downloading and reapplying on a
|
|
278
|
+
move. Local single-writer files need none of this.
|
|
279
|
+
- **Packaging and ignore rules are judged against paths and environments, not your tree**: before
|
|
280
|
+
a release, pack the real tarball, install it clean with lifecycle scripts
|
|
281
|
+
ON, and smoke it — a postinstall hook fine in the repo can delete the
|
|
282
|
+
shipped runtime where the build toolchain is absent. After moving or
|
|
283
|
+
renaming a directory, every ignore rule is void for the new paths:
|
|
284
|
+
re-check there and read the staged diff's file count, since old-path
|
|
285
|
+
patterns stop matching and excluded data enters the stage.
|
|
170
286
|
- **Shared live config has concurrent writers**: before concluding your edit
|
|
171
287
|
to a shared state/config file was lost or corrupting, rule out concurrent
|
|
172
288
|
writers with a short live observation (mtime plus the fields you changed),
|
|
173
289
|
and scope merge/union operations to the intended fields only.
|
|
290
|
+
- **Uniform failure is structural — read the persisted reason, then check the built artifact**: when
|
|
291
|
+
every item in a batch fails and the per-item error is persisted outside
|
|
292
|
+
the log (a status column, a result record), read it before blaming keys,
|
|
293
|
+
quota, or model availability. And when code reads sibling files from its
|
|
294
|
+
working directory, prove they exist inside the built image by listing or
|
|
295
|
+
hashing them there: a selective copy passes every repo-side check and
|
|
296
|
+
fails only at runtime.
|
|
174
297
|
- **Production probes expose data**: default diagnostic queries against
|
|
175
298
|
production stores to read-only server-side aggregation (counts, types,
|
|
176
299
|
presence, hashes) — never pull raw payloads into logs, prompts, or
|
|
177
300
|
transcripts — and delete scratch probe resources after the decision.
|
|
301
|
+
- **Build-context ignore patterns anchor at the root**: in a `.dockerignore`
|
|
302
|
+
or any root-anchored filter, a bare filename matches only at the context
|
|
303
|
+
root and never in a subdirectory, and an extension glob misses the
|
|
304
|
+
same-purpose credential file carrying another extension. Never conclude a
|
|
305
|
+
shipped image is secret-free from the patterns — list the built artifact's
|
|
306
|
+
own filesystem for credential-shaped files (env files, keys,
|
|
307
|
+
service-account JSON) as a named negative control, repeated whenever the
|
|
308
|
+
build context or ignore file changes.
|
|
309
|
+
- **A revoke is grant-wide, not token-wide**: revoking anything issued under
|
|
310
|
+
a client id the user's live sessions share invalidates those sessions too,
|
|
311
|
+
and the failure surfaces later, elsewhere, with no re-auth prompt. A probe
|
|
312
|
+
may clean up only what it alone owns — use minimum scopes and a dedicated
|
|
313
|
+
client id, and let a probe token expire rather than revoking it. Where a
|
|
314
|
+
revoke on a shared grant is unavoidable, state the blast radius and time
|
|
315
|
+
it with the user.
|
|
178
316
|
|
|
179
317
|
## Own what you spawn
|
|
180
318
|
|
|
@@ -186,3 +324,32 @@ Instance of the global rule: own the full lifecycle of what you create.
|
|
|
186
324
|
orphans the real child), and await exit. An unref'd child handle or open
|
|
187
325
|
stdin pipe keeps the parent's event loop alive and hangs otherwise-complete
|
|
188
326
|
commands.
|
|
327
|
+
- **A handle issued to a human is a commitment**: once a consent URL is
|
|
328
|
+
handed over, the listener behind it must not be restarted, re-ported, or
|
|
329
|
+
replaced while the person may still act — keep it alive until the callback
|
|
330
|
+
lands, or say plainly that the link is dead. The handler captures the
|
|
331
|
+
credential write-once and ignores later hits, since a browser's favicon
|
|
332
|
+
request clears a naive one. Before asking a human to click again, drive
|
|
333
|
+
the handler with a simulated callback.
|
|
334
|
+
- **A stop is confirmed at the sink, not in the process list**: a kill that
|
|
335
|
+
misses one descendant lets that stage finish and publish, and the process
|
|
336
|
+
list looks clean either way. After stopping a multi-stage run, list the
|
|
337
|
+
output sink for anything written after the stop instant and roll it back
|
|
338
|
+
as contaminated. Read the rollback path's retention — noncurrent-version
|
|
339
|
+
expiry is a recovery window, not a backup — and snapshot before a risky
|
|
340
|
+
run, choosing the restore point by timestamp.
|
|
341
|
+
- **Detach what must outlive the call**: a process started inside a harness
|
|
342
|
+
tool call belongs to that call's process group: a trailing `&` is reaped
|
|
343
|
+
when the call returns, and the harness may signal the group when another
|
|
344
|
+
background task finishes. Launch anything meant to outlive one call
|
|
345
|
+
through the harness's background facility or fully detached
|
|
346
|
+
(nohup/setsid), writing progress to durable files a resume can read. A
|
|
347
|
+
zero-byte output file means it never survived. Detached still means owned
|
|
348
|
+
— PID file and stop path.
|
|
349
|
+
- **Name-substring lookup is not a liveness check**: `pgrep -f <name>` and
|
|
350
|
+
`ps | grep <name>` match the argv of the shell running the query itself,
|
|
351
|
+
and match unrelated processes carrying the same name — another session, a
|
|
352
|
+
sibling dispatch, the launcher's own plan text. To decide whether a
|
|
353
|
+
dispatched job is alive, use the PID or handle captured at launch, its
|
|
354
|
+
process-group state, or growth of its own output artifact. Substring
|
|
355
|
+
lookup is for discovery only, confirmed against one of those first.
|
|
@@ -34,11 +34,21 @@ the depth first:
|
|
|
34
34
|
over persisted real artifacts rather than re-running the whole pipeline to observe it.
|
|
35
35
|
- Probe at N=1 with the inputs precondition-checked. A single well-chosen case that reaches the
|
|
36
36
|
real path outranks a hundred that stop short of it.
|
|
37
|
+
- Bound the N=1 probe to reachability. One case settles whether a path works — never how often a
|
|
38
|
+
stochastic behavior holds. Before quoting a rate-shaped property (determinism, flake rate, an
|
|
39
|
+
A/B effect), confirm the mechanism honors your controls, establish the noise floor from runs on
|
|
40
|
+
hand, then sample enough to bound the rate against a no-treatment control on the real path. A
|
|
41
|
+
floor larger than the effect breaks the premise — record it before redesigning.
|
|
37
42
|
- Reserve the full design-review-plus-live-verification treatment for first-of-kind work and for
|
|
38
43
|
changes that move authority — who may decide, who may write, what is irreversible.
|
|
39
44
|
- Proportion assurance to the deployment context. A single-user tool operating on its owner's own
|
|
40
45
|
data does not warrant production-grade assurance, and treating it as if it did buys nothing
|
|
41
46
|
while delaying delivery. Prefer shipping.
|
|
47
|
+
- Census the population before a costed or irreversible batch. Read its current state cheaply and
|
|
48
|
+
deterministically — status distribution, version, presence of the artifacts the plan expects —
|
|
49
|
+
and compare that against the premise the batch rests on, halting to re-diagnose when the
|
|
50
|
+
distribution contradicts it. A probe validates the path; only the census validates that the
|
|
51
|
+
population is what the plan assumes. A cheap idempotent batch over a few items needs none.
|
|
42
52
|
|
|
43
53
|
The failure this prevents is not under-testing. It is spending the verification budget on the
|
|
44
54
|
cheap half of the risk and having nothing left for the part that could actually hurt.
|
|
@@ -62,10 +72,14 @@ cheapest one to write.
|
|
|
62
72
|
- Config or data: real parsers, schema checks, fixture validation, and sample transformations.
|
|
63
73
|
- Spreadsheets: static workbook checks, fixture-based output checks, cross-sheet flow checks, visual/layout checks, and real Microsoft Excel engine recalculation for formula-dependent results.
|
|
64
74
|
- Docs: links, terminology, current behavior alignment, and references to isolated historical notes.
|
|
75
|
+
- Multi-subject prose: when deliverables draw on material about different people, companies, or cases, bind every captured item to its subject and confirmation status at capture, and narrate nothing under a subject until that binding is confirmed. Cross-check subject-specific proper nouns and figures across all outputs before delivery — one subject's term under another is the signature of context bleed. Keep a figure in the same sentence as its composition, since a detached number is read at its worst.
|
|
65
76
|
- Release or distribution: after publishing to multiple independently writable channels (signed manifest, object storage, release host, embedded updater), digest-verify every referenced object against the staging original per channel — publish success and upload order are not evidence — and run the real installer/updater through its default path.
|
|
66
77
|
- A/B or on/off measurements: before accepting a null result, verify the arms actually received different treatment in the mechanism under test — a shared default or unconditional upstream step can silently apply the treatment to both arms.
|
|
78
|
+
- Multi-stage pipelines with nondeterministic stages: a final-output diff cannot attribute an effect or a regression to a stage — it conflates the change with run-to-run variance. Persist every stage's output, tabulate what each creates, may edit, and only guards, restrict the suspects to the stages that edit the content in question, and find the first stage where the intended effect disappears or the defect appears. Fix there, preferring a structural recheck over another prompt-level instruction that already failed.
|
|
79
|
+
- Before/after comparisons: pin the input to an immutable copy — a snapshot or versioned artifact — and run both arms against it, because a live artifact (a growing log, a regenerated upstream stage) drifts between runs and any diff over it, a matching one included, is evidence of nothing; when the arms are metered, restore the baseline's exact upstream inputs and re-run only the changed stage. This is input identity, not the separate unit-and-denominator basis rule.
|
|
67
80
|
- Model-behavior guardrails: verify by changed behavior, not recitation — a staged battery from named-trigger cases through disguised, deconfounded, category-wide, and single-variable framings; a clean pass means "no known defect", so re-run the battery when the model changes.
|
|
68
81
|
- Branch/version test builds against real data: explicitly separate every state sink the app touches (files, DB, OS-level stores that ignore env overrides), confirm the launch path propagates the isolation to child processes, and back up live data before the first run — a mismatched schema that drops unknown fields on write is data loss, not a no-op.
|
|
82
|
+
- Sandbox, replay, or re-adjudication runs on production-derived config: enumerate every outbound channel the stage can reach — publish, upload, notify, external write — and disable or redirect each one before the run, proving each disarm fires as you would prove a path guard; a guard on the input or target path alone leaves egress armed. Fingerprint every external destination before the run and diff it after, so an escaped write is caught by the run rather than by a recipient.
|
|
69
83
|
- Irreversible capture switches: when activation itself has unreproducible cost (a capture window that cannot be replayed), prove the downstream consumption path against existing samples before enabling — reversibility of the code path alone is not enough.
|
|
70
84
|
|
|
71
85
|
## Deriving the case space
|
|
@@ -108,6 +122,12 @@ someone typed once.
|
|
|
108
122
|
interrupted, a restore sitting after it never runs and the plant survives into a
|
|
109
123
|
commit. Plant in a copy where the shape allows it, and when it must be in place, snapshot
|
|
110
124
|
first and restore from the snapshot as its own step rather than trusting the probe to finish.
|
|
125
|
+
- Attribute a fault by a controlled contrast before choosing a remedy. Hold everything constant —
|
|
126
|
+
principal, path, input — vary only the candidate variable, and read a countable difference
|
|
127
|
+
(element dimensions, request count, status-code family). A missing error report is not
|
|
128
|
+
exoneration: channels can be suppressed, unobserved, or unwired from the mechanism at fault, so
|
|
129
|
+
"no violations logged" rules a cause out only when the contrast shows none either. Read the
|
|
130
|
+
contrast the evidence already holds before proposing a policy loosening.
|
|
111
131
|
|
|
112
132
|
## When a green means nothing
|
|
113
133
|
|
|
@@ -121,17 +141,65 @@ produce a green with no evidence behind it:
|
|
|
121
141
|
confirm its inputs satisfy the live branch's entry guard. A copied fixture that fails the new
|
|
122
142
|
guard routes silently into the about-to-be-deleted dead branch and stays green after the real
|
|
123
143
|
behavior breaks.
|
|
144
|
+
- **The fixture the producer never emits.** A test's input is evidence only when the thing that
|
|
145
|
+
produces it in production produced it. A wire fixture must be a raw response captured through
|
|
146
|
+
the same client that will parse it, never a rendered listing or a hand-written payload; a gated
|
|
147
|
+
feature's E2E must run with the gate in its production setting, on the state its real upstream
|
|
148
|
+
leaves. Replay against the live producer once, then probe every sibling built the same way.
|
|
124
149
|
- **The permissive fallback in the checker.** A `a || b` inside a gate absorbs a wrong assumption
|
|
125
150
|
and keeps passing. Checker code must assert the shape it expects and fail loud.
|
|
126
151
|
- **The suspiciously fast or empty run.** When a check goes green unexpectedly quickly, or reports
|
|
127
152
|
nothing at all, dump what it actually ran over before believing it. A harness that crashed early
|
|
128
153
|
and one that found nothing produce the same exit code.
|
|
154
|
+
- **The control that failed by crashing.** A negative control is evidence only when it fails
|
|
155
|
+
through the assertion it names: a traceback and a caught violation share an exit code, and an
|
|
156
|
+
early crash can pre-empt every control after it. Treat each traceback in a control run as a
|
|
157
|
+
defect until the run reports one named failure per planted violation. Run any unattended gate
|
|
158
|
+
with stdin closed, so a path reaching a prompt fails at once instead of hanging.
|
|
129
159
|
- **The control that went quiet.** A negative control indexing a live list stops testing when that
|
|
130
160
|
list empties, and says nothing about it. New controls build their own subject; resolving an item
|
|
131
161
|
means re-reading the controls for ones that have gone silent.
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
162
|
+
- **The silent log.** Absence of activity is not evidence of non-use. Before retiring an
|
|
163
|
+
identity, key, or endpoint on a quiet log, show that the queried field is the one recording the
|
|
164
|
+
subject — delegated actions are attributed to the caller, with the target in another field, so
|
|
165
|
+
the wrong field returns clean silence — and that no live binding still references it, since a
|
|
166
|
+
reference proves use while emitting no traffic. Only a demonstrated "this use would have been
|
|
167
|
+
logged" evidences non-use.
|
|
168
|
+
- **The population that shrank.** A gate scanning an explicit list of files sees only "listed but
|
|
169
|
+
empty": a subject that migrates to an unlisted surface leaves the scanned set smaller yet
|
|
170
|
+
non-empty, so the empty-subject guard never fires while coverage erodes. Retarget the gate in
|
|
171
|
+
the same change that moves the code, and pin a floor so any decrease fails loudly — proved by
|
|
172
|
+
moving one subject out. A floor is a ratchet: never lower it to pass a run.
|
|
173
|
+
- **The mutant that never ran.** A mutation verdict counts only if the mutant is valid: it
|
|
174
|
+
compiled, sits on a path the exercised test traverses, and changes the guarded behavior, not
|
|
175
|
+
healed downstream or coinciding with a default. The runner must report build failure,
|
|
176
|
+
unreachable, and equivalent distinctly from KILLED and SURVIVED, and halt on a moved anchor.
|
|
177
|
+
More tests red than the mutation should touch indicts it; classify a survivor (rebuild,
|
|
178
|
+
discard, genuine gap) before writing a test.
|
|
179
|
+
- **The probe that measured the original.** A copied script that derives its root or targets from
|
|
180
|
+
its own location (`$0`, `BASH_SOURCE`, a `cd` to its parent) scans the original tree, not the
|
|
181
|
+
copy, so its verdict says nothing about the mutation you planted. Pin the subject in the copy,
|
|
182
|
+
copy the whole tree, or plant in place with a snapshot-restore. The tell is a result identical
|
|
183
|
+
to the unmutated run; a script taking its subject as an argument is safe to copy.
|
|
184
|
+
|
|
185
|
+
The discipline that covers every shape above: after adding a check, revert the fix it guards and
|
|
186
|
+
watch the check fail. A control that survives a faithful revert was never testing the thing it
|
|
187
|
+
names. Prove the plant landed: assert the altered input differs from the original and that the
|
|
188
|
+
control's cases reach the mutated branch before asserting rejection. Construct corruption
|
|
189
|
+
deterministically: scanning for a flip site yields no-op mutations reporting a rejection nothing
|
|
190
|
+
exercised.
|
|
191
|
+
|
|
192
|
+
## Before blaming code for a metric change
|
|
193
|
+
|
|
194
|
+
When a live metric collapses or crosses a pre-declared threshold, localize the change in time
|
|
195
|
+
before diagnosing the feature: confirm every inbound source is alive, compare only
|
|
196
|
+
contemporaneous cohorts, never re-processed rows, then bracket the transition to the finest unit
|
|
197
|
+
available and read the deploy log around that instant.
|
|
198
|
+
|
|
199
|
+
A change landing seconds from the last-good point is the prime suspect; a step in a window with
|
|
200
|
+
no deploys is an input-population shift, fixed by scoping the measured population, not the model
|
|
201
|
+
or the code. Declare the threshold before looking, and confirm a failure from the fleet's vantage
|
|
202
|
+
rather than your own — this does not replace fixing the comparison basis, which comes first.
|
|
135
203
|
|
|
136
204
|
## Keeping E2E honest
|
|
137
205
|
|
package/install.sh
CHANGED
|
@@ -68,6 +68,21 @@ PRIOR_MANIFEST="$STATE_DIR/manifest.prev.txt"
|
|
|
68
68
|
ZSHRC="${ZDOTDIR:-$HOME}/.zshrc"
|
|
69
69
|
ZSH_HOOK='[ -r "$HOME/.config/agent-launch/shell.zsh" ] && source "$HOME/.config/agent-launch/shell.zsh"'
|
|
70
70
|
HOOK_MARK='agent-launch/shell.zsh'
|
|
71
|
+
# .zshrc is the only file this installer WRITES, but it is not the only file zsh reads, and a
|
|
72
|
+
# user who moves the line to .zshenv (where non-interactive shells see it too) had every
|
|
73
|
+
# question about it answered wrongly: `status` said "zsh hook absent" while the integration was
|
|
74
|
+
# demonstrably running, `uninstall` said there was nothing to remove, and the next `install`
|
|
75
|
+
# appended a second copy to .zshrc. Detection therefore looks everywhere zsh would; ownership
|
|
76
|
+
# stays with .zshrc alone.
|
|
77
|
+
ZSH_HOOK_FILES="${ZDOTDIR:-$HOME}/.zshrc ${ZDOTDIR:-$HOME}/.zshenv ${ZDOTDIR:-$HOME}/.zprofile"
|
|
78
|
+
|
|
79
|
+
# Echo every startup file that carries the hook line, one per line, or nothing.
|
|
80
|
+
zsh_hook_locations() {
|
|
81
|
+
local f
|
|
82
|
+
for f in $ZSH_HOOK_FILES; do
|
|
83
|
+
if [ -f "$f" ] && grep -qF "$HOOK_MARK" "$f"; then printf '%s\n' "$f"; fi
|
|
84
|
+
done
|
|
85
|
+
}
|
|
71
86
|
|
|
72
87
|
DRY_RUN=0
|
|
73
88
|
BACKUP_DIR=""
|
|
@@ -390,8 +405,11 @@ assemble_corpus() {
|
|
|
390
405
|
}
|
|
391
406
|
|
|
392
407
|
add_zsh_hook() {
|
|
393
|
-
|
|
394
|
-
|
|
408
|
+
local found
|
|
409
|
+
found="$(zsh_hook_locations)"
|
|
410
|
+
if [ -n "$found" ]; then
|
|
411
|
+
# Appending beside an existing copy would source the integration twice per shell.
|
|
412
|
+
info "zsh hook present $(printf '%s' "$found" | tr '\n' ' ')"
|
|
395
413
|
return
|
|
396
414
|
fi
|
|
397
415
|
if [ "$DRY_RUN" = 1 ]; then info "[dry-run] append zsh hook to $ZSHRC"; return; fi
|
|
@@ -689,8 +707,13 @@ PY
|
|
|
689
707
|
}
|
|
690
708
|
|
|
691
709
|
remove_zsh_hook() {
|
|
710
|
+
local other
|
|
711
|
+
# Named, never edited: a line this installer did not write is the user's, and uninstall
|
|
712
|
+
# removing it would be editing a file it does not own.
|
|
713
|
+
other="$(zsh_hook_locations | grep -vF "$ZSHRC" || true)"
|
|
714
|
+
[ -n "$other" ] && info "zsh hook also in $(printf '%s' "$other" | tr '\n' ' ') — left in place (not written by this installer)"
|
|
692
715
|
if [ ! -f "$ZSHRC" ] || ! grep -qF "$HOOK_MARK" "$ZSHRC"; then
|
|
693
|
-
info "no zsh hook to remove"
|
|
716
|
+
info "no zsh hook to remove $ZSHRC"
|
|
694
717
|
return
|
|
695
718
|
fi
|
|
696
719
|
if [ "$DRY_RUN" = 1 ]; then info "[dry-run] remove zsh hook from $ZSHRC"; return; fi
|
|
@@ -975,6 +998,13 @@ cmd_verify() {
|
|
|
975
998
|
verify_match "$REPO/launch/agent-launch.py" "$BIN_DIR/agent-launch" || fail=1
|
|
976
999
|
verify_match "$REPO/launch/agent-launch.toml" "$LAUNCH_DIR/profiles.toml" || fail=1
|
|
977
1000
|
verify_match "$REPO/launch/agent-launch.zsh" "$LAUNCH_DIR/shell.zsh" || fail=1
|
|
1001
|
+
# Evidence, never a verdict: a shadow that was repaired is the mechanism working.
|
|
1002
|
+
shell_shadow_summary
|
|
1003
|
+
# Not `whence -v`: a function redefined at preexec carries no "from FILE" annotation,
|
|
1004
|
+
# so in exactly the shell the reassert repaired, whence reads as "not ours". And not a
|
|
1005
|
+
# substring of the body either — a foreign body can quote it. Byte-equality with the
|
|
1006
|
+
# body the interception captured when it was sourced is the same test the reassert runs.
|
|
1007
|
+
info "live check (run in the terminal you use): [[ \"\$functions[claude]\" == \"\$_agent_launch_body[claude]\" ]] && print OURS → expect OURS"
|
|
978
1008
|
# The catalogs were the one deploy-managed artifact nothing verified, which is
|
|
979
1009
|
# the state that renders key names on screen: the deployed launcher keeps no
|
|
980
1010
|
# catalog beside itself, so whatever is here IS the UI text. Byte-identity, the
|
|
@@ -1055,7 +1085,19 @@ PY
|
|
|
1055
1085
|
fi
|
|
1056
1086
|
fi
|
|
1057
1087
|
# Repo-internal mirror parity is a maintainer gate; only meaningful from a clone.
|
|
1058
|
-
|
|
1088
|
+
#
|
|
1089
|
+
# Not from inside the install-scenario suite, though. That suite points REPO at its own
|
|
1090
|
+
# checkout and writes only into a sandbox HOME, so the tree this gate would judge is the
|
|
1091
|
+
# very tree the umbrella that launched the suite already judged — and the suite performs
|
|
1092
|
+
# twelve verifies. Measured 2026-08-31: 3m37s each, 43m22s of a 50m26s commit, 86% of it,
|
|
1093
|
+
# for twelve re-derivations of one unchanged answer that no assertion in
|
|
1094
|
+
# gates/test-install-guides.sh reads. The skip announces itself, because a leg that goes
|
|
1095
|
+
# quiet is how a suite reports clean over something it never ran. The payload and
|
|
1096
|
+
# prompting-target gates above are deliberately NOT skipped: at 0.16s together they buy
|
|
1097
|
+
# back no time, and running them keeps proving that verify still wires its gates up.
|
|
1098
|
+
if [ "${AGENT_BIOS_IN_INSTALL_TEST:-0}" = 1 ]; then
|
|
1099
|
+
info "SKIP: repo mirror parity — the umbrella running this suite already judged this tree"
|
|
1100
|
+
elif [ -d "$REPO/ko" ] && [ -x "$REPO/gates/check-parity.sh" ]; then
|
|
1059
1101
|
# Output kept, not discarded — the third place in this repo where a gate's own
|
|
1060
1102
|
# explanation went to /dev/null and left "it failed" as the entire report. The umbrella
|
|
1061
1103
|
# and the install-scenario harness each learned this after a failure cost an eleven-
|
|
@@ -1273,6 +1315,9 @@ archive_and_purge() {
|
|
|
1273
1315
|
# the file keeps the thing that names it. Everything else goes either way.
|
|
1274
1316
|
if [ "${UNBACKED:-0}" -eq 0 ]; then
|
|
1275
1317
|
rm -rf "$STATE_DIR"
|
|
1318
|
+
else
|
|
1319
|
+
# The dir survives for the manifest's sake; the shell-shadow evidence is not that.
|
|
1320
|
+
rm -f "$STATE_DIR/shell-shadow.log"
|
|
1276
1321
|
fi
|
|
1277
1322
|
rm -rf "$HOME/.cache/agent-launch" \
|
|
1278
1323
|
"${AGENT_LAUNCH_VENV:-$HOME/.local/share/agent-launch}"
|
|
@@ -1528,7 +1573,44 @@ cmd_status() {
|
|
|
1528
1573
|
"$LAUNCH_DIR/profiles.toml" "$LAUNCH_DIR/shell.zsh"; do
|
|
1529
1574
|
if [ -e "$p" ]; then info "present $p"; else info "MISSING $p"; fi
|
|
1530
1575
|
done
|
|
1531
|
-
|
|
1576
|
+
zsh_found="$(zsh_hook_locations)"
|
|
1577
|
+
if [ -z "$zsh_found" ]; then
|
|
1578
|
+
info "zsh hook absent"
|
|
1579
|
+
elif printf '%s\n' "$zsh_found" | grep -qxF "$ZSHRC"; then
|
|
1580
|
+
info "zsh hook present $(printf '%s' "$zsh_found" | tr '\n' ' ')"
|
|
1581
|
+
else
|
|
1582
|
+
# Present and working, but somewhere this installer will neither update nor remove.
|
|
1583
|
+
info "zsh hook present $(printf '%s' "$zsh_found" | tr '\n' ' ') (unmanaged location: not $ZSHRC)"
|
|
1584
|
+
fi
|
|
1585
|
+
shell_shadow_summary
|
|
1586
|
+
}
|
|
1587
|
+
|
|
1588
|
+
# The interception in launch/agent-launch.zsh reasserts `claude`/`codex` at preexec when a
|
|
1589
|
+
# tool redefined them after the rc files (cmux does, on its first precmd) and appends one
|
|
1590
|
+
# line per shadowed host per shell to $STATE_DIR/shell-shadow.log. That log is the only
|
|
1591
|
+
# evidence of shadowing status/verify can read: a child shell cannot reproduce the
|
|
1592
|
+
# terminal's own bootstrap, so a canary run from here would report PASS while the live
|
|
1593
|
+
# shell is shadowed — which is why there is none.
|
|
1594
|
+
shell_shadow_summary() {
|
|
1595
|
+
local f="$STATE_DIR/shell-shadow.log" n last
|
|
1596
|
+
# A dir that cannot be written records nothing, and "nothing" must not read as "clean" —
|
|
1597
|
+
# so this branch is exclusive: no "no shadowing observed" beside it.
|
|
1598
|
+
if [ -L "$f" ] || { [ -e "$f" ] && [ ! -f "$f" ]; }; then
|
|
1599
|
+
info "shell interception: evidence log is not a regular file ($f) — nothing is recorded there, so absence here is not evidence"
|
|
1600
|
+
elif { [ -d "$STATE_DIR" ] && [ ! -w "$STATE_DIR" ]; } || { [ -e "$f" ] && [ ! -w "$f" ]; }; then
|
|
1601
|
+
info "shell interception: evidence log not writable ($f) — a shadowing cannot be recorded, so absence here is not evidence"
|
|
1602
|
+
elif [ -s "$f" ]; then
|
|
1603
|
+
# One line is one host's FIRST observation in one shell, never a count of repairs: a
|
|
1604
|
+
# tool that redefines on every prompt is repaired on every command and logged once.
|
|
1605
|
+
n=$(wc -l <"$f" | tr -d ' ')
|
|
1606
|
+
# Control bytes are stripped again on display: the log is written by a shell the
|
|
1607
|
+
# user does not fully own, and a terminal-control sequence in a "foreign body" field
|
|
1608
|
+
# could repaint this very line.
|
|
1609
|
+
last=$(tail -n 1 "$f" | tr -d '\000-\010\013-\037\177')
|
|
1610
|
+
info "shell interception: shadowing observed $n time(s) (first observation per host per shell), last $(printf '%s' "$last" | cut -f1) — $(printf '%s' "$last" | cut -f2) was shadowed by: $(printf '%s' "$last" | cut -f4-)"
|
|
1611
|
+
else
|
|
1612
|
+
info "shell interception: no shadowing observed"
|
|
1613
|
+
fi
|
|
1532
1614
|
}
|
|
1533
1615
|
|
|
1534
1616
|
cmd_update() {
|