@tech-leads-club/harness-toolkit 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +95 -0
- package/LICENSE +84 -0
- package/NOTICE +9 -0
- package/README.md +480 -0
- package/bin/tlc +10 -0
- package/bin/tlc-build +80 -0
- package/bin/tlc-cli.ts +1509 -0
- package/bin/tlc-exec +10 -0
- package/bin/tlc-exec.cmd +4 -0
- package/bin/tlc-exec.d.mts +60 -0
- package/bin/tlc-exec.mjs +248 -0
- package/bin/tlc.cmd +4 -0
- package/bin/tlc.mjs +7 -0
- package/bin/write-user-hooks.d.mts +40 -0
- package/bin/write-user-hooks.mjs +123 -0
- package/capabilities/catalog.json +418 -0
- package/config.example.json +51 -0
- package/dist/compact-before.mjs +7839 -0
- package/dist/doctor.mjs +8263 -0
- package/dist/help-topic.mjs +82 -0
- package/dist/init-project.mjs +993 -0
- package/dist/install-runtime.mjs +969 -0
- package/dist/lessons-cli.mjs +7258 -0
- package/dist/obs-cli.mjs +7025 -0
- package/dist/price-lookup.mjs +221 -0
- package/dist/prompt-submit.mjs +7860 -0
- package/dist/refresh-model-prices.mjs +158 -0
- package/dist/response-after.mjs +7879 -0
- package/dist/run.mjs +7830 -0
- package/dist/session-end.mjs +7937 -0
- package/dist/session-start.mjs +8056 -0
- package/dist/shim.mjs +52 -0
- package/dist/stop.mjs +8494 -0
- package/dist/subagent-start.mjs +7861 -0
- package/dist/subagent-stop.mjs +7851 -0
- package/dist/support.mjs +7045 -0
- package/dist/tlc-cli.mjs +8063 -0
- package/dist/tool-after.mjs +8113 -0
- package/dist/tool-before.mjs +7985 -0
- package/dist/tool-failure.mjs +7850 -0
- package/dist/uninstall-runtime.mjs +1307 -0
- package/docs/architecture.md +270 -0
- package/docs/concepts.md +562 -0
- package/docs/coverage.md +55 -0
- package/docs/decisions/ad-001.md +32 -0
- package/docs/decisions/ad-002.md +51 -0
- package/docs/decisions/ad-003.md +30 -0
- package/docs/decisions/ad-004.md +37 -0
- package/docs/decisions/ad-005.md +36 -0
- package/docs/decisions/ad-006.md +49 -0
- package/docs/decisions/ad-007.md +36 -0
- package/docs/decisions/ad-008.md +54 -0
- package/docs/decisions/ad-009.md +61 -0
- package/docs/decisions/ad-010.md +45 -0
- package/docs/decisions/ad-011.md +59 -0
- package/docs/decisions/ad-012.md +71 -0
- package/docs/decisions/ad-013.md +87 -0
- package/docs/decisions/ad-014.md +56 -0
- package/docs/decisions/ad-015.md +33 -0
- package/docs/decisions/ad-016.md +98 -0
- package/docs/decisions/ad-017.md +65 -0
- package/docs/decisions/ad-018.md +77 -0
- package/docs/decisions/ad-019.md +75 -0
- package/docs/decisions/ad-020.md +88 -0
- package/docs/decisions/ad-021.md +57 -0
- package/docs/decisions/ad-022.md +120 -0
- package/docs/decisions/ad-023.md +87 -0
- package/docs/decisions/ad-024.md +85 -0
- package/docs/decisions/ad-025.md +92 -0
- package/docs/decisions/ad-026.md +131 -0
- package/docs/decisions/ad-027.md +90 -0
- package/docs/decisions/ad-028.md +90 -0
- package/docs/decisions/ad-029.md +73 -0
- package/docs/decisions/ad-030.md +97 -0
- package/docs/decisions/ad-031.md +94 -0
- package/docs/decisions/ad-032.md +83 -0
- package/docs/decisions/ad-033.md +89 -0
- package/docs/decisions/ad-034.md +86 -0
- package/docs/decisions/ad-035.md +86 -0
- package/docs/decisions/ad-036.md +68 -0
- package/docs/decisions/ad-037.md +47 -0
- package/docs/decisions/ad-038.md +52 -0
- package/docs/decisions/ad-039.md +69 -0
- package/docs/decisions/ad-040.md +89 -0
- package/docs/decisions/ad-041.md +98 -0
- package/docs/decisions/ad-042.md +82 -0
- package/docs/decisions/ad-043.md +79 -0
- package/docs/decisions/ad-044.md +61 -0
- package/docs/decisions/ad-045.md +94 -0
- package/docs/decisions/ad-046.md +111 -0
- package/docs/decisions/ad-047.md +96 -0
- package/docs/decisions/ad-048.md +85 -0
- package/docs/decisions/ad-049.md +66 -0
- package/docs/decisions/ad-050.md +94 -0
- package/docs/decisions/ad-051.md +69 -0
- package/docs/decisions/ad-052.md +69 -0
- package/docs/decisions/ad-053.md +78 -0
- package/docs/decisions/ad-054.md +98 -0
- package/docs/decisions/ad-055.md +74 -0
- package/docs/decisions/ad-056.md +85 -0
- package/docs/decisions/ad-057.md +68 -0
- package/docs/decisions/ad-058.md +97 -0
- package/docs/decisions/ad-059.md +82 -0
- package/docs/decisions/ad-060.md +75 -0
- package/docs/decisions/ad-061.md +68 -0
- package/docs/decisions/ad-062.md +72 -0
- package/docs/decisions/ad-063.md +84 -0
- package/docs/decisions/ad-064.md +79 -0
- package/docs/decisions/ad-065.md +81 -0
- package/docs/decisions/ad-066.md +111 -0
- package/docs/decisions/ad-067.md +64 -0
- package/docs/decisions/ad-068.md +79 -0
- package/docs/decisions/ad-069.md +74 -0
- package/docs/decisions/ad-070.md +86 -0
- package/docs/decisions/ad-071.md +93 -0
- package/docs/decisions/ad-072.md +82 -0
- package/docs/decisions/ad-073.md +102 -0
- package/docs/decisions/ad-074.md +91 -0
- package/docs/decisions/ad-075.md +79 -0
- package/docs/decisions/ad-076.md +102 -0
- package/docs/decisions/ad-077.md +94 -0
- package/docs/decisions/ad-078.md +84 -0
- package/docs/decisions/ad-079.md +73 -0
- package/docs/decisions/ad-080.md +86 -0
- package/docs/decisions/ad-081.md +70 -0
- package/docs/decisions/ad-082.md +79 -0
- package/docs/decisions/ad-083.md +88 -0
- package/docs/decisions/index.md +111 -0
- package/docs/diagnose.md +248 -0
- package/docs/index.md +52 -0
- package/docs/init.md +101 -0
- package/docs/lessons.md +294 -0
- package/docs/log.md +139 -0
- package/docs/measure.md +178 -0
- package/docs/providers/claude-code.md +118 -0
- package/docs/providers/cursor.md +110 -0
- package/docs/providers/index.md +71 -0
- package/docs/troubleshooting.md +138 -0
- package/model-aliases.json +12 -0
- package/model-prices.cursor.json +410 -0
- package/model-prices.json +1 -0
- package/package.json +57 -0
- package/skills/harness-init/SKILL.md +241 -0
- package/skills/harness-init/evals/trigger_evals.json +86 -0
- package/skills/harness-init/references/capabilities.md +113 -0
- package/src/contracts/capabilities.ts +30 -0
- package/src/contracts/decision.ts +21 -0
- package/src/contracts/effort.ts +15 -0
- package/src/contracts/harness-event.ts +87 -0
- package/src/contracts/index.ts +6 -0
- package/src/contracts/policy-defaults.ts +14 -0
- package/src/contracts/wiring.ts +20 -0
- package/src/core/attest/attest.service.ts +0 -0
- package/src/core/capability/capability.service.ts +62 -0
- package/src/core/capability/capability.store.ts +51 -0
- package/src/core/capability/capability.types.ts +56 -0
- package/src/core/comment-policy/comment-policy.service.ts +307 -0
- package/src/core/comment-policy/comment-policy.types.ts +6 -0
- package/src/core/comment-policy/comment-resolvability.ts +134 -0
- package/src/core/comment-policy/comment-syntax.catalog.ts +310 -0
- package/src/core/comment-policy/comment-syntax.store.ts +65 -0
- package/src/core/comment-policy/comment-syntax.types.ts +24 -0
- package/src/core/core.facade.ts +461 -0
- package/src/core/duplication/duplication.service.ts +219 -0
- package/src/core/floor/floor.catalog.ts +54 -0
- package/src/core/floor/floor.paths.ts +90 -0
- package/src/core/floor/floor.policy-surface.ts +311 -0
- package/src/core/floor/floor.service.ts +306 -0
- package/src/core/floor/floor.tokenize.ts +179 -0
- package/src/core/floor/floor.verb.ts +28 -0
- package/src/core/floor/index.ts +1 -0
- package/src/core/gate/gate.artifact.ts +157 -0
- package/src/core/gate/gate.command.ts +130 -0
- package/src/core/gate/gate.findings.ts +201 -0
- package/src/core/gate/gate.inputs.ts +0 -0
- package/src/core/gate/gate.lock.ts +283 -0
- package/src/core/gate/gate.service.ts +28 -0
- package/src/core/gate/gate.types.ts +67 -0
- package/src/core/handoff/handoff.service.ts +43 -0
- package/src/core/handoff/handoff.store.ts +59 -0
- package/src/core/handoff/handoff.types.ts +89 -0
- package/src/core/index.ts +52 -0
- package/src/core/integrity/state-seal.ts +81 -0
- package/src/core/lesson/lesson.authored.ts +96 -0
- package/src/core/lesson/lesson.credit.ts +51 -0
- package/src/core/lesson/lesson.garden.ts +283 -0
- package/src/core/lesson/lesson.link.ts +93 -0
- package/src/core/lesson/lesson.score.ts +61 -0
- package/src/core/lesson/lesson.select.ts +209 -0
- package/src/core/lesson/lesson.service.ts +110 -0
- package/src/core/lesson/lesson.store.ts +317 -0
- package/src/core/lesson/lesson.sync.ts +70 -0
- package/src/core/lesson/lesson.types.ts +92 -0
- package/src/core/lesson/lesson.validity.ts +44 -0
- package/src/core/observability/observability.report.ts +326 -0
- package/src/core/observability/observability.service.ts +328 -0
- package/src/core/observability/observability.store.ts +269 -0
- package/src/core/observability/observability.types.ts +193 -0
- package/src/core/observability/observability.why.ts +206 -0
- package/src/core/observe/observe.service.ts +76 -0
- package/src/core/plan/plan.detect.ts +47 -0
- package/src/core/plan/plan.service.ts +81 -0
- package/src/core/plan/plan.types.ts +25 -0
- package/src/core/policy/policy.defaults.ts +103 -0
- package/src/core/policy/policy.guard.ts +34 -0
- package/src/core/policy/policy.integrity.ts +238 -0
- package/src/core/policy/policy.loader.ts +118 -0
- package/src/core/policy/policy.operator.ts +82 -0
- package/src/core/policy/policy.posture.ts +79 -0
- package/src/core/policy/policy.rails.ts +45 -0
- package/src/core/policy/policy.types.ts +157 -0
- package/src/core/presence/presence.service.ts +111 -0
- package/src/core/presence/presence.store.ts +59 -0
- package/src/core/presence/presence.types.ts +9 -0
- package/src/core/release/release.decisions.ts +147 -0
- package/src/core/release/release.seen.ts +38 -0
- package/src/core/shell-policy/shell-policy.service.ts +215 -0
- package/src/core/shell-policy/shell-policy.stall.ts +57 -0
- package/src/core/shell-policy/shell-policy.types.ts +26 -0
- package/src/core/ship/ship.ledger.ts +76 -0
- package/src/core/ship/ship.service.ts +137 -0
- package/src/core/ship/ship.types.ts +19 -0
- package/src/core/stagnation/stagnation.resolution.ts +87 -0
- package/src/core/stagnation/stagnation.service.ts +20 -0
- package/src/core/stagnation/stagnation.store.ts +49 -0
- package/src/core/stagnation/stagnation.types.ts +12 -0
- package/src/core/subagent-policy/subagent-policy.parent-model.ts +202 -0
- package/src/core/subagent-policy/subagent-policy.service.ts +139 -0
- package/src/core/subagent-policy/subagent-policy.types.ts +22 -0
- package/src/core/supply-chain/supply-chain.catalog.ts +51 -0
- package/src/core/supply-chain/supply-chain.service.ts +217 -0
- package/src/core/turn/turn.activity.ts +94 -0
- package/src/core/turn/turn.autopilot.ts +131 -0
- package/src/core/turn/turn.failure-signals.ts +193 -0
- package/src/core/turn/turn.loop-counter.ts +71 -0
- package/src/core/turn/turn.types.ts +14 -0
- package/src/core/untrusted/untrusted.detect.ts +53 -0
- package/src/core/untrusted/untrusted.recall.ts +87 -0
- package/src/core/untrusted/untrusted.service.ts +129 -0
- package/src/core/untrusted/untrusted.store.ts +75 -0
- package/src/core/untrusted/untrusted.types.ts +42 -0
- package/src/entrypoints/compact-before.ts +14 -0
- package/src/entrypoints/prompt-submit.ts +25 -0
- package/src/entrypoints/response-after.ts +60 -0
- package/src/entrypoints/run.ts +179 -0
- package/src/entrypoints/session-end.ts +58 -0
- package/src/entrypoints/session-start.ts +172 -0
- package/src/entrypoints/shim.ts +46 -0
- package/src/entrypoints/stop.ts +876 -0
- package/src/entrypoints/subagent-start.ts +16 -0
- package/src/entrypoints/subagent-stop.ts +32 -0
- package/src/entrypoints/support.ts +182 -0
- package/src/entrypoints/tool-after.ts +111 -0
- package/src/entrypoints/tool-before.ts +186 -0
- package/src/entrypoints/tool-failure.ts +26 -0
- package/src/platform/backoff.ts +49 -0
- package/src/platform/cli-output.ts +30 -0
- package/src/platform/env-scope.ts +22 -0
- package/src/platform/fs-atomic.ts +142 -0
- package/src/platform/fs-jsonl.ts +24 -0
- package/src/platform/git.ts +187 -0
- package/src/platform/paths.ts +80 -0
- package/src/platform/pricing.ts +229 -0
- package/src/platform/process.ts +64 -0
- package/src/platform/sanitize.ts +23 -0
- package/src/platform/screen.ts +46 -0
- package/src/platform/style.ts +103 -0
- package/src/providers/claude/claude.capabilities.ts +25 -0
- package/src/providers/claude/claude.detect.ts +14 -0
- package/src/providers/claude/claude.inbound.ts +252 -0
- package/src/providers/claude/claude.lessons-view.ts +33 -0
- package/src/providers/claude/claude.outbound.ts +65 -0
- package/src/providers/claude/claude.policy-defaults.ts +9 -0
- package/src/providers/claude/claude.transcript.ts +62 -0
- package/src/providers/claude/claude.wiring.ts +245 -0
- package/src/providers/claude/index.ts +17 -0
- package/src/providers/cursor/cursor.capabilities.ts +28 -0
- package/src/providers/cursor/cursor.detect.ts +14 -0
- package/src/providers/cursor/cursor.inbound.ts +203 -0
- package/src/providers/cursor/cursor.lessons-view.ts +30 -0
- package/src/providers/cursor/cursor.outbound.ts +43 -0
- package/src/providers/cursor/cursor.policy-defaults.ts +9 -0
- package/src/providers/cursor/cursor.wiring.ts +214 -0
- package/src/providers/cursor/index.ts +17 -0
- package/src/providers/index.ts +9 -0
- package/src/providers/provider.degrade.ts +152 -0
- package/src/providers/provider.port.ts +20 -0
- package/src/providers/provider.registry.ts +28 -0
- package/tools/doctor.ts +631 -0
- package/tools/help-topic.ts +74 -0
- package/tools/init-project.ts +277 -0
- package/tools/install-runtime.ts +164 -0
- package/tools/lessons-cli.ts +474 -0
- package/tools/obs-cli.ts +175 -0
- package/tools/price-lookup.ts +57 -0
- package/tools/refresh-model-prices.ts +210 -0
- package/tools/test-env.mjs +28 -0
- package/tools/test-env.names.d.mts +1 -0
- package/tools/test-env.names.mjs +14 -0
- package/tools/uninstall-runtime.ts +463 -0
|
@@ -0,0 +1,120 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: Decision
|
|
3
|
+
title: "AD-022 — The policy surface is a floor rule, detection sits behind interception, and the operator/agent line is structural"
|
|
4
|
+
description: "The guard denied agent writes to the policy file with the words 'a gate an agent can switch off is not a gate' and enforced that for five tool names only, so one shell line went around it. The invariant moves to the floor, a per-session hash covers what shell parsing cannot see, and the missing CLI route that produced the bypass is opened."
|
|
5
|
+
tags: [decision, floor, policy, gate, cli]
|
|
6
|
+
timestamp: "2026-07-31"
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# AD-022 — The policy surface is a floor rule, detection sits behind interception, and the operator/agent line is structural
|
|
10
|
+
|
|
11
|
+
- **status**: active
|
|
12
|
+
- **date**: 2026-07-31
|
|
13
|
+
|
|
14
|
+
## Decision
|
|
15
|
+
|
|
16
|
+
1. **The policy surface is a floor rule, `policy-surface-write`.** It covers shell routes to
|
|
17
|
+
`.tlc/harness/config.json`, `flags/` and `state/`, and the mutating `tlc harness` subcommands
|
|
18
|
+
(`pause`, `resume`, `grind`, `mode`, `init`, `gate`). The tool guard stays for tool writes, where it
|
|
19
|
+
produces a better-targeted message, but it is no longer the only thing standing there.
|
|
20
|
+
2. **The surface includes the runtime config and runtime state**, `~/.tlc/harness/config.json` and
|
|
21
|
+
`~/.tlc/harness/state/`, not only the project's. `loadPolicy` merges the runtime config *under* the
|
|
22
|
+
project one, so every field the project leaves unset is decided there — for every repository on the
|
|
23
|
+
machine.
|
|
24
|
+
3. **Detection is an allowlist of proven readers, never an enumeration of writers.** A verb not in the set
|
|
25
|
+
cannot be shown to only read, so it is denied. `awk` and `sort` are excluded despite looking like readers.
|
|
26
|
+
`git` is a reader only for `show`, `diff`, `log`, `status`, `ls-files`, `cat-file` and `blame`.
|
|
27
|
+
4. **The operator/agent distinction is structural.** The floor runs only inside a hook, and a hook exists
|
|
28
|
+
only inside an agent session, so the operator's own terminal never reaches the check and needs no
|
|
29
|
+
exemption. No TTY sniffing and no environment marker decide who is asking.
|
|
30
|
+
5. **A per-session baseline hash detects what parsing cannot intercept.** Every source `loadPolicy` reads is
|
|
31
|
+
hashed at session start; an intra-session change with no `tlc harness` command behind it blocks the next
|
|
32
|
+
tool call and reports the path. The check is unconditional.
|
|
33
|
+
6. **`tlc harness gate test-command | lint-command` is the legitimate route** to `grind.testCommand` and
|
|
34
|
+
`grind.lintCommand`, refusing an empty argv, an unresolvable binary and a non-interactive invocation.
|
|
35
|
+
|
|
36
|
+
## Why
|
|
37
|
+
|
|
38
|
+
The guard's message asserted an invariant the guard did not hold. `guardPolicySurface` was reached only from
|
|
39
|
+
`case "tool.before"` and keyed off `WRITE_TOOLS`, a set of five tool names, so anything that was not one of
|
|
40
|
+
those five bypassed it by construction. An agent replaced `grind.testCommand` with `python3 -c` in the shell,
|
|
41
|
+
unopposed, and was refused minutes later when it attempted the same write through `Edit`. The gate that fires
|
|
42
|
+
second is not a gate.
|
|
43
|
+
|
|
44
|
+
The floor is where the invariant belonged all along: it is evaluated before any policy is loaded, it already
|
|
45
|
+
receives `command`, and its own comment forbids giving it a config parameter. Moving the rule there made the
|
|
46
|
+
message true instead of aspirational.
|
|
47
|
+
|
|
48
|
+
The sanctioned route was the wider hole. `evaluateShellCommand` handles only catastrophic commands and stall
|
|
49
|
+
detection, so `tlc harness pause` — one word, fully supported, writing `flags/skip-verify` — turned every stop
|
|
50
|
+
check off from the agent's shell. `isPolicySurface` already protected `flags/` from tool writes, which shows
|
|
51
|
+
the intent; the CLI wrote them for whoever asked. Closing the file route while leaving that open would have
|
|
52
|
+
moved the bypass rather than fixed it.
|
|
53
|
+
|
|
54
|
+
The third defect was the incentive behind the first. `grind.testCommand` had no CLI subcommand, so the guard
|
|
55
|
+
forbade the edit and offered nothing instead. An agent that needs a field it is not allowed to set will find
|
|
56
|
+
the shell. The route and the refusal had to ship together.
|
|
57
|
+
|
|
58
|
+
## Trade-offs
|
|
59
|
+
|
|
60
|
+
Prove-safe misfires toward denial, and four of those misfires were real — each one caught by the gate
|
|
61
|
+
refusing its own author, and each fixed rather than worked around.
|
|
62
|
+
|
|
63
|
+
1. The heredoc check first asked whether the segment's verb was a proven reader, which denied
|
|
64
|
+
`git commit -F -` for a message that merely named the policy path. Writing about this rule tripped it, and
|
|
65
|
+
every commit in this change would have had to route around the gate being added. The question is not
|
|
66
|
+
whether the verb reads but whether it *executes* what it reads, so the check is an interpreter set.
|
|
67
|
+
2. It then attributed a body to the whole command, which denied `cat >> f <<'T' … T; node --test` because a
|
|
68
|
+
different segment ran node. Bodies are now paired with the text before their marker, so the owning verb is
|
|
69
|
+
the one that actually receives them. Found when the gate refused to let its own test file be written.
|
|
70
|
+
3. `printf '{…config.json…}' | node dist/tool-before.mjs` was denied, which blocked the rule's own end-to-end
|
|
71
|
+
verification. `echo` and `printf` cannot modify a file on their own — only a redirect can, and redirects
|
|
72
|
+
are judged separately — so both joined the reader set.
|
|
73
|
+
4. The redirect scanner read a `>` inside a quoted argument as a redirect, so a JSON payload carrying
|
|
74
|
+
`"command":"echo x > cfg"` was refused. No shell performs that redirect: quoting makes the operator
|
|
75
|
+
literal. The tokenizer now reports whether a word *opened* with a quote, and only such words are skipped —
|
|
76
|
+
`>"$f"` begins unquoted and remains a real redirect.
|
|
77
|
+
|
|
78
|
+
Each is a regression test now, with the opposite direction pinned beside it: the interpreter heredoc, the
|
|
79
|
+
cross-segment attribution, `echo '{}' > cfg` and `echo x > "cfg"` all still deny.
|
|
80
|
+
|
|
81
|
+
Reading stays ordinary work. The bootstrap instructs the agent to read `handoff.json`, and inspecting the
|
|
82
|
+
config is normal investigation, so a proven reader on those paths passes. The cost is an allowlist that will
|
|
83
|
+
need a new entry occasionally — three entries already arrived this way — and the alternative was a rule that
|
|
84
|
+
punished the reading it asks for.
|
|
85
|
+
|
|
86
|
+
The project root is excluded from the containment check by name. It contains the surface, so counting it
|
|
87
|
+
would deny `find .` and `grep -r x .` — the most common commands there are. Destruction of the whole project
|
|
88
|
+
remains the concern of `outside-project-destruction` and its siblings.
|
|
89
|
+
|
|
90
|
+
Integrity detection is intra-session only. Flagging every out-of-band change would fire on the operator
|
|
91
|
+
editing a tracked file between sessions, which is their prerogative and the misfire this repository forbids.
|
|
92
|
+
Re-recording is the entire receipt: every mutator refreshes all live baselines, so "a harness command did
|
|
93
|
+
this" and "the baseline matches" are one fact rather than two that can drift, and the CLI refreshes every
|
|
94
|
+
session because it cannot know which are live.
|
|
95
|
+
|
|
96
|
+
The TTY requirement on `gate` is a second layer, not the distinction. Existing mutators keep working
|
|
97
|
+
non-interactively — nothing in CI or the installers calls them, which was verified before the choice.
|
|
98
|
+
|
|
99
|
+
## Not decided here
|
|
100
|
+
|
|
101
|
+
- **Fully dynamic path construction.** `P=$(…); echo x > "$P"` is unreachable by the floor, as are
|
|
102
|
+
`bash script.sh` and compiled binaries. The floor already documents that class as a coverage limit rather
|
|
103
|
+
than evasion; the baseline hash catches the effect after the fact. Detection after the write is inherent,
|
|
104
|
+
and it is why interception exists in front of it.
|
|
105
|
+
- **`rm -rf .` still reaches the state directory.** It is allowed by the existing destruction rules as
|
|
106
|
+
destruction inside the project, and narrowing that is a change to those rules, not to this one. Note this
|
|
107
|
+
is a pre-existing property of `outside-project-destruction`, not something this decision introduced: a
|
|
108
|
+
command that erases the whole repository is a larger problem than the policy files inside it.
|
|
109
|
+
|
|
110
|
+
## Corrected after first writing
|
|
111
|
+
|
|
112
|
+
This decision first left the runtime config as "not decided here", on the reasoning that `isPolicySurface`
|
|
113
|
+
is per-project by design. That was wrong, and adding `echo` and `printf` to the reader set is what exposed it:
|
|
114
|
+
`echo '{}' > ~/.tlc/harness/config.json` resolved outside the project, so the redirect check did not match,
|
|
115
|
+
and the head verb was a proven reader, so nothing else did either. It was allowed, verified against the built
|
|
116
|
+
bundle. The substring fallback caught the *interpreter* spelling of the same write, which is why it read as
|
|
117
|
+
covered.
|
|
118
|
+
|
|
119
|
+
Deferring it would have left the machine-wide half of the surface open while the project half was closed —
|
|
120
|
+
the same defect one directory up, reachable by the reader set this decision had just widened.
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: Decision
|
|
3
|
+
title: "AD-023 — One finding per failure, and lesson relevance is recurrence rather than exposure"
|
|
4
|
+
description: "A single failing test reached an operator as three gaps — the assertion header, the test name and the tally '1 fail' — while a lesson whose cause had been fixed four days earlier outranked everything. Findings are now grouped per failure, and lesson decay reads the field the world writes instead of the field the harness writes by reading."
|
|
5
|
+
tags: [decision, gate, lessons, signal]
|
|
6
|
+
timestamp: "2026-08-03"
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# AD-023 — One finding per failure, and lesson relevance is recurrence rather than exposure
|
|
10
|
+
|
|
11
|
+
- **status**: active
|
|
12
|
+
- **date**: 2026-08-03
|
|
13
|
+
|
|
14
|
+
## Decision
|
|
15
|
+
|
|
16
|
+
1. **A finding corresponds to a failure, not to a matched line.** `extractFindingsFromOutput` classifies each
|
|
17
|
+
line by role — `count`, `test`, `assertion`, `other` — groups an assertion onto the test it explains, and
|
|
18
|
+
deduplicates. A tally is never a finding on its own.
|
|
19
|
+
2. **A tally is a line composed only of tally clauses.** `1 fail — runCascade …` is not a tally, because the
|
|
20
|
+
name is the failure's only identity.
|
|
21
|
+
3. **A bare `error:` is not an assertion.** Assertion vocabulary (`expect(`, `toEqual`, `AssertionError`,
|
|
22
|
+
`Expected`, `received`) decides detail; `error: cannot find module` stays a failure of its own.
|
|
23
|
+
4. **An unrecognised runner degrades to one finding per line** — today's behaviour minus the tally. Unknown
|
|
24
|
+
formats become noisier, never silent.
|
|
25
|
+
5. **The cap discloses what it omitted.** The disclosure occupies one slot, so the omitted count is always two
|
|
26
|
+
or more.
|
|
27
|
+
6. **Lesson decay and pruning measure `lastSeenAt`, never `lastAccessedAt`.** `lastSeenAt` moves only when the
|
|
28
|
+
failure signature recurs; `lastAccessedAt` moves when a lesson is injected and is telemetry only.
|
|
29
|
+
7. **A lesson does not restate the gate's own suggestion.** `recordLessonFromFailure` no longer takes
|
|
30
|
+
`suggestion`; the lesson carries the signature and the retry guidance.
|
|
31
|
+
8. **The garden retires a `verification` lesson whose signal is an unresolved gate command**, because AD-021
|
|
32
|
+
made that class classify as `config` and it can no longer legitimately recur.
|
|
33
|
+
|
|
34
|
+
## Why
|
|
35
|
+
|
|
36
|
+
Both halves were observed together on one real gate run: `28 pass, 1 fail` in a project's `bun test`, reported
|
|
37
|
+
to the agent as three gaps plus a lesson from a dead cause.
|
|
38
|
+
|
|
39
|
+
**The findings.** `extractFindingsFromOutput` returned one `GateFinding` per line matching `FAIL_HINT`. For
|
|
40
|
+
that run it returned three: `error: expect(received).toEqual(expected)`, `(fail) runCascade — métricas …`, and
|
|
41
|
+
`1 fail`. Those are the assertion, the test's identity, and a counter — three views of one failure. This is
|
|
42
|
+
not a display nicety: `gapsFromArtifact` maps findings 1:1 to gaps, the autopilot prints *"Read the
|
|
43
|
+
PREVIOUS_GAPS list and fix each item explicitly"*, and `mergeGaps` carries the list into the handoff. One
|
|
44
|
+
assertion failure became three tracked problems that persisted across turns, with a counter among them as
|
|
45
|
+
something to go fix.
|
|
46
|
+
|
|
47
|
+
**The lesson.** `decayedConfidence` read `lastAccessedAt || lastSeenAt`, and `touchAccessed` writes
|
|
48
|
+
`lastAccessedAt` for every lesson *selected for injection*. Showing a lesson therefore reset the clock that
|
|
49
|
+
decided whether to keep showing it: injected → touched → decay back to zero → still ranked high → injected
|
|
50
|
+
again. The garden's prune expression read the same field, so it never pruned either. Relevance was being
|
|
51
|
+
measured by "was recently injected", and injecting is what writes that field.
|
|
52
|
+
|
|
53
|
+
That loop is why a lesson from a config fault AD-021 had already fixed was still ranked top four days later,
|
|
54
|
+
with `lastAccessedAt` renewed daily by its own injection. Its `instruction` was also, by construction, the
|
|
55
|
+
gate's own `next_action`: the text was built as `` `${suggestion} Recurrent failure signature…` `` while
|
|
56
|
+
`stop.ts` handed the same `suggestionFor(category, gate)` value to both the plan and the lesson. The
|
|
57
|
+
highest-ranked entry in a budgeted list restated the sentence printed directly above it.
|
|
58
|
+
|
|
59
|
+
## Trade-offs
|
|
60
|
+
|
|
61
|
+
Role classification is a heuristic over text, not a parser. The alternative — requiring each project to emit
|
|
62
|
+
JUnit XML or a JSON reporter — would change every project's configured command, and `reportPath` already
|
|
63
|
+
serves projects that want structured findings. The heuristic is therefore built to fail toward noise: an
|
|
64
|
+
unknown format keeps every matched line, and an assertion with no test line ever seen becomes its own finding
|
|
65
|
+
rather than being dropped with the buffer. Losing a real failure is the only unacceptable outcome on this
|
|
66
|
+
channel.
|
|
67
|
+
|
|
68
|
+
`lastAccessedAt` is kept and still written. Removing it would break existing stores for no reader benefit, and
|
|
69
|
+
"was this lesson ever used" is worth knowing — it just is not evidence that the lesson is still relevant. The
|
|
70
|
+
defect was reading it, not writing it.
|
|
71
|
+
|
|
72
|
+
Stored lessons keep their old prefixed instructions. A migration that rewrites them buys no reader anything
|
|
73
|
+
once the decay clock is honest; they fade on their own.
|
|
74
|
+
|
|
75
|
+
The AD-021 retirement is narrow on purpose. Only that class is provable from what the store retains — the
|
|
76
|
+
store keeps a signal snippet, not raw output and exit codes — so a general "would this classify differently
|
|
77
|
+
today" engine would be guesswork. `exitCode: 0` is passed to `isCommandResolutionFailure` so that only its
|
|
78
|
+
message patterns can decide; the real function returns true for 127, and stored text can contain any number.
|
|
79
|
+
|
|
80
|
+
## Not decided here
|
|
81
|
+
|
|
82
|
+
- **Grouping across the cap.** When more than `FINDINGS_MAX` distinct failures exist, the omitted ones are
|
|
83
|
+
counted, not summarised. Clustering them by file or suite would need more structure than the text gives.
|
|
84
|
+
- **Retiring lessons whose cause was fixed in general.** Beyond the AD-021 signature there is no stored
|
|
85
|
+
evidence to decide it, and inventing one would put guesswork into the channel this decision exists to clean.
|
|
86
|
+
- **`lastSeenAt` on the six built-in core lessons** is the epoch. It never reaches the maths because
|
|
87
|
+
`decayedConfidence` returns early for `source === "core"`, so it was left alone rather than backfilled.
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: Decision
|
|
3
|
+
title: "AD-024 — The gate tells the truth about its own environment, its own lock, and the cause of a failure"
|
|
4
|
+
description: "One stop hook produced three wrong statements at once: 22 tests failed for environmental reasons, the gate could not run because a lock named a dead process, and the plan blamed the only file that had changed. The suite is now hermetic, pid liveness is host-scoped, and the named files come from the failure rather than from the diff."
|
|
5
|
+
tags: [decision, gate, testing, lock, signal]
|
|
6
|
+
timestamp: "2026-08-04"
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# AD-024 — The gate tells the truth about its own environment, its own lock, and the cause of a failure
|
|
10
|
+
|
|
11
|
+
- **status**: active
|
|
12
|
+
- **date**: 2026-08-04
|
|
13
|
+
|
|
14
|
+
## Decision
|
|
15
|
+
|
|
16
|
+
1. **The test suite is hermetic with respect to project-identifying environment.** A setup module loaded via
|
|
17
|
+
`node --import` deletes `CLAUDE_PROJECT_DIR` and `TLC_PROJECT_DIR` before any test file runs, for both
|
|
18
|
+
suites. `TLC_HOME` is deliberately kept: it names which runtime, CI sets it on purpose, and the suite
|
|
19
|
+
exercises it.
|
|
20
|
+
2. **A leak fails one test that names itself**, and the wiring is asserted separately, so dropping the
|
|
21
|
+
`--import` fails rather than silently restoring 22 misleading failures.
|
|
22
|
+
3. **A test that asserts what happens when a variable is absent makes it absent itself**, rather than
|
|
23
|
+
inheriting that from the shell.
|
|
24
|
+
4. **Lock reclaim consults process liveness, scoped to the host that wrote the lock.** `LockBody` gains
|
|
25
|
+
`host`; `process.kill(pid, 0)` decides; `ESRCH` is gone, `EPERM` is alive, and a foreign host, a missing
|
|
26
|
+
host or an unusable pid all fall back to the age rule.
|
|
27
|
+
5. **`describeHolder` applies the same rule as reclaim.** The two must agree or the reclaim path is
|
|
28
|
+
unreachable.
|
|
29
|
+
6. **The files a plan names come from the failure output when it names any.** Only that list may read as an
|
|
30
|
+
instruction; the changed files are labelled as coming from the diff, and neither list is called "focus".
|
|
31
|
+
|
|
32
|
+
## Why
|
|
33
|
+
|
|
34
|
+
A single stop hook made three false statements, and each traced to a different defect.
|
|
35
|
+
|
|
36
|
+
**The suite answered the shell, not its fixtures.** `projectDirFor` prefers `CLAUDE_PROJECT_DIR` over the
|
|
37
|
+
payload's `cwd`, which is correct in production — the variable is the project root, `cwd` can be a
|
|
38
|
+
subdirectory. Inside a Claude Code hook it is always set, so every test that built a fixture in a temp
|
|
39
|
+
directory and passed `cwd` resolved against the real repository. Measured: 947 pass from a shell, 925 pass and
|
|
40
|
+
22 fail with the variable set, reproduced on a clean HEAD. The 22 spanned presence collisions, minEffort
|
|
41
|
+
denial, untrusted-content framing, policy integrity and cost estimation, so the gate read as five product bugs
|
|
42
|
+
rather than one leak. The resolver is untouched; the tests were the defect.
|
|
43
|
+
|
|
44
|
+
**The lock honoured a dead owner.** The gate refused to run because the lock named a session whose pid `ps`
|
|
45
|
+
reported gone. The body carried the pid and nothing consulted the operating system: `isLockStale` decides by
|
|
46
|
+
file age alone, against a thirty-minute window. The half of this that actually unblocked the gate was
|
|
47
|
+
`describeHolder`, which already carried a comment saying *"an abandoned lock must not read as held … callers
|
|
48
|
+
short-circuit on this answer before withGateLock can reach stealIfStale"* — and implemented that for staleness
|
|
49
|
+
only. A fresh lock with a dead owner was reported as held, the caller short-circuited, and no change to
|
|
50
|
+
reclaimability could have been reached.
|
|
51
|
+
|
|
52
|
+
**The plan blamed proximity.** `Focus files` was rendered from the changed files, under a category whose steps
|
|
53
|
+
say to fix each item explicitly. It named a new file no test imports while the output named the three real
|
|
54
|
+
ones. The evidence was in the same text and was passed over in favour of the diff. AD-021 recorded this harm
|
|
55
|
+
once already, arriving through a malformed command; this is the same harm arriving through the diff.
|
|
56
|
+
|
|
57
|
+
## Trade-offs
|
|
58
|
+
|
|
59
|
+
Cleaning environment in a setup module is a blunt instrument, and the blunt part is the point: enumerating
|
|
60
|
+
which tests need which variable absent is what produced a suite that passed in one place and failed in
|
|
61
|
+
another. The cleaned set stays minimal — only variables that name *which project* — and `TLC_HOME` is excluded
|
|
62
|
+
by name and by test, because deleting it would hide a different class of bug than the one this module exists
|
|
63
|
+
for.
|
|
64
|
+
|
|
65
|
+
Splitting the module in two (`test-env.mjs` performs the deletion, `test-env.names.mjs` holds the list) looks
|
|
66
|
+
redundant and is not. The first draft had the guard import the list from the module that deletes, so importing
|
|
67
|
+
it ran the cleanup and the assertion could never fail — verified: it passed with the variable set and the
|
|
68
|
+
`--import` absent.
|
|
69
|
+
|
|
70
|
+
Host-scoped liveness is narrower than checking every pid, deliberately. A pid means nothing on another
|
|
71
|
+
machine, and reclaiming a lock whose owner is alive elsewhere would let two gates run at once, which is the
|
|
72
|
+
thing the lock exists to prevent. `GATE_LOCK_STALE_MS` is unchanged for the same reason: age remains the answer
|
|
73
|
+
whenever liveness cannot be established.
|
|
74
|
+
|
|
75
|
+
File extraction is a heuristic over text and is scoped to paths the runner itself printed. It does not attempt
|
|
76
|
+
to infer a culprit from a stack, a symbol or a diff — inferring is what this decision exists to stop.
|
|
77
|
+
|
|
78
|
+
## Not decided here
|
|
79
|
+
|
|
80
|
+
- **Cursor's project-identifying variables**, if any exist, are not in the cleaned set. None were found; the
|
|
81
|
+
set grows when one is.
|
|
82
|
+
- **Orphan lock sweeping** on a schedule. Reclaim is checked when the lock is contended, not proactively;
|
|
83
|
+
presence has `sweepStale` and the lock does not, which is a difference this decision leaves standing.
|
|
84
|
+
- **Whether the changed-file list belongs in the plan at all.** It is kept, relabelled. Removing it would lose
|
|
85
|
+
context that is occasionally the right lead.
|
|
@@ -0,0 +1,92 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: Decision
|
|
3
|
+
title: "AD-025 — Posture governs surfacing only, and each posture has exactly one name"
|
|
4
|
+
description: "Three postures existed by name and two by behaviour: the deepest one was reachable under a second spelling, forced grind on, narrowed the test gate, and stated a threshold that the invariant block contradicted. Posture now varies one thing — what earns an interruption — and one word names each level everywhere."
|
|
5
|
+
tags: [decision, policy, posture, cli, vocabulary]
|
|
6
|
+
timestamp: "2026-08-04"
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# AD-025 — Posture governs surfacing only, and each posture has exactly one name
|
|
10
|
+
|
|
11
|
+
- **status**: active
|
|
12
|
+
- **date**: 2026-08-04
|
|
13
|
+
|
|
14
|
+
## Decision
|
|
15
|
+
|
|
16
|
+
1. **One word per posture: `paired`, `solo`, `focus`.** It is the word the operator types, the word the config
|
|
17
|
+
field holds, the name of the flag file, and the word `status` prints. No alias, no translation.
|
|
18
|
+
2. **`resolvePosture` is the only thing that answers which posture is in effect**, and it answers *where from* in
|
|
19
|
+
the same call. The loader, `status` and `doctor` read that one result.
|
|
20
|
+
3. **A value that is not a posture is refused, not absorbed.** The CLI rejects it with the three accepted words;
|
|
21
|
+
a config value falls back to `solo` with origin `fallback` and the rejected word carried alongside, which
|
|
22
|
+
`status` prints and `doctor` reports as a `warn`.
|
|
23
|
+
4. **Posture changes what the agent surfaces and nothing else.** The evidence bar, the gates and the
|
|
24
|
+
done-criteria are identical at all three, and the harness says so in its own bootstrap.
|
|
25
|
+
5. **The interruption threshold is stated per posture, not once for all three.** `paired` adds a pre-check
|
|
26
|
+
before any sizable move; `solo` names three stops; `focus` names two and hands ambiguity to the agent under a
|
|
27
|
+
stated assumption.
|
|
28
|
+
6. **`paired` is enforced, not only described.** A shell command classified `write` or `network` is asked about
|
|
29
|
+
before it runs. `solo` and `focus` share enforcement, because what separates them is whether *ambiguity*
|
|
30
|
+
reaches the operator — the agent's judgement, not a hook event.
|
|
31
|
+
7. **No posture raises grind and no posture narrows the test gate.** Grind has its own switch, its own flag and
|
|
32
|
+
its own documented trade-off; the test gate runs whenever test files or `codePaths` files changed.
|
|
33
|
+
|
|
34
|
+
## Why
|
|
35
|
+
|
|
36
|
+
The harness advertised three postures and implemented two. Measured on the config this repository ships:
|
|
37
|
+
`"mode": "focus"` — the word the README documents and the CLI accepts — matched no branch in the operator
|
|
38
|
+
bootstrap, because the field stored a second spelling. The result was a policy with **no posture line at all**
|
|
39
|
+
and grind left off, reported by nothing. A vocabulary split between the command and the file it writes is not a
|
|
40
|
+
cosmetic defect; it is a value the operator can express and the system cannot hold.
|
|
41
|
+
|
|
42
|
+
The deepest posture also carried three things that were not surfacing. It forced grind on, so a preference about
|
|
43
|
+
interruptions silently switched on a capability with its own switch — the AD-020 defect class, arriving from the
|
|
44
|
+
policy loader instead of from `status`. It narrowed the test gate to *test files changed*, which skips exactly
|
|
45
|
+
the change that most needs testing: a code change with no test in the diff. And its own line told the agent not
|
|
46
|
+
to ask about reversible work while the invariant block, which applies to every posture, demanded escalating
|
|
47
|
+
ambiguity. A rule that varies cannot sit in the block that does not vary, so the threshold moved out of `BASE`
|
|
48
|
+
and is now stated once per posture.
|
|
49
|
+
|
|
50
|
+
`paired` was the posture that promised the most and enforced the least: it said it would check in before a
|
|
51
|
+
sizable move and then relied entirely on the agent reading its own instructions. `classifyShell` already answers
|
|
52
|
+
"is this a write or a network call" and is already tested, so the promise became a decision at the hook. The
|
|
53
|
+
first draft of this feature recorded that enforcement was impossible here; that reading was wrong and is
|
|
54
|
+
recorded as refuted.
|
|
55
|
+
|
|
56
|
+
## Trade-offs
|
|
57
|
+
|
|
58
|
+
`focus` won as the canonical name over the alternative spelling because it is the operator-facing word and it
|
|
59
|
+
matches the shape of its siblings — three single words for three levels. The earlier draft chose the other way
|
|
60
|
+
on the grounds that the word came from an external methodology the harness was modelled on. That rationale was
|
|
61
|
+
withdrawn: the harness must not know that anything outside it exists, so the only remaining criterion is
|
|
62
|
+
internal coherence.
|
|
63
|
+
|
|
64
|
+
Refusing the second spelling breaks existing configs that carry it, by choice. AD-003 forbids back-compat
|
|
65
|
+
shims, and an alias is precisely the translation layer this decision exists to remove. The cost is one word in
|
|
66
|
+
one file per project, and `doctor` names the file and the fix. Absorbing it silently is what produced the
|
|
67
|
+
original bug.
|
|
68
|
+
|
|
69
|
+
`solo` and `focus` being identical in enforcement is deliberate and will read as a gap. What separates them is
|
|
70
|
+
whether the agent raises ambiguity, and no hook observes ambiguity — text is the honest mechanism there. Giving
|
|
71
|
+
them different enforcement would mean inventing a signal that does not exist.
|
|
72
|
+
|
|
73
|
+
Removing grind forcing means an operator who relied on the deepest posture implying grind now sets grind
|
|
74
|
+
explicitly. Both known installs already enable it, so the measured impact is zero, and the alternative is a
|
|
75
|
+
posture that quietly changes what "verified" means.
|
|
76
|
+
|
|
77
|
+
Two things about verifying this are worth keeping. **A second derivation of the same fact can pass an entire
|
|
78
|
+
suite.** The precedence between the project and the user config was written twice — once where the policy is
|
|
79
|
+
loaded, once where it is reported — and reversing one of them passed all 982 tests in that suite. The test that
|
|
80
|
+
now pins it makes the two configs *disagree*, because with both saying the same thing a reversed precedence
|
|
81
|
+
passes. **And a module can ship with no test file at all.** `policy.operator.ts` was rewritten with the
|
|
82
|
+
threshold moved out of its invariant block, and the green suite said nothing, because there was never a test
|
|
83
|
+
file to fail. A passing gate is evidence about the tests that exist and about nothing else.
|
|
84
|
+
|
|
85
|
+
## Not decided here
|
|
86
|
+
|
|
87
|
+
- **Whether `init` should validate `mode` before writing it.** It validates no other field, and a bad value now
|
|
88
|
+
surfaces through the fallback and the `doctor` row rather than disappearing.
|
|
89
|
+
- **A fourth posture**, or per-project thresholds inside a posture. Three levels of one dimension is the whole
|
|
90
|
+
model; a second dimension would be a different decision.
|
|
91
|
+
- **Whether `paired` should also pre-check non-shell actions** such as an Edit to a large file. Only the shell
|
|
92
|
+
is classified today.
|
|
@@ -0,0 +1,131 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: Decision
|
|
3
|
+
title: "AD-026 — An interruption is worth what it costs: narrower asks, a deadline on questions, and a rate the operator can see"
|
|
4
|
+
description: "Published 2026 work confirmed that verification must not weaken with autonomy and contradicted three details of the posture model: the paired ask was broad enough to become reflex, no posture stated when a question stops being worth asking, and the interruption rate was unmeasurable while the report printed a number for it. Approval fatigue is treated as a security defect, ambiguity gets a deadline stated in words, and a rail that was pointed at the wrong phase is connected."
|
|
5
|
+
tags: [decision, policy, posture, security, observability]
|
|
6
|
+
timestamp: "2026-08-04"
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# AD-026 — An interruption is worth what it costs
|
|
10
|
+
|
|
11
|
+
- **status**: active
|
|
12
|
+
- **date**: 2026-08-04
|
|
13
|
+
|
|
14
|
+
## Decision
|
|
15
|
+
|
|
16
|
+
1. **The `paired` ask marks a command that can lose something or widen access, not one that merely changes
|
|
17
|
+
something.** The shell classifier gains two classes: `write-preserving` (an append — content survives, so it
|
|
18
|
+
leaves the asking tier) and `privilege` (`chmod`, `chown` — no byte is lost and access moves, so it stays).
|
|
19
|
+
`cp`, `mv` and `tee` stay in `write`, because each can overwrite a destination and the harness cannot know
|
|
20
|
+
whether that destination exists. `privilege` outranks `write`: an overwrite inside a repository is recoverable
|
|
21
|
+
and visible in a diff, a widened permission is neither.
|
|
22
|
+
2. **`>` and `>>` are different classes.** They collapsed into one branch, so an append was indistinguishable
|
|
23
|
+
from an overwrite — the exact line the new class draws.
|
|
24
|
+
3. **`PAIRED_ASK` is unchanged.** The narrowing happens in the classifier, because what a command can do is
|
|
25
|
+
decided in one place.
|
|
26
|
+
4. **Every posture states a deadline for ambiguity, not only a threshold.** An unclear goal belongs in the first
|
|
27
|
+
actions; once the work is under way, the agent takes the most reasonable reading and states the assumption
|
|
28
|
+
instead of asking. `focus` admits exactly one early question — the goal it cannot read before starting.
|
|
29
|
+
5. **No posture line contains a digit.** The harness cannot measure where in a trajectory it is, so the deadline
|
|
30
|
+
is stated in words. A position it cannot compute would be a claim it cannot honour.
|
|
31
|
+
6. **The autopilot's posture step states the settle-and-state form at every posture and invites no question.** It
|
|
32
|
+
renders only after a gate has already failed, so it is late by construction — the one place where the deadline
|
|
33
|
+
is a fact rather than a guess.
|
|
34
|
+
7. **A `deny` or `ask` decision may name the rule that produced it.** `Decision.rule` is optional; a consumer
|
|
35
|
+
that wants attribution and finds none reports it as unattributed rather than guessing.
|
|
36
|
+
8. **Every shell decision is recorded with its permission, the active posture and its rule.** An `allow` grades
|
|
37
|
+
as debug and `debugEnabled` is false by default, so the common path computes and drops it.
|
|
38
|
+
9. **The session report attributes interruptions by rule**, ordered by weight, rendered only when non-empty.
|
|
39
|
+
10. **The harness does not claim Ask-F1, or any precision or recall.** It records its own decisions. It never sees
|
|
40
|
+
the operator's answer, and it cannot see whether a question it did not ask would have helped.
|
|
41
|
+
|
|
42
|
+
## Why
|
|
43
|
+
|
|
44
|
+
Three findings from published 2026 work, and one confirmation.
|
|
45
|
+
|
|
46
|
+
**Confirmed: verification must not weaken as autonomy rises.** Graduated oversight for agentic code generation
|
|
47
|
+
(arXiv 2606.22484) names that coupling — more autonomy, less verification rigour — as the common pitfall, and
|
|
48
|
+
recommends gates stay constant or strengthen. That is AD-025, and nothing here touches it.
|
|
49
|
+
|
|
50
|
+
**Approval fatigue is a security defect, not an ergonomics complaint.** A survey reframing agent security as an
|
|
51
|
+
agent-human interaction problem (arXiv 2605.24309) finds the cost asymmetry decides the outcome: rejecting an
|
|
52
|
+
action means understanding it well enough to say why, while approving is one keystroke. After enough
|
|
53
|
+
interruptions the human approves by reflex — and an attacker who can influence agent output hides the
|
|
54
|
+
consequential action inside the stream of routine ones, where a habituated reviewer waves it through. The
|
|
55
|
+
convergent mitigation across every source is the same: reduce the number of approvals so the remaining ones get
|
|
56
|
+
real attention. `PAIRED_ASK` covered the whole `write` class, which meant asking before `chmod +x` and before
|
|
57
|
+
appending a line to a file. Those are the routine ones.
|
|
58
|
+
|
|
59
|
+
Measured against the shipped classifier rather than assumed. The research note that opened this work claimed
|
|
60
|
+
`mkdir` asked; it never did — `mkdir` and `touch` classify as `read`. What actually asked was `cp`, `mv`,
|
|
61
|
+
`chmod`, `chown`, `rm`, `rmdir`, `tee`, `truncate`, `sed -i`, and both redirects. Three of those cannot lose a
|
|
62
|
+
byte that existed.
|
|
63
|
+
|
|
64
|
+
**A late question is worse than no question.** Clarification timing across 6,000+ runs, four models and four
|
|
65
|
+
information dimensions (arXiv 2605.07937) measures the decay: a goal question loses nearly all its value once
|
|
66
|
+
work is under way, an input question holds longer, and deferring any clarification past the middle of the work is
|
|
67
|
+
worse than never asking at all. Cross-model agreement (Kendall τ 0.78–0.87) says this is the structure of the
|
|
68
|
+
task, not a quirk of one model. Every posture line stated a threshold with no deadline, which licensed precisely
|
|
69
|
+
the worst case: asking at the twentieth action about a goal misread at the first. `focus` already had the correct
|
|
70
|
+
late behaviour and never said that a goal unclear *before starting* is the one question worth the interruption.
|
|
71
|
+
`focus` means ask early or not at all — not never ask.
|
|
72
|
+
|
|
73
|
+
**The interruption rate was unmeasurable, and the report said otherwise.** HiL-Bench (arXiv 2604.09408) defines
|
|
74
|
+
Ask-F1 — the harmonic mean of question precision and blocker recall — because a benchmark rewarding only
|
|
75
|
+
execution scores a lucky guess the same as a correct escalation. This harness measured nothing about asking, and
|
|
76
|
+
the shape of that gap was worse than absence. `attrs.permission` was read in two places and written in none, so
|
|
77
|
+
`shell.ask` and `shell.deny` were structurally zero while the report printed `Shell allow/ask/deny` — a
|
|
78
|
+
truthful-looking `0` for every ask that ever happened. And the rail was not merely unconnected: the level branch
|
|
79
|
+
keyed on `shell.end` alone, and a command that was denied or asked about never reaches an "after" event. The only
|
|
80
|
+
permission that phase could ever carry was `allow`, so the counter could not have moved even once something wrote
|
|
81
|
+
the attribute. The decision exists at `shell.start`, and that is where it is now recorded.
|
|
82
|
+
|
|
83
|
+
## Trade-offs
|
|
84
|
+
|
|
85
|
+
The narrowing is small, and worth stating plainly rather than dressing up: **one shape left the asking tier** —
|
|
86
|
+
the append. `cp` and `mv` stayed because the destination may exist, which is unknowable without touching the
|
|
87
|
+
filesystem, and `tee` stayed with or without `-a` because the flagless form truncates and a rule whose failure
|
|
88
|
+
mode is silent data loss does not get a special case. So the fatigue reduction this delivers is close to nothing;
|
|
89
|
+
what it delivers is the structure and the correct question. Whether `paired` actually interrupts too often is
|
|
90
|
+
unanswered, and decision 8 is the instrument that will answer it, with no data in it yet.
|
|
91
|
+
|
|
92
|
+
**The first version of this decision had the wrong axis, and it shipped for one revision.** `chmod` and `chown`
|
|
93
|
+
were classified as preserving on the reasoning that they lose no bytes. That is true and beside the point:
|
|
94
|
+
`chmod -R 777 .` went through `paired` unremarked. Losing content and widening access are separate questions, and
|
|
95
|
+
the second is the one whose answer never appears in a diff — which makes it worse to miss, not better. The
|
|
96
|
+
correction is recorded here rather than quietly folded in, because the failure was in the criterion, not in the
|
|
97
|
+
code, and a criterion is the kind of thing that gets reused.
|
|
98
|
+
|
|
99
|
+
**Sticky approval was rejected, using the same source that motivated the narrowing.** The obvious next step —
|
|
100
|
+
remember that this command was approved and stop asking — is the pattern the survey names as having the *worst
|
|
101
|
+
long-term security profile*, because each click monotonically widens the trusted surface. Worse, this harness
|
|
102
|
+
cannot tell a human approval from a host auto-approval: `shell.after` fires either way, so a trust cache built on
|
|
103
|
+
it would record consent that was never given. Every sticky approval needs scope and expiry, and the honest
|
|
104
|
+
sequence is to measure the rate first — which is what decision 8 makes possible, with data rather than taste.
|
|
105
|
+
|
|
106
|
+
**Sandbox auto-approval was rejected for a simpler reason.** It is the pattern with the strongest reported
|
|
107
|
+
reduction, and it delegates the decision to a boundary this harness does not have. Claiming it without the
|
|
108
|
+
mechanism would advertise a protection that does not exist.
|
|
109
|
+
|
|
110
|
+
`focus` gaining a stop reads as a regression against "fewest interruptions" and is not: the stop is bounded to
|
|
111
|
+
before the first action, and one question there is cheaper than everything built on a misreading. A test asserts
|
|
112
|
+
the late form is still settle-and-state, so the bound holds.
|
|
113
|
+
|
|
114
|
+
Widening `ShellEffectClass` touched five references in one file, and `tsc` proves the exhaustiveness. Adding an
|
|
115
|
+
optional field to two `Decision` members is source-compatible; no existing consumer reads it.
|
|
116
|
+
|
|
117
|
+
## Not decided here
|
|
118
|
+
|
|
119
|
+
- **`policy.deny` has the same shape of defect.** `rollup.denials` and the report's "Policy denials" are fed by
|
|
120
|
+
an obs kind nothing emits. Fixing it means deciding which layer owns the emission and accepting that emitting
|
|
121
|
+
`tool.start` creates empty per-tool slots in the report. Recorded, not fixed.
|
|
122
|
+
- **Floor denials are not recorded.** `rm -rf /` never reaches a policy layer, so the rate covers policy
|
|
123
|
+
decisions only. The catastrophic-attribution test states this in the command it chooses.
|
|
124
|
+
- **Whether `paired` should pre-check non-shell actions.** Unchanged from AD-025: only the shell is classified,
|
|
125
|
+
and asking before every one-line `Edit` turns a posture into a permission prompt.
|
|
126
|
+
- **Plan-level approval as the middle posture.** `planGate` already checks declared scope against the diff, which
|
|
127
|
+
covers the harm; swapping `solo` for a plan-approval model would redesign a feature that just stabilised.
|
|
128
|
+
- **Autonomy that accrues with trust.** Posture is the operator's declared preference, not a score the harness
|
|
129
|
+
assigns itself.
|
|
130
|
+
- **A second agent detecting underspecification** (arXiv 2603.26233). Demonstrated, and out of scale for a hook:
|
|
131
|
+
it needs a monitor running alongside every turn.
|
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: Decision
|
|
3
|
+
title: "AD-027 — Evidence is ordered against the code, every rail's firing is recorded, and a checker can run with its rule off"
|
|
4
|
+
description: "The ship gate compared evidence to the clock instead of to the work it certified. Three declared observability rails had no producer while the report printed numbers for two of them. And nothing could tell a rail that works from a rail that was never needed — so a checker can now run while its rule is switched off, which is the reading that makes deleting a rail a decision rather than a feeling."
|
|
5
|
+
migration: "Re-run your verification after the last code change before citing the verdict — the ship gate now refuses evidence written before the code it certifies, so a claim that used to pass can block. Nothing in doctor can see this one; it shows up as a blocked stop."
|
|
6
|
+
tags: [decision, ship, observability, rails, measurement]
|
|
7
|
+
timestamp: "2026-08-04"
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
# AD-027 — Measure the rails, and measure them honestly
|
|
11
|
+
|
|
12
|
+
- **status**: active
|
|
13
|
+
- **date**: 2026-08-04
|
|
14
|
+
|
|
15
|
+
## Decision
|
|
16
|
+
|
|
17
|
+
1. **Evidence must be newer than the code it certifies.** `hasRecentEvidence` takes the newest mtime among the
|
|
18
|
+
turn's changed files; a verdict older than that does not count. Age remains the second criterion, and a missing
|
|
19
|
+
timestamp falls back to age rather than failing closed.
|
|
20
|
+
2. **`gate.outcome` is emitted**, from inside `runLockedGate` so a gate added later cannot be forgotten, and
|
|
21
|
+
outside the lock so a measurement never widens the window in which one gate blocks another.
|
|
22
|
+
3. **`policy.deny` is emitted** from the one place every entrypoint's decision passes through, after `degrade`, so
|
|
23
|
+
what is recorded is what the provider actually rendered.
|
|
24
|
+
4. **A floor denial carries its rule structurally.** The floor already wrote `rule=<name>` into its reason prose;
|
|
25
|
+
attributing a refusal no longer requires parsing English.
|
|
26
|
+
5. **One rail owns every shell decision, including the ones the floor makes.** The shared refusal path skips
|
|
27
|
+
`shell.before` to avoid double counting, so the shell rail records the floor's shell denials itself.
|
|
28
|
+
6. **Refusals are counted by rule in one place**, whichever rail produced them, and gates are counted by name.
|
|
29
|
+
7. **The session report answers three questions that need no inference**: which enabled rail never fired, which
|
|
30
|
+
fires constantly, and what the injected prose cost in characters. The active-rail list is a parameter, because
|
|
31
|
+
core cannot see which rails a project enabled and a report that guessed would accuse a rail nobody switched on.
|
|
32
|
+
8. **Observation mode runs a rail's checker while that rail is not enforcing**, records the reading, and never
|
|
33
|
+
returns a decision. Opt-in, default empty. An enforcing rail is not observed, and observations carry their own
|
|
34
|
+
obs kind so the refusal counters stay honest.
|
|
35
|
+
|
|
36
|
+
## Why
|
|
37
|
+
|
|
38
|
+
**Freshness is not what makes evidence evidence.** The ship gate asked `now - mtime > maxAgeHours`. A verdict
|
|
39
|
+
written ten minutes ago passed while the code it certified changed five minutes ago — evidence that predates the
|
|
40
|
+
change proves nothing about it, and the gate accepted it in silence. Ordering is the question that matters, and the
|
|
41
|
+
comparison timestamp was already in hand: the stop path lists the turn's changed files on every run.
|
|
42
|
+
|
|
43
|
+
**Three rails were declared and unfed, and two of them printed numbers.** `gate.outcome` is consumed by the rollup
|
|
44
|
+
counter and by the report's "Gates pass/fail" line, and was emitted by nothing — so the report has printed a
|
|
45
|
+
truthful-looking `0 / 0` for every gate this harness has ever run. `policy.deny` fed `rollup.denials` and "Policy
|
|
46
|
+
denials" with no producer, so a harness whose whole purpose is refusing things reported zero refusals. This is the
|
|
47
|
+
third instance of the family recorded in AD-020 and AD-024: a field advertised and never written. The pattern is
|
|
48
|
+
consistent enough to name — a consumer written before its producer reads as working, because zero is a plausible
|
|
49
|
+
number.
|
|
50
|
+
|
|
51
|
+
**And a firing rate cannot tell you whether a rule was needed.** A rail that never fires while its prose is
|
|
52
|
+
injected is either working perfectly or unnecessary, and the count is identical in both cases. That matters because
|
|
53
|
+
prose is not free: the harness injects at every session start, paid in input tokens on every turn, and it had never
|
|
54
|
+
told the operator the size of that bill.
|
|
55
|
+
|
|
56
|
+
Running the checker with the prose *absent* separates the two. If the property holds anyway, the model was already
|
|
57
|
+
honouring it and the rule is paying for context and returning nothing. This is possible here and almost nowhere
|
|
58
|
+
else, because in this harness the checker and the instruction are different things; in a system where the rule *is*
|
|
59
|
+
the mechanism there is nothing to hold apart.
|
|
60
|
+
|
|
61
|
+
## Trade-offs
|
|
62
|
+
|
|
63
|
+
Observation costs a diff scan per turn per observed rail, and it is opt-in for exactly that reason: it answers a
|
|
64
|
+
question only an operator who is asking it needs answered. It never blocks and never changes a decision, and a test
|
|
65
|
+
asserts the turn's outcome is identical with it on — a measurement that can change what it measures is not a
|
|
66
|
+
measurement.
|
|
67
|
+
|
|
68
|
+
The `held-with-prose` reading is ambiguous by construction and stays that way. Resolving it would mean withholding
|
|
69
|
+
the prose on a fraction of real turns to compare, which pays for the measurement in the operator's own outcomes.
|
|
70
|
+
The harness's job is to make this turn better.
|
|
71
|
+
|
|
72
|
+
Recording the injected character count is a proxy for tokens, not a token count. It is honest about being a
|
|
73
|
+
character count, and it moves in the same direction as the thing that is billed.
|
|
74
|
+
|
|
75
|
+
The ordering check compares mtimes, which a filesystem with coarse timestamps or a checkout that rewrites them can
|
|
76
|
+
confuse. Age remains as the second criterion for that reason, and an absent timestamp falls back to it rather than
|
|
77
|
+
blocking a stop on missing input.
|
|
78
|
+
|
|
79
|
+
## Not decided here
|
|
80
|
+
|
|
81
|
+
- **A probe that runs a task N times with and without a rule.** It is the clean way to measure a rule's effect and
|
|
82
|
+
it needs task repetition. Real work does not repeat: a bug is fixed once. That design belongs to a benchmark.
|
|
83
|
+
- **A placebo arm** — injecting filler of equal token count to separate "the content helped" from "more context
|
|
84
|
+
helped". Methodologically the sharpest instrument available and it degrades real work to obtain a reading.
|
|
85
|
+
- **Inference over the recorded data.** Mixed models, Bayes factors and equivalence tests are sized for a
|
|
86
|
+
benchmark, and the load-bearing element is a paired design that task repetition would provide and real work does
|
|
87
|
+
not — while task identity is the largest confound there is. Inference earns its keep at fleet scale, where a
|
|
88
|
+
random intercept per repository is meaningful. Two installations are not a fleet.
|
|
89
|
+
- **Per-rail token attribution.** The report gives one number for all injected prose. Splitting it by rail means
|
|
90
|
+
threading provenance through the assembly, and the single number already answers "is this bill large or small".
|