session-orchestrator 5.3.0 → 5.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/claude-md-drift-check/SKILL.md +1 -1
- package/.agents/skills/test-audit/SKILL.md +21 -0
- package/.agents/skills/test-audit/agents/openai.yaml +5 -0
- package/.claude-plugin/marketplace.json +1 -1
- package/.claude-plugin/plugin.json +1 -1
- package/.codex-plugin/plugin.json +1 -1
- package/.codex-plugin/skills/claude-md-drift-check/SKILL.md +1 -1
- package/.codex-plugin/skills/test-audit/SKILL.md +21 -0
- package/.codex-plugin/skills/test-audit/agents/openai.yaml +5 -0
- package/.cursor/commands/test-audit.md +14 -0
- package/.cursor/rules/060-evolve.mdc +28 -13
- package/.cursor/skills/claude-md-drift-check/SKILL.md +1 -1
- package/.cursor/skills/test-audit/SKILL.md +12 -0
- package/.cursor-plugin/plugin.json +1 -1
- package/.orchestrator/policy/blocked-commands.json +1 -1
- package/AGENTS.md +9 -31
- package/CHANGELOG.md +162 -0
- package/README.md +14 -10
- package/docs/agent-authoring.md +2 -2
- package/docs/baseline.md +10 -1
- package/docs/ci-setup.md +154 -4
- package/docs/components.md +11 -11
- package/docs/events-schema.md +25 -19
- package/docs/install.md +14 -2
- package/docs/migration-v4.md +24 -17
- package/docs/persona-panel.md +5 -4
- package/docs/plugin-architecture-v3.md +1 -1
- package/docs/rule-authoring.md +27 -12
- package/docs/scope-collision-guard.md +40 -16
- package/docs/session-config-reference.md +61 -12
- package/docs/session-config-template.md +11 -2
- package/docs/vault-docs-architecture.md +8 -4
- package/hooks/_lib/hook-import-set.json +42 -29
- package/hooks/_lib/lock-bootstrap.mjs +118 -26
- package/hooks/config-protection.mjs +1 -1
- package/hooks/enforce-commands.mjs +28 -13
- package/hooks/enforce-scope.mjs +201 -23
- package/hooks/hooks-codex.json +1 -1
- package/hooks/hooks-cursor.json +5 -0
- package/hooks/hooks-pi.json +5 -0
- package/hooks/hooks.json +6 -1
- package/hooks/on-session-start.mjs +57 -2
- package/hooks/on-stop.mjs +70 -168
- package/hooks/post-bash-write-verify.mjs +98 -17
- package/hooks/post-edit-validate.mjs +15 -5
- package/hooks/post-tool-batch-wave-signal.mjs +32 -152
- package/hooks/pre-bash-memory-propose-audit.mjs +71 -10
- package/hooks/pre-bash-sessions-ledger-guard.mjs +28 -2
- package/hooks/pre-bash-staging-fence.mjs +29 -8
- package/hooks/pre-edit-sessions-ledger-guard.mjs +162 -0
- package/hooks/pre-task-scope-disjoint.mjs +1143 -279
- package/hooks/subagent-telemetry.mjs +96 -19
- package/hooks/wave-scope-commit-guard.mjs +168 -72
- package/package.json +2 -6
- package/pi/prompts/test-audit.md +12 -0
- package/rules/README.md +6 -1
- package/rules/_index.md +3 -3
- package/rules/always-on/bash-harness-pitfalls.md +7 -1
- package/rules/opt-in-domain/prompt-caching.md +5 -1
- package/rules/opt-in-stack/backend-data.md +9 -1
- package/rules/opt-in-stack/backend.md +7 -1
- package/rules/opt-in-stack/frontend.md +6 -1
- package/rules/opt-in-stack/security-web.md +7 -1
- package/rules/opt-in-stack/swift.md +5 -1
- package/scripts/apply-session-learnings.mjs +513 -0
- package/scripts/archive-closed-prds.mjs +40 -12
- package/scripts/autopilot.mjs +57 -54
- package/scripts/backfill-abandoned-sessions.mjs +81 -6
- package/scripts/backfill-learnings-from-vault.mjs +49 -22
- package/scripts/baseline-archetypes.mjs +1 -1
- package/scripts/check-package-manager.mjs +62 -5
- package/scripts/codex-install.mjs +3 -0
- package/scripts/cursor-install.mjs +47 -6
- package/scripts/emit-session.mjs +298 -2
- package/scripts/export-hw-learnings.mjs +40 -10
- package/scripts/fleet-checkin.mjs +170 -0
- package/scripts/lib/agent-frontmatter.mjs +3 -3
- package/scripts/lib/auto-dream.mjs +7 -6
- package/scripts/lib/autopilot/flags.mjs +6 -1
- package/scripts/lib/autopilot/loop.mjs +32 -13
- package/scripts/lib/autopilot/stall-sampler.mjs +13 -9
- package/scripts/lib/baseline-archetypes.mjs +41 -22
- package/scripts/lib/blocked-commands-policy.mjs +40 -0
- package/scripts/lib/common.mjs +41 -0
- package/scripts/lib/config/host-paths.mjs +79 -0
- package/scripts/lib/config/persona-gate-wave.mjs +3 -3
- package/scripts/lib/config/reaper.mjs +7 -6
- package/scripts/lib/config/vault-integration.mjs +151 -16
- package/scripts/lib/config-schema.mjs +13 -0
- package/scripts/lib/config.mjs +75 -10
- package/scripts/lib/eval/engine.mjs +13 -0
- package/scripts/lib/events-retention-banner.mjs +393 -0
- package/scripts/lib/events-rotation.mjs +21 -6
- package/scripts/lib/events.mjs +72 -5
- package/scripts/lib/fleet-protocol.mjs +288 -0
- package/scripts/lib/gates/gate-baseline.mjs +13 -3
- package/scripts/lib/gates/gate-full.mjs +7 -2
- package/scripts/lib/gates/gate-helpers.mjs +93 -2
- package/scripts/lib/gates/gate-incremental.mjs +23 -22
- package/scripts/lib/gates/gate-per-file.mjs +16 -7
- package/scripts/lib/hardening.mjs +1 -0
- package/scripts/lib/harness-audit/categories/category4.mjs +8 -6
- package/scripts/lib/instruction-budget-guard.mjs +77 -11
- package/scripts/lib/learnings/expiry-sweep.mjs +420 -22
- package/scripts/lib/learnings/io.mjs +274 -37
- package/scripts/lib/learnings/schema.mjs +163 -15
- package/scripts/lib/learnings/sizing-subject.mjs +2 -2
- package/scripts/lib/lock-reaper.mjs +38 -14
- package/scripts/lib/maintenance-due-banner.mjs +23 -8
- package/scripts/lib/memory-proposals/schema.mjs +11 -5
- package/scripts/lib/named-baseline-resolver.mjs +53 -28
- package/scripts/lib/named-vault-resolver.mjs +147 -5
- package/scripts/lib/orphan-reaper/defaults.mjs +41 -0
- package/scripts/lib/orphan-reaper/no-follow-append.mjs +87 -0
- package/scripts/lib/orphan-reaper/ps-snapshot.mjs +181 -0
- package/scripts/lib/orphan-reaper/reaper-audit.mjs +323 -0
- package/scripts/lib/orphan-reaper/reaper-decide.mjs +395 -0
- package/scripts/lib/orphan-reaper/scan-throttle.mjs +144 -0
- package/scripts/lib/orphan-reaper/scan.mjs +510 -0
- package/scripts/lib/orphan-reaper/trigger.mjs +173 -0
- package/scripts/lib/orphan-reaper.mjs +47 -1315
- package/scripts/lib/owner-yaml.mjs +104 -6
- package/scripts/lib/persona-panel/catalog-loader.mjs +1 -1
- package/scripts/lib/platform.mjs +206 -2
- package/scripts/lib/plugin-update-banner.mjs +3 -1
- package/scripts/lib/process-group.mjs +134 -22
- package/scripts/lib/project-hygiene.mjs +21 -1
- package/scripts/lib/quality-gate.mjs +45 -5
- package/scripts/lib/resource-probe.mjs +1 -1
- package/scripts/lib/rule-loader.mjs +100 -62
- package/scripts/lib/rules-sync.mjs +27 -12
- package/scripts/lib/scope-baseline.mjs +119 -29
- package/scripts/lib/scope-echo.mjs +7 -0
- package/scripts/lib/scope-gate.mjs +194 -27
- package/scripts/lib/session-close-backfill.mjs +129 -4
- package/scripts/lib/session-end/tail-runner.mjs +10 -5
- package/scripts/lib/session-lock-shape.mjs +6 -3
- package/scripts/lib/session-lock.mjs +320 -67
- package/scripts/lib/session-record-repair.mjs +183 -26
- package/scripts/lib/session-registry.mjs +67 -17
- package/scripts/lib/session-schema/constants.mjs +30 -4
- package/scripts/lib/session-schema/validator.mjs +28 -1
- package/scripts/lib/session-start-probes.mjs +242 -68
- package/scripts/lib/session-token-rollup.mjs +252 -54
- package/scripts/lib/sessions-canonical.mjs +100 -9
- package/scripts/lib/sessions-staleness-banner.mjs +3 -1
- package/scripts/lib/sh-quote.mjs +29 -0
- package/scripts/lib/subagents-schema.mjs +30 -3
- package/scripts/lib/tail-window.mjs +27 -3
- package/scripts/lib/telemetry/pricing.mjs +121 -71
- package/scripts/lib/telemetry/sync.mjs +17 -0
- package/scripts/lib/tests-src-ratio.mjs +215 -23
- package/scripts/lib/tmux-layout/layouts.mjs +1 -11
- package/scripts/lib/tmux-layout/telemetry-stats.mjs +12 -0
- package/scripts/lib/tmux-layout/tmux-shell.mjs +2 -11
- package/scripts/lib/ux-grill/reconcile.mjs +1 -1
- package/scripts/lib/validate/check-agents.mjs +1 -1
- package/scripts/lib/validate/check-hooks-symmetry.mjs +5 -0
- package/scripts/lib/validate/check-learnings-shell-writes.mjs +214 -0
- package/scripts/lib/validate/check-test-value-bans.mjs +388 -27
- package/scripts/lib/validate/check-unicode-safety.mjs +1 -0
- package/scripts/lib/validate/check-unwired-features.mjs +1 -1
- package/scripts/lib/validate/check-vcs-repo-flag.mjs +1 -1
- package/scripts/lib/validate-vendored-rules.mjs +206 -81
- package/scripts/lib/vault-mirror/auto-commit.mjs +5 -3
- package/scripts/lib/vault-mirror/namespace.mjs +47 -26
- package/scripts/lib/vault-mirror/process.mjs +350 -62
- package/scripts/lib/vault-mirror/pseudonym-map.mjs +14 -5
- package/scripts/lib/vault-mirror/render-learnings.mjs +66 -3
- package/scripts/lib/vault-mirror/render-sessions.mjs +28 -4
- package/scripts/lib/vault-mirror/telemetry.mjs +25 -11
- package/scripts/lib/vault-status/board-writer.mjs +53 -12
- package/scripts/lib/vault-status/narrative-mirror.mjs +40 -9
- package/scripts/lib/vcs-repo-spec.mjs +8 -0
- package/scripts/lib/wave-executor/dispatch-common.mjs +10 -18
- package/scripts/lib/wave-executor/foreign-dispatch.mjs +1 -1
- package/scripts/lib/wave-executor/remote-dispatch.mjs +1 -1
- package/scripts/lib/wave-resource-gate.mjs +34 -8
- package/scripts/materialize-wave-scope.mjs +7 -4
- package/scripts/memory-propose.mjs +18 -5
- package/scripts/release-session-lock.mjs +2 -1
- package/scripts/release.mjs +11 -3
- package/scripts/repair-invalid-sessions.mjs +37 -4
- package/scripts/run-migrate-v2-cross-repo.mjs +171 -31
- package/scripts/run-quality-gate.mjs +331 -307
- package/scripts/self-update.mjs +229 -0
- package/scripts/site-numbers.mjs +163 -5
- package/scripts/sweep-expired-learnings.mjs +603 -20
- package/scripts/sweep-expired-rules.mjs +2 -1
- package/scripts/validate-plugin.mjs +7 -0
- package/scripts/validate-wave-scope.mjs +21 -2
- package/scripts/vault-backfill.mjs +50 -37
- package/scripts/vault-integration-watcher.mjs +2 -1
- package/scripts/vault-mirror.mjs +238 -75
- package/scripts/wave-scope-binding.mjs +2 -1
- package/skills/_shared/config-reading.md +3 -3
- package/skills/_shared/fleet-protocol.md +134 -0
- package/skills/_shared/state-ownership.md +41 -12
- package/skills/autopilot/SKILL.md +25 -11
- package/skills/bootstrap/SKILL.md +6 -4
- package/skills/bootstrap/_shared-template.md +97 -2
- package/skills/bootstrap/fast-template.md +4 -1
- package/skills/bootstrap/private-contract.md +2 -1
- package/skills/bootstrap/public-fallback.md +6 -3
- package/skills/bootstrap/references/bootstrap-refresh-lock-flow.md +5 -3
- package/skills/bootstrap/references/bootstrap-retroactive-flow.md +2 -1
- package/skills/bootstrap/standard-template.md +2 -0
- package/skills/claude-md-drift-check/SKILL.md +10 -10
- package/skills/claude-md-drift-check/checker.mjs +26 -18
- package/skills/discovery/probes/ssot-code-diff.mjs +12 -6
- package/skills/discovery/probes/vault-narrative-staleness.mjs +6 -2
- package/skills/discovery/probes/vault-staleness.mjs +9 -3
- package/skills/docs-orchestrator/SKILL.md +12 -5
- package/skills/docs-orchestrator/audience-mapping.md +5 -3
- package/skills/evolve/SKILL.md +13 -4
- package/skills/evolve/references/evolve-analyze-mode.md +42 -18
- package/skills/memory-cleanup/SKILL.md +1 -1
- package/skills/persona-panel/SKILL.md +2 -2
- package/skills/persona-panel/persona-format.md +2 -2
- package/skills/persona-panel/presets/designer-lens.md +1 -1
- package/skills/persona-panel/presets/engineer-lens.md +1 -1
- package/skills/persona-panel/presets/pm-lens.md +1 -1
- package/skills/plan/mode-retro.md +19 -4
- package/skills/quality-gates/SKILL.md +6 -4
- package/skills/remote-offload/SKILL.md +4 -2
- package/skills/session-end/learning-patterns.md +23 -13
- package/skills/session-end/metrics-collection.md +6 -7
- package/skills/session-end/phase-3-2-docs-verification.md +7 -6
- package/skills/session-end/plan-verification.md +25 -9
- package/skills/session-end/references/phase-3-documentation-updates.md +13 -3
- package/skills/session-end/references/phase-4a-worktree-cleanup.md +2 -0
- package/skills/session-end/references/session-summary-template.md +1 -1
- package/skills/session-end/session-metrics-write.md +65 -68
- package/skills/session-plan/SKILL.md +0 -3
- package/skills/session-start/SKILL.md +8 -2
- package/skills/session-start/references/operations-contract.md +6 -0
- package/skills/session-start/references/phase-1-2-session-lock.md +32 -7
- package/skills/session-start/references/phase-1-7-vault-status-board.md +1 -1
- package/skills/session-start/references/phase-4-ssot-environment-check.md +10 -3
- package/skills/test-audit/SKILL.md +152 -0
- package/skills/test-audit/references/campaign.md +111 -0
- package/skills/test-audit/references/mutate.sh +50 -0
- package/skills/vault-mirror/SKILL.md +21 -11
- package/skills/wave-executor/SKILL.md +2 -2
- package/skills/wave-executor/circuit-breaker.md +1 -1
- package/skills/wave-executor/references/wave-executor-state-init.md +3 -0
- package/skills/wave-executor/references/wave-loop-dispatch.md +11 -1
- package/skills/wave-executor/references/wave-loop-review.md +4 -6
- package/skills/wave-executor/references/wave-loop-scope-manifest.md +13 -5
- package/templates/personas/README.md +5 -5
- package/templates/personas/accounting-compliance.v1.md +1 -1
- package/templates/personas/accounting-tax-advisor.v1.md +1 -1
- package/templates/personas/buyer-p1-cto.v1.md +1 -1
- package/templates/personas/buyer-p2-kanzlei.v1.md +1 -1
- package/templates/personas/buyer-p3-build.v1.md +1 -1
- package/templates/personas/buyer-p4-tech-veto.v1.md +1 -1
- package/templates/personas/buyer-p5-solo.v1.md +1 -1
- package/templates/personas/buyer-p6-ld.v1.md +1 -1
- package/templates/personas/klima-ai-expert.v1.md +1 -1
- package/templates/personas/klima-physicist.v1.md +1 -1
|
@@ -0,0 +1,152 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: test-audit
|
|
3
|
+
description: >
|
|
4
|
+
Use when writing, reviewing, or pruning tests. Three modes, one value bar: a four-question gate
|
|
5
|
+
before any new test is written, a focused audit of one scope, and a campaign over a whole repo or
|
|
6
|
+
subsystem that removes the least useful tests under a coverage guard. Every delete, merge, or
|
|
7
|
+
repair carries written evidence and a caught mutation. Triggered by "audit the tests", "prune the
|
|
8
|
+
test suite", "remove low-value tests", "test diet", "is this test worth adding", /test-audit.
|
|
9
|
+
user-invocable: true
|
|
10
|
+
argument-hint: "[gate|audit|campaign] [scope] [--goal \"<goal>\"]"
|
|
11
|
+
model: inherit
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
# Test Audit
|
|
15
|
+
|
|
16
|
+
A test earns its place by catching a regression nothing else catches. This skill applies that bar at three depths. It complements `.claude/rules/test-value.md` (TV-001..TV-005) and the repo's test-quality / test-hygiene rules; it replaces none of them.
|
|
17
|
+
|
|
18
|
+
Rule references below (`test-value.md`, `receiving-review.md`, `parallel-sessions.md`, `bash-harness-pitfalls.md`) name the repo's `.claude/rules/`. A consumer repo often lacks them; then read them from the plugin's `rules/always-on/`.
|
|
19
|
+
|
|
20
|
+
## Invocation
|
|
21
|
+
|
|
22
|
+
Invoked as `/test-audit [gate|audit|campaign] [scope] [--goal "<goal>"]` with arguments: **$ARGUMENTS**.
|
|
23
|
+
|
|
24
|
+
- `gate` — run the write gate on the test about to be written. Default when no mode is given and the conversation is about to add a test.
|
|
25
|
+
- `audit <scope>` — one focused pass over a path, glob, or module. Default when a scope is given without a mode.
|
|
26
|
+
- `campaign <scope>` — a whole repo or subsystem, split into lanes. Only on the explicit word `campaign`; read [references/campaign.md](references/campaign.md) before starting.
|
|
27
|
+
- A missing scope for `audit` or `campaign` is asked for once via `AskUserQuestion`, with the largest test directories as options.
|
|
28
|
+
|
|
29
|
+
## Set an ambitious goal first (audit and campaign)
|
|
30
|
+
|
|
31
|
+
State the goal in the first output and repeat it in the report. With no `--goal`, use:
|
|
32
|
+
|
|
33
|
+
> Remove at least 20 % of the least useful tests, measured in test lines. Total coverage stays within 2 percentage points of M0, and no production file drops below its own M0 coverage.
|
|
34
|
+
|
|
35
|
+
Test lines count only for files the CI actually runs; tracked test files outside that scope are reported on their own line, as dead or as live outside the include, and never counted toward the goal ([references/campaign.md](references/campaign.md) § Measurements).
|
|
36
|
+
|
|
37
|
+
The goal exists because a bare "clean up" stops far too early. It never licenses a deletion without evidence. What counts toward it, and what is reported beside it (same-file reshaping, mock ballast, F/N growth, the O sum), is fixed in [references/campaign.md](references/campaign.md) § Goal accounting. A goal that is only reachable by breaking the evidence rules below is not reached: the report says which rule blocked it and where the audit stopped. An audit that ends with zero changes is valid when the ledger shows why.
|
|
38
|
+
|
|
39
|
+
## Mode 1: write gate
|
|
40
|
+
|
|
41
|
+
Before adding any test, answer all four. A missing answer means the test is not written yet.
|
|
42
|
+
|
|
43
|
+
1. Which observable behaviour, invariant, or independent contract does it protect?
|
|
44
|
+
2. Which credible regression turns it red?
|
|
45
|
+
3. Why does no existing test catch that regression? Grep first (TV-004); extend a table case or shared fixture instead of adding a near-duplicate.
|
|
46
|
+
4. Does it need a production seam (export, flag, hook, injection parameter) that no production caller uses? Then no: move the test to the real boundary. An export or parameter with one production caller is API when the package's public entry (`exports`, index, another repo) reaches it, and a seam otherwise.
|
|
47
|
+
|
|
48
|
+
Then check it against the junk patterns below; a match fails the gate unless the keep list names the contract it alone guards. A regression test must be shown red on the pre-fix code, for the intended reason, before the fix lands; one that never failed proves the mock, not the fix. One regression at the owning boundary covers the bug; do not replay it at every layer. `no-tests-needed: <reason>` is a success outcome (TV-001).
|
|
49
|
+
|
|
50
|
+
## The value bar
|
|
51
|
+
|
|
52
|
+
Judge each test by what its assertions can catch, never by its name.
|
|
53
|
+
|
|
54
|
+
**Junk patterns** (candidates for D, C, or F):
|
|
55
|
+
|
|
56
|
+
- no assertion, or an assertion that cannot fail (self-comparison, identity copy);
|
|
57
|
+
- copied inventories, export lists, manifests, or fixtures that restate the source;
|
|
58
|
+
- a test-local copy of another repo's schema or contract, checked against test-local payloads: it proves the copy, not the contract; the keeper is the synced artefact or the other repo's test;
|
|
59
|
+
- exact source, import, or string greps where a behavioural check exists;
|
|
60
|
+
- the expected value is produced by the helper under test;
|
|
61
|
+
- the mock implements the behaviour being asserted, or one mock stands in for different APIs;
|
|
62
|
+
- a negative control that passes for an unrelated reason (a different guard rejects first, or the production path never reaches the rejection); for a security check, one that only hits the early exit (wrong length, wrong prefix) and never the comparison itself is usually F;
|
|
63
|
+
- the name promises more than the assertion checks;
|
|
64
|
+
- an assertion that depends on the current date (a fixture expiring next quarter turns it red on its own; F with a fixed clock, and urgent);
|
|
65
|
+
- prose or structure pinned in a `.md` file (TV-002c), unless the project instruction file makes that structure a rule (then R, citing it);
|
|
66
|
+
- the test is the only caller of dead production code: D only together with the removal of that code (its own `code-implementer` commit, owner decision when the code is public API), otherwise O.
|
|
67
|
+
|
|
68
|
+
**Keep list** (the counterweight to TV-002): a test that is the independent proof for a public API, protocol, a config value production reads (a config field whose only reader is the test is a seam, not a contract), migration, storage format, security control, release step, or data-protection rule stays. So do observable call ordering and regressions with a credible failure mode. Static or slow is not a reason to delete. When in doubt, mark R.
|
|
69
|
+
|
|
70
|
+
**Accidental coverage** is not a proof: a line that runs only because a mock is incomplete (a missing method sends the code into its catch branch) or because a timer is left dangling after the test. Deleting its duplicate loses no contract but trips the coverage backstop; answer that with a contract test (fake timers, an explicit failing mock), never by restoring the accidental one.
|
|
71
|
+
|
|
72
|
+
## Ledger marks
|
|
73
|
+
|
|
74
|
+
Every test declaration in scope gets exactly one mark and one evidence line. A parametrised table is one declaration unless its rows need different marks.
|
|
75
|
+
|
|
76
|
+
| Mark | Meaning | Required note |
|
|
77
|
+
|---|---|---|
|
|
78
|
+
| R | keep | the contract and the bug it catches |
|
|
79
|
+
| F | keep the contract, repair the assertion | what the assertion misses today, and the mutation the repair must catch |
|
|
80
|
+
| C | consolidate into a keeper | the keeper test that absorbs it |
|
|
81
|
+
| D | delete | the remaining proof, or why no contract exists |
|
|
82
|
+
| N | new test | only after passing the write gate |
|
|
83
|
+
| B | red on the baseline (from the P1 / M0 pass/fail list only) | possible product bug: never deleted, becomes an issue |
|
|
84
|
+
| O | owner decision pending: keep | the open question and the candidate's line count; listed in the MR under open decisions, never folded into R |
|
|
85
|
+
|
|
86
|
+
Suspected product bugs found while reading, with no test behind them, go into a separate _Issues_ section of the ledger (`file:line`, expected vs. actual). They become issues, never red tests in the MR.
|
|
87
|
+
|
|
88
|
+
## Evidence before any C or D
|
|
89
|
+
|
|
90
|
+
All eight fields, written into the ledger before the edit. A missing field means no deletion.
|
|
91
|
+
|
|
92
|
+
1. Test location (`file:line`) and exact name.
|
|
93
|
+
2. The failure it can actually detect.
|
|
94
|
+
3. Non-test callers of the code it covers.
|
|
95
|
+
4. The stronger proof that remains (the keeper), or why none is needed.
|
|
96
|
+
5. History: `git log -S "<identifier>" -- <file>` and the reason the test or seam exists.
|
|
97
|
+
6. What the deletion unlocks (a test-only export, a wrapper, a dead path).
|
|
98
|
+
7. Risk and the focused command that re-checks it.
|
|
99
|
+
8. The proposed contract mutation (id, `src/file:line`, and the diff as a patch file) under which the keeper must go red; falsification executes it. A D that removes no contract (dead code, a duplicate proven elsewhere, a test-local copy) says so here and becomes a `NOCONTRACT` manifest row instead ([references/campaign.md](references/campaign.md) § Mutation manifest).
|
|
100
|
+
|
|
101
|
+
## Falsification
|
|
102
|
+
|
|
103
|
+
For every contract that a C, D, or F relies on, make one targeted mutation in the production code: flip the condition, drop the field, change the byte the contract names. Several marks may share one mutation when they name the same keeper and contract; the ledger maps every mark to at least one mutation id. Replacing a whole function body with `throw` kills every caller and proves nothing; use it at most as a pre-filter. The keeper (for C/D) or the repaired test (for F) must turn red under the mutation.
|
|
104
|
+
|
|
105
|
+
- Apply the mutation as a patch (`git apply`), run the one test, reverse it (`git apply -R`), then prove the restore byte for byte: `git diff --exit-code -- <prod-file>`, or a checksum taken before the mutation when the file already carried edits. A leftover diff aborts the audit. [references/mutate.sh](references/mutate.sh) runs a manifest of such patches ([references/campaign.md](references/campaign.md) § Mutation manifest). Prove the red with the runner's failure summary (vitest: `Tests N failed`), never with the exit code alone, and keep the output files.
|
|
106
|
+
- Never mutate a checkout another session or agent is using, and never edit while the test runner is live in that checkout. The binding run is the coordinator's, in an offload worktree: `offload run <repo> -H <host> --job ta-<repo>-<purpose> -- <command>`, reused with `--no-sync` (hosts come from `remote-hosts:` in Session Config; see the `remote-offload` skill; path and expansion pitfalls in [references/campaign.md](references/campaign.md) § Measurements). An agent's own pre-check runs only in a private copy outside the repo, under [references/campaign.md](references/campaign.md) § Pre-checks, and never counts as proof.
|
|
107
|
+
- Log each mutation: `file:line`, the mutation diff, the test that went red, the restore proof.
|
|
108
|
+
- Per-file coverage is the mechanical backstop, compared on all four metrics (lines, branches, functions, statements); lines alone miss a lost callback. When any of them drops for a production file, locate the line, branch, or function in the detailed report (vitest `--coverage.reporter=json`) and restore the contract in the keeper with a caught mutation. Restore the deleted test verbatim only when it was the genuine proof; accidental coverage (§ The value bar) gets a contract test instead.
|
|
109
|
+
|
|
110
|
+
Bash harnesses are falsified before their tests are judged: feed a known-broken input and require a non-zero exit and a FAIL in the written artefact, under `bash`, not zsh (`.claude/rules/bash-harness-pitfalls.md`).
|
|
111
|
+
|
|
112
|
+
## Mode 2: audit
|
|
113
|
+
|
|
114
|
+
1. Read the project instruction file and the test rules it points at. Check the coverage provider and the CI scope first, then record the M0 numbers for the scope at a pinned SHA (recipes in [references/campaign.md](references/campaign.md) § P1 and § Measurements).
|
|
115
|
+
2. Read every test in scope in full, plus the production owner, its entry point, callers, and history. For more than a handful of files, dispatch `qa-strategist` read-only to draft the ledger; it writes `/tmp/<audit>/ledger-<lane>.md` itself and answers with the count line, and the coordinator assembles the ledger file. A scope above one lane (about 2,200 test lines or 150 declarations, [references/campaign.md](references/campaign.md) § P2) needs campaign mode: say so and ask for the word `campaign` before step 3.
|
|
116
|
+
3. Write the ledger with marks and evidence. Prefer a few well-proven candidates over a long speculative list.
|
|
117
|
+
4. Keepers first, in one uncommitted pass: `test-writer` repairs every F, absorbs every C into its keeper, and deletes the C and D. Agents never commit (PSA-007 in `.claude/rules/parallel-sessions.md`).
|
|
118
|
+
5. Falsify every C, D, and F against that tree; commit only when every run row is `CAUGHT`. Removing a test-only production seam is a separate `code-implementer` task and commit.
|
|
119
|
+
6. Independent preservation review, read-only (`session-reviewer`, or `pr-review-toolkit:pr-test-analyzer` when installed): contracts that lost their only proof, and new assertions that cannot fail. Every restored contract needs a caught mutation.
|
|
120
|
+
7. Measure M1 with the same commands, then report.
|
|
121
|
+
|
|
122
|
+
## Mode 3: campaign
|
|
123
|
+
|
|
124
|
+
Phases P1 to P9 with lanes, a separate layer-plan pass, an owner checkpoint after P4, the sub-campaign threshold, the pre-check rules, and the full measurement set live in [references/campaign.md](references/campaign.md). Read it completely before P1.
|
|
125
|
+
|
|
126
|
+
## Never without the owner
|
|
127
|
+
|
|
128
|
+
Ask via `AskUserQuestion`, and do not work around a refusal. A headless run without that tool takes the answer the brief or this list already gives and logs it; anything else becomes an O mark plus an entry in the run's parking file (the report's _Parked_ section when the brief names none), never a workaround:
|
|
129
|
+
|
|
130
|
+
- changing coverage thresholds, include/exclude lists, or runner config (never `autoUpdate`);
|
|
131
|
+
- deleting or skipping e2e, Playwright, or real-database suites;
|
|
132
|
+
- editing CI config or git hooks;
|
|
133
|
+
- updating snapshots with `-u`;
|
|
134
|
+
- adding any dependency, a mutation-testing tool or a missing coverage provider included (raised in P1, with the three options in [references/campaign.md](references/campaign.md) § P1);
|
|
135
|
+
- fixing a product bug found on the way (it becomes an issue);
|
|
136
|
+
- removing a seam whose removal changes a public contract (`stop-and-escalate`, RCR-007 in `.claude/rules/receiving-review.md`);
|
|
137
|
+
- merging.
|
|
138
|
+
|
|
139
|
+
## Stop and success
|
|
140
|
+
|
|
141
|
+
Stop when: a change you did not make, or a lock, appears in scope (PSA-002); a restore leaves a diff; the baseline does not reproduce; two review cycles pass without fewer open findings (RCR-008: land the smallest safe subset); a lane exceeds its budget.
|
|
142
|
+
|
|
143
|
+
Done when: the gate is green at the MR SHA; flakes and per-file coverage are no worse than M0; every C, D, and F has evidence and a caught mutation (a D without a contract may instead have a `NOCONTRACT` row with ledger evidence); the production diff against `main` outside the test globs and `docs/audits/` is empty, seam commits and owner-approved changes excepted; the preservation review has no open gap; the goal is reached, or the report names why it was not; the MR pipeline has finished and its result stands in the report.
|
|
144
|
+
|
|
145
|
+
## Output
|
|
146
|
+
|
|
147
|
+
- `docs/audits/<date>-test-audit.md` (report) and `docs/audits/<date>-test-audit-ledger.md` (ledger) in the target repo.
|
|
148
|
+
- A draft MR titled `test(audit): <scope> Test-Audit <date>` (subject case per the repo's commitlint config) carrying: M0/M1 table, R/F/C/D/N/B/O counts per lane, the keeper per contract, the mutation table, kept false alarms and why, production and test lines counted separately, the zero-production-diff proof, owner decisions taken and open (every O), and the pipeline result.
|
|
149
|
+
- Commits by the coordinator: one per lane, owner lanes before the cross-cutting lane, seam removal separate, the audit documents separate (commitlint and clone depth: [references/campaign.md](references/campaign.md) § P6, § P9).
|
|
150
|
+
- Follow-up issues: every B, every entry of the ledger's _Issues_ section, every flake, every seam whose removal changes a contract, every N gap not implemented.
|
|
151
|
+
|
|
152
|
+
Inspired by openclaw/openclaw `.agents/skills/test-audit` @ 80930af (MIT, Copyright (c) 2026 OpenClaw Foundation); rewritten for session-orchestrator.
|
|
@@ -0,0 +1,111 @@
|
|
|
1
|
+
# Test-audit campaign
|
|
2
|
+
|
|
3
|
+
A campaign covers a whole repo or subsystem in one draft MR. Everything in [../SKILL.md](../SKILL.md) applies to every lane: the goal, the value bar, the ledger marks, the eight evidence fields, falsification, and the owner list. This file adds the order of work, the measurements, and the mutation runner. Each phase ends on its done-criterion. After every phase, write its numbers into the report file before the next one starts: a session limit can stop the coordinator itself, and a resumed session must be able to continue from the files alone (`git log`, `git status`, the ledgers and manifests under `/tmp/<campaign>/`). Reviewers and lane agents create their result file as a skeleton first and fill it per lane, so a run cut off by a time cap leaves a partial state instead of nothing (one campaign lost a whole P4 round that way). P3 may start as soon as the P1 file list exists and the last CI pipeline at the pinned SHA is green; B marks are reconciled when the M0 pass/fail list lands. P5 and later never start before the P4 owner checkpoint is answered.
|
|
4
|
+
|
|
5
|
+
**Splitting a large campaign.** Above about 25,000 test lines or 15 lanes, P1 to P4 remain one campaign and P5 to P9 run as sub-campaigns with one draft MR each (measured once: 50k lines, 25 lanes; P3 alone took 2 h 46 min at three agents in parallel, and a 3-hour session cap cut P4 in the middle). Sub-campaigns that supply keepers to others go first; the goal applies to the sum.
|
|
6
|
+
|
|
7
|
+
**Briefs.** A campaign brief (the run's instruction from an operator or a coordinating session) wins over this file wherever it is stricter: when it forbids a pre-check or a local run, that stays forbidden. Headless rules: SKILL.md § Never without the owner.
|
|
8
|
+
|
|
9
|
+
## Phases
|
|
10
|
+
|
|
11
|
+
**P1 Baseline (M0).** Pin a SHA. Check the scope against the CI configuration, not against the default test script alone: a suite the pipeline runs under another script (`test:shell-suites`, integration jobs) is in scope and its lines count toward the goal, and one the pipeline never runs is reported as such (measured once: 32 lines under `pnpm test`, 7,850 under a second CI script). Check the coverage provider before anything expensive: a runner that answers `MISSING DEPENDENCY` to `--coverage` (vitest without `@vitest/coverage-v8`) makes the goal's coverage clause and the per-file backstop unmeasurable, and a provider is a dependency, so it is an owner decision. Raise it now with three options: (1) a devDependency pinned to the runner's version, (2) a transient install in the offload worktree only, (3) mutations as the only backstop, named as such in the MR. P1 to P4 continue meanwhile with the coverage clause marked not verifiable; P5 waits for the answer (3 of 6 pilot repos lacked the provider, and it surfaced after P4 each time). Record every measurement below and the pass/fail state of every in-scope test file. Baseline-red tests go on their own list as B: they are possible product bugs, not stale tests.
|
|
12
|
+
Done when every in-scope test file has a recorded result, the coverage provider is confirmed or its owner decision is open, and the M0 table carries its commands and the SHA.
|
|
13
|
+
|
|
14
|
+
**P2 Lanes.** Split the tests along production ownership (the module a test protects), not file-name prefixes. Every test file belongs to exactly one lane; the lane file counts add up to the P1 total. Cross-cutting test files (error-path or contract suites spanning several modules) form their own lane, scheduled after the owner lanes. Budget each lane by lines and declarations: at most about 2,200 test lines or about 150 declarations, whichever comes first (a single unsplittable file may exceed it). Measured over five TypeScript/vitest campaigns (2026-09-24/25): a read-only lane agent takes 10–29 min and 220–500k tokens; Bash suites cost more per line (1,000–1,600 lines, 300–400k). Each lane agent gets one run plus one retry; a third run is the budget overrun that stops the lane (SKILL.md § Stop and success). The declaration count at P2 is a grep estimate; the P3 agent's count is binding, and the runner's own test count stands beside it (a `describe` block is not a declaration, and `numTotalTestSuites` counts blocks, not files). Run at most three agents in parallel: three never hit the account limit (HTTP 429) in five campaigns, five did once. Plan `ceil(lanes / 3)` rounds of about 20 min each.
|
|
15
|
+
Done when the lane table sums to the total.
|
|
16
|
+
|
|
17
|
+
**P3 Ledger per lane.** One read-only `qa-strategist` per lane, in parallel, all from one shared brief. It reads every assigned test in full, including parameter tables, plus the production owner, its callers, history, and CI routing, and writes one mark with evidence per declaration. For a duplicate whose twin lies in another lane, the default keeper is the file the production module owns: owner file beats cross-cutting file, real boundary beats mocked collaborator, exact assertion beats `typeof`. Mark the other side C and name the keeper; never mark both sides C. When the non-owner twin must stay for a contract of its own, the owner side may be C into it only if the twin runs at a boundary at least as real; otherwise both stay R. Cite a cross-lane keeper by its declaration line, never by an assertion line inside another declaration. The cross-cutting lane receives the owner lanes' ledgers as input. `qa-strategist` has Bash: it writes `/tmp/<campaign>/ledger-<lane>.md` itself (heredoc) and answers with the count line, the path, and at most five sentences; the lane brief says that this path and the pre-check copy override the agent definition's report path and read-only Bash rule, and nothing else does. The coordinator assembles `docs/audits/<date>-test-audit-ledger.md`. Suspected product bugs go into the ledger's _Issues_ section; nobody files them yet. Evidence field 8 is written as a patch file under `/tmp/<campaign>/evidence-<lane>/`. A lane agent may pre-check that patch, but only under § Pre-checks; nothing else runs under a mutation before P5.
|
|
18
|
+
Done when every declaration in the lane has a mark and an evidence line.
|
|
19
|
+
|
|
20
|
+
**P4 Layer plan and owner checkpoint.** A different read-only agent (`architect-reviewer`, or a fresh `qa-strategist` that did not write the ledger) takes the ledger as input and looks for redundant layers rather than single tests: which suite is the keeper for each contract, which assertions move into it, which test-only seams become free. One reviewer covers at most eight or nine lanes; above that, several reviewers share one brief and each receives its predecessors' result files. Prefer the real boundary with a fake transport over a mocked collaborator. It re-reads at least 15 % of the C and D declarations against the source, the largest and the weakest-argued first, resolves every cross-lane keeper reference against the owner lane's mark (a keeper must end as R or F; a keeper that is itself C or D, or a wrong line number, invalidates the chain, 16 cases in one campaign), and records every fix in a corrections table (`file:line`, old mark, new mark, evidence); it never edits the lane ledgers in place.
|
|
21
|
+
P4 ends at an owner checkpoint, written into the report: the goal estimate (§ Goal accounting), every O with its line count, the coverage decision from P1 if still open, and the bundled issue list from the ledgers. When the estimate lies far below the goal, the owner chooses between continuing, a repair-and-close goal (F repairs, N tests, issues, no deletion path), or stopping; the threshold for "far below" is not fixed yet. A headless run takes the brief's answer; without one it stops after P4 with the checkpoint in the report. Issues are filed after this checkpoint, bundled by cause, by the session that owns the run unless the brief delegates it. P4 reviewers run no pre-checks.
|
|
22
|
+
Done when each lane names its retired files, its keeper per contract, the assertions to carry over, and the seams unlocked, and the checkpoint is answered.
|
|
23
|
+
|
|
24
|
+
**P5 Keepers and falsification.** One `test-writer` per lane with disjoint file scopes repairs every F, absorbs every C into its keeper, and deletes its C and D, all uncommitted in one pass, and writes the lane's mutation manifest (§ Mutation manifest). Changes to shared helpers and fixtures go through one owner, serialised. Then sync and mutate. A lane is ready when every run row is `CAUGHT` (`REF` and `NOCONTRACT` rows are listed only); a `SURVIVED`, `NOPROOF`, or `APPLYFAIL` row sends it back to its agent. The control run therefore sees the deletions, which is the point: three campaigns ran this order and it saved one agent round per lane. The binding proof is the coordinator's run in the offload worktree (§ Pre-checks for what agents may run on their own).
|
|
25
|
+
Done when every C and F maps to at least one caught mutation id with a restore proof, and every D to one or to a `NOCONTRACT` row with its ledger evidence.
|
|
26
|
+
|
|
27
|
+
**P6 Cutover.** Read the repo's commitlint configuration before the first commit and run commitlint on every lane commit: the repo's config beats the brief and the MR title in SKILL.md (one repo demanded sentence case where the brief said lower case; another had a 100-character subject limit, and the chain had to be rebuilt). Agents never run git write commands (PSA-007); the coordinator commits ready lanes one commit per lane, the owner lanes before the lane that deletes cross-cutting duplicates, so every intermediate commit still carries each contract's proof. Removing a test-only production seam is a separate `code-implementer` task and a separate commit; a D whose evidence field 6 names dead production code (the test is its only caller) lands with or after that removal commit, and a removed code path leaves the coverage denominator instead of counting as a drop.
|
|
28
|
+
Done when every ready lane is committed and each lane's keepers pass.
|
|
29
|
+
|
|
30
|
+
**P7 Preservation review.** Independent, read-only, one reviewer per group of lanes (`session-reviewer`, or `pr-review-toolkit:pr-test-analyzer` when installed). It compares deleted coverage against the keepers and looks for contracts that lost their only proof, for new assertions that cannot fail, and for any production change outside the seam commits. Each restored contract gets a caught mutation. Commit every lane before P7 starts, and give the reviewers the commit list and the current file state: reviewers cite HEAD line numbers and otherwise take the campaign's own edits for another session's, and one brief called files deleted that still existed.
|
|
31
|
+
Done when every reported gap is restored or rejected with source evidence.
|
|
32
|
+
|
|
33
|
+
**P8 Final measurement (M1).** Same commands, same gate, same host as M0.
|
|
34
|
+
Done when the M0/M1 table is complete and per-file coverage is compared.
|
|
35
|
+
|
|
36
|
+
**P9 Hand-off.** Draft MR and follow-up issues as listed in the SKILL.md § Output. Before the push, prove the production change is zero: `git diff --stat origin/main...HEAD -- . ':(exclude)<test globs>' ':(exclude)docs/audits/'` prints nothing except the seam commits and owner-approved dependency or config changes (a coverage provider from P1), each named in the MR. Count the branch's commits against the pipeline's clone depth (`GIT_DEPTH`): commitlint fails on a shallow clone that cannot reach the merge base (one 20-commit branch against depth 20); an overrun goes into the MR as an owner point, never a squash on your own. Then wait for the MR pipeline in the foreground, blocking, and write MR number, pipeline id and job list into the report; the report is the last thing the run produces (two pilots ended their turn while the pipeline still ran and left no final report). A campaign that outlives many `main` commits merges `main` instead of rebasing; when `main` changed a file the campaign deleted, the deletion stays and the new contract moves into the keeper.
|
|
37
|
+
|
|
38
|
+
**Calibration** (one TypeScript/vitest campaign, 2026-09-24; 5.8k test lines, 138 C/D/F marks): 105 mutations covered the 138 marks and took about 3 minutes on the offload host, each a focused `-t` run (81 of them in 63 s). P7 still found 4 lost proofs (2.9 % of the marks) and the per-file backstop 2 more; two later campaigns put P7's yield at 2.9–4.8 % even after a 45 % P4 sample, so neither replaces the other. The whole campaign took about 2 h wall-clock and about 3.2 M subagent tokens. Re-checked with `mutate.sh` at the MR head: control green, 105 of 105 mutations CAUGHT with a red summary, 11 cross-lane `REF` rows.
|
|
39
|
+
|
|
40
|
+
## Goal accounting
|
|
41
|
+
|
|
42
|
+
The 20 % goal counts removed test lines. The estimate after P3 and P4 counts only D, C into another file, and the header lines (imports, setup) of retired files. Reshaping inside the same file (C-same, parametrising a table) is net about zero and is reported on its own line; so is mock ballast without a mark, and so is the F/N growth, which eats into the goal (one campaign: about +430 lines). The O sum stands beside the estimate, never inside it. In three of the six pilot repos (tests:src 0.55–0.80, well inside the TV-003 corridor) the evidence rules held the removable share at 2–5 %; a fourth, at 0.62, estimated 26 % and delivered 12.8 % in its first sub-campaign; the one above the corridor (1.88) reached 31 %. A low ratio predicts a short deletion path but does not decide it; that is what the P4 checkpoint is for.
|
|
43
|
+
|
|
44
|
+
## Pre-checks
|
|
45
|
+
|
|
46
|
+
A pre-check is a mutation run by a lane agent (P3 or P5) on its own, before the coordinator's binding run. One campaign got 90 % of its binding mutations ready-made from lane pre-checks and ran P5 with zero retries; one leaked an in-repo pre-check into a parallel lane for a few seconds; one was halted because its brief forbade pre-checks and the lane prompt never said so. The rules:
|
|
47
|
+
|
|
48
|
+
- Only in a private copy outside the repo: `d=$(mktemp -d /tmp/<campaign>/evidence-<lane>/copy.XXXX) && git archive HEAD | tar -x -C "$d"`; in P5, copy the agent's own uncommitted files over it, since `git archive` exports committed files only. Never in the repo, never in any checkout another agent or the runner uses, never against a `node_modules` symlinked into the repo (a package manager will install through the link).
|
|
49
|
+
- Only for the agent's own patch files, and the result is marked `pre-checked, not binding` in the manifest row or the ledger. The binding proof is the coordinator's `mutate.sh` run in the offload worktree.
|
|
50
|
+
- Artefacts under `/tmp/<campaign>/evidence-<lane>/`, nothing in the campaign root; the coordinator removes the copies (a destructive-command guard blocks `rm -rf` for agents).
|
|
51
|
+
- The lane prompt repeats these rules verbatim and states that they override the agent definition's read-only Bash rule for the copy only; a brief that forbids pre-checks wins.
|
|
52
|
+
|
|
53
|
+
## Measurements
|
|
54
|
+
|
|
55
|
+
Every number carries the command that produced it and the SHA it was measured at. Heavy runs (runtime, flakes, coverage) go to the offload host: `offload run <repo> -H <host> --job <id> -- <command>`, then `--job <id> --no-sync` for the remaining runs on the same worktree (`--rm` removes it only after a successful run).
|
|
56
|
+
|
|
57
|
+
- Job names carry the repo: `ta-<repo>-<purpose>`. Host artefacts live under `<offload-worktree>/.ta-<job>/`, local ones under `/tmp/<campaign>/`, never in the working tree (a `.sh` helper in the tree turned one repo's lint red) and never under a shared host path (two campaigns overwrote each other's M0 under `/tmp/ta-m0` on the same host).
|
|
58
|
+
- A sync deletes untracked files in the job worktree, including `.ta-<job>/`, `mutate.sh`, and copied manifests, but does not reset local changes. So: copy `mutate.sh` and the manifests in after every sync, copy results out before the next one, and after every sync check `git status --porcelain` and that the snapshot SHA changed (one campaign ran two rounds with 42 false `SURVIVED` on a stale worktree). A reference worktree created inside the offload clone falls into the runner's include (770 instead of 277 tests): create it elsewhere or remove it right after the measurement.
|
|
59
|
+
- Measurement and mutation scripts exit non-zero when any run is red, so `--rm` keeps the worktree, and print the per-run summary and the failing test names to stdout: files left in a removed worktree are gone before anyone reads them.
|
|
60
|
+
- The command is not shell-expanded on the host: `~` and `$HOME` resolve locally or not at all. Pass literal remote paths, or wrap in `bash -c '…'`.
|
|
61
|
+
- Record both SHAs the host prints (`SHA=… SNAPSHOT=…`). The snapshot SHA differs from the pinned one; the tracked tree must not (`FILES=` lists only untracked files).
|
|
62
|
+
- `timeout` is not present on every macOS host (rc 127 turns the control run red). Check `command -v timeout || command -v gtimeout` first; otherwise rely on `offload --timeout`.
|
|
63
|
+
- A suite under about 60 s per run may be measured locally instead, sequentially, under `nice -n 10` and `timeout -k 5 600`: for short suites, offload sync and install cost more than the runs (measured once: 13 min waiting for 3.7 min of runs). M0 and M1 still run on the same host. Mutations (P5) always go to the offload worktree.
|
|
64
|
+
|
|
65
|
+
| Measure | How |
|
|
66
|
+
|---|---|
|
|
67
|
+
| Test count | the runner's own count (vitest: `--reporter=json`, field `numTotalTests`; file count `testResults.length`) |
|
|
68
|
+
| Test lines vs production lines | at M0: line totals of the tracked test files the runner actually includes (vitest `include`, pytest `testpaths`, …) and of the rest, counted separately. Tracked test files outside the include are _dead test files_ only when they are neither a setup file, nor imported by an included test, nor a documented manual or live suite; those are listed as _outside include, live_ with their own total. Dead files count neither in M0 nor toward the goal, and their removal is reported on its own line. M0 to M1: `git diff --numstat <M0-sha> <M1-sha>` split by the same globs |
|
|
69
|
+
| Suite runtime | median of 3 full runs on the measuring host |
|
|
70
|
+
| Flakes | 5 runs in shuffled order. A run is flaky when its exit code is non-zero or any test fails, runner-level unhandled errors included (exit 1 with `numFailedTests: 0` is a flake). M1 may not have more than M0. Fake timers next to an unmocked network client (ioredis, fetch) are a known cause |
|
|
71
|
+
| Coverage | total and per production file, on all four metrics (lines, branches, functions, statements); thresholds and config unchanged |
|
|
72
|
+
| tests:src ratio | per TV-003 (`test-value.md`); by hand when the script below does not know the stack |
|
|
73
|
+
|
|
74
|
+
`scripts/lib/tests-src-ratio.mjs` counts JavaScript and TypeScript (`.mjs`/`.js`/`.cjs`/`.jsx`/`.ts`/`.tsx`/`.mts`/`.cts`; `*.d.ts` on neither side). Its test side is a top-level `tests/` plus co-located tests anywhere: `*.test.<ext>`, `*.spec.<ext>`, and files under `__tests__/`. Its `--json` envelope prints that definition; quote it with the number. Two limits remain. A nested test directory (`packages/x/tests/`) whose helpers are not named `*.test.*` counts as src. Other stacks (Python, Swift, Go, …) are invisible to it: the envelope then says `reason: "no-tests-found"` (ratio `null`) or measures only the JS/TS code it happens to find. There, `null` or a low ratio means not measurable, never within the corridor. Compute those by hand with the TV-003 recipe: numerator the physical lines of the tracked test files (`git ls-files '<testdir>/**'`, code files only), denominator every other tracked code file. State both numbers and the globs.
|
|
75
|
+
|
|
76
|
+
Stack recipes (check each flag against the installed version before use):
|
|
77
|
+
|
|
78
|
+
- **TypeScript / vitest:** count `vitest run --reporter=json --outputFile=<dir>/count.json` (without `--outputFile` the JSON mixes into stdout; fields `numTotalTests`, per file `testResults[].status`); flakes the same with `--sequence.shuffle --sequence.seed=$i` for `i` in 1..5; coverage `vitest run --coverage --coverage.reporter=json-summary` → `coverage/coverage-summary.json`. Its keys are absolute paths: strip the worktree prefix before comparing M0 and M1. Thresholds stay enforced, so a red threshold fails the run. For a backstop hit, `--coverage.reporter=json` (`coverage/coverage-final.json`) names the uncovered line, branch, or function.
|
|
79
|
+
- **Python / pytest:** count `--collect-only -q`; coverage `--cov --cov-report=json`; leave `fail_under` as it is. Shuffle only with a plugin that is already installed; otherwise 5 plain runs.
|
|
80
|
+
- **Swift / XCTest:** count `swift test list`; coverage via `swift test --enable-code-coverage` and `xcrun xccov`.
|
|
81
|
+
- **Bash harness:** falsify the harness first (known-broken input, non-zero exit, FAIL in the artefact), under `bash`, then count and time it.
|
|
82
|
+
|
|
83
|
+
## Mutation manifest
|
|
84
|
+
|
|
85
|
+
One directory per lane, `/tmp/<campaign>/mutations/<lane>/`, holding `manifest.tsv` and the patch files it names. One tab-separated line per mutation; `#` lines and an `id` header are skipped. Every row fills all six columns (an empty column shifts the rest left): `REF`, `NOCONTRACT`, and type-check rows put `-` in `testfiles` and `pattern`.
|
|
86
|
+
|
|
87
|
+
| Column | Content |
|
|
88
|
+
|---|---|
|
|
89
|
+
| `id` | unique mutation id, cited by the ledger |
|
|
90
|
+
| `patch` | patch file in the lane directory: `git diff -- <prod-file>` of the mutated tree; `EXTERN:<lane>` when the contract's mutation lives in another lane's manifest (logged as `REF`, not run); or `NOCONTRACT` for a D that removes no contract (dead production code, a duplicate whose keeper is proven outside this campaign, a test-local copy), logged and not run, with the evidence in the ledger; a keeper proven in another lane's manifest is `EXTERN:<lane>` |
|
|
91
|
+
| `testfiles` | space-separated keeper test files that must go red; `-` when the keeper is the whole command (a type check) |
|
|
92
|
+
| `pattern` | the runner's name filter for the keeper (vitest `-t`), or `-` for whole files. It is a regex: escape `.`, `(`, `$`; vitest quotes `it.each` `$name` values, so match them with `.?` around the name |
|
|
93
|
+
| `contract` | one sentence |
|
|
94
|
+
| `ledger_refs` | the marks it proves (`file:line`, comma-separated); every C, D, and F appears in at least one line |
|
|
95
|
+
|
|
96
|
+
[mutate.sh](mutate.sh) runs from the offload worktree root: `bash mutate.sh <mutations-dir>`. The host does not see local `/tmp`, so copy the script and the mutations directory there after every sync and pass literal remote paths (§ Measurements). It runs the whole suite once as control and stops on red (exit 2). Per line it applies the patch, runs the focused test, reverses it, and compares the SHA-256 of the touched files before and after; a mismatch aborts at once (exit 3). Log verdicts: `CAUGHT` needs a non-zero exit and the failure summary; `NOPROOF` is a non-zero exit without it (a compile or collection error proves nothing); `SURVIVED` is a green run under the mutation; `APPLYFAIL` is a patch that does not apply; `REF` and `NOCONTRACT` rows are listed and not run. Any verdict other than `CAUGHT`, or no manifest line at all, ends with exit 1. It keeps `<id>.out` next to each patch for the mutation table. Paths with spaces are not supported.
|
|
97
|
+
|
|
98
|
+
Runner defaults: `MUTATE_CMD` is `pnpm exec vitest run` when a `pnpm-lock.yaml` exists, otherwise `node_modules/.bin/vitest run` (never `npx`, which may download); set it explicitly for anything else. Other stacks set `MUTATE_CMD`, `MUTATE_PATTERN_FLAG`, and `MUTATE_RED_RE` (pytest: `-k`, `[0-9]+ failed`). A type checker as keeper (a mutation that changes a shape a fresh object literal must satisfy): `MUTATE_CMD="timeout -k 5 300 pnpm exec tsgo --noEmit"`, `MUTATE_RED_RE='error TS[0-9]+'`, `testfiles=-`, `pattern=-`, measured with 14 caught mutations in one campaign. tsgo catches only what the type graph sees: not `.mjs` under `allowJs: false`, not runtime schemas (`zod`), so those contracts need a test keeper.
|
|
99
|
+
|
|
100
|
+
## MR body
|
|
101
|
+
|
|
102
|
+
- Goal as stated at the start, and reached or not with the reason; the estimate per § Goal accounting with C-same, ballast, F/N growth and the O sum on their own lines.
|
|
103
|
+
- M0/M1 table with command and SHA per row.
|
|
104
|
+
- R/F/C/D/N/B/O counts per lane.
|
|
105
|
+
- Keeper per contract, and the retired layers.
|
|
106
|
+
- Mutation table: `file:line`, mutation, red test, restore proof; `NOCONTRACT` rows with their ledger evidence.
|
|
107
|
+
- Kept false alarms: tests that looked like junk and stayed, with the contract they guard.
|
|
108
|
+
- Production lines and test lines, separately; dead test files and live files outside the include on their own lines; the zero-production-diff command and its empty output.
|
|
109
|
+
- The P4 corrections table and the ledger's _Issues_ section.
|
|
110
|
+
- Owner decisions taken during the campaign, and the open ones: every O with its line count, so a follow-up MR finds the candidates; the coverage-provider decision when it was open; the commit count against the CI clone depth when it overran.
|
|
111
|
+
- MR number, pipeline id and job list, written after the pipeline finished.
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# Mutation runner for the test-audit skill; manifest format in campaign.md, section "Mutation manifest".
|
|
3
|
+
# Usage: bash mutate.sh <mutations-dir> from the offload worktree root, never a live checkout.
|
|
4
|
+
# Reads <mutations-dir>/<lane>/manifest.tsv (tab-separated):
|
|
5
|
+
# id patch testfiles pattern contract ledger_refs
|
|
6
|
+
# Env: MUTATE_CMD (default vitest via pnpm when pnpm-lock.yaml exists, else node_modules/.bin), MUTATE_PATTERN_FLAG, MUTATE_RED_RE.
|
|
7
|
+
# patch=EXTERN:<lane> and patch=NOCONTRACT rows are listed, never run. testfiles=- and pattern=- run CMD alone.
|
|
8
|
+
# Exit 0 only when the control run is green and every mutation is CAUGHT; 1 otherwise,
|
|
9
|
+
# 2 red control, 3 restore mismatch (aborts at once). CMD and testfiles split on spaces.
|
|
10
|
+
set -u
|
|
11
|
+
MDIR="${1:?usage: mutate.sh <mutations-dir>}"
|
|
12
|
+
if [ -n "${MUTATE_CMD:-}" ]; then CMD="$MUTATE_CMD"; elif [ -f pnpm-lock.yaml ]; then CMD="pnpm exec vitest run"; else CMD="node_modules/.bin/vitest run"; fi # never npx: it may download
|
|
13
|
+
PFLAG="${MUTATE_PATTERN_FLAG:--t}"
|
|
14
|
+
RED_RE="${MUTATE_RED_RE:-Tests +[0-9]+ failed}"
|
|
15
|
+
export NO_COLOR=1; ESC=$(printf '\033') # colour codes are stripped too: a runner may force colour
|
|
16
|
+
sha() { if command -v sha256sum >/dev/null 2>&1; then sha256sum "$@"; else shasum -a 256 "$@"; fi; }
|
|
17
|
+
echo "SHA=$(git rev-parse HEAD) HOST=$(hostname) CMD=[$CMD] START=$(date -u +%FT%TZ)"
|
|
18
|
+
$CMD >"$MDIR/control.out" 2>&1 </dev/null || { echo "CONTROLRED rc=$? see $MDIR/control.out"; exit 2; }
|
|
19
|
+
echo "CONTROL green"
|
|
20
|
+
bad=0; n=0; nc=0
|
|
21
|
+
for M in "$MDIR"/*/manifest.tsv; do
|
|
22
|
+
[ -f "$M" ] || continue
|
|
23
|
+
L=$(dirname "$M"); lane=$(basename "$L")
|
|
24
|
+
while IFS=$'\t' read -r id patch testfiles pattern contract refs || [ -n "${id:-}" ]; do
|
|
25
|
+
case "$id" in (''|\#*|id) continue ;; esac
|
|
26
|
+
case "$patch" in
|
|
27
|
+
(EXTERN:*) echo "REF $lane $id -> ${patch#EXTERN:} refs=$refs"; continue ;;
|
|
28
|
+
(NOCONTRACT) nc=$((nc + 1)); echo "NOCONTRACT $lane $id refs=$refs"; continue ;;
|
|
29
|
+
esac
|
|
30
|
+
n=$((n + 1)); P="$L/$patch"
|
|
31
|
+
files=$(git apply --numstat "$P" 2>/dev/null | awk '{print $3}')
|
|
32
|
+
if [ -z "$files" ] || ! before=$(sha $files) || ! git apply "$P" 2>"$L/$id.apply.err"; then
|
|
33
|
+
echo "APPLYFAIL $lane $id patch=$patch (see $id.apply.err if present)"; bad=1; continue
|
|
34
|
+
fi
|
|
35
|
+
[ "$testfiles" = "-" ] && testfiles=""
|
|
36
|
+
if [ "$pattern" = "-" ]; then $CMD $testfiles; else $CMD $testfiles "$PFLAG" "$pattern"; fi >"$L/$id.out" 2>&1 </dev/null
|
|
37
|
+
rc=$?
|
|
38
|
+
git apply -R "$P" 2>"$L/$id.revert.err"; rrc=$?
|
|
39
|
+
after=$(sha $files)
|
|
40
|
+
if [ "$rrc" -ne 0 ] || [ "$before" != "$after" ]; then
|
|
41
|
+
echo "RESTOREFAIL $lane $id files=[$files] see $L/$id.revert.err"; exit 3
|
|
42
|
+
fi
|
|
43
|
+
red=$(sed "s/$ESC\[[0-9;]*m//g" "$L/$id.out" | grep -E "$RED_RE" | head -n 1 | tr -s ' ')
|
|
44
|
+
if [ "$rc" -ne 0 ] && [ -n "$red" ]; then v=CAUGHT; elif [ "$rc" -ne 0 ]; then v=NOPROOF; bad=1; else v=SURVIVED; bad=1; fi
|
|
45
|
+
echo "$v $lane $id rc=$rc restore=OK [$red] files=[$files] refs=$refs"
|
|
46
|
+
done <"$M"
|
|
47
|
+
done
|
|
48
|
+
[ "$n" -gt 0 ] || [ "$nc" -gt 0 ] || { echo "NOMUTATIONS in $MDIR"; bad=1; }
|
|
49
|
+
echo "END=$(date -u +%FT%TZ) mutations=$n nocontract=$nc"; git status --porcelain
|
|
50
|
+
exit "$bad"
|
|
@@ -32,20 +32,21 @@ Both call sites are conditional: vault-mirror runs only when `vault-integration.
|
|
|
32
32
|
|
|
33
33
|
## Inputs
|
|
34
34
|
|
|
35
|
-
`scripts/vault-mirror.mjs` is the implementation.
|
|
35
|
+
`scripts/vault-mirror.mjs` is the implementation (call it by its plugin path, `"$PLUGIN_ROOT/scripts/vault-mirror.mjs"` — never `cd` into the plugin checkout to run it). `--vault-dir`, `--source` and `--kind` are required; `node "$PLUGIN_ROOT/scripts/vault-mirror.mjs" --help` lists every flag.
|
|
36
36
|
|
|
37
37
|
| Flag | Type | Required | Description |
|
|
38
38
|
|---|---|---|---|
|
|
39
39
|
| `--vault-dir` | path | yes | Absolute path to the Meta-Vault root directory. Must exist. |
|
|
40
|
-
| `--source` | path | yes | Path to the JSONL file to read (one JSON object per line). Must exist. |
|
|
40
|
+
| `--source` | path | yes | Path to the JSONL file to read (one JSON object per line). Must exist. Pass it absolute — a relative path resolves against the cwd. |
|
|
41
41
|
| `--kind` | `session` or `learning` | yes | Determines which generator and target path are used. |
|
|
42
|
+
| `--repo-root` | path | no | The repo whose ledger `--source` is (#1503). Its `.vault.yaml` slug / git remote decides the vault namespace. Default: the project holding `--source` (`<root>/.orchestrator/metrics/<file>` → `<root>`, so a monorepo package with its own `.orchestrator/` keeps its own namespace), else the git repo containing it, else the cwd (only for a loose source outside any git repo, with a stderr `WARN`). Callers pass `--repo-root "$PWD"` from the repo root. |
|
|
42
43
|
| `--dry-run` | flag | no | Parse and resolve paths but do not write any files. Emits action lines as normal. |
|
|
43
44
|
|
|
44
45
|
Empty lines in the JSONL source are silently skipped.
|
|
45
46
|
|
|
46
47
|
## Outputs
|
|
47
48
|
|
|
48
|
-
One JSON line is written to stdout for each non-empty JSONL entry processed. Exit code reflects the run outcome.
|
|
49
|
+
One JSON line is written to stdout for each non-empty JSONL entry processed. Exit code reflects the run outcome. Diagnostics (`SKIP …`, `WARN …`, refusals) go to stderr and are NOT JSON — a caller that captures `2>&1` must parse line by line (`jq -R -r 'fromjson? | objects | …'`), or one `SKIP` line ends the parse and drops every action after it.
|
|
49
50
|
|
|
50
51
|
### Action values
|
|
51
52
|
|
|
@@ -56,6 +57,7 @@ One JSON line is written to stdout for each non-empty JSONL entry processed. Exi
|
|
|
56
57
|
| `skipped-noop` | Entry existed, same id, `updated` date not advanced; file unchanged. |
|
|
57
58
|
| `skipped-handwritten` | A file at the target path has no `_generator` marker (or an unknown generator); left untouched. |
|
|
58
59
|
| `skipped-collision-resolved` | A file at the target path has the generator marker but a different `id`; a disambiguated slug was used instead. |
|
|
60
|
+
| `skipped-foreign-owner` | A generator note at the target path was rendered from another record (#1503), and this write must not replace it. **Sessions:** its `source-record` differs from this write and occurs nowhere in this `--source` ledger (`reason: "source-record mismatch"`), or its `id` names a different session (`reason: "id mismatch"`). **Learnings:** only for the disambiguated `<slug>-<uuid8>.md` file — a mismatch on the main `<slug>.md` is routed to disambiguation instead (`skipped-collision-resolved`, see Idempotency rule 6). One `SKIP foreign owner` line goes to stderr; `--force` does not bypass it. A leak in the foreign note is masked in place — owner, content and `source-record` kept — and the line carries `healed_leak: true`. A note without `source-record` (written before #1503) stays writable. |
|
|
59
61
|
| `skipped-invalid` | Entry is missing one or more required fields; entry skipped, processing continues. |
|
|
60
62
|
| `skipped-quality-low` | Entry failed the quality gate (PRD F1.2): learning `confidence` below `vault-mirror.quality.min-confidence` (CLI: `--quality-min-confidence`, default `0.5`), or session rendered-narrative length below `vault-mirror.quality.min-narrative-chars` (CLI: `--quality-min-narrative-chars`, default `400`). The emitted JSON line includes a `reason` field describing the violated threshold and `path: null` (no file was created). The quality gate runs **before** `--force`; `--force` does not bypass it. |
|
|
61
63
|
|
|
@@ -73,7 +75,7 @@ One JSON line is written to stdout for each non-empty JSONL entry processed. Exi
|
|
|
73
75
|
|---|---|
|
|
74
76
|
| `0` | Success (including idempotent no-ops and per-entry skips). |
|
|
75
77
|
| `1` | Malformed JSON on a JSONL line — fatal, processing stops. Also returned when required CLI args are missing. |
|
|
76
|
-
| `2` | Filesystem error: `--vault-dir` not found, `--source` not found, or an unexpected write error. |
|
|
78
|
+
| `2` | Filesystem error or pre-loop refusal: `--vault-dir` not found, vault not canonical, `--source` not found, `--source` in a different repo than `--repo-root` (`source-repo-mismatch`, #1503), or an unexpected write error. |
|
|
77
79
|
|
|
78
80
|
## Target Paths
|
|
79
81
|
|
|
@@ -86,7 +88,7 @@ Subdirectories are created automatically with `mkdirSync({ recursive: true })` w
|
|
|
86
88
|
|
|
87
89
|
The numeric prefix (`50-sessions/`, `40-learnings/`) follows the vault folder ordering convention so that sessions and learnings appear in the correct position in the vault tree relative to other note types.
|
|
88
90
|
|
|
89
|
-
**Per-project namespacing (#660).** New writes are namespaced under a per-repo subdirectory `<repo>/`, so a single shared vault can hold notes from multiple projects without cross-repo slug/id collisions. `<repo>` is resolved by `resolveRepoNamespace()` (`scripts/lib/vault-mirror/namespace.mjs`): the optional `vault-integration.vault-name` Session Config key (CLI: `--vault-name`) when set, else the
|
|
91
|
+
**Per-project namespacing (#660).** New writes are namespaced under a per-repo subdirectory `<repo>/`, so a single shared vault can hold notes from multiple projects without cross-repo slug/id collisions. `<repo>` is resolved by `resolveRepoNamespace()` (`scripts/lib/vault-mirror/namespace.mjs`): the optional `vault-integration.vault-name` Session Config key (CLI: `--vault-name`) when set, else the repo root's `.vault.yaml` slug, else its preferred git remote via `deriveRepo()`, sanitised to a single kebab segment. The repo root is `--repo-root`, else the project holding `--source` (`<root>/.orchestrator/metrics/`), else the git repo containing it, else the cwd — never the cwd when the source belongs to a project (#1503: running from another repo's checkout once wrote one project's ledger over another project's notes). Owner-privacy leaks (personal home path / private project slug / personal name) are redacted to `redacted-repo` before any write. The legacy **flat** layout (`40-learnings/<slug>.md`) is still read — the writer dual-probes the flat path so a pre-existing flat note is never duplicated; a one-time relocation of the historical flat corpus is tracked as a follow-up.
|
|
90
92
|
|
|
91
93
|
**Path contract note:** Issue #187 shipped the flat numeric-prefix layout (deferring the per-`<repo>` subfolder it sketched). Issue #660 adds that per-repo subfolder for write-isolation (above) while keeping the numeric-prefix ordering. If you need a different layout, file a new issue — do NOT silently change the script.
|
|
92
94
|
|
|
@@ -102,7 +104,8 @@ vault-mirror is safe to run multiple times against the same JSONL source:
|
|
|
102
104
|
2. **File exists, `_generator` marker present, same id, `updated` date not advanced** → `skipped-noop`, file unchanged.
|
|
103
105
|
3. **File exists, `_generator` marker present, same id, `updated` date advanced** → `updated`, file overwritten.
|
|
104
106
|
4. **File exists, no `_generator` marker** → `skipped-handwritten`, file untouched. The absence of the marker means the file was written by a human and must not be overwritten automatically.
|
|
105
|
-
5. **File exists, `_generator` marker present, different id** → slug collision. A disambiguated slug is derived by appending `-<first-8-chars-of-entry-uuid>` (hyphens stripped from the UUID before taking the prefix). The original file is left unchanged; the new note is written at the disambiguated path with action `skipped-collision-resolved`.
|
|
107
|
+
5. **File exists, `_generator` marker present, different id** → slug collision. A disambiguated slug is derived by appending `-<first-8-chars-of-entry-uuid>` (hyphens stripped from the UUID before taking the prefix). The original file is left unchanged; the new note is written at the disambiguated path with action `skipped-collision-resolved`. Sessions have no disambiguation: a different-id session note is `skipped-foreign-owner`.
|
|
108
|
+
6. **File exists, `_generator` marker present, different record** (#1503). Every note records the record it was rendered from in `source-record` (learnings: the record `id`; sessions: `raw_session_id`, else `started_at`). Checked before rules 2–3, because an id match proves nothing — subjects and semantic session ids recur across repos, and an expired learning re-learned in the same repo gets a new id. **Learnings:** handled as the slug collision of rule 5 — the new record lands at `<slug>-<uuid8>.md` (`skipped-collision-resolved` with `reason: "source-record mismatch"`), the existing note is not overwritten. **Sessions:** `skipped-foreign-owner`, unless the note's key occurs on any line of this `--source` ledger (superseded duplicates included — a backfill stub replaced by its real record stays this repo's note). `source-repo` is deliberately not an owner signal: a note at the target path already sits in the writer's namespace folder, so a different `source-repo` only means the note was moved or the repo relabelled. **A leak in a foreign note is masked in place, never rewritten from this record:** the note keeps its owner, content and `source-record`, only the values the current masker knows become `[REDACTED]`, and this record is routed exactly as for any foreign note (the action carries `healed_leak: true`). If the masked frontmatter would no longer parse as YAML (a value that starts with the secret), the note is left unchanged and one stderr `WARN` names it. Rewriting it from this record would be the #1503 overwrite, triggered by a secret. Limitation: this cannot protect a note written before `source-record` existed; the repo-root resolution and the `source-repo-mismatch` refusal are what prevent the incident class.
|
|
106
109
|
|
|
107
110
|
The generator marker value is `session-orchestrator-vault-mirror@1` and appears in the YAML frontmatter as `_generator: session-orchestrator-vault-mirror@1`.
|
|
108
111
|
|
|
@@ -127,31 +130,38 @@ Per `.claude/rules/development.md` Git Safety Protocol — *"Never skip hooks (`
|
|
|
127
130
|
| Malformed JSON on a JSONL line | Error written to stderr, exit 1 (processing stops). |
|
|
128
131
|
| `--vault-dir` not found | Error written to stderr, exit 2. |
|
|
129
132
|
| `--source` file not found | Error written to stderr, exit 2. |
|
|
133
|
+
| `--source` lies in a different repo than `--repo-root` | Refusal written to stderr (both namespaces named), exit 2, nothing written; run event `aborted: "source-repo-mismatch"`. |
|
|
134
|
+
| Session note rendered from a record outside this ledger | `skipped-foreign-owner` emitted on stdout with a `reason`, `SKIP foreign owner` line on stderr, file left unchanged. |
|
|
135
|
+
| `--source` outside any git repo | One `WARN` line on stderr naming the namespace used (it comes from the cwd or an unverifiable `--repo-root`); the run proceeds. |
|
|
130
136
|
| Hand-written file at target path | `skipped-handwritten` emitted on stdout, note written to stderr, file left unchanged. |
|
|
131
137
|
| Unknown `_generator` value in existing file | Treated as hand-written: `skipped-handwritten`, file left unchanged. |
|
|
132
138
|
| Unexpected filesystem write error | Error written to stderr, exit 2. |
|
|
133
139
|
|
|
134
140
|
## Examples
|
|
135
141
|
|
|
136
|
-
Mirror the sessions.jsonl for the current project into a vault:
|
|
142
|
+
Mirror the sessions.jsonl for the current project into a vault — run from the project's repo root, call the script by its plugin path:
|
|
137
143
|
|
|
138
144
|
```bash
|
|
139
|
-
node scripts/vault-mirror.mjs \
|
|
145
|
+
node "$PLUGIN_ROOT/scripts/vault-mirror.mjs" \
|
|
140
146
|
--vault-dir ~/Projects/vault \
|
|
141
|
-
--
|
|
147
|
+
--repo-root "$PWD" \
|
|
148
|
+
--source "$PWD/.orchestrator/metrics/sessions.jsonl" \
|
|
142
149
|
--kind session
|
|
143
150
|
```
|
|
144
151
|
|
|
145
152
|
Dry-run a learnings mirror to preview actions without writing:
|
|
146
153
|
|
|
147
154
|
```bash
|
|
148
|
-
node scripts/vault-mirror.mjs \
|
|
155
|
+
node "$PLUGIN_ROOT/scripts/vault-mirror.mjs" \
|
|
149
156
|
--vault-dir ~/Projects/vault \
|
|
150
|
-
--
|
|
157
|
+
--repo-root "$PWD" \
|
|
158
|
+
--source "$PWD/.orchestrator/metrics/learnings.jsonl" \
|
|
151
159
|
--kind learning \
|
|
152
160
|
--dry-run
|
|
153
161
|
```
|
|
154
162
|
|
|
163
|
+
Never `cd` into the plugin checkout to run it: the namespace comes from the repo the ledger belongs to, and an explicit `--repo-root` that disagrees with `--source` is refused with exit 2 (`source-repo-mismatch`).
|
|
164
|
+
|
|
155
165
|
## Live State
|
|
156
166
|
|
|
157
167
|
**Phase 1 shipped and actively running.** As of 2026-05-09, the skill has produced:
|
|
@@ -93,7 +93,7 @@ Before starting the first wave (Discovery role):
|
|
|
93
93
|
|
|
94
94
|
**Execution Config shortcut:** If the session-plan output contains an `### Execution Config` section, its execution-level fields (waves, agents-per-wave, isolation, enforcement, max-turns) take precedence over `$CONFIG`. Session-level fields (persistence, pencil) always come from `$CONFIG`. If the Execution Config section is missing, use `$CONFIG` alone.
|
|
95
95
|
6. **Initialize session metrics** (if `persistence` enabled): Prepare a metrics tracking object for this session:
|
|
96
|
-
- `session_id`: `<branch>-<YYYY-MM-DD>-<HHmm
|
|
96
|
+
- `session_id`: the STATE.md frontmatter `session` value, verbatim (the semantic label, = `semantic_session_id` in `.orchestrator/current-session.json`) — never a constructed `<branch>-<YYYY-MM-DD>-<HHmm>`; `scripts/emit-session.mjs` attaches the raw UUID and token rollup only when the labels match
|
|
97
97
|
- `session_type`: from Session Config
|
|
98
98
|
- `started_at`: ISO 8601 timestamp
|
|
99
99
|
- `waves`: empty array (populated after each wave)
|
|
@@ -230,7 +230,7 @@ MUST prefix with `SO_WAVE_AGENT=1` — without it the CLI returns exit 3 `reject
|
|
|
230
230
|
|
|
231
231
|
Exit code 0 = queued (the coordinator will present at session-end via AskUserQuestion); 1 = quota-exceeded; 2 = rejected-low-confidence (below floor 0.5); 3 = rejected-wrong-context (STATE.md not active OR SO_WAVE_AGENT != "1"); 4 = error (arg validation or internal).
|
|
232
232
|
|
|
233
|
-
Use ONLY when you find a recurring pattern, anti-pattern, or constraint worth carrying into future sessions. The coordinator confirms each proposal before it lands in learnings.jsonl. Do NOT over-propose — quota is
|
|
233
|
+
Use ONLY when you find a recurring pattern, anti-pattern, or constraint worth carrying into future sessions. The coordinator confirms each proposal before it lands in learnings.jsonl. Do NOT over-propose — the quota is shared by every agent of the wave: once the wave holds `quota-per-wave` proposals (default 5), the next one exits 1 whichever agent sends it.
|
|
234
234
|
|
|
235
235
|
Analyzer-only learning types, including `autonomy-verdict`, are intentionally not valid here; those are emitted by `/evolve` after their analyzer-specific evidence gates pass.
|
|
236
236
|
```
|
|
@@ -97,7 +97,7 @@ The function never throws — it always returns a result object. Treat `skipped:
|
|
|
97
97
|
|
|
98
98
|
Rationale: the verified learning `coordinator-over-worktree-on-shared-files` (confidence 0.75) shows that small waves on partitioned scopes merge cleaner when run in-place. Two consecutive deep-session regressions (2026-04-20 07:30, 09:00) were worktree base-ref staleness on ≤2-agent waves editing the same SKILL.md. Graduated default makes worktree the tool for parallelism, not the default tax on every wave.
|
|
99
99
|
|
|
100
|
-
1a. **After ANY commit made during this session, run the wave IN-PLACE** (omit `isolation`) regardless of what the agent-count table above resolves
|
|
100
|
+
1a. **After ANY commit made during this session, run the wave IN-PLACE** (omit `isolation`) regardless of what the agent-count table above resolves. The harness picks the agent worktree's base — the `Agent` payload has no base-ref field: `origin/HEAD` (origin's default branch) unless `worktree.baseRef: "head"` is set, then the local HEAD; this repo sets `"head"` in `.claude/settings.json`. Under the default, a dispatch from a HEAD that origin's default branch lacks hands the agent the OLD code and its test run is structurally red on top; under either base, uncommitted work reaches no worktree. The measurement, the fix-agent case and the base-verification one-liner are in `.claude/rules/review-and-adapter-contracts.md` § "Ein Review-Panel im frischen Worktree prueft den ALTEN Code"; `hooks/pre-task-scope-disjoint.mjs` DENIES a worktree dispatch whose base it measures as missing commits of HEAD, naming the remedies, and allows + records one it cannot determine (#1485). In-place then auto-promotes `warn` → `strict` via step 2, which is the intended trade: the scope hook becomes the barrier the worktree no longer is.
|
|
101
101
|
|
|
102
102
|
2. **Enforcement auto-promote (#194)**: Call `resolveEnforcement({ isolation, configEnforcement })` from the same module. When isolation resolves to `none` and the user has not explicitly set `configEnforcement: 'off'`, enforcement auto-promotes from `warn` → `strict`. Worktrees provide filesystem-level isolation; in-place dispatch relies on the scope hook as the only barrier — it must be hard, not informational. Write the resolved value into `wave-scope.json` `enforcement`.
|
|
103
103
|
|
|
@@ -21,6 +21,7 @@ issues: [<issue numbers from plan>]
|
|
|
21
21
|
session: <semantic session label, e.g. main-2026-01-01-deep-1>
|
|
22
22
|
session-id: <raw session_id from .orchestrator/session.lock — OMIT the key when no lock exists>
|
|
23
23
|
started_at: <resolveSessionStartedAt({ repoRoot }) — the lock's own started_at, NEVER new Date() at write time (#1368)>
|
|
24
|
+
session-start-ref: <full sha from git rev-parse HEAD at init>
|
|
24
25
|
status: active
|
|
25
26
|
current-wave: 0
|
|
26
27
|
total-waves: <from session plan>
|
|
@@ -51,6 +52,8 @@ node -e "import('./scripts/lib/state-md.mjs').then(m => console.log(JSON.stringi
|
|
|
51
52
|
|
|
52
53
|
`session-id` is the raw/native harness id (`resolveSessionIds().session_id`), `session` the semantic label (`.semantic_session_id`). Both are OPTIONAL (`schema-version` stays `1`): when the lock yields `null`, OMIT the key entirely — never write a placeholder. Neither field grants lock or registry ownership; see `skills/_shared/state-ownership.md`.
|
|
53
54
|
|
|
55
|
+
`session-start-ref` is the commit the session started on (`git rev-parse HEAD`, full SHA); `scripts/emit-session.mjs` derives the ledger's `session_start_ref` from it and, when the key is absent, falls back to the `head_sha` of this session's own `orchestrator.session.started` event (#1443). `hooks/pre-task-scope-disjoint.mjs` no longer reads it: since #1485 its stale-worktree check on `isolation: "worktree"` dispatches compares HEAD against the base the harness actually branches from (`origin/HEAD` unless `worktree.baseRef: "head"`, then HEAD), and DENIES a measured mismatch. The other reader is `scripts/lib/scope-baseline.mjs` (`readBaseline()` reads the key into `sessionStartRef`, `computeDrift()` consumes it): with the key present its numerator is the union of the committed `<ref>..HEAD` diff, the working tree vs HEAD (staged + unstaged) and the untracked files (#1027 Pkt 2), all in the coordinator's own checkout; absent, it resolves a base itself (the preferred remote's default branch, else a local `main`/`master`; nothing resolvable skips with `no-baseline-ref`); a present-but-dead ref skips with `unresolvable-ref`, and so does any failed, oversized or timed-out numerator query. All three git queries run with `cwd` = repoRoot, so they never read an agent worktree's contents (the worktree directory itself is excluded from the untracked list). The ref bounds only the committed source, which gives the ratio two edges in opposite directions. Under-report: this template takes HEAD at wave-executor init rather than at session start, so files changed by a commit landed between session start and wave-executor init are missing from the committed source. Over-report: the working-tree and untracked sources are not tied to the ref at all, so files already dirty when the session started and a same-checkout peer session's files count as drift and can raise a false WARN. That second edge is a named ceiling in `computeDrift()`'s JSDoc; removing it needs a start-dirty snapshot persisted at freeze time, a storage change outside this template.
|
|
56
|
+
|
|
54
57
|
Create the `<state-dir>` directory if needed (`mkdir -p <state-dir>`) before writing. This file is the persistent state record — other skills and resumed sessions read it.
|
|
55
58
|
|
|
56
59
|
**Then VALIDATE `total-waves` against the resolved shape — do not skip this.** A plan whose wave count the shape does not produce must never be dispatched silently:
|
|
@@ -89,9 +89,11 @@ const gate = await evaluateWaveResourceGate({
|
|
|
89
89
|
|
|
90
90
|
Passing neither is a valid choice, not a bug: the gate then behaves exactly as it did before #1160 and reduces locally.
|
|
91
91
|
|
|
92
|
+
With `offload-first: true` (#1465) the witness is mandatory for the switch to act: the gate may then place an offloadable role even on `proceed`, but without `remoteReady`/`probeFn` everything stays local exactly as before. In that case `gate.reasons` carries an extra `offload-first: … — staying local` line naming why — include it in the wave progress update's "Resource gate:" bullet like any other reason.
|
|
93
|
+
|
|
92
94
|
Reasons MUST appear in the wave's progress update under a "Resource gate:" bullet. Measurements (RAM free GB, CPU %, concurrent sessions) appear verbatim so the user can trust the decision.
|
|
93
95
|
|
|
94
|
-
Probe failures never block a wave — the gate returns `proceed` with a "probe failed (ignored)" reason and the wave continues at the planned count. A config without `resource-thresholds` (legacy pre-#166) returns `proceed` with `"resource-thresholds missing from config — gate skipped"` — a defensive fallback so the gate never crashes the dispatch loop.
|
|
96
|
+
Probe failures never block a wave — the gate returns `proceed` with a "probe failed (ignored)" reason and the wave continues at the planned count (with `offload-first: true` that `proceed` still goes through placement, so the answer can be `offload` with the same reason kept). A config without `resource-thresholds` (legacy pre-#166) returns `proceed` with `"resource-thresholds missing from config — gate skipped"` — a defensive fallback so the gate never crashes the dispatch loop.
|
|
95
97
|
|
|
96
98
|
**STATE.md deviation contract (#193):** when the gate returns `reduce` or `coordinator-direct`, append a single timestamped entry to `## Deviations` in `<state-dir>/STATE.md`. Use this exact format so future sessions and the evolve skill can mine for hardware-pattern learnings:
|
|
97
99
|
|
|
@@ -425,6 +427,14 @@ The helper is stdlib-only and cross-platform. `mode=block` is recommended when t
|
|
|
425
427
|
|
|
426
428
|
This is a downstream backstop: the underlying worktree merge-back strategy lives in the Claude Code harness and is outside this plugin's control. The correct fix (preserve untracked coordinator files during merge-back) must come upstream. Until then, this check is the only defense.
|
|
427
429
|
|
|
430
|
+
#### Worktree Base Check (#1485)
|
|
431
|
+
|
|
432
|
+
The harness, not this plugin, picks the commit an `isolation: "worktree"` agent branches from. Per the Claude Code docs (code.claude.com/docs/en/worktrees § "Choose the base branch", read 2026-10-02), subagent worktrees follow `worktree.baseRef`: `"fresh"` (the default) branches from `origin/HEAD`, `"head"` from the current local HEAD. On a feature branch with commits the default branch on origin lacks, `"fresh"` hands the agent OLD code — #1485 measured worktrees on `main`, six commits behind the branch HEAD.
|
|
433
|
+
|
|
434
|
+
- **Root cause, removed here:** this repo's `.claude/settings.json` sets `"worktree": {"baseRef": "head"}`. Uncommitted coordinator work still reaches no worktree under either base — a wave that builds on uncommitted predecessor results runs in-place.
|
|
435
|
+
- **Mechanical guard:** `hooks/pre-task-scope-disjoint.mjs` checks every `isolation: "worktree"` dispatch before it runs. It reads `worktree.baseRef` in the documented precedence order — `.claude/settings.local.json`, then `.claude/settings.json`, then user settings; the first file that sets it decides — at the settings root (`$CLAUDE_PROJECT_DIR`, else the git toplevel), not at a `cwd` the session has `cd`-ed into. When that selects `"fresh"`, it runs `git merge-base --is-ancestor HEAD origin/HEAD` in `cwd`'s repo; exit 1 (the worktree would miss commits) DENIES the dispatch, naming the deciding settings file and the remedies: re-dispatch in-place (omit `isolation`), set `baseRef: "head"`, or get HEAD into origin's default branch. A denied dispatch is not written to the scope ledger, so the in-place re-dispatch does not collide with it. Anything it cannot measure (no cached `origin/HEAD`, a shallow clone, a git error, or a would-be deny while `FETCH_HEAD` is older than 24 h or absent — the harness then fetches the default branch before branching, so the cached ref is not the real base: `skipped: 'stale-remote-ref'`) is allowed. Every worktree dispatch leaves one `orchestrator.wave_dispatch.worktree_base_checked` record carrying `decision` and `base_ref_source` (`docs/events-schema.md`).
|
|
436
|
+
- **Blind spots:** managed settings and the `--settings` flag outrank the files above and are not read — a `"head"` set only there makes the hook see `"fresh"` and can DENY wrongly (that deny names "the default" as its source, `base_ref_source: "default"`); re-dispatch in-place or mirror the value into `.claude/settings.local.json`. Two rarer divergences remain: a repo without an `origin/HEAD` symref (the harness falls back to `origin/main|master`, the hook skips), and a stale fetch whose harness refetch then fails offline.
|
|
437
|
+
|
|
428
438
|
#### Pre-Dispatch Coordinator Snapshot (#196)
|
|
429
439
|
|
|
430
440
|
Before dispatching agents for this wave, checkpoint any uncommitted coordinator work as a git stash snapshot. This is a backup — it does NOT touch the working tree and does NOT block dispatch on failure.
|