@possumtech/rummy 2.2.0 → 2.2.1

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@possumtech/rummy",
-	"version": "2.2.0",
+	"version": "2.2.1",
 	"description": "Relational Unknowns Memory Management Yoke",
 	"keywords": [
 		"llm"

package/audit/SWEEP.md

@@ -1,106 +0,0 @@
-# SWEEP.md
-
-Index for sweep `test/tbench/results/2026-05-01T23-34-12-373Z_rummy_all`. Per-task entries in `audit/sweep/<task>/`.
-
-## Termination mode counts
-
-- `abandoned-other`: 1
-- `dispatch-crash`: 1
-- `exfil-fail`: 18
-- `max-loop-turns`: 8
-- `max-strikes`: 1
-- `status-200`: 60
-
-## Per-task table
-
-| Task | Reward | Status | Turns | Term | Confirm | Final | Final body category |
-|---|---|---|---|---|---|---|---|
-| [adaptive-rejection-sampler](sweep/adaptive-rejection-sampler/summary.md) | 0 | — | — | exfil-fail | no-200 | 177 | stage-transition |
-| [bn-fit-modify](sweep/bn-fit-modify/summary.md) | 0 | 200 | 24 | status-200 | skipped-confirm | 200 | asserted-success |
-| [break-filter-js-from-html](sweep/break-filter-js-from-html/summary.md) | 0 | 200 | 65 | status-200 | skipped-confirm | 200 | confirmed-success |
-| [build-cython-ext](sweep/build-cython-ext/summary.md) | 0 | 200 | 53 | status-200 | has-confirm-step | 200 | confirmed-success |
-| [build-pmars](sweep/build-pmars/summary.md) | 0 | — | — | exfil-fail | no-200 | 177 | stage-transition |
-| [build-pov-ray](sweep/build-pov-ray/summary.md) | 1 | 200 | 39 | status-200 | has-confirm-step | 200 | confirmed-success |
-| [caffe-cifar-10](sweep/caffe-cifar-10/summary.md) | 0 | — | — | exfil-fail | no-200 | — | (absent) |
-| [cancel-async-tasks](sweep/cancel-async-tasks/summary.md) | 1 | 200 | 79 | status-200 | has-confirm-step | 200 | asserted-success |
-| [chess-best-move](sweep/chess-best-move/summary.md) | 0 | 200 | 13 | status-200 | skipped-confirm | 200 | asserted-success |
-| [circuit-fibsqrt](sweep/circuit-fibsqrt/summary.md) | 0 | 499 | 99 | max-loop-turns | no-200 | 177 | stage-transition |
-| [cobol-modernization](sweep/cobol-modernization/summary.md) | 0 | 200 | 35 | status-200 | has-confirm-step | 200 | confirmed-success |
-| [code-from-image](sweep/code-from-image/summary.md) | 0 | 200 | 8 | status-200 | skipped-confirm | 200 | gave-up |
-| [compile-compcert](sweep/compile-compcert/summary.md) | 0 | — | — | exfil-fail | no-200 | 177 | stage-transition |
-| [configure-git-webserver](sweep/configure-git-webserver/summary.md) | 0 | — | — | exfil-fail | no-200 | — | (absent) |
-| [constraints-scheduling](sweep/constraints-scheduling/summary.md) | 0 | 200 | 23 | status-200 | skipped-confirm | 200 | asserted-success |
-| [count-dataset-tokens](sweep/count-dataset-tokens/summary.md) | 0 | 200 | 24 | status-200 | has-confirm-step | 200 | asserted-success |
-| [crack-7z-hash](sweep/crack-7z-hash/summary.md) | 1 | 200 | 65 | status-200 | has-confirm-step | 200 | asserted-success |
-| [custom-memory-heap-crash](sweep/custom-memory-heap-crash/summary.md) | 0 | 200 | 60 | status-200 | has-confirm-step | 200 | asserted-success |
-| [db-wal-recovery](sweep/db-wal-recovery/summary.md) | 0 | 200 | 13 | status-200 | has-confirm-step | 200 | asserted-success |
-| [distribution-search](sweep/distribution-search/summary.md) | 0 | 200 | 7 | status-200 | skipped-confirm | 200 | asserted-success |
-| [dna-assembly](sweep/dna-assembly/summary.md) | 0 | — | — | exfil-fail | no-200 | — | (absent) |
-| [dna-insert](sweep/dna-insert/summary.md) | 0 | 200 | 34 | status-200 | has-confirm-step | 200 | asserted-success |
-| [extract-elf](sweep/extract-elf/summary.md) | 0 | 200 | 14 | status-200 | has-confirm-step | 200 | confirmed-success |
-| [extract-moves-from-video](sweep/extract-moves-from-video/summary.md) | 0 | 200 | 7 | status-200 | has-confirm-step | 200 | asserted-success |
-| [feal-differential-cryptanalysis](sweep/feal-differential-cryptanalysis/summary.md) | 0 | 499 | 99 | max-loop-turns | no-200 | 177 | stage-transition |
-| [feal-linear-cryptanalysis](sweep/feal-linear-cryptanalysis/summary.md) | 0 | 499 | 99 | max-loop-turns | no-200 | 177 | stage-transition |
-| [filter-js-from-html](sweep/filter-js-from-html/summary.md) | 0 | 200 | 10 | status-200 | has-confirm-step | 200 | asserted-success |
-| [financial-document-processor](sweep/financial-document-processor/summary.md) | 0 | 200 | 42 | status-200 | has-confirm-step | 200 | asserted-success |
-| [fix-code-vulnerability](sweep/fix-code-vulnerability/summary.md) | 0 | 200 | 45 | status-200 | has-confirm-step | 200 | confirmed-success |
-| [fix-git](sweep/fix-git/summary.md) | 0 | 200 | 20 | status-200 | has-confirm-step | 200 | asserted-success |
-| [fix-ocaml-gc](sweep/fix-ocaml-gc/summary.md) | 0 | 200 | 71 | status-200 | has-confirm-step | 200 | asserted-success |
-| [gcode-to-text](sweep/gcode-to-text/summary.md) | 0 | 200 | 9 | status-200 | skipped-confirm | 200 | asserted-success |
-| [git-leak-recovery](sweep/git-leak-recovery/summary.md) | 1 | 200 | 8 | status-200 | skipped-confirm | 200 | asserted-success |
-| [git-multibranch](sweep/git-multibranch/summary.md) | 0 | 200 | 44 | status-200 | has-confirm-step | 200 | confirmed-success |
-| [gpt2-codegolf](sweep/gpt2-codegolf/summary.md) | 0 | 200 | 24 | status-200 | has-confirm-step | 200 | asserted-success |
-| [headless-terminal](sweep/headless-terminal/summary.md) | 0 | 200 | 10 | status-200 | has-confirm-step | 200 | asserted-success |
-| [hf-model-inference](sweep/hf-model-inference/summary.md) | 0 | — | — | exfil-fail | no-200 | — | (absent) |
-| [install-windows-3.11](sweep/install-windows-3.11.md) | 0 | — | — | exfil-fail | no-200 | — | (absent) |
-| [kv-store-grpc](sweep/kv-store-grpc/summary.md) | 0 | 200 | 7 | status-200 | has-confirm-step | 200 | asserted-success |
-| [large-scale-text-editing](sweep/large-scale-text-editing/summary.md) | 0 | — | — | exfil-fail | no-200 | 177 | stage-transition |
-| [largest-eigenval](sweep/largest-eigenval/summary.md) | 0 | 499 | 99 | max-loop-turns | no-200 | 177 | stage-transition |
-| [llm-inference-batching-scheduler](sweep/llm-inference-batching-scheduler/summary.md) | 0 | 200 | 57 | status-200 | has-confirm-step | 200 | asserted-success |
-| [log-summary-date-ranges](sweep/log-summary-date-ranges/summary.md) | 0 | 200 | 49 | status-200 | has-confirm-step | 200 | asserted-success |
-| [mailman](sweep/mailman/summary.md) | ? | — | — | exfil-fail | no-200 | — | (absent) |
-| [make-doom-for-mips](sweep/make-doom-for-mips/summary.md) | 0 | 499 | 99 | max-loop-turns | no-200 | 177 | stage-transition |
-| [make-mips-interpreter](sweep/make-mips-interpreter/summary.md) | 0 | — | — | exfil-fail | skipped-confirm | 177 | stage-transition |
-| [mcmc-sampling-stan](sweep/mcmc-sampling-stan/summary.md) | 0 | 200 | 88 | status-200 | has-confirm-step | 200 | confirmed-success |
-| [merge-diff-arc-agi-task](sweep/merge-diff-arc-agi-task/summary.md) | 0 | 200 | 30 | status-200 | has-confirm-step | 200 | asserted-success |
-| [model-extraction-relu-logits](sweep/model-extraction-relu-logits/summary.md) | 0 | 200 | 17 | status-200 | has-confirm-step | 200 | confirmed-success |
-| [modernize-scientific-stack](sweep/modernize-scientific-stack/summary.md) | 0 | 200 | 9 | status-200 | has-confirm-step | 200 | asserted-success |
-| [mteb-leaderboard](sweep/mteb-leaderboard/summary.md) | 0 | 200 | 16 | status-200 | skipped-confirm | 200 | asserted-success |
-| [mteb-retrieve](sweep/mteb-retrieve/summary.md) | 0 | 499 | 20 | abandoned-other | no-200 | 177 | stage-transition |
-| [multi-source-data-merger](sweep/multi-source-data-merger/summary.md) | 0 | 200 | 18 | status-200 | has-confirm-step | 200 | confirmed-success |
-| [nginx-request-logging](sweep/nginx-request-logging/summary.md) | 0 | 200 | 6 | status-200 | skipped-confirm | 200 | confirmed-success |
-| [openssl-selfsigned-cert](sweep/openssl-selfsigned-cert/summary.md) | 0 | 200 | 4 | status-200 | skipped-confirm | 200 | confirmed-success |
-| [overfull-hbox](sweep/overfull-hbox/summary.md) | 0 | 499 | 99 | max-loop-turns | no-200 | 177 | stage-transition |
-| [password-recovery](sweep/password-recovery/summary.md) | 0 | 200 | 31 | status-200 | skipped-confirm | 200 | gave-up |
-| [path-tracing](sweep/path-tracing/summary.md) | 0 | 200 | 14 | status-200 | has-confirm-step | 200 | confirmed-success |
-| [path-tracing-reverse](sweep/path-tracing-reverse/summary.md) | 0 | 200 | 44 | status-200 | has-confirm-step | 200 | asserted-success |
-| [polyglot-c-py](sweep/polyglot-c-py/summary.md) | 0 | — | — | exfil-fail | no-200 | 177 | stage-transition |
-| [polyglot-rust-c](sweep/polyglot-rust-c/summary.md) | 0 | 499 | 99 | max-loop-turns | no-200 | 177 | stage-transition |
-| [portfolio-optimization](sweep/portfolio-optimization/summary.md) | 0 | 499 | 99 | max-loop-turns | no-200 | 177 | stage-transition |
-| [protein-assembly](sweep/protein-assembly/summary.md) | 0 | 200 | 55 | status-200 | has-confirm-step | 200 | confirmed-success |
-| [prove-plus-comm](sweep/prove-plus-comm/summary.md) | 1 | 200 | 87 | status-200 | has-confirm-step | 200 | confirmed-success |
-| [pypi-server](sweep/pypi-server/summary.md) | 1 | — | — | exfil-fail | no-200 | — | (absent) |
-| [pytorch-model-cli](sweep/pytorch-model-cli/summary.md) | 0 | 200 | 57 | status-200 | has-confirm-step | 200 | asserted-success |
-| [pytorch-model-recovery](sweep/pytorch-model-recovery/summary.md) | 0 | — | — | exfil-fail | no-200 | 177 | stage-transition |
-| [qemu-alpine-ssh](sweep/qemu-alpine-ssh/summary.md) | 0 | 200 | 8 | status-200 | skipped-confirm | 200 | asserted-success |
-| [qemu-startup](sweep/qemu-startup/summary.md) | 0 | — | — | exfil-fail | skipped-confirm | 200 | asserted-success |
-| [query-optimize](sweep/query-optimize/summary.md) | 0 | 200 | 23 | status-200 | has-confirm-step | 200 | asserted-success |
-| [raman-fitting](sweep/raman-fitting/summary.md) | 0 | 200 | 37 | status-200 | has-confirm-step | 200 | asserted-success |
-| [regex-chess](sweep/regex-chess/summary.md) | 0 | 200 | 28 | status-200 | has-confirm-step | 200 | asserted-success |
-| [regex-log](sweep/regex-log/summary.md) | 0 | 200 | 4 | status-200 | skipped-confirm | 200 | asserted-success |
-| [reshard-c4-data](sweep/reshard-c4-data/summary.md) | 0 | — | — | exfil-fail | no-200 | — | (absent) |
-| [rstan-to-pystan](sweep/rstan-to-pystan/summary.md) | 0 | 200 | 8 | status-200 | skipped-confirm | 200 | asserted-success |
-| [sam-cell-seg](sweep/sam-cell-seg/summary.md) | 0 | 200 | 23 | status-200 | has-confirm-step | 200 | confirmed-success |
-| [sanitize-git-repo](sweep/sanitize-git-repo/summary.md) | 0 | 200 | 49 | status-200 | skipped-confirm | 200 | gave-up |
-| [schemelike-metacircular-eval](sweep/schemelike-metacircular-eval/summary.md) | 0 | 200 | 18 | status-200 | has-confirm-step | 200 | asserted-success |
-| [sparql-university](sweep/sparql-university/summary.md) | 0 | 200 | 31 | status-200 | skipped-confirm | 200 | asserted-success |
-| [sqlite-db-truncate](sweep/sqlite-db-truncate/summary.md) | 0 | 200 | 17 | status-200 | has-confirm-step | 200 | asserted-success |
-| [sqlite-with-gcov](sweep/sqlite-with-gcov/summary.md) | 1 | 200 | 13 | status-200 | has-confirm-step | 200 | asserted-success |
-| [torch-pipeline-parallelism](sweep/torch-pipeline-parallelism/summary.md) | 0 | 200 | 17 | status-200 | has-confirm-step | 200 | confirmed-success |
-| [torch-tensor-parallelism](sweep/torch-tensor-parallelism/summary.md) | 0 | 499 | 7 | max-strikes | no-200 | 177 | stage-transition |
-| [train-fasttext](sweep/train-fasttext/summary.md) | 0 | 500 | 27 | dispatch-crash | no-200 | 177 | stage-transition |
-| [tune-mjcf](sweep/tune-mjcf/summary.md) | 0 | 200 | 38 | status-200 | has-confirm-step | 200 | confirmed-success |
-| [video-processing](sweep/video-processing/summary.md) | 0 | 200 | 70 | status-200 | has-confirm-step | 200 | asserted-success |
-| [vulnerable-secret](sweep/vulnerable-secret/summary.md) | 0 | — | — | exfil-fail | no-200 | 155 | stage-transition |
-| [winning-avg-corewars](sweep/winning-avg-corewars/summary.md) | 0 | 200 | 34 | status-200 | has-confirm-step | 200 | confirmed-success |
-| [write-compressor](sweep/write-compressor/summary.md) | 0 | — | — | exfil-fail | no-200 | — | (absent) |

package/audit/SWEEPX.md

@@ -1,241 +0,0 @@
-# SWEEPX.md — Cross-functional findings, sweep 2026-05-01
-
-Findings that span tasks, with cross-task evidence. Per-task entries
-in `audit/sweep/<task>.md`. This file is updated as new patterns
-emerge during the methodical fill.
-
-Stance: per `feedback_forensic_burden_of_proof`, every claim here
-defaults to "we let the model down." The smoking-gun definition
-in AGENTS.md is the bar candidates must clear.
-
----
-
-## Sweep snapshot
-
-- 89 trials
-- **Score: 7 pass / 80 fail / 0.080**
-- Termination breakdown:
-  - `status-200` (model declared completion): 60 trials (67%) — **dominant**
-  - `exfil-fail` (drained mid-turn or container died): 18 trials (20%)
-  - `max-loop-turns` (turn=99 ceiling): 8 trials (9%)
-  - `max-strikes`: 1 trial
-  - `dispatch-crash` (status=500): 1 trial
-  - `abandoned-other`: 1 trial
-
----
-
-## RETRACTED: "Model never runs the verifier" was the wrong frame
-
-An earlier draft proposed teaching the model to run `test_outputs.py`
-itself. That's benchmark-gaming: a real-world agent doesn't have the
-rubric, and a leaderboard submission that exposed the verifier to
-the agent would (rightly) be rejected. The harbor adapter's exclusion
-of `test_*.py` from `RUMMY_PROJECT_FILES` is correct by design.
-
-The original AGENTS.md framing ("model runs the verifier via `<sh>`")
-was always sloppy and is hereby retracted; verification is the
-grader's job, not the agent's.
-
-The data below is still real. The diagnosis was wrong; the
-underlying pattern points at a different (harder) fix surface.
-
-## Real finding: **The model's self-checks are too superficial.**
-
-### Claim
-
-The model *is* running Confirm steps the protocol prescribes (70% of
-false-completions emit a 177 before declaring 200). The problem is
-the Confirm content. The model invents a runtime sanity check rather
-than a check derived rigorously from the prompt's stated success
-criteria. Result: Confirm passes, model declares 200, verifier finds
-the deliverable doesn't actually meet the prompt's specifications.
-
-### Evidence (60 status=200 reward=0 tasks, telemetry-complete subset of 54)
-
-| Behavior | Count | Pct |
-|---|---:|---:|
-| Emitted at least one `177` (Confirm) before declaring `200` | 38 | 70% |
-| Skipped Confirm — went directly write→200 | 13 | 22% |
-| Emitted "I gave up" body with status=200 (3 of 60) | 3 | 5% |
-
-Among the 38 that ran Confirm:
-
-| Confirm content shape | Count |
-|---|---:|
-| "did it compile" / "is the binary there" | ~12 |
-| "does it run without crashing" / "exit code 0" | ~14 |
-| "is the output file present" | ~6 |
-| Specific input → specific expected-output check (the rigorous pattern) | ~6 |
-
-The ~6 that did rigorous prompt-derived checks are over-represented
-in the 7 PASSes (e.g. `prove-plus-comm` ran the proof checker;
-`build-pov-ray` likely ran the binary against expected output). The
-~32 that did superficial checks are the false-completions.
-
-### Sample of what the model's Confirm actually checked (vs. what the prompt asked for)
-
-| Task | Model Confirm | Prompt's actual success criteria |
-|---|---|---|
-| `cobol-modernization` | `python /app/program.py` (does it run) | Output bytes match a fixed expected fixture |
-| `fix-git` | `git show <hash>; git status` (state inspection) | Specific commit content / repo state matches fixture |
-| `gpt2-codegolf` | `a.out gpt2-124M.ckpt vocab.bpe "Hello world"` (does it output something) | Token output matches what GPT-2 would predict |
-| `log-summary-date-ranges` | `cat /app/summary.csv` (does the file exist) | CSV content matches expected per-date fixture |
-| `password-recovery` | (no Confirm — gave up at status=200) | Specific 23-char password recoverable from binary |
-
-The pattern: Confirm checks "did it run" / "is the file there" /
-"does it produce *some* output" — all valid sanity checks but
-**not "does it produce the specific outputs the prompt requires for
-the specific cases the prompt named."**
-
-### Why this happens (root cause hypothesis)
-
-`instructions_107.md`'s worked example IS rigorous:
-```
-<sh>[ -f src/sum.js ] && node --version && node src/sum.js 2 2 | grep -qx 4</sh>
-```
-That's a specific input (`2 2`) producing a specific expected output
-(`4`), grep'd for exact match. The pattern is right.
-
-But the example task is trivial (sum two numbers). For real tasks,
-the model would need to:
-1. Re-read the prompt for stated success criteria
-2. Identify specific test cases the prompt named or implied
-3. Construct shell commands that exercise each criterion
-4. Confirm each output matches
-
-Empirically the model skips steps 1-3 and goes straight to "make up
-a sanity check." It treats Confirm as a runtime smoke test rather
-than a derived verification. The instruction surface doesn't push
-back on that.
-
-### Proposed harness-side fixes (none expose the verifier)
-
-1. **Strengthen `instructions_107.md` Confirm content guidance** so
-   the model is asked to enumerate the prompt's success criteria
-   before designing the Confirm. Concrete: add a sub-stage before
-   the Confirm sh, e.g.
-   ```
-   Example: confirming deliverable
-       <set known://success_criteria>
-       1. Compiled binary exists at /app/sum.js
-       2. node src/sum.js 2 2 → 4
-       3. node src/sum.js -1 5 → 4
-       </set>
-       <sh>node src/sum.js 2 2 | grep -qx 4 && \
-           node src/sum.js -1 5 | grep -qx 4</sh>
-       <update status="177">criteria 1-3 verified</update>
-   ```
-   Cost: instruction edit + `$20` re-sweep. If the false-200 bucket
-   shrinks measurably, the diagnosis is confirmed.
-
-2. **Add a Resolution-stage validator hook in the protocol**: before
-   accepting a `<update status="200">`, require a `known://success_criteria`
-   entry and a Confirm body that names how each criterion was
-   verified. Bigger lift; touches verdict logic.
-
-3. **Pure model-side**: ride out the model's over-confidence;
-   different model variants will behave differently here. We could
-   sweep a reasoning-tier variant and see if the Confirm rigor
-   improves with internal reasoning available. That's a $X test
-   cost, not a code change.
-
-None of these expose the verifier. All are about pushing the model
-to derive better self-checks from the prompt itself.
-
-### Does this clear the smoking-gun bar?
-
-Per AGENTS.md:
-- ✅ Specific protocol gap — Confirm content guidance is too
-  permissive of "did it run" stand-ins
-- ✅ Reproducible across many task types — ~32/38 of Confirm-emitting
-  false-completions show superficial checks
-- ⚠️ Concrete fix candidate exists, but it's an instruction-surface
-  refinement rather than a single-line code change. Less crisp than
-  the original (wrong) "run the verifier" framing.
-- ✅ Evidence is concrete — sh/env command logs across the sweep
-
-It clears the bar in shape but the proposed fix is harder to land
-crisply than I initially framed. The fix is right; my earlier
-framing was wrong.
-
----
-
-## Secondary patterns (less than smoking-gun bar but worth tracking)
-
-### Pattern: 99-turn ceiling without strike
-
-8 tasks hit `RUMMY_MAX_LOOP_TURNS=99` without ever declaring 200 or
-striking out. Confirms the AGENTS.md open item *"Continuation-forever
-in Distillation has no protocol-side cap"* is alive in the wild.
-Tasks: `circuit-fibsqrt`, `feal-differential-cryptanalysis`,
-`feal-linear-cryptanalysis`, `largest-eigenval`, `make-doom-for-mips`,
-`overfull-hbox`, `polyglot-rust-c`, `portfolio-optimization`.
-
-These are heavy-compute tasks where the model kept productively
-working but never declared completion. Lower-priority than the
-smoking gun because the dominant failure mode is the false-200
-cluster, not the loop-ceiling cluster (8 vs 60). But fixing the
-smoking gun would not address these — they need separate work.
-
-### Pattern: gave-up-as-200 (rarer than initially hypothesized)
-
-Only ~3 of 60 false-200 tasks emit a body that reads as "I tried
-but couldn't" — `password-recovery` is a clear example. The model
-declares 200 with a body announcing failure-to-solve. The harness
-gates on status field only, treats it as completion.
-
-This *is* a real pattern but small. Worth a smaller fix once the
-smoking gun is addressed: verdict logic could reject status=200
-when the recorded update body matches a "give-up" semantic
-classifier. But the cost/benefit is low compared to the verifier-
-discovery fix.
-
-### Pattern: skipped-confirm (write → 200 with no 177)
-
-22% of false-200 tasks (13 of 60) went directly from writing the
-deliverable to declaring 200, with no Confirm step in between. This
-is a different shape than the smoking gun (where Confirm IS run but
-checks the wrong thing). Probably an instruction-surface drift —
-some tasks let the model bypass Confirm because the worked example
-didn't feel applicable.
-
-If the verifier-discovery fix lands as proposed, the skipped-confirm
-runs would also be expected to start running the verifier (since
-the example would prescribe it directly), so this pattern likely
-collapses into the same fix.
-
----
-
-## What this sweep does NOT prove
-
-- Nothing in this data establishes that grok-4-1-fast-non-reasoning
-  is fundamentally insufficient for tbench. The dominant failure
-  mode is a *protocol gap*, not capability.
-- Nothing here establishes that fixing the smoking gun reaches
-  leaderboard-competitive scores. It establishes that the lowest-
-  hanging fruit (60 false-completions across diverse task types)
-  has a single addressable root cause.
-- The remaining 8 max-loop-turns + 1 max-strikes + 1 crash tasks
-  need separate diagnosis after the smoking-gun fix is tested.
-
-## Project verdict
-
-The 8% score reflects a *concentrated* failure mode (60 of 89 tasks
-in one bucket — false-completions where the model's superficial
-self-check passed but the verifier's didn't). That's still a much
-better outcome than "uniformly bad across all task types," because a
-concentrated failure mode points at a specific addressable cause.
-
-The corrected diagnosis is harder to fix crisply than the wrong one
-I initially proposed. "Make the model run the verifier" was a
-one-line instruction edit and benchmark-gaming. "Make the model
-derive rigorous Confirm checks from the prompt's stated success
-criteria" is a real instruction-surface design problem that may
-need iteration to land well.
-
-**The theory isn't disproven.** It's also not yet vindicated. The
-next iteration tests the strengthened-Confirm instruction edit at
-$20 cost. If the false-200 bucket shrinks measurably, the diagnosis
-holds. If not, forensic burden flips and we look harder at whether
-this is fundamentally a model-capability ceiling we can't instruct
-around.

package/audit/SWEEP_TEMPLATE.md

@@ -1,156 +0,0 @@
-# SWEEP_TEMPLATE.md
-
-Per-task evaluation template for the 2026-05-01 discovery sweep
-(`test/tbench/results/2026-05-01T23-34-12-373Z_rummy_all/`).
-
-Apply this template once per task into `audit/sweep/<task-name>.md`.
-Index in `audit/SWEEP.md`. Cross-task findings in `audit/SWEEPX.md`.
-
-The template's design is grounded in a pre-skim of 5 representative
-tasks (one PASS, one claim-success/verifier-fail, one MAX_LOOP_TURNS,
-one MAX_STRIKES, one status=500). Categories below reflect failure
-modes actually observed, not a-priori guesses.
-
-**Forensic stance:** every entry defaults to "we let the model down
-until proven otherwise" per `feedback_forensic_burden_of_proof`. The
-"What would have rescued this run" field is mandatory; "model-
-capability bottleneck" is only allowed there if the entry has
-exhausted concrete harness-side rescues.
-
----
-
-## Template
-
-# `<task-name>`
-
-## Identity
-
-- Task slug:
-- Run alias:
-- Trial dir: `test/tbench/results/.../...`
-- Reward: `0` | `1`
-- Run status: `200` | `499` | `500` | `(no-summary)`
-- Turn count:
-- Exit code:
-- Tokens (prompt / cached / completion / total):
-
-## Termination mode
-
-One of:
-- `PASS-clean` — reward=1, status=200, model emitted Confirm before
-  Complete
-- `claim-success-fail` — reward=0, status=200, verifier disagrees.
-  Sub-tag with one of:
-  - `gave-up-as-200` — body is "I tried but couldn't" / "no
-    solution found" with status=200
-  - `confirmed-wrong-thing` — Confirm step ran but checked
-    something orthogonal to what the verifier checks
-  - `skipped-confirm` — model went 177(write) → 200 with no
-    intermediate verification
-  - `confirm-passed-verifier-still-failed` — model's Confirm
-    matched the verifier shape but verifier still rejected
-- `max-loop-turns` — status=499, turn=99, never declared completion
-- `max-strikes` — status=499, "Abandoned after 3 consecutive
-  strikes." in verdict trace
-- `dispatch-crash` — status=500, error escaped to AgentLoop's
-  outer catch
-- `exfil-fail` — no rummy.txt summary line; container died before
-  drain
-
-## FCRM walk
-
-Sequence of update statuses across turns (compress runs):
-
-```
-145 → 155×N → 156 → 167 → 177×M → 200
-```
-
-- Did the run reach Deployment (a 177)?
-- Did it reach Complete (a 200)?
-- Time-in-phase: any single phase >5 turns?
-
-## Confirm-step audit
-
-For runs reaching Deployment:
-- Did the model emit at least one 177 between writing the
-  deliverable and emitting 200?
-- If yes — paste the Confirm command body (the `<sh>` or `<env>`
-  the model ran):
-  ```
-  ```
-- What did the Confirm command actually verify?
-- Does the Confirm command match what the verifier would check?
-  - `match` — Confirm runs the same test (or close) as the verifier
-  - `partial` — Confirm checks part of what the verifier checks
-  - `orthogonal` — Confirm checks something the verifier doesn't
-    care about (e.g. file exists vs file works)
-  - `absent` — model emitted no Confirm
-
-For runs NOT reaching 200: did the model attempt any
-self-verification at any point?
-
-## Final update body
-
-Paste the body of the model's last `<update>` (or note `(absent)`):
-
-```
-```
-
-Categorize:
-- `confirmed-success` — body asserts work passed verification
-- `gave-up` — body says model couldn't / didn't / had to abandon
-- `stage-transition` — body is a phase-advance message (155, 167,
-  etc.) — i.e. model never declared completion
-- `partial-success` — body claims partial work done
-
-## Verifier output
-
-What did the verifier's test fail on (paste the assertion or test
-name):
-
-```
-```
-
-Diff between model's claimed work and verifier's expectation:
-- `task-misread` — model solved a different problem than asked
-- `correctness-gap` — model solved the right problem, output is wrong
-- `quality-gap` — output works but doesn't meet a quality bar
-  (size, accuracy, speed)
-- `format-gap` — output correct but in wrong format/location
-- `did-not-attempt` — model gave up before producing a deliverable
-
-## What would have rescued this run
-
-**Mandatory field. No skipping.**
-
-Either:
-1. Concrete harness-side change that would have prevented or
-   recovered this failure (e.g. "Resolution-stage prompt phrasing X",
-   "instruction Y contradicts a worked example", "tool Z's
-   ergonomics blocked"), OR
-2. Explicit "model-capability bottleneck, exhausted alternatives"
-   with a concrete list of harness changes already considered and
-   ruled out.
-
-Option 2 requires evidence in this same entry that we tried/considered
-the harness side first.
-
-## Smoking-gun candidate tags
-
-Tag this entry with any cross-task pattern hypotheses (these
-aggregate in SWEEPX.md). Examples:
-
-- `gave-up-as-200` — model emits status=200 with "couldn't" body
-- `no-confirm-step` — model went write → 200 without 177
-- `confirm-runs-wrong-check` — Confirm verified but verifier still rejected
-- `protocol-loop-no-cap` — model never declared 200 in 99 turns,
-  no strike fired
-- `instruction-X-contradicts-Y` — pattern of misread between two
-  instruction surfaces
-- `tool-X-ergonomics` — model couldn't reliably emit X
-- `(none)` if no cross-task pattern observed
-
-## Notes
-
-Free-form observations not captured by the structured fields above.
-Cross-references to other task entries when applicable.