odd-tool 0.5.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
odd_tool-0.5.0/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Varun (github.com/varunkan)
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,488 @@
1
+ Metadata-Version: 2.4
2
+ Name: odd-tool
3
+ Version: 0.5.0
4
+ Summary: Outcome-Driven Development: an outcome counts as delivered only when its proof fails without the change.
5
+ License: MIT
6
+ Project-URL: Homepage, https://github.com/varunkan/ODD-Outcome-Driven-Development
7
+ Project-URL: Documentation, https://github.com/varunkan/ODD-Outcome-Driven-Development/blob/main/DESIGN.md
8
+ Keywords: ai,coding-agent,claude-code,cursor,copilot,agents-md,testing,ci
9
+ Classifier: Development Status :: 3 - Alpha
10
+ Classifier: Environment :: Console
11
+ Classifier: Intended Audience :: Developers
12
+ Classifier: License :: OSI Approved :: MIT License
13
+ Classifier: Operating System :: POSIX
14
+ Classifier: Programming Language :: Python :: 3
15
+ Classifier: Topic :: Software Development :: Quality Assurance
16
+ Classifier: Topic :: Software Development :: Testing
17
+ Requires-Python: >=3.8
18
+ Description-Content-Type: text/markdown
19
+ License-File: LICENSE
20
+ Dynamic: license-file
21
+
22
+ # Outcome-Driven Development (ODD)
23
+
24
+ **An outcome counts as delivered only when its proof fails without the change.**
25
+
26
+ > **The rule:** nothing counts as done until a check that runs says so: a check that fails without the change, run
27
+ > by someone other than the one claiming it. **No proof, no done.** A proof reaches only as far as the requirement it
28
+ > was written from ([evidence 18](docs/evidence/18-real-repositories.md)). Why it holds, what it does not promise, and
29
+ > how to follow it with or without odd: [THE-RULE.md](THE-RULE.md).
30
+
31
+ Work is done when a proof says so, not when the agent says so. (Not to be confused with the product-management use of
32
+ "outcome-driven", aligning work with business goals, or with observability-driven development, also abbreviated ODD.
33
+ Here an outcome is something a command can check.)
34
+
35
+ `odd` is one small command-line tool and a half-page method for building software with a coding agent. It works with
36
+ Claude Code, Cursor, Copilot, Codex or anything else that can run a shell command, because it needs only a shell and
37
+ git.
38
+
39
+
40
+ **A reasoner for the hard parts** ([DESIGN.md](DESIGN.md) section 3j). Set one agent
41
+ command for judgement, `"reasoner": "<your agent command, any model>"`: it plans a ticket
42
+ into checkable outcomes and their build order (`odd plan <ticket>`), writes proofs (`odd spec`) and challenges them
43
+ (`odd challenge`), while cheap agents write the code. Nothing it writes is trusted: a plan must cover every
44
+ requirement with observable outcomes, and a proof must fail before the code.
45
+
46
+ **The proof, in one page:** [docs/PROOF.md](docs/PROOF.md) lists every claim odd makes with the tests that show it
47
+ (all of them, run on Linux and macOS on every push), grouped by what they prove, and what the measurements show and do not show. It is generated from the test suite,
48
+ and CI fails when it is out of date.
49
+
50
+ ## The problem
51
+
52
+ **A coding agent says "done" when the work is not done, and nothing in the loop checks the claim.** Measured on 24
53
+ real bug fixes (pytest, sphinx, sympy and pylint issues, with hidden tests), a plain agent told to write tests, run
54
+ them and say DONE only if they pass said DONE wrongly on 5. It happens in three ways, and each needs a different
55
+ answer:
56
+
57
+ | what goes wrong | how often it was seen | what odd does about it |
58
+ |---|---|---|
59
+ | **Nothing was changed.** The agent wrote a test that passes on the code as it was, and said DONE | 3 of those 5 wrong DONE | `odd check` says `NO CHANGE`; `odd done` refuses it |
60
+ | **The tests prove nothing.** They pass whether the change is there or not | once in 48 runs | the take-out check: the proof must fail with the change removed |
61
+ | **The fix is wrong the same way as its tests.** One author misread the request and wrote both | 6 of 30 changes whose tests did depend on them | no check on those tests can see it. `odd reread` gets a second reading from another agent and reports a disagreement for a person to settle |
62
+
63
+ Writing more rules for the agent does not fix any of these, because a rule is something the model may or may not
64
+ follow. In the project this method came from, 1,856 instructions were given to agents over three months, and the
65
+ two most expensive mistakes were the process's own rules. ODD keeps the part that does not depend on the model
66
+ agreeing: "done" is decided by a command, on the code.
67
+
68
+ **What it does not solve.** It does not make the work correct. A request that does not say what is wanted stays
69
+ unsolved by any check, and in the measurements a stronger model did more for correctness than odd did
70
+ ([25](docs/evidence/25-the-six-that-fail.md)). odd narrows what an agent can claim without having shown it.
71
+
72
+ ## Where to use it
73
+
74
+ | the situation | the risk | what to run |
75
+ |---|---|---|
76
+ | **You work with an agent at your desk** | it reports done and you take its word | `odd check` after it finishes: one command, no setup. Or `odd init`, and the agent runs `odd done` before it reports |
77
+ | **Agents open pull requests** | a reviewer cannot tell whether the PR's tests show anything | one GitHub Actions step. `mode: check` needs no setup; the full gate requires every changed file to sit inside a proven outcome |
78
+ | **Agents run unattended** (overnight, in a queue, in CI) | nobody reads the "done" claims | the gate decides, and `odd park` gives the agent an honest "blocked" in place of a false done |
79
+ | **A Claude Code session should not end on a false done** | the last message says fixed | `odd check --stop-hook`, or `odd init --claude-hooks`: a turn that would end on it is sent back once, with the result |
80
+ | **A bug fix that matters** (money, data, security) | the fix and its tests share one misreading | `odd reread <slug> --by "<a stronger model>" --ticket <the request>`: a second reading, written without seeing the fix |
81
+ | **You use Spec Kit, BMAD or Superpowers** | specs and plans, and "done" is still the model's word | odd sits underneath as the last check; it replaces none of them |
82
+ | **You compare agents or models** | pass rates alone hide false claims | the harness in [`bench/`](bench/SPEC.md) counts right DONE, wrong DONE and unfinished apart |
83
+
84
+ **Where it is not worth it.** Easy tasks, where every way of working got everything right
85
+ ([15](docs/evidence/15-comparison.md)). Work you review line by line anyway. A change with no test and no way to
86
+ write one: odd then has nothing to run. And as a replacement for a better model or a clearer request, it is none.
87
+
88
+ It is not a planning framework, a prompt library or an agent orchestrator. It is the one check the others leave to
89
+ the model. Questions and objections: [FAQ](FAQ.md).
90
+
91
+ > **Status: early (0.5.0).** The tool passes its own tests (418), including the ways an agent could fake "done". It
92
+ > has been through a design review (23 changes, all applied) and a red team (48 confirmed defects, each now a test).
93
+ >
94
+ > **Does it work better? Measured, and mostly no difference.** In a same-task comparison (16 tasks with hidden
95
+ > acceptance tests, five ways of working, three runs each, two models, 480 runs) every arm, plain Claude Code included, got every doable task right on both Sonnet and Haiku (42 of 42 runs; one Superpowers miss on Sonnet). ODD did not make the code more correct; nothing did. The arms separated only on the impossible tasks with Haiku: ODD and Superpowers claimed "done" falsely in 2 of 6 runs, plain, BMAD and Spec Kit in 3 of 6, and ODD never edited a visible test: a hint, not a finding. ODD cost about a quarter to a half more than plain; BMAD and Spec Kit two to five times as much. These runs used an early copy of ODD (commit 0e38e3e), without proof strength, sealed proofs, or the race, cascade and spec commands.
96
+ > Full results and both grades: [15](docs/evidence/15-comparison.md).
97
+ >
98
+ > **On real bug fixes** (24 issues from pytest, sphinx, sympy and pylint, one run each, one model): against a plain
99
+ > agent told in one sentence to write and run tests, lean odd passed the hidden tests on 18 against 12 and said DONE
100
+ > wrongly once against five times. Neither difference reached the threshold fixed before the runs, lean odd cost
101
+ > about a tenth more, and on a small task the one sentence did as well for less. Three of the plain agent's five
102
+ > wrong DONE were runs that changed only tests and no product code. Counts, not rates:
103
+ > [21](docs/evidence/21-is-the-core-worth-more-than-a-sentence.md), [22](docs/evidence/22-real-bug-fixes.md),
104
+ > [23](docs/evidence/23-real-bug-fixes-pass-rate.md), [24](docs/evidence/24-the-check-with-no-setup.md).
105
+
106
+ ## The idea
107
+
108
+ 1. **State the outcome before the code.** One sentence saying who observes what, a command that exits 0 only when
109
+ that is true, and the files the change may touch.
110
+ 2. **Make the change.**
111
+ 3. **`odd done` decides.** It passes only when the proof passes on your code as it is, *fails on the same code with
112
+ your change taken out*, and passes again once the change is put back. A proof that would pass anyway (`true`, a
113
+ test that asserts nothing) is rejected.
114
+ 4. **`odd gate` checks the whole change.** Every changed product file must be inside the boundary of one verified
115
+ outcome. In CI it recomputes everything from the repository and trusts nothing the agent recorded.
116
+
117
+ Small edits (25 changed lines by default) need no outcome. If the work cannot be done as specified, `odd park` says
118
+ so, and that is an acceptable result.
119
+
120
+ ## Try it
121
+
122
+ **The one check, with no setup.** In any git repository, after you or an agent made a change with tests:
123
+
124
+ ```bash
125
+ pipx install odd-tool # or copy odd.py and run: python3 odd.py check
126
+ odd check # runs the tests this change brings, takes the change out, runs them again
127
+ ```
128
+
129
+ It ends with one line: `SHOWN` (the tests fail without the change), `NOT SHOWN` (they pass without it, or do not
130
+ pass), `NO CHANGE` (only tests changed: nothing was fixed) or `NOT CHECKED` (the change has no test). Nothing is
131
+ written and nothing is recorded. It reports; what it has and has not been shown to catch is in
132
+ [evidence 24](docs/evidence/24-the-check-with-no-setup.md), and its options in [DESIGN.md](DESIGN.md) section 3s.
133
+
134
+ **The whole method.** Install it either way; both give the same single file.
135
+
136
+ ```bash
137
+ pipx install odd-tool # or: pip install odd-tool (then the command is `odd`)
138
+ curl -fsSLO https://raw.githubusercontent.com/varunkan/ODD-Outcome-Driven-Development/main/odd.py # or just the file
139
+ ```
140
+
141
+ ```bash
142
+ cd your-project # any git repository; needs a POSIX shell, git 2.23+ and Python 3.8+
143
+ odd init # (or: python3 ../odd.py init) writes .odd/, with its own copy of the tool, and the method into AGENTS.md
144
+
145
+ python3 .odd/odd.py new split-bill \
146
+ "A cashier who splits a bill three ways sees three totals that add up to the bill." \
147
+ --proof "pytest tests/test_split.py -q" --in "src/billing/**"
148
+
149
+ # write tests/test_split.py, then make the change in src/billing/
150
+
151
+ python3 .odd/odd.py done split-bill
152
+ # proof passed in 0.4s; two more runs (without the change, then with it again), about 0.8s
153
+ # split-bill: verified. took out 2 file(s), 41 line(s); without them the proof said: 1 failed in 0.04s; 0.4+0.4+0.4 s
154
+
155
+ python3 .odd/odd.py gate # exit 0 only when every changed file is inside a verified outcome
156
+ ```
157
+
158
+ From here your coding agent follows the method in `AGENTS.md` by itself.
159
+
160
+ | command | what it does |
161
+ |---|---|
162
+ | `odd init [--claude-hooks] [--mcp]` | set up or repair a project; prints the step to add to CI and a proof suggestion |
163
+ | `odd new <slug> "<outcome>" --proof CMD --in "GLOB, GLOB"` | state an outcome (a three-line file in `.odd/outcomes/`); `--proof auto` picks a narrow command from your test runner and prints what it detected; `--covers R1,R3` names the ticket requirements it covers; `--kind bug` requires the proof to fail on the unfixed code; `--kind feature` and `--kind refactor` fill their defaults and hints; prints the files to read first |
164
+ | `odd done <slug> [--base REF]` | verify it: passes, fails with the change taken out, passes again |
165
+ | `odd race <slug> [--n N] [--agent CMD]` | run N agent attempts in parallel, each in its own git worktree; the verified one with the strongest proof and fewest changed lines is applied to your tree |
166
+ | `odd cascade <slug> [--agents C1,C2] [--race N]` | try the agents in order, cheapest first, each attempt in its own worktree; stop at the first verified one and apply it, else escalate with the failure in the next prompt, and park after the last |
167
+ | `odd chain <ticket> [--race N] [--cascade]` | do a long task as the outcomes listed in `.odd/chains/<ticket>.md`, in order: each one not yet proven is raced (or cascaded), verified and committed as the checkpoint `odd: <slug> proven`; an unproven one stops the chain, is parked with its reasons, and a rerun continues from it |
168
+ | `odd spec <slug> --proof-by CMD [--tries N]` | another agent (in practice a stronger model) writes only the proof in its own worktree; it is brought back and sealed red before any code, retried up to 3 times, then parked |
169
+ | `odd challenge <slug> [--agent CMD]` | advisory: another agent tries to make the verified change wrong while the proof still passes; prints its claim and the variant, applies nothing |
170
+ | `odd gate [--recompute]` | is every changed file inside a verified outcome? (`--recompute` for CI) |
171
+ | `odd park <slug> "<reason>"` | the honest exit: blocked, or cannot be done as specified |
172
+ | `odd ask <slug> "<question>"` | the outcome cannot be made checkable from the ticket: record one precise question; the gate says "waiting for an answer" until the `asked:` line is deleted |
173
+ | `odd escaped <slug> "<what, where found>"` | a defect got past a verified outcome: record it on that outcome |
174
+ | `odd stats [--json]` | counts, self-reported, including escaped defects by kind of proof, the escape rate per outcome kind, seal, strength band and proof author, and dollars and seconds per proven outcome from race, cascade and chain attempts |
175
+
176
+ **Racing agents.** With `"agents": ["<agent command>", ...]` in `.odd/config.json` (any CLI agent; `{prompt_file}`
177
+ in the command becomes the prompt's path, else the prompt goes to stdin), `odd race <slug>` starts one attempt per
178
+ command in its own working tree under `.git/` (a `git clone --shared`, so its git config, hooks and refs stay its
179
+ own), checks each one as `odd done` would, prints a table of attempts and
180
+ applies the winner: verified, then the most small breaks caught, then the fewest changed lines, then the earliest
181
+ finish. If your tree already has changes inside the boundary it writes the winner as a patch instead. On fake agents
182
+ (one correct, one with a weaker proof, one broken, one that edits the test) the correct one won 10 races of 10, and
183
+ setting up and removing an attempt's tree costs 0.06-0.08 s per attempt on a three-file repository. An attempt is
184
+ refused if it left a process running (found by its `ODD_ATTEMPT` mark on Linux, and killed), and ignored files it made
185
+ that your tree lacks are removed before the check, since the applied patch would not carry them. Not measured yet:
186
+ real agents, and whether a race of cheap models beats one strong one (BACKLOG item 1). Details: [DESIGN.md](DESIGN.md) section 3b.
187
+
188
+ **Cheapest agent first.** `odd cascade <slug>` reads the same `"agents"` list as levels, cheapest first (or
189
+ `--agents "C1,C2"`): the first agent attempts the outcome in a fresh worktree; if the attempt is verified it is applied
190
+ and nothing dearer runs; otherwise the next agent gets the precise failure (the proof's output, the surviving small
191
+ break and its line, the line the proof never ran) in its prompt. After the last level the outcome is parked with every
192
+ reason, and nothing unverified is applied. `--race N` runs N attempts per level. Each attempt's cost is shown when the
193
+ agent prints a final line `odd-cost: <usd>`, else "not reported". On fake agents: 0.65-0.71 s wall when level 1 is
194
+ verified, 0.74-0.87 s when it escalates to level 2, 0.42-0.49 s to park after two (5 runs each, three-file repository).
195
+ Every race, cascade or chain attempt also records its reported cost and its seconds locally, and `odd stats` prints
196
+ dollars and seconds per outcome and per proven outcome, failed attempts included, labelled self-reported (a floor
197
+ when some agents report no cost; seconds are summed per attempt, not wall clock).
198
+ Not measured: real agents and whether Haiku-then-Sonnet is cheaper than Sonnet alone (BACKLOG item 2). Details:
199
+ [DESIGN.md](DESIGN.md) section 3c.
200
+
201
+ **Long tasks as a chain.** Split a ticket into small outcomes (`odd new` each, proofs written), list their slugs in
202
+ order in `.odd/chains/<ticket>.md`, and run `odd chain <ticket>` (`--race N`, or `--cascade` to use the agents as
203
+ levels). Each outcome is attempted in worktrees as race or cascade would, verified again in your tree, and committed
204
+ as its own checkpoint `odd: <slug> proven`; the next one starts from that commit. When one cannot be proven the chain
205
+ stops, the tree is at the last checkpoint, the outcome is parked with the reasons, and the status lists every outcome
206
+ as proven, parked or not started. Run it again to continue. It refuses to start if your tree or index has changes outside the
207
+ chain's outcomes and their tests, and it only ever reverses the patch it applied itself (and unstages what it staged);
208
+ an uncommitted edit to a test the proof names goes into that outcome's checkpoint. On fake agents, three
209
+ outcomes took 3.9-4.2 s with three checkpoint commits; a chain stopping at the second took 1.7-1.9 s and its resume
210
+ 2.8-3.1 s (5 runs each, a repository of six small files). Not measured: real agents, and how much longer an unattended task gets
211
+ with chains than without (BACKLOG item 10). Details: [DESIGN.md](DESIGN.md) section 3e.
212
+
213
+ **Every requirement covered.** Write the ticket's requirements once, one per line, in
214
+ `.odd/requirements/<ticket>.md` (`R1: ...`, `R2: ...`), and let each outcome name the ones it covers (`covers: R1, R3`,
215
+ or `odd new ... --covers R1,R3`). With `"coverage": "report"` in `.odd/config.json`, `odd gate` and the pull-request summary print `requirements <ticket>: 1 of 2
216
+ covered; not covered: R2 (...)`, and a `covers:` id that is not in the file fails the gate. `"enforce"` makes
217
+ an uncovered requirement a Fail too. The default is `"off"`.
218
+ It catches "works, but not what was asked" only as far as the list is complete and each proof tests
219
+ what its outcome says; odd checks neither. Not measured: requirements missed with and without it (BACKLOG item 4).
220
+ Details: [DESIGN.md](DESIGN.md) section 5a.
221
+
222
+ **Earlier proven work stays proven.** A merged outcome keeps its file, so its proof stays in the repository. When a
223
+ later change touches a file inside an earlier outcome's `in:`, `odd gate` (local and `--recompute`) runs that proof
224
+ again (green only) and fails with `<slug> (proven in <commit>) now fails: <last line>` when it no longer passes; the
225
+ pull-request summary lists every guard run. Parked, asked and skipped outcomes are not guards. The cost is one proof
226
+ run per guard. `"guard": "report"` only prints, `"off"` turns it off (the default `"enforce"`).
227
+ Editing only an outcome's `escaped:` line keeps it a guard. Deleting or parking a merged outcome runs its
228
+ base proof as a reported guard that never fails the gate, prints `deleted <slug> (merged in <commit>)` for a deletion,
229
+ and sends the auto lane to review. Overlap is by path: a break through a file outside the earlier `in:` is not seen. Not measured: earlier
230
+ outcomes broken with and without it (BACKLOG item 9). Details: [DESIGN.md](DESIGN.md) section 5b.
231
+
232
+ **A verification is not repeated when nothing it reads changed.** `odd done`, `race`, `cascade`, `chain` and `gate
233
+ --recompute` key each verification on the tool, the settings, the base commit and every file that differs from it; a
234
+ match with an earlier pass prints `cached (verified at <time>)` and runs no proof, and a one-byte change in any of
235
+ them runs it in full (`--no-cache` always does). Locally the cache is under the git directory. In CI it is used only
236
+ with `"cache_file"` set on the target branch to a path outside the repository, holding
237
+ what an earlier `--recompute` wrote; restore it on pull requests and save it only from the target branch. The gate
238
+ also says how many earlier proofs the regression guard skipped as unaffected. Measured on the test fixture: a second
239
+ `done` 0.22 s against 0.91 s, a second `gate --recompute` 0.16 s against 0.73 s (5.66 s with a proof that sleeps 1 s).
240
+ Ignored files, installed packages and the environment are not in the key. Details: [DESIGN.md](DESIGN.md) section 5d.
241
+
242
+ **The agent starts from the boundary.** After `odd new` (and the MCP `new` tool) writes the outcome, it prints
243
+ `read these first`: the files inside `in:` that exist, at most ten, the largest change against the base first, then
244
+ the largest file, and the files the proof names. The race, cascade and chain prompt carries the same list (no
245
+ setting). Whether it saves an agent turns or tokens is not measured yet. Details:
246
+ [DESIGN.md](DESIGN.md) section 3f.
247
+
248
+ **An auto-merge lane, off by default.** With `"auto_lane": true` the gate also says
249
+ `"lane": "auto"` or `"lane": "review"` with its reasons: auto only when the gate passed, every outcome in the change
250
+ is verified with every small break caught (`"auto_min_strength"`, default 1.0) and no changed line its proof never
251
+ runs, the change is at most `"auto_max_lines"` product lines (default 120) all inside outcomes, and there is no
252
+ control file, no `test-change:`, no skipped take-out check, no parked or asked outcome, no deleted merged outcome file, and every
253
+ regression guard passed. It shows in `--json`, in the pull-request summary (`Auto-merge lane: yes` or `no, because ...`) and as the
254
+ Action output `odd_lane`. odd never merges: a repository wires its own auto-merge job on `odd_lane == 'auto'`, and a
255
+ change that takes it merges with nobody reading its code, so the outcome sentence and the proof are all that is
256
+ checked. Not measured: review minutes saved or defects escaping through it (BACKLOG item 11). Details:
257
+ [DESIGN.md](DESIGN.md) section 5c.
258
+
259
+ **Bugs are reproduced first.** `odd new <slug> ... --kind bug` marks a bug fix. Then `odd seal` and `odd done` require
260
+ the proof to fail on the unfixed code (the outcome's changed files put back to the base) and print
261
+ `reproduced: <last line of the failure>`; a proof that passes there is refused ("the proof does not reproduce the
262
+ bug"). odd shows the failure line; whether it is the reported symptom is for the reviewer
263
+ (a missing new file also fails). Details: [DESIGN.md](DESIGN.md) section 4a.
264
+
265
+ **Each kind of change starts from its proof shape.** `odd new ... --kind bug|feature|refactor`
266
+ writes `kind:` and prints a one-line hint: bug, reproduce first; feature, the proof covers the new behaviour
267
+ and the existing suite stays green; refactor, the existing suite is the proof. `--kind refactor` may leave out
268
+ `--proof` (it becomes the detected runner's whole suite) and writes `counterfactual: skip - behaviour-preserving
269
+ refactor`, so `done` checks only that the suite passes and no test changed unexplained. odd does not run the suite for
270
+ a feature, and does not show that the suite exercises a refactored line. Details: [DESIGN.md](DESIGN.md) section 4c.
271
+
272
+ **A vague outcome is stopped, and a question is a valid answer.** `odd new` refuses a sentence with nothing anyone
273
+ would see, get or measure ("Refactor the billing module.") and suggests "<who> <observes> <what>, measurably". It is a
274
+ small English word list, not a model: wide on purpose (state words such as "is disabled", "keeps", "can export"
275
+ count), so it misses vague sentences that contain a listed word and still refuses ordinary sentences built only from
276
+ unlisted words; "so the tests pass" does not count as an observable. `"ambiguity_stop": false` turns it off. When the ticket does not say what should be observable, `odd ask <slug>
277
+ "<question>"` records the question; the gate reports that outcome as waiting for an answer, not failed, until the
278
+ `asked:` line is deleted; until then it covers no files and no requirement. Details: [DESIGN.md](DESIGN.md) section 4b.
279
+
280
+ **A stronger model writes the proof, a cheaper one the code.** `odd spec <slug> --proof-by "<command>"` gives the
281
+ outcome to one agent command to write only the proof, in its own worktree. odd accepts it only if it changes test
282
+ files alone and the proof fails before the code; it brings the tests back, seals them (`odd seal`) and writes
283
+ `proof-by: <name>` in the outcome file (credentials on the command line are left out). A proof that passes before the
284
+ code is refused and the reason goes to the next try; after 3 tries (`"proof_tries"`) the outcome is parked. Then
285
+ `odd race` or `odd cascade` with cheaper agents works against that sealed proof, and an attempt that edits it is
286
+ refused. Why: in one study, tests from a stronger model raised a fixed repair agent's resolution from 61.2% to 65.3%,
287
+ while tests from the weak model itself lowered it to 57.3% (arXiv 2609.09133, see
288
+ [docs/evidence/14-research.md](docs/evidence/14-research.md)). On odd's harness ([16](docs/evidence/16-new-arms.md#after-the-fixes)) it
289
+ did not: with Sonnet's proofs, Haiku delivered 30 of 42 doable tasks and 4 of Sonnet's proofs passed work the hidden
290
+ tests reject; cascade (39 of 42 at a third of plain Sonnet's cost) and race (no false DONE) did better. Details: [DESIGN.md](DESIGN.md) section 3d.
291
+
292
+ **An attempt budget per outcome.** `"max_attempts": N` (default 0, off) parks an outcome
293
+ after N failed attempts at it, counted on this machine from odd's local evidence: failed `odd done` runs and refused
294
+ `race`, `cascade` or `chain` attempts, since the outcome was last verified. The `parked:` line lists each reason; a
295
+ cascade runs no further level once the budget is reached. While parked for its budget, `done`, `race`, `cascade`
296
+ and `chain` refuse the outcome (exit 1) and run no proof and no agent, until a person deletes the `parked:` line. The
297
+ number is never written into `AGENTS.md` or an agent's prompt (in our pilot a visible budget made an agent skip work),
298
+ but it is in `.odd/config.json`, which an agent can read in its copy of the repository. Whether it
299
+ saves money on tasks that end unsolved is not measured. Details: [DESIGN.md](DESIGN.md) section 3g.
300
+
301
+ **A proof the agent never sees.** `"held_out": [{"name", "proof", "in"}]` lists proofs
302
+ kept outside the repository, for example in a private checkout a CI step provides. Only `odd gate --recompute` runs
303
+ them, with the list read from the target branch, and their output is never shown: a change fitted to its visible
304
+ proof (a lookup table for the cases it checks) fails there without learning why. Why: visible-only checks scored 97%
305
+ against 0% held out on such a solution (arXiv 2605.21384). Details: [DESIGN.md](DESIGN.md) section 5e.
306
+
307
+ **Files the proof does not need.** `"inert": "report"` or `"enforce"` (default off) takes
308
+ each changed file out on its own after the main check; a file whose removal leaves the proof passing is named, or
309
+ fails the check under `"enforce"`. Smaller changes, nothing riding along unproven. Details: [DESIGN.md](DESIGN.md)
310
+ section 4f.
311
+
312
+ **Checks earned from real projects** ([DESIGN.md](DESIGN.md) section 5f): standing
313
+ project rules as commands every gate runs (`"checks"`, a secrets scan for example); `"repeat"` so a flaky proof
314
+ cannot fake the take-out check; `target:` so a proof must show it ran against this commit's deploy or build;
315
+ `owed:` and `odd owed` for steps still owed on the real system, closed only with evidence; `"max_cost"`, a budget in
316
+ dollars; and proofs piped into `tail` or `grep -q .` refused, since the pipe hides their exit code.
317
+
318
+ **Refactors checked by comparison, not skipped.** `same: <command>` in an outcome runs the
319
+ command with the change and with it taken out; its exit code and whole output must be identical, or `done` names the
320
+ first differing line. Details: [DESIGN.md](DESIGN.md) section 4e.
321
+
322
+ ## What it checks, and what it does not
323
+
324
+ It checks that the proof **depends on the change**: with the change removed, the proof fails. It checks that nothing
325
+ was changed outside the stated boundary, including files written from a shell.
326
+
327
+ It then makes up to eight small breaks in the changed lines (a `<` for a `<=`, `and` for `or`, a number plus one),
328
+ one per changed line first and spread across files, one at a time, and names each one the proof does not notice: a
329
+ changed line the proof does not depend on. The score reads "proof strength k of n" (k breaks caught of n), never
330
+ "verified". For a Python proof (`python -m unittest`, `pytest`, a `.py` file) it also runs the proof once under a
331
+ stdlib trace hook and names each changed line the proof never executed ("not run by the proof: calc.py:6"); for other
332
+ languages it says "not measured". A proof that only prints (`echo`, or a Python script of imports, `print(...)`
333
+ and plain assignments, with no assert, raise, exit, subscript or division) is refused. All of this is reported by default and can fail the gate (`"strength":
334
+ "enforce"` in `.odd/config.json`, or `"off"`).
335
+
336
+ **A fix that is wrong the same way as its tests** passes all of the above: one author read the request and wrote
337
+ both. `odd reread <slug> --by "<agent command>" --ticket <file>` has another agent write a proof from the request
338
+ without seeing the change, and runs it on the change. A disagreement means one of the two readings is wrong (52 of
339
+ 56 times on kept runs) and a person decides which; agreement is not proof. Give the reader a stronger model than the one that wrote the
340
+ change: with one, it disagreed with 40 of 52 wrong-fix comparisons and 9 of 38 right ones; with the same model, it
341
+ flagged right fixes nearly half the time. Advisory: [evidence 26](docs/evidence/26-a-second-reading.md),
342
+ [27](docs/evidence/27-odd-reread-with-a-model.md).
343
+
344
+ It does not check that the outcome is worth having, or that the proof is thorough. For new code, a proof that only
345
+ imports the new function already fails without it; the small breaks catch some of that, not all. The outcome sentence and the proof command are a few lines, and
346
+ the gate prints what the proof said when it failed; a person should read both.
347
+
348
+ **Where the enforcement is real.** On a developer's machine, an agent with shell access can bypass any local check.
349
+ The gate is the authority only in CI, and only when (1) it is a required check, (2) the tool, its settings and the CI
350
+ configuration need a person's review to change, and (3) the agent cannot merge or push to the target branch. The CI
351
+ step takes the tool and its settings from the base branch, so a pull request cannot loosen its own gate. `odd init`
352
+ prints that step.
353
+
354
+ **Sealed proofs (optional).** Write the test, run `odd seal <slug>`, then write the code. `seal` refuses a proof that
355
+ already passes, and writes `sealed: proof@<hash>, tests/test_x.py@<hash>` into the outcome file. If the proof command
356
+ or a sealed test file changes afterwards, `odd done` and the gate fail until it is resealed, and a reseal changes the
357
+ outcome file, so it shows in the diff. It makes fitting the test to the code visible; it does not stop it. A reseal
358
+ needs the proof red again, so take the product change out first. Unsealed outcomes work as before.
359
+
360
+ **Changed tests and empty proofs fail closed.** A change that edits or deletes a test file that existed at the base
361
+ fails `odd done` and the gate unless an outcome in the change says why with `test-change: <reason>`; the gate and the
362
+ review comment print the reason; a reason that names test files (`test-change: tests/test_keep.py - ...`) covers
363
+ only those, and the gate shows which reason covers which file. A new or changed `conftest.py` or `pytest.ini`, a
364
+ changed pytest section in `pyproject.toml`/`setup.cfg`/`tox.ini` or a changed `load_tests` needs a reason too:
365
+ turning a test off is editing it. Other new test files need none. A proof that exits 0 but ran no test, skipped every test
366
+ it ran (or only had expected failures: xfail, `expectedFailure`), or stopped before its runner printed its summary (`sys.exit(0)` at import, an empty suite, `go test` with no
367
+ test files) is refused, for unittest, pytest, jest and go test, read from the proof's own output and deliberately
368
+ conservative ([DESIGN](DESIGN.md) section 4 lists what is not detected). On seeded fixtures of the systemic hacks,
369
+ the tool before these rules accepted 6 of 8 and this version refuses all 8; three honest fixes still pass.
370
+
371
+ **Escaped defects.** When a defect gets past a verified outcome, run `odd escaped <slug> "<what went wrong, where it
372
+ was found>"`. It appends a dated entry to that outcome's `escaped:` line and tells you the next step: a new outcome
373
+ whose proof fails on the escaped behaviour. `odd stats` then counts escapes against outcomes that were sealed or not,
374
+ that skipped the take-out check or not, and whose proof caught every small break or not (when that was measured on
375
+ this machine). Over time that shows which kinds of proof let defects through. It counts only what people record, so
376
+ it is self-reported, not a measurement. It also prints a table of escape rates (escaped
377
+ defects per outcome) grouped by outcome kind (bug, feature, refactor, none), sealed or not, proof strength band (full,
378
+ partial, not measured) and whether another agent wrote the proof (`odd spec --proof-by`), and says under each group of
379
+ fewer than 10 outcomes that its counts are small; `odd stats --json` gives the same as JSON.
380
+
381
+ ## In CI
382
+
383
+ On GitHub Actions the gate is one step at the end of your test job:
384
+
385
+ ```yaml
386
+ - uses: actions/checkout@v4
387
+ - uses: varunkan/ODD-Outcome-Driven-Development@v0.5.0
388
+ ```
389
+
390
+ With `with: mode: check` the step runs `odd check` instead: no outcome files and no `.odd` folder are needed, and
391
+ it fails when the tests a pull request brings pass without the rest of it, or do not pass.
392
+
393
+ It fetches the history it needs, judges the change with the action's own pinned copy of the tool (never the copy the
394
+ pull request carries) and reads the settings at the tip of the target branch. Pin a tag or a commit, and make the step
395
+ a required check. Other CI systems use the two lines `odd init` prints; they do the same.
396
+
397
+ The result is written for the reviewer: per outcome its sentence, its proof, what the proof said with the change
398
+ taken out and its strength score, then the exemptions and any control files changed. It goes to the job's step
399
+ summary always, and on a pull request into one comment that each run updates. To allow the comment, give the job:
400
+
401
+ ```yaml
402
+ permissions:
403
+ contents: read
404
+ pull-requests: write
405
+ ```
406
+
407
+ Without it (or on a pull request from a fork, whose token is read-only) the log says the comment could not be posted
408
+ and the check's result is unchanged. `with: comment: false` turns the comment off. The step that runs the proofs
409
+ is never given the token, only the comment step after it; only odd's own comment (by `github-actions[bot]`, or
410
+ `with: author:` for a personal token) is updated, and text from the change is rendered inert. Elsewhere, `odd gate --summary
411
+ FILE` writes the same Markdown and `--json FILE` the same result as data. Whether it saves review time is not yet
412
+ measured ([protocol](docs/review-time.md)).
413
+
414
+ ## Works with
415
+
416
+ `odd init` writes the method (about thirty lines) into the root `AGENTS.md`; running it again replaces that block in
417
+ place. Its last step tells the agent that when the task cannot be done as asked or the proof cannot honestly pass, it
418
+ should STOP and explain with `odd park`, and that this is the expected answer, not a failure (wording from
419
+ ImpossibleBench's abort option; its effect here is not measured). Step 4 tells it to stop once `odd done` and
420
+ `odd gate` both pass. `odd done` and `odd gate` end with one `VERDICT:` line (`DONE`, `DONE, WITH n POINTS NOT
421
+ PROVEN`, or `NOT DONE`), and step 5 tells the agent to put that line in its report, word for word.
422
+
423
+ | reads `AGENTS.md` without configuration | needs one line |
424
+ |---|---|
425
+ | Codex, Cursor, GitHub Copilot in VS Code (and its cloud agent and CLI), Claude Code, Junie, Windsurf/Devin | Gemini CLI: set `context.fileName` to `AGENTS.md`. Aider: `read: AGENTS.md` in `.aider.conf.yml`. Copilot Chat outside VS Code: point `.github/copilot-instructions.md` at `AGENTS.md`. |
426
+
427
+ The table rests on each tool's documentation ([sources](docs/evidence/10-design-review.md)). Only Claude Code has
428
+ been run with it so far. The command and the gate are the same for every tool and for a person.
429
+
430
+ Optional, for Claude Code only: `odd init --claude-hooks` adds two hooks for your machine (never committed) that give
431
+ one line of status at session start and stop a turn from ending on unproven work.
432
+
433
+ Optional, for any MCP client: `odd init --mcp` adds odd's MCP server to the project's `.mcp.json` (other servers are
434
+ kept), so an agent calls `new`, `seal`, `done`, `gate`, `park` and `escaped` as tools and gets odd's exit code and
435
+ output back; a failed `done` ends with a `next:` block (the proof's failing line, each surviving break, each
436
+ unrun changed line, each refused test edit, the command to rerun; a failure with none of these items has no block,
437
+ since the remedy is in the message; DESIGN.md section 4d),
438
+ also returned as `structuredContent.next`. It is a thin wrapper that runs `.odd/odd.py`; so far it has been driven only by the test suite, not by
439
+ a real client ([details and limits](DESIGN.md#7-how-it-reaches-each-tool)).
440
+
441
+ ## Why, and the evidence
442
+
443
+ The method was distilled from building a production point-of-sale system with coding agents: 1,856 instructions
444
+ given over three months, each measured and ruled on. Most were dropped.
445
+
446
+ - [FAQ.md](FAQ.md): short answers, the differentiator, objections, how to adopt it, and a script for explaining it
447
+ in a session.
448
+ - [WHY.md](WHY.md): what the evidence supports, what it counts against, and what is not claimed.
449
+ - [What went wrong](docs/evidence/02-pos-evidence-ledger.md): the waste record of the source project. The two most
450
+ expensive mistakes were its own rules.
451
+ - [The comparison with Spec Kit, BMAD and Superpowers](docs/evidence/07-validation-and-verdict.md), and
452
+ [the research behind it](docs/evidence/04-spec-kit-bmad-superpowers.md).
453
+ - [All of it](docs/evidence/README.md), including every ruling and the data.
454
+ - [DESIGN.md](DESIGN.md): how the tool works and its known limits, and
455
+ [the review that shaped it](docs/evidence/10-design-review.md).
456
+
457
+ ## Limits
458
+
459
+ - The method came from one project, measured by itself. The comparisons since are small: one model, one run a
460
+ task, and no difference on real bug fixes has reached its threshold (see the status note above). The harness is
461
+ in [`bench/`](bench/SPEC.md). Its report states trust as numbers per arm: claim
462
+ precision, silent failures, the cheat and park rates on impossible tasks, and the own check against the hidden tests
463
+ by change size. A cheat is counted only by the routes a script can see (a DONE claim, an edited visible test, a test
464
+ input quoted into the code).
465
+ - The check costs three runs of the proof, plus one per small break (at most eight by default; under `report` it
466
+ stops once the breaks took about as long as the core runs), so the proof should be the narrowest command that
467
+ shows the outcome. With a 0.8 s jest proof that is about 2.4 s for the core and about as much again for strength.
468
+ A Python proof adds one traced run: on the demo 0.1 s (core runs 0.1 s each), on a pytest fixture 0.4 s (core
469
+ runs 0.4 s each), measured on one machine with Python 3.13; tracing a long suite slows it more.
470
+ - While the change is taken out, files are rewritten in place. They are put back on interruption and an edit made
471
+ meanwhile is never overwritten, but a dev server or file watcher will see the revert.
472
+ - State outside tracked files (installed dependencies, caches) is not reverted; a dependency bump or a pure refactor
473
+ uses `counterfactual: skip - <reason>`, which the gate names in its result.
474
+ - A proof that needs a real device, account or secret cannot be recomputed in CI.
475
+ - The verification cache does not see ignored files, installed packages, the environment or the clock; use
476
+ `--no-cache` when the result depends on them. Locally, anything that can write the git directory can plant a
477
+ cached pass for `odd done`; only `gate --recompute`, which ignores the local cache, decides.
478
+ - Windows is untested.
479
+
480
+ The full list is in [DESIGN.md](DESIGN.md#10-limits).
481
+
482
+ ## Develop
483
+
484
+ ```bash
485
+ python3 -m unittest discover -s tests
486
+ ```
487
+
488
+ MIT licence. See [LICENSE](LICENSE).