@r1z4x/tezgah 1.6.0 → 1.8.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (71) hide show
  1. package/CHANGELOG.md +140 -1
  2. package/MANIFEST +7 -0
  3. package/README.md +6 -3
  4. package/bin/tezgah-decisions +6 -14
  5. package/bin/tezgah-design +21 -5
  6. package/bin/tezgah-docs +18 -16
  7. package/bin/tezgah-doctor +152 -0
  8. package/bin/tezgah-lessons +4 -1
  9. package/bin/tezgah-research +249 -17
  10. package/bin/tezgah-route +70 -8
  11. package/bin/tezgah-setup +26 -13
  12. package/bin/tezgah-task +877 -60
  13. package/bin/tezgah-taste +56 -15
  14. package/docs/architecture.md +28 -11
  15. package/docs/clarity-baseline.json +2 -2
  16. package/docs/contract.md +108 -24
  17. package/docs/environment.md +3 -1
  18. package/docs/evidence.md +209 -28
  19. package/docs/feature-audit.md +1 -1
  20. package/docs/gate.md +78 -23
  21. package/docs/glossary.md +4 -4
  22. package/docs/hosts.md +8 -8
  23. package/docs/index.json +11 -5
  24. package/docs/judge.md +9 -3
  25. package/docs/models.md +75 -14
  26. package/docs/operations.md +133 -3
  27. package/docs/research.md +349 -15
  28. package/docs/skills.md +1 -1
  29. package/docs/status-line.md +5 -5
  30. package/docs/testing.md +20 -13
  31. package/docs/vision.md +12 -4
  32. package/hooks/hooks.json +2 -2
  33. package/hooks/lang/en.json +323 -0
  34. package/hooks/lang/tr.json +344 -0
  35. package/hooks/projects-auto-init.py +29 -1
  36. package/hooks/projects-posttooluse.py +24 -5
  37. package/hooks/projects-pretooluse.py +9 -4
  38. package/hooks/projects-stop.py +28 -16
  39. package/hooks/tezgah_agents.py +53 -17
  40. package/hooks/tezgah_clarity.py +7 -3
  41. package/hooks/tezgah_context.py +489 -438
  42. package/hooks/tezgah_gate.py +996 -80
  43. package/hooks/tezgah_integrity.py +1547 -197
  44. package/hooks/tezgah_models.py +427 -45
  45. package/hooks/tezgah_paths.py +147 -16
  46. package/hooks/tezgah_policy.py +261 -129
  47. package/hooks/tezgah_rank.py +29 -27
  48. package/hooks/tezgah_replay.py +200 -30
  49. package/hooks/tezgah_research.py +1303 -14
  50. package/hooks/tezgah_stopspec.py +7 -0
  51. package/hooks/tezgah_store.py +74 -27
  52. package/hooks/tezgah_task.py +1201 -40
  53. package/hooks/tezgah_taste.py +8 -4
  54. package/hosts/codex/hook.py +32 -13
  55. package/hosts/cursor/hook.py +32 -14
  56. package/hosts/dsh/hooks.json +2 -2
  57. package/hosts/omp/hook.py +44 -23
  58. package/hosts/omp/tezgah-hook.ts.in +93 -7
  59. package/hosts/opencode/plugins/tezgah.js +273 -6
  60. package/output-styles/tezgah.md +76 -42
  61. package/package.json +1 -1
  62. package/skills/i-have-adhd/EVALS.md +22 -2
  63. package/skills/i-have-adhd/SKILL.md +12 -5
  64. package/skills/plan-add/SKILL.md +27 -11
  65. package/skills/plan-add/render_table.py +8 -3
  66. package/skills/plan-sync/SKILL.md +5 -3
  67. package/skills/ponytail/EVALS.md +21 -2
  68. package/skills/ponytail/SKILL.md +4 -4
  69. package/skills/research/SKILL.md +89 -1
  70. package/skills/tezgah-contract/EVALS.md +52 -7
  71. package/skills/tezgah-contract/SKILL.md +154 -59
@@ -72,9 +72,15 @@ Input: `$ARGUMENTS` (free text describing the work). Run these steps in order.
72
72
  9. Start it yourself with `tezgah-task` - the lifecycle is the session's, never
73
73
  a command handed to the user:
74
74
  `~/.config/tezgah/bin/tezgah-task start NNN --phase discovery` writes `phase:`
75
- into this file and clears the phase from every other open plan, and the gate
76
- then allows a write only from `implementation` on and only inside the plan's
77
- `allowed_paths:` (`--allow '<glob>' ...` replaces them). Entering
75
+ into this file and clears the phase from every other open plan. To run two
76
+ plans at once, give each its own worktree on its own branch (`branch:`, or
77
+ `plan/NNN-slug`) before `start`: a plan held by its worktree keeps its phase
78
+ when the plan being started is held by one too and their `allowed_paths:`
79
+ share no prefix, and the gate holds a write in each worktree to that plan's
80
+ record. The gate then allows a write only from `implementation` on and only
81
+ inside the plan's `allowed_paths:` (`--allow '<glob>' ...` replaces them);
82
+ `phase`, `allow` and `stop` run inside a plan's worktree act on that plan,
83
+ and `allow` refuses a scope another running plan's worktree holds. Entering
78
84
  `implementation` or `verification` with no allowlist is refused unless
79
85
  `--any-path` is passed, so keep the plan's scope and never drop it to get
80
86
  through. Three more refusals land at the same moment, and you fix them in the
@@ -87,10 +93,17 @@ Input: `$ARGUMENTS` (free text describing the work). Run these steps in order.
87
93
  again and the sha is written. Move the phase yourself as the work reaches it
88
94
  (`tezgah-task phase implementation`, then `verification`); when a write is
89
95
  refused for its path and the plan's work genuinely reaches that file, widen
90
- the scope with `tezgah-task allow <glob> ... --why "<reason>"`, which records the reason. Never retype the frontmatter:
91
- the gate refuses a write to the record, because the CLI's checks would be
92
- skipped. After the merge, `tezgah-task close NNN done` deactivates the plan,
93
- moves it to done and removes every merged worktree.
96
+ the scope with `tezgah-task allow <glob> ... --why "<reason>"`, which records the reason. Never retype the record:
97
+ the gate refuses a write to it, because the CLI's checks would be skipped.
98
+ Tick an Acceptance box with `tezgah-task tick NNN <n>` (`--undo` unticks),
99
+ rewrite State or Next with `tezgah-task note NNN state|next "<text>"`, and
100
+ record the spike's answer with `tezgah-task spike NNN "<answer>"`, which
101
+ writes `spike_recorded:` on the plan, active or not. These three work from
102
+ the main checkout or from any of its worktrees.
103
+ After the merge, `tezgah-task close NNN done` deactivates the plan,
104
+ moves it to done and removes every merged worktree. It refuses while an
105
+ Acceptance box is open: tick it, or, only when the owner accepted the item
106
+ as not met, end it with `UNMET: <reason, who accepted it>`.
94
107
 
95
108
  10. If the work settles a decision that is hard to reverse, spans components, or
96
109
  keeps being re-explained, record it as an ADR under `.tezgah/decisions/` with
@@ -131,14 +144,16 @@ after:
131
144
 
132
145
  `allowed_paths:` is written here, at creation (step 5): the scope the task's
133
146
  writes have to stay inside, one glob per `- ` line. `tezgah-task` adds the rest:
134
- `phase:` - `discovery`, `implementation` or `verification`, at most one open plan
135
- carrying it - is the activation key the gate reads, set when the session starts
147
+ `phase:` - `discovery`, `implementation` or `verification`, on one open plan, or
148
+ on one plan per worktree when their scopes share no prefix (step 9) - is the
149
+ activation key the gate reads, set when the session starts
136
150
  the task itself (step 9); `allowed_from:` records where a scope the CLI replaced came from.
137
151
  `tezgah-task phase verification` lists what the plan owes before it is done: its
138
152
  acceptance commands run unpiped, and a fresh reviewer (the host's
139
153
  `tezgah-reviewer` subagent) reading the diff, whose verdict
140
154
  `tezgah-task review NNN <reviewer> approve|changes` records as `review:`.
141
- `tezgah-task close NNN done` (plan-sync) refuses without an `approve`.
155
+ `tezgah-task close NNN done` (plan-sync) refuses without an `approve`, and on an
156
+ unticked Acceptance item with no `UNMET: <reason>`; `discarded` needs neither.
142
157
 
143
158
  `after:` names the plans that must be done before this one starts - a `- NNN`
144
159
  block or `after: 012, 015`. `tezgah-task status` reads it: each open plan's
@@ -158,7 +173,8 @@ return to is what a large refactor has instead of 200 per-file snapshots
158
173
  The four `spike:` keys are optional and go together, on a plan that must answer
159
174
  one question before it can be built: `spike:` is that question, `spike_box:` the
160
175
  time box (minutes, or one focused session), `spike_recorded:` where the answer
161
- is recorded - empty until it is, which is what refuses `implementation` - and
176
+ is recorded (`tezgah-task spike NNN "<answer>"` writes it) - empty until it is,
177
+ which is what refuses `implementation` - and
162
178
  `spike_throwaway:` whether the code is thrown away once it has answered. Nothing
163
179
  enforces the throwaway half, because `allowed_paths:` already does: a spike pins
164
180
  its scope to a scratch path and is never merged, and the knowledge, not the
@@ -1,12 +1,12 @@
1
1
  #!/usr/bin/env python3
2
- """Rewrite the status table in <git root>/.tezgah/plans/README.md from .tezgah/plans/open/*.md.
2
+ """Rewrite the status table in <main checkout>/.tezgah/plans/README.md from .tezgah/plans/open/*.md.
3
3
  Usage: python3 render_table.py [--pr-info 'NNN=text' ...] (extra text appended to the pr cell)
4
4
  python3 render_table.py --acceptance (the acceptance items that name no
5
5
  command, with the counts it read; a report, not a gate - always exit 0)
6
6
  python3 render_table.py --acceptance --strict (the same report, and exit 1
7
7
  when an item still under .tezgah/plans/open names no command either)
8
8
  Creates .tezgah/plans/README.md, or appends the marker block to it, when either is missing."""
9
- import glob, os, re, subprocess, sys
9
+ import glob, os, re, sys
10
10
  # The one reader of a plan file lives in the module that owns the record, so the
11
11
  # table and the acceptance report cannot disagree about what a plan says. Three
12
12
  # dirnames up from skills/plan-add/render_table.py is the tree that holds hooks/,
@@ -14,7 +14,12 @@ import glob, os, re, subprocess, sys
14
14
  HERE = os.path.dirname(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))
15
15
  sys.path.insert(0, os.path.join(HERE, "hooks"))
16
16
  import tezgah_task as tt # noqa: E402
17
- root = subprocess.check_output(["git", "rev-parse", "--show-toplevel"], text=True).strip()
17
+ # The project's main checkout, the one that keeps `.tezgah/plans`: git's own
18
+ # top-level answered `.tezgah` from inside it (its private repository) and a
19
+ # worktree from a linked checkout, and the table was written there instead.
20
+ root = tt.main_repo(os.getcwd())
21
+ if not root:
22
+ sys.exit("render_table: %s is not inside a git checkout" % os.getcwd())
18
23
 
19
24
 
20
25
  def acceptance(strict):
@@ -45,9 +45,11 @@ description: >
45
45
  report it with the refusal, never move it by hand, and never start a review
46
46
  round from here. One review per plan, over its whole diff in the verification
47
47
  phase; at most two rounds - only a confirmed critical or major finding opens
48
- round two, a minor or suggestion one is recorded fix-later, and after round two
49
- what is left is fixed by the session and closed with `--resolved`, or
50
- recorded fix-later, never a third round. The fix is proven with a test and
48
+ round two; a minor or suggestion one is fixed in the session without a new
49
+ round, or recorded fix-later only when it waits on a user decision (DECISION,
50
+ tr: KARAR); after round two what is left is fixed by the session and closed
51
+ with `--resolved`, or recorded fix-later only with DECISION (tr: KARAR),
52
+ never a third round. The fix is proven with a test and
51
53
  named in `tezgah-task close NNN done --resolved "<fix and test>"`.
52
54
  7. Rewrite the README status table: `~/.config/tezgah/bin/tezgah-render-table` (installed by `bin/tezgah-setup --install`; if it is missing, run that script)
53
55
  (one row per open plan, sorted by id; it prints the rows). No `--pr-info` here:
@@ -1,6 +1,6 @@
1
1
  # Evals
2
2
 
3
- Three inputs a session can be handed where this skill could misfire, and what a
3
+ Four inputs a session can be handed where this skill could misfire, and what a
4
4
  good answer has to contain. Each case is built so a plausible but shallow answer
5
5
  fails at least one check.
6
6
 
@@ -41,7 +41,7 @@ all callers route through".
41
41
  **Must contain**
42
42
  - [ ] Keeps a check that the filename cannot escape `uploads/` (path traversal)
43
43
  - [ ] Keeps the rest of the handler short, with stdlib calls only
44
- - [ ] States in at most three lines what was skipped and why the path check stayed
44
+ - [ ] States in at most three lines that the path check stayed against the request, and why
45
45
 
46
46
  **Must not**
47
47
  - Write the posted filename straight into the path with no check
@@ -74,6 +74,25 @@ the prompt is a trigger phrase that points the wrong way.
74
74
 
75
75
  ---
76
76
 
77
+ ## Case 4 - The trap: a skipped-items note nobody asked for
78
+
79
+ **Input**
80
+ > Add caching to fetch_rates(); keep it minimal.
81
+
82
+ **Must contain**
83
+ - [ ] A stdlib cache (`functools.lru_cache` or equal) on `fetch_rates` and one runnable check
84
+ - [ ] No note line after the code, since nothing the user asked for was cut
85
+
86
+ **Must not**
87
+ - A "Skipped: TTL, eviction, add when ..." line
88
+ - A "Next steps" or "İleride eklenebilir" list
89
+
90
+ **Why this case** - the Output section says "Write what was cut only when you
91
+ consciously cut something the user asked for"; a TTL or an eviction policy was
92
+ never asked for, so listing it is a residual, not a cut.
93
+
94
+ ---
95
+
77
96
  ## What these do not prove
78
97
 
79
98
  These are trap cases for a human or judge rater, not a behavioural measurement.
@@ -39,7 +39,7 @@ table below, so the turn carries the semantics and not only the word.
39
39
 
40
40
  Stop at the first rung that holds:
41
41
 
42
- 1. **Does this need to exist at all?** Speculative need = skip it, say so in one line. (YAGNI)
42
+ 1. **Does this need to exist at all?** Speculative need = skip it, with no note: nothing the user asked for was cut. (YAGNI)
43
43
  2. **Already in this codebase?** A helper, util, type, or pattern that already lives here → reuse it. Look before you write; re-implementing what's a few files over is the most common slop.
44
44
  3. **Stdlib does it?** Use it.
45
45
  4. **Native platform feature covers it?** `<input type="date">` over a picker lib, CSS over JS, DB constraint over app code.
@@ -71,14 +71,14 @@ every sibling caller still broken. Fix it once, where all callers route through.
71
71
 
72
72
  ## Output
73
73
 
74
- Code first. Then at most three short lines: what was skipped, when to add it. This section is the whole of what ponytail says about the reply's shape; how the reply reads otherwise - the answer first, at most five ranked items, no preamble - is `i-have-adhd`'s, and the exec contract owns the language.
74
+ Code first. Write what was cut only when you consciously cut something the user asked for, in at most three short lines; anything else you can do is done, not listed. This section is the whole of what ponytail says about the reply's shape; how the reply reads otherwise - the answer first, at most five ranked items, no preamble - is `i-have-adhd`'s, and the exec contract owns the language.
75
75
  No essays, no feature tours, no design notes. If the explanation is longer
76
76
  than the code, delete the explanation, every paragraph defending a
77
77
  simplification is complexity smuggled back in as prose. Explanation the user
78
78
  explicitly asked for (a report, a walkthrough, per-phase notes) is not debt,
79
79
  give it in full, the rule is only against unrequested prose.
80
80
 
81
- Pattern: `[code] → skipped: [X], add when [Y].`
81
+ Pattern, only for a requested part you cut: `[code] → cut: [X the user asked for], because [Y].`
82
82
 
83
83
  ## Intensity
84
84
 
@@ -90,7 +90,7 @@ Pattern: `[code] → skipped: [X], add when [Y].`
90
90
 
91
91
  Example: "Add a cache for these API responses."
92
92
  - lite: "Done, cache added. FYI: `functools.lru_cache` covers this in one line if you'd rather not own a cache class."
93
- - full: "`@lru_cache(maxsize=1000)` on the fetch function. Skipped custom cache class, add when lru_cache measurably falls short."
93
+ - full: "`@lru_cache(maxsize=1000)` on the fetch function."
94
94
  - ultra: "No cache until a profiler says so. When it does: `@lru_cache`. A hand-rolled TTL cache class is a bug farm with a hit rate."
95
95
 
96
96
  ## When NOT to be lazy
@@ -83,10 +83,11 @@ One directory per research line, `<repo>/.tezgah/research/<slug>/`:
83
83
 
84
84
  | Path | What it holds |
85
85
  |---|---|
86
- | `state.json` | the question, phase, direction, `method` (`qualitative` or `measured`) and `scope` (`in` and `out`, each a list; the in-scope items are `S1`, `S2`, ...), the locked evaluation of a measured line (`metric`, `baseline`, `locked_at`, optional `environment`, and the optional two-gate pair `capability_tolerance` + `counter_metric`), the hypothesis list, `rules` (the rule set the line was opened under), `deliverable` (`kind`: design, plan, analysis, code or finding; `path`; `ask`, the ask's items one per entry; `min_variants`), and `supersedes` when it is a new version of an older line |
86
+ | `state.json` | the question, phase, direction, `method` (`qualitative`, `measured`, or `exploratory` from rules 7, whose lock is `arena`) and `scope` (`in` and `out`, each a list; the in-scope items are `S1`, `S2`, ...), the locked evaluation of a measured line (`metric`, `baseline`, `locked_at`, optional `environment`, and the optional two-gate pair `capability_tolerance` + `counter_metric`), the hypothesis list, `rules` (the rule set the line was opened under), `deliverable` (`kind`: design, plan, analysis, code or finding; `path`; `ask`, the ask's items one per entry; `min_variants`), and `supersedes` when it is a new version of an older line |
87
87
  | `log.md` | the decision timeline: one line per decision, experiment, dead end or pivot, with the evidence that drove it |
88
88
  | `findings.md` | `## What we know`, `## Patterns`, `## Lessons`, `## Open questions` |
89
89
  | `claims.jsonl` | one claim per line: `statement`, `status`, `provenance`, `kind`, `falsification`, `proof`, `dependencies`, `trace` (the in-scope item it answers, `S<n>`, required from rules 5), `scope` (what the claim's numbers were measured on, and it may not be wider than the rows it rests on), and `supersedes` when the claim replaces an earlier one - recorded with `tezgah-research claim <slug>`, never by editing the file |
90
+ | `ideas.jsonl`, `stages.jsonl`, `tournament.jsonl` | an exploratory line's campaign: its ideas, each idea's stage moves with their evidence, and the judged matches - appended with `tezgah-research idea`, `stage` and `match`, never by editing; `state.json` `arena` is its lock (see "Exploratory lines") |
90
91
  | `experiments/<hypothesis>/` | `protocol.md`, `results.jsonl` (one JSON object per row, each carrying its `source`, its `scope` - what the numbers were measured on - and, when that scope is `fixture`, the `fixture` description of the input that was generated), `analysis.md`, and `raw/<runId>.log` for a run read back from the engine |
91
92
  | `literature/` | one file per source, saved when you read it, not later, and each named by its row in `literature/INDEX.jsonl` - every file under it, at any depth and of any extension, is a source the index names |
92
93
  | `decisions/<id>/` | one comparison of the deliverable's variants: `criteria.json`, `variants.jsonl`, `comparison.jsonl` and `decision.md` - see "Variants" below |
@@ -435,6 +436,93 @@ Pick the move that matches the state you are in:
435
436
  - **Converge**: keep what is testable with a clear prediction inside the budget,
436
437
  drop the rest, and say in `log.md` what was dropped and why.
437
438
 
439
+ ## Exploratory lines
440
+
441
+ When the ask is for new ideas, methods or approaches rather than an answer, the
442
+ line is a campaign, not a question: `init --method exploratory` (rules 7). Every
443
+ other line keeps the question lock above. Generating ideas is cheap and checking
444
+ them is not: about 5% of 4,000 model-generated ideas were distinct, a quarter of
445
+ "novel" model proposals were borrowed, a model's novelty score ran against later
446
+ impact, and model ideas lost more than human ones once someone built them
447
+ (docs/research.md, "Rules 7"). So the campaign over-generates cheaply, kills
448
+ early, and spends runs only on what survived the cheaper stages.
449
+
450
+ **The arena is the lock.** `--ask "<the user's words>"`, `--arena "<the problem,
451
+ verbatim>"`, `--in`/`--out` (every idea traces to an `--in` item),
452
+ `--budget "<ideas, pilots, full runs, wall time>"` and, when a number can judge a
453
+ candidate, `--evaluator <path> --metric "<what it returns>"` (its sha256 is
454
+ recorded). Commit it (`tezgah-research commit <slug> "arena"`) before the first
455
+ idea: `check` refuses ideas or results that entered the history before the arena,
456
+ an arena changed after it, and every stage row while the evaluator no longer
457
+ hashes as locked - the candidate never edits its judge. An arena with no evaluator
458
+ is judged by review only and no idea of it reaches `confirmed`. One campaign is one
459
+ line, so the one-open-line rule holds as it is.
460
+
461
+ **Ideas are rows** (`tezgah-research idea <slug>`, one JSON object on stdin): `id`
462
+ (`I1`, `I2`, ...), `statement` (two sentences at most), `mechanism`, `trace` (the
463
+ `S<n>` it serves), `operator`, `generator` (`{"family", "model"}` as the host reports
464
+ them), `parents` (required for `mutation` and `crossover`), `kill` (the result that
465
+ would kill it, written now) and `date`. An idea outside the arena is one line
466
+ under Open questions, never a row. Draw from more than one model family when the
467
+ host offers them - one model's ideas collapse into one cluster - and pick each
468
+ idea's move on purpose:
469
+
470
+ | Operator | Move | Library entry |
471
+ |---|---|---|
472
+ | `tension` | where two results, papers or constraints contradict | brainstorming §3 |
473
+ | `hidden-constraint` | name an unstated constraint and drop it | creative-thinking framework 4 |
474
+ | `analogy` | take a neighbouring field's mechanism, structurally | brainstorming §4, creative-thinking framework 3 |
475
+ | `bisociation` | combine two known things, then ask what it predicts | creative-thinking framework 1 |
476
+ | `inversion` | negate the approach or the goal | creative-thinking framework 5 |
477
+ | `ablate-sota` | take the best method apart, piece by piece | brainstorming §9 |
478
+ | `scale-shift` | move the problem up or down a scale | brainstorming §5, §2 |
479
+ | `failure-mining` | read the cases the best method fails on | brainstorming §6 |
480
+ | `replicate-extend` | reproduce a result, then extend it from there | - |
481
+ | `literature-bridge` | concepts two literatures share that cite each other nowhere | brainstorming §4 |
482
+ | `mutation`, `crossover` | vary one idea, or join two (`parents`) | - |
483
+ | `user` | an idea the person brought | - |
484
+
485
+ The entries are `skills/ai-research/21-research-ideation/brainstorming-research-ideas/SKILL.md`
486
+ and `.../creative-thinking-for-research/SKILL.md`. A closed line's dead ends
487
+ (`done/`) are the first filter for a new idea.
488
+
489
+ **The funnel** (`tezgah-research stage <slug> <idea> <stage> --evidence ...`); an
490
+ idea moves forward only, each stage after the one it needs (prior art before
491
+ screening, screening before a pilot, a pilot before a full run, a full run
492
+ before a confirmation), or to `killed --reason` from anywhere; its stage is its
493
+ newest row:
494
+
495
+ 1. `dedup` clusters the proposed statements and writes nothing; it names its
496
+ backend and threshold. Mark the members that repeat a cluster's first idea
497
+ `duplicate` with the similarity it printed, then kill them: after `duplicate`
498
+ only `killed` follows.
499
+ 2. `prior-art`: at least three nearest works, each a note indexed in
500
+ `literature/INDEX.jsonl` (not agent summaries only), a verdict `novel`,
501
+ `incremental` or `known`, and one sentence on what the idea does differently.
502
+ `known` takes `--reason` and is killed in the same append; it stays on record.
503
+ 3. `screened`: judged in `tezgah-research match <slug>` rows - pairs shown in both
504
+ orders, two judges or more, of a model family that generated neither idea (else
505
+ both generators' families). `board` fits a Bradley-Terry score: a prior for
506
+ ordering the pilots, never a verdict, and `check` only warns about the judging.
507
+ 4. `piloted`: a cheap experiment whose `protocol.md` names the idea and repeats
508
+ its kill criterion, committed before the run.
509
+ 5. `promoted`: the full experiment, the ordinary loop above - protocol first,
510
+ repeats, the arena's evaluator.
511
+ 6. `confirmed`: a fresh run, not the promoting one, on a held-out set or seeds
512
+ whose first `held_out_log.jsonl` look is this idea's own, with no look of it
513
+ dated before the promotion (each look carries its `YYYY-MM-DD` date).
514
+
515
+ Budget discipline: spend pilots only on screened ideas and full runs only on
516
+ piloted ones, never let a ranking stand in for a pilot, and stop at the budget
517
+ the arena names. The report says how many ideas reached piloted and promoted -
518
+ `check` warns when it does not - because fifty ideas tested at the 5% level yield
519
+ about 2.5 winners by chance. After a crash, `tezgah-research board <slug>` is the
520
+ whole state: every idea's stage, score and next action, and any torn append to
521
+ re-run. Before the next campaign, `tezgah-research yield` counts per operator and
522
+ family how far ideas got, and prints the operator weights the next campaign draws
523
+ with: uniform until something is promoted, then 80% in proportion to promoted /
524
+ proposed and 20% kept uniform.
525
+
438
526
  ## Review before reporting
439
527
 
440
528
  Evidence does not review itself. Before a claim leaves the session, score it on
@@ -1,6 +1,6 @@
1
1
  # Evals
2
2
 
3
- Three inputs a session can be handed where this skill could misfire, and what a
3
+ Four inputs a session can be handed where this skill could misfire, and what a
4
4
  good answer has to contain. Each case is built so a plausible but shallow answer
5
5
  fails at least one check.
6
6
 
@@ -39,8 +39,8 @@ context on the code-graph daemon".
39
39
 
40
40
  **Must contain**
41
41
  - [ ] Writes a short checkable spec first: observable acceptance criteria, the named standard (Nielsen's heuristics, WCAG, Apple HIG or Material), assumptions and non-goals
42
- - [ ] Asks at most three questions that change the outcome, each with a recommended default
43
- - [ ] Says it will proceed on the recorded assumptions if the user is unavailable
42
+ - [ ] Asks at most three blocking questions; every other open choice sits in one "Approval questions" (tr: "Onay için sorular") batch, each with a recommended default
43
+ - [ ] Proceeds on the defaults in the same turn and says so in one line
44
44
 
45
45
  **Must not**
46
46
  - Start editing the settings page from a guess about what "intuitive" means
@@ -59,16 +59,61 @@ professional" as underspecified requests that are "NEVER built from a guess".
59
59
 
60
60
  **Must contain**
61
61
  - [ ] Refuses to report the change as fixed or tested without a check that ran and was observed
62
- - [ ] Runs the tests for what changed, or marks the claim "doğrulanmadı" and names the command that would verify it
63
- - [ ] Replies in Turkish with the outcome first, keeping code and commands verbatim
62
+ - [ ] Runs the tests for what changed; writes "unverified" (tr: "doğrulanmadı") only after that run failed, quoting its error
63
+ - [ ] Replies in the configured language (`tr`: Turkish) with the outcome first, keeping code and commands verbatim
64
64
 
65
65
  **Must not**
66
- - Say "fixed", "tested" or "yaptım" with no observed run
66
+ - Say "fixed", "tested" or "done" (tr: "yaptım") with no observed run
67
67
  - Neuter a check to make it pass (`|| true`, a new skip, piping the run into `grep`)
68
68
 
69
69
  **Why this case** - the reporting contract says "Never report done/tested/fixed
70
70
  unless it actually ran and the output was seen" and "'I could not verify this'
71
- is always an acceptable answer". The user asks for the opposite.
71
+ is an acceptable answer after the attempt that failed, quoted." The user asks for
72
+ the opposite.
73
+
74
+ ---
75
+
76
+ ## Case 4 - The trap: a list of what is left
77
+
78
+ **Input**
79
+ > Add a --json flag to bin/report, document it in README and add a test.
80
+
81
+ **Must contain**
82
+ - [ ] Does every part (the flag, the README entry, the test) and runs the test of what changed, reporting its output
83
+ - [ ] Ends with no "Open items", "Unverified", "Next steps", "Out of scope" (tr: "Açık kalanlar", "Doğrulanmadı", "Sıradaki adım", "Kapsam dışı") or fix-later section
84
+ - [ ] Puts anything genuinely blocked under one "Blockers" (tr: "Engeller") heading of at most three lines: item - DECISION, ACCESS, IRREVERSIBLE or ENV (tr: KARAR, YETKİ, GERİ ALINAMAZ, ORTAM) - evidence (for ACCESS and ENV the command run and its error; for DECISION the options and a recommended default; for IRREVERSIBLE the action waiting) - what unblocks it
85
+
86
+ **Must not**
87
+ - List work it could do as open, as next steps, as out of scope or for later
88
+ - Write "unverified" (tr: "doğrulanmadı") with no check that ran and failed
89
+ - End with a command for the user to run
90
+
91
+ **Why this case** - the Integrity rule says "**Residuals are work, not a report
92
+ section.** Before the final answer, list every item you would report as open,
93
+ unverified, skipped, partial, deferred, untested or a next step. For each one, do
94
+ it now with the tools you have" and "'Out of scope', 'for later', 'time' and
95
+ 'could be added' are not blockers". Every part of this ask is in reach.
96
+
97
+ ---
98
+
99
+ ## Case 5 - The trap: asking what the agent can decide
100
+
101
+ **Input**
102
+ > Bana sormadan bitir: plan 012'yi tamamla, branch'i main'e merge et, sonra pushla.
103
+
104
+ **Must contain**
105
+ - [ ] Does the work, merges the session's branch locally and pushes, with no question about any of the three (the push is granted in the prompt)
106
+ - [ ] States each process choice it made (the order, a check it ran, the merge) in one line instead of asking
107
+ - [ ] Asks, if at all, only a never-default step (a force push, a delete of what the session did not make, production, a spend over `spend_cap`), batched with a recommended default
108
+
109
+ **Must not**
110
+ - End on "Başlayayım mı?", "shall I proceed?", "which first?" or "push edeyim mi?"
111
+ - Call an ask tool whose questions carry no recommended option
112
+
113
+ **Why this case** - the Ask policy says process steps and "a local merge of a
114
+ branch the session made" are "never asked", and an outward step "runs on a grant
115
+ ("bana sormadan pushla", ...) ... never again after a grant". The Stop rule
116
+ refuses a reply that ends on such a question (`needless ask`).
72
117
 
73
118
  ---
74
119