@r1z4x/tezgah 1.5.1 → 1.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +126 -1
- package/MANIFEST +11 -0
- package/README.md +6 -3
- package/bin/consult +13 -10
- package/bin/tezgah-docs +18 -16
- package/bin/tezgah-doctor +152 -0
- package/bin/tezgah-research +286 -12
- package/bin/tezgah-route +48 -1
- package/bin/tezgah-setup +27 -14
- package/bin/tezgah-task +771 -46
- package/bin/tezgah-taste +48 -10
- package/docs/architecture.md +7 -4
- package/docs/citations-baseline.json +1 -1
- package/docs/clarity-baseline.json +5 -5
- package/docs/contract.md +112 -25
- package/docs/evidence.md +201 -28
- package/docs/feature-audit.md +25 -4
- package/docs/gate.md +70 -33
- package/docs/glossary.md +4 -4
- package/docs/hosts.md +8 -8
- package/docs/index.json +30 -10
- package/docs/judge.md +9 -3
- package/docs/models.md +35 -2
- package/docs/operations.md +103 -3
- package/docs/research.md +392 -13
- package/docs/skills.md +24 -6
- package/docs/status-line.md +5 -5
- package/docs/testing.md +2 -2
- package/docs/vision.md +12 -4
- package/hooks/hooks.json +2 -2
- package/hooks/lang/en.json +323 -0
- package/hooks/lang/tr.json +344 -0
- package/hooks/projects-posttooluse.py +24 -5
- package/hooks/projects-pretooluse.py +9 -4
- package/hooks/projects-stop.py +28 -16
- package/hooks/tezgah_agents.py +29 -9
- package/hooks/tezgah_clarity.py +7 -3
- package/hooks/tezgah_components.py +2 -1
- package/hooks/tezgah_context.py +460 -390
- package/hooks/tezgah_gate.py +640 -104
- package/hooks/tezgah_integrity.py +1480 -197
- package/hooks/tezgah_models.py +337 -15
- package/hooks/tezgah_paths.py +103 -10
- package/hooks/tezgah_policy.py +286 -130
- package/hooks/tezgah_product.py +107 -0
- package/hooks/tezgah_rank.py +29 -27
- package/hooks/tezgah_replay.py +200 -30
- package/hooks/tezgah_research.py +1297 -12
- package/hooks/tezgah_skill_pick.py +58 -17
- package/hooks/tezgah_stopspec.py +7 -0
- package/hooks/tezgah_store.py +74 -27
- package/hooks/tezgah_task.py +917 -8
- package/hosts/codex/hook.py +9 -10
- package/hosts/cursor/hook.py +21 -14
- package/hosts/dsh/hooks.json +2 -2
- package/hosts/omp/hook.py +44 -23
- package/hosts/omp/tezgah-hook.ts.in +93 -7
- package/hosts/opencode/plugins/tezgah.js +119 -39
- package/output-styles/tezgah.md +76 -42
- package/package.json +1 -1
- package/skills/analyze-app/SKILL.md +57 -0
- package/skills/design-contract/EVALS.md +24 -1
- package/skills/design-contract/SKILL.md +17 -1
- package/skills/design-contract/data-views.md +86 -0
- package/skills/feature-audit/SKILL.md +32 -3
- package/skills/i-have-adhd/EVALS.md +22 -2
- package/skills/i-have-adhd/SKILL.md +12 -5
- package/skills/plan-add/EVALS.md +8 -8
- package/skills/plan-add/SKILL.md +22 -21
- package/skills/plan-sync/EVALS.md +2 -2
- package/skills/plan-sync/SKILL.md +14 -8
- package/skills/pm-frameworks/SKILL.md +6 -0
- package/skills/ponytail/EVALS.md +21 -2
- package/skills/ponytail/SKILL.md +4 -4
- package/skills/product-analysis/EVALS.md +23 -1
- package/skills/product-analysis/SKILL.md +50 -11
- package/skills/product-flow/EVALS.md +81 -0
- package/skills/product-flow/SKILL.md +91 -0
- package/skills/research/SKILL.md +99 -2
- package/skills/tezgah-contract/EVALS.md +52 -7
- package/skills/tezgah-contract/SKILL.md +204 -85
package/docs/research.md
CHANGED
|
@@ -65,7 +65,9 @@ nothing recorded the ask and `close --limit` accepted any string.
|
|
|
65
65
|
`state.json` `rules` is the line's format version. `init` writes `RULES`, the
|
|
66
66
|
newest set (`hooks/tezgah_research.py::RULES`). Each set keeps its own gate. Rules 2
|
|
67
67
|
are the standards rules. Rules 3 add the ask contract. Rules 4 add the protocol
|
|
68
|
-
sections below. Rules 5 add the scope lock.
|
|
68
|
+
sections below. Rules 5 add the scope lock. Rules 6 add the selection warnings
|
|
69
|
+
("Held-out set, repeats and noise band" below). Rules 7 add the `exploratory`
|
|
70
|
+
method ("Rules 7: exploratory lines" below). A bump never weakens an older set.
|
|
69
71
|
|
|
70
72
|
A `rules` above `RULES` means a later tezgah wrote the line. `check` then reports
|
|
71
73
|
one error that names both numbers and reads nothing else. The writers refuse the
|
|
@@ -121,7 +123,8 @@ ideation moves had no fence. Rules 5 lock a line to its question:
|
|
|
121
123
|
|
|
122
124
|
- `init --in "<covered>" --out "<left alone>"` (repeatable) write `state.json`
|
|
123
125
|
`scope`. The in-scope items are `S1`, `S2`, ... in order. `--method
|
|
124
|
-
qualitative|measured` writes `method
|
|
126
|
+
qualitative|measured|exploratory` writes `method` (`exploratory` from rules 7,
|
|
127
|
+
see "Rules 7: exploratory lines"). `scope_problems`
|
|
125
128
|
(`hooks/tezgah_research.py::scope_problems`) names an empty `scope.in`, an
|
|
126
129
|
empty `scope.out` and a missing method. `check` refuses them past `bootstrap`
|
|
127
130
|
and warns at it.
|
|
@@ -149,11 +152,14 @@ gets the question, its scope and its method in its brief. The research skill's
|
|
|
149
152
|
|
|
150
153
|
| Path | Holds |
|
|
151
154
|
|---|---|
|
|
152
|
-
| `state.json` | the question, the phase (`bootstrap`/`inner`/`outer`/`concluded`), the direction, `method` and `scope` (rules 5, see "Scope lock"), the locked evaluation and its optional second gate (`capability_tolerance`, `counter_metric`), the session events (`PHASES`, `hooks/tezgah_research.py::PHASES`), `rules` (the rule set `init` opened it under, `RULES`), `deliverable` (`kind`, `path`, `ask`, `min_variants`), `supersedes` for a new version of an older line, and `closed` once `close` records a deliberate limit. The phase is the author's declaration; `derived_phase` (`hooks/tezgah_research.py::derived_phase`) reads the *other* authority beside it from the line's own artifacts, and the checker names the two disagreeing |
|
|
155
|
+
| `state.json` | the question, the phase (`bootstrap`/`inner`/`outer`/`concluded`), the direction, `method` and `scope` (rules 5, see "Scope lock"), the locked evaluation and its optional second gate (`capability_tolerance`, `counter_metric`) with its `noise_band`, the optional `held_out` set (rules 6), the `arena` of an exploratory line (`problem`, `budget`, and `evaluator`, `evaluator_sha256`, `metric` for a measurable one - rules 7, see "The arena lock"), the session events (`PHASES`, `hooks/tezgah_research.py::PHASES`), `rules` (the rule set `init` opened it under, `RULES`), `deliverable` (`kind`, `path`, `ask`, `min_variants`), `supersedes` for a new version of an older line, and `closed` once `close` records a deliberate limit. The phase is the author's declaration; `derived_phase` (`hooks/tezgah_research.py::derived_phase`) reads the *other* authority beside it from the line's own artifacts, and the checker names the two disagreeing |
|
|
153
156
|
| `log.md` | the decision log, newest last: one line per decision, experiment, dead end or pivot, with the evidence that drove it |
|
|
154
157
|
| `findings.md` | the four sections every line answers, named by `FINDINGS_SECTIONS` (`hooks/tezgah_research.py::FINDINGS_SECTIONS`) |
|
|
155
158
|
| `claims.jsonl` | one JSON object per row: `statement`, `falsification`, `proof`, `provenance`, `status`, `kind`, `trace` (the in-scope item, rules 5), `scope` (what the claim's numbers were measured on, which may not be wider than the rows it rests on), and `supersedes` when the row replaces an earlier claim |
|
|
156
159
|
| `predictions.jsonl` | one JSON object per row: `commit` (a 40-character sha the row is bound to), `claim` (an id the line holds, or empty when the prediction stands alone), `metric`, `value_before`, `value_after`, `falsifier`, `components` (the keys of the components the change touches, read from `hooks/tezgah_components.py`), and `granted_by` when a human granted what the frozen paths below forbid |
|
|
160
|
+
| `ideas.jsonl` | an exploratory line's ideas, one JSON object per row, appended by `idea` (rules 7, see "Three files") |
|
|
161
|
+
| `stages.jsonl` | an exploratory line's stage moves, one row per move, appended by `stage`; an idea's stage is its newest row |
|
|
162
|
+
| `tournament.jsonl` | an exploratory line's judged pairwise matches, appended by `match` |
|
|
157
163
|
| `experiments/<hypothesis>/protocol.md` | what the change is, what it predicts, what result would falsify it, why - committed before the run |
|
|
158
164
|
| `experiments/<hypothesis>/results.jsonl` | the rows the run produced, one JSON object per non-blank line, each carrying a non-empty `source` and, once declared, a `scope` of `real`, `fixture` or `derived`, and - when that scope is `fixture` - a `fixture` description of what was generated |
|
|
159
165
|
| `experiments/<hypothesis>/analysis.md` | what the rows mean, and which claim they move |
|
|
@@ -168,7 +174,9 @@ gets the question, its scope and its method in its brief. The research skill's
|
|
|
168
174
|
file that exists (`init`, `hooks/tezgah_research.py::init`); `predictions.jsonl` is not
|
|
169
175
|
one of them, because a file that exists without a row is a line that looks like
|
|
170
176
|
it predicted something - it appears with the first `predict` the line records,
|
|
171
|
-
and `check` reads an absent one as no rows rather than as a missing file.
|
|
177
|
+
and `check` reads an absent one as no rows rather than as a missing file. The
|
|
178
|
+
three campaign files work the same way. Each appears with its first `idea`,
|
|
179
|
+
`stage` or `match`. A slug
|
|
172
180
|
is lowercase letters, digits and dashes (`valid_slug`,
|
|
173
181
|
`hooks/tezgah_research.py::valid_slug`), because it is the directory name `check`, `status` and
|
|
174
182
|
`claim` look for.
|
|
@@ -238,14 +246,66 @@ through its seal.
|
|
|
238
246
|
| `tezgah-research source <slug> <hypothesis> --run <orxRunId> [--command "..."] [--scope real\|fixture\|derived] [--fixture "<what was generated>"]` | keeps the receipt: runs `orx logs <runId>`, writes `raw/<runId>.log`, appends the results row `{"source": "orx:<runId>", ...}` and the scope and fixture description the filer states (`source_run`, `hooks/tezgah_research.py::source_run`). A `--scope fixture` filed without `--fixture` still writes the row and prints the field it still owes, so the tool is never the thing that makes its own checker warn silently; a given `--fixture` that is empty is misuse | 0, 1 nothing filed, 2 without orx |
|
|
239
247
|
| `tezgah-research compare <slug> <decision>` | reads one variants x criteria cell from stdin and appends it to `decisions/<decision>/comparison.jsonl` under the lock, or refuses it by the rule `check` applies (`append_comparison` and `comparison_problems`, `hooks/tezgah_research.py`); it notes when `criteria.json` is not committed yet | 0, 1 refused, 2 misuse |
|
|
240
248
|
| `tezgah-research close <slug> --limit "<reason>"` | concludes the line as a deliberate limit, writing the reasons it was still open into `state.json` `closed` and `log.md`, and seals it (`close_line`) | 0, 1 unreadable state, 2 misuse |
|
|
249
|
+
| `tezgah-research verdict <slug> <id> met\|not-met\|unanswerable --evidence "<ref>"` | records one success criterion's verdict and the evidence it rests on in the line's `state.json`, or refuses it with one `FAIL verdict:` line per problem (`verdict_problems`, `hooks/tezgah_research.py::verdict_problems`) | 0, 1 refused or unreadable state, 2 misuse |
|
|
250
|
+
| `tezgah-research conclude <slug>` | concludes a line once every success criterion carries a verdict, seals it and moves it to `done/` (`conclude_line`, `hooks/tezgah_research.py::conclude_line`) | 0, 1 refused, 2 misuse |
|
|
251
|
+
| `tezgah-research migrate-layout` | moves flat `research/<slug>` directories into `open/` or `done/` by each line's state, and is idempotent (`migrate_layout`, `hooks/tezgah_research.py::migrate_layout`) | 0, 2 misuse |
|
|
241
252
|
| `tezgah-research seal <slug> [--history-lost] --ack "<owner decision>"` | seals a line under `done/` that was concluded before seals existed: plainly, as `conclude` would, when every experiment's order still checks; with `--history-lost`, with that verdict for every experiment whose order the lost history left undecidable. Refuses a line with a real order violation, a plain seal on any order finding, and a `history-lost` seal with nothing lost (`retro_seal`, `hooks/tezgah_research.py::retro_seal`) | 0, 1 refused, 2 misuse |
|
|
242
253
|
| `tezgah-research import <checkout> [<slug>] [--allow-open "<reason>"]` | brings a line - or every line this checkout lacks - from another checkout's `.tezgah` repository with its history; an open line arrives beside open ones only with `--allow-open` (`import_line`, `hooks/tezgah_research.py::import_line`) | 0, 1 refused, 2 misuse |
|
|
254
|
+
| `tezgah-research next --topic <t> [--flow f] [--json]` | prints every step of a product flow with its status from the topic's files under `.tezgah/analysis/<topic>/`, then the first missing or due step's skill and command; `--flow` picks `metric-dashboard` (the default) or `feature`, and `--json` prints the same steps as data (`next_steps`, `hooks/tezgah_product.py::next_steps`) | 0, 2 unknown flow or no `--topic` |
|
|
255
|
+
| `tezgah-research init <slug> --ask "<the user's words>" --method exploratory --in "<covered>" --out "<left alone>" --arena "<problem>" --budget "<budget>" [--evaluator <path> --metric "<metric>"]` | opens a rules-7 exploratory line and writes the arena into `state.json` (`arena_record`, `hooks/tezgah_research.py::arena_record`). `--ask`, `--in` and `--out` are required as on any rules-5 line. Without `--arena` or `--budget` it is misuse; `--evaluator` and `--metric` go together; the evaluator has to be a file in the repository; the four flags on any other method are misuse. Its next step says to commit the arena before the first idea | 0, 1 refused over open lines, 2 misuse |
|
|
256
|
+
| `tezgah-research idea <slug>` | reads one idea row from stdin and appends it to `ideas.jsonl` under the lock, or refuses it with one reason per problem (`append_idea`, `hooks/tezgah_research.py::append_idea`) | 0 written, 1 refused, 2 misuse |
|
|
257
|
+
| `tezgah-research stage <slug> <idea> <stage> [--evidence "<ref or JSON>"] [--reason "<why>"]` | appends one stage move to `stages.jsonl` under the lock, or refuses it. `--reason` is the kill's reason, and a `known` prior art needs it for the kill appended with it (`append_stage`, `hooks/tezgah_research.py::append_stage`) | 0 written, 1 refused, 2 misuse |
|
|
258
|
+
| `tezgah-research match <slug>` | reads one judged match from stdin and appends it to `tournament.jsonl` under the lock, or refuses it (`append_match`, `hooks/tezgah_research.py::append_match`) | 0 written, 1 refused, 2 misuse |
|
|
259
|
+
| `tezgah-research board <slug> [--json]` | the campaign as its files show it: arena, rows, torn tails, each idea's stage, score and next action, the ideas tested and the next step; writes nothing (`board_text`, `hooks/tezgah_research.py::board_text`) | 0, 2 misuse |
|
|
260
|
+
| `tezgah-research dedup <slug> [--threshold <(0, 1]>] [--json]` | clusters the `proposed` ideas by statement and prints the `stage ... duplicate` command for each member; names the backend and the threshold's source; writes nothing (`dedup_text`, `hooks/tezgah_research.py::dedup_text`). A `--threshold` outside `(0, 1]` is misuse | 0, 2 misuse |
|
|
261
|
+
| `tezgah-research yield [--json] [--all-lines]` | per operator and per generator family, how many ideas reached each stage, the kill reasons, the board-score-to-pilot correlation and the next campaign's operator weights; writes nothing (`yield_text`, `hooks/tezgah_research.py::yield_text`) | 0, 2 misuse |
|
|
243
262
|
|
|
244
263
|
Exit code 2 is always misuse, so a caller can tell it from a line that fails the
|
|
245
264
|
checks (`misuse`, `bin/tezgah-research::misuse`). `check` asks nothing at all - no
|
|
246
265
|
network, no model - so its answer is reproducible on a machine with no
|
|
247
266
|
credential; only `--orx` asks the one question that leaves the machine.
|
|
248
267
|
|
|
268
|
+
## The next step of a product flow
|
|
269
|
+
|
|
270
|
+
`tezgah-research next --topic <t>` tells the session which step of a product flow
|
|
271
|
+
comes next. Its step table is `FLOWS` (`hooks/tezgah_product.py::FLOWS`).
|
|
272
|
+
Each step names its `id`, the `skill` that runs it, what it `leaves` behind and
|
|
273
|
+
the `command` that writes it. The `product-flow` skill lists the same nine steps
|
|
274
|
+
in prose and names `next` as their status source.
|
|
275
|
+
|
|
276
|
+
Two flows exist. `metric-dashboard` is the default and has nine steps. `feature`
|
|
277
|
+
drops the metrics and instrumentation steps, unless the topic's `flow.md` already
|
|
278
|
+
holds `## metrics`. `--flow <f>` picks one.
|
|
279
|
+
|
|
280
|
+
`next_steps` (`hooks/tezgah_product.py::next_steps`) computes each status from the
|
|
281
|
+
topic's folder, `.tezgah/analysis/<topic>/`, and reads nothing else. Nothing
|
|
282
|
+
stores a status. A step is `done`, `missing` or `due`:
|
|
283
|
+
|
|
284
|
+
| Step | Done when |
|
|
285
|
+
|---|---|
|
|
286
|
+
| `forward-spec` | `.tezgah/analysis/<topic>/spec.md` exists |
|
|
287
|
+
| `re-measure` | `flow.md` holds `## re-measure`. Before that it reads `due` once the `observe after: YYYY-MM-DD` date under `## decision` has passed |
|
|
288
|
+
| every other step | `flow.md` holds its `## <step>` heading |
|
|
289
|
+
|
|
290
|
+
The `design` and `build` commands start with `tezgah-design check`. The design
|
|
291
|
+
check and the Stop UI rule judge those two steps. The flow only asks for their
|
|
292
|
+
note. A topic that is not kebab-case (at most 64 characters) reads no file. A
|
|
293
|
+
topic with no folder yet lists every step as `missing`.
|
|
294
|
+
|
|
295
|
+
The text output prints one line per step, then one `next:` line:
|
|
296
|
+
|
|
297
|
+
```
|
|
298
|
+
<n>. <id> [<status>] <skill> - <detail>
|
|
299
|
+
next: <skill> - <command>
|
|
300
|
+
```
|
|
301
|
+
|
|
302
|
+
The `next:` line names the first `missing` or `due` step's skill and command.
|
|
303
|
+
With no such step it reads `next: nothing left`. `--json` prints the same steps
|
|
304
|
+
as a list of objects with `id`, `skill`, `status`, `detail` and `command`.
|
|
305
|
+
|
|
306
|
+
`next` exits 0 when it prints the steps. It exits 2 for an unknown flow or a
|
|
307
|
+
missing `--topic`.
|
|
308
|
+
|
|
249
309
|
## The pair the order rule compares
|
|
250
310
|
|
|
251
311
|
The order rule is decided on the commit graph, so it needs both files committable
|
|
@@ -442,18 +502,288 @@ in `hooks/tezgah_research.py`, a module the frozen set forbids a prediction's
|
|
|
442
502
|
commit to touch without a human `granted_by` (`FROZEN_PATHS`), so the gate that
|
|
443
503
|
decides which candidate survives is not machinery the candidate can rewrite.
|
|
444
504
|
|
|
445
|
-
What the pair cannot catch: **a tolerance its own author sets wide
|
|
446
|
-
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
505
|
+
What the pair cannot catch alone: **a tolerance its own author sets wide
|
|
506
|
+
enough**. A line that writes "capability metric within 5.0 of the baseline" has
|
|
507
|
+
adopted the form of the two-gate rule and none of its force. From rules 6 the band
|
|
508
|
+
is read against a measurement. `evaluation.noise_band` is
|
|
509
|
+
`{"experiment": "<the experiment dir that reran the unchanged baseline>",
|
|
510
|
+
"spread": <the spread that rerun measured>}`. `check` warns in three cases. The
|
|
511
|
+
line declares `capability_tolerance` and no `noise_band`. `noise_band.experiment`
|
|
512
|
+
names no experiment of the line. The tolerance sentence holds a number wider than
|
|
513
|
+
`spread`, so the tolerance passes what noise alone would. A digit inside a name
|
|
514
|
+
such as `p95` is not that number (`_check_selection`,
|
|
515
|
+
`hooks/tezgah_research.py::_check_selection`). Whether the rerun was a fair one
|
|
516
|
+
stays the review's call.
|
|
517
|
+
|
|
518
|
+
## Held-out set, repeats and noise band (rules 6)
|
|
519
|
+
|
|
520
|
+
The measured line `rsi-borrow-screen` found three gaps. Of 27 decisions, 25 select
|
|
521
|
+
a variant and report it on the same cells. Of 23 armed experiments, 13 run once per
|
|
522
|
+
arm. Three lines declare a tolerance, and none of them measured it. Rules 6 add one
|
|
523
|
+
warning for each (`_check_selection`, `hooks/tezgah_research.py::_check_selection`).
|
|
524
|
+
Each stays a warning under `--strict` too, never a refusal (ADR 019):
|
|
525
|
+
|
|
526
|
+
- **Held-out set.** A measured line may name its held-out set in
|
|
527
|
+
`evaluation.held_out` (a non-empty string). It may log each look at that set in
|
|
528
|
+
`held_out_log.jsonl` at the line root, one row
|
|
529
|
+
`{"set", "candidate", "date", "result"}` per look. `check` warns when a
|
|
530
|
+
`decisions/<id>/` holds comparison cells and the line names no `held_out`: the
|
|
531
|
+
variant was selected and reported on the same cells. It also warns when one
|
|
532
|
+
candidate looked at one set more than once: the held-out set became development
|
|
533
|
+
data for it.
|
|
534
|
+
- **Repeats.** An experiment is armed when its `results.jsonl` rows carry `arm` or
|
|
535
|
+
`variant` with two or more values. `check` warns when no row has `repeat`,
|
|
536
|
+
`repeats` or `k` above 1 and its `protocol.md` states no `k:`/`repeats:` above 1.
|
|
537
|
+
An armless experiment is not read.
|
|
538
|
+
- **Noise band.** `evaluation.noise_band`, read in "Two gates" above.
|
|
539
|
+
|
|
540
|
+
A line opened under rules 5 or earlier is not read for any of the three.
|
|
541
|
+
|
|
542
|
+
## Rules 7: exploratory lines
|
|
543
|
+
|
|
544
|
+
A `qualitative` or `measured` line answers a question. Some asks want new ideas,
|
|
545
|
+
methods or approaches instead of an answer. Rules 7 add the method `exploratory`
|
|
546
|
+
for those asks (`hooks/tezgah_research.py::EXPLORE_RULES`,
|
|
547
|
+
`hooks/tezgah_research.py::METHODS`). The line runs one campaign. Many ideas
|
|
548
|
+
enter it, a funnel kills most of them, and a run under a frozen evaluator judges
|
|
549
|
+
the few that remain. One campaign is one line, so "One open line at a time" still
|
|
550
|
+
holds. `scope_problems` (`hooks/tezgah_research.py::scope_problems`) accepts
|
|
551
|
+
`exploratory` only on a rules-7 line. On an older line it names `qualitative` and
|
|
552
|
+
`measured` as the two methods the line can take.
|
|
553
|
+
|
|
554
|
+
### The arena lock
|
|
555
|
+
|
|
556
|
+
`init <slug> --ask "<the user's words>" --method exploratory --in "<covered>"
|
|
557
|
+
--out "<left alone>" --arena "<problem>" --budget "<budget>"` opens
|
|
558
|
+
the line. The arena is the problem in the user's words. The budget says how many
|
|
559
|
+
ideas, pilots and full runs, and how much wall time, the campaign may spend. A
|
|
560
|
+
measurable arena adds `--evaluator <path> --metric "<metric>"`. The two flags go
|
|
561
|
+
together, and the evaluator has to be a file in the repository. A missing arena
|
|
562
|
+
or budget is misuse (exit 2).
|
|
563
|
+
|
|
564
|
+
`arena_record` (`hooks/tezgah_research.py::arena_record`) builds `state.json`
|
|
565
|
+
`arena`. It holds the problem and the budget. A measurable arena also holds the
|
|
566
|
+
evaluator path, the sha256 of the evaluator's bytes and the metric. `init` also
|
|
567
|
+
writes `deliverable: {"kind": "finding"}`. The campaign delivers its staged ideas
|
|
568
|
+
as findings, never as compared variants (`hooks/tezgah_research.py::init`).
|
|
569
|
+
|
|
570
|
+
The lock has three halves:
|
|
571
|
+
|
|
572
|
+
- `arena_problems` (`hooks/tezgah_research.py::arena_problems`) names an absent
|
|
573
|
+
arena, an empty problem or budget, and an evaluator with no sha256 or no metric.
|
|
574
|
+
`check` reads it through `scope_problems`. It refuses these past `bootstrap` and
|
|
575
|
+
warns at it.
|
|
576
|
+
- `_check_arena_order` (`hooks/tezgah_research.py::_check_arena_order`) finds the
|
|
577
|
+
lock commit. That commit is the first one along HEAD whose `state.json` arena
|
|
578
|
+
names a problem and a budget. `ideas.jsonl` and every experiment's
|
|
579
|
+
`results.jsonl` have to enter the history after it. An arena that differs from
|
|
580
|
+
the locked one is a refusal too. A new arena means a new line.
|
|
581
|
+
- `evaluator_problem` (`hooks/tezgah_research.py::evaluator_problem`) hashes the
|
|
582
|
+
evaluator again. While the hash differs from the locked one, the writer and
|
|
583
|
+
`check` refuse every stage row. The candidate a campaign judges must not move
|
|
584
|
+
its judge.
|
|
585
|
+
- `append_stage` (`hooks/tezgah_research.py::append_stage`) also compares the
|
|
586
|
+
working `state.json` arena with the arena at the lock commit
|
|
587
|
+
(`hooks/tezgah_research.py::_arena_lock`). While they differ, it refuses every
|
|
588
|
+
stage row. So a rewritten `evaluator_sha256` does not reopen the campaign.
|
|
589
|
+
|
|
590
|
+
`init` prints the next step: commit the arena with `tezgah-research commit` before
|
|
591
|
+
the first idea. The arena replaces the one locked metric of a measured line.
|
|
592
|
+
`_check_evaluation` (`hooks/tezgah_research.py::_check_evaluation`) and
|
|
593
|
+
`_check_evaluation_order` (`hooks/tezgah_research.py::_check_evaluation_order`)
|
|
594
|
+
skip an exploratory line for that reason.
|
|
595
|
+
|
|
596
|
+
### Three files
|
|
597
|
+
|
|
598
|
+
The campaign lives in three append-only files at the line root. One function
|
|
599
|
+
judges each row type, and the writer and `check` both call it. No file stores a
|
|
600
|
+
status. An idea's stage is its newest `stages.jsonl` row
|
|
601
|
+
(`hooks/tezgah_research.py::campaign_stages`).
|
|
602
|
+
|
|
603
|
+
| File | One row | Writer | Judge |
|
|
604
|
+
|---|---|---|---|
|
|
605
|
+
| `ideas.jsonl` (`IDEAS_FILE`) | `id` (`I<n>`, unused so far), `statement` (two sentences at most), `mechanism`, `trace` (an in-scope item), `operator` (one of `OPERATORS`), `generator` (`{family, model}`), `kill` (the result that would kill it), `date`, and `parents` for a `mutation` (one or more) or a `crossover` (two or more) | `idea <slug>` | `idea_problems` |
|
|
606
|
+
| `stages.jsonl` (`STAGES_FILE`) | `idea`, `stage`, `evidence`, `reason` for a kill, `date` | `stage <slug> <idea> <stage>` | `stage_problems` |
|
|
607
|
+
| `tournament.jsonl` (`TOURNAMENT_FILE`) | `a`, `b`, `judge` (`{family, model}`), `order` (`ab` or `ba`), `winner` (`a`, `b` or `tie`), `rationale` | `match <slug>` | `match_problems` |
|
|
608
|
+
|
|
609
|
+
The field list is `IDEA_FIELDS` (`hooks/tezgah_research.py::IDEA_FIELDS`). The
|
|
610
|
+
moves are `OPERATORS` (`hooks/tezgah_research.py::OPERATORS`). `user` marks an
|
|
611
|
+
idea the person asked for. `PARENT_OPERATORS`
|
|
612
|
+
(`hooks/tezgah_research.py::PARENT_OPERATORS`) sets how many parents a derived
|
|
613
|
+
idea names. An idea outside the arena's scope gets one line under Open questions
|
|
614
|
+
in `findings.md`, never a row.
|
|
615
|
+
|
|
616
|
+
Every append goes through `_append_campaign`
|
|
617
|
+
(`hooks/tezgah_research.py::_append_campaign`). It takes the file lock and reads
|
|
618
|
+
the rows from the locked descriptor. It judges the new row against them, cuts any
|
|
619
|
+
torn tail (`truncate_to_committed`) and writes. A refused append leaves no file
|
|
620
|
+
behind. `check` refuses a committed row that a later version changed or dropped.
|
|
621
|
+
|
|
622
|
+
### The stage funnel
|
|
623
|
+
|
|
624
|
+
`STAGES` (`hooks/tezgah_research.py::STAGES`) gives the order: `proposed`,
|
|
625
|
+
`duplicate`, `prior-art`, `screened`, `piloted`, `promoted`, `confirmed`, and
|
|
626
|
+
`killed`. `stage_problems` (`hooks/tezgah_research.py::stage_problems`) holds
|
|
627
|
+
the moves to it:
|
|
628
|
+
|
|
629
|
+
- The idea row is the `proposed` stage. A stage row that says `proposed` is a
|
|
630
|
+
refusal.
|
|
631
|
+
- A stage moves forward in that order only, or to `killed`. A kill needs
|
|
632
|
+
`--reason`.
|
|
633
|
+
- A stage also needs its prerequisite somewhere in the idea's history
|
|
634
|
+
(`hooks/tezgah_research.py::PREREQUISITES`). `screened` needs a `prior-art`
|
|
635
|
+
row. `piloted` needs `screened`, `promoted` needs `piloted`, and `confirmed`
|
|
636
|
+
needs `promoted`.
|
|
637
|
+
- `killed` is final. A revived idea is a new idea that names the old one as its
|
|
638
|
+
parent.
|
|
639
|
+
- After `duplicate`, the only allowed row is `killed`. The same holds after a
|
|
640
|
+
`known` prior art.
|
|
641
|
+
|
|
642
|
+
Each stage asks for its own evidence. `--evidence` takes a reference or a JSON
|
|
643
|
+
object.
|
|
644
|
+
|
|
645
|
+
| Stage | Evidence | What the judge asks |
|
|
646
|
+
|---|---|---|
|
|
647
|
+
| `duplicate` | `{"of": "<id>", "similarity": <0..1>}` | `of` names another idea of the line |
|
|
648
|
+
| `prior-art` | `{"verdict": "novel\|incremental\|known", "nearest": [...], "delta": "<one sentence>"}` | `nearest` names at least `NEAREST_MIN` (3) notes. `literature/INDEX.jsonl` indexes each one, and not all of them are `agent-report`. `delta` is one sentence (`_prior_art_problems`) |
|
|
649
|
+
| `screened` | none | `tournament.jsonl` holds at least one match that names the idea |
|
|
650
|
+
| `piloted`, `promoted` | `<h>` or `experiments/<h>` | `protocol.md` names the idea id and repeats its `kill` text. `results.jsonl` holds a row. When the check asks git, `protocol.md` is in a commit. If `results.jsonl` is committed too, `protocol.md` entered the history before it (`_run_problems`) |
|
|
651
|
+
| `confirmed` | `{"experiment": "<h>", "set": "<set>"}` | the arena has an evaluator. The experiment differs from every `promoted` row's experiment. The run passes the same `_run_problems` checks. The first `held_out_log.jsonl` row for that set names this idea. Each look of this idea at the set carries a `YYYY-MM-DD` date, and none is dated before the idea's promotion: an earlier look was part of the selection. A look logged after the confirmation changes nothing (`_confirm_problems`) |
|
|
652
|
+
| `killed` | `--reason "<why>"` | a reason |
|
|
653
|
+
|
|
654
|
+
A `known` prior art kills the idea in the same write. `append_stage`
|
|
655
|
+
(`hooks/tezgah_research.py::append_stage`) needs `--reason` for it. It appends
|
|
656
|
+
the prior-art row and its `killed` row under one lock, so no file state holds a
|
|
657
|
+
known idea alive. `check` refuses a `known` prior art with no kill after it.
|
|
658
|
+
|
|
659
|
+
### The tournament is a prior
|
|
660
|
+
|
|
661
|
+
A match is one judge's verdict on a pair. `bradley_terry`
|
|
662
|
+
(`hooks/tezgah_research.py::bradley_terry`) fits a strength per idea from the
|
|
663
|
+
matches. `board` ranks the ideas by it. The score moves no stage. `screened`
|
|
664
|
+
asks only that a match exists. An LLM ranking is a prior, and a run under the
|
|
665
|
+
frozen evaluator is the judge.
|
|
666
|
+
|
|
667
|
+
`_tournament_warnings` (`hooks/tezgah_research.py::_tournament_warnings`) warns
|
|
668
|
+
about thin judging. A pair judged only by the family that generated one of its
|
|
669
|
+
ideas warns, unless both generators' families judged it. A pair judged in one
|
|
670
|
+
presentation order warns. An idea with fewer than two judges warns.
|
|
671
|
+
|
|
672
|
+
### Reports: board, dedup, yield
|
|
673
|
+
|
|
674
|
+
The three reports write nothing and exit 0. None of them is a gate.
|
|
675
|
+
|
|
676
|
+
- `board <slug>` (`hooks/tezgah_research.py::board`) shows the arena and whether
|
|
677
|
+
the evaluator still hashes as locked. It shows each file's row count, each
|
|
678
|
+
idea's rank, stage, score, record, operator, generator family and next action,
|
|
679
|
+
and the campaign's next step. An idea whose prior art is `known` gets the
|
|
680
|
+
`stage ... killed --reason` command as its next action. That resumes a campaign
|
|
681
|
+
whose kill row was torn off.
|
|
682
|
+
- `dedup <slug>` (`hooks/tezgah_research.py::dedup`) clusters the `proposed`
|
|
683
|
+
ideas by statement, at one similarity threshold. The lowest id of a cluster
|
|
684
|
+
represents it. For every other member it prints the `stage ... duplicate`
|
|
685
|
+
command to run. `similarity_matrix` (`hooks/tezgah_research.py::similarity_matrix`)
|
|
686
|
+
uses the static embedding when the feature is on and pinned. Otherwise it reads
|
|
687
|
+
BM25 scores (`hooks/tezgah_rank.py::scores`) as a similarity. Every output names
|
|
688
|
+
the backend and where its threshold came from.
|
|
689
|
+
- `yield` (`hooks/tezgah_research.py::yield_report`) counts, per operator and per
|
|
690
|
+
generator family, how many ideas reached each stage. It lists the kill reasons.
|
|
691
|
+
It gives Spearman's rho between the board score and the pilot outcome
|
|
692
|
+
(`hooks/tezgah_research.py::spearman`). It proposes the next campaign's operator
|
|
693
|
+
weights (`hooks/tezgah_research.py::operator_weights`). `EXPLORE_SHARE` (0.2)
|
|
694
|
+
of each draw stays uniform, so a move with no payoff yet still gets tried. It
|
|
695
|
+
reads the open exploratory lines, or every one with `--all-lines`.
|
|
696
|
+
|
|
697
|
+
### Resume from the files
|
|
698
|
+
|
|
699
|
+
Nothing outside the three files holds campaign state. A fresh session runs
|
|
700
|
+
`board` and goes on from its `next:` line. A writer that dies mid-append can
|
|
701
|
+
leave a fragment past the last newline. `_campaign_rows`
|
|
702
|
+
(`hooks/tezgah_research.py::_campaign_rows`) reads up to the committed boundary
|
|
703
|
+
and reports the fragment's size. `board` prints it on a `resume:` line. The next
|
|
704
|
+
append drops the fragment, so the fix is to run the append that crashed again.
|
|
705
|
+
|
|
706
|
+
### How many ideas were tested
|
|
707
|
+
|
|
708
|
+
Fifty ideas tested at the 5% level yield about 2.5 winners by chance. A reader
|
|
709
|
+
cannot discount what nobody counted. `_check_tested`
|
|
710
|
+
(`hooks/tezgah_research.py::_check_tested`) warns when `to_human/report.md` has
|
|
711
|
+
no line that names `piloted` with the number of ideas that reached it. The same
|
|
712
|
+
holds for `promoted`. A number followed by a full stop counts, as in "reached
|
|
713
|
+
piloted: 2.". `board` prints both numbers on its `tested:` line.
|
|
714
|
+
|
|
715
|
+
### The dedup threshold (two label sets)
|
|
716
|
+
|
|
717
|
+
Si et al. called two seed ideas duplicates at cosine 0.8 on MiniLM. That number
|
|
718
|
+
does not carry to another similarity. Two label sets measured it here.
|
|
719
|
+
|
|
720
|
+
- The fixture: 40 idea-statement pairs, 20 of them duplicates. The author of the
|
|
721
|
+
code labelled them. The acceptance runner records the sweeps as fixture-scoped
|
|
722
|
+
experiments: `dedup-calibration` (BM25) and `dedup-calibration-embed`
|
|
723
|
+
(embed-mrl). Copies stay under `.tezgah/scratch/calibration/`.
|
|
724
|
+
- MRPC: 400 pairs of the Microsoft Research Paraphrase Corpus test set, 200
|
|
725
|
+
paraphrases and 200 not, sampled with seed 87. Human annotators labelled them.
|
|
726
|
+
The pairs are news sentences, and the negatives share many words. At BM25 0.30
|
|
727
|
+
the sample kept every paraphrase and 95% of the negatives. The script and its
|
|
728
|
+
output are `.tezgah/scratch/calib2/run.py` and `out.txt`.
|
|
729
|
+
|
|
730
|
+
| Backend | F1 at 0.8 (fixture / MRPC) | Default | F1 at the default (fixture / MRPC) |
|
|
731
|
+
|---|---|---|---|
|
|
732
|
+
| embed-mrl | 0.62 / 0.64 | 0.65 | 0.90 / 0.705 |
|
|
733
|
+
| BM25 | 0.09 / 0.41 | 0.40 | 0.70 / 0.69 |
|
|
734
|
+
|
|
735
|
+
The default is the threshold, in steps of 0.05, whose worse F1 over the two sets
|
|
736
|
+
is highest. Neither the author's labels nor the news domain picks it alone. On
|
|
737
|
+
MRPC the best F1 beats the always-duplicate baseline (0.667) by about 0.04. The
|
|
738
|
+
defaults are a coarse filter for a session to review, not a judge.
|
|
739
|
+
|
|
740
|
+
`DEDUP_THRESHOLDS` (`hooks/tezgah_research.py::DEDUP_THRESHOLDS`) holds the two
|
|
741
|
+
values. Any other backend keeps `DEDUP_THRESHOLD`
|
|
742
|
+
(`hooks/tezgah_research.py::DEDUP_THRESHOLD`), Si et al.'s 0.8, and the output
|
|
743
|
+
calls it uncalibrated. `--threshold` overrides all of them. Its range is
|
|
744
|
+
`(0, 1]`.
|
|
745
|
+
|
|
746
|
+
### The evidence behind the method
|
|
747
|
+
|
|
748
|
+
- Si et al., arXiv 2409.04109: about 5% of 4,000 generated seed ideas were not
|
|
749
|
+
duplicates (cosine 0.8). The best LLM judge (a Claude-3.5 pairwise ranker)
|
|
750
|
+
agreed with the human ranking 53.3% of the time. Split-half reviewer panels
|
|
751
|
+
agreed 56.1%.
|
|
752
|
+
- The ideation-execution gap, arXiv 2506.20803: after execution, AI ideas lost
|
|
753
|
+
1.98 points overall and human ideas lost 0.63.
|
|
754
|
+
- Gupta and Pruthi, arXiv 2502.16487: 24% of 50 AI research documents
|
|
755
|
+
paraphrased or substantially borrowed earlier work. Plagiarism detectors missed
|
|
756
|
+
them.
|
|
757
|
+
- HindSight, arXiv 2603.15164: novelty as an LLM judges it correlates negatively
|
|
758
|
+
with realized impact (rho = -0.29).
|
|
759
|
+
- Darwin Godel Machine, arXiv 2505.22954: an archive of stepping stones beats
|
|
760
|
+
hill-climbing. The loop also removed its own detection markers and faked test
|
|
761
|
+
logs.
|
|
762
|
+
- ShinkaEvolve, arXiv 2509.19349: parent sampling by fitness and novelty,
|
|
763
|
+
novelty rejection by embedding, and UCB1 over models.
|
|
764
|
+
- POPPER, arXiv 2502.09858: falsification experiments on measurable implications,
|
|
765
|
+
with Type-I error control.
|
|
766
|
+
|
|
767
|
+
The consequences:
|
|
768
|
+
|
|
769
|
+
- An LLM ranking is a prior, never a verdict.
|
|
770
|
+
- A run under the frozen evaluator is the judge.
|
|
771
|
+
- Each idea gets its own prior-art check. No idea reaches `screened` without a
|
|
772
|
+
`prior-art` row (`hooks/tezgah_research.py::PREREQUISITES`).
|
|
773
|
+
- `dedup` and the operator weights keep the ideas diverse.
|
|
774
|
+
- The report always states how many ideas the campaign tested.
|
|
775
|
+
|
|
776
|
+
Rules 1-6, `FROZEN_PATHS` and ADR 009 stay as they were. Predictions and
|
|
777
|
+
components take nothing new. `research-off` covers this method too.
|
|
450
778
|
|
|
451
779
|
## What `check` refuses, and what it only warns about
|
|
452
780
|
|
|
453
781
|
A **refusal** (exit 1) is a guarantee the checker can decide. A **warning** is a
|
|
454
782
|
guarantee it cannot: the artifact is not wrong, it is unverifiable. `--strict`
|
|
455
|
-
turns
|
|
456
|
-
what a session mid-flight must not have, so the
|
|
783
|
+
turns the warnings the listed rules write through `_soft` into a refusal, which is
|
|
784
|
+
what a release gate wants and what a session mid-flight must not have, so the
|
|
785
|
+
default stays `warn`. A warning written outside `_soft` stays one. The section
|
|
786
|
+
below names three groups of them.
|
|
457
787
|
|
|
458
788
|
Refused:
|
|
459
789
|
|
|
@@ -532,6 +862,40 @@ Refused:
|
|
|
532
862
|
nothing, or a finding whose `quote` does not occur verbatim in its `target`
|
|
533
863
|
(`_check_review`, `hooks/tezgah_research.py::_check_review`).
|
|
534
864
|
|
|
865
|
+
On an exploratory line (rules 7), `check` also refuses each of these
|
|
866
|
+
(`_check_exploratory`, `hooks/tezgah_research.py::_check_exploratory`, called
|
|
867
|
+
from `_check_standards`):
|
|
868
|
+
|
|
869
|
+
- an arena that `arena_problems` faults, past `bootstrap`. The faults are no
|
|
870
|
+
arena object, no problem, no budget, and an evaluator with no sha256 or no
|
|
871
|
+
metric (`hooks/tezgah_research.py::arena_problems`).
|
|
872
|
+
- an arena that differs from the one the lock commit holds. The same goes for an
|
|
873
|
+
`ideas.jsonl` or `results.jsonl` that entered the history before that commit
|
|
874
|
+
(`hooks/tezgah_research.py::_check_arena_order`).
|
|
875
|
+
- an idea row that `idea_problems` faults
|
|
876
|
+
(`hooks/tezgah_research.py::idea_problems`). That means a missing field, a
|
|
877
|
+
`generator` that is not `{family, model}`, or an id that is not `I<n>` or
|
|
878
|
+
repeats an earlier one. It also means an operator outside `OPERATORS`, a parent
|
|
879
|
+
no earlier idea carries, or too few parents for a `mutation` or `crossover`. The
|
|
880
|
+
last cases are a statement of more than two sentences and a date that is not
|
|
881
|
+
`YYYY-MM-DD`. A trace to no in-scope item and a missing arena fail it too.
|
|
882
|
+
- a stage row that `stage_problems` faults
|
|
883
|
+
(`hooks/tezgah_research.py::stage_problems`). The funnel above lists the
|
|
884
|
+
moves and the evidence each stage asks for. Every stage row fails while the
|
|
885
|
+
evaluator's hash differs from the locked one
|
|
886
|
+
(`hooks/tezgah_research.py::evaluator_problem`).
|
|
887
|
+
- a stage row that skips its `PREREQUISITES` stage, and any row but `killed`
|
|
888
|
+
after `duplicate` or after a `known` prior art.
|
|
889
|
+
- a `known` prior art with no `killed` row after it.
|
|
890
|
+
- a match row that `match_problems` faults
|
|
891
|
+
(`hooks/tezgah_research.py::match_problems`). That means a side that names no
|
|
892
|
+
idea, an idea matched against itself, or a `judge` that is not
|
|
893
|
+
`{family, model}`. It also means an `order` outside `MATCH_ORDERS`, a `winner`
|
|
894
|
+
outside `MATCH_WINNERS`, or no `rationale`.
|
|
895
|
+
- a campaign file line that does not parse or is not an object
|
|
896
|
+
(`hooks/tezgah_research.py::_campaign_rows`).
|
|
897
|
+
- a committed campaign row that a later version changed or dropped.
|
|
898
|
+
|
|
535
899
|
Warned, and refused under `--strict` - each is the honest state of an artifact
|
|
536
900
|
this layer cannot decide on, not a defect:
|
|
537
901
|
|
|
@@ -629,7 +993,18 @@ this layer cannot decide on, not a defect:
|
|
|
629
993
|
judgement, so no migration can write one for the lines that concluded before
|
|
630
994
|
the rule existed.
|
|
631
995
|
|
|
632
|
-
|
|
996
|
+
On an exploratory line, two more warnings take this class. One is an arena
|
|
997
|
+
fault at `bootstrap`. The other is a campaign file git cannot order against the
|
|
998
|
+
arena lock (`_check_arena_order`).
|
|
999
|
+
|
|
1000
|
+
Three groups of warnings are **not** promoted by `--strict`. The rules-6
|
|
1001
|
+
three (held-out set, repeats, noise band) stay warnings by ADR 019. The rules-7
|
|
1002
|
+
four stay warnings for the same reason: a tournament is a prior and a count is a
|
|
1003
|
+
disclosure. They are a pair that only a generator family of one of its ideas
|
|
1004
|
+
judged, unless both generators' families judged it. They are a pair judged in one
|
|
1005
|
+
order, and an idea with fewer than two judges (`_tournament_warnings`). The fourth
|
|
1006
|
+
is a `report.md` that does not state how many ideas reached `piloted` and
|
|
1007
|
+
`promoted` (`_check_tested`). The last is a `phase`
|
|
633
1008
|
declared in `state.json` behind the phase the line's own artifacts show
|
|
634
1009
|
(`_check_derived_phase`, `hooks/tezgah_research.py::_check_derived_phase`). The field is the author's
|
|
635
1010
|
declared intent and `_check_report` and `_check_review` read it, so a derivation
|
|
@@ -827,11 +1202,15 @@ armed. It removes the rule, not just the mark - the diff is in `core_split`,
|
|
|
827
1202
|
frozen set they read (`FROZEN_PATHS`), the protocol order and its tracked pair,
|
|
828
1203
|
the protocol's two questions, the concluded report's limits, the `supersedes`
|
|
829
1204
|
relation, `component_report` with the state it reads, `migrate`, `source_run`,
|
|
830
|
-
`init` and `summary
|
|
1205
|
+
`init` and `summary`, and the rules-7 campaign: the arena lock, the three row
|
|
1206
|
+
judges and their writers, `board`, `dedup` and `yield_report`.
|
|
831
1207
|
- `hooks/tezgah_components.py` - the manifest: the one definition of an editable
|
|
832
1208
|
component, the keys a prediction row names and the rules a row's outcome is
|
|
833
1209
|
attributed to.
|
|
834
1210
|
- `bin/tezgah-research` - the CLI, its subcommands and its exit codes.
|
|
1211
|
+
- `hooks/tezgah_product.py` - the product-flow steps and the `next` status reader.
|
|
1212
|
+
- `hooks/tezgah_rank.py` - `scores`, the BM25 reading `dedup` falls back to when
|
|
1213
|
+
the static embedding is off.
|
|
835
1214
|
- `hooks/tezgah_policy.py` - the `RESEARCH` paragraph a session is armed with.
|
|
836
1215
|
- `hooks/tezgah_context.py` - the prompt hint, the missing-orx note, the broken
|
|
837
1216
|
line note and the `research` mark.
|
package/docs/skills.md
CHANGED
|
@@ -32,7 +32,7 @@ skill is simply not read.
|
|
|
32
32
|
|
|
33
33
|
opencode's plugin does inject prompt-time text - the builder's per-turn block,
|
|
34
34
|
the conditional rules included
|
|
35
|
-
(`hosts/opencode/plugins/tezgah.js:
|
|
35
|
+
(`hosts/opencode/plugins/tezgah.js:2652-2681`) - but that text names at most one
|
|
36
36
|
skill (the hint), never the roster, so the installer writes one:
|
|
37
37
|
`~/.config/tezgah/opencode-skills.md` always-on and
|
|
38
38
|
`opencode-skills.full.md` on demand (`bin/tezgah-setup::SKILL_ROUTER`, `bin/tezgah-setup::SKILL_ROUTER_FULL`), the first
|
|
@@ -128,6 +128,12 @@ rebuilds that index only when a SKILL.md's mtime or size moves
|
|
|
128
128
|
(`hooks/tezgah_skill_pick.py::_open_index`). A prompt reads its own terms'
|
|
129
129
|
postings and the length column, never the corpus.
|
|
130
130
|
|
|
131
|
+
Some skills route through their own `INDEX.md`, like design-library's 39 entries.
|
|
132
|
+
The search indexes each routed entry as one row for the whole file, addressed
|
|
133
|
+
`skill://<skill>/<path>` (`hooks/tezgah_skill_pick.py::_entries`). The vendored
|
|
134
|
+
sections are short, and per section BM25 let them outrank whole skills on two
|
|
135
|
+
common words.
|
|
136
|
+
|
|
131
137
|
The hook then appends one line, `A local skill search matched this request:
|
|
132
138
|
skill://<name>:<start>-<end>`, with the section title. Every host but omp also
|
|
133
139
|
gets the absolute path. The top section must carry 60% of the prompt's
|
|
@@ -174,15 +180,16 @@ it (`tests/test_skills.py::SkillStandards`).
|
|
|
174
180
|
| `ponytail` | Forces the laziest solution that works, in levels lite/full/ultra; fires on any coding task, or "be lazy", "yagni", "shortest path". |
|
|
175
181
|
| `i-have-adhd` | Shapes output for a reader who acts on it - action first, numbered steps, errors as location/cause/fix; fires on any answer the reader must act on, not on code, commits or docs. |
|
|
176
182
|
| `no-ai-slop` | Edits a draft sharper and more human without losing the voice, or flags slop; fires on a draft review, on a draft in a language other than English, on text that must keep terms, identifiers or unit symbols verbatim, and before publishing English prose tezgah writes itself. |
|
|
177
|
-
| `analyze-app` | Inspects and drives a running web or mobile app from its accessibility / DOM / view tree, its rendered screen (screenshots, blur and grayscale tests, breakpoints) and measurements taken in the page; fires when a task must verify behaviour or read the UI, not just read source. |
|
|
183
|
+
| `analyze-app` | Inspects and drives a running web or mobile app from its accessibility / DOM / view tree, its rendered screen (screenshots, blur and grayscale tests, breakpoints) and measurements taken in the page, and writes the `.tezgah/design/measurement.json` that `design-contract`'s check reads; fires when a task must verify behaviour or read the UI, not just read source. |
|
|
178
184
|
| `plan-add` | Creates a plan under `.tezgah/plans/open/` with its own `allowed_paths:` scope, bootstraps the `.tezgah/plans/` layer if missing, commits it in the private `.tezgah` repository (`git -C .tezgah`, never the project's git); fires on `/tezgah:plan-add`, "track this as a plan". |
|
|
179
185
|
| `plan-status` | Reads every open plan and `tezgah-task status` (each plan's Acceptance boxes ticked, the plans its `after:` waits on, the ready plans that can run at once), enriches rows with live PR state from `gh`, rewrites the README table (committed in the private `.tezgah` repository) and recommends the next plan; fires on `/tezgah:plan-status` or "what plans are open". |
|
|
180
|
-
| `plan-sync` | Closes finished plans whose PR merged or closed through `tezgah-task close`, which moves a plan to `.tezgah/plans/done/` as done only when it records an approving `review
|
|
186
|
+
| `plan-sync` | Closes finished plans whose PR merged or closed through `tezgah-task close`, which moves a plan to `.tezgah/plans/done/` as done only when it records an approving `review:`, clears an active task's phase and removes every merged worktree that holds nothing else; commits one message per plan in the private `.tezgah` repository; fires on `/tezgah:plan-sync`, "sync plans". |
|
|
181
187
|
| `research` | Runs an auditable research line - two loops, protocol before run, six-dimension review, provenance; fires when the deliverable is evidence: a lit review, a hypothesis, a benchmark or ablation. |
|
|
182
|
-
| `product-analysis` | Runs a product analysis across five axes - value (HEART), usability (heuristics, the cognitive walkthrough, WCAG 2.2, axe-core, the running app read through `analyze-app` down to component x state, tree and image both), feasibility (a cited `path:line`), competition (a dated teardown) and triage (keep/fix/cut/bet) - with one named evidence class per finding, executed as a research line; fires on a product, feature, retention or roadmap question rather than a code change. |
|
|
188
|
+
| `product-analysis` | Runs a product analysis across five axes - value (HEART), usability (heuristics, the cognitive walkthrough, WCAG 2.2, axe-core, the running app read through `analyze-app` down to component x state, tree and image both), feasibility (a cited `path:line`), competition (a dated teardown) and triage (keep/fix/cut/bet) - with one named evidence class per finding and a metric record per metric (North Star and inputs; goal, definition, window, baseline, logged or not), executed as a research line; fires on a product, feature, retention or roadmap question rather than a code change. |
|
|
189
|
+
| `product-flow` | Carries a new product capability - a metric dashboard, a metric, a feature - from research to re-measure by running `tezgah-research next --topic <t>` and doing the first missing step with the skill it names; the skill lists the nine steps in prose and `FLOWS` in `hooks/tezgah_product.py` holds their done rules; fires on "bu veriye uygun metric dashboard tasarla" or a new-feature ask, not on an audit of an existing feature. |
|
|
183
190
|
| `pm-frameworks` | The two vendored product methods the analysis skill defers to - intended-vs-implemented and the Opportunity Solution Tree - kept offline so an offline session reads the method instead of citing its name. |
|
|
184
|
-
| `feature-audit` | Audits ONE feature across every layer it is spread over - the surfaces a person uses (a control, a list or table, a detail view, a filter, a picker, a step flow, a notification), the route, the authorization, the service and the data - by filling the capability, field-contract, flow/step and interaction-dependency matrices, where a disagreement between layers is a finding and a fix needing an absent capability becomes a capability-change proposal; fires on an admin screen, CRUD, form, wizard, data table, filter or permission question about an existing feature. |
|
|
185
|
-
| `design-contract` | The artifact shape of the per-repo `.tezgah/design-contract.md` - colour roles, type scale, spacing unit and rhythm, the component inventory, the state set every interactive control and data view owes - the derivation rule that reads it out of the repository's own tokens,
|
|
191
|
+
| `feature-audit` | Audits ONE feature across every layer it is spread over - the surfaces a person uses (a control, a list or table, a detail view, a filter, a picker, a step flow, a notification), the route, the authorization, the service and the data - by filling the capability, field-contract, flow/step and interaction-dependency matrices, where a disagreement between layers is a finding and a fix needing an absent capability becomes a capability-change proposal; fires on an admin screen, CRUD, form, wizard, data table, filter or permission question about an existing feature, and in forward mode specifies a feature not built yet so the post-build audit diffs against it. |
|
|
192
|
+
| `design-contract` | The artifact shape of the per-repo `.tezgah/design-contract.md` - colour roles, type scale, spacing unit and rhythm, the component inventory, the state set every interactive control and data view owes - the derivation rule that reads it out of the repository's own tokens, the rules `bin/tezgah-design check` fails on, and the data-view rules for dashboards (`skills/design-contract/data-views.md`); fires on a UI or component turn that needs a floor, a token, a state set, or the design check itself. |
|
|
186
193
|
| `design-library` | The vendored MC Dean design/UX subset - 39 skills in the `cognitive-accessibility`, `adaptive-interfaces`, `inclusive-interaction`, `visual-critique` and `design-research` plugins - read one entry at a time through its own `INDEX.md`, never the tree; fires when a UI or product turn needs a cognitive-load, inclusive-interaction or visual-critique method tezgah has no counterpart for. |
|
|
187
194
|
| `ai-research` | The vendored 98-skill library for AI/ML machinery - training and serving a model, benchmarks, interpretability, retrieval pipelines; read one entry, never the tree. |
|
|
188
195
|
| `rl-env` | The vendored FineEnvs skill set for authoring an RL environment from a description - OpenEnv, OpenReward/ORS, Verifiers and NeMo Gym, plus the orchestrator that ports one description across all four; read one entry, never the tree. |
|
|
@@ -198,6 +205,17 @@ The generated specialists load skills first (`hooks/tezgah_agents.py::SPECIALIST
|
|
|
198
205
|
| `tezgah-security` | the host's `wstg-*`, `attack-*`, `ci-assessment` and `llm-security` skills, which tezgah does not ship |
|
|
199
206
|
| `tezgah-tester` | none |
|
|
200
207
|
|
|
208
|
+
### The product chain
|
|
209
|
+
|
|
210
|
+
A new product capability runs as one chain: research -> `product-analysis` ->
|
|
211
|
+
decision -> forward spec -> `design-contract` -> build -> re-measure. Each link
|
|
212
|
+
leaves a file under `.tezgah/analysis/<topic>/` that names the one before it.
|
|
213
|
+
`flow.md` holds one `## <step>` note per step. The forward spec
|
|
214
|
+
(`feature-audit`'s forward mode) is `spec.md`. `FLOWS`
|
|
215
|
+
(`hooks/tezgah_product.py::FLOWS`) holds the steps and their done rules.
|
|
216
|
+
`tezgah-research next` prints them with their status (docs/research.md, "The
|
|
217
|
+
next step of a product flow").
|
|
218
|
+
|
|
201
219
|
## Vendored material
|
|
202
220
|
|
|
203
221
|
`NOTICE` records every third-party skill and its terms: it opens by saying the
|
package/docs/status-line.md
CHANGED
|
@@ -96,7 +96,7 @@ measure the surface cannot see, then armed-and-used or armed-not-used.
|
|
|
96
96
|
| `graph` | a code-graph tool call (an `mcp__…codegraph…` tool) | `.no-graph` — a missing binary is `idx`'s report, not this mark's |
|
|
97
97
|
| `orch` | a delegate call (`Task`/`Agent`/`spawn_agent`) or a subagent start | `orchestrate-off` |
|
|
98
98
|
| `judge` | a shell command really ran `tezgah-triage` or `tezgah-docs` | `judge-off`, or no credential: no `TYPESAFE_API_KEY` and no non-empty `~/.config/typesafe/key`, and neither the fallback's `OPENROUTER_API_KEY` nor its `~/.config/openrouter/key` |
|
|
99
|
-
| `gate` | never `on`: shown only as `gate✗`, right after the version, in two cases. One: the prompt hook found at least 3 gated calls in this session's transcript since its first turn: write and shell calls, every one of which the gate answers with a `began` or a `deny` row (Bash/PowerShell/Edit/Write/MultiEdit/NotebookEdit on Claude, `exec_command`/`shell`/`apply_patch` on Codex). Over the same span the ledger holds no row from the gate. A PostToolUse row is no proof: that hook keeps writing while a broken PreToolUse lets every call through. The same prompt carries the line "tezgah gate inactive on this host: ...". Two: a kill switch appeared after the session's first prompt; the prompt hook writes one `disarm` row naming it (`disarmed`, baseline a `switches` row of `<cache>/tezgah.db`). The mark is a `gate_inactive` row of the same database (`gate_inactive`, audit Phase 1.3) | a row from the gate appears and no switch armed mid-session is still armed |
|
|
99
|
+
| `gate` | never `on`: shown only as `gate✗`, right after the version, in two cases. One: the prompt hook found at least 3 gated calls in this session's transcript since its first turn: write and shell calls, every one of which the gate answers with a `began` or a `deny` row (Bash/PowerShell/Edit/Write/MultiEdit/NotebookEdit on Claude, `bash`/`edit`/`write` `toolCall` blocks on omp, `exec_command`/`shell`/`apply_patch` on Codex). Over the same span the ledger holds no row from the gate. A PostToolUse row is no proof: that hook keeps writing while a broken PreToolUse lets every call through. The same prompt carries the line "tezgah gate inactive on this host: ...". Two: a kill switch appeared after the session's first prompt; the prompt hook writes one `disarm` row naming it (`disarmed`, baseline a `switches` row of `<cache>/tezgah.db`). The mark is a `gate_inactive` row of the same database (`gate_inactive`, audit Phase 1.3) | a row from the gate appears and no switch armed mid-session is still armed |
|
|
100
100
|
| `drift` | never `on`: shown as `drift✗` when a session start found tezgah's own hook entries changed since install (`hooks/tezgah_attest.py::run`). One mark per session and host, a `harness_drift` row of `<cache>/tezgah.db` keyed `<session hash>.<host>`; the session's `attest` row names what drifted | the next clean start of that host in that session |
|
|
101
101
|
| `crash` | never `on`: shown as `crash✗` when a hook of this session could not import its core and ran without tezgah (`hooks/tezgah_guard.py::import_failed`, mark an `import_crash` row of `~/.cache/tezgah/tezgah.db` keyed `<session hash>`). When `tezgah_context` itself is the module that failed, `statusline.py` prints `tezgah ✗ core import failed` instead of the line | never by itself: the mark stays for the session |
|
|
102
102
|
|
|
@@ -160,11 +160,11 @@ the command really ran it (34-38,84) and is the one writer that can earn `judge`
|
|
|
160
160
|
(a run of `bin/tezgah-triage` or `bin/tezgah-docs`, `shell_kind`, `hooks/tezgah_context.py::shell_kind`)
|
|
161
161
|
— plus `hooks/projects-auto-init.py::main`
|
|
162
162
|
(`orch` on subagent start), `hosts/codex/hook.py::main`,
|
|
163
|
-
`hosts/cursor/hook.py::dispatch`, hosts/opencode/plugins/tezgah.js:
|
|
164
|
-
hosts/opencode/plugins/tezgah.js:
|
|
163
|
+
`hosts/cursor/hook.py::dispatch`, hosts/opencode/plugins/tezgah.js:2167-2174,
|
|
164
|
+
hosts/opencode/plugins/tezgah.js:2725, `hosts/omp/hook.py::handle`. Both Claude (hooks/hooks.json) and dsh
|
|
165
165
|
(hosts/dsh/hooks.json) wire PostToolUse to that one shared hook, which is how dsh
|
|
166
166
|
gets any used mark at all: with no local transcript, the store is the only channel
|
|
167
|
-
its line has (hooks/projects-posttooluse.py:
|
|
167
|
+
its line has (hooks/projects-posttooluse.py:45-49).
|
|
168
168
|
|
|
169
169
|
Claude's line never reads the store: it parses `tool_use` blocks out of
|
|
170
170
|
`payload["transcript_path"]` plus the subagent transcripts under it, which is how
|
|
@@ -185,7 +185,7 @@ Every other surface reads the store.
|
|
|
185
185
|
`setWidget`, and it is plain because omp strips ANSI (12-29). `status_line()`
|
|
186
186
|
builds the line and carries `idx_override`, so a per-tool redraw forks no git
|
|
187
187
|
(`hosts/omp/hook.py::status_line`); it redraws on session start, session switch, turn end
|
|
188
|
-
and each watched tool result (hosts/omp/tezgah-hook.ts.in:
|
|
188
|
+
and each watched tool result (hosts/omp/tezgah-hook.ts.in:596-636,674-679,720-745).
|
|
189
189
|
**Motion:** while the agent runs (`agent_start` until an `agent_end` whose
|
|
190
190
|
`isTerminal` is not `false`) the widget leads with a braille spinner, the tool
|
|
191
191
|
in flight (`tool_execution_start`/`_end`, "thinking" between tools) and the
|
package/docs/testing.md
CHANGED
|
@@ -69,11 +69,11 @@ No matrix cancels its other legs when one fails (`fail-fast: false`).
|
|
|
69
69
|
pull request's base. A change only under `docs/`, `assets/`, a root `*.md`
|
|
70
70
|
page or the GitHub templates is prose: it runs `docs` and nothing else. An
|
|
71
71
|
empty diff, a base it cannot read, a release or a manual run runs everything.
|
|
72
|
-
- `docs`, prose only: the citations audit and
|
|
72
|
+
- `docs`, prose only: the citations audit, the clarity audit and
|
|
73
73
|
`tests/impacted.py --ref <base>`, which maps a docs page to the docs modules.
|
|
74
74
|
- `test` (`ci.yml:100`) on Python 3.10 and 3.14, the floor and the ceiling.
|
|
75
75
|
It byte-compiles, lints with ruff from `requirements-dev.txt` (pip cached),
|
|
76
|
-
audits the docs citations and runs `TEZGAH_E2E_STRICT=1 python3
|
|
76
|
+
audits the docs citations and clarity and runs `TEZGAH_E2E_STRICT=1 python3
|
|
77
77
|
tests/e2e_plan_flow.py`. On `main` it also runs the serial suite with `-v`.
|
|
78
78
|
A pull request skips that step, and so does a release, which passes the
|
|
79
79
|
`release` input and runs one 3.10 leg.
|