@r1z4x/tezgah 1.3.0 → 1.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +103 -1
- package/MANIFEST +3 -0
- package/agents/tezgah-reviewer.md +19 -10
- package/bin/tezgah-docs +3 -3
- package/bin/tezgah-doctor +75 -18
- package/bin/tezgah-route +5 -2
- package/bin/tezgah-setup +30 -18
- package/bin/tezgah-status +3 -1
- package/bin/tezgah-task +130 -37
- package/bin/tezgah-taste +74 -55
- package/bin/tezgah-triage +12 -7
- package/docs/architecture.md +4 -4
- package/docs/clarity-baseline.json +9 -9
- package/docs/contract.md +22 -12
- package/docs/environment.md +4 -1
- package/docs/evidence.md +71 -34
- package/docs/gate.md +21 -21
- package/docs/glossary.md +5 -5
- package/docs/hosts.md +8 -8
- package/docs/judge.md +93 -27
- package/docs/operations.md +69 -30
- package/docs/skills.md +2 -2
- package/docs/status-line.md +7 -7
- package/docs/vision.md +10 -5
- package/hooks/tezgah_agents.py +22 -11
- package/hooks/tezgah_attest.py +15 -22
- package/hooks/tezgah_context.py +84 -124
- package/hooks/tezgah_gate.py +26 -23
- package/hooks/tezgah_guard.py +9 -11
- package/hooks/tezgah_integrity.py +77 -178
- package/hooks/tezgah_judge.py +270 -122
- package/hooks/tezgah_lessons.py +18 -63
- package/hooks/tezgah_paths.py +183 -55
- package/hooks/tezgah_policy.py +35 -8
- package/hooks/tezgah_replay.py +57 -33
- package/hooks/tezgah_shapes.py +4 -4
- package/hooks/tezgah_skill_pick.py +6 -35
- package/hooks/tezgah_snapshot.py +25 -46
- package/hooks/tezgah_store.py +1268 -0
- package/hooks/tezgah_task.py +146 -40
- package/hooks/tezgah_taste.py +21 -18
- package/hooks/tezgah_taste_ledger.py +63 -88
- package/hooks/tezgah_update.py +6 -26
- package/hosts/codex/hook.py +5 -0
- package/hosts/cursor/hook.py +19 -31
- package/hosts/opencode/plugins/tezgah.js +309 -108
- package/package.json +1 -1
- package/skills/harness/SKILL.md +4 -0
- package/skills/plan-add/SKILL.md +23 -7
- package/skills/plan-add/render_table.py +3 -2
- package/skills/plan-status/SKILL.md +12 -4
- package/skills/plan-sync/SKILL.md +19 -9
- package/skills/tezgah-contract/SKILL.md +25 -12
package/bin/tezgah-task
CHANGED
|
@@ -5,7 +5,7 @@
|
|
|
5
5
|
tezgah-task phase P [--any-path] change the active task's phase
|
|
6
6
|
tezgah-task allow GLOB ... | --any-path replace the allowlist
|
|
7
7
|
tezgah-task stop deactivate (clear phase)
|
|
8
|
-
tezgah-task status [--json] the active task,
|
|
8
|
+
tezgah-task status [--json] the active task, then every open plan: progress, what it waits on, what can run at once
|
|
9
9
|
tezgah-task review <NNN|path> <reviewer> approve|changes record a fresh review's verdict
|
|
10
10
|
tezgah-task close <NNN|path> done|discarded move a plan to .tezgah/plans/done
|
|
11
11
|
|
|
@@ -22,21 +22,27 @@ creates the plan, and `start` without `--allow` keeps it. A writing phase with
|
|
|
22
22
|
no allowlist is refused unless `--any-path` says so explicitly, and the record
|
|
23
23
|
keeps where its scope came from (`allowed_from:`), which `status` prints.
|
|
24
24
|
|
|
25
|
-
`review` and `close` are the session's: the verification phase asks for
|
|
26
|
-
|
|
25
|
+
`review` and `close` are the session's: the verification phase asks for one
|
|
26
|
+
review of the plan's whole diff, `review` records its verdict as
|
|
27
27
|
`review: <reviewer> <verdict> <date>` together with `reviewed_sha:` (the
|
|
28
28
|
project's HEAD it read) and `review_round:`, so the next round reads only
|
|
29
|
-
`git diff <reviewed_sha>..HEAD
|
|
30
|
-
NNN done` refuses a plan whose review is missing or not
|
|
31
|
-
refuses the active task - that one is the user's to stop first.
|
|
29
|
+
`git diff <reviewed_sha>..HEAD`, and refuses a third round: the budget is two
|
|
30
|
+
rounds per plan; `close NNN done` refuses a plan whose review is missing or not
|
|
31
|
+
`approve`. `close` refuses the active task - that one is the user's to stop first.
|
|
32
32
|
|
|
33
33
|
Two things the plan owes are asked here, at the only moment they can still be
|
|
34
34
|
fixed - the phase move - because the record is the user's and the refusal is
|
|
35
|
-
this CLI's: a writing phase needs
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
`implementation` needs the plan's spike
|
|
39
|
-
evidence rather than on the guess the spike
|
|
35
|
+
this CLI's: a writing phase needs every Acceptance item to name the command
|
|
36
|
+
proving it (or say `unverifiable` and why), since the report CI runs afterwards
|
|
37
|
+
refuses an open plan on any one item that does neither, through the same
|
|
38
|
+
predicate (`tezgah_task.unproven`); and `implementation` needs the plan's spike
|
|
39
|
+
answered, so the build starts on evidence rather than on the guess the spike
|
|
40
|
+
exists to retire.
|
|
41
|
+
|
|
42
|
+
`status` lists every open plan after the active task: its Acceptance boxes
|
|
43
|
+
ticked, the plans its `after:` still waits on, and - on a `parallel:` line - the
|
|
44
|
+
ready plans whose `allowed_paths:` share no prefix, which can run at once. A
|
|
45
|
+
status outside open|blocked, or an `after:` id no plan carries, is a `FAIL` line.
|
|
40
46
|
|
|
41
47
|
Entering `implementation` also records where the risky work returns to: a
|
|
42
48
|
`checkpoint:` field holding the pre-work commit's sha, or `pending <sha>` while
|
|
@@ -139,6 +145,14 @@ def write_fields(path, **fields):
|
|
|
139
145
|
|
|
140
146
|
ANY_PATH = "--any-path"
|
|
141
147
|
VERDICTS = ("approve", "changes")
|
|
148
|
+
# The review budget, enforced here and nowhere else: `review` refuses a third
|
|
149
|
+
# round, and every text that states the rule (the reviewer agent, the contract,
|
|
150
|
+
# plan-sync) says the same two rounds.
|
|
151
|
+
REVIEW_ROUNDS = 2
|
|
152
|
+
# Under this many changed lines one reviewer reads every dimension; at or above
|
|
153
|
+
# it the round fans out. ponytail: a line count, not a risk model - a small diff
|
|
154
|
+
# that touches a trust boundary is the brief's to route to security.
|
|
155
|
+
SMALL_DIFF = 300
|
|
142
156
|
|
|
143
157
|
|
|
144
158
|
def allow_source(text):
|
|
@@ -279,13 +293,14 @@ def phase_problems(phase, text, path):
|
|
|
279
293
|
items = tt.acceptance_gap(text)
|
|
280
294
|
if items:
|
|
281
295
|
out.append(
|
|
282
|
-
"%s cannot enter %s:
|
|
283
|
-
"command and
|
|
284
|
-
"hooks/tezgah_task.py:acceptance_items), so
|
|
285
|
-
"to be held to. Give
|
|
296
|
+
"%s cannot enter %s: %d of its %d Acceptance items name no "
|
|
297
|
+
"command and declare no `unverifiable` (read by "
|
|
298
|
+
"hooks/tezgah_task.py:acceptance_items; at %s), so that work has "
|
|
299
|
+
"no check to be held to. Give each one the command that proves "
|
|
286
300
|
"it, or `unverifiable` and why; "
|
|
287
|
-
"`tezgah-render-table --acceptance`
|
|
288
|
-
% (path, phase, len(items)))
|
|
301
|
+
"`tezgah-render-table --acceptance --strict` refuses the same items"
|
|
302
|
+
% (path, phase, len(items), len(tt.acceptance_items(text)),
|
|
303
|
+
", ".join("%s:%d" % (path, i["line"]) for i in items)))
|
|
289
304
|
if phase == "implementation" and tt.spike_unanswered(text):
|
|
290
305
|
out.append(
|
|
291
306
|
"%s cannot enter implementation: its spike is unanswered - `spike:` "
|
|
@@ -295,12 +310,23 @@ def phase_problems(phase, text, path):
|
|
|
295
310
|
return out
|
|
296
311
|
|
|
297
312
|
|
|
313
|
+
def changed_lines(root, base):
|
|
314
|
+
"""Lines `git diff <base>..HEAD` adds and removes, or None when git cannot
|
|
315
|
+
say (no base, not a repository)."""
|
|
316
|
+
stat = git_out(root, "diff", "--shortstat", base + "..HEAD") \
|
|
317
|
+
if base != "<base>" else None
|
|
318
|
+
if stat is None:
|
|
319
|
+
return None
|
|
320
|
+
return sum(int(n) for n in re.findall(r"(\d+) (?:insertion|deletion)", stat))
|
|
321
|
+
|
|
322
|
+
|
|
298
323
|
def verification_steps(text, path):
|
|
299
324
|
"""What the verification phase requires before the plan is done: the plan's
|
|
300
|
-
own acceptance commands run once on the final tree, unpiped, and
|
|
301
|
-
|
|
302
|
-
without
|
|
303
|
-
|
|
325
|
+
own acceptance commands run once on the final tree, unpiped, and ONE review
|
|
326
|
+
of the plan's whole diff, recorded with `review` - `close NNN done` refuses
|
|
327
|
+
without an approve. Round one fans out one reviewer per dimension, round two
|
|
328
|
+
reads only the delta since the sha round one recorded, and REVIEW_ROUNDS is
|
|
329
|
+
the budget `review` enforces."""
|
|
304
330
|
fields = tt.frontmatter(text)
|
|
305
331
|
ident = fields.get("id") or os.path.basename(path).split("-")[0]
|
|
306
332
|
# every item, not only the ones the command reader recognises: `pnpm test`
|
|
@@ -311,27 +337,51 @@ def verification_steps(text, path):
|
|
|
311
337
|
for i in tt.acceptance_items(text)]
|
|
312
338
|
# the project, not the private repository `.tezgah` may hold
|
|
313
339
|
root = os.path.realpath(os.path.join(os.path.dirname(path), "..", "..", ".."))
|
|
314
|
-
reviewed = fields.get("reviewed_sha") or ""
|
|
340
|
+
reviewed = fields.get("reviewed_sha") or "(no commit)"
|
|
315
341
|
prior = fields.get("review_round") or ""
|
|
316
342
|
rounds = int(prior) if prior.isdigit() else 0
|
|
317
343
|
print("verification - required before this plan is done:")
|
|
318
344
|
print(" 1. prove the acceptance items ONCE, on the final tree - not again per "
|
|
319
345
|
"review round - every command run unpiped "
|
|
320
|
-
"(`cmd > /tmp/check.log 2>&1`, then read the file)
|
|
346
|
+
"(`cmd > /tmp/check.log 2>&1`, then read the file); items that share "
|
|
347
|
+
"no state go out as tool calls in ONE message, one suite run at a time:")
|
|
321
348
|
for item in items or ["(the plan has no Acceptance items)"]:
|
|
322
349
|
print(" - %s" % item)
|
|
323
350
|
if os.path.isfile(os.path.join(root, "tests", "impacted.py")):
|
|
324
351
|
print(" the one full run here is `python3 tests/impacted.py --all`")
|
|
325
|
-
if
|
|
326
|
-
print(" 2. review round %d
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
|
|
352
|
+
if rounds and review_verdict(text) == "approve":
|
|
353
|
+
print(" 2. the review approved in round %d at %s: no further round"
|
|
354
|
+
% (rounds, reviewed))
|
|
355
|
+
print(" 3. `tezgah-task close %s done` once the work lands" % ident)
|
|
356
|
+
return
|
|
357
|
+
if rounds >= REVIEW_ROUNDS:
|
|
358
|
+
print(" 2. the %d-round budget is spent: `tezgah-task review` refuses a "
|
|
359
|
+
"third round. A critical or major finding still confirmed goes to "
|
|
360
|
+
"the user (block); a minor or suggestion one is recorded fix-later "
|
|
361
|
+
"under the plan's ## State" % REVIEW_ROUNDS)
|
|
362
|
+
return
|
|
363
|
+
if rounds:
|
|
364
|
+
print(" 2. review round 2 (the last): each confirmed critical or major "
|
|
365
|
+
"finding is fixed with a regression test that failed first; one "
|
|
366
|
+
"fresh tezgah-reviewer reads only the delta `git diff %s..HEAD`, the "
|
|
367
|
+
"round-1 hunks and direct callers of the symbols it touches, and the "
|
|
368
|
+
"test output above, and marks every round-1 finding closed or open; "
|
|
369
|
+
"it does not re-run the suite" % reviewed)
|
|
332
370
|
else:
|
|
333
|
-
|
|
334
|
-
|
|
371
|
+
base = diff_base(root)
|
|
372
|
+
size = changed_lines(root, base)
|
|
373
|
+
print(" 2. one review of the whole plan, at most %d rounds: spawn fresh "
|
|
374
|
+
"tezgah-reviewer subagents in ONE message, each owning one dimension "
|
|
375
|
+
"- correctness and contract, security, tests and performance - or one "
|
|
376
|
+
"file group of a large diff, over `git diff %s..HEAD` and the test "
|
|
377
|
+
"output above, then merge their findings into one verdict. Under %d "
|
|
378
|
+
"changed lines one reviewer reads every dimension (this diff: %s), "
|
|
379
|
+
"unless the diff touches auth, secrets or a gate, which keeps a "
|
|
380
|
+
"security reviewer" % (REVIEW_ROUNDS, base, SMALL_DIFF,
|
|
381
|
+
"unknown" if size is None else size))
|
|
382
|
+
print(" only a confirmed critical or major finding is `changes` and "
|
|
383
|
+
"opens round 2; a minor or suggestion one is recorded fix-later "
|
|
384
|
+
"under ## State and the verdict is `approve`")
|
|
335
385
|
print(" 3. record its verdict: `tezgah-task review %s <reviewer> approve|changes`"
|
|
336
386
|
" (records the reviewed sha);"
|
|
337
387
|
" `tezgah-task close %s done` refuses without an approve" % (ident, ident))
|
|
@@ -495,6 +545,16 @@ def review_verdict(text):
|
|
|
495
545
|
return parts[1] if len(parts) >= 2 and parts[1] in VERDICTS else None
|
|
496
546
|
|
|
497
547
|
|
|
548
|
+
def budget_spent(path, ident):
|
|
549
|
+
"""The refusal once REVIEW_ROUNDS rounds are recorded: what is left belongs
|
|
550
|
+
to the user, never to a third round."""
|
|
551
|
+
return ("%s has spent its %d-round review budget: never a third round. A "
|
|
552
|
+
"critical or major finding still confirmed goes to the user, who "
|
|
553
|
+
"decides (block); a minor or suggestion one is recorded fix-later "
|
|
554
|
+
"under the plan's ## State; `tezgah-task close %s discarded` needs no "
|
|
555
|
+
"review" % (path, REVIEW_ROUNDS, ident))
|
|
556
|
+
|
|
557
|
+
|
|
498
558
|
def cmd_review(args):
|
|
499
559
|
if len(args) != 3 or args[2] not in VERDICTS:
|
|
500
560
|
return misuse("review takes <NNN|path> <reviewer> %s" % "|".join(VERDICTS))
|
|
@@ -503,6 +563,8 @@ def cmd_review(args):
|
|
|
503
563
|
return problems([problem])
|
|
504
564
|
fields = tt.frontmatter(read(path))
|
|
505
565
|
prior = fields.get("review_round") or ""
|
|
566
|
+
if prior.isdigit() and int(prior) >= REVIEW_ROUNDS:
|
|
567
|
+
return problems([budget_spent(path, os.path.basename(path).split("-")[0])])
|
|
506
568
|
# the project's HEAD, never the private `.tezgah` repository's: the next
|
|
507
569
|
# round's delta is `git diff <this>..HEAD` over the project
|
|
508
570
|
root = os.path.realpath(os.path.join(os.path.dirname(path), "..", "..", ".."))
|
|
@@ -535,6 +597,9 @@ def cmd_close(args):
|
|
|
535
597
|
return problems(["%s is the active task: the user stops it first "
|
|
536
598
|
"(`tezgah-task stop`)" % path])
|
|
537
599
|
if args[1] == "done" and review_verdict(text) != "approve":
|
|
600
|
+
prior = tt.frontmatter(text).get("review_round") or ""
|
|
601
|
+
if prior.isdigit() and int(prior) >= REVIEW_ROUNDS:
|
|
602
|
+
return problems([budget_spent(path, ident)])
|
|
538
603
|
return problems(["%s records no approving review: a fresh reviewer reads "
|
|
539
604
|
"the diff, then `tezgah-task review %s <reviewer> approve`"
|
|
540
605
|
% (path, ident)])
|
|
@@ -561,22 +626,50 @@ def cmd_status(args):
|
|
|
561
626
|
bad = []
|
|
562
627
|
for path in plans(root):
|
|
563
628
|
fields = tt.frontmatter(read(path))
|
|
629
|
+
ident = fields.get("id") or os.path.basename(path).split("-")[0]
|
|
564
630
|
phase = fields.get("phase") or ""
|
|
565
631
|
if phase and phase not in tt.TASK_PHASES:
|
|
566
|
-
bad.append({"id":
|
|
567
|
-
"
|
|
632
|
+
bad.append({"id": ident, "phase": phase, "path": path,
|
|
633
|
+
"problem": "phase %r is not one of %s, so the plan is ignored (%s)"
|
|
634
|
+
% (phase, ", ".join(tt.TASK_PHASES), path)})
|
|
635
|
+
status = fields.get("status") or ""
|
|
636
|
+
if status not in tt.OPEN_STATUSES:
|
|
637
|
+
bad.append({"id": ident, "path": path,
|
|
638
|
+
"problem": "status %r is not one of %s, which is what a plan "
|
|
639
|
+
"under .tezgah/plans/open carries (%s)"
|
|
640
|
+
% (status, ", ".join(tt.OPEN_STATUSES), path)})
|
|
641
|
+
rows = tt.queue(root)
|
|
642
|
+
for row in rows:
|
|
643
|
+
for ident, state in row["after"]:
|
|
644
|
+
if state == "missing":
|
|
645
|
+
bad.append({"id": row["id"], "path": row["path"],
|
|
646
|
+
"problem": "`after: %s` names no plan under .tezgah/plans "
|
|
647
|
+
"(%s)" % (ident, row["path"])})
|
|
648
|
+
together = tt.parallel(rows)
|
|
568
649
|
if task:
|
|
569
650
|
task["allowed_from"] = allow_source(read(task["path"]))[1]
|
|
570
651
|
if as_json:
|
|
571
|
-
print(json.dumps({"active": task, "malformed": bad
|
|
652
|
+
print(json.dumps({"active": task, "malformed": bad, "open": rows,
|
|
653
|
+
"parallel": together}, indent=2))
|
|
572
654
|
else:
|
|
573
655
|
if task:
|
|
574
656
|
show(task["path"])
|
|
575
657
|
else:
|
|
576
658
|
print("no active task")
|
|
659
|
+
if rows:
|
|
660
|
+
print("open plans: %d, %d ready" % (len(rows), sum(r["ready"] for r in rows)))
|
|
661
|
+
for row in rows:
|
|
662
|
+
waits = ["%s (%s)" % (ident, state) for ident, state in row["after"]
|
|
663
|
+
if state != "done"]
|
|
664
|
+
print(" %s %s, acceptance %d/%d checked, %s" % (
|
|
665
|
+
row["id"], row["status"] or "(no status)", row["checked"], row["items"],
|
|
666
|
+
"ready" if row["ready"] else
|
|
667
|
+
"waits on %s" % ", ".join(waits) if waits else row["status"]))
|
|
668
|
+
if together:
|
|
669
|
+
print("parallel: %s - ready, and no two share an allowed_paths prefix"
|
|
670
|
+
% ", ".join(together))
|
|
577
671
|
for row in bad:
|
|
578
|
-
print("FAIL %s:
|
|
579
|
-
% (row["id"], row["phase"], ", ".join(tt.TASK_PHASES), row["path"]))
|
|
672
|
+
print("FAIL %s: %s" % (row["id"], row["problem"]))
|
|
580
673
|
return 1 if bad else 0
|
|
581
674
|
|
|
582
675
|
|
package/bin/tezgah-taste
CHANGED
|
@@ -34,15 +34,17 @@ judge's provider (`hooks/tezgah_judge.py` names it); with no credential or
|
|
|
34
34
|
`judge-off` set they send nothing and exit 2.
|
|
35
35
|
|
|
36
36
|
`learn` is taste-1's pipeline on tezgah (research line taste-architecture,
|
|
37
|
-
design T1P). Each prompt after a writing turn - from the capture rows in
|
|
38
|
-
`<repo>/.tezgah/taste/
|
|
39
|
-
repository's omp and Claude transcripts - becomes one structured signal
|
|
40
|
-
code: the prompt, the files written, the agent's edits and the user's
|
|
41
|
-
between two agent writes. One typed decision per signal, required from
|
|
42
|
-
(`only=("
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
37
|
+
design T1P). Each prompt after a writing turn - from the capture rows in the
|
|
38
|
+
`signals` table of `<repo>/.tezgah/taste/taste.db`, or with `--from-transcripts`
|
|
39
|
+
from this repository's omp and Claude transcripts - becomes one structured signal
|
|
40
|
+
built by code: the prompt, the files written, the agent's edits and the user's
|
|
41
|
+
own edit between two agent writes. One typed decision per signal, required from
|
|
42
|
+
the Jev model (`only=("jev",)`: TypeSafe, OpenRouter System One, Cloudflare
|
|
43
|
+
Workers AI or a compatible endpoint - whichever carrier the judge seam resolves):
|
|
44
|
+
preference, defect or none; the category; the scope; and how it relates to each
|
|
45
|
+
learning held. A decision any other provider gave is recorded `unverified` and
|
|
46
|
+
never applied. A defect goes to the `defects`
|
|
47
|
+
table, never into the ledger. The ledger itself is `hooks/tezgah_taste_ledger.py`. A
|
|
46
48
|
learning that becomes active gets one readable line from a generative provider
|
|
47
49
|
(a `text` question; the user's own words when none answers).
|
|
48
50
|
|
|
@@ -70,6 +72,7 @@ sys.path.insert(0, os.path.join(HERE, "hooks"))
|
|
|
70
72
|
import tezgah_integrity as ti # noqa: E402
|
|
71
73
|
import tezgah_judge # noqa: E402
|
|
72
74
|
import tezgah_models # noqa: E402
|
|
75
|
+
import tezgah_paths # noqa: E402
|
|
73
76
|
import tezgah_taste_ledger as tl # noqa: E402
|
|
74
77
|
|
|
75
78
|
WRITE_TOOLS = {"omp": {"edit", "write"},
|
|
@@ -319,7 +322,7 @@ def classify(texts, model, only=None):
|
|
|
319
322
|
|
|
320
323
|
The rule rides the state, so each request pays for it once and the
|
|
321
324
|
questions carry only their own prompt."""
|
|
322
|
-
labels, tokens_in, tokens_out, used = [], 0, 0, model
|
|
325
|
+
labels, tokens_in, tokens_out, used, spent = [], 0, 0, model, 0.0
|
|
323
326
|
for start in range(0, len(texts), BATCH):
|
|
324
327
|
chunk = texts[start:start + BATCH]
|
|
325
328
|
questions = {"p%d" % i: {"type": "choice", "criteria": CRITERIA,
|
|
@@ -331,24 +334,36 @@ def classify(texts, model, only=None):
|
|
|
331
334
|
tokens_in += result["usage"]["input_tokens"]
|
|
332
335
|
tokens_out += result["usage"]["output_tokens"]
|
|
333
336
|
used = result.get("model", used)
|
|
337
|
+
one = result_cost(result)
|
|
338
|
+
spent = None if spent is None or one is None else spent + one
|
|
334
339
|
for i in range(len(chunk)):
|
|
335
340
|
label = tezgah_judge.choice(result, "p%d" % i)
|
|
336
341
|
labels.append(label if label in CLASSES else None)
|
|
337
342
|
return labels, {"model": used, "input_tokens": tokens_in,
|
|
338
|
-
"output_tokens": tokens_out,
|
|
339
|
-
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
def cost(model, tokens_in, tokens_out):
|
|
343
|
-
"""Dollars at the model's listed price per 1M tokens, None when unlisted.
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
343
|
+
"output_tokens": tokens_out,
|
|
344
|
+
"cost_usd": None if spent is None else round(spent, 6)}
|
|
345
|
+
|
|
346
|
+
|
|
347
|
+
def cost(model, tokens_in, tokens_out, provider=None):
|
|
348
|
+
"""Dollars at the model's listed price per 1M tokens, None when unlisted. A
|
|
349
|
+
typed carrier (`tezgah_judge.input_price`) is priced by the carrier that
|
|
350
|
+
answered: Clef's price is not Jev's."""
|
|
351
|
+
typed = tezgah_judge.input_price(provider)
|
|
352
|
+
price = (typed, 0.0) if typed is not None else JEV_PRICE \
|
|
353
|
+
if model == tezgah_judge.MODEL else (
|
|
354
|
+
tezgah_models.overlay().get("prices", {}).get(model)
|
|
355
|
+
or tezgah_models.SNAPSHOT.get(model))
|
|
347
356
|
if not (isinstance(price, (list, tuple)) and len(price) == 2):
|
|
348
357
|
return None
|
|
349
358
|
return round((tokens_in * price[0] + tokens_out * price[1]) / 1e6, 6)
|
|
350
359
|
|
|
351
360
|
|
|
361
|
+
def result_cost(result):
|
|
362
|
+
"""One judge result's cost, priced by the provider that answered it."""
|
|
363
|
+
return cost(result.get("model"), result["usage"]["input_tokens"],
|
|
364
|
+
result["usage"]["output_tokens"], result.get("provider"))
|
|
365
|
+
|
|
366
|
+
|
|
352
367
|
def usage_line(usage):
|
|
353
368
|
money = ("$%.6f" % usage["cost_usd"] if usage["cost_usd"] is not None
|
|
354
369
|
else "cost unknown (no listed price)")
|
|
@@ -369,7 +384,7 @@ def cmd_measure(args):
|
|
|
369
384
|
if not tezgah_judge.available():
|
|
370
385
|
return misuse("no judge: no credential resolves or judge-off is set; nothing sent")
|
|
371
386
|
got, usage = classify([text for text, _gold in pairs], args.model,
|
|
372
|
-
only=("
|
|
387
|
+
only=("jev",) if args.typed else None)
|
|
373
388
|
confusion = {g: {p: 0 for p in CLASSES} for g in CLASSES}
|
|
374
389
|
unjudged = 0
|
|
375
390
|
for (_text, gold), pred in zip(pairs, got):
|
|
@@ -466,18 +481,13 @@ def iso_day(ts):
|
|
|
466
481
|
|
|
467
482
|
|
|
468
483
|
def _snapshots():
|
|
469
|
-
"""{absolute path: [(ts, id, after)]} over the snapshot
|
|
470
|
-
import
|
|
484
|
+
"""{absolute path: [(ts, id, after)]} over the snapshot index, oldest first."""
|
|
485
|
+
import tezgah_store
|
|
471
486
|
out = {}
|
|
472
|
-
for
|
|
473
|
-
|
|
474
|
-
with open(os.path.join(d, "meta.json"), encoding="utf-8") as fh:
|
|
475
|
-
meta = json.load(fh)
|
|
476
|
-
except (OSError, ValueError):
|
|
477
|
-
continue
|
|
478
|
-
if isinstance(meta, dict) and meta.get("path"):
|
|
487
|
+
for meta in tezgah_store.snapshots():
|
|
488
|
+
if isinstance(meta, dict) and meta.get("path") and meta.get("id"):
|
|
479
489
|
out.setdefault(meta["path"], []).append(
|
|
480
|
-
(int(meta.get("ts") or 0),
|
|
490
|
+
(int(meta.get("ts") or 0), meta["id"], bool(meta.get("after"))))
|
|
481
491
|
for rows in out.values():
|
|
482
492
|
rows.sort()
|
|
483
493
|
return out
|
|
@@ -522,7 +532,7 @@ def user_diff(repo, rel, after_sid, snaps):
|
|
|
522
532
|
def capture_signals(repo):
|
|
523
533
|
"""One signal per prompt that followed a writing turn, from the capture rows."""
|
|
524
534
|
snaps, turns, out = None, {}, []
|
|
525
|
-
for row in tl.rows(
|
|
535
|
+
for row in tl.rows(repo, "signals"):
|
|
526
536
|
session = row.get("session") or ""
|
|
527
537
|
turn = turns.setdefault(session, {"edits": [], "after": []})
|
|
528
538
|
if row.get("kind") == "edit":
|
|
@@ -632,12 +642,12 @@ def picked(result, qid, default):
|
|
|
632
642
|
|
|
633
643
|
|
|
634
644
|
def decide(sig, learnings, fallback=True):
|
|
635
|
-
"""(decision, result): the typed decision,
|
|
645
|
+
"""(decision, result): the typed decision, the Jev model required; when it cannot
|
|
636
646
|
answer, the seam's own providers, recorded unverified - only on a signal's
|
|
637
647
|
first try, since an unverified answer is never applied and a rerun would pay
|
|
638
648
|
for it again."""
|
|
639
649
|
state, questions = decision_request(sig, learnings)
|
|
640
|
-
result = tezgah_judge.ask(state, questions, only=("
|
|
650
|
+
result = tezgah_judge.ask(state, questions, only=("jev",), timeout=60)
|
|
641
651
|
if result is None and fallback:
|
|
642
652
|
result = tezgah_judge.ask(state, questions, timeout=120)
|
|
643
653
|
if result is None:
|
|
@@ -647,7 +657,7 @@ def decide(sig, learnings, fallback=True):
|
|
|
647
657
|
"scope": picked(result, "scope", "repository"),
|
|
648
658
|
"relations": {lr["id"]: picked(result, "rel_" + lr["id"], "unrelated")
|
|
649
659
|
for lr in learnings},
|
|
650
|
-
"verified": result.get("provider")
|
|
660
|
+
"verified": tezgah_judge.is_jev(result.get("provider"))}
|
|
651
661
|
return dec, result
|
|
652
662
|
|
|
653
663
|
|
|
@@ -666,9 +676,15 @@ def write_line(learning):
|
|
|
666
676
|
|
|
667
677
|
|
|
668
678
|
def store_or_exit(args):
|
|
679
|
+
"""The repository whose taste store a command opens. Refused when its
|
|
680
|
+
`.tezgah` came with the clone: those files are the project's data, and
|
|
681
|
+
opening the store would import, rename or create files in them."""
|
|
669
682
|
repo = repo_root(os.path.abspath(args.repo or os.getcwd()))
|
|
670
683
|
if not repo:
|
|
671
684
|
raise SystemExit(misuse("not inside a git repository"))
|
|
685
|
+
if tezgah_paths.workspace_from_repo(repo):
|
|
686
|
+
raise SystemExit(misuse("%s/.tezgah came with the repository; its taste "
|
|
687
|
+
"store is not opened" % repo))
|
|
672
688
|
return repo
|
|
673
689
|
|
|
674
690
|
|
|
@@ -676,7 +692,7 @@ def learn_refusal(repo):
|
|
|
676
692
|
"""Why `learn` sends and writes nothing here, or None - read before the
|
|
677
693
|
store lock, so a refused run leaves no lock file behind."""
|
|
678
694
|
if tezgah_judge.tp.off("judge-off") or not (
|
|
679
|
-
tezgah_judge.available() or tezgah_judge.named(("
|
|
695
|
+
tezgah_judge.available() or tezgah_judge.named(("jev",))):
|
|
680
696
|
return "no judge: no credential resolves or judge-off is set; nothing sent"
|
|
681
697
|
import tezgah_context
|
|
682
698
|
if ".no-taste" in tezgah_context.repo_marks(repo)[1]:
|
|
@@ -689,17 +705,17 @@ def cmd_learn(args):
|
|
|
689
705
|
signals = (transcript_signals(repo, args.host) if args.from_transcripts
|
|
690
706
|
else capture_signals(repo))
|
|
691
707
|
decided = {}
|
|
692
|
-
for r in tl.rows(
|
|
693
|
-
decided[r.get("id")] = decided.get(r.get("id")) or r.get("provider")
|
|
694
|
-
# a typed decision is final; an unverified one is retried,
|
|
708
|
+
for r in tl.rows(repo, "decisions"):
|
|
709
|
+
decided[r.get("id")] = decided.get(r.get("id")) or tezgah_judge.is_jev(r.get("provider"))
|
|
710
|
+
# a typed decision is final; an unverified one is retried, the Jev model only
|
|
695
711
|
todo = [s for s in signals if not decided.get(s["id"])]
|
|
696
712
|
if args.limit is not None:
|
|
697
713
|
todo = todo[:args.limit]
|
|
698
714
|
ledgers = tl.load(repo)
|
|
699
715
|
before = {k for k, v in tl.every(ledgers).items() if v["state"] == "active"}
|
|
700
|
-
counts = {"signals": len(todo), "
|
|
716
|
+
counts = {"signals": len(todo), "jev": 0, "unverified": 0, "unanswered": 0,
|
|
701
717
|
"preference": 0, "defect": 0, "none": 0}
|
|
702
|
-
tokens = [0, 0]
|
|
718
|
+
tokens, spent = [0, 0], 0.0
|
|
703
719
|
for sig in todo:
|
|
704
720
|
dec, result = decide(sig, held(ledgers),
|
|
705
721
|
fallback=not args.no_fallback and sig["id"] not in decided)
|
|
@@ -708,8 +724,10 @@ def cmd_learn(args):
|
|
|
708
724
|
continue
|
|
709
725
|
tokens[0] += result["usage"]["input_tokens"]
|
|
710
726
|
tokens[1] += result["usage"]["output_tokens"]
|
|
727
|
+
one = result_cost(result)
|
|
728
|
+
spent = None if spent is None or one is None else spent + one
|
|
711
729
|
day = tl.now_day()
|
|
712
|
-
tl.append(
|
|
730
|
+
tl.append(repo, "decisions", {
|
|
713
731
|
"id": sig["id"], "day": day, "at": sig.get("at"), "session": sig["session"],
|
|
714
732
|
"provider": result.get("provider"), "model": result.get("model"),
|
|
715
733
|
"verified": dec["verified"], "kind": dec["kind"][0], "p": dec["kind"][1],
|
|
@@ -719,10 +737,10 @@ def cmd_learn(args):
|
|
|
719
737
|
if not dec["verified"]:
|
|
720
738
|
counts["unverified"] += 1
|
|
721
739
|
continue
|
|
722
|
-
counts["
|
|
740
|
+
counts["jev"] += 1
|
|
723
741
|
counts[dec["kind"][0] if dec["kind"][0] in CLASSES else "none"] += 1
|
|
724
742
|
if dec["kind"][0] == "defect":
|
|
725
|
-
tl.append(
|
|
743
|
+
tl.append(repo, "defects",
|
|
726
744
|
{"id": sig["id"], "day": day, "session": sig["session"],
|
|
727
745
|
"text": ti.redact(sig["text"])[:600], "paths": sig["paths"][:8]},
|
|
728
746
|
strict=True)
|
|
@@ -741,16 +759,18 @@ def cmd_learn(args):
|
|
|
741
759
|
for learning in tl.every(ledgers).values():
|
|
742
760
|
states[learning["state"]] = states.get(learning["state"], 0) + 1
|
|
743
761
|
report = dict(counts, states=states, input_tokens=tokens[0], output_tokens=tokens[1],
|
|
744
|
-
cost_usd=
|
|
762
|
+
cost_usd=None if spent is None else round(spent, 6))
|
|
745
763
|
if args.json:
|
|
746
764
|
print(json.dumps(report, indent=2))
|
|
747
765
|
return 0
|
|
748
|
-
print("signals %d:
|
|
749
|
-
counts["signals"], counts["
|
|
766
|
+
print("signals %d: jev %d, unverified %d, unanswered %d" % (
|
|
767
|
+
counts["signals"], counts["jev"], counts["unverified"], counts["unanswered"]))
|
|
750
768
|
print("decided: preference %d, defect %d, none %d" % (
|
|
751
769
|
counts["preference"], counts["defect"], counts["none"]))
|
|
752
770
|
print("ledger: " + (", ".join("%s %d" % kv for kv in sorted(states.items())) or "empty"))
|
|
753
|
-
|
|
771
|
+
money = report["cost_usd"]
|
|
772
|
+
print("typed decisions: %d input tokens, %s" % (
|
|
773
|
+
tokens[0], "$%s" % money if money is not None else "cost unknown (no listed price)"))
|
|
754
774
|
return 0
|
|
755
775
|
|
|
756
776
|
|
|
@@ -806,17 +826,16 @@ def cmd_label(args):
|
|
|
806
826
|
sample of unlabelled typed decisions - drawn without regard to what was
|
|
807
827
|
injected, so the calibration bound rests on an unselected sample."""
|
|
808
828
|
repo = store_or_exit(args)
|
|
809
|
-
|
|
810
|
-
|
|
811
|
-
|
|
812
|
-
labelled = {r.get("id") for r in tl.rows(os.path.join(store, "labels.jsonl"))}
|
|
829
|
+
decided = {r["id"]: r for r in tl.rows(repo, "decisions", *tl.jev_rows())
|
|
830
|
+
if r.get("id")}
|
|
831
|
+
labelled = {r.get("id") for r in tl.rows(repo, "labels")}
|
|
813
832
|
if args.id:
|
|
814
833
|
if args.label not in CLASSES:
|
|
815
834
|
return misuse("label is one of %s" % ", ".join(CLASSES))
|
|
816
835
|
row = decided.get(args.id)
|
|
817
836
|
if row is None:
|
|
818
837
|
return misuse("no typed decision %s" % args.id)
|
|
819
|
-
tl.append(
|
|
838
|
+
tl.append(repo, "labels",
|
|
820
839
|
{"id": args.id, "label": args.label, "decided": row.get("kind"),
|
|
821
840
|
"provider": row.get("provider"), "day": tl.now_day()}, strict=True)
|
|
822
841
|
print("%s labelled %s (decided %s)" % (args.id, args.label, row.get("kind")))
|
|
@@ -872,7 +891,7 @@ def main(argv):
|
|
|
872
891
|
measure.add_argument("--model", default=tezgah_judge.MODEL)
|
|
873
892
|
measure.add_argument("--json", action="store_true")
|
|
874
893
|
measure.add_argument("--typed", action="store_true",
|
|
875
|
-
help="require
|
|
894
|
+
help="require the Jev model (the decision T1P's ledger trusts)")
|
|
876
895
|
rate = sub.add_parser("rate")
|
|
877
896
|
rate.add_argument("--in", dest="inp", required=True)
|
|
878
897
|
rate.add_argument("--limit", type=int)
|
|
@@ -882,7 +901,7 @@ def main(argv):
|
|
|
882
901
|
learn.add_argument("--host", choices=("omp", "claude", "all"), default="all")
|
|
883
902
|
learn.add_argument("--limit", type=int)
|
|
884
903
|
learn.add_argument("--no-fallback", action="store_true",
|
|
885
|
-
help="
|
|
904
|
+
help="the Jev model only: never pay another provider for a decision")
|
|
886
905
|
listing = sub.add_parser("list")
|
|
887
906
|
listing.add_argument("--all", action="store_true")
|
|
888
907
|
for name in ("show", "accept", "reject", "edit"):
|
|
@@ -910,7 +929,7 @@ def main(argv):
|
|
|
910
929
|
"gate": cmd_gate}[args.cmd]
|
|
911
930
|
if args.cmd not in ("learn", "accept", "reject", "edit", "label"):
|
|
912
931
|
return handler(args)
|
|
913
|
-
# the commands that rewrite
|
|
932
|
+
# the commands that rewrite the ledgers: one at a time, or the last one
|
|
914
933
|
# to finish would silently drop what the other wrote
|
|
915
934
|
try:
|
|
916
935
|
repo = store_or_exit(args)
|
package/bin/tezgah-triage
CHANGED
|
@@ -4,7 +4,8 @@
|
|
|
4
4
|
A `browser_snapshot` / `mobile_list_elements_on_screen` result is the whole screen
|
|
5
5
|
as text - one measured 40-row work-order table came back as 13,045 characters in
|
|
6
6
|
356 lines - and the analyze-app loop pays for all of it to find the dozen lines
|
|
7
|
-
that carry the controls. This flattens the snapshot, asks
|
|
7
|
+
that carry the controls. This flattens the snapshot, asks the judge seam (the
|
|
8
|
+
session's own CLI or the Jev model through any carrier) about the
|
|
8
9
|
units of the tree (a row, a table cell, a control on its own) in ONE batched
|
|
9
10
|
request, and prints the line ids under the selected units with their original
|
|
10
11
|
element refs.
|
|
@@ -90,10 +91,13 @@ MEASURED = ("100% of the control lines at a 94% read on a reproduced 356-line "
|
|
|
90
91
|
"screen")
|
|
91
92
|
CUT = 96
|
|
92
93
|
USAGE = "tezgah-triage --select FILE --task TEXT | --states FILE [--component TEXT]"
|
|
93
|
-
NO_CREDENTIAL = ("no judge: no session CLI (omp
|
|
94
|
-
"and no third party the `fallback`
|
|
95
|
-
"
|
|
96
|
-
"
|
|
94
|
+
NO_CREDENTIAL = ("no judge: no session CLI (omp, claude, opencode, cursor or codex, "
|
|
95
|
+
"the one this session runs under), and no third party the `fallback` "
|
|
96
|
+
"setting allows - the Jev model through TypeSafe (TYPESAFE_API_KEY or "
|
|
97
|
+
"~/.config/typesafe/key), OpenRouter System One or the OpenRouter chat "
|
|
98
|
+
"fallback (OPENROUTER_API_KEY or ~/.config/openrouter/key), Cloudflare "
|
|
99
|
+
"Workers AI, or a compatible endpoint (JEV_API_BASE_URL), the carrier "
|
|
100
|
+
"picked by TEZGAH_JEV_PROVIDER or config `jev` - or the judge-off "
|
|
97
101
|
"switch is on")
|
|
98
102
|
TRIAGE_OFF = "the triage-off switch is on"
|
|
99
103
|
FAILED = ("the call failed - a timeout, a refusal, or a reply that is not the "
|
|
@@ -197,10 +201,11 @@ def trim(line):
|
|
|
197
201
|
|
|
198
202
|
def judge_line(result):
|
|
199
203
|
usage = result["usage"]
|
|
204
|
+
# the carrier's own price (Clef's is not Jev's); Jev's for a chat stand-in
|
|
205
|
+
price = tezgah_judge.input_price(result.get("provider")) or PRICE_PER_MILLION
|
|
200
206
|
print("judge: %d input tokens, %d output, %d ms; $%.6f at $%.3f/1M input"
|
|
201
207
|
% (usage["input_tokens"], usage["output_tokens"], result["latency_ms"],
|
|
202
|
-
usage["input_tokens"] *
|
|
203
|
-
PRICE_PER_MILLION))
|
|
208
|
+
usage["input_tokens"] * price / 1000000, price))
|
|
204
209
|
|
|
205
210
|
|
|
206
211
|
def units(numbered):
|
package/docs/architecture.md
CHANGED
|
@@ -154,10 +154,10 @@ lowest-value blocks are dropped in a fixed order rather than the rules
|
|
|
154
154
|
|
|
155
155
|
| Store | Path | Writer | Authoritative for |
|
|
156
156
|
|---|---|---|---|
|
|
157
|
-
| Evidence ledger | `~/.cache/tezgah/evidence/<session>.jsonl` (`hooks/tezgah_integrity.py::_path`) | `note_tool` from each host's PostToolUse (`hooks/tezgah_integrity.py::note_tool`) | what a session actually ran, and therefore the Stop verdict |
|
|
158
|
-
| Session store (used marks) | the
|
|
159
|
-
| Snapshot store | `<cache dir>/snapshots` (`hooks/tezgah_snapshot.py::_store`) | `capture` on the write path (`hooks/tezgah_snapshot.py::capture`) | the pre-write bytes; the rollback source |
|
|
160
|
-
| Turn stamp |
|
|
157
|
+
| Evidence ledger | the `evidence` rows of `~/.cache/tezgah/tezgah.db`, keyed by the session's `evidence/<session>.jsonl` path (`hooks/tezgah_integrity.py::_path`, `hooks/tezgah_store.py::EVIDENCE_SCHEMA`); an old JSONL file at that path is imported (`hooks/tezgah_store.py::_sync`) | `note_tool` from each host's PostToolUse (`hooks/tezgah_integrity.py::note_tool`) | what a session actually ran, and therefore the Stop verdict |
|
|
158
|
+
| Session store (used marks) | the `used` rows of the cache dir's `tezgah.db`, one per session and kind (`hooks/tezgah_store.py::use`) | `record` (`hooks/tezgah_context.py::record`) from every adapter | nothing evidential: a convenience channel for a surface that has no transcript; an `orch` mark is also written to the evidence ledger (`hooks/tezgah_context.py::record`) so `fanout` can count it |
|
|
159
|
+
| Snapshot store | the blobs under `<cache dir>/snapshots` (`hooks/tezgah_snapshot.py::_store`), their manifests the `snapshots` rows of `tezgah.db` (`hooks/tezgah_store.py::add_snapshot`) | `capture` on the write path (`hooks/tezgah_snapshot.py::capture`) | the pre-write bytes; the rollback source |
|
|
160
|
+
| Turn stamp | the `turns` rows of the cache dir's `tezgah.db`, one per session (`hooks/tezgah_context.py::write_stamp`) | `write_stamp` (`hooks/tezgah_context.py::write_stamp`) | the comparison behind the one-line delta, nothing else |
|
|
161
161
|
| Installed config dir | `~/.config/tezgah` (`hooks/tezgah_paths.py::CONFIG_DIR`) | the installer and the user | which roots are armed, which kill switches are on |
|
|
162
162
|
| Plugin copy | `~/.claude/plugins/cache/<marketplace>/tezgah/<version>/` (`bin/tezgah-setup::plugin_copies`) | `tezgah-setup --sync` (`bin/tezgah-setup::sync`) | what Claude Code actually executes — a copy, never this checkout |
|
|
163
163
|
|