@r1z4x/tezgah 1.4.0 → 1.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +120 -1
- package/MANIFEST +0 -1
- package/README.md +6 -4
- package/bin/tezgah-docs +3 -3
- package/bin/tezgah-route +12 -6
- package/bin/tezgah-setup +67 -24
- package/bin/tezgah-status +3 -1
- package/bin/tezgah-task +132 -39
- package/bin/tezgah-taste +49 -31
- package/bin/tezgah-triage +12 -7
- package/docs/architecture.md +1 -1
- package/docs/clarity-baseline.json +7 -7
- package/docs/contract.md +21 -11
- package/docs/environment.md +3 -1
- package/docs/evidence.md +4 -3
- package/docs/glossary.md +1 -1
- package/docs/hosts.md +9 -0
- package/docs/judge.md +95 -27
- package/docs/layers.md +1 -1
- package/docs/models.md +17 -7
- package/docs/operations.md +23 -13
- package/docs/skills.md +1 -1
- package/docs/testing.md +1 -1
- package/docs/vision.md +10 -5
- package/hooks/tezgah_agents.py +149 -105
- package/hooks/tezgah_context.py +42 -7
- package/hooks/tezgah_judge.py +252 -79
- package/hooks/tezgah_models.py +50 -3
- package/hooks/tezgah_paths.py +180 -27
- package/hooks/tezgah_policy.py +41 -10
- package/hooks/tezgah_replay.py +26 -15
- package/hooks/tezgah_task.py +160 -46
- package/hooks/tezgah_taste.py +2 -2
- package/hooks/tezgah_taste_ledger.py +13 -4
- package/hosts/codex/hook.py +5 -0
- package/package.json +1 -2
- package/skills/harness/SKILL.md +4 -0
- package/skills/plan-add/SKILL.md +23 -7
- package/skills/plan-add/render_table.py +3 -2
- package/skills/plan-status/SKILL.md +12 -4
- package/skills/plan-sync/SKILL.md +19 -9
- package/skills/tezgah-contract/SKILL.md +29 -14
- package/agents/tezgah-reviewer.md +0 -70
package/bin/tezgah-task
CHANGED
|
@@ -5,7 +5,7 @@
|
|
|
5
5
|
tezgah-task phase P [--any-path] change the active task's phase
|
|
6
6
|
tezgah-task allow GLOB ... | --any-path replace the allowlist
|
|
7
7
|
tezgah-task stop deactivate (clear phase)
|
|
8
|
-
tezgah-task status [--json] the active task,
|
|
8
|
+
tezgah-task status [--json] the active task, then every open plan: progress, what it waits on, what can run at once
|
|
9
9
|
tezgah-task review <NNN|path> <reviewer> approve|changes record a fresh review's verdict
|
|
10
10
|
tezgah-task close <NNN|path> done|discarded move a plan to .tezgah/plans/done
|
|
11
11
|
|
|
@@ -22,21 +22,27 @@ creates the plan, and `start` without `--allow` keeps it. A writing phase with
|
|
|
22
22
|
no allowlist is refused unless `--any-path` says so explicitly, and the record
|
|
23
23
|
keeps where its scope came from (`allowed_from:`), which `status` prints.
|
|
24
24
|
|
|
25
|
-
`review` and `close` are the session's: the verification phase asks for
|
|
26
|
-
|
|
25
|
+
`review` and `close` are the session's: the verification phase asks for one
|
|
26
|
+
review of the plan's whole diff, `review` records its verdict as
|
|
27
27
|
`review: <reviewer> <verdict> <date>` together with `reviewed_sha:` (the
|
|
28
28
|
project's HEAD it read) and `review_round:`, so the next round reads only
|
|
29
|
-
`git diff <reviewed_sha>..HEAD
|
|
30
|
-
NNN done` refuses a plan whose review is missing or not
|
|
31
|
-
refuses the active task - that one is the user's to stop first.
|
|
29
|
+
`git diff <reviewed_sha>..HEAD`, and refuses a third round: the budget is two
|
|
30
|
+
rounds per plan; `close NNN done` refuses a plan whose review is missing or not
|
|
31
|
+
`approve`. `close` refuses the active task - that one is the user's to stop first.
|
|
32
32
|
|
|
33
33
|
Two things the plan owes are asked here, at the only moment they can still be
|
|
34
34
|
fixed - the phase move - because the record is the user's and the refusal is
|
|
35
|
-
this CLI's: a writing phase needs
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
`implementation` needs the plan's spike
|
|
39
|
-
evidence rather than on the guess the spike
|
|
35
|
+
this CLI's: a writing phase needs every Acceptance item to name the command
|
|
36
|
+
proving it (or say `unverifiable` and why), since the report CI runs afterwards
|
|
37
|
+
refuses an open plan on any one item that does neither, through the same
|
|
38
|
+
predicate (`tezgah_task.unproven`); and `implementation` needs the plan's spike
|
|
39
|
+
answered, so the build starts on evidence rather than on the guess the spike
|
|
40
|
+
exists to retire.
|
|
41
|
+
|
|
42
|
+
`status` lists every open plan after the active task: its Acceptance boxes
|
|
43
|
+
ticked, the plans its `after:` still waits on, and - on a `parallel:` line - the
|
|
44
|
+
ready plans whose `allowed_paths:` share no prefix, which can run at once. A
|
|
45
|
+
status outside open|blocked, or an `after:` id no plan carries, is a `FAIL` line.
|
|
40
46
|
|
|
41
47
|
Entering `implementation` also records where the risky work returns to: a
|
|
42
48
|
`checkpoint:` field holding the pre-work commit's sha, or `pending <sha>` while
|
|
@@ -139,6 +145,14 @@ def write_fields(path, **fields):
|
|
|
139
145
|
|
|
140
146
|
ANY_PATH = "--any-path"
|
|
141
147
|
VERDICTS = ("approve", "changes")
|
|
148
|
+
# The review budget, enforced here and nowhere else: `review` refuses a third
|
|
149
|
+
# round, and every text that states the rule (the reviewer agent, the contract,
|
|
150
|
+
# plan-sync) says the same two rounds.
|
|
151
|
+
REVIEW_ROUNDS = 2
|
|
152
|
+
# Under this many changed lines one reviewer reads every dimension; at or above
|
|
153
|
+
# it the round fans out. ponytail: a line count, not a risk model - a small diff
|
|
154
|
+
# that touches a trust boundary is the brief's to route to security.
|
|
155
|
+
SMALL_DIFF = 300
|
|
142
156
|
|
|
143
157
|
|
|
144
158
|
def allow_source(text):
|
|
@@ -279,13 +293,14 @@ def phase_problems(phase, text, path):
|
|
|
279
293
|
items = tt.acceptance_gap(text)
|
|
280
294
|
if items:
|
|
281
295
|
out.append(
|
|
282
|
-
"%s cannot enter %s:
|
|
283
|
-
"command and
|
|
284
|
-
"hooks/tezgah_task.py:acceptance_items), so
|
|
285
|
-
"to be held to. Give
|
|
296
|
+
"%s cannot enter %s: %d of its %d Acceptance items name no "
|
|
297
|
+
"command and declare no `unverifiable` (read by "
|
|
298
|
+
"hooks/tezgah_task.py:acceptance_items; at %s), so that work has "
|
|
299
|
+
"no check to be held to. Give each one the command that proves "
|
|
286
300
|
"it, or `unverifiable` and why; "
|
|
287
|
-
"`tezgah-render-table --acceptance`
|
|
288
|
-
% (path, phase, len(items)))
|
|
301
|
+
"`tezgah-render-table --acceptance --strict` refuses the same items"
|
|
302
|
+
% (path, phase, len(items), len(tt.acceptance_items(text)),
|
|
303
|
+
", ".join("%s:%d" % (path, i["line"]) for i in items)))
|
|
289
304
|
if phase == "implementation" and tt.spike_unanswered(text):
|
|
290
305
|
out.append(
|
|
291
306
|
"%s cannot enter implementation: its spike is unanswered - `spike:` "
|
|
@@ -295,43 +310,78 @@ def phase_problems(phase, text, path):
|
|
|
295
310
|
return out
|
|
296
311
|
|
|
297
312
|
|
|
313
|
+
def changed_lines(root, base):
|
|
314
|
+
"""Lines `git diff <base>..HEAD` adds and removes, or None when git cannot
|
|
315
|
+
say (no base, not a repository)."""
|
|
316
|
+
stat = git_out(root, "diff", "--shortstat", base + "..HEAD") \
|
|
317
|
+
if base != "<base>" else None
|
|
318
|
+
if stat is None:
|
|
319
|
+
return None
|
|
320
|
+
return sum(int(n) for n in re.findall(r"(\d+) (?:insertion|deletion)", stat))
|
|
321
|
+
|
|
322
|
+
|
|
298
323
|
def verification_steps(text, path):
|
|
299
324
|
"""What the verification phase requires before the plan is done: the plan's
|
|
300
|
-
own acceptance commands run once on the final tree, unpiped, and
|
|
301
|
-
|
|
302
|
-
without
|
|
303
|
-
|
|
325
|
+
own acceptance commands run once on the final tree, unpiped, and ONE review
|
|
326
|
+
of the plan's whole diff, recorded with `review` - `close NNN done` refuses
|
|
327
|
+
without an approve. Round one fans out one reviewer per dimension, round two
|
|
328
|
+
reads only the delta since the sha round one recorded, and REVIEW_ROUNDS is
|
|
329
|
+
the budget `review` enforces."""
|
|
304
330
|
fields = tt.frontmatter(text)
|
|
305
331
|
ident = fields.get("id") or os.path.basename(path).split("-")[0]
|
|
306
|
-
# every item, not only the ones the command reader recognises:
|
|
307
|
-
#
|
|
332
|
+
# every item, not only the ones the command reader recognises: a bare
|
|
333
|
+
# `pytest` is a command it does not read as one, and must not vanish here
|
|
308
334
|
items = ["%s%s" % (i["text"], "" if i["state"] == "checkable" else
|
|
309
335
|
" [%s]" % ("unverifiable" if i["state"] == "unverifiable"
|
|
310
336
|
else "names no command"))
|
|
311
337
|
for i in tt.acceptance_items(text)]
|
|
312
338
|
# the project, not the private repository `.tezgah` may hold
|
|
313
339
|
root = os.path.realpath(os.path.join(os.path.dirname(path), "..", "..", ".."))
|
|
314
|
-
reviewed = fields.get("reviewed_sha") or ""
|
|
340
|
+
reviewed = fields.get("reviewed_sha") or "(no commit)"
|
|
315
341
|
prior = fields.get("review_round") or ""
|
|
316
342
|
rounds = int(prior) if prior.isdigit() else 0
|
|
317
343
|
print("verification - required before this plan is done:")
|
|
318
344
|
print(" 1. prove the acceptance items ONCE, on the final tree - not again per "
|
|
319
345
|
"review round - every command run unpiped "
|
|
320
|
-
"(`cmd > /tmp/check.log 2>&1`, then read the file)
|
|
346
|
+
"(`cmd > /tmp/check.log 2>&1`, then read the file); items that share "
|
|
347
|
+
"no state go out as tool calls in ONE message, one suite run at a time:")
|
|
321
348
|
for item in items or ["(the plan has no Acceptance items)"]:
|
|
322
349
|
print(" - %s" % item)
|
|
323
350
|
if os.path.isfile(os.path.join(root, "tests", "impacted.py")):
|
|
324
351
|
print(" the one full run here is `python3 tests/impacted.py --all`")
|
|
325
|
-
if
|
|
326
|
-
print(" 2. review round %d
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
|
|
352
|
+
if rounds and review_verdict(text) == "approve":
|
|
353
|
+
print(" 2. the review approved in round %d at %s: no further round"
|
|
354
|
+
% (rounds, reviewed))
|
|
355
|
+
print(" 3. `tezgah-task close %s done` once the work lands" % ident)
|
|
356
|
+
return
|
|
357
|
+
if rounds >= REVIEW_ROUNDS:
|
|
358
|
+
print(" 2. the %d-round budget is spent: `tezgah-task review` refuses a "
|
|
359
|
+
"third round. A critical or major finding still confirmed goes to "
|
|
360
|
+
"the user (block); a minor or suggestion one is recorded fix-later "
|
|
361
|
+
"under the plan's ## State" % REVIEW_ROUNDS)
|
|
362
|
+
return
|
|
363
|
+
if rounds:
|
|
364
|
+
print(" 2. review round 2 (the last): each confirmed critical or major "
|
|
365
|
+
"finding is fixed with a regression test that failed first; one "
|
|
366
|
+
"fresh tezgah-reviewer reads only the delta `git diff %s..HEAD`, the "
|
|
367
|
+
"round-1 hunks and direct callers of the symbols it touches, and the "
|
|
368
|
+
"test output above, and marks every round-1 finding closed or open; "
|
|
369
|
+
"it does not re-run the suite" % reviewed)
|
|
332
370
|
else:
|
|
333
|
-
|
|
334
|
-
|
|
371
|
+
base = diff_base(root)
|
|
372
|
+
size = changed_lines(root, base)
|
|
373
|
+
print(" 2. one review of the whole plan, at most %d rounds: spawn fresh "
|
|
374
|
+
"tezgah-reviewer subagents in ONE message, each owning one dimension "
|
|
375
|
+
"- correctness and contract, security, tests and performance - or one "
|
|
376
|
+
"file group of a large diff, over `git diff %s..HEAD` and the test "
|
|
377
|
+
"output above, then merge their findings into one verdict. Under %d "
|
|
378
|
+
"changed lines one reviewer reads every dimension (this diff: %s), "
|
|
379
|
+
"unless the diff touches auth, secrets or a gate, which keeps a "
|
|
380
|
+
"security reviewer" % (REVIEW_ROUNDS, base, SMALL_DIFF,
|
|
381
|
+
"unknown" if size is None else size))
|
|
382
|
+
print(" only a confirmed critical or major finding is `changes` and "
|
|
383
|
+
"opens round 2; a minor or suggestion one is recorded fix-later "
|
|
384
|
+
"under ## State and the verdict is `approve`")
|
|
335
385
|
print(" 3. record its verdict: `tezgah-task review %s <reviewer> approve|changes`"
|
|
336
386
|
" (records the reviewed sha);"
|
|
337
387
|
" `tezgah-task close %s done` refuses without an approve" % (ident, ident))
|
|
@@ -495,6 +545,16 @@ def review_verdict(text):
|
|
|
495
545
|
return parts[1] if len(parts) >= 2 and parts[1] in VERDICTS else None
|
|
496
546
|
|
|
497
547
|
|
|
548
|
+
def budget_spent(path, ident):
|
|
549
|
+
"""The refusal once REVIEW_ROUNDS rounds are recorded: what is left belongs
|
|
550
|
+
to the user, never to a third round."""
|
|
551
|
+
return ("%s has spent its %d-round review budget: never a third round. A "
|
|
552
|
+
"critical or major finding still confirmed goes to the user, who "
|
|
553
|
+
"decides (block); a minor or suggestion one is recorded fix-later "
|
|
554
|
+
"under the plan's ## State; `tezgah-task close %s discarded` needs no "
|
|
555
|
+
"review" % (path, REVIEW_ROUNDS, ident))
|
|
556
|
+
|
|
557
|
+
|
|
498
558
|
def cmd_review(args):
|
|
499
559
|
if len(args) != 3 or args[2] not in VERDICTS:
|
|
500
560
|
return misuse("review takes <NNN|path> <reviewer> %s" % "|".join(VERDICTS))
|
|
@@ -503,6 +563,8 @@ def cmd_review(args):
|
|
|
503
563
|
return problems([problem])
|
|
504
564
|
fields = tt.frontmatter(read(path))
|
|
505
565
|
prior = fields.get("review_round") or ""
|
|
566
|
+
if prior.isdigit() and int(prior) >= REVIEW_ROUNDS:
|
|
567
|
+
return problems([budget_spent(path, os.path.basename(path).split("-")[0])])
|
|
506
568
|
# the project's HEAD, never the private `.tezgah` repository's: the next
|
|
507
569
|
# round's delta is `git diff <this>..HEAD` over the project
|
|
508
570
|
root = os.path.realpath(os.path.join(os.path.dirname(path), "..", "..", ".."))
|
|
@@ -535,6 +597,9 @@ def cmd_close(args):
|
|
|
535
597
|
return problems(["%s is the active task: the user stops it first "
|
|
536
598
|
"(`tezgah-task stop`)" % path])
|
|
537
599
|
if args[1] == "done" and review_verdict(text) != "approve":
|
|
600
|
+
prior = tt.frontmatter(text).get("review_round") or ""
|
|
601
|
+
if prior.isdigit() and int(prior) >= REVIEW_ROUNDS:
|
|
602
|
+
return problems([budget_spent(path, ident)])
|
|
538
603
|
return problems(["%s records no approving review: a fresh reviewer reads "
|
|
539
604
|
"the diff, then `tezgah-task review %s <reviewer> approve`"
|
|
540
605
|
% (path, ident)])
|
|
@@ -561,22 +626,50 @@ def cmd_status(args):
|
|
|
561
626
|
bad = []
|
|
562
627
|
for path in plans(root):
|
|
563
628
|
fields = tt.frontmatter(read(path))
|
|
629
|
+
ident = fields.get("id") or os.path.basename(path).split("-")[0]
|
|
564
630
|
phase = fields.get("phase") or ""
|
|
565
631
|
if phase and phase not in tt.TASK_PHASES:
|
|
566
|
-
bad.append({"id":
|
|
567
|
-
"
|
|
632
|
+
bad.append({"id": ident, "phase": phase, "path": path,
|
|
633
|
+
"problem": "phase %r is not one of %s, so the plan is ignored (%s)"
|
|
634
|
+
% (phase, ", ".join(tt.TASK_PHASES), path)})
|
|
635
|
+
status = fields.get("status") or ""
|
|
636
|
+
if status not in tt.OPEN_STATUSES:
|
|
637
|
+
bad.append({"id": ident, "path": path,
|
|
638
|
+
"problem": "status %r is not one of %s, which is what a plan "
|
|
639
|
+
"under .tezgah/plans/open carries (%s)"
|
|
640
|
+
% (status, ", ".join(tt.OPEN_STATUSES), path)})
|
|
641
|
+
rows = tt.queue(root)
|
|
642
|
+
for row in rows:
|
|
643
|
+
for ident, state in row["after"]:
|
|
644
|
+
if state == "missing":
|
|
645
|
+
bad.append({"id": row["id"], "path": row["path"],
|
|
646
|
+
"problem": "`after: %s` names no plan under .tezgah/plans "
|
|
647
|
+
"(%s)" % (ident, row["path"])})
|
|
648
|
+
together = tt.parallel(rows)
|
|
568
649
|
if task:
|
|
569
650
|
task["allowed_from"] = allow_source(read(task["path"]))[1]
|
|
570
651
|
if as_json:
|
|
571
|
-
print(json.dumps({"active": task, "malformed": bad
|
|
652
|
+
print(json.dumps({"active": task, "malformed": bad, "open": rows,
|
|
653
|
+
"parallel": together}, indent=2))
|
|
572
654
|
else:
|
|
573
655
|
if task:
|
|
574
656
|
show(task["path"])
|
|
575
657
|
else:
|
|
576
658
|
print("no active task")
|
|
659
|
+
if rows:
|
|
660
|
+
print("open plans: %d, %d ready" % (len(rows), sum(r["ready"] for r in rows)))
|
|
661
|
+
for row in rows:
|
|
662
|
+
waits = ["%s (%s)" % (ident, state) for ident, state in row["after"]
|
|
663
|
+
if state != "done"]
|
|
664
|
+
print(" %s %s, acceptance %d/%d checked, %s" % (
|
|
665
|
+
row["id"], row["status"] or "(no status)", row["checked"], row["items"],
|
|
666
|
+
"ready" if row["ready"] else
|
|
667
|
+
"waits on %s" % ", ".join(waits) if waits else row["status"]))
|
|
668
|
+
if together:
|
|
669
|
+
print("parallel: %s - ready, and no two share an allowed_paths prefix"
|
|
670
|
+
% ", ".join(together))
|
|
577
671
|
for row in bad:
|
|
578
|
-
print("FAIL %s:
|
|
579
|
-
% (row["id"], row["phase"], ", ".join(tt.TASK_PHASES), row["path"]))
|
|
672
|
+
print("FAIL %s: %s" % (row["id"], row["problem"]))
|
|
580
673
|
return 1 if bad else 0
|
|
581
674
|
|
|
582
675
|
|
package/bin/tezgah-taste
CHANGED
|
@@ -39,9 +39,11 @@ design T1P). Each prompt after a writing turn - from the capture rows in the
|
|
|
39
39
|
from this repository's omp and Claude transcripts - becomes one structured signal
|
|
40
40
|
built by code: the prompt, the files written, the agent's edits and the user's
|
|
41
41
|
own edit between two agent writes. One typed decision per signal, required from
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
42
|
+
the Jev model (`only=("jev",)`: TypeSafe, OpenRouter System One, Cloudflare
|
|
43
|
+
Workers AI or a compatible endpoint - whichever carrier the judge seam resolves):
|
|
44
|
+
preference, defect or none; the category; the scope; and how it relates to each
|
|
45
|
+
learning held. A decision any other provider gave is recorded `unverified` and
|
|
46
|
+
never applied. A defect goes to the `defects`
|
|
45
47
|
table, never into the ledger. The ledger itself is `hooks/tezgah_taste_ledger.py`. A
|
|
46
48
|
learning that becomes active gets one readable line from a generative provider
|
|
47
49
|
(a `text` question; the user's own words when none answers).
|
|
@@ -320,7 +322,7 @@ def classify(texts, model, only=None):
|
|
|
320
322
|
|
|
321
323
|
The rule rides the state, so each request pays for it once and the
|
|
322
324
|
questions carry only their own prompt."""
|
|
323
|
-
labels, tokens_in, tokens_out, used = [], 0, 0, model
|
|
325
|
+
labels, tokens_in, tokens_out, used, spent = [], 0, 0, model, 0.0
|
|
324
326
|
for start in range(0, len(texts), BATCH):
|
|
325
327
|
chunk = texts[start:start + BATCH]
|
|
326
328
|
questions = {"p%d" % i: {"type": "choice", "criteria": CRITERIA,
|
|
@@ -332,24 +334,36 @@ def classify(texts, model, only=None):
|
|
|
332
334
|
tokens_in += result["usage"]["input_tokens"]
|
|
333
335
|
tokens_out += result["usage"]["output_tokens"]
|
|
334
336
|
used = result.get("model", used)
|
|
337
|
+
one = result_cost(result)
|
|
338
|
+
spent = None if spent is None or one is None else spent + one
|
|
335
339
|
for i in range(len(chunk)):
|
|
336
340
|
label = tezgah_judge.choice(result, "p%d" % i)
|
|
337
341
|
labels.append(label if label in CLASSES else None)
|
|
338
342
|
return labels, {"model": used, "input_tokens": tokens_in,
|
|
339
|
-
"output_tokens": tokens_out,
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
def cost(model, tokens_in, tokens_out):
|
|
344
|
-
"""Dollars at the model's listed price per 1M tokens, None when unlisted.
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
343
|
+
"output_tokens": tokens_out,
|
|
344
|
+
"cost_usd": None if spent is None else round(spent, 6)}
|
|
345
|
+
|
|
346
|
+
|
|
347
|
+
def cost(model, tokens_in, tokens_out, provider=None):
|
|
348
|
+
"""Dollars at the model's listed price per 1M tokens, None when unlisted. A
|
|
349
|
+
typed carrier (`tezgah_judge.input_price`) is priced by the carrier that
|
|
350
|
+
answered: Clef's price is not Jev's."""
|
|
351
|
+
typed = tezgah_judge.input_price(provider)
|
|
352
|
+
price = (typed, 0.0) if typed is not None else JEV_PRICE \
|
|
353
|
+
if model == tezgah_judge.MODEL else (
|
|
354
|
+
tezgah_models.overlay().get("prices", {}).get(model)
|
|
355
|
+
or tezgah_models.SNAPSHOT.get(model))
|
|
348
356
|
if not (isinstance(price, (list, tuple)) and len(price) == 2):
|
|
349
357
|
return None
|
|
350
358
|
return round((tokens_in * price[0] + tokens_out * price[1]) / 1e6, 6)
|
|
351
359
|
|
|
352
360
|
|
|
361
|
+
def result_cost(result):
|
|
362
|
+
"""One judge result's cost, priced by the provider that answered it."""
|
|
363
|
+
return cost(result.get("model"), result["usage"]["input_tokens"],
|
|
364
|
+
result["usage"]["output_tokens"], result.get("provider"))
|
|
365
|
+
|
|
366
|
+
|
|
353
367
|
def usage_line(usage):
|
|
354
368
|
money = ("$%.6f" % usage["cost_usd"] if usage["cost_usd"] is not None
|
|
355
369
|
else "cost unknown (no listed price)")
|
|
@@ -370,7 +384,7 @@ def cmd_measure(args):
|
|
|
370
384
|
if not tezgah_judge.available():
|
|
371
385
|
return misuse("no judge: no credential resolves or judge-off is set; nothing sent")
|
|
372
386
|
got, usage = classify([text for text, _gold in pairs], args.model,
|
|
373
|
-
only=("
|
|
387
|
+
only=("jev",) if args.typed else None)
|
|
374
388
|
confusion = {g: {p: 0 for p in CLASSES} for g in CLASSES}
|
|
375
389
|
unjudged = 0
|
|
376
390
|
for (_text, gold), pred in zip(pairs, got):
|
|
@@ -628,12 +642,12 @@ def picked(result, qid, default):
|
|
|
628
642
|
|
|
629
643
|
|
|
630
644
|
def decide(sig, learnings, fallback=True):
|
|
631
|
-
"""(decision, result): the typed decision,
|
|
645
|
+
"""(decision, result): the typed decision, the Jev model required; when it cannot
|
|
632
646
|
answer, the seam's own providers, recorded unverified - only on a signal's
|
|
633
647
|
first try, since an unverified answer is never applied and a rerun would pay
|
|
634
648
|
for it again."""
|
|
635
649
|
state, questions = decision_request(sig, learnings)
|
|
636
|
-
result = tezgah_judge.ask(state, questions, only=("
|
|
650
|
+
result = tezgah_judge.ask(state, questions, only=("jev",), timeout=60)
|
|
637
651
|
if result is None and fallback:
|
|
638
652
|
result = tezgah_judge.ask(state, questions, timeout=120)
|
|
639
653
|
if result is None:
|
|
@@ -643,7 +657,7 @@ def decide(sig, learnings, fallback=True):
|
|
|
643
657
|
"scope": picked(result, "scope", "repository"),
|
|
644
658
|
"relations": {lr["id"]: picked(result, "rel_" + lr["id"], "unrelated")
|
|
645
659
|
for lr in learnings},
|
|
646
|
-
"verified": result.get("provider")
|
|
660
|
+
"verified": tezgah_judge.is_jev(result.get("provider"))}
|
|
647
661
|
return dec, result
|
|
648
662
|
|
|
649
663
|
|
|
@@ -678,7 +692,7 @@ def learn_refusal(repo):
|
|
|
678
692
|
"""Why `learn` sends and writes nothing here, or None - read before the
|
|
679
693
|
store lock, so a refused run leaves no lock file behind."""
|
|
680
694
|
if tezgah_judge.tp.off("judge-off") or not (
|
|
681
|
-
tezgah_judge.available() or tezgah_judge.named(("
|
|
695
|
+
tezgah_judge.available() or tezgah_judge.named(("jev",))):
|
|
682
696
|
return "no judge: no credential resolves or judge-off is set; nothing sent"
|
|
683
697
|
import tezgah_context
|
|
684
698
|
if ".no-taste" in tezgah_context.repo_marks(repo)[1]:
|
|
@@ -692,16 +706,16 @@ def cmd_learn(args):
|
|
|
692
706
|
else capture_signals(repo))
|
|
693
707
|
decided = {}
|
|
694
708
|
for r in tl.rows(repo, "decisions"):
|
|
695
|
-
decided[r.get("id")] = decided.get(r.get("id")) or r.get("provider")
|
|
696
|
-
# a typed decision is final; an unverified one is retried,
|
|
709
|
+
decided[r.get("id")] = decided.get(r.get("id")) or tezgah_judge.is_jev(r.get("provider"))
|
|
710
|
+
# a typed decision is final; an unverified one is retried, the Jev model only
|
|
697
711
|
todo = [s for s in signals if not decided.get(s["id"])]
|
|
698
712
|
if args.limit is not None:
|
|
699
713
|
todo = todo[:args.limit]
|
|
700
714
|
ledgers = tl.load(repo)
|
|
701
715
|
before = {k for k, v in tl.every(ledgers).items() if v["state"] == "active"}
|
|
702
|
-
counts = {"signals": len(todo), "
|
|
716
|
+
counts = {"signals": len(todo), "jev": 0, "unverified": 0, "unanswered": 0,
|
|
703
717
|
"preference": 0, "defect": 0, "none": 0}
|
|
704
|
-
tokens = [0, 0]
|
|
718
|
+
tokens, spent = [0, 0], 0.0
|
|
705
719
|
for sig in todo:
|
|
706
720
|
dec, result = decide(sig, held(ledgers),
|
|
707
721
|
fallback=not args.no_fallback and sig["id"] not in decided)
|
|
@@ -710,6 +724,8 @@ def cmd_learn(args):
|
|
|
710
724
|
continue
|
|
711
725
|
tokens[0] += result["usage"]["input_tokens"]
|
|
712
726
|
tokens[1] += result["usage"]["output_tokens"]
|
|
727
|
+
one = result_cost(result)
|
|
728
|
+
spent = None if spent is None or one is None else spent + one
|
|
713
729
|
day = tl.now_day()
|
|
714
730
|
tl.append(repo, "decisions", {
|
|
715
731
|
"id": sig["id"], "day": day, "at": sig.get("at"), "session": sig["session"],
|
|
@@ -721,7 +737,7 @@ def cmd_learn(args):
|
|
|
721
737
|
if not dec["verified"]:
|
|
722
738
|
counts["unverified"] += 1
|
|
723
739
|
continue
|
|
724
|
-
counts["
|
|
740
|
+
counts["jev"] += 1
|
|
725
741
|
counts[dec["kind"][0] if dec["kind"][0] in CLASSES else "none"] += 1
|
|
726
742
|
if dec["kind"][0] == "defect":
|
|
727
743
|
tl.append(repo, "defects",
|
|
@@ -743,16 +759,18 @@ def cmd_learn(args):
|
|
|
743
759
|
for learning in tl.every(ledgers).values():
|
|
744
760
|
states[learning["state"]] = states.get(learning["state"], 0) + 1
|
|
745
761
|
report = dict(counts, states=states, input_tokens=tokens[0], output_tokens=tokens[1],
|
|
746
|
-
cost_usd=
|
|
762
|
+
cost_usd=None if spent is None else round(spent, 6))
|
|
747
763
|
if args.json:
|
|
748
764
|
print(json.dumps(report, indent=2))
|
|
749
765
|
return 0
|
|
750
|
-
print("signals %d:
|
|
751
|
-
counts["signals"], counts["
|
|
766
|
+
print("signals %d: jev %d, unverified %d, unanswered %d" % (
|
|
767
|
+
counts["signals"], counts["jev"], counts["unverified"], counts["unanswered"]))
|
|
752
768
|
print("decided: preference %d, defect %d, none %d" % (
|
|
753
769
|
counts["preference"], counts["defect"], counts["none"]))
|
|
754
770
|
print("ledger: " + (", ".join("%s %d" % kv for kv in sorted(states.items())) or "empty"))
|
|
755
|
-
|
|
771
|
+
money = report["cost_usd"]
|
|
772
|
+
print("typed decisions: %d input tokens, %s" % (
|
|
773
|
+
tokens[0], "$%s" % money if money is not None else "cost unknown (no listed price)"))
|
|
756
774
|
return 0
|
|
757
775
|
|
|
758
776
|
|
|
@@ -808,8 +826,8 @@ def cmd_label(args):
|
|
|
808
826
|
sample of unlabelled typed decisions - drawn without regard to what was
|
|
809
827
|
injected, so the calibration bound rests on an unselected sample."""
|
|
810
828
|
repo = store_or_exit(args)
|
|
811
|
-
decided = {r["id"]: r for r in tl.rows(repo, "decisions",
|
|
812
|
-
|
|
829
|
+
decided = {r["id"]: r for r in tl.rows(repo, "decisions", *tl.jev_rows())
|
|
830
|
+
if r.get("id")}
|
|
813
831
|
labelled = {r.get("id") for r in tl.rows(repo, "labels")}
|
|
814
832
|
if args.id:
|
|
815
833
|
if args.label not in CLASSES:
|
|
@@ -873,7 +891,7 @@ def main(argv):
|
|
|
873
891
|
measure.add_argument("--model", default=tezgah_judge.MODEL)
|
|
874
892
|
measure.add_argument("--json", action="store_true")
|
|
875
893
|
measure.add_argument("--typed", action="store_true",
|
|
876
|
-
help="require
|
|
894
|
+
help="require the Jev model (the decision T1P's ledger trusts)")
|
|
877
895
|
rate = sub.add_parser("rate")
|
|
878
896
|
rate.add_argument("--in", dest="inp", required=True)
|
|
879
897
|
rate.add_argument("--limit", type=int)
|
|
@@ -883,7 +901,7 @@ def main(argv):
|
|
|
883
901
|
learn.add_argument("--host", choices=("omp", "claude", "all"), default="all")
|
|
884
902
|
learn.add_argument("--limit", type=int)
|
|
885
903
|
learn.add_argument("--no-fallback", action="store_true",
|
|
886
|
-
help="
|
|
904
|
+
help="the Jev model only: never pay another provider for a decision")
|
|
887
905
|
listing = sub.add_parser("list")
|
|
888
906
|
listing.add_argument("--all", action="store_true")
|
|
889
907
|
for name in ("show", "accept", "reject", "edit"):
|
package/bin/tezgah-triage
CHANGED
|
@@ -4,7 +4,8 @@
|
|
|
4
4
|
A `browser_snapshot` / `mobile_list_elements_on_screen` result is the whole screen
|
|
5
5
|
as text - one measured 40-row work-order table came back as 13,045 characters in
|
|
6
6
|
356 lines - and the analyze-app loop pays for all of it to find the dozen lines
|
|
7
|
-
that carry the controls. This flattens the snapshot, asks
|
|
7
|
+
that carry the controls. This flattens the snapshot, asks the judge seam (the
|
|
8
|
+
session's own CLI or the Jev model through any carrier) about the
|
|
8
9
|
units of the tree (a row, a table cell, a control on its own) in ONE batched
|
|
9
10
|
request, and prints the line ids under the selected units with their original
|
|
10
11
|
element refs.
|
|
@@ -90,10 +91,13 @@ MEASURED = ("100% of the control lines at a 94% read on a reproduced 356-line "
|
|
|
90
91
|
"screen")
|
|
91
92
|
CUT = 96
|
|
92
93
|
USAGE = "tezgah-triage --select FILE --task TEXT | --states FILE [--component TEXT]"
|
|
93
|
-
NO_CREDENTIAL = ("no judge: no session CLI (omp
|
|
94
|
-
"and no third party the `fallback`
|
|
95
|
-
"
|
|
96
|
-
"
|
|
94
|
+
NO_CREDENTIAL = ("no judge: no session CLI (omp, claude, opencode, cursor or codex, "
|
|
95
|
+
"the one this session runs under), and no third party the `fallback` "
|
|
96
|
+
"setting allows - the Jev model through TypeSafe (TYPESAFE_API_KEY or "
|
|
97
|
+
"~/.config/typesafe/key), OpenRouter System One or the OpenRouter chat "
|
|
98
|
+
"fallback (OPENROUTER_API_KEY or ~/.config/openrouter/key), Cloudflare "
|
|
99
|
+
"Workers AI, or a compatible endpoint (JEV_API_BASE_URL), the carrier "
|
|
100
|
+
"picked by TEZGAH_JEV_PROVIDER or config `jev` - or the judge-off "
|
|
97
101
|
"switch is on")
|
|
98
102
|
TRIAGE_OFF = "the triage-off switch is on"
|
|
99
103
|
FAILED = ("the call failed - a timeout, a refusal, or a reply that is not the "
|
|
@@ -197,10 +201,11 @@ def trim(line):
|
|
|
197
201
|
|
|
198
202
|
def judge_line(result):
|
|
199
203
|
usage = result["usage"]
|
|
204
|
+
# the carrier's own price (Clef's is not Jev's); Jev's for a chat stand-in
|
|
205
|
+
price = tezgah_judge.input_price(result.get("provider")) or PRICE_PER_MILLION
|
|
200
206
|
print("judge: %d input tokens, %d output, %d ms; $%.6f at $%.3f/1M input"
|
|
201
207
|
% (usage["input_tokens"], usage["output_tokens"], result["latency_ms"],
|
|
202
|
-
usage["input_tokens"] *
|
|
203
|
-
PRICE_PER_MILLION))
|
|
208
|
+
usage["input_tokens"] * price / 1000000, price))
|
|
204
209
|
|
|
205
210
|
|
|
206
211
|
def units(numbered):
|
package/docs/architecture.md
CHANGED
|
@@ -34,7 +34,7 @@ nothing about a host's event names or output envelope.
|
|
|
34
34
|
| `tezgah_paths.py` | where tezgah is armed: [roots](glossary.md#root) and the linked worktrees of a repository under one, kill switches, the config dir, and the writable cache dir - the fallback resolved on use rather than at import, so a gated call does not pay for `tempfile`, `shutil` or `sqlite3` | `hooks/tezgah_paths.py:2-14`, `hooks/tezgah_paths.py::cache_dir`, `hooks/tezgah_paths.py::fallback_cache` |
|
|
35
35
|
| `tezgah_snapshot.py` | pre-write bytes of every file a write is about to change, and the one explicit restore | `hooks/tezgah_snapshot.py:2-18`, `hooks/tezgah_snapshot.py::capture` |
|
|
36
36
|
| `tezgah_untrusted.py` | the untrusted-content label on a result from outside the user and workspace, and the taint notice on the next effect | `hooks/tezgah_untrusted.py:2-22`, `hooks/tezgah_untrusted.py::marks` |
|
|
37
|
-
| `tezgah_agents.py` |
|
|
37
|
+
| `tezgah_agents.py` | subagent definitions generated into each host that has an agent surface: user-level for Claude, Codex and omp (written at install), per repo for Cursor and opencode | `hooks/tezgah_agents.py:2-30`, `hooks/tezgah_agents.py::sync_root`, `hooks/tezgah_agents.py::user_agents` |
|
|
38
38
|
| `tezgah_research.py` | the in-repo research workspace and the check that a protocol predates its results | `hooks/tezgah_research.py:2-85`, `hooks/tezgah_research.py::check` |
|
|
39
39
|
| `tezgah_index.py` | the detached graph auto-index worker (flock-guarded, bounded retry) | `hooks/tezgah_index.py:2-18`, `hooks/tezgah_index.py::main` |
|
|
40
40
|
|
|
@@ -18,10 +18,10 @@
|
|
|
18
18
|
"semicolons": 9
|
|
19
19
|
},
|
|
20
20
|
"docs/contract.md": {
|
|
21
|
-
"long_sentences":
|
|
22
|
-
"passive":
|
|
21
|
+
"long_sentences": 42,
|
|
22
|
+
"passive": 25,
|
|
23
23
|
"replace_words": 0,
|
|
24
|
-
"semicolons":
|
|
24
|
+
"semicolons": 30
|
|
25
25
|
},
|
|
26
26
|
"docs/environment.md": {
|
|
27
27
|
"long_sentences": 3,
|
|
@@ -61,9 +61,9 @@
|
|
|
61
61
|
},
|
|
62
62
|
"docs/judge.md": {
|
|
63
63
|
"long_sentences": 22,
|
|
64
|
-
"passive":
|
|
64
|
+
"passive": 14,
|
|
65
65
|
"replace_words": 0,
|
|
66
|
-
"semicolons":
|
|
66
|
+
"semicolons": 13
|
|
67
67
|
},
|
|
68
68
|
"docs/layers.md": {
|
|
69
69
|
"long_sentences": 20,
|
|
@@ -72,13 +72,13 @@
|
|
|
72
72
|
"semicolons": 6
|
|
73
73
|
},
|
|
74
74
|
"docs/models.md": {
|
|
75
|
-
"long_sentences":
|
|
75
|
+
"long_sentences": 18,
|
|
76
76
|
"passive": 8,
|
|
77
77
|
"replace_words": 0,
|
|
78
78
|
"semicolons": 9
|
|
79
79
|
},
|
|
80
80
|
"docs/operations.md": {
|
|
81
|
-
"long_sentences":
|
|
81
|
+
"long_sentences": 66,
|
|
82
82
|
"passive": 21,
|
|
83
83
|
"replace_words": 3,
|
|
84
84
|
"semicolons": 42
|
package/docs/contract.md
CHANGED
|
@@ -38,7 +38,9 @@ rules and all.
|
|
|
38
38
|
The skill suggestion is the one surface a judgement writes rather than a constant.
|
|
39
39
|
The roster reaches a session as an index of host-truncated one-liners, so which
|
|
40
40
|
entry to look at first is the turn's own weak spot; one batched judgement request
|
|
41
|
-
answers it -
|
|
41
|
+
answers it - the session's own CLI, else the Jev model through whichever carrier
|
|
42
|
+
resolves (TypeSafe, OpenRouter System One, Cloudflare Workers AI, a compatible
|
|
43
|
+
endpoint), else the OpenRouter chat fallback
|
|
42
44
|
(`credential()`, `hooks/tezgah_judge.py::credential`) - a Choice across the installed
|
|
43
45
|
skill names plus `none`, and one Noul
|
|
44
46
|
asking whether the turn wants a skill at all - and the winner is appended after
|
|
@@ -158,11 +160,14 @@ never told this; and a ledger with no such row costs no bytes. `verify-off` drop
|
|
|
158
160
|
it with the integrity rule it restates.
|
|
159
161
|
|
|
160
162
|
The on-demand contract also carries detail the core has no budget for. One is
|
|
161
|
-
the review rule
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
163
|
+
the review rule. A plan gets one review over its whole diff, in at most two
|
|
164
|
+
rounds. Round one fans out one reviewer per dimension. Only a confirmed critical
|
|
165
|
+
or major finding opens round two, and `tezgah-task review` refuses a third. A
|
|
166
|
+
change with an executable oracle - a checker, a gate rule, a parser - spends the
|
|
167
|
+
same budget. Its round one is four direction-led reviewers, and each claim
|
|
168
|
+
becomes a fixture that runs before it counts. That population is a tentative
|
|
169
|
+
default, measured on two research lines with a written flip
|
|
170
|
+
(`skills/tezgah-contract/SKILL.md:519-550`).
|
|
166
171
|
|
|
167
172
|
## Composition order of the always-on core
|
|
168
173
|
|
|
@@ -272,12 +277,17 @@ sentence of the rule.
|
|
|
272
277
|
`PROMPT_HINTS` (`hooks/tezgah_context.py::PROMPT_HINTS`) is one compiled pattern per key
|
|
273
278
|
— `spec`, `consult`, `research`, `product`, `graph`, `fanout` — and `classify_prompt()` returns the keys
|
|
274
279
|
a prompt matches (`hooks/tezgah_context.py::classify_prompt`). `fanout` arms on the
|
|
275
|
-
shape of the request, not a word. Two
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
280
|
+
shape of the request, not a word. Two list entries on their own lines arm it.
|
|
281
|
+
An entry starts with `1.`, `1)`, `1 -`, `1:`, `a)` or a bullet. An inline
|
|
282
|
+
`1. ... 2. ...` arms it too. So do three files named as one list, "First, ...
|
|
283
|
+
Second, ..." and three imperative clauses in one sentence. An explicit
|
|
284
|
+
subagent, parallel or worktree ask arms it as well. The owner no longer has to
|
|
285
|
+
say "do it with subagents, in parallel". On that turn only, the matching
|
|
286
|
+
paragraphs are appended after the reminder
|
|
279
287
|
(`hooks/tezgah_context.py::context_for`); a session that never asks such a question
|
|
280
|
-
pays the one-line pointer instead.
|
|
288
|
+
pays the one-line pointer instead. A later match in the same session pays one
|
|
289
|
+
line. The fan-out line still names the action
|
|
290
|
+
(`hooks/tezgah_context.py::armed_again`). The patterns carry Turkish stems because the
|
|
281
291
|
user writes Turkish, and a plain prompt arms nothing. The same prompt arms the
|
|
282
292
|
same rules on every host (`tests/test_context.py::ArmingConformance.test_all_hosts_arm_the_same_rules`) and each advisory
|
|
283
293
|
rule keeps a pointer line in the always-on text (`tests/test_context.py::ArmingConformance.test_every_advisory_rule_has_an_always_on_pointer`).
|