@r1z4x/tezgah 1.0.0 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +92 -1
- package/MANIFEST +10 -0
- package/bin/tezgah-agents +10 -1
- package/bin/tezgah-docs +1 -1
- package/bin/tezgah-doctor +11 -0
- package/bin/tezgah-research +31 -14
- package/bin/tezgah-setup +154 -29
- package/bin/tezgah-skill +59 -0
- package/bin/tezgah-skill.py +59 -0
- package/bin/tezgah-status +62 -1
- package/bin/tezgah-taste +504 -5
- package/bin/tezgah-triage +5 -3
- package/docs/README.md +1 -0
- package/docs/architecture.md +5 -2
- package/docs/citations-baseline.json +1 -1
- package/docs/clarity-baseline.json +9 -9
- package/docs/contract.md +68 -26
- package/docs/environment.md +6 -3
- package/docs/evidence.md +7 -0
- package/docs/gate.md +13 -7
- package/docs/glossary.md +1 -1
- package/docs/hosts.md +48 -1
- package/docs/index.json +34 -3
- package/docs/judge.md +75 -60
- package/docs/models.md +14 -8
- package/docs/operations.md +80 -25
- package/docs/orca.md +126 -0
- package/docs/research.md +38 -3
- package/docs/skills.md +74 -0
- package/docs/status-line.md +3 -1
- package/docs/testing.md +33 -21
- package/docs/vision.md +12 -8
- package/hooks/projects-posttooluse.py +7 -0
- package/hooks/projects-pretooluse.py +17 -5
- package/hooks/tezgah_agents.py +135 -13
- package/hooks/tezgah_components.py +3 -0
- package/hooks/tezgah_context.py +84 -22
- package/hooks/tezgah_gate.py +133 -17
- package/hooks/tezgah_integrity.py +60 -13
- package/hooks/tezgah_judge.py +288 -66
- package/hooks/tezgah_models.py +41 -5
- package/hooks/tezgah_orca.py +144 -0
- package/hooks/tezgah_paths.py +40 -6
- package/hooks/tezgah_policy.py +109 -35
- package/hooks/tezgah_research.py +104 -8
- package/hooks/tezgah_skill_pick.py +459 -8
- package/hooks/tezgah_stack.py +382 -0
- package/hooks/tezgah_stopspec.py +2 -2
- package/hooks/tezgah_taste.py +35 -9
- package/hooks/tezgah_taste_ledger.py +628 -0
- package/hosts/codex/hook.py +12 -1
- package/hosts/cursor/hook.py +12 -4
- package/hosts/omp/hook.py +18 -6
- package/hosts/omp/tezgah-hook.ts.in +20 -7
- package/hosts/opencode/plugins/tezgah.js +28 -15
- package/output-styles/tezgah.md +34 -19
- package/package.json +1 -1
- package/skills/i-have-adhd/SKILL.md +2 -2
- package/skills/research/SKILL.md +60 -17
- package/skills/tezgah-contract/SKILL.md +55 -13
package/bin/tezgah-taste
CHANGED
|
@@ -1,9 +1,17 @@
|
|
|
1
1
|
#!/usr/bin/env python3
|
|
2
|
-
"""tezgah-taste - mine and
|
|
2
|
+
"""tezgah-taste - mine, measure, learn and apply the coding-taste signals.
|
|
3
3
|
|
|
4
4
|
tezgah-taste mine [--host omp|claude|all] [--root DIR] [--stats] [--out FILE]
|
|
5
5
|
tezgah-taste measure --samples FILE --labels FILE [--model M] [--json]
|
|
6
6
|
tezgah-taste rate --in MINED [--limit N] [--json]
|
|
7
|
+
tezgah-taste learn [--repo DIR] [--from-transcripts] [--host omp|claude|all]
|
|
8
|
+
[--limit N] [--json]
|
|
9
|
+
tezgah-taste list [--repo DIR] [--all] [--json]
|
|
10
|
+
tezgah-taste show|accept|reject ID [--repo DIR]
|
|
11
|
+
tezgah-taste edit ID --text TEXT [--repo DIR]
|
|
12
|
+
tezgah-taste export [--repo DIR]
|
|
13
|
+
tezgah-taste label [ID LABEL] [--n N] [--repo DIR]
|
|
14
|
+
tezgah-taste calibrate|gate [--repo DIR] [--json]
|
|
7
15
|
|
|
8
16
|
`mine` reads the session transcripts on this machine - omp's top-level session
|
|
9
17
|
files under `~/.omp/agent/sessions/*/` (a nested file is a subagent session) and
|
|
@@ -25,13 +33,36 @@ writing-turn prompt, per repo and overall. Both send the sample text to the
|
|
|
25
33
|
judge's provider (`hooks/tezgah_judge.py` names it); with no credential or
|
|
26
34
|
`judge-off` set they send nothing and exit 2.
|
|
27
35
|
|
|
36
|
+
`learn` is taste-1's pipeline on tezgah (research line taste-architecture,
|
|
37
|
+
design T1P). Each prompt after a writing turn - from the capture rows in
|
|
38
|
+
`<repo>/.tezgah/taste/signals.jsonl`, or with `--from-transcripts` from this
|
|
39
|
+
repository's omp and Claude transcripts - becomes one structured signal built by
|
|
40
|
+
code: the prompt, the files written, the agent's edits and the user's own edit
|
|
41
|
+
between two agent writes. One typed decision per signal, required from TypeSafe
|
|
42
|
+
(`only=("typesafe",)`): preference, defect or none; the category; the scope; and
|
|
43
|
+
how it relates to each learning held. A decision any other provider gave is
|
|
44
|
+
recorded `unverified` and never applied. A defect goes to `defects.jsonl`, never
|
|
45
|
+
into the ledger. The ledger itself is `hooks/tezgah_taste_ledger.py`. A
|
|
46
|
+
learning that becomes active gets one readable line from a generative provider
|
|
47
|
+
(a `text` question; the user's own words when none answers).
|
|
48
|
+
|
|
49
|
+
`list`, `show`, `accept`, `reject` and `edit` are the user's control; `export`
|
|
50
|
+
writes the accepted learnings into AGENTS.md. `label` draws an independent
|
|
51
|
+
random sample of decided signals for the user to label, `calibrate` prints the
|
|
52
|
+
Clopper-Pearson bound a rule needs, and `gate` runs the pooled before/after test
|
|
53
|
+
that stops injection when the corrections do not fall.
|
|
54
|
+
|
|
28
55
|
Exit codes: 0 done, 2 misuse or no judge.
|
|
29
56
|
"""
|
|
30
57
|
import argparse
|
|
31
58
|
import datetime
|
|
59
|
+
import difflib
|
|
32
60
|
import glob
|
|
61
|
+
import hashlib
|
|
33
62
|
import json
|
|
34
63
|
import os
|
|
64
|
+
import random
|
|
65
|
+
import subprocess
|
|
35
66
|
import sys
|
|
36
67
|
|
|
37
68
|
HERE = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
|
|
@@ -39,6 +70,7 @@ sys.path.insert(0, os.path.join(HERE, "hooks"))
|
|
|
39
70
|
import tezgah_integrity as ti # noqa: E402
|
|
40
71
|
import tezgah_judge # noqa: E402
|
|
41
72
|
import tezgah_models # noqa: E402
|
|
73
|
+
import tezgah_taste_ledger as tl # noqa: E402
|
|
42
74
|
|
|
43
75
|
WRITE_TOOLS = {"omp": {"edit", "write"},
|
|
44
76
|
"claude": {"Edit", "Write", "MultiEdit", "NotebookEdit"}}
|
|
@@ -282,7 +314,7 @@ def read_jsonl(path):
|
|
|
282
314
|
raise SystemExit(misuse("cannot read %s: %s" % (path, exc)))
|
|
283
315
|
|
|
284
316
|
|
|
285
|
-
def classify(texts, model):
|
|
317
|
+
def classify(texts, model, only=None):
|
|
286
318
|
"""One label per text (None when the judge gave none) and the usage.
|
|
287
319
|
|
|
288
320
|
The rule rides the state, so each request pays for it once and the
|
|
@@ -294,7 +326,7 @@ def classify(texts, model):
|
|
|
294
326
|
"instructions": "Label this prompt under the rule "
|
|
295
327
|
"in the state:\n" + ti.redact(text)}
|
|
296
328
|
for i, text in enumerate(chunk)}
|
|
297
|
-
result = tezgah_judge.ask(RULE, questions, model=model, timeout=120)
|
|
329
|
+
result = tezgah_judge.ask(RULE, questions, model=model, timeout=120, only=only)
|
|
298
330
|
if result is not None:
|
|
299
331
|
tokens_in += result["usage"]["input_tokens"]
|
|
300
332
|
tokens_out += result["usage"]["output_tokens"]
|
|
@@ -336,7 +368,8 @@ def cmd_measure(args):
|
|
|
336
368
|
return misuse("no sample joins a label on (set, n)")
|
|
337
369
|
if not tezgah_judge.available():
|
|
338
370
|
return misuse("no judge: no credential resolves or judge-off is set; nothing sent")
|
|
339
|
-
got, usage = classify([text for text, _gold in pairs], args.model
|
|
371
|
+
got, usage = classify([text for text, _gold in pairs], args.model,
|
|
372
|
+
only=("typesafe",) if args.typed else None)
|
|
340
373
|
confusion = {g: {p: 0 for p in CLASSES} for g in CLASSES}
|
|
341
374
|
unjudged = 0
|
|
342
375
|
for (_text, gold), pred in zip(pairs, got):
|
|
@@ -396,6 +429,430 @@ def cmd_rate(args):
|
|
|
396
429
|
print(usage_line(usage))
|
|
397
430
|
return 0
|
|
398
431
|
|
|
432
|
+
# ------------------------------------------------------------ learn, apply ----
|
|
433
|
+
|
|
434
|
+
SCOPE_CRITERIA = {
|
|
435
|
+
"path": "only files under the directory the agent wrote",
|
|
436
|
+
"language": "files of the language or extension the agent wrote",
|
|
437
|
+
"repository": "everything in this repository",
|
|
438
|
+
"user": "every repository this user works in"}
|
|
439
|
+
RELATION_CRITERIA = {
|
|
440
|
+
"supports": "the prompt asks for the same preference as this learning",
|
|
441
|
+
"contradicts": "the prompt asks for the opposite of this learning",
|
|
442
|
+
"narrows": "the prompt keeps this learning but makes an exception for a narrower place",
|
|
443
|
+
"unrelated": "the prompt is about something else"}
|
|
444
|
+
CATEGORY_CRITERIA = {c: c.replace("-", " ") for c in tl.CATEGORIES}
|
|
445
|
+
HELD_MAX = 30
|
|
446
|
+
DIFF_MAX = 1500
|
|
447
|
+
|
|
448
|
+
|
|
449
|
+
def repo_root(path):
|
|
450
|
+
out = subprocess.run(["git", "-C", path, "rev-parse", "--show-toplevel"],
|
|
451
|
+
capture_output=True, text=True)
|
|
452
|
+
return out.stdout.strip() if out.returncode == 0 and out.stdout.strip() else None
|
|
453
|
+
|
|
454
|
+
|
|
455
|
+
def sig_id(*parts):
|
|
456
|
+
return hashlib.sha1("\0".join(str(p) for p in parts).encode()).hexdigest()[:12]
|
|
457
|
+
|
|
458
|
+
|
|
459
|
+
def iso_day(ts):
|
|
460
|
+
"""A capture row's ISO timestamp as days since the epoch, or None."""
|
|
461
|
+
try:
|
|
462
|
+
return datetime.datetime.fromisoformat(str(ts).replace("Z", "+00:00")) \
|
|
463
|
+
.timestamp() / 86400.0
|
|
464
|
+
except ValueError:
|
|
465
|
+
return None
|
|
466
|
+
|
|
467
|
+
|
|
468
|
+
def _snapshots():
|
|
469
|
+
"""{absolute path: [(ts, id, after)]} over the snapshot store, oldest first."""
|
|
470
|
+
import tezgah_snapshot as ts
|
|
471
|
+
out = {}
|
|
472
|
+
for d in ts._dirs():
|
|
473
|
+
try:
|
|
474
|
+
with open(os.path.join(d, "meta.json"), encoding="utf-8") as fh:
|
|
475
|
+
meta = json.load(fh)
|
|
476
|
+
except (OSError, ValueError):
|
|
477
|
+
continue
|
|
478
|
+
if isinstance(meta, dict) and meta.get("path"):
|
|
479
|
+
out.setdefault(meta["path"], []).append(
|
|
480
|
+
(int(meta.get("ts") or 0), os.path.basename(d), bool(meta.get("after"))))
|
|
481
|
+
for rows in out.values():
|
|
482
|
+
rows.sort()
|
|
483
|
+
return out
|
|
484
|
+
|
|
485
|
+
|
|
486
|
+
def _read_blob(sid):
|
|
487
|
+
import tezgah_snapshot as ts
|
|
488
|
+
try:
|
|
489
|
+
with open(ts._blob(sid), "rb") as fh:
|
|
490
|
+
return fh.read().decode("utf-8", "replace")
|
|
491
|
+
except OSError:
|
|
492
|
+
return None
|
|
493
|
+
|
|
494
|
+
|
|
495
|
+
def user_diff(repo, rel, after_sid, snaps):
|
|
496
|
+
"""The user's own change to `rel` after the agent's write: the after-blob
|
|
497
|
+
against the pre-write bytes of the next agent write to it, or against the
|
|
498
|
+
file on disk when none came. "" when nothing changed in between."""
|
|
499
|
+
after = _read_blob(after_sid)
|
|
500
|
+
if after is None:
|
|
501
|
+
return ""
|
|
502
|
+
apath = os.path.realpath(os.path.join(repo, rel))
|
|
503
|
+
rows = snaps.get(apath) or []
|
|
504
|
+
mine = next((t for t, sid, _a in rows if sid == after_sid), None)
|
|
505
|
+
later = next((sid for t, sid, a in rows if mine is not None and t >= mine
|
|
506
|
+
and not a and sid != after_sid), None)
|
|
507
|
+
if later:
|
|
508
|
+
now = _read_blob(later)
|
|
509
|
+
else:
|
|
510
|
+
try:
|
|
511
|
+
with open(apath, encoding="utf-8", errors="replace") as fh:
|
|
512
|
+
now = fh.read()
|
|
513
|
+
except OSError:
|
|
514
|
+
return ""
|
|
515
|
+
if now is None or now == after:
|
|
516
|
+
return ""
|
|
517
|
+
diff = "".join(difflib.unified_diff(after.splitlines(True), now.splitlines(True),
|
|
518
|
+
"agent/" + rel, "user/" + rel, n=1))
|
|
519
|
+
return ti.redact(diff)[:DIFF_MAX]
|
|
520
|
+
|
|
521
|
+
|
|
522
|
+
def capture_signals(repo):
|
|
523
|
+
"""One signal per prompt that followed a writing turn, from the capture rows."""
|
|
524
|
+
snaps, turns, out = None, {}, []
|
|
525
|
+
for row in tl.rows(os.path.join(tl.store_dir(repo), "signals.jsonl")):
|
|
526
|
+
session = row.get("session") or ""
|
|
527
|
+
turn = turns.setdefault(session, {"edits": [], "after": []})
|
|
528
|
+
if row.get("kind") == "edit":
|
|
529
|
+
turn["edits"].append(row)
|
|
530
|
+
elif row.get("kind") == "after":
|
|
531
|
+
turn["after"].append(row)
|
|
532
|
+
elif row.get("kind") == "prompt":
|
|
533
|
+
if turn["edits"]:
|
|
534
|
+
if snaps is None:
|
|
535
|
+
snaps = _snapshots()
|
|
536
|
+
diffs = [d for d in (user_diff(repo, a.get("path"), a.get("snapshot"), snaps)
|
|
537
|
+
for a in turn["after"] if a.get("path")) if d]
|
|
538
|
+
out.append({
|
|
539
|
+
"id": sig_id(session, row.get("ts"), row.get("text")),
|
|
540
|
+
"at": iso_day(row.get("ts")),
|
|
541
|
+
"session": session, "text": row.get("text") or "",
|
|
542
|
+
"paths": sorted({e.get("path") for e in turn["edits"] if e.get("path")}),
|
|
543
|
+
"edits": [(e.get("path"), (e.get("old") or "")[:300],
|
|
544
|
+
(e.get("new") or "")[:300]) for e in turn["edits"][-4:]],
|
|
545
|
+
"diffs": diffs})
|
|
546
|
+
turns[session] = {"edits": [], "after": []}
|
|
547
|
+
return out
|
|
548
|
+
|
|
549
|
+
|
|
550
|
+
def _call_path(args):
|
|
551
|
+
for key in ("path", "file_path", "notebook_path"):
|
|
552
|
+
if isinstance(args.get(key), str):
|
|
553
|
+
return args[key]
|
|
554
|
+
return None
|
|
555
|
+
|
|
556
|
+
|
|
557
|
+
def transcript_signals(repo, host):
|
|
558
|
+
"""The same signals from this repository's top-level host transcripts: the
|
|
559
|
+
prompt and the files written; no edit text or user diff (the history
|
|
560
|
+
bootstrap, Command Code's `/learn-taste`)."""
|
|
561
|
+
out = []
|
|
562
|
+
for name in (("omp", "claude") if host == "all" else (host,)):
|
|
563
|
+
for path, cwd in sessions(name, os.path.dirname(repo)):
|
|
564
|
+
if cwd != repo and not cwd.startswith(repo + os.sep):
|
|
565
|
+
continue
|
|
566
|
+
session = session_id(name, path)[0] or os.path.basename(path)
|
|
567
|
+
written, n, k, at = [], 0, 0, None
|
|
568
|
+
for event in HOSTS[name][2](path, calls=True):
|
|
569
|
+
if event[0] in ("call", "reply") and isinstance(event[-1], (int, float)):
|
|
570
|
+
at = event[-1] / 86400.0
|
|
571
|
+
if event[0] == "call" and event[1] in WRITE_TOOLS[name]:
|
|
572
|
+
target = _call_path(event[2])
|
|
573
|
+
if target:
|
|
574
|
+
rel = os.path.relpath(os.path.join(cwd, target), repo)
|
|
575
|
+
if not rel.startswith(".."):
|
|
576
|
+
written.append(rel)
|
|
577
|
+
elif event[0] == "write":
|
|
578
|
+
n += 1
|
|
579
|
+
elif event[0] == "user":
|
|
580
|
+
if n and not harness_prompt(event[1]):
|
|
581
|
+
text = ti.redact(event[1])[:CUT]
|
|
582
|
+
# keyed inside its own session, so another session's
|
|
583
|
+
# new turns never shift this id and re-decide it
|
|
584
|
+
k += 1
|
|
585
|
+
out.append({"id": sig_id(name, session, k, text), "at": at,
|
|
586
|
+
"session": session, "text": text,
|
|
587
|
+
"paths": sorted(set(written)), "edits": [], "diffs": []})
|
|
588
|
+
written, n = [], 0
|
|
589
|
+
return out
|
|
590
|
+
|
|
591
|
+
|
|
592
|
+
def held(ledgers):
|
|
593
|
+
"""The learnings a new signal is compared against: every one not retired,
|
|
594
|
+
the most supported first, capped so one call stays one call."""
|
|
595
|
+
alive = [v for v in tl.every(ledgers).values() if v["state"] != "retired"]
|
|
596
|
+
alive.sort(key=lambda v: -v.get("r", 0.0))
|
|
597
|
+
return alive[:HELD_MAX]
|
|
598
|
+
|
|
599
|
+
|
|
600
|
+
def decision_request(sig, learnings):
|
|
601
|
+
state = ["A user prompt that followed a turn in which a coding agent wrote files.",
|
|
602
|
+
"Prompt: " + ti.redact(sig["text"])[:CUT]]
|
|
603
|
+
if sig["paths"]:
|
|
604
|
+
state.append("Files the agent wrote: " + ", ".join(sig["paths"][:12]))
|
|
605
|
+
for path, old, new in sig.get("edits") or ():
|
|
606
|
+
state.append("Agent edit to %s:\n- %s\n+ %s" % (path, ti.redact(old), ti.redact(new)))
|
|
607
|
+
for diff in sig.get("diffs") or ():
|
|
608
|
+
state.append("The user's own edit after the agent's write:\n" + diff)
|
|
609
|
+
questions = {
|
|
610
|
+
"kind": {"type": "choice", "instructions": RULE, "criteria": CRITERIA},
|
|
611
|
+
"category": {"type": "choice", "criteria": CATEGORY_CRITERIA,
|
|
612
|
+
"instructions": "If the prompt states a coding preference, which "
|
|
613
|
+
"category does it belong to?"},
|
|
614
|
+
"scope": {"type": "choice", "criteria": SCOPE_CRITERIA,
|
|
615
|
+
"instructions": "Where should this preference apply, judging only from "
|
|
616
|
+
"what the prompt says?"}}
|
|
617
|
+
for learning in learnings:
|
|
618
|
+
questions["rel_" + learning["id"]] = {
|
|
619
|
+
"type": "choice", "criteria": RELATION_CRITERIA,
|
|
620
|
+
"instructions": "How does the prompt relate to this learning (%s, %s): %s"
|
|
621
|
+
% (learning["category"], learning["scope"],
|
|
622
|
+
" ".join(learning["text"].split())[:300])}
|
|
623
|
+
return "\n\n".join(state), questions
|
|
624
|
+
|
|
625
|
+
|
|
626
|
+
def picked(result, qid, default):
|
|
627
|
+
choice = tezgah_judge.choice(result, qid)
|
|
628
|
+
if choice is None:
|
|
629
|
+
return default, 0.0
|
|
630
|
+
p = tezgah_judge.noul(result, qid, choice)
|
|
631
|
+
return choice, (p if p is not None else 0.5)
|
|
632
|
+
|
|
633
|
+
|
|
634
|
+
def decide(sig, learnings, fallback=True):
|
|
635
|
+
"""(decision, result): the typed decision, TypeSafe required; when it cannot
|
|
636
|
+
answer, the seam's own providers, recorded unverified - only on a signal's
|
|
637
|
+
first try, since an unverified answer is never applied and a rerun would pay
|
|
638
|
+
for it again."""
|
|
639
|
+
state, questions = decision_request(sig, learnings)
|
|
640
|
+
result = tezgah_judge.ask(state, questions, only=("typesafe",), timeout=60)
|
|
641
|
+
if result is None and fallback:
|
|
642
|
+
result = tezgah_judge.ask(state, questions, timeout=120)
|
|
643
|
+
if result is None:
|
|
644
|
+
return None, None
|
|
645
|
+
dec = {"kind": picked(result, "kind", "none"),
|
|
646
|
+
"category": picked(result, "category", ""),
|
|
647
|
+
"scope": picked(result, "scope", "repository"),
|
|
648
|
+
"relations": {lr["id"]: picked(result, "rel_" + lr["id"], "unrelated")
|
|
649
|
+
for lr in learnings},
|
|
650
|
+
"verified": result.get("provider") == "typesafe"}
|
|
651
|
+
return dec, result
|
|
652
|
+
|
|
653
|
+
|
|
654
|
+
def write_line(learning):
|
|
655
|
+
"""One readable line for a learning that became active, from a generative
|
|
656
|
+
provider; None when none answers (the user's own words stay)."""
|
|
657
|
+
evidence = " ".join(learning["text"].split())[:600]
|
|
658
|
+
result = tezgah_judge.ask(
|
|
659
|
+
"A user corrected a coding agent with: " + evidence,
|
|
660
|
+
{"line": {"type": "text", "instructions":
|
|
661
|
+
"Write the coding preference this correction states as one imperative "
|
|
662
|
+
"line for a coding agent, under 20 words. Use only what the correction "
|
|
663
|
+
"says; add no preference it does not state."}},
|
|
664
|
+
timeout=120)
|
|
665
|
+
return tezgah_judge.text(result, "line")
|
|
666
|
+
|
|
667
|
+
|
|
668
|
+
def store_or_exit(args):
|
|
669
|
+
repo = repo_root(os.path.abspath(args.repo or os.getcwd()))
|
|
670
|
+
if not repo:
|
|
671
|
+
raise SystemExit(misuse("not inside a git repository"))
|
|
672
|
+
return repo
|
|
673
|
+
|
|
674
|
+
|
|
675
|
+
def learn_refusal(repo):
|
|
676
|
+
"""Why `learn` sends and writes nothing here, or None - read before the
|
|
677
|
+
store lock, so a refused run leaves no lock file behind."""
|
|
678
|
+
if tezgah_judge.tp.off("judge-off") or not (
|
|
679
|
+
tezgah_judge.available() or tezgah_judge.named(("typesafe",))):
|
|
680
|
+
return "no judge: no credential resolves or judge-off is set; nothing sent"
|
|
681
|
+
import tezgah_context
|
|
682
|
+
if ".no-taste" in tezgah_context.repo_marks(repo)[1]:
|
|
683
|
+
return ".no-taste marks this repository; nothing learned"
|
|
684
|
+
return None
|
|
685
|
+
|
|
686
|
+
|
|
687
|
+
def cmd_learn(args):
|
|
688
|
+
repo = store_or_exit(args)
|
|
689
|
+
signals = (transcript_signals(repo, args.host) if args.from_transcripts
|
|
690
|
+
else capture_signals(repo))
|
|
691
|
+
decided = {}
|
|
692
|
+
for r in tl.rows(os.path.join(tl.store_dir(repo), "decisions.jsonl")):
|
|
693
|
+
decided[r.get("id")] = decided.get(r.get("id")) or r.get("provider") == "typesafe"
|
|
694
|
+
# a typed decision is final; an unverified one is retried, TypeSafe only
|
|
695
|
+
todo = [s for s in signals if not decided.get(s["id"])]
|
|
696
|
+
if args.limit is not None:
|
|
697
|
+
todo = todo[:args.limit]
|
|
698
|
+
ledgers = tl.load(repo)
|
|
699
|
+
before = {k for k, v in tl.every(ledgers).items() if v["state"] == "active"}
|
|
700
|
+
counts = {"signals": len(todo), "typesafe": 0, "unverified": 0, "unanswered": 0,
|
|
701
|
+
"preference": 0, "defect": 0, "none": 0}
|
|
702
|
+
tokens = [0, 0]
|
|
703
|
+
for sig in todo:
|
|
704
|
+
dec, result = decide(sig, held(ledgers), fallback=sig["id"] not in decided)
|
|
705
|
+
if dec is None:
|
|
706
|
+
counts["unanswered"] += 1
|
|
707
|
+
continue
|
|
708
|
+
tokens[0] += result["usage"]["input_tokens"]
|
|
709
|
+
tokens[1] += result["usage"]["output_tokens"]
|
|
710
|
+
day = tl.now_day()
|
|
711
|
+
tl.append(os.path.join(tl.store_dir(repo), "decisions.jsonl"), {
|
|
712
|
+
"id": sig["id"], "day": day, "at": sig.get("at"), "session": sig["session"],
|
|
713
|
+
"provider": result.get("provider"), "model": result.get("model"),
|
|
714
|
+
"verified": dec["verified"], "kind": dec["kind"][0], "p": dec["kind"][1],
|
|
715
|
+
"category": dec["category"][0], "scope": dec["scope"][0],
|
|
716
|
+
"relations": {k: v[0] for k, v in dec["relations"].items() if v[0] != "unrelated"},
|
|
717
|
+
"text": ti.redact(sig["text"])[:300], "paths": sig["paths"][:8]}, strict=True)
|
|
718
|
+
if not dec["verified"]:
|
|
719
|
+
counts["unverified"] += 1
|
|
720
|
+
continue
|
|
721
|
+
counts["typesafe"] += 1
|
|
722
|
+
counts[dec["kind"][0] if dec["kind"][0] in CLASSES else "none"] += 1
|
|
723
|
+
if dec["kind"][0] == "defect":
|
|
724
|
+
tl.append(os.path.join(tl.store_dir(repo), "defects.jsonl"),
|
|
725
|
+
{"id": sig["id"], "day": day, "session": sig["session"],
|
|
726
|
+
"text": ti.redact(sig["text"])[:600], "paths": sig["paths"][:8]},
|
|
727
|
+
strict=True)
|
|
728
|
+
tl.apply(ledgers, sig, dec, day, applied=tl.injected(repo, sig["session"]))
|
|
729
|
+
for lid, learning in tl.every(ledgers).items():
|
|
730
|
+
if learning["state"] == "active" and lid not in before \
|
|
731
|
+
and not learning.get("written") and not learning.get("edited"):
|
|
732
|
+
line = write_line(learning)
|
|
733
|
+
if line:
|
|
734
|
+
learning.update(text=" ".join(line.split())[:400], written=True)
|
|
735
|
+
tl.save(repo, ledgers)
|
|
736
|
+
tl.markdown(repo)
|
|
737
|
+
states = {}
|
|
738
|
+
for learning in tl.every(ledgers).values():
|
|
739
|
+
states[learning["state"]] = states.get(learning["state"], 0) + 1
|
|
740
|
+
report = dict(counts, states=states, input_tokens=tokens[0], output_tokens=tokens[1],
|
|
741
|
+
cost_usd=cost(tezgah_judge.MODEL, tokens[0], tokens[1]))
|
|
742
|
+
if args.json:
|
|
743
|
+
print(json.dumps(report, indent=2))
|
|
744
|
+
return 0
|
|
745
|
+
print("signals %d: typesafe %d, unverified %d, unanswered %d" % (
|
|
746
|
+
counts["signals"], counts["typesafe"], counts["unverified"], counts["unanswered"]))
|
|
747
|
+
print("decided: preference %d, defect %d, none %d" % (
|
|
748
|
+
counts["preference"], counts["defect"], counts["none"]))
|
|
749
|
+
print("ledger: " + (", ".join("%s %d" % kv for kv in sorted(states.items())) or "empty"))
|
|
750
|
+
print("typed decisions: %d input tokens, $%s" % (tokens[0], report["cost_usd"]))
|
|
751
|
+
return 0
|
|
752
|
+
|
|
753
|
+
|
|
754
|
+
def cmd_list(args):
|
|
755
|
+
repo = store_or_exit(args)
|
|
756
|
+
ledgers = tl.load(repo)
|
|
757
|
+
modes = {m[1]["id"]: m[0] for m in tl.usable(repo)}
|
|
758
|
+
rows = [v for v in tl.every(ledgers).values() if args.all or v["state"] != "retired"]
|
|
759
|
+
rows.sort(key=lambda v: (v["state"] != "active", -tl.confidence(v)))
|
|
760
|
+
if args.json:
|
|
761
|
+
print(json.dumps([dict(v, confidence=round(tl.confidence(v), 4),
|
|
762
|
+
mode=modes.get(v["id"])) for v in rows], indent=2))
|
|
763
|
+
return 0
|
|
764
|
+
if not rows:
|
|
765
|
+
print("no learnings")
|
|
766
|
+
return 0
|
|
767
|
+
for v in rows:
|
|
768
|
+
print("%-6s %-11s %-4s %.2f %-10s %-17s %s" % (
|
|
769
|
+
v["id"], v["state"], modes.get(v["id"], "-"), tl.confidence(v),
|
|
770
|
+
v["scope"] + ("" if v["scope"] in ("repository", "user") else ":" + v["where"]),
|
|
771
|
+
v["category"], " ".join(v["text"].split())[:80]))
|
|
772
|
+
return 0
|
|
773
|
+
|
|
774
|
+
|
|
775
|
+
def cmd_one(args):
|
|
776
|
+
repo = store_or_exit(args)
|
|
777
|
+
ledgers = tl.load(repo)
|
|
778
|
+
if args.cmd == "show":
|
|
779
|
+
learning = tl.every(ledgers).get(args.id)
|
|
780
|
+
if learning is None:
|
|
781
|
+
return misuse("no learning %s" % args.id)
|
|
782
|
+
print(json.dumps(dict(learning, confidence=round(tl.confidence(learning), 4)),
|
|
783
|
+
indent=2, ensure_ascii=False))
|
|
784
|
+
return 0
|
|
785
|
+
if not tl.decide_user(ledgers, args.id, args.cmd, tl.now_day(),
|
|
786
|
+
text=getattr(args, "text", None)):
|
|
787
|
+
return misuse("no learning %s" % args.id)
|
|
788
|
+
tl.save(repo, ledgers)
|
|
789
|
+
tl.markdown(repo)
|
|
790
|
+
print("%s %s" % (args.id, {"accept": "accepted", "reject": "rejected",
|
|
791
|
+
"edit": "edited"}[args.cmd]))
|
|
792
|
+
return 0
|
|
793
|
+
|
|
794
|
+
|
|
795
|
+
def cmd_export(args):
|
|
796
|
+
repo = store_or_exit(args)
|
|
797
|
+
print("AGENTS.md: %d accepted learning(s)" % tl.export_agents(repo))
|
|
798
|
+
return 0
|
|
799
|
+
|
|
800
|
+
|
|
801
|
+
def cmd_label(args):
|
|
802
|
+
"""With ID and LABEL, record the user's label; without, print a random
|
|
803
|
+
sample of unlabelled typed decisions - drawn without regard to what was
|
|
804
|
+
injected, so the calibration bound rests on an unselected sample."""
|
|
805
|
+
repo = store_or_exit(args)
|
|
806
|
+
store = tl.store_dir(repo)
|
|
807
|
+
decided = {r["id"]: r for r in tl.rows(os.path.join(store, "decisions.jsonl"))
|
|
808
|
+
if r.get("provider") == "typesafe" and r.get("id")}
|
|
809
|
+
labelled = {r.get("id") for r in tl.rows(os.path.join(store, "labels.jsonl"))}
|
|
810
|
+
if args.id:
|
|
811
|
+
if args.label not in CLASSES:
|
|
812
|
+
return misuse("label is one of %s" % ", ".join(CLASSES))
|
|
813
|
+
row = decided.get(args.id)
|
|
814
|
+
if row is None:
|
|
815
|
+
return misuse("no typed decision %s" % args.id)
|
|
816
|
+
tl.append(os.path.join(store, "labels.jsonl"),
|
|
817
|
+
{"id": args.id, "label": args.label, "decided": row.get("kind"),
|
|
818
|
+
"provider": row.get("provider"), "day": tl.now_day()}, strict=True)
|
|
819
|
+
print("%s labelled %s (decided %s)" % (args.id, args.label, row.get("kind")))
|
|
820
|
+
return 0
|
|
821
|
+
pool = sorted(i for i in decided if i not in labelled)
|
|
822
|
+
for i in random.Random().sample(pool, min(args.n, len(pool))):
|
|
823
|
+
print("%s %s" % (i, " ".join(decided[i].get("text", "").split())[:200]))
|
|
824
|
+
print("label each: tezgah-taste label <id> preference|defect|none")
|
|
825
|
+
return 0
|
|
826
|
+
|
|
827
|
+
|
|
828
|
+
def cmd_calibrate(args):
|
|
829
|
+
repo = store_or_exit(args)
|
|
830
|
+
x, n, bound = tl.calibration(repo)
|
|
831
|
+
report = {"agree": x, "labelled_preference_decisions": n, "bound": round(bound, 4),
|
|
832
|
+
"rules_allowed": bound >= tl.BOUND_AT}
|
|
833
|
+
if args.json:
|
|
834
|
+
print(json.dumps(report, indent=2))
|
|
835
|
+
else:
|
|
836
|
+
print("preference precision %d/%d, Clopper-Pearson 95%% lower bound %.4f: %s" % (
|
|
837
|
+
x, n, bound, "rules allowed" if report["rules_allowed"]
|
|
838
|
+
else "hints only (a rule needs >= %.1f)" % tl.BOUND_AT))
|
|
839
|
+
return 0
|
|
840
|
+
|
|
841
|
+
|
|
842
|
+
def cmd_gate(args):
|
|
843
|
+
repo = store_or_exit(args)
|
|
844
|
+
report = tl.gate(repo)
|
|
845
|
+
if args.json:
|
|
846
|
+
print(json.dumps(report, indent=2))
|
|
847
|
+
return 0
|
|
848
|
+
b, a = report["before"], report["after"]
|
|
849
|
+
print("before: %d/%d preference per writing turn; after: %d/%d; one-sided p %s; "
|
|
850
|
+
"need %d per arm; injection %s" % (
|
|
851
|
+
b["preference"], b["turns"], a["preference"], a["turns"], report["p"],
|
|
852
|
+
report["need"], "STOPPED" if report["stopped"] else "on"))
|
|
853
|
+
return 0
|
|
854
|
+
|
|
855
|
+
|
|
399
856
|
|
|
400
857
|
def main(argv):
|
|
401
858
|
parser = argparse.ArgumentParser(prog="tezgah-taste", description=__doc__,
|
|
@@ -411,12 +868,54 @@ def main(argv):
|
|
|
411
868
|
measure.add_argument("--labels", required=True)
|
|
412
869
|
measure.add_argument("--model", default=tezgah_judge.MODEL)
|
|
413
870
|
measure.add_argument("--json", action="store_true")
|
|
871
|
+
measure.add_argument("--typed", action="store_true",
|
|
872
|
+
help="require TypeSafe (the decision T1P's ledger trusts)")
|
|
414
873
|
rate = sub.add_parser("rate")
|
|
415
874
|
rate.add_argument("--in", dest="inp", required=True)
|
|
416
875
|
rate.add_argument("--limit", type=int)
|
|
417
876
|
rate.add_argument("--json", action="store_true")
|
|
877
|
+
learn = sub.add_parser("learn")
|
|
878
|
+
learn.add_argument("--from-transcripts", action="store_true")
|
|
879
|
+
learn.add_argument("--host", choices=("omp", "claude", "all"), default="all")
|
|
880
|
+
learn.add_argument("--limit", type=int)
|
|
881
|
+
listing = sub.add_parser("list")
|
|
882
|
+
listing.add_argument("--all", action="store_true")
|
|
883
|
+
for name in ("show", "accept", "reject", "edit"):
|
|
884
|
+
one = sub.add_parser(name)
|
|
885
|
+
one.add_argument("id")
|
|
886
|
+
if name == "edit":
|
|
887
|
+
one.add_argument("--text", required=True)
|
|
888
|
+
sub.add_parser("export")
|
|
889
|
+
label = sub.add_parser("label")
|
|
890
|
+
label.add_argument("id", nargs="?")
|
|
891
|
+
label.add_argument("label", nargs="?")
|
|
892
|
+
label.add_argument("--n", type=int, default=10)
|
|
893
|
+
sub.add_parser("calibrate")
|
|
894
|
+
sub.add_parser("gate")
|
|
895
|
+
for name, p in sub.choices.items():
|
|
896
|
+
if name not in ("mine", "measure", "rate"):
|
|
897
|
+
p.add_argument("--repo")
|
|
898
|
+
if name in ("learn", "list", "calibrate", "gate"):
|
|
899
|
+
p.add_argument("--json", action="store_true")
|
|
418
900
|
args = parser.parse_args(argv[1:])
|
|
419
|
-
|
|
901
|
+
handler = {"mine": cmd_mine, "measure": cmd_measure, "rate": cmd_rate,
|
|
902
|
+
"learn": cmd_learn, "list": cmd_list, "show": cmd_one, "accept": cmd_one,
|
|
903
|
+
"reject": cmd_one, "edit": cmd_one, "export": cmd_export,
|
|
904
|
+
"label": cmd_label, "calibrate": cmd_calibrate,
|
|
905
|
+
"gate": cmd_gate}[args.cmd]
|
|
906
|
+
if args.cmd not in ("learn", "accept", "reject", "edit", "label"):
|
|
907
|
+
return handler(args)
|
|
908
|
+
# the commands that rewrite ledger.json: one at a time, or the last one
|
|
909
|
+
# to finish would silently drop what the other wrote
|
|
910
|
+
try:
|
|
911
|
+
repo = store_or_exit(args)
|
|
912
|
+
refused = learn_refusal(repo) if args.cmd == "learn" else None
|
|
913
|
+
if refused:
|
|
914
|
+
return misuse(refused)
|
|
915
|
+
with tl.locked():
|
|
916
|
+
return handler(args)
|
|
917
|
+
except tl.Busy as busy:
|
|
918
|
+
return misuse("the taste ledger is in use (%s); run this again when it ends" % busy)
|
|
420
919
|
|
|
421
920
|
|
|
422
921
|
if __name__ == "__main__":
|
package/bin/tezgah-triage
CHANGED
|
@@ -90,9 +90,11 @@ MEASURED = ("100% of the control lines at a 94% read on a reproduced 356-line "
|
|
|
90
90
|
"screen")
|
|
91
91
|
CUT = 96
|
|
92
92
|
USAGE = "tezgah-triage --select FILE --task TEXT | --states FILE [--component TEXT]"
|
|
93
|
-
NO_CREDENTIAL = ("no
|
|
94
|
-
"the
|
|
95
|
-
"~/.config/
|
|
93
|
+
NO_CREDENTIAL = ("no judge: no session CLI (omp or claude, from OMPCODE/CLAUDECODE), "
|
|
94
|
+
"and no third party the `fallback` setting allows (TYPESAFE_API_KEY "
|
|
95
|
+
"or ~/.config/typesafe/key, or the OpenRouter fallback's "
|
|
96
|
+
"OPENROUTER_API_KEY or ~/.config/openrouter/key), or the judge-off "
|
|
97
|
+
"switch is on")
|
|
96
98
|
TRIAGE_OFF = "the triage-off switch is on"
|
|
97
99
|
FAILED = ("the call failed - a timeout, a refusal, or a reply that is not the "
|
|
98
100
|
"documented shape")
|
package/docs/README.md
CHANGED
|
@@ -36,6 +36,7 @@ citation you can open: `path::symbol` into Python code, `path:line` elsewhere.
|
|
|
36
36
|
| Which model runs which subagent, and how does `tezgah-route` pick one? | [models](models.md) |
|
|
37
37
|
| Where does a research line live, and what does `tezgah-research check` refuse? | [research](research.md) |
|
|
38
38
|
| What does each host get, and what can it observe? | [hosts](hosts.md) |
|
|
39
|
+
| Inside Orca, who owns worktrees and terminals, and which checkouts does Orca track? | [orca](orca.md) |
|
|
39
40
|
| Which skills ship, and how does one reach a host? | [skills](skills.md) |
|
|
40
41
|
| How is one feature audited across its layers, and what does a capability change need to carry? | [feature-audit](feature-audit.md) |
|
|
41
42
|
| How do I check a change, and how do I test it? | [testing](testing.md) |
|
package/docs/architecture.md
CHANGED
|
@@ -93,6 +93,7 @@ One session, in order. Each step names the file that handles the event on Claude
|
|
|
93
93
|
the graph index status and the detached index spawn (`hooks/tezgah_context.py::context_for`),
|
|
94
94
|
the open plans, the sibling checkouts of a `git worktree` with their open plan
|
|
95
95
|
and research line counts (`sibling_line`, `hooks/tezgah_context.py::sibling_line`),
|
|
96
|
+
inside Orca the `Orca:` line ([orca](orca.md), `hooks/tezgah_orca.py::context_line`),
|
|
96
97
|
the lessons ledger, and the active kill switches. The CORE is
|
|
97
98
|
dropped when the host's own file already carries it (`TEZGAH_CORE_IN_FILE`,
|
|
98
99
|
declared by that manifest row: `~/.claude/CLAUDE.md` on Claude). Outside a
|
|
@@ -110,9 +111,11 @@ One session, in order. Each step names the file that handles the event on Claude
|
|
|
110
111
|
session block's last five, see
|
|
111
112
|
[operations.md](operations.md#opt-in-embedding-relevance); each once per
|
|
112
113
|
session, remembered in the turn stamp and forgotten at a compaction),
|
|
113
|
-
and the stale-index notice
|
|
114
|
+
and the stale-index notice, which also starts the incremental re-index
|
|
115
|
+
(`hooks/tezgah_context.py::index_notice`).
|
|
114
116
|
3. **PreToolUse** — `hooks/projects-pretooluse.py::main` calls `decision` and emits
|
|
115
|
-
the deny envelope
|
|
117
|
+
the deny envelope, or, for a code-symbol search it lets through, the graph line
|
|
118
|
+
as `additionalContext` (`hooks/tezgah_gate.py::graph_advice`). The gate is the same
|
|
116
119
|
object on every host: `hosts/omp/hook.py::handle`, `hosts/codex/hook.py::gate_reason`.
|
|
117
120
|
4. **PostToolUse** — `hooks/projects-posttooluse.py::main` writes the evidence row
|
|
118
121
|
through `note_tool` (`hooks/projects-posttooluse.py::main`), records the used
|