@azure-id/orc 0.56.1 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +219 -0
- package/README-id.md +49 -107
- package/README.md +694 -719
- package/bin/build-agents.js +18 -11
- package/bin/cli.js +33699 -29855
- package/bin/onboarding-content.js +0 -3
- package/bin/test-run.js +491 -0
- package/bin/verify-contracts.js +4330 -3277
- package/bin/verify-package.js +76 -6
- package/bin/webui/api.js +1201 -1163
- package/bin/webui/app.html +210 -198
- package/bin/webui/css/06-responsive.css +120 -108
- package/bin/webui/css/panels/extra.css +29 -0
- package/bin/webui/css/panels/lanes.css +134 -0
- package/bin/webui/css/panels/settings.css +79 -3
- package/bin/webui/css/panels/wait.css +123 -0
- package/bin/webui/fixtures/extra.js +2036 -1934
- package/bin/webui/fixtures/flow.js +2 -2
- package/bin/webui/fixtures/index.js +510 -486
- package/bin/webui/fixtures/lanes.js +218 -0
- package/bin/webui/fixtures/settings.js +261 -34
- package/bin/webui/fixtures/stats.js +3 -3
- package/bin/webui/fixtures/wait.js +97 -0
- package/bin/webui/i18n/TERMS.md +135 -134
- package/bin/webui/i18n/en/extra.json +345 -313
- package/bin/webui/i18n/en/lanes.json +29 -0
- package/bin/webui/i18n/en/nav.json +21 -19
- package/bin/webui/i18n/en/overview.json +2 -0
- package/bin/webui/i18n/en/settings.json +16 -5
- package/bin/webui/i18n/en/wait.json +41 -0
- package/bin/webui/i18n/id/extra.json +345 -313
- package/bin/webui/i18n/id/lanes.json +29 -0
- package/bin/webui/i18n/id/nav.json +21 -19
- package/bin/webui/i18n/id/overview.json +2 -0
- package/bin/webui/i18n/id/settings.json +15 -4
- package/bin/webui/i18n/id/wait.json +41 -0
- package/bin/webui/js/01-i18n.js +151 -149
- package/bin/webui/js/panels/extra.js +3199 -3011
- package/bin/webui/js/panels/lanes.js +259 -0
- package/bin/webui/js/panels/overview.js +10 -0
- package/bin/webui/js/panels/settings.js +149 -3
- package/bin/webui/js/panels/wait.js +253 -0
- package/package.json +39 -39
- package/templates/agents/MODEL-MAPPING.md +23 -35
- package/templates/agents/orc-executor-opus-4-7-high.md +1 -1
- package/templates/agents/orc-executor-opus-4-7-med.md +1 -1
- package/templates/agents/orc-executor-opus-4-8-high.md +1 -1
- package/templates/agents/orc-executor-opus-5-high.md +1 -1
- package/templates/agents/orc-executor-opus-5-low.md +1 -1
- package/templates/agents/orc-executor-opus-5-med.md +1 -1
- package/templates/agents/orc-retro-opus-5-med.md +73 -73
- package/templates/agents/orc-retro-sonnet-5-high.md +72 -72
- package/templates/agents/orc-trace-writer-haiku-4-5.md +107 -107
- package/templates/commands/orc-route.md +1 -1
- package/templates/commands/orc-wait.md +19 -0
- package/templates/commands/orc.md +1 -1
- package/templates/hooks/orc-statusline.js +39 -0
- package/templates/skills/_shared/README.md +7 -3
- package/templates/skills/_shared/config-precedence.md +198 -0
- package/templates/skills/_shared/drift-recovery.md +1 -1
- package/templates/skills/_shared/extra-dispatch.md +159 -7
- package/templates/skills/_shared/interview.md +1 -1
- package/templates/skills/_shared/opus5-only.md +9 -7
- package/templates/skills/_shared/phases/README.md +82 -0
- package/templates/skills/{orc/references → _shared/phases}/analyst-gates.md +15 -3
- package/templates/skills/_shared/phases/execution.md +143 -0
- package/templates/skills/{orc/references → _shared/phases}/house-rules.md +13 -1
- package/templates/skills/{orc/references → _shared/phases}/intake.md +15 -2
- package/templates/skills/_shared/phases/mock-example.md +56 -0
- package/templates/skills/{orc/references → _shared/phases}/plan-handoff.md +16 -3
- package/templates/skills/_shared/phases/planning.md +135 -0
- package/templates/skills/_shared/phases/preflight.md +98 -0
- package/templates/skills/_shared/phases/review.md +53 -0
- package/templates/skills/_shared/phases/scoring.md +65 -0
- package/templates/skills/{orc/references → _shared/phases}/security-checklist.md +12 -1
- package/templates/skills/_shared/phases/security.md +55 -0
- package/templates/skills/_shared/phases/ship.md +88 -0
- package/templates/skills/{orc/references/stop-and-resume.md → _shared/phases/stop-resume.md} +169 -160
- package/templates/skills/_shared/phases/summary.md +48 -0
- package/templates/skills/_shared/phases/testgen.md +51 -0
- package/templates/skills/{orc/references/trace-protocol.md → _shared/phases/trace.md} +49 -6
- package/templates/skills/_shared/phases/verify.md +70 -0
- package/templates/skills/{orc/references → _shared/phases}/wave-grouping.md +14 -2
- package/templates/skills/{orc/references → _shared/phases}/wiki-consult.md +13 -1
- package/templates/skills/_shared/read-ladder.md +48 -0
- package/templates/skills/_shared/wait.md +240 -0
- package/templates/skills/context-combiner/SKILL.md +214 -187
- package/templates/skills/orc/README.md +148 -150
- package/templates/skills/orc/SKILL.md +75 -358
- package/templates/skills/orc/config.md +137 -389
- package/templates/skills/orc/examples/full-run-mock.md +1 -1
- package/templates/skills/orc/references/effort-and-mode.md +16 -18
- package/templates/skills/orc/references/phases/intake.md +47 -0
- package/templates/skills/orc/references/phases/integration.md +19 -0
- package/templates/skills/orc/references/preflight-report.md +4 -4
- package/templates/skills/orc/references/ultra-mode.md +117 -123
- package/templates/skills/orc/schemas/planning-output.md +279 -279
- package/templates/skills/orc/subskills/orc-planner/SKILL.md +260 -264
- package/templates/skills/orc/subskills/orc-pr/stack-gate.md +4 -1
- package/templates/skills/orc-aftermath/SKILL.md +158 -136
- package/templates/skills/orc-analyze/SKILL.md +252 -220
- package/templates/skills/orc-analyze/references/branching.md +51 -51
- package/templates/skills/orc-analyze/references/thin-input.md +1 -1
- package/templates/skills/orc-analyze-mini/SKILL.md +128 -112
- package/templates/skills/orc-boundary/SKILL.md +249 -208
- package/templates/skills/orc-brainstorm/SKILL.md +377 -339
- package/templates/skills/orc-budget/SKILL.md +248 -225
- package/templates/skills/orc-challenge/README.md +1 -1
- package/templates/skills/orc-challenge/SKILL.md +298 -255
- package/templates/skills/orc-challenge/examples/council-full-roster.md +1 -1
- package/templates/skills/orc-claude/SKILL.md +221 -199
- package/templates/skills/orc-diy/README.md +2 -1
- package/templates/skills/orc-diy/SKILL.md +132 -68
- package/templates/skills/orc-diy/references/blocks/analyze.md +1 -1
- package/templates/skills/orc-diy/references/compile.md +21 -7
- package/templates/skills/orc-diy/references/flow-schema.md +2 -2
- package/templates/skills/orc-doc/SKILL.md +49 -7
- package/templates/skills/orc-explain/SKILL.md +24 -0
- package/templates/skills/orc-export/SKILL.md +31 -5
- package/templates/skills/orc-fast/SKILL.md +218 -185
- package/templates/skills/orc-grill/SKILL.md +238 -207
- package/templates/skills/orc-handoff/SKILL.md +223 -204
- package/templates/skills/orc-learn/SKILL.md +181 -156
- package/templates/skills/orc-mini/SKILL.md +47 -24
- package/templates/skills/orc-pact/SKILL.md +255 -218
- package/templates/skills/orc-pattern/SKILL.md +154 -124
- package/templates/skills/orc-poly/SKILL.md +242 -216
- package/templates/skills/orc-poly/references/gather.md +1 -1
- package/templates/skills/orc-pr-driver/SKILL.md +196 -167
- package/templates/skills/orc-pr-driver/references/orc-run-split.md +1 -1
- package/templates/skills/orc-pr-setup/SKILL.md +212 -184
- package/templates/skills/orc-quick/README.md +2 -2
- package/templates/skills/orc-quick/SKILL.md +51 -13
- package/templates/skills/orc-quick/references/dispatch-gate.md +2 -3
- package/templates/skills/orc-retro/SKILL.md +249 -222
- package/templates/skills/orc-retro/examples/retro-mock.md +171 -171
- package/templates/skills/orc-route/SKILL.md +188 -165
- package/templates/skills/orc-verify/SKILL.md +103 -84
- package/templates/skills/orc-wait/SKILL.md +163 -0
- package/templates/skills/orc-wiki/SKILL.md +60 -206
- package/templates/skills/orc-wiki/references/phases/phase-0.md +71 -0
- package/templates/skills/orc-wiki/references/phases/phase-1.md +35 -0
- package/templates/skills/orc-wiki/references/phases/phase-2.md +52 -0
- package/templates/skills/orc-wiki/references/phases/phase-3.md +57 -0
- package/templates/skills/orc-wiki/references/phases/phase-3c.md +36 -0
- package/templates/agents/orc-advisor-fable-5.md +0 -50
- package/templates/agents/orc-analyst-fable-5.md +0 -115
- package/templates/agents/orc-judge-fable-5.md +0 -79
- package/templates/agents/orc-planner-fable-5.md +0 -152
- package/templates/agents/orc-reviewer-fable-5.md +0 -57
- package/templates/skills/_shared/fable5-override.md +0 -56
- package/templates/skills/orc-diy/references/blocks/execution.md +0 -42
- package/templates/skills/orc-diy/references/blocks/mock-example.md +0 -22
- package/templates/skills/orc-diy/references/blocks/planning.md +0 -34
- package/templates/skills/orc-diy/references/blocks/review.md +0 -18
- package/templates/skills/orc-diy/references/blocks/scoring.md +0 -16
- package/templates/skills/orc-diy/references/blocks/security.md +0 -25
- package/templates/skills/orc-diy/references/blocks/ship.md +0 -25
- package/templates/skills/orc-diy/references/blocks/summary.md +0 -18
- package/templates/skills/orc-diy/references/blocks/testgen.md +0 -17
- package/templates/skills/orc-diy/references/blocks/trace.md +0 -28
- package/templates/skills/orc-diy/references/blocks/verify.md +0 -25
|
@@ -1,222 +1,249 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: orc-retro
|
|
3
|
-
description: >
|
|
4
|
-
Retro miner for ORC — closes the behavior-trace flywheel. Use for
|
|
5
|
-
"/orc-retro", "analyze the orc traces", "how well is orc scoring", or "what
|
|
6
|
-
should we tune from the logged runs". Reads the persistent behavior traces
|
|
7
|
-
in log_dir (behavior-trace logging is permanent — every run writes one), aggregates per-band
|
|
8
|
-
outcomes (retries, requeues, needs_context, unmet, downgrades, findings),
|
|
9
|
-
and produces a calibration report with recommendations. READ-ONLY and
|
|
10
|
-
REPORT-ONLY against the local system: it never edits the rubric, the skills,
|
|
11
|
-
or project code. The report is DELIVERED upstream — filed as a PR (issue
|
|
12
|
-
fallback) to the ORC repo (`retro_repo` config, default azure-id/orc) in
|
|
13
|
-
AI-readable markdown, via the gh CLI or a GitHub MCP. P0 preflight: if
|
|
14
|
-
NEITHER delivery channel exists, the retro does not run at all. The
|
|
15
|
-
orchestrator dispatches the mining to a subagent — it never mines itself.
|
|
16
|
-
---
|
|
17
|
-
|
|
18
|
-
# ORC-RETRO (trace miner)
|
|
19
|
-
|
|
20
|
-
The behavior trace exists "for post-hoc skill improvement" — this skill is the
|
|
21
|
-
return spoke of that flywheel. It turns the raw `.txt` traces into an answer to
|
|
22
|
-
three questions: **is the scoring rubric calibrated? are the workers honest?
|
|
23
|
-
where does the pipeline leak?**
|
|
24
|
-
|
|
25
|
-
Run as Opus 4.8 high (orchestrator). The mining itself is dispatched to
|
|
26
|
-
`orc-retro-sonnet-5-high` — cheap, because it reads trace text, not code — or
|
|
27
|
-
to `orc-retro-opus-5-med` when `opus5_only: true` forces it
|
|
28
|
-
(`../_shared/opus5-only.md`).
|
|
29
|
-
|
|
30
|
-
**Worked example** (orient only — never execute from it): `examples/retro-mock.md`.
|
|
31
|
-
|
|
32
|
-
## Hard rules
|
|
33
|
-
|
|
34
|
-
0. **P0 preflight — a delivery channel or no retro at all.** The report exists
|
|
35
|
-
to land in the ORC repo's PRs/issues where the maintainer (or an AI reading
|
|
36
|
-
the repo) can act on it; a retro that can't deliver is pointless. BEFORE
|
|
37
|
-
resolving traces or dispatching anything, probe in order:
|
|
38
|
-
- **gh CLI:** `gh --version` succeeds AND `gh auth status` reports a logged-in
|
|
39
|
-
account → channel is `gh`.
|
|
40
|
-
- **GitHub MCP:** otherwise, check the session's available tools for a GitHub
|
|
41
|
-
MCP server (tool names like `mcp__github__*` / create_pull_request /
|
|
42
|
-
create_issue) → channel is `mcp`.
|
|
43
|
-
- **Neither → STOP.** Do not mine, do not spawn, do not write a local report.
|
|
44
|
-
Tell the user: install + auth the gh CLI (`gh auth login`) or connect a
|
|
45
|
-
GitHub MCP server, then re-run `/orc-retro`.
|
|
46
|
-
1. **Read-only, report-only against the local system.** Never edit
|
|
47
|
-
`effort-and-mode.md`, any skill, config, or project code. Recommendations
|
|
48
|
-
are phrased for a HUMAN (or the ORC repo's AI) to apply; the retro never
|
|
49
|
-
self-tunes the system it measures. Its ONLY write outside `log_dir` is the
|
|
50
|
-
upstream PR/issue delivery below.
|
|
51
|
-
2. **You never mine yourself — you spawn.** Dispatch the retro agent with the
|
|
52
|
-
trace file list; you validate the return and write the report.
|
|
53
|
-
3. **No traces → say so and stop.** Requires ORC runs to have happened (logging
|
|
54
|
-
is permanent, so any past run left a trace): resolve `log_dir`
|
|
55
|
-
(`../orc/config.md` default + `.claude/orc.config.yaml`) and list its `*.txt`.
|
|
56
|
-
Empty → tell the user no runs have been traced yet and stop. Never invent
|
|
57
|
-
findings from zero data.
|
|
58
|
-
4. **This lane does NOT write a trace of its own.** It is the reader of the
|
|
59
|
-
trace system, not a run — it writes no run pointer and emits no markers
|
|
60
|
-
(tracing the retro would pollute the very data it mines).
|
|
61
|
-
5. **Small-sample honesty:** every aggregate states its n. A recommendation
|
|
62
|
-
from n<3 runs is labeled "weak signal — gather more runs", never stated as
|
|
63
|
-
a conclusion.
|
|
64
|
-
|
|
65
|
-
## Procedure
|
|
66
|
-
|
|
67
|
-
0. **Preflight (hard rule 0):** establish the delivery channel (`gh` or `mcp`).
|
|
68
|
-
No channel → stop here. Resolve `retro_repo` with the other config keys.
|
|
69
|
-
1. Resolve `log_dir`; collect `*.txt` traces (all, or the user-named subset /
|
|
70
|
-
date range from `$ARGUMENTS`). Show the count and ask nothing else.
|
|
71
|
-
2. Dispatch `orc-retro-sonnet-5-high` (or `orc-retro-opus-5-med` under
|
|
72
|
-
`opus5_only`) with the slice: trace file paths + the
|
|
73
|
-
verb reference (`../
|
|
74
|
-
`<trace>.jsonl` sidecar first when present (structured — no regex over free
|
|
75
|
-
text) and falls back to `.txt` parsing for pre-v0.32.0 traces, merging the
|
|
76
|
-
hook's `.txt`-only skeleton lines by timestamp. It parses the CLOSED verb set
|
|
77
|
-
and aggregates:
|
|
78
|
-
- **Band calibration** (from `OUTCOME` lines): per band — task count, avg
|
|
79
|
-
retries/requeues/needs_context/unmet. High retries in a band = the band's
|
|
80
|
-
model is too weak (or slices too big); all-zeros in a high band = maybe
|
|
81
|
-
over-tiered (cost leak).
|
|
82
|
-
- **Tier integrity** (from `VERIFY` lines): every `⛔ DOWNGRADE`, grouped by
|
|
83
|
-
agent — the "main session below Opus" bug leaves this exact fingerprint.
|
|
84
|
-
- **Gate bounce rates** (from `GATE` lines): per gate name (grounding /
|
|
85
|
-
coverage / graph / evidence / derivation) — pass vs bounce counts. A high
|
|
86
|
-
bounce rate on one gate localizes which role's instructions leak (e.g.
|
|
87
|
-
planner orphans → planner coverage discipline needs tuning).
|
|
88
|
-
- **Per-lane aggregation** (free, from the filename grammar
|
|
89
|
-
`run-<lane>-<slug>-<DDMMYY>-<HHMMSS>.txt`): runs/tasks/unfinished per lane,
|
|
90
|
-
so an orc run is never averaged together with a mini or fast one.
|
|
91
|
-
- **Pipeline leaks:** `QUESTION`/`CONTEXT-GAP` clusters (over-asking, slices
|
|
92
|
-
missing context), `REPLAN` reasons, `FINDING p0..p3` and `VERDICT fail`
|
|
93
|
-
rates per run, runs with `SPAWN`s but no `FINISH` (aborted/never closed).
|
|
94
|
-
- **Trace hygiene → narration coverage:** the hook's `PHASE-EDGE` lines
|
|
95
|
-
segment every run with zero model cooperation, so a missing narration is
|
|
96
|
-
now DETERMINISTICALLY visible: count the phases whose edge-interval
|
|
97
|
-
contains no trace-writer `SPAWN`. Report `covered/total` + the unnarrated
|
|
98
|
-
phases. The question is no longer "were rich markers forgotten?" but
|
|
99
|
-
"which phases never dispatched their writer?" — a run with edges and zero
|
|
100
|
-
writer spawns is a total narration failure and is named as such.
|
|
101
|
-
2a. **Repair memory as calibration input (READ-ONLY).** Probe `orc gotcha status`
|
|
102
|
-
(exit 0 = entries, 1 = none — never a `find`). On a hit, add
|
|
103
|
-
`.claude/orc/gotchas.md` to the agent's slice as one more read-only source:
|
|
104
|
-
a `kind`/`area` cluster is evidence that a band, a slice size or a playbook is
|
|
105
|
-
under-serving that area, and a high-`hits` entry is a repair this project keeps
|
|
106
|
-
re-paying for. Cite the entry IDs in the recommendation's evidence like any
|
|
107
|
-
other line count. **This lane NEVER writes, prunes, or edits a gotcha** — it is
|
|
108
|
-
report-only against the local system by hard rule 2, and repair memory is no
|
|
109
|
-
exception. Canonical: `../_shared/gotchas.md`.
|
|
110
|
-
3. Validate the return (contract below). Write the report to
|
|
111
|
-
`log_dir/retro/<DDMMYY>-report.md` (the `retro/` subfolder keeps the trace
|
|
112
|
-
folder's top level to run traces + their sidecars) in the format below, and
|
|
113
|
-
show the user the summary: verdict per question, the per-band table, and
|
|
114
|
-
each recommendation with its evidence line counts + n.
|
|
115
|
-
4. **Deliver upstream (the point of the retro).** File the report to
|
|
116
|
-
`retro_repo` (config, default `azure-id/orc`) — **PR preferred, issue
|
|
117
|
-
fallback**, over the channel from step 0:
|
|
118
|
-
- **`gh` channel:** if the cwd's `git remote` already IS `retro_repo`,
|
|
119
|
-
branch `retro/<DDMMYY>` from the default branch, add the report as
|
|
120
|
-
`retro/incoming/<DDMMYY>-<project>-report.md`, push, `gh pr create`.
|
|
121
|
-
Otherwise shallow-clone `retro_repo` into a temp dir and do the same
|
|
122
|
-
there. If push or PR creation fails (e.g. no write access), fall back to
|
|
123
|
-
`gh issue create -R <retro_repo>` with the full report as the body.
|
|
124
|
-
- **`mcp` channel:** same shape with the MCP's branch/file/PR tools;
|
|
125
|
-
fallback its create-issue tool.
|
|
126
|
-
- PR/issue title: `orc-retro: <DDMMYY> — <n> runs, <k> recommendations`.
|
|
127
|
-
- Either way, end by showing the user the created PR/issue URL. If delivery
|
|
128
|
-
itself errors after the preflight passed, surface the error verbatim and
|
|
129
|
-
point at the local report copy — never claim it was filed.
|
|
130
|
-
|
|
131
|
-
## Foreign dispatches — the `EXTRA` verb (v0.50.0)
|
|
132
|
-
|
|
133
|
-
A task can execute on a **non-Claude worker** (`_shared/extra-dispatch.md`). When
|
|
134
|
-
it does, the trace carries an `EXTRA` line per dispatch and **no `SPAWN` /
|
|
135
|
-
`RETURN` at all** — a foreign worker is not a Claude subagent, so the hook has
|
|
136
|
-
nothing to observe (P7, the `/orc-quick` ad-hoc-recon precedent).
|
|
137
|
-
|
|
138
|
-
**Read `EXTRA` or every foreign dispatch reads as a MISSING RETURN.** That is
|
|
139
|
-
the concrete failure this section exists to prevent: a leak count inflated by
|
|
140
|
-
work that completed perfectly well somewhere else. A foreign dispatch also
|
|
141
|
-
contributes nothing to NARRATION COVERAGE, and must not be counted against it.
|
|
142
|
-
|
|
143
|
-
Do not parse the lines yourself — **run `orc extra stats --json`** and report
|
|
144
|
-
what it computed (the `computeWikiFreshness` rule: one engine, and the skill is
|
|
145
|
-
not it). It groups **per profile per band**, which is the pair a routing decision
|
|
146
|
-
was actually made in: a per-provider total cannot tell you the `[0,30)` row was
|
|
147
|
-
fine and the `[30,70)` row was a false economy.
|
|
148
|
-
|
|
149
|
-
Report four things beside the ordinary per-band table:
|
|
150
|
-
|
|
151
|
-
| what | why it is its own number |
|
|
152
|
-
|---|---|
|
|
153
|
-
| outcome mix per band | `done` / `partial` / `failed` / `fallback`. The fallback rate IS the answer to "did routing this band off Claude work" |
|
|
154
|
-
| **SUBSTITUTION** count | the endpoint answered with a **different model**. Never aggregate this into a failure rate — the dispatch may have succeeded; you got something you did not ask for |
|
|
155
|
-
| **REROUTE** count | the model id held and a **different company** served it. Only engine `api` can see this at all; on the other two engines the absence of reroutes is **not** evidence there were none |
|
|
156
|
-
| repairs AFTER a foreign dispatch | fix cycles, `TDD-RED` iterations, reviewer P0/P1 and `REPLAN` lines whose task id matches a foreign task. **This is the whole point.** A run that cost a tenth as much and then needed two repair rounds was not cheaper |
|
|
157
|
-
|
|
158
|
-
**`tok=none` is a real value and must never be averaged as zero.** Engine `cli`
|
|
159
|
-
often reports no token counts; `orc extra stats` reports the vector plus *how
|
|
160
|
-
many dispatches it came from*, and the retro must carry that denominator through.
|
|
161
|
-
A cost total assembled from six of ten dispatches is not that band's cost.
|
|
162
|
-
|
|
163
|
-
**A dollar figure only where a rate exists.** Every `models` map in the shipped
|
|
164
|
-
price table is EMPTY on purpose (`orc extra rates` explains why and prints the
|
|
165
|
-
JSON to paste), so `usd: null` is the normal state and is reported as an em dash,
|
|
166
|
-
never as zero and never as an estimate.
|
|
167
|
-
|
|
168
|
-
**Never rank providers by quality.** `/orc-retro` reports outcomes and lets the
|
|
169
|
-
user decide — a benchmark ORC ran itself would be a benchmark ORC was motivated
|
|
170
|
-
to like.
|
|
171
|
-
|
|
172
|
-
## Report format (AI-readable — the PR/issue payload)
|
|
173
|
-
|
|
174
|
-
The report is written so the ORC repo's maintainer OR an AI session reading
|
|
175
|
-
the repo can act on it without parsing prose. YAML frontmatter mirrors the
|
|
176
|
-
return contract EXACTLY (machine layer), followed by short human sections:
|
|
177
|
-
|
|
178
|
-
```markdown
|
|
179
|
-
---
|
|
180
|
-
schema: orc-retro/v1
|
|
181
|
-
generated: <ISO date>
|
|
182
|
-
project: <cwd project name>
|
|
183
|
-
orc_version: <installed ORC version if known, else unknown>
|
|
184
|
-
runs_analyzed: <n>
|
|
185
|
-
tasks_analyzed: <n>
|
|
186
|
-
lane_stats: [...] # verbatim from the return contract
|
|
187
|
-
narration_coverage: {...}
|
|
188
|
-
band_stats: [...]
|
|
189
|
-
downgrades: [...]
|
|
190
|
-
leaks: [...]
|
|
191
|
-
extra_stats: {...} # `orc extra stats --json` verbatim, or null when no EXTRA line exists
|
|
192
|
-
recommendations: [...] # each with finding, suggested_change, confidence
|
|
193
|
-
actual_model: <...>
|
|
194
|
-
actual_effort: <...>
|
|
195
|
-
---
|
|
196
|
-
## Verdicts (the three questions, one line each)
|
|
197
|
-
## Per-band table
|
|
198
|
-
## Recommendations (one subsection each: evidence lines, suggested edit, confidence + n)
|
|
199
|
-
```
|
|
200
|
-
|
|
201
|
-
## Return contract (the agent emits EXACTLY this; you validate)
|
|
202
|
-
|
|
203
|
-
- `runs_analyzed`, `tasks_analyzed` — the n behind everything
|
|
204
|
-
- `lane_stats[]` — {lane, runs, tasks, unfinished} (lane from the filename)
|
|
205
|
-
- `narration_coverage` — {phases_total, phases_narrated, pct, unnarrated[]:
|
|
206
|
-
{run, role_family, first_agent}} — from the hook's `PHASE-EDGE` segmentation
|
|
207
|
-
- `band_stats[]` — {band, model, tasks, avg_retries, avg_requeues,
|
|
208
|
-
avg_needs_context, avg_unmet}
|
|
209
|
-
- `downgrades[]` — {agent, expected, actual, run}
|
|
210
|
-
- `leaks[]` — {kind: question-cluster | context-gap | replan | verdict-fail |
|
|
211
|
-
unfinished-run | hygiene, evidence (trace file + line numbers), count}
|
|
212
|
-
- `recommendations[]` — {finding, suggested_change (which file/table a human
|
|
213
|
-
would edit — e.g. an effort-and-mode.md facet weight (mined from the SCORE
|
|
214
|
-
line's `facets=` vector) or a band boundary in
|
|
215
|
-
config.md), confidence: strong|weak (weak when n<3)}
|
|
216
|
-
- `actual_model` — quoted VERBATIM from the system prompt's "The exact model
|
|
217
|
-
ID is …" line (`unknown` if absent, never guessed)
|
|
218
|
-
- `actual_effort` — `$CLAUDE_EFFORT`
|
|
219
|
-
|
|
220
|
-
Malformed = failure (re-dispatch once, then surface). A recommendation without
|
|
221
|
-
evidence line references is dropped — the retro obeys the same
|
|
222
|
-
evidence-or-advisory rule it audits.
|
|
1
|
+
---
|
|
2
|
+
name: orc-retro
|
|
3
|
+
description: >
|
|
4
|
+
Retro miner for ORC — closes the behavior-trace flywheel. Use for
|
|
5
|
+
"/orc-retro", "analyze the orc traces", "how well is orc scoring", or "what
|
|
6
|
+
should we tune from the logged runs". Reads the persistent behavior traces
|
|
7
|
+
in log_dir (behavior-trace logging is permanent — every run writes one), aggregates per-band
|
|
8
|
+
outcomes (retries, requeues, needs_context, unmet, downgrades, findings),
|
|
9
|
+
and produces a calibration report with recommendations. READ-ONLY and
|
|
10
|
+
REPORT-ONLY against the local system: it never edits the rubric, the skills,
|
|
11
|
+
or project code. The report is DELIVERED upstream — filed as a PR (issue
|
|
12
|
+
fallback) to the ORC repo (`retro_repo` config, default azure-id/orc) in
|
|
13
|
+
AI-readable markdown, via the gh CLI or a GitHub MCP. P0 preflight: if
|
|
14
|
+
NEITHER delivery channel exists, the retro does not run at all. The
|
|
15
|
+
orchestrator dispatches the mining to a subagent — it never mines itself.
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
# ORC-RETRO (trace miner)
|
|
19
|
+
|
|
20
|
+
The behavior trace exists "for post-hoc skill improvement" — this skill is the
|
|
21
|
+
return spoke of that flywheel. It turns the raw `.txt` traces into an answer to
|
|
22
|
+
three questions: **is the scoring rubric calibrated? are the workers honest?
|
|
23
|
+
where does the pipeline leak?**
|
|
24
|
+
|
|
25
|
+
Run as Opus 4.8 high (orchestrator). The mining itself is dispatched to
|
|
26
|
+
`orc-retro-sonnet-5-high` — cheap, because it reads trace text, not code — or
|
|
27
|
+
to `orc-retro-opus-5-med` when `opus5_only: true` forces it
|
|
28
|
+
(`../_shared/opus5-only.md`).
|
|
29
|
+
|
|
30
|
+
**Worked example** (orient only — never execute from it): `examples/retro-mock.md`.
|
|
31
|
+
|
|
32
|
+
## Hard rules
|
|
33
|
+
|
|
34
|
+
0. **P0 preflight — a delivery channel or no retro at all.** The report exists
|
|
35
|
+
to land in the ORC repo's PRs/issues where the maintainer (or an AI reading
|
|
36
|
+
the repo) can act on it; a retro that can't deliver is pointless. BEFORE
|
|
37
|
+
resolving traces or dispatching anything, probe in order:
|
|
38
|
+
- **gh CLI:** `gh --version` succeeds AND `gh auth status` reports a logged-in
|
|
39
|
+
account → channel is `gh`.
|
|
40
|
+
- **GitHub MCP:** otherwise, check the session's available tools for a GitHub
|
|
41
|
+
MCP server (tool names like `mcp__github__*` / create_pull_request /
|
|
42
|
+
create_issue) → channel is `mcp`.
|
|
43
|
+
- **Neither → STOP.** Do not mine, do not spawn, do not write a local report.
|
|
44
|
+
Tell the user: install + auth the gh CLI (`gh auth login`) or connect a
|
|
45
|
+
GitHub MCP server, then re-run `/orc-retro`.
|
|
46
|
+
1. **Read-only, report-only against the local system.** Never edit
|
|
47
|
+
`effort-and-mode.md`, any skill, config, or project code. Recommendations
|
|
48
|
+
are phrased for a HUMAN (or the ORC repo's AI) to apply; the retro never
|
|
49
|
+
self-tunes the system it measures. Its ONLY write outside `log_dir` is the
|
|
50
|
+
upstream PR/issue delivery below.
|
|
51
|
+
2. **You never mine yourself — you spawn.** Dispatch the retro agent with the
|
|
52
|
+
trace file list; you validate the return and write the report.
|
|
53
|
+
3. **No traces → say so and stop.** Requires ORC runs to have happened (logging
|
|
54
|
+
is permanent, so any past run left a trace): resolve `log_dir`
|
|
55
|
+
(`../orc/config.md` default + `.claude/orc.config.yaml`) and list its `*.txt`.
|
|
56
|
+
Empty → tell the user no runs have been traced yet and stop. Never invent
|
|
57
|
+
findings from zero data.
|
|
58
|
+
4. **This lane does NOT write a trace of its own.** It is the reader of the
|
|
59
|
+
trace system, not a run — it writes no run pointer and emits no markers
|
|
60
|
+
(tracing the retro would pollute the very data it mines).
|
|
61
|
+
5. **Small-sample honesty:** every aggregate states its n. A recommendation
|
|
62
|
+
from n<3 runs is labeled "weak signal — gather more runs", never stated as
|
|
63
|
+
a conclusion.
|
|
64
|
+
|
|
65
|
+
## Procedure
|
|
66
|
+
|
|
67
|
+
0. **Preflight (hard rule 0):** establish the delivery channel (`gh` or `mcp`).
|
|
68
|
+
No channel → stop here. Resolve `retro_repo` with the other config keys.
|
|
69
|
+
1. Resolve `log_dir`; collect `*.txt` traces (all, or the user-named subset /
|
|
70
|
+
date range from `$ARGUMENTS`). Show the count and ask nothing else.
|
|
71
|
+
2. Dispatch `orc-retro-sonnet-5-high` (or `orc-retro-opus-5-med` under
|
|
72
|
+
`opus5_only`) with the slice: trace file paths + the
|
|
73
|
+
verb reference (`../_shared/phases/trace.md`). The agent mines the
|
|
74
|
+
`<trace>.jsonl` sidecar first when present (structured — no regex over free
|
|
75
|
+
text) and falls back to `.txt` parsing for pre-v0.32.0 traces, merging the
|
|
76
|
+
hook's `.txt`-only skeleton lines by timestamp. It parses the CLOSED verb set
|
|
77
|
+
and aggregates:
|
|
78
|
+
- **Band calibration** (from `OUTCOME` lines): per band — task count, avg
|
|
79
|
+
retries/requeues/needs_context/unmet. High retries in a band = the band's
|
|
80
|
+
model is too weak (or slices too big); all-zeros in a high band = maybe
|
|
81
|
+
over-tiered (cost leak).
|
|
82
|
+
- **Tier integrity** (from `VERIFY` lines): every `⛔ DOWNGRADE`, grouped by
|
|
83
|
+
agent — the "main session below Opus" bug leaves this exact fingerprint.
|
|
84
|
+
- **Gate bounce rates** (from `GATE` lines): per gate name (grounding /
|
|
85
|
+
coverage / graph / evidence / derivation) — pass vs bounce counts. A high
|
|
86
|
+
bounce rate on one gate localizes which role's instructions leak (e.g.
|
|
87
|
+
planner orphans → planner coverage discipline needs tuning).
|
|
88
|
+
- **Per-lane aggregation** (free, from the filename grammar
|
|
89
|
+
`run-<lane>-<slug>-<DDMMYY>-<HHMMSS>.txt`): runs/tasks/unfinished per lane,
|
|
90
|
+
so an orc run is never averaged together with a mini or fast one.
|
|
91
|
+
- **Pipeline leaks:** `QUESTION`/`CONTEXT-GAP` clusters (over-asking, slices
|
|
92
|
+
missing context), `REPLAN` reasons, `FINDING p0..p3` and `VERDICT fail`
|
|
93
|
+
rates per run, runs with `SPAWN`s but no `FINISH` (aborted/never closed).
|
|
94
|
+
- **Trace hygiene → narration coverage:** the hook's `PHASE-EDGE` lines
|
|
95
|
+
segment every run with zero model cooperation, so a missing narration is
|
|
96
|
+
now DETERMINISTICALLY visible: count the phases whose edge-interval
|
|
97
|
+
contains no trace-writer `SPAWN`. Report `covered/total` + the unnarrated
|
|
98
|
+
phases. The question is no longer "were rich markers forgotten?" but
|
|
99
|
+
"which phases never dispatched their writer?" — a run with edges and zero
|
|
100
|
+
writer spawns is a total narration failure and is named as such.
|
|
101
|
+
2a. **Repair memory as calibration input (READ-ONLY).** Probe `orc gotcha status`
|
|
102
|
+
(exit 0 = entries, 1 = none — never a `find`). On a hit, add
|
|
103
|
+
`.claude/orc/gotchas.md` to the agent's slice as one more read-only source:
|
|
104
|
+
a `kind`/`area` cluster is evidence that a band, a slice size or a playbook is
|
|
105
|
+
under-serving that area, and a high-`hits` entry is a repair this project keeps
|
|
106
|
+
re-paying for. Cite the entry IDs in the recommendation's evidence like any
|
|
107
|
+
other line count. **This lane NEVER writes, prunes, or edits a gotcha** — it is
|
|
108
|
+
report-only against the local system by hard rule 2, and repair memory is no
|
|
109
|
+
exception. Canonical: `../_shared/gotchas.md`.
|
|
110
|
+
3. Validate the return (contract below). Write the report to
|
|
111
|
+
`log_dir/retro/<DDMMYY>-report.md` (the `retro/` subfolder keeps the trace
|
|
112
|
+
folder's top level to run traces + their sidecars) in the format below, and
|
|
113
|
+
show the user the summary: verdict per question, the per-band table, and
|
|
114
|
+
each recommendation with its evidence line counts + n.
|
|
115
|
+
4. **Deliver upstream (the point of the retro).** File the report to
|
|
116
|
+
`retro_repo` (config, default `azure-id/orc`) — **PR preferred, issue
|
|
117
|
+
fallback**, over the channel from step 0:
|
|
118
|
+
- **`gh` channel:** if the cwd's `git remote` already IS `retro_repo`,
|
|
119
|
+
branch `retro/<DDMMYY>` from the default branch, add the report as
|
|
120
|
+
`retro/incoming/<DDMMYY>-<project>-report.md`, push, `gh pr create`.
|
|
121
|
+
Otherwise shallow-clone `retro_repo` into a temp dir and do the same
|
|
122
|
+
there. If push or PR creation fails (e.g. no write access), fall back to
|
|
123
|
+
`gh issue create -R <retro_repo>` with the full report as the body.
|
|
124
|
+
- **`mcp` channel:** same shape with the MCP's branch/file/PR tools;
|
|
125
|
+
fallback its create-issue tool.
|
|
126
|
+
- PR/issue title: `orc-retro: <DDMMYY> — <n> runs, <k> recommendations`.
|
|
127
|
+
- Either way, end by showing the user the created PR/issue URL. If delivery
|
|
128
|
+
itself errors after the preflight passed, surface the error verbatim and
|
|
129
|
+
point at the local report copy — never claim it was filed.
|
|
130
|
+
|
|
131
|
+
## Foreign dispatches — the `EXTRA` verb (v0.50.0)
|
|
132
|
+
|
|
133
|
+
A task can execute on a **non-Claude worker** (`_shared/extra-dispatch.md`). When
|
|
134
|
+
it does, the trace carries an `EXTRA` line per dispatch and **no `SPAWN` /
|
|
135
|
+
`RETURN` at all** — a foreign worker is not a Claude subagent, so the hook has
|
|
136
|
+
nothing to observe (P7, the `/orc-quick` ad-hoc-recon precedent).
|
|
137
|
+
|
|
138
|
+
**Read `EXTRA` or every foreign dispatch reads as a MISSING RETURN.** That is
|
|
139
|
+
the concrete failure this section exists to prevent: a leak count inflated by
|
|
140
|
+
work that completed perfectly well somewhere else. A foreign dispatch also
|
|
141
|
+
contributes nothing to NARRATION COVERAGE, and must not be counted against it.
|
|
142
|
+
|
|
143
|
+
Do not parse the lines yourself — **run `orc extra stats --json`** and report
|
|
144
|
+
what it computed (the `computeWikiFreshness` rule: one engine, and the skill is
|
|
145
|
+
not it). It groups **per profile per band**, which is the pair a routing decision
|
|
146
|
+
was actually made in: a per-provider total cannot tell you the `[0,30)` row was
|
|
147
|
+
fine and the `[30,70)` row was a false economy.
|
|
148
|
+
|
|
149
|
+
Report four things beside the ordinary per-band table:
|
|
150
|
+
|
|
151
|
+
| what | why it is its own number |
|
|
152
|
+
|---|---|
|
|
153
|
+
| outcome mix per band | `done` / `partial` / `failed` / `fallback`. The fallback rate IS the answer to "did routing this band off Claude work" |
|
|
154
|
+
| **SUBSTITUTION** count | the endpoint answered with a **different model**. Never aggregate this into a failure rate — the dispatch may have succeeded; you got something you did not ask for |
|
|
155
|
+
| **REROUTE** count | the model id held and a **different company** served it. Only engine `api` can see this at all; on the other two engines the absence of reroutes is **not** evidence there were none |
|
|
156
|
+
| repairs AFTER a foreign dispatch | fix cycles, `TDD-RED` iterations, reviewer P0/P1 and `REPLAN` lines whose task id matches a foreign task. **This is the whole point.** A run that cost a tenth as much and then needed two repair rounds was not cheaper |
|
|
157
|
+
|
|
158
|
+
**`tok=none` is a real value and must never be averaged as zero.** Engine `cli`
|
|
159
|
+
often reports no token counts; `orc extra stats` reports the vector plus *how
|
|
160
|
+
many dispatches it came from*, and the retro must carry that denominator through.
|
|
161
|
+
A cost total assembled from six of ten dispatches is not that band's cost.
|
|
162
|
+
|
|
163
|
+
**A dollar figure only where a rate exists.** Every `models` map in the shipped
|
|
164
|
+
price table is EMPTY on purpose (`orc extra rates` explains why and prints the
|
|
165
|
+
JSON to paste), so `usd: null` is the normal state and is reported as an em dash,
|
|
166
|
+
never as zero and never as an estimate.
|
|
167
|
+
|
|
168
|
+
**Never rank providers by quality.** `/orc-retro` reports outcomes and lets the
|
|
169
|
+
user decide — a benchmark ORC ran itself would be a benchmark ORC was motivated
|
|
170
|
+
to like.
|
|
171
|
+
|
|
172
|
+
## Report format (AI-readable — the PR/issue payload)
|
|
173
|
+
|
|
174
|
+
The report is written so the ORC repo's maintainer OR an AI session reading
|
|
175
|
+
the repo can act on it without parsing prose. YAML frontmatter mirrors the
|
|
176
|
+
return contract EXACTLY (machine layer), followed by short human sections:
|
|
177
|
+
|
|
178
|
+
```markdown
|
|
179
|
+
---
|
|
180
|
+
schema: orc-retro/v1
|
|
181
|
+
generated: <ISO date>
|
|
182
|
+
project: <cwd project name>
|
|
183
|
+
orc_version: <installed ORC version if known, else unknown>
|
|
184
|
+
runs_analyzed: <n>
|
|
185
|
+
tasks_analyzed: <n>
|
|
186
|
+
lane_stats: [...] # verbatim from the return contract
|
|
187
|
+
narration_coverage: {...}
|
|
188
|
+
band_stats: [...]
|
|
189
|
+
downgrades: [...]
|
|
190
|
+
leaks: [...]
|
|
191
|
+
extra_stats: {...} # `orc extra stats --json` verbatim, or null when no EXTRA line exists
|
|
192
|
+
recommendations: [...] # each with finding, suggested_change, confidence
|
|
193
|
+
actual_model: <...>
|
|
194
|
+
actual_effort: <...>
|
|
195
|
+
---
|
|
196
|
+
## Verdicts (the three questions, one line each)
|
|
197
|
+
## Per-band table
|
|
198
|
+
## Recommendations (one subsection each: evidence lines, suggested edit, confidence + n)
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
## Return contract (the agent emits EXACTLY this; you validate)
|
|
202
|
+
|
|
203
|
+
- `runs_analyzed`, `tasks_analyzed` — the n behind everything
|
|
204
|
+
- `lane_stats[]` — {lane, runs, tasks, unfinished} (lane from the filename)
|
|
205
|
+
- `narration_coverage` — {phases_total, phases_narrated, pct, unnarrated[]:
|
|
206
|
+
{run, role_family, first_agent}} — from the hook's `PHASE-EDGE` segmentation
|
|
207
|
+
- `band_stats[]` — {band, model, tasks, avg_retries, avg_requeues,
|
|
208
|
+
avg_needs_context, avg_unmet}
|
|
209
|
+
- `downgrades[]` — {agent, expected, actual, run}
|
|
210
|
+
- `leaks[]` — {kind: question-cluster | context-gap | replan | verdict-fail |
|
|
211
|
+
unfinished-run | hygiene, evidence (trace file + line numbers), count}
|
|
212
|
+
- `recommendations[]` — {finding, suggested_change (which file/table a human
|
|
213
|
+
would edit — e.g. an effort-and-mode.md facet weight (mined from the SCORE
|
|
214
|
+
line's `facets=` vector) or a band boundary in
|
|
215
|
+
config.md), confidence: strong|weak (weak when n<3)}
|
|
216
|
+
- `actual_model` — quoted VERBATIM from the system prompt's "The exact model
|
|
217
|
+
ID is …" line (`unknown` if absent, never guessed)
|
|
218
|
+
- `actual_effort` — `$CLAUDE_EFFORT`
|
|
219
|
+
|
|
220
|
+
Malformed = failure (re-dispatch once, then surface). A recommendation without
|
|
221
|
+
evidence line references is dropped — the retro obeys the same
|
|
222
|
+
evidence-or-advisory rule it audits.
|
|
223
|
+
|
|
224
|
+
## Config
|
|
225
|
+
|
|
226
|
+
**ONE resolver, and it is not you:** `orc lane config orc-retro --json`. Obey
|
|
227
|
+
`effective`, print every line in `announce[]` VERBATIM at preflight, and honour
|
|
228
|
+
`stops[]` before wave 1. Never re-derive a value, a precedence or an inertness
|
|
229
|
+
from `.claude/orc.config.yaml` — a key this lane does not read is not in the
|
|
230
|
+
answer, and a key another key shadows comes back already marked. Exit ≠ 0 → say
|
|
231
|
+
the CLI is unavailable and fall back to `../_shared/config-precedence.md`'s
|
|
232
|
+
documented defaults, out loud. Priorities and families:
|
|
233
|
+
`../_shared/config-precedence.md`.
|
|
234
|
+
|
|
235
|
+
## Calls
|
|
236
|
+
|
|
237
|
+
**ONE catalogue, and it is not you:** `orc lane calls orc-retro --json` names every
|
|
238
|
+
CLI call this lane makes, each with its exit-code contract, its cost, when to run
|
|
239
|
+
it, and what an EMPTY answer means. Never invent a spelling, never re-word an
|
|
240
|
+
exit code, and never re-derive a state word — the CLI's state words are the only
|
|
241
|
+
state words, and **an exit code is an ANSWER wherever that contract says so, not
|
|
242
|
+
a failure**. A call the answer does not name is a call this lane does not make.
|
|
243
|
+
Exit ≠ 0 from the catalogue itself → say the CLI is unavailable and name the
|
|
244
|
+
command you are about to run, out loud, before running it.
|
|
245
|
+
|
|
246
|
+
## Waiting mid-run (`/orc-wait`)
|
|
247
|
+
|
|
248
|
+
Canonical: `../_shared/wait.md`. **`a lane that waits without a hand-back` has broken this contract.**
|
|
249
|
+
Checkpoint **none** · safe point **read-only, seconds long**. Nothing here to checkpoint, so all three modes behave identically — say so rather than asking. Never begin a wait between a dispatch and its validated return, or before the smoke gate has reported.
|