@azure-id/orc 0.56.1 → 1.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (162) hide show
  1. package/CHANGELOG.md +219 -0
  2. package/README-id.md +49 -107
  3. package/README.md +694 -719
  4. package/bin/build-agents.js +18 -11
  5. package/bin/cli.js +33699 -29855
  6. package/bin/onboarding-content.js +0 -3
  7. package/bin/test-run.js +491 -0
  8. package/bin/verify-contracts.js +4330 -3277
  9. package/bin/verify-package.js +76 -6
  10. package/bin/webui/api.js +1201 -1163
  11. package/bin/webui/app.html +210 -198
  12. package/bin/webui/css/06-responsive.css +120 -108
  13. package/bin/webui/css/panels/extra.css +29 -0
  14. package/bin/webui/css/panels/lanes.css +134 -0
  15. package/bin/webui/css/panels/settings.css +79 -3
  16. package/bin/webui/css/panels/wait.css +123 -0
  17. package/bin/webui/fixtures/extra.js +2036 -1934
  18. package/bin/webui/fixtures/flow.js +2 -2
  19. package/bin/webui/fixtures/index.js +510 -486
  20. package/bin/webui/fixtures/lanes.js +218 -0
  21. package/bin/webui/fixtures/settings.js +261 -34
  22. package/bin/webui/fixtures/stats.js +3 -3
  23. package/bin/webui/fixtures/wait.js +97 -0
  24. package/bin/webui/i18n/TERMS.md +135 -134
  25. package/bin/webui/i18n/en/extra.json +345 -313
  26. package/bin/webui/i18n/en/lanes.json +29 -0
  27. package/bin/webui/i18n/en/nav.json +21 -19
  28. package/bin/webui/i18n/en/overview.json +2 -0
  29. package/bin/webui/i18n/en/settings.json +16 -5
  30. package/bin/webui/i18n/en/wait.json +41 -0
  31. package/bin/webui/i18n/id/extra.json +345 -313
  32. package/bin/webui/i18n/id/lanes.json +29 -0
  33. package/bin/webui/i18n/id/nav.json +21 -19
  34. package/bin/webui/i18n/id/overview.json +2 -0
  35. package/bin/webui/i18n/id/settings.json +15 -4
  36. package/bin/webui/i18n/id/wait.json +41 -0
  37. package/bin/webui/js/01-i18n.js +151 -149
  38. package/bin/webui/js/panels/extra.js +3199 -3011
  39. package/bin/webui/js/panels/lanes.js +259 -0
  40. package/bin/webui/js/panels/overview.js +10 -0
  41. package/bin/webui/js/panels/settings.js +149 -3
  42. package/bin/webui/js/panels/wait.js +253 -0
  43. package/package.json +39 -39
  44. package/templates/agents/MODEL-MAPPING.md +23 -35
  45. package/templates/agents/orc-executor-opus-4-7-high.md +1 -1
  46. package/templates/agents/orc-executor-opus-4-7-med.md +1 -1
  47. package/templates/agents/orc-executor-opus-4-8-high.md +1 -1
  48. package/templates/agents/orc-executor-opus-5-high.md +1 -1
  49. package/templates/agents/orc-executor-opus-5-low.md +1 -1
  50. package/templates/agents/orc-executor-opus-5-med.md +1 -1
  51. package/templates/agents/orc-retro-opus-5-med.md +73 -73
  52. package/templates/agents/orc-retro-sonnet-5-high.md +72 -72
  53. package/templates/agents/orc-trace-writer-haiku-4-5.md +107 -107
  54. package/templates/commands/orc-route.md +1 -1
  55. package/templates/commands/orc-wait.md +19 -0
  56. package/templates/commands/orc.md +1 -1
  57. package/templates/hooks/orc-statusline.js +39 -0
  58. package/templates/skills/_shared/README.md +7 -3
  59. package/templates/skills/_shared/config-precedence.md +198 -0
  60. package/templates/skills/_shared/drift-recovery.md +1 -1
  61. package/templates/skills/_shared/extra-dispatch.md +159 -7
  62. package/templates/skills/_shared/interview.md +1 -1
  63. package/templates/skills/_shared/opus5-only.md +9 -7
  64. package/templates/skills/_shared/phases/README.md +82 -0
  65. package/templates/skills/{orc/references → _shared/phases}/analyst-gates.md +15 -3
  66. package/templates/skills/_shared/phases/execution.md +143 -0
  67. package/templates/skills/{orc/references → _shared/phases}/house-rules.md +13 -1
  68. package/templates/skills/{orc/references → _shared/phases}/intake.md +15 -2
  69. package/templates/skills/_shared/phases/mock-example.md +56 -0
  70. package/templates/skills/{orc/references → _shared/phases}/plan-handoff.md +16 -3
  71. package/templates/skills/_shared/phases/planning.md +135 -0
  72. package/templates/skills/_shared/phases/preflight.md +98 -0
  73. package/templates/skills/_shared/phases/review.md +53 -0
  74. package/templates/skills/_shared/phases/scoring.md +65 -0
  75. package/templates/skills/{orc/references → _shared/phases}/security-checklist.md +12 -1
  76. package/templates/skills/_shared/phases/security.md +55 -0
  77. package/templates/skills/_shared/phases/ship.md +88 -0
  78. package/templates/skills/{orc/references/stop-and-resume.md → _shared/phases/stop-resume.md} +169 -160
  79. package/templates/skills/_shared/phases/summary.md +48 -0
  80. package/templates/skills/_shared/phases/testgen.md +51 -0
  81. package/templates/skills/{orc/references/trace-protocol.md → _shared/phases/trace.md} +49 -6
  82. package/templates/skills/_shared/phases/verify.md +70 -0
  83. package/templates/skills/{orc/references → _shared/phases}/wave-grouping.md +14 -2
  84. package/templates/skills/{orc/references → _shared/phases}/wiki-consult.md +13 -1
  85. package/templates/skills/_shared/read-ladder.md +48 -0
  86. package/templates/skills/_shared/wait.md +240 -0
  87. package/templates/skills/context-combiner/SKILL.md +214 -187
  88. package/templates/skills/orc/README.md +148 -150
  89. package/templates/skills/orc/SKILL.md +75 -358
  90. package/templates/skills/orc/config.md +137 -389
  91. package/templates/skills/orc/examples/full-run-mock.md +1 -1
  92. package/templates/skills/orc/references/effort-and-mode.md +16 -18
  93. package/templates/skills/orc/references/phases/intake.md +47 -0
  94. package/templates/skills/orc/references/phases/integration.md +19 -0
  95. package/templates/skills/orc/references/preflight-report.md +4 -4
  96. package/templates/skills/orc/references/ultra-mode.md +117 -123
  97. package/templates/skills/orc/schemas/planning-output.md +279 -279
  98. package/templates/skills/orc/subskills/orc-planner/SKILL.md +260 -264
  99. package/templates/skills/orc/subskills/orc-pr/stack-gate.md +4 -1
  100. package/templates/skills/orc-aftermath/SKILL.md +158 -136
  101. package/templates/skills/orc-analyze/SKILL.md +252 -220
  102. package/templates/skills/orc-analyze/references/branching.md +51 -51
  103. package/templates/skills/orc-analyze/references/thin-input.md +1 -1
  104. package/templates/skills/orc-analyze-mini/SKILL.md +128 -112
  105. package/templates/skills/orc-boundary/SKILL.md +249 -208
  106. package/templates/skills/orc-brainstorm/SKILL.md +377 -339
  107. package/templates/skills/orc-budget/SKILL.md +248 -225
  108. package/templates/skills/orc-challenge/README.md +1 -1
  109. package/templates/skills/orc-challenge/SKILL.md +298 -255
  110. package/templates/skills/orc-challenge/examples/council-full-roster.md +1 -1
  111. package/templates/skills/orc-claude/SKILL.md +221 -199
  112. package/templates/skills/orc-diy/README.md +2 -1
  113. package/templates/skills/orc-diy/SKILL.md +132 -68
  114. package/templates/skills/orc-diy/references/blocks/analyze.md +1 -1
  115. package/templates/skills/orc-diy/references/compile.md +21 -7
  116. package/templates/skills/orc-diy/references/flow-schema.md +2 -2
  117. package/templates/skills/orc-doc/SKILL.md +49 -7
  118. package/templates/skills/orc-explain/SKILL.md +24 -0
  119. package/templates/skills/orc-export/SKILL.md +31 -5
  120. package/templates/skills/orc-fast/SKILL.md +218 -185
  121. package/templates/skills/orc-grill/SKILL.md +238 -207
  122. package/templates/skills/orc-handoff/SKILL.md +223 -204
  123. package/templates/skills/orc-learn/SKILL.md +181 -156
  124. package/templates/skills/orc-mini/SKILL.md +47 -24
  125. package/templates/skills/orc-pact/SKILL.md +255 -218
  126. package/templates/skills/orc-pattern/SKILL.md +154 -124
  127. package/templates/skills/orc-poly/SKILL.md +242 -216
  128. package/templates/skills/orc-poly/references/gather.md +1 -1
  129. package/templates/skills/orc-pr-driver/SKILL.md +196 -167
  130. package/templates/skills/orc-pr-driver/references/orc-run-split.md +1 -1
  131. package/templates/skills/orc-pr-setup/SKILL.md +212 -184
  132. package/templates/skills/orc-quick/README.md +2 -2
  133. package/templates/skills/orc-quick/SKILL.md +51 -13
  134. package/templates/skills/orc-quick/references/dispatch-gate.md +2 -3
  135. package/templates/skills/orc-retro/SKILL.md +249 -222
  136. package/templates/skills/orc-retro/examples/retro-mock.md +171 -171
  137. package/templates/skills/orc-route/SKILL.md +188 -165
  138. package/templates/skills/orc-verify/SKILL.md +103 -84
  139. package/templates/skills/orc-wait/SKILL.md +163 -0
  140. package/templates/skills/orc-wiki/SKILL.md +60 -206
  141. package/templates/skills/orc-wiki/references/phases/phase-0.md +71 -0
  142. package/templates/skills/orc-wiki/references/phases/phase-1.md +35 -0
  143. package/templates/skills/orc-wiki/references/phases/phase-2.md +52 -0
  144. package/templates/skills/orc-wiki/references/phases/phase-3.md +57 -0
  145. package/templates/skills/orc-wiki/references/phases/phase-3c.md +36 -0
  146. package/templates/agents/orc-advisor-fable-5.md +0 -50
  147. package/templates/agents/orc-analyst-fable-5.md +0 -115
  148. package/templates/agents/orc-judge-fable-5.md +0 -79
  149. package/templates/agents/orc-planner-fable-5.md +0 -152
  150. package/templates/agents/orc-reviewer-fable-5.md +0 -57
  151. package/templates/skills/_shared/fable5-override.md +0 -56
  152. package/templates/skills/orc-diy/references/blocks/execution.md +0 -42
  153. package/templates/skills/orc-diy/references/blocks/mock-example.md +0 -22
  154. package/templates/skills/orc-diy/references/blocks/planning.md +0 -34
  155. package/templates/skills/orc-diy/references/blocks/review.md +0 -18
  156. package/templates/skills/orc-diy/references/blocks/scoring.md +0 -16
  157. package/templates/skills/orc-diy/references/blocks/security.md +0 -25
  158. package/templates/skills/orc-diy/references/blocks/ship.md +0 -25
  159. package/templates/skills/orc-diy/references/blocks/summary.md +0 -18
  160. package/templates/skills/orc-diy/references/blocks/testgen.md +0 -17
  161. package/templates/skills/orc-diy/references/blocks/trace.md +0 -28
  162. package/templates/skills/orc-diy/references/blocks/verify.md +0 -25
@@ -1,222 +1,249 @@
1
- ---
2
- name: orc-retro
3
- description: >
4
- Retro miner for ORC — closes the behavior-trace flywheel. Use for
5
- "/orc-retro", "analyze the orc traces", "how well is orc scoring", or "what
6
- should we tune from the logged runs". Reads the persistent behavior traces
7
- in log_dir (behavior-trace logging is permanent — every run writes one), aggregates per-band
8
- outcomes (retries, requeues, needs_context, unmet, downgrades, findings),
9
- and produces a calibration report with recommendations. READ-ONLY and
10
- REPORT-ONLY against the local system: it never edits the rubric, the skills,
11
- or project code. The report is DELIVERED upstream — filed as a PR (issue
12
- fallback) to the ORC repo (`retro_repo` config, default azure-id/orc) in
13
- AI-readable markdown, via the gh CLI or a GitHub MCP. P0 preflight: if
14
- NEITHER delivery channel exists, the retro does not run at all. The
15
- orchestrator dispatches the mining to a subagent — it never mines itself.
16
- ---
17
-
18
- # ORC-RETRO (trace miner)
19
-
20
- The behavior trace exists "for post-hoc skill improvement" — this skill is the
21
- return spoke of that flywheel. It turns the raw `.txt` traces into an answer to
22
- three questions: **is the scoring rubric calibrated? are the workers honest?
23
- where does the pipeline leak?**
24
-
25
- Run as Opus 4.8 high (orchestrator). The mining itself is dispatched to
26
- `orc-retro-sonnet-5-high` — cheap, because it reads trace text, not code — or
27
- to `orc-retro-opus-5-med` when `opus5_only: true` forces it
28
- (`../_shared/opus5-only.md`).
29
-
30
- **Worked example** (orient only — never execute from it): `examples/retro-mock.md`.
31
-
32
- ## Hard rules
33
-
34
- 0. **P0 preflight — a delivery channel or no retro at all.** The report exists
35
- to land in the ORC repo's PRs/issues where the maintainer (or an AI reading
36
- the repo) can act on it; a retro that can't deliver is pointless. BEFORE
37
- resolving traces or dispatching anything, probe in order:
38
- - **gh CLI:** `gh --version` succeeds AND `gh auth status` reports a logged-in
39
- account → channel is `gh`.
40
- - **GitHub MCP:** otherwise, check the session's available tools for a GitHub
41
- MCP server (tool names like `mcp__github__*` / create_pull_request /
42
- create_issue) → channel is `mcp`.
43
- - **Neither → STOP.** Do not mine, do not spawn, do not write a local report.
44
- Tell the user: install + auth the gh CLI (`gh auth login`) or connect a
45
- GitHub MCP server, then re-run `/orc-retro`.
46
- 1. **Read-only, report-only against the local system.** Never edit
47
- `effort-and-mode.md`, any skill, config, or project code. Recommendations
48
- are phrased for a HUMAN (or the ORC repo's AI) to apply; the retro never
49
- self-tunes the system it measures. Its ONLY write outside `log_dir` is the
50
- upstream PR/issue delivery below.
51
- 2. **You never mine yourself — you spawn.** Dispatch the retro agent with the
52
- trace file list; you validate the return and write the report.
53
- 3. **No traces → say so and stop.** Requires ORC runs to have happened (logging
54
- is permanent, so any past run left a trace): resolve `log_dir`
55
- (`../orc/config.md` default + `.claude/orc.config.yaml`) and list its `*.txt`.
56
- Empty → tell the user no runs have been traced yet and stop. Never invent
57
- findings from zero data.
58
- 4. **This lane does NOT write a trace of its own.** It is the reader of the
59
- trace system, not a run — it writes no run pointer and emits no markers
60
- (tracing the retro would pollute the very data it mines).
61
- 5. **Small-sample honesty:** every aggregate states its n. A recommendation
62
- from n<3 runs is labeled "weak signal — gather more runs", never stated as
63
- a conclusion.
64
-
65
- ## Procedure
66
-
67
- 0. **Preflight (hard rule 0):** establish the delivery channel (`gh` or `mcp`).
68
- No channel → stop here. Resolve `retro_repo` with the other config keys.
69
- 1. Resolve `log_dir`; collect `*.txt` traces (all, or the user-named subset /
70
- date range from `$ARGUMENTS`). Show the count and ask nothing else.
71
- 2. Dispatch `orc-retro-sonnet-5-high` (or `orc-retro-opus-5-med` under
72
- `opus5_only`) with the slice: trace file paths + the
73
- verb reference (`../orc/references/trace-protocol.md`). The agent mines the
74
- `<trace>.jsonl` sidecar first when present (structured — no regex over free
75
- text) and falls back to `.txt` parsing for pre-v0.32.0 traces, merging the
76
- hook's `.txt`-only skeleton lines by timestamp. It parses the CLOSED verb set
77
- and aggregates:
78
- - **Band calibration** (from `OUTCOME` lines): per band — task count, avg
79
- retries/requeues/needs_context/unmet. High retries in a band = the band's
80
- model is too weak (or slices too big); all-zeros in a high band = maybe
81
- over-tiered (cost leak).
82
- - **Tier integrity** (from `VERIFY` lines): every `⛔ DOWNGRADE`, grouped by
83
- agent — the "main session below Opus" bug leaves this exact fingerprint.
84
- - **Gate bounce rates** (from `GATE` lines): per gate name (grounding /
85
- coverage / graph / evidence / derivation) — pass vs bounce counts. A high
86
- bounce rate on one gate localizes which role's instructions leak (e.g.
87
- planner orphans → planner coverage discipline needs tuning).
88
- - **Per-lane aggregation** (free, from the filename grammar
89
- `run-<lane>-<slug>-<DDMMYY>-<HHMMSS>.txt`): runs/tasks/unfinished per lane,
90
- so an orc run is never averaged together with a mini or fast one.
91
- - **Pipeline leaks:** `QUESTION`/`CONTEXT-GAP` clusters (over-asking, slices
92
- missing context), `REPLAN` reasons, `FINDING p0..p3` and `VERDICT fail`
93
- rates per run, runs with `SPAWN`s but no `FINISH` (aborted/never closed).
94
- - **Trace hygiene → narration coverage:** the hook's `PHASE-EDGE` lines
95
- segment every run with zero model cooperation, so a missing narration is
96
- now DETERMINISTICALLY visible: count the phases whose edge-interval
97
- contains no trace-writer `SPAWN`. Report `covered/total` + the unnarrated
98
- phases. The question is no longer "were rich markers forgotten?" but
99
- "which phases never dispatched their writer?" — a run with edges and zero
100
- writer spawns is a total narration failure and is named as such.
101
- 2a. **Repair memory as calibration input (READ-ONLY).** Probe `orc gotcha status`
102
- (exit 0 = entries, 1 = none — never a `find`). On a hit, add
103
- `.claude/orc/gotchas.md` to the agent's slice as one more read-only source:
104
- a `kind`/`area` cluster is evidence that a band, a slice size or a playbook is
105
- under-serving that area, and a high-`hits` entry is a repair this project keeps
106
- re-paying for. Cite the entry IDs in the recommendation's evidence like any
107
- other line count. **This lane NEVER writes, prunes, or edits a gotcha** — it is
108
- report-only against the local system by hard rule 2, and repair memory is no
109
- exception. Canonical: `../_shared/gotchas.md`.
110
- 3. Validate the return (contract below). Write the report to
111
- `log_dir/retro/<DDMMYY>-report.md` (the `retro/` subfolder keeps the trace
112
- folder's top level to run traces + their sidecars) in the format below, and
113
- show the user the summary: verdict per question, the per-band table, and
114
- each recommendation with its evidence line counts + n.
115
- 4. **Deliver upstream (the point of the retro).** File the report to
116
- `retro_repo` (config, default `azure-id/orc`) — **PR preferred, issue
117
- fallback**, over the channel from step 0:
118
- - **`gh` channel:** if the cwd's `git remote` already IS `retro_repo`,
119
- branch `retro/<DDMMYY>` from the default branch, add the report as
120
- `retro/incoming/<DDMMYY>-<project>-report.md`, push, `gh pr create`.
121
- Otherwise shallow-clone `retro_repo` into a temp dir and do the same
122
- there. If push or PR creation fails (e.g. no write access), fall back to
123
- `gh issue create -R <retro_repo>` with the full report as the body.
124
- - **`mcp` channel:** same shape with the MCP's branch/file/PR tools;
125
- fallback its create-issue tool.
126
- - PR/issue title: `orc-retro: <DDMMYY> — <n> runs, <k> recommendations`.
127
- - Either way, end by showing the user the created PR/issue URL. If delivery
128
- itself errors after the preflight passed, surface the error verbatim and
129
- point at the local report copy — never claim it was filed.
130
-
131
- ## Foreign dispatches — the `EXTRA` verb (v0.50.0)
132
-
133
- A task can execute on a **non-Claude worker** (`_shared/extra-dispatch.md`). When
134
- it does, the trace carries an `EXTRA` line per dispatch and **no `SPAWN` /
135
- `RETURN` at all** — a foreign worker is not a Claude subagent, so the hook has
136
- nothing to observe (P7, the `/orc-quick` ad-hoc-recon precedent).
137
-
138
- **Read `EXTRA` or every foreign dispatch reads as a MISSING RETURN.** That is
139
- the concrete failure this section exists to prevent: a leak count inflated by
140
- work that completed perfectly well somewhere else. A foreign dispatch also
141
- contributes nothing to NARRATION COVERAGE, and must not be counted against it.
142
-
143
- Do not parse the lines yourself — **run `orc extra stats --json`** and report
144
- what it computed (the `computeWikiFreshness` rule: one engine, and the skill is
145
- not it). It groups **per profile per band**, which is the pair a routing decision
146
- was actually made in: a per-provider total cannot tell you the `[0,30)` row was
147
- fine and the `[30,70)` row was a false economy.
148
-
149
- Report four things beside the ordinary per-band table:
150
-
151
- | what | why it is its own number |
152
- |---|---|
153
- | outcome mix per band | `done` / `partial` / `failed` / `fallback`. The fallback rate IS the answer to "did routing this band off Claude work" |
154
- | **SUBSTITUTION** count | the endpoint answered with a **different model**. Never aggregate this into a failure rate — the dispatch may have succeeded; you got something you did not ask for |
155
- | **REROUTE** count | the model id held and a **different company** served it. Only engine `api` can see this at all; on the other two engines the absence of reroutes is **not** evidence there were none |
156
- | repairs AFTER a foreign dispatch | fix cycles, `TDD-RED` iterations, reviewer P0/P1 and `REPLAN` lines whose task id matches a foreign task. **This is the whole point.** A run that cost a tenth as much and then needed two repair rounds was not cheaper |
157
-
158
- **`tok=none` is a real value and must never be averaged as zero.** Engine `cli`
159
- often reports no token counts; `orc extra stats` reports the vector plus *how
160
- many dispatches it came from*, and the retro must carry that denominator through.
161
- A cost total assembled from six of ten dispatches is not that band's cost.
162
-
163
- **A dollar figure only where a rate exists.** Every `models` map in the shipped
164
- price table is EMPTY on purpose (`orc extra rates` explains why and prints the
165
- JSON to paste), so `usd: null` is the normal state and is reported as an em dash,
166
- never as zero and never as an estimate.
167
-
168
- **Never rank providers by quality.** `/orc-retro` reports outcomes and lets the
169
- user decide — a benchmark ORC ran itself would be a benchmark ORC was motivated
170
- to like.
171
-
172
- ## Report format (AI-readable — the PR/issue payload)
173
-
174
- The report is written so the ORC repo's maintainer OR an AI session reading
175
- the repo can act on it without parsing prose. YAML frontmatter mirrors the
176
- return contract EXACTLY (machine layer), followed by short human sections:
177
-
178
- ```markdown
179
- ---
180
- schema: orc-retro/v1
181
- generated: <ISO date>
182
- project: <cwd project name>
183
- orc_version: <installed ORC version if known, else unknown>
184
- runs_analyzed: <n>
185
- tasks_analyzed: <n>
186
- lane_stats: [...] # verbatim from the return contract
187
- narration_coverage: {...}
188
- band_stats: [...]
189
- downgrades: [...]
190
- leaks: [...]
191
- extra_stats: {...} # `orc extra stats --json` verbatim, or null when no EXTRA line exists
192
- recommendations: [...] # each with finding, suggested_change, confidence
193
- actual_model: <...>
194
- actual_effort: <...>
195
- ---
196
- ## Verdicts (the three questions, one line each)
197
- ## Per-band table
198
- ## Recommendations (one subsection each: evidence lines, suggested edit, confidence + n)
199
- ```
200
-
201
- ## Return contract (the agent emits EXACTLY this; you validate)
202
-
203
- - `runs_analyzed`, `tasks_analyzed` — the n behind everything
204
- - `lane_stats[]` — {lane, runs, tasks, unfinished} (lane from the filename)
205
- - `narration_coverage` — {phases_total, phases_narrated, pct, unnarrated[]:
206
- {run, role_family, first_agent}} — from the hook's `PHASE-EDGE` segmentation
207
- - `band_stats[]` — {band, model, tasks, avg_retries, avg_requeues,
208
- avg_needs_context, avg_unmet}
209
- - `downgrades[]` — {agent, expected, actual, run}
210
- - `leaks[]` — {kind: question-cluster | context-gap | replan | verdict-fail |
211
- unfinished-run | hygiene, evidence (trace file + line numbers), count}
212
- - `recommendations[]` — {finding, suggested_change (which file/table a human
213
- would edit — e.g. an effort-and-mode.md facet weight (mined from the SCORE
214
- line's `facets=` vector) or a band boundary in
215
- config.md), confidence: strong|weak (weak when n<3)}
216
- - `actual_model` — quoted VERBATIM from the system prompt's "The exact model
217
- ID is …" line (`unknown` if absent, never guessed)
218
- - `actual_effort` — `$CLAUDE_EFFORT`
219
-
220
- Malformed = failure (re-dispatch once, then surface). A recommendation without
221
- evidence line references is dropped — the retro obeys the same
222
- evidence-or-advisory rule it audits.
1
+ ---
2
+ name: orc-retro
3
+ description: >
4
+ Retro miner for ORC — closes the behavior-trace flywheel. Use for
5
+ "/orc-retro", "analyze the orc traces", "how well is orc scoring", or "what
6
+ should we tune from the logged runs". Reads the persistent behavior traces
7
+ in log_dir (behavior-trace logging is permanent — every run writes one), aggregates per-band
8
+ outcomes (retries, requeues, needs_context, unmet, downgrades, findings),
9
+ and produces a calibration report with recommendations. READ-ONLY and
10
+ REPORT-ONLY against the local system: it never edits the rubric, the skills,
11
+ or project code. The report is DELIVERED upstream — filed as a PR (issue
12
+ fallback) to the ORC repo (`retro_repo` config, default azure-id/orc) in
13
+ AI-readable markdown, via the gh CLI or a GitHub MCP. P0 preflight: if
14
+ NEITHER delivery channel exists, the retro does not run at all. The
15
+ orchestrator dispatches the mining to a subagent — it never mines itself.
16
+ ---
17
+
18
+ # ORC-RETRO (trace miner)
19
+
20
+ The behavior trace exists "for post-hoc skill improvement" — this skill is the
21
+ return spoke of that flywheel. It turns the raw `.txt` traces into an answer to
22
+ three questions: **is the scoring rubric calibrated? are the workers honest?
23
+ where does the pipeline leak?**
24
+
25
+ Run as Opus 4.8 high (orchestrator). The mining itself is dispatched to
26
+ `orc-retro-sonnet-5-high` — cheap, because it reads trace text, not code — or
27
+ to `orc-retro-opus-5-med` when `opus5_only: true` forces it
28
+ (`../_shared/opus5-only.md`).
29
+
30
+ **Worked example** (orient only — never execute from it): `examples/retro-mock.md`.
31
+
32
+ ## Hard rules
33
+
34
+ 0. **P0 preflight — a delivery channel or no retro at all.** The report exists
35
+ to land in the ORC repo's PRs/issues where the maintainer (or an AI reading
36
+ the repo) can act on it; a retro that can't deliver is pointless. BEFORE
37
+ resolving traces or dispatching anything, probe in order:
38
+ - **gh CLI:** `gh --version` succeeds AND `gh auth status` reports a logged-in
39
+ account → channel is `gh`.
40
+ - **GitHub MCP:** otherwise, check the session's available tools for a GitHub
41
+ MCP server (tool names like `mcp__github__*` / create_pull_request /
42
+ create_issue) → channel is `mcp`.
43
+ - **Neither → STOP.** Do not mine, do not spawn, do not write a local report.
44
+ Tell the user: install + auth the gh CLI (`gh auth login`) or connect a
45
+ GitHub MCP server, then re-run `/orc-retro`.
46
+ 1. **Read-only, report-only against the local system.** Never edit
47
+ `effort-and-mode.md`, any skill, config, or project code. Recommendations
48
+ are phrased for a HUMAN (or the ORC repo's AI) to apply; the retro never
49
+ self-tunes the system it measures. Its ONLY write outside `log_dir` is the
50
+ upstream PR/issue delivery below.
51
+ 2. **You never mine yourself — you spawn.** Dispatch the retro agent with the
52
+ trace file list; you validate the return and write the report.
53
+ 3. **No traces → say so and stop.** Requires ORC runs to have happened (logging
54
+ is permanent, so any past run left a trace): resolve `log_dir`
55
+ (`../orc/config.md` default + `.claude/orc.config.yaml`) and list its `*.txt`.
56
+ Empty → tell the user no runs have been traced yet and stop. Never invent
57
+ findings from zero data.
58
+ 4. **This lane does NOT write a trace of its own.** It is the reader of the
59
+ trace system, not a run — it writes no run pointer and emits no markers
60
+ (tracing the retro would pollute the very data it mines).
61
+ 5. **Small-sample honesty:** every aggregate states its n. A recommendation
62
+ from n<3 runs is labeled "weak signal — gather more runs", never stated as
63
+ a conclusion.
64
+
65
+ ## Procedure
66
+
67
+ 0. **Preflight (hard rule 0):** establish the delivery channel (`gh` or `mcp`).
68
+ No channel → stop here. Resolve `retro_repo` with the other config keys.
69
+ 1. Resolve `log_dir`; collect `*.txt` traces (all, or the user-named subset /
70
+ date range from `$ARGUMENTS`). Show the count and ask nothing else.
71
+ 2. Dispatch `orc-retro-sonnet-5-high` (or `orc-retro-opus-5-med` under
72
+ `opus5_only`) with the slice: trace file paths + the
73
+ verb reference (`../_shared/phases/trace.md`). The agent mines the
74
+ `<trace>.jsonl` sidecar first when present (structured — no regex over free
75
+ text) and falls back to `.txt` parsing for pre-v0.32.0 traces, merging the
76
+ hook's `.txt`-only skeleton lines by timestamp. It parses the CLOSED verb set
77
+ and aggregates:
78
+ - **Band calibration** (from `OUTCOME` lines): per band — task count, avg
79
+ retries/requeues/needs_context/unmet. High retries in a band = the band's
80
+ model is too weak (or slices too big); all-zeros in a high band = maybe
81
+ over-tiered (cost leak).
82
+ - **Tier integrity** (from `VERIFY` lines): every `⛔ DOWNGRADE`, grouped by
83
+ agent — the "main session below Opus" bug leaves this exact fingerprint.
84
+ - **Gate bounce rates** (from `GATE` lines): per gate name (grounding /
85
+ coverage / graph / evidence / derivation) — pass vs bounce counts. A high
86
+ bounce rate on one gate localizes which role's instructions leak (e.g.
87
+ planner orphans → planner coverage discipline needs tuning).
88
+ - **Per-lane aggregation** (free, from the filename grammar
89
+ `run-<lane>-<slug>-<DDMMYY>-<HHMMSS>.txt`): runs/tasks/unfinished per lane,
90
+ so an orc run is never averaged together with a mini or fast one.
91
+ - **Pipeline leaks:** `QUESTION`/`CONTEXT-GAP` clusters (over-asking, slices
92
+ missing context), `REPLAN` reasons, `FINDING p0..p3` and `VERDICT fail`
93
+ rates per run, runs with `SPAWN`s but no `FINISH` (aborted/never closed).
94
+ - **Trace hygiene → narration coverage:** the hook's `PHASE-EDGE` lines
95
+ segment every run with zero model cooperation, so a missing narration is
96
+ now DETERMINISTICALLY visible: count the phases whose edge-interval
97
+ contains no trace-writer `SPAWN`. Report `covered/total` + the unnarrated
98
+ phases. The question is no longer "were rich markers forgotten?" but
99
+ "which phases never dispatched their writer?" — a run with edges and zero
100
+ writer spawns is a total narration failure and is named as such.
101
+ 2a. **Repair memory as calibration input (READ-ONLY).** Probe `orc gotcha status`
102
+ (exit 0 = entries, 1 = none — never a `find`). On a hit, add
103
+ `.claude/orc/gotchas.md` to the agent's slice as one more read-only source:
104
+ a `kind`/`area` cluster is evidence that a band, a slice size or a playbook is
105
+ under-serving that area, and a high-`hits` entry is a repair this project keeps
106
+ re-paying for. Cite the entry IDs in the recommendation's evidence like any
107
+ other line count. **This lane NEVER writes, prunes, or edits a gotcha** — it is
108
+ report-only against the local system by hard rule 2, and repair memory is no
109
+ exception. Canonical: `../_shared/gotchas.md`.
110
+ 3. Validate the return (contract below). Write the report to
111
+ `log_dir/retro/<DDMMYY>-report.md` (the `retro/` subfolder keeps the trace
112
+ folder's top level to run traces + their sidecars) in the format below, and
113
+ show the user the summary: verdict per question, the per-band table, and
114
+ each recommendation with its evidence line counts + n.
115
+ 4. **Deliver upstream (the point of the retro).** File the report to
116
+ `retro_repo` (config, default `azure-id/orc`) — **PR preferred, issue
117
+ fallback**, over the channel from step 0:
118
+ - **`gh` channel:** if the cwd's `git remote` already IS `retro_repo`,
119
+ branch `retro/<DDMMYY>` from the default branch, add the report as
120
+ `retro/incoming/<DDMMYY>-<project>-report.md`, push, `gh pr create`.
121
+ Otherwise shallow-clone `retro_repo` into a temp dir and do the same
122
+ there. If push or PR creation fails (e.g. no write access), fall back to
123
+ `gh issue create -R <retro_repo>` with the full report as the body.
124
+ - **`mcp` channel:** same shape with the MCP's branch/file/PR tools;
125
+ fallback its create-issue tool.
126
+ - PR/issue title: `orc-retro: <DDMMYY> — <n> runs, <k> recommendations`.
127
+ - Either way, end by showing the user the created PR/issue URL. If delivery
128
+ itself errors after the preflight passed, surface the error verbatim and
129
+ point at the local report copy — never claim it was filed.
130
+
131
+ ## Foreign dispatches — the `EXTRA` verb (v0.50.0)
132
+
133
+ A task can execute on a **non-Claude worker** (`_shared/extra-dispatch.md`). When
134
+ it does, the trace carries an `EXTRA` line per dispatch and **no `SPAWN` /
135
+ `RETURN` at all** — a foreign worker is not a Claude subagent, so the hook has
136
+ nothing to observe (P7, the `/orc-quick` ad-hoc-recon precedent).
137
+
138
+ **Read `EXTRA` or every foreign dispatch reads as a MISSING RETURN.** That is
139
+ the concrete failure this section exists to prevent: a leak count inflated by
140
+ work that completed perfectly well somewhere else. A foreign dispatch also
141
+ contributes nothing to NARRATION COVERAGE, and must not be counted against it.
142
+
143
+ Do not parse the lines yourself — **run `orc extra stats --json`** and report
144
+ what it computed (the `computeWikiFreshness` rule: one engine, and the skill is
145
+ not it). It groups **per profile per band**, which is the pair a routing decision
146
+ was actually made in: a per-provider total cannot tell you the `[0,30)` row was
147
+ fine and the `[30,70)` row was a false economy.
148
+
149
+ Report four things beside the ordinary per-band table:
150
+
151
+ | what | why it is its own number |
152
+ |---|---|
153
+ | outcome mix per band | `done` / `partial` / `failed` / `fallback`. The fallback rate IS the answer to "did routing this band off Claude work" |
154
+ | **SUBSTITUTION** count | the endpoint answered with a **different model**. Never aggregate this into a failure rate — the dispatch may have succeeded; you got something you did not ask for |
155
+ | **REROUTE** count | the model id held and a **different company** served it. Only engine `api` can see this at all; on the other two engines the absence of reroutes is **not** evidence there were none |
156
+ | repairs AFTER a foreign dispatch | fix cycles, `TDD-RED` iterations, reviewer P0/P1 and `REPLAN` lines whose task id matches a foreign task. **This is the whole point.** A run that cost a tenth as much and then needed two repair rounds was not cheaper |
157
+
158
+ **`tok=none` is a real value and must never be averaged as zero.** Engine `cli`
159
+ often reports no token counts; `orc extra stats` reports the vector plus *how
160
+ many dispatches it came from*, and the retro must carry that denominator through.
161
+ A cost total assembled from six of ten dispatches is not that band's cost.
162
+
163
+ **A dollar figure only where a rate exists.** Every `models` map in the shipped
164
+ price table is EMPTY on purpose (`orc extra rates` explains why and prints the
165
+ JSON to paste), so `usd: null` is the normal state and is reported as an em dash,
166
+ never as zero and never as an estimate.
167
+
168
+ **Never rank providers by quality.** `/orc-retro` reports outcomes and lets the
169
+ user decide — a benchmark ORC ran itself would be a benchmark ORC was motivated
170
+ to like.
171
+
172
+ ## Report format (AI-readable — the PR/issue payload)
173
+
174
+ The report is written so the ORC repo's maintainer OR an AI session reading
175
+ the repo can act on it without parsing prose. YAML frontmatter mirrors the
176
+ return contract EXACTLY (machine layer), followed by short human sections:
177
+
178
+ ```markdown
179
+ ---
180
+ schema: orc-retro/v1
181
+ generated: <ISO date>
182
+ project: <cwd project name>
183
+ orc_version: <installed ORC version if known, else unknown>
184
+ runs_analyzed: <n>
185
+ tasks_analyzed: <n>
186
+ lane_stats: [...] # verbatim from the return contract
187
+ narration_coverage: {...}
188
+ band_stats: [...]
189
+ downgrades: [...]
190
+ leaks: [...]
191
+ extra_stats: {...} # `orc extra stats --json` verbatim, or null when no EXTRA line exists
192
+ recommendations: [...] # each with finding, suggested_change, confidence
193
+ actual_model: <...>
194
+ actual_effort: <...>
195
+ ---
196
+ ## Verdicts (the three questions, one line each)
197
+ ## Per-band table
198
+ ## Recommendations (one subsection each: evidence lines, suggested edit, confidence + n)
199
+ ```
200
+
201
+ ## Return contract (the agent emits EXACTLY this; you validate)
202
+
203
+ - `runs_analyzed`, `tasks_analyzed` — the n behind everything
204
+ - `lane_stats[]` — {lane, runs, tasks, unfinished} (lane from the filename)
205
+ - `narration_coverage` — {phases_total, phases_narrated, pct, unnarrated[]:
206
+ {run, role_family, first_agent}} — from the hook's `PHASE-EDGE` segmentation
207
+ - `band_stats[]` — {band, model, tasks, avg_retries, avg_requeues,
208
+ avg_needs_context, avg_unmet}
209
+ - `downgrades[]` — {agent, expected, actual, run}
210
+ - `leaks[]` — {kind: question-cluster | context-gap | replan | verdict-fail |
211
+ unfinished-run | hygiene, evidence (trace file + line numbers), count}
212
+ - `recommendations[]` — {finding, suggested_change (which file/table a human
213
+ would edit — e.g. an effort-and-mode.md facet weight (mined from the SCORE
214
+ line's `facets=` vector) or a band boundary in
215
+ config.md), confidence: strong|weak (weak when n<3)}
216
+ - `actual_model` — quoted VERBATIM from the system prompt's "The exact model
217
+ ID is …" line (`unknown` if absent, never guessed)
218
+ - `actual_effort` — `$CLAUDE_EFFORT`
219
+
220
+ Malformed = failure (re-dispatch once, then surface). A recommendation without
221
+ evidence line references is dropped — the retro obeys the same
222
+ evidence-or-advisory rule it audits.
223
+
224
+ ## Config
225
+
226
+ **ONE resolver, and it is not you:** `orc lane config orc-retro --json`. Obey
227
+ `effective`, print every line in `announce[]` VERBATIM at preflight, and honour
228
+ `stops[]` before wave 1. Never re-derive a value, a precedence or an inertness
229
+ from `.claude/orc.config.yaml` — a key this lane does not read is not in the
230
+ answer, and a key another key shadows comes back already marked. Exit ≠ 0 → say
231
+ the CLI is unavailable and fall back to `../_shared/config-precedence.md`'s
232
+ documented defaults, out loud. Priorities and families:
233
+ `../_shared/config-precedence.md`.
234
+
235
+ ## Calls
236
+
237
+ **ONE catalogue, and it is not you:** `orc lane calls orc-retro --json` names every
238
+ CLI call this lane makes, each with its exit-code contract, its cost, when to run
239
+ it, and what an EMPTY answer means. Never invent a spelling, never re-word an
240
+ exit code, and never re-derive a state word — the CLI's state words are the only
241
+ state words, and **an exit code is an ANSWER wherever that contract says so, not
242
+ a failure**. A call the answer does not name is a call this lane does not make.
243
+ Exit ≠ 0 from the catalogue itself → say the CLI is unavailable and name the
244
+ command you are about to run, out loud, before running it.
245
+
246
+ ## Waiting mid-run (`/orc-wait`)
247
+
248
+ Canonical: `../_shared/wait.md`. **`a lane that waits without a hand-back` has broken this contract.**
249
+ Checkpoint **none** · safe point **read-only, seconds long**. Nothing here to checkpoint, so all three modes behave identically — say so rather than asking. Never begin a wait between a dispatch and its validated return, or before the smoke gate has reported.