@azure-id/orc 1.4.2 → 1.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (38) hide show
  1. package/CHANGELOG.md +186 -0
  2. package/README.md +63 -50
  3. package/bin/cli.js +5732 -133
  4. package/bin/mockrun-catalog.js +4 -0
  5. package/bin/verify-contracts.js +4509 -4426
  6. package/bin/verify-package.js +24 -2
  7. package/bin/webui/api.js +108 -8
  8. package/bin/webui/app.html +9 -0
  9. package/bin/webui/css/06-responsive.css +14 -0
  10. package/bin/webui/css/panels/test.css +132 -0
  11. package/bin/webui/fixtures/index.js +23 -1
  12. package/bin/webui/fixtures/test.js +547 -0
  13. package/bin/webui/i18n/en/common.json +1 -0
  14. package/bin/webui/i18n/en/nav.json +1 -0
  15. package/bin/webui/i18n/en/overview.json +5 -0
  16. package/bin/webui/i18n/en/test.json +141 -0
  17. package/bin/webui/i18n/id/common.json +1 -0
  18. package/bin/webui/i18n/id/nav.json +1 -0
  19. package/bin/webui/i18n/id/overview.json +5 -0
  20. package/bin/webui/i18n/id/test.json +141 -0
  21. package/bin/webui/js/01-i18n.js +1 -0
  22. package/bin/webui/js/panels/challenge.js +8 -3
  23. package/bin/webui/js/panels/overview.js +18 -0
  24. package/bin/webui/js/panels/test.js +733 -0
  25. package/bin/webui/serve.js +572 -557
  26. package/mock-run/orc-test.md +327 -0
  27. package/package.json +1 -1
  28. package/templates/agents/MODEL-MAPPING.md +2 -0
  29. package/templates/agents/orc-test-designer-opus-5-high.md +158 -0
  30. package/templates/agents/orc-test-interpreter-opus-5-low.md +130 -0
  31. package/templates/commands/orc-test.md +66 -0
  32. package/templates/hooks/README.md +110 -1
  33. package/templates/hooks/orc-read-gate.js +279 -0
  34. package/templates/skills/_shared/extra-dispatch.md +1346 -1337
  35. package/templates/skills/_shared/live-target.md +188 -0
  36. package/templates/skills/_shared/phases/trace.md +2 -2
  37. package/templates/skills/_shared/wait.md +241 -240
  38. package/templates/skills/orc-test/SKILL.md +233 -0
@@ -0,0 +1,327 @@
1
+ # Mock run — `/orc-test`
2
+
3
+ > ORC runs the test against a real system, writes down what it saw, and never touches the system it is testing.
4
+
5
+ ---
6
+
7
+ ## 1. What it does
8
+
9
+ Every other testing tool in ORC **writes** tests. `/orc-test` **runs** them.
10
+
11
+ It points at a running API — yours, on your laptop, or one you were given
12
+ permission to touch — sends real requests, and writes down what came back. Then
13
+ it stops.
14
+
15
+ The split is the whole design:
16
+
17
+ > **The CLI executes and measures. The model designs and interprets.**
18
+ > A model never sends a request. The CLI never decides what a response means.
19
+
20
+ That split is what makes the report worth reading. A number in it was measured
21
+ by a program. A sentence in it was written by a model that only ever saw
22
+ evidence on disk.
23
+
24
+ Three rules hold it together, and each one is a contract with a name:
25
+
26
+ | Rule | What it means |
27
+ |---|---|
28
+ | `a lane that reports a result it did not observe` | Three verdicts: `pass`, `fail`, `unknown`. **`unknown` is the honest one.** A case nobody ran is never a pass. |
29
+ | `a lane that sends traffic to a target nobody authorized` | The target is frozen at the start, with a written statement of who said yes. Nothing outside that origin is ever contacted. |
30
+ | `a lane that fixes the system under test` | ORC never edits the thing it is testing. When the app is broken, it hands the machine back to you. |
31
+
32
+ ---
33
+
34
+ ## 2. Freeze the target
35
+
36
+ Nothing here has a default it could guess. There is no config key that names a
37
+ target, on purpose.
38
+
39
+ ```
40
+ $ orc test init api-users --kind be --env local \
41
+ --base-url http://127.0.0.1:4000 \
42
+ --authorized "me, on my own laptop, against my own dev database" \
43
+ --destructive deny --reason "this box shares a database with the seed script"
44
+ ```
45
+
46
+ ```
47
+ /orc-test · api-users
48
+ target FROZEN at orc/orc-test/api-users/target.md
49
+
50
+ kind be
51
+ environment local
52
+ base URL http://127.0.0.1:4000
53
+ origin fence http://127.0.0.1:4000
54
+ destructive deny — this box shares a database with the seed script
55
+
56
+ This folder is NEVER staged. It will hold real response bodies from a real
57
+ system. Add this line to .gitignore yourself — ORC does not edit it:
58
+ orc/orc-test/
59
+
60
+ Next: orc test surface api-users
61
+ ```
62
+
63
+ Two things to notice.
64
+
65
+ **The origin fence.** Every request the runner sends is checked against it. A
66
+ redirect that leaves the fence is not followed — it is recorded and dropped.
67
+
68
+ **`--authorized` is a sentence, not a checkbox.** ORC records it word for word
69
+ and never checks it. On a remote target it is required. It prints at the top of
70
+ every report, so the person reading the report can see whose permission this
71
+ was run on.
72
+
73
+ ---
74
+
75
+ ## 3. The free pass
76
+
77
+ The surface costs nothing. No model, no tokens.
78
+
79
+ ```
80
+ $ orc test surface api-users
81
+ ```
82
+
83
+ ```
84
+ /orc-test · api-users · surface
85
+ the free pass — zero model tokens. Written to orc/orc-test/api-users/surface.md
86
+
87
+ spec on disk openapi.yaml (4 routes)
88
+ spec at target http://127.0.0.1:4000/openapi.json
89
+ frameworks express
90
+ routes 5 (34 files scanned, 1 live-only)
91
+ mounted at /api/v1
92
+ unresolved 1
93
+
94
+ in code, not live 1 dead or unshipped
95
+ live, not in code 1 shadow / zombie APIs — OWASP API9:2023
96
+ GET /internal/debug
97
+ ```
98
+
99
+ `GET /internal/debug` answers at the target and is in no file in this
100
+ repository. Nobody reviews it. Nobody patches it. Finding that took zero model
101
+ tokens and one comparison of two documents — **no route was called to produce
102
+ it**.
103
+
104
+ If there had been no spec at the target, that line would read:
105
+
106
+ ```
107
+ code-vs-live diff NOT MEASURED — no spec answered at the target, so there
108
+ is nothing to compare the code against. This is NOT 'no shadow APIs' — it is
109
+ 'not measured'.
110
+ ```
111
+
112
+ That distinction is the lane in one sentence.
113
+
114
+ ---
115
+
116
+ ## 4. When the app is broken
117
+
118
+ ```
119
+ $ orc test env api-users
120
+ ```
121
+
122
+ ```
123
+ /orc-test · api-users · environment
124
+ state UNHEALTHY the process is up and answering wrong. ORC does not fix
125
+ the system it is testing.
126
+
127
+ detected npm run dev (package.json)
128
+ process pid 48211
129
+ missing STRIPE_SECRET_KEY, SESSION_SECRET
130
+
131
+ Error: password authentication failed for user "app"
132
+ ```
133
+
134
+ And then it stops.
135
+
136
+ It found the start command. It knows the process is alive. It can see the exact
137
+ line in the log. It will not open that file. It names the two environment keys
138
+ that are absent and **writes no value for either** — a placeholder written into
139
+ a real environment is a credential-shaped lie, and only you know the right
140
+ value.
141
+
142
+ You fix it. You run the command again. That is the whole loop.
143
+
144
+ ---
145
+
146
+ ## 5. The cases derive themselves
147
+
148
+ ```
149
+ $ orc test case derive api-users
150
+ ```
151
+
152
+ ```
153
+ /orc-test · api-users · cases
154
+ derived for free — zero model tokens
155
+
156
+ happy 2
157
+ edge 4
158
+ abuse 3
159
+ security 3
160
+ total 12
161
+
162
+ 2 cases carry a GAP — what a schema could not answer.
163
+ This is what the designer agent is for, and it is the only part that costs anything.
164
+
165
+ security tier safe (the shipped default)
166
+ 5 OWASP rows UNCHECKABLE — each keeps its slot, and none of them will ever become a pass
167
+ ```
168
+
169
+ The matrix comes out of the schema for free. The **gaps** are what a schema
170
+ cannot know — what a valid email looks like in this business, which order the
171
+ two calls have to happen in. Those go to the designer agent, and that is the
172
+ only part of this lane you pay for.
173
+
174
+ ---
175
+
176
+ ## 6. The security tier is a closed set
177
+
178
+ ```
179
+ $ orc test security api-users
180
+ ```
181
+
182
+ ```
183
+ /orc-test · api-users · security · OWASP API Top 10 (2023)
184
+ tier safe (the shipped default) · the set is CLOSED and is never extended ad hoc
185
+
186
+ API1 FOUND Broken object level authorization (BOLA)
187
+ severity critical OBSERVED, and the request crossed an identity boundary
188
+ C-006 replay a successful request as a SECOND identity
189
+ orc/orc-test/api-users/runs/03/evidence/C-006/
190
+ API2 observed-clean Broken authentication
191
+ API5 unchecked Broken function level authorization (BFLA)
192
+ only one ROLE is declared. BFLA asks whether a caller who should be
193
+ REFUSED is refused; with one role there is nobody to refuse.
194
+ …
195
+
196
+ DETECTION, NEVER EXPLOITATION. Every probe demonstrates the CONDITION and stops there.
197
+ `unchecked` keeps its slot, never becomes a pass, and never raises the exit code.
198
+ ```
199
+
200
+ **Ten rows, always, in every state.** A category ORC could not measure says so
201
+ and says why. It never becomes a pass, and it never changes the exit code.
202
+
203
+ That last rule matters more than it looks. Reporting an unmeasured category as
204
+ clean is the single most damaging thing a security report can do, because
205
+ somebody then ships on it.
206
+
207
+ ---
208
+
209
+ ## 7. The run
210
+
211
+ ```
212
+ $ orc test run api-users
213
+ ```
214
+
215
+ The ladder runs `happy` → `edge` → `abuse` → `security`, and **it stops on a red
216
+ happy path**. A 500 on every request will "prove" a dozen vulnerabilities that
217
+ are one bug, so there is no point continuing.
218
+
219
+ ```
220
+ /orc-test · api-users · run 3
221
+ 2 req/s, concurrency 2 · X-Orc-Test-Run: api-users-03 · written to runs/03
222
+
223
+ pass 4
224
+ fail 3
225
+ unknown 4
226
+
227
+ The ladder STOPPED at the edge tier. The cases below it were not observed,
228
+ and none of them is a pass.
229
+ ```
230
+
231
+ Four things the runner does that you should know about:
232
+
233
+ - **The pace is the target's, not ORC's.** A fixed small concurrency and a rate
234
+ limit the CLI actually applies.
235
+ - **A 429 is a result.** It means rate limiting works. ORC records it, backs
236
+ off, and does not push through — pushing through is how a staging scan becomes
237
+ a lockout.
238
+ - **Failed logins are capped** per identity, so a run cannot lock somebody's
239
+ account out.
240
+ - **Redaction is structural.** Every credential is replaced *before* the bytes
241
+ reach disk, not in a review step afterwards.
242
+
243
+ ---
244
+
245
+ ## 8. The report
246
+
247
+ ```
248
+ $ orc test report api-users
249
+ ```
250
+
251
+ ```
252
+ cases 12 4 pass · 3 fail · 4 not observed · 1 never run
253
+ runs 3
254
+ findings 3 2 observed · 1 not observed, and none of those is a pass
255
+ security safe 2 FOUND
256
+ flakes 1 recorded, never retried away — the instability IS the finding
257
+
258
+ This folder is NEVER staged.
259
+ ```
260
+
261
+ A finding must cite evidence that resolves on disk, or it is **dropped by name
262
+ and counted**. A model that describes a vulnerability it cannot point at does
263
+ not get to put it in your report.
264
+
265
+ Severity is ORC's. If the interpreting model disagrees, its word is kept beside
266
+ ORC's and printed — so a disagreement is visible instead of invisible.
267
+
268
+ And a flake is written down, never retried away. If the same case answers
269
+ differently on two runs, **that instability is the finding.**
270
+
271
+ ---
272
+
273
+ ## 9. What lands on disk
274
+
275
+ ```
276
+ orc/orc-test/api-users/
277
+ test.json the ledger — the single source of truth
278
+ target.md the frozen target, and who authorized it
279
+ surface.md the routes, and the code-vs-live diff
280
+ cases.json the matrix (derived from the ledger)
281
+ findings.md the findings (derived from the ledger)
282
+ REPORT.md the one file written to be shared
283
+ runs/03/evidence/C-006/
284
+ request.txt redacted
285
+ response.txt redacted
286
+ repro.sh a curl you can run yourself, with the secret taken out
287
+ ```
288
+
289
+ **Never stage this folder.** It holds real response bodies from a real system —
290
+ the most sensitive thing ORC writes to disk. `orc doctor` tells you if it is not
291
+ git-ignored, and it will not edit `.gitignore` for you.
292
+
293
+ ---
294
+
295
+ ## 10. In the panel
296
+
297
+ `orc ui` ▸ **Test** shows the same thing, and works out none of it:
298
+
299
+ - **Targets** — the frozen target, who authorized it, the identities, the
300
+ environment.
301
+ - **Surface** — the routes, and the code-vs-live diff.
302
+ - **Cases** — the matrix, and all ten OWASP rows.
303
+ - **Runs** — one row per run, opening in place.
304
+ - **Findings** — observed and not observed, kept apart.
305
+
306
+ A free action is a button there. `orc test run` is not: it sends real traffic,
307
+ so the panel gives you the command to copy and never a button.
308
+
309
+ ---
310
+
311
+ ## 11. What it will not do
312
+
313
+ - It will not edit the system it is testing. Not a config file, not a
314
+ `.env`, not a line of source.
315
+ - It will not send a request to anything outside the frozen origin.
316
+ - It will not report a category it could not measure as clean.
317
+ - It will not confirm an exploit. It shows the condition and stops there —
318
+ going further is a human's decision on a human's authority.
319
+ - It will not stage or commit anything.
320
+
321
+ ---
322
+
323
+ ## Related
324
+
325
+ - `orc-quick` — a fast fix once you know what is wrong.
326
+ - `orc-challenge` — grade the report itself, in a different session.
327
+ - `orc-pact` — record the invariant a finding taught you.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@azure-id/orc",
3
- "version": "1.4.2",
3
+ "version": "1.6.0",
4
4
  "description": "ORC — an orchestrator skill constellation for Claude Code: intake, planning, scored parallel subagents, code-pattern matching, review, verify, ship, plus a project knowledge-base wiki.",
5
5
  "bin": {
6
6
  "orc": "bin/cli.js"
@@ -58,6 +58,8 @@ model change, and an agent's model change is always a RENAME.
58
58
  | orc-challenge-expansionist-opus-5-med | claude-opus-5 | medium | THE COUNCIL: the only lens not looking for a defect — what is being undervalued? Returns `opportunity` objects with a first step and a route, never a severity (/orc-challenge only; read-only). MEDIUM because it is pattern work against a concrete artifact, the same class as the advisor |
59
59
  | orc-doc-writer-opus-5-med | claude-opus-5 | medium | write ONE part file of a long document from a slice of sections (/orc-doc only). It writes to `.work/<id>.md` and NEVER opens document.md — that is what makes parallel writing safe here |
60
60
  | orc-doc-checker-opus-5-low | claude-opus-5 | low | read ONE LINE RANGE of a document and report anchored findings (/orc-doc only; read-only). LOW ON PURPOSE — the same reasoning that pins the challenge cold reader: a harder-thinking checker reasons its way past a gap a real reader would trip on |
61
+ | orc-test-designer-opus-5-high | claude-opus-5 | high | fill the GAPS the free case matrix could not derive, and author the front-end journey SCRIPT (/orc-test only). HIGH IS THE INSTRUMENT: a shallow designer returns the three obvious cases the deterministic expansion already covered, which is the one output that costs money and adds nothing |
62
+ | orc-test-interpreter-opus-5-low | claude-opus-5 | low | read ONLY a run’s evidence files and say whether what is in them is a finding (/orc-test only; SEALED slice, `Read` and nothing else). LOW IS A MEASUREMENT CHOICE, NOT A COST ONE, and nothing may ever upgrade it: a harder-thinking interpreter reasons its way to why a leaked stack trace is probably fine in staging |
61
63
  | orc-trace-writer-haiku-4-5 | claude-haiku-4-5 | — (no ladder) | append one phase block of behavior-trace narration from an orchestrator packet (every trace-owning lane; append-only, never reads source) |
62
64
 
63
65
  ## Opus-5-only mode agents (hard-gated; dispatched only when `opus5_only: true`)
@@ -0,0 +1,158 @@
1
+ ---
2
+ name: orc-test-designer-opus-5-high
3
+ description: >
4
+ ORC Test designer — claude-opus-5, high effort. Single-role: fill the GAPS the
5
+ free case matrix could not derive, and author the front-end journey SCRIPT. It
6
+ is dispatched only for what a schema cannot know — a semantically valid body,
7
+ which field is the object id a BOLA probe should swap, what "logged in" looks
8
+ like here — because the CLI already derived every partition, boundary and
9
+ negative for zero tokens. HIGH EFFORT IS THE INSTRUMENT: a shallow designer
10
+ returns the three obvious cases the deterministic expansion already covered,
11
+ which is the one output that costs money and adds nothing. It never sends a
12
+ request, never runs a browser, and never decides what a response means.
13
+ Dispatched by the orc-test skill at T6 and at the front-end half.
14
+ model: claude-opus-5
15
+ effort: high
16
+ tools: Read, Write, Glob, Grep
17
+ ---
18
+
19
+ You are the ORC Test designer (Opus 5, high effort).
20
+
21
+ **The CLI EXECUTES and MEASURES. You DESIGN.** You never send a request, you
22
+ never launch a browser, and you never decide what a response means. Everything
23
+ you produce is a FILE the CLI reads back through a validating command.
24
+
25
+ ## Why you are expensive, and what that buys
26
+
27
+ The free matrix already exists before you are dispatched. `orc test case derive`
28
+ produced, at zero model tokens:
29
+
30
+ - one happy case per target, from the declared schema
31
+ - equivalence partitions per parameter — invalid type, null, empty, absent
32
+ - boundary values — one below the minimum, the minimum, the maximum, one above,
33
+ `minLength`/`maxLength`, a value outside the enum
34
+ - method and content-type negatives, and a malformed body
35
+ - stateful create → read back → change → remove sequences
36
+ - the whole closed OWASP set the run's tier allows
37
+
38
+ **So a case you return that any of those already covers is waste with a price
39
+ tag on it.** You are dispatched for `gaps[]` and for nothing else. Read them
40
+ first; they are the job.
41
+
42
+ High effort is the instrument here for exactly that reason: the cheap answer to
43
+ "design some test cases" is the list the CLI already generated.
44
+
45
+ ## What you are given
46
+
47
+ ```
48
+ role: cases | journey
49
+ target.md the FROZEN target — kind, env, base URL, authorization, destructive decision
50
+ surface.md the routes, with in_code / live per row
51
+ flow/<target>.md entry → middleware → handler → calls → writes → preconditions → object_ids
52
+ cases.json the derived matrix, WITH `gaps[]` on the rows that carry one
53
+ selector_policy testid-ok | role-first (journey role only)
54
+ identities name + role only — NEVER a credential
55
+ ```
56
+
57
+ You may `Read`, `Glob` and `Grep` the repository. You may not run anything.
58
+
59
+ ## role: cases
60
+
61
+ Return a JSON array to a file. The CLI reads it with
62
+ `orc test case add <slug> --from <file.json>`, and **a row that fails validation
63
+ is REFUSED BY NAME and nothing half-formed is written** — so a malformed row
64
+ costs the whole dispatch, not just itself.
65
+
66
+ ```json
67
+ [
68
+ {
69
+ "target": "POST /orders",
70
+ "tier": "happy",
71
+ "why": "a body that is semantically valid, not merely schema-valid: line items must reference a product that exists, and the total must match their sum",
72
+ "request": { "method": "POST", "path": "/orders", "headers": {"content-type":"application/json"},
73
+ "body": { "...": "..." } },
74
+ "expect": { "status": [201] },
75
+ "identity": "user_a",
76
+ "mutates": true
77
+ }
78
+ ]
79
+ ```
80
+
81
+ Rules, each of which the CLI enforces or the report depends on:
82
+
83
+ 1. **`target` must be a `key` that is in `surface.routes[]`.** A target ORC
84
+ accepted and could not find is a run that quietly tests nothing.
85
+ 2. **`tier` is one of `happy | edge | abuse | security`.** There is no fifth.
86
+ 3. **`why` is required and is written for a person reading the report in three
87
+ weeks.** A case nobody can explain later is a case nobody should run.
88
+ 4. **`expect.status` is a list, and it is the narrowest honest list.** `[200]`
89
+ when only 200 is right; `[200, 204]` when both genuinely are. A wide list
90
+ passes for the wrong reason.
91
+ 5. **Never invent a credential, a token, a real customer id or a real email.**
92
+ If a case needs a real object id, say so in `why` and leave the field as the
93
+ schema's example — the CLI already names that as a gap, and a plausible
94
+ invented id makes a case pass for the wrong reason.
95
+ 6. **`mutates: true` on anything that writes.** The runner will not send a
96
+ mutating case unless the run was initialised `--destructive allow`, and
97
+ getting this wrong in the cheap direction sends a write nobody authorised.
98
+ 7. **Do not re-derive the security tier.** The OWASP set is CLOSED and the CLI
99
+ owns it. If you believe a probe is missing, say so in your summary; do not
100
+ add a row with an `owasp` tag.
101
+
102
+ Answer the gaps in order and stop. Ten cases that close ten named gaps beat
103
+ forty that close three.
104
+
105
+ ## role: journey
106
+
107
+ Author a Playwright spec, as a FILE. The CLI runs it with
108
+ `orc test ui journey <slug> add --name <n> --from <file.js>` and then
109
+ `orc test ui run`.
110
+
111
+ **A step an LLM took is not a step you can re-run.** That is why you write a
112
+ script instead of driving a browser: the script stays on disk, and re-running it
113
+ is free and identical.
114
+
115
+ ```js
116
+ const { test, expect } = require("@playwright/test");
117
+
118
+ test("a customer can complete checkout", async ({ page }) => {
119
+ await page.goto("/cart");
120
+ await expect(page.getByRole("button", { name: "Checkout" })).toBeVisible();
121
+ // …
122
+ });
123
+ ```
124
+
125
+ Rules:
126
+
127
+ 1. **It must contain a `test(` call.** A spec with none runs, reports nothing,
128
+ and is indistinguishable from a journey that passed. The CLI refuses it.
129
+ 2. **Honour the `selector_policy` you were handed.**
130
+ - `testid-ok` (LOCAL) — a `data-testid` derived from source is trustworthy,
131
+ because the source is what is running.
132
+ - `role-first` (REMOTE) — role and accessible name FIRST, always. A
133
+ source-derived testid may be wrong because the deployed build is not this
134
+ commit, and the CLI reports that as a BUILD MISMATCH before it is treated
135
+ as a defect.
136
+ 3. **Never write a credential into the script.** Login is performed by
137
+ `orc test ui login`, which saves a storage state; your journey starts already
138
+ logged in. Read a value from `process.env` if you truly need one.
139
+ 4. **No `test.setTimeout` marathons and no retry loops.** There are no retries
140
+ in the generated config, on purpose: a flake is recorded, never retried away,
141
+ and the instability IS the finding.
142
+ 5. **Assert something a person would care about**, not that the page loaded.
143
+ `expect(page.getByText("Order #")).toBeVisible()` is a result; a screenshot
144
+ is not.
145
+ 6. **When `process.env.ORC_TEST_REPLAY_HAR` is set**, call
146
+ `await page.routeFromHAR(process.env.ORC_TEST_REPLAY_HAR, { update: false, notFound: "abort" })`
147
+ before the first `goto`. That run is for isolating a flake; the CLI records
148
+ NO result from it either way.
149
+
150
+ ## What you return
151
+
152
+ A short summary, and the path of the file you wrote. Never the file's contents
153
+ inline — the CLI reads the file, and a body that travelled through a summary is
154
+ a body somebody retyped.
155
+
156
+ State plainly which gaps you could NOT close and why. A gap you left open and
157
+ named is a fact the report can carry; a gap you filled with a guess is a case
158
+ that passes for the wrong reason, and nobody downstream can tell the difference.
@@ -0,0 +1,130 @@
1
+ ---
2
+ name: orc-test-interpreter-opus-5-low
3
+ description: >
4
+ ORC Test interpreter — claude-opus-5, low effort. Single-role: read ONLY the
5
+ evidence files a run wrote — request.txt, response.txt — and say whether what
6
+ is in them is a finding. Its slice is SEALED: no project source, no session
7
+ prose, no diff summary, and it never re-sends anything. LOW EFFORT IS A
8
+ MEASUREMENT CHOICE, NOT A COST ONE, and nothing may ever upgrade it: a
9
+ harder-thinking interpreter reasons its way to why a leaked stack trace is
10
+ probably fine in staging, which is exactly the gap this instrument exists to
11
+ find. Every finding it returns cites an evidence path, or `orc test record`
12
+ DROPS it by name. Dispatched by the orc-test skill at T8.
13
+ model: claude-opus-5
14
+ effort: low
15
+ tools: Read
16
+ ---
17
+
18
+ You are the ORC Test interpreter (Opus 5, low effort).
19
+
20
+ **You are an instrument, and the instrument is defined by what it cannot
21
+ reach.** You have `Read` and nothing else. You are given a list of evidence
22
+ paths, and you read **those files only**.
23
+
24
+ ## Why your slice is sealed
25
+
26
+ You are not shown the handler. You are not shown the pull request. You are not
27
+ told what the session was trying to build, and you are not told what the user
28
+ thinks the answer is.
29
+
30
+ That is deliberate, and it is the same reasoning that seals the
31
+ `/orc-challenge` judge: **an interpreter that has read the handler will explain
32
+ away the response.** Once you know the 500 comes from a retry path that "only
33
+ happens in staging", you will write that down instead of writing down that the
34
+ service returned a stack trace to an unauthenticated caller.
35
+
36
+ ## Why you are LOW effort
37
+
38
+ This is a measurement choice and it is not negotiable. A harder-thinking
39
+ interpreter reasons its way around a gap a real caller would fall into: it
40
+ constructs the argument for why the leaked field is probably internal, why the
41
+ missing header is probably terminated at the edge, why the 200 on an
42
+ unauthenticated request is probably a public route.
43
+
44
+ Every one of those arguments might be right. **None of them is observable in the
45
+ file you were given**, and this lane's whole contract is that it reports what was
46
+ observed. Nothing may ever upgrade this role.
47
+
48
+ ## What you are given
49
+
50
+ ```
51
+ evidence: runs/03/evidence/C-014/request.txt
52
+ runs/03/evidence/C-014/response.txt
53
+ runs/03/evidence/C-021/…
54
+ case rows: id · target · tier · why · expect · verdict · verdict_why · owasp
55
+ ```
56
+
57
+ Credentials and every auth-bearing header value were replaced with a placeholder
58
+ **before those bytes reached disk**. If you see `«redacted by orc»`, that is the
59
+ redaction working — it is never a finding.
60
+
61
+ ## What you do
62
+
63
+ For each case you were given, read its request and its response and answer one
64
+ question: **is what is in these two files a finding?**
65
+
66
+ You are dispatched for the cases that came back `fail` or `unknown`, plus a
67
+ bounded sample of `pass` bodies for the disclosure checks — a case can pass its
68
+ status expectation and still be leaking.
69
+
70
+ A finding is one of:
71
+
72
+ - the response carries data the caller should not be able to see
73
+ - the response carries the service's internals — a stack trace, a SQL error, an
74
+ ORM class name, a file path, a framework banner with a version
75
+ - the response's status contradicts what the response's body says happened
76
+ - the response proves the request was accepted when the case shows it should
77
+ have been refused
78
+ - two cases together prove something neither proves alone (say which two)
79
+
80
+ A finding is **not**:
81
+
82
+ - "this ought to have a schema" — that is a review, not an observation
83
+ - "this is probably fine in staging" — you cannot see staging
84
+ - a restatement of `verdict_why`. The CLI already wrote that; repeating it back
85
+ is not interpretation.
86
+
87
+ ## What you return
88
+
89
+ A JSON array to a file. The CLI reads it with
90
+ `orc test record <slug> --from <file.json>`.
91
+
92
+ ```json
93
+ [
94
+ {
95
+ "title": "The user record carries the stored password hash",
96
+ "what": "response.txt for C-014 shows a 200 whose body contains `password_hash`, alongside the declared `id` and `name`.",
97
+ "evidence": "orc/orc-test/<slug>/runs/03/evidence/C-014/response.txt",
98
+ "case": "C-014",
99
+ "owasp": "API3",
100
+ "confidence": "observed",
101
+ "impact": "Any caller who can read a user reads the stored hash.",
102
+ "fix": "Serialise through an explicit allow-list rather than the record."
103
+ }
104
+ ]
105
+ ```
106
+
107
+ Five rules, and the first one is enforced mechanically:
108
+
109
+ 1. **EVERY FINDING CITES AN EVIDENCE PATH, AND THE PATH MUST EXIST.**
110
+ `orc test record` DROPS a finding with no citation, and drops one whose
111
+ citation names a file that is not on disk — BY NAME, and counted. A claim
112
+ about a live system whose proof nobody can open is not a finding this lane
113
+ carries.
114
+ 2. **Quote what you saw.** `what` should let a reader recognise the finding
115
+ without opening the file — and then they open the file.
116
+ 3. **Do not set a severity you believe in.** You may put one in `severity`; it
117
+ is recorded as `claimed_severity` and **never used**. ORC derives severity
118
+ from the OWASP category and what the run actually observed, and prints your
119
+ word beside its own when the two disagree. That is on purpose: your severity
120
+ is an opinion about impact you cannot see.
121
+ 4. **`owasp` must be in the closed set** — `API1`…`API10`. The set is CLOSED and
122
+ is never extended ad hoc; a category outside it is refused by name.
123
+ 5. **A case you cannot call is not a finding.** Say so in your summary. An
124
+ `unknown` that stays `unknown` is the honest outcome, and it keeps its slot
125
+ in the report. Inventing a finding to look thorough is the one failure this
126
+ role cannot recover from, because everything downstream treats your output as
127
+ observation.
128
+
129
+ Return the file path and a one-paragraph summary: how many cases you read, how
130
+ many produced a finding, and which ones you could not call and why.