@azure-id/orc 1.4.1 → 1.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +3154 -2907
- package/README-id.md +39 -2
- package/README.md +647 -631
- package/bin/cli.js +42495 -36866
- package/bin/mockrun-catalog.js +4 -0
- package/bin/verify-contracts.js +4475 -4410
- package/bin/verify-package.js +21 -2
- package/bin/webui/api.js +1388 -1283
- package/bin/webui/app.html +9 -0
- package/bin/webui/css/04-motion.css +18 -0
- package/bin/webui/css/06-responsive.css +17 -1
- package/bin/webui/css/panels/hookui.css +220 -41
- package/bin/webui/css/panels/test.css +132 -0
- package/bin/webui/fixtures/index.js +23 -1
- package/bin/webui/fixtures/test.js +547 -0
- package/bin/webui/i18n/en/common.json +1 -0
- package/bin/webui/i18n/en/hookui.json +17 -3
- package/bin/webui/i18n/en/nav.json +1 -0
- package/bin/webui/i18n/en/overview.json +4 -0
- package/bin/webui/i18n/en/test.json +141 -0
- package/bin/webui/i18n/en/tour.json +1 -1
- package/bin/webui/i18n/id/common.json +1 -0
- package/bin/webui/i18n/id/hookui.json +17 -3
- package/bin/webui/i18n/id/nav.json +1 -0
- package/bin/webui/i18n/id/overview.json +4 -0
- package/bin/webui/i18n/id/test.json +141 -0
- package/bin/webui/i18n/id/tour.json +1 -1
- package/bin/webui/js/01-i18n.js +1 -0
- package/bin/webui/js/panels/challenge.js +8 -3
- package/bin/webui/js/panels/hookui.js +586 -208
- package/bin/webui/js/panels/overview.js +10 -0
- package/bin/webui/js/panels/test.js +733 -0
- package/bin/webui/serve.js +572 -557
- package/mock-run/orc-test.md +327 -0
- package/package.json +1 -1
- package/templates/agents/MODEL-MAPPING.md +2 -0
- package/templates/agents/orc-test-designer-opus-5-high.md +158 -0
- package/templates/agents/orc-test-interpreter-opus-5-low.md +130 -0
- package/templates/commands/orc-test.md +66 -0
- package/templates/hooks/README.md +8 -1
- package/templates/hooks/orc-statusline.js +15 -4
- package/templates/hooks/orc-subagent-line.js +5 -2
- package/templates/skills/_shared/extra-dispatch.md +1346 -1337
- package/templates/skills/_shared/live-target.md +188 -0
- package/templates/skills/_shared/phases/trace.md +2 -2
- package/templates/skills/_shared/wait.md +241 -240
- package/templates/skills/orc-test/SKILL.md +233 -0
|
@@ -0,0 +1,327 @@
|
|
|
1
|
+
# Mock run — `/orc-test`
|
|
2
|
+
|
|
3
|
+
> ORC runs the test against a real system, writes down what it saw, and never touches the system it is testing.
|
|
4
|
+
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## 1. What it does
|
|
8
|
+
|
|
9
|
+
Every other testing tool in ORC **writes** tests. `/orc-test` **runs** them.
|
|
10
|
+
|
|
11
|
+
It points at a running API — yours, on your laptop, or one you were given
|
|
12
|
+
permission to touch — sends real requests, and writes down what came back. Then
|
|
13
|
+
it stops.
|
|
14
|
+
|
|
15
|
+
The split is the whole design:
|
|
16
|
+
|
|
17
|
+
> **The CLI executes and measures. The model designs and interprets.**
|
|
18
|
+
> A model never sends a request. The CLI never decides what a response means.
|
|
19
|
+
|
|
20
|
+
That split is what makes the report worth reading. A number in it was measured
|
|
21
|
+
by a program. A sentence in it was written by a model that only ever saw
|
|
22
|
+
evidence on disk.
|
|
23
|
+
|
|
24
|
+
Three rules hold it together, and each one is a contract with a name:
|
|
25
|
+
|
|
26
|
+
| Rule | What it means |
|
|
27
|
+
|---|---|
|
|
28
|
+
| `a lane that reports a result it did not observe` | Three verdicts: `pass`, `fail`, `unknown`. **`unknown` is the honest one.** A case nobody ran is never a pass. |
|
|
29
|
+
| `a lane that sends traffic to a target nobody authorized` | The target is frozen at the start, with a written statement of who said yes. Nothing outside that origin is ever contacted. |
|
|
30
|
+
| `a lane that fixes the system under test` | ORC never edits the thing it is testing. When the app is broken, it hands the machine back to you. |
|
|
31
|
+
|
|
32
|
+
---
|
|
33
|
+
|
|
34
|
+
## 2. Freeze the target
|
|
35
|
+
|
|
36
|
+
Nothing here has a default it could guess. There is no config key that names a
|
|
37
|
+
target, on purpose.
|
|
38
|
+
|
|
39
|
+
```
|
|
40
|
+
$ orc test init api-users --kind be --env local \
|
|
41
|
+
--base-url http://127.0.0.1:4000 \
|
|
42
|
+
--authorized "me, on my own laptop, against my own dev database" \
|
|
43
|
+
--destructive deny --reason "this box shares a database with the seed script"
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
```
|
|
47
|
+
/orc-test · api-users
|
|
48
|
+
target FROZEN at orc/orc-test/api-users/target.md
|
|
49
|
+
|
|
50
|
+
kind be
|
|
51
|
+
environment local
|
|
52
|
+
base URL http://127.0.0.1:4000
|
|
53
|
+
origin fence http://127.0.0.1:4000
|
|
54
|
+
destructive deny — this box shares a database with the seed script
|
|
55
|
+
|
|
56
|
+
This folder is NEVER staged. It will hold real response bodies from a real
|
|
57
|
+
system. Add this line to .gitignore yourself — ORC does not edit it:
|
|
58
|
+
orc/orc-test/
|
|
59
|
+
|
|
60
|
+
Next: orc test surface api-users
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
Two things to notice.
|
|
64
|
+
|
|
65
|
+
**The origin fence.** Every request the runner sends is checked against it. A
|
|
66
|
+
redirect that leaves the fence is not followed — it is recorded and dropped.
|
|
67
|
+
|
|
68
|
+
**`--authorized` is a sentence, not a checkbox.** ORC records it word for word
|
|
69
|
+
and never checks it. On a remote target it is required. It prints at the top of
|
|
70
|
+
every report, so the person reading the report can see whose permission this
|
|
71
|
+
was run on.
|
|
72
|
+
|
|
73
|
+
---
|
|
74
|
+
|
|
75
|
+
## 3. The free pass
|
|
76
|
+
|
|
77
|
+
The surface costs nothing. No model, no tokens.
|
|
78
|
+
|
|
79
|
+
```
|
|
80
|
+
$ orc test surface api-users
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
```
|
|
84
|
+
/orc-test · api-users · surface
|
|
85
|
+
the free pass — zero model tokens. Written to orc/orc-test/api-users/surface.md
|
|
86
|
+
|
|
87
|
+
spec on disk openapi.yaml (4 routes)
|
|
88
|
+
spec at target http://127.0.0.1:4000/openapi.json
|
|
89
|
+
frameworks express
|
|
90
|
+
routes 5 (34 files scanned, 1 live-only)
|
|
91
|
+
mounted at /api/v1
|
|
92
|
+
unresolved 1
|
|
93
|
+
|
|
94
|
+
in code, not live 1 dead or unshipped
|
|
95
|
+
live, not in code 1 shadow / zombie APIs — OWASP API9:2023
|
|
96
|
+
GET /internal/debug
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
`GET /internal/debug` answers at the target and is in no file in this
|
|
100
|
+
repository. Nobody reviews it. Nobody patches it. Finding that took zero model
|
|
101
|
+
tokens and one comparison of two documents — **no route was called to produce
|
|
102
|
+
it**.
|
|
103
|
+
|
|
104
|
+
If there had been no spec at the target, that line would read:
|
|
105
|
+
|
|
106
|
+
```
|
|
107
|
+
code-vs-live diff NOT MEASURED — no spec answered at the target, so there
|
|
108
|
+
is nothing to compare the code against. This is NOT 'no shadow APIs' — it is
|
|
109
|
+
'not measured'.
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
That distinction is the lane in one sentence.
|
|
113
|
+
|
|
114
|
+
---
|
|
115
|
+
|
|
116
|
+
## 4. When the app is broken
|
|
117
|
+
|
|
118
|
+
```
|
|
119
|
+
$ orc test env api-users
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
```
|
|
123
|
+
/orc-test · api-users · environment
|
|
124
|
+
state UNHEALTHY the process is up and answering wrong. ORC does not fix
|
|
125
|
+
the system it is testing.
|
|
126
|
+
|
|
127
|
+
detected npm run dev (package.json)
|
|
128
|
+
process pid 48211
|
|
129
|
+
missing STRIPE_SECRET_KEY, SESSION_SECRET
|
|
130
|
+
|
|
131
|
+
Error: password authentication failed for user "app"
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
And then it stops.
|
|
135
|
+
|
|
136
|
+
It found the start command. It knows the process is alive. It can see the exact
|
|
137
|
+
line in the log. It will not open that file. It names the two environment keys
|
|
138
|
+
that are absent and **writes no value for either** — a placeholder written into
|
|
139
|
+
a real environment is a credential-shaped lie, and only you know the right
|
|
140
|
+
value.
|
|
141
|
+
|
|
142
|
+
You fix it. You run the command again. That is the whole loop.
|
|
143
|
+
|
|
144
|
+
---
|
|
145
|
+
|
|
146
|
+
## 5. The cases derive themselves
|
|
147
|
+
|
|
148
|
+
```
|
|
149
|
+
$ orc test case derive api-users
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
```
|
|
153
|
+
/orc-test · api-users · cases
|
|
154
|
+
derived for free — zero model tokens
|
|
155
|
+
|
|
156
|
+
happy 2
|
|
157
|
+
edge 4
|
|
158
|
+
abuse 3
|
|
159
|
+
security 3
|
|
160
|
+
total 12
|
|
161
|
+
|
|
162
|
+
2 cases carry a GAP — what a schema could not answer.
|
|
163
|
+
This is what the designer agent is for, and it is the only part that costs anything.
|
|
164
|
+
|
|
165
|
+
security tier safe (the shipped default)
|
|
166
|
+
5 OWASP rows UNCHECKABLE — each keeps its slot, and none of them will ever become a pass
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
The matrix comes out of the schema for free. The **gaps** are what a schema
|
|
170
|
+
cannot know — what a valid email looks like in this business, which order the
|
|
171
|
+
two calls have to happen in. Those go to the designer agent, and that is the
|
|
172
|
+
only part of this lane you pay for.
|
|
173
|
+
|
|
174
|
+
---
|
|
175
|
+
|
|
176
|
+
## 6. The security tier is a closed set
|
|
177
|
+
|
|
178
|
+
```
|
|
179
|
+
$ orc test security api-users
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
```
|
|
183
|
+
/orc-test · api-users · security · OWASP API Top 10 (2023)
|
|
184
|
+
tier safe (the shipped default) · the set is CLOSED and is never extended ad hoc
|
|
185
|
+
|
|
186
|
+
API1 FOUND Broken object level authorization (BOLA)
|
|
187
|
+
severity critical OBSERVED, and the request crossed an identity boundary
|
|
188
|
+
C-006 replay a successful request as a SECOND identity
|
|
189
|
+
orc/orc-test/api-users/runs/03/evidence/C-006/
|
|
190
|
+
API2 observed-clean Broken authentication
|
|
191
|
+
API5 unchecked Broken function level authorization (BFLA)
|
|
192
|
+
only one ROLE is declared. BFLA asks whether a caller who should be
|
|
193
|
+
REFUSED is refused; with one role there is nobody to refuse.
|
|
194
|
+
…
|
|
195
|
+
|
|
196
|
+
DETECTION, NEVER EXPLOITATION. Every probe demonstrates the CONDITION and stops there.
|
|
197
|
+
`unchecked` keeps its slot, never becomes a pass, and never raises the exit code.
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
**Ten rows, always, in every state.** A category ORC could not measure says so
|
|
201
|
+
and says why. It never becomes a pass, and it never changes the exit code.
|
|
202
|
+
|
|
203
|
+
That last rule matters more than it looks. Reporting an unmeasured category as
|
|
204
|
+
clean is the single most damaging thing a security report can do, because
|
|
205
|
+
somebody then ships on it.
|
|
206
|
+
|
|
207
|
+
---
|
|
208
|
+
|
|
209
|
+
## 7. The run
|
|
210
|
+
|
|
211
|
+
```
|
|
212
|
+
$ orc test run api-users
|
|
213
|
+
```
|
|
214
|
+
|
|
215
|
+
The ladder runs `happy` → `edge` → `abuse` → `security`, and **it stops on a red
|
|
216
|
+
happy path**. A 500 on every request will "prove" a dozen vulnerabilities that
|
|
217
|
+
are one bug, so there is no point continuing.
|
|
218
|
+
|
|
219
|
+
```
|
|
220
|
+
/orc-test · api-users · run 3
|
|
221
|
+
2 req/s, concurrency 2 · X-Orc-Test-Run: api-users-03 · written to runs/03
|
|
222
|
+
|
|
223
|
+
pass 4
|
|
224
|
+
fail 3
|
|
225
|
+
unknown 4
|
|
226
|
+
|
|
227
|
+
The ladder STOPPED at the edge tier. The cases below it were not observed,
|
|
228
|
+
and none of them is a pass.
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
Four things the runner does that you should know about:
|
|
232
|
+
|
|
233
|
+
- **The pace is the target's, not ORC's.** A fixed small concurrency and a rate
|
|
234
|
+
limit the CLI actually applies.
|
|
235
|
+
- **A 429 is a result.** It means rate limiting works. ORC records it, backs
|
|
236
|
+
off, and does not push through — pushing through is how a staging scan becomes
|
|
237
|
+
a lockout.
|
|
238
|
+
- **Failed logins are capped** per identity, so a run cannot lock somebody's
|
|
239
|
+
account out.
|
|
240
|
+
- **Redaction is structural.** Every credential is replaced *before* the bytes
|
|
241
|
+
reach disk, not in a review step afterwards.
|
|
242
|
+
|
|
243
|
+
---
|
|
244
|
+
|
|
245
|
+
## 8. The report
|
|
246
|
+
|
|
247
|
+
```
|
|
248
|
+
$ orc test report api-users
|
|
249
|
+
```
|
|
250
|
+
|
|
251
|
+
```
|
|
252
|
+
cases 12 4 pass · 3 fail · 4 not observed · 1 never run
|
|
253
|
+
runs 3
|
|
254
|
+
findings 3 2 observed · 1 not observed, and none of those is a pass
|
|
255
|
+
security safe 2 FOUND
|
|
256
|
+
flakes 1 recorded, never retried away — the instability IS the finding
|
|
257
|
+
|
|
258
|
+
This folder is NEVER staged.
|
|
259
|
+
```
|
|
260
|
+
|
|
261
|
+
A finding must cite evidence that resolves on disk, or it is **dropped by name
|
|
262
|
+
and counted**. A model that describes a vulnerability it cannot point at does
|
|
263
|
+
not get to put it in your report.
|
|
264
|
+
|
|
265
|
+
Severity is ORC's. If the interpreting model disagrees, its word is kept beside
|
|
266
|
+
ORC's and printed — so a disagreement is visible instead of invisible.
|
|
267
|
+
|
|
268
|
+
And a flake is written down, never retried away. If the same case answers
|
|
269
|
+
differently on two runs, **that instability is the finding.**
|
|
270
|
+
|
|
271
|
+
---
|
|
272
|
+
|
|
273
|
+
## 9. What lands on disk
|
|
274
|
+
|
|
275
|
+
```
|
|
276
|
+
orc/orc-test/api-users/
|
|
277
|
+
test.json the ledger — the single source of truth
|
|
278
|
+
target.md the frozen target, and who authorized it
|
|
279
|
+
surface.md the routes, and the code-vs-live diff
|
|
280
|
+
cases.json the matrix (derived from the ledger)
|
|
281
|
+
findings.md the findings (derived from the ledger)
|
|
282
|
+
REPORT.md the one file written to be shared
|
|
283
|
+
runs/03/evidence/C-006/
|
|
284
|
+
request.txt redacted
|
|
285
|
+
response.txt redacted
|
|
286
|
+
repro.sh a curl you can run yourself, with the secret taken out
|
|
287
|
+
```
|
|
288
|
+
|
|
289
|
+
**Never stage this folder.** It holds real response bodies from a real system —
|
|
290
|
+
the most sensitive thing ORC writes to disk. `orc doctor` tells you if it is not
|
|
291
|
+
git-ignored, and it will not edit `.gitignore` for you.
|
|
292
|
+
|
|
293
|
+
---
|
|
294
|
+
|
|
295
|
+
## 10. In the panel
|
|
296
|
+
|
|
297
|
+
`orc ui` ▸ **Test** shows the same thing, and works out none of it:
|
|
298
|
+
|
|
299
|
+
- **Targets** — the frozen target, who authorized it, the identities, the
|
|
300
|
+
environment.
|
|
301
|
+
- **Surface** — the routes, and the code-vs-live diff.
|
|
302
|
+
- **Cases** — the matrix, and all ten OWASP rows.
|
|
303
|
+
- **Runs** — one row per run, opening in place.
|
|
304
|
+
- **Findings** — observed and not observed, kept apart.
|
|
305
|
+
|
|
306
|
+
A free action is a button there. `orc test run` is not: it sends real traffic,
|
|
307
|
+
so the panel gives you the command to copy and never a button.
|
|
308
|
+
|
|
309
|
+
---
|
|
310
|
+
|
|
311
|
+
## 11. What it will not do
|
|
312
|
+
|
|
313
|
+
- It will not edit the system it is testing. Not a config file, not a
|
|
314
|
+
`.env`, not a line of source.
|
|
315
|
+
- It will not send a request to anything outside the frozen origin.
|
|
316
|
+
- It will not report a category it could not measure as clean.
|
|
317
|
+
- It will not confirm an exploit. It shows the condition and stops there —
|
|
318
|
+
going further is a human's decision on a human's authority.
|
|
319
|
+
- It will not stage or commit anything.
|
|
320
|
+
|
|
321
|
+
---
|
|
322
|
+
|
|
323
|
+
## Related
|
|
324
|
+
|
|
325
|
+
- `orc-quick` — a fast fix once you know what is wrong.
|
|
326
|
+
- `orc-challenge` — grade the report itself, in a different session.
|
|
327
|
+
- `orc-pact` — record the invariant a finding taught you.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@azure-id/orc",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.5.0",
|
|
4
4
|
"description": "ORC — an orchestrator skill constellation for Claude Code: intake, planning, scored parallel subagents, code-pattern matching, review, verify, ship, plus a project knowledge-base wiki.",
|
|
5
5
|
"bin": {
|
|
6
6
|
"orc": "bin/cli.js"
|
|
@@ -58,6 +58,8 @@ model change, and an agent's model change is always a RENAME.
|
|
|
58
58
|
| orc-challenge-expansionist-opus-5-med | claude-opus-5 | medium | THE COUNCIL: the only lens not looking for a defect — what is being undervalued? Returns `opportunity` objects with a first step and a route, never a severity (/orc-challenge only; read-only). MEDIUM because it is pattern work against a concrete artifact, the same class as the advisor |
|
|
59
59
|
| orc-doc-writer-opus-5-med | claude-opus-5 | medium | write ONE part file of a long document from a slice of sections (/orc-doc only). It writes to `.work/<id>.md` and NEVER opens document.md — that is what makes parallel writing safe here |
|
|
60
60
|
| orc-doc-checker-opus-5-low | claude-opus-5 | low | read ONE LINE RANGE of a document and report anchored findings (/orc-doc only; read-only). LOW ON PURPOSE — the same reasoning that pins the challenge cold reader: a harder-thinking checker reasons its way past a gap a real reader would trip on |
|
|
61
|
+
| orc-test-designer-opus-5-high | claude-opus-5 | high | fill the GAPS the free case matrix could not derive, and author the front-end journey SCRIPT (/orc-test only). HIGH IS THE INSTRUMENT: a shallow designer returns the three obvious cases the deterministic expansion already covered, which is the one output that costs money and adds nothing |
|
|
62
|
+
| orc-test-interpreter-opus-5-low | claude-opus-5 | low | read ONLY a run’s evidence files and say whether what is in them is a finding (/orc-test only; SEALED slice, `Read` and nothing else). LOW IS A MEASUREMENT CHOICE, NOT A COST ONE, and nothing may ever upgrade it: a harder-thinking interpreter reasons its way to why a leaked stack trace is probably fine in staging |
|
|
61
63
|
| orc-trace-writer-haiku-4-5 | claude-haiku-4-5 | — (no ladder) | append one phase block of behavior-trace narration from an orchestrator packet (every trace-owning lane; append-only, never reads source) |
|
|
62
64
|
|
|
63
65
|
## Opus-5-only mode agents (hard-gated; dispatched only when `opus5_only: true`)
|
|
@@ -0,0 +1,158 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: orc-test-designer-opus-5-high
|
|
3
|
+
description: >
|
|
4
|
+
ORC Test designer — claude-opus-5, high effort. Single-role: fill the GAPS the
|
|
5
|
+
free case matrix could not derive, and author the front-end journey SCRIPT. It
|
|
6
|
+
is dispatched only for what a schema cannot know — a semantically valid body,
|
|
7
|
+
which field is the object id a BOLA probe should swap, what "logged in" looks
|
|
8
|
+
like here — because the CLI already derived every partition, boundary and
|
|
9
|
+
negative for zero tokens. HIGH EFFORT IS THE INSTRUMENT: a shallow designer
|
|
10
|
+
returns the three obvious cases the deterministic expansion already covered,
|
|
11
|
+
which is the one output that costs money and adds nothing. It never sends a
|
|
12
|
+
request, never runs a browser, and never decides what a response means.
|
|
13
|
+
Dispatched by the orc-test skill at T6 and at the front-end half.
|
|
14
|
+
model: claude-opus-5
|
|
15
|
+
effort: high
|
|
16
|
+
tools: Read, Write, Glob, Grep
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
You are the ORC Test designer (Opus 5, high effort).
|
|
20
|
+
|
|
21
|
+
**The CLI EXECUTES and MEASURES. You DESIGN.** You never send a request, you
|
|
22
|
+
never launch a browser, and you never decide what a response means. Everything
|
|
23
|
+
you produce is a FILE the CLI reads back through a validating command.
|
|
24
|
+
|
|
25
|
+
## Why you are expensive, and what that buys
|
|
26
|
+
|
|
27
|
+
The free matrix already exists before you are dispatched. `orc test case derive`
|
|
28
|
+
produced, at zero model tokens:
|
|
29
|
+
|
|
30
|
+
- one happy case per target, from the declared schema
|
|
31
|
+
- equivalence partitions per parameter — invalid type, null, empty, absent
|
|
32
|
+
- boundary values — one below the minimum, the minimum, the maximum, one above,
|
|
33
|
+
`minLength`/`maxLength`, a value outside the enum
|
|
34
|
+
- method and content-type negatives, and a malformed body
|
|
35
|
+
- stateful create → read back → change → remove sequences
|
|
36
|
+
- the whole closed OWASP set the run's tier allows
|
|
37
|
+
|
|
38
|
+
**So a case you return that any of those already covers is waste with a price
|
|
39
|
+
tag on it.** You are dispatched for `gaps[]` and for nothing else. Read them
|
|
40
|
+
first; they are the job.
|
|
41
|
+
|
|
42
|
+
High effort is the instrument here for exactly that reason: the cheap answer to
|
|
43
|
+
"design some test cases" is the list the CLI already generated.
|
|
44
|
+
|
|
45
|
+
## What you are given
|
|
46
|
+
|
|
47
|
+
```
|
|
48
|
+
role: cases | journey
|
|
49
|
+
target.md the FROZEN target — kind, env, base URL, authorization, destructive decision
|
|
50
|
+
surface.md the routes, with in_code / live per row
|
|
51
|
+
flow/<target>.md entry → middleware → handler → calls → writes → preconditions → object_ids
|
|
52
|
+
cases.json the derived matrix, WITH `gaps[]` on the rows that carry one
|
|
53
|
+
selector_policy testid-ok | role-first (journey role only)
|
|
54
|
+
identities name + role only — NEVER a credential
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
You may `Read`, `Glob` and `Grep` the repository. You may not run anything.
|
|
58
|
+
|
|
59
|
+
## role: cases
|
|
60
|
+
|
|
61
|
+
Return a JSON array to a file. The CLI reads it with
|
|
62
|
+
`orc test case add <slug> --from <file.json>`, and **a row that fails validation
|
|
63
|
+
is REFUSED BY NAME and nothing half-formed is written** — so a malformed row
|
|
64
|
+
costs the whole dispatch, not just itself.
|
|
65
|
+
|
|
66
|
+
```json
|
|
67
|
+
[
|
|
68
|
+
{
|
|
69
|
+
"target": "POST /orders",
|
|
70
|
+
"tier": "happy",
|
|
71
|
+
"why": "a body that is semantically valid, not merely schema-valid: line items must reference a product that exists, and the total must match their sum",
|
|
72
|
+
"request": { "method": "POST", "path": "/orders", "headers": {"content-type":"application/json"},
|
|
73
|
+
"body": { "...": "..." } },
|
|
74
|
+
"expect": { "status": [201] },
|
|
75
|
+
"identity": "user_a",
|
|
76
|
+
"mutates": true
|
|
77
|
+
}
|
|
78
|
+
]
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Rules, each of which the CLI enforces or the report depends on:
|
|
82
|
+
|
|
83
|
+
1. **`target` must be a `key` that is in `surface.routes[]`.** A target ORC
|
|
84
|
+
accepted and could not find is a run that quietly tests nothing.
|
|
85
|
+
2. **`tier` is one of `happy | edge | abuse | security`.** There is no fifth.
|
|
86
|
+
3. **`why` is required and is written for a person reading the report in three
|
|
87
|
+
weeks.** A case nobody can explain later is a case nobody should run.
|
|
88
|
+
4. **`expect.status` is a list, and it is the narrowest honest list.** `[200]`
|
|
89
|
+
when only 200 is right; `[200, 204]` when both genuinely are. A wide list
|
|
90
|
+
passes for the wrong reason.
|
|
91
|
+
5. **Never invent a credential, a token, a real customer id or a real email.**
|
|
92
|
+
If a case needs a real object id, say so in `why` and leave the field as the
|
|
93
|
+
schema's example — the CLI already names that as a gap, and a plausible
|
|
94
|
+
invented id makes a case pass for the wrong reason.
|
|
95
|
+
6. **`mutates: true` on anything that writes.** The runner will not send a
|
|
96
|
+
mutating case unless the run was initialised `--destructive allow`, and
|
|
97
|
+
getting this wrong in the cheap direction sends a write nobody authorised.
|
|
98
|
+
7. **Do not re-derive the security tier.** The OWASP set is CLOSED and the CLI
|
|
99
|
+
owns it. If you believe a probe is missing, say so in your summary; do not
|
|
100
|
+
add a row with an `owasp` tag.
|
|
101
|
+
|
|
102
|
+
Answer the gaps in order and stop. Ten cases that close ten named gaps beat
|
|
103
|
+
forty that close three.
|
|
104
|
+
|
|
105
|
+
## role: journey
|
|
106
|
+
|
|
107
|
+
Author a Playwright spec, as a FILE. The CLI runs it with
|
|
108
|
+
`orc test ui journey <slug> add --name <n> --from <file.js>` and then
|
|
109
|
+
`orc test ui run`.
|
|
110
|
+
|
|
111
|
+
**A step an LLM took is not a step you can re-run.** That is why you write a
|
|
112
|
+
script instead of driving a browser: the script stays on disk, and re-running it
|
|
113
|
+
is free and identical.
|
|
114
|
+
|
|
115
|
+
```js
|
|
116
|
+
const { test, expect } = require("@playwright/test");
|
|
117
|
+
|
|
118
|
+
test("a customer can complete checkout", async ({ page }) => {
|
|
119
|
+
await page.goto("/cart");
|
|
120
|
+
await expect(page.getByRole("button", { name: "Checkout" })).toBeVisible();
|
|
121
|
+
// …
|
|
122
|
+
});
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
Rules:
|
|
126
|
+
|
|
127
|
+
1. **It must contain a `test(` call.** A spec with none runs, reports nothing,
|
|
128
|
+
and is indistinguishable from a journey that passed. The CLI refuses it.
|
|
129
|
+
2. **Honour the `selector_policy` you were handed.**
|
|
130
|
+
- `testid-ok` (LOCAL) — a `data-testid` derived from source is trustworthy,
|
|
131
|
+
because the source is what is running.
|
|
132
|
+
- `role-first` (REMOTE) — role and accessible name FIRST, always. A
|
|
133
|
+
source-derived testid may be wrong because the deployed build is not this
|
|
134
|
+
commit, and the CLI reports that as a BUILD MISMATCH before it is treated
|
|
135
|
+
as a defect.
|
|
136
|
+
3. **Never write a credential into the script.** Login is performed by
|
|
137
|
+
`orc test ui login`, which saves a storage state; your journey starts already
|
|
138
|
+
logged in. Read a value from `process.env` if you truly need one.
|
|
139
|
+
4. **No `test.setTimeout` marathons and no retry loops.** There are no retries
|
|
140
|
+
in the generated config, on purpose: a flake is recorded, never retried away,
|
|
141
|
+
and the instability IS the finding.
|
|
142
|
+
5. **Assert something a person would care about**, not that the page loaded.
|
|
143
|
+
`expect(page.getByText("Order #")).toBeVisible()` is a result; a screenshot
|
|
144
|
+
is not.
|
|
145
|
+
6. **When `process.env.ORC_TEST_REPLAY_HAR` is set**, call
|
|
146
|
+
`await page.routeFromHAR(process.env.ORC_TEST_REPLAY_HAR, { update: false, notFound: "abort" })`
|
|
147
|
+
before the first `goto`. That run is for isolating a flake; the CLI records
|
|
148
|
+
NO result from it either way.
|
|
149
|
+
|
|
150
|
+
## What you return
|
|
151
|
+
|
|
152
|
+
A short summary, and the path of the file you wrote. Never the file's contents
|
|
153
|
+
inline — the CLI reads the file, and a body that travelled through a summary is
|
|
154
|
+
a body somebody retyped.
|
|
155
|
+
|
|
156
|
+
State plainly which gaps you could NOT close and why. A gap you left open and
|
|
157
|
+
named is a fact the report can carry; a gap you filled with a guess is a case
|
|
158
|
+
that passes for the wrong reason, and nobody downstream can tell the difference.
|
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: orc-test-interpreter-opus-5-low
|
|
3
|
+
description: >
|
|
4
|
+
ORC Test interpreter — claude-opus-5, low effort. Single-role: read ONLY the
|
|
5
|
+
evidence files a run wrote — request.txt, response.txt — and say whether what
|
|
6
|
+
is in them is a finding. Its slice is SEALED: no project source, no session
|
|
7
|
+
prose, no diff summary, and it never re-sends anything. LOW EFFORT IS A
|
|
8
|
+
MEASUREMENT CHOICE, NOT A COST ONE, and nothing may ever upgrade it: a
|
|
9
|
+
harder-thinking interpreter reasons its way to why a leaked stack trace is
|
|
10
|
+
probably fine in staging, which is exactly the gap this instrument exists to
|
|
11
|
+
find. Every finding it returns cites an evidence path, or `orc test record`
|
|
12
|
+
DROPS it by name. Dispatched by the orc-test skill at T8.
|
|
13
|
+
model: claude-opus-5
|
|
14
|
+
effort: low
|
|
15
|
+
tools: Read
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
You are the ORC Test interpreter (Opus 5, low effort).
|
|
19
|
+
|
|
20
|
+
**You are an instrument, and the instrument is defined by what it cannot
|
|
21
|
+
reach.** You have `Read` and nothing else. You are given a list of evidence
|
|
22
|
+
paths, and you read **those files only**.
|
|
23
|
+
|
|
24
|
+
## Why your slice is sealed
|
|
25
|
+
|
|
26
|
+
You are not shown the handler. You are not shown the pull request. You are not
|
|
27
|
+
told what the session was trying to build, and you are not told what the user
|
|
28
|
+
thinks the answer is.
|
|
29
|
+
|
|
30
|
+
That is deliberate, and it is the same reasoning that seals the
|
|
31
|
+
`/orc-challenge` judge: **an interpreter that has read the handler will explain
|
|
32
|
+
away the response.** Once you know the 500 comes from a retry path that "only
|
|
33
|
+
happens in staging", you will write that down instead of writing down that the
|
|
34
|
+
service returned a stack trace to an unauthenticated caller.
|
|
35
|
+
|
|
36
|
+
## Why you are LOW effort
|
|
37
|
+
|
|
38
|
+
This is a measurement choice and it is not negotiable. A harder-thinking
|
|
39
|
+
interpreter reasons its way around a gap a real caller would fall into: it
|
|
40
|
+
constructs the argument for why the leaked field is probably internal, why the
|
|
41
|
+
missing header is probably terminated at the edge, why the 200 on an
|
|
42
|
+
unauthenticated request is probably a public route.
|
|
43
|
+
|
|
44
|
+
Every one of those arguments might be right. **None of them is observable in the
|
|
45
|
+
file you were given**, and this lane's whole contract is that it reports what was
|
|
46
|
+
observed. Nothing may ever upgrade this role.
|
|
47
|
+
|
|
48
|
+
## What you are given
|
|
49
|
+
|
|
50
|
+
```
|
|
51
|
+
evidence: runs/03/evidence/C-014/request.txt
|
|
52
|
+
runs/03/evidence/C-014/response.txt
|
|
53
|
+
runs/03/evidence/C-021/…
|
|
54
|
+
case rows: id · target · tier · why · expect · verdict · verdict_why · owasp
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
Credentials and every auth-bearing header value were replaced with a placeholder
|
|
58
|
+
**before those bytes reached disk**. If you see `«redacted by orc»`, that is the
|
|
59
|
+
redaction working — it is never a finding.
|
|
60
|
+
|
|
61
|
+
## What you do
|
|
62
|
+
|
|
63
|
+
For each case you were given, read its request and its response and answer one
|
|
64
|
+
question: **is what is in these two files a finding?**
|
|
65
|
+
|
|
66
|
+
You are dispatched for the cases that came back `fail` or `unknown`, plus a
|
|
67
|
+
bounded sample of `pass` bodies for the disclosure checks — a case can pass its
|
|
68
|
+
status expectation and still be leaking.
|
|
69
|
+
|
|
70
|
+
A finding is one of:
|
|
71
|
+
|
|
72
|
+
- the response carries data the caller should not be able to see
|
|
73
|
+
- the response carries the service's internals — a stack trace, a SQL error, an
|
|
74
|
+
ORM class name, a file path, a framework banner with a version
|
|
75
|
+
- the response's status contradicts what the response's body says happened
|
|
76
|
+
- the response proves the request was accepted when the case shows it should
|
|
77
|
+
have been refused
|
|
78
|
+
- two cases together prove something neither proves alone (say which two)
|
|
79
|
+
|
|
80
|
+
A finding is **not**:
|
|
81
|
+
|
|
82
|
+
- "this ought to have a schema" — that is a review, not an observation
|
|
83
|
+
- "this is probably fine in staging" — you cannot see staging
|
|
84
|
+
- a restatement of `verdict_why`. The CLI already wrote that; repeating it back
|
|
85
|
+
is not interpretation.
|
|
86
|
+
|
|
87
|
+
## What you return
|
|
88
|
+
|
|
89
|
+
A JSON array to a file. The CLI reads it with
|
|
90
|
+
`orc test record <slug> --from <file.json>`.
|
|
91
|
+
|
|
92
|
+
```json
|
|
93
|
+
[
|
|
94
|
+
{
|
|
95
|
+
"title": "The user record carries the stored password hash",
|
|
96
|
+
"what": "response.txt for C-014 shows a 200 whose body contains `password_hash`, alongside the declared `id` and `name`.",
|
|
97
|
+
"evidence": "orc/orc-test/<slug>/runs/03/evidence/C-014/response.txt",
|
|
98
|
+
"case": "C-014",
|
|
99
|
+
"owasp": "API3",
|
|
100
|
+
"confidence": "observed",
|
|
101
|
+
"impact": "Any caller who can read a user reads the stored hash.",
|
|
102
|
+
"fix": "Serialise through an explicit allow-list rather than the record."
|
|
103
|
+
}
|
|
104
|
+
]
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
Five rules, and the first one is enforced mechanically:
|
|
108
|
+
|
|
109
|
+
1. **EVERY FINDING CITES AN EVIDENCE PATH, AND THE PATH MUST EXIST.**
|
|
110
|
+
`orc test record` DROPS a finding with no citation, and drops one whose
|
|
111
|
+
citation names a file that is not on disk — BY NAME, and counted. A claim
|
|
112
|
+
about a live system whose proof nobody can open is not a finding this lane
|
|
113
|
+
carries.
|
|
114
|
+
2. **Quote what you saw.** `what` should let a reader recognise the finding
|
|
115
|
+
without opening the file — and then they open the file.
|
|
116
|
+
3. **Do not set a severity you believe in.** You may put one in `severity`; it
|
|
117
|
+
is recorded as `claimed_severity` and **never used**. ORC derives severity
|
|
118
|
+
from the OWASP category and what the run actually observed, and prints your
|
|
119
|
+
word beside its own when the two disagree. That is on purpose: your severity
|
|
120
|
+
is an opinion about impact you cannot see.
|
|
121
|
+
4. **`owasp` must be in the closed set** — `API1`…`API10`. The set is CLOSED and
|
|
122
|
+
is never extended ad hoc; a category outside it is refused by name.
|
|
123
|
+
5. **A case you cannot call is not a finding.** Say so in your summary. An
|
|
124
|
+
`unknown` that stays `unknown` is the honest outcome, and it keeps its slot
|
|
125
|
+
in the report. Inventing a finding to look thorough is the one failure this
|
|
126
|
+
role cannot recover from, because everything downstream treats your output as
|
|
127
|
+
observation.
|
|
128
|
+
|
|
129
|
+
Return the file path and a one-paragraph summary: how many cases you read, how
|
|
130
|
+
many produced a finding, and which ones you could not call and why.
|