@openwop/openwop-conformance 2.46.6 → 2.46.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +17 -0
- package/README.md +2 -2
- package/dist/spec-artifacts.lock.json +2 -2
- package/fixtures/eval-suites/conformance-eval-fixture-miss.json +82 -0
- package/fixtures/eval-suites/conformance-eval-trajectory.json +202 -0
- package/fixtures/eval-suites/conformance-eval-trials.json +57 -0
- package/fixtures.md +19 -0
- package/package.json +2 -2
- package/requirements.json +240 -36
- package/scenario-majors.json +5 -2
- package/schemas/CORPUS-STAMP.json +15 -15
- package/src/lib/eval-trajectory.ts +146 -0
- package/src/scenarios/v2-eval-trajectory-trials.test.ts +253 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,20 @@
|
|
|
1
|
+
## [2.46.8] — 2026-10-14 — RFC 0244: eval trajectories, repeated trials, fail-closed fixture miss
|
|
2
|
+
|
|
3
|
+
- **New `v2-eval-trajectory-trials`:**
|
|
4
|
+
- Server-free: the RFC 0244 suite, summary and `eval.*` fields validate; the negatives and the content-free negatives (`eval-summary-no-content-leak`) are refused; the three eval-suite fixtures validate.
|
|
5
|
+
- Host legs, each gated on its `agents.evalSuite` member (`trajectory`, `trials`, `fixtureMatching`) and its fixture: `0244.trajectory-graded`, `0244.trials-per-task`, `0244.trial-stats-formula`, `0244.fixture-miss-fails`, and `0244.unadvertised-feature-refused` (a trials suite on a host without `trials` → `400 validation_error`).
|
|
6
|
+
- **New `lib/eval-trajectory.ts`:** the reference grader the host legs compare against, with self-tests (the four modes, `forbidden`, `argsMatch`, the mode-independent `matchedCount`, and the §B formulas over a fixed `(n, c, k)` table).
|
|
7
|
+
- **New `fixtures/eval-suites/`:** `conformance-eval-trajectory`, `conformance-eval-trials`, `conformance-eval-fixture-miss`. These are `AgentEvalSuite` documents run against the suite-defined scripted agent `core.conformance.agents.scripted` (contract in `fixtures.md`).
|
|
8
|
+
- **What a host's next bundle shows:** no host advertises the new members or seeds the fixtures, so every host leg is `inapplicable`. The server-free legs pass everywhere.
|
|
9
|
+
|
|
10
|
+
## [2.46.7] — 2026-10-14 — one id per leg; RFC 0243 G2 closes
|
|
11
|
+
|
|
12
|
+
- **RFC 0243 G2 closes.** No assertion changes; the packed `spec/v1/gaps.json` changes. openwop-app's certified 2.46.6 cut (build `44960bca5`) records all three G2 ids `executed-pass`.
|
|
13
|
+
- **Registry: a requirement id passed through a helper is registered.** A scenario writing `assert(ID_X, judge(o))`, where the helper calls `req(id, …)`, registered `explicitId: null`. `src/setup.ts` labels a soft-skipped row from `explicitId`, so on a host where the leg was `inapplicable` the row read `openwop.it.<file>.<slug>`, and on a host where it ran, the requirement id. One leg, two ids, depending on the host. Found by openwop-app-8f on `v2-agents-fan-out-waves`.
|
|
14
|
+
- The generator now takes the first `openwop.requirement.*` const passed as a call's first argument when the body has no `req()`. 20 records in 7 files gain their id, and no existing id changes: `v2-agents-fan-out-waves`, `v2-dispatch-loop`, `v2-dispatch-input-mapping`, `v2-subworkflow-input-mapping`, `v2-subworkflow-linkage`, `v2-host-event-delivery`, `v2-safefetch-ssrf-refused`.
|
|
15
|
+
- **What a host's next bundle shows:** those legs' non-executed rows are renamed from `openwop.it.*` to their requirement ids. Executed rows are unchanged. A bundle cut on an older suite lacks rows for the three 2.46.6 ids under `check-cut-gates` Witness, which is the usual currency effect.
|
|
16
|
+
- A self-test pins the registered ids for two of the files.
|
|
17
|
+
|
|
1
18
|
## [2.46.6] — 2026-10-13 — RFC 0243 G2: fan-out waves and per-item input
|
|
2
19
|
|
|
3
20
|
- **New `v2-agents-fan-out-waves`** (three ids, legs in `lib/fan-out-witness.ts`). The v1 twins drove a seam v2 does not carry; these run fixtures whose supervisor emits its `mockDispatchPlan`.
|
package/README.md
CHANGED
|
@@ -137,7 +137,7 @@ Each scenario file records its disposition and assertion count in `<report-dir>/
|
|
|
137
137
|
|
|
138
138
|
## What's Covered
|
|
139
139
|
|
|
140
|
-
The current suite has
|
|
140
|
+
The current suite has 658 scenario files under `src/scenarios/`. By target major ([`scenario-majors.json`](./scenario-majors.json)): 218 files run against a v2 host and 455 against a v1 host; 15 of them run at both.
|
|
141
141
|
|
|
142
142
|
The v2 scenarios (`src/scenarios/v2-*.test.ts`) cover, by area:
|
|
143
143
|
|
|
@@ -166,7 +166,7 @@ Three kinds of scenario share the tree:
|
|
|
166
166
|
- **Host** checks drive a live host over HTTP.
|
|
167
167
|
- **Gated** checks run only when the host advertises the family, profile or fixture they need, and otherwise record `inapplicable` or `blocked` with a reason. `--require-behavior` turns an advertised-but-unwitnessable behaviour into a failure.
|
|
168
168
|
|
|
169
|
-
Current source tree:
|
|
169
|
+
Current source tree: 658 scenario files. [`coverage.md`](./coverage.md) maps each spec document and every OpenAPI operation to its scenarios. [`CHANGELOG.md`](./CHANGELOG.md) records when each scenario landed.
|
|
170
170
|
|
|
171
171
|
## How the 2.x suite is built
|
|
172
172
|
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
{
|
|
2
|
+
"suiteId": "core.openwop.evals.conformance-eval-fixture-miss",
|
|
3
|
+
"version": "1.0.0",
|
|
4
|
+
"targetAgentId": "core.conformance.agents.scripted",
|
|
5
|
+
"modes": [
|
|
6
|
+
"golden"
|
|
7
|
+
],
|
|
8
|
+
"tasks": [
|
|
9
|
+
{
|
|
10
|
+
"taskId": "served",
|
|
11
|
+
"input": {
|
|
12
|
+
"script": [
|
|
13
|
+
{
|
|
14
|
+
"tool": "conformance.eval:lookup",
|
|
15
|
+
"arguments": {
|
|
16
|
+
"orderId": "o-1"
|
|
17
|
+
}
|
|
18
|
+
}
|
|
19
|
+
],
|
|
20
|
+
"output": "done"
|
|
21
|
+
},
|
|
22
|
+
"expected": {
|
|
23
|
+
"kind": "golden",
|
|
24
|
+
"match": {
|
|
25
|
+
"strategy": "contains",
|
|
26
|
+
"value": "done"
|
|
27
|
+
}
|
|
28
|
+
},
|
|
29
|
+
"fixtures": {
|
|
30
|
+
"onMiss": "fail",
|
|
31
|
+
"toolResponses": [
|
|
32
|
+
{
|
|
33
|
+
"tool": "conformance.eval:lookup",
|
|
34
|
+
"args": {
|
|
35
|
+
"orderId": "o-1"
|
|
36
|
+
},
|
|
37
|
+
"argsMatch": "exact",
|
|
38
|
+
"response": {
|
|
39
|
+
"status": "shipped"
|
|
40
|
+
}
|
|
41
|
+
}
|
|
42
|
+
]
|
|
43
|
+
}
|
|
44
|
+
},
|
|
45
|
+
{
|
|
46
|
+
"taskId": "missed",
|
|
47
|
+
"input": {
|
|
48
|
+
"script": [
|
|
49
|
+
{
|
|
50
|
+
"tool": "conformance.eval:lookup",
|
|
51
|
+
"arguments": {
|
|
52
|
+
"orderId": "o-2"
|
|
53
|
+
}
|
|
54
|
+
}
|
|
55
|
+
],
|
|
56
|
+
"output": "done"
|
|
57
|
+
},
|
|
58
|
+
"expected": {
|
|
59
|
+
"kind": "golden",
|
|
60
|
+
"match": {
|
|
61
|
+
"strategy": "contains",
|
|
62
|
+
"value": "done"
|
|
63
|
+
}
|
|
64
|
+
},
|
|
65
|
+
"fixtures": {
|
|
66
|
+
"onMiss": "fail",
|
|
67
|
+
"toolResponses": [
|
|
68
|
+
{
|
|
69
|
+
"tool": "conformance.eval:lookup",
|
|
70
|
+
"args": {
|
|
71
|
+
"orderId": "o-1"
|
|
72
|
+
},
|
|
73
|
+
"argsMatch": "exact",
|
|
74
|
+
"response": {
|
|
75
|
+
"status": "shipped"
|
|
76
|
+
}
|
|
77
|
+
}
|
|
78
|
+
]
|
|
79
|
+
}
|
|
80
|
+
}
|
|
81
|
+
]
|
|
82
|
+
}
|
|
@@ -0,0 +1,202 @@
|
|
|
1
|
+
{
|
|
2
|
+
"suiteId": "core.openwop.evals.conformance-eval-trajectory",
|
|
3
|
+
"version": "1.0.0",
|
|
4
|
+
"targetAgentId": "core.conformance.agents.scripted",
|
|
5
|
+
"modes": [
|
|
6
|
+
"golden"
|
|
7
|
+
],
|
|
8
|
+
"thresholds": {
|
|
9
|
+
"passScore": 0.5
|
|
10
|
+
},
|
|
11
|
+
"tasks": [
|
|
12
|
+
{
|
|
13
|
+
"taskId": "in-order-match",
|
|
14
|
+
"input": {
|
|
15
|
+
"script": [
|
|
16
|
+
{
|
|
17
|
+
"tool": "conformance.eval:lookup",
|
|
18
|
+
"arguments": {
|
|
19
|
+
"orderId": "o-1"
|
|
20
|
+
}
|
|
21
|
+
},
|
|
22
|
+
{
|
|
23
|
+
"tool": "conformance.eval:note",
|
|
24
|
+
"arguments": {
|
|
25
|
+
"text": "checked"
|
|
26
|
+
}
|
|
27
|
+
},
|
|
28
|
+
{
|
|
29
|
+
"tool": "conformance.eval:refund",
|
|
30
|
+
"arguments": {
|
|
31
|
+
"orderId": "o-1",
|
|
32
|
+
"amountMinor": 4200
|
|
33
|
+
}
|
|
34
|
+
}
|
|
35
|
+
],
|
|
36
|
+
"output": "done"
|
|
37
|
+
},
|
|
38
|
+
"expected": {
|
|
39
|
+
"kind": "golden",
|
|
40
|
+
"match": {
|
|
41
|
+
"strategy": "contains",
|
|
42
|
+
"value": "done"
|
|
43
|
+
},
|
|
44
|
+
"trajectory": {
|
|
45
|
+
"match": "in-order",
|
|
46
|
+
"calls": [
|
|
47
|
+
{
|
|
48
|
+
"tool": "conformance.eval:lookup"
|
|
49
|
+
},
|
|
50
|
+
{
|
|
51
|
+
"tool": "conformance.eval:refund",
|
|
52
|
+
"args": {
|
|
53
|
+
"amountMinor": 4200
|
|
54
|
+
},
|
|
55
|
+
"argsMatch": "subset"
|
|
56
|
+
}
|
|
57
|
+
]
|
|
58
|
+
}
|
|
59
|
+
}
|
|
60
|
+
},
|
|
61
|
+
{
|
|
62
|
+
"taskId": "in-order-wrong-order",
|
|
63
|
+
"input": {
|
|
64
|
+
"script": [
|
|
65
|
+
{
|
|
66
|
+
"tool": "conformance.eval:refund",
|
|
67
|
+
"arguments": {
|
|
68
|
+
"orderId": "o-1",
|
|
69
|
+
"amountMinor": 4200
|
|
70
|
+
}
|
|
71
|
+
},
|
|
72
|
+
{
|
|
73
|
+
"tool": "conformance.eval:lookup",
|
|
74
|
+
"arguments": {
|
|
75
|
+
"orderId": "o-1"
|
|
76
|
+
}
|
|
77
|
+
}
|
|
78
|
+
],
|
|
79
|
+
"output": "done"
|
|
80
|
+
},
|
|
81
|
+
"expected": {
|
|
82
|
+
"kind": "golden",
|
|
83
|
+
"match": {
|
|
84
|
+
"strategy": "contains",
|
|
85
|
+
"value": "done"
|
|
86
|
+
},
|
|
87
|
+
"trajectory": {
|
|
88
|
+
"match": "in-order",
|
|
89
|
+
"calls": [
|
|
90
|
+
{
|
|
91
|
+
"tool": "conformance.eval:lookup"
|
|
92
|
+
},
|
|
93
|
+
{
|
|
94
|
+
"tool": "conformance.eval:refund"
|
|
95
|
+
}
|
|
96
|
+
]
|
|
97
|
+
}
|
|
98
|
+
}
|
|
99
|
+
},
|
|
100
|
+
{
|
|
101
|
+
"taskId": "forbidden-hit",
|
|
102
|
+
"input": {
|
|
103
|
+
"script": [
|
|
104
|
+
{
|
|
105
|
+
"tool": "conformance.eval:lookup",
|
|
106
|
+
"arguments": {
|
|
107
|
+
"orderId": "o-1"
|
|
108
|
+
}
|
|
109
|
+
},
|
|
110
|
+
{
|
|
111
|
+
"tool": "conformance.eval:refund-all",
|
|
112
|
+
"arguments": {}
|
|
113
|
+
}
|
|
114
|
+
],
|
|
115
|
+
"output": "done"
|
|
116
|
+
},
|
|
117
|
+
"expected": {
|
|
118
|
+
"kind": "golden",
|
|
119
|
+
"match": {
|
|
120
|
+
"strategy": "contains",
|
|
121
|
+
"value": "done"
|
|
122
|
+
},
|
|
123
|
+
"trajectory": {
|
|
124
|
+
"match": "any-order",
|
|
125
|
+
"calls": [
|
|
126
|
+
{
|
|
127
|
+
"tool": "conformance.eval:lookup"
|
|
128
|
+
}
|
|
129
|
+
],
|
|
130
|
+
"forbidden": [
|
|
131
|
+
"conformance.eval:refund-all"
|
|
132
|
+
]
|
|
133
|
+
}
|
|
134
|
+
}
|
|
135
|
+
},
|
|
136
|
+
{
|
|
137
|
+
"taskId": "trajectory-only-subset",
|
|
138
|
+
"input": {
|
|
139
|
+
"script": [
|
|
140
|
+
{
|
|
141
|
+
"tool": "conformance.eval:lookup",
|
|
142
|
+
"arguments": {
|
|
143
|
+
"orderId": "o-1"
|
|
144
|
+
}
|
|
145
|
+
},
|
|
146
|
+
{
|
|
147
|
+
"tool": "conformance.eval:lookup",
|
|
148
|
+
"arguments": {
|
|
149
|
+
"orderId": "o-2"
|
|
150
|
+
}
|
|
151
|
+
}
|
|
152
|
+
],
|
|
153
|
+
"output": "done"
|
|
154
|
+
},
|
|
155
|
+
"expected": {
|
|
156
|
+
"kind": "trajectory",
|
|
157
|
+
"trajectory": {
|
|
158
|
+
"match": "subset",
|
|
159
|
+
"calls": [
|
|
160
|
+
{
|
|
161
|
+
"tool": "conformance.eval:lookup"
|
|
162
|
+
},
|
|
163
|
+
{
|
|
164
|
+
"tool": "conformance.eval:refund"
|
|
165
|
+
}
|
|
166
|
+
]
|
|
167
|
+
}
|
|
168
|
+
}
|
|
169
|
+
},
|
|
170
|
+
{
|
|
171
|
+
"taskId": "exact-args-mismatch",
|
|
172
|
+
"input": {
|
|
173
|
+
"script": [
|
|
174
|
+
{
|
|
175
|
+
"tool": "conformance.eval:refund",
|
|
176
|
+
"arguments": {
|
|
177
|
+
"orderId": "o-1",
|
|
178
|
+
"amountMinor": 100
|
|
179
|
+
}
|
|
180
|
+
}
|
|
181
|
+
],
|
|
182
|
+
"output": "done"
|
|
183
|
+
},
|
|
184
|
+
"expected": {
|
|
185
|
+
"kind": "trajectory",
|
|
186
|
+
"trajectory": {
|
|
187
|
+
"match": "exact",
|
|
188
|
+
"calls": [
|
|
189
|
+
{
|
|
190
|
+
"tool": "conformance.eval:refund",
|
|
191
|
+
"args": {
|
|
192
|
+
"orderId": "o-1",
|
|
193
|
+
"amountMinor": 4200
|
|
194
|
+
},
|
|
195
|
+
"argsMatch": "exact"
|
|
196
|
+
}
|
|
197
|
+
]
|
|
198
|
+
}
|
|
199
|
+
}
|
|
200
|
+
}
|
|
201
|
+
]
|
|
202
|
+
}
|
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
{
|
|
2
|
+
"suiteId": "core.openwop.evals.conformance-eval-trials",
|
|
3
|
+
"version": "1.0.0",
|
|
4
|
+
"targetAgentId": "core.conformance.agents.scripted",
|
|
5
|
+
"modes": [
|
|
6
|
+
"golden"
|
|
7
|
+
],
|
|
8
|
+
"trials": {
|
|
9
|
+
"count": 3,
|
|
10
|
+
"k": 2,
|
|
11
|
+
"passRule": "all"
|
|
12
|
+
},
|
|
13
|
+
"tasks": [
|
|
14
|
+
{
|
|
15
|
+
"taskId": "always-passes",
|
|
16
|
+
"input": {
|
|
17
|
+
"script": [
|
|
18
|
+
{
|
|
19
|
+
"tool": "conformance.eval:lookup",
|
|
20
|
+
"arguments": {
|
|
21
|
+
"orderId": "o-1"
|
|
22
|
+
}
|
|
23
|
+
}
|
|
24
|
+
],
|
|
25
|
+
"output": "done"
|
|
26
|
+
},
|
|
27
|
+
"expected": {
|
|
28
|
+
"kind": "golden",
|
|
29
|
+
"match": {
|
|
30
|
+
"strategy": "contains",
|
|
31
|
+
"value": "done"
|
|
32
|
+
}
|
|
33
|
+
}
|
|
34
|
+
},
|
|
35
|
+
{
|
|
36
|
+
"taskId": "always-fails",
|
|
37
|
+
"input": {
|
|
38
|
+
"script": [
|
|
39
|
+
{
|
|
40
|
+
"tool": "conformance.eval:lookup",
|
|
41
|
+
"arguments": {
|
|
42
|
+
"orderId": "o-1"
|
|
43
|
+
}
|
|
44
|
+
}
|
|
45
|
+
],
|
|
46
|
+
"output": "done"
|
|
47
|
+
},
|
|
48
|
+
"expected": {
|
|
49
|
+
"kind": "golden",
|
|
50
|
+
"match": {
|
|
51
|
+
"strategy": "contains",
|
|
52
|
+
"value": "never-in-output"
|
|
53
|
+
}
|
|
54
|
+
}
|
|
55
|
+
}
|
|
56
|
+
]
|
|
57
|
+
}
|
package/fixtures.md
CHANGED
|
@@ -69,6 +69,9 @@ All fixtures MUST advertise:
|
|
|
69
69
|
| Prompt All Four Kinds | `conformance-prompt-all-four-kinds` | RFC 0027 §A four-kind dispatch coverage with a MULTI-ENTRY `fewShotPromptRefs[]` array. Single `mock-ai` node with one ref per singular-kind slot (`systemPromptRef`, `userPromptRef`, `schemaHintPromptRef`) + two distinct templateIds in `fewShotPromptRefs[]`; host MUST emit 5 `agent.promptResolved` events (one per slot) AND 5 `prompt.composed` events. Multi-entry few-shot is the regression pin for `fewShotPromptRefs[slotIndex]` per-index resolution — a host that hard-codes `[0]` would emit the same template twice in the few-shot events and fail the per-templateId assertion. Capability-gated on `capabilities.prompts.supported`. | `completed` | ≤ 10s |
|
|
70
70
|
| Dispatch Loop | `conformance-dispatch-loop` | Verifies `core.dispatch` loop mechanism | `completed` | ≤ 30s |
|
|
71
71
|
| Dispatch Fan-Out Race | `conformance-dispatch-fanout-race` | RFC 0243 §A.2 — a `core.dispatch` node pinning `fanOutPolicy: 'parallel'` with a `race` join. A host that does not admit it refuses it at load, or at `createRun` with `400 validation_error` and `details.requiredCapability` (`v2-agents-fan-out`) | refused; no run | ≤ 5s |
|
|
72
|
+
| Eval — Trajectory | `conformance-eval-trajectory` | RFC 0244 §A — an `AgentEvalSuite` under `eval-suites/` (not a workflow), run with `mode: eval` against the scripted agent; five tasks whose tool paths match / miss each mode and a forbidden call (`v2-eval-trajectory-trials`) | eval run `completed`; per-task grade as the reference grader | ≤ 30s |
|
|
73
|
+
| Eval — Trials | `conformance-eval-trials` | RFC 0244 §B — two tasks × `trials.count: 3`, `k: 2`; one always passes, one always fails (`v2-eval-trajectory-trials`) | eval run `completed`, or `400` where `trials` is unadvertised | ≤ 30s |
|
|
74
|
+
| Eval — Fixture Miss | `conformance-eval-fixture-miss` | RFC 0244 §D — `onMiss: "fail"`; one task served by an args-matched fixture, one that misses it (`v2-eval-trajectory-trials`) | eval run `completed`; the missed task fails | ≤ 30s |
|
|
72
75
|
| Dispatch Fan-Out Waves | `conformance-dispatch-fanout-waves` | RFC 0243 §A.3 — a parallel `core.dispatch` (`maxConcurrency: 2`, `wait-all`/`collect`) whose supervisor's `mockDispatchPlan` names three `conformance-delay` children. At most `min(2, maxFanOut)` run at once and none is dropped. Gated on `agents.fanOut` admitting `wait-all` with `collect`. | `completed` | ≤ 30s |
|
|
73
76
|
| Dispatch Per-Item Input | `conformance-dispatch-per-item` | RFC 0243 §A.4 — a sequential `core.dispatch` whose decision dispatches `conformance-dispatch-input-mapping-child` twice with `nextWorkerInputs` `item-a`, `item-b`. At `agents.fanOut.perItemInput: true` each child receives its item; otherwise the dispatch node fails `validation_error` and no child is dispatched. | `completed`, or `failed` without `perItemInput` | ≤ 30s |
|
|
74
77
|
| Dispatch Per-Item Length Mismatch | `conformance-dispatch-per-item-mismatch` | `orchestrator-decision.schema.json` `nextWorkerInputs` — one item for two children. Every dispatch host fails the dispatch node `validation_error` and dispatches no child. | `failed` | ≤ 30s |
|
|
@@ -1012,6 +1015,22 @@ Unlike the other sub-directories, this one also carries **deliberately-invalid**
|
|
|
1012
1015
|
|
|
1013
1016
|
---
|
|
1014
1017
|
|
|
1018
|
+
## Eval-suite fixtures (RFC 0244)
|
|
1019
|
+
|
|
1020
|
+
The `fixtures/eval-suites/` sub-directory holds `AgentEvalSuite` documents (`../schemas/v2/agent-eval-suite.schema.json`), validated server-free by `v2-eval-trajectory-trials.test.ts`. They are not workflows, so the top-level sweep never reads them. A host that runs them seeds each verbatim, lists its id (the file stem) in discovery `fixtures`, and resolves `evalSuiteRef: "https://conformance.openwop.dev/eval-suites/<id>"` to it.
|
|
1021
|
+
|
|
1022
|
+
Every task targets **the scripted agent `core.conformance.agents.scripted`**. Its contract:
|
|
1023
|
+
|
|
1024
|
+
- `input.script` is an ordered list of `{ tool, arguments? }`. The agent makes exactly those tool calls, in order, each emitting `agent.tool-called` with `agentId: "core.conformance.agents.scripted"`, `toolName: tool` and `inputs: arguments`.
|
|
1025
|
+
- Each call goes through the host's eval tool boundary. In a task with `fixtures.toolResponses`, the RFC 0244 §D selection and `onMiss` rules apply. Otherwise the host answers the call with its own `arguments`. No scripted call reaches a live tool or an external effect.
|
|
1026
|
+
- After the last call the agent decides, and its output is `input.output`.
|
|
1027
|
+
|
|
1028
|
+
| Fixture | Facet | What the suite asserts |
|
|
1029
|
+
| --- | --- | --- |
|
|
1030
|
+
| `conformance-eval-trajectory` | `agents.evalSuite.trajectory` | each task's `trajectory` counts equal `lib/eval-trajectory.ts`'s grade of the script; an unmatched trajectory fails; a `kind: "trajectory"` task scores 1 or 0; the log carries one `agent.tool-called` per scripted call |
|
|
1031
|
+
| `conformance-eval-trials` | `agents.evalSuite.trials` | `eval.started.trialCount: 3`; three `eval.scored` per task with `trialIndex` 0..2; the per-task `trials` block and `trialStats` equal the §B formulas. Without the facet: `400 validation_error` at `createRun` |
|
|
1032
|
+
| `conformance-eval-fixture-miss` | `agents.evalSuite.fixtureMatching` | `missed` reports `fixtureMissCount: 1` and `passed: false`; `served` passes with no miss. That no live tool ran is not observable here (RFC 0244 G2) |
|
|
1033
|
+
|
|
1015
1034
|
## OAuth provider fixtures
|
|
1016
1035
|
|
|
1017
1036
|
RFC 0199 adds `provider.issuer`, `provider.pkce` and the two issuer-less providers (`issuerlessProviders[]`) the distinct-redirect-URI leg needs; the endpoints a host actually talks to are supplied at run time by the `oauth/authorize-start` seam, which points them at the suite's authorization-server double (`src/lib/oauth-as-double.ts`).
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@openwop/openwop-conformance",
|
|
3
|
-
"version": "2.46.
|
|
3
|
+
"version": "2.46.8",
|
|
4
4
|
"description": "Black-box conformance suite for OpenWOP hosts (v2 current; v1 scenarios retained for the frozen v1 tree).",
|
|
5
5
|
"repository": {
|
|
6
6
|
"type": "git",
|
|
@@ -56,6 +56,6 @@
|
|
|
56
56
|
"@openwop/spec-artifacts": "file:../spec-artifacts"
|
|
57
57
|
},
|
|
58
58
|
"peerDependencies": {
|
|
59
|
-
"@openwop/spec-artifacts": "2.46.
|
|
59
|
+
"@openwop/spec-artifacts": "2.46.8"
|
|
60
60
|
}
|
|
61
61
|
}
|