overseer-testing-protocol 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/LICENSE +19 -0
- data/README.md +75 -0
- data/SOURCE.json +6 -0
- data/bin/overseer-testing-conformance +53 -0
- data/docs/testing-control-implementation-guide.md +116 -0
- data/docs/testing-control-protocol.md +432 -0
- data/lib/overseer/testing_control/conformance/case_file.rb +120 -0
- data/lib/overseer/testing_control/conformance/http_transport.rb +114 -0
- data/lib/overseer/testing_control/conformance/report.rb +104 -0
- data/lib/overseer/testing_control/conformance/runner.rb +783 -0
- data/lib/overseer/testing_control/discovery.rb +79 -0
- data/lib/overseer/testing_control/json_subset_matcher.rb +120 -0
- data/lib/overseer/testing_control/protocol_v3.rb +298 -0
- data/lib/overseer/testing_control/redaction.rb +111 -0
- data/lib/overseer/testing_protocol.rb +9 -0
- data/protocol/testing-control/v3/conformance-case.schema.json +151 -0
- data/protocol/testing-control/v3/conformance-report.schema.json +255 -0
- data/protocol/testing-control/v3/fixtures/capabilities-response.json +147 -0
- data/protocol/testing-control/v3/fixtures/conformance-case.json +23 -0
- data/protocol/testing-control/v3/fixtures/conformance-report.json +88 -0
- data/protocol/testing-control/v3/fixtures/error-response.json +20 -0
- data/protocol/testing-control/v3/fixtures/manifest.json +13 -0
- data/protocol/testing-control/v3/fixtures/probe-request.json +20 -0
- data/protocol/testing-control/v3/fixtures/probe-response.json +22 -0
- data/protocol/testing-control/v3/fixtures/reset-response.json +15 -0
- data/protocol/testing-control/v3/fixtures/sink-query-request.json +26 -0
- data/protocol/testing-control/v3/fixtures/sink-query-response.json +39 -0
- data/protocol/testing-control/v3/fixtures/state-request.json +20 -0
- data/protocol/testing-control/v3/fixtures/state-response.json +23 -0
- data/protocol/testing-control/v3/openapi.yaml +343 -0
- data/protocol/testing-control/v3/schema.json +772 -0
- metadata +85 -0
|
@@ -0,0 +1,432 @@
|
|
|
1
|
+
# Testing-control protocol v3
|
|
2
|
+
|
|
3
|
+
Status: canonical framework-neutral wire specification. Testing-control v3 is
|
|
4
|
+
the sole current testing-control protocol.
|
|
5
|
+
|
|
6
|
+
The normative artifacts are:
|
|
7
|
+
|
|
8
|
+
- [`../protocol/testing-control/v3/openapi.yaml`](../protocol/testing-control/v3/openapi.yaml)
|
|
9
|
+
for routes, methods, headers, media types, and response statuses;
|
|
10
|
+
- [`../protocol/testing-control/v3/schema.json`](../protocol/testing-control/v3/schema.json)
|
|
11
|
+
for wire request and response envelopes;
|
|
12
|
+
- [`../protocol/testing-control/v3/fixtures/`](../protocol/testing-control/v3/fixtures/)
|
|
13
|
+
for framework-neutral examples;
|
|
14
|
+
- [`../protocol/testing-control/v3/conformance-case.schema.json`](../protocol/testing-control/v3/conformance-case.schema.json)
|
|
15
|
+
for trusted declarative backend scenarios; and
|
|
16
|
+
- [`../protocol/testing-control/v3/conformance-report.schema.json`](../protocol/testing-control/v3/conformance-report.schema.json)
|
|
17
|
+
for bounded conformance diagnostic output.
|
|
18
|
+
|
|
19
|
+
The OpenAPI document and wire schema take precedence if prose differs. The
|
|
20
|
+
case and report formats support conformance; they are not additional
|
|
21
|
+
testing-control protocols. The [framework-neutral implementation
|
|
22
|
+
guide](./testing-control-implementation-guide.md) explains direct adoption.
|
|
23
|
+
|
|
24
|
+
Requirement words **MUST**, **MUST NOT**, **SHOULD**, and **MAY** are normative.
|
|
25
|
+
|
|
26
|
+
## 1. Scope and non-goals
|
|
27
|
+
|
|
28
|
+
Testing control is a test-only mechanism for:
|
|
29
|
+
|
|
30
|
+
- discovering the exact protocol and safe product-owned capabilities;
|
|
31
|
+
- witnessing application and test-environment identity;
|
|
32
|
+
- returning a synthetic runtime to its declared baseline;
|
|
33
|
+
- arranging allowlisted, schema-validated synthetic preconditions;
|
|
34
|
+
- observing bounded, sanitized, read-only projections; and
|
|
35
|
+
- querying bounded records captured by fake external-effect sinks.
|
|
36
|
+
|
|
37
|
+
Testing control MUST NOT become a public business API, general database
|
|
38
|
+
seeding language, factory API, model API, arbitrary query interface, command
|
|
39
|
+
runner, workflow engine, authorization bypass, or replacement for the public
|
|
40
|
+
OpenAPI contract. Public API calls and public workflow execution remain
|
|
41
|
+
outside this protocol.
|
|
42
|
+
|
|
43
|
+
The protocol calls product-owned state capabilities **arrangements** and
|
|
44
|
+
product-owned probe and fake-sink capabilities **observations**. Their current
|
|
45
|
+
wire route names remain `states`, `probes`, and `sinks`.
|
|
46
|
+
|
|
47
|
+
## 2. Trust and deployment assumptions
|
|
48
|
+
|
|
49
|
+
An implementation MUST expose testing control only inside a verified
|
|
50
|
+
synthetic test environment. It MUST fail closed unless the backend's normal
|
|
51
|
+
test-mode checks and the test runner's supplied runtime identity are valid.
|
|
52
|
+
Authentication inside a production-capable environment is not a substitute.
|
|
53
|
+
|
|
54
|
+
The runtime MUST use synthetic data, MUST omit real provider and user
|
|
55
|
+
credentials, MUST NOT persist secrets, and MUST block or explicitly allowlist
|
|
56
|
+
external network access. An implementation MUST NOT accept arbitrary commands,
|
|
57
|
+
scripts, SQL, repository paths, runtime URLs, direct database operations, or
|
|
58
|
+
credentials through any testing-control payload.
|
|
59
|
+
|
|
60
|
+
The test runner owns runtime creation or explicit attachment, network containment,
|
|
61
|
+
source and process verification, lifecycle cleanup, and evidence sanitization.
|
|
62
|
+
The product owns its domain invariants, capability handlers, read-only
|
|
63
|
+
projections, safe fields, and fake-effect interception.
|
|
64
|
+
|
|
65
|
+
## 3. Transport and endpoints
|
|
66
|
+
|
|
67
|
+
The backend and test client agree on one testing base path. Exactly
|
|
68
|
+
these routes are relative to that base path:
|
|
69
|
+
|
|
70
|
+
```text
|
|
71
|
+
GET /capabilities
|
|
72
|
+
PUT /reset
|
|
73
|
+
POST /states
|
|
74
|
+
POST /probes
|
|
75
|
+
POST /sinks/query
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
Clients and servers MUST use HTTP over the already verified loopback origin.
|
|
79
|
+
Testing-control redirects and arbitrary absolute targets are unsupported.
|
|
80
|
+
Requests with bodies and all responses MUST use UTF-8 JSON with media type
|
|
81
|
+
`application/json`. `GET /capabilities` and `PUT /reset` MUST have no body.
|
|
82
|
+
Malformed JSON and a non-JSON request media type MUST fail as `invalid-request`
|
|
83
|
+
with HTTP 400.
|
|
84
|
+
|
|
85
|
+
An adapter MUST enforce its advertised `maxRequestBytes` and
|
|
86
|
+
`maxResponseBytes`. It MUST reject an oversized request before executing a
|
|
87
|
+
handler and MUST NOT construct or stream an unbounded response. Clients MUST
|
|
88
|
+
use finite connection and response timeouts and MUST stop reading after the
|
|
89
|
+
applicable byte limit. Exact client timeout values are runner policy, not wire
|
|
90
|
+
semantics.
|
|
91
|
+
|
|
92
|
+
Protocol envelopes are closed: fields not admitted by `schema.json` MUST be
|
|
93
|
+
rejected. Product-owned capability payloads follow their advertised schemas;
|
|
94
|
+
the product schema deliberately controls whether its own additional fields are
|
|
95
|
+
allowed.
|
|
96
|
+
|
|
97
|
+
## 4. Exact version negotiation
|
|
98
|
+
|
|
99
|
+
Every request MUST carry:
|
|
100
|
+
|
|
101
|
+
| Header | Requirement |
|
|
102
|
+
| ---------------------------------- | ------------------------------------------------ |
|
|
103
|
+
| `Overseer-Testing-Control-Version` | Exact value `3`. |
|
|
104
|
+
| `Overseer-Run-Id` | Opaque valid identifier for the current attempt. |
|
|
105
|
+
| `Overseer-Correlation-Id` | Opaque valid identifier for this control call. |
|
|
106
|
+
|
|
107
|
+
`Overseer-Step-Id` MAY identify attributable setup or public-step context.
|
|
108
|
+
`Overseer-Idempotency-Key` MAY be used for reset and MUST be used when an
|
|
109
|
+
arrangement advertises required idempotency.
|
|
110
|
+
|
|
111
|
+
Identifiers contain 1 through 128 ASCII characters, begin with a letter or
|
|
112
|
+
digit, and then contain only letters, digits, `.`, `_`, `:`, or `-`.
|
|
113
|
+
|
|
114
|
+
Every response, including an error, MUST set
|
|
115
|
+
`Overseer-Testing-Control-Version: 3` and a valid
|
|
116
|
+
`Overseer-Correlation-Id`. For a valid request correlation ID, the response
|
|
117
|
+
MUST echo it. The body protocol member MUST be exactly:
|
|
118
|
+
|
|
119
|
+
```json
|
|
120
|
+
{
|
|
121
|
+
"name": "overseer-testing-control",
|
|
122
|
+
"version": "3"
|
|
123
|
+
}
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
A request for any version other than `3` MUST return HTTP 409 with
|
|
127
|
+
`protocol-version-unsupported`. It MUST NOT downgrade, dispatch to a legacy
|
|
128
|
+
implementation, or serve a simultaneous v1/v2 compatibility path. Testing
|
|
129
|
+
control v4 does not exist.
|
|
130
|
+
|
|
131
|
+
## 5. Envelope identity
|
|
132
|
+
|
|
133
|
+
Every normal request and successful response has `protocol`, `meta`, and
|
|
134
|
+
`data`; every error has `protocol`, `meta`, and `errors`. `meta.runId` and
|
|
135
|
+
`meta.correlationId` MUST match their headers. `meta.stepId` MUST be present
|
|
136
|
+
and equal to its header when supplied and MUST otherwise be absent.
|
|
137
|
+
|
|
138
|
+
When required identity headers are absent or invalid, an adapter MUST return a
|
|
139
|
+
schema-valid safe error identity. Generated placeholder identifiers MAY be
|
|
140
|
+
used; raw invalid values MUST NOT be echoed.
|
|
141
|
+
|
|
142
|
+
## 6. Runtime identity and independently verified facts
|
|
143
|
+
|
|
144
|
+
`GET /capabilities` owns these testing-control identity facts:
|
|
145
|
+
|
|
146
|
+
- `data.application.id`: stable backend integration ID;
|
|
147
|
+
- `data.application.version`: exact source identity injected into this
|
|
148
|
+
runtime;
|
|
149
|
+
- `data.environment.id`: opaque environment ID injected by the runtime owner;
|
|
150
|
+
- `data.environment.kind`: exact value `test`; and
|
|
151
|
+
- `data.environment.isolated`: exact value `true` for a verified isolated
|
|
152
|
+
runtime.
|
|
153
|
+
|
|
154
|
+
The test client MUST compare those values with the identities independently
|
|
155
|
+
established when starting or attaching to the backend, before reset,
|
|
156
|
+
arrangement, or observation. A mismatch is protocol nonconformance and MUST
|
|
157
|
+
stop dependent stateful checks.
|
|
158
|
+
|
|
159
|
+
Health status, loopback origin, source version resolution, whether the test
|
|
160
|
+
runner created the process, container/network facts, and cleanup result are
|
|
161
|
+
facts the runtime owner verifies independently. They MUST NOT be invented by the
|
|
162
|
+
testing-control response or treated as product business evidence.
|
|
163
|
+
|
|
164
|
+
## 7. Capability discovery
|
|
165
|
+
|
|
166
|
+
`GET /capabilities` MUST return:
|
|
167
|
+
|
|
168
|
+
- application and environment identity;
|
|
169
|
+
- one reset strategy and its sink-clearing guarantee;
|
|
170
|
+
- the required replay features;
|
|
171
|
+
- size and record limits;
|
|
172
|
+
- the safety declaration;
|
|
173
|
+
- zero or more arrangement descriptors in `states`;
|
|
174
|
+
- zero or more read-only observation descriptors in `probes`; and
|
|
175
|
+
- zero or more fake-sink observation descriptors in `sinks`.
|
|
176
|
+
|
|
177
|
+
The required features are `correlation-propagation`,
|
|
178
|
+
`deterministic-replay`, `reset`, `schema-validation`, and `sink-clearing`.
|
|
179
|
+
The safety declaration MUST require synthetic data, forbid real credentials
|
|
180
|
+
and secret persistence, and declare external networking as `blocked` or
|
|
181
|
+
`allowlisted`.
|
|
182
|
+
|
|
183
|
+
Every capability is selected by an opaque product-owned `id` and positive
|
|
184
|
+
decimal `version`. An `id@version` pair MUST be unique inside its family.
|
|
185
|
+
The test client MUST NOT enumerate or interpret product capability names.
|
|
186
|
+
|
|
187
|
+
Arrangement descriptors MUST advertise input and output JSON Schemas and an
|
|
188
|
+
idempotency policy of `required`, `supported`, or `none`. Probe descriptors
|
|
189
|
+
MUST advertise `readOnly: true` plus input and output schemas. Sink descriptors
|
|
190
|
+
MUST advertise effect kinds, outcomes, a query-input schema, and a record
|
|
191
|
+
payload schema.
|
|
192
|
+
|
|
193
|
+
Advertised schemas MUST use JSON Schema Draft 2020-12, MUST have an object
|
|
194
|
+
root, MUST fit `maxSchemaBytes`, MUST compile, and MUST use only local fragment
|
|
195
|
+
references. The test client MUST NOT fetch an external schema URI supplied by a
|
|
196
|
+
backend. The schemas remain product-owned and opaque even though the generic
|
|
197
|
+
client validates values against them.
|
|
198
|
+
|
|
199
|
+
## 8. Reset
|
|
200
|
+
|
|
201
|
+
The advertised reset strategy is exactly `in-process` or `runtime-restart`.
|
|
202
|
+
Both strategies MUST return the product to its declared synthetic baseline and
|
|
203
|
+
MUST clear arrangement idempotency state and fake-sink records before the next
|
|
204
|
+
run.
|
|
205
|
+
|
|
206
|
+
For `in-process`, `PUT /reset` MUST return HTTP 200 with `resetAt`, matching
|
|
207
|
+
strategy `in-process`, and `sinksCleared: true`. Repeating reset in the same
|
|
208
|
+
isolated runtime MUST be safe and return the same declared baseline.
|
|
209
|
+
|
|
210
|
+
For `runtime-restart`, `PUT /reset` MUST return HTTP 409 with
|
|
211
|
+
`reset-requires-restart`. The runtime driver, not a product handler, recreates
|
|
212
|
+
the isolated process and writable data, repeats normal test setup and health
|
|
213
|
+
verification, and negotiates capabilities again. The replacement MUST preserve
|
|
214
|
+
application source identity, environment identity, protocol v3, and
|
|
215
|
+
the declared reset strategy. An explicitly attached process cannot satisfy
|
|
216
|
+
runtime-restart unless its owner supplies a verified restart callback.
|
|
217
|
+
|
|
218
|
+
Reset failure MUST prevent later arrangements and observations that depend on
|
|
219
|
+
a clean baseline. Independent non-mutating protocol checks MAY continue.
|
|
220
|
+
|
|
221
|
+
## 9. Arrangement
|
|
222
|
+
|
|
223
|
+
`POST /states` selects one advertised arrangement:
|
|
224
|
+
|
|
225
|
+
```json
|
|
226
|
+
{
|
|
227
|
+
"protocol": { "name": "overseer-testing-control", "version": "3" },
|
|
228
|
+
"meta": {
|
|
229
|
+
"runId": "run-001",
|
|
230
|
+
"correlationId": "arrange-001",
|
|
231
|
+
"stepId": "setup-open-item"
|
|
232
|
+
},
|
|
233
|
+
"data": {
|
|
234
|
+
"capability": { "id": "open-item", "version": "1" },
|
|
235
|
+
"input": { "title": "Synthetic item" }
|
|
236
|
+
}
|
|
237
|
+
}
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
The adapter MUST locate the exact advertised `id@version`, validate `input`
|
|
241
|
+
against that descriptor's schema, execute only the registered product-owned
|
|
242
|
+
handler, validate the output, and return the same capability reference plus
|
|
243
|
+
bounded canonical identifiers and output. Product execution remains opaque to
|
|
244
|
+
the client.
|
|
245
|
+
|
|
246
|
+
An unknown capability MUST return HTTP 404 with `capability-not-found`. A
|
|
247
|
+
schema-invalid input MUST return HTTP 422 with `schema-validation-failed`.
|
|
248
|
+
Required idempotency MUST fail when its header is absent. Retrying an unchanged
|
|
249
|
+
key and input MUST NOT create a second logical arrangement; reusing a key with
|
|
250
|
+
different input MUST fail safely.
|
|
251
|
+
|
|
252
|
+
An arrangement is setup, not a public product mutation or evidence that a
|
|
253
|
+
consumer can create the same state.
|
|
254
|
+
|
|
255
|
+
## 10. Observation
|
|
256
|
+
|
|
257
|
+
`POST /probes` selects one advertised read-only probe, validates its input,
|
|
258
|
+
executes only its registered product-owned observation, validates the result,
|
|
259
|
+
and returns the same capability, `observedAt`, and a bounded `projection`.
|
|
260
|
+
|
|
261
|
+
A probe MUST NOT mutate product state, enqueue work, or trigger an external
|
|
262
|
+
effect. Unknown probes and invalid inputs use the same 404 and 422 behavior as
|
|
263
|
+
arrangements.
|
|
264
|
+
|
|
265
|
+
`POST /sinks/query` selects one advertised fake sink. It MUST require a public
|
|
266
|
+
correlation filter and MAY narrow by step, advertised effect kind, capture
|
|
267
|
+
time, opaque cursor, and bounded limit. Product query input and each record's
|
|
268
|
+
payload MUST satisfy the advertised schemas. Record count and bytes MUST fit
|
|
269
|
+
the negotiated limits.
|
|
270
|
+
|
|
271
|
+
Fake-sink records contain a safe summary and bounded product-owned payload.
|
|
272
|
+
They prove only that the contained product adapter recorded an attempted
|
|
273
|
+
effect; they never prove provider delivery.
|
|
274
|
+
|
|
275
|
+
## 11. Error model
|
|
276
|
+
|
|
277
|
+
Errors MUST use the schema-defined envelope and MUST NOT return an HTML
|
|
278
|
+
framework exception page, stack trace, source path, SQL, model attributes,
|
|
279
|
+
credentials, raw personal data, or an unlimited body.
|
|
280
|
+
|
|
281
|
+
| HTTP | Code | Meaning |
|
|
282
|
+
| ---- | ------------------------------ | ------------------------------------------------------- |
|
|
283
|
+
| 400 | `invalid-request` | Invalid headers, JSON, media type, or generic envelope. |
|
|
284
|
+
| 404 | `capability-not-found` | Exact capability ID/version is not advertised. |
|
|
285
|
+
| 409 | `protocol-version-unsupported` | Requested protocol is not version 3. |
|
|
286
|
+
| 409 | `reset-requires-restart` | Runtime driver must recreate the runtime. |
|
|
287
|
+
| 413 | `payload-too-large` | A transport or advertised byte bound was exceeded. |
|
|
288
|
+
| 422 | `schema-validation-failed` | Product-owned input violates its schema. |
|
|
289
|
+
| 422 | `limit-exceeded` | A declared semantic or query limit was exceeded. |
|
|
290
|
+
| 500 | `internal-error` | Sanitized unexpected adapter failure. |
|
|
291
|
+
| 503 | `runtime-unavailable` | Isolated test runtime cannot currently serve the call. |
|
|
292
|
+
|
|
293
|
+
Every error item MUST contain `code`, bounded `title`, bounded `detail`, and
|
|
294
|
+
`retryable`. A safe JSON Pointer MAY identify a schema location; the invalid
|
|
295
|
+
value MUST NOT be copied into the error.
|
|
296
|
+
|
|
297
|
+
## 12. Lifecycle ordering
|
|
298
|
+
|
|
299
|
+
For every runtime, the test client MUST:
|
|
300
|
+
|
|
301
|
+
1. create or explicitly attach to the test runtime and verify its health;
|
|
302
|
+
2. negotiate protocol v3 and validate runtime identity and capabilities;
|
|
303
|
+
3. reset before arrangements for a run or case;
|
|
304
|
+
4. execute declared arrangements;
|
|
305
|
+
5. execute public API operations outside testing control;
|
|
306
|
+
6. evaluate configured read-only observations; and
|
|
307
|
+
7. let the runtime driver release or clean the runtime.
|
|
308
|
+
|
|
309
|
+
To demonstrate ordinary API behavior, setup mutations belong before the public
|
|
310
|
+
API sequence. Inserting testing-control mutations between its calls does not
|
|
311
|
+
prove that the same result is reachable through the public API alone. The
|
|
312
|
+
runtime owner is responsible for cleanup after the test.
|
|
313
|
+
|
|
314
|
+
### Optional discovery snapshots
|
|
315
|
+
|
|
316
|
+
A client can use the Ruby library's `Discovery.build` method to produce a
|
|
317
|
+
stable snapshot and SHA-256 digest from an already validated capabilities
|
|
318
|
+
response. This is useful for detecting a changed testing API when the client
|
|
319
|
+
restarts a backend or reuses an earlier discovery result. It adds no HTTP route
|
|
320
|
+
and requires no backend registration file.
|
|
321
|
+
|
|
322
|
+
The current library retains protocol and application identity, including the
|
|
323
|
+
source version; test/isolation kind; features, limits, safety and reset facts;
|
|
324
|
+
and the advertised capabilities, including their titles, descriptions, schemas
|
|
325
|
+
and behavioral flags. It excludes per-request metadata and the environment
|
|
326
|
+
instance ID, sorts set-like fields and capability families, and hashes a
|
|
327
|
+
canonical JSON representation. Its limits are 128 entries per capability
|
|
328
|
+
family and 256 KiB. The shared conformance runner requires the same digest
|
|
329
|
+
after a runtime restart.
|
|
330
|
+
|
|
331
|
+
Computing or storing a snapshot is a client implementation choice. Every client
|
|
332
|
+
must still perform the live identity and capability checks required above.
|
|
333
|
+
Discovery validates the advertised contract; behavioral tests are needed to
|
|
334
|
+
check that its operations do what they promise.
|
|
335
|
+
|
|
336
|
+
## 13. Safety, privacy, and evidence
|
|
337
|
+
|
|
338
|
+
Clients and adapters MUST reject credentials according to the application and
|
|
339
|
+
test runner's configured policy. Capability payloads MAY be used transiently for
|
|
340
|
+
validation and execution, but conformance reports MUST NOT retain them.
|
|
341
|
+
Reports MAY retain safe capability names and redacted, bounded advertised
|
|
342
|
+
schemas.
|
|
343
|
+
|
|
344
|
+
Response bodies, error messages, and evidence MUST be byte-bounded before
|
|
345
|
+
persistence or presentation. Authorization headers, cookies, tokens,
|
|
346
|
+
passwords, secrets, credentials, non-synthetic email addresses, and configured
|
|
347
|
+
sensitive fields MUST be redacted. Invalid or raw framework responses MUST be
|
|
348
|
+
omitted from persisted diagnostics.
|
|
349
|
+
|
|
350
|
+
The test client MUST NOT decide product authorization or product outcome validity.
|
|
351
|
+
Those decisions remain in the product and its public API.
|
|
352
|
+
|
|
353
|
+
## 14. Extensibility
|
|
354
|
+
|
|
355
|
+
A product MAY add a new capability without changing the test client. It does so
|
|
356
|
+
by registering an opaque capability name/version, advertising bounded Draft
|
|
357
|
+
2020-12 input and output schemas, and implementing its product-owned handler.
|
|
358
|
+
Incompatible capability semantics require a new capability version, not a
|
|
359
|
+
testing-control protocol version.
|
|
360
|
+
|
|
361
|
+
An implementation MAY use an optional framework adapter or implement the HTTP
|
|
362
|
+
contract directly. No official helper is required for discovery, execution,
|
|
363
|
+
or conformance.
|
|
364
|
+
|
|
365
|
+
## 15. Black-box conformance
|
|
366
|
+
|
|
367
|
+
The shared runner checks an implementation through HTTP calls to testing-control
|
|
368
|
+
routes. It does not import an adapter, inspect backend source, connect to a
|
|
369
|
+
database, call factories or models, or execute backend commands.
|
|
370
|
+
|
|
371
|
+
In package `0.1.0`, core checks cover discovery, exact version behavior, runtime
|
|
372
|
+
identity, advertised capability schemas, reset responses, unknown arrangements
|
|
373
|
+
and probes, malformed JSON, invalid request media types, structured errors,
|
|
374
|
+
response formats and raw-exception exclusion. A product-input rejection check
|
|
375
|
+
uses the first advertised state only when its schema rejects an empty object.
|
|
376
|
+
The client enforces response byte limits and finite timeouts on its requests;
|
|
377
|
+
it does not exhaustively test the backend's request-size enforcement.
|
|
378
|
+
Independent safe checks continue after a mismatch. Stateful capability cases
|
|
379
|
+
run only after discovery and capability validation succeed and the reset check
|
|
380
|
+
passes.
|
|
381
|
+
|
|
382
|
+
The current runner does not call `/sinks/query`, check idempotent retries, or
|
|
383
|
+
verify that reset actually erased captured records or application data. For
|
|
384
|
+
an in-process reset it validates the success response and `sinksCleared: true`
|
|
385
|
+
declaration. It also does not prove that probes have no side effects. These
|
|
386
|
+
protocol requirements still apply; application request and end-to-end tests
|
|
387
|
+
must supply the missing behavioral evidence. A passing report describes the
|
|
388
|
+
checks performed, not exhaustive compliance with this specification.
|
|
389
|
+
|
|
390
|
+
The caller MAY supply a trusted, checked-in case file. The standalone CLI
|
|
391
|
+
loads that file as bounded JSON. Selection of the file and isolation of the
|
|
392
|
+
target belong to the caller; the file cannot execute commands. The case format permits only bounded arrangement and
|
|
393
|
+
probe data, top-level probe input bindings from returned
|
|
394
|
+
arrangement identifiers, and a JSON subset or JSON Schema assertion. It rejects
|
|
395
|
+
commands, scripts, SQL, credentials, repository paths, runtime URLs, and
|
|
396
|
+
arbitrary executable fields.
|
|
397
|
+
|
|
398
|
+
Expected-subset assertions use one shared recursive JSON matcher. Expected
|
|
399
|
+
objects are key subsets: every expected key must be present and recursively
|
|
400
|
+
match, while additional actual object keys are allowed. Expected arrays are
|
|
401
|
+
ordered exact-cardinality sequences: both arrays must have the same length and
|
|
402
|
+
each expected element recursively matches the actual element at the same
|
|
403
|
+
index. Object fields inside array elements may therefore be omitted from the
|
|
404
|
+
declaration, but additional or missing array elements and different ordering
|
|
405
|
+
fail. Empty arrays match only empty arrays.
|
|
406
|
+
|
|
407
|
+
Parsed JSON scalars and null retain their existing value equality. The matcher
|
|
408
|
+
does not coerce, stringify, convert, or apply truthiness; JSON numbers retain
|
|
409
|
+
numeric equality. A missing object key is distinct from a present null, false,
|
|
410
|
+
zero, empty string, empty object, or empty array. Protocol-envelope validation
|
|
411
|
+
and advertised capability response-schema validation run before subset
|
|
412
|
+
matching. This positional structural matcher has no wildcard, containment,
|
|
413
|
+
unordered, predicate, transformation, or general assertion-language behavior.
|
|
414
|
+
|
|
415
|
+
The standalone command in this repository attaches to one explicit loopback
|
|
416
|
+
origin. It compares supplied application, source and environment identities
|
|
417
|
+
before mutating data. `--allow-reset` is required because conformance resets
|
|
418
|
+
synthetic data. It does not start or destroy runtimes. A `runtime-restart`
|
|
419
|
+
adapter needs a caller that can recreate its runtime; the attached CLI reports
|
|
420
|
+
that prerequisite as unmet.
|
|
421
|
+
|
|
422
|
+
Exit codes: `0` passed, `4` nonconforming, `2` configuration failure, `3` invalid
|
|
423
|
+
invocation. Passing conformance proves the observed HTTP behavior, not network
|
|
424
|
+
containment, credential absence, or correctness of untested product effects.
|
|
425
|
+
|
|
426
|
+
Spies are product-owned observations of internal events or calls. Expose their
|
|
427
|
+
bounded, sanitized read-only projections through named `/probes` capabilities.
|
|
428
|
+
There is no additional `/spies` route in v3. Sinks capture supported effects in
|
|
429
|
+
place of real delivery and expose records through `/sinks/query`. A recorded
|
|
430
|
+
intent before enqueue is not evidence of worker completion or provider delivery.
|
|
431
|
+
Run identity partitions observations; it is not an authorization credential or
|
|
432
|
+
a substitute for a separate runtime for concurrent test suites.
|
|
@@ -0,0 +1,120 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'json'
|
|
4
|
+
require 'json_schemer'
|
|
5
|
+
|
|
6
|
+
module Overseer
|
|
7
|
+
module TestingControl
|
|
8
|
+
module Conformance
|
|
9
|
+
module CaseFile
|
|
10
|
+
MAX_BYTES = 256 * 1024
|
|
11
|
+
MAX_DEPTH = 24
|
|
12
|
+
SCHEMA_PATH = File.expand_path(
|
|
13
|
+
'../../../../protocol/testing-control/v3/conformance-case.schema.json',
|
|
14
|
+
__dir__
|
|
15
|
+
)
|
|
16
|
+
FORBIDDEN_KEY = Regexp.new(
|
|
17
|
+
'(?:command|script|sql|credential|password|token|secret|authorization|runtimeurl|repositorypath)\\z',
|
|
18
|
+
Regexp::IGNORECASE
|
|
19
|
+
)
|
|
20
|
+
FORBIDDEN_VALUE = %r{
|
|
21
|
+
(?:\A|\s)(?:https?|file)://|
|
|
22
|
+
\bBearer\s+|
|
|
23
|
+
\beyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\b|
|
|
24
|
+
(?:\A|\s)/(?:Users|home|etc|var|tmp)/|
|
|
25
|
+
\A\s*\#!|
|
|
26
|
+
\b(?:select\s+.+\s+from|insert\s+into|update\s+.+\s+set|delete\s+from|drop\s+table)\b
|
|
27
|
+
}ix
|
|
28
|
+
|
|
29
|
+
class Error < StandardError; end
|
|
30
|
+
|
|
31
|
+
module_function
|
|
32
|
+
|
|
33
|
+
def load(path)
|
|
34
|
+
source = File.binread(path, MAX_BYTES + 1)
|
|
35
|
+
parse(source, label: path)
|
|
36
|
+
rescue Errno::ENOENT, Errno::EACCES, Errno::EISDIR => e
|
|
37
|
+
raise Error, "Cannot read conformance case file: #{e.message}"
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
def parse(source, label: 'conformance case file')
|
|
41
|
+
raise Error, "Conformance case file exceeds #{MAX_BYTES} bytes" if source.bytesize > MAX_BYTES
|
|
42
|
+
|
|
43
|
+
document = JSON.parse(source)
|
|
44
|
+
errors = schemer.validate(document).take(5)
|
|
45
|
+
raise Error, "Invalid #{label}: #{details(errors)}" unless errors.empty?
|
|
46
|
+
|
|
47
|
+
validate_unique_ids!(document.fetch('cases'))
|
|
48
|
+
validate_safe_data!(document)
|
|
49
|
+
deep_freeze(document)
|
|
50
|
+
rescue JSON::ParserError => e
|
|
51
|
+
raise Error, "#{label.capitalize} is not valid JSON: #{e.message}"
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
def validate_schema!
|
|
55
|
+
schema = JSON.parse(File.read(SCHEMA_PATH, encoding: 'UTF-8'))
|
|
56
|
+
errors = JSONSchemer.validate_schema(schema).take(5)
|
|
57
|
+
raise Error, "Invalid conformance case schema: #{details(errors)}" unless errors.empty?
|
|
58
|
+
|
|
59
|
+
schema
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
def schemer
|
|
63
|
+
@schemer ||= JSONSchemer.schema(validate_schema!)
|
|
64
|
+
end
|
|
65
|
+
private_class_method :schemer
|
|
66
|
+
|
|
67
|
+
def validate_unique_ids!(cases)
|
|
68
|
+
duplicates = cases.group_by { |item| item.fetch('id') }.select { |_id, items| items.length > 1 }.keys
|
|
69
|
+
raise Error, "Duplicate conformance case IDs: #{duplicates.join(', ')}" unless duplicates.empty?
|
|
70
|
+
end
|
|
71
|
+
private_class_method :validate_unique_ids!
|
|
72
|
+
|
|
73
|
+
# rubocop:disable Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
|
|
74
|
+
def validate_safe_data!(value, depth = 0, field: nil)
|
|
75
|
+
raise Error, "Conformance case data exceeds nesting depth #{MAX_DEPTH}" if depth > MAX_DEPTH
|
|
76
|
+
|
|
77
|
+
case value
|
|
78
|
+
when Hash
|
|
79
|
+
value.each do |key, child|
|
|
80
|
+
if key.match?(FORBIDDEN_KEY)
|
|
81
|
+
raise Error,
|
|
82
|
+
"Conformance case field #{key.inspect} is executable or sensitive"
|
|
83
|
+
end
|
|
84
|
+
|
|
85
|
+
validate_safe_data!(child, depth + 1, field: key)
|
|
86
|
+
end
|
|
87
|
+
when Array
|
|
88
|
+
value.each { |child| validate_safe_data!(child, depth + 1, field:) }
|
|
89
|
+
when String
|
|
90
|
+
return if field == '$schema' && value == 'https://json-schema.org/draft/2020-12/schema'
|
|
91
|
+
|
|
92
|
+
raise Error, 'Conformance case data contains a URL, path, credential, script, or SQL-like value' if
|
|
93
|
+
value.match?(FORBIDDEN_VALUE)
|
|
94
|
+
end
|
|
95
|
+
end
|
|
96
|
+
# rubocop:enable Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
|
|
97
|
+
private_class_method :validate_safe_data!
|
|
98
|
+
|
|
99
|
+
def details(errors)
|
|
100
|
+
errors.map { |error| "#{error.fetch('data_pointer', '/')} #{error.fetch('type')}" }.join('; ')
|
|
101
|
+
end
|
|
102
|
+
private_class_method :details
|
|
103
|
+
|
|
104
|
+
def deep_freeze(value)
|
|
105
|
+
case value
|
|
106
|
+
when Hash
|
|
107
|
+
value.each do |key, child|
|
|
108
|
+
key.freeze
|
|
109
|
+
deep_freeze(child)
|
|
110
|
+
end
|
|
111
|
+
when Array
|
|
112
|
+
value.each { |child| deep_freeze(child) }
|
|
113
|
+
end
|
|
114
|
+
value.freeze
|
|
115
|
+
end
|
|
116
|
+
private_class_method :deep_freeze
|
|
117
|
+
end
|
|
118
|
+
end
|
|
119
|
+
end
|
|
120
|
+
end
|
|
@@ -0,0 +1,114 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'net/http'
|
|
4
|
+
require 'uri'
|
|
5
|
+
|
|
6
|
+
module Overseer
|
|
7
|
+
module TestingControl
|
|
8
|
+
module Conformance
|
|
9
|
+
class HTTPTransport
|
|
10
|
+
Response = Data.define(:status, :headers, :body, :duration_ms)
|
|
11
|
+
OPEN_TIMEOUT_SECONDS = 2
|
|
12
|
+
READ_TIMEOUT_SECONDS = 10
|
|
13
|
+
|
|
14
|
+
attr_reader :origin
|
|
15
|
+
|
|
16
|
+
def initialize(origin:, base_path:)
|
|
17
|
+
@origin = self.class.origin!(origin)
|
|
18
|
+
@base_path = normalized_base_path(base_path)
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
def request(method:, path:, headers:, body:, max_response_bytes:)
|
|
22
|
+
uri = request_uri(path)
|
|
23
|
+
request = request_class(method).new(uri)
|
|
24
|
+
headers.each { |name, value| request[name] = value }
|
|
25
|
+
request.body = body unless body.nil?
|
|
26
|
+
started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
27
|
+
response = perform(uri, request, max_response_bytes:)
|
|
28
|
+
response.with(duration_ms: elapsed_milliseconds(started))
|
|
29
|
+
rescue Net::OpenTimeout, Net::ReadTimeout
|
|
30
|
+
raise TimeoutError, 'Testing-control request exceeded its bounded timeout'
|
|
31
|
+
rescue ResponseTooLarge, Error
|
|
32
|
+
raise
|
|
33
|
+
rescue IOError, SystemCallError => e
|
|
34
|
+
raise ConnectionError, "Testing-control loopback request failed (#{e.class.name})"
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
# rubocop:disable Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
|
|
38
|
+
def self.origin!(value)
|
|
39
|
+
uri = URI.parse(value)
|
|
40
|
+
valid = uri.is_a?(URI::HTTP) && uri.scheme == 'http' && uri.host == '127.0.0.1' &&
|
|
41
|
+
uri.authority&.match?(/:\d+\z/) && uri.port.between?(1, 65_535) && !uri.userinfo &&
|
|
42
|
+
uri.path.empty? && !uri.query && !uri.fragment
|
|
43
|
+
raise Error, 'Conformance requires an explicit bare 127.0.0.1 HTTP origin' unless valid
|
|
44
|
+
|
|
45
|
+
uri.to_s
|
|
46
|
+
rescue URI::InvalidURIError
|
|
47
|
+
raise Error, 'Conformance requires an explicit bare 127.0.0.1 HTTP origin'
|
|
48
|
+
end
|
|
49
|
+
# rubocop:enable Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
|
|
50
|
+
|
|
51
|
+
private
|
|
52
|
+
|
|
53
|
+
def normalized_base_path(value)
|
|
54
|
+
unless value.is_a?(String) && value.start_with?('/') && !value.include?('?') && !value.include?('#')
|
|
55
|
+
raise Error, 'Testing-control base path must be an absolute path'
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
value.sub(%r{/+\z}, '')
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
def request_uri(path)
|
|
62
|
+
raise Error, 'Testing-control route must be a fixed absolute path' unless path.start_with?('/')
|
|
63
|
+
|
|
64
|
+
uri = URI.parse(origin)
|
|
65
|
+
uri.path = "#{@base_path}#{path}"
|
|
66
|
+
uri
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
def request_class(method)
|
|
70
|
+
{
|
|
71
|
+
get: Net::HTTP::Get,
|
|
72
|
+
put: Net::HTTP::Put,
|
|
73
|
+
post: Net::HTTP::Post
|
|
74
|
+
}.fetch(method)
|
|
75
|
+
rescue KeyError
|
|
76
|
+
raise Error, "Unsupported testing-control HTTP method #{method.inspect}"
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
def perform(uri, request, max_response_bytes:)
|
|
80
|
+
client = Net::HTTP.new(uri.host, uri.port, nil)
|
|
81
|
+
client.open_timeout = OPEN_TIMEOUT_SECONDS
|
|
82
|
+
client.read_timeout = READ_TIMEOUT_SECONDS
|
|
83
|
+
result = nil
|
|
84
|
+
client.request(request) do |response|
|
|
85
|
+
body = +''
|
|
86
|
+
response.read_body do |chunk|
|
|
87
|
+
if body.bytesize + chunk.bytesize > max_response_bytes
|
|
88
|
+
raise ResponseTooLarge, "Testing-control response exceeds #{max_response_bytes} bytes"
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
body << chunk
|
|
92
|
+
end
|
|
93
|
+
result = Response.new(
|
|
94
|
+
status: response.code.to_i,
|
|
95
|
+
headers: response.each_header.to_h,
|
|
96
|
+
body:,
|
|
97
|
+
duration_ms: nil
|
|
98
|
+
)
|
|
99
|
+
end
|
|
100
|
+
result || raise(ConnectionError, 'Testing-control loopback response was unavailable')
|
|
101
|
+
end
|
|
102
|
+
|
|
103
|
+
def elapsed_milliseconds(started)
|
|
104
|
+
((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started) * 1000).round(3)
|
|
105
|
+
end
|
|
106
|
+
|
|
107
|
+
class Error < StandardError; end
|
|
108
|
+
class TimeoutError < Error; end
|
|
109
|
+
class ResponseTooLarge < Error; end
|
|
110
|
+
class ConnectionError < Error; end
|
|
111
|
+
end
|
|
112
|
+
end
|
|
113
|
+
end
|
|
114
|
+
end
|