overseer-testing-protocol 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (33) hide show
  1. checksums.yaml +7 -0
  2. data/LICENSE +19 -0
  3. data/README.md +75 -0
  4. data/SOURCE.json +6 -0
  5. data/bin/overseer-testing-conformance +53 -0
  6. data/docs/testing-control-implementation-guide.md +116 -0
  7. data/docs/testing-control-protocol.md +432 -0
  8. data/lib/overseer/testing_control/conformance/case_file.rb +120 -0
  9. data/lib/overseer/testing_control/conformance/http_transport.rb +114 -0
  10. data/lib/overseer/testing_control/conformance/report.rb +104 -0
  11. data/lib/overseer/testing_control/conformance/runner.rb +783 -0
  12. data/lib/overseer/testing_control/discovery.rb +79 -0
  13. data/lib/overseer/testing_control/json_subset_matcher.rb +120 -0
  14. data/lib/overseer/testing_control/protocol_v3.rb +298 -0
  15. data/lib/overseer/testing_control/redaction.rb +111 -0
  16. data/lib/overseer/testing_protocol.rb +9 -0
  17. data/protocol/testing-control/v3/conformance-case.schema.json +151 -0
  18. data/protocol/testing-control/v3/conformance-report.schema.json +255 -0
  19. data/protocol/testing-control/v3/fixtures/capabilities-response.json +147 -0
  20. data/protocol/testing-control/v3/fixtures/conformance-case.json +23 -0
  21. data/protocol/testing-control/v3/fixtures/conformance-report.json +88 -0
  22. data/protocol/testing-control/v3/fixtures/error-response.json +20 -0
  23. data/protocol/testing-control/v3/fixtures/manifest.json +13 -0
  24. data/protocol/testing-control/v3/fixtures/probe-request.json +20 -0
  25. data/protocol/testing-control/v3/fixtures/probe-response.json +22 -0
  26. data/protocol/testing-control/v3/fixtures/reset-response.json +15 -0
  27. data/protocol/testing-control/v3/fixtures/sink-query-request.json +26 -0
  28. data/protocol/testing-control/v3/fixtures/sink-query-response.json +39 -0
  29. data/protocol/testing-control/v3/fixtures/state-request.json +20 -0
  30. data/protocol/testing-control/v3/fixtures/state-response.json +23 -0
  31. data/protocol/testing-control/v3/openapi.yaml +343 -0
  32. data/protocol/testing-control/v3/schema.json +772 -0
  33. metadata +85 -0
@@ -0,0 +1,432 @@
1
+ # Testing-control protocol v3
2
+
3
+ Status: canonical framework-neutral wire specification. Testing-control v3 is
4
+ the sole current testing-control protocol.
5
+
6
+ The normative artifacts are:
7
+
8
+ - [`../protocol/testing-control/v3/openapi.yaml`](../protocol/testing-control/v3/openapi.yaml)
9
+ for routes, methods, headers, media types, and response statuses;
10
+ - [`../protocol/testing-control/v3/schema.json`](../protocol/testing-control/v3/schema.json)
11
+ for wire request and response envelopes;
12
+ - [`../protocol/testing-control/v3/fixtures/`](../protocol/testing-control/v3/fixtures/)
13
+ for framework-neutral examples;
14
+ - [`../protocol/testing-control/v3/conformance-case.schema.json`](../protocol/testing-control/v3/conformance-case.schema.json)
15
+ for trusted declarative backend scenarios; and
16
+ - [`../protocol/testing-control/v3/conformance-report.schema.json`](../protocol/testing-control/v3/conformance-report.schema.json)
17
+ for bounded conformance diagnostic output.
18
+
19
+ The OpenAPI document and wire schema take precedence if prose differs. The
20
+ case and report formats support conformance; they are not additional
21
+ testing-control protocols. The [framework-neutral implementation
22
+ guide](./testing-control-implementation-guide.md) explains direct adoption.
23
+
24
+ Requirement words **MUST**, **MUST NOT**, **SHOULD**, and **MAY** are normative.
25
+
26
+ ## 1. Scope and non-goals
27
+
28
+ Testing control is a test-only mechanism for:
29
+
30
+ - discovering the exact protocol and safe product-owned capabilities;
31
+ - witnessing application and test-environment identity;
32
+ - returning a synthetic runtime to its declared baseline;
33
+ - arranging allowlisted, schema-validated synthetic preconditions;
34
+ - observing bounded, sanitized, read-only projections; and
35
+ - querying bounded records captured by fake external-effect sinks.
36
+
37
+ Testing control MUST NOT become a public business API, general database
38
+ seeding language, factory API, model API, arbitrary query interface, command
39
+ runner, workflow engine, authorization bypass, or replacement for the public
40
+ OpenAPI contract. Public API calls and public workflow execution remain
41
+ outside this protocol.
42
+
43
+ The protocol calls product-owned state capabilities **arrangements** and
44
+ product-owned probe and fake-sink capabilities **observations**. Their current
45
+ wire route names remain `states`, `probes`, and `sinks`.
46
+
47
+ ## 2. Trust and deployment assumptions
48
+
49
+ An implementation MUST expose testing control only inside a verified
50
+ synthetic test environment. It MUST fail closed unless the backend's normal
51
+ test-mode checks and the test runner's supplied runtime identity are valid.
52
+ Authentication inside a production-capable environment is not a substitute.
53
+
54
+ The runtime MUST use synthetic data, MUST omit real provider and user
55
+ credentials, MUST NOT persist secrets, and MUST block or explicitly allowlist
56
+ external network access. An implementation MUST NOT accept arbitrary commands,
57
+ scripts, SQL, repository paths, runtime URLs, direct database operations, or
58
+ credentials through any testing-control payload.
59
+
60
+ The test runner owns runtime creation or explicit attachment, network containment,
61
+ source and process verification, lifecycle cleanup, and evidence sanitization.
62
+ The product owns its domain invariants, capability handlers, read-only
63
+ projections, safe fields, and fake-effect interception.
64
+
65
+ ## 3. Transport and endpoints
66
+
67
+ The backend and test client agree on one testing base path. Exactly
68
+ these routes are relative to that base path:
69
+
70
+ ```text
71
+ GET /capabilities
72
+ PUT /reset
73
+ POST /states
74
+ POST /probes
75
+ POST /sinks/query
76
+ ```
77
+
78
+ Clients and servers MUST use HTTP over the already verified loopback origin.
79
+ Testing-control redirects and arbitrary absolute targets are unsupported.
80
+ Requests with bodies and all responses MUST use UTF-8 JSON with media type
81
+ `application/json`. `GET /capabilities` and `PUT /reset` MUST have no body.
82
+ Malformed JSON and a non-JSON request media type MUST fail as `invalid-request`
83
+ with HTTP 400.
84
+
85
+ An adapter MUST enforce its advertised `maxRequestBytes` and
86
+ `maxResponseBytes`. It MUST reject an oversized request before executing a
87
+ handler and MUST NOT construct or stream an unbounded response. Clients MUST
88
+ use finite connection and response timeouts and MUST stop reading after the
89
+ applicable byte limit. Exact client timeout values are runner policy, not wire
90
+ semantics.
91
+
92
+ Protocol envelopes are closed: fields not admitted by `schema.json` MUST be
93
+ rejected. Product-owned capability payloads follow their advertised schemas;
94
+ the product schema deliberately controls whether its own additional fields are
95
+ allowed.
96
+
97
+ ## 4. Exact version negotiation
98
+
99
+ Every request MUST carry:
100
+
101
+ | Header | Requirement |
102
+ | ---------------------------------- | ------------------------------------------------ |
103
+ | `Overseer-Testing-Control-Version` | Exact value `3`. |
104
+ | `Overseer-Run-Id` | Opaque valid identifier for the current attempt. |
105
+ | `Overseer-Correlation-Id` | Opaque valid identifier for this control call. |
106
+
107
+ `Overseer-Step-Id` MAY identify attributable setup or public-step context.
108
+ `Overseer-Idempotency-Key` MAY be used for reset and MUST be used when an
109
+ arrangement advertises required idempotency.
110
+
111
+ Identifiers contain 1 through 128 ASCII characters, begin with a letter or
112
+ digit, and then contain only letters, digits, `.`, `_`, `:`, or `-`.
113
+
114
+ Every response, including an error, MUST set
115
+ `Overseer-Testing-Control-Version: 3` and a valid
116
+ `Overseer-Correlation-Id`. For a valid request correlation ID, the response
117
+ MUST echo it. The body protocol member MUST be exactly:
118
+
119
+ ```json
120
+ {
121
+ "name": "overseer-testing-control",
122
+ "version": "3"
123
+ }
124
+ ```
125
+
126
+ A request for any version other than `3` MUST return HTTP 409 with
127
+ `protocol-version-unsupported`. It MUST NOT downgrade, dispatch to a legacy
128
+ implementation, or serve a simultaneous v1/v2 compatibility path. Testing
129
+ control v4 does not exist.
130
+
131
+ ## 5. Envelope identity
132
+
133
+ Every normal request and successful response has `protocol`, `meta`, and
134
+ `data`; every error has `protocol`, `meta`, and `errors`. `meta.runId` and
135
+ `meta.correlationId` MUST match their headers. `meta.stepId` MUST be present
136
+ and equal to its header when supplied and MUST otherwise be absent.
137
+
138
+ When required identity headers are absent or invalid, an adapter MUST return a
139
+ schema-valid safe error identity. Generated placeholder identifiers MAY be
140
+ used; raw invalid values MUST NOT be echoed.
141
+
142
+ ## 6. Runtime identity and independently verified facts
143
+
144
+ `GET /capabilities` owns these testing-control identity facts:
145
+
146
+ - `data.application.id`: stable backend integration ID;
147
+ - `data.application.version`: exact source identity injected into this
148
+ runtime;
149
+ - `data.environment.id`: opaque environment ID injected by the runtime owner;
150
+ - `data.environment.kind`: exact value `test`; and
151
+ - `data.environment.isolated`: exact value `true` for a verified isolated
152
+ runtime.
153
+
154
+ The test client MUST compare those values with the identities independently
155
+ established when starting or attaching to the backend, before reset,
156
+ arrangement, or observation. A mismatch is protocol nonconformance and MUST
157
+ stop dependent stateful checks.
158
+
159
+ Health status, loopback origin, source version resolution, whether the test
160
+ runner created the process, container/network facts, and cleanup result are
161
+ facts the runtime owner verifies independently. They MUST NOT be invented by the
162
+ testing-control response or treated as product business evidence.
163
+
164
+ ## 7. Capability discovery
165
+
166
+ `GET /capabilities` MUST return:
167
+
168
+ - application and environment identity;
169
+ - one reset strategy and its sink-clearing guarantee;
170
+ - the required replay features;
171
+ - size and record limits;
172
+ - the safety declaration;
173
+ - zero or more arrangement descriptors in `states`;
174
+ - zero or more read-only observation descriptors in `probes`; and
175
+ - zero or more fake-sink observation descriptors in `sinks`.
176
+
177
+ The required features are `correlation-propagation`,
178
+ `deterministic-replay`, `reset`, `schema-validation`, and `sink-clearing`.
179
+ The safety declaration MUST require synthetic data, forbid real credentials
180
+ and secret persistence, and declare external networking as `blocked` or
181
+ `allowlisted`.
182
+
183
+ Every capability is selected by an opaque product-owned `id` and positive
184
+ decimal `version`. An `id@version` pair MUST be unique inside its family.
185
+ The test client MUST NOT enumerate or interpret product capability names.
186
+
187
+ Arrangement descriptors MUST advertise input and output JSON Schemas and an
188
+ idempotency policy of `required`, `supported`, or `none`. Probe descriptors
189
+ MUST advertise `readOnly: true` plus input and output schemas. Sink descriptors
190
+ MUST advertise effect kinds, outcomes, a query-input schema, and a record
191
+ payload schema.
192
+
193
+ Advertised schemas MUST use JSON Schema Draft 2020-12, MUST have an object
194
+ root, MUST fit `maxSchemaBytes`, MUST compile, and MUST use only local fragment
195
+ references. The test client MUST NOT fetch an external schema URI supplied by a
196
+ backend. The schemas remain product-owned and opaque even though the generic
197
+ client validates values against them.
198
+
199
+ ## 8. Reset
200
+
201
+ The advertised reset strategy is exactly `in-process` or `runtime-restart`.
202
+ Both strategies MUST return the product to its declared synthetic baseline and
203
+ MUST clear arrangement idempotency state and fake-sink records before the next
204
+ run.
205
+
206
+ For `in-process`, `PUT /reset` MUST return HTTP 200 with `resetAt`, matching
207
+ strategy `in-process`, and `sinksCleared: true`. Repeating reset in the same
208
+ isolated runtime MUST be safe and return the same declared baseline.
209
+
210
+ For `runtime-restart`, `PUT /reset` MUST return HTTP 409 with
211
+ `reset-requires-restart`. The runtime driver, not a product handler, recreates
212
+ the isolated process and writable data, repeats normal test setup and health
213
+ verification, and negotiates capabilities again. The replacement MUST preserve
214
+ application source identity, environment identity, protocol v3, and
215
+ the declared reset strategy. An explicitly attached process cannot satisfy
216
+ runtime-restart unless its owner supplies a verified restart callback.
217
+
218
+ Reset failure MUST prevent later arrangements and observations that depend on
219
+ a clean baseline. Independent non-mutating protocol checks MAY continue.
220
+
221
+ ## 9. Arrangement
222
+
223
+ `POST /states` selects one advertised arrangement:
224
+
225
+ ```json
226
+ {
227
+ "protocol": { "name": "overseer-testing-control", "version": "3" },
228
+ "meta": {
229
+ "runId": "run-001",
230
+ "correlationId": "arrange-001",
231
+ "stepId": "setup-open-item"
232
+ },
233
+ "data": {
234
+ "capability": { "id": "open-item", "version": "1" },
235
+ "input": { "title": "Synthetic item" }
236
+ }
237
+ }
238
+ ```
239
+
240
+ The adapter MUST locate the exact advertised `id@version`, validate `input`
241
+ against that descriptor's schema, execute only the registered product-owned
242
+ handler, validate the output, and return the same capability reference plus
243
+ bounded canonical identifiers and output. Product execution remains opaque to
244
+ the client.
245
+
246
+ An unknown capability MUST return HTTP 404 with `capability-not-found`. A
247
+ schema-invalid input MUST return HTTP 422 with `schema-validation-failed`.
248
+ Required idempotency MUST fail when its header is absent. Retrying an unchanged
249
+ key and input MUST NOT create a second logical arrangement; reusing a key with
250
+ different input MUST fail safely.
251
+
252
+ An arrangement is setup, not a public product mutation or evidence that a
253
+ consumer can create the same state.
254
+
255
+ ## 10. Observation
256
+
257
+ `POST /probes` selects one advertised read-only probe, validates its input,
258
+ executes only its registered product-owned observation, validates the result,
259
+ and returns the same capability, `observedAt`, and a bounded `projection`.
260
+
261
+ A probe MUST NOT mutate product state, enqueue work, or trigger an external
262
+ effect. Unknown probes and invalid inputs use the same 404 and 422 behavior as
263
+ arrangements.
264
+
265
+ `POST /sinks/query` selects one advertised fake sink. It MUST require a public
266
+ correlation filter and MAY narrow by step, advertised effect kind, capture
267
+ time, opaque cursor, and bounded limit. Product query input and each record's
268
+ payload MUST satisfy the advertised schemas. Record count and bytes MUST fit
269
+ the negotiated limits.
270
+
271
+ Fake-sink records contain a safe summary and bounded product-owned payload.
272
+ They prove only that the contained product adapter recorded an attempted
273
+ effect; they never prove provider delivery.
274
+
275
+ ## 11. Error model
276
+
277
+ Errors MUST use the schema-defined envelope and MUST NOT return an HTML
278
+ framework exception page, stack trace, source path, SQL, model attributes,
279
+ credentials, raw personal data, or an unlimited body.
280
+
281
+ | HTTP | Code | Meaning |
282
+ | ---- | ------------------------------ | ------------------------------------------------------- |
283
+ | 400 | `invalid-request` | Invalid headers, JSON, media type, or generic envelope. |
284
+ | 404 | `capability-not-found` | Exact capability ID/version is not advertised. |
285
+ | 409 | `protocol-version-unsupported` | Requested protocol is not version 3. |
286
+ | 409 | `reset-requires-restart` | Runtime driver must recreate the runtime. |
287
+ | 413 | `payload-too-large` | A transport or advertised byte bound was exceeded. |
288
+ | 422 | `schema-validation-failed` | Product-owned input violates its schema. |
289
+ | 422 | `limit-exceeded` | A declared semantic or query limit was exceeded. |
290
+ | 500 | `internal-error` | Sanitized unexpected adapter failure. |
291
+ | 503 | `runtime-unavailable` | Isolated test runtime cannot currently serve the call. |
292
+
293
+ Every error item MUST contain `code`, bounded `title`, bounded `detail`, and
294
+ `retryable`. A safe JSON Pointer MAY identify a schema location; the invalid
295
+ value MUST NOT be copied into the error.
296
+
297
+ ## 12. Lifecycle ordering
298
+
299
+ For every runtime, the test client MUST:
300
+
301
+ 1. create or explicitly attach to the test runtime and verify its health;
302
+ 2. negotiate protocol v3 and validate runtime identity and capabilities;
303
+ 3. reset before arrangements for a run or case;
304
+ 4. execute declared arrangements;
305
+ 5. execute public API operations outside testing control;
306
+ 6. evaluate configured read-only observations; and
307
+ 7. let the runtime driver release or clean the runtime.
308
+
309
+ To demonstrate ordinary API behavior, setup mutations belong before the public
310
+ API sequence. Inserting testing-control mutations between its calls does not
311
+ prove that the same result is reachable through the public API alone. The
312
+ runtime owner is responsible for cleanup after the test.
313
+
314
+ ### Optional discovery snapshots
315
+
316
+ A client can use the Ruby library's `Discovery.build` method to produce a
317
+ stable snapshot and SHA-256 digest from an already validated capabilities
318
+ response. This is useful for detecting a changed testing API when the client
319
+ restarts a backend or reuses an earlier discovery result. It adds no HTTP route
320
+ and requires no backend registration file.
321
+
322
+ The current library retains protocol and application identity, including the
323
+ source version; test/isolation kind; features, limits, safety and reset facts;
324
+ and the advertised capabilities, including their titles, descriptions, schemas
325
+ and behavioral flags. It excludes per-request metadata and the environment
326
+ instance ID, sorts set-like fields and capability families, and hashes a
327
+ canonical JSON representation. Its limits are 128 entries per capability
328
+ family and 256 KiB. The shared conformance runner requires the same digest
329
+ after a runtime restart.
330
+
331
+ Computing or storing a snapshot is a client implementation choice. Every client
332
+ must still perform the live identity and capability checks required above.
333
+ Discovery validates the advertised contract; behavioral tests are needed to
334
+ check that its operations do what they promise.
335
+
336
+ ## 13. Safety, privacy, and evidence
337
+
338
+ Clients and adapters MUST reject credentials according to the application and
339
+ test runner's configured policy. Capability payloads MAY be used transiently for
340
+ validation and execution, but conformance reports MUST NOT retain them.
341
+ Reports MAY retain safe capability names and redacted, bounded advertised
342
+ schemas.
343
+
344
+ Response bodies, error messages, and evidence MUST be byte-bounded before
345
+ persistence or presentation. Authorization headers, cookies, tokens,
346
+ passwords, secrets, credentials, non-synthetic email addresses, and configured
347
+ sensitive fields MUST be redacted. Invalid or raw framework responses MUST be
348
+ omitted from persisted diagnostics.
349
+
350
+ The test client MUST NOT decide product authorization or product outcome validity.
351
+ Those decisions remain in the product and its public API.
352
+
353
+ ## 14. Extensibility
354
+
355
+ A product MAY add a new capability without changing the test client. It does so
356
+ by registering an opaque capability name/version, advertising bounded Draft
357
+ 2020-12 input and output schemas, and implementing its product-owned handler.
358
+ Incompatible capability semantics require a new capability version, not a
359
+ testing-control protocol version.
360
+
361
+ An implementation MAY use an optional framework adapter or implement the HTTP
362
+ contract directly. No official helper is required for discovery, execution,
363
+ or conformance.
364
+
365
+ ## 15. Black-box conformance
366
+
367
+ The shared runner checks an implementation through HTTP calls to testing-control
368
+ routes. It does not import an adapter, inspect backend source, connect to a
369
+ database, call factories or models, or execute backend commands.
370
+
371
+ In package `0.1.0`, core checks cover discovery, exact version behavior, runtime
372
+ identity, advertised capability schemas, reset responses, unknown arrangements
373
+ and probes, malformed JSON, invalid request media types, structured errors,
374
+ response formats and raw-exception exclusion. A product-input rejection check
375
+ uses the first advertised state only when its schema rejects an empty object.
376
+ The client enforces response byte limits and finite timeouts on its requests;
377
+ it does not exhaustively test the backend's request-size enforcement.
378
+ Independent safe checks continue after a mismatch. Stateful capability cases
379
+ run only after discovery and capability validation succeed and the reset check
380
+ passes.
381
+
382
+ The current runner does not call `/sinks/query`, check idempotent retries, or
383
+ verify that reset actually erased captured records or application data. For
384
+ an in-process reset it validates the success response and `sinksCleared: true`
385
+ declaration. It also does not prove that probes have no side effects. These
386
+ protocol requirements still apply; application request and end-to-end tests
387
+ must supply the missing behavioral evidence. A passing report describes the
388
+ checks performed, not exhaustive compliance with this specification.
389
+
390
+ The caller MAY supply a trusted, checked-in case file. The standalone CLI
391
+ loads that file as bounded JSON. Selection of the file and isolation of the
392
+ target belong to the caller; the file cannot execute commands. The case format permits only bounded arrangement and
393
+ probe data, top-level probe input bindings from returned
394
+ arrangement identifiers, and a JSON subset or JSON Schema assertion. It rejects
395
+ commands, scripts, SQL, credentials, repository paths, runtime URLs, and
396
+ arbitrary executable fields.
397
+
398
+ Expected-subset assertions use one shared recursive JSON matcher. Expected
399
+ objects are key subsets: every expected key must be present and recursively
400
+ match, while additional actual object keys are allowed. Expected arrays are
401
+ ordered exact-cardinality sequences: both arrays must have the same length and
402
+ each expected element recursively matches the actual element at the same
403
+ index. Object fields inside array elements may therefore be omitted from the
404
+ declaration, but additional or missing array elements and different ordering
405
+ fail. Empty arrays match only empty arrays.
406
+
407
+ Parsed JSON scalars and null retain their existing value equality. The matcher
408
+ does not coerce, stringify, convert, or apply truthiness; JSON numbers retain
409
+ numeric equality. A missing object key is distinct from a present null, false,
410
+ zero, empty string, empty object, or empty array. Protocol-envelope validation
411
+ and advertised capability response-schema validation run before subset
412
+ matching. This positional structural matcher has no wildcard, containment,
413
+ unordered, predicate, transformation, or general assertion-language behavior.
414
+
415
+ The standalone command in this repository attaches to one explicit loopback
416
+ origin. It compares supplied application, source and environment identities
417
+ before mutating data. `--allow-reset` is required because conformance resets
418
+ synthetic data. It does not start or destroy runtimes. A `runtime-restart`
419
+ adapter needs a caller that can recreate its runtime; the attached CLI reports
420
+ that prerequisite as unmet.
421
+
422
+ Exit codes: `0` passed, `4` nonconforming, `2` configuration failure, `3` invalid
423
+ invocation. Passing conformance proves the observed HTTP behavior, not network
424
+ containment, credential absence, or correctness of untested product effects.
425
+
426
+ Spies are product-owned observations of internal events or calls. Expose their
427
+ bounded, sanitized read-only projections through named `/probes` capabilities.
428
+ There is no additional `/spies` route in v3. Sinks capture supported effects in
429
+ place of real delivery and expose records through `/sinks/query`. A recorded
430
+ intent before enqueue is not evidence of worker completion or provider delivery.
431
+ Run identity partitions observations; it is not an authorization credential or
432
+ a substitute for a separate runtime for concurrent test suites.
@@ -0,0 +1,120 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'json'
4
+ require 'json_schemer'
5
+
6
+ module Overseer
7
+ module TestingControl
8
+ module Conformance
9
+ module CaseFile
10
+ MAX_BYTES = 256 * 1024
11
+ MAX_DEPTH = 24
12
+ SCHEMA_PATH = File.expand_path(
13
+ '../../../../protocol/testing-control/v3/conformance-case.schema.json',
14
+ __dir__
15
+ )
16
+ FORBIDDEN_KEY = Regexp.new(
17
+ '(?:command|script|sql|credential|password|token|secret|authorization|runtimeurl|repositorypath)\\z',
18
+ Regexp::IGNORECASE
19
+ )
20
+ FORBIDDEN_VALUE = %r{
21
+ (?:\A|\s)(?:https?|file)://|
22
+ \bBearer\s+|
23
+ \beyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\b|
24
+ (?:\A|\s)/(?:Users|home|etc|var|tmp)/|
25
+ \A\s*\#!|
26
+ \b(?:select\s+.+\s+from|insert\s+into|update\s+.+\s+set|delete\s+from|drop\s+table)\b
27
+ }ix
28
+
29
+ class Error < StandardError; end
30
+
31
+ module_function
32
+
33
+ def load(path)
34
+ source = File.binread(path, MAX_BYTES + 1)
35
+ parse(source, label: path)
36
+ rescue Errno::ENOENT, Errno::EACCES, Errno::EISDIR => e
37
+ raise Error, "Cannot read conformance case file: #{e.message}"
38
+ end
39
+
40
+ def parse(source, label: 'conformance case file')
41
+ raise Error, "Conformance case file exceeds #{MAX_BYTES} bytes" if source.bytesize > MAX_BYTES
42
+
43
+ document = JSON.parse(source)
44
+ errors = schemer.validate(document).take(5)
45
+ raise Error, "Invalid #{label}: #{details(errors)}" unless errors.empty?
46
+
47
+ validate_unique_ids!(document.fetch('cases'))
48
+ validate_safe_data!(document)
49
+ deep_freeze(document)
50
+ rescue JSON::ParserError => e
51
+ raise Error, "#{label.capitalize} is not valid JSON: #{e.message}"
52
+ end
53
+
54
+ def validate_schema!
55
+ schema = JSON.parse(File.read(SCHEMA_PATH, encoding: 'UTF-8'))
56
+ errors = JSONSchemer.validate_schema(schema).take(5)
57
+ raise Error, "Invalid conformance case schema: #{details(errors)}" unless errors.empty?
58
+
59
+ schema
60
+ end
61
+
62
+ def schemer
63
+ @schemer ||= JSONSchemer.schema(validate_schema!)
64
+ end
65
+ private_class_method :schemer
66
+
67
+ def validate_unique_ids!(cases)
68
+ duplicates = cases.group_by { |item| item.fetch('id') }.select { |_id, items| items.length > 1 }.keys
69
+ raise Error, "Duplicate conformance case IDs: #{duplicates.join(', ')}" unless duplicates.empty?
70
+ end
71
+ private_class_method :validate_unique_ids!
72
+
73
+ # rubocop:disable Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
74
+ def validate_safe_data!(value, depth = 0, field: nil)
75
+ raise Error, "Conformance case data exceeds nesting depth #{MAX_DEPTH}" if depth > MAX_DEPTH
76
+
77
+ case value
78
+ when Hash
79
+ value.each do |key, child|
80
+ if key.match?(FORBIDDEN_KEY)
81
+ raise Error,
82
+ "Conformance case field #{key.inspect} is executable or sensitive"
83
+ end
84
+
85
+ validate_safe_data!(child, depth + 1, field: key)
86
+ end
87
+ when Array
88
+ value.each { |child| validate_safe_data!(child, depth + 1, field:) }
89
+ when String
90
+ return if field == '$schema' && value == 'https://json-schema.org/draft/2020-12/schema'
91
+
92
+ raise Error, 'Conformance case data contains a URL, path, credential, script, or SQL-like value' if
93
+ value.match?(FORBIDDEN_VALUE)
94
+ end
95
+ end
96
+ # rubocop:enable Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
97
+ private_class_method :validate_safe_data!
98
+
99
+ def details(errors)
100
+ errors.map { |error| "#{error.fetch('data_pointer', '/')} #{error.fetch('type')}" }.join('; ')
101
+ end
102
+ private_class_method :details
103
+
104
+ def deep_freeze(value)
105
+ case value
106
+ when Hash
107
+ value.each do |key, child|
108
+ key.freeze
109
+ deep_freeze(child)
110
+ end
111
+ when Array
112
+ value.each { |child| deep_freeze(child) }
113
+ end
114
+ value.freeze
115
+ end
116
+ private_class_method :deep_freeze
117
+ end
118
+ end
119
+ end
120
+ end
@@ -0,0 +1,114 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'net/http'
4
+ require 'uri'
5
+
6
+ module Overseer
7
+ module TestingControl
8
+ module Conformance
9
+ class HTTPTransport
10
+ Response = Data.define(:status, :headers, :body, :duration_ms)
11
+ OPEN_TIMEOUT_SECONDS = 2
12
+ READ_TIMEOUT_SECONDS = 10
13
+
14
+ attr_reader :origin
15
+
16
+ def initialize(origin:, base_path:)
17
+ @origin = self.class.origin!(origin)
18
+ @base_path = normalized_base_path(base_path)
19
+ end
20
+
21
+ def request(method:, path:, headers:, body:, max_response_bytes:)
22
+ uri = request_uri(path)
23
+ request = request_class(method).new(uri)
24
+ headers.each { |name, value| request[name] = value }
25
+ request.body = body unless body.nil?
26
+ started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
27
+ response = perform(uri, request, max_response_bytes:)
28
+ response.with(duration_ms: elapsed_milliseconds(started))
29
+ rescue Net::OpenTimeout, Net::ReadTimeout
30
+ raise TimeoutError, 'Testing-control request exceeded its bounded timeout'
31
+ rescue ResponseTooLarge, Error
32
+ raise
33
+ rescue IOError, SystemCallError => e
34
+ raise ConnectionError, "Testing-control loopback request failed (#{e.class.name})"
35
+ end
36
+
37
+ # rubocop:disable Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
38
+ def self.origin!(value)
39
+ uri = URI.parse(value)
40
+ valid = uri.is_a?(URI::HTTP) && uri.scheme == 'http' && uri.host == '127.0.0.1' &&
41
+ uri.authority&.match?(/:\d+\z/) && uri.port.between?(1, 65_535) && !uri.userinfo &&
42
+ uri.path.empty? && !uri.query && !uri.fragment
43
+ raise Error, 'Conformance requires an explicit bare 127.0.0.1 HTTP origin' unless valid
44
+
45
+ uri.to_s
46
+ rescue URI::InvalidURIError
47
+ raise Error, 'Conformance requires an explicit bare 127.0.0.1 HTTP origin'
48
+ end
49
+ # rubocop:enable Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
50
+
51
+ private
52
+
53
+ def normalized_base_path(value)
54
+ unless value.is_a?(String) && value.start_with?('/') && !value.include?('?') && !value.include?('#')
55
+ raise Error, 'Testing-control base path must be an absolute path'
56
+ end
57
+
58
+ value.sub(%r{/+\z}, '')
59
+ end
60
+
61
+ def request_uri(path)
62
+ raise Error, 'Testing-control route must be a fixed absolute path' unless path.start_with?('/')
63
+
64
+ uri = URI.parse(origin)
65
+ uri.path = "#{@base_path}#{path}"
66
+ uri
67
+ end
68
+
69
+ def request_class(method)
70
+ {
71
+ get: Net::HTTP::Get,
72
+ put: Net::HTTP::Put,
73
+ post: Net::HTTP::Post
74
+ }.fetch(method)
75
+ rescue KeyError
76
+ raise Error, "Unsupported testing-control HTTP method #{method.inspect}"
77
+ end
78
+
79
+ def perform(uri, request, max_response_bytes:)
80
+ client = Net::HTTP.new(uri.host, uri.port, nil)
81
+ client.open_timeout = OPEN_TIMEOUT_SECONDS
82
+ client.read_timeout = READ_TIMEOUT_SECONDS
83
+ result = nil
84
+ client.request(request) do |response|
85
+ body = +''
86
+ response.read_body do |chunk|
87
+ if body.bytesize + chunk.bytesize > max_response_bytes
88
+ raise ResponseTooLarge, "Testing-control response exceeds #{max_response_bytes} bytes"
89
+ end
90
+
91
+ body << chunk
92
+ end
93
+ result = Response.new(
94
+ status: response.code.to_i,
95
+ headers: response.each_header.to_h,
96
+ body:,
97
+ duration_ms: nil
98
+ )
99
+ end
100
+ result || raise(ConnectionError, 'Testing-control loopback response was unavailable')
101
+ end
102
+
103
+ def elapsed_milliseconds(started)
104
+ ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started) * 1000).round(3)
105
+ end
106
+
107
+ class Error < StandardError; end
108
+ class TimeoutError < Error; end
109
+ class ResponseTooLarge < Error; end
110
+ class ConnectionError < Error; end
111
+ end
112
+ end
113
+ end
114
+ end