overseer-testing-protocol 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 1733fd2abba19f5f65ad617c29110553112503f8797b527a9d59b72023a281d5
+  data.tar.gz: a737cfad69c9b6fb01523d7e9273f8bdb9589ef7c89f984749768863388b8480
+SHA512:
+  metadata.gz: e5c6d9bdc242edac4bd68815fbb6d7ce8a59f2b99ead35b1515c4a52eddc6e0b72a0e140a7be121b726fd736ec49cfdca1b46c6d8c9e7ae50d9371ddfb4264e7
+  data.tar.gz: 8c2eaeaebf9d1d0d17dbd4660d4664f980844da6b8212cc70c8248fda38fd494395b7d9f27e3d22883462e890b31ebfaadfda5bbd465a9f7479ac02b6e91bf47

data/LICENSE

@@ -0,0 +1,19 @@
+Copyright 2026 Olistik
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,75 @@
+# Overseer testing protocol
+
+The standalone home of **Overseer testing-control v3**. The Overseer names in
+headers and JSON identify the protocol. Frontend tests, backend tests, curl and
+CI can all use it without installing a particular test runner or application.
+
+The protocol gives tests a common way to prepare synthetic data, call the
+application's normal API, and inspect the result. The backend defines the
+available operations and their business meaning.
+
+The [specification](docs/testing-control-protocol.md),
+[OpenAPI](protocol/testing-control/v3/openapi.yaml),
+[JSON Schema](protocol/testing-control/v3/schema.json) and
+[fixtures](protocol/testing-control/v3/fixtures/manifest.json) define the wire
+contract. The [implementation guide](docs/testing-control-implementation-guide.md)
+explains states, spies, probes and sinks. Product capability names and their
+schemas belong to each backend.
+
+| Route relative to the testing base path | Purpose |
+| --- | --- |
+| `GET /capabilities` | Discover supported setup, observations and limits |
+| `PUT /reset` | Restore the declared baseline or require a restart |
+| `POST /states` | Arrange named synthetic preconditions |
+| `POST /probes` | Read a safe projection, including an internal spy |
+| `POST /sinks/query` | Read captured fake effects without consuming them |
+
+## Using the protocol in an application
+
+For most backend developers, the main entry point to this gem is the
+`overseer-testing-conformance` command: run it against an implemented testing
+API in your local tests or CI.
+
+You can implement that API yourself using the specification. A Rails app can
+instead use the optional
+[`overseer-testing-control-rails` helper](https://source.olisti.co/overseer/helper-rails)
+for routing, JSON validation, correlation and bounded storage. The application
+still owns its setup handlers, read-only observations, fake providers and reset
+behavior. The helper uses the protocol's Ruby methods internally; application
+handlers normally do not need to call `Overseer::TestingControl::ProtocolV3`.
+
+Direct use of those methods is useful when writing your own adapter. They
+validate incoming and outgoing JSON and construct protocol envelopes. Request
+validation runs before a handler changes data; the command checks the HTTP
+behavior exercised by its tests. One does not replace the other.
+
+Adding the protocol gem alone creates no Rails routes, migrations, model hooks
+or fake providers. A custom backend can follow the specification without
+loading this gem and install the command only in its test tooling.
+
+The optional Ruby package `overseer-testing-protocol` requires Ruby 3.2 or newer.
+Its only runtime dependency is `json_schemer`. Other languages can use the data
+artifacts without Ruby or a framework helper.
+Package version `0.1.0` carries wire version `3`; these are separate versions.
+
+See the [implementation guide](docs/testing-control-implementation-guide.md)
+for an example test flow, the command invocation, and its current coverage
+limits. Passing the command is a baseline check, not proof that every protocol
+requirement or business operation has been tested.
+
+## Working on this package
+
+```sh
+bundle install
+bundle exec rake test
+gem build overseer-testing-protocol.gemspec
+bin/overseer-testing-conformance --help
+```
+
+The caller owns test-server startup, isolation and cleanup. The command checks
+an already isolated loopback server, requires explicit expected identity plus
+`--allow-reset`, and uses finite timeouts and bounded reads. Release consumers
+should pin the package or a reviewed source commit. A local package build does
+not publish a release.
+
+Made with ❤️ by [olistik](https://olisti.co)

data/SOURCE.json

@@ -0,0 +1,6 @@
+{
+  "extractedFrom": "overseer/studio",
+  "sourceCommit": "5aa6e6e7664d772ad4e55bc97e3ca118031306ef",
+  "wireVersion": "3",
+  "schemaSha256": "9e0591133fd38c15e96383eae11b202cb27ea8f7ddc98d2a64f2abff326cb924"
+}

data/bin/overseer-testing-conformance

@@ -0,0 +1,53 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require 'optparse'
+require_relative '../lib/overseer/testing_protocol'
+
+options = { base_path: '/v1/testing', format: 'text' }
+parser = OptionParser.new do |cli|
+  cli.banner = 'Usage: overseer-testing-conformance --origin http://127.0.0.1:PORT --application ID --source VERSION --environment ID --allow-reset [--cases FILE]'
+  cli.on('--origin ORIGIN') { |value| options[:origin] = value }
+  cli.on('--base-path PATH') { |value| options[:base_path] = value }
+  cli.on('--application ID') { |value| options[:application] = value }
+  cli.on('--source VERSION') { |value| options[:source] = value }
+  cli.on('--environment ID') { |value| options[:environment] = value }
+  cli.on('--cases FILE') { |value| options[:cases] = value }
+  cli.on('--format FORMAT', %w[text json]) { |value| options[:format] = value }
+  cli.on('--allow-reset', 'Allow resets of this caller-owned isolated synthetic runtime') { options[:allow_reset] = true }
+  cli.on('-h', '--help') { puts cli; exit 0 }
+end
+
+begin
+  parser.parse!
+  raise OptionParser::InvalidArgument, 'Unexpected positional arguments' unless ARGV.empty?
+  %i[origin application source environment allow_reset].each do |key|
+    raise OptionParser::MissingArgument, "--#{key.to_s.tr('_', '-')}" unless options[key]
+  end
+  protocol = Overseer::TestingControl::ProtocolV3
+  protocol.identifier!(options[:application], label: 'Application ID')
+  protocol.identifier!(options[:environment], label: 'Environment ID')
+  protocol.application_version!(options[:source])
+  conformance = Overseer::TestingControl::Conformance
+  cases = options[:cases] && conformance::CaseFile.load(options[:cases])
+  report = conformance::Runner.new(
+    origin: options[:origin], base_path: options[:base_path], target_label: options[:application],
+    expected_identity: conformance::Runner::ExpectedIdentity.new(
+      application_id: options[:application], application_version: options[:source], environment_id: options[:environment]
+    ),
+    required_capabilities: { 'states' => {}, 'probes' => {}, 'sinks' => {} }, cases:,
+    lifecycle: { 'ownership' => 'explicitly-attached', 'started' => true, 'completed' => true, 'cleanup' => 'not-owned' },
+    redaction_policy: Overseer::TestingControl::Redaction::Policy.new(
+      header_names: %w[authorization cookie], field_patterns: %w[password token secret credential], max_body_bytes: 65_536
+    )
+  ).call
+  puts(options[:format] == 'json' ? JSON.pretty_generate(report) : conformance::Report.render(report))
+  exit(report.fetch('aggregateStatus') == 'passed' ? 0 : 4)
+rescue OptionParser::ParseError => error
+  warn error.message
+  warn parser
+  exit 3
+rescue StandardError => error
+  warn "Conformance could not start (#{error.class.name}); check the supplied configuration."
+  exit 2
+end

data/docs/testing-control-implementation-guide.md

@@ -0,0 +1,116 @@
+# Implementing Overseer testing-control v3
+
+Use this contract from frontend tests, backend tests, curl or CI. The backend
+and test client agree on the testing API; no particular runner is required.
+The [wire specification](testing-control-protocol.md) defines exact behavior.
+
+## Choose how to implement the API
+
+| Component | What it does |
+| --- | --- |
+| Your application | Defines synthetic setups, read-only observations, fake effects and reset behavior. |
+| An optional framework helper | Handles HTTP routes, protocol validation, correlation and responses. |
+| The protocol command | Calls the running testing API and checks the behavior exercised by its tests. |
+
+For Rails, the optional
+[`overseer-testing-control-rails` helper](https://source.olisti.co/overseer/helper-rails)
+also supplies bounded memory stores for fake effects and arrangement
+idempotency. Configure the helper and provide your own handlers and schemas.
+It calls the protocol's Ruby methods internally, so application handlers
+normally do not need a `ProtocolV3` reference.
+
+If you build your own adapter, `Overseer::TestingControl::ProtocolV3` offers
+optional methods such as `validate_document!`, `validate_value!` and `envelope`.
+These save you from repeating JSON validation and response construction. For
+example, the adapter can reject an invalid pending-order input before calling
+the application's order setup handler. You still implement dispatch, identity
+checks and the other HTTP requirements.
+
+You can also implement the specification without using this gem in the backend
+and install only the command in your test tooling. The backend must still
+validate requests: an external check covers the requests it sends, while the
+adapter validates each request it receives. Installing the protocol gem alone
+does not add Rails endpoints, migrations, spies or fake providers.
+
+## Implement the testing API
+
+1. Start one isolated synthetic runtime per independent test suite. Block
+   external egress, omit real credentials, and keep the control API off deployed
+   environments. The library does not create this isolation for you.
+2. Mount the five relative routes under a test-only base path. Verify the test
+   environment, activation flag, source identity and runtime identity before
+   exposing them. Discover capabilities before changing data.
+3. Register named states that preserve domain invariants. Validate closed,
+   bounded input and output schemas. Return identifiers and safe fields;
+   passwords and tokens never belong in responses or observations.
+4. Reset all owned state or advertise `runtime-restart`. A failed reset must
+   stop the scenario. Reset includes captured observations and idempotency.
+5. Call the ordinary public API with normal authentication and the same run ID.
+   Give every public call its own correlation ID; the optional step ID is a
+   useful test label. The public API does not acquire a protocol envelope.
+6. Observe database state or captured internal calls through named probes.
+   Capture supported external effects through product-owned fake adapters.
+   Query a sink using the public call's correlation ID in `data.filter` and a
+   new query correlation ID in `meta`. Queries never consume records.
+7. Validate response envelopes and advertised result schemas before assertions.
+   Never assume asynchronous work is complete merely because enqueue succeeded.
+
+For example, an order test discovers `pending-order`, resets the test server,
+creates a synthetic order through `/states`, and pays through the normal order
+API with normal authentication. It then reads the order status through a probe
+and queries a fake email sink for the confirmation attempt. The application
+defines those names and payloads; the protocol does not know about orders.
+
+## Expose spies and fake effects
+
+An internal spy should record only the named event and allowlisted synthetic
+identifiers needed for a test. Bound its storage, partition by run, clear it on
+reset, and expose a read-only probe. Do not expose method names, call stacks,
+arbitrary arguments, SQL, model serializers or a general mocking endpoint.
+
+A sink should intercept the real effect boundary, preserve domain behavior and
+callbacks, record only an allowlisted projection, and return the response the
+application expects from the fake provider. Distinguish requested, accepted,
+completed and failed outcomes according to what was actually observed. A
+pre-enqueue sink proves the request intent only. Unsupported effects must stay
+contained by the runtime even when there is no corresponding sink.
+
+## Check the running API
+
+Run HTTP conformance against your already isolated disposable server:
+
+```sh
+bundle exec overseer-testing-conformance \
+  --origin http://127.0.0.1:3100 --base-path /v1/testing \
+  --application my-api --source source-snapshot-001 \
+  --environment local-test-001 --allow-reset
+```
+
+Replace the example application, source and environment IDs with the exact
+values configured for your running backend. The command checks those identities
+before stateful checks. It resets synthetic data and does not start or stop the
+server. It accepts an explicit `http://127.0.0.1:PORT` origin. A backend using
+`runtime-restart` needs a caller that can recreate the runtime; this attached
+command cannot perform that restart itself.
+
+For application-specific state/probe checks, append
+`--cases test/conformance.json` and provide that file in your application.
+The case format is demonstrated by
+[`conformance-case.json`](../protocol/testing-control/v3/fixtures/conformance-case.json).
+It allows named setup, read-only probes, identifier bindings and bounded JSON
+assertions. It cannot run scripts, public API calls or sink queries.
+
+The current command checks discovery, protocol and identity handling, JSON
+responses, several invalid requests, reset responses, and any supplied
+state/probe cases. It does not exercise `/sinks/query`, prove that reset clears
+captured records, or check repeated requests with the same idempotency key.
+It also cannot prove that a probe is read-only or that network containment is
+in place. The specification's [conformance section](testing-control-protocol.md#15-black-box-conformance)
+records the current coverage and its limits.
+
+Keep application request tests for those protocol requirements, plus an
+end-to-end test that exercises the normal API and checks the resulting spies
+and sinks. For a Rails app using the helper, use the helper in the isolated
+test environment, the protocol command in CI, and application tests for the
+business behavior and remaining coverage. A passing command does not replace
+request validation or prove that every advertised operation works.