setspec 0.2.0__tar.gz → 0.3.0__tar.gz

{setspec-0.2.0 → setspec-0.3.0}/.github/workflows/ci.yml

@@ -91,6 +91,44 @@ jobs:
       - run: pip install . --no-deps
       - run: pytest -m contract
 
+  # The three published guarantees get their own jobs rather than sharing `contracts`' single
+  # red mark, because each answers a different question and the answers are acted on differently:
+  # a snapshot diff means "bump the version", a golden failure means "the artifact is wrong", and
+  # a cross-version failure means "a consumer at another version just broke".
+  schema-snapshot-diff:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with: { python-version: "3.12" }
+      - run: pip install --require-hashes -r requirements/ci.lock
+      - run: pip install . --no-deps
+      # ADR-0009 rule 7: a schema change without a version bump fails CI. The test regenerates
+      # every published schema from its writer model and diffs it against the committed snapshot.
+      - run: pytest tests/contract/test_schema_snapshots.py -m contract
+
+  golden-validation:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with: { python-version: "3.12" }
+      - run: pip install --require-hashes -r requirements/ci.lock
+      - run: pip install . --no-deps
+      - run: pytest tests/contract/test_goldens.py -m contract
+
+  cross-version-compatibility:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with: { python-version: "3.12" }
+      - run: pip install --require-hashes -r requirements/ci.lock
+      - run: pip install . --no-deps
+      # Testing Standards §8.3, proven by the repository that makes the promise: a v1.0 reader
+      # accepts a synthetic v1.1 golden without loss and refuses a synthetic v2.0 by major.
+      - run: pytest tests/contract/test_cross_version.py -m contract
+
   security:
     runs-on: ubuntu-latest
     steps:
@@ -148,3 +186,18 @@ jobs:
           marker = pathlib.Path(setspec.__file__).parent / 'py.typed'
           assert marker.is_file(), 'py.typed missing from the installed wheel'
           "
+      # The schemas and goldens are package data (ADR-0009 rule 7), and package data is exactly
+      # what a build is most likely to leave out silently: every test in this repository passes
+      # against the source tree whether or not the files reach the wheel. This is the only job
+      # that would notice, which is why it asserts against the *installed* distribution.
+      - name: schemas and goldens ship in the wheel
+        run: |
+          python -c "
+          from setspec import PUBLISHED_SCHEMAS, golden_payloads, json_schema_for
+          for schema, versions in PUBLISHED_SCHEMAS.items():
+              for version in versions:
+                  assert json_schema_for(schema, version)['title'] == schema
+                  goldens = golden_payloads(schema, version)
+                  assert len(goldens) >= 3, f'{schema} {version} ships {len(goldens)} goldens'
+          print(f'{len(PUBLISHED_SCHEMAS)} schemas with goldens loaded from the installed wheel')
+          "

{setspec-0.2.0 → setspec-0.3.0}/CHANGELOG.md

@@ -7,7 +7,84 @@ packaging and release standards §3.
 
 ## [Unreleased]
 
+## [0.3.0] — 2026-08-28
+
 ### Added
+
+- **The v1.0 freeze.** `DRAFT_SCHEMAS` is now empty. Every registered payload type —
+  `model.identity`, `machine.profile`, `benchmark.result`, `benchmark.run_summary`,
+  `capability.evidence`, `benchmark.evidence_bundle`, `benchmark.goal_pack` and
+  `benchmark.calibration_report` — is a published `1.0` that may no longer be reshaped in place.
+  FreeWeight's real output (P6 onward, and P8A–8B for the two goal payloads) demanded no field
+  change, so the freeze is a promotion rather than a correction; the one field the pass did add,
+  `MetricValueFields.metric_key`, landed before it and is frozen *with* the schema.
+
+  The set survives its own emptiness deliberately: it is the mechanism a future draft uses, and
+  deleting it would leave the next provisional payload type either shipping silently provisional or
+  inventing a second way to say so.
+
+- **Generated JSON Schema for every published version**, committed as package data under
+  `setspec/schemas/<schema>/<version>.json` and reachable through `json_schema_for()`. Generated
+  from the **writer** (`extra="forbid"`) half, so a published document says
+  `additionalProperties: false` — which is the contract API standards §7 rule 5 states and the one
+  a producer's test suite asserts its output against. The reader policy stays where it belongs, in
+  the reader: `load_envelope` accepts an unknown minor by comparing majors, never by loosening a
+  version's schema to admit fields it cannot describe.
+
+  Every `$ref` resolves inside the document's own `$defs`. Nothing is fetched (spec §14).
+
+- **25 golden payloads**, three or more per version, under
+  `setspec/goldens/<schema>/<version>/<name>.json` and reachable through `golden_payloads()` and
+  `golden_names()`. Each version ships a `minimal` (only required fields), a `full` (every field
+  populated) and, wherever the payload carries a measurement at all, an `unsupported`-heavy example.
+  `capability.evidence` adds `goal`, a calibrated `user.noir_tech_voice` record with the whole
+  goal-sourced group populated; `benchmark.goal_pack` adds `starter_unforked`; and
+  `benchmark.calibration_report` adds `gate_failed` — the outcome `capability.evidence`
+  deliberately cannot express, because a goal below its gate emits no evidence record at all.
+
+  Which versions need an `unsupported` golden is derived from the published schema rather than
+  listed in the test: a payload whose document contains no `{"const": "unsupported"}` branch has no
+  measurement field to leave unsupported, so `benchmark.goal_pack` and
+  `benchmark.calibration_report` are excused by the artifact itself rather than by an exception
+  someone maintains.
+
+- **`setspec.artifacts`** — `PUBLISHED_SCHEMAS`, `json_schema_for()`, `golden_payloads()`,
+  `golden_names()`, `payload_pair()`, `build_json_schema()` and `render_schema_document()`. The
+  first four are re-exported from `setspec` directly, matching spec §7's "schema artefacts" group:
+  an accessor that *returns* a version's artifacts is not itself versioned.
+
+  `PUBLISHED_SCHEMAS` records **exact** versions, unlike `SUPPORTED_SCHEMAS`, which records
+  supported majors. An artifact is a file that either exists or does not, so "highest minor known"
+  is the wrong shape for it — and a contract test asserts the two agree, so a schema that can be
+  negotiated but not validated, or validated but not negotiated, fails the build.
+
+- **Three contract jobs, one per published guarantee.** `schema-snapshot-diff` regenerates every
+  schema from its models and diffs it against the committed snapshot — ADR-0009 rule 7, *a schema
+  change without a version bump fails CI*, made mechanical and proven by a deliberate test-only
+  mutation. `golden-validation` checks every golden against its writer model, its reader model, the
+  published JSON Schema and a canonical round trip. `cross-version-compatibility` builds a synthetic
+  `1.1` document from each `full` golden and asserts a `1.0` reader accepts it, preserves the field
+  it does not know, and re-exports it intact — then builds a synthetic `2.0` and asserts the refusal
+  names both versions. They are separate jobs rather than steps because the three failures are acted
+  on differently: bump the version, fix the artifact, or go and warn a consumer.
+
+- **`install-check` now asserts the package data reaches the wheel.** Every test in this repository
+  passes against the source tree whether or not the schemas and goldens are built into the
+  distribution, so this job is the only one that would notice. It loads all eight schemas and their
+  goldens out of the *installed* wheel.
+
+- **`docs/schemas.md`** — the human-readable catalogue: what ships and where, every payload type
+  with its required-field count and its goldens, how to consume the artifacts from a repository that
+  shares no code with this one, and a table of every cross-field rule the JSON Schema **cannot**
+  express and the models therefore still own.
+
+- **`jsonschema` and `types-jsonschema` in the `dev` extra.** Test-only, and justified by the
+  assertion they exist for: a golden is validated against the *published document* a non-Python
+  consumer receives, not only against the model that generated it. Without a real validator that
+  check would be a hand-rolled subset of draft 2020-12 maintained in this repository — the thing
+  under test also serving as the thing testing it. Nothing under `src/` imports either, and the base
+  install still pulls in no validator. `requirements/ci.lock` regenerated.
+
 - **`metric_key` on `MetricValueFields`.** The model declared value, unit, aggregation, direction,
   sample count and dispersion — and nothing saying *which metric it is*. Both
   `BenchmarkResultFields.metrics` and `BenchmarkRunSummaryFields.aggregate_metrics` carry sequences
@@ -58,6 +135,14 @@ packaging and release standards §3.
 
 ### Changed
 
+- The five payload modules' status notes read **frozen (`1.0`)** rather than **draft (`1.0`)**, and
+  say what the freeze binds them to: a new optional field is a minor bump, and a removal, rename,
+  retype or tightening is a major — never an edit in place.
+- The two tests that asserted the draft state now assert the frozen one
+  (`test_no_registered_schema_is_still_draft`, `test_the_schema_is_frozen`). They are the same
+  guarantee read from the other side, and leaving them asserting `DRAFT_SCHEMAS == SUPPORTED_SCHEMAS`
+  would have made the freeze a change the test suite refused.
+
 - A bare reserved root is now refused by `validate_capability` and reported `False` by
   `is_known_capability`. `user` is a namespace, not a capability: a payload claiming
   `capability_id: "user"` has lost the identity that is the entire point of the namespace.
@@ -71,6 +156,7 @@ packaging and release standards §3.
   declaring `1.2` or later is unaffected. Producers on `1.1` must emit roots that exist at `1.1`.
 
 ### Fixed
+
 - **Coverage measured a directory nothing imports.** `[tool.coverage.run] source` named
   `src/setspec`, the source *path*, while CI installs the built distribution — so the moment the
   jobs stopped using an editable install, coverage reported **0 %** and failed the 95 % floor for a
@@ -89,6 +175,13 @@ packaging and release standards §3.
   (`baseaicore`, `modelrack`, `sweatmeter`). Without it, `mypy --strict` in a consuming repository
   cannot see this package's types at all and treats every import from it as untyped.
 
+### Known gaps
+
+- `event.envelope` and `error.envelope` (Phase 3) and `prompt.record` / `prompt.manifest`
+  (Phase 5) are **not** part of this freeze, because they do not exist yet: a schema is frozen by
+  being published, and an unwritten one has nothing to publish. The freeze covers the eight payload
+  types that cross an application boundary today, not the eleven ADR-0009 anticipates.
+
 ## [0.2.0] — 2026-08-23
 
 Phase 2 of the [development plan](docs/packages/setspec/development-plan.md): provisional

setspec-0.3.0/PHASE4_ISSUES.md

@@ -0,0 +1,99 @@
+# Phase 4 — issues to address
+
+Written at the end of Phase 4 (freeze v1.0, publish schemas and goldens; `setspec 0.3.0`). Each
+entry is something a later phase, a docs change, a consumer or a release step has to resolve.
+Nothing here blocks Phase 4's acceptance criteria; everything here would become a defect if it were
+forgotten.
+
+---
+
+## Status — 2026-08-28
+
+| # | Issue | Status |
+|---|---|---|
+| 1 | The freeze covers eight payload types, not ADR-0009's eleven | **Open — by design.** Phases 3 and 5 own the other three. |
+| 2 | Goldens are authored inputs, but a SetSpec writer emits every declared key | **Needs a decision.** |
+| 3 | `ContributingMetricFields.metric_key` carries no pattern while `MetricValueFields.metric_key` does | **Needs a decision.** |
+| 4 | Cross-field rules are invisible in the published JSON Schema | **Open — documented.** `docs/schemas.md` §4 lists every one. |
+| 5 | `jsonschema` joined the `dev` extra; `requirements/ci.lock` regenerated | **Closed** — verify `pip-audit` in CI. |
+| 6 | The release is not tagged or published yet | **Owner: you.** Commands below. |
+
+---
+
+## 1. The freeze covers eight payload types, not eleven
+
+ADR-0009 lists eleven initial payload types. `event.envelope` and `error.envelope` (Phase 3) and
+`prompt.record` / `prompt.manifest` (Phase 5) are not written, so they are not frozen — a schema is
+frozen by being published, and an unwritten one has nothing to publish. `DRAFT_SCHEMAS` is empty
+**now**; when Phase 3 or 5 lands a new payload type it should re-enter that set until its own
+freeze, which is exactly the mechanism the set survives its emptiness for. The changelog's *Known
+gaps* section says the same. Nothing to do until those phases start, except not to read "empty"
+as "finished".
+
+## 2. Goldens are authored inputs, but a SetSpec writer emits every declared key
+
+`dump_envelope(CapabilityEvidenceOut(...))` serialises through `model_dump()`, which writes every
+declared field — a non-goal record carries `goal_hash: null`, `uncalibrated: false`, and so on. The
+`capability.evidence/1.0/full` golden was authored the other way round: it omits the goal group
+entirely, because "fully populated" was read as "every field a *non-goal* record populates". Both
+forms validate under both models, so the contract is not broken, but a producer's structural test
+("my keys match the full golden's keys") fails against the file and passes against the file *as a
+SetSpec writer would dump it*. FreeWeight's contract test compares against the latter.
+
+**Decision needed:** should goldens be committed *as written* (every declared key present, nulls
+included), so that "matches the golden structurally" is a byte-level statement? If yes, regenerate
+the 25 goldens through their `Out` models once and add a contract test that a golden equals its own
+writer-dumped form. If no, say so in `docs/schemas.md` §3 so consumers compare against the dumped
+form, as FreeWeight now does.
+
+## 3. `ContributingMetricFields.metric_key` has no pattern
+
+`MetricValueFields.metric_key` is constrained to lower snake case, dot-separable
+(`^[a-z][a-z0-9_]*(\.[a-z][a-z0-9_]*)*$`). `ContributingMetricFields.metric_key` — the key inside
+`capability.evidence.contributing_metrics` — is only `min_length=1`. FreeWeight writes
+`<suite_key>.<metric_key>` there (`native.tool_use.task_success`), `criterion.<key>` for a goal's
+own record and `goal.<slug>.composite_score` for a goal contributing to a shipped capability; all
+three happen to satisfy the stricter pattern. Tightening the field is a **major** change now that
+`1.0` is frozen, so it can only land as `capability.evidence 2.0`, and only if a consumer ever
+needs the guarantee. Recorded so the asymmetry is a decision rather than an accident.
+
+## 4. Cross-field rules are invisible in the published JSON Schema
+
+Pydantic renders types, ranges, patterns and required keys; it cannot render a `model_validator`.
+`runtime_profile_hash` agreeing with its profile, `score_method_mix` summing to one,
+`measured_at ≤ computed_at`, the goal group's five coherence rules — every one is enforced by the
+models only. `docs/schemas.md` §4 tables them, and every golden is validated against *both* the
+schema and the model for exactly this reason. A non-Python consumer that needs those rules needs a
+second validation step this package does not ship. Nothing to do unless such a consumer appears.
+
+## 5. `jsonschema` in the `dev` extra
+
+Added so the golden contract test validates against the *published* document with a real
+draft-2020-12 validator rather than a hand-rolled subset. Test only — nothing under `src/`
+imports it and the base install pulls in no validator. `requirements/ci.lock` was regenerated with
+`pip-compile --generate-hashes` (jsonschema 4.26.0, jsonschema-specifications, referencing,
+rpds-py, attrs, types-jsonschema). `release.lock` is unchanged. CI's `security` job audits both
+locks; the first run after this lands is the one to watch.
+
+## 6. Tagging and publishing 0.3.0
+
+Nothing was tagged or published. `__about__.py` says `0.3.0` and the changelog carries a dated
+`[0.3.0]` section. The release procedure (packaging standards §6), from the SetSpec repository:
+
+```bash
+cd ~/ai/suite/py/SetSpec
+git add -A
+git commit -m "feat(setspec): freeze v1.0, publish JSON Schema and goldens (Phase 4, 0.3.0)"
+git push origin main
+# wait for CI to be green on main, then:
+git tag -a v0.3.0 -m "setspec 0.3.0 — v1.0 contracts frozen; JSON Schema and goldens published"
+git push origin v0.3.0
+# release.yml builds, tests the wheel, publishes via Trusted Publishing and creates the release.
+# Verify:
+python -m venv /tmp/setspec-check && /tmp/setspec-check/bin/pip install setspec==0.3.0 && \
+  /tmp/setspec-check/bin/python -c "from setspec import PUBLISHED_SCHEMAS, golden_payloads, SchemaVersion; \
+  print(len(PUBLISHED_SCHEMAS), len(golden_payloads('capability.evidence', SchemaVersion(1, 0))))"
+```
+
+FreeWeight's `pyproject.toml` already pins `setspec>=0.3,<0.4`; its `install-check` job cannot
+resolve until this release is on PyPI.

{setspec-0.2.0 → setspec-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: setspec
-Version: 0.2.0
+Version: 0.3.0
 Summary: Every versioned data contract that crosses an application boundary: benchmark results, capability evidence, event/error envelopes, prompt records.
 Project-URL: Homepage, https://github.com/JPKell/SetSpec
 Project-URL: Documentation, https://github.com/JPKell/SetSpec/tree/main/docs
@@ -21,25 +21,31 @@ Requires-Dist: jinja2<4,>=3.1
 Requires-Dist: pydantic<3,>=2.9
 Provides-Extra: dev
 Requires-Dist: import-linter<3,>=2.0; extra == 'dev'
+Requires-Dist: jsonschema<5,>=4.23; extra == 'dev'
 Requires-Dist: mypy<2,>=1.11; extra == 'dev'
 Requires-Dist: pytest-cov<6,>=5; extra == 'dev'
 Requires-Dist: pytest-randomly<4,>=3; extra == 'dev'
 Requires-Dist: pytest<10,>=9.0.3; extra == 'dev'
 Requires-Dist: respx<1,>=0.21; extra == 'dev'
 Requires-Dist: ruff<1,>=0.6; extra == 'dev'
+Requires-Dist: types-jsonschema<5,>=4.23; extra == 'dev'
 Description-Content-Type: text/markdown
 
 # SetSpec
 
 Every versioned data contract that crosses an application boundary: benchmark results, capability evidence, event/error envelopes, prompt records.
 
-**Status:** `0.2.0` — Phases 1–2 complete. The envelope, version negotiation and serialization core
-are implemented and tested; `model.identity`, `machine.profile`, `benchmark.result`,
-`benchmark.run_summary`, `capability.evidence` and `benchmark.evidence_bundle` are registered in
-`SUPPORTED_SCHEMAS`, but **draft**: Phase 4 may still reshape a field once FreeWeight has produced
-real results against these payloads. Which schemas are still provisional is readable at runtime
-from `setspec.DRAFT_SCHEMAS`, not only stated here — freezing one is a deletion from that set.
-Event and error envelopes arrive in Phase 3. See the
+**Status:** `0.3.0` — Phases 1–2, 3A and 4 complete, and **the v1.0 contracts are frozen.** Eight
+payload types are published at `1.0` — `model.identity`, `machine.profile`, `benchmark.result`,
+`benchmark.run_summary`, `capability.evidence`, `benchmark.evidence_bundle`,
+`benchmark.goal_pack` and `benchmark.calibration_report` — each with generated JSON Schema and at
+least three golden payloads shipped as package data. `setspec.DRAFT_SCHEMAS` is empty, which is
+where the freeze is readable at runtime rather than only stated here; from now on a new optional
+field is a minor bump and anything else is a major, enforced by a snapshot diff in CI.
+
+The [schema catalogue](docs/schemas.md) lists every payload type, its artifacts, and the
+cross-field rules the JSON Schema cannot express. Event and error envelopes (Phase 3) and prompt
+records (Phase 5) are not yet written and are therefore not part of the freeze. See the
 [development plan](docs/packages/setspec/development-plan.md) for what each phase adds.
 
 Part of the **Local AI Suite**.

{setspec-0.2.0 → setspec-0.3.0}/README.md

@@ -2,13 +2,17 @@
 
 Every versioned data contract that crosses an application boundary: benchmark results, capability evidence, event/error envelopes, prompt records.
 
-**Status:** `0.2.0` — Phases 1–2 complete. The envelope, version negotiation and serialization core
-are implemented and tested; `model.identity`, `machine.profile`, `benchmark.result`,
-`benchmark.run_summary`, `capability.evidence` and `benchmark.evidence_bundle` are registered in
-`SUPPORTED_SCHEMAS`, but **draft**: Phase 4 may still reshape a field once FreeWeight has produced
-real results against these payloads. Which schemas are still provisional is readable at runtime
-from `setspec.DRAFT_SCHEMAS`, not only stated here — freezing one is a deletion from that set.
-Event and error envelopes arrive in Phase 3. See the
+**Status:** `0.3.0` — Phases 1–2, 3A and 4 complete, and **the v1.0 contracts are frozen.** Eight
+payload types are published at `1.0` — `model.identity`, `machine.profile`, `benchmark.result`,
+`benchmark.run_summary`, `capability.evidence`, `benchmark.evidence_bundle`,
+`benchmark.goal_pack` and `benchmark.calibration_report` — each with generated JSON Schema and at
+least three golden payloads shipped as package data. `setspec.DRAFT_SCHEMAS` is empty, which is
+where the freeze is readable at runtime rather than only stated here; from now on a new optional
+field is a minor bump and anything else is a major, enforced by a snapshot diff in CI.
+
+The [schema catalogue](docs/schemas.md) lists every payload type, its artifacts, and the
+cross-field rules the JSON Schema cannot express. Event and error envelopes (Phase 3) and prompt
+records (Phase 5) are not yet written and are therefore not part of the freeze. See the
 [development plan](docs/packages/setspec/development-plan.md) for what each phase adds.
 
 Part of the **Local AI Suite**.

{setspec-0.2.0 → setspec-0.3.0}/docs/README.md

@@ -4,5 +4,6 @@ This directory contains the maintained documentation for SetSpec.
 
 ## Documents
 
+- [Schema catalogue](schemas.md) — every payload type and version, with its artifacts
 - [Development plan](packages/setspec/development-plan.md)
 - [Specification](packages/setspec/spec.md)

{setspec-0.2.0 → setspec-0.3.0}/docs/packages/setspec/spec.md

@@ -124,9 +124,11 @@ RESERVED_ROOTS: frozenset[str]        # roots valid ONLY as a specialization; {"
 validate_capability(capability_id: str) -> CapabilityId
 is_known_capability(capability_id: str) -> bool
 
-# Schema artefacts
-json_schema_for(schema: str, version: SchemaVersion) -> dict
-golden_payloads(schema: str, version: SchemaVersion) -> list[dict]
+# Schema artefacts (package data; every published version keeps both for the life of its major)
+PUBLISHED_SCHEMAS: Mapping[str, tuple[SchemaVersion, ...]]   # exact versions with committed artefacts
+json_schema_for(schema: str, version: SchemaVersion) -> dict   # the committed JSON Schema snapshot
+golden_payloads(schema: str, version: SchemaVersion) -> list[dict]   # ≥ 3 per version, by name
+golden_names(schema: str, version: SchemaVersion) -> tuple[str, ...]  # the same order as above
 
 # Prompts (ADR-0028) — one implementation of a determinism contract, not three
 class PromptLibrary:

setspec-0.3.0/docs/schemas.md

@@ -0,0 +1,195 @@
+# Schema catalogue
+
+Every payload type `setspec` publishes, at every version, with the artifacts that make it usable
+from a repository that shares no code with this one.
+
+**Status: frozen at `1.0`** (Phase 4, `setspec 0.3.0`). `DRAFT_SCHEMAS` is empty. From here every
+change follows the ordinary rules: a new optional field is a **minor** bump, and a removed,
+renamed, retyped or newly-tightened field is a **major**. Neither happens by editing a payload
+module in place — the snapshot contract test fails the build if the generated schema stops matching
+the committed one, which is ADR-0009 rule 7 made mechanical.
+
+---
+
+## 1. What ships, and where it lives
+
+| Artifact | Location in the installed package | Accessor |
+|---|---|---|
+| JSON Schema, one per version | `setspec/schemas/<schema>/<version>.json` | `json_schema_for(schema, version)` |
+| Golden payloads, ≥ 3 per version | `setspec/goldens/<schema>/<version>/<name>.json` | `golden_payloads(schema, version)` |
+| The names of those goldens | — | `golden_names(schema, version)` |
+| Every published schema and version | — | `PUBLISHED_SCHEMAS` |
+
+All four are importable from `setspec` directly. The files are **package data**, loadable through
+`importlib.resources` and present in the built wheel — the `install-check` CI job asserts that
+against the installed distribution, because every test in this repository passes against the source
+tree whether or not the files reach the wheel.
+
+Nothing is fetched. A published schema resolves every `$ref` inside its own `$defs`, so validating
+an export never depends on a registry being reachable (spec §14).
+
+## 2. The catalogue
+
+| Schema | Version | Python module | Writer / reader | Goldens |
+|---|---|---|---|---|
+| `model.identity` | 1.0 | `setspec.model.v1` | `ModelIdentityOut` / `ModelIdentityIn` | `minimal`, `full`, `unsupported` |
+| `machine.profile` | 1.0 | `setspec.machine.v1` | `MachineProfileOut` / `MachineProfileIn` | `minimal`, `full`, `unsupported` |
+| `benchmark.result` | 1.0 | `setspec.benchmark.v1` | `BenchmarkResultOut` / `BenchmarkResultIn` | `minimal`, `full`, `unsupported` |
+| `benchmark.run_summary` | 1.0 | `setspec.benchmark.v1` | `BenchmarkRunSummaryOut` / `BenchmarkRunSummaryIn` | `minimal`, `full`, `unsupported` |
+| `capability.evidence` | 1.0 | `setspec.capability.v1` | `CapabilityEvidenceOut` / `CapabilityEvidenceIn` | `minimal`, `full`, `goal`, `unsupported` |
+| `benchmark.evidence_bundle` | 1.0 | `setspec.capability.v1` | `EvidenceBundleOut` / `EvidenceBundleIn` | `minimal`, `full`, `unsupported` |
+| `benchmark.goal_pack` | 1.0 | `setspec.goal.v1` | `GoalPackOut` / `GoalPackIn` | `minimal`, `full`, `starter_unforked` |
+| `benchmark.calibration_report` | 1.0 | `setspec.goal.v1` | `CalibrationReportOut` / `CalibrationReportIn` | `minimal`, `full`, `gate_failed` |
+
+### `model.identity` — which weights, plus what the provider says about them
+
+5 required fields of 25. The identity triple (`provider_kind`, `provider_model_name`,
+`artifact_digest`) plus its two derived fields, then the refreshable descriptor. `canonical_id` and
+`identity_confidence` are **recomputed and checked** against the triple on validation: they are
+pure functions of it, carried on the wire for convenience rather than as independent facts
+(ADR-0024). Every descriptor quantity defaults to `"unsupported"`, because a provider that reports
+no layer count has not reported zero layers.
+
+### `machine.profile` — where a measurement happened
+
+7 required of 14. A field-for-field mirror of `baseaicore.MachineProfile`, including which fields
+are required-but-nullable: a producer must say it could not read the hostname rather than omit the
+key. `machine_fingerprint` is the one hash-shaped field this package carries **without**
+recomputing it — the policy deciding which fields feed a fingerprint may change, while a profile
+read back years later must still reconstruct exactly as stored.
+
+### `benchmark.result` — one benchmark, one subject, one machine
+
+12 required of 23, the largest payload in the suite. Carries Machine Identity §6's minimum
+provenance set as nested objects: `suite`, `execution`, `environment`, `application`,
+`reproducibility`, plus the optional `machine_profile` and `telemetry_summary`.
+`runtime_profile_hash` is recomputed from the embedded `runtime_profile` and must agree
+(ADR-0023). Four further cross-field rules apply — see §4.
+
+### `benchmark.run_summary` — the roll-up of many results
+
+10 required of 15. Deliberately lighter than a result: the same subject/suite/environment building
+blocks, a run state, three timestamps and `aggregate_metrics`. `INTERRUPTED` is a distinct status
+from `FAILED` and means the process died and the run is resumable, not that it failed.
+
+### `capability.evidence` — the record LoadCoach routes on
+
+14 required of 26, and the suite's most load-bearing contract (ADR-0022). Identity, score,
+confidence, the counts behind them, the hard-separation inputs (`benchmark_versions`,
+`dataset_hashes`, `prompt_subset_hashes`, `goal_hash`, `judge_set`) and the two timestamps whose
+order is enforced: `measured_at` is what freshness decays from and can never follow `computed_at`.
+
+The **goal-sourced group** (ADR-0032 §5) is optional and absent on an ordinary record:
+`judge_validity_factor` (exactly `1.0` for every rung 1–4 measurement), `goal_hash`,
+`goal_pack_version`, `score_method_mix`, `judge_set`, `calibration`, `uncalibrated`. The `goal`
+golden populates all of it; `uncalibrated: true` is **refused** rather than published, because a
+goal below its calibration gate emits no record at all.
+
+### `benchmark.evidence_bundle` — the FreeWeight → LoadCoach payload
+
+2 required of 3: `source_id`, `complete`, `evidence`. `complete: true` is what lets a consumer
+infer removal — evidence held locally for this `source_id` and absent from a complete bundle is
+marked superseded, never deleted, and never inferred from a partial bundle. The bundle carries no
+`generated_at`: that lives on the envelope, and a client stores *that* value to send back as its
+next `?since=`.
+
+### `benchmark.goal_pack` — a user-authored goal, portable and hash-pinned
+
+6 required of 12. Criteria with their ladder rung and weight, tasks with their prompt hashes, and
+the jury when any criterion is judged. Weights must sum to `1`; a judged criterion needs an
+anchored ordinal scale and a `judge_set`. The author's calibration **grades do not travel** — an
+importer who wants the rubric held to their own taste recalibrates against their own grades.
+
+### `benchmark.calibration_report` — how well the judge agreed with the author
+
+11 required of 14. A payload in its own right rather than a field group on evidence, because it is
+meaningful precisely when **no** evidence was emitted: `gate_failed` is the golden that shows the
+outcome `capability.evidence` deliberately cannot express (ADR-0032 §3). The verdict must follow
+from the numbers — a `passed_gate` that contradicts `weighted_kappa_w` against `min_agreement` is
+refused.
+
+## 3. Consuming these from another repository
+
+No shared code, no import of the producing application, no database:
+
+```python
+from setspec import SchemaVersion, golden_payloads, json_schema_for, load_envelope
+from setspec.capability.v1 import EvidenceBundleIn
+
+# The contract test: read every golden this build publishes.
+for payload in golden_payloads("benchmark.evidence_bundle", SchemaVersion(1, 0)):
+    EvidenceBundleIn.model_validate(payload)
+
+# The real thing: a file a producer exported.
+envelope = load_envelope(exported_bytes, expect="benchmark.evidence_bundle")
+bundle = EvidenceBundleIn.model_validate(envelope.payload)
+```
+
+Read with the **`In`** model, always. It preserves fields this build has not heard of, so an older
+consumer in the middle of a pipeline is a relay rather than a sink (ADR-0009 rule 4). Write with the
+**`Out`** model, which refuses a field the schema does not declare (rule 5).
+
+A non-Python consumer validates against `json_schema_for(...)`, or against the file directly:
+
+```
+setspec/schemas/benchmark.evidence_bundle/1.0.json
+```
+
+The documents declare `$schema: https://json-schema.org/draft/2020-12/schema`.
+
+## 4. What the JSON Schema does not say
+
+The published documents are generated from the **writer** model, so they carry
+`additionalProperties: false` at the top level — that is the writer's contract, and a producer's
+test suite asserts its output validates against it (testing standards §8.2). The reader policy is a
+*reader behaviour*: `load_envelope` accepts any minor within a supported major, including one it has
+never heard of. That is not a loosening of any one version's schema. A `1.1` document is described
+by the `1.1` schema; a `1.0` reader accepts it by comparing majors.
+
+Pydantic renders types, ranges, patterns and required keys. It cannot render a `model_validator`, so
+none of the following appears in the published documents and all of them are enforced by the models:
+
+| Payload | Rule the schema cannot state |
+|---|---|
+| `metric.value` (nested) | A real value needs ≥ 1 supported sample; an unsupported value needs 0; dispersion needs ≥ 2 |
+| `model.identity` | `canonical_id` and `identity_confidence` must agree with the identity triple |
+| `benchmark.result` | `runtime_profile_hash` must recompute from `runtime_profile`; `completed_at ≥ started_at`; `completed_cases ≤ total_cases`; `skip_reason` iff skipped; a completed result has metrics |
+| `benchmark.run_summary` | `runtime_profile_hash` agreement; timing order |
+| `capability.evidence` | `capability_id` in the vocabulary; `measured_at ≤ computed_at`; the five goal-group coherence rules; `score_method_mix` sums to 1 over known rungs |
+| `benchmark.goal_pack` | Criterion weights sum to 1; no duplicate keys; rung-appropriate fields; a judged criterion needs a jury |
+| `benchmark.calibration_report` | The gate verdict must follow from `weighted_kappa_w` against `min_agreement` |
+
+This is why every golden is validated **twice** — once against the schema a non-Python consumer
+uses, once against the model — and why a consumer that can run Python should use the model.
+
+## 5. Versioning and regeneration
+
+A schema version is `MAJOR.MINOR`, independent of this package's own version and of any HTTP API
+version. `SUPPORTED_SCHEMAS` declares supported **majors**; `PUBLISHED_SCHEMAS` declares the exact
+versions that have artifacts. A contract test asserts the two agree, so a schema that can be
+negotiated but not validated — or validated but not negotiated — fails the build.
+
+Adding a version means adding a module (`setspec.benchmark.v2`), registering it, and committing its
+schema and goldens. The old version stays importable for at least one minor release of every
+consumer (ADR-0009 rule 6).
+
+Regenerate the snapshots after any deliberate model change:
+
+```bash
+python - <<'PY'
+from pathlib import Path
+from setspec.artifacts import PUBLISHED_SCHEMAS, build_json_schema, render_schema_document
+
+for schema, versions in PUBLISHED_SCHEMAS.items():
+    for version in versions:
+        path = Path("src/setspec/schemas", schema, f"{version}.json")
+        path.parent.mkdir(parents=True, exist_ok=True)
+        path.write_text(
+            render_schema_document(build_json_schema(schema, version)), encoding="utf-8"
+        )
+PY
+```
+
+Then answer the question CI just asked: was that change additive, or did it deserve a major bump?
+Goldens are **not** regenerated. They are authored, committed and left alone — a golden that is
+rebuilt from the model whenever the model changes is not a golden, it is a mirror.

{setspec-0.2.0 → setspec-0.3.0}/pyproject.toml

@@ -36,6 +36,13 @@ dev = [
     "ruff>=0.6,<1",
     "import-linter>=2.0,<3",
     "respx>=0.21,<1",
+    # Phase 4's contract tests validate every golden against the *published* JSON Schema, not only
+    # against the model that generated it. Without a real validator that assertion would be a
+    # hand-rolled subset of draft 2020-12 written in this repository — which is to say, the thing
+    # under test would also be the thing testing it. Test-only: nothing under `src/` imports it,
+    # and `setspec` itself pulls in no validator (packaging standards §4).
+    "jsonschema>=4.23,<5",
+    "types-jsonschema>=4.23,<5",
 ]
 
 [project.urls]