setspec 0.3.0__tar.gz → 0.5.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {setspec-0.3.0 → setspec-0.5.0}/CHANGELOG.md +99 -0
- {setspec-0.3.0 → setspec-0.5.0}/PKG-INFO +23 -12
- {setspec-0.3.0 → setspec-0.5.0}/README.md +21 -10
- {setspec-0.3.0 → setspec-0.5.0}/docs/packages/setspec/development-plan.md +149 -15
- {setspec-0.3.0 → setspec-0.5.0}/docs/packages/setspec/spec.md +52 -5
- setspec-0.5.0/docs/prompts-adoption-checklist.md +86 -0
- {setspec-0.3.0 → setspec-0.5.0}/docs/schemas.md +86 -11
- {setspec-0.3.0 → setspec-0.5.0}/pyproject.toml +7 -1
- {setspec-0.3.0 → setspec-0.5.0}/requirements/ci.lock +3 -3
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/__about__.py +1 -1
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/artifacts.py +20 -2
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/capability/v1.py +67 -8
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/envelope.py +30 -18
- setspec-0.5.0/src/setspec/goldens/capability.evidence/1.1/full.json +85 -0
- setspec-0.5.0/src/setspec/goldens/capability.evidence/1.1/minimal.json +25 -0
- setspec-0.5.0/src/setspec/goldens/capability.evidence/1.1/unsupported.json +61 -0
- setspec-0.5.0/src/setspec/goldens/governance.egress_decision/1.0/denied_no_ceiling.json +20 -0
- setspec-0.5.0/src/setspec/goldens/governance.egress_decision/1.0/full.json +20 -0
- setspec-0.5.0/src/setspec/goldens/governance.egress_decision/1.0/minimal.json +20 -0
- setspec-0.5.0/src/setspec/goldens/governance.egress_decision/1.0/violation.json +20 -0
- setspec-0.5.0/src/setspec/goldens/model.adapter_manifest/1.0/full.json +19 -0
- setspec-0.5.0/src/setspec/goldens/model.adapter_manifest/1.0/minimal.json +16 -0
- setspec-0.5.0/src/setspec/goldens/model.adapter_manifest/1.0/name_only.json +19 -0
- setspec-0.5.0/src/setspec/goldens/prompt.manifest/1.0/full.json +19 -0
- setspec-0.5.0/src/setspec/goldens/prompt.manifest/1.0/minimal.json +14 -0
- setspec-0.5.0/src/setspec/goldens/prompt.record/1.0/full.json +54 -0
- setspec-0.5.0/src/setspec/goldens/prompt.record/1.0/minimal.json +10 -0
- setspec-0.5.0/src/setspec/governance/v1.py +139 -0
- setspec-0.5.0/src/setspec/model/v1.py +389 -0
- setspec-0.5.0/src/setspec/prompts.py +940 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/schemas/benchmark.evidence_bundle/1.0.json +2 -2
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/schemas/benchmark.result/1.0.json +2 -2
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/schemas/benchmark.run_summary/1.0.json +2 -2
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/schemas/capability.evidence/1.0.json +2 -2
- setspec-0.5.0/src/setspec/schemas/capability.evidence/1.1.json +743 -0
- setspec-0.5.0/src/setspec/schemas/governance.egress_decision/1.0.json +158 -0
- setspec-0.5.0/src/setspec/schemas/model.adapter_manifest/1.0.json +135 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/schemas/model.identity/1.0.json +2 -2
- setspec-0.5.0/src/setspec/schemas/prompt.manifest/1.0.json +37 -0
- setspec-0.5.0/src/setspec/schemas/prompt.record/1.0.json +70 -0
- setspec-0.5.0/tests/contract/test_adapter_axis_i15.py +96 -0
- {setspec-0.3.0 → setspec-0.5.0}/tests/contract/test_cross_version.py +5 -1
- {setspec-0.3.0 → setspec-0.5.0}/tests/contract/test_goldens.py +8 -2
- {setspec-0.3.0 → setspec-0.5.0}/tests/contract/test_schema_snapshots.py +28 -3
- {setspec-0.3.0 → setspec-0.5.0}/tests/unit/test_payloads_capability.py +211 -5
- setspec-0.5.0/tests/unit/test_payloads_governance.py +197 -0
- setspec-0.5.0/tests/unit/test_prompts.py +350 -0
- setspec-0.3.0/src/setspec/model/v1.py +0 -172
- {setspec-0.3.0 → setspec-0.5.0}/.editorconfig +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/.github/workflows/ci.yml +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/.github/workflows/release.yml +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/.gitignore +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/.importlinter +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/.pre-commit-config.yaml +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/CONTRIBUTING.md +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/LICENSE +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/PHASE4_ISSUES.md +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/SECURITY.md +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/docs/README.md +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/requirements/README.md +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/requirements/release.in +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/requirements/release.lock +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/__init__.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/base.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/benchmark/v1.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/error/v1.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/errors.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/event/v1.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goal/v1.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/benchmark.calibration_report/1.0/full.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/benchmark.calibration_report/1.0/gate_failed.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/benchmark.calibration_report/1.0/minimal.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/benchmark.evidence_bundle/1.0/full.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/benchmark.evidence_bundle/1.0/minimal.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/benchmark.evidence_bundle/1.0/unsupported.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/benchmark.goal_pack/1.0/full.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/benchmark.goal_pack/1.0/minimal.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/benchmark.goal_pack/1.0/starter_unforked.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/benchmark.result/1.0/full.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/benchmark.result/1.0/minimal.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/benchmark.result/1.0/unsupported.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/benchmark.run_summary/1.0/full.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/benchmark.run_summary/1.0/minimal.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/benchmark.run_summary/1.0/unsupported.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/capability.evidence/1.0/full.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/capability.evidence/1.0/goal.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/capability.evidence/1.0/minimal.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/capability.evidence/1.0/unsupported.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/machine.profile/1.0/full.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/machine.profile/1.0/minimal.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/machine.profile/1.0/unsupported.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/model.identity/1.0/full.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/model.identity/1.0/minimal.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/goldens/model.identity/1.0/unsupported.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/machine/v1.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/metrics.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/provenance.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/py.typed +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/schemas/benchmark.calibration_report/1.0.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/schemas/benchmark.goal_pack/1.0.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/schemas/machine.profile/1.0.json +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/serialization.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/src/setspec/vocabulary.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/tests/conftest.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/tests/contract/test_version_negotiation.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/tests/unit/test_base.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/tests/unit/test_envelope.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/tests/unit/test_errors.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/tests/unit/test_events.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/tests/unit/test_metrics.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/tests/unit/test_payloads_benchmark.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/tests/unit/test_payloads_goal.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/tests/unit/test_serialization.py +0 -0
- {setspec-0.3.0 → setspec-0.5.0}/tests/unit/test_vocabulary.py +0 -0
|
@@ -7,6 +7,105 @@ packaging and release standards §3.
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.5.0] — 2026-09-02
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- **`capability.evidence` `1.1`** (Phase 6, ADR-0058): an optional `adapter` field naming the
|
|
15
|
+
adapter axis a measurement was taken under. Absent — and byte-for-byte identical to `1.0` — on
|
|
16
|
+
every record measured on the bare base; a `@model_serializer` drops the field from the dump
|
|
17
|
+
entirely when unset, rather than emitting `"adapter": null`, so a non-adapter record written
|
|
18
|
+
through the new model is indistinguishable from what `0.4.0` writes (LA0 exit condition I15).
|
|
19
|
+
Lives on a new sibling class, `CapabilityEvidenceV1_1Fields`
|
|
20
|
+
(`CapabilityEvidenceV1_1Out`/`CapabilityEvidenceV1_1In`), rather than an edit to the existing
|
|
21
|
+
`CapabilityEvidenceFields`/`CapabilityEvidenceOut`/`CapabilityEvidenceIn`, which keep meaning
|
|
22
|
+
`1.0`: `benchmark.evidence_bundle` nests the `capability.evidence` shape by reference, so an
|
|
23
|
+
edit in place would have silently moved that schema's own committed `1.0` snapshot too. The
|
|
24
|
+
mechanism and the naming rule it implies — a bare exported name keeps the version it was frozen
|
|
25
|
+
at, so adopting `1.1` is an explicit import of `CapabilityEvidenceV1_1Out`/`In` and never
|
|
26
|
+
something a dependency upgrade delivers — are ADR-0068.
|
|
27
|
+
- **`model.adapter_manifest` `1.0`** (Phase 6, ADR-0061): the operator-reviewed record behind one
|
|
28
|
+
adapter — `name`, `artifact_file`, `artifact_sha256`, optional `source_sha256`, `base` (a
|
|
29
|
+
provider model name plus an optional artifact digest, at `digest`-or-`name_only` confidence),
|
|
30
|
+
`declared_capabilities`, `data_classification`, `format`, `created_at`, `notes`. `name` and the
|
|
31
|
+
two digest fields are validated by reconstructing a `baseaicore.AdapterIdentity`, reusing that
|
|
32
|
+
type's own name-pattern and digest-normalization rules rather than a second implementation.
|
|
33
|
+
**`data_classification` is required, with no schema default** (ADR-0065 rule 1): a manifest that
|
|
34
|
+
omits it is invalid, never silently defaulted closed.
|
|
35
|
+
- **`governance.egress_decision` `1.0`** (Phase 6, ADR-0051 §4, ADR-0054): one recorded egress
|
|
36
|
+
verdict — `decision_id`, an embedded request (`run_id`, `source_ref`, `data_classification`,
|
|
37
|
+
`target{name, remote, max_data_classification, provider_kind}`, `requested_at`), `verdict`
|
|
38
|
+
(`approved`/`denied`/`violation`), `reason`, `policy_name`, `policy_version`, `decided_at`.
|
|
39
|
+
SetSpec's first payload under a root other than `benchmark`/`capability`/`machine`/`model` —
|
|
40
|
+
added because the shape has a named second reader (IdeaPress's S4 egress badge reads decisions
|
|
41
|
+
PromptCadence exported, with SpotCheck not installed). Two nullable fields, both one level down.
|
|
42
|
+
`target.max_data_classification` is deliberate: "remote with no declared ceiling" is the
|
|
43
|
+
fail-closed case a policy must be able to deny and record. `request.requested_at` mirrors
|
|
44
|
+
SpotCheck spec §7's own `None` default and is on the wire so that its §11 contract 4 —
|
|
45
|
+
`from_payload(to_payload(d))` preserves **every** field — is keepable: a value-object field with
|
|
46
|
+
nowhere to land makes that round trip silently lossy. It is not a second record timestamp;
|
|
47
|
+
`decided_at` is the record's, and it stays required. SpotCheck does not exist as code yet;
|
|
48
|
+
nothing here imports it.
|
|
49
|
+
- JSON Schema and goldens for all three: `capability.evidence/1.1.json` (`minimal`, `full`,
|
|
50
|
+
`unsupported`), `model.adapter_manifest/1.0.json` (`minimal`, `full`, `name_only`),
|
|
51
|
+
`governance.egress_decision/1.0.json` (`minimal`, `full`, `denied_no_ceiling`, `violation`).
|
|
52
|
+
The `capability.evidence/1.0` and `benchmark.evidence_bundle/1.0` schemas and goldens are
|
|
53
|
+
byte-for-byte unchanged — asserted by a dedicated contract test (I15) rather than by inspection.
|
|
54
|
+
|
|
55
|
+
### Changed
|
|
56
|
+
|
|
57
|
+
- **`baseaicore>=0.4.1,<0.5`** (was `>=0.4,<0.5`). `setspec.governance.v1` and
|
|
58
|
+
`setspec.model.v1` import `DataClassification` and the adapter value objects at module scope,
|
|
59
|
+
and neither name exists in `baseaicore 0.4.0` — the old floor permitted an install that raises
|
|
60
|
+
`ImportError` on import. The same floor is what makes the snapshots below reproducible.
|
|
61
|
+
- Five already-committed `1.0` JSON Schema snapshots (`model.identity`, `benchmark.result`,
|
|
62
|
+
`benchmark.run_summary`, `capability.evidence`, `benchmark.evidence_bundle`) were regenerated to
|
|
63
|
+
pick up two `baseaicore 0.4.1` enum docstring edits (`IdentityConfidence`,
|
|
64
|
+
`ModelCapabilityFlag`) that the raised floor above surfaces in their nested `$defs`
|
|
65
|
+
descriptions. **Structurally identical** — no property, type or required-field changed,
|
|
66
|
+
confirmed by a description-stripped diff before committing; the regeneration is a byproduct of
|
|
67
|
+
the dependency bump, not a payload change (`docs/schemas.md` §5).
|
|
68
|
+
- Bumped to 0.5.0 for the `capability.evidence` `1.1` addition and the two new payload types
|
|
69
|
+
(packaging standards §3: additive, non-breaking).
|
|
70
|
+
|
|
71
|
+
## [0.4.0] — 2026-08-29
|
|
72
|
+
|
|
73
|
+
### Added
|
|
74
|
+
|
|
75
|
+
- **`setspec.prompts`** (Phase 5): prompt records and prompt packs, moved from
|
|
76
|
+
`freeweight.services.prompts` — `PromptRecord`, `PromptLibrary`, `load_record()`, `load_pack()`,
|
|
77
|
+
`build_manifest()`/`write_manifest()`, and the pack's three hashes
|
|
78
|
+
(`prompt_record_hash`/`prompt_subset_hash`/`pack_hash`, ADR-0028). Rendering runs through a
|
|
79
|
+
sandboxed, loader-less Jinja2 `Environment` with `StrictUndefined` (prompt-management-standards
|
|
80
|
+
§2.1): a referenced-but-unsupplied variable is an error, a template cannot reach the filesystem,
|
|
81
|
+
and dunder attribute access is refused. `load_pack()`'s `root` parameter is required — the
|
|
82
|
+
package carries no opinion about an application's pack layout; an application wanting a default
|
|
83
|
+
location supplies its own thin wrapper (FreeWeight's `freeweight.services.prompts.load_pack`
|
|
84
|
+
keeps its old default this way).
|
|
85
|
+
|
|
86
|
+
FreeWeight's own prompt pack adopts this module in the same change (pulling its P12 forward):
|
|
87
|
+
`freeweight.services.prompts` is now a re-export shim over `setspec.prompts`, verified to
|
|
88
|
+
produce a byte-identical `pack_hash` and per-record hash for every shipped prompt before and
|
|
89
|
+
after the move, with FreeWeight's own gate green afterwards. Two applications reading the same
|
|
90
|
+
pack format from two processes is the scenario ADR-0028's hashes exist for; the FreeWeight move
|
|
91
|
+
is the first proof of it.
|
|
92
|
+
|
|
93
|
+
- **JSON Schema for `prompt.record` and `prompt.manifest`**, committed as package data under
|
|
94
|
+
`setspec/schemas/prompt.record/1.0.json` and `setspec/schemas/prompt.manifest/1.0.json`, with
|
|
95
|
+
golden examples under `setspec/goldens/prompt.{record,manifest}/1.0/{full,minimal}.json`. Both
|
|
96
|
+
are hand-authored rather than generated: `PromptRecord` is a plain dataclass, not a pydantic
|
|
97
|
+
envelope payload — a prompt record is a pack file on disk, never wrapped in a `SchemaEnvelope` —
|
|
98
|
+
so neither schema is registered in `PUBLISHED_SCHEMAS`/reachable through `json_schema_for()`,
|
|
99
|
+
and the goldens are illustrative examples rather than inputs to the envelope contract-test
|
|
100
|
+
matrix in `tests/contract/test_goldens.py`. Every golden round-trips through the real
|
|
101
|
+
`load_pack()` loader (not just schema validation) and its `pack_sha256` is the module's own
|
|
102
|
+
computed hash, not a hand-typed value.
|
|
103
|
+
|
|
104
|
+
### Changed
|
|
105
|
+
|
|
106
|
+
- Bumped to 0.4.0 for the `setspec.prompts` addition (packaging standards §3: additive,
|
|
107
|
+
non-breaking).
|
|
108
|
+
|
|
10
109
|
## [0.3.0] — 2026-08-28
|
|
11
110
|
|
|
12
111
|
### Added
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: setspec
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.5.0
|
|
4
4
|
Summary: Every versioned data contract that crosses an application boundary: benchmark results, capability evidence, event/error envelopes, prompt records.
|
|
5
5
|
Project-URL: Homepage, https://github.com/JPKell/SetSpec
|
|
6
6
|
Project-URL: Documentation, https://github.com/JPKell/SetSpec/tree/main/docs
|
|
@@ -16,7 +16,7 @@ Classifier: Programming Language :: Python :: 3.12
|
|
|
16
16
|
Classifier: Programming Language :: Python :: 3.13
|
|
17
17
|
Classifier: Typing :: Typed
|
|
18
18
|
Requires-Python: >=3.12
|
|
19
|
-
Requires-Dist: baseaicore<0.5,>=0.4
|
|
19
|
+
Requires-Dist: baseaicore<0.5,>=0.4.1
|
|
20
20
|
Requires-Dist: jinja2<4,>=3.1
|
|
21
21
|
Requires-Dist: pydantic<3,>=2.9
|
|
22
22
|
Provides-Extra: dev
|
|
@@ -35,18 +35,29 @@ Description-Content-Type: text/markdown
|
|
|
35
35
|
|
|
36
36
|
Every versioned data contract that crosses an application boundary: benchmark results, capability evidence, event/error envelopes, prompt records.
|
|
37
37
|
|
|
38
|
-
**Status:** `0.
|
|
39
|
-
|
|
40
|
-
`
|
|
41
|
-
`benchmark.
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
38
|
+
**Status:** `0.5.0` — Phases 1–2, 3A, 4, 5 and 6 complete, and **the v1.0 contracts are frozen**,
|
|
39
|
+
with the first additive minor now published on top of that freeze. Eight payload types remain
|
|
40
|
+
frozen at `1.0` — `model.identity`, `machine.profile`, `benchmark.result`,
|
|
41
|
+
`benchmark.run_summary`, `benchmark.evidence_bundle`,
|
|
42
|
+
`benchmark.goal_pack` and `benchmark.calibration_report`, plus `capability.evidence` itself — each
|
|
43
|
+
with generated JSON Schema and at least three golden payloads shipped as package data.
|
|
44
|
+
`setspec.DRAFT_SCHEMAS` is empty, which is where the freeze is readable at runtime rather than only
|
|
45
|
+
stated here; from now on a new optional field is a minor bump and anything else is a major,
|
|
46
|
+
enforced by a snapshot diff in CI.
|
|
47
|
+
|
|
48
|
+
Phase 6 (the adapter arc's LA0 checkpoint) adds three things without touching any of the above:
|
|
49
|
+
`capability.evidence` gains an additive `1.1` (an optional `adapter` field, absent — and
|
|
50
|
+
byte-identical to `1.0` — on every record with no adapter); `model.adapter_manifest` `1.0`
|
|
51
|
+
publishes the operator-reviewed record behind one adapter; and `governance.egress_decision` `1.0`
|
|
52
|
+
is the package's first payload under a root other than `benchmark`/`capability`/`machine`/`model`,
|
|
53
|
+
carrying a recorded egress verdict for a reader that has SpotCheck installed or not.
|
|
45
54
|
|
|
46
55
|
The [schema catalogue](docs/schemas.md) lists every payload type, its artifacts, and the
|
|
47
|
-
cross-field rules the JSON Schema cannot express. Event and error envelopes (Phase 3)
|
|
48
|
-
|
|
49
|
-
|
|
56
|
+
cross-field rules the JSON Schema cannot express. Event and error envelopes (Phase 3) are not yet
|
|
57
|
+
written and are therefore not part of the freeze. Prompt records (`setspec.prompts`, Phase 5,
|
|
58
|
+
added in 0.4.0) are shipped: prompt packs with their three content hashes and sandboxed
|
|
59
|
+
rendering; they carry their own record schema version rather than joining the frozen payload types.
|
|
60
|
+
See the [development plan](docs/packages/setspec/development-plan.md) for what each phase adds.
|
|
50
61
|
|
|
51
62
|
Part of the **Local AI Suite**.
|
|
52
63
|
|
|
@@ -2,18 +2,29 @@
|
|
|
2
2
|
|
|
3
3
|
Every versioned data contract that crosses an application boundary: benchmark results, capability evidence, event/error envelopes, prompt records.
|
|
4
4
|
|
|
5
|
-
**Status:** `0.
|
|
6
|
-
|
|
7
|
-
`
|
|
8
|
-
`benchmark.
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
5
|
+
**Status:** `0.5.0` — Phases 1–2, 3A, 4, 5 and 6 complete, and **the v1.0 contracts are frozen**,
|
|
6
|
+
with the first additive minor now published on top of that freeze. Eight payload types remain
|
|
7
|
+
frozen at `1.0` — `model.identity`, `machine.profile`, `benchmark.result`,
|
|
8
|
+
`benchmark.run_summary`, `benchmark.evidence_bundle`,
|
|
9
|
+
`benchmark.goal_pack` and `benchmark.calibration_report`, plus `capability.evidence` itself — each
|
|
10
|
+
with generated JSON Schema and at least three golden payloads shipped as package data.
|
|
11
|
+
`setspec.DRAFT_SCHEMAS` is empty, which is where the freeze is readable at runtime rather than only
|
|
12
|
+
stated here; from now on a new optional field is a minor bump and anything else is a major,
|
|
13
|
+
enforced by a snapshot diff in CI.
|
|
14
|
+
|
|
15
|
+
Phase 6 (the adapter arc's LA0 checkpoint) adds three things without touching any of the above:
|
|
16
|
+
`capability.evidence` gains an additive `1.1` (an optional `adapter` field, absent — and
|
|
17
|
+
byte-identical to `1.0` — on every record with no adapter); `model.adapter_manifest` `1.0`
|
|
18
|
+
publishes the operator-reviewed record behind one adapter; and `governance.egress_decision` `1.0`
|
|
19
|
+
is the package's first payload under a root other than `benchmark`/`capability`/`machine`/`model`,
|
|
20
|
+
carrying a recorded egress verdict for a reader that has SpotCheck installed or not.
|
|
12
21
|
|
|
13
22
|
The [schema catalogue](docs/schemas.md) lists every payload type, its artifacts, and the
|
|
14
|
-
cross-field rules the JSON Schema cannot express. Event and error envelopes (Phase 3)
|
|
15
|
-
|
|
16
|
-
|
|
23
|
+
cross-field rules the JSON Schema cannot express. Event and error envelopes (Phase 3) are not yet
|
|
24
|
+
written and are therefore not part of the freeze. Prompt records (`setspec.prompts`, Phase 5,
|
|
25
|
+
added in 0.4.0) are shipped: prompt packs with their three content hashes and sandboxed
|
|
26
|
+
rendering; they carry their own record schema version rather than joining the frozen payload types.
|
|
27
|
+
See the [development plan](docs/packages/setspec/development-plan.md) for what each phase adds.
|
|
17
28
|
|
|
18
29
|
Part of the **Local AI Suite**.
|
|
19
30
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
**Sequence position:** second component. Depends on BaseAiCore Phase 4.
|
|
4
4
|
**Target:** `setspec 0.3.0` by the end of Phase 4; `0.4.0` adds `setspec.prompts` during LoadCoach P4
|
|
5
|
-
(ADR-0028).
|
|
5
|
+
([ADR-0028](../../adr/0028-prompt-pack-granularity.md)).
|
|
6
6
|
|
|
7
7
|
Note on ordering: SetSpec's *benchmark* payloads cannot be finalized before FreeWeight knows what a
|
|
8
8
|
result contains. This plan therefore ships the envelope and negotiation machinery first (Phase 1),
|
|
@@ -20,7 +20,7 @@ correctly when its major version is unsupported — demonstrated with a trivial
|
|
|
20
20
|
**Prerequisites:** `baseaicore>=0.4,<0.5`.
|
|
21
21
|
|
|
22
22
|
**Work**
|
|
23
|
-
* Repository skeleton (as per Packaging Standards).
|
|
23
|
+
* Repository skeleton (as per [Packaging Standards](../../standards/packaging-and-release-standards.md)).
|
|
24
24
|
* `envelope.py`: `SchemaEnvelope`, `GeneratorInfo`, `SchemaVersion`, `SUPPORTED_SCHEMAS`,
|
|
25
25
|
`load_envelope`, `dump_envelope`, `SchemaVersionUnsupported`.
|
|
26
26
|
* `serialization.py`: canonical JSON dump, `Unsupported` codec, RFC 3339 codec, depth/size guards.
|
|
@@ -28,7 +28,7 @@ correctly when its major version is unsupported — demonstrated with a trivial
|
|
|
28
28
|
`PreservingPayload` (`extra="allow"`, for inbound), plus the generator that produces an `Out` and an
|
|
29
29
|
`In` class from one payload definition. The round-trip contract is asserted **per class**: an `In`
|
|
30
30
|
model preserves unknown keys through load → dump, an `Out` model rejects them
|
|
31
|
-
(ADR-0009 rule 4).
|
|
31
|
+
([ADR-0009 rule 4](../../adr/0009-setspec-schema-strategy.md)).
|
|
32
32
|
* `MetricValue` model (value | `"unsupported"`, unit, aggregation, `higher_is_better`, sample count,
|
|
33
33
|
dispersion).
|
|
34
34
|
|
|
@@ -72,7 +72,7 @@ fields.
|
|
|
72
72
|
(model identity, runtime profile, machine, suite, dataset hashes, prompt pack, effective
|
|
73
73
|
parameters, reproducibility fingerprint).
|
|
74
74
|
* `capability/v1.py`: `CapabilityEvidence` with the full normative field set from
|
|
75
|
-
ADR-0022 §1 — including `measured_at`
|
|
75
|
+
[ADR-0022 §1](../../adr/0022-capability-evidence-record-contract.md) — including `measured_at`
|
|
76
76
|
**and** `computed_at` as distinct fields, `vocabulary_version`, `dataset_hashes` and
|
|
77
77
|
`prompt_subset_hashes` — and `EvidenceBundle` with its `complete` flag.
|
|
78
78
|
* `machine/v1.py`: `MachineProfilePayload`; `model/v1.py`: `ModelIdentityPayload`.
|
|
@@ -90,7 +90,7 @@ tests/unit/test_vocabulary.py
|
|
|
90
90
|
**Tests**
|
|
91
91
|
* Round-trip for every model with all-optional and all-populated variants.
|
|
92
92
|
* Provenance completeness: a result missing any required provenance field is rejected, with the field
|
|
93
|
-
named (Machine Identity §6).
|
|
93
|
+
named ([Machine Identity §6](../../architecture/machine-identity-and-reproducibility.md)).
|
|
94
94
|
* Score/confidence range validation; negative sample counts rejected.
|
|
95
95
|
* A `CapabilityEvidence` missing `measured_at`, `policy_version` or `vocabulary_version` is rejected
|
|
96
96
|
with the field named; `measured_at` later than `computed_at` is rejected as incoherent.
|
|
@@ -121,7 +121,7 @@ constrained models that accept results with missing provenance.
|
|
|
121
121
|
* `event/v1.py`: the event **payload** (`event_id`, `sequence`, `type`, `entity`, `timestamp`,
|
|
122
122
|
`message`, `progress`, `data`), carried inside the standard envelope whose `generator` identifies
|
|
123
123
|
the producing application — so the payload has no separate `source` field
|
|
124
|
-
(ADR-0025 §3). Event-type naming enforced by validation.
|
|
124
|
+
([ADR-0025 §3](../../adr/0025-envelope-boundaries.md)). Event-type naming enforced by validation.
|
|
125
125
|
* `error/v1.py`: `ErrorEnvelope` (`code`, `message`, `details`, `request_id`, `timestamp`) — the
|
|
126
126
|
object inside `{"error": {…}}`, transported **unwrapped**; a test asserts it is never emitted inside
|
|
127
127
|
a SetSpec envelope.
|
|
@@ -139,11 +139,11 @@ tests/unit/{test_events,test_errors}.py
|
|
|
139
139
|
* Error envelope: `code` is `SCREAMING_SNAKE_CASE`; `details` never contains a secret-shaped key
|
|
140
140
|
(asserted by a redaction test).
|
|
141
141
|
* Serialization matches the examples in
|
|
142
|
-
API Standards §4 and §8 exactly.
|
|
142
|
+
[API Standards §4 and §8](../../standards/api-and-contract-standards.md) exactly.
|
|
143
143
|
|
|
144
144
|
**Acceptance criteria**
|
|
145
|
-
1. The documented example payloads in API Standards §4 and §8
|
|
146
|
-
and Observability §4.1 validate against these models
|
|
145
|
+
1. The documented example payloads in [API Standards §4 and §8](../../standards/api-and-contract-standards.md)
|
|
146
|
+
and [Observability §4.1](../../standards/observability-standards.md) validate against these models
|
|
147
147
|
verbatim — the check that keeps the three documents from drifting into three wire formats again.
|
|
148
148
|
2. A secret-shaped key in `details` fails the test suite.
|
|
149
149
|
|
|
@@ -161,14 +161,14 @@ per application and testing the registry.
|
|
|
161
161
|
the "where do user-defined capabilities live" question permanently.
|
|
162
162
|
|
|
163
163
|
**Prerequisites:** P2. Independent of P3.
|
|
164
|
-
**Driven by:** ADR-0031,
|
|
165
|
-
ADR-0032.
|
|
164
|
+
**Driven by:** [ADR-0031](../../adr/0031-user-defined-goal-benchmarks.md),
|
|
165
|
+
[ADR-0032](../../adr/0032-judge-validity-and-user-capability-namespace.md).
|
|
166
166
|
**Ships in `setspec 0.3.0`** alongside Phase 4 — no separate release. FreeWeight already pins
|
|
167
167
|
`setspec>=0.3,<0.4` and its P11 already requires SetSpec P4, so the existing prerequisite chain
|
|
168
168
|
delivers vocabulary 1.1 before the M3 contract freeze without any change to either pin. The
|
|
169
169
|
package version is **not** bumped when this phase lands: the work accumulates under
|
|
170
170
|
`## [Unreleased]` and `CAPABILITY_VOCABULARY_VERSION` moves on its own axis
|
|
171
|
-
(Packaging Standards §3.1 — distribution,
|
|
171
|
+
([Packaging Standards §3.1](../../standards/packaging-and-release-standards.md) — distribution,
|
|
172
172
|
HTTP API and schema versions are independent, and the vocabulary is a fourth).
|
|
173
173
|
|
|
174
174
|
**Note for the 0.3.0 release.** This phase narrows forward-compatibility leniency: a payload
|
|
@@ -187,7 +187,7 @@ appear in the changelog rather than passing as an implementation detail.
|
|
|
187
187
|
* `setspec.goal.v1`: `benchmark.goal_pack` and `benchmark.calibration_report`, registered in
|
|
188
188
|
`SUPPORTED_SCHEMAS` and listed in `DRAFT_SCHEMAS`.
|
|
189
189
|
* The goal-sourced field group on `capability.evidence`, every field optional, with the coherence
|
|
190
|
-
validators from ADR-0032 §5.
|
|
190
|
+
validators from [ADR-0032 §5](../../adr/0032-judge-validity-and-user-capability-namespace.md).
|
|
191
191
|
|
|
192
192
|
**Tests**
|
|
193
193
|
* `user.anything_at_all` validates; bare `user` is refused with the reserved roots named.
|
|
@@ -258,7 +258,7 @@ serialization; documented and enforced version rules.
|
|
|
258
258
|
**Deferred:** `routing.decision` export payload; compression; generated non-Python types.
|
|
259
259
|
**Not created:** `production.feedback` — `POST /jobs/{id}/feedback` is an ordinary body of LoadCoach's
|
|
260
260
|
own `/api/v1`, versioned by its path and contracted through the committed OpenAPI snapshot
|
|
261
|
-
(ADR-0025 §5). Recorded as a deliberate rejection.
|
|
261
|
+
([ADR-0025 §5](../../adr/0025-envelope-boundaries.md)). Recorded as a deliberate rejection.
|
|
262
262
|
|
|
263
263
|
---
|
|
264
264
|
|
|
@@ -289,5 +289,139 @@ needing the same.
|
|
|
289
289
|
2. FreeWeight P12 adopts it with no hash change.
|
|
290
290
|
|
|
291
291
|
**Known risks:** Jinja2 entering SetSpec's dependency set. Accepted in
|
|
292
|
-
ADR-0028; the alternative was three implementations of a
|
|
292
|
+
[ADR-0028](../../adr/0028-prompt-pack-granularity.md); the alternative was three implementations of a
|
|
293
293
|
determinism contract.
|
|
294
|
+
|
|
295
|
+
---
|
|
296
|
+
|
|
297
|
+
## Phase 6 — The adapter axis's contracts (LA0, joint with PromptCadence Phase 0)
|
|
298
|
+
|
|
299
|
+
**Goal:** the adapter arc has a wire form before any component serves, routes on, or benchmarks an
|
|
300
|
+
adapter — contracts precede consumers, the standing rule the adapter roadmap states outright for
|
|
301
|
+
LA0. A `setspec`-only reader can validate a manifest and an adapter-bearing evidence record, and
|
|
302
|
+
today's evidence records are provably unmoved.
|
|
303
|
+
|
|
304
|
+
**Prerequisites:** Phase 4 (the `1.0` freeze this phase's additive minor builds on); `baseaicore
|
|
305
|
+
0.4.1` published, carrying `AdapterIdentity`, `DataClassification` and `normalize_digest`
|
|
306
|
+
([ADR-0058](../../adr/0058-the-execution-subject-gains-an-adapter-axis.md)).
|
|
307
|
+
|
|
308
|
+
**Work**
|
|
309
|
+
* `capability.evidence` `1.1`: one optional field, `adapter: AdapterIdentityFields | None`, on a
|
|
310
|
+
**new sibling class** (`CapabilityEvidenceV1_1Fields`) rather than an edit to the frozen
|
|
311
|
+
`CapabilityEvidenceFields` — the latter is nested by reference inside `EvidenceBundleFields`, so
|
|
312
|
+
editing it in place would have silently moved `benchmark.evidence_bundle`'s own committed `1.0`
|
|
313
|
+
schema. `benchmark.evidence_bundle` is untouched by this phase; it keeps nesting the `1.0` shape,
|
|
314
|
+
and gaining the adapter field there is FreeWeight 1.1's work (LA3), not LA0's. A
|
|
315
|
+
`@model_serializer` drops `adapter` from the dump entirely when absent, so a non-adapter record
|
|
316
|
+
written through the `1.1` model is byte-identical to what `1.0` writes — not merely
|
|
317
|
+
structurally compatible, literally the same bytes.
|
|
318
|
+
* `AdapterIdentityFields` (`setspec.model.v1`): the nested exchange form of
|
|
319
|
+
`baseaicore.AdapterIdentity` — `name`, `artifact_digest`, optional `source_digest`,
|
|
320
|
+
`canonical_suffix` (materialized and checked, like `model.identity`'s `canonical_id`).
|
|
321
|
+
* `model.adapter_manifest` `1.0` (`setspec.model.v1`): `AdapterManifestFields` per
|
|
322
|
+
[ADR-0061](../../adr/0061-the-adapter-registry-is-a-directory-and-a-manifest.md) rule 1 —
|
|
323
|
+
`name`, `artifact_file`, `artifact_sha256`, optional `source_sha256`, `base`
|
|
324
|
+
(`AdapterManifestBaseFields`: provider model name, optional artifact digest, checked identity
|
|
325
|
+
confidence), `declared_capabilities` (validated strictly against the current vocabulary, no
|
|
326
|
+
forward-compatibility exception), `data_classification` (**required, no schema default** —
|
|
327
|
+
[ADR-0065](../../adr/0065-an-adapter-is-classified-and-local-only.md) rule 1), `format`
|
|
328
|
+
(fixed `"gguf"`), `created_at`, `notes`. `name`/`artifact_sha256`/`source_sha256` are validated
|
|
329
|
+
by reconstructing a `baseaicore.AdapterIdentity`, reusing its name-pattern and
|
|
330
|
+
digest-normalization rules instead of a second implementation.
|
|
331
|
+
* `governance.egress_decision` `1.0` (new module, `setspec.governance.v1`): SetSpec's fifth owned
|
|
332
|
+
root, per [ADR-0051](../../adr/0051-plans-stay-internal-and-one-payload-travels.md) §4 —
|
|
333
|
+
`decision_id`, `request` (`run_id`, `source_ref`, `data_classification`, `target{name, remote,
|
|
334
|
+
max_data_classification, provider_kind}`, `requested_at`), `verdict`
|
|
335
|
+
(`approved`/`denied`/`violation`, a local `EgressVerdict` `StrEnum` — SetSpec cannot import
|
|
336
|
+
SpotCheck, which does not exist as code yet), `reason`, `policy_name`, `policy_version`,
|
|
337
|
+
`decided_at`. Two nullable fields, both one level down. `target.max_data_classification` carries
|
|
338
|
+
the fail-closed "remote with no declared ceiling" case
|
|
339
|
+
([ADR-0054](../../adr/0054-spotcheck-records-egress-it-does-not-enforce-it.md) rule 3), which
|
|
340
|
+
must be representable rather than schema-refused. `request.requested_at` mirrors SpotCheck spec
|
|
341
|
+
§7's own `None` default and exists so that its §11 contract 4 — `from_payload(to_payload(d))`
|
|
342
|
+
preserves **every** field — can be kept: a value object field with no wire field would make that
|
|
343
|
+
round trip silently lossy. It is not a second record timestamp; `decided_at` remains the
|
|
344
|
+
required one, and the two answer different questions (when the caller built the request, when
|
|
345
|
+
the policy answered it). No cross-field "approved implies a ceiling" validator was added: that is
|
|
346
|
+
`OrderedClassificationPolicy`'s behaviour, and a caller running a different, legitimate policy
|
|
347
|
+
must not have its decisions rejected by the wire shape.
|
|
348
|
+
* Register all three in `SUPPORTED_SCHEMAS` (`capability.evidence` → highest known minor `1.1`;
|
|
349
|
+
the two new schemas → `1.0`) and in `artifacts._REGISTRY`/`PUBLISHED_SCHEMAS`. JSON Schema and
|
|
350
|
+
≥ 3 goldens each: `capability.evidence/1.1` (`minimal`, `full` — adapter-bearing, `unsupported`),
|
|
351
|
+
`model.adapter_manifest/1.0` (`minimal` — name-only base, `full` — digest-verified base,
|
|
352
|
+
`name_only` — a second name-only example with lineage and multiple declared capabilities),
|
|
353
|
+
`governance.egress_decision/1.0` (`minimal`, `full`, `denied_no_ceiling`, `violation`).
|
|
354
|
+
|
|
355
|
+
**Files/subsystems**
|
|
356
|
+
```text
|
|
357
|
+
src/setspec/model/v1.py # + AdapterIdentityFields, AdapterManifestBaseFields,
|
|
358
|
+
# AdapterManifestFields, AdapterManifestOut/In
|
|
359
|
+
src/setspec/capability/v1.py # + CapabilityEvidenceV1_1Fields, …V1_1Out/In
|
|
360
|
+
src/setspec/governance/v1.py # new module
|
|
361
|
+
src/setspec/envelope.py # SUPPORTED_SCHEMAS entries
|
|
362
|
+
src/setspec/artifacts.py # _REGISTRY entries
|
|
363
|
+
src/setspec/schemas/{capability.evidence/1.1,model.adapter_manifest/1.0,
|
|
364
|
+
governance.egress_decision/1.0}.json
|
|
365
|
+
src/setspec/goldens/{capability.evidence/1.1,model.adapter_manifest/1.0,
|
|
366
|
+
governance.egress_decision/1.0}/*.json
|
|
367
|
+
tests/unit/{test_payloads_capability.py (extended), test_payloads_governance.py (new)}
|
|
368
|
+
tests/contract/test_adapter_axis_i15.py # the exit condition, spelled out explicitly
|
|
369
|
+
```
|
|
370
|
+
|
|
371
|
+
**Tests**
|
|
372
|
+
* Round-trip for every new/extended model, including the adapter-absent and adapter-present cases.
|
|
373
|
+
* `test_adapter_axis_i15.py`: every committed `capability.evidence/1.0` golden dumps
|
|
374
|
+
byte-identically whether validated through the `1.0` model or the `1.1` model; a manifest golden
|
|
375
|
+
and an adapter-bearing evidence golden validate through `setspec` alone.
|
|
376
|
+
* `AdapterManifestFields`: digest-verified base and name-only base both validate; a missing
|
|
377
|
+
`data_classification` is refused and names the field; a bare reserved capability root is
|
|
378
|
+
refused; a malformed name or digest is refused via the underlying `AdapterIdentity` check.
|
|
379
|
+
* `GovernanceEgressDecisionFields`: all three verdicts validate; a remote target with no declared
|
|
380
|
+
ceiling validates (the fail-closed case is representable); an unknown verdict is refused;
|
|
381
|
+
`request.requested_at` may be omitted, may be null, normalizes to UTC, refuses a naive datetime,
|
|
382
|
+
and survives the canonical round trip — the last one is the assertion that justifies the field
|
|
383
|
+
existing at all (SpotCheck spec §11 contract 4), and `decided_at` stays required beside it.
|
|
384
|
+
* Updated cross-cutting contract tests: the schema-snapshot suite's "everything is frozen at 1.0"
|
|
385
|
+
assertion is split into "no major has ever moved" (permanent) and "every pre-Phase-6 schema is
|
|
386
|
+
still exactly 1.0" (an explicit, named exception for `capability.evidence`); the
|
|
387
|
+
minimal-vs-full golden-size check compares serialized bytes rather than top-level key count,
|
|
388
|
+
since `governance.egress_decision` has no optional top-level field for "minimal" to omit.
|
|
389
|
+
|
|
390
|
+
**Acceptance criteria**
|
|
391
|
+
1. I15 (adapter-roadmap §7): a `setspec`-only reader validates the manifest and adapter-evidence
|
|
392
|
+
goldens, and today's evidence records round-trip byte-identically — proven by a dedicated test
|
|
393
|
+
over the *existing* committed `1.0` goldens, not by argument.
|
|
394
|
+
2. `capability.evidence/1.0.json` and every `benchmark.evidence_bundle` artifact are unchanged
|
|
395
|
+
(modulo the unrelated `baseaicore 0.4.1` docstring refresh described below).
|
|
396
|
+
3. Every §20 criterion in the [spec](spec.md) continues to hold, including for the two new schemas.
|
|
397
|
+
4. Full gate green: `ruff format --check`, `ruff check`, `mypy --strict`, `lint-imports`,
|
|
398
|
+
`pytest -m "not live and not performance"`, coverage ≥ 95 %.
|
|
399
|
+
|
|
400
|
+
**Known risks:** a dependency-version bump rippling into a frozen schema's committed snapshot
|
|
401
|
+
through no fault of this phase's own field changes — realized during this phase, not merely
|
|
402
|
+
anticipated: `baseaicore 0.4.1` changed two enum docstrings
|
|
403
|
+
(`IdentityConfidence`, `ModelCapabilityFlag`), and pydantic embeds an enum's docstring as its JSON
|
|
404
|
+
Schema `description`. Five already-frozen snapshots
|
|
405
|
+
(`model.identity`, `benchmark.result`, `benchmark.run_summary`, `capability.evidence/1.0`,
|
|
406
|
+
`benchmark.evidence_bundle`) therefore needed regenerating — confirmed, before committing, to be
|
|
407
|
+
identical once `description` fields are stripped from the comparison, i.e. no property, type or
|
|
408
|
+
required-field moved. This is not a payload shape change and not a version bump trigger; it is
|
|
409
|
+
recorded here because "the committed snapshot didn't match the model" is exactly the failure this
|
|
410
|
+
phase's own machinery is built to catch, and the reason mattered enough to state rather than to
|
|
411
|
+
silently regenerate past.
|
|
412
|
+
**Likely failure modes:** editing `CapabilityEvidenceFields` in place instead of subclassing it
|
|
413
|
+
(silently moves `benchmark.evidence_bundle`'s frozen schema); a schema default on
|
|
414
|
+
`data_classification` (reintroduces the fail-open gap ADR-0065 rule 1 closes); validating
|
|
415
|
+
`declared_capabilities` with the forward-compatibility exception (the manifest carries no
|
|
416
|
+
`vocabulary_version` to justify it); a cross-field "approved requires a ceiling" validator on
|
|
417
|
+
`governance.egress_decision` (rejects a legitimate document from a caller's own, non-shipped
|
|
418
|
+
policy — SetSpec carries the shape, not SpotCheck's opinion of it).
|
|
419
|
+
**Gold standards:** additive means byte-identical where nothing changed, not merely
|
|
420
|
+
schema-compatible; a payload's shape says only what every legitimate producer could write, never
|
|
421
|
+
one policy's opinion of what a valid decision looks like.
|
|
422
|
+
**Deferred:** `benchmark.evidence_bundle` gaining the adapter field — its own minor, by the
|
|
423
|
+
same sibling-class mechanism ([ADR-0068](../../adr/0068-a-post-freeze-minor-is-a-sibling-class.md)
|
|
424
|
+
rule 5), scheduled as **Phase 7 / `0.6.0`** (row B5 of
|
|
425
|
+
[outstanding-work §1](../../roadmap/outstanding-work.md)) and required before FreeWeight 1.1 can
|
|
426
|
+
export adapter evidence bundled; SpotCheck
|
|
427
|
+
itself, which this phase publishes the shape for but does not implement.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# SetSpec — Specification
|
|
2
2
|
|
|
3
3
|
**Type:** Python package · **Import/distribution name:** `setspec` · **Layer:** 2 (contract package)
|
|
4
|
-
**Status:** Specified, not implemented. **Decision record:** ADR-0009.
|
|
4
|
+
**Status:** Specified, not implemented. **Decision record:** [ADR-0009](../../adr/0009-setspec-schema-strategy.md).
|
|
5
5
|
|
|
6
6
|
---
|
|
7
7
|
|
|
@@ -48,7 +48,7 @@ applications can exchange data without sharing code or a database.
|
|
|
48
48
|
|
|
49
49
|
`baseaicore`, `pydantic>=2.9,<3`, `jinja2>=3.1,<4`. Nothing else.
|
|
50
50
|
|
|
51
|
-
Jinja2 arrives with `setspec.prompts` (ADR-0028): prompt
|
|
51
|
+
Jinja2 arrives with `setspec.prompts` ([ADR-0028](../../adr/0028-prompt-pack-granularity.md)): prompt
|
|
52
52
|
records are versioned, hash-bearing documents whose hashes appear in cross-application evidence, so
|
|
53
53
|
they belong with the other contracts rather than being implemented three times. Consumers that want
|
|
54
54
|
schemas without a template engine are served by making it an extra if that need ever appears; today
|
|
@@ -99,6 +99,11 @@ CapabilityEvidence # normative field set in ADR-0022: identity triple + ca
|
|
|
99
99
|
# measurement), goal_hash, goal_pack_version, score_method_mix,
|
|
100
100
|
# judge_set, calibration, uncalibrated (refused as True: the gate
|
|
101
101
|
# withholds the record rather than discounting it)
|
|
102
|
+
# `1.1` (ADR-0058, Phase 6) adds one further optional field on a sibling
|
|
103
|
+
# class, CapabilityEvidenceV1_1 (…V1_1Out/…V1_1In): adapter, the
|
|
104
|
+
# optional adapter axis (AdapterIdentity below), absent — and
|
|
105
|
+
# byte-identical to `1.0` — on a bare-base record. The bare
|
|
106
|
+
# CapabilityEvidence name keeps meaning `1.0`.
|
|
102
107
|
GoalPack # benchmark.goal_pack — a user-authored goal, portable and hash-pinned
|
|
103
108
|
# (ADR-0031 §6): slug, goal_hash, criteria with their ladder rung and
|
|
104
109
|
# weight, tasks, judge_set. The author's grades do NOT travel with it.
|
|
@@ -109,6 +114,21 @@ PromptRecord # the prompt record schema (ADR-0028)
|
|
|
109
114
|
PromptManifest # pack manifest, prompt_subset_hash helper
|
|
110
115
|
MachineProfilePayload # exchange form of baseaicore.MachineProfile
|
|
111
116
|
ModelIdentityPayload # exchange form of baseaicore.ModelIdentity + descriptor snapshot
|
|
117
|
+
AdapterIdentity # model.identity's adapter-axis sibling (ADR-0058): the nested exchange
|
|
118
|
+
# form of baseaicore.AdapterIdentity — name, artifact_digest,
|
|
119
|
+
# source_digest, canonical_suffix (materialized and checked, like
|
|
120
|
+
# ModelIdentity's canonical_id). Embedded on CapabilityEvidence `1.1`.
|
|
121
|
+
AdapterManifest # model.adapter_manifest (ADR-0061 rule 1, Phase 6): name, artifact_file,
|
|
122
|
+
# artifact_sha256, source_sha256, base (provider model name + optional
|
|
123
|
+
# artifact digest, at digest-or-name_only confidence),
|
|
124
|
+
# declared_capabilities, data_classification (required, no default —
|
|
125
|
+
# ADR-0065 rule 1), format (fixed "gguf"), created_at, notes
|
|
126
|
+
GovernanceEgressDecision # governance.egress_decision (ADR-0051 §4, ADR-0054, Phase 6): decision_id,
|
|
127
|
+
# request (run_id, source_ref, data_classification, target{name, remote,
|
|
128
|
+
# max_data_classification, provider_kind}, requested_at), verdict
|
|
129
|
+
# (approved | denied | violation), reason, policy_name, policy_version,
|
|
130
|
+
# decided_at. SetSpec's first payload outside the
|
|
131
|
+
# benchmark/capability/machine/model roots.
|
|
112
132
|
EventEnvelope # SSE and job/run event payloads
|
|
113
133
|
ErrorEnvelope # the object inside {"error": {…}} — a shape, transported UNWRAPPED
|
|
114
134
|
# (ADR-0025 §4); never itself put in a SetSpec envelope
|
|
@@ -166,13 +186,28 @@ Owns the **schemas**, not the data. It never persists anything.
|
|
|
166
186
|
7a. A prompt record's `sha256`, a rendering's `rendered_sha256` and a `prompt_subset_hash` are
|
|
167
187
|
byte-stable across platforms, Python versions and installed `setspec` versions within a major —
|
|
168
188
|
they appear in evidence that another application reads, so they are contract-grade, golden-tested
|
|
169
|
-
determinism (ADR-0028 §3).
|
|
189
|
+
determinism ([ADR-0028 §3](../../adr/0028-prompt-pack-granularity.md)).
|
|
170
190
|
8. The capability vocabulary is versioned: additions are minor, removals/redefinitions are major.
|
|
171
191
|
At **1.1** the reserved root `user` was added, whose specializations (`user.<slug>`) carry
|
|
172
192
|
FreeWeight's user-authored goal evidence. It is the only root a user's own measurement ever
|
|
173
193
|
needs: the existing open-ended specialization rule accepts every future goal without a
|
|
174
194
|
further vocabulary change, and no shipped benchmark maps onto it
|
|
175
|
-
(ADR-0032 §1).
|
|
195
|
+
([ADR-0032 §1](../../adr/0032-judge-validity-and-user-capability-namespace.md)).
|
|
196
|
+
9. An additive minor on an **already-published** payload is byte-identical on every document that
|
|
197
|
+
does not use the new field, proved by golden — not merely structurally compatible. Since
|
|
198
|
+
[Phase 4](development-plan.md)'s freeze, this is implemented as a sibling field-definition
|
|
199
|
+
class in the same module (`CapabilityEvidenceV1_1Fields`, subclassing the untouched
|
|
200
|
+
`CapabilityEvidenceFields`), never as an edit to the published class in place: a definition
|
|
201
|
+
nested by reference inside another payload (`capability.evidence` inside
|
|
202
|
+
`benchmark.evidence_bundle`) must not have its shape move when the *other* schema is untouched.
|
|
203
|
+
Where the new field's own default is not itself a wire value the old version could produce
|
|
204
|
+
(`None` meaning "absent," not "present and null"), the model suppresses it from the dump rather
|
|
205
|
+
than emitting it — first exercised at `capability.evidence` `1.1` (ADR-0058). The mechanism and
|
|
206
|
+
its naming rule are [ADR-0068](../../adr/0068-a-post-freeze-minor-is-a-sibling-class.md): the
|
|
207
|
+
bare `CapabilityEvidenceOut`/`In` keep meaning `1.0` permanently, and adopting a minor is an
|
|
208
|
+
explicit import of the version-qualified name, never something a dependency upgrade delivers.
|
|
209
|
+
Carrying a nested payload's new minor into the payload that nests it is that outer payload's
|
|
210
|
+
own minor, versioned separately (rule 5).
|
|
176
211
|
|
|
177
212
|
## 12. Configuration
|
|
178
213
|
|
|
@@ -246,7 +281,19 @@ Coverage floor: **95 %**.
|
|
|
246
281
|
* A schema change without a version bump fails CI (snapshot diff).
|
|
247
282
|
* Coexisting majors live in versioned modules (`setspec.benchmark.v1`, `.v2`) and both are exported
|
|
248
283
|
for at least one minor release of every consumer.
|
|
284
|
+
* Coexisting **minors** of an already-published payload live as sibling classes in the *same*
|
|
285
|
+
module (`CapabilityEvidenceFields` for `1.0`, `CapabilityEvidenceV1_1Fields` for `1.1`), the
|
|
286
|
+
older one left untouched, and the bare exported name keeps the version it was frozen at
|
|
287
|
+
([ADR-0068](../../adr/0068-a-post-freeze-minor-is-a-sibling-class.md)). First exercised at
|
|
288
|
+
[Phase 6](development-plan.md): `capability.evidence` went from a single published version to two
|
|
289
|
+
(`1.0` and `1.1`) without either's committed JSON Schema or goldens moving.
|
|
249
290
|
* The schema compatibility job validates every published version against its goldens on every CI run.
|
|
291
|
+
* A dependency version bump can change a generated schema's `description` text without changing any
|
|
292
|
+
payload's shape, when that text is drawn from the dependency's own docstrings (pydantic embeds an
|
|
293
|
+
enum's `__doc__` in its JSON Schema). This is not a schema version bump trigger — the committed
|
|
294
|
+
snapshot is still regenerated and re-committed, and the change is verified structurally identical
|
|
295
|
+
(same properties, types and required fields) before it is. Observed at Phase 6 against
|
|
296
|
+
`baseaicore 0.4.1`.
|
|
250
297
|
|
|
251
298
|
## 20. Acceptance criteria
|
|
252
299
|
|
|
@@ -264,7 +311,7 @@ Coverage floor: **95 %**.
|
|
|
264
311
|
|
|
265
312
|
* ~~`production.feedback` payload~~ — **not created.** `POST /jobs/{id}/feedback` is an ordinary body
|
|
266
313
|
of LoadCoach's own `/api/v1`, versioned by its path and contracted through the committed OpenAPI
|
|
267
|
-
snapshot (ADR-0025 §5). Recorded as a deliberate rejection
|
|
314
|
+
snapshot ([ADR-0025 §5](../../adr/0025-envelope-boundaries.md)). Recorded as a deliberate rejection
|
|
268
315
|
rather than an oversight.
|
|
269
316
|
* `routing.decision` export payload for sharing routing explanations between tools.
|
|
270
317
|
* `benchmark.suite_manifest` as a shareable payload so suites can be exchanged.
|