driftproof 0.5.0 → 0.7.1

package/spec/receipt.schema.json

@@ -2,7 +2,7 @@
   "$schema": "https://json-schema.org/draft/2020-12/schema",
   "$id": "https://driftproofhq.com/spec/receipt.schema.json",
   "title": "driftproof receipt",
-  "description": "A signed, dated record of running one agent skill's eval suite with and without the skill on one model version, with sampled judge scores and confidence bands. Receipt spec v0.4 — additive over v0.3.1: adds per-case, per-arm generation `usage` (input/output/cached tokens + measured wall_ms) captured from the surfaces that report it, a separate per-case `judge_usage` (measurement overhead, EXCLUDED from every skill-value figure by construction — `economics.judge_excluded` is const true), a run-level `run.pricing_snapshot` freezing the registry prices the derived dollar figures were computed from (so a receipt keeps its meaning when prices later change), and a derived `economics` block (per-arm mean cost/call, skill incremental cost per call and per 1k calls, output-length delta, median wall_ms with IQR). The three value axes — accuracy lift, cost, latency — are recorded separately and NEVER combined into a composite score. All v0.3.1 semantics are unchanged and every prior receipt still validates against its own frozen schema (v0.1, v0.2, v0.3, v0.3.1). The TESTED tightening (see allOf) is unchanged: the interop relaxations remain available only below TESTED.",
+  "description": "A hash-verified, dated record of running one agent skill eval suite with and without the skill on one model version, with the GENERATION sampled n times per arm and the judge sampled k times inside each draw. Receipt spec v0.5 — additive over v0.4: adds per-case `generation` carrying the full draw list (each draw with its own generation_hash, nested judge samples, mean and stddev), the across-draw mean and sd, the mean judge-level sd, and the variance_ratio between them (null when the judge sd is zero, never a division result); adds the sampling policy actually applied (n_planned, n_drawn, n_measured, n_unmeasured, stopping_reason); records a timed-out draw as status `unmeasured` with NO score, excluded from every statistic rather than counted as zero; and adds a per-suite canary. All v0.4 semantics are unchanged and every prior receipt still validates against its own frozen schema (v0.1, v0.2, v0.3, v0.3.1, v0.4).",
   "type": "object",
   "additionalProperties": false,
   "required": [
@@ -115,12 +115,108 @@
           }
         }
       }
+    },
+    {
+      "$comment": "driftproof/generation-sampled-declared",
+      "description": "F-014-F, first direction: a receipt carrying a draw set or a variance ratio must declare the capability. Without this a third-party emitter validates as v0.5 while carrying none of what v0.5 exists to add.",
+      "if": {
+        "required": [
+          "results"
+        ],
+        "properties": {
+          "results": {
+            "required": [
+              "cases"
+            ],
+            "properties": {
+              "cases": {
+                "contains": {
+                  "required": [
+                    "generation"
+                  ]
+                }
+              }
+            }
+          }
+        }
+      },
+      "then": {
+        "required": [
+          "generation_sampled"
+        ],
+        "properties": {
+          "generation_sampled": {
+            "const": true
+          }
+        }
+      }
+    },
+    {
+      "$comment": "driftproof/generation-sampled-honest",
+      "description": "F-014-F, second direction: a receipt that declares the capability must carry it. A flag assertable by a receipt carrying nothing would be a second way to claim a capability falsely, and would close the exposure in one direction only.",
+      "if": {
+        "required": [
+          "generation_sampled"
+        ],
+        "properties": {
+          "generation_sampled": {
+            "const": true
+          }
+        }
+      },
+      "then": {
+        "required": [
+          "results"
+        ],
+        "properties": {
+          "results": {
+            "required": [
+              "cases"
+            ],
+            "properties": {
+              "cases": {
+                "contains": {
+                  "required": [
+                    "generation"
+                  ]
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    {
+      "$comment": "driftproof/generation-sampled-canary",
+      "description": "F-015-A, the other half of F-014-F: a receipt declaring the capability must also carry the per-suite canary. F-014-F named BOTH results.cases[].generation and suite.canary; spec 015 bound the first and left this one, so a receipt could still claim v0.5 conformance while carrying only half of what v0.5 adds. A receipt that declares generation_sampled has by construction run a suite of ours, so it has a canary to record; omitting it is the same false claim in the other half. Legacy is untouched: a v0.4-and-earlier receipt is governed by its own frozen schema, and a v0.5 receipt that ran no generation sampling declares nothing and is unaffected.",
+      "if": {
+        "required": [
+          "generation_sampled"
+        ],
+        "properties": {
+          "generation_sampled": {
+            "const": true
+          }
+        }
+      },
+      "then": {
+        "required": [
+          "suite"
+        ],
+        "properties": {
+          "suite": {
+            "required": [
+              "canary"
+            ]
+          }
+        }
+      }
     }
   ],
   "properties": {
     "schema_version": {
       "type": "string",
-      "const": "0.4"
+      "const": "0.5"
     },
     "skill": {
       "type": "object",
@@ -179,6 +275,11 @@
         "case_count": {
           "type": "integer",
           "minimum": 0
+        },
+        "canary": {
+          "type": "string",
+          "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$",
+          "description": "Per-suite canary GUID: a leaked suite is detectable in a corpus. Detection aid, not a control."
         }
       }
     },
@@ -247,7 +348,7 @@
         "failed_case_count": {
           "type": "integer",
           "minimum": 0,
-          "description": "v0.3.1 (optional). Number of cases marked failed_timeout (excluded from aggregates)."
+          "description": "v0.3.1 (optional; redefined in v0.5). The number of CASES excluded from the aggregates: exactly the length of results.aggregates.excluded_cases, computed from that list so the two cannot disagree. Exclusion is PAIRWISE since v0.5 — a case with any unmeasured arm leaves BOTH arms together and is counted ONCE here, however many of its arms failed."
         },
         "runner_version": {
           "type": "string",
@@ -626,6 +727,226 @@
                     "minimum": 0
                   }
                 }
+              },
+              "generation": {
+                "type": "object",
+                "additionalProperties": true,
+                "required": [
+                  "n_drawn",
+                  "n_measured",
+                  "n_unmeasured",
+                  "stopping_reason",
+                  "draws"
+                ],
+                "properties": {
+                  "n_planned": {
+                    "type": "integer",
+                    "minimum": 1
+                  },
+                  "n_drawn": {
+                    "type": "integer",
+                    "minimum": 0
+                  },
+                  "n_measured": {
+                    "type": "integer",
+                    "minimum": 0
+                  },
+                  "n_unmeasured": {
+                    "type": "integer",
+                    "minimum": 0
+                  },
+                  "stopping_reason": {
+                    "enum": [
+                      "min_reached",
+                      "stabilised",
+                      "max_reached",
+                      "unmeasured_exhausted",
+                      "below_min",
+                      "escalating"
+                    ]
+                  },
+                  "mean": {
+                    "type": [
+                      "number",
+                      "null"
+                    ]
+                  },
+                  "sd": {
+                    "type": [
+                      "number",
+                      "null"
+                    ]
+                  },
+                  "judge_sd_mean": {
+                    "type": [
+                      "number",
+                      "null"
+                    ]
+                  },
+                  "variance_ratio": {
+                    "type": [
+                      "number",
+                      "null"
+                    ]
+                  },
+                  "draws": {
+                    "type": "array",
+                    "items": {
+                      "type": "object",
+                      "additionalProperties": true,
+                      "required": [
+                        "draw_index",
+                        "status"
+                      ],
+                      "properties": {
+                        "draw_index": {
+                          "type": "integer",
+                          "minimum": 0
+                        },
+                        "status": {
+                          "enum": [
+                            "measured",
+                            "unmeasured"
+                          ]
+                        },
+                        "generation_hash": {
+                          "type": [
+                            "string",
+                            "null"
+                          ]
+                        },
+                        "samples": {
+                          "type": "array",
+                          "items": {
+                            "type": "number"
+                          }
+                        },
+                        "judge_sample_hashes": {
+                          "type": "array",
+                          "items": {
+                            "type": "string"
+                          }
+                        },
+                        "mean": {
+                          "type": [
+                            "number",
+                            "null"
+                          ]
+                        },
+                        "stddev": {
+                          "type": [
+                            "number",
+                            "null"
+                          ]
+                        },
+                        "reason": {
+                          "type": "string"
+                        },
+                        "usage": {
+                          "type": "object"
+                        },
+                        "judge_usage": {
+                          "type": "object"
+                        }
+                      },
+                      "allOf": [
+                        {
+                          "if": {
+                            "properties": {
+                              "status": {
+                                "const": "unmeasured"
+                              }
+                            },
+                            "required": [
+                              "status"
+                            ]
+                          },
+                          "then": {
+                            "properties": {
+                              "mean": {
+                                "type": "null"
+                              },
+                              "stddev": {
+                                "type": "null"
+                              },
+                              "samples": {
+                                "maxItems": 0
+                              }
+                            }
+                          }
+                        },
+                        {
+                          "if": {
+                            "properties": {
+                              "status": {
+                                "const": "measured"
+                              }
+                            },
+                            "required": [
+                              "status"
+                            ]
+                          },
+                          "then": {
+                            "required": [
+                              "generation_hash",
+                              "samples",
+                              "mean",
+                              "stddev"
+                            ],
+                            "properties": {
+                              "mean": {
+                                "type": "number"
+                              },
+                              "generation_hash": {
+                                "type": "string"
+                              }
+                            }
+                          }
+                        }
+                      ]
+                    }
+                  },
+                  "variance_ratio_unavailable": {
+                    "type": [
+                      "string",
+                      "null"
+                    ],
+                    "enum": [
+                      "single_judge_sample",
+                      "judge_sd_zero",
+                      "no_measured_draws",
+                      "judge_samples_unknown",
+                      null
+                    ],
+                    "description": "WHICH null the variance_ratio is (F-014-C). `null` when a ratio was formed. `single_judge_sample`: k=1, so the per-draw judge spread is 0 by construction and the ratio is undefined — this is the shape every pre-v0.5 example produced. `judge_sd_zero`: k>=2 and the judge agreed with itself perfectly inside every measured draw. `no_measured_draws`: nothing was measured. `judge_samples_unknown`: the draws carry no sample list, so the count cannot be established and saying which null it is would assert a cause this control cannot reach. Each names what was OBSERVED and none names a cause."
+                  }
+                },
+                "allOf": [
+                  {
+                    "$comment": "driftproof/variance-ratio-cause",
+                    "description": "F-014-C: a null ratio must say WHICH null it is. Bound to the null rather than required outright, so a receipt that formed a ratio carries no reason and nothing in the archive is retroactively incomplete.",
+                    "if": {
+                      "required": [
+                        "variance_ratio"
+                      ],
+                      "properties": {
+                        "variance_ratio": {
+                          "type": "null"
+                        }
+                      }
+                    },
+                    "then": {
+                      "required": [
+                        "variance_ratio_unavailable"
+                      ],
+                      "properties": {
+                        "variance_ratio_unavailable": {
+                          "type": "string"
+                        }
+                      }
+                    }
+                  }
+                ]
               }
             }
           }
@@ -643,6 +964,38 @@
             },
             "baseline": {
               "$ref": "#/$defs/modeAggregate"
+            },
+            "excluded_cases": {
+              "type": "array",
+              "description": "Cases excluded from BOTH arms of the aggregate because at least one of their arms had no measured result (spec 017 AC-6). `comparison.delta` is paired by construction, so a case that cannot be measured on one side is removed from both rather than from one — otherwise the delta is a mean over one case set minus a mean over another. The case itself remains in results.cases: it is removed from the mean, not from the record. Absent when nothing was excluded.",
+              "items": {
+                "type": "object",
+                "additionalProperties": true,
+                "required": [
+                  "id",
+                  "reason"
+                ],
+                "properties": {
+                  "id": {
+                    "type": "string",
+                    "description": "The case id excluded from both arms."
+                  },
+                  "modes": {
+                    "type": "array",
+                    "items": {
+                      "enum": [
+                        "with_skill",
+                        "baseline"
+                      ]
+                    },
+                    "description": "Which arm or arms were unusable."
+                  },
+                  "reason": {
+                    "type": "string",
+                    "description": "What was observed. Names no cause the receipt cannot establish."
+                  }
+                }
+              }
             }
           }
         }
@@ -920,6 +1273,10 @@
           }
         }
       }
+    },
+    "generation_sampled": {
+      "type": "boolean",
+      "description": "CAPABILITY FLAG (v0.5). A receipt that carries across-draw statistics — any results.cases[] entry with a `generation` block — MUST declare `generation_sampled: true`, and a receipt that declares it MUST carry at least one AND MUST carry `suite.canary` (F-015-A: F-014-F named both blocks). ABSENT MEANS LEGACY: a v0.5 receipt that ran no generation sampling (an imported DECLARED receipt, for instance) omits this field and stays valid, and every v0.4-and-earlier receipt is unaffected. The flag exists because v0.5 otherwise let a receipt claim conformance while carrying none of what v0.5 adds (F-014-F); binding the requirement to what the receipt DECLARES rather than to its verification_level closes that without invalidating a single archived receipt."
     }
   },
   "$defs": {