autorate 0.1.0__tar.gz

autorate-0.1.0/.gitignore

@@ -0,0 +1,13 @@
+.venv/
+__pycache__/
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+cert.json
+
+# The composite action's badge output. The *committed* badge lives in `.github/badges/`; this is the
+# per-run copy a workflow uploads as an artefact.
+.autorate/
+
+# SDD implementation worktrees (isolated lanes); never part of the published tree.
+.worktrees/

autorate-0.1.0/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 the autorate authors
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

autorate-0.1.0/PKG-INFO

@@ -0,0 +1,417 @@
+Metadata-Version: 2.5
+Name: autorate
+Version: 0.1.0
+Summary: Certified automation rates for decision models: confidence is not permission.
+License: Apache-2.0
+License-File: LICENSE
+Requires-Python: >=3.10
+Requires-Dist: numpy>=1.24
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# autorate
+
+[![autorate gate](.github/badges/autorate-badge.svg)](.github/badges/autorate-badge.json)
+
+Certified automation rates for decision models: confidence is not permission.
+
+A benchmark answers "how good is the model". A certificate answers a question an
+operator can actually be held to: "what share of this model's decisions may run
+unattended while keeping the probability that an automated decision is wrong at most
+`beta`, given the held-out data we really have". When the data cannot support that
+claim, `autorate` prints `NOT_CERTIFIABLE` with a reason code, the observation at the
+requested coverage that caused it, and a diagnostic sweep of the neighbouring coverage
+levels -- evidence for the refusal, not a certificate. It exits 3 and does not publish a
+number about nothing.
+
+## Verification-first navigation
+
+- [Verification status](docs/VERIFICATION.md) · [Known errata](docs/ERRATA.md) · [Results](docs/RESULTS.md)
+- The present implementation covers the decision-certification primitives described below; planned certificate families and guarantees are **not** implemented claims.
+- This is a public repository. No public licensing or release rights are implied.
+
+## What a certificate carries
+
+Stored: `feasible`, `reason`, `backend`, `method`, `alpha`, `beta`, `delta`, `autorate`,
+`autorate_ci`, `conditional_error_bound`, `marginal_error_bound`, `mean_set_size`,
+`qhat`, `n_cal`, `n_eval`, `envelope`, `meta`.
+
+Derived, never stored, because each is a pure function of a stored field -- storing them
+gave one number two owners and a guard that could not see the disagreement: `refusal_rate`
+= `1 - autorate` (every evaluated case is empty, singleton, or multi-label) and `level`
+= `1 - N_CLAIMS * delta` (three bounds are published per certificate).
+
+Two of those rows carry the whole argument:
+
+- `conditional_error_bound` is `P(wrong | automated) <= beta`. `marginal_error_bound`
+  is `P(wrong AND automated)`. They are different quantities, and a paper or a dashboard
+  that quotes one while meaning the other is the failure mode this tool exists to stop.
+- Both are over **cases**, not archive rows. Repeats of one case are one observation, and on this
+  corpus they are not even evenly counted: laya's archived choice slice is 64 cases x 3 rows (192
+  rows), while jev's is 191 rows over the same 64 cases -- one case carries two rows, not three -- and
+  a jev case carries between one and three *distinct* probability vectors. What a repeat shares is the
+  measurement, not the row: laya's three repeats per case carry byte-identical probability vectors,
+  scores and gold labels (not identical rows: the repeat counter and its own elapsed time differ),
+  while jev's repeats need not carry the same vector at all -- so the `n` in every bound counts cases
+  and the record counts are published beside it instead of standing in for it.
+- `envelope` is the machine-readable statement of what the bound assumed -- coverage
+  level, group conditioning, drift, and how empty sets were counted. A certificate read
+  without its envelope is invalid output, so `certify`, `compare` and `gate` all print it.
+- `alpha` is the coverage floor the certificate is held to, not a label on it. An earlier version
+  swept a grid of coverage levels for the largest automation rate and tested the winner on
+  the same evaluation half, then printed "at alpha=0.050" while operating at 0.89
+  coverage (fixed in `241e395`). The pre-fix quantity is still checkable rather than asserted:
+  `data/cert_*.json` carries `meta.diagnostic_curve`, the sweep of coverage levels the old code
+  maximised over, and `meta.closest_attempt`, the point that came closest to the budget.
+  Selecting on the test half is snooping, so the set size now comes from the
+  calibration half at exactly the requested `alpha`; the grid survives only as diagnostic
+  evidence attached to a refusal.
+
+`refusal_rate` and `mean_set_size` exist because an empty certified set is a refusal,
+not a free correct answer: counting it as automation would inflate the rate.
+
+## Installing the wheel, and what it cannot do
+
+`pip install autorate` installs the library and the console script. It does **not** install this
+repository's evidence bundle. `autorate verify` re-derives *this* tree's published numbers, so it
+needs `protocol.json` and the committed `data/` set and refuses with exit `4` when an artefact the
+tables declare is absent -- by design, because the six `data/records_*.jsonl` files carry dataset
+rows (labels, per-option probabilities, gold answers) and are not put in a package that any index
+can host. For the evidence, take the repository, not the wheel:
+
+```bash
+uv add "autorate @ git+https://github.com/Jackxiaozhiren/autorate"
+```
+
+## Commands
+
+```bash
+uv sync --frozen --extra dev
+
+uv run --frozen autorate snapshot --out data/inventory.md   # corpus -> committed inventory
+uv run --frozen autorate export-feishu --out data/records_feishu.jsonl
+uv run --frozen autorate certify data/records_feishu.jsonl --backend laya --out cert.json
+uv run --frozen autorate compare data/cert_laya.json data/cert_jev.json
+uv run --frozen autorate gate data/records_feishu.jsonl --cert data/cert_jev.json --limit 5
+uv run --frozen autorate report --out docs/RESULTS.md   # write the published numbers
+uv run --frozen autorate report --full               # write docs/REPORT.md, one document for a reviewer
+uv run --frozen autorate verify                          # re-derive them, fail on drift
+```
+
+The `uv run --frozen` prefix is not decoration: the console script is not on `PATH`
+after `uv sync`, and a bare `pytest` on this machine resolves to a different
+interpreter entirely.
+
+Exit codes: `0` certified, `3` `NOT_CERTIFIABLE`, `2` usage error -- an input this
+command was handed is wrong, whether you named its path or a default supplied it --
+`4` the committed state cannot reproduce what it publishes: a number that disagrees,
+or a published artefact this repository's own tables declare that this run cannot
+read -- because nothing is at its path, because the path itself refuses the open, or
+because what is there is not the artefact that table publishes: a corrupt certificate,
+a records line that is not a record, a document that is not text, a certificate stored
+in a shape this build does not read, a committed inventory snapshot that will not load
+or is not the mapping its renderers index.
+One table-declared path stays `2` by ruling rather than oversight:
+`data/records_feishu.jsonl` is the default every command reads -- `verify` and `report`
+with no flag, `audit --records` with one -- absent or unreadable alike, and a default is
+something its operator can name and repair.
+A missing `data/inventory.md.json` alongside no corpus is `2` for a reason the refusal
+does not cover -- there is nothing to verify against, which is a different statement
+from a published artefact that cannot be read -- while a snapshot that is present and
+will not load is `4` like any other published artefact.
+
+`snapshot` and `export-feishu` read the corpus: pass `--root`, set
+`AUTORATE_FEISHU_ROOT`, or place the checkout as a sibling of this repo
+(`<workspace>/laya*/research/benchmarks/feishu_zh`). No source file carries an absolute
+machine path, and `tests/test_cli.py` fails the build if one reappears -- a literal like
+that passes every local gate and only breaks in the environment nobody can see.
+
+Everything else runs offline on committed files; no command reaches the network.
+
+## GitHub Action and badge
+
+[`action.yml`](action.yml) is a composite action. It shells out to the CLI and implements **no**
+statistics: no bound, rate or aggregate is computed in the action or in
+[`tools/action_report.py`](tools/action_report.py), which only maps the CLI's own exit code to a state and
+formats what the CLI printed. The one number it parses is the denominator out of the tool's
+`re-derived N /M values` line, so the badge can carry it instead of standing alone.
+
+| input | default | meaning |
+|---|---|---|
+| `version` | empty | run `autorate==<version>` from PyPI through `uvx` |
+| `ref` | empty | run the CLI from a detached worktree at that ref of this repository (needs full history) |
+| `alpha`, `beta` | empty | passed to `certify`; empty means "use the frozen protocol's rubric" |
+| `path` | `protocol.json` | read from the **working tree**, never `git show <rev>:protocol.json` |
+| `records` | empty | when set, `certify` runs too, and its exit 3 is the verdict |
+| `fail_on` | `drift,not-run` | states that fail the job |
+| `badge_dir` | `.autorate` | where the `.svg` and its `.json` record are written |
+| `comment` | `true` | post a PR comment quoting `autorate compare` verbatim (both denominators, no bound diff) |
+
+The states are a closed vocabulary, and none of them is a boolean:
+
+| state | when | what it does **not** mean |
+|---|---|---|
+| `certified` | `verify` exited 0 (and `certify` exited 0, if a records file was given) | not "the model is certified": `protocol.json`'s `claims[]` is where a certificate claim is made or refused |
+| `not-certifiable` | `certify` exited 3 | not a broken pipeline -- a refusal is a result, and the badge says so |
+| `drift` | `verify` exited 4 | not "the numbers moved slightly": the tree cannot reproduce what it publishes |
+| `not-run` | no verdict was produced -- no CLI, no network, a usage error, a shallow checkout for `ref`, or a missing denominator line | never a pass |
+
+`fail_on` defaults to `drift,not-run`: exit 4 fails the job, exit 3 is reported rather than red, and a run
+that produced no verdict is never green. Add `not-certifiable` to make a refusal fail the job as well.
+
+**What the PR comment is, and is not.** F8 rule 4 asks for a bound-versus-previous-bound diff with both
+denominators. The shipped `autorate compare` prints each side's own `n_cal` / `n_eval` but states in its
+own source that it "has no authority to pair anything", and returns a usage error rather than a
+before/after table when the two certificates were frozen under different protocols. The comment therefore
+quotes the command's output verbatim, carries both denominators as the tool printed them, and says plainly
+that it computes no bound; claiming a bound diff would assert a capability the tool does not have
+(PRD §13.3 item 4). When no certificate under `data/` changed — the usual case here — the comment says
+`not-run`, and a comment that cannot be delivered is a named `not-run` in the job summary rather than a
+silent stderr line. The literal rule 4 wording is recorded as a deviation in the milestone report.
+
+Two things about checkouts, because both have already broken this repository's CI once:
+
+- Every checkout in a workflow that uses the action sets `fetch-depth: 0`
+  ([`.github/workflows/action-gate.yml`](.github/workflows/action-gate.yml)). A shallow runner keeps one
+  commit in its object database, so `git show <rev>:<path>` fails there while passing in every real clone.
+  The action reads `protocol.json` from the working tree and states, in the job summary, which store it
+  read the *base revision* from: `history source: object-db` (full history), `shallow`, or `no-object-db`.
+  `tests/test_action_f8.py` refuses both a workflow that drops the flag and a summary that stops saying it.
+- The committed badge in `.github/badges/` is produced by the same code path the action runs, and it
+  records the protocol digest it was minted under. A re-freeze makes it stale and the test that reads it
+  goes red rather than leaving a green pill describing a superseded protocol.
+
+  It is minted in the **corpus-absent** lane, because that is the lane a clone and the runner see: the
+  corpus is not in the repository, so a stranger reproduces `re-derived 612 /612 values`, and the badge
+  carries that number and `corpus absent` in its tooltip and in `autorate-badge.json`. On a machine that
+  also holds the corpus the same command prints `re-derived 649 /649 values` and writes a badge saying
+  `corpus present`; both are the tool's own labels, and the badge never carries one lane's number under
+  the other lane's name. Re-mint the committed one from an export:
+
+  ```bash
+  exp=$(mktemp -d) && git archive HEAD | tar -x -C "$exp"
+  (cd "$exp" && uv run --frozen python3 tools/action_report.py --path protocol.json --badge-dir badges)
+  cp "$exp"/badges/autorate-badge.svg "$exp"/badges/autorate-badge.json .github/badges/
+  ```
+
+## Reproduction
+
+`docs/RESULTS.md` is generated, never typed: `autorate report` emits it and `autorate
+verify` re-derives every value in it and prints `re-derived N /N values`. It carries no
+timestamp, so a re-run is byte-comparable, and it quotes no path or machine name, so a
+machine without the corpus reproduces the same bytes -- which is what lets CI run the
+gate at all.
+
+`docs/REPORT.md` is the same kind of file for a reader who wants one document instead of
+seven: `autorate report --full` emits it, `verify` re-derives it byte for byte under the same
+rule, and every figure in it is read out of an artefact rather than typed into a sentence --
+the certificate shape from `certificate.py`, the five gates from `protocol.json`'s `claims`
+array, each refusal from the artefact that refused, the related work from the rows of
+`docs/VERIFICATION.md` that are `verified`. The flag is additive on purpose: `autorate report`
+with no flag still writes `docs/RESULTS.md`, byte for byte the file this gate has always
+compared.
+
+That fallback has a stated limit, and `verify` prints it: with the corpus absent, the
+upstream `SOURCE.json` digests and the inventory snapshot are transcribed from
+`data/inventory.md.json`, not recomputed. The snapshot is policed from the environment
+that can see the corpus, where a stale key goes red.
+
+Those two environments are not two speeds of the same gate, and the denominators `verify`
+prints say so: `re-derived 612 /612 values` where the corpus is absent, `re-derived 649 /649
+values` where it is present. Both exit 0, both are green, and they are not the same claim.
+
+The corpus is not a committed file. It is the sibling `laya*/research/benchmarks/feishu_zh`
+checkout, which lives outside this repository, so a runner that checks out this repository
+alone can never reach the 649: every CI run is a 612 run, on both of its arms. Green CI is
+therefore evidence about the transcribed lane and nothing else.
+
+A 612 run proves a lot inside that lane. Every published number that is a function of a
+committed input re-derives there -- all seven published certificates, each from the committed
+records it was actually made from, G4's slice run from the four committed MASSIVE record files
+it names inside itself, the whole
+of `docs/RESULTS.md` byte for byte, `data/inventory.md` byte for byte, `docs/AUDIT.md` byte
+for byte, `docs/SEVERITY.md` byte for byte, `docs/RESPLIT.md` byte for byte, `docs/SLICES.md` byte
+for byte, `docs/CONTAMINATION.md` byte for byte, `docs/REPORT.md` byte for byte, and the MASSIVE rows
+from their committed pin. What a 612 run
+cannot prove is what its own disclosure says in so many words --
+`this run cannot see whether that snapshot went stale`. The gap is exactly 37
+checks: the 9 upstream `SOURCE.json` digests, the 26 snapshot keys a corpus-present run
+recomputes instead of reading back. On a machine without the archive those 35 do not run at
+all; the figures they would test exist there only as the transcription, so a snapshot that
+had drifted from the corpus would agree with itself and report 612 /612.
+
+So the corpus-present run is not a slower, fussier version of the CI job; it is the only one
+in which the transcription has an owner, and it is a local obligation, run before a push,
+because no runner can run it for you.
+
+## The five criteria, and what this tree can show for each
+
+v1 is defined by five criteria, G1–G5. `autorate verify` prints one line per criterion with a status that
+can say no -- `not_certifiable`, `measured_not_bounded`, `not_measured` -- read from `protocol.json`, the
+frozen protocol document, which is emitted by a command and never typed. The statuses below are what that
+command prints at this revision; the document is their owner, so this section is a reading of it and not a
+second place to keep them.
+
+- **G1, at least three decision tasks certified at `autorate(α = 0.05)` with a 200-resplit empirical
+  violation rate: `not_certifiable`.** Six decision tasks have been scored and their records committed;
+  zero of the six yield a certificate, and the refusals are published as artefacts rather than rounded
+  away. The binding quantity is the evaluation half's case count: the exact one-sided bound on a small
+  number of automated cases sits above a 5% budget no matter how few are wrong. The criterion's second
+  column is now measured and published rather than promised: `tools/resplit_experiment.py` re-splits every
+  scored task 200 times at the rubric this freeze owns, and `data/resplit_200.json` with `docs/RESPLIT.md`
+  carries what came out. All six published tasks certify in **0 of 200** splits, so their empirical violation
+  rate is recorded as undefined with a zero denominator instead of being imputed as `0.000`; the one row that
+  does produce a rate is the synthetic control (200 of 200 splits certify, rate 0.000 over 200 certificates),
+  which tests the machinery and is labelled as nothing else. `verify` compares the artefact's own arithmetic —
+  certificates plus refusals per task, the rate against its denominator, the rows against the splits — and the
+  rubric it recorded against the freeze's, in both lanes. The row is still not close to met: the second column
+  exists, the first is zero. What replaces the missing certificate is the ceiling, published rather than
+  asserted: `tools/g1_feasibility.py` computes, for each pool size this checkout can actually build, how wide
+  the exact one-sided bound is and what error rate would therefore be certifiable at `α = 0.05`, and
+  `verify` recomputes every figure in both lanes from `src/autorate/tails.py` and the published certificates.
+  So `not_certifiable` is now a measured claim about n and bound width, not an absence of an answer -- which
+  is the outcome spec §14 names as a result ("bound width is a first-class output; 'the bound is useless
+  here' *is* a certification result"). §14's kill criterion for this row is not evaluable rather than
+  fired: its condition compares a worst-group certified `autorate` with 0.30, and with nothing certified there
+  is no certified `autorate` to put in that comparison -- the same refusal that keeps the violation rate
+  undefined. Both halves of the pivot are published now: the audit multiple G2 measures, and the
+  contamination gap section 9 defines, frozen as `data/contamination_probe.json` with its interval and its
+  `claim = False` (see the G4 bullet). So the headline this project ships is the audit multiple G2
+  measures and the worst-slice refusal G4 names. Read the G1 row as unmet; that is what it says.
+- **G2, the reference README's gate demonstrably violating its implied error rate: `measured_not_bounded`.**
+  `autorate audit` measures the gate upstream instructs and `docs/AUDIT.md` publishes the multiplier, but the
+  sample cannot bound the claim: on the 2 cases one backend automates, one is wrong, and the exact one-sided
+  bound is 0.975. The point estimate misses upward; what the corpus says is that it cannot carry the claim.
+- **G3, at least one ordinal guarantee multiclass conformal prediction cannot express: `not_measured`.** The
+  one-sided severity certificate (`P(Y > â(X)) ≤ α`, three canonical bounds, its own refusal reasons) is a
+  command, an artefact and a gate row now: `autorate severity` writes `data/cert_severity_ordinal.json` from
+  the committed ordinal population, `docs/SEVERITY.md` renders it from those bytes, and `verify` re-derives
+  every published quantity of it in both lanes. The status does not move, because the criterion is a contrast
+  and not a certificate. A first measured contrast is committed (`tests/test_g3_contrast.py`): on the published
+  ordinal population, at the rubric this freeze owns, this repository's ordinal set gate refuses --
+  `empty_feasible_set`, publishing no `qhat`, no automation rate and no error bound of any kind -- while the
+  severity gate certifies the one-sided bound, reports its measured held-out violation rate and states the
+  rule's price; the set gate refuses on all four population variants that file walks. That is silence in a
+  named direction, and it is not the criterion -- because the criterion's own wording lost. On the corner of
+  the generator where the set gate does certify (`shift = 0`, `error_rate = 0`; 4 of the 64 knob cells
+  searched), its interval's top endpoint sits above the gold level in none of the 600 evaluated records
+  (200 cases), so
+  `P(Y > hi(X)) = 0` lies inside that certificate's own marginal bound of 0.014867 -- computed by
+  `tests/test_g3_contrast.py`'s builder in memory and written to no artefact, which is why the digit names its
+  producer instead of calling itself published: a coverage claim
+  over ordered intervals *does* yield a one-sided bound, and "multiclass conformal prediction cannot express
+  one" is false for that construction. The difference that survives is narrower, and it is measured rather
+  than asserted: taking that rule costs reporting above the model's own modal level on a large share of cases
+  and leaves 36 evaluated records with an empty certified set and so with no top endpoint at all, while the set
+  certificate publishes no price, no override rate and no named rule -- the severity artefact certifies a rule
+  and states both quantities beside its bound, at the same `delta`. The status stays `not_measured` because the
+  criterion is the wider sentence, and the narrowing lives in the frozen document that carries it
+  (`protocol.json`'s G3 note, at this digest) rather than in a re-worded criterion: retiring the wider wording
+  is a decision with a new freeze, and this section reads its statuses off that document instead of keeping a
+  second copy of them.
+- **G4, worst-slice certification refusing rather than averaging: `not_met`.** The behaviour is published now
+  rather than printed and discarded: `tools/language_slices.py` certifies the four committed MASSIVE record
+  files (11,896 cases) as slices of `slices.language` -- the key the producer declares, not a key invented for
+  the run -- and `verify` recomputes that run from the same bytes, 92 field comparisons plus 5 rubric
+  rows, in
+  both lanes. What it publishes is a refusal: 4 slices, 0 certified, each slice refused at
+  the corrected budget (alpha and delta 0.05 / 4 = 0.0125, joint level 0.85), and `n_min_violations` empty --
+  so this is a bound failure, not the size floor. The pooled marginal certificate over the same records is
+  published beside the slices and refuses too, which is the "rather than averaging" half: nothing in the
+  artefact lets the pooled number stand in for the worst slice. Two things keep the row short of met: it is
+  per-language only, because `option_count` is 20 for every committed MASSIVE row -- Khmer included, measured
+  across all 11,896 rows -- so the (language x option-count) cross the criterion names would be the same four
+  slices; and the ranking the criterion's word `worst` asks for cannot be published from a run where
+  nothing certifies: `run.worst_claim.worst.group` is `de`, but that is a tie-break, not a measurement -- its
+  `run.worst_claim.metric` is `autorate`, and `autorate` is null in all four rows, so the artefact names a
+  slice without demonstrating that it is the worst one. The slice the criterion names by example, Khmer, is no longer a gap:
+  `data/records_massive_km.jsonl` commits 2,974 per-case rows and this tree measures it, refusing like the
+  other three at the same rubric. Upstream's `accuracy 0.000 at mean confidence 0.952` remains upstream's own
+  100-case read, which these 1,487 evaluated Khmer cases neither replicate nor contradict. What the run does
+  say is the criterion's own word:
+  every refused slice's summary carries `NOT_CERTIFIABLE: reason=empty_feasible_set`, and the command over
+  these records exits 3, `EXIT_NOT_CERTIFIABLE`. The criterion's second named output is published beside
+  that run: `tools/contamination_probe.py` pools `data/pin_app_benchmark.json` -- upstream's own per-suite
+  `in_training` declarations, 27 usable rows and 3 excluded by name because their rounded rate is not a whole
+  number of cases -- and publishes `delta_hat` = -0.235208 over [-0.262646, -0.207502], so the in-training
+  stratum errs less and the interval is clear of zero. `verify` re-pools it (29 quantities plus 4 budget
+  values in both lanes, and re-digests the upstream file where the corpus is visible), and the artefact's
+  `claim` is false: the strata are different suites, so task difficulty is confounded with the flag and a gap
+  between aggregate rates is a diagnostic, not a proof of leakage. The four marginal certificates, read from their own `detail`, are the at-a-glance version of
+  the same evidence: `de` automates 0 of 1,487 cases (no singleton at that coverage, so nothing to bound);
+  `en` automates 709 with 43 wrong, upper bound 0.078; `km` 11 with 5 wrong, 0.729; `sw` 39 with 13 wrong,
+  0.477. Every non-empty slice is above the 0.050 budget, and none of the four certifies.
+- **G5, reproducible from a commit with no GPU, no network, no model: `certified`, and the proof is a
+  committed transcript.**
+  `docs/G5.md` is the record: `tools/g5_export.sh` unpacks a `git archive` of commit `9e6363a` into a scratch
+  directory with its own HOME and runs `autorate verify --protocol protocol.json` there --
+  `249/249` values with the corpus absent at rc=0 and `284/284` with
+  it present at rc=0, and four planted defects that each come back inside the documented
+  exit-code contract (rc=2, 4, 4, 4) and are restored byte-identically with the restore
+  checked. `tests/test_g5_transcript.py` refuses the file if a recorded code escapes the contract, if a
+  lane's two halves disagree, if the digest printed is not the digest of the commit the transcript names,
+  if the sentence counting the plants disagrees with the number of plant blocks, or if a single line still
+  names a machine path. Both CI jobs run the `--protocol` form on every push, so each tip gets its own
+  platform witness of the frozen comparison; `docs/G5.md` states that workflow fact without quoting a run
+  id, because a run belongs to the push that produced it and the transcript names a commit instead.
+  Three limits stay stated: the archive lane is a local obligation, the
+  no-network claim covers the gate steps and not the dependency install, and the drift and contamination
+  figures are unpublished, so they are absent from both denominators instead of re-derived.
+
+The honest summary is that the gate's reproduction story is the strongest part of this tree today. G4's
+slice criterion now has published evidence and is short of its own wording, not short of an artefact; the
+ordinal criterion is the one whose engine, evidence and wording disagree in a measured way; and G1 remains
+the criterion this corpus cannot meet at all.
+
+## What is here, and what is not
+
+The first paired certificate pair is published in `docs/RESULTS.md`: on the archived
+Chinese workplace corpus **both backends refuse** at alpha = beta = 0.05, for different
+reasons. One automates 2 of its 32 evaluation cases and one of those is wrong, so its case
+bound is 0.975. The other automates all 32 with one wrong, and the exact case bound is 0.140
+-- above a 0.05 budget. Even had that one request been right, a zero-error run over n=32
+would have bounded at 0.089, so the corpus cannot support a 5% claim at any error count.
+
+That is the result, not a shortfall: the corpus holds 64 independent cases, and the
+project's own feasibility table -- computed as a scratch probe before any certificate was written, and now a
+published artefact, `tools/g1_feasibility.py` writing `data/g1_feasibility.json` and `docs/FEASIBILITY.md`
+with every figure recomputed by `verify` in both lanes -- said a 5% claim was out of reach here.
+A tool that certified anyway would have been selling the exact thing this one
+refuses to sell. Both evaluation halves cover the same 32 cases, so the pair is compared
+case by case on identical inputs.
+
+Covered by code and by tests: the `choice` primitive, marginal two-sided control, APS and LAC set sizes,
+Clopper-Pearson one-sided bounds, and the gate that turns a certificate into
+`auto` / `escalate` / `abstain`.
+
+Written and tested, but not yet published as evidence: the threshold certifier (`certify_threshold.py`),
+drift-adaptive conformal (`drift.py`), and the contamination stratification (`contamination.py`). Each of
+them is a library with unit tests and no committed artefact that `verify` re-derives, which is the
+difference between "implemented" and "certified" in this project -- see the criteria above. The
+group-conditional (Mondrian) surface and its simultaneous per-slice correction
+(`certify_choice.certify_groups`, `correction.py`) left that list when `tools/language_slices.py` published
+`data/slices_language.json` and `docs/SLICES.md`: those are the artefacts the G4 row above is about, and
+`verify` recomputes the run from the committed records field by field.
+
+Published, re-derived, and still not the evidence its criterion asks for: the ordinal one-sided severity
+certificate (`severity.py`), reachable as `autorate severity`, committed as `data/cert_severity_ordinal.json`
+over `data/records_ordinal_synthetic.jsonl`, rendered into `docs/SEVERITY.md` and re-derived by `verify` in
+both lanes. `claim G3` prints `not_measured` beside an artefact that exists, because the criterion is the
+wider sentence -- "multiclass conformal prediction *cannot express* a one-sided ordinal guarantee" -- and the
+control that would establish the contrast has now been run and did not go that way: at the corner of the
+generator where the set gate certifies at all, its interval already yields the bound. What survives is
+narrower and measured, and the narrowing lives in the frozen G3 note rather than in a re-worded criterion. Its
+population is synthetic ordinal data with known answers: no committed records file in this checkout carries a
+`qtype="score"` record, and the artefact's own envelope says so, so nothing here is a measurement of a shipped
+model's ordinal risk.
+
+Not implemented at all: per-ask `noul` budgets. The `noul` question type is read, counted and quarantined
+by the same loaders as the others, and there is no certifier for it.
+
+`docs/superpowers/specs/` states the design; `docs/superpowers/plans/` states the order.
+
+Read `docs/ERRATA.md` before quoting any number from this project's history, and
+`docs/VERIFICATION.md` for which external claims were checked against sources and which
+were withdrawn.