greencheck 0.3.0__tar.gz

greencheck-0.3.0/CITATION.cff

@@ -0,0 +1,22 @@
+cff-version: 1.2.0
+message: "If you use this software or the case-study data, please cite it as below."
+title: "greencheck: auditing self-reported instruments in long-running agents"
+abstract: >-
+  A dependency-free checker for one rule: a number is not a measurement until it
+  has been observed to take different values on inputs that are known to differ.
+  Ships a taxonomy of six ways a self-reported metric can be correct code and
+  still measure nothing, and a case study over six instruments of an autonomous
+  agent that ran continuously for roughly two months.
+authors:
+  - name: Simon
+version: 0.2.0
+date-released: 2026-09-22
+license: MIT
+type: software
+keywords:
+  - autonomous-agents
+  - self-report
+  - observability
+  - falsification
+  - metrics
+  - reproducibility

greencheck-0.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Simon
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

greencheck-0.3.0/MANIFEST.in

@@ -0,0 +1,20 @@
+include README.md
+include LICENSE
+include CITATION.cff
+include pyproject.toml
+
+# The skills ship with the source tree rather than inside the package: they are
+# meant to be read as files (and browsed on GitHub), not imported. `greencheck
+# skills` locates this directory relative to the package, so a source install
+# finds them; a bare `pip install greencheck` from a wheel would not.
+recursive-include skills *.md
+recursive-include data *.json *.jsonl *.md
+recursive-include docs *.md
+recursive-include case_study *.py *.md *.json
+recursive-include tests *.py
+recursive-include tools *.py *.example.json *.example.txt
+
+# Never ship these, whatever else changes.
+exclude tools/*.local.json
+exclude tools/*.local.txt
+prune .git

greencheck-0.3.0/PKG-INFO

@@ -0,0 +1,379 @@
+Metadata-Version: 2.4
+Name: greencheck
+Version: 0.3.0
+Summary: Your validator passes inputs it should reject. greencheck finds them.
+Author: Simon
+License: MIT
+Project-URL: Homepage, https://github.com/simin-yuan/greencheck
+Project-URL: Repository, https://github.com/simin-yuan/greencheck
+Project-URL: Issues, https://github.com/simin-yuan/greencheck/issues
+Project-URL: Changelog, https://github.com/simin-yuan/greencheck/releases
+Keywords: mutation-testing,observability,metrics,agent,monitoring,evaluation,verification,test,testing,audit
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: System :: Monitoring
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Dynamic: license-file
+
+# greencheck
+
+[![tests](https://github.com/simin-yuan/greencheck/actions/workflows/tests.yml/badge.svg)](https://github.com/simin-yuan/greencheck/actions/workflows/tests.yml)
+[![licence: MIT](https://img.shields.io/badge/licence-MIT-blue.svg)](LICENSE)
+[![dependencies: none](https://img.shields.io/badge/dependencies-none-brightgreen.svg)](#)
+
+### Your validator passes inputs it should reject. This finds them.
+
+```console
+$ pip install https://github.com/simin-yuan/greencheck/releases/download/v0.3.0/greencheck-0.3.0-py3-none-any.whl
+```
+
+Then, from a checkout of this repository (both halves run against files that
+ship with it, so a reader can reproduce every line below):
+
+```console
+$ python -m greencheck.cli mutate --gate "python examples/mutate-demo/gate.py {target}/config.json" --target examples/mutate-demo/input
+```
+
+```
+baseline rc: 0  PASS (baseline is clean, mutating)
+mutants    : 12
+----------------------------------------------------------------------
+      caught  drop-file:config.json
+      caught  empty-file:config.json
+      caught  drop-line:config.json:4:"replicas": 3,
+      caught  blank-value:config.json:2:"service":
+      caught  blank-value:config.json:3:"region":
+   * ESCAPED  blank-value:config.json:4:"replicas":
+      caught  blank-value:config.json:5:"owner":
+----------------------------------------------------------------------
+caught 11 / 12
+
+mutants the gate let through (1):
+  *  blank-value:config.json:4:"replicas":
+```
+
+The escaped mutant turns `"replicas": 3` into `"replicas": 0` — a service that
+never starts. The gate checks that every key is present and every value is
+non-empty. Zero is neither missing nor empty, so it says yes.
+
+**Every structural property the gate tests for still holds.** That is exactly
+why this class of gap survives review: it is invisible to anything that only
+looks at shape. → [the runnable example](examples/mutate-demo/)
+
+## It also audits metrics
+
+Point it at a metric you already collect — a self-score, a recall rate, a health
+probe — and it tells you whether the number varies with the thing it is named
+after, or only with something adjacent: input existence, wall-clock freshness, a
+single boolean.
+
+```console
+$ python -m greencheck.cli audit data/ledger_metric_samples.jsonl --field identity.identity_score
+```
+
+The same question, asked of numbers instead of gates: *has this ever been
+observed to take a different value on input that differs?* If it has not, it is
+not a measurement. It is a decoration that updates on a schedule.
+
+---
+
+## The one rule
+
+> **A number is not a measurement until it has been observed to take different
+> values on inputs that are known to differ.**
+
+A metric that reads `1.0` forever is not telling you things are perfect. It is
+not telling you anything. A dashboard that is green because it has never been
+asked a question it could fail is a specific kind of silence — and this tool
+exists to make that silence audible.
+
+Point it at a metric you already collect: a self-score, a recall rate, a guard,
+a health probe. It tells you whether the number varies with the thing it is
+named after, or only with something adjacent — input existence, wall-clock
+freshness, a single boolean.
+
+---
+
+## 30 seconds
+
+```console
+$ python -m greencheck.cli demo
+
+[ok  ] content_length          (2 samples)
+[FAIL] content_presence_score  (2 samples)
+       - IDENTITY: 0/1 probe pairs separated within the claimed dimension
+         'content'. The instrument responds to something other than what it
+         claims to measure.
+[FAIL] always_one (control)    (2 samples)
+```
+
+The middle line is the point. `content_presence_score` **does** separate one of
+the probe pairs — `1.0` for text, `0.0` for empty. A naive check ("did any two
+outputs differ?") passes it. But the pair it separates differs in *existence*,
+not in *content*, and `content` is what the instrument is named after. A
+discriminability test that is not dimension-scoped is itself a decoration.
+
+Audit a real ledger (this one ships in `data/`):
+
+```console
+$ python -m greencheck.cli audit data/ledger_metric_samples.jsonl --field identity.identity_score
+
+[FAIL] identity.identity_score  (84 samples)
+       - CONSTANT: 84/84 samples report the identical value 1.0.
+         Zero variance over the observed window.
+       stats: n_samples=84, distinct_values=1
+```
+
+Check whether an assertion has ever fired. This reads the per-day aggregate
+log in `data/` — the same file the case study table is computed from, so the
+figure is yours to re-add:
+
+```console
+$ python -m greencheck.cli gate data/gate_daily.jsonl --fire-event block_issued
+
+[FAIL] data/gate_daily.jsonl — 13 full days with zero firings  (27097 samples)
+       - DEAD_GATE: 27,097 events recorded across 13 days, of which 25,594 were
+         the guard reporting itself healthy, and the denial path fired 0 times.
+         The assertion has never been observed to fire. Its existence is
+         documented; its behaviour is not.
+       stats: total_events=27097, health_events=25594, firings=0, silent_days_before_first_fire=13
+
+[ok  ] data/gate_daily.jsonl — full window (16 days)  (33228 samples)
+       - PASS: The path fired 5 times, the first on 2026-09-19. That first
+         firing came from a positive control written specifically to exercise
+         the path, not from ordinary traffic.
+       stats: total_events=33228, health_events=31642, firings=5
+```
+
+Both rows are the same guard. The code did not change between them; what
+changed is that somebody finally built it an input it was known to have to
+reject. Full window: 5 firings out of 33,228 events. Without that positive
+control, all 33,228 would have read as health.
+
+Zero dependencies. Python 3.9+. Nothing to install beyond the repo.
+
+---
+
+## What it catches
+
+| verdict | signature | how you notice |
+|---|---|---|
+| `CONSTANT` | Zero variance over the whole window | 84/84 samples report the same value |
+| `IDENTITY` | Separates inputs by existence, not by the claimed dimension | Empty vs non-empty moves it; good vs contradictory content does not |
+| `BARE_ZERO` | `n = 0` and `n > 0, 0 hits` produce the same output | 63 of 84 "recall" samples examined nothing and reported `0.0` |
+| `LIVENESS_BIT` | All variance traces to one boolean | A composite score whose two values are exactly a freshness flag |
+| `DEAD_GATE` | A guard exists, logs health, has never fired | 25,594 healthy events, 0 firings, 13 full days |
+| `NO_DATA` | Nothing matched | Reported instead of `PASS`, on purpose |
+
+**`NO_DATA` is the failure mode of the audit tool itself.** An empty input
+returning `PASS` is precisely the class of bug this repository exists to catch,
+so the tool refuses to commit it. And `IDENTITY` is not "the values differ
+somewhere" — discrimination is scoped to the dimension the instrument *claims*
+to measure, which is a strictly harder bar and the one that actually matters.
+
+None of these is hypothetical. Each was found on a real instrument, in
+production, in the case study below.
+
+---
+
+## The case study
+
+`data/` holds two anonymised datasets from a continuously-running autonomous
+agent over roughly two months. Everything in this repo is reproducible from
+them:
+
+```bash
+python case_study/run_case_study.py
+```
+
+| instrument | n | verdict |
+|---|---:|---|
+| `self_assessment.identity_score` | 84 | **CONSTANT** |
+| `self_assessment.memory_recall.recall_rate` | 84 | **CONSTANT; BARE_ZERO** |
+| `self_assessment.reflection_score` | 84 | **LIVENESS_BIT** |
+| `self_assessment.overall (composite)` | 84 | **LIVENESS_BIT** |
+| `guard denial path — 13 full days with zero firings` | 27,097 | **DEAD_GATE** |
+| `guard denial path — full window` | 33,228 | **PASS** |
+
+Six instruments. Five flagged. Full write-up:
+[`case_study/results/CASE_STUDY.md`](case_study/results/CASE_STUDY.md).
+
+The last two rows are **the same guard**. It reads `DEAD_GATE` for the thirteen
+days before anyone constructed an input it was *known* to have to block, and
+`PASS` afterwards. Nothing about the guard changed at that moment. What changed
+is that somebody finally asked it to prove itself.
+
+Every number above can be recomputed from the two files in `data/`. The headline
+is 27,097 and not a larger figure, because the larger one counted events with a
+timestamp earlier than the first firing — which needs event-level data, and only
+aggregates are published. A reader could not have checked it. Summing the
+zero-firing rows of [`data/gate_daily.jsonl`](data/gate_daily.jsonl) gives
+exactly 27,097, and that is the number used everywhere in this repository.
+
+That is the whole thesis. A control is not validated by its presence in the
+code, nor by its green light. It is validated by having been observed to fire.
+
+### The same loop, on a validator nobody here wrote
+
+[`case_study/third_party/`](case_study/third_party/) points it at third-party
+artifacts instead: a real `package.json` from a public project, checked by
+SchemaStore's official schema. 168 mutants, 36 caught, 132 let through — and
+then the part that decides anything: `npm` refuses two of the escapes and
+accepts a third. The count is not the finding. The triage is.
+
+---
+
+## Skills
+
+The CLI audits a ledger **after** the fact. `greencheck/skills/` is for **before** —
+agent-readable instructions that keep the broken instrument from being built at
+all.
+
+```console
+$ python -m greencheck.cli skills
+greencheck skills — 5 available
+
+  bare-zero                  Use when a metric, counter, rate or score reports zero...
+  dead-check                 Use when writing or reviewing a monitoring rule...
+  dimension-scope            Use when testing whether an instrument measures what its name claims...
+  measurement-or-decoration  Use when a system reports a score, confidence, health value about itself...
+  positive-control           Use when writing a guard, assertion, test, alert or validation rule...
+
+Read one in full:   python -m greencheck.cli skills <name>
+```
+
+Plain `SKILL.md` files with YAML frontmatter — the format used by Claude Code,
+Codex, OpenClaw and most agent harnesses. Copy one where your harness looks for
+skills, or let your agent read it directly:
+
+```console
+$ python -m greencheck.cli skills positive-control   # prints the whole file
+```
+
+| skill | use when |
+|---|---|
+| [`measurement-or-decoration`](greencheck/skills/measurement-or-decoration/SKILL.md) | the system reports a score about itself. The root rule; the others are special cases. |
+| [`positive-control`](greencheck/skills/positive-control/SKILL.md) | writing a guard, assertion or alert that is supposed to reject bad input |
+| [`dead-check`](greencheck/skills/dead-check/SKILL.md) | writing a monitor. Covers rules that can never fire, and rules with inverted polarity. |
+| [`dimension-scope`](greencheck/skills/dimension-scope/SKILL.md) | testing whether an instrument measures the dimension its name claims |
+| [`bare-zero`](greencheck/skills/bare-zero/SKILL.md) | a metric reports `0` |
+
+## Install / use
+
+No dependencies. Python 3.9+.
+
+From a checkout of this repository:
+
+```console
+$ python -m unittest discover -s tests      # 45 tests, stdlib only
+$ python -m greencheck.cli demo
+```
+
+Audit your own ledger:
+
+```bash
+python -m greencheck.cli audit my_ledger.jsonl \
+    --field identity.identity_score \
+    --count memory_recall.sampled \
+    --fresh reflection.fresh
+```
+
+Or drive it as a library:
+
+```python
+from greencheck import ProbePair, discriminate
+
+discriminate(my_metric, [
+    ProbePair("rich vs degenerate", rich_text, "aaaa", dimension="content"),
+    ProbePair("present vs absent", "hello", "", dimension="content"),
+], subject="my_metric", claimed_dimensions=["content"])
+```
+
+---
+
+## Where the evidence comes from
+
+The underlying logs are from production and contain filesystem paths, host
+identifiers and operator-specific strings. **They are not published.** What is
+published is aggregated:
+
+- `data/ledger_metric_samples.jsonl` — 84 samples, field names neutralised; no
+  value, timestamp or count altered
+- `data/gate_summary.json`, `data/gate_daily.jsonl` — per-day counts only
+
+`tools/anonymise.py` produces them and **fails the run** if any forbidden pattern
+survives. The leak gate has its own positive control (`--self-test`), because a
+checker that has never been observed to fire is not a checker — and that rule
+applies to this repository before it applies to yours. `tools/leakscan.py`
+re-verifies the entire tree independently. Rules and provenance:
+[`data/PROVENANCE.md`](data/PROVENANCE.md).
+
+---
+
+## Standing on
+
+This did not start from nothing, and it would be dishonest to imply otherwise:
+
+- **Dimension-scoped discriminability and falsification-first verification** —
+  adapted from [obra/superpowers](https://github.com/obra/superpowers) (MIT).
+- **`n = 1` case-study discipline** — report a single system honestly instead of
+  inflating it into a general claim.
+
+The strongest contribution in the other direction: if you have a metric that
+passes `greencheck` and still lies to you, that is a bug here, and the issue is
+welcome.
+
+---
+
+## Limits
+
+- **This tests instruments, not systems.** A `PASS` means the instrument
+  separates inputs within its claimed dimension. It does not mean the thing
+  measured is good, or that the dimension is the right one to measure.
+- **`n = 1` system.** One agent, roughly two months. The signatures recur across
+  its instruments, but cross-system replication is open. Treat the taxonomy as a
+  starting set, not a closed one.
+- **The tool only sees what you record.** A metric that is never written down
+  cannot be audited. Absence of records is itself a finding — a different one.
+- **A positive control is part of the procedure, not an optional extra.** A
+  guard's firing count is meaningless until someone has shown it fires when it
+  should. See [`docs/METHOD.md`](docs/METHOD.md).
+- **The instruments were written by the system being measured.** That is the
+  condition under study, not a flaw in the study.
+- **The judging standard is not set by the thing being judged.** The verdicts in
+  the case study are proposed by this tool's author and should be read as a
+  draft. Disagreeing with one of them is the useful move.
+
+## Falsifiers
+
+1. If any instrument in the case study is shown to separate inputs within the
+   dimension it claims to measure, the corresponding finding is false.
+2. If the guard is shown to have fired before `2026-09-19T14:52:23Z` on an input
+   that should have been blocked, the `DEAD_GATE` finding is false.
+3. If anonymisation changed any *value* (as opposed to a field name), every
+   number in the case study is void. `tools/anonymise.py` is the only transform
+   applied; its leak check is mandatory.
+
+## Citation
+
+See [`CITATION.cff`](CITATION.cff).
+
+## License
+
+MIT — see [`LICENSE`](LICENSE).
+
+**Simon**, 2026.