nullcov 0.1.0__tar.gz

nullcov-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,61 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.11", "3.12", "3.13"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install
+        run: pip install -e ".[dev]"
+
+      # nullcov gates its own suite. An unexercised declared layer fails the
+      # build, which is the behaviour the library exists to produce.
+      - name: Test
+        run: pytest --nullcov-strict --nullcov-json nullcov-report.json
+
+      - name: Upload coverage report
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: nullcov-report-py${{ matrix.python-version }}
+          path: nullcov-report.json
+          if-no-files-found: ignore
+
+  example:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install
+        run: pip install -e ".[dev]"
+
+      # The example is deliberately incomplete: it has an untested escalation
+      # layer and a real boundary bug behind it. Strict mode is expected to
+      # fail here, so a passing run would mean the gate stopped working.
+      - name: Example demonstrates the gate
+        working-directory: examples/refund_agent
+        run: |
+          if pytest --nullcov-strict --nullcov-incidents incidents.json; then
+            echo "::error::The example was expected to fail the nullcov gate."
+            exit 1
+          fi
+          echo "Gate correctly rejected the run."

nullcov-0.1.0/.gitignore

@@ -0,0 +1,16 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+build/
+dist/
+.venv/
+venv/
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.coverage
+htmlcov/
+nullcov-report.json
+.DS_Store
+.idea/
+.vscode/

nullcov-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Shay
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

nullcov-0.1.0/PKG-INFO

@@ -0,0 +1,270 @@
+Metadata-Version: 2.5
+Name: nullcov
+Version: 0.1.0
+Summary: Honest coverage for agent scaffolds: unexercised layers report unknown, never 100%.
+License: MIT License
+        
+        Copyright (c) 2026 Shay
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+License-File: LICENSE
+Keywords: agents,coverage,evaluation,llm,pytest,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Framework :: Pytest
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Requires-Dist: pytest>=8.0
+Provides-Extra: dev
+Requires-Dist: langgraph>=0.2; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Provides-Extra: langgraph
+Requires-Dist: langgraph>=0.2; extra == 'langgraph'
+Description-Content-Type: text/markdown
+
+# nullcov
+
+**Your agent test suite reports 94%. This tells you which part of that number is a lie.**
+
+Conventional coverage divides passing cases by total cases. A layer of your agent
+that nobody wrote a case for contributes nothing to either side of that division —
+so it vanishes from the summary entirely. The suite looks strongest exactly where
+it is blindest.
+
+nullcov reports those layers as `unknown`. Never `100%`, never silently dropped.
+
+```
+============================= nullcov ==============================
+  [ok  ] routing                  3 slice(s), 41 case(s)
+  [ok  ] schema                   2 slice(s), 18 case(s)
+  [????] escalation               no cases -- coverage unknown, not 100%
+  [FAIL] safety                   1 slice(s), 6 case(s), 1 failing
+  [----] reasoning                generative -- not scored, by design
+
+  65 case(s) ran, 1 failing (98.5% of executed cases passed).
+  1 declared layer(s) have no coverage. The number above says nothing about them.
+```
+
+## Why this exists
+
+In a June 2026 survey of 157 enterprises (VentureBeat VB Pulse), **half had shipped
+an agent that passed their own evaluations and still failed in front of a customer.**
+Only 5% fully trusted the evaluations they were shipping on. When asked what was
+wrong with those evaluations, the most common answer was not "not enough coverage" —
+it was **poor alignment with real-world outcomes**.
+
+The same pattern shows up in the research. A production agent team
+([arXiv:2606.11686](https://arxiv.org/abs/2606.11686)) documented a confirmation-gate
+bug that caused roughly half of confirmed orders to silently not execute — while
+their aggregate quality metric barely moved. Under controlled fault injection,
+end-to-end pass rates dropped 1.7–5.9% while layer-isolated slices dropped 25–91%.
+
+One aggregate number hides a broken layer. That is the problem nullcov is built for.
+
+## Install
+
+```bash
+pip install nullcov
+```
+
+## Use
+
+**1. Declare your layers.** In `pyproject.toml` (or a `nullcov.toml`):
+
+```toml
+[tool.nullcov]
+layers = ["routing", "schema", "escalation", "safety"]
+```
+
+This declaration is the point. Without a statement of what *should* exist, an
+untested layer is indistinguishable from a layer that does not exist, and no tool
+can tell you the difference. Keeping it in version control means deleting a layer
+to make a report go green is a diff a reviewer sees — not a number that quietly
+improves.
+
+Layers that are genuinely generative are declared as such and are never scored:
+
+```toml
+[tool.nullcov.layers]
+routing = { description = "picks the next node" }
+reasoning = { deterministic = false }
+```
+
+**2. Attach tests to layers.**
+
+```python
+import nullcov
+
+@nullcov.case(layer="escalation", slice_id="high_value_requires_human")
+def test_high_value_transactions_go_to_a_human():
+    with nullcov.pure_mode():
+        result = run_scaffold(graph, {"amount": 1200})
+    assert result.next_node == "human_review"
+```
+
+**3. Run.**
+
+```bash
+pytest --nullcov
+```
+
+Add `--nullcov-strict` to make it a CI gate. An unexercised declared layer fails
+the run, because a suite that cannot speak to a layer has not verified it.
+
+## Pure mode
+
+`pure_mode()` blocks outbound network connections at the socket layer for the
+duration of a block. It is not a stub you have to remember to install — it holds
+regardless of which SDK the agent reaches for, and raises `PureModeViolation`
+naming the host if anything tries to escape.
+
+```python
+with nullcov.pure_mode() as report:
+    result = graph.invoke(state)
+assert report.clean
+```
+
+A test that passes under pure mode **provably** made zero model calls. Not by
+convention, not because someone remembered to patch the client — because nothing
+could reach the network. That makes it deterministic, free, and fast enough to run
+on every commit. Loopback stays open, so local fixtures and recorded-cassette
+servers still work.
+
+## Reality alignment
+
+Coverage tells you what you looked at. It cannot tell you whether you looked at the
+right thing. Feed nullcov your production incidents and it separates two failures
+that look identical on a dashboard:
+
+```bash
+pytest --nullcov --nullcov-incidents incidents.json
+```
+
+```
+reality alignment
+  29 attributed incident(s).
+
+  Blind spots -- production fails here and no case looks:
+    escalation: 11 incidents (38% of all attributed) and no coverage at all.
+    The suite cannot speak to this layer.
+
+  Misaligned -- cases pass here and production fails anyway:
+    routing: 9 incidents (31% of all attributed) despite 41 passing cases.
+    The cases pass and production still fails -- distrust these cases before
+    adding more.
+```
+
+**Blind spot**: no coverage. Write cases.
+
+**Misalignment**: full, passing coverage — and production fails there anyway. More
+cases of the same shape make this worse, not better. The existing cases are the
+thing to distrust. This is the finding that explains a green pipeline and a
+customer-visible outage on the same afternoon, and it is the reason this tool is
+not just another eval runner.
+
+`incidents.json` is a list of records — an export from your incident tracker, or a
+handful of postmortem entries:
+
+```json
+[
+  {"incident_id": "INC-4471", "layer": "escalation", "severity": "sev2",
+   "summary": "refund over threshold auto-approved"}
+]
+```
+
+Attribution is the one input nullcov cannot derive for itself, and the quality of
+everything above depends on it.
+
+## Structural coverage for LangGraph
+
+Everything above depends on a human typing a truthful layer name. Nothing stops
+that from drifting: add a node to the graph, forget to tag a test for it, and
+it is invisible everywhere -- not `[????]`, not undeclared, just absent.
+
+`nullcov.langgraph` removes the human from that specific step. It reads the
+topology straight off your **compiled** graph object -- the actual nodes and
+conditional branches LangGraph built, not a file you maintain by hand -- and
+tells you which of them any test run ever actually traversed.
+
+```python
+from nullcov.langgraph import CoverageAccumulator, GraphTopology
+
+graph = build_graph()  # your compiled LangGraph graph
+accumulator = CoverageAccumulator(topology=GraphTopology.extract(graph))
+
+# in your tests, run the graph through the accumulator instead of graph.invoke()
+accumulator.record_run(graph, {"amount": 50, "reason": "wrong size"})
+accumulator.record_run(graph, {"amount": 600, "reason": "wrong size"})
+
+coverage = accumulator.coverage()
+print(coverage.branch_rate())        # 0.5
+print(coverage.untaken_branches)     # {Branch("validate", "reject")}
+print(coverage.unvisited_nodes)      # {"reject"}
+```
+
+Run that against the example agent in this repo and it finds a real gap the
+manual layer system misses entirely: `validate()` has a passing unit test for
+an invalid payload, and `reject` is a fully-implemented node — but no test ever
+runs the **compiled graph** end-to-end with a bad input, so the
+`validate → reject` transition has never once fired in the test suite. A unit
+test on the routing function proves the function is correct. It proves nothing
+about whether the graph, as wired, ever takes that path.
+
+We checked before building this: neither AWS Bedrock AgentCore Evaluations nor
+`fasteval-langgraph` (Intuit) compute this. Both score individual test cases
+against a trajectory the developer writes by hand — a real capability, but a
+different one. Neither aggregates, across a whole suite, what fraction of the
+graph's actual branch topology was ever exercised. `nullcov.langgraph` is
+narrower than either of them and answers a question they don't ask.
+
+Requires `langgraph` (`pip install nullcov[langgraph]`). This module is
+LangGraph-specific by design — the ground truth it reads only exists because
+`CompiledGraph.get_graph()` exposes it. CrewAI and AutoGen would each need
+their own adapter reading their own internal representation; none exists yet.
+
+## What this does not do
+
+- **It does not score generative output.** LLM-as-judge is not statistically
+  calibrated — position, verbosity and self-enhancement bias are documented, and
+  calibrating judge confidence needs logprobs most providers do not expose. nullcov
+  declines to put a number on free-form reasoning rather than pretending.
+- **It does not prove your agent is correct.** It proves which contracts held and,
+  more usefully, which ones nobody checked.
+- **It does not replace your eval suite.** It tells you where that suite is blind.
+
+## Prior art
+
+nullcov leans on published work rather than inventing around it. The
+scaffold/generative split and the coverage-honesty criterion come from
+[arXiv:2606.11686](https://arxiv.org/abs/2606.11686). Deterministic record-and-replay
+of agent I/O is solved by [agrepl](https://arxiv.org/abs/2607.16200) and
+`vcr-langchain`; multi-agent tracing by
+[MAESTRO](https://github.com/sands-lab/maestro). Signed action receipts exist in
+`nobulex` and `protect-mcp`. If you need those, use those.
+
+What nullcov adds is the honesty constraint on the aggregate, and the alignment
+pass against production.
+
+## License
+
+MIT

nullcov-0.1.0/README.md

@@ -0,0 +1,226 @@
+# nullcov
+
+**Your agent test suite reports 94%. This tells you which part of that number is a lie.**
+
+Conventional coverage divides passing cases by total cases. A layer of your agent
+that nobody wrote a case for contributes nothing to either side of that division —
+so it vanishes from the summary entirely. The suite looks strongest exactly where
+it is blindest.
+
+nullcov reports those layers as `unknown`. Never `100%`, never silently dropped.
+
+```
+============================= nullcov ==============================
+  [ok  ] routing                  3 slice(s), 41 case(s)
+  [ok  ] schema                   2 slice(s), 18 case(s)
+  [????] escalation               no cases -- coverage unknown, not 100%
+  [FAIL] safety                   1 slice(s), 6 case(s), 1 failing
+  [----] reasoning                generative -- not scored, by design
+
+  65 case(s) ran, 1 failing (98.5% of executed cases passed).
+  1 declared layer(s) have no coverage. The number above says nothing about them.
+```
+
+## Why this exists
+
+In a June 2026 survey of 157 enterprises (VentureBeat VB Pulse), **half had shipped
+an agent that passed their own evaluations and still failed in front of a customer.**
+Only 5% fully trusted the evaluations they were shipping on. When asked what was
+wrong with those evaluations, the most common answer was not "not enough coverage" —
+it was **poor alignment with real-world outcomes**.
+
+The same pattern shows up in the research. A production agent team
+([arXiv:2606.11686](https://arxiv.org/abs/2606.11686)) documented a confirmation-gate
+bug that caused roughly half of confirmed orders to silently not execute — while
+their aggregate quality metric barely moved. Under controlled fault injection,
+end-to-end pass rates dropped 1.7–5.9% while layer-isolated slices dropped 25–91%.
+
+One aggregate number hides a broken layer. That is the problem nullcov is built for.
+
+## Install
+
+```bash
+pip install nullcov
+```
+
+## Use
+
+**1. Declare your layers.** In `pyproject.toml` (or a `nullcov.toml`):
+
+```toml
+[tool.nullcov]
+layers = ["routing", "schema", "escalation", "safety"]
+```
+
+This declaration is the point. Without a statement of what *should* exist, an
+untested layer is indistinguishable from a layer that does not exist, and no tool
+can tell you the difference. Keeping it in version control means deleting a layer
+to make a report go green is a diff a reviewer sees — not a number that quietly
+improves.
+
+Layers that are genuinely generative are declared as such and are never scored:
+
+```toml
+[tool.nullcov.layers]
+routing = { description = "picks the next node" }
+reasoning = { deterministic = false }
+```
+
+**2. Attach tests to layers.**
+
+```python
+import nullcov
+
+@nullcov.case(layer="escalation", slice_id="high_value_requires_human")
+def test_high_value_transactions_go_to_a_human():
+    with nullcov.pure_mode():
+        result = run_scaffold(graph, {"amount": 1200})
+    assert result.next_node == "human_review"
+```
+
+**3. Run.**
+
+```bash
+pytest --nullcov
+```
+
+Add `--nullcov-strict` to make it a CI gate. An unexercised declared layer fails
+the run, because a suite that cannot speak to a layer has not verified it.
+
+## Pure mode
+
+`pure_mode()` blocks outbound network connections at the socket layer for the
+duration of a block. It is not a stub you have to remember to install — it holds
+regardless of which SDK the agent reaches for, and raises `PureModeViolation`
+naming the host if anything tries to escape.
+
+```python
+with nullcov.pure_mode() as report:
+    result = graph.invoke(state)
+assert report.clean
+```
+
+A test that passes under pure mode **provably** made zero model calls. Not by
+convention, not because someone remembered to patch the client — because nothing
+could reach the network. That makes it deterministic, free, and fast enough to run
+on every commit. Loopback stays open, so local fixtures and recorded-cassette
+servers still work.
+
+## Reality alignment
+
+Coverage tells you what you looked at. It cannot tell you whether you looked at the
+right thing. Feed nullcov your production incidents and it separates two failures
+that look identical on a dashboard:
+
+```bash
+pytest --nullcov --nullcov-incidents incidents.json
+```
+
+```
+reality alignment
+  29 attributed incident(s).
+
+  Blind spots -- production fails here and no case looks:
+    escalation: 11 incidents (38% of all attributed) and no coverage at all.
+    The suite cannot speak to this layer.
+
+  Misaligned -- cases pass here and production fails anyway:
+    routing: 9 incidents (31% of all attributed) despite 41 passing cases.
+    The cases pass and production still fails -- distrust these cases before
+    adding more.
+```
+
+**Blind spot**: no coverage. Write cases.
+
+**Misalignment**: full, passing coverage — and production fails there anyway. More
+cases of the same shape make this worse, not better. The existing cases are the
+thing to distrust. This is the finding that explains a green pipeline and a
+customer-visible outage on the same afternoon, and it is the reason this tool is
+not just another eval runner.
+
+`incidents.json` is a list of records — an export from your incident tracker, or a
+handful of postmortem entries:
+
+```json
+[
+  {"incident_id": "INC-4471", "layer": "escalation", "severity": "sev2",
+   "summary": "refund over threshold auto-approved"}
+]
+```
+
+Attribution is the one input nullcov cannot derive for itself, and the quality of
+everything above depends on it.
+
+## Structural coverage for LangGraph
+
+Everything above depends on a human typing a truthful layer name. Nothing stops
+that from drifting: add a node to the graph, forget to tag a test for it, and
+it is invisible everywhere -- not `[????]`, not undeclared, just absent.
+
+`nullcov.langgraph` removes the human from that specific step. It reads the
+topology straight off your **compiled** graph object -- the actual nodes and
+conditional branches LangGraph built, not a file you maintain by hand -- and
+tells you which of them any test run ever actually traversed.
+
+```python
+from nullcov.langgraph import CoverageAccumulator, GraphTopology
+
+graph = build_graph()  # your compiled LangGraph graph
+accumulator = CoverageAccumulator(topology=GraphTopology.extract(graph))
+
+# in your tests, run the graph through the accumulator instead of graph.invoke()
+accumulator.record_run(graph, {"amount": 50, "reason": "wrong size"})
+accumulator.record_run(graph, {"amount": 600, "reason": "wrong size"})
+
+coverage = accumulator.coverage()
+print(coverage.branch_rate())        # 0.5
+print(coverage.untaken_branches)     # {Branch("validate", "reject")}
+print(coverage.unvisited_nodes)      # {"reject"}
+```
+
+Run that against the example agent in this repo and it finds a real gap the
+manual layer system misses entirely: `validate()` has a passing unit test for
+an invalid payload, and `reject` is a fully-implemented node — but no test ever
+runs the **compiled graph** end-to-end with a bad input, so the
+`validate → reject` transition has never once fired in the test suite. A unit
+test on the routing function proves the function is correct. It proves nothing
+about whether the graph, as wired, ever takes that path.
+
+We checked before building this: neither AWS Bedrock AgentCore Evaluations nor
+`fasteval-langgraph` (Intuit) compute this. Both score individual test cases
+against a trajectory the developer writes by hand — a real capability, but a
+different one. Neither aggregates, across a whole suite, what fraction of the
+graph's actual branch topology was ever exercised. `nullcov.langgraph` is
+narrower than either of them and answers a question they don't ask.
+
+Requires `langgraph` (`pip install nullcov[langgraph]`). This module is
+LangGraph-specific by design — the ground truth it reads only exists because
+`CompiledGraph.get_graph()` exposes it. CrewAI and AutoGen would each need
+their own adapter reading their own internal representation; none exists yet.
+
+## What this does not do
+
+- **It does not score generative output.** LLM-as-judge is not statistically
+  calibrated — position, verbosity and self-enhancement bias are documented, and
+  calibrating judge confidence needs logprobs most providers do not expose. nullcov
+  declines to put a number on free-form reasoning rather than pretending.
+- **It does not prove your agent is correct.** It proves which contracts held and,
+  more usefully, which ones nobody checked.
+- **It does not replace your eval suite.** It tells you where that suite is blind.
+
+## Prior art
+
+nullcov leans on published work rather than inventing around it. The
+scaffold/generative split and the coverage-honesty criterion come from
+[arXiv:2606.11686](https://arxiv.org/abs/2606.11686). Deterministic record-and-replay
+of agent I/O is solved by [agrepl](https://arxiv.org/abs/2607.16200) and
+`vcr-langchain`; multi-agent tracing by
+[MAESTRO](https://github.com/sands-lab/maestro). Signed action receipts exist in
+`nobulex` and `protect-mcp`. If you need those, use those.
+
+What nullcov adds is the honesty constraint on the aggregate, and the alignment
+pass against production.
+
+## License
+
+MIT

nullcov-0.1.0/examples/refund_agent/README.md

@@ -0,0 +1,82 @@
+# Example: a refund agent with a blind spot
+
+A small LangGraph refund agent, a test suite a competent team would plausibly
+write for it, and a real bug that the suite cannot see.
+
+## Run it
+
+```bash
+pytest --nullcov --nullcov-incidents incidents.json
+```
+
+```
+  [????] escalation               no cases -- coverage unknown, not 100%
+  [ok  ] routing                  1 slice(s), 2 case(s)
+  [ok  ] safety                   1 slice(s), 2 case(s)
+  [ok  ] schema                   2 slice(s), 5 case(s)
+  [----] summary                  generative -- not scored, by design
+
+  9 case(s) ran, 0 failing (100.0% of executed cases passed).
+  1 declared layer(s) have no coverage. The number above says nothing about them.
+
+reality alignment
+  6 attributed incident(s).
+
+  Blind spots -- production fails here and no case looks:
+    escalation: 4 incidents (67% of all attributed) and no coverage at all.
+    The suite cannot speak to this layer.
+
+  Misaligned -- cases pass here and production fails anyway:
+    routing: 1 incidents (17% of all attributed) despite 2 passing cases.
+    safety: 1 incidents (17% of all attributed) despite 2 passing cases.
+```
+
+Every test passes. A conventional runner would print `9 passed` and stop there.
+
+## The bug
+
+`decide()` in `agent.py` routes large refunds to a human:
+
+```python
+if amount > AUTO_APPROVE_LIMIT:   # should be >=
+    return {**state, "decision": "human_review"}
+```
+
+Policy says the limit itself requires approval. The comparison is strictly
+greater-than, so a refund of exactly £500 is auto-approved:
+
+```
+at limit (500.0)  -> auto_approve    <-- policy says human_review
+above (500.01)    -> human_review
+below (499.99)    -> auto_approve
+```
+
+An off-by-one on a boundary. End-to-end sampling almost never lands exactly on
+it, and an aggregate pass rate will not notice when it does.
+
+## What each half of the output tells you
+
+**The blind spot** is the escalation layer: four of six production incidents,
+and not one test case. Nobody decided that layer was safe — the person writing
+the suite covered what they thought of, and the boundary rule was not it. That
+is what `????` means, and why it is not `100%`.
+
+**The misalignments** are routing and safety: cases pass, production fails
+anyway. Those layers *are* tested. The tests are looking at the wrong thing.
+Writing more cases in the same shape will not help; the existing ones are what
+to distrust.
+
+Half of enterprises surveyed in June 2026 had shipped an agent that passed their
+own evaluations and still failed in front of a customer (VentureBeat VB Pulse,
+n=157). Misalignment is the name for that, and it does not show up on a
+coverage percentage.
+
+## Fix it
+
+Uncomment the last case in `test_refund_agent.py` and run again. The escalation
+layer goes from `????` to `FAIL` — the tool stops saying "I do not know" and
+starts saying "this is broken." Change `>` to `>=` in `agent.py` and it goes
+green.
+
+The point is the first transition, not the second. `????` to `FAIL` is a suite
+learning to see. A conventional report never leaves `100%`.