agent-reliability-ci 0.8.0__py3-none-any.whl

agent_reliability_ci-0.8.0.dist-info/METADATA

@@ -0,0 +1,506 @@
+Metadata-Version: 2.5
+Name: agent-reliability-ci
+Version: 0.8.0
+Summary: Regression testing for stochastic AI agents: frozen experiments, fault injection, replayable reduced failures.
+Project-URL: Homepage, https://github.com/ajaysurya1221/agent-reliability-ci
+Project-URL: Source, https://github.com/ajaysurya1221/agent-reliability-ci
+Project-URL: Changelog, https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/CHANGELOG.md
+Project-URL: Issues, https://github.com/ajaysurya1221/agent-reliability-ci/issues
+Project-URL: Design records, https://github.com/ajaysurya1221/agent-reliability-ci/tree/main/docs/design
+License-Expression: Apache-2.0
+License-File: LICENSE
+License-File: NOTICE
+Keywords: ai-agents,evaluation,fault-injection,reliability,reproducibility,statistics
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: POSIX
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: pydantic<3,>=2.7
+Requires-Dist: pyyaml<7,>=6
+Description-Content-Type: text/markdown
+
+# Did this change make your agent less reliable?
+
+ARCI (agent-reliability-ci) runs your baseline and candidate repeatedly under the same predeclared
+tasks and injected tool faults, grades each run with an independent success check, returns `PASS`,
+`BLOCK`, `INCONCLUSIVE` or `ERROR` with an exit code, and can replay a recorded failure offline.
+
+Try it without cloning (seeded fixture, no model call):
+
+```bash
+uvx --from agent-reliability-ci arci demo
+```
+
+Replace these example references with your project's importable Python functions:
+
+```bash
+uv tool install agent-reliability-ci && arci init checkout-retry --baseline app.v1:run --candidate app.v2:run --toolset tests.world:make_world --oracle tests.world:passed && arci run arci.yaml
+```
+
+`baseline + candidate → same predeclared tasks/faults → repeated trials → verdict → minimise/replay`
+
+**PASS means only this:** for this predeclared experiment, the evidence supports that any reliability
+drop from baseline to candidate is smaller than the allowed drop. It is **not** a claim of absolute
+reliability.
+
+A small first run will often be **INCONCLUSIVE**. That is a valid result; use `arci plan` before
+choosing a larger confirmatory run.
+
+ARCI observes calls that cross its tool boundary. It is not a sandbox, it does not replay model
+inference, and trace divergence shows where runs differ, not why.
+
+## Try the offline demo
+
+Python 3.11+, POSIX and [uv](https://docs.astral.sh/uv/). Installation needs network; the fixture makes
+no model calls.
+
+```bash
+uvx --from agent-reliability-ci arci demo
+```
+
+`arci demo` runs a packaged copy of the seeded inventory fixture (`arci.demo`): the same world, agents,
+seeds and sample size as the recorded run below, so it computes **192/200** baseline successes and
+**132/200** retry-removed successes fresh on your machine and says whether they matched the recorded
+run. It is a reproduction, not the recorded evidence. The six steps go through the real CLI and the
+Python API, each trial in a fresh process with a separate grader process. Runtime depends on the
+machine and current load (measured: 38 s on a 15-core Apple Silicon Mac with 8 workers; 72 s through
+`uvx` from a local wheel including environment creation; CI runners are slower). `just demo` runs the checkout script
+`examples/retry_agent/hero_demo.py`, the same six steps. Lines from the recorded run:
+
+```text
+=== 2. The frozen experiment, N=200 per arm, fault: tool_timeout on reserve
+  VERDICT: BLOCK (exit 1)   [A vs agent_b]
+  VERDICT: PASS (exit 0)   [A vs agent_a]
+
+=== 3. Where do a passing and a failing run part ways?
+  left: tool_start tool="reserve" arguments={"order_id":"order-7","quantity":2,"sku":"widget"}
+  right: tool_start tool="confirm" arguments={"order_id":"order-7","sku":"widget"}
+  after_injection: tool_timeout
+
+=== 4. Shrink the failing condition (3 injected faults)
+  kept: tool_timeout
+  removed: empty_result, tool_error_once
+  minimality: 1-minimal
+
+=== 5. Replay the reduced bundle offline
+  REPRODUCED: failure reproduced
+```
+
+Copied from the unedited, committed log
+[hero-demo-2026-10-05.txt](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/reports/ci-gate-2026-10-06/logs/hero-demo-2026-10-05.txt), without
+the `ok` lines. Step 1 passes both agents on one clean run; step 6 passes the repaired agent C on
+the exact reproducer and on its own frozen experiment; the run ends `All six steps held`, exit 0.
+The experiment is also a frozen acceptance test ([test_hero.py](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/tests/acceptance/test_hero.py)).
+
+Agent B is not rigged with dice. It is agent A with the retry around `reserve` removed, a plausible
+refactoring slip: when `reserve` times out it carries on to `confirm` and still reports success. Its
+failure rate comes from the environment. Reservations already exist with configured probability
+65% (135 of the default 200 seeds, plus 8 scenarios with a naturally flaky `confirm`; shares set in
+[world.py](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/src/arci/demo/world.py), counts asserted in
+[test_retry_example.py](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/tests/unit/examples/test_retry_example.py)), and in those the missing retry
+never matters. That is exactly why one run hides it.
+
+### Recorded result and evidence
+
+**Recorded fixture: 200 trials per arm; timeout on `reserve`.**
+Baseline: **192/200**. Retry removed: **132/200**. Verdict: **BLOCK (exit 1)**.
+[Recorded results](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/results/retry-demo-n200.md) · [Trace, reduction and replay](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/reports/ci-gate-2026-10-06/logs/hero-demo-2026-10-05.txt)
+
+**Evidence:** [18 archived decisions re-derived](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/reports/ci-gate-2026-10-06/REPORT.md) · [Frozen contracts](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/FROZEN.sha256) · [Design records](https://github.com/ajaysurya1221/agent-reliability-ci/tree/main/docs/design) · [CI and macOS statistics checks](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/.github/workflows/ci.yml)
+
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/ajaysurya1221/agent-reliability-ci/main/docs/assets/hero-dark.svg">
+  <img src="https://raw.githubusercontent.com/ajaysurya1221/agent-reliability-ci/main/docs/assets/hero-light.svg" alt="arci, agent-reliability-ci: Did this change make the agent less reliable, and what broke? A frozen experiment, an exact verdict with its uncertainty, and a failure case you can replay offline. Seeded retry demo, 200 trials per arm, timeout on the first reserve call: A 192/200, B 132/200; bounds on the difference minus 0.405 to minus 0.183 against a margin of minus 0.10; VERDICT BLOCK, exit 1; 3 injected faults reduced to 1, 1-minimal; offline replay REPRODUCED. B is A with one retry removed. Source: docs/results/retry-demo-n200.md.">
+</picture>
+
+Re-derived means each archived `decision.json` regenerates byte for byte from its committed
+inputs, whatever its verdict: reproducing a BLOCK, INCONCLUSIVE or ERROR counts, and no archived
+run is turned green ([reproduction recipe](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/reports/ci-gate-2026-10-06/reproduction/README.md),
+[CITATION.cff](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/CITATION.cff)).
+
+> **Review a retry-policy change before merging**
+>
+> In the recorded inventory-reservation fixture, the candidate loses its retry after a timeout. ARCI reports BLOCK, locates the trace divergence, reduces three injected faults to one, and reproduces the failure offline. The repaired candidate passes the reproducer and its own separately frozen experiment.
+>
+> Scaffold your own experiment with `arci init` (below), or start from the example’s manifest builder. Define the oracle, fault conditions, margin and sample size before running. A small experiment may remain INCONCLUSIVE.
+
+Manifest builder of the recorded run: [examples/retry_agent/experiment.py](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/examples/retry_agent/experiment.py).
+
+## Verdicts and exit codes
+
+Every gating command ends in a verdict and an exit code. Nothing else decides.
+
+| Verdict | Exit | Meaning |
+|---|---:|---|
+| `PASS` | 0 | The candidate is non-inferior to the baseline within `delta`. Nothing about absolute reliability. |
+| `BLOCK` | 1 | The candidate is worse by more than `delta`, or it broke a hard invariant. |
+| `INCONCLUSIVE` | 2 | Not enough evidence either way. CI stays non-green. No regression is claimed. |
+| `ERROR` | 3 | The experiment is invalid: a harness or grader fault, a missing, duplicated or foreign trial, a broken seal. It can never pass. |
+
+An agent crash, timeout, blown budget or uncaught tool fault is a `FAIL`. A broken environment,
+injector, grader or event sink is an `ERROR`, and it invalidates the experiment instead of quietly
+leaving the denominator. `arci replay` uses its own codes: 0 reproduced, 1 not reproduced, 3 invalid.
+
+## How it works
+
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/ajaysurya1221/agent-reliability-ci/main/docs/assets/where-dark.svg">
+  <img src="https://raw.githubusercontent.com/ajaysurya1221/agent-reliability-ci/main/docs/assets/where-light.svg" alt="Where arci sits. Before the run, a frozen manifest declares the baseline and candidate, oracle, faults, seeds, N and the decision rule. In CI, the recorded trials (seeded trials per arm, injected tool faults, sealed boundary records) feed the arci gate, which computes exact Clopper-Pearson bounds on the difference against a frozen margin and returns PASS, BLOCK, INCONCLUSIVE or ERROR with exit 0, 1, 2 or 3; merge on PASS, while BLOCK and ERROR fail the job and INCONCLUSIVE fails by default. A separate branch reads the same records for any verdict: trace comparison (the first divergent step between a passing and a failing trial), fault reduction (ddmin over injected faults, same failure fingerprint) and offline replay of the recorded boundary calls, not fresh model inference.">
+</picture>
+
+In words: the manifest (task, toolset, oracle, contract, both arms, fault conditions, `alpha`,
+`delta`, `n_per_arm`, seeds) is sealed before the run. Each trial runs as a Python worker or a
+command agent behind an MCP boundary, with a separate grader process; the tool boundary applies
+budgets and seeded faults and records every call into sealed `trials.jsonl`. The gate is a pure
+function of manifest and trials. Diagnosis reads the same records, whatever the verdict: `diff`
+(first divergence), `minimize` (ddmin over injected faults, same failure fingerprint), `bundle`
+and `replay` (re-serves recorded boundary calls: REPRODUCED, NOT_REPRODUCED or INVALID).
+
+A Python agent is a plain function, `agent(task, tools, rng) -> dict`. It calls `tools.call("name", ...)`
+and may annotate its reasoning with `tools.note_model_step(...)`. Success is decided by an oracle
+over the environment's final state, never by what the agent says about itself.
+
+**A gate that can say "I do not know".** The rule is fixed-sample and deliberately boring
+([docs/STATISTICS.md](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/STATISTICS.md)): per arm, an exact Clopper-Pearson interval at tail
+`alpha / (4K)`; bounds on the difference by Bonferroni; `PASS` iff the lower bound is above
+`-delta`, `BLOCK` iff the upper bound is below it. No peeking, no extending a run, no rerunning
+until green. The decision rule is bound into every trial's identity, so a manifest resealed after
+the run with a different rule no longer matches its trials and the gate returns `ERROR`: choosing
+the rule after the data is detected, not merely forbidden. Earlier runs on the same candidate are
+listed in `prior_runs` and printed in the decision and the Markdown report; `arci` does not
+discover them or enforce a cross-run error budget for you. Wilson intervals are shown for
+readability and never decide.
+
+**Command agents.** An agent does not have to be a Python function: it can be any program
+that speaks MCP over stdio. `arci` owns the one MCP server behind it, so every tool call is
+recorded, budgeted, fault-injected and replayable from outside the agent's process.
+
+```text
+your agent (any argv) --> arci.mcp_shim --> arci.mcp_boundary --> MCP server (the environment)
+                          byte relay        recorder, faults,     yours, or any python toolset via
+                                            budgets, latches      python -m arci.mcp_toolset_server
+```
+
+[docs/REAL_AGENTS.md](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/REAL_AGENTS.md) has the recipe and the local-model worked example.
+
+**Decision calls.** A command agent may call a System One decision endpoint (TypeSafe's Jev
+API). The same boundary serves `POST /v1/systemone` on loopback, records each admitted request as
+a tool call named `decision:systemone`, and applies budgets, perturbations, diff, minimisation,
+bundles and exact replay to it. The agent is redirected with `TYPESAFE_BASE_URL` and a per-trial token
+while any real key stays in the harness; the official Python and JavaScript SDKs are exercised
+unchanged by the acceptance tests. `arci preflight` checks key, endpoint, pinned model and one
+smoke decision before a live run.
+
+![The agent process, any SDK unchanged, reads two environment variables that point at a loopback port and a per-trial token. The harness-owned boundary serves the MCP tools and the decision endpoint, records every request and answer, budgets them and injects decision_low_confidence or decision_unavailable on schedule. The upstream is a seeded fixture in CI or the real API with the key held by the harness. Sealed records keep the served answer and the raw upstream answer, never the key, and feed the same gate, replay, preflight and report. Provenance: built from the vendor's documentation and both official SDKs, dated 2026-09-22; run against Jev on 2026-10-03, see docs/results/jev.](https://raw.githubusercontent.com/ajaysurya1221/agent-reliability-ci/main/docs/images/decision-boundary.png)
+
+**Run against Jev on 2026-10-03.** Built from documentation dated 2026-09-22, the boundary ran the
+pre-registered [day one](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/design/0004-day-one.md) live. Four things did not survive contact
+and are fixed: alias-only model lists (`arci preflight --allow-unlisted-model`), smoke questions
+needing `instructions`, CONNECT-only egress (`ARCI_HTTPS_PROXY`), and the console script needing
+the repository root on `PYTHONPATH`. Prices and provider limits are still as documented, not
+measured, and the gateway route is still untested. See [docs/DECISIONS.md](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/DECISIONS.md).
+
+## Where it has been run
+
+All runs are the maintainer's own; no external users yet. Failed and invalid attempts stay in
+the linked records.
+
+- **Seeded retry demo** (2026-09-23): the result above; A against itself and the repaired C both
+  PASS ([retry-demo-n200.md](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/results/retry-demo-n200.md)).
+- **Support-triage agent, live Jev endpoint** (2026-10-03, `jev-1.13.0` pinned, Python and Node
+  SDKs unchanged; planted regression, the harness caps confidence and B stops escalating): A vs B
+  at N=200, 200/200 vs 0/200, bounds [-1.000, -0.957], BLOCK; one minimised failure replays
+  REPRODUCED; 902 recorded decisions ([docs/results/jev](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/results/jev/README.md)). Offline
+  against a seeded fixture: `.venv/bin/python examples/jev_triage_agent/hero_demo.py --n 50`.
+- **The tool-call guard of the maintainer's other project, frontier-scout** (2026-10-05,
+  `hook_runtime.py` verbatim at `58b0b17`, run through the harness boundary, not inside a
+  coding-agent session; outages injected): fail-closed against fail-open under
+  `decision_unavailable` at N=200, 200/200 vs 130/200, bounds [-0.4304, -0.2537], BLOCK; diffed,
+  minimised (`reduced`) and replayed REPRODUCED ([docs/results/guardrail](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/results/guardrail/README.md)).
+- **A local model, one sentence of prompt** (`qwen3.5:4b-mlx` through Ollama, one machine,
+  exploratory): N=400, 376/400 vs 308/400, bounds [-0.2445, -0.0921], INCONCLUSIVE; BLOCK needed
+  the upper bound below -0.10 and missed by 0.008. Not re-run until it blocks
+  ([docs/results](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/results/README.md)).
+
+**What went wrong and was kept.**
+
+- Ollama run 1: a harness bug (tools advertised with no parameters), found from the failure
+  clusters ([ollama-run1-n30-bridge-bug.md](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/results/ollama-run1-n30-bridge-bug.md)). Run 3:
+  the model server died and one ERROR trial invalidated the run
+  ([ollama-run3-n400-invalid.md](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/results/ollama-run3-n400-invalid.md)); hence
+  `infra_exit_codes` (75 by default), which turns an agent's own infrastructure failure into ERROR.
+- Jev attempt 1: every trial crashed on an import error, so equally broken arms "passed"; the
+  invalid stores are kept in [attempt-1-invalid](https://github.com/ajaysurya1221/agent-reliability-ci/tree/main/docs/results/jev/2026-10-03/attempt-1-invalid/).
+- Guardrail at N=50: 50/50 vs 38/50, bounds [-0.4016, -0.0348], INCONCLUSIVE against the
+  pre-registered BLOCK, reported as is ([docs/results/guardrail](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/results/guardrail/README.md)).
+
+**Is the model's confidence something you can gate on?** A separate, pre-registered audit of
+`jev-1.13.0` on CLINC150 and Banking77 answers that for intent routing, not permission
+calibration: zero-shot accuracy 0.921 with ECE 0.024 on CLINC150, and 0.801 with ECE 0.084 on
+Banking77, where confidence is over-stated in the middle of the range. Write-up, full table and
+limits: [docs/results/jev-calibration](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/results/jev-calibration/README.md); protocol and code:
+[bench/jev_calibration](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/bench/jev_calibration/PROTOCOL.md).
+
+## The gate, measured against itself
+
+`just selfcheck` (or `.venv/bin/python bench/selfcheck.py`) computes the gate's own operating
+characteristics by exact enumeration (`alpha` 0.05, `delta` 0.10, one condition). Probabilities of
+PASS / BLOCK / INCONCLUSIVE:
+
+| baseline -> candidate | N=20 | N=100 | N=200 | N=400 |
+|---|---|---|---|---|
+| 0.95 -> 0.95 | .000 / .000 / 1.000 | .443 / .000 / .557 | .889 / .000 / .111 | .999 / .000 / .001 |
+| 0.95 -> 0.85 (the margin) | .000 / .000 / 1.000 | .001 / .000 / .999 | .001 / .000 / .999 | .001 / .001 / .999 |
+| 0.95 -> 0.75 | .000 / .000 / 1.000 | .000 / .093 / .907 | .000 / .365 / .635 | .000 / .828 / .172 |
+| 0.95 -> 0.65 | .000 / .008 / .992 | .000 / .685 / .315 | .000 / .985 / .015 | .000 / 1.000 / .000 |
+| 0.80 -> 0.80 | .003 / .000 / .997 | .059 / .000 / .941 | .218 / .000 / .782 | .615 / .000 / .385 |
+
+![Verdict probabilities by exact enumeration for the default gate: a 95 to 65 percent drop is INCONCLUSIVE 99.2 percent of the time at N=20 and BLOCK 98.5 percent at N=200; a 95 to 75 percent drop is BLOCK 36.5 percent at N=200; two equal 80 percent agents PASS 21.8 percent at N=200.](https://raw.githubusercontent.com/ajaysurya1221/agent-reliability-ci/main/docs/images/calibration.png)
+
+- False PASS at the margin and false BLOCK under no change are both far below `alpha` (worst
+  0.004 and 0.001 over the boundary sweep). The price is conservatism: two equal agents at 80%
+  reach PASS only 21.8% of the time at N=200.
+- **Twenty runs decide almost nothing.** At N=20 a 95% -> 65% collapse is still INCONCLUSIVE 99.2%
+  of the time, and showing a failure rate below 0.5% needs at least 598 clean trials. A
+  twenty-point drop from 0.95 to 0.75 BLOCKs only 36.5% of the time at N=200.
+- `interval_method: newcombe` (about twice the power, calibrated by exact enumeration rather than
+  proved) and pre-registered `looks` must be frozen before the run; `arci plan` shows a design's
+  verdict probabilities before you spend trials. Details: [docs/STATISTICS.md](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/STATISTICS.md),
+  [report section 3](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/reports/ci-gate-2026-10-06/REPORT.md#3-operating-characteristics).
+
+## Gate your own agent
+
+Scaffold one experiment from four references, then run it:
+
+```bash
+uv tool install agent-reliability-ci
+arci init checkout-retry --baseline app.v1:run --candidate app.v2:run --toolset tests.world:make_world --oracle tests.world:passed
+arci run arci.yaml
+```
+
+`arci init` imports the four references once (in a child process, as a trial would), checks that every
+fault names a tool the toolset provides, and writes `arci.yaml`: a thin, schema-checked descriptor that
+`arci run` seals into the same JSON manifest the Python builder produces. The scaffold is a **smoke
+design** (8 trials per arm): under the default rule two perfect arms cannot PASS below N=42 and a
+total collapse cannot BLOCK below N=8, so expect INCONCLUSIVE, then plan a confirmatory N with
+`arci plan` before spending trials. Without `--fault` the scaffold writes one clean condition (no
+injected fault); `--fault tool_timeout:reserve` adds a timeout on the first `reserve` call, and `--task '{...}'`
+sets the task. A hand-written descriptor with one fault looks like this:
+
+```yaml
+experiment_id: checkout-retry
+task: {order_id: "order-7"}
+toolset: tests.world:make_world
+oracle: tests.world:passed
+baseline: app.v1:run
+candidate: app.v2:run
+conditions:
+  - id: reserve-timeout
+    fault: {name: tool_timeout, tool: reserve, occurrence: 0}
+alpha: 0.05
+delta: 0.10
+n_per_arm: 8
+```
+
+The descriptor expands documented shorthands (`id`, `fault`, `occurrence`, `oracle`) into a sealed
+manifest; the key reference lists its defaults. `arci schema` prints the JSON Schema, and [docs/DESCRIPTOR.md](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/DESCRIPTOR.md)
+has the key reference, command-agent experiments and the YAML quoting rule. The toolset is `(task, seed) -> object with .tools and .snapshot()`, the
+oracle `(task, final_state) -> bool`, and each agent a plain function `agent(task, tools, rng) -> dict`.
+
+### Advanced: the Python manifest builder
+
+A manifest is a sealed JSON document built in Python. The retry example's builder,
+[examples/retry_agent/experiment.py](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/examples/retry_agent/experiment.py), is the template; its shape:
+
+```python
+Manifest.create(
+    experiment_id="retry-agent-agent_b-reserve-timeout-200",
+    task_id="confirm-inventory-order",
+    task={"order_id": "order-7", "sku": "widget", "quantity": 2, "stock": 10},
+    toolset="examples.retry_agent.world:make_world",  # (task, seed) -> tools + snapshot
+    contract=ContractSpec(
+        oracle="examples.retry_agent.world:oracle"
+    ),  # (task, final_state) -> bool
+    baseline=ArmSpec(label="agent_a", agent="examples.retry_agent.agents:agent_a"),
+    candidate=ArmSpec(label="agent_b", agent="examples.retry_agent.agents:agent_b"),
+    conditions=(
+        Condition(
+            condition_id="reserve-timeout",
+            faults=(
+                FaultSpec(
+                    name="tool_timeout", bucket=Bucket.FALSIFY, tool="reserve", at_occurrence=0
+                ),
+            ),
+        ),
+    ),
+    n_per_arm=200,
+    base_seed=12_000,
+    budgets=Budgets(max_tool_calls=10, max_model_steps=10, max_seconds=15.0),
+)
+```
+
+A behaviour contract names the oracle plus the invariants the tool boundary can observe
+(`max_tool_calls`, `forbidden_tools`, `required_tools`, `max_calls_per_tool`). A contract that asks
+for something the boundary cannot see is rejected, not silently skipped. Fault conditions are tagged
+`falsify` (a robust agent should survive), `benign` (must not change the outcome) or `ceiling`
+(known-unrecoverable: reported, never gated on rates). Built in: `tool_timeout`, `tool_error_once`
+and `empty_result` at the tool boundary, plus `decision_low_confidence` and `decision_unavailable`
+at the decision boundary; a `"module:function"` reference plugs in your own. For a command agent,
+replace `toolset` with `mcp_server` and the arms' `agent` with `command`, as
+[docs/REAL_AGENTS.md](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/REAL_AGENTS.md) shows.
+
+```bash
+arci init NAME --baseline M:F --candidate M:F --toolset M:F --oracle M:F   # write arci.yaml (smoke N=8)
+arci schema                                        # the descriptor's JSON Schema, generated from code
+arci demo                                          # the packaged six-step retry story, no clone
+arci plan --help                                   # verdict probabilities of a design, before any trial
+arci preflight manifest.json                       # live decision upstreams: key, model, smoke call
+arci run arci.yaml --workers 8                     # run + gate; exit code is the verdict (--out defaults to runs)
+arci gate runs/<experiment> --junit junit.xml --markdown report.md
+arci report runs/<experiment>                      # gate, failure clusters, tokens, estimated cost
+arci diff runs/<experiment> <passing_trial> <failing_trial>    # --all-steps to include annotations
+arci minimize runs/<experiment> <failing_trial> --out min.json
+arci bundle runs/<experiment> <failing_trial> --out bundle.json --root . --include my_agent.py
+arci replay bundle.json                            # 0 reproduced, 1 not reproduced, 3 invalid
+```
+
+A run store is one directory per experiment with the sealed manifest, `events.jsonl` and
+`trials.jsonl`. Keep it with the evidence it supports; it can hold the agent's full task state and,
+for decision calls, the full request state, so treat stores and bundles as sensitive data. Records
+sealed by an earlier schema version are not readable after an upgrade (v0.2 and v0.5 changed the
+schema); re-run the experiment rather than trust an unreadable store.
+
+### GitHub Actions
+
+This example pins `v0.8.0`, the released feature set (the Action installs that tag's source; the
+same code is on PyPI as `agent-reliability-ci==0.8.0`). See
+[CHANGELOG.md](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/CHANGELOG.md).
+
+```yaml
+- uses: actions/setup-python@v7
+  with: { python-version: "3.12" }
+- run: pip install -e .            # your agent, importable
+- uses: ajaysurya1221/agent-reliability-ci@v0.8.0
+  id: gate
+  with:
+    manifest: reliability/manifest.json
+    workers: "4"                   # out: arci-runs by default
+- run: echo "verdict=${{ steps.gate.outputs.verdict }}"
+```
+
+This assumes your agent's repository is checked out and installed, and that you pin action versions
+you have verified. The report lands in the job summary (a start-up failure shows only in the step's
+stderr). `BLOCK` and `ERROR` always fail the job; `INCONCLUSIVE` fails it unless you set
+`allow-inconclusive: "true"`. The `verdict` output (`PASS`, `BLOCK`, `INCONCLUSIVE` or `ERROR`) is
+there for steps that run after the gate. This repository's own CI runs the test suite,
+re-derives the archived decisions, smoke-tests the built wheel on Linux and macOS, and runs the Action
+on a committed fixture that must BLOCK.
+
+## What it does not do
+
+- **It is not a sandbox and not a security boundary.** Python agents share a process with their tool
+  boundary. Command agents use a separate harness-owned MCP boundary, which is isolation from
+  accident, not containment. Both assume buggy, non-hostile agent code; a hostile agent could still
+  forge or bypass its record. See [docs/TRUST_MODEL.md](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/TRUST_MODEL.md).
+- **It only sees the tool boundary.** Files, network and subprocesses the agent touches directly are
+  invisible, as is any decision call that does not go through the injected base URL.
+- **It replays the boundary, not the model.** Deterministic replay is guaranteed for seeded,
+  fixture-backed agents. A live model will not repeat itself token for token, so replaying its
+  bundle is usually INVALID; what survives is the recorded trace and the reduced fault condition.
+- **INCONCLUSIVE is common at small N.** That is the honest answer, not a failure of the run: at
+  N=20 even a 95% -> 65% collapse is INCONCLUSIVE 99.2% of the time, and one real 400-trial run
+  missed BLOCK by 0.008 and stayed INCONCLUSIVE.
+- **A bundle embeds code, and replaying it runs that code.** Hashes are integrity checks, not
+  signatures. Replay bundles only from sources you trust. A bundle embeds only the files you
+  `--include` (the minimiser's output embeds none), so portable replay needs the referenced code and
+  a compatible environment.
+- **A divergence is evidence, not proof of cause.** It shows where two runs part ways.
+- **The decision boundary has been run against one vendor model, on one day, on three seeded
+  tickets.** The live model answered every clean decision with confidence 1.0, so the runs in
+  `docs/results/jev` show the harness, the pacing, both SDKs and the gate working end to end and the
+  planted regression caught; they say nothing about the model's calibration or about ambiguous
+  tickets. Live answers are sampled, so live minimisation is reported as `reduced`, never
+  `1-minimal`, and a decision model's confidence never enters a verdict except through the agent's
+  actions.
+- **Reference validation executes user code.** `arci init` validates Python references by importing
+  them, and calls the toolset factory once, in a child process; import-time code runs.
+- **Declaring a fault does not guarantee injection.** A fault names a tool; if the agent never calls
+  that tool at that occurrence, the condition runs clean and nothing says so. Inspect the recorded
+  calls to confirm injection.
+- Command agents get one stdio MCP server (revision 2026-07-28), serial tool calls, no HTTP
+  transport, no concurrent decisions.
+- No importers (OTLP, Claude Code, Codex), no HTML report yet. POSIX only: on Windows the package
+  installs and `arci --help` runs, but experiments are unsupported.
+
+**Related work:** [Documentation comparison, checked 2026-10-05; not a matched benchmark.](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/reports/ci-gate-2026-10-06/REPORT.md#6-related-work-and-honest-positioning)
+
+**Independence.** ARCI's job is preregistered stochastic regression testing. The manifest is a
+prospective contract supplied by its author: it fixes the task, fault conditions, sample size and
+decision rule before the trials run, and a rule changed after the fact is detected, not merely
+forbidden. It does not independently authenticate when the manifest was written or that every earlier
+run was disclosed; that discipline stays with the author. A `PASS` reports relative non-inferiority
+against the baseline within the registered margin under those conditions, never absolute reliability
+or a general quality score; `BLOCK`, `INCONCLUSIVE` and `ERROR` are defined above. ARCI is one of the
+maintainer's independent projects alongside the earlier
+[evalopt-graph](https://github.com/ajaysurya1221/evalopt-graph) experiment, whose standalone feature
+development has ended. ARCI declares no package or runtime dependency on evalopt and imports nothing
+from it, and it shares no runtime layer with the maintainer's sibling projects Actseal, Dorian and
+Frontier Scout. [NOTICE](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/NOTICE) records two adaptations from evalopt under MIT with attribution: the
+canonical JSON hashing helpers in `src/arci/hashing.py` and the self-sealing decision pattern in
+`src/arci/schema.py`, both attributed source adaptation inside the frozen manifest, not a dependency.
+ARCI is not a successor to, conformance target for or drop-in replacement for evalopt, and nothing
+here claims adoption, performance or superiority.
+
+## Roadmap
+
+Most valuable first: per-request pacing through parent IPC; counterfactual replay of recorded
+decisions, if it can be done without overstating what a re-perturbed recording proves; extending
+the intent-routing calibration audit to labelled coding-agent permission decisions, with a
+separate protocol and explicit limits. Full list in [docs/ROADMAP.md](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/ROADMAP.md).
+
+## Development
+
+**How it was built.** One maintainer, September to October 2026, with Claude Code and Codex as
+pair-programmers (recorded in the commit messages). The experiment design, the decision rule and
+its calibration, the trust model and the acceptance tests are the maintainer's own; the retry
+demo, the calibration table and the test suite reproduce from a clean clone. Issues and PRs welcome.
+
+```bash
+uv sync
+just check        # ruff format --check, ruff, basedpyright strict, pytest
+just frozen       # the frozen contract files are unchanged
+just selfcheck    # the calibration table above
+just demo
+npm ci --prefix examples/jev_triage_agent   # only for the Node agent and its test (Node 20+)
+python docs/assets/src/make_figures.py --check   # the README figures match their generator
+```
+
+The acceptance suite is hash-pinned in `FROZEN.sha256`. CI checks formatting, lint, strict typing
+and tests on Python 3.11 and 3.14, with separate macOS statistics checks. See
+[the workflow](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/.github/workflows/ci.yml) for coverage and
+[the evidence report](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/reports/ci-gate-2026-10-06/REPORT.md) for measured results. Design
+records live in [docs/design](https://github.com/ajaysurya1221/agent-reliability-ci/tree/main/docs/design/), the changelog in [CHANGELOG.md](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/CHANGELOG.md).
+
+### Availability and project status
+
+**Availability:** v0.8.0 is on PyPI as `agent-reliability-ci` and as the Action tag `v0.8.0`; it
+adds the packaged demo (`arci demo`), the descriptor (`arci.yaml`, `arci schema`), `arci init`, the
+planner (`arci plan`), opt-in pass^k reporting (`arci report --pass-k`) and portability fixes over
+v0.7.0. **Boundary:** PASS means relative non-inferiority, not absolute reliability. This is not a
+sandbox.
+
+[![CI](https://img.shields.io/github/actions/workflow/status/ajaysurya1221/agent-reliability-ci/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/ajaysurya1221/agent-reliability-ci/actions/workflows/ci.yml)
+[![Release](https://img.shields.io/github/v/release/ajaysurya1221/agent-reliability-ci?sort=semver&style=flat-square)](https://github.com/ajaysurya1221/agent-reliability-ci/releases/tag/v0.8.0)
+[![Evidence package](https://img.shields.io/badge/evidence_package-2026--10--06-555?style=flat-square)](https://github.com/ajaysurya1221/agent-reliability-ci/releases/tag/evidence-2026-10-06)
+[![License](https://img.shields.io/badge/license-Apache--2.0-blue?style=flat-square)](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/LICENSE)
+[![Python](https://img.shields.io/badge/python-3.11%2B-blue?style=flat-square)](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/pyproject.toml)
+[Security policy](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/SECURITY.md)
+
+[Apache-2.0](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/LICENSE). See [NOTICE](https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/NOTICE) for adapted code.

agent_reliability_ci-0.8.0.dist-info/RECORD

@@ -0,0 +1,40 @@
+arci/__init__.py,sha256=147fAPiBqNlvDat8cKcagr8X7PFNoh8kq8se8m4jEJg,96
+arci/__main__.py,sha256=hj411yFeJie6ShpC-MfAlKStBnZpBp3iunbmcRjFnks,52
+arci/cli.py,sha256=rEPzytPgv-7kvvXow4Kuh4lOlyz5jco4q15-2utP7Mc,31837
+arci/contracts.py,sha256=DZZJHrfFMFtFOAE8fQtN8j4nCvMEavM2GmKW9hvUj2I,3798
+arci/descriptor.py,sha256=6MbJiYXnwHHkbPl1rtNZhza8GajJ1lQd066SHkR2UQg,41886
+arci/diff.py,sha256=cR6j2T9--kiq36cmeDh29EnLrGrPoQ92uxiieV_NSMQ,6425
+arci/fingerprint.py,sha256=CUJV40UKFxuPAGzN1uJjxZufc_9ORg-BN9pFBlGJd4I,1222
+arci/gate.py,sha256=okZU-LGK4scss6tfr4TozLXUDsUePf_NjkrJzeazTfg,14632
+arci/grade.py,sha256=YDy8NQNN61RtQQ6gVYkbt6PtTZxplY2cZOP-fXwDGd0,2907
+arci/hashing.py,sha256=Iw2EfoDBvgXJDgaSHiwCIgFjkKUjVl5pwdqDLhpiz7M,906
+arci/init.py,sha256=EA6POR6wYD5uCRzmLCDOouSgj1fkNts4ldDSgJm2k18,24638
+arci/interfaces.py,sha256=QWYFeJOaHgC8Z02CnA4buJ5-93gFSFlYC5K8JA3dBSU,6338
+arci/mcp_boundary.py,sha256=eyj2LnIrSF8NwxuaTGHCwA-ITFNQuv0WS8H7HsLUP-I,85308
+arci/mcp_shim.py,sha256=4L-HjnfNx5RN9zGSfjeAQ_hATzWU5g0UItNX69lI884,1800
+arci/mcp_toolset_server.py,sha256=lHb5CgkJkYGBYwc507Fxp6N5r0OmfcEIund-OAQ6ddY,9869
+arci/minimize.py,sha256=dqadc0fr8rAgds2IRtrrgVRmex-995bhPOaF1BdoljE,5057
+arci/perturb.py,sha256=JjTGJJCW8SMI-n9oB4OsTOUU_ueCEejYwkrO0MsPBf8,7963
+arci/planning.py,sha256=evHVNNK64bwln-I7Db_zWme91TX4mm8iEDD8L-YNTgM,21283
+arci/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+arci/reliability.py,sha256=g9fRG4iTTNEUQQu4QO9Xv3Crk_tgB-TDVR0cXBcI-9Y,4498
+arci/replay.py,sha256=zbhWx6IicuZ9Ynf6IQDOfO6wYsPOHF6REBzvitRiYcY,4722
+arci/report.py,sha256=Yqa-xNsn77NETrRqYx_UBYol7o_WP83_-YzyNf9z_kM,13886
+arci/runner.py,sha256=abUVfidq1905xMfu0pKxtSRtDZaVOrqB8NmOwv_-dIg,64220
+arci/schedule.py,sha256=S9sTrxH_s7M_XjVLQ8we6HyErOdpSO0VS3c3-AyJsvw,2387
+arci/schema.py,sha256=1wCMzFGKLoGvjSR3aIb9agPqu2FeDCdVOh__e8Y9V1Y,26192
+arci/stats.py,sha256=x5twT7Kg2t5rX6wfAfnQMdjwAJk_FyNQcJIIDa-n7zQ,6064
+arci/storage.py,sha256=PyngbQbeArw8By7CQKj6FKNARN142qEtPxO8vIQoVUc,1647
+arci/toolbox.py,sha256=y-Gw2cCuS7LNh10vQbjINfFc-_iiCk1XNpaO0gI5mJ0,12021
+arci/worker.py,sha256=WB74W04lt8lFC6fhYwIyR3BaXbjaiGjhlhnoCEw4I5I,7586
+arci/demo/__init__.py,sha256=H5slPkgA4VfenosAPV-k9owfgPTohnwg8zwQtRJNTKM,192
+arci/demo/agents.py,sha256=o5VpHISDoJlA_epzeNmNTp-DJFW1PeLCIUfoQEg7i2A,4180
+arci/demo/experiment.py,sha256=u0D4pQ2AWJ7Jwe3WPPq8WEvQlLEq8AdbAiMDgP117Hk,2489
+arci/demo/run.py,sha256=XR0JiRfyYIfdJxnUiX9DkX6ZWgmxiiL-7kBUrwvK6Dg,13043
+arci/demo/world.py,sha256=GxpGvJY5r9p0FtePpL7CNBKmz7wuE9YeY0BozcsghNU,4453
+agent_reliability_ci-0.8.0.dist-info/METADATA,sha256=2zR9W4aYRnMFslsZF_cIczYLvG4s8UEmpaNdMjnajj4,36499
+agent_reliability_ci-0.8.0.dist-info/WHEEL,sha256=W3fkpkm7-wf9vBI5Z-7s0eWkeM-spu78I8Neb98DeEg,87
+agent_reliability_ci-0.8.0.dist-info/entry_points.txt,sha256=L1AYQhsIQJzWoG3Q_SIHy5lzc3bnfMQpeou36c5OUnY,39
+agent_reliability_ci-0.8.0.dist-info/licenses/LICENSE,sha256=EGsLk1OHMpu4Y4mYkaUBUVXaeDeSm4QX3O8MYFHgV4Q,11353
+agent_reliability_ci-0.8.0.dist-info/licenses/NOTICE,sha256=a9LQB8vhSxcTc7KtQFKZcH1vgx6TpERz3S1BXrhuXR0,420
+agent_reliability_ci-0.8.0.dist-info/RECORD,,

agent_reliability_ci-0.8.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.32.4
+Root-Is-Purelib: true
+Tag: py3-none-any

agent_reliability_ci-0.8.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+arci = arci.cli:main