agent-reliability-ci 0.8.0__tar.gz

agent_reliability_ci-0.8.0/.gitignore

@@ -0,0 +1,12 @@
+.venv/
+__pycache__/
+*.pyc
+.pytest_cache/
+.ruff_cache/
+dist/
+build/
+*.egg-info/
+.DS_Store
+runs/
+.env
+node_modules/

agent_reliability_ci-0.8.0/CHANGELOG.md

@@ -0,0 +1,375 @@
+# Changelog
+
+## Unreleased
+
+(nothing yet)
+
+## v0.8.0 (2026-10-13)
+
+The onboarding release: install from PyPI, run the recorded retry story with one `uvx` command, scaffold
+a first experiment from four references, and read the whole point on one screen. No change to the
+decision rule, its calibration, the schema (`arci/0.5`) or any recorded result.
+
+- **Added:** the PyPI package `agent-reliability-ci` and `.github/workflows/release.yml`: a version
+  tag builds the wheel and sdist once, smoke-tests that wheel on Ubuntu and macOS (Python 3.11 and
+  3.14) from an empty directory, and publishes the same files through PyPI trusted publishing with
+  attestations. The guard refuses a tag that is not `v` + the package version. `docs/RELEASING.md`
+  has the steps; `docs/design/0005-v0.8.0-packaging-and-version-bump.md` records the frozen-file
+  changes (version, project URLs, classifiers, keywords, an sdist restricted by `only-include` to
+  the package, README, LICENSE, NOTICE, CHANGELOG and pyproject, plus hatchling's own `PKG-INFO`
+  and `.gitignore`).
+- **Added:** `arci demo` (and `python -m arci`): the six-step retry story from an installed wheel with
+  no checkout, through the real `run`, `diff`, `minimize` and `replay`. The seeded world and agents
+  now live in `arci.demo` (`examples/retry_agent/{world,agents}.py` re-export them); the demo's
+  manifests use `arci.demo` references and an `arci-demo-` experiment id, so their seals differ from
+  the recorded run while the seeds, counts (192/200 vs 132/200) and verdicts are the same, and the
+  demo says whether its counts matched. Run stores go to a temporary directory unless `--keep DIR`.
+- **Added:** the `arci.yaml` descriptor (`src/arci/descriptor.py`, `docs/DESCRIPTOR.md`,
+  `docs/design/0007-descriptor.md`): a thin, `extra="forbid"` front-end whose keys are the `Manifest`'s
+  fields with the same defaults, compiled and sealed with `Manifest.create` exactly as the Python
+  builders do. `arci run` accepts a `.yaml`/`.yml` descriptor or a sealed `.json` manifest, and its
+  `--out` now defaults to `runs`. `arci schema` prints the JSON Schema generated from the model;
+  the bytes are committed at `docs/schema/arci.schema.json` and tested for staleness. Pydantic is
+  authoritative; the YAML loader resolves only `true`/`false` as booleans and decimal, `0x`, `0o`,
+  `0b` as integers, so `yes` and `on` stay strings and `010` is ten, not octal eight; explicit
+  `!!int`, `!!float` and `!!null` tags accept only those same forms, and alias expansion is capped.
+- **Added:** `arci init EXPERIMENT_ID --baseline --candidate --toolset --oracle [...]`
+  (`docs/design/0008-arci-init.md`): validates the four references in a child process with the
+  runner's import rule (the toolset factory is called once; agents and the oracle are only imported),
+  checks every `--fault` tool against the toolset's tools, and writes a smoke-design descriptor
+  (`n_per_arm` 8, the smallest N at which a total collapse can still BLOCK under the default rule;
+  two perfect arms cannot PASS below N=42). One line on stderr and exit 3 on the first problem. The
+  file is published atomically (a hard link, or a rename with `--force`); without `--force` an
+  existing file is never replaced.
+- **Added:** a smoke-design notice on stderr from `arci run` when a gating condition has fewer than
+  100 trials per arm, pointing at `arci plan`. stdout, reports, decisions and stores are unchanged.
+- **Changed:** the runner appends the current directory to the import path of every child it starts
+  (worker, grader, boundary, command agent, MCP server;
+  `docs/design/0006-cwd-on-the-child-import-path.md`), so references into the project directory
+  resolve under the `arci` console script as they already did under `python -m arci.cli`. Appended,
+  never prepended; an MCP server whose `env` sets `PYTHONPATH` keeps that configured value, as
+  before; records are unaffected.
+- **Added:** CI jobs that build the wheel and run `arci demo` from an empty directory on Ubuntu and
+  macOS, a Windows job that only installs the wheel and runs `arci --help` (execution stays
+  unsupported on Windows), and an Action fixture (`examples/action_fixture/`) that runs the composite
+  Action on the demo's A-vs-B manifest and asserts `BLOCK`. CI also runs on `sprint/**` branches.
+- **Added:** `docs/measurement/`: the time-to-first-verdict and cold-read protocol, a receipt
+  template and a receipts directory; maintainer measurements are labelled and never counted as
+  cold readers.
+- **Documentation:** the README opens with the question, one sentence, the two commands, the pipeline
+  line, what PASS means, what a small run means and the boundary (`docs/design/0009-readme-first-screen.md`);
+  the recorded result, the figure and the badges moved below the fold; "What it does not do" gains two
+  items (reference validation imports your code; a fault naming a tool the agent never calls silently
+  never fires). Links from the old first screen are absolute so the PyPI page renders them.
+- **Documentation:** two recipes, each a runnable tree under `examples/recipes/` with a narrative in
+  `docs/recipes/` and a test that runs it at smoke N from a copied directory without `PYTHONPATH`:
+  `python_agent` (a blank project with `app/v1.py`, `app/v2.py` and `tests/world.py`; `arci init`,
+  the smoke run, `arci plan`, the confirmatory run, then `diff`, `minimize` and `replay`, with the
+  recorded output) and `mcp_agent` (a stdlib newline-JSON-RPC client against
+  `arci.mcp_toolset_server`, a hand-written command-arm descriptor and an Inspector preflight). The
+  Inspector step is recorded as blocked on 2026-10-13: Inspector 2.10.1 (TypeScript SDK 1.29.0)
+  rejects the bridge's protocol revision `2026-07-28` at `initialize`; the recipe keeps the verbatim
+  result and the ARCI steps run without it.
+- Everything that was already on `main` and unreleased is in this version:
+  - **Documentation:** the README presents the recorded retry result once (200 trials per arm,
+  192/200 vs 132/200, BLOCK) with its boundary, links to the evidence, and the offline demo with
+  lines from the committed demo log (in this release these sections sit below the new first
+  screen); a "Where it has been run" summary links every campaign's records, including the
+  guardrail run and the invalid and inconclusive attempts. The hero PNG is replaced by
+  generated light and dark SVGs (`docs/assets/hero-*.svg`) and the how-it-works exposition by a
+  generated pipeline figure (`docs/assets/where-*.svg`), both written by
+  `docs/assets/src/make_figures.py` (stdlib only), whose `--check` mode runs in CI and in
+  `tests/unit/docs/test_figures.py`.
+  - **Added:** the technical report package `docs/reports/ci-gate-2026-10-06/` (report, plan of
+  record, enumerated operating-characteristics metrics for both interval methods, the Linux
+  container re-derivation log for all 18 archived decisions, the offline replay probe log for
+  both committed failure bundles, and the two-stage Docker reproduction recipe) and
+  `CITATION.cff`.
+  - **Fixed:** platform-dependent rounding in 95% Wilson intervals by pinning the inverse-normal
+  value used by the existing Linux-produced records (CPython's C accelerator on arm64 macOS
+  differs by one ulp from the pure-Python fallback). Restores reproducible decision seals for
+  the archived Clopper-Pearson stores without changing any verdict. Regression tests re-derive
+  every committed `docs/results/jev` decision byte for byte.
+  - **Fixed:** platform-dependent Clopper-Pearson bounds by comparing binomial tails with exact
+  integer arithmetic, including zero-success and all-success bounds. The existing 60-step
+  bisection, statistical method, confidence allocation, and gate rules are unchanged. Re-derived
+  six archived Jev decisions and five guardrail decisions from unchanged sealed manifests and
+  trials. Only CP bounds, their derived difference bounds, and decision seals changed; verdicts,
+  exit codes, counts, stopping histories, and rendered Markdown/JUnit reports are unchanged.
+  Byte-for-byte regeneration is checked on macOS and Ubuntu.
+  - **Added:** `arci plan` enumerates PASS, BLOCK and INCONCLUSIVE probabilities under the
+  existing Clopper-Pearson gate for independent binomial arms, one gating condition, one
+  fixed-sample look and N <= 400 per arm, reporting assumptions and the smallest tested N meeting
+  a target or "target not reached"; probabilities are enumerated with exact integer arithmetic
+  and converted to floating point once, so the output is identical on every platform.
+  - **Added:** opt-in Markdown `arci report --pass-k` estimates using C(successes,k)/C(n,k)
+  separately per arm and condition, with explicit IID assumptions and unavailability for n < k,
+  ERROR verdicts and multi-look designs; default report output, gate decisions and stored records
+  are unchanged.
+  - **Guardrail experiment:** published the pre-registered live run of
+  `examples/guardrail_agent/` (2026-10-05, `run.sh` unchanged): the sealed stores of all six
+  runs, the preflight receipt, per-run gate reports, and the N=200 failure diffed, minimised
+  and replayed (REPRODUCED), in `docs/results/guardrail/`. The write-up and `metrics.json` are
+  generated from the committed evidence by `docs/results/guardrail/summarize.py`
+  (`--check` fails if they are stale), including the N=50 `decision_unavailable` run that came
+  out INCONCLUSIVE rather than the pre-registered BLOCK. Added
+  `tests/unit/examples/test_guardrail_results.py`, an offline test that re-derives every
+  committed guardrail decision byte for byte with `arci gate`.
+  - **Documentation:** the offline demo section was renamed "Try the offline demo" (this release's
+  README now points it at `arci demo`); the availability note that separated `main` from the
+  `v0.7.0` Action is replaced by this release's Availability line; unsupported comparison and
+  reviewer-count wording removed; the calibration roadmap item now extends the published
+  intent-routing audit. `CITATION.cff` pins the report URL to the `evidence-2026-10-06` tag (its
+  version and date are now 0.8.0 and 2026-10-13). Added `SECURITY.md` (private reporting through
+  GitHub security advisories, the trusted-code assumption, no private stores or credentials in
+  reports).
+
+## v0.7.0 (2026-10-05)
+
+- **Calibration audit of the decision model (2026-10-03):** `bench/jev_calibration/` is a
+  pre-registered audit of `jev-1.13.0` on CLINC150 (4,500 in-scope + 1,000 out-of-scope) and
+  Banking77 (3,080), three passes each plus a bundling check, 26,140 requests, about USD 4.6
+  at list price. `PROTOCOL.md` (hash posted on PR #2 before the first request) fixes the
+  questions, passes, measures and seven expectations; `run.py` seals every answer in a
+  hash-chained store; `metrics.py` (exact binomial intervals, ECE/MCE/Brier/log loss,
+  tie-aware AUROC and average precision, bootstrap, agreement) is unit tested; `analyze.py`
+  verifies the chains and writes `metrics.json`, `report.md`, reliability diagrams and a
+  self-verifying compact store from which every number re-derives. All seven expectations
+  met: zero-shot accuracy 0.921 [0.912, 0.928] on CLINC150 (a supervised TF-IDF+LR trained on
+  15,000 examples: 0.925) with ECE 0.024; 0.801 on Banking77 with ECE 0.084 and mid-range
+  over-confidence; 0.970 accuracy at confidence >= 0.9 on 83% coverage; out-of-scope AUROC
+  0.977 (yes/no question) against 0.911 from 1 - p_max; 99.4% repeat agreement; p50 322 ms.
+  Write-up: `docs/results/jev-calibration/README.md`.
+- **Added:** `examples/jev_triage_agent/experiment.py` builds live manifests
+  (`live_manifest`, `decisions.upstream = "http"`, pinned model, paced trial starts, prior
+  runs) and writes them from the command line
+  (`python -m examples.jev_triage_agent.experiment --out manifest.json`), so the day-one
+  sequence in `docs/design/0004-day-one.md` runs from files. `examples/jev_triage_agent/day_one.sh`
+  drives it end to end: preflight, the clean pair, the N=50 trio, the Node agent through the
+  official SDK, the pre-registered N=200 run, and a minimised, replayed failure, with a gate
+  report per run and a refusal to finish if the key appears in any output. Results:
+  `docs/results/jev/`.
+- **Run against Jev (2026-10-03):** the pre-registered day one ran end to end against
+  `https://api.typesafe.ai` with `jev-1.13.0` pinned. A vs B under `decision_low_confidence` at
+  N=200: 200/200 vs 0/200, bounds [-1.000, -0.957], BLOCK; A vs A and A vs C PASS at N=50; the
+  Node agent through the unchanged official SDK BLOCK; the minimised failure replays REPRODUCED;
+  an exploratory clean run at N=50 routed all 100 decisions correctly at confidence 1.0. 902
+  recorded decisions, about USD 0.018 at list price. Sealed stores, the preflight receipt, the
+  invalid first attempt and one write-up per run: `docs/results/jev/`; every `decision.json`
+  re-derives from its committed `trials.jsonl`. Findings: `docs/DECISIONS.md`, "What day one
+  found".
+- **Added:** `arci preflight --allow-unlisted-model`. The account's `GET /v1/models` lists only
+  the aliases `jev-latest` and `jev-preview`, so the pre-registered preflight exits 2 on a pinned
+  versioned id. With the flag, preflight prints the listed names, sends the smoke request, and
+  exits 0 only if the validated response reports exactly the pinned id (3 otherwise). Without the
+  flag nothing changes except a hint after the exit-2 message. Tests:
+  `tests/unit/cli/test_preflight_unlisted_model.py`.
+- **Added:** `ARCI_HTTPS_PROXY=http://host:port` makes the decision upstream sender tunnel through
+  an HTTP CONNECT proxy, for environments whose only egress is such a proxy; unset, the direct
+  connection is unchanged, and a malformed value is a transport error. Tests:
+  `tests/unit/decisions/test_upstream_proxy.py`.
+- **Fixed:** the three preflight smoke questions carry `instructions`; the vendor rejects a
+  question with neither `instructions` nor `criteria` (400), so the pre-registered smoke request
+  could not be accepted as written.
+- **Fixed:** `examples/jev_triage_agent/day_one.sh` puts the repository root on `PYTHONPATH`
+  (the manifests name `examples.jev_triage_agent.*` modules, invisible to the `arci` console
+  script), insists on running from the repository root, and stops with exit 4 unless both arms of
+  the N=1 clean pair PASS, so a harness fault that breaks both arms equally cannot be mistaken
+  for a measurement.
+- **Fixed:** two reader threads, the grader's response reader and the MCP boundary's frame
+  reader, could end a read on a "not ready" poll verdict computed before the child wrote and
+  before the parent asked them to stop. On a loaded machine the thread was descheduled across
+  the child's exit, the stale verdict dropped the whole response, and the trial ended `ERROR`
+  (`grader returned invalid output`, or `boundary exited before returning` for command agents)
+  at about 1 trial in 400 with 8 workers on 4 cores, which is enough to turn a 200-per-arm
+  experiment into an `ERROR` verdict on a busy CI runner. The worker reader already followed
+  the correct rule; both readers now snapshot the stop flag before polling, and only a poll
+  made after the stop was observed may end the read. Regression tests:
+  `tests/unit/runner/test_reader_stop_race.py`.
+
+## v0.6.0 (2026-09-22)
+
+Day-one Jev readiness. The upstream behavior below is derived from vendor and official SDK
+documentation and tested against shaped fixtures; it was not run against Jev because no key was
+available.
+
+- **A1:** HTTP decision upstreams accept a path-prefixed base URL, including the Vercel AI Gateway
+  TypeSafe route and its `typesafe-ai/jev` model id.
+- **A2:** Response validation accepts the tested SDK response shapes: unknown metadata is tolerated and rounded
+  probability sums are accepted within `1e-3`, while required keys and pinned-model checks remain.
+- **A3:** Harness-owned request headers, allowlisted response headers, deterministic transport
+  diagnostics, and one recorded bounded wait on an upstream 429/529 make gateway attempts auditable.
+- **B4:** `arci preflight MANIFEST` checks credentials, model discovery and one tiny smoke decision,
+  then emits a credential-free receipt with exit 0/2/3 semantics.
+- **B5:** HTTP experiments pace trial starts through a shared parent token bucket; users divide the
+  provider rate by the maximum decisions per trial.
+- **B6:** Markdown reports count completed decision recordings and their available token usage,
+  and show estimated input cost at the list price read on 2026-09-22 (fixtures and replays are
+  free; this is an estimate, not billed spend). Incomplete admissions are excluded; before-injected
+  results contribute zero tokens.
+- **B7:** The Jev guide now covers the three-command day-one workflow, provider limits, privacy,
+  sampled answers and live-run minimality; `docs/results/TEMPLATE-jev.md` standardises write-ups.
+- **C8:** The triage example gains a Node 20+ ES-module agent using `@typesafe-ai/sdk` 0.6.0 through
+  environment-only `new TypeSafeClient()`, plus CI coverage on Python 3.14 / Node 22.
+- **C9:** Decision records retain an independent raw upstream snapshot before perturbation; live
+  HTTP upstreams, removal trials ending in ERROR, and replay INVALID results report `reduced`, never
+  `1-minimal`.
+- **D10:** The offline triage fixture can inject seeded synthetic miscalibration for testing
+  confidence gates; it is explicitly not evidence about Jev's calibration.
+
+## v0.5.0 (2026-09-22)
+
+Command agents can now test the decision logic around TypeSafe's Jev/System One API.
+
+- `Manifest.decisions: DecisionSpec` adds a harness-owned loopback `POST /v1/systemone` boundary.
+  Agents are redirected through `TYPESAFE_BASE_URL` and a per-trial token; the real upstream key
+  stays in the harness (the MCP server child does not inherit it, diagnostics are scrubbed). The
+  Python `typesafe-sdk` is exercised unchanged by the acceptance tests; the JavaScript SDK reads the
+  same variables but is untested here.
+- Admitted decisions use the existing `ToolCall`, `ToolResult` and `RecordedCall` records under
+  `decision:systemone`, so budgets, events, diff, minimisation, bundles and exact replay apply.
+- `decision_low_confidence` mixes choice probabilities toward uniform without changing the winner;
+  `decision_unavailable` returns an injected 529 from a selected occurrence onward.
+- The boundary now lives until the command agent exits when decisions are configured, including
+  decisions made after the MCP session closes. Decision/MCP overlap and concurrent decisions are
+  rejected as harness faults.
+- Schema `arci/0.5` adds decision configuration to sealed manifests and trial specs. Records sealed
+  by earlier schema versions are not readable; as with every field addition, experiments must be
+  re-run.
+- The official `typesafe-sdk` is a development-only dependency used by an acceptance test against
+  the boundary. Runtime code and `examples/jev_triage_agent` do not import it.
+- `examples/jev_triage_agent` is a six-step offline triage demo: a confidence-gating regression is
+  blocked, reduced, replayed, and repaired. `docs/DECISIONS.md` documents fixture and real-Jev use.
+
+## v0.4.0 (2026-09-22)
+
+One thing: pre-registered looks, so an experiment can stop early without losing error control.
+
+- `Manifest.looks = (N_1, ..., N_L)`: cumulative pairs per condition, last equal to `n_per_arm`.
+  Equal Bonferroni spending, `alpha / L` per look, with either interval method. The run stops at
+  the first look that is PASS or BLOCK; the final look is INCONCLUSIVE otherwise. No `looks` is
+  the fixed design, unchanged.
+- The gate replays every look and treats the first decisive one as a mandatory stop: a store that
+  continued past it, stopped without a decision, or is not a whole-pair prefix at a declared look
+  is ERROR. The plan is bound into every trial's identity; the look history is sealed into the
+  decision.
+- `bench/selfcheck.py --looks 50,100,200`: exact enumeration over sequential paths (a
+  (x_A, x_B) state table, at most (N+1)^2 states), with the same false-PASS / false-BLOCK <= alpha
+  sweep. Measured, K=1: a 0.95 -> 0.65 regression costs 302 expected trials with Clopper-Pearson
+  (P(BLOCK) .985 -> .943) and 159 with Newcombe, against 400 for the fixed design; two equal 0.95
+  agents cost 350 and 239. The Markdown report shows the look history and the stopping look.
+
+## v0.3.0 (2026-09-22)
+
+One thing: a tighter interval for the margin, opt-in.
+
+- `Manifest.interval_method`: `clopper_pearson` (default, unchanged) or `newcombe` (Newcombe's
+  hybrid score interval, method 10, no continuity correction, Wilson per arm at `1 - alpha/K`).
+  Frozen with the manifest; printed in every decision and report.
+- Calibration by exact enumeration in `bench/selfcheck.py --method newcombe`: a sweep of the
+  boundary and the no-change line for p in [0.5, 0.99], N in {20, 50, 100, 200, 400}. Worst
+  directional error 0.031 (alpha 0.05). The bench exits non-zero if any sweep fails. Head-to-head
+  table in `docs/STATISTICS.md`.
+- Why: the clean v0.2 confirmatory run (376/400 vs 308/400) was INCONCLUSIVE by 0.008 under the
+  default rule; Clopper-Pearson with Bonferroni was spending about a tenth of its allowed alpha.
+  Re-analysed under `newcombe` it blocks. Its sealed verdict stays INCONCLUSIVE.
+
+## v0.2.0 (2026-09-21)
+
+Command agents: test any program that speaks MCP over stdio, not only Python functions.
+
+### Added
+
+- **The out-of-process boundary.** A trial can be any subprocess (`CommandSpec`). Its tools are ONE
+  stdio MCP server owned by the harness (`McpServerSpec`). The agent's process tree holds only a
+  byte-relay shim (`arci.mcp_shim`); the recorder, fault injection, budgets and fault latches live in
+  a harness-owned boundary process (`arci.mcp_boundary`). Same frame protocol, latch precedence,
+  deadlines, process-group cleanup and sealed envelopes as v0.1.
+- Supported MCP subset (revision 2026-07-28): `initialize`, `notifications/initialized`, `ping`,
+  `tools/list`, serial `tools/call` with `resultType: "complete"`. Task results, sampling,
+  elicitation and concurrent calls are a harness ERROR, never a silent pass-through.
+- Replay for command agents: the recording includes `initialize` and `tools/list`, the server is
+  never started, JSON-RPC ids are normalised, consumption must be exact.
+- `arci.mcp_toolset_server`: serve any v0.1 python toolset as an MCP server, with a trusted
+  `snapshot` for grading.
+- `examples/ollama_mcp_agent`: a real tool-calling agent on a local model, with a one-sentence
+  prompt regression between its arms. `docs/REAL_AGENTS.md`, `docs/design/0002-...md`.
+- Schema `arci/0.2`: `ArmSpec.command`, `Manifest.mcp_server`, `McpServerSpec.snapshot`. An
+  experiment is all python agents or all command agents. v0.1 run stores must be re-run.
+
+- `CommandSpec.infra_exit_codes` (default 75, EX_TEMPFAIL): an agent can say "my infrastructure
+  failed, not me". The trial is ERROR and invalidates the experiment instead of counting against the
+  agent. Added after a local model server died mid-experiment and 435 trials "crashed".
+
+### Evidence (docs/results)
+
+A real local-LLM agent, two arms differing by one sentence of the system prompt, one injected tool
+timeout. Four runs are reported, including two invalid ones that each exposed a defect (a bridge
+that advertised no tool parameters; a dead model server plus a harness shutdown race). The clean
+confirmatory run: 376/400 (94.0%) versus 308/400 (77.0%), bounds [-0.2445, -0.0921], verdict
+INCONCLUSIVE by 0.008 under the pre-registered rule. Reported as is.
+
+### Fixed during review
+
+Eleven findings from an adversarial review of the boundary (false REPRODUCED at a deadline kill,
+boundary crashes blamed on the agent, client mistakes invalidating experiments, malformed server
+results accepted, id reuse, pipe deadlocks, recordings over 1 MiB, path leaks), a shim deadlock, and
+a shutdown race that turned about 5% of trials into false ERRORs on a loaded machine.
+
+### Trust model
+
+For command agents the record can no longer be corrupted by accident from inside the agent's
+process. This is isolation from accident, not a sandbox: the agent still holds the socket it was
+given, and whatever it does outside MCP is invisible.
+
+### Known limits
+
+- One MCP server per trial, stdio only. No HTTP transport yet.
+- Tool replay is not agent replay: a live model rarely repeats its calls, so replaying its bundle is
+  usually INVALID. Deterministic replay is demonstrated only with scripted clients.
+- The Claude Code recipe in `docs/REAL_AGENTS.md` is an untested sketch; no paid-agent trials were
+  run for this release. The local-model results are exploratory (one machine, one small model).
+- POSIX only (unix sockets, process groups).
+
+## v0.1.0 (2026-09-21)
+
+First release. One claim: a measured regression becomes an executable, reduced failure case.
+
+### What is in it
+
+- Frozen, sealed experiments (`Manifest`), a deterministic paired schedule, one fresh process per
+  trial with a separate grader process, a single parent writer for `events.jsonl` and `trials.jsonl`.
+- The tool boundary (`ToolBox`): budgets, seeded fault injection (`tool_timeout`, `tool_error_once`,
+  `empty_result`, or your own `"module:function"`), record and replay.
+- The gate: exact Clopper-Pearson at tail `alpha / (4K)`, Bonferroni bounds on the difference,
+  verdicts PASS / BLOCK / INCONCLUSIVE / ERROR with exit codes 0 / 1 / 2 / 3. It never raises.
+- `bench/selfcheck.py`: the gate's own operating characteristics by exact enumeration.
+- Boundary-aligned trace divergence, ddmin fault minimisation with an honest 1-minimal claim,
+  portable replay bundles with hash-verified embedded code.
+- CLI (`arci run | gate | report | bundle | replay | diff | minimize`), Markdown and JUnit reports,
+  a composite GitHub Action.
+- `examples/retry_agent`: agents A, B (A minus one retry) and C (the repair), with a six-step demo.
+
+### How it was checked
+
+229 tests, basedpyright strict, CI on Python 3.11 and 3.14. The acceptance suite was written before
+the implementation and is hash-pinned (`FROZEN.sha256`). An adversarial reviewer attacked the design and
+the code in four rounds, with reproductions: about 45 findings. They were fixed, or, where out of
+scope (a hostile in-process agent, detached descendants), documented as non-goals. The behavioural
+fixes have frozen regression tests (`tests/acceptance/test_hardening*.py` and the gate, replay and
+schema suites); documentation corrections and the protocol queue bound do not. Every published
+calibration number was recomputed independently by the reviewer and matched.
+
+### Known limits and residual risks
+
+- Trust model: the agent is assumed buggy, not hostile. It shares a process with the tool boundary,
+  so a deliberately hostile agent can forge its result. See `docs/TRUST_MODEL.md`.
+- Only the tool boundary is observed. Direct file, network or subprocess use by the agent is
+  invisible. Descendants that start their own session survive cleanup.
+- The gate is conservative by design: mid-range success rates need large N to reach PASS, and
+  N=20 decides almost nothing. `prior_runs` is author-supplied; no cross-run error budget is enforced.
+- Supported statistics range: `alpha` in [1e-6, 0.5] with `alpha >= K x 1e-6`, `n_per_arm <= 10000`.
+- Bundles embed only the files you include; the minimiser's output embeds none.
+- The final reviewer re-check ran on Python 3.13 in a read-only sandbox, so writing bundle payloads
+  to disk was exercised by the test suite and CI (3.11, 3.14) but not by that review.
+- POSIX only (process groups, `ps` in tests). Windows is untested.

agent_reliability_ci-0.8.0/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 Ajay Surya Senthilrajan
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

agent_reliability_ci-0.8.0/NOTICE

@@ -0,0 +1,9 @@
+agent-reliability-ci
+Copyright 2026 Ajay Surya Senthilrajan
+
+This product includes software adapted from evalopt-graph
+(https://github.com/ajaysurya1221/evalopt-graph), Copyright (c) 2026
+Ajay Surya Senthilrajan, MIT License: the canonical JSON hashing helpers in
+src/arci/hashing.py (from src/evalopt_graph/attestation.py) and the
+self-sealing decision pattern in src/arci/schema.py (from
+src/evalopt_graph/kernel.py).