agentcrash-core 0.2.0__tar.gz → 0.3.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- agentcrash_core-0.3.0/.github/workflows/release.yml +96 -0
- agentcrash_core-0.3.0/AGENTS.md +5 -0
- agentcrash_core-0.3.0/CHANGELOG.md +83 -0
- agentcrash_core-0.3.0/LICENSE +21 -0
- agentcrash_core-0.3.0/PKG-INFO +173 -0
- agentcrash_core-0.3.0/README.md +140 -0
- agentcrash_core-0.3.0/bench/run_overhead.py +164 -0
- agentcrash_core-0.3.0/docs/adlc/01-planning-and-feasibility.md +92 -0
- agentcrash_core-0.3.0/docs/adlc/02-prd-and-project-brief.md +103 -0
- agentcrash_core-0.3.0/docs/adlc/03-requirements-and-srs.md +238 -0
- agentcrash_core-0.3.0/docs/adlc/04-glossary-and-domain-model.md +175 -0
- agentcrash_core-0.3.0/docs/adlc/05-system-design.md +235 -0
- agentcrash_core-0.3.0/docs/adlc/06-data-design.md +257 -0
- agentcrash_core-0.3.0/docs/adlc/07-api-contracts.md +380 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/docs/adlc/08-adr-template.md +20 -64
- agentcrash_core-0.3.0/docs/adlc/09-agent-context.md +237 -0
- agentcrash_core-0.3.0/docs/adlc/10-testing-strategy.md +174 -0
- agentcrash_core-0.3.0/docs/adlc/11-security-and-compliance.md +177 -0
- agentcrash_core-0.3.0/docs/adlc/12-non-functional-requirements.md +158 -0
- agentcrash_core-0.3.0/docs/adlc/13-contributing-and-workflow.md +215 -0
- agentcrash_core-0.3.0/docs/adlc/14-runbook.md +214 -0
- agentcrash_core-0.3.0/docs/adlc/15-roadmap-and-changelog.md +127 -0
- agentcrash_core-0.3.0/docs/adlc/README.md +137 -0
- agentcrash_core-0.3.0/docs/architecture-decoupling.md +76 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/docs/overview.md +1 -1
- agentcrash_core-0.3.0/docs/phase7-evidence/bench.json +18 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/docs/phase7-evidence/cost-sheet.md +1 -1
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/docs/plans/2026-10-29-agentcrash-implementation.md +14 -82
- agentcrash_core-0.3.0/docs/plans/2026-11-14-sota-roadmap.md +113 -0
- agentcrash_core-0.3.0/docs/setup.md +137 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/examples/README.md +9 -5
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/examples/minimal_agent/app.py +1 -1
- agentcrash_core-0.3.0/pyproject.toml +69 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/src/agentcrash/__init__.py +8 -6
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/src/agentcrash/_registry.py +1 -1
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/src/agentcrash/cli.py +57 -4
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/src/agentcrash/injector.py +12 -21
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/src/agentcrash/invariants.py +1 -1
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/src/agentcrash/lab/index.html +1 -1
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/src/agentcrash/ledger.py +2 -2
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/src/agentcrash/policy.py +18 -104
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/src/agentcrash/sdk.py +143 -54
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/src/agentcrash/server.py +34 -19
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/src/agentcrash/tracer.py +42 -20
- agentcrash_core-0.3.0/src/agentcrash/verifiers.py +289 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/tests/test_cli_help.py +1 -0
- agentcrash_core-0.3.0/tests/test_gaps.py +365 -0
- agentcrash_core-0.3.0/tests/test_gaps_2.py +546 -0
- agentcrash_core-0.3.0/tests/test_generic_core.py +161 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/tests/test_lab.py +5 -5
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/tests/test_ledger.py +15 -15
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/tests/test_sdk.py +13 -3
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/tests/test_second_app.py +7 -11
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/tests/test_server_ingestion.py +33 -5
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/tests/test_tracer.py +7 -7
- agentcrash_core-0.3.0/uv.lock +803 -0
- agentcrash_core-0.2.0/AGENTS.md +0 -3
- agentcrash_core-0.2.0/PKG-INFO +0 -106
- agentcrash_core-0.2.0/README.md +0 -81
- agentcrash_core-0.2.0/bench/run_overhead.py +0 -125
- agentcrash_core-0.2.0/docs/adlc/01-planning-and-feasibility.md +0 -181
- agentcrash_core-0.2.0/docs/adlc/02-prd-and-project-brief.md +0 -154
- agentcrash_core-0.2.0/docs/adlc/03-requirements-and-srs.md +0 -182
- agentcrash_core-0.2.0/docs/adlc/04-glossary-and-domain-model.md +0 -159
- agentcrash_core-0.2.0/docs/adlc/05-system-design.md +0 -194
- agentcrash_core-0.2.0/docs/adlc/06-data-design.md +0 -216
- agentcrash_core-0.2.0/docs/adlc/07-api-contracts.md +0 -229
- agentcrash_core-0.2.0/docs/adlc/09-agent-context.md +0 -249
- agentcrash_core-0.2.0/docs/adlc/10-testing-strategy.md +0 -200
- agentcrash_core-0.2.0/docs/adlc/11-security-and-compliance.md +0 -160
- agentcrash_core-0.2.0/docs/adlc/12-non-functional-requirements.md +0 -155
- agentcrash_core-0.2.0/docs/adlc/13-contributing-and-workflow.md +0 -167
- agentcrash_core-0.2.0/docs/adlc/14-runbook.md +0 -228
- agentcrash_core-0.2.0/docs/adlc/15-roadmap-and-changelog.md +0 -159
- agentcrash_core-0.2.0/docs/adlc/README.md +0 -159
- agentcrash_core-0.2.0/docs/architecture-decoupling.md +0 -109
- agentcrash_core-0.2.0/docs/nl-to-odrl-sequence.html +0 -14998
- agentcrash_core-0.2.0/docs/phase7-evidence/bench.json +0 -12
- agentcrash_core-0.2.0/docs/phase7-evidence/demo-bundle/README.md +0 -34
- agentcrash_core-0.2.0/docs/phase7-evidence/demo-bundle/fixed-policy.yaml +0 -15
- agentcrash_core-0.2.0/docs/phase7-evidence/demo-bundle/post-fix-state.json +0 -1
- agentcrash_core-0.2.0/docs/phase7-evidence/demo-bundle/post-fix-trace.jsonl +0 -11
- agentcrash_core-0.2.0/docs/phase7-evidence/demo-bundle/post-fix-verdict.json +0 -23
- agentcrash_core-0.2.0/docs/phase7-evidence/demo-bundle/pre-fix-state.json +0 -1
- agentcrash_core-0.2.0/docs/phase7-evidence/demo-bundle/pre-fix-trace.jsonl +0 -13
- agentcrash_core-0.2.0/docs/phase7-evidence/demo-bundle/pre-fix-verdict.json +0 -23
- agentcrash_core-0.2.0/docs/phase7-evidence/rehearsals.md +0 -64
- agentcrash_core-0.2.0/docs/plans/2026-11-01-next-steps.md +0 -74
- agentcrash_core-0.2.0/docs/setup.md +0 -151
- agentcrash_core-0.2.0/pyproject.toml +0 -47
- agentcrash_core-0.2.0/src/agentcrash/db.py +0 -411
- agentcrash_core-0.2.0/src/agentcrash/verifiers.py +0 -216
- agentcrash_core-0.2.0/tests/fixtures/odrl_nl.yaml +0 -85
- agentcrash_core-0.2.0/tests/fixtures/traces/odrl_convert_timeout-fixed-seed42.jsonl +0 -15
- agentcrash_core-0.2.0/tests/fixtures/traces/odrl_convert_timeout-fixed-seed42.state.json +0 -1
- agentcrash_core-0.2.0/tests/fixtures/traces/odrl_convert_timeout-naive-seed42.jsonl +0 -13
- agentcrash_core-0.2.0/tests/fixtures/traces/odrl_convert_timeout-naive-seed42.state.json +0 -1
- agentcrash_core-0.2.0/uv.lock +0 -2908
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/.github/workflows/ci.yml +0 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/.gitignore +0 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/docs/phase7-evidence/regression-report.json +0 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/src/agentcrash/__main__.py +0 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/src/agentcrash/clock.py +0 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/src/agentcrash/evaluate.py +0 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/src/agentcrash/events.py +0 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/src/agentcrash/lab/app.js +0 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/src/agentcrash/lab/styles.css +0 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/src/agentcrash/scan_pii.py +0 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/src/agentcrash/schemas/policy-patch.json +0 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/src/agentcrash/schemas/policy.json +0 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/tests/test_acr001.py +0 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/tests/test_clock.py +0 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/tests/test_no_static_bundle.py +0 -0
- {agentcrash_core-0.2.0 → agentcrash_core-0.3.0}/tests/test_skeleton.py +0 -0
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
name: release
|
|
2
|
+
|
|
3
|
+
# Publishes agentcrash-core to PyPI when a version tag is pushed:
|
|
4
|
+
# git tag v0.3.0 && git push origin v0.3.0
|
|
5
|
+
# Uses PyPI trusted publishing (OIDC): no API token is stored in GitHub.
|
|
6
|
+
# One-time setup (PyPI trusted publisher + GitHub environment "pypi") is in
|
|
7
|
+
# docs/adlc/13-contributing-and-workflow.md, section 8.
|
|
8
|
+
|
|
9
|
+
on:
|
|
10
|
+
push:
|
|
11
|
+
tags: ["v*"]
|
|
12
|
+
|
|
13
|
+
permissions:
|
|
14
|
+
contents: read
|
|
15
|
+
|
|
16
|
+
concurrency:
|
|
17
|
+
group: release-${{ github.ref }}
|
|
18
|
+
cancel-in-progress: false
|
|
19
|
+
|
|
20
|
+
jobs:
|
|
21
|
+
gates:
|
|
22
|
+
name: Gates (py${{ matrix.python-version }})
|
|
23
|
+
runs-on: ubuntu-latest
|
|
24
|
+
strategy:
|
|
25
|
+
fail-fast: true
|
|
26
|
+
matrix:
|
|
27
|
+
python-version: ["3.11", "3.12", "3.13"]
|
|
28
|
+
steps:
|
|
29
|
+
- uses: actions/checkout@v4
|
|
30
|
+
- uses: actions/setup-python@v5
|
|
31
|
+
with:
|
|
32
|
+
python-version: ${{ matrix.python-version }}
|
|
33
|
+
- uses: astral-sh/setup-uv@v5
|
|
34
|
+
- name: Install (with dev extras)
|
|
35
|
+
run: uv sync --extra dev
|
|
36
|
+
- name: Lint
|
|
37
|
+
run: uv run --no-sync ruff check .
|
|
38
|
+
- name: Format
|
|
39
|
+
run: uv run --no-sync black --check .
|
|
40
|
+
- name: Tests
|
|
41
|
+
run: uv run --no-sync pytest -q
|
|
42
|
+
|
|
43
|
+
build:
|
|
44
|
+
name: Build sdist + wheel
|
|
45
|
+
needs: gates
|
|
46
|
+
runs-on: ubuntu-latest
|
|
47
|
+
steps:
|
|
48
|
+
- uses: actions/checkout@v4
|
|
49
|
+
- uses: actions/setup-python@v5
|
|
50
|
+
with:
|
|
51
|
+
python-version: "3.12"
|
|
52
|
+
- uses: astral-sh/setup-uv@v5
|
|
53
|
+
- name: Tag must match the package version
|
|
54
|
+
run: |
|
|
55
|
+
tag="${GITHUB_REF_NAME#v}"
|
|
56
|
+
pyproject=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])")
|
|
57
|
+
module=$(python -c "import re; print(re.search(r'__version__ = \"([^\"]+)\"', open('src/agentcrash/__init__.py').read()).group(1))")
|
|
58
|
+
echo "tag=$tag pyproject=$pyproject module=$module"
|
|
59
|
+
if [ "$tag" != "$pyproject" ] || [ "$tag" != "$module" ]; then
|
|
60
|
+
echo "::error::tag v$tag does not match pyproject ($pyproject) and __version__ ($module)"
|
|
61
|
+
exit 1
|
|
62
|
+
fi
|
|
63
|
+
- name: Build
|
|
64
|
+
run: |
|
|
65
|
+
uv build --out-dir dist
|
|
66
|
+
rm -f dist/.gitignore # uv writes one; it is not a distribution
|
|
67
|
+
ls -l dist
|
|
68
|
+
- name: Check metadata
|
|
69
|
+
run: uvx twine check --strict dist/*.whl dist/*.tar.gz
|
|
70
|
+
- name: Smoke-test the wheel in a clean venv
|
|
71
|
+
run: |
|
|
72
|
+
uv venv --python 3.12 /tmp/smoke
|
|
73
|
+
uv pip install --python /tmp/smoke/bin/python dist/*.whl
|
|
74
|
+
/tmp/smoke/bin/python -m agentcrash --version
|
|
75
|
+
/tmp/smoke/bin/python -c "import agentcrash, sys; assert agentcrash.__version__ == '${GITHUB_REF_NAME#v}'; print('import ok')"
|
|
76
|
+
- uses: actions/upload-artifact@v4
|
|
77
|
+
with:
|
|
78
|
+
name: dist
|
|
79
|
+
path: dist/
|
|
80
|
+
if-no-files-found: error
|
|
81
|
+
|
|
82
|
+
publish:
|
|
83
|
+
name: Publish to PyPI
|
|
84
|
+
needs: build
|
|
85
|
+
runs-on: ubuntu-latest
|
|
86
|
+
environment:
|
|
87
|
+
name: pypi
|
|
88
|
+
url: https://pypi.org/project/agentcrash-core/
|
|
89
|
+
permissions:
|
|
90
|
+
id-token: write # OIDC token for PyPI trusted publishing; nothing else
|
|
91
|
+
steps:
|
|
92
|
+
- uses: actions/download-artifact@v4
|
|
93
|
+
with:
|
|
94
|
+
name: dist
|
|
95
|
+
path: dist/
|
|
96
|
+
- uses: pypa/gh-action-pypi-publish@release/v1
|
|
@@ -0,0 +1,5 @@
|
|
|
1
|
+
# Agent instructions pointer
|
|
2
|
+
|
|
3
|
+
Canonical docs live in `docs/`. New agent sessions: start with `docs/overview.md`, then `docs/architecture-decoupling.md` (core vs host boundary), then the roadmap in `docs/plans/2026-11-14-sota-roadmap.md` (draft status — read it for direction, but do not implement until the owner replies `go`). `docs/adlc/` and `docs/plans/2026-10-29-agentcrash-implementation.md` are the historical lifecycle record.
|
|
4
|
+
|
|
5
|
+
The core package must stay domain-free: `tests/test_generic_core.py` enforces it.
|
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to `agentcrash-core`. Versions follow [SemVer](https://semver.org/);
|
|
4
|
+
while the version is `0.x`, minor releases may break the API.
|
|
5
|
+
|
|
6
|
+
## 0.3.0 — unreleased
|
|
7
|
+
|
|
8
|
+
Chaos engineering for any agentic workflow: the core no longer ships domain code.
|
|
9
|
+
|
|
10
|
+
### Breaking changes and how to migrate
|
|
11
|
+
|
|
12
|
+
| Removed | Why | Do this instead |
|
|
13
|
+
|---|---|---|
|
|
14
|
+
| `agentcrash.db` | Its tables were shaped for one demo app (`nl_text`, `odrl_docs`) and nothing in core used it | Use the control plane (`agentcrash serve`, `/v1/runs`, `/v1/events`) or persist `ac.trace(run_id)` yourself |
|
|
15
|
+
| `agentcrash.policy.reference_patch_for` and `reference_patch_s4_*` | Hard-coded fixes for one demo's scenarios | Build a `policy_patch/v1` dict for your workflow and apply it with `policy.apply_patch`; or opt in to model-proposed patches via `fetch_live_patch` |
|
|
16
|
+
| `agentcrash.verifiers.SAVE_ODRL` | Demo tool name | `FreshWriteOnly(tools=["your_store_tool"])` |
|
|
17
|
+
| `host-odrl` extra (`pyshacl`, `dspy`, `numpy`) | Domain stacks do not belong in core | Depend on them directly in your own package |
|
|
18
|
+
| Fault `default_params` naming a tool | Silently targeted a demo tool | Name the tool yourself: `{"tool": "payments.charge", ...}`; `concurrent_update` now requires `key` and `updates` |
|
|
19
|
+
| Exporter de-duplication by `event_id` | Unbounded memory in long-running hosts | Nothing to do: the server de-duplicates on ingestion |
|
|
20
|
+
|
|
21
|
+
`LLAMA_BASE_URL` / `LLAMA_API_KEY` keep working; the preferred names are
|
|
22
|
+
`AGENTCRASH_LLM_BASE_URL` / `AGENTCRASH_LLM_API_KEY`.
|
|
23
|
+
|
|
24
|
+
### Added
|
|
25
|
+
|
|
26
|
+
- `.github/workflows/release.yml`: on a `v*` tag, run the gates (py3.11-3.13), check the
|
|
27
|
+
tag matches the package version, build sdist + wheel, `twine check --strict`,
|
|
28
|
+
smoke-install the wheel, and publish to PyPI via trusted publishing (OIDC, no token).
|
|
29
|
+
- `verify_sdk_chain(events)` and `agentcrash check --verify-chain` (SDK runs and
|
|
30
|
+
`TraceWriter` files; a broken chain exits 3).
|
|
31
|
+
- `check --invariant fresh_write_only:<tool>[,<tool>]` names your store tools on the CLI.
|
|
32
|
+
- `agentcrash.verifiers.tool_results(trace)`: one normalized view of tool calls from
|
|
33
|
+
either trace format, for writing your own invariants.
|
|
34
|
+
- `tests/test_gaps_2.py`: workflow error path, `enabled=False`, event validation and
|
|
35
|
+
invalid ingestion, `FaultSchedule` refusal and matching, `fault_injection=False`,
|
|
36
|
+
`run_verifiers` error paths, runtime verifier purity, FastAPI bearer auth, Lab
|
|
37
|
+
traversal, `load_policy`, `diff_policies`, live-patch transport and output gating.
|
|
38
|
+
Every FR in doc 03 now cites a test. Dev extra adds `httpx2` (FastAPI test client).
|
|
39
|
+
- `tests/test_gaps.py`: `--scan-pii`, card redaction, `NoLostUpdate`, `check` exit codes
|
|
40
|
+
2 and 3, patch rejection, report byte-stability.
|
|
41
|
+
- `agentcrash check` registers the built-in invariants (`duplicate_mutating_effect`,
|
|
42
|
+
`fresh_write_only`, `no_lost_update`, `required_step:<name>`), so it works without plugins.
|
|
43
|
+
- `FreshWriteOnly(tools=...)` for your own versioned-store tools.
|
|
44
|
+
- MIT `LICENSE`; PyPI classifiers, keywords and project URLs.
|
|
45
|
+
- `tests/test_generic_core.py`: fails if domain vocabulary returns to `src/`.
|
|
46
|
+
|
|
47
|
+
### Fixed
|
|
48
|
+
|
|
49
|
+
- `fresh_write_only` and `no_lost_update` only understood `TraceWriter` envelopes, so on
|
|
50
|
+
SDK traces they passed with nothing to check. Both now read SDK events too
|
|
51
|
+
(`kind: tool` / `kind: fault`); `fresh_write_only` says `nothing to judge (vacuous)`
|
|
52
|
+
when no write matched, and `no_lost_update` raises (exit 3) instead of passing when a
|
|
53
|
+
rival writer exists but `--state` has no `record_versions`.
|
|
54
|
+
- SDK `seq` and the hash chain were per `AgentCrash` instance, so one run's trace could not
|
|
55
|
+
be verified alone. Both are now per run (`seq` from 1, chain from `GENESIS`).
|
|
56
|
+
Behavior change: `seq` restarts at 1 in every run.
|
|
57
|
+
- `maybe_fault` always checked call 1, so `occurrence: 2+` never fired. It now counts
|
|
58
|
+
calls per boundary per run (`occurrence: N` fires on exactly the Nth call); pass
|
|
59
|
+
`call_index` to override. Behavior change: an `occurrence: 1` fault no longer fires
|
|
60
|
+
on every call, only the first.
|
|
61
|
+
- Stdlib server lacked `GET /v1/runs/{id}/evaluation`; FastAPI app lacked
|
|
62
|
+
`GET /v1/reliability/{project}` and returned a different `GET /v1/runs` row shape.
|
|
63
|
+
Both servers now expose the same routes and rows.
|
|
64
|
+
|
|
65
|
+
- Tracer redacted its own `prev_hash`/`event_id` fields; a hash with 13+ digits in a row
|
|
66
|
+
could be rewritten as a "card number" and break the chain.
|
|
67
|
+
- SDK hash chain could fork under concurrent steps (`prev_hash` updated outside the lock).
|
|
68
|
+
- `record_tool` raised `TypeError` into host code for non-JSON outputs (objects, dates).
|
|
69
|
+
- Server reset the connection on 401 before reading the body (Windows; flaky auth test).
|
|
70
|
+
- Bench crashed on Windows (unclosed trace files).
|
|
71
|
+
|
|
72
|
+
### Performance
|
|
73
|
+
|
|
74
|
+
- Tracer: ~75 → ~31 µs per event (compiled key matcher, cached key checks, cheap
|
|
75
|
+
pre-checks before the PII regexes, one serialization per event).
|
|
76
|
+
- SDK: ~14 µs per event (`deque` exporter, prebuilt encoder, no redundant
|
|
77
|
+
per-event validation).
|
|
78
|
+
- `bench/run_overhead.py` now measures NFR-PERF-01 as written (per-run overhead vs a bare
|
|
79
|
+
12-step agent loop): 5.7% at a 2 ms stub step, target ≤ 10%.
|
|
80
|
+
|
|
81
|
+
## 0.2.0
|
|
82
|
+
|
|
83
|
+
First installable split of the SDK, control plane and `check` CLI.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Sahil Satyam
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,173 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: agentcrash-core
|
|
3
|
+
Version: 0.3.0
|
|
4
|
+
Summary: Chaos engineering for tool-using AI agents: deterministic fault injection, hash-chained traces, invariant verdicts, and policy replay.
|
|
5
|
+
Project-URL: Homepage, https://github.com/SahilSatyam/AgentCrash
|
|
6
|
+
Project-URL: Issues, https://github.com/SahilSatyam/AgentCrash/issues
|
|
7
|
+
Author: Sahil Satyam
|
|
8
|
+
License-Expression: MIT
|
|
9
|
+
License-File: LICENSE
|
|
10
|
+
Keywords: agentic-workflows,ai-agents,chaos-engineering,evaluation,fault-injection,llm,reliability
|
|
11
|
+
Classifier: Development Status :: 3 - Alpha
|
|
12
|
+
Classifier: Intended Audience :: Developers
|
|
13
|
+
Classifier: Programming Language :: Python :: 3
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
17
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
18
|
+
Classifier: Topic :: Software Development :: Testing
|
|
19
|
+
Requires-Python: >=3.11
|
|
20
|
+
Requires-Dist: jsonschema
|
|
21
|
+
Requires-Dist: pyyaml
|
|
22
|
+
Provides-Extra: dev
|
|
23
|
+
Requires-Dist: black; extra == 'dev'
|
|
24
|
+
Requires-Dist: fastapi; extra == 'dev'
|
|
25
|
+
Requires-Dist: httpx2; extra == 'dev'
|
|
26
|
+
Requires-Dist: pytest; extra == 'dev'
|
|
27
|
+
Requires-Dist: ruff; extra == 'dev'
|
|
28
|
+
Requires-Dist: uvicorn; extra == 'dev'
|
|
29
|
+
Provides-Extra: server
|
|
30
|
+
Requires-Dist: fastapi; extra == 'server'
|
|
31
|
+
Requires-Dist: uvicorn; extra == 'server'
|
|
32
|
+
Description-Content-Type: text/markdown
|
|
33
|
+
|
|
34
|
+
# AgentCrash — chaos engineering for AI agents
|
|
35
|
+
|
|
36
|
+
[](https://github.com/SahilSatyam/AgentCrash/actions/workflows/ci.yml)
|
|
37
|
+
[](https://pypi.org/project/agentcrash-core/)
|
|
38
|
+
[](https://www.python.org/downloads/)
|
|
39
|
+
[](LICENSE)
|
|
40
|
+
|
|
41
|
+
Most agent evals grade the final answer. A tool-using agent can give a
|
|
42
|
+
correct-looking answer while it charges a customer twice, acts on stale
|
|
43
|
+
data, or follows instructions hidden in a document. AgentCrash tests the
|
|
44
|
+
**path**: it breaks your agent's tools on purpose, records every step in
|
|
45
|
+
a hash-chained trace, and judges the trace with deterministic invariants.
|
|
46
|
+
A model may suggest a fix. Only the trace decides whether it passes.
|
|
47
|
+
|
|
48
|
+
It works with any agentic workflow: plain Python, LangGraph, OpenAI
|
|
49
|
+
Agents SDK, CrewAI, MCP tools. You wrap the tool boundary; AgentCrash
|
|
50
|
+
never runs your agent for you.
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
pip install agentcrash-core
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
## 60-second example
|
|
57
|
+
|
|
58
|
+
```python
|
|
59
|
+
from agentcrash import AgentCrash, DuplicateMutatingEffect, WorkflowConfig, evaluate_trace
|
|
60
|
+
|
|
61
|
+
ac = AgentCrash(WorkflowConfig(project="checkout-agent", fault_injection=True, seed=42))
|
|
62
|
+
# The first call to payments.charge commits, then reports a timeout.
|
|
63
|
+
ac.configure_faults({"payments.charge": {"kind": "timeout_after_commit", "occurrence": 1}})
|
|
64
|
+
|
|
65
|
+
def charge(order_id: str) -> dict:
|
|
66
|
+
return {"charged": order_id} # your real side effect
|
|
67
|
+
|
|
68
|
+
with ac.run(name="checkout") as run, run.step("charge") as step:
|
|
69
|
+
result = charge("o-1")
|
|
70
|
+
fault = run.maybe_fault("payments.charge")
|
|
71
|
+
step.record_tool("payments.charge", inputs={"order": "o-1"}, outputs=result)
|
|
72
|
+
if fault: # a naive agent sees TIMEOUT and blindly retries
|
|
73
|
+
step.record_tool("payments.charge", inputs={"order": "o-1"}, outputs=charge("o-1"))
|
|
74
|
+
|
|
75
|
+
report = evaluate_trace(ac.trace(run.run_id), [DuplicateMutatingEffect()], run_id=run.run_id)
|
|
76
|
+
print(report["verdict"]) # FAIL
|
|
77
|
+
print(report["results"][0]["evidence"]) # duplicate effect(s): payments.charge at steps ...
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
Fix the retry (check status first, or send an idempotency key), run it
|
|
81
|
+
again, and the same invariant passes.
|
|
82
|
+
|
|
83
|
+
## What you get
|
|
84
|
+
|
|
85
|
+
| Part | What it does |
|
|
86
|
+
|---|---|
|
|
87
|
+
| SDK (`AgentCrash`) | `run` / `step` / `record_tool` / `maybe_fault`; sync and async; no-op when `AGENTCRASH_DISABLED=1` |
|
|
88
|
+
| Fault injection | Seeded and reproducible; fires on tool + op + Nth call. Kinds: `timeout_after_commit`, `timeout_before_commit`, `concurrent_update`, `poisoned_doc` |
|
|
89
|
+
| Traces | Versioned events, `prev_hash` chain, secrets and PII redacted at write time |
|
|
90
|
+
| Invariants | Pure `(trace, state) -> Verdict`; built-ins `required_step:<step>`, `duplicate_mutating_effect`, `fresh_write_only:<tools>`, `no_lost_update`; register your own |
|
|
91
|
+
| `agentcrash check` | Judge a JSONL trace; JSON output, exit codes, `--plugin` for your invariants, `--verify-chain`, `--scan-pii` |
|
|
92
|
+
| Recovery policy | `policy/v1` YAML + RFC 7396 merge patches; optional model-proposed patches, schema-gated |
|
|
93
|
+
| Control plane | `agentcrash serve`: `/v1` API, idempotent ingestion, Lab UI, bearer auth, retention |
|
|
94
|
+
|
|
95
|
+
## How it fits together
|
|
96
|
+
|
|
97
|
+
```mermaid
|
|
98
|
+
flowchart LR
|
|
99
|
+
H[Your agent] -->|tool calls| SDK[AgentCrash SDK]
|
|
100
|
+
F[Fault policy<br/>seeded] -.->|inject| SDK
|
|
101
|
+
SDK --> T[Hash-chained trace<br/>+ state snapshot]
|
|
102
|
+
SDK -->|batch flush| S[Control plane<br/>agentcrash serve]
|
|
103
|
+
T --> C[Invariants<br/>no model in the loop]
|
|
104
|
+
C --> V[Verdict + evidence<br/>PASS / FAIL]
|
|
105
|
+
V -.->|optional| P[Policy patch<br/>model may propose]
|
|
106
|
+
P -.->|replay| H
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
## Your own invariants
|
|
110
|
+
|
|
111
|
+
```python
|
|
112
|
+
from agentcrash.verifiers import Verdict, register_verifier
|
|
113
|
+
|
|
114
|
+
class NoRefundOverLimit:
|
|
115
|
+
id = "no_refund_over_limit"
|
|
116
|
+
|
|
117
|
+
def check(self, trace, state):
|
|
118
|
+
big = [e["step"] for e in trace
|
|
119
|
+
if e.get("tool") == "refund" and e.get("inputs", {}).get("amount", 0) > 500]
|
|
120
|
+
return Verdict(self.id, not big, f"refunds over limit at {big}" if big else "ok")
|
|
121
|
+
|
|
122
|
+
register_verifier(NoRefundOverLimit())
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
```bash
|
|
126
|
+
agentcrash check --trace run.jsonl --plugin my_agent.invariants \
|
|
127
|
+
--invariant no_refund_over_limit --format json --report-out verdict.json
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
## Install
|
|
131
|
+
|
|
132
|
+
| You want | Command |
|
|
133
|
+
|---|---|
|
|
134
|
+
| SDK + `check` | `pip install agentcrash-core` |
|
|
135
|
+
| Control-plane service | `pip install "agentcrash-core[server]"` |
|
|
136
|
+
| Contributors | `uv venv && uv pip install -e ".[dev]"` |
|
|
137
|
+
|
|
138
|
+
Python 3.11+. No API key needed; the whole test suite runs offline.
|
|
139
|
+
|
|
140
|
+
On Windows with Smart App Control or WDAC, unsigned console-script
|
|
141
|
+
launchers (`agentcrash.exe`, and any other pip/uv-installed `.exe`) may be
|
|
142
|
+
blocked. Use `python -m agentcrash ...`, which is equivalent.
|
|
143
|
+
|
|
144
|
+
Upgrading from 0.2? See [CHANGELOG.md](CHANGELOG.md) for the migration table.
|
|
145
|
+
|
|
146
|
+
## Configuration
|
|
147
|
+
|
|
148
|
+
| Variable | Effect |
|
|
149
|
+
|---|---|
|
|
150
|
+
| `AGENTCRASH_DISABLED=1` | Every SDK call becomes a no-op passthrough |
|
|
151
|
+
| `AGENTCRASH_API_TOKENS` | Comma-separated bearer tokens required on `/v1/*`; empty = loopback mode |
|
|
152
|
+
| `AGENTCRASH_LLM_BASE_URL` | Opt-in: send failing traces to an OpenAI-compatible `/chat/completions` endpoint for a candidate policy patch |
|
|
153
|
+
| `AGENTCRASH_LLM_API_KEY` | Bearer token for that call; sent as a header only, never traced |
|
|
154
|
+
|
|
155
|
+
`LLAMA_BASE_URL` / `LLAMA_API_KEY` still work as fallbacks.
|
|
156
|
+
|
|
157
|
+
## Develop
|
|
158
|
+
|
|
159
|
+
```bash
|
|
160
|
+
git clone https://github.com/SahilSatyam/AgentCrash.git && cd AgentCrash
|
|
161
|
+
uv venv && uv pip install -e ".[dev]"
|
|
162
|
+
pytest -q && ruff check . && black --check .
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
## Layout
|
|
166
|
+
|
|
167
|
+
- `src/agentcrash/`: `sdk`, `events`, `injector`, `tracer`, `clock`,
|
|
168
|
+
`invariants`, `verifiers`, `evaluate`, `policy`, `ledger`, `server`,
|
|
169
|
+
`cli`, `scan_pii`, `lab/`, `schemas/`
|
|
170
|
+
- `examples/`: reference integrations (`minimal_agent/`)
|
|
171
|
+
- `docs/`: [overview](docs/overview.md), [setup](docs/setup.md),
|
|
172
|
+
[architecture](docs/architecture-decoupling.md),
|
|
173
|
+
[roadmap](docs/plans/2026-11-14-sota-roadmap.md)
|
|
@@ -0,0 +1,140 @@
|
|
|
1
|
+
# AgentCrash — chaos engineering for AI agents
|
|
2
|
+
|
|
3
|
+
[](https://github.com/SahilSatyam/AgentCrash/actions/workflows/ci.yml)
|
|
4
|
+
[](https://pypi.org/project/agentcrash-core/)
|
|
5
|
+
[](https://www.python.org/downloads/)
|
|
6
|
+
[](LICENSE)
|
|
7
|
+
|
|
8
|
+
Most agent evals grade the final answer. A tool-using agent can give a
|
|
9
|
+
correct-looking answer while it charges a customer twice, acts on stale
|
|
10
|
+
data, or follows instructions hidden in a document. AgentCrash tests the
|
|
11
|
+
**path**: it breaks your agent's tools on purpose, records every step in
|
|
12
|
+
a hash-chained trace, and judges the trace with deterministic invariants.
|
|
13
|
+
A model may suggest a fix. Only the trace decides whether it passes.
|
|
14
|
+
|
|
15
|
+
It works with any agentic workflow: plain Python, LangGraph, OpenAI
|
|
16
|
+
Agents SDK, CrewAI, MCP tools. You wrap the tool boundary; AgentCrash
|
|
17
|
+
never runs your agent for you.
|
|
18
|
+
|
|
19
|
+
```bash
|
|
20
|
+
pip install agentcrash-core
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
## 60-second example
|
|
24
|
+
|
|
25
|
+
```python
|
|
26
|
+
from agentcrash import AgentCrash, DuplicateMutatingEffect, WorkflowConfig, evaluate_trace
|
|
27
|
+
|
|
28
|
+
ac = AgentCrash(WorkflowConfig(project="checkout-agent", fault_injection=True, seed=42))
|
|
29
|
+
# The first call to payments.charge commits, then reports a timeout.
|
|
30
|
+
ac.configure_faults({"payments.charge": {"kind": "timeout_after_commit", "occurrence": 1}})
|
|
31
|
+
|
|
32
|
+
def charge(order_id: str) -> dict:
|
|
33
|
+
return {"charged": order_id} # your real side effect
|
|
34
|
+
|
|
35
|
+
with ac.run(name="checkout") as run, run.step("charge") as step:
|
|
36
|
+
result = charge("o-1")
|
|
37
|
+
fault = run.maybe_fault("payments.charge")
|
|
38
|
+
step.record_tool("payments.charge", inputs={"order": "o-1"}, outputs=result)
|
|
39
|
+
if fault: # a naive agent sees TIMEOUT and blindly retries
|
|
40
|
+
step.record_tool("payments.charge", inputs={"order": "o-1"}, outputs=charge("o-1"))
|
|
41
|
+
|
|
42
|
+
report = evaluate_trace(ac.trace(run.run_id), [DuplicateMutatingEffect()], run_id=run.run_id)
|
|
43
|
+
print(report["verdict"]) # FAIL
|
|
44
|
+
print(report["results"][0]["evidence"]) # duplicate effect(s): payments.charge at steps ...
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
Fix the retry (check status first, or send an idempotency key), run it
|
|
48
|
+
again, and the same invariant passes.
|
|
49
|
+
|
|
50
|
+
## What you get
|
|
51
|
+
|
|
52
|
+
| Part | What it does |
|
|
53
|
+
|---|---|
|
|
54
|
+
| SDK (`AgentCrash`) | `run` / `step` / `record_tool` / `maybe_fault`; sync and async; no-op when `AGENTCRASH_DISABLED=1` |
|
|
55
|
+
| Fault injection | Seeded and reproducible; fires on tool + op + Nth call. Kinds: `timeout_after_commit`, `timeout_before_commit`, `concurrent_update`, `poisoned_doc` |
|
|
56
|
+
| Traces | Versioned events, `prev_hash` chain, secrets and PII redacted at write time |
|
|
57
|
+
| Invariants | Pure `(trace, state) -> Verdict`; built-ins `required_step:<step>`, `duplicate_mutating_effect`, `fresh_write_only:<tools>`, `no_lost_update`; register your own |
|
|
58
|
+
| `agentcrash check` | Judge a JSONL trace; JSON output, exit codes, `--plugin` for your invariants, `--verify-chain`, `--scan-pii` |
|
|
59
|
+
| Recovery policy | `policy/v1` YAML + RFC 7396 merge patches; optional model-proposed patches, schema-gated |
|
|
60
|
+
| Control plane | `agentcrash serve`: `/v1` API, idempotent ingestion, Lab UI, bearer auth, retention |
|
|
61
|
+
|
|
62
|
+
## How it fits together
|
|
63
|
+
|
|
64
|
+
```mermaid
|
|
65
|
+
flowchart LR
|
|
66
|
+
H[Your agent] -->|tool calls| SDK[AgentCrash SDK]
|
|
67
|
+
F[Fault policy<br/>seeded] -.->|inject| SDK
|
|
68
|
+
SDK --> T[Hash-chained trace<br/>+ state snapshot]
|
|
69
|
+
SDK -->|batch flush| S[Control plane<br/>agentcrash serve]
|
|
70
|
+
T --> C[Invariants<br/>no model in the loop]
|
|
71
|
+
C --> V[Verdict + evidence<br/>PASS / FAIL]
|
|
72
|
+
V -.->|optional| P[Policy patch<br/>model may propose]
|
|
73
|
+
P -.->|replay| H
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
## Your own invariants
|
|
77
|
+
|
|
78
|
+
```python
|
|
79
|
+
from agentcrash.verifiers import Verdict, register_verifier
|
|
80
|
+
|
|
81
|
+
class NoRefundOverLimit:
|
|
82
|
+
id = "no_refund_over_limit"
|
|
83
|
+
|
|
84
|
+
def check(self, trace, state):
|
|
85
|
+
big = [e["step"] for e in trace
|
|
86
|
+
if e.get("tool") == "refund" and e.get("inputs", {}).get("amount", 0) > 500]
|
|
87
|
+
return Verdict(self.id, not big, f"refunds over limit at {big}" if big else "ok")
|
|
88
|
+
|
|
89
|
+
register_verifier(NoRefundOverLimit())
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
```bash
|
|
93
|
+
agentcrash check --trace run.jsonl --plugin my_agent.invariants \
|
|
94
|
+
--invariant no_refund_over_limit --format json --report-out verdict.json
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
## Install
|
|
98
|
+
|
|
99
|
+
| You want | Command |
|
|
100
|
+
|---|---|
|
|
101
|
+
| SDK + `check` | `pip install agentcrash-core` |
|
|
102
|
+
| Control-plane service | `pip install "agentcrash-core[server]"` |
|
|
103
|
+
| Contributors | `uv venv && uv pip install -e ".[dev]"` |
|
|
104
|
+
|
|
105
|
+
Python 3.11+. No API key needed; the whole test suite runs offline.
|
|
106
|
+
|
|
107
|
+
On Windows with Smart App Control or WDAC, unsigned console-script
|
|
108
|
+
launchers (`agentcrash.exe`, and any other pip/uv-installed `.exe`) may be
|
|
109
|
+
blocked. Use `python -m agentcrash ...`, which is equivalent.
|
|
110
|
+
|
|
111
|
+
Upgrading from 0.2? See [CHANGELOG.md](CHANGELOG.md) for the migration table.
|
|
112
|
+
|
|
113
|
+
## Configuration
|
|
114
|
+
|
|
115
|
+
| Variable | Effect |
|
|
116
|
+
|---|---|
|
|
117
|
+
| `AGENTCRASH_DISABLED=1` | Every SDK call becomes a no-op passthrough |
|
|
118
|
+
| `AGENTCRASH_API_TOKENS` | Comma-separated bearer tokens required on `/v1/*`; empty = loopback mode |
|
|
119
|
+
| `AGENTCRASH_LLM_BASE_URL` | Opt-in: send failing traces to an OpenAI-compatible `/chat/completions` endpoint for a candidate policy patch |
|
|
120
|
+
| `AGENTCRASH_LLM_API_KEY` | Bearer token for that call; sent as a header only, never traced |
|
|
121
|
+
|
|
122
|
+
`LLAMA_BASE_URL` / `LLAMA_API_KEY` still work as fallbacks.
|
|
123
|
+
|
|
124
|
+
## Develop
|
|
125
|
+
|
|
126
|
+
```bash
|
|
127
|
+
git clone https://github.com/SahilSatyam/AgentCrash.git && cd AgentCrash
|
|
128
|
+
uv venv && uv pip install -e ".[dev]"
|
|
129
|
+
pytest -q && ruff check . && black --check .
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
## Layout
|
|
133
|
+
|
|
134
|
+
- `src/agentcrash/`: `sdk`, `events`, `injector`, `tracer`, `clock`,
|
|
135
|
+
`invariants`, `verifiers`, `evaluate`, `policy`, `ledger`, `server`,
|
|
136
|
+
`cli`, `scan_pii`, `lab/`, `schemas/`
|
|
137
|
+
- `examples/`: reference integrations (`minimal_agent/`)
|
|
138
|
+
- `docs/`: [overview](docs/overview.md), [setup](docs/setup.md),
|
|
139
|
+
[architecture](docs/architecture-decoupling.md),
|
|
140
|
+
[roadmap](docs/plans/2026-11-14-sota-roadmap.md)
|