faultbench 0.1.0__tar.gz

faultbench-0.1.0/.env.example

@@ -0,0 +1,4 @@
+# Copy to .env. Only the example agent needs a model key; faultbench itself makes no LLM calls.
+# Pydantic AI reads the provider key for whichever model you name in examples/shop/agent.py.
+OPENAI_API_KEY=
+ANTHROPIC_API_KEY=

faultbench-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,27 @@
+name: ci
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install
+        run: uv sync --all-extras
+      - name: Lint
+        run: uv run ruff check .
+      - name: Format check
+        run: uv run ruff format --check .
+      - name: Test (no API key needed)
+        run: uv run pytest -q

faultbench-0.1.0/.gitignore

@@ -0,0 +1,12 @@
+__pycache__/
+*.pyc
+.venv/
+dist/
+build/
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+.traces/
+.shop_state.json
+.env
+.DS_Store

faultbench-0.1.0/.python-version

@@ -0,0 +1 @@
+3.11

faultbench-0.1.0/CHANGELOG.md

@@ -0,0 +1,4 @@
+# Changelog
+
+## Unreleased
+- Scaffold: layout, pyproject, CLAUDE.md, architecture/decisions/status docs, smoke tests.

faultbench-0.1.0/CLAUDE.md

@@ -0,0 +1,52 @@
+# faultbench
+
+Fake, stateful worlds with fault injection for testing tool-using AI agents.
+A user declares services and records in `world.yaml`; faultbench serves them as MCP
+tools, injects latency/errors/clock skew on demand, records every call, and lets
+pytest assert on the world's end state.
+
+**Read `docs/STATUS.md` before doing anything.** It says which milestone is active
+and what the next session should do. Then read `docs/ARCHITECTURE.md` for the
+component you are touching and `docs/DECISIONS.md` so you don't re-argue settled choices.
+
+## Non-goals (do not build these)
+- No simulated user / persona engine. Users bring LangWatch Scenario, DeepEval or openevals.
+- No LLM judge or transcript scoring. Assert on world state and the trace only.
+- No dashboard or web UI. Stay inside pytest and the CLI.
+- No clever YAML. Logic goes in Python handlers, not in templates or conditionals.
+- No support for every framework at once. Pydantic AI first, one more via HTTP, then stop.
+
+## Stack
+Python 3.11+, uv, pydantic v2, official `mcp` SDK, faker, pytest, ruff. No database.
+Target: under 3,000 lines for v0.1.
+
+## Commands
+    uv sync --all-extras          # install
+    uv run pytest                 # harness unit tests
+    uv run pytest examples/shop   # example agent tests (needs a model API key)
+    uv run ruff check . && uv run ruff format .
+    uv run faultbench serve examples/shop/worlds/shop.yaml
+
+## Session protocol
+1. Read STATUS.md, ARCHITECTURE.md (relevant section), DECISIONS.md.
+2. Propose a short plan for this session's milestone; wait for approval.
+3. One component per session. Do not start the next milestone.
+4. Write tests alongside code in `tests/`.
+5. Before ending: run `uv run pytest` and `uv run ruff check .`, update STATUS.md
+   (state + "next session should"), append to DECISIONS.md if anything was decided,
+   and update docs/JOURNEY.md: flip the finished milestone to ✅ with a 2-line plain-language
+   "Built." note (what exists now, not how), move the "Now" marker to the next one. Commit.
+
+## Layout
+    src/faultbench/world/    schema.py engine.py seed.py       # YAML -> stateful tables
+    src/faultbench/faults/   profile.py injector.py clock.py   # latency, errors, fake time
+    src/faultbench/server/   mcp_server.py operations.py       # world -> MCP tools
+    src/faultbench/trace/    recorder.py queries.py            # JSONL trace of every call
+    src/faultbench/pytest_plugin.py                            # fixtures, markers, --runs
+    src/faultbench/cli.py                                      # `faultbench serve`
+    examples/shop/           the refund agent demo (world, rules, agent, tests)
+    tests/                   unit tests for faultbench itself
+    docs/                    ARCHITECTURE, DECISIONS, STATUS, user docs
+
+Background (why this exists, market landscape, 12-weekend plan):
+https://claude.ai/code/artifact/058e73f3-39fa-4d36-8f96-f1b29efe8de2

faultbench-0.1.0/LICENSE

@@ -0,0 +1,175 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.

faultbench-0.1.0/PKG-INFO

@@ -0,0 +1,137 @@
+Metadata-Version: 2.5
+Name: faultbench
+Version: 0.1.0
+Summary: Fake, stateful worlds with fault injection for testing tool-using AI agents. Exposed over MCP, asserted from pytest.
+Author: mithun
+License-Expression: Apache-2.0
+License-File: LICENSE
+Keywords: ai-agents,fault-injection,mcp,pytest,simulation,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Framework :: Pytest
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Requires-Dist: faker>=25.0
+Requires-Dist: mcp>=2.2
+Requires-Dist: pydantic>=2.7
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: examples
+Requires-Dist: pydantic-ai>=2.0; extra == 'examples'
+Provides-Extra: openai-agents
+Requires-Dist: openai-agents>=0.2; extra == 'openai-agents'
+Provides-Extra: pydantic-ai
+Requires-Dist: pydantic-ai>=2.0; extra == 'pydantic-ai'
+Description-Content-Type: text/markdown
+
+# faultbench
+
+**Fake, stateful worlds with fault injection for testing tool-using AI agents.**
+
+Declare your services in a YAML file. faultbench serves them as MCP tools your agent can call,
+makes them slow or broken on purpose, records every call, and lets you assert on the world's
+end state from pytest, twenty runs at a time.
+
+> Status: pre-alpha, but it works end to end. Write a world, serve it over MCP (stdio or
+> HTTP), inject faults, and assert on state from pytest across N seeded runs.
+> **Guides:** [write a world](docs/WORLDS.md) · [test an agent](docs/TESTING.md) ·
+> examples: [`examples/shop`](examples/shop) (refund agent), [`examples/bank`](examples/bank).
+> Validated against **Pydantic AI** and the **OpenAI Agents SDK** (any MCP framework works).
+
+## The problem
+
+Your support agent handles "return my order and refund me." It works when you test by hand.
+In production the refund API times out once, the agent retries, and a customer is refunded twice.
+You can't make the real payments API time out on command, so you never tested it.
+
+## What it looks like
+
+```yaml
+# world.yaml
+services:
+  orders:
+    records: { order: { id: str, status: enum[placed, delivered, returned], total: float } }
+    operations: { get_order: { kind: get, record: order } }
+  payments:
+    records: { refund: { id: str, order_id: str, amount: float } }
+    operations: { issue_refund: { kind: create, record: refund } }
+faults:
+  payments.issue_refund: { errors: { timeout: 0.10 } }
+```
+
+```python
+from faultbench.integrations.pydantic_ai import run_agent  # or wire any MCP framework
+
+
+@pytest.mark.world("world.yaml")
+@pytest.mark.faults("world.yaml")  # inject the faults: block above
+@pytest.mark.runs(20)
+@pytest.mark.min_pass_rate(0.95)
+async def test_refund_issued_exactly_once(world, mcp_server, trace):
+    order = world.orders.pick(status="delivered")
+    await run_agent(
+        "openai:gpt-5-mini",
+        f"Return order {order.id} and refund me",
+        mcp=mcp_server,
+        system_prompt="You are a refund agent.",
+    )
+    assert len(world.refunds.where(order_id=order.id)) == 1  # a timeout+retry breaks this
+```
+
+```
+faultbench: pass rate over runs
+test_refund_issued_exactly_once: 17/20 passed (85%)  min_pass_rate=95% -> FAIL
+    run4:  get_order → create_return → issue_refund!timeout → issue_refund
+    run11: get_order → create_return → issue_refund!timeout → issue_refund
+    run18: get_order → create_return → issue_refund!timeout → issue_refund
+```
+
+The timeout fired *after* the refund was written, the agent retried, and the customer was
+refunded twice — the production bug you couldn't trigger on the real payments API, now a red
+test with the trace that explains it.
+
+## What it can and can't model
+faultbench models services as **flat records** (fields: `str/int/float/bool/datetime/enum/ref`)
+with built-in CRUD plus **custom Python operations** for anything else.
+
+- **Fits well:** entities with enums and `ref` relationships; CRUD and list-by-field; business
+  rules, state machines, and multi-record writes as custom handlers; array *inputs* via a custom
+  op that flattens into a related record type; money as integer minor-units. (See
+  [`examples/stripe`](examples/stripe) — a Stripe-style payments API with partial-refund rules.)
+- **Caveat:** records are flat — there are **no nested objects or array fields**. Model a
+  one-to-many as a related record type + a `ref` (invoice ← line items); a GET returns the
+  parent without children inline, so if your agent's correctness depends on a nested *response*
+  shape, the fake's shape differs.
+- **Not in 0.1:** pagination/cursors, non-equality filters, auth, webhooks, per-request
+  idempotency (that last is a bug faultbench helps you *catch*, not prevent). Generated seed
+  values are type-correct but not domain-aware — set realistic values in a handler or your test.
+
+## Security
+A world file can name Python to import and run (custom handlers, `handler: module.func`), so
+loading or serving one executes that code. Only use world files you trust, like any script.
+
+## Not in scope
+Simulated users, LLM judges, dashboards. Use LangWatch Scenario / DeepEval for users and your own
+judge for scoring; faultbench is the environment.
+
+## Quickstart
+
+    uv sync --all-extras
+    uv run pytest                                             # the harness test suite (no API key)
+    uv run faultbench serve examples/shop/worlds/shop.yaml    # serve a world over MCP (stdio)
+    uv run faultbench serve examples/bank/worlds/bank.yaml --http   # ...or over HTTP
+
+Then write your own: [docs/WORLDS.md](docs/WORLDS.md) and [docs/TESTING.md](docs/TESTING.md).
+
+## Development
+See `CLAUDE.md` for the session protocol and `docs/` for architecture, decisions and status.
+
+    uv sync --all-extras
+    uv run pytest
+    uv run pytest examples/shop     # the example agent test (needs a model API key)
+
+## License
+Apache 2.0

faultbench-0.1.0/README.md

@@ -0,0 +1,108 @@
+# faultbench
+
+**Fake, stateful worlds with fault injection for testing tool-using AI agents.**
+
+Declare your services in a YAML file. faultbench serves them as MCP tools your agent can call,
+makes them slow or broken on purpose, records every call, and lets you assert on the world's
+end state from pytest, twenty runs at a time.
+
+> Status: pre-alpha, but it works end to end. Write a world, serve it over MCP (stdio or
+> HTTP), inject faults, and assert on state from pytest across N seeded runs.
+> **Guides:** [write a world](docs/WORLDS.md) · [test an agent](docs/TESTING.md) ·
+> examples: [`examples/shop`](examples/shop) (refund agent), [`examples/bank`](examples/bank).
+> Validated against **Pydantic AI** and the **OpenAI Agents SDK** (any MCP framework works).
+
+## The problem
+
+Your support agent handles "return my order and refund me." It works when you test by hand.
+In production the refund API times out once, the agent retries, and a customer is refunded twice.
+You can't make the real payments API time out on command, so you never tested it.
+
+## What it looks like
+
+```yaml
+# world.yaml
+services:
+  orders:
+    records: { order: { id: str, status: enum[placed, delivered, returned], total: float } }
+    operations: { get_order: { kind: get, record: order } }
+  payments:
+    records: { refund: { id: str, order_id: str, amount: float } }
+    operations: { issue_refund: { kind: create, record: refund } }
+faults:
+  payments.issue_refund: { errors: { timeout: 0.10 } }
+```
+
+```python
+from faultbench.integrations.pydantic_ai import run_agent  # or wire any MCP framework
+
+
+@pytest.mark.world("world.yaml")
+@pytest.mark.faults("world.yaml")  # inject the faults: block above
+@pytest.mark.runs(20)
+@pytest.mark.min_pass_rate(0.95)
+async def test_refund_issued_exactly_once(world, mcp_server, trace):
+    order = world.orders.pick(status="delivered")
+    await run_agent(
+        "openai:gpt-5-mini",
+        f"Return order {order.id} and refund me",
+        mcp=mcp_server,
+        system_prompt="You are a refund agent.",
+    )
+    assert len(world.refunds.where(order_id=order.id)) == 1  # a timeout+retry breaks this
+```
+
+```
+faultbench: pass rate over runs
+test_refund_issued_exactly_once: 17/20 passed (85%)  min_pass_rate=95% -> FAIL
+    run4:  get_order → create_return → issue_refund!timeout → issue_refund
+    run11: get_order → create_return → issue_refund!timeout → issue_refund
+    run18: get_order → create_return → issue_refund!timeout → issue_refund
+```
+
+The timeout fired *after* the refund was written, the agent retried, and the customer was
+refunded twice — the production bug you couldn't trigger on the real payments API, now a red
+test with the trace that explains it.
+
+## What it can and can't model
+faultbench models services as **flat records** (fields: `str/int/float/bool/datetime/enum/ref`)
+with built-in CRUD plus **custom Python operations** for anything else.
+
+- **Fits well:** entities with enums and `ref` relationships; CRUD and list-by-field; business
+  rules, state machines, and multi-record writes as custom handlers; array *inputs* via a custom
+  op that flattens into a related record type; money as integer minor-units. (See
+  [`examples/stripe`](examples/stripe) — a Stripe-style payments API with partial-refund rules.)
+- **Caveat:** records are flat — there are **no nested objects or array fields**. Model a
+  one-to-many as a related record type + a `ref` (invoice ← line items); a GET returns the
+  parent without children inline, so if your agent's correctness depends on a nested *response*
+  shape, the fake's shape differs.
+- **Not in 0.1:** pagination/cursors, non-equality filters, auth, webhooks, per-request
+  idempotency (that last is a bug faultbench helps you *catch*, not prevent). Generated seed
+  values are type-correct but not domain-aware — set realistic values in a handler or your test.
+
+## Security
+A world file can name Python to import and run (custom handlers, `handler: module.func`), so
+loading or serving one executes that code. Only use world files you trust, like any script.
+
+## Not in scope
+Simulated users, LLM judges, dashboards. Use LangWatch Scenario / DeepEval for users and your own
+judge for scoring; faultbench is the environment.
+
+## Quickstart
+
+    uv sync --all-extras
+    uv run pytest                                             # the harness test suite (no API key)
+    uv run faultbench serve examples/shop/worlds/shop.yaml    # serve a world over MCP (stdio)
+    uv run faultbench serve examples/bank/worlds/bank.yaml --http   # ...or over HTTP
+
+Then write your own: [docs/WORLDS.md](docs/WORLDS.md) and [docs/TESTING.md](docs/TESTING.md).
+
+## Development
+See `CLAUDE.md` for the session protocol and `docs/` for architecture, decisions and status.
+
+    uv sync --all-extras
+    uv run pytest
+    uv run pytest examples/shop     # the example agent test (needs a model API key)
+
+## License
+Apache 2.0

faultbench-0.1.0/conftest.py

@@ -0,0 +1,4 @@
+"""Repo-root conftest: enable pytest's `pytester` fixture so tests/test_runs.py can run the
+runs/pass-rate machinery in an inner pytest session."""
+
+pytest_plugins = ["pytester"]

faultbench-0.1.0/docs/ARCHITECTURE.md

@@ -0,0 +1,50 @@
+# Architecture
+
+Five components. The agent under test never knows it is talking to a fake; it sees MCP tools.
+
+    Agent --MCP--> mcp_server --> faults.injector --> world.engine
+                      |
+                      +--> trace.recorder
+    pytest_plugin ----> engine (assert state), injector (configure), recorder (read trace)
+
+## 1. World engine (`world/`)
+- `schema.py`: pydantic models for `world.yaml` (World, Service, RecordType, Field, Operation, SeedSpec).
+  Field types: `str`, `int`, `float`, `bool`, `datetime`, `enum[a, b, c]`, `ref[service.record]`.
+- `engine.py`: `World.load(path) -> World`. Builds one in-memory `Table` per record type.
+  `Table` API: `get(id)`, `all()`, `where(**eq)`, `pick(**eq)` (deterministic first match),
+  `insert(record)`, `update(id, **fields)`, `delete(id)`.
+  `World` API: `snapshot() -> dict`, `diff(snapshot) -> Diff`, `revision: int` (bumps on every write).
+- `seed.py`: faker-based generators per field type, seeded by `world.seed`. CSV/JSON loaders.
+- Invariant: same YAML + same seed => identical initial state, byte for byte.
+
+## 2. Fault layer (`faults/`)
+- `profile.py`: pydantic models for the `faults:` block. Keyed by `service.operation` or `default`.
+  Fields: `latency_ms: [lo, hi]`, `errors: {kind: probability}`, `rate_limit: {calls, per_seconds}`,
+  conditional errors (e.g. `not_found_if_newer_than: 2h`).
+- `injector.py`: wraps an operation callable. Seeded RNG (`seed + run_index`). Error catalogue:
+  `timeout` (actually sleeps past the client deadline), `http_500`, `http_429` (with retry-after),
+  `malformed_json`, `empty_result`, `not_found`. Every decision is recorded to the trace.
+- `clock.py`: `FakeClock` with `now()`, `advance(seconds)`. The engine and injector read time only from it.
+- Invariant: same profile + same seed => identical fault sequence.
+
+## 3. MCP server (`server/`)
+- `mcp_server.py`: builds a `mcp.server.Server` from a World + FaultProfile + Recorder.
+  One tool per operation, JSON schema derived from the record type. Transports: stdio, streamable HTTP.
+- `operations.py`: built-in kinds `get`, `list`, `create`, `update`, `delete`; `custom` resolves
+  `handler: "module.func"` and calls `func(world, clock, **args)`.
+
+## 4. Trace (`trace/`)
+- `recorder.py`: JSON Lines, one event per tool call:
+  `{run, seq, ts, tool, args, fault, latency_ms, ok, result|error, world_rev}`.
+- `queries.py`: `Trace.count(tool)`, `calls_to(tool)`, `faults()`, `failures()`.
+
+## 5. pytest plugin (`pytest_plugin.py`)
+- Markers: `@pytest.mark.world("path.yaml")`, `@pytest.mark.faults("path.yaml" | dict)`,
+  `@pytest.mark.runs(N)`, `@pytest.mark.min_pass_rate(0.9)`.
+- Fixtures: `world`, `faults`, `clock`, `mcp_url` (server subprocess per test), `trace`.
+- `--runs=N` overrides the marker; report shows `runs / passed / failed / pass rate` and a
+  per-failure trace summary. `min_pass_rate` turns the rate into pass/fail for CI.
+
+## Flow of one test
+pytest builds World + FaultProfile -> starts MCP server subprocess -> yields `mcp_url`
+-> test runs the agent -> test asserts on `world` and `trace` -> server stops, world discarded.

faultbench-0.1.0/docs/DECISIONS.md

@@ -0,0 +1,51 @@
+# Decisions
+
+One line each. Append; do not rewrite history. Format: date, decision, reason.
+
+- 2026-09-19 — Name: `faultbench`. Reason: describes the thing (fake world + benchmarking); free on PyPI/GitHub at time of check.
+- 2026-09-19 — License: Apache 2.0. Reason: matches peers (Scenario, DeepEval, tau2-bench), patent grant, leaves room for a hosted product later.
+- 2026-09-19 — Language: Python 3.11+, packaged with uv. Reason: largest agent ecosystem; pytest is the natural runner.
+- 2026-09-19 — Scope: fake world + faults + trace + pytest. Explicitly NOT simulated users, judges, dashboards. Reason: those markets are full; the environment layer is empty (see market doc).
+- 2026-09-19 — Fake services are hand-written from YAML, not LLM-emulated. Reason: determinism is the feature. `kind: emulated` may come later on request.
+- 2026-09-19 — Logic lives in Python handlers (`kind: custom`), never in YAML templating. Reason: every YAML feature is something users must learn and something that breaks sharing.
+- 2026-09-19 — Server runs as a subprocess per test, stdio by default, HTTP when the agent is remote. Reason: isolation over speed at this stage.
+- 2026-09-19 — Tests report a pass rate over N seeded runs; `min_pass_rate` converts to pass/fail for CI. Reason: agents are probabilistic; a single pass/fail lies.
+- 2026-09-19 — Toy agent uses Pydantic AI. Reason: small, typed, speaks MCP natively, model-agnostic.
+- 2026-09-19 — Time is read only from `FakeClock`. Reason: "order is 2 hours old" must be testable without waiting.
+- 2026-09-19 — Installed `mcp` is 2.x (2.2.0): `FastMCP` is now `mcp.server.mcpserver.MCPServer` (`@server.tool()`, `server.run("stdio"|"streamable-http")`). Weekend 3 targets this API; `pyproject` `mcp>=1.2` should be tightened to `>=2` before the server milestone.
+- 2026-09-19 — Pydantic AI is 2.x (2.45): MCP client is `MCPToolset(FastMCPClient(StdioTransport(...)))`, not `MCPServerStdio`. Agent connects via `async with agent:`.
+- 2026-09-19 — Example agent model is set via `FAULTBENCH_DEMO_MODEL` (default `openai:gpt-5-mini`; the key on hand this session was OpenAI). Provider prefix picks the required key. Reason: keep the example key-agnostic rather than pinning one provider.
+- 2026-09-19 — Schema pydantic root is `WorldSpec` (with `ServiceSpec`/`RecordType`/`FieldType`/`OperationSpec`/`SeedSpec`), not `World`. Reason: avoid two classes named `World` — `engine.World` is the runtime, stateful world.
+- 2026-09-19 — Tables are reached on the runtime `World` by record-type name (`world.orders`), so record names must be unique across the world (validated). `shop.yaml` records are named as collections (`orders`, `refunds`). Reason: matches the documented target test `world.orders.pick(...)` / `world.refunds.where(...)` with no pluralization magic ("no clever" rule).
+- 2026-09-19 — Seeded `datetime` fields are drawn from a FIXED absolute window (2020–2025), never a now-relative one, and stored as ISO strings. Reason: the byte-identical invariant must hold on any day; verified identical across two separate processes.
+- 2026-09-19 — Seeding does not bump `revision`; a freshly loaded world is at revision 0. Only insert/update/delete bump it. Reason: revision should count agent-caused writes, not initial state.
+- 2026-09-19 — MCP tool schemas are derived by giving a generated wrapper function an explicit `__signature__` (mcp 2.x `func_metadata` reads `inspect.signature`), one tool per operation. Reason: operations are known only at runtime from YAML; this reuses the SDK's schema generation instead of hand-building JSON schema.
+- 2026-09-19 — Tool params: get/delete → `id`; update → `id` + optional fields; create → record fields (optional, id auto-assigned); list → the operation's `filter` fields (optional); custom → the handler's params minus `world`/`clock`. Create/list fields are optional so the agent can call with only what it knows (e.g. `issue_refund(order_id, amount)`). Reason: keep the toy agent working unchanged.
+- 2026-09-19 — Business-rule failures (ValueError/LookupError/KeyError from an op or handler) are converted to MCP `ToolError`; other exceptions propagate as crashes. Reason: the agent sees a clean, readable tool error for "not found"/"not delivered" without a server traceback, while real bugs stay loud.
+- 2026-09-19 — Custom handler modules resolve by bare name; the server puts the world file's directory and its parent on `sys.path`. Reason: lets `shop.yaml` (in worlds/) reference `shop_rules` (in examples/shop/) without packaging.
+- 2026-09-19 — `examples/shop/tools_dict.py` deleted; the agent now spawns `python -m faultbench.server <world.yaml>`. Reason: the generated server replaces the milestone-1 throwaway.
+- 2026-09-19 — Fault timing models reality: `timeout` runs the op (side effect happens) THEN fails; `http_500`/`http_429`/`not_found` reject before running; `empty_result`/`malformed_json` corrupt the response without running. Reason: the double-refund bug requires the write to have happened before the client gives up and retries.
+- 2026-09-19 — Injected timeouts/errors are surfaced to the client as MCP `ToolError`, not a real transport timeout. Reason: a retrying agent re-calls the op (reproducing the double write) without tests depending on real wall-clock deadlines; verified live (agent double-refunded order 2 at run_index 4).
+- 2026-09-19 — Fault RNG is `Random(f"{seed}:{run_index}")`, one per server build; `seed` comes from the world. Reason: same profile + seed + run_index => identical fault sequence, and `run_index` walks distinct deterministic runs (for weekend-6 `--runs=N`).
+- 2026-09-19 — Conditional faults (`not_found_if_newer_than`) and rate limits are parsed but NOT enforced in weekend 4; deferred to weekend 7 with the clock/seed-window eligibility work. Reason: keep milestone 4 to probabilistic faults + the demo; shop.yaml still loads.
+- 2026-09-19 — pytest plugin fixture for the server is `mcp_server` (an in-process MCPServer sharing the `world` object), NOT `mcp_url`. Reason: our transport is stdio/in-memory, not HTTP; sharing the World object lets tests assert on state directly. `mcp_url`/HTTP is deferred to weekend 8.
+- 2026-09-19 — The `mcp_server` fixture builds with `clock=None` (return-window check off), even though a `clock` fixture exists. Reason: threading the clock activates the 30-day window, but seed dates (2020–2025) are outside 30 days of the default clock (2025-06-01), so the refund test would fail; aligning them is weekend-7 eligibility work.
+- 2026-09-19 — Example agent test (`examples/shop/test_refunds.py`) connects the agent to the fixture's in-process server via `run_agent(..., server=...)`, and lives outside `testpaths` so `uv run pytest` stays key-free; `uv run pytest examples/shop` runs it (skipped without a key, `.env` loaded by conftest). Reason: harness CI needs no API key; the real-agent test is opt-in.
+- 2026-09-19 — Each of the N runs is a normal pytest item (`run_index` parametrised 0..N-1), not an internal loop. Reason: pytest-asyncio runs each item normally, so async tests work without fighting the pyfunc_call hook.
+- 2026-09-19 — With `min_pass_rate(r)`, individual run failures are flipped to "passed" in `pytest_runtest_makereport` and the session fails (in `pytest_sessionfinish`) only if the aggregate rate < r. Without `min_pass_rate`, each run's pass/fail stands. Reason: DECISION that a single pass/fail lies — the aggregate is the CI verdict; but a strict test (no marker) should still go red on any failure.
+- 2026-09-19 — Trace event per call carries the injector's decision via a `call(..., on_decision=)` hook; `build_server(..., recorder=)` owns writing the full event (adds ok/result-or-error/world_rev). Reason: one event per call assembled server-side, since fault/latency are known before the op and result/rev after.
+- 2026-09-19 — Seed datetimes fall in the 20 days ending at `WORLD_EPOCH` (2025-06-01), and `faults.clock.DEFAULT_NOW == WORLD_EPOCH` (drift-guarded by a test, no cross-layer import). Reason: the deferred clock/seed alignment — so `create_return`'s 30-day window has eligible orders at the default clock. Threaded the clock into `mcp_server`/`serve_stdio`.
+- 2026-09-19 — Injector check order per call: latency → conditional not-found → rate limit → probabilistic error → execute. Conditional (`not_found_if_newer_than`) and rate limits are deterministic (data/clock-driven, no RNG); the record's timestamp is supplied by the server (first datetime field of the target record, get/update/delete only). Reason: conditional/limit faults model real, non-random behaviour and must be reproducible from the clock.
+- 2026-09-19 — HTTP transport is streamable-HTTP via `serve_http` / `faultbench serve --http`. The `mcp_url` pytest fixture serves the world in a subprocess and exposes end state through the world's state-file mirror (`.snapshot()`), since an out-of-process server can't share the `world` object. Reason: HTTP is for remote/other-framework agents; in-process `mcp_server` stays the fast path for state assertions.
+- 2026-09-19 — Framework-neutrality is proven at the MCP protocol level: the reference `mcp.Client` (not Pydantic AI) drives the world over HTTP in CI, keyless. Reason: a real second agent framework needs another dependency + API key + flaky LLM calls; protocol-level proof is deterministic and per the "one more via HTTP, then stop" non-goal.
+- 2026-09-19 — Second example world is `examples/bank/` (accounts + transfers, custom transfer rule). Reason: proves the world format generalises beyond the shop; validated keylessly in `tests/` so it can't rot.
+- 2026-09-19 — User docs live in `docs/WORLDS.md` (authoring) and `docs/TESTING.md` (markers/fixtures/CLI); the README links them. Reason: separate the "how do I use it" guides from the internal ARCHITECTURE/DECISIONS/STATUS docs.
+- 2026-09-19 — Framework adapters live in `faultbench.integrations.<framework>` as optional imports (each needs its own extra, e.g. `faultbench[pydantic-ai]`); the core stays framework-neutral. The Pydantic AI adapter's `run_agent(model, prompt, mcp=...)` accepts the in-process `mcp_server` or the `mcp_url` string. Reason: smooth the first-run experience without making pydantic-ai a core dependency.
+- 2026-09-19 — Repo-local git identity is `mithun <mithunmsman@gmail.com>` (global untouched); the whole existing history was rewritten to this author before any push. Publishing (GitHub push, PyPI) is held for the `mit-hun-k` account token and explicit user confirmation — outward-facing/irreversible. Reason: correct attribution for the new owner; no accidental publish.
+- 2026-09-19 — `faultbench.__version__` and `pyproject` version must match, guarded by `test_version_matches_pyproject`. Reason: they drifted (code 0.0.1 vs wheel 0.1.0) and shipped the wrong `--version`.
+- 2026-09-19 — Pre-launch validation with a real second agent framework: the OpenAI Agents SDK drives the shop world over HTTP (`examples/shop/test_openai_agents.py`, opt-in; `openai-agents` extra). Fault-free it does a clean refund; under a forced timeout it retried and wrote 3 phantom refunds while reporting failure — faultbench caught them. Reason: substantiates "works with any framework" with a genuine rival agent, not just the reference client.
+- 2026-09-19 — The server mirrors world state to the state file after EVERY attempt, including faulted ones (a timeout runs the op then fails). Found during the OpenAI Agents SDK validation: the success-only mirror under-reported write-then-timeout over HTTP — the exact bug class faultbench targets. Guarded by `test_faulted_write_is_mirrored_to_state_file`.
+- 2026-09-19 — Records using `enum[a, b, c]` must be written in YAML block style; flow style (`{ status: enum[a, b, c] }`) is a YAML parse error (commas/brackets). Documented in WORLDS.md. Reason: real papercut hit while authoring a world.
+- 2026-09-19 — Enum values are validated on writes: `create`/`update` reject a value outside the field's members (`run_builtin._check_enums`), and enum tool params advertise their members as a JSON-schema enum (Literal). Reason: real APIs reject bad enums; a docs-only tester found `status="banana"` was silently accepted.
+- 2026-09-19 — Documented (README + WORLDS.md) that world files execute Python via custom handlers, so only trusted world files should be loaded/served. Reason: loading a world runs `handler: module.func` imports; users must know.
+- 2026-09-19 — Documented the keyless testing path (`await mcp_server.call_tool(...)` + assert on `world`/`trace`), guarded by `tests/test_keyless.py`. Reason: docs-only tester couldn't test operations/faults without an LLM; the free deterministic loop was undocumented.