api-shakedown 0.1.0__tar.gz

api_shakedown-0.1.0/.gitignore

@@ -0,0 +1,22 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+build/
+dist/
+.venv/
+venv/
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.coverage
+htmlcov/
+
+# secrets: never commit real keys
+.env
+
+# generated harnesses
+shakedown-output/
+
+.idea/
+.vscode/
+.DS_Store

api_shakedown-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Anzar Ahsan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

api_shakedown-0.1.0/PKG-INFO

@@ -0,0 +1,431 @@
+Metadata-Version: 2.5
+Name: api-shakedown
+Version: 0.1.0
+Summary: Multi-agent generator of WireMock stubs, contract tests, and chaos scenarios from an OpenAPI spec.
+Project-URL: Homepage, https://github.com/anzar-ahsan-commits/api-shakedown
+Project-URL: Documentation, https://github.com/anzar-ahsan-commits/api-shakedown#readme
+Project-URL: Issues, https://github.com/anzar-ahsan-commits/api-shakedown/issues
+Project-URL: Changelog, https://github.com/anzar-ahsan-commits/api-shakedown/blob/main/CHANGELOG.md
+Author: Anzar Ahsan
+License-Expression: MIT
+License-File: LICENSE
+Keywords: chaos-engineering,contract-testing,langgraph,llm,mcp,openapi,wiremock
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: agentpanel<0.2,>=0.1.0
+Requires-Dist: httpx>=0.27
+Requires-Dist: openapi-spec-validator>=0.7
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: typer>=0.12
+Provides-Extra: anthropic
+Requires-Dist: agentpanel[anthropic]<0.2,>=0.1.0; extra == 'anthropic'
+Provides-Extra: cloud
+Requires-Dist: agentpanel[cloud]<0.2,>=0.1.0; extra == 'cloud'
+Provides-Extra: dev
+Requires-Dist: agentpanel[cloud]<0.2,>=0.1.0; extra == 'dev'
+Requires-Dist: build>=1.2; extra == 'dev'
+Requires-Dist: mcp<3,>=2.3; extra == 'dev'
+Requires-Dist: mypy>=1.11; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Requires-Dist: twine>=5; extra == 'dev'
+Requires-Dist: types-pyyaml; extra == 'dev'
+Provides-Extra: mcp
+Requires-Dist: mcp<3,>=2.3; extra == 'mcp'
+Provides-Extra: openai
+Requires-Dist: agentpanel[openai]<0.2,>=0.1.0; extra == 'openai'
+Description-Content-Type: text/markdown
+
+# API Shakedown
+
+**Give it an OpenAPI spec. Get back a ready-to-run test harness: WireMock stubs, contract
+tests, robustness probes, and chaos scenarios, written by a team of LLM agents that
+review each other's work.**
+
+[![CI](https://github.com/anzar-ahsan-commits/api-shakedown/actions/workflows/ci.yml/badge.svg)](https://github.com/anzar-ahsan-commits/api-shakedown/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/api-shakedown.svg)](https://pypi.org/project/api-shakedown/)
+[![Python](https://img.shields.io/pypi/pyversions/api-shakedown.svg)](https://pypi.org/project/api-shakedown/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/anzar-ahsan-commits/api-shakedown/blob/main/LICENSE)
+
+Runs **locally on [Ollama](https://ollama.com) by default**: no API key, no data leaves
+your machine. Cloud models (OpenAI, Anthropic) are bring-your-own-key. Available as a
+CLI, a Docker image, and an **MCP server** so an AI assistant can drive it.
+
+> Status: alpha (0.1.0). Works end to end on the example spec with a local 8B model.
+
+---
+
+## Contents
+
+- [Why](#why)
+- [Quick start](#quick-start)
+- [What you get](#what-you-get)
+- [How it works](#how-it-works)
+- [Install](#install)
+- [Usage](#usage)
+- [Configuration](#configuration)
+- [MCP server](#mcp-server)
+- [Docker](#docker)
+- [Architecture](#architecture)
+- [Limitations](#limitations)
+- [Roadmap](#roadmap)
+- [Development](#development)
+- [Contributing](#contributing)
+- [License](#license)
+
+## Why
+
+Teams that consume an API need three things that are tedious to write by hand and go
+stale quickly:
+
+1. **A realistic mock** of the API to develop and test against.
+2. **Contract tests** that tell you when the real API stops matching its spec.
+3. **Failure drills**: what does your client do when the API is slow, flaky, rate
+   limited, returns garbage, or goes down?
+
+API Shakedown generates all three from the spec you already have, and keeps them honest:
+**the LLMs never write code that runs**. They propose *data* (response bodies, test
+cases, failure scenarios, test records), and code validates every item against the
+spec, feeds problems back to the model, and falls back to deterministic output when a
+model cannot get it right. Without any model at all (`--offline`), every part still works.
+
+## Quick start
+
+Prerequisites: Python 3.11+, Docker (for WireMock), and [Ollama](https://ollama.com) with
+a model (`ollama pull qwen3:8b`) unless you use `--offline` or a cloud model.
+
+```bash
+pip install api-shakedown
+shakedown doctor                                   # is the model ready?
+shakedown run openapi.yaml --out my-harness        # minutes on a local model
+cd my-harness
+docker compose up -d                               # WireMock with your stubs on :8080
+pip install -r requirements.txt
+SHAKEDOWN_SELECT_STATUS=1 pytest tests/test_contract.py
+python chaos/chaos.py on flaky-start               # now point your client at :8080
+python chaos/chaos.py off
+```
+
+Try it on the bundled example: `shakedown run examples/petstore.yaml --out petstore`.
+
+A real run on the example spec with `qwen3:8b`:
+
+```text
+Petstore 1.0.0: 5 operations (ollama / qwen3:8b)
+Generating test data...
+Test data ready (3 Pet).
+Generating stubs, contract tests, and chaos scenarios...
+  stubs: done
+  contracts: done
+  chaos: done
+  review contracts: approved
+  review stubs: approved
+  review chaos: approved
+
+Wrote 14 files to harness in 436s: 15 stubs, 11 contract cases, 11 chaos scenarios, 30 probes.
+Next: cd harness && docker compose up -d   (then see README.md)
+```
+
+## What you get
+
+```text
+my-harness/
+├── README.md                  # generated: how to run it, every scenario, review notes
+├── docker-compose.yml         # WireMock 3 serving the stubs (port: SHAKEDOWN_WIREMOCK_PORT)
+├── wiremock/mappings/         # stub mappings
+├── tests/
+│   ├── test_contract.py       # contract tests (cases in contract_cases.json)
+│   └── test_robustness.py     # malformed-request probes (cases in robustness_cases.json)
+├── chaos/
+│   ├── chaos.py               # switch failure scenarios on/off (standard library only)
+│   └── chaos_scenarios.json
+├── data/dataset.json          # test records and edge-case values per field
+├── requirements.txt           # pytest, httpx, openapi-schema-validator
+└── shakedown.json             # manifest: counts, review outcome, generation notes
+```
+
+| Part | What it does |
+|---|---|
+| **Stubs** | Every documented response of every operation. Known ids return their own record, unknown ids return the documented 404, and any other response can be requested with `X-Shakedown-Status: <status>`. |
+| **Contract tests** | Happy paths, documented error statuses, and boundary values. Each checks the status code, `Content-Type`, and the body against the response schema. Run them against the stubs or a real deployment. |
+| **Robustness probes** | Invalid bodies and parameters derived from the schemas (missing required fields, wrong types, out-of-range values, broken JSON). A robust server answers 4xx, never 5xx. |
+| **Chaos scenarios** | Seven baseline failures every client should survive, plus scenarios the LLM targets at your API. |
+| **Test data** | Realistic, consistent records per entity, shared by the stubs and the tests, plus edge-case values for every field. |
+
+Baseline chaos scenarios:
+
+| Scenario | What happens | A resilient client should |
+|---|---|---|
+| `total-outage` | Every endpoint returns 503 | Fail fast, back off, not hammer the API |
+| `slow-responses` | Every response takes 3 s | Stay responsive |
+| `timeouts` | Every response takes 30 s | Give up on its own and free resources |
+| `connection-resets` | Connections are reset | Retry idempotent calls with backoff |
+| `flaky-start` | First 2 calls fail with 503, then recover | Retry and succeed on the third attempt |
+| `rate-limited` | First 3 calls return 429 | Honour `Retry-After` |
+| `truncated-json` | Success bodies are cut off mid-JSON | Report a parse error, never crash |
+
+LLM-targeted scenarios from a real run include *createPet fails 3 times with 503 then
+recovers*, *listPets is rate limited*, and *getPet loses required fields*.
+
+## How it works
+
+```mermaid
+flowchart LR
+    spec[/"OpenAPI spec"/] --> load["Load, validate,<br/>normalize, chunk"]
+    load --> data["Test data agent"]
+    data --> stubs["Stub agent"]
+    data --> contracts["Contract agent"]
+    data --> chaos["Chaos agent"]
+    stubs --> review{"Reviewer agent"}
+    contracts --> review
+    chaos --> review
+    review -- "blocker: revise with feedback" --> stubs
+    review -- "blocker" --> contracts
+    review -- "blocker" --> chaos
+    review -- "approved, or budget spent" --> coord["Coordinator"]
+    coord --> harness[/"Harness directory"/]
+```
+
+1. **Spec layer**: loads YAML/JSON from a path or URL, validates it (problems become
+   warnings), inlines `$ref`s safely, and splits operations into prompt-sized chunks so
+   large specs fit small models.
+2. **Test data agent** runs first: the stubs and tests share its records.
+3. **Stub, contract, and chaos agents** run in parallel.
+4. **Reviewer** judges each artifact. Blockers send it back to its author with the
+   feedback, for a bounded number of rounds; suggestions are kept as notes.
+5. **Coordinator** assembles the harness and its README, including anything the
+   reviewer still rejected when the revision budget ran out.
+
+Every agent follows the same pattern, which is what keeps small local models reliable:
+
+```mermaid
+sequenceDiagram
+    participant A as Agent
+    participant M as LLM
+    participant C as Code checks
+    A->>M: task + JSON schema (constrained decoding on Ollama)
+    M-->>A: proposal (bodies, cases, scenarios, records)
+    A->>C: validate against the spec
+    alt problems found
+        C-->>A: exact errors, e.g. "listPets 200: 'id' is a required property"
+        A->>M: fix these problems
+        M-->>A: corrected proposal
+    end
+    A->>A: keep valid items, fall back to deterministic ones for the rest
+```
+
+Things code guarantees, so the model never has to:
+
+- Request matching, URL patterns, priorities, and path-parameter types in the stubs.
+- Response bodies valid against their schemas.
+- Test inputs that route (valid path parameters, required query parameters), success
+  cases that use ids that exist, 404 cases that use ids that do not.
+- Authentication from the spec's security schemes.
+- Chaos scenarios that are internally consistent (a "flaky" failure must recover, a rate
+  limit is a 429, a partial outage leaves some endpoints up).
+- Reviewer findings that cite real items, and blockers only on what the author controls.
+
+## Install
+
+```bash
+pip install api-shakedown              # local models via Ollama
+pip install "api-shakedown[cloud]"     # plus OpenAI and Anthropic
+pip install "api-shakedown[mcp]"       # plus the MCP server
+```
+
+Or use the Docker image, which includes everything: see [Docker](#docker).
+
+Generated harnesses need Docker (for WireMock) and Python for their tests.
+
+## Usage
+
+```text
+shakedown doctor                     Check the LLM provider is reachable and ready
+shakedown inspect SPEC [--json]      Show the operations the agents will see
+shakedown run SPEC [options]         Generate a harness
+shakedown version
+```
+
+`shakedown run` options:
+
+| Option | Default | |
+|---|---|---|
+| `--out PATH` | `shakedown-output` | Where to write the harness (refuses a non-empty directory) |
+| `--offline` | off | No LLM: deterministic output in seconds |
+| `--no-review` | review on | Skip the reviewer (about 3x faster on the example) |
+| `--max-revisions N` | `1` | Review/revise rounds |
+| `--force` | off | Write into a non-empty directory (only overwrites harness files) |
+| `--provider`, `--model` | from settings | Override the LLM for this run |
+
+Using the harness:
+
+```bash
+docker compose up -d                                     # stubs on http://localhost:8080
+SHAKEDOWN_SELECT_STATUS=1 pytest tests/test_contract.py  # against the stubs
+python chaos/chaos.py list | on <scenario> | off         # failure drills
+SHAKEDOWN_TARGET_URL=https://api.example.com/v1 pytest tests   # against a real server
+```
+
+PowerShell: `$env:SHAKEDOWN_SELECT_STATUS = "1"; pytest tests/test_contract.py`.
+
+## Configuration
+
+Settings come from environment variables or a `.env` file
+(see [.env.example](https://github.com/anzar-ahsan-commits/api-shakedown/blob/main/.env.example)). Everything is optional.
+
+| Variable | Default | |
+|---|---|---|
+| `SHAKEDOWN_PROVIDER` | `ollama` | `ollama`, `openai`, or `anthropic` |
+| `SHAKEDOWN_MODEL` | per provider | `qwen3:8b`, `gpt-5-mini`, `claude-sonnet-5-5` |
+| `SHAKEDOWN_OLLAMA_URL` | `http://localhost:11434` | Where Ollama runs |
+| `SHAKEDOWN_BASE_URL` | | Endpoint override, e.g. an OpenAI-compatible server |
+| `SHAKEDOWN_TIMEOUT` | 600 (Ollama) / 120 | Seconds per request |
+| `SHAKEDOWN_CONTEXT_WINDOW` | `16384` | Ollama context size |
+| `SHAKEDOWN_REASONING` | model default | Ollama: turn a reasoning model's thinking on/off |
+| `SHAKEDOWN_AGENT_MODELS` | | Per-agent models, e.g. `{"reviewer": "anthropic:claude-sonnet-5-5"}` |
+| `SHAKEDOWN_AGENT_REASONING` | thinking off for generators | Per-agent thinking, e.g. `{"reviewer": true}` |
+| `OPENAI_API_KEY`, `ANTHROPIC_API_KEY` | | Bring your own key |
+
+Mixing models works well: generate locally, review with a stronger cloud model:
+
+```bash
+SHAKEDOWN_AGENT_MODELS='{"reviewer": "anthropic:claude-sonnet-5-5"}' shakedown run openapi.yaml
+```
+
+Generated harnesses read `SHAKEDOWN_TARGET_URL`, `SHAKEDOWN_SELECT_STATUS`,
+`SHAKEDOWN_API_KEY`, `SHAKEDOWN_BASIC_AUTH`, `SHAKEDOWN_TIMEOUT`, and
+`SHAKEDOWN_WIREMOCK_PORT` / `SHAKEDOWN_WIREMOCK_URL`.
+
+## MCP server
+
+`shakedown-mcp` (stdio) lets an MCP client such as Claude Code or Claude Desktop
+generate harnesses and run failure drills:
+
+| Tool | |
+|---|---|
+| `doctor` | Is the LLM provider ready? |
+| `inspect_spec` | List a spec's operations and warnings |
+| `generate_harness` | Generate a harness, with progress notifications |
+| `list_chaos_scenarios` | Scenarios in a harness |
+| `activate_chaos` / `deactivate_chaos` | Switch a scenario on/off in a running WireMock |
+
+```bash
+pip install "api-shakedown[mcp]"
+claude mcp add api-shakedown -- shakedown-mcp          # Claude Code
+```
+
+Other clients (`mcpServers` in their config):
+
+```json
+{
+  "mcpServers": {
+    "api-shakedown": {
+      "command": "shakedown-mcp",
+      "env": { "SHAKEDOWN_PROVIDER": "ollama" }
+    }
+  }
+}
+```
+
+Pass absolute paths. Generating with a local model takes minutes; if your client gives up
+on long tool calls, use `offline` or `review: false`, or raise its tool timeout.
+
+## Docker
+
+```bash
+docker build -t api-shakedown .
+docker run --rm -v "$PWD:/work" api-shakedown run /work/openapi.yaml --out /work/harness
+docker run -i --rm -v "$PWD:/work" --entrypoint shakedown-mcp api-shakedown   # MCP server
+```
+
+The image reaches Ollama on the host at `host.docker.internal`. Details for Linux,
+PowerShell, and cloud keys: [DOCKER.md](https://github.com/anzar-ahsan-commits/api-shakedown/blob/main/DOCKER.md).
+
+## Architecture
+
+```mermaid
+flowchart TB
+    subgraph app["api-shakedown"]
+        cli["CLI: shakedown"] --> pipeline["pipeline"]
+        mcp["MCP server: shakedown-mcp"] --> pipeline
+        pipeline --> agents["agents: testdata, stubs, contracts,<br/>chaos, reviewer, coordinator"]
+        agents --> domain["spec, wiremock, contracts,<br/>chaos, testdata (code checks)"]
+        pipeline --> harness["harness writer + templates"]
+    end
+    subgraph core["agentpanel (reusable core)"]
+        workflow["Workflow: parallel specialists,<br/>cross-review, bounded revision"]
+        structured["structured output:<br/>validate, feed back, retry"]
+        llm["providers: Ollama, OpenAI, Anthropic"]
+    end
+    agents --> workflow
+    agents --> structured
+    structured --> llm
+```
+
+The orchestration core is published separately as
+[**agentpanel**](https://github.com/anzar-ahsan-commits/api-shakedown/blob/main/packages/agentpanel/README.md): a small, domain-agnostic library for
+"specialists draft, reviewers critique, authors revise, a coordinator decides" workflows
+on LangGraph. It never imports the application; a test enforces that.
+
+```text
+packages/agentpanel/     # the reusable core (its own PyPI package)
+src/api_shakedown/       # the application
+  spec/                  # loading, validation, $ref resolution, edge cases, samples
+  agents/                # the six agents
+  templates/             # runners copied into every harness
+examples/petstore.yaml   # example spec
+tests/                   # offline tests; `-m live` for Ollama + Docker tests
+```
+
+## Limitations
+
+- **Speed on local models**: a full run on the 5-operation example takes about 7 minutes
+  with `qwen3:8b`, most of it review; `--no-review` takes about 2.5 minutes.
+- **Small reviewers are imperfect**: an 8B reviewer sometimes raises a false blocker,
+  which costs a revision round. Use a stronger model for the reviewer if you can.
+- **Specs**: OpenAPI 3.0-3.2. Swagger 2.0 must be converted first; references to other
+  files are not followed; self-referencing schemas are not validated.
+- **Stubs are static**: a create call returns a new record, but stubs do not keep state
+  between calls.
+- **Probes against stubs** only cover operations that document a 4xx; they are meant for
+  real servers.
+- Tested so far mainly on the bundled example. Reports from larger real-world specs are
+  very welcome.
+
+## Roadmap
+
+- Larger real-world specs: batching and timing at 30+ operations
+- Stateful stubs (WireMock response templating, created records visible to later calls)
+- Auth-failure scenarios (401/403) and "valid but nasty input must not 5xx" probes
+- Review of the test dataset
+- More harness targets (k6 load profiles, Postman collections)
+
+## Development
+
+```bash
+git clone https://github.com/anzar-ahsan-commits/api-shakedown.git
+cd api-shakedown
+python -m venv .venv && source .venv/bin/activate     # Windows: .venv\Scripts\activate
+pip install -e packages/agentpanel -e ".[dev]"
+ruff check . && ruff format --check . && mypy
+pytest                 # offline: LLMs are faked, about 20 s
+pytest -m live         # also uses your local Ollama and Docker
+```
+
+## Contributing
+
+Contributions are welcome: see [CONTRIBUTING.md](https://github.com/anzar-ahsan-commits/api-shakedown/blob/main/CONTRIBUTING.md). Security issues:
+see [SECURITY.md](https://github.com/anzar-ahsan-commits/api-shakedown/blob/main/SECURITY.md).
+
+## License
+
+[MIT](https://github.com/anzar-ahsan-commits/api-shakedown/blob/main/LICENSE) © 2026 Anzar Ahsan