pytest-wirecheck 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- pytest_wirecheck-0.1.0/LICENSE +21 -0
- pytest_wirecheck-0.1.0/PKG-INFO +175 -0
- pytest_wirecheck-0.1.0/README.md +145 -0
- pytest_wirecheck-0.1.0/pyproject.toml +50 -0
- pytest_wirecheck-0.1.0/setup.cfg +4 -0
- pytest_wirecheck-0.1.0/src/pytest_wirecheck.egg-info/PKG-INFO +175 -0
- pytest_wirecheck-0.1.0/src/pytest_wirecheck.egg-info/SOURCES.txt +25 -0
- pytest_wirecheck-0.1.0/src/pytest_wirecheck.egg-info/dependency_links.txt +1 -0
- pytest_wirecheck-0.1.0/src/pytest_wirecheck.egg-info/entry_points.txt +5 -0
- pytest_wirecheck-0.1.0/src/pytest_wirecheck.egg-info/requires.txt +4 -0
- pytest_wirecheck-0.1.0/src/pytest_wirecheck.egg-info/top_level.txt +1 -0
- pytest_wirecheck-0.1.0/src/wirecheck/__init__.py +3 -0
- pytest_wirecheck-0.1.0/src/wirecheck/__main__.py +3 -0
- pytest_wirecheck-0.1.0/src/wirecheck/capture.py +231 -0
- pytest_wirecheck-0.1.0/src/wirecheck/cli.py +93 -0
- pytest_wirecheck-0.1.0/src/wirecheck/clock.py +17 -0
- pytest_wirecheck-0.1.0/src/wirecheck/plugin.py +314 -0
- pytest_wirecheck-0.1.0/src/wirecheck/report.py +183 -0
- pytest_wirecheck-0.1.0/src/wirecheck/rules.py +239 -0
- pytest_wirecheck-0.1.0/src/wirecheck/scan.py +320 -0
- pytest_wirecheck-0.1.0/src/wirecheck/versions.py +65 -0
- pytest_wirecheck-0.1.0/tests/test_capture.py +162 -0
- pytest_wirecheck-0.1.0/tests/test_plugin.py +198 -0
- pytest_wirecheck-0.1.0/tests/test_rules.py +206 -0
- pytest_wirecheck-0.1.0/tests/test_scan.py +180 -0
- pytest_wirecheck-0.1.0/tests/test_sdk.py +181 -0
- pytest_wirecheck-0.1.0/tests/test_versions.py +31 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Ishan Yash
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,175 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: pytest-wirecheck
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Record what your code really sends to the Claude API in tests, and fail the build if a request would break under API version 2026-10-21.
|
|
5
|
+
Author: Ishan Yash
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/ishanyash/wirecheck
|
|
8
|
+
Project-URL: Source, https://github.com/ishanyash/wirecheck
|
|
9
|
+
Project-URL: Issues, https://github.com/ishanyash/wirecheck/issues
|
|
10
|
+
Keywords: pytest,anthropic,claude,claude-api,migration,ci,lint
|
|
11
|
+
Classifier: Development Status :: 3 - Alpha
|
|
12
|
+
Classifier: Framework :: Pytest
|
|
13
|
+
Classifier: Intended Audience :: Developers
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
21
|
+
Classifier: Topic :: Software Development :: Quality Assurance
|
|
22
|
+
Classifier: Topic :: Software Development :: Testing
|
|
23
|
+
Requires-Python: >=3.10
|
|
24
|
+
Description-Content-Type: text/markdown
|
|
25
|
+
License-File: LICENSE
|
|
26
|
+
Requires-Dist: pytest>=8.0
|
|
27
|
+
Provides-Extra: test
|
|
28
|
+
Requires-Dist: anthropic>=2.0.0; extra == "test"
|
|
29
|
+
Dynamic: license-file
|
|
30
|
+
|
|
31
|
+
# WireCheck
|
|
32
|
+
|
|
33
|
+
**Your tests pass. Your Claude API calls return 400. WireCheck checks what your code really sends to the Claude API in tests, and fails the build if a request would break under API version 2026-10-21.**
|
|
34
|
+
|
|
35
|
+

|
|
36
|
+
|
|
37
|
+
[PyPI: pytest-wirecheck](https://pypi.org/project/pytest-wirecheck/) · [GitHub Marketplace: WireCheck for Claude API](https://github.com/marketplace/actions/wirecheck-for-claude-api) · [Demo](demo/)
|
|
38
|
+
|
|
39
|
+
## Why
|
|
40
|
+
|
|
41
|
+
On 21 Oct 2026 Anthropic released Claude API version `2026-10-21`. The new SDK majors (Python `anthropic` 2.0.0 and later) send it by default. Under it, a Messages request that sends `temperature`, `top_p` or `top_k` returns **400, even at the default value** ([migration guide](https://platform.claude.com/docs/en/api/migration-2026-10-21)).
|
|
42
|
+
|
|
43
|
+
Most test suites mock the API, so they never see that 400. And these parameters are hard to grep for. Since `anthropic` 1.0.0, `temperature=` is no longer a keyword of `messages.create` (passing it is a `TypeError`, so tests catch it), and the SDK 1.0 migration guide said to pass it through `extra_body` instead. `extra_body` is merged into the JSON at runtime, so no type checker sees it, and it often sits in a shared wrapper or a defaults dict, far from the call. That is exactly how [Nojoin broke](https://github.com/Valtora/Nojoin/issues/554): a loose version floor pulled in `anthropic` 2.x, a wrapper still sent `temperature: 0.2` through `extra_body`, and the tests mocked the client, so they all passed.
|
|
44
|
+
|
|
45
|
+
WireCheck looks at the final request as it leaves the HTTP client, after the SDK has merged every wrapper, default and `extra_body`.
|
|
46
|
+
|
|
47
|
+
## Quick start
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
pip install pytest-wirecheck
|
|
51
|
+
pytest --wirecheck
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
A test that sends a breaking request now fails, and the run ends with a short report. This is the [demo](demo/) on `anthropic` 2.3.0:
|
|
55
|
+
|
|
56
|
+
```
|
|
57
|
+
3 requests to the Claude API checked: 3 errors, 0 warnings
|
|
58
|
+
anthropic SDK: 2.3.0; API versions seen: 2026-10-21 x3
|
|
59
|
+
|
|
60
|
+
RULE LEVEL COUNT TEST CALL SITE WHAT
|
|
61
|
+
WC001 error 1 broken/test_broken.py::test_direct_call broken/app_direct.py:12 temperature
|
|
62
|
+
WC001 error 1 broken/test_broken.py::test_extra_body broken/app_extra_body.py:18 temperature
|
|
63
|
+
WC001 error 1 broken/test_broken.py::test_wrapper_factory broken/app_factory.py:21 top_p
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Each row shows the rule, the test that sent the request, and the file and line in your code that made the call. Works with `pytest-xdist` too.
|
|
67
|
+
|
|
68
|
+
### GitHub Action
|
|
69
|
+
|
|
70
|
+
```yaml
|
|
71
|
+
name: WireCheck
|
|
72
|
+
on: pull_request
|
|
73
|
+
permissions:
|
|
74
|
+
contents: read
|
|
75
|
+
pull-requests: write # to post the findings comment
|
|
76
|
+
jobs:
|
|
77
|
+
wirecheck:
|
|
78
|
+
runs-on: ubuntu-latest
|
|
79
|
+
steps:
|
|
80
|
+
- uses: actions/checkout@v7
|
|
81
|
+
- uses: actions/setup-python@v7
|
|
82
|
+
with:
|
|
83
|
+
python-version: "3.12"
|
|
84
|
+
- run: pip install -e ".[test]" # your project and its test deps
|
|
85
|
+
- uses: ishanyash/wirecheck@v0.1.0
|
|
86
|
+
with:
|
|
87
|
+
pytest-args: tests/
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
The Action runs the static scan and your tests with `--wirecheck`, writes the findings to the job summary, and posts **one** pull request comment with a findings table (it updates the same comment on later pushes).
|
|
91
|
+
|
|
92
|
+
| Input | Default | Meaning |
|
|
93
|
+
|---|---|---|
|
|
94
|
+
| `path` | `.` | Folders for the static scan |
|
|
95
|
+
| `run-pytest` | `true` | Run `pytest --wirecheck` |
|
|
96
|
+
| `pytest-args` | | Extra pytest arguments |
|
|
97
|
+
| `working-directory` | `.` | Where to run |
|
|
98
|
+
| `fail-on` | `error` | `error`, `warn` or `never` |
|
|
99
|
+
| `comment` | `true` | Post the PR comment |
|
|
100
|
+
| `install` | `action` | `action` installs WireCheck from the tagged Action source; `skip` if you install it yourself |
|
|
101
|
+
|
|
102
|
+
## What it checks
|
|
103
|
+
|
|
104
|
+
Rules come from the [migration guide](https://platform.claude.com/docs/en/api/migration-2026-10-21) and the [release notes](https://platform.claude.com/docs/en/release-notes/overview).
|
|
105
|
+
|
|
106
|
+
**In the real requests your tests send (`pytest --wirecheck`):**
|
|
107
|
+
|
|
108
|
+
| Rule | What | Level |
|
|
109
|
+
|---|---|---|
|
|
110
|
+
| WC001 | `temperature`, `top_p` or `top_k` in a Messages, token count or batch request | error on 2026-10-21 |
|
|
111
|
+
| WC002 | `document` block directly in an `assistant` message | error on 2026-10-21 |
|
|
112
|
+
| WC003 | `text` block with non-empty `citations` in a `user` message, including inside `search_result` | error on 2026-10-21 |
|
|
113
|
+
| WC004 | `X-Api-Key` header set by your code. SDK 2.x sends `Authorization: Bearer`, so your key no longer replaces the client key (both are sent) | warning |
|
|
114
|
+
| WC005 | `claude-opus-4-8` with `speed: "fast"` | warning now, error from 2026-11-30 |
|
|
115
|
+
|
|
116
|
+
**In source (`wirecheck scan`):**
|
|
117
|
+
|
|
118
|
+
| Rule | What | Level |
|
|
119
|
+
|---|---|---|
|
|
120
|
+
| WCS001 | `MessageParam(...)` / `BetaMessageParam(...)` called or subclassed (TypeError in SDK 2.x) | error |
|
|
121
|
+
| WCS002 | `X-Api-Key` in `extra_headers` / `default_headers` | warning |
|
|
122
|
+
| WCS003 | `ant --api-key` / `ant --auth-token` (removed in the `ant` CLI 2.0.0) | error |
|
|
123
|
+
| WCS004 | Text Completions calls (`client.completions.create`, `/v1/complete`) | error |
|
|
124
|
+
| WCS005 | `temperature` / `top_p` / `top_k` visible in a Messages call (a hint; the runtime check is the real one) | warning |
|
|
125
|
+
|
|
126
|
+
```bash
|
|
127
|
+
wirecheck scan . # text
|
|
128
|
+
wirecheck scan src --format json # or markdown
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
## Which API version applies
|
|
132
|
+
|
|
133
|
+
For each request WireCheck reads the `anthropic-version` header. If there is none, it uses the installed SDK (`anthropic` 2.0.0 and later means `2026-10-21`).
|
|
134
|
+
|
|
135
|
+
- On `2026-10-21`, WC001 to WC003 are **errors** and fail the test that sent the request.
|
|
136
|
+
- On `2023-06-01` (SDK 1.x and older), the same findings are **warnings**. Your code works today and breaks when you upgrade. Use `--wirecheck-fail-on=warn` to fail on them now.
|
|
137
|
+
|
|
138
|
+
## How it works
|
|
139
|
+
|
|
140
|
+
When enabled, the plugin wraps `Client.send` and `AsyncClient.send` of `httpx2` (used by `anthropic` 1.x and 2.x) and `httpx` (older SDKs). Every request to `api.anthropic.com`, `$ANTHROPIC_BASE_URL` or a `--wirecheck-base-url` is recorded before it reaches the transport, so it is seen even when the test mocks the response with a mock transport, `respx`, or the `wirecheck_stub` fixture, and even when there is no network. API keys are redacted before anything is stored. The hook is removed when the run ends.
|
|
141
|
+
|
|
142
|
+
## Options
|
|
143
|
+
|
|
144
|
+
| Option | Meaning |
|
|
145
|
+
|---|---|
|
|
146
|
+
| `--wirecheck` | Turn it on (or `wirecheck = true` in your pytest ini, or `WIRECHECK=1`) |
|
|
147
|
+
| `--wirecheck-fail-on` | `error` (default), `warn`, or `never` |
|
|
148
|
+
| `--wirecheck-base-url URL` | Also capture a gateway or proxy URL (repeatable) |
|
|
149
|
+
| `--wirecheck-report PATH` | Write findings as JSON |
|
|
150
|
+
| `--wirecheck-markdown PATH` | Write findings as Markdown |
|
|
151
|
+
| `WIRECHECK_TODAY=YYYY-MM-DD` | Pin the date used by date-based rules |
|
|
152
|
+
|
|
153
|
+
Fixtures and markers:
|
|
154
|
+
|
|
155
|
+
- `wirecheck_stub` answers every Claude API request in a test with a canned reply, with no network. It mocks the response, not the client, so WireCheck still sees what you send. Set `wirecheck_stub["text"]` to change the reply.
|
|
156
|
+
- `wirecheck_requests` lists the requests the current test sent.
|
|
157
|
+
- `@pytest.mark.wirecheck_skip` ignores requests from one test.
|
|
158
|
+
|
|
159
|
+
## Known limits
|
|
160
|
+
|
|
161
|
+
- **It only sees requests that are really built.** If a test replaces `client.messages.create` (or the whole client) with a `MagicMock`, no request exists and WireCheck sees nothing. It prints a hint when a run captured no requests. Mock the HTTP response instead.
|
|
162
|
+
- Python only, and only the `httpx` / `httpx2` HTTP clients (the SDK default). The optional `aiohttp` backend is not hooked.
|
|
163
|
+
- Bedrock and Vertex requests go to other hosts and carry the version differently. Add their URL with `--wirecheck-base-url` if you want them recorded; version detection there falls back to the SDK version.
|
|
164
|
+
- `wirecheck_stub` answers non-streaming calls only.
|
|
165
|
+
- WC003 checks the places the guide names (user `text` blocks and `search_result` blocks). It does not flag bare `text` blocks inside `tool_result`, because the guide does not say.
|
|
166
|
+
- The static scanner is a set of simple patterns. It misses anything assembled at runtime, which is why the pytest plugin exists.
|
|
167
|
+
- No auto-fix and no TypeScript or other SDKs in v0.1.
|
|
168
|
+
|
|
169
|
+
## Licence
|
|
170
|
+
|
|
171
|
+
MIT. See [LICENSE](LICENSE).
|
|
172
|
+
|
|
173
|
+
---
|
|
174
|
+
|
|
175
|
+
Built by [Ishan Yash](https://ishanyash.com)
|
|
@@ -0,0 +1,145 @@
|
|
|
1
|
+
# WireCheck
|
|
2
|
+
|
|
3
|
+
**Your tests pass. Your Claude API calls return 400. WireCheck checks what your code really sends to the Claude API in tests, and fails the build if a request would break under API version 2026-10-21.**
|
|
4
|
+
|
|
5
|
+

|
|
6
|
+
|
|
7
|
+
[PyPI: pytest-wirecheck](https://pypi.org/project/pytest-wirecheck/) · [GitHub Marketplace: WireCheck for Claude API](https://github.com/marketplace/actions/wirecheck-for-claude-api) · [Demo](demo/)
|
|
8
|
+
|
|
9
|
+
## Why
|
|
10
|
+
|
|
11
|
+
On 21 Oct 2026 Anthropic released Claude API version `2026-10-21`. The new SDK majors (Python `anthropic` 2.0.0 and later) send it by default. Under it, a Messages request that sends `temperature`, `top_p` or `top_k` returns **400, even at the default value** ([migration guide](https://platform.claude.com/docs/en/api/migration-2026-10-21)).
|
|
12
|
+
|
|
13
|
+
Most test suites mock the API, so they never see that 400. And these parameters are hard to grep for. Since `anthropic` 1.0.0, `temperature=` is no longer a keyword of `messages.create` (passing it is a `TypeError`, so tests catch it), and the SDK 1.0 migration guide said to pass it through `extra_body` instead. `extra_body` is merged into the JSON at runtime, so no type checker sees it, and it often sits in a shared wrapper or a defaults dict, far from the call. That is exactly how [Nojoin broke](https://github.com/Valtora/Nojoin/issues/554): a loose version floor pulled in `anthropic` 2.x, a wrapper still sent `temperature: 0.2` through `extra_body`, and the tests mocked the client, so they all passed.
|
|
14
|
+
|
|
15
|
+
WireCheck looks at the final request as it leaves the HTTP client, after the SDK has merged every wrapper, default and `extra_body`.
|
|
16
|
+
|
|
17
|
+
## Quick start
|
|
18
|
+
|
|
19
|
+
```bash
|
|
20
|
+
pip install pytest-wirecheck
|
|
21
|
+
pytest --wirecheck
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
A test that sends a breaking request now fails, and the run ends with a short report. This is the [demo](demo/) on `anthropic` 2.3.0:
|
|
25
|
+
|
|
26
|
+
```
|
|
27
|
+
3 requests to the Claude API checked: 3 errors, 0 warnings
|
|
28
|
+
anthropic SDK: 2.3.0; API versions seen: 2026-10-21 x3
|
|
29
|
+
|
|
30
|
+
RULE LEVEL COUNT TEST CALL SITE WHAT
|
|
31
|
+
WC001 error 1 broken/test_broken.py::test_direct_call broken/app_direct.py:12 temperature
|
|
32
|
+
WC001 error 1 broken/test_broken.py::test_extra_body broken/app_extra_body.py:18 temperature
|
|
33
|
+
WC001 error 1 broken/test_broken.py::test_wrapper_factory broken/app_factory.py:21 top_p
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
Each row shows the rule, the test that sent the request, and the file and line in your code that made the call. Works with `pytest-xdist` too.
|
|
37
|
+
|
|
38
|
+
### GitHub Action
|
|
39
|
+
|
|
40
|
+
```yaml
|
|
41
|
+
name: WireCheck
|
|
42
|
+
on: pull_request
|
|
43
|
+
permissions:
|
|
44
|
+
contents: read
|
|
45
|
+
pull-requests: write # to post the findings comment
|
|
46
|
+
jobs:
|
|
47
|
+
wirecheck:
|
|
48
|
+
runs-on: ubuntu-latest
|
|
49
|
+
steps:
|
|
50
|
+
- uses: actions/checkout@v7
|
|
51
|
+
- uses: actions/setup-python@v7
|
|
52
|
+
with:
|
|
53
|
+
python-version: "3.12"
|
|
54
|
+
- run: pip install -e ".[test]" # your project and its test deps
|
|
55
|
+
- uses: ishanyash/wirecheck@v0.1.0
|
|
56
|
+
with:
|
|
57
|
+
pytest-args: tests/
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
The Action runs the static scan and your tests with `--wirecheck`, writes the findings to the job summary, and posts **one** pull request comment with a findings table (it updates the same comment on later pushes).
|
|
61
|
+
|
|
62
|
+
| Input | Default | Meaning |
|
|
63
|
+
|---|---|---|
|
|
64
|
+
| `path` | `.` | Folders for the static scan |
|
|
65
|
+
| `run-pytest` | `true` | Run `pytest --wirecheck` |
|
|
66
|
+
| `pytest-args` | | Extra pytest arguments |
|
|
67
|
+
| `working-directory` | `.` | Where to run |
|
|
68
|
+
| `fail-on` | `error` | `error`, `warn` or `never` |
|
|
69
|
+
| `comment` | `true` | Post the PR comment |
|
|
70
|
+
| `install` | `action` | `action` installs WireCheck from the tagged Action source; `skip` if you install it yourself |
|
|
71
|
+
|
|
72
|
+
## What it checks
|
|
73
|
+
|
|
74
|
+
Rules come from the [migration guide](https://platform.claude.com/docs/en/api/migration-2026-10-21) and the [release notes](https://platform.claude.com/docs/en/release-notes/overview).
|
|
75
|
+
|
|
76
|
+
**In the real requests your tests send (`pytest --wirecheck`):**
|
|
77
|
+
|
|
78
|
+
| Rule | What | Level |
|
|
79
|
+
|---|---|---|
|
|
80
|
+
| WC001 | `temperature`, `top_p` or `top_k` in a Messages, token count or batch request | error on 2026-10-21 |
|
|
81
|
+
| WC002 | `document` block directly in an `assistant` message | error on 2026-10-21 |
|
|
82
|
+
| WC003 | `text` block with non-empty `citations` in a `user` message, including inside `search_result` | error on 2026-10-21 |
|
|
83
|
+
| WC004 | `X-Api-Key` header set by your code. SDK 2.x sends `Authorization: Bearer`, so your key no longer replaces the client key (both are sent) | warning |
|
|
84
|
+
| WC005 | `claude-opus-4-8` with `speed: "fast"` | warning now, error from 2026-11-30 |
|
|
85
|
+
|
|
86
|
+
**In source (`wirecheck scan`):**
|
|
87
|
+
|
|
88
|
+
| Rule | What | Level |
|
|
89
|
+
|---|---|---|
|
|
90
|
+
| WCS001 | `MessageParam(...)` / `BetaMessageParam(...)` called or subclassed (TypeError in SDK 2.x) | error |
|
|
91
|
+
| WCS002 | `X-Api-Key` in `extra_headers` / `default_headers` | warning |
|
|
92
|
+
| WCS003 | `ant --api-key` / `ant --auth-token` (removed in the `ant` CLI 2.0.0) | error |
|
|
93
|
+
| WCS004 | Text Completions calls (`client.completions.create`, `/v1/complete`) | error |
|
|
94
|
+
| WCS005 | `temperature` / `top_p` / `top_k` visible in a Messages call (a hint; the runtime check is the real one) | warning |
|
|
95
|
+
|
|
96
|
+
```bash
|
|
97
|
+
wirecheck scan . # text
|
|
98
|
+
wirecheck scan src --format json # or markdown
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
## Which API version applies
|
|
102
|
+
|
|
103
|
+
For each request WireCheck reads the `anthropic-version` header. If there is none, it uses the installed SDK (`anthropic` 2.0.0 and later means `2026-10-21`).
|
|
104
|
+
|
|
105
|
+
- On `2026-10-21`, WC001 to WC003 are **errors** and fail the test that sent the request.
|
|
106
|
+
- On `2023-06-01` (SDK 1.x and older), the same findings are **warnings**. Your code works today and breaks when you upgrade. Use `--wirecheck-fail-on=warn` to fail on them now.
|
|
107
|
+
|
|
108
|
+
## How it works
|
|
109
|
+
|
|
110
|
+
When enabled, the plugin wraps `Client.send` and `AsyncClient.send` of `httpx2` (used by `anthropic` 1.x and 2.x) and `httpx` (older SDKs). Every request to `api.anthropic.com`, `$ANTHROPIC_BASE_URL` or a `--wirecheck-base-url` is recorded before it reaches the transport, so it is seen even when the test mocks the response with a mock transport, `respx`, or the `wirecheck_stub` fixture, and even when there is no network. API keys are redacted before anything is stored. The hook is removed when the run ends.
|
|
111
|
+
|
|
112
|
+
## Options
|
|
113
|
+
|
|
114
|
+
| Option | Meaning |
|
|
115
|
+
|---|---|
|
|
116
|
+
| `--wirecheck` | Turn it on (or `wirecheck = true` in your pytest ini, or `WIRECHECK=1`) |
|
|
117
|
+
| `--wirecheck-fail-on` | `error` (default), `warn`, or `never` |
|
|
118
|
+
| `--wirecheck-base-url URL` | Also capture a gateway or proxy URL (repeatable) |
|
|
119
|
+
| `--wirecheck-report PATH` | Write findings as JSON |
|
|
120
|
+
| `--wirecheck-markdown PATH` | Write findings as Markdown |
|
|
121
|
+
| `WIRECHECK_TODAY=YYYY-MM-DD` | Pin the date used by date-based rules |
|
|
122
|
+
|
|
123
|
+
Fixtures and markers:
|
|
124
|
+
|
|
125
|
+
- `wirecheck_stub` answers every Claude API request in a test with a canned reply, with no network. It mocks the response, not the client, so WireCheck still sees what you send. Set `wirecheck_stub["text"]` to change the reply.
|
|
126
|
+
- `wirecheck_requests` lists the requests the current test sent.
|
|
127
|
+
- `@pytest.mark.wirecheck_skip` ignores requests from one test.
|
|
128
|
+
|
|
129
|
+
## Known limits
|
|
130
|
+
|
|
131
|
+
- **It only sees requests that are really built.** If a test replaces `client.messages.create` (or the whole client) with a `MagicMock`, no request exists and WireCheck sees nothing. It prints a hint when a run captured no requests. Mock the HTTP response instead.
|
|
132
|
+
- Python only, and only the `httpx` / `httpx2` HTTP clients (the SDK default). The optional `aiohttp` backend is not hooked.
|
|
133
|
+
- Bedrock and Vertex requests go to other hosts and carry the version differently. Add their URL with `--wirecheck-base-url` if you want them recorded; version detection there falls back to the SDK version.
|
|
134
|
+
- `wirecheck_stub` answers non-streaming calls only.
|
|
135
|
+
- WC003 checks the places the guide names (user `text` blocks and `search_result` blocks). It does not flag bare `text` blocks inside `tool_result`, because the guide does not say.
|
|
136
|
+
- The static scanner is a set of simple patterns. It misses anything assembled at runtime, which is why the pytest plugin exists.
|
|
137
|
+
- No auto-fix and no TypeScript or other SDKs in v0.1.
|
|
138
|
+
|
|
139
|
+
## Licence
|
|
140
|
+
|
|
141
|
+
MIT. See [LICENSE](LICENSE).
|
|
142
|
+
|
|
143
|
+
---
|
|
144
|
+
|
|
145
|
+
Built by [Ishan Yash](https://ishanyash.com)
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["setuptools>=77"]
|
|
3
|
+
build-backend = "setuptools.build_meta"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
name = "pytest-wirecheck"
|
|
7
|
+
version = "0.1.0"
|
|
8
|
+
description = "Record what your code really sends to the Claude API in tests, and fail the build if a request would break under API version 2026-10-21."
|
|
9
|
+
readme = "README.md"
|
|
10
|
+
requires-python = ">=3.10"
|
|
11
|
+
license = "MIT"
|
|
12
|
+
license-files = ["LICENSE"]
|
|
13
|
+
authors = [{ name = "Ishan Yash" }]
|
|
14
|
+
keywords = ["pytest", "anthropic", "claude", "claude-api", "migration", "ci", "lint"]
|
|
15
|
+
classifiers = [
|
|
16
|
+
"Development Status :: 3 - Alpha",
|
|
17
|
+
"Framework :: Pytest",
|
|
18
|
+
"Intended Audience :: Developers",
|
|
19
|
+
"Programming Language :: Python :: 3",
|
|
20
|
+
"Programming Language :: Python :: 3 :: Only",
|
|
21
|
+
"Programming Language :: Python :: 3.10",
|
|
22
|
+
"Programming Language :: Python :: 3.11",
|
|
23
|
+
"Programming Language :: Python :: 3.12",
|
|
24
|
+
"Programming Language :: Python :: 3.13",
|
|
25
|
+
"Programming Language :: Python :: 3.14",
|
|
26
|
+
"Topic :: Software Development :: Quality Assurance",
|
|
27
|
+
"Topic :: Software Development :: Testing",
|
|
28
|
+
]
|
|
29
|
+
dependencies = ["pytest>=8.0"]
|
|
30
|
+
|
|
31
|
+
[project.optional-dependencies]
|
|
32
|
+
test = ["anthropic>=2.0.0"]
|
|
33
|
+
|
|
34
|
+
[project.urls]
|
|
35
|
+
Homepage = "https://github.com/ishanyash/wirecheck"
|
|
36
|
+
Source = "https://github.com/ishanyash/wirecheck"
|
|
37
|
+
Issues = "https://github.com/ishanyash/wirecheck/issues"
|
|
38
|
+
|
|
39
|
+
[project.scripts]
|
|
40
|
+
wirecheck = "wirecheck.cli:main"
|
|
41
|
+
|
|
42
|
+
[project.entry-points.pytest11]
|
|
43
|
+
wirecheck = "wirecheck.plugin"
|
|
44
|
+
|
|
45
|
+
[tool.setuptools.packages.find]
|
|
46
|
+
where = ["src"]
|
|
47
|
+
|
|
48
|
+
[tool.pytest.ini_options]
|
|
49
|
+
testpaths = ["tests"]
|
|
50
|
+
addopts = "-p pytester"
|
|
@@ -0,0 +1,175 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: pytest-wirecheck
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Record what your code really sends to the Claude API in tests, and fail the build if a request would break under API version 2026-10-21.
|
|
5
|
+
Author: Ishan Yash
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/ishanyash/wirecheck
|
|
8
|
+
Project-URL: Source, https://github.com/ishanyash/wirecheck
|
|
9
|
+
Project-URL: Issues, https://github.com/ishanyash/wirecheck/issues
|
|
10
|
+
Keywords: pytest,anthropic,claude,claude-api,migration,ci,lint
|
|
11
|
+
Classifier: Development Status :: 3 - Alpha
|
|
12
|
+
Classifier: Framework :: Pytest
|
|
13
|
+
Classifier: Intended Audience :: Developers
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
21
|
+
Classifier: Topic :: Software Development :: Quality Assurance
|
|
22
|
+
Classifier: Topic :: Software Development :: Testing
|
|
23
|
+
Requires-Python: >=3.10
|
|
24
|
+
Description-Content-Type: text/markdown
|
|
25
|
+
License-File: LICENSE
|
|
26
|
+
Requires-Dist: pytest>=8.0
|
|
27
|
+
Provides-Extra: test
|
|
28
|
+
Requires-Dist: anthropic>=2.0.0; extra == "test"
|
|
29
|
+
Dynamic: license-file
|
|
30
|
+
|
|
31
|
+
# WireCheck
|
|
32
|
+
|
|
33
|
+
**Your tests pass. Your Claude API calls return 400. WireCheck checks what your code really sends to the Claude API in tests, and fails the build if a request would break under API version 2026-10-21.**
|
|
34
|
+
|
|
35
|
+

|
|
36
|
+
|
|
37
|
+
[PyPI: pytest-wirecheck](https://pypi.org/project/pytest-wirecheck/) · [GitHub Marketplace: WireCheck for Claude API](https://github.com/marketplace/actions/wirecheck-for-claude-api) · [Demo](demo/)
|
|
38
|
+
|
|
39
|
+
## Why
|
|
40
|
+
|
|
41
|
+
On 21 Oct 2026 Anthropic released Claude API version `2026-10-21`. The new SDK majors (Python `anthropic` 2.0.0 and later) send it by default. Under it, a Messages request that sends `temperature`, `top_p` or `top_k` returns **400, even at the default value** ([migration guide](https://platform.claude.com/docs/en/api/migration-2026-10-21)).
|
|
42
|
+
|
|
43
|
+
Most test suites mock the API, so they never see that 400. And these parameters are hard to grep for. Since `anthropic` 1.0.0, `temperature=` is no longer a keyword of `messages.create` (passing it is a `TypeError`, so tests catch it), and the SDK 1.0 migration guide said to pass it through `extra_body` instead. `extra_body` is merged into the JSON at runtime, so no type checker sees it, and it often sits in a shared wrapper or a defaults dict, far from the call. That is exactly how [Nojoin broke](https://github.com/Valtora/Nojoin/issues/554): a loose version floor pulled in `anthropic` 2.x, a wrapper still sent `temperature: 0.2` through `extra_body`, and the tests mocked the client, so they all passed.
|
|
44
|
+
|
|
45
|
+
WireCheck looks at the final request as it leaves the HTTP client, after the SDK has merged every wrapper, default and `extra_body`.
|
|
46
|
+
|
|
47
|
+
## Quick start
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
pip install pytest-wirecheck
|
|
51
|
+
pytest --wirecheck
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
A test that sends a breaking request now fails, and the run ends with a short report. This is the [demo](demo/) on `anthropic` 2.3.0:
|
|
55
|
+
|
|
56
|
+
```
|
|
57
|
+
3 requests to the Claude API checked: 3 errors, 0 warnings
|
|
58
|
+
anthropic SDK: 2.3.0; API versions seen: 2026-10-21 x3
|
|
59
|
+
|
|
60
|
+
RULE LEVEL COUNT TEST CALL SITE WHAT
|
|
61
|
+
WC001 error 1 broken/test_broken.py::test_direct_call broken/app_direct.py:12 temperature
|
|
62
|
+
WC001 error 1 broken/test_broken.py::test_extra_body broken/app_extra_body.py:18 temperature
|
|
63
|
+
WC001 error 1 broken/test_broken.py::test_wrapper_factory broken/app_factory.py:21 top_p
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Each row shows the rule, the test that sent the request, and the file and line in your code that made the call. Works with `pytest-xdist` too.
|
|
67
|
+
|
|
68
|
+
### GitHub Action
|
|
69
|
+
|
|
70
|
+
```yaml
|
|
71
|
+
name: WireCheck
|
|
72
|
+
on: pull_request
|
|
73
|
+
permissions:
|
|
74
|
+
contents: read
|
|
75
|
+
pull-requests: write # to post the findings comment
|
|
76
|
+
jobs:
|
|
77
|
+
wirecheck:
|
|
78
|
+
runs-on: ubuntu-latest
|
|
79
|
+
steps:
|
|
80
|
+
- uses: actions/checkout@v7
|
|
81
|
+
- uses: actions/setup-python@v7
|
|
82
|
+
with:
|
|
83
|
+
python-version: "3.12"
|
|
84
|
+
- run: pip install -e ".[test]" # your project and its test deps
|
|
85
|
+
- uses: ishanyash/wirecheck@v0.1.0
|
|
86
|
+
with:
|
|
87
|
+
pytest-args: tests/
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
The Action runs the static scan and your tests with `--wirecheck`, writes the findings to the job summary, and posts **one** pull request comment with a findings table (it updates the same comment on later pushes).
|
|
91
|
+
|
|
92
|
+
| Input | Default | Meaning |
|
|
93
|
+
|---|---|---|
|
|
94
|
+
| `path` | `.` | Folders for the static scan |
|
|
95
|
+
| `run-pytest` | `true` | Run `pytest --wirecheck` |
|
|
96
|
+
| `pytest-args` | | Extra pytest arguments |
|
|
97
|
+
| `working-directory` | `.` | Where to run |
|
|
98
|
+
| `fail-on` | `error` | `error`, `warn` or `never` |
|
|
99
|
+
| `comment` | `true` | Post the PR comment |
|
|
100
|
+
| `install` | `action` | `action` installs WireCheck from the tagged Action source; `skip` if you install it yourself |
|
|
101
|
+
|
|
102
|
+
## What it checks
|
|
103
|
+
|
|
104
|
+
Rules come from the [migration guide](https://platform.claude.com/docs/en/api/migration-2026-10-21) and the [release notes](https://platform.claude.com/docs/en/release-notes/overview).
|
|
105
|
+
|
|
106
|
+
**In the real requests your tests send (`pytest --wirecheck`):**
|
|
107
|
+
|
|
108
|
+
| Rule | What | Level |
|
|
109
|
+
|---|---|---|
|
|
110
|
+
| WC001 | `temperature`, `top_p` or `top_k` in a Messages, token count or batch request | error on 2026-10-21 |
|
|
111
|
+
| WC002 | `document` block directly in an `assistant` message | error on 2026-10-21 |
|
|
112
|
+
| WC003 | `text` block with non-empty `citations` in a `user` message, including inside `search_result` | error on 2026-10-21 |
|
|
113
|
+
| WC004 | `X-Api-Key` header set by your code. SDK 2.x sends `Authorization: Bearer`, so your key no longer replaces the client key (both are sent) | warning |
|
|
114
|
+
| WC005 | `claude-opus-4-8` with `speed: "fast"` | warning now, error from 2026-11-30 |
|
|
115
|
+
|
|
116
|
+
**In source (`wirecheck scan`):**
|
|
117
|
+
|
|
118
|
+
| Rule | What | Level |
|
|
119
|
+
|---|---|---|
|
|
120
|
+
| WCS001 | `MessageParam(...)` / `BetaMessageParam(...)` called or subclassed (TypeError in SDK 2.x) | error |
|
|
121
|
+
| WCS002 | `X-Api-Key` in `extra_headers` / `default_headers` | warning |
|
|
122
|
+
| WCS003 | `ant --api-key` / `ant --auth-token` (removed in the `ant` CLI 2.0.0) | error |
|
|
123
|
+
| WCS004 | Text Completions calls (`client.completions.create`, `/v1/complete`) | error |
|
|
124
|
+
| WCS005 | `temperature` / `top_p` / `top_k` visible in a Messages call (a hint; the runtime check is the real one) | warning |
|
|
125
|
+
|
|
126
|
+
```bash
|
|
127
|
+
wirecheck scan . # text
|
|
128
|
+
wirecheck scan src --format json # or markdown
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
## Which API version applies
|
|
132
|
+
|
|
133
|
+
For each request WireCheck reads the `anthropic-version` header. If there is none, it uses the installed SDK (`anthropic` 2.0.0 and later means `2026-10-21`).
|
|
134
|
+
|
|
135
|
+
- On `2026-10-21`, WC001 to WC003 are **errors** and fail the test that sent the request.
|
|
136
|
+
- On `2023-06-01` (SDK 1.x and older), the same findings are **warnings**. Your code works today and breaks when you upgrade. Use `--wirecheck-fail-on=warn` to fail on them now.
|
|
137
|
+
|
|
138
|
+
## How it works
|
|
139
|
+
|
|
140
|
+
When enabled, the plugin wraps `Client.send` and `AsyncClient.send` of `httpx2` (used by `anthropic` 1.x and 2.x) and `httpx` (older SDKs). Every request to `api.anthropic.com`, `$ANTHROPIC_BASE_URL` or a `--wirecheck-base-url` is recorded before it reaches the transport, so it is seen even when the test mocks the response with a mock transport, `respx`, or the `wirecheck_stub` fixture, and even when there is no network. API keys are redacted before anything is stored. The hook is removed when the run ends.
|
|
141
|
+
|
|
142
|
+
## Options
|
|
143
|
+
|
|
144
|
+
| Option | Meaning |
|
|
145
|
+
|---|---|
|
|
146
|
+
| `--wirecheck` | Turn it on (or `wirecheck = true` in your pytest ini, or `WIRECHECK=1`) |
|
|
147
|
+
| `--wirecheck-fail-on` | `error` (default), `warn`, or `never` |
|
|
148
|
+
| `--wirecheck-base-url URL` | Also capture a gateway or proxy URL (repeatable) |
|
|
149
|
+
| `--wirecheck-report PATH` | Write findings as JSON |
|
|
150
|
+
| `--wirecheck-markdown PATH` | Write findings as Markdown |
|
|
151
|
+
| `WIRECHECK_TODAY=YYYY-MM-DD` | Pin the date used by date-based rules |
|
|
152
|
+
|
|
153
|
+
Fixtures and markers:
|
|
154
|
+
|
|
155
|
+
- `wirecheck_stub` answers every Claude API request in a test with a canned reply, with no network. It mocks the response, not the client, so WireCheck still sees what you send. Set `wirecheck_stub["text"]` to change the reply.
|
|
156
|
+
- `wirecheck_requests` lists the requests the current test sent.
|
|
157
|
+
- `@pytest.mark.wirecheck_skip` ignores requests from one test.
|
|
158
|
+
|
|
159
|
+
## Known limits
|
|
160
|
+
|
|
161
|
+
- **It only sees requests that are really built.** If a test replaces `client.messages.create` (or the whole client) with a `MagicMock`, no request exists and WireCheck sees nothing. It prints a hint when a run captured no requests. Mock the HTTP response instead.
|
|
162
|
+
- Python only, and only the `httpx` / `httpx2` HTTP clients (the SDK default). The optional `aiohttp` backend is not hooked.
|
|
163
|
+
- Bedrock and Vertex requests go to other hosts and carry the version differently. Add their URL with `--wirecheck-base-url` if you want them recorded; version detection there falls back to the SDK version.
|
|
164
|
+
- `wirecheck_stub` answers non-streaming calls only.
|
|
165
|
+
- WC003 checks the places the guide names (user `text` blocks and `search_result` blocks). It does not flag bare `text` blocks inside `tool_result`, because the guide does not say.
|
|
166
|
+
- The static scanner is a set of simple patterns. It misses anything assembled at runtime, which is why the pytest plugin exists.
|
|
167
|
+
- No auto-fix and no TypeScript or other SDKs in v0.1.
|
|
168
|
+
|
|
169
|
+
## Licence
|
|
170
|
+
|
|
171
|
+
MIT. See [LICENSE](LICENSE).
|
|
172
|
+
|
|
173
|
+
---
|
|
174
|
+
|
|
175
|
+
Built by [Ishan Yash](https://ishanyash.com)
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
LICENSE
|
|
2
|
+
README.md
|
|
3
|
+
pyproject.toml
|
|
4
|
+
src/pytest_wirecheck.egg-info/PKG-INFO
|
|
5
|
+
src/pytest_wirecheck.egg-info/SOURCES.txt
|
|
6
|
+
src/pytest_wirecheck.egg-info/dependency_links.txt
|
|
7
|
+
src/pytest_wirecheck.egg-info/entry_points.txt
|
|
8
|
+
src/pytest_wirecheck.egg-info/requires.txt
|
|
9
|
+
src/pytest_wirecheck.egg-info/top_level.txt
|
|
10
|
+
src/wirecheck/__init__.py
|
|
11
|
+
src/wirecheck/__main__.py
|
|
12
|
+
src/wirecheck/capture.py
|
|
13
|
+
src/wirecheck/cli.py
|
|
14
|
+
src/wirecheck/clock.py
|
|
15
|
+
src/wirecheck/plugin.py
|
|
16
|
+
src/wirecheck/report.py
|
|
17
|
+
src/wirecheck/rules.py
|
|
18
|
+
src/wirecheck/scan.py
|
|
19
|
+
src/wirecheck/versions.py
|
|
20
|
+
tests/test_capture.py
|
|
21
|
+
tests/test_plugin.py
|
|
22
|
+
tests/test_rules.py
|
|
23
|
+
tests/test_scan.py
|
|
24
|
+
tests/test_sdk.py
|
|
25
|
+
tests/test_versions.py
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
wirecheck
|