httpx2-audit 0.1.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- httpx2_audit-0.1.0.dist-info/METADATA +108 -0
- httpx2_audit-0.1.0.dist-info/RECORD +7 -0
- httpx2_audit-0.1.0.dist-info/WHEEL +5 -0
- httpx2_audit-0.1.0.dist-info/entry_points.txt +2 -0
- httpx2_audit-0.1.0.dist-info/licenses/LICENSE +9 -0
- httpx2_audit-0.1.0.dist-info/top_level.txt +1 -0
- httpx2_audit.py +598 -0
|
@@ -0,0 +1,108 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: httpx2-audit
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Find what breaks when your Python stack moves from httpx to httpx2: static audit, import/dependency codemod and a mixed-stack/TLS environment report.
|
|
5
|
+
Author-email: agent-one <rafal.baranowski.iot@gmail.com>
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/rafal-one/httpx2-audit
|
|
8
|
+
Keywords: httpx,httpx2,migration,codemod,audit,openai,anthropic,respx,pytest-httpx
|
|
9
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
10
|
+
Classifier: Programming Language :: Python :: 3
|
|
11
|
+
Classifier: Topic :: Software Development :: Quality Assurance
|
|
12
|
+
Requires-Python: >=3.9
|
|
13
|
+
Description-Content-Type: text/markdown
|
|
14
|
+
License-File: LICENSE
|
|
15
|
+
Dynamic: license-file
|
|
16
|
+
|
|
17
|
+
# httpx2-audit
|
|
18
|
+
|
|
19
|
+
Find what breaks when your Python stack moves from **httpx** to **httpx2** (the pydantic fork; openai 3.x,
|
|
20
|
+
anthropic 1.x, Starlette, the MCP SDK and huggingface_hub already switched). Pure stdlib, Python 3.9+
|
|
21
|
+
(it analyses code, it does not import httpx), no network except the optional TLS probe.
|
|
22
|
+
|
|
23
|
+
Unofficial community tool, not affiliated with httpx, httpx2 or pydantic. State as of 2026-10-18.
|
|
24
|
+
|
|
25
|
+
## Why
|
|
26
|
+
|
|
27
|
+
The move breaks things *silently*: `import httpx` fails once an SDK stops pulling it in; `respx` and
|
|
28
|
+
`pytest-httpx` mocks stop intercepting, so tests hit the live API; `except httpx.ConnectError` no longer
|
|
29
|
+
catches errors raised by httpx2 code; httpx clients passed as `http_client=` are the wrong type; TLS switches
|
|
30
|
+
from the certifi bundle to the OS trust store.
|
|
31
|
+
|
|
32
|
+
```
|
|
33
|
+
pip install httpx2-audit # or: pip install git+https://github.com/rafal-one/httpx2-audit
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
## `check`: static audit
|
|
37
|
+
|
|
38
|
+
```
|
|
39
|
+
$ httpx2-audit check src/
|
|
40
|
+
src/app.py:1:1: HX001 imports httpx; httpx2 is a separate package with its own module name [info]
|
|
41
|
+
src/app.py:4:36: HX004 httpx object or http_client passed on; an httpx2-based SDK needs httpx2 objects (http_client) [warning]
|
|
42
|
+
src/app.py:7:8: HX003 except clause names an httpx exception; errors raised by httpx2-based code are not caught (httpx.ConnectError) [warning]
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
| Rule | Severity | Meaning |
|
|
46
|
+
|---|---|---|
|
|
47
|
+
| HX001 | info | `import httpx` / `from httpx import ...` |
|
|
48
|
+
| HX002 | error | httpx imported but not declared in pyproject/requirements (transitive install is gone) |
|
|
49
|
+
| HX003 | warning | `except httpx.X` (also aliases and tuples) |
|
|
50
|
+
| HX004 | warning | `http_client=` or any httpx object (`httpx.Timeout`, `httpx.Limits`, ...) passed to a non-httpx call |
|
|
51
|
+
| HX005 | warning | `isinstance`/`issubclass` against httpx classes |
|
|
52
|
+
| HX006 | warning | `import httpcore` (httpx2 uses httpcore2) |
|
|
53
|
+
| HX007 | warning | `respx` / `pytest_httpx` imports: they do not mock httpx2 |
|
|
54
|
+
| HX008 | info | `certifi.where()` passed explicitly while httpx is in use |
|
|
55
|
+
| HX010 | warning | file imports both httpx and httpx2 |
|
|
56
|
+
|
|
57
|
+
Options: `--format text|json|sarif` (SARIF for GitHub code scanning), `--min-severity`, `--fail-on
|
|
58
|
+
info|warning|error|never` (default `warning`), `-q` (no footer). Exit codes: `0` nothing at or above
|
|
59
|
+
`--fail-on`, `1` findings, `2` bad usage or missing path. Directories like `.venv`, `node_modules`, `build`
|
|
60
|
+
are skipped; files that do not parse are reported on stderr and skipped. Imports already wrapped in
|
|
61
|
+
`try: import httpx2 ... except ImportError: import httpx` are treated as handled.
|
|
62
|
+
|
|
63
|
+
CI: `httpx2-audit check . --format sarif > audit.sarif`, or just `httpx2-audit check . --fail-on error`.
|
|
64
|
+
|
|
65
|
+
## `fix`: import and dependency codemod
|
|
66
|
+
|
|
67
|
+
Dry run by default (prints a unified diff); `--write` applies it.
|
|
68
|
+
|
|
69
|
+
```
|
|
70
|
+
httpx2-audit fix . # diff
|
|
71
|
+
httpx2-audit fix . --write # apply
|
|
72
|
+
httpx2-audit fix . --dual --write # libraries: try httpx2, fall back to httpx
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
* `import httpx` -> `import httpx2 as httpx` (so the rest of the file keeps working), `from httpx import X`
|
|
76
|
+
-> `from httpx2 import X`; same for `httpcore` -> `httpcore2` and `pytest_httpx` -> `pytest_httpx2`.
|
|
77
|
+
* `requirements*.txt` and `pyproject.toml` requirement strings: `httpx` -> `httpx2>=2.0`, `httpcore`,
|
|
78
|
+
`pytest-httpx` (extras and markers are kept).
|
|
79
|
+
* `--dual` wraps each import in a `try/except ImportError` and leaves dependencies alone.
|
|
80
|
+
* Private modules (`httpx._models`) and dotted imports are reported as `manual:` and left untouched.
|
|
81
|
+
* It does not touch `except` clauses, `respx` mocks or TLS settings; run `check` afterwards. httpx2 needs Python 3.10+.
|
|
82
|
+
|
|
83
|
+
## `env`: mixed stacks and TLS
|
|
84
|
+
|
|
85
|
+
```
|
|
86
|
+
$ httpx2-audit env --probe api.openai.com --probe intranet.corp:8443
|
|
87
|
+
installed:
|
|
88
|
+
httpx 0.28.1
|
|
89
|
+
httpx2 2.14.0
|
|
90
|
+
depend on httpx: respx
|
|
91
|
+
depend on httpx2: pytest-httpx2
|
|
92
|
+
tls intranet.corp:8443: openssl-default=ok, certifi=FAIL: CERTIFICATE_VERIFY_FAILED, truststore=ok
|
|
93
|
+
WARNING: TLS trust differs for intranet.corp:8443: ...
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
Lists the relevant installed distributions, which installed packages require httpx or httpx2 (a mixed
|
|
97
|
+
environment cannot pass clients between them), and compares what the system store, certifi and truststore
|
|
98
|
+
(if installed) accept for the hosts you probe. Exit code `1` when warnings are printed.
|
|
99
|
+
|
|
100
|
+
## Development
|
|
101
|
+
|
|
102
|
+
`pytest` (37 tests). Tested on Python 3.9 and 3.14; clean installs of wheel and sdist checked.
|
|
103
|
+
|
|
104
|
+
## Donate
|
|
105
|
+
|
|
106
|
+
If this saved your migration a day: https://paypal.me/RafalBaranowski
|
|
107
|
+
|
|
108
|
+
MIT licensed.
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
httpx2_audit.py,sha256=a9uJCWNlIA9gG6fK4qa6pmQeWgtYCBsTR891bgg96o0,24474
|
|
2
|
+
httpx2_audit-0.1.0.dist-info/licenses/LICENSE,sha256=-hZriFiksFsGGM2WbhdnQzoXUOFuMGve5P3y4pvA0Ws,1066
|
|
3
|
+
httpx2_audit-0.1.0.dist-info/METADATA,sha256=Qcgu6nUbcb7cY0zc3At8OoPHzLRNg29MlOvCR08EPl0,5207
|
|
4
|
+
httpx2_audit-0.1.0.dist-info/WHEEL,sha256=YVMoNqKzERt-wjUZwJ33xBGAwnFl-4cqbYkTtWa4itE,91
|
|
5
|
+
httpx2_audit-0.1.0.dist-info/entry_points.txt,sha256=6ehiU6UNvfmy90hhE6TLmVgcbDYz9zKxXj41kNbGfec,51
|
|
6
|
+
httpx2_audit-0.1.0.dist-info/top_level.txt,sha256=BfRObduluR_FQQDRTFdEq4SuTKxawIVsYSiTprWPl2U,13
|
|
7
|
+
httpx2_audit-0.1.0.dist-info/RECORD,,
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 agent-one
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
|
|
6
|
+
|
|
7
|
+
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
|
|
8
|
+
|
|
9
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
httpx2_audit
|
httpx2_audit.py
ADDED
|
@@ -0,0 +1,598 @@
|
|
|
1
|
+
"""httpx2-audit: find what breaks when a Python stack moves from httpx to httpx2."""
|
|
2
|
+
import argparse
|
|
3
|
+
import ast
|
|
4
|
+
import json
|
|
5
|
+
import os
|
|
6
|
+
import re
|
|
7
|
+
import socket
|
|
8
|
+
import ssl
|
|
9
|
+
import sys
|
|
10
|
+
from importlib import metadata
|
|
11
|
+
|
|
12
|
+
__version__ = "0.1.0"
|
|
13
|
+
|
|
14
|
+
DONATE = "https://paypal.me/RafalBaranowski"
|
|
15
|
+
SKIP_DIRS = {".git", ".hg", ".venv", "venv", "env", "node_modules", "build", "dist",
|
|
16
|
+
"site-packages", "__pycache__", ".tox", ".nox", ".mypy_cache", ".eggs"}
|
|
17
|
+
SEV_ORDER = {"info": 0, "warning": 1, "error": 2}
|
|
18
|
+
|
|
19
|
+
RULES = {
|
|
20
|
+
"HX001": ("info", "imports httpx; httpx2 is a separate package with its own module name"),
|
|
21
|
+
"HX002": ("error", "httpx is imported but not declared as a dependency; SDKs that moved to httpx2 no longer install it"),
|
|
22
|
+
"HX003": ("warning", "except clause names an httpx exception; errors raised by httpx2-based code are not caught"),
|
|
23
|
+
"HX004": ("warning", "httpx object or http_client passed on; an httpx2-based SDK needs httpx2 objects"),
|
|
24
|
+
"HX005": ("warning", "isinstance/issubclass against an httpx class fails for httpx2 objects"),
|
|
25
|
+
"HX006": ("warning", "imports httpcore; httpx2 is built on httpcore2"),
|
|
26
|
+
"HX007": ("warning", "test mock library for httpx; it does not intercept httpx2 requests, so tests may reach the live API"),
|
|
27
|
+
"HX008": ("info", "certifi bundle passed explicitly; httpx2 defaults to the OS trust store (truststore)"),
|
|
28
|
+
"HX010": ("warning", "file imports both httpx and httpx2"),
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
# import name -> (distribution, replacement import name, replacement distribution spec)
|
|
32
|
+
SWAP_MODULES = {
|
|
33
|
+
"httpx": "httpx2",
|
|
34
|
+
"httpcore": "httpcore2",
|
|
35
|
+
"pytest_httpx": "pytest_httpx2",
|
|
36
|
+
}
|
|
37
|
+
SWAP_DISTS = {
|
|
38
|
+
"httpx": ("httpx2", ">=2.0"),
|
|
39
|
+
"httpcore": ("httpcore2", ">=2.0"),
|
|
40
|
+
"pytest-httpx": ("pytest-httpx2", ">=1.0"),
|
|
41
|
+
}
|
|
42
|
+
MOCK_MODULES = {"respx", "pytest_httpx"}
|
|
43
|
+
WATCH_DISTS = {"httpx", "httpx2", "httpcore", "httpcore2", "respx", "pytest-httpx", "pytest-httpx2"}
|
|
44
|
+
|
|
45
|
+
|
|
46
|
+
def norm(name):
|
|
47
|
+
return re.sub(r"[-_.]+", "-", name).lower()
|
|
48
|
+
|
|
49
|
+
|
|
50
|
+
class Finding(dict):
|
|
51
|
+
pass
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
def finding(rule, path, line, col, detail=""):
|
|
55
|
+
sev, msg = RULES[rule]
|
|
56
|
+
return Finding(rule=rule, severity=sev, path=path, line=line, col=col,
|
|
57
|
+
message=msg + (" (%s)" % detail if detail else ""))
|
|
58
|
+
|
|
59
|
+
|
|
60
|
+
# ----------------------------------------------------------------- manifests
|
|
61
|
+
|
|
62
|
+
REQ_RE = re.compile(r"^\s*([A-Za-z0-9][A-Za-z0-9._-]*)\s*(\[[^\]]*\])?\s*([<>=!~].*?)?\s*(;.*)?$")
|
|
63
|
+
|
|
64
|
+
|
|
65
|
+
def iter_files(roots, pred):
|
|
66
|
+
for root in roots:
|
|
67
|
+
if os.path.isfile(root):
|
|
68
|
+
if pred(os.path.basename(root)):
|
|
69
|
+
yield root
|
|
70
|
+
continue
|
|
71
|
+
for dp, dns, fns in os.walk(root):
|
|
72
|
+
dns[:] = sorted(d for d in dns if d not in SKIP_DIRS and not d.endswith(".egg-info"))
|
|
73
|
+
for fn in sorted(fns):
|
|
74
|
+
if pred(fn):
|
|
75
|
+
yield os.path.join(dp, fn)
|
|
76
|
+
|
|
77
|
+
|
|
78
|
+
def is_manifest(fn):
|
|
79
|
+
return fn == "pyproject.toml" or (fn.startswith("requirements") and fn.endswith((".txt", ".in")))
|
|
80
|
+
|
|
81
|
+
|
|
82
|
+
def manifest_requirements(path):
|
|
83
|
+
"""Yield (lineno, requirement-name) for requirement-looking lines/strings."""
|
|
84
|
+
try:
|
|
85
|
+
with open(path, encoding="utf-8") as f:
|
|
86
|
+
lines = f.read().splitlines()
|
|
87
|
+
except (OSError, UnicodeDecodeError):
|
|
88
|
+
return
|
|
89
|
+
for i, line in enumerate(lines, 1):
|
|
90
|
+
if path.endswith("pyproject.toml"):
|
|
91
|
+
for m in re.finditer(r"""["']([^"'\n]+)["']""", line):
|
|
92
|
+
r = REQ_RE.match(m.group(1))
|
|
93
|
+
if r and norm(r.group(1)) in WATCH_DISTS:
|
|
94
|
+
yield i, norm(r.group(1))
|
|
95
|
+
else:
|
|
96
|
+
s = line.split("#", 1)[0].strip()
|
|
97
|
+
if s and not s.startswith("-"):
|
|
98
|
+
r = REQ_RE.match(s)
|
|
99
|
+
if r:
|
|
100
|
+
yield i, norm(r.group(1))
|
|
101
|
+
|
|
102
|
+
|
|
103
|
+
def declared_deps(roots):
|
|
104
|
+
found = set()
|
|
105
|
+
manifests = list(iter_files(roots, is_manifest))
|
|
106
|
+
for m in manifests:
|
|
107
|
+
for _, name in manifest_requirements(m):
|
|
108
|
+
found.add(name)
|
|
109
|
+
return found, manifests
|
|
110
|
+
|
|
111
|
+
|
|
112
|
+
# ------------------------------------------------------------------- scanner
|
|
113
|
+
|
|
114
|
+
class Scanner(ast.NodeVisitor):
|
|
115
|
+
def __init__(self, path, tree):
|
|
116
|
+
self.path = path
|
|
117
|
+
self.out = []
|
|
118
|
+
self.mod_alias = {} # local name -> module ("httpx"/"httpcore"/...)
|
|
119
|
+
self.name_alias = {} # local name -> "httpx.Client"
|
|
120
|
+
self.dual = set() # ids of import nodes inside try/except ImportError
|
|
121
|
+
self.imports = [] # (node, module) for httpx imports (for HX002/HX010)
|
|
122
|
+
self.uses_httpx = False
|
|
123
|
+
self.uses_httpx2 = False
|
|
124
|
+
self._mark_dual(tree)
|
|
125
|
+
self.tree = tree
|
|
126
|
+
|
|
127
|
+
def add(self, rule, node, detail=""):
|
|
128
|
+
self.out.append(finding(rule, self.path, node.lineno, node.col_offset + 1, detail))
|
|
129
|
+
|
|
130
|
+
def _mark_dual(self, tree):
|
|
131
|
+
for node in ast.walk(tree):
|
|
132
|
+
if isinstance(node, ast.Try):
|
|
133
|
+
catches = False
|
|
134
|
+
for h in node.handlers:
|
|
135
|
+
names = []
|
|
136
|
+
if h.type is not None:
|
|
137
|
+
names = [n for n in ast.walk(h.type) if isinstance(n, ast.Name)]
|
|
138
|
+
if h.type is None or any(n.id in ("ImportError", "ModuleNotFoundError") for n in names):
|
|
139
|
+
catches = True
|
|
140
|
+
if not catches:
|
|
141
|
+
continue
|
|
142
|
+
for part in (node.body, *[h.body for h in node.handlers]):
|
|
143
|
+
for st in part:
|
|
144
|
+
if isinstance(st, (ast.Import, ast.ImportFrom)):
|
|
145
|
+
self.dual.add(id(st))
|
|
146
|
+
|
|
147
|
+
def resolve(self, node):
|
|
148
|
+
"""Dotted name for Name/Attribute chains, with import aliases resolved."""
|
|
149
|
+
parts = []
|
|
150
|
+
while isinstance(node, ast.Attribute):
|
|
151
|
+
parts.append(node.attr)
|
|
152
|
+
node = node.value
|
|
153
|
+
if not isinstance(node, ast.Name):
|
|
154
|
+
return None
|
|
155
|
+
head = node.id
|
|
156
|
+
if head in self.name_alias:
|
|
157
|
+
base = self.name_alias[head]
|
|
158
|
+
elif head in self.mod_alias:
|
|
159
|
+
base = self.mod_alias[head]
|
|
160
|
+
else:
|
|
161
|
+
return None
|
|
162
|
+
return ".".join([base] + parts[::-1])
|
|
163
|
+
|
|
164
|
+
def mentions(self, node, prefix="httpx"):
|
|
165
|
+
for n in ast.walk(node):
|
|
166
|
+
if isinstance(n, (ast.Name, ast.Attribute)):
|
|
167
|
+
r = self.resolve(n)
|
|
168
|
+
if r and (r == prefix or r.startswith(prefix + ".")):
|
|
169
|
+
return True
|
|
170
|
+
return False
|
|
171
|
+
|
|
172
|
+
def visit_Import(self, node):
|
|
173
|
+
for a in node.names:
|
|
174
|
+
top = a.name.split(".")[0]
|
|
175
|
+
local = a.asname or top
|
|
176
|
+
if top in ("httpx", "httpx2", "httpcore", "httpcore2", "respx", "pytest_httpx", "certifi"):
|
|
177
|
+
self.mod_alias[local] = top if a.asname is None or "." not in a.name else a.name
|
|
178
|
+
self._module_seen(node, a.name)
|
|
179
|
+
self.generic_visit(node)
|
|
180
|
+
|
|
181
|
+
def visit_ImportFrom(self, node):
|
|
182
|
+
if node.level == 0 and node.module:
|
|
183
|
+
top = node.module.split(".")[0]
|
|
184
|
+
for a in node.names:
|
|
185
|
+
if top in ("httpx", "httpx2", "httpcore", "httpcore2", "respx", "pytest_httpx"):
|
|
186
|
+
self.name_alias[a.asname or a.name] = node.module + "." + a.name
|
|
187
|
+
self._module_seen(node, node.module)
|
|
188
|
+
self.generic_visit(node)
|
|
189
|
+
|
|
190
|
+
def _module_seen(self, node, module):
|
|
191
|
+
top = module.split(".")[0]
|
|
192
|
+
is_dual = id(node) in self.dual
|
|
193
|
+
if top == "httpx2":
|
|
194
|
+
self.uses_httpx2 = True
|
|
195
|
+
if top == "httpx":
|
|
196
|
+
self.uses_httpx = not is_dual or self.uses_httpx
|
|
197
|
+
if not is_dual:
|
|
198
|
+
self.imports.append((node, top))
|
|
199
|
+
self.add("HX001", node)
|
|
200
|
+
elif top == "httpcore" and not is_dual:
|
|
201
|
+
self.add("HX006", node)
|
|
202
|
+
elif top in MOCK_MODULES:
|
|
203
|
+
self.add("HX007", node, top)
|
|
204
|
+
|
|
205
|
+
def visit_ExceptHandler(self, node):
|
|
206
|
+
if node.type is not None:
|
|
207
|
+
elts = node.type.elts if isinstance(node.type, ast.Tuple) else [node.type]
|
|
208
|
+
for e in elts:
|
|
209
|
+
r = self.resolve(e)
|
|
210
|
+
if r and r.startswith("httpx."):
|
|
211
|
+
self.add("HX003", e, r)
|
|
212
|
+
self.generic_visit(node)
|
|
213
|
+
|
|
214
|
+
def visit_Call(self, node):
|
|
215
|
+
fn = self.resolve(node.func)
|
|
216
|
+
if isinstance(node.func, ast.Name) and node.func.id in ("isinstance", "issubclass") and len(node.args) >= 2:
|
|
217
|
+
if self.mentions(node.args[1]):
|
|
218
|
+
self.add("HX005", node, self.resolve(node.args[1]) or "")
|
|
219
|
+
callee_httpx = bool(fn and (fn == "httpx" or fn.startswith("httpx.")))
|
|
220
|
+
for kw in node.keywords:
|
|
221
|
+
if kw.arg is None or callee_httpx:
|
|
222
|
+
continue
|
|
223
|
+
if kw.arg in ("http_client", "http_async_client") or self.mentions(kw.value):
|
|
224
|
+
self.add("HX004", kw.value, kw.arg)
|
|
225
|
+
if fn == "certifi.where" and (self.uses_httpx or self.uses_httpx2):
|
|
226
|
+
self.add("HX008", node)
|
|
227
|
+
self.generic_visit(node)
|
|
228
|
+
|
|
229
|
+
|
|
230
|
+
def scan_file(path):
|
|
231
|
+
try:
|
|
232
|
+
with open(path, "rb") as f:
|
|
233
|
+
src = f.read()
|
|
234
|
+
tree = ast.parse(src, filename=path)
|
|
235
|
+
except (OSError, SyntaxError, ValueError) as e:
|
|
236
|
+
return None, str(e)
|
|
237
|
+
sc = Scanner(path, tree)
|
|
238
|
+
# first pass records imports/aliases so later uses resolve; imports come first in practice
|
|
239
|
+
sc.visit(tree)
|
|
240
|
+
if sc.uses_httpx and sc.uses_httpx2:
|
|
241
|
+
sc.add("HX010", sc.imports[0][0] if sc.imports else tree.body[0])
|
|
242
|
+
return sc, None
|
|
243
|
+
|
|
244
|
+
|
|
245
|
+
def run_check(paths, declared=None):
|
|
246
|
+
findings, skipped, nfiles = [], [], 0
|
|
247
|
+
if declared is None:
|
|
248
|
+
declared, manifests = declared_deps(paths)
|
|
249
|
+
else:
|
|
250
|
+
manifests = []
|
|
251
|
+
for p in iter_files(paths, lambda fn: fn.endswith(".py")):
|
|
252
|
+
nfiles += 1
|
|
253
|
+
sc, err = scan_file(p)
|
|
254
|
+
if sc is None:
|
|
255
|
+
skipped.append({"path": p, "error": err})
|
|
256
|
+
continue
|
|
257
|
+
findings.extend(sc.out)
|
|
258
|
+
if sc.imports and manifests and not ({"httpx", "httpx2"} & declared):
|
|
259
|
+
node = sc.imports[0][0]
|
|
260
|
+
findings.append(finding("HX002", p, node.lineno, node.col_offset + 1))
|
|
261
|
+
findings.sort(key=lambda f: (f["path"], f["line"], f["col"], f["rule"]))
|
|
262
|
+
return {"findings": findings, "skipped": skipped, "files": nfiles,
|
|
263
|
+
"manifests": manifests, "declared": sorted(declared)}
|
|
264
|
+
|
|
265
|
+
|
|
266
|
+
# ---------------------------------------------------------------------- fix
|
|
267
|
+
|
|
268
|
+
def _swap_import(node, lines, dual):
|
|
269
|
+
"""Return (start_line, end_line, new_text) or (None, reason)."""
|
|
270
|
+
start, end = node.lineno - 1, node.end_lineno - 1
|
|
271
|
+
seg = lines[start:end + 1]
|
|
272
|
+
first = seg[0]
|
|
273
|
+
indent = first[:node.col_offset]
|
|
274
|
+
if first[:node.col_offset].strip():
|
|
275
|
+
return None, "statement does not start its line"
|
|
276
|
+
text = "\n".join(seg)
|
|
277
|
+
text = text[node.col_offset:]
|
|
278
|
+
if ";" in text.split("#", 1)[0]:
|
|
279
|
+
return None, "multiple statements on one line"
|
|
280
|
+
if isinstance(node, ast.Import):
|
|
281
|
+
if any("." in a.name for a in node.names):
|
|
282
|
+
return None, "dotted import of a private/submodule"
|
|
283
|
+
if dual:
|
|
284
|
+
if len(node.names) != 1:
|
|
285
|
+
return None, "multi-name import in dual mode"
|
|
286
|
+
a = node.names[0]
|
|
287
|
+
local = a.asname or a.name
|
|
288
|
+
new = SWAP_MODULES[a.name]
|
|
289
|
+
body = "%stry:\n%s import %s as %s\n%sexcept ImportError: # pragma: no cover\n%s %s" % (
|
|
290
|
+
indent, indent, new, local, indent, indent, text)
|
|
291
|
+
return (start, end, body), None
|
|
292
|
+
parts = []
|
|
293
|
+
for a in node.names:
|
|
294
|
+
if a.name in SWAP_MODULES:
|
|
295
|
+
parts.append("%s as %s" % (SWAP_MODULES[a.name], a.asname or a.name))
|
|
296
|
+
else:
|
|
297
|
+
parts.append(a.name + (" as " + a.asname if a.asname else ""))
|
|
298
|
+
return (start, end, indent + "import " + ", ".join(parts)), None
|
|
299
|
+
# ImportFrom
|
|
300
|
+
if node.module != node.module.split(".")[0] and node.module.split(".")[0] in SWAP_MODULES:
|
|
301
|
+
return None, "import from a private/submodule (%s)" % node.module
|
|
302
|
+
new_mod = SWAP_MODULES[node.module]
|
|
303
|
+
new_text = re.sub(r"\bfrom(\s+)%s\b" % re.escape(node.module), r"from\g<1>%s" % new_mod, text, count=1)
|
|
304
|
+
if dual:
|
|
305
|
+
body = "%stry:\n%s %s\n%sexcept ImportError: # pragma: no cover\n%s %s" % (
|
|
306
|
+
indent, indent, new_text.replace("\n", "\n "), indent, indent, text.replace("\n", "\n "))
|
|
307
|
+
return (start, end, body), None
|
|
308
|
+
return (start, end, indent + new_text), None
|
|
309
|
+
|
|
310
|
+
|
|
311
|
+
def fix_source(src, dual=False):
|
|
312
|
+
"""Return (new_source, changes, unfixable) for one Python file's source text."""
|
|
313
|
+
try:
|
|
314
|
+
tree = ast.parse(src)
|
|
315
|
+
except SyntaxError:
|
|
316
|
+
return src, [], []
|
|
317
|
+
sc = Scanner("<s>", tree)
|
|
318
|
+
sc.visit(tree)
|
|
319
|
+
lines = src.split("\n")
|
|
320
|
+
edits, unfixable = [], []
|
|
321
|
+
for node in ast.walk(tree):
|
|
322
|
+
if id(node) in sc.dual:
|
|
323
|
+
continue
|
|
324
|
+
mod = None
|
|
325
|
+
if isinstance(node, ast.Import):
|
|
326
|
+
mods = [a.name.split(".")[0] for a in node.names]
|
|
327
|
+
if any(m in SWAP_MODULES for m in mods):
|
|
328
|
+
mod = True
|
|
329
|
+
elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module:
|
|
330
|
+
if node.module.split(".")[0] in SWAP_MODULES:
|
|
331
|
+
mod = True
|
|
332
|
+
if not mod:
|
|
333
|
+
continue
|
|
334
|
+
res, reason = _swap_import(node, lines, dual)
|
|
335
|
+
if res is None:
|
|
336
|
+
unfixable.append((node.lineno, reason))
|
|
337
|
+
else:
|
|
338
|
+
edits.append((res, node.lineno))
|
|
339
|
+
changes = []
|
|
340
|
+
for (s, e, text), ln in sorted(edits, key=lambda x: -x[0][0]):
|
|
341
|
+
lines[s:e + 1] = text.split("\n")
|
|
342
|
+
changes.append(ln)
|
|
343
|
+
return "\n".join(lines), sorted(changes), unfixable
|
|
344
|
+
|
|
345
|
+
|
|
346
|
+
def fix_manifest_text(text, is_pyproject):
|
|
347
|
+
"""Swap httpx/httpcore/pytest-httpx requirements for their *2 counterparts."""
|
|
348
|
+
changes = []
|
|
349
|
+
|
|
350
|
+
def repl_req(s):
|
|
351
|
+
m = REQ_RE.match(s)
|
|
352
|
+
if not m:
|
|
353
|
+
return s
|
|
354
|
+
n = norm(m.group(1))
|
|
355
|
+
if n not in SWAP_DISTS:
|
|
356
|
+
return s
|
|
357
|
+
new, spec = SWAP_DISTS[n]
|
|
358
|
+
return "%s%s%s%s" % (new, m.group(2) or "", spec, (" " + m.group(4).strip()) if m.group(4) else "")
|
|
359
|
+
|
|
360
|
+
out = []
|
|
361
|
+
for i, line in enumerate(text.split("\n"), 1):
|
|
362
|
+
new = line
|
|
363
|
+
if is_pyproject:
|
|
364
|
+
new = re.sub(r"""(["'])([^"'\n]+)\1""", lambda m: m.group(1) + repl_req(m.group(2)) + m.group(1), line)
|
|
365
|
+
else:
|
|
366
|
+
body, sep, comment = line.partition("#")
|
|
367
|
+
if body.strip() and not body.strip().startswith("-"):
|
|
368
|
+
lead = body[:len(body) - len(body.lstrip())]
|
|
369
|
+
trail = " " if sep and body.endswith(" ") else ""
|
|
370
|
+
new = lead + repl_req(body.strip()) + trail + sep + comment
|
|
371
|
+
if new != line:
|
|
372
|
+
changes.append(i)
|
|
373
|
+
out.append(new)
|
|
374
|
+
return "\n".join(out), changes
|
|
375
|
+
|
|
376
|
+
|
|
377
|
+
def run_fix(paths, write=False, dual=False):
|
|
378
|
+
import difflib
|
|
379
|
+
report = {"files": [], "unfixable": [], "diff": ""}
|
|
380
|
+
diffs = []
|
|
381
|
+
for p in iter_files(paths, lambda fn: fn.endswith(".py")):
|
|
382
|
+
try:
|
|
383
|
+
with open(p, encoding="utf-8") as f:
|
|
384
|
+
src = f.read()
|
|
385
|
+
except (OSError, UnicodeDecodeError):
|
|
386
|
+
continue
|
|
387
|
+
new, changes, unfix = fix_source(src, dual)
|
|
388
|
+
for ln, why in unfix:
|
|
389
|
+
report["unfixable"].append({"path": p, "line": ln, "reason": why})
|
|
390
|
+
if new != src:
|
|
391
|
+
report["files"].append(p)
|
|
392
|
+
diffs.append("".join(difflib.unified_diff(src.splitlines(True), new.splitlines(True), p, p)))
|
|
393
|
+
if write:
|
|
394
|
+
with open(p, "w", encoding="utf-8") as f:
|
|
395
|
+
f.write(new)
|
|
396
|
+
if not dual:
|
|
397
|
+
for m in iter_files(paths, is_manifest):
|
|
398
|
+
try:
|
|
399
|
+
with open(m, encoding="utf-8") as f:
|
|
400
|
+
src = f.read()
|
|
401
|
+
except (OSError, UnicodeDecodeError):
|
|
402
|
+
continue
|
|
403
|
+
new, changes = fix_manifest_text(src, m.endswith("pyproject.toml"))
|
|
404
|
+
if new != src:
|
|
405
|
+
report["files"].append(m)
|
|
406
|
+
diffs.append("".join(difflib.unified_diff(src.splitlines(True), new.splitlines(True), m, m)))
|
|
407
|
+
if write:
|
|
408
|
+
with open(m, "w", encoding="utf-8") as f:
|
|
409
|
+
f.write(new)
|
|
410
|
+
report["diff"] = "".join(diffs)
|
|
411
|
+
return report
|
|
412
|
+
|
|
413
|
+
|
|
414
|
+
# ----------------------------------------------------------------------- env
|
|
415
|
+
|
|
416
|
+
def installed_report():
|
|
417
|
+
dists = {}
|
|
418
|
+
for d in metadata.distributions():
|
|
419
|
+
name = norm(d.metadata["Name"] or "")
|
|
420
|
+
if name:
|
|
421
|
+
dists[name] = d
|
|
422
|
+
rows = {n: dists[n].version for n in sorted(WATCH_DISTS | {"certifi", "truststore", "openai", "anthropic",
|
|
423
|
+
"mcp", "starlette", "huggingface-hub"}) if n in dists}
|
|
424
|
+
uses = {"httpx": [], "httpx2": []}
|
|
425
|
+
for n, d in sorted(dists.items()):
|
|
426
|
+
for r in d.requires or []:
|
|
427
|
+
m = REQ_RE.match(r.split(";")[0].strip())
|
|
428
|
+
if not m or "extra ==" in r.replace('"', "'").replace("extra==", "extra =="):
|
|
429
|
+
continue
|
|
430
|
+
rn = norm(m.group(1))
|
|
431
|
+
if rn in uses and n not in ("httpx", "httpx2"):
|
|
432
|
+
uses[rn].append(n)
|
|
433
|
+
return rows, uses
|
|
434
|
+
|
|
435
|
+
|
|
436
|
+
def tls_probe(host, port=443, timeout=5.0):
|
|
437
|
+
result = {}
|
|
438
|
+
ctxs = {"openssl-default": ssl.create_default_context()}
|
|
439
|
+
try:
|
|
440
|
+
import certifi
|
|
441
|
+
ctxs["certifi"] = ssl.create_default_context(cafile=certifi.where())
|
|
442
|
+
except ImportError:
|
|
443
|
+
pass
|
|
444
|
+
try:
|
|
445
|
+
import truststore
|
|
446
|
+
ctxs["truststore"] = truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
|
|
447
|
+
except ImportError:
|
|
448
|
+
pass
|
|
449
|
+
for name, ctx in ctxs.items():
|
|
450
|
+
try:
|
|
451
|
+
with socket.create_connection((host, port), timeout=timeout) as s:
|
|
452
|
+
with ctx.wrap_socket(s, server_hostname=host):
|
|
453
|
+
result[name] = "ok"
|
|
454
|
+
except (ssl.SSLError, OSError) as e:
|
|
455
|
+
result[name] = "FAIL: %s" % (getattr(e, "reason", None) or e)
|
|
456
|
+
return result
|
|
457
|
+
|
|
458
|
+
|
|
459
|
+
def run_env(probes):
|
|
460
|
+
rows, uses = installed_report()
|
|
461
|
+
warnings = []
|
|
462
|
+
if uses["httpx"] and uses["httpx2"]:
|
|
463
|
+
warnings.append("mixed stack: %s need httpx, %s need httpx2; objects cannot cross over" %
|
|
464
|
+
(", ".join(uses["httpx"]), ", ".join(uses["httpx2"])))
|
|
465
|
+
if "respx" in rows or "pytest-httpx" in rows:
|
|
466
|
+
if "httpx2" in rows and "pytest-httpx2" not in rows:
|
|
467
|
+
warnings.append("httpx2 installed but only the httpx mock libraries are: tests will not be intercepted")
|
|
468
|
+
if "httpx2" in rows and sys.version_info < (3, 10):
|
|
469
|
+
warnings.append("httpx2 requires Python 3.10+")
|
|
470
|
+
tls = {}
|
|
471
|
+
for p in probes:
|
|
472
|
+
host, _, port = p.partition(":")
|
|
473
|
+
tls[p] = tls_probe(host, int(port) if port else 443)
|
|
474
|
+
vals = list(tls[p].values())
|
|
475
|
+
if len(set(v == "ok" for v in vals)) > 1:
|
|
476
|
+
warnings.append("TLS trust differs for %s: %s" % (p, ", ".join("%s=%s" % kv for kv in tls[p].items())))
|
|
477
|
+
return {"installed": rows, "httpx_dependents": uses["httpx"], "httpx2_dependents": uses["httpx2"],
|
|
478
|
+
"tls": tls, "warnings": warnings}
|
|
479
|
+
|
|
480
|
+
|
|
481
|
+
# -------------------------------------------------------------------- output
|
|
482
|
+
|
|
483
|
+
def to_sarif(findings):
|
|
484
|
+
rules = [{"id": k, "shortDescription": {"text": v[1]}} for k, v in RULES.items()]
|
|
485
|
+
lvl = {"info": "note", "warning": "warning", "error": "error"}
|
|
486
|
+
return {"version": "2.1.0", "$schema": "https://json.schemastore.org/sarif-2.1.0.json",
|
|
487
|
+
"runs": [{"tool": {"driver": {"name": "httpx2-audit", "version": __version__,
|
|
488
|
+
"informationUri": "https://github.com/rafal-one/httpx2-audit",
|
|
489
|
+
"rules": rules}},
|
|
490
|
+
"results": [{"ruleId": f["rule"], "level": lvl[f["severity"]],
|
|
491
|
+
"message": {"text": f["message"]},
|
|
492
|
+
"locations": [{"physicalLocation": {
|
|
493
|
+
"artifactLocation": {"uri": f["path"].replace(os.sep, "/")},
|
|
494
|
+
"region": {"startLine": f["line"], "startColumn": f["col"]}}}]}
|
|
495
|
+
for f in findings]}]}
|
|
496
|
+
|
|
497
|
+
|
|
498
|
+
def footer():
|
|
499
|
+
return "httpx2-audit is free (MIT). If it saved you time: %s" % DONATE
|
|
500
|
+
|
|
501
|
+
|
|
502
|
+
def cmd_check(a):
|
|
503
|
+
for p in a.paths:
|
|
504
|
+
if not os.path.exists(p):
|
|
505
|
+
print("httpx2-audit: no such path: %s" % p, file=sys.stderr)
|
|
506
|
+
return 2
|
|
507
|
+
res = run_check(a.paths)
|
|
508
|
+
floor = SEV_ORDER[a.min_severity]
|
|
509
|
+
shown = [f for f in res["findings"] if SEV_ORDER[f["severity"]] >= floor]
|
|
510
|
+
if a.format == "json":
|
|
511
|
+
print(json.dumps({"findings": shown, "skipped": res["skipped"], "files": res["files"],
|
|
512
|
+
"declared": res["declared"]}, indent=2))
|
|
513
|
+
elif a.format == "sarif":
|
|
514
|
+
print(json.dumps(to_sarif(shown), indent=2))
|
|
515
|
+
else:
|
|
516
|
+
for f in shown:
|
|
517
|
+
print("%s:%d:%d: %s %s [%s]" % (f["path"], f["line"], f["col"], f["rule"], f["message"], f["severity"]))
|
|
518
|
+
for s in res["skipped"]:
|
|
519
|
+
print("%s: skipped (%s)" % (s["path"], s["error"]), file=sys.stderr)
|
|
520
|
+
print("\n%d finding(s) in %d file(s) scanned." % (len(shown), res["files"]))
|
|
521
|
+
if not a.quiet:
|
|
522
|
+
print(footer())
|
|
523
|
+
if a.fail_on == "never":
|
|
524
|
+
return 0
|
|
525
|
+
bad = [f for f in res["findings"] if SEV_ORDER[f["severity"]] >= SEV_ORDER[a.fail_on]]
|
|
526
|
+
return 1 if bad else 0
|
|
527
|
+
|
|
528
|
+
|
|
529
|
+
def cmd_fix(a):
|
|
530
|
+
for p in a.paths:
|
|
531
|
+
if not os.path.exists(p):
|
|
532
|
+
print("httpx2-audit: no such path: %s" % p, file=sys.stderr)
|
|
533
|
+
return 2
|
|
534
|
+
rep = run_fix(a.paths, write=a.write, dual=a.dual)
|
|
535
|
+
if rep["diff"]:
|
|
536
|
+
print(rep["diff"], end="")
|
|
537
|
+
for u in rep["unfixable"]:
|
|
538
|
+
print("manual: %s:%d: %s" % (u["path"], u["line"], u["reason"]), file=sys.stderr)
|
|
539
|
+
verb = "rewrote" if a.write else "would rewrite (use --write)"
|
|
540
|
+
print("%s %d file(s)." % (verb, len(rep["files"])))
|
|
541
|
+
if a.dual:
|
|
542
|
+
print("dual mode: imports prefer httpx2 and fall back to httpx; dependencies unchanged.")
|
|
543
|
+
elif rep["files"]:
|
|
544
|
+
print("note: httpx2 needs Python 3.10+. Then run `httpx2-audit check` for what imports cannot fix "
|
|
545
|
+
"(except clauses, mocks, TLS).")
|
|
546
|
+
return 0
|
|
547
|
+
|
|
548
|
+
|
|
549
|
+
def cmd_env(a):
|
|
550
|
+
rep = run_env(a.probe)
|
|
551
|
+
if a.format == "json":
|
|
552
|
+
print(json.dumps(rep, indent=2))
|
|
553
|
+
else:
|
|
554
|
+
print("installed:")
|
|
555
|
+
for n, v in rep["installed"].items():
|
|
556
|
+
print(" %-18s %s" % (n, v))
|
|
557
|
+
print("depend on httpx: %s" % (", ".join(rep["httpx_dependents"]) or "-"))
|
|
558
|
+
print("depend on httpx2: %s" % (", ".join(rep["httpx2_dependents"]) or "-"))
|
|
559
|
+
for h, r in rep["tls"].items():
|
|
560
|
+
print("tls %s: %s" % (h, ", ".join("%s=%s" % kv for kv in r.items())))
|
|
561
|
+
for w in rep["warnings"]:
|
|
562
|
+
print("WARNING: " + w)
|
|
563
|
+
print(footer())
|
|
564
|
+
return 1 if rep["warnings"] else 0
|
|
565
|
+
|
|
566
|
+
|
|
567
|
+
def build_parser():
|
|
568
|
+
p = argparse.ArgumentParser(prog="httpx2-audit", description=__doc__)
|
|
569
|
+
p.add_argument("--version", action="version", version="httpx2-audit " + __version__)
|
|
570
|
+
sub = p.add_subparsers(dest="cmd", required=True)
|
|
571
|
+
c = sub.add_parser("check", help="scan Python files for httpx -> httpx2 problems")
|
|
572
|
+
c.add_argument("paths", nargs="*", default=["."])
|
|
573
|
+
c.add_argument("--format", choices=["text", "json", "sarif"], default="text")
|
|
574
|
+
c.add_argument("--min-severity", choices=list(SEV_ORDER), default="info")
|
|
575
|
+
c.add_argument("--fail-on", choices=["info", "warning", "error", "never"], default="warning")
|
|
576
|
+
c.add_argument("-q", "--quiet", action="store_true", help="omit the footer")
|
|
577
|
+
c.set_defaults(func=cmd_check)
|
|
578
|
+
f = sub.add_parser("fix", help="rewrite imports and dependencies (dry run unless --write)")
|
|
579
|
+
f.add_argument("paths", nargs="*", default=["."])
|
|
580
|
+
f.add_argument("--write", action="store_true")
|
|
581
|
+
f.add_argument("--dual", action="store_true", help="try httpx2, fall back to httpx (for libraries)")
|
|
582
|
+
f.set_defaults(func=cmd_fix)
|
|
583
|
+
e = sub.add_parser("env", help="report installed httpx/httpx2 stack and optionally probe TLS")
|
|
584
|
+
e.add_argument("--probe", action="append", default=[], metavar="HOST[:PORT]")
|
|
585
|
+
e.add_argument("--format", choices=["text", "json"], default="text")
|
|
586
|
+
e.set_defaults(func=cmd_env)
|
|
587
|
+
return p
|
|
588
|
+
|
|
589
|
+
|
|
590
|
+
def main(argv=None):
|
|
591
|
+
a = build_parser().parse_args(argv)
|
|
592
|
+
if a.cmd in ("check", "fix") and not a.paths:
|
|
593
|
+
a.paths = ["."]
|
|
594
|
+
return a.func(a)
|
|
595
|
+
|
|
596
|
+
|
|
597
|
+
if __name__ == "__main__":
|
|
598
|
+
sys.exit(main())
|