syshttp 0.1.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- syshttp/AGENTS.md +817 -0
- syshttp/README.md +177 -0
- syshttp/__init__.py +71 -0
- syshttp/_fileio.py +78 -0
- syshttp/_parse.py +30 -0
- syshttp/auth.py +17 -0
- syshttp/autopatch.py +382 -0
- syshttp/client/__init__.py +70 -0
- syshttp/client/_body.py +156 -0
- syshttp/client/_client.py +413 -0
- syshttp/client/_context.py +107 -0
- syshttp/client/_deadline.py +94 -0
- syshttp/client/_default.py +72 -0
- syshttp/client/_fetch.py +98 -0
- syshttp/client/_follow.py +243 -0
- syshttp/client/_headers.py +238 -0
- syshttp/client/_pool.py +170 -0
- syshttp/client/_prepare.py +77 -0
- syshttp/client/_response.py +301 -0
- syshttp/client/_route.py +75 -0
- syshttp/client/_settings.py +12 -0
- syshttp/client/_state.py +322 -0
- syshttp/client/_status.py +59 -0
- syshttp/client/_tlstunnel.py +114 -0
- syshttp/client/_transport.py +550 -0
- syshttp/cookies.py +391 -0
- syshttp/env.py +144 -0
- syshttp/exceptions.py +29 -0
- syshttp/hooks.py +314 -0
- syshttp/jar.py +133 -0
- syshttp/proxy.py +341 -0
- syshttp/py.typed +0 -0
- syshttp/retry.py +34 -0
- syshttp/tls.py +60 -0
- syshttp-0.1.0.dist-info/METADATA +219 -0
- syshttp-0.1.0.dist-info/RECORD +39 -0
- syshttp-0.1.0.dist-info/WHEEL +4 -0
- syshttp-0.1.0.dist-info/entry_points.txt +6 -0
- syshttp-0.1.0.dist-info/licenses/LICENSE +21 -0
syshttp/AGENTS.md
ADDED
|
@@ -0,0 +1,817 @@
|
|
|
1
|
+
# `syshttp` — public API header
|
|
2
|
+
|
|
3
|
+
Header-file-style reference for the `syshttp` package: every public export with
|
|
4
|
+
its signature, arguments, contract and gotchas, so the package can be used
|
|
5
|
+
without reading its source. It ships inside the package and is
|
|
6
|
+
self-contained. Development documentation lives with the source at
|
|
7
|
+
<https://github.com/jose-pr/syshttp>.
|
|
8
|
+
|
|
9
|
+
Import names from their modules: `syshttp.env`, `syshttp.proxy`,
|
|
10
|
+
`syshttp.client`, `syshttp.jar`, `syshttp.cookies`, `syshttp.hooks`,
|
|
11
|
+
`syshttp.tls`, `syshttp.retry`, `syshttp.auth`, `syshttp.exceptions`,
|
|
12
|
+
`syshttp.autopatch`. Each declares `__all__`: a name outside it is not API,
|
|
13
|
+
and a module whose name starts with `_` is private. The root re-exports
|
|
14
|
+
`HTTPEnv`, the exceptions, `NoReachableProxyError`, `proxy_map`,
|
|
15
|
+
`configured_proxy`, `proxy_type` and `fetch` (the function of
|
|
16
|
+
`syshttp.client`, imported when it is asked for); there is no `syshttp.fetch`
|
|
17
|
+
module.
|
|
18
|
+
|
|
19
|
+
`pip install syshttp` brings proxylib 2.0.2+ (proxy routing and the system's proxy configuration),
|
|
20
|
+
sslcontext (the system's TLS client; truststore on Python 3.10+) and
|
|
21
|
+
importpatch (import-time patching of other HTTP clients, off unless
|
|
22
|
+
`SYSHTTP_IMPORTPATCH` asks for it). The extra `pac`
|
|
23
|
+
adds a JavaScript engine for PAC scripts proxylib evaluates itself (Linux
|
|
24
|
+
without libproxy). The client needs nothing else: `requests` is no dependency
|
|
25
|
+
of any feature; only `syshttp.autopatch` touches it, in a program that imports
|
|
26
|
+
it, when `SYSHTTP_IMPORTPATCH` asks for it.
|
|
27
|
+
|
|
28
|
+
`syshttp.__version__` is the installed distribution's version.
|
|
29
|
+
|
|
30
|
+
## Settings (`syshttp.env`)
|
|
31
|
+
|
|
32
|
+
Every setting is a variable with a full name: syshttp's own carry
|
|
33
|
+
`SYSHTTP_` (`SYSHTTP_PROXY`, `SYSHTTP_HOME`, `SYSHTTP_HOOKS_<KEY>`),
|
|
34
|
+
conventional ones keep theirs (`HTTPS_PROXY`, `NO_PROXY`, `CA_BUNDLE`). Each
|
|
35
|
+
is read as `<PREFIX><NAME>` for every prefix in turn, the first one set (and
|
|
36
|
+
not blank) winning, each name on its own. The prefixes are the `prefixes`
|
|
37
|
+
argument of an `HTTPEnv`, else `SYSHTTP_PREFIX` (a `:`-separated list), else
|
|
38
|
+
one empty prefix: the names as written. A prefix is used literally, so it
|
|
39
|
+
carries its own separator (`AGENTS_`). Order: an explicit argument, then the
|
|
40
|
+
variable, then the built-in default. Nothing is read at import; each call
|
|
41
|
+
reads the environment as it is then. Below, every name is given as written,
|
|
42
|
+
under the empty prefix.
|
|
43
|
+
|
|
44
|
+
```python
|
|
45
|
+
class HTTPEnv:
|
|
46
|
+
def __init__(self, prefixes: str | Iterable[str] | None = None,
|
|
47
|
+
environ: Mapping[str, str] | None = None) -> None
|
|
48
|
+
environ: Mapping[str, str] # property: the mapping read (os.environ when None)
|
|
49
|
+
prefixes: tuple[str, ...] # property: the prefixes in force, in lookup order
|
|
50
|
+
def name(self, name: str) -> str
|
|
51
|
+
def find(self, *names: str) -> tuple[str, str] | None
|
|
52
|
+
def get(self, *names: str) -> str | None
|
|
53
|
+
def family(self, stem: str) -> Iterator[tuple[str, str, str]]
|
|
54
|
+
PREFIX_VAR = "SYSHTTP_PREFIX"
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
- `SYSHTTP_PREFIX=MYTOOL_` reads `MYTOOL_SYSHTTP_PROXY`, `MYTOOL_HTTPS_PROXY`,
|
|
58
|
+
`MYTOOL_CA_BUNDLE` and so on, and nothing unprefixed: a tool opts out of
|
|
59
|
+
the defaults. `MYTOOL_:` reads `MYTOOL_<NAME>` first, then `<NAME>`: a tool
|
|
60
|
+
layers its own settings over them. `SYSHTTP_PREFIX` itself has no prefix.
|
|
61
|
+
- `prefixes=None` reads `SYSHTTP_PREFIX` at every lookup; blank or unset is
|
|
62
|
+
`("",)`. A string is split at `:`; an iterable is taken as the list; an
|
|
63
|
+
empty iterable reads nothing. A prefix is letters, digits and underscores
|
|
64
|
+
(possibly none), upper-cased; one listed twice counts once. A malformed
|
|
65
|
+
argument raises `ValueError` in the constructor; a malformed
|
|
66
|
+
`SYSHTTP_PREFIX` raises `ConfigurationError` naming it at the first lookup.
|
|
67
|
+
- A CGI program receives every request header as an `HTTP_*` variable, so a
|
|
68
|
+
client could set `HTTP_PROXY`, `HTTP_PROXY_AUTH` and the like: under CGI,
|
|
69
|
+
set a prefix list without the empty entry.
|
|
70
|
+
- `environ` isolates an instance: two `HTTPEnv`s with different prefixes or
|
|
71
|
+
mappings never see each other's settings.
|
|
72
|
+
- `find(*names)` returns `(variable, value)` for the first of `names` set
|
|
73
|
+
under the first prefix that has one (every name tried under a prefix before
|
|
74
|
+
the next prefix), the value stripped; `None` when none is. `get` returns
|
|
75
|
+
the value only. `name(name)` is `name` under the first prefix: what a
|
|
76
|
+
message names when nothing is set, and what a child process is given.
|
|
77
|
+
- `family(stem)` yields `(rest, value, variable)` for every variable named
|
|
78
|
+
`<PREFIX><stem><rest>` with a non-empty `rest` and a value that is not
|
|
79
|
+
blank, the first prefix winning for a `rest` (compared upper-cased).
|
|
80
|
+
|
|
81
|
+
Every function below that reads a setting takes `env: HTTPEnv | None = None`
|
|
82
|
+
(keyword-only) and uses `HTTPEnv()` when it is `None`.
|
|
83
|
+
|
|
84
|
+
## Proxy (`syshttp.proxy`)
|
|
85
|
+
|
|
86
|
+
Routing is proxylib's (2.x): syshttp turns its settings into one proxylib
|
|
87
|
+
proxy map, and proxylib's client integrations send through it. Read an
|
|
88
|
+
answer with `proxylib.lookup(m, url)`; every proxylib integration takes the
|
|
89
|
+
map (`ProxyMapAdapter`, `ProxyMapHandler`, `ProxyTransport`, `ProxySession`).
|
|
90
|
+
|
|
91
|
+
```python
|
|
92
|
+
def proxy_map(proxy: str | None = None, noproxy: str | None = None, proxy_user: str | None = None,
|
|
93
|
+
*, env=None) -> proxylib.ProxyMap
|
|
94
|
+
def configured_proxy(*, env=None) -> proxylib.Proxy | None
|
|
95
|
+
def system_map() -> proxylib.ProxyMap | None
|
|
96
|
+
def system_proxy_enabled(*, env=None) -> bool
|
|
97
|
+
def auth_header(*, env=None) -> str
|
|
98
|
+
def proxy_type(*, env=None) -> tuple[str, str | None]
|
|
99
|
+
NoReachableProxyError = proxylib.NoReachableProxyError
|
|
100
|
+
SYSTEM_PROXY_MAX_AGE = 60
|
|
101
|
+
PROXY = "PROXY"; PROXY_AUTH = "PROXY_AUTH"; PROXY_AUTH_HEADER = "PROXY_AUTH_HEADER"
|
|
102
|
+
PROXY_TYPE = "PROXY_TYPE"; SYSTEM_PROXY = "SYSTEM_PROXY" # setting names, as `SYSHTTP_<NAME>`
|
|
103
|
+
DEFAULT_AUTH_HEADER = "Proxy-Authorization"; MACHINE_AUTH_VAR = "HTTP_PROXY_AUTH"
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
- `proxy_map()` reads the settings at every lookup, so one map serves a
|
|
107
|
+
long-lived client, and always answers (never `KeyError`). Its answer is a
|
|
108
|
+
tuple of entries to try in order, `None` for DIRECT:
|
|
109
|
+
1. `NO_PROXY` / `no_proxy` hosts: DIRECT (proxylib's `BypassRules`:
|
|
110
|
+
suffixes, CIDR, ports, `*`; loopback always).
|
|
111
|
+
2. `SYSHTTP_PROXY` (`configured_proxy()`), alone.
|
|
112
|
+
3. Else the machine's proxy: when any of `HTTP_PROXY`, `HTTPS_PROXY`,
|
|
113
|
+
`FTP_PROXY`, `WS_PROXY`, `WSS_PROXY`, `ALL_PROXY` is set (either case),
|
|
114
|
+
those variables alone decide, per scheme (proxylib's `EnvProxyConfig`:
|
|
115
|
+
the lowercase spelling first under each prefix, a PAC result such as
|
|
116
|
+
`PROXY a:1; DIRECT` read as a list); else the system's
|
|
117
|
+
configuration for the URL (`system_map()`), unless `SYSHTTP_SYSTEM_PROXY`
|
|
118
|
+
is off. `HTTP_PROXY_AUTH`, when set, is every such proxy's
|
|
119
|
+
`Proxy-Authorization` (SOCKS entries excepted).
|
|
120
|
+
4. Else DIRECT.
|
|
121
|
+
- `proxy` (curl's `-x`) replaces 2 and 3 with that proxy, a plain one whose
|
|
122
|
+
credential is only its userinfo; `""` is DIRECT. `noproxy` (`--noproxy`)
|
|
123
|
+
replaces `NO_PROXY`, the variables' own included; listing `<-loopback>`
|
|
124
|
+
turns proxylib's implicit loopback bypass off, as curl has none. `proxy_user` (`-U user:password`) becomes the
|
|
125
|
+
credential of whichever proxy answers.
|
|
126
|
+
- A malformed `SYSHTTP_PROXY`, `SYSHTTP_PROXY_TYPE` or `SYSHTTP_PROXY_AUTH_HEADER`
|
|
127
|
+
raises `ConfigurationError` (proxylib's `ProxyParseError` for a URL) at
|
|
128
|
+
lookup, only when `SYSHTTP_PROXY` is set (the header name only when
|
|
129
|
+
`SYSHTTP_PROXY_AUTH` is too); a bad `SYSHTTP_SYSTEM_PROXY` only when the system
|
|
130
|
+
would be asked.
|
|
131
|
+
- `configured_proxy()` is `SYSHTTP_PROXY` as a proxylib `Proxy` (a bare
|
|
132
|
+
`host:port` is `http`). Its credential is `SYSHTTP_PROXY_AUTH`, a header value
|
|
133
|
+
sent verbatim in `auth_header()` (`SYSHTTP_PROXY_AUTH_HEADER`, default
|
|
134
|
+
`Proxy-Authorization`; a value with `:` or whitespace raises `ValueError`)
|
|
135
|
+
in place of the userinfo, else the userinfo as Basic. A credential (this
|
|
136
|
+
or `HTTP_PROXY_AUTH`) containing a CR, LF or NUL raises `ValueError` naming
|
|
137
|
+
the variable, never the value. It goes to that proxy only: a `CONNECT`, or a plain-http request to it, never through the tunnel.
|
|
138
|
+
`SYSHTTP_PROXY_TYPE=prefix[:/endpoint]` (`proxy_type()`) makes it a
|
|
139
|
+
`prefix+http(s)` gateway: the request for `T` goes to
|
|
140
|
+
`<proxy><endpoint>T`, the credential a header of that request. `syshttp.client`
|
|
141
|
+
does not send an `https:` origin through a plain `prefix+http` gateway.
|
|
142
|
+
- `system_map()` is `proxylib.auto_proxy(environ={})` (the OS resolver --
|
|
143
|
+
WinHTTP, CFNetwork, libproxy -- then the OS settings, PAC and WPAD
|
|
144
|
+
included), built once per `SYSTEM_PROXY_MAX_AGE` seconds, each URL's
|
|
145
|
+
answer cached as long. A configuration that cannot be read is `None`, and
|
|
146
|
+
a lookup that fails is no opinion: DIRECT, logged at DEBUG, never a failed
|
|
147
|
+
request.
|
|
148
|
+
- Proxies never print their secrets: a `Proxy`'s `repr()` hides the password
|
|
149
|
+
and headers, `.url` has no credentials.
|
|
150
|
+
- How an answer is used is the client integration's: requests, httpx and
|
|
151
|
+
aiohttp try the entries in order and move on when one does not connect
|
|
152
|
+
(the last error is raised; a DIRECT entry reached is direct); a client
|
|
153
|
+
that sends once, `syshttp.client` among them, takes the first reachable
|
|
154
|
+
entry through `proxylib.Probed(proxy_map())`, which raises
|
|
155
|
+
`NoReachableProxyError` (a
|
|
156
|
+
`LookupError`) when none answers and the list holds no DIRECT.
|
|
157
|
+
|
|
158
|
+
## Client (`syshttp.client`)
|
|
159
|
+
|
|
160
|
+
A synchronous HTTP client on the standard library (`http.client`, `ssl`): no
|
|
161
|
+
third-party package, `requests` included. It routes by the proxy map
|
|
162
|
+
(`syshttp.proxy`), verifies with the system's TLS, and keeps the cookies and
|
|
163
|
+
tokens of each origin. `import syshttp.client` loads `http.client`, `ssl`,
|
|
164
|
+
proxylib and sslcontext; `import syshttp` does not.
|
|
165
|
+
|
|
166
|
+
```python
|
|
167
|
+
class HTTPClient:
|
|
168
|
+
headers: dict
|
|
169
|
+
timeout: float | tuple | None
|
|
170
|
+
deadline: float | None
|
|
171
|
+
user_agent: str
|
|
172
|
+
def __init__(self, transport=None, *, headers=None, timeout=(10, 60), deadline=None, user_agent=None,
|
|
173
|
+
env: HTTPEnv | None = None, proxymap=None, ssl_context=None, proxy_ssl_context=None,
|
|
174
|
+
verify: bool | str | PathLike = True,
|
|
175
|
+
cookies: bool | str | CookieJar | None = False, tokens: bool | str | TokenJar | None = False,
|
|
176
|
+
auth_provider: AuthProvider | None = None, retries: int = 3, backoff: float = 1,
|
|
177
|
+
status_forcelist: Iterable[int] = TRANSIENT_STATUSES,
|
|
178
|
+
cookie_hosts: Iterable[Iterable[str]] | None = None)
|
|
179
|
+
transport # the transport in use
|
|
180
|
+
def request(self, method, url, *, data=None, headers=None, stream=False, allow_redirects=None,
|
|
181
|
+
timeout=..., deadline=..., auth=None) -> HTTPResponse
|
|
182
|
+
def redirect(self, method, url, resp, *, headers=None, stream=False, timeout=..., deadline=...,
|
|
183
|
+
auth=None) -> HTTPResponse
|
|
184
|
+
def close(self) -> None
|
|
185
|
+
|
|
186
|
+
def fetch(url, method="GET", *, headers=None, data=None, json=None, timeout=DEFAULT_TIMEOUT,
|
|
187
|
+
client=None, transport=None, reauth=None, **client_options) -> HTTPResponse
|
|
188
|
+
|
|
189
|
+
DEFAULT_TIMEOUT = (10, 60)
|
|
190
|
+
MAX_REPLY_BYTES = 8388608
|
|
191
|
+
CA_BUNDLE = "CA_BUNDLE"; COOKIE_HOSTS = "SYSHTTP_COOKIE_HOSTS"
|
|
192
|
+
DEFAULT_TRANSPORT = "SYSHTTP_DEFAULT_TRANSPORT" # setting names
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
- **`request(method, url, *, data=None, headers=None, stream=False,
|
|
196
|
+
allow_redirects=None, timeout, deadline, auth=None)`** returns the final
|
|
197
|
+
`HTTPResponse`. `data` is `bytes`, `str` (UTF-8), a binary file or an
|
|
198
|
+
iterable of `bytes`; one that is not `bytes` cannot be sent twice, so it is
|
|
199
|
+
not sent again after a redirect, a dropped connection or a retry. `auth` is a
|
|
200
|
+
`(user, password)` pair sent as `Authorization: Basic`, or a `str` sent as
|
|
201
|
+
the `Authorization` value (`"Bearer <token>"`), to the origin of `url`
|
|
202
|
+
unless a header names an `Authorization`; it beats a token of the client. A
|
|
203
|
+
`url` that carries credentials, a URL or header value with a control
|
|
204
|
+
character (CR and LF included) and another scheme than `http:` or `https:`
|
|
205
|
+
are `ValueError`, before anything is sent. A status of 400 or more is
|
|
206
|
+
**not** an error: it is the reply (`raise_for_status()`). With
|
|
207
|
+
`stream=False` the body has been read when `request()` returns, and one
|
|
208
|
+
that decodes to more than `MAX_REPLY_BYTES` is `OSError(EFBIG)`; with
|
|
209
|
+
`stream=True` the caller reads the body and closes the response.
|
|
210
|
+
- **Headers and time.** `headers` of the client and of the call are merged by
|
|
211
|
+
name (case-blind, a value of `None` removes the name) over
|
|
212
|
+
`User-Agent: <user_agent>` (default `syshttp/<version>`) and `Accept: */*`.
|
|
213
|
+
Every request gets `timeout=(10, 60)` seconds (`connect`, `read`) unless
|
|
214
|
+
the client or the call gives one; `timeout=None` waits without limit.
|
|
215
|
+
`timeout` bounds each wait; `deadline` (seconds, `None` for none) bounds the
|
|
216
|
+
whole request, redirects, retries and the reading of the body included, and
|
|
217
|
+
`TimeoutError` is raised once it has passed. A call's value replaces the
|
|
218
|
+
client's; a `deadline` that is not a number or `None` is `TypeError`, one
|
|
219
|
+
that is not positive `ValueError`. `repr()` of a client shows no header.
|
|
220
|
+
- **Redirects.** `GET`, `HEAD` and `OPTIONS` follow up to 30 redirects (a
|
|
221
|
+
`303` becomes a `GET`); `allow_redirects=False` returns the first answer. A
|
|
222
|
+
redirect whose `Location` is missing or cannot be requested is
|
|
223
|
+
`OSError(EIO)`. A redirect that changes the scheme, host or port drops
|
|
224
|
+
`Authorization`, `Cookie` and `Proxy-Authorization` for that hop and every
|
|
225
|
+
later one (the jar's cookies are chosen again for each hop), and nothing
|
|
226
|
+
else: every other header goes to the new host as well, so keep a secret in
|
|
227
|
+
one of those three. Every other method is sent without
|
|
228
|
+
following: a 307 or 308 (for `PROPFIND` also a 301, 302 or 303) to the same
|
|
229
|
+
scheme, host and port is sent once more with the same method, headers and
|
|
230
|
+
body, and any other 3xx, a redirect to another origin or a second redirect
|
|
231
|
+
is `OSError(EIO)` naming the status and the `Location` (userinfo, query and
|
|
232
|
+
fragment removed). `redirect(method, url, resp, ...)` sends `method` to where
|
|
233
|
+
`resp`, the answer to a request for `url` made with `allow_redirects=False`,
|
|
234
|
+
redirects to, with the same rules; cookies apply to every hop, the token,
|
|
235
|
+
hooks and retries belong to `request()`.
|
|
236
|
+
- **Routing.** Each request asks the proxy map (`proxymap`, any proxylib
|
|
237
|
+
`ProxyMap` or value `ProxyMap()` builds one from; default
|
|
238
|
+
`proxy.proxy_map(env=env)`) for the entries it may use and takes the first
|
|
239
|
+
that accepts a TCP connection (`proxylib.Probed`; with no entry reachable and
|
|
240
|
+
no `DIRECT`, `proxylib.NoReachableProxyError`). A direct entry connects to
|
|
241
|
+
the origin; an `http:` or `https:` proxy gets an absolute-form request for
|
|
242
|
+
an `http:` origin and a `CONNECT` for an `https:` one, an `https:` proxy
|
|
243
|
+
over TLS and the origin's TLS inside it; a `prefix+http(s)` gateway gets
|
|
244
|
+
`<gateway><path><URL>` with its credential as a header of that request, and
|
|
245
|
+
the response keeps the URL asked for; a plain `prefix+http` gateway is not
|
|
246
|
+
offered an `https:` origin, whose tokens and Secure cookies would cross it in
|
|
247
|
+
clear text: the entry is skipped like a SOCKS one, and with nothing else left
|
|
248
|
+
the request is `proxylib.UnsupportedProxyError`. A proxy's headers (its credential) go
|
|
249
|
+
to the proxy only: in the `CONNECT`, or on the plain-http request to it,
|
|
250
|
+
never through a tunnel. SOCKS entries are skipped; an answer with nothing
|
|
251
|
+
else is `proxylib.UnsupportedProxyError`. A map with no opinion for a URL
|
|
252
|
+
(`{}` for every URL) means a direct connection and the settings are not
|
|
253
|
+
read. Every hop of a redirect is routed again; a connection to a proxy this
|
|
254
|
+
thread already holds is reused without probing it. A `proxymap` of your own
|
|
255
|
+
adds no `SYSHTTP_PROXY_AUTH` credential. A malformed proxy setting
|
|
256
|
+
(`ConfigurationError`) is refused when the client is made.
|
|
257
|
+
- **TLS.** `https:` origins are verified against `ssl_context` when given,
|
|
258
|
+
used as given and never changed (its `verify_mode`, trust store, client
|
|
259
|
+
identity, ciphers and protocol versions decide; `verify` must stay `True`
|
|
260
|
+
beside it, else `ValueError`; a non-`SSLContext` is `TypeError`); else
|
|
261
|
+
against the CA bundle that `verify=<path>` or the `CA_BUNDLE` setting (read
|
|
262
|
+
at each new connection, through the prefixes) names, a context trusting only
|
|
263
|
+
that file (a file that cannot be read is `ConfigurationError` naming where
|
|
264
|
+
the path came from); else against the system's context
|
|
265
|
+
(`tls.os_ssl_context()`: the OS verifies, and a server that asks for a client
|
|
266
|
+
certificate gets one from the OS store). `verify=False` turns verification
|
|
267
|
+
off (a warning is logged under the `syshttp.client` logger when the client's
|
|
268
|
+
transport is made). requests' own `REQUESTS_CA_BUNDLE` / `CURL_CA_BUNDLE` are
|
|
269
|
+
not read. An `https:` proxy or gateway is spoken to with
|
|
270
|
+
`proxy_ssl_context`, else the system's context; the gateway never gets the
|
|
271
|
+
origin's `ssl_context` or client certificate. There is no per-request
|
|
272
|
+
`verify` or client certificate: build another client or load the identity
|
|
273
|
+
into the context.
|
|
274
|
+
- **Cookies.** A client keeps the cookies its responses set, per origin host
|
|
275
|
+
and never per proxy or gateway, on every hop: parsed as RFC 6265 reads them
|
|
276
|
+
(`cookies.set_cookie_specs`), sent as `cookies.cookie_applies` decides
|
|
277
|
+
(longer paths first), expired and deleted ones dropped. A `Cookie` header the
|
|
278
|
+
call or client names replaces the jar's for that request. They live in
|
|
279
|
+
memory for the client's life; `cookies=True` also keeps them in a file jar
|
|
280
|
+
under `jar.store_root(env=env)` by host, a string names one key for every
|
|
281
|
+
host, a jar object is used as given. A host's rows are loaded from the jar
|
|
282
|
+
the first time the host is asked for, and written back only when they differ
|
|
283
|
+
from what was loaded or written last; a jar that fails to read or write is
|
|
284
|
+
ignored. A by-host file gives its host (or group) only the rows whose domain
|
|
285
|
+
covers it; a named key holds every row. A `Set-Cookie` with a control
|
|
286
|
+
character (the tab apart) is ignored whole. A Secure cookie from plain http
|
|
287
|
+
(the loopback hosts apart, as curl has it) is ignored, and plain http cannot
|
|
288
|
+
set, replace or delete a stored Secure cookie of the same name whose domain
|
|
289
|
+
(either containing the other) and path match (RFC 6265bis 5.7). `cookie_hosts` (default `SYSHTTP_COOKIE_HOSTS`, read when the client
|
|
290
|
+
is made: groups split by `;`, hosts by `,`:
|
|
291
|
+
`app.example.com,app.example.net;wiki,wiki.corp.example`; `[]` for none)
|
|
292
|
+
declares groups of hosts that are one for cookies (`cookies.host_aliases`):
|
|
293
|
+
- a host-only cookie a member sets, and one whose `Domain` names a member,
|
|
294
|
+
belongs to the group's first host (stored there, kept in its jar file);
|
|
295
|
+
- a request to any member carries what a request to any member would,
|
|
296
|
+
`Secure` and the path still applying per request;
|
|
297
|
+
- a member deletes such a cookie as the first host would;
|
|
298
|
+
- a `Domain` outside the group is judged on the host that sent it, so a
|
|
299
|
+
group never lets a host set a cookie for a domain that is not its own.
|
|
300
|
+
|
|
301
|
+
Without groups the RFC 6265 rule stands: no host sets or deletes another
|
|
302
|
+
host's cookie. Hosts are names or IP addresses, without a port; a host in
|
|
303
|
+
two groups raises `ValueError`, a malformed variable `ConfigurationError`
|
|
304
|
+
naming it, a group given as one string `TypeError`.
|
|
305
|
+
- **Tokens.** `tokens=True` reads `Authorization` values from file jars under
|
|
306
|
+
`jar.store_root(env=env)` keyed by the origin's host (read at each request),
|
|
307
|
+
a string names one key for every host, a jar object is used as given. A
|
|
308
|
+
token is a per-request `Authorization`: `Bearer <token>`, or verbatim when it
|
|
309
|
+
already names a scheme, never a client header, so it follows no redirect to
|
|
310
|
+
another origin. An explicit `Authorization` header or `auth=` wins. A token
|
|
311
|
+
containing a CR, LF or NUL is `ConfigurationError` naming the jar key, never
|
|
312
|
+
the value.
|
|
313
|
+
- **Auth provider and hooks.** In `request()` (not `redirect()`),
|
|
314
|
+
`auth_provider.ensure(client, url)` runs first and its errors are swallowed;
|
|
315
|
+
then the `_PY` hooks (see "URL hooks") run on the caller's URL with the
|
|
316
|
+
keyword arguments the call gave (`data`, `headers`, `stream`,
|
|
317
|
+
`allow_redirects`, `timeout`, `deadline`, `auth`) as a mutable `kwargs`; one
|
|
318
|
+
that sets another name is `TypeError`. A hook may return a response, which
|
|
319
|
+
is used instead of sending. A request a provider or a hook makes through the
|
|
320
|
+
client runs neither again; the guard is per thread, so another thread's
|
|
321
|
+
requests still run them.
|
|
322
|
+
- **Retries.** A status in `status_forcelist` (default `TRANSIENT_STATUSES`),
|
|
323
|
+
or a failed connection (`ConnectionError`, not a timeout, not
|
|
324
|
+
`NoReachableProxyError`), is sent again up to `retries` times (default 3) for
|
|
325
|
+
`GET`, `HEAD`, `OPTIONS`, `PUT`, `DELETE`, `TRACE` and `PROPFIND` with a body
|
|
326
|
+
that can be sent twice. None waits before the first retry, then `backoff`
|
|
327
|
+
seconds doubled each time, never less than the reply's `Retry-After`, at most
|
|
328
|
+
120 seconds; the `deadline` bounds the sum. A status that outlasts the
|
|
329
|
+
retries is returned, not raised; a connection that never succeeds raises
|
|
330
|
+
its last error. `retries=0` sends once.
|
|
331
|
+
- **Close.** `close()` (or leaving a `with` block) closes a transport the
|
|
332
|
+
client made, never one it was given; a closed client makes a new one when
|
|
333
|
+
used again.
|
|
334
|
+
|
|
335
|
+
```python
|
|
336
|
+
class HTTPResponse:
|
|
337
|
+
status: int
|
|
338
|
+
reason: str
|
|
339
|
+
headers: Headers
|
|
340
|
+
url: str
|
|
341
|
+
def __init__(self, status, reason, headers, url, body=b"", *, decode=True): ...
|
|
342
|
+
def read(self, size=-1): ...
|
|
343
|
+
def iter_bytes(self, chunk_size=65536): ...
|
|
344
|
+
def buffered(self): ...
|
|
345
|
+
content: bytes # property
|
|
346
|
+
def text(self, encoding=None, errors="replace") -> str
|
|
347
|
+
def json(self) -> Any
|
|
348
|
+
def close(self): ...
|
|
349
|
+
|
|
350
|
+
class Headers: # read-only Mapping, names case-blind
|
|
351
|
+
def get_all(self, name): ...
|
|
352
|
+
|
|
353
|
+
class HTTPStatusError(Exception):
|
|
354
|
+
response: HTTPResponse
|
|
355
|
+
|
|
356
|
+
def raise_for_status(resp): ...
|
|
357
|
+
def read_limited(resp, what, limit=None): ...
|
|
358
|
+
def json_body(resp, what, limit=None): ...
|
|
359
|
+
def drop_body(resp, limit=None): ...
|
|
360
|
+
def peer_text(text, limit=120): ...
|
|
361
|
+
def split_userinfo(url): ...
|
|
362
|
+
def is_redirect(resp): ...
|
|
363
|
+
```
|
|
364
|
+
|
|
365
|
+
- **`HTTPResponse`**: `status`, `reason`, `headers` (names compare without
|
|
366
|
+
case, a name sent twice reads as `"a, b"` and `headers.get_all()` lists them;
|
|
367
|
+
`repr()` shows names only), `url` (the URL requested, without fragment, an
|
|
368
|
+
empty path `/`; through a gateway still the URL asked for), `read(size=-1)`,
|
|
369
|
+
`iter_bytes(chunk_size)`, `buffered(limit=None)` (reads the rest of the body
|
|
370
|
+
into memory, closes the connection and returns the response, which stays
|
|
371
|
+
readable; more than `limit` decoded bytes, default `MAX_REPLY_BYTES`, is
|
|
372
|
+
`OSError(EFBIG)`), `close()` and a context manager. A `gzip` or `deflate` body is
|
|
373
|
+
decoded, in steps of at most 256 KiB, unless the request named an
|
|
374
|
+
`Accept-Encoding` of its own (`identity` counts as none); `headers` stay as
|
|
375
|
+
received, so `Content-Length` counts the encoded bytes. A body that ends
|
|
376
|
+
before the length the server stated, a chunked body cut short and a coded
|
|
377
|
+
body that is not valid are `OSError(EIO)`, never a shorter read.
|
|
378
|
+
`content` is the rest of the decoded body, read on first use (at most
|
|
379
|
+
`MAX_REPLY_BYTES`, else `OSError(EFBIG)` naming the URL as scheme, host,
|
|
380
|
+
port and path) and kept, so `content`, `text()`
|
|
381
|
+
and `json()` agree however often they are used; bytes already taken with
|
|
382
|
+
`read()` are not in it. `text()` decodes with `encoding`, else the
|
|
383
|
+
`charset` of `Content-Type`, else UTF-8, replacing what does not decode.
|
|
384
|
+
`json()` parses it (`None` for an empty body); a body that is not JSON is
|
|
385
|
+
`ResponseError` naming the status. A response built from `bytes` by a hook
|
|
386
|
+
or a fake transport has all of this.
|
|
387
|
+
- `raise_for_status(resp)` raises `HTTPStatusError` (holding `response`) for a
|
|
388
|
+
status of 400 or more. `read_limited(resp, what, limit=None)` is the whole
|
|
389
|
+
decoded body, at most `limit` bytes (default `MAX_REPLY_BYTES`, 8 MiB after
|
|
390
|
+
decoding; assign `syshttp.client.MAX_REPLY_BYTES` to allow more, it is read
|
|
391
|
+
each time): more is `OSError(EFBIG)` naming `what`, and the connection is
|
|
392
|
+
closed; `json_body()` is that, parsed (`ValueError` when it is not JSON).
|
|
393
|
+
`drop_body(resp, limit=None)` gives up the body of a reply nobody looks at: it
|
|
394
|
+
is read up to `limit` bytes so the connection can serve the next request,
|
|
395
|
+
and `resp` is closed. `peer_text(text, limit=120)` is text from the other end
|
|
396
|
+
(a reason phrase, a `Location`) as an error message may show it: control and
|
|
397
|
+
bidirectional characters escaped, cut to `limit`. `split_userinfo(url)` is
|
|
398
|
+
`(url_without_userinfo, (user, password) | None)`, the parts the octets their
|
|
399
|
+
percent-escapes stand for; `is_redirect(resp)` says whether a reply is a
|
|
400
|
+
301, 302, 303, 307 or 308 that names a `Location`.
|
|
401
|
+
- **`fetch(url, method="GET", *, headers=None, data=None, json=None,
|
|
402
|
+
timeout=DEFAULT_TIMEOUT, client=None, transport=None, reauth=None,
|
|
403
|
+
**client_options)`** sends one request and returns the `HTTPResponse` with
|
|
404
|
+
its body read whole (`MAX_REPLY_BYTES` bounds it), whatever the status: take
|
|
405
|
+
`.status`, `.headers`, `.content`, `.text()` or `.json()` from it. The return
|
|
406
|
+
type never depends on an argument. `data` is bytes, text, a mapping
|
|
407
|
+
(form-encoded, `Content-Type: application/x-www-form-urlencoded`) or a binary
|
|
408
|
+
file; `json` is a value sent as JSON (`Content-Type: application/json`;
|
|
409
|
+
`None` sends nothing); both is `ValueError`. It sends through `client` when
|
|
410
|
+
given, left open, else through a new `HTTPClient(transport, **client_options)`
|
|
411
|
+
closed after the call: `transport` is any object implementing the transport
|
|
412
|
+
protocol (`None`: the `SYSHTTP_DEFAULT_TRANSPORT` setting's, else syshttp's
|
|
413
|
+
own) and `client_options` are the other keywords of `HTTPClient`. A `client`
|
|
414
|
+
beside `transport` or `client_options` is `ValueError`: a given client has
|
|
415
|
+
them already. `timeout` is for this call, whatever the client's own is.
|
|
416
|
+
`reauth` is called as `reauth(client, url)` (an object with
|
|
417
|
+
`ensure(client, url)`, an `AuthProvider`, through that) once after a 401 or
|
|
418
|
+
403, and the request is sent one more time; its errors are the caller's, and
|
|
419
|
+
the body must be one that can be sent twice. `syshttp.fetch` is the same
|
|
420
|
+
function: the root imports the client only when it is asked for.
|
|
421
|
+
|
|
422
|
+
### The transport protocol and the default transport (`syshttp.client`)
|
|
423
|
+
|
|
424
|
+
```python
|
|
425
|
+
class Transport(Protocol):
|
|
426
|
+
def send(self, method, url, *, headers, body=None, timeout, stream=False) -> HTTPResponse: ...
|
|
427
|
+
def close(self) -> None: ...
|
|
428
|
+
|
|
429
|
+
class PreparedRequest(NamedTuple):
|
|
430
|
+
method: str; url: str; headers: dict[str, str]; body: Any
|
|
431
|
+
timeout: tuple[float | None, float | None]; decode: bool
|
|
432
|
+
|
|
433
|
+
def prepare_request(method, url, *, headers, body=None, timeout) -> PreparedRequest
|
|
434
|
+
|
|
435
|
+
class HTTPTransport:
|
|
436
|
+
def __init__(self, *, proxymap=None, env=None, ssl_context=None, proxy_ssl_context=None, verify=True)
|
|
437
|
+
def send(self, method, url, *, headers, body=None, timeout, stream=False) -> HTTPResponse
|
|
438
|
+
def close(self) -> None
|
|
439
|
+
```
|
|
440
|
+
|
|
441
|
+
`HTTPClient(transport=...)` takes any object with `send()` (a transport that
|
|
442
|
+
calls `requests`, `httpx`, a fake in a test); the client owns redirects,
|
|
443
|
+
credentials, cookies, tokens, hooks, retries and the deadline above it, so a
|
|
444
|
+
transport does none of them. Without one the client builds the transport the
|
|
445
|
+
`SYSHTTP_DEFAULT_TRANSPORT` setting names (read through the prefixes when the
|
|
446
|
+
client is made): an object reference `module:attr` or `<file>.py:attr` (a
|
|
447
|
+
dotted `attr` is allowed), resolved as a `_PY` hook is (`hooks.load_object`).
|
|
448
|
+
The object is the transport when it has `send()`; a class or factory (anything
|
|
449
|
+
else callable) is called with no arguments to make one, which the client then
|
|
450
|
+
closes. A value that cannot be imported, resolved or called, or whose result
|
|
451
|
+
has no `send()`, is `ConfigurationError` naming the variable, never a silent
|
|
452
|
+
fallback; `proxymap`, `ssl_context`, `proxy_ssl_context` and `verify` beside a
|
|
453
|
+
setting, or beside a `transport` of your own, are refused. Precedence: an
|
|
454
|
+
explicit `transport=`, then the setting, then `HTTPTransport`. The setting
|
|
455
|
+
names code to import: never let an untrusted party set the environment of a
|
|
456
|
+
process using syshttp.
|
|
457
|
+
|
|
458
|
+
- **The contract.** `send()` makes exactly one request: it follows no
|
|
459
|
+
redirect, adds no credentials of the caller's and retries nothing but what is
|
|
460
|
+
said below. `body` is `None`, `bytes`, a binary file or an iterable of
|
|
461
|
+
`bytes` (never `str`: `TypeError`), sent under a `Content-Length` when its
|
|
462
|
+
length is known without reading it (`bytes`, a list or tuple of `bytes`, a
|
|
463
|
+
seekable file, a regular file), else `Transfer-Encoding: chunked`; a
|
|
464
|
+
`Content-Length` in `headers` is trusted (a longer body is cut there, a count
|
|
465
|
+
larger than a known length or not a number is `ValueError`, a body that runs
|
|
466
|
+
out first is `OSError(EIO)`). `timeout` is seconds, a `(connect, read)` pair
|
|
467
|
+
or `None`; the read half bounds each wait on the server. With `stream=False`
|
|
468
|
+
the body has been read when `send()` returns (a body that decodes to more
|
|
469
|
+
than `MAX_REPLY_BYTES` is `OSError(EFBIG)`); with `stream=True` it is read
|
|
470
|
+
from the response, which the caller closes. A transport raises only
|
|
471
|
+
`TimeoutError`, `ConnectionError`, `ssl.SSLError`, `ValueError` (before
|
|
472
|
+
anything is sent: a header or URL with a control character, a URL with
|
|
473
|
+
credentials, another scheme) or else `OSError(EIO)`; no third-party
|
|
474
|
+
exception and no URL or header value in its text. A reply below 200 is never
|
|
475
|
+
the one it returns. It builds the answer as `HTTPResponse(status, reason,
|
|
476
|
+
headers, url, body, decode=...)` with `body` as `bytes` or an object with
|
|
477
|
+
`read(amount)` and `close()`.
|
|
478
|
+
- **`prepare_request(method, url, *, headers, body=None, timeout)`** is the
|
|
479
|
+
arguments of `send()` as the contract reads them, so a transport that starts
|
|
480
|
+
with it refuses what `HTTPTransport` refuses and frames a body the same way:
|
|
481
|
+
`method` upper-cased, the fragment dropped from `url`, `headers` a `dict` of
|
|
482
|
+
text values with the `Content-Length` of a body whose length is known (`0`
|
|
483
|
+
for a `PATCH`, `POST` or `PUT` without one) and `Accept-Encoding: gzip,
|
|
484
|
+
deflate` unless the caller named an encoding, `body` an object that yields
|
|
485
|
+
exactly `Content-Length` bytes or an iterable to send chunked, `timeout` the
|
|
486
|
+
pair, `decode` what to pass as `HTTPResponse(..., decode=decode)`. It raises
|
|
487
|
+
`ValueError` for a method or header name that is not a token, a value or
|
|
488
|
+
URL with a control character, a URL with credentials or another scheme, a
|
|
489
|
+
`Content-Length` that is not a count of bytes or exceeds the body, and a
|
|
490
|
+
timeout that is not a positive finite number, `None` or a pair of them;
|
|
491
|
+
`TypeError` for a `str` body. No message quotes the URL or a header value.
|
|
492
|
+
- **What `HTTPTransport` does that another transport must do itself** (the
|
|
493
|
+
client does none of it): routing by the proxy map (`proxymap`, `env`: probing,
|
|
494
|
+
failover, proxies, `CONNECT`, gateways, a proxy's credentials), TLS
|
|
495
|
+
verification (`ssl_context`, `proxy_ssl_context`, `verify`, `CA_BUNDLE`, the
|
|
496
|
+
system's context), connection reuse (one idle connection per route and
|
|
497
|
+
thread, at most 8, closed when the thread ends or on `close()`; reused only
|
|
498
|
+
if the server has not closed it or sent bytes on it; an idempotent request
|
|
499
|
+
that fails before any byte of a reply on a reused connection is sent once
|
|
500
|
+
more on a new one, any other request reuses a connection only if it carried
|
|
501
|
+
a reply within the last second), skipping interim (1xx) replies (at most 8),
|
|
502
|
+
decoding `gzip` and `deflate`. A transport written on another library
|
|
503
|
+
therefore gets the library's own proxy and TLS behaviour, not syshttp's,
|
|
504
|
+
unless it implements them; it can still be used by `fetch(transport=...)`.
|
|
505
|
+
`HTTPTransport` adds to the contract's exceptions
|
|
506
|
+
`proxylib.NoReachableProxyError`, `proxylib.UnsupportedProxyError` and
|
|
507
|
+
`ConfigurationError`.
|
|
508
|
+
|
|
509
|
+
## Exceptions (`syshttp.exceptions`)
|
|
510
|
+
|
|
511
|
+
```python
|
|
512
|
+
class SysHTTPError(Exception)
|
|
513
|
+
class ConfigurationError(SysHTTPError, ValueError)
|
|
514
|
+
class HookError(SysHTTPError, RuntimeError)
|
|
515
|
+
class ResponseError(SysHTTPError, ValueError)
|
|
516
|
+
```
|
|
517
|
+
|
|
518
|
+
- `SysHTTPError` is the base of what goes wrong at run time. proxylib's own
|
|
519
|
+
errors (`NoReachableProxyError`, `UnsupportedProxyError`, `ProxyParseError`)
|
|
520
|
+
pass through as they are.
|
|
521
|
+
- `ConfigurationError`: a setting or stored credential that cannot be used
|
|
522
|
+
(a malformed `SYSHTTP_PROXY_TYPE`, a regular expression that does not compile,
|
|
523
|
+
a hook that cannot be loaded, a token with a line break, a malformed
|
|
524
|
+
`SYSHTTP_PREFIX`). The message names the variable or key, never a secret.
|
|
525
|
+
- `HookError`: a `_PY` hook failed; the hook's own exception is `__cause__`.
|
|
526
|
+
- `ResponseError`: a body that is not the JSON `HTTPResponse.json()` was asked for.
|
|
527
|
+
- A caller's own mistake is a plain `ValueError` or `TypeError`, not one of
|
|
528
|
+
these: a malformed `prefixes=` argument, `verify=False` beside `ssl_context`,
|
|
529
|
+
a `client` beside `client_options` in `fetch()`, an unknown patch target.
|
|
530
|
+
|
|
531
|
+
## Jars (`syshttp.jar`)
|
|
532
|
+
|
|
533
|
+
```python
|
|
534
|
+
def store_root(*, env=None) -> Path
|
|
535
|
+
def default_root(*, env=None) -> Path
|
|
536
|
+
def safe_name(key: str) -> str
|
|
537
|
+
class FileCookieJar: def __init__(self, root: Path | None = None, *, env=None)
|
|
538
|
+
class FileTokenJar: def __init__(self, root: Path | None = None, *, env=None)
|
|
539
|
+
class MemoryCookieJar(dict)
|
|
540
|
+
class MemoryTokenJar(dict)
|
|
541
|
+
class CookieJar(Protocol); class TokenJar(Protocol) # __getitem__, __setitem__, get
|
|
542
|
+
```
|
|
543
|
+
|
|
544
|
+
- `store_root()` returns `SYSHTTP_HOME` (a leading `~` expanded), else
|
|
545
|
+
`default_root()`: a directory `syshttp` under `%LOCALAPPDATA%` (Windows),
|
|
546
|
+
`~/Library/Application Support` (macOS) or `$XDG_STATE_HOME` /
|
|
547
|
+
`~/.local/state` (elsewhere).
|
|
548
|
+
- Cookies live in `<root>/cookies/<host>.txt` (Netscape), tokens in
|
|
549
|
+
`<root>/tokens/<host>.token`.
|
|
550
|
+
- Both are secrets and are written as such: mode `0600` (directories this
|
|
551
|
+
library creates `0700`; an existing directory keeps its mode), replaced
|
|
552
|
+
whole by a rename so a reader sees the old file or the new one, with LF line
|
|
553
|
+
ends on every platform. `cookies.save_netscape` does the same. The
|
|
554
|
+
constructors do nothing on disk: the directory is created at the first
|
|
555
|
+
write.
|
|
556
|
+
- `get()` returns the default on any read error. Token values are never
|
|
557
|
+
logged.
|
|
558
|
+
- A client saves a host's cookie file only when its cookies differ from
|
|
559
|
+
those last loaded or saved. A token file is read at every request.
|
|
560
|
+
|
|
561
|
+
## Cookies (`syshttp.cookies`)
|
|
562
|
+
|
|
563
|
+
```python
|
|
564
|
+
@dataclass(frozen=True)
|
|
565
|
+
class CookieSpec: domain: str; path: str; secure: bool; expires: int | None; name: str; value: str; http_only: bool = False; subdomains: bool = False
|
|
566
|
+
def load_netscape(path: Path) -> list[CookieSpec]
|
|
567
|
+
def save_netscape(session_cookies: Iterable[CookieLike], path: Path) -> None
|
|
568
|
+
def cookie_applies(cookie: CookieSpec, url: str, now: float | None = None) -> bool
|
|
569
|
+
def set_cookie_specs(values: Iterable[str], default_domain: str, now: float | None = None, *,
|
|
570
|
+
request_path: str = "/",
|
|
571
|
+
aliases: Mapping[str, str] | None = None) -> list[tuple[CookieSpec, bool]]
|
|
572
|
+
def domain_covers(domain: str, host: str) -> bool
|
|
573
|
+
def host_aliases(groups: Iterable[Iterable[str]]) -> dict[str, str]
|
|
574
|
+
HTTP_ONLY_PREFIX = "#HttpOnly_"
|
|
575
|
+
MAX_COOKIE_AGE = 400 * 24 * 3600
|
|
576
|
+
```
|
|
577
|
+
|
|
578
|
+
- `load_netscape` returns `[]` for a missing file, reads `#HttpOnly_` rows,
|
|
579
|
+
keeps a row whose value is empty and treats an unreadable expiry as a
|
|
580
|
+
session cookie. Lines end at `\n` only (one trailing `\r` is taken off), so
|
|
581
|
+
a character `str.splitlines()` ends a line at (`\x0b`, `\x0c`, `\x85`)
|
|
582
|
+
cannot start a row inside a value; a row whose name or value holds a control
|
|
583
|
+
character is skipped. `save_netscape` writes `#HttpOnly_` rows as curl does,
|
|
584
|
+
and writes no row with a control character or a tab in a field.
|
|
585
|
+
- `cookie_applies` decides as curl does: the domain (a subdomain only with a
|
|
586
|
+
leading dot or the include-subdomains flag, and an IP only matches
|
|
587
|
+
itself), the RFC 6265 path match, `Secure` only over https or to loopback,
|
|
588
|
+
not expired (`0` / `None` is a session cookie).
|
|
589
|
+
- `set_cookie_specs` returns `(cookie, gone)` per `Set-Cookie` value, parsed
|
|
590
|
+
as RFC 6265 section 5.2 reads it: the name and value up to the first `;`
|
|
591
|
+
(a value may hold spaces, `=` or JSON), attribute names in any case,
|
|
592
|
+
unknown attributes (`Priority`, `Partitioned`) ignored, `Max-Age` beating
|
|
593
|
+
`Expires` in either order, an HTTP date in any of its three forms read as
|
|
594
|
+
UTC, and an expiry beyond 400 days cut to `MAX_COOKIE_AGE`. A header with
|
|
595
|
+
no `name=value` is skipped. `gone` marks a cookie the server expired (an
|
|
596
|
+
empty value, `Max-Age` of zero or less, a past date).
|
|
597
|
+
- `default_domain` is the host asked. A `Domain` attribute is kept only
|
|
598
|
+
when that host domain-matches it, on a set and on a deletion alike;
|
|
599
|
+
otherwise the cookie is skipped. It then gives `subdomains=True`. With no
|
|
600
|
+
`Domain`, or with one equal to an IP-literal host, the cookie is
|
|
601
|
+
host-only. A `Domain` that is a public suffix is skipped unless it
|
|
602
|
+
equals the host (`localhost`, `wiki`), which gives a host-only cookie.
|
|
603
|
+
- A public suffix is recognised without the Public Suffix List: a name
|
|
604
|
+
with no dot (a top level domain) or a registry under a two-letter country
|
|
605
|
+
code (`co.uk`, `com.au`). A shared suffix such as `github.io` is not
|
|
606
|
+
recognised, so a site there can set a `Domain` cookie for it.
|
|
607
|
+
- A cookie with no `Path` gets the directory of `request_path` (the
|
|
608
|
+
request's path; RFC 6265 section 5.1.4: `/app/login` is `/app`).
|
|
609
|
+
- `FileCookieJar` keeps `HttpOnly` and the include-subdomains flag. A
|
|
610
|
+
by-host file holds every cookie that applies to its host, one a host set
|
|
611
|
+
for a parent domain (`Domain=example.com` from `www.example.com`) included.
|
|
612
|
+
- `host_aliases(groups)` maps each host of each group (lower case, IPv6
|
|
613
|
+
brackets and a trailing dot removed) to the group's first host; a group of
|
|
614
|
+
one host is no alias. A host that is not a name or an IP address (a port,
|
|
615
|
+
a path, a blank) or that is in two groups raises `ValueError`; a group
|
|
616
|
+
given as one string, `TypeError`. `set_cookie_specs` takes the mapping as
|
|
617
|
+
`aliases`, and `HTTPClient` applies its `cookie_hosts` rules with it.
|
|
618
|
+
|
|
619
|
+
## URL hooks (`syshttp.hooks`)
|
|
620
|
+
|
|
621
|
+
```
|
|
622
|
+
SYSHTTP_HOOKS_<KEY>=<regex> re.search against the caller's URL
|
|
623
|
+
SYSHTTP_HOOKS_<KEY>_PY=<module:callable> syshttp calls callable(client, method, url, kwargs)
|
|
624
|
+
SYSHTTP_HOOKS_<KEY>_CURL=<command> a curl front end runs this in its place, its argv appended
|
|
625
|
+
```
|
|
626
|
+
|
|
627
|
+
```python
|
|
628
|
+
class Hook: key, pattern, curl, py, variable; matches(url) -> bool
|
|
629
|
+
def hooks_from_env(*, env=None) -> list[Hook]
|
|
630
|
+
def matching(url: str, *, kind: str | None = None, env=None) -> list[Hook]
|
|
631
|
+
def skipped(*, env=None) -> set[str]
|
|
632
|
+
def call_py_hooks(client, method, url, kwargs, *, env=None) -> HTTPResponse | None
|
|
633
|
+
def load_callable(spec: str) -> PyHook
|
|
634
|
+
def load_object(spec: str, what: str = "an object") -> Any
|
|
635
|
+
def split_command(value: str) -> list[str]
|
|
636
|
+
def curl_command(hook: Hook, python: str | None = None) -> list[str]
|
|
637
|
+
def wrapper_environment(hook: Hook, url: str, entry: str, *, env=None) -> dict[str, str]
|
|
638
|
+
```
|
|
639
|
+
|
|
640
|
+
- Hooks run in KEY order. A `_PY` hook returning non-`None` is used as the
|
|
641
|
+
response and ends the chain; the response is an `HTTPResponse`. `kwargs`
|
|
642
|
+
holds the keyword arguments the `HTTPClient.request()` call gave and is
|
|
643
|
+
mutable (a name `request()` does not take is `TypeError`). A hook declared without
|
|
644
|
+
a `_PY` or `_CURL` target is ignored, and so is a target without a
|
|
645
|
+
pattern, or a blank one.
|
|
646
|
+
- KEYs are compared without regard to case (`Hook.key` is upper-case, as
|
|
647
|
+
Windows holds variable names): `HOOKS_mykey` and `HOOKS_MYKEY` are one
|
|
648
|
+
hook, and `HOOK_SKIP=mykey` suppresses it.
|
|
649
|
+
- `SYSHTTP_HOOK_SKIP` (a comma list of KEYs) suppresses hooks.
|
|
650
|
+
- `load_callable` accepts `module:attr` or `<file>.py:attr` (`attr` may be
|
|
651
|
+
dotted); `load_object` is the same lookup without the callable check, the
|
|
652
|
+
loader `SYSHTTP_DEFAULT_TRANSPORT` shares. A file is loaded once per process, by its resolved path, as a
|
|
653
|
+
`module` is imported once: its module-level state lives on between requests
|
|
654
|
+
and a later edit is not picked up. Two files named alike are two modules.
|
|
655
|
+
- A failing `_PY` hook raises `RuntimeError` naming the variable, the target,
|
|
656
|
+
the method and the URL as scheme, host, port and path (no userinfo, query
|
|
657
|
+
or fragment); the hook's own exception is chained and not rewritten.
|
|
658
|
+
- `curl_command` splits a shell-quoted command line. Backslashes are
|
|
659
|
+
literal, a bare existing path is one program, and a leading `.py` runs
|
|
660
|
+
under `python`.
|
|
661
|
+
- `wrapper_environment` returns the `SYSHTTP_HOOK_URL`, `SYSHTTP_HOOK_KEY`,
|
|
662
|
+
`SYSHTTP_HOOK_CURL` and `SYSHTTP_HOOK_SKIP` (this KEY added) variables a
|
|
663
|
+
`_CURL` wrapper run needs, each under the first prefix.
|
|
664
|
+
|
|
665
|
+
## TLS and retry (`syshttp.tls`, `syshttp.retry`)
|
|
666
|
+
|
|
667
|
+
```python
|
|
668
|
+
def new_os_context(cafile: str | None = None, capath: str | None = None, *,
|
|
669
|
+
client_certificates: bool = True) -> ssl.SSLContext
|
|
670
|
+
def os_ssl_context() -> ssl.SSLContext
|
|
671
|
+
OS_CONTEXT_MAX_AGE = 60
|
|
672
|
+
TRANSIENT_STATUSES = (408, 429, 500, 502, 503, 504)
|
|
673
|
+
def retry_after(headers: Iterable[tuple[str, str]], now: float | None = None) -> float | None
|
|
674
|
+
```
|
|
675
|
+
|
|
676
|
+
- `new_os_context()` without `cafile`/`capath` is
|
|
677
|
+
`sslcontext.system_context()`. On Python 3.10+ it is a
|
|
678
|
+
`truststore.SSLContext`: the operating system verifies the chain
|
|
679
|
+
(CryptoAPI on Windows, the Security framework on macOS, OpenSSL over the
|
|
680
|
+
system store elsewhere), so OpenSSL's CRL flags have no effect on it. On
|
|
681
|
+
3.9 OpenSSL verifies over the default store. A server that asks for a
|
|
682
|
+
client certificate gets one from the OS store (the Windows Personal store
|
|
683
|
+
and smart cards, the macOS keychains, the default PKCS#11 module), chosen
|
|
684
|
+
as a browser chooses (the remembered choice, the Edge/Chrome
|
|
685
|
+
`AutoSelectCertificateForUrls` policy, the only candidate), or none.
|
|
686
|
+
`client_certificates=False` never offers one: pass it before loading a
|
|
687
|
+
certificate of your own, which a store certificate would otherwise
|
|
688
|
+
replace. With `cafile`/`capath` it is a plain `ssl` context trusting only
|
|
689
|
+
them, offering no store certificate.
|
|
690
|
+
- `os_ssl_context()` is shared for `OS_CONTEXT_MAX_AGE` seconds, because
|
|
691
|
+
the clients that pool connections by context object (urllib3) would never
|
|
692
|
+
reuse one otherwise. It is what `syshttp.client` verifies with by default.
|
|
693
|
+
Never load a client certificate into it; use
|
|
694
|
+
`new_os_context(client_certificates=False)`.
|
|
695
|
+
- `retry_after` reads a number of seconds (the digits `0-9` only) or an HTTP
|
|
696
|
+
date, a date without a zone being UTC. It returns `None` when the header
|
|
697
|
+
is absent or unreadable.
|
|
698
|
+
|
|
699
|
+
## Import-time patching (`syshttp.autopatch`)
|
|
700
|
+
|
|
701
|
+
```python
|
|
702
|
+
SETTING = "SYSHTTP_IMPORTPATCH"
|
|
703
|
+
TARGETS = {"requests": "requests", "urllib": "urllib.request", "httpx": "httpx", "aiohttp": "aiohttp",
|
|
704
|
+
"pip": "pip._vendor.requests"}
|
|
705
|
+
def setting_name(*, env=None) -> str
|
|
706
|
+
def enabled(*, env=None) -> frozenset[str]
|
|
707
|
+
def importpatch(name: str, module: ModuleType) -> None # importpatch's entry point
|
|
708
|
+
def patch(targets: Iterable[str] | None = None) -> tuple[str, ...]
|
|
709
|
+
def unpatch() -> None
|
|
710
|
+
def patched() -> tuple[str, ...]
|
|
711
|
+
```
|
|
712
|
+
|
|
713
|
+
- `SYSHTTP_IMPORTPATCH` patches client libraries as they are imported, in
|
|
714
|
+
any program run by an interpreter of the environment. It is read under the
|
|
715
|
+
prefixes like every setting (`SYSHTTP_PREFIX=AGENTS_`:
|
|
716
|
+
`AGENTS_SYSHTTP_IMPORTPATCH`; `setting_name()` is the variable found, else
|
|
717
|
+
the name under the first prefix). Values:
|
|
718
|
+
`1`/`true`/`yes`/`on` for every target, `0`/`false`/`no`/`off`, empty or
|
|
719
|
+
unset for none, else target names separated by `:` (`requests:httpx`;
|
|
720
|
+
`urllib.request` means `urllib`). An unknown name is
|
|
721
|
+
logged at WARNING and skipped. Read when the library is imported.
|
|
722
|
+
- The mechanism is importpatch (a dependency): its `.pth` hooks imports at
|
|
723
|
+
interpreter start, and syshttp's `importpatch` entry points name
|
|
724
|
+
`syshttp.autopatch` for `requests`, `urllib.request`, `httpx` and
|
|
725
|
+
`aiohttp`. `IMPORTPATCH_DISABLE=1` turns every importpatch hook off.
|
|
726
|
+
`patch(targets=None)` applies the patches from code whatever the setting
|
|
727
|
+
says (every installed target by default; a named one that is not installed
|
|
728
|
+
raises its `ImportError`, an unknown name `ValueError`); `unpatch()` undoes
|
|
729
|
+
them, though a client built while patched keeps what it was given.
|
|
730
|
+
- What a patched library does: routing is proxylib's integration for that
|
|
731
|
+
client (`proxylib.patch(proxy.proxy_map(), targets=[name])`, the newest
|
|
732
|
+
patch winning, as proxylib's own `PROXYLIB_IMPORTPATCH` would install it);
|
|
733
|
+
TLS is syshttp's: with verification on and no TLS argument of the
|
|
734
|
+
caller's, the system's context.
|
|
735
|
+
- **requests**: new `Session`s mount proxylib's `ProxyMapAdapter`:
|
|
736
|
+
entries tried in order. A proxy the caller gives (`proxies=` on the
|
|
737
|
+
call, or `session.proxies`) is used as given, without syshttp's
|
|
738
|
+
credential, and a session with `trust_env = False` is left to requests.
|
|
739
|
+
Every `HTTPAdapter` with `verify=True` and no context of its own pool
|
|
740
|
+
manager verifies with `tls.os_ssl_context()`.
|
|
741
|
+
- **urllib**: proxylib's global opener (installed when the program has
|
|
742
|
+
not installed its own) routes `urlopen`, through the first reachable
|
|
743
|
+
entry (`Probed`; none reachable: `URLError` whose `reason` is
|
|
744
|
+
`NoReachableProxyError`). A credential under another header than
|
|
745
|
+
`Proxy-Authorization` cannot go to an https proxy:
|
|
746
|
+
`proxylib.UnsupportedProxyError`. `ssl._create_default_https_context`
|
|
747
|
+
returns a system context, so `http.client` and every caller that gives
|
|
748
|
+
no context verify with it (one context of its own, shared 60 s:
|
|
749
|
+
http.client sets ALPN on it).
|
|
750
|
+
- **httpx**: `Client` / `AsyncClient` built without `transport`, `proxy`,
|
|
751
|
+
`proxies` or `mounts` get proxylib's `ProxyTransport` /
|
|
752
|
+
`AsyncProxyTransport`; `verify` left at `True` becomes the system
|
|
753
|
+
context (a fresh one offering no store certificate when `cert=` is
|
|
754
|
+
given).
|
|
755
|
+
- **aiohttp**: `ClientSession` requests without `proxy=` are routed by
|
|
756
|
+
proxylib, with failover. A URL-prefix gateway is not followed:
|
|
757
|
+
`proxylib.UnsupportedProxyError`. `ssl` left at `True` with no
|
|
758
|
+
`ssl_context` / `verify_ssl` / `fingerprint` becomes the system context.
|
|
759
|
+
- **pip** (`pip._vendor.requests`): every adapter pip mounts routes
|
|
760
|
+
through the map, as proxylib's `pip` target does; `pip --proxy` and
|
|
761
|
+
`--no-proxy-env` keep pip's own choice. pip keeps its own TLS context
|
|
762
|
+
(truststore).
|
|
763
|
+
- Not patched: urllib3 used directly, websocket libraries.
|
|
764
|
+
- proxylib's own `PROXYLIB_AUTOPROXY` names clients that use
|
|
765
|
+
`proxylib.auto_proxy()`; a client syshttp patches uses syshttp's map
|
|
766
|
+
instead (the newer patch wins).
|
|
767
|
+
|
|
768
|
+
## Environment variables
|
|
769
|
+
|
|
770
|
+
| Variable | Sets | Default |
|
|
771
|
+
| --- | --- | --- |
|
|
772
|
+
| `SYSHTTP_PREFIX` | the prefixes every variable below is read under, `:`-separated, in order; read as written | none (the names as written) |
|
|
773
|
+
| `SYSHTTP_PROXY` | the proxy URL (`http://`, `https://`, or a bare `host:port`) | the request's scheme's `*_PROXY` variable, else `ALL_PROXY` (lowercase first); with none set, the system's configuration for the URL |
|
|
774
|
+
| `SYSHTTP_SYSTEM_PROXY` | whether the system's proxy configuration is consulted (boolean) | on |
|
|
775
|
+
| `SYSHTTP_PROXY_AUTH` | **secret**: `SYSHTTP_PROXY`'s credential, a header value sent verbatim | the URL's userinfo as Basic |
|
|
776
|
+
| `HTTP_PROXY_AUTH` | **secret**: the machine's proxy's credential (a `*_PROXY` variable's or the system's), as `Proxy-Authorization` | the URL's userinfo as Basic |
|
|
777
|
+
| `SYSHTTP_PROXY_AUTH_HEADER` | the header name `SYSHTTP_PROXY_AUTH` rides in | `Proxy-Authorization` |
|
|
778
|
+
| `SYSHTTP_PROXY_TYPE` | `connect` or `prefix[:/endpoint]` | `connect` |
|
|
779
|
+
| `SYSHTTP_HOME` | the jar root | the per-user state directory |
|
|
780
|
+
| `SYSHTTP_COOKIE_HOSTS` | groups of hosts that share cookies: hosts split by `,`, groups by `;` (`HTTPClient`) | none |
|
|
781
|
+
| `SYSHTTP_DEFAULT_TRANSPORT` | the transport of an `HTTPClient` (and `fetch()`) given none: `module:attr` or `<file>.py:attr`, an object with `send()` or something that makes one. It names code to import | syshttp's `HTTPTransport` |
|
|
782
|
+
| `SYSHTTP_IMPORTPATCH` | which client libraries are patched on import (`1` for all, or `:`-separated names: `requests`, `urllib`, `httpx`, `aiohttp`, `pip`) | none |
|
|
783
|
+
| `SYSHTTP_HOOKS_<KEY>` (+ `_PY`, `_CURL`) | a URL hook | none |
|
|
784
|
+
| `SYSHTTP_HOOK_SKIP` | KEYs of hooks not to run, comma list | none |
|
|
785
|
+
| `HTTP_PROXY`, `HTTPS_PROXY`, `FTP_PROXY`, `WS_PROXY`, `WSS_PROXY`, `ALL_PROXY` (either case, lowercase first) | the machine's proxy, per scheme | none |
|
|
786
|
+
| `NO_PROXY` / `no_proxy` | hosts that go direct | none |
|
|
787
|
+
| `CA_BUNDLE` | the CA bundle a client's `verify=True` uses (`REQUESTS_CA_BUNDLE` / `CURL_CA_BUNDLE` are not read) | the OS verifier |
|
|
788
|
+
|
|
789
|
+
Each of these is read under every prefix of `SYSHTTP_PREFIX`. A `_CURL`
|
|
790
|
+
hook's wrapper runs with `SYSHTTP_HOOK_URL`, `SYSHTTP_HOOK_KEY`,
|
|
791
|
+
`SYSHTTP_HOOK_CURL` and `SYSHTTP_HOOK_SKIP` set under the first prefix
|
|
792
|
+
(`hooks.wrapper_environment`); a curl front end reads them back.
|
|
793
|
+
|
|
794
|
+
Read as written, never prefixed: `LOCALAPPDATA` (Windows) and
|
|
795
|
+
`XDG_STATE_HOME` (Linux and other POSIX) for the per-user state directory;
|
|
796
|
+
importpatch's `IMPORTPATCH_DISABLE` (`1` turns every import-time patch off);
|
|
797
|
+
proxylib's own `PROXYLIB_IMPORTPATCH` and `PROXYLIB_AUTOPROXY`.
|
|
798
|
+
|
|
799
|
+
## Gotchas
|
|
800
|
+
|
|
801
|
+
- With no proxy variable set, the system's configuration routes requests:
|
|
802
|
+
on a machine with a corporate PAC script, traffic goes where the browser's
|
|
803
|
+
would. `SYSHTTP_SYSTEM_PROXY=0` stops that.
|
|
804
|
+
- `SYSHTTP_PREFIX=MYTOOL_` reads `MYTOOL_<NAME>` only: `SYSHTTP_PROXY` and
|
|
805
|
+
the machine's `HTTPS_PROXY` are then ignored. Add the empty entry
|
|
806
|
+
(`MYTOOL_:`) to fall back to them.
|
|
807
|
+
- A malformed `SYSHTTP_PROXY_TYPE` is harmless until `SYSHTTP_PROXY` is set.
|
|
808
|
+
- Pass `root=` to keep jar files out of the user's state directory.
|
|
809
|
+
- A client certificate loaded into the context you pass is used for every
|
|
810
|
+
https host the client talks to, redirects included. Give a client an
|
|
811
|
+
identity only for hosts that should see it.
|
|
812
|
+
- `verify=False` logs a warning under the `syshttp.client` logger when the
|
|
813
|
+
transport is made, and changes no process-wide warning filter.
|
|
814
|
+
- A `_PY` hook and `SYSHTTP_DEFAULT_TRANSPORT` name code to import. Never let
|
|
815
|
+
an untrusted party set the environment of a process using syshttp.
|
|
816
|
+
- A client keeps the cookies its responses set for its whole life, as a
|
|
817
|
+
session does: build one per identity, or pass `cookies=` a jar of its own.
|