syshttp 0.1.0__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
syshttp/AGENTS.md ADDED
@@ -0,0 +1,817 @@
1
+ # `syshttp` — public API header
2
+
3
+ Header-file-style reference for the `syshttp` package: every public export with
4
+ its signature, arguments, contract and gotchas, so the package can be used
5
+ without reading its source. It ships inside the package and is
6
+ self-contained. Development documentation lives with the source at
7
+ <https://github.com/jose-pr/syshttp>.
8
+
9
+ Import names from their modules: `syshttp.env`, `syshttp.proxy`,
10
+ `syshttp.client`, `syshttp.jar`, `syshttp.cookies`, `syshttp.hooks`,
11
+ `syshttp.tls`, `syshttp.retry`, `syshttp.auth`, `syshttp.exceptions`,
12
+ `syshttp.autopatch`. Each declares `__all__`: a name outside it is not API,
13
+ and a module whose name starts with `_` is private. The root re-exports
14
+ `HTTPEnv`, the exceptions, `NoReachableProxyError`, `proxy_map`,
15
+ `configured_proxy`, `proxy_type` and `fetch` (the function of
16
+ `syshttp.client`, imported when it is asked for); there is no `syshttp.fetch`
17
+ module.
18
+
19
+ `pip install syshttp` brings proxylib 2.0.2+ (proxy routing and the system's proxy configuration),
20
+ sslcontext (the system's TLS client; truststore on Python 3.10+) and
21
+ importpatch (import-time patching of other HTTP clients, off unless
22
+ `SYSHTTP_IMPORTPATCH` asks for it). The extra `pac`
23
+ adds a JavaScript engine for PAC scripts proxylib evaluates itself (Linux
24
+ without libproxy). The client needs nothing else: `requests` is no dependency
25
+ of any feature; only `syshttp.autopatch` touches it, in a program that imports
26
+ it, when `SYSHTTP_IMPORTPATCH` asks for it.
27
+
28
+ `syshttp.__version__` is the installed distribution's version.
29
+
30
+ ## Settings (`syshttp.env`)
31
+
32
+ Every setting is a variable with a full name: syshttp's own carry
33
+ `SYSHTTP_` (`SYSHTTP_PROXY`, `SYSHTTP_HOME`, `SYSHTTP_HOOKS_<KEY>`),
34
+ conventional ones keep theirs (`HTTPS_PROXY`, `NO_PROXY`, `CA_BUNDLE`). Each
35
+ is read as `<PREFIX><NAME>` for every prefix in turn, the first one set (and
36
+ not blank) winning, each name on its own. The prefixes are the `prefixes`
37
+ argument of an `HTTPEnv`, else `SYSHTTP_PREFIX` (a `:`-separated list), else
38
+ one empty prefix: the names as written. A prefix is used literally, so it
39
+ carries its own separator (`AGENTS_`). Order: an explicit argument, then the
40
+ variable, then the built-in default. Nothing is read at import; each call
41
+ reads the environment as it is then. Below, every name is given as written,
42
+ under the empty prefix.
43
+
44
+ ```python
45
+ class HTTPEnv:
46
+ def __init__(self, prefixes: str | Iterable[str] | None = None,
47
+ environ: Mapping[str, str] | None = None) -> None
48
+ environ: Mapping[str, str] # property: the mapping read (os.environ when None)
49
+ prefixes: tuple[str, ...] # property: the prefixes in force, in lookup order
50
+ def name(self, name: str) -> str
51
+ def find(self, *names: str) -> tuple[str, str] | None
52
+ def get(self, *names: str) -> str | None
53
+ def family(self, stem: str) -> Iterator[tuple[str, str, str]]
54
+ PREFIX_VAR = "SYSHTTP_PREFIX"
55
+ ```
56
+
57
+ - `SYSHTTP_PREFIX=MYTOOL_` reads `MYTOOL_SYSHTTP_PROXY`, `MYTOOL_HTTPS_PROXY`,
58
+ `MYTOOL_CA_BUNDLE` and so on, and nothing unprefixed: a tool opts out of
59
+ the defaults. `MYTOOL_:` reads `MYTOOL_<NAME>` first, then `<NAME>`: a tool
60
+ layers its own settings over them. `SYSHTTP_PREFIX` itself has no prefix.
61
+ - `prefixes=None` reads `SYSHTTP_PREFIX` at every lookup; blank or unset is
62
+ `("",)`. A string is split at `:`; an iterable is taken as the list; an
63
+ empty iterable reads nothing. A prefix is letters, digits and underscores
64
+ (possibly none), upper-cased; one listed twice counts once. A malformed
65
+ argument raises `ValueError` in the constructor; a malformed
66
+ `SYSHTTP_PREFIX` raises `ConfigurationError` naming it at the first lookup.
67
+ - A CGI program receives every request header as an `HTTP_*` variable, so a
68
+ client could set `HTTP_PROXY`, `HTTP_PROXY_AUTH` and the like: under CGI,
69
+ set a prefix list without the empty entry.
70
+ - `environ` isolates an instance: two `HTTPEnv`s with different prefixes or
71
+ mappings never see each other's settings.
72
+ - `find(*names)` returns `(variable, value)` for the first of `names` set
73
+ under the first prefix that has one (every name tried under a prefix before
74
+ the next prefix), the value stripped; `None` when none is. `get` returns
75
+ the value only. `name(name)` is `name` under the first prefix: what a
76
+ message names when nothing is set, and what a child process is given.
77
+ - `family(stem)` yields `(rest, value, variable)` for every variable named
78
+ `<PREFIX><stem><rest>` with a non-empty `rest` and a value that is not
79
+ blank, the first prefix winning for a `rest` (compared upper-cased).
80
+
81
+ Every function below that reads a setting takes `env: HTTPEnv | None = None`
82
+ (keyword-only) and uses `HTTPEnv()` when it is `None`.
83
+
84
+ ## Proxy (`syshttp.proxy`)
85
+
86
+ Routing is proxylib's (2.x): syshttp turns its settings into one proxylib
87
+ proxy map, and proxylib's client integrations send through it. Read an
88
+ answer with `proxylib.lookup(m, url)`; every proxylib integration takes the
89
+ map (`ProxyMapAdapter`, `ProxyMapHandler`, `ProxyTransport`, `ProxySession`).
90
+
91
+ ```python
92
+ def proxy_map(proxy: str | None = None, noproxy: str | None = None, proxy_user: str | None = None,
93
+ *, env=None) -> proxylib.ProxyMap
94
+ def configured_proxy(*, env=None) -> proxylib.Proxy | None
95
+ def system_map() -> proxylib.ProxyMap | None
96
+ def system_proxy_enabled(*, env=None) -> bool
97
+ def auth_header(*, env=None) -> str
98
+ def proxy_type(*, env=None) -> tuple[str, str | None]
99
+ NoReachableProxyError = proxylib.NoReachableProxyError
100
+ SYSTEM_PROXY_MAX_AGE = 60
101
+ PROXY = "PROXY"; PROXY_AUTH = "PROXY_AUTH"; PROXY_AUTH_HEADER = "PROXY_AUTH_HEADER"
102
+ PROXY_TYPE = "PROXY_TYPE"; SYSTEM_PROXY = "SYSTEM_PROXY" # setting names, as `SYSHTTP_<NAME>`
103
+ DEFAULT_AUTH_HEADER = "Proxy-Authorization"; MACHINE_AUTH_VAR = "HTTP_PROXY_AUTH"
104
+ ```
105
+
106
+ - `proxy_map()` reads the settings at every lookup, so one map serves a
107
+ long-lived client, and always answers (never `KeyError`). Its answer is a
108
+ tuple of entries to try in order, `None` for DIRECT:
109
+ 1. `NO_PROXY` / `no_proxy` hosts: DIRECT (proxylib's `BypassRules`:
110
+ suffixes, CIDR, ports, `*`; loopback always).
111
+ 2. `SYSHTTP_PROXY` (`configured_proxy()`), alone.
112
+ 3. Else the machine's proxy: when any of `HTTP_PROXY`, `HTTPS_PROXY`,
113
+ `FTP_PROXY`, `WS_PROXY`, `WSS_PROXY`, `ALL_PROXY` is set (either case),
114
+ those variables alone decide, per scheme (proxylib's `EnvProxyConfig`:
115
+ the lowercase spelling first under each prefix, a PAC result such as
116
+ `PROXY a:1; DIRECT` read as a list); else the system's
117
+ configuration for the URL (`system_map()`), unless `SYSHTTP_SYSTEM_PROXY`
118
+ is off. `HTTP_PROXY_AUTH`, when set, is every such proxy's
119
+ `Proxy-Authorization` (SOCKS entries excepted).
120
+ 4. Else DIRECT.
121
+ - `proxy` (curl's `-x`) replaces 2 and 3 with that proxy, a plain one whose
122
+ credential is only its userinfo; `""` is DIRECT. `noproxy` (`--noproxy`)
123
+ replaces `NO_PROXY`, the variables' own included; listing `<-loopback>`
124
+ turns proxylib's implicit loopback bypass off, as curl has none. `proxy_user` (`-U user:password`) becomes the
125
+ credential of whichever proxy answers.
126
+ - A malformed `SYSHTTP_PROXY`, `SYSHTTP_PROXY_TYPE` or `SYSHTTP_PROXY_AUTH_HEADER`
127
+ raises `ConfigurationError` (proxylib's `ProxyParseError` for a URL) at
128
+ lookup, only when `SYSHTTP_PROXY` is set (the header name only when
129
+ `SYSHTTP_PROXY_AUTH` is too); a bad `SYSHTTP_SYSTEM_PROXY` only when the system
130
+ would be asked.
131
+ - `configured_proxy()` is `SYSHTTP_PROXY` as a proxylib `Proxy` (a bare
132
+ `host:port` is `http`). Its credential is `SYSHTTP_PROXY_AUTH`, a header value
133
+ sent verbatim in `auth_header()` (`SYSHTTP_PROXY_AUTH_HEADER`, default
134
+ `Proxy-Authorization`; a value with `:` or whitespace raises `ValueError`)
135
+ in place of the userinfo, else the userinfo as Basic. A credential (this
136
+ or `HTTP_PROXY_AUTH`) containing a CR, LF or NUL raises `ValueError` naming
137
+ the variable, never the value. It goes to that proxy only: a `CONNECT`, or a plain-http request to it, never through the tunnel.
138
+ `SYSHTTP_PROXY_TYPE=prefix[:/endpoint]` (`proxy_type()`) makes it a
139
+ `prefix+http(s)` gateway: the request for `T` goes to
140
+ `<proxy><endpoint>T`, the credential a header of that request. `syshttp.client`
141
+ does not send an `https:` origin through a plain `prefix+http` gateway.
142
+ - `system_map()` is `proxylib.auto_proxy(environ={})` (the OS resolver --
143
+ WinHTTP, CFNetwork, libproxy -- then the OS settings, PAC and WPAD
144
+ included), built once per `SYSTEM_PROXY_MAX_AGE` seconds, each URL's
145
+ answer cached as long. A configuration that cannot be read is `None`, and
146
+ a lookup that fails is no opinion: DIRECT, logged at DEBUG, never a failed
147
+ request.
148
+ - Proxies never print their secrets: a `Proxy`'s `repr()` hides the password
149
+ and headers, `.url` has no credentials.
150
+ - How an answer is used is the client integration's: requests, httpx and
151
+ aiohttp try the entries in order and move on when one does not connect
152
+ (the last error is raised; a DIRECT entry reached is direct); a client
153
+ that sends once, `syshttp.client` among them, takes the first reachable
154
+ entry through `proxylib.Probed(proxy_map())`, which raises
155
+ `NoReachableProxyError` (a
156
+ `LookupError`) when none answers and the list holds no DIRECT.
157
+
158
+ ## Client (`syshttp.client`)
159
+
160
+ A synchronous HTTP client on the standard library (`http.client`, `ssl`): no
161
+ third-party package, `requests` included. It routes by the proxy map
162
+ (`syshttp.proxy`), verifies with the system's TLS, and keeps the cookies and
163
+ tokens of each origin. `import syshttp.client` loads `http.client`, `ssl`,
164
+ proxylib and sslcontext; `import syshttp` does not.
165
+
166
+ ```python
167
+ class HTTPClient:
168
+ headers: dict
169
+ timeout: float | tuple | None
170
+ deadline: float | None
171
+ user_agent: str
172
+ def __init__(self, transport=None, *, headers=None, timeout=(10, 60), deadline=None, user_agent=None,
173
+ env: HTTPEnv | None = None, proxymap=None, ssl_context=None, proxy_ssl_context=None,
174
+ verify: bool | str | PathLike = True,
175
+ cookies: bool | str | CookieJar | None = False, tokens: bool | str | TokenJar | None = False,
176
+ auth_provider: AuthProvider | None = None, retries: int = 3, backoff: float = 1,
177
+ status_forcelist: Iterable[int] = TRANSIENT_STATUSES,
178
+ cookie_hosts: Iterable[Iterable[str]] | None = None)
179
+ transport # the transport in use
180
+ def request(self, method, url, *, data=None, headers=None, stream=False, allow_redirects=None,
181
+ timeout=..., deadline=..., auth=None) -> HTTPResponse
182
+ def redirect(self, method, url, resp, *, headers=None, stream=False, timeout=..., deadline=...,
183
+ auth=None) -> HTTPResponse
184
+ def close(self) -> None
185
+
186
+ def fetch(url, method="GET", *, headers=None, data=None, json=None, timeout=DEFAULT_TIMEOUT,
187
+ client=None, transport=None, reauth=None, **client_options) -> HTTPResponse
188
+
189
+ DEFAULT_TIMEOUT = (10, 60)
190
+ MAX_REPLY_BYTES = 8388608
191
+ CA_BUNDLE = "CA_BUNDLE"; COOKIE_HOSTS = "SYSHTTP_COOKIE_HOSTS"
192
+ DEFAULT_TRANSPORT = "SYSHTTP_DEFAULT_TRANSPORT" # setting names
193
+ ```
194
+
195
+ - **`request(method, url, *, data=None, headers=None, stream=False,
196
+ allow_redirects=None, timeout, deadline, auth=None)`** returns the final
197
+ `HTTPResponse`. `data` is `bytes`, `str` (UTF-8), a binary file or an
198
+ iterable of `bytes`; one that is not `bytes` cannot be sent twice, so it is
199
+ not sent again after a redirect, a dropped connection or a retry. `auth` is a
200
+ `(user, password)` pair sent as `Authorization: Basic`, or a `str` sent as
201
+ the `Authorization` value (`"Bearer <token>"`), to the origin of `url`
202
+ unless a header names an `Authorization`; it beats a token of the client. A
203
+ `url` that carries credentials, a URL or header value with a control
204
+ character (CR and LF included) and another scheme than `http:` or `https:`
205
+ are `ValueError`, before anything is sent. A status of 400 or more is
206
+ **not** an error: it is the reply (`raise_for_status()`). With
207
+ `stream=False` the body has been read when `request()` returns, and one
208
+ that decodes to more than `MAX_REPLY_BYTES` is `OSError(EFBIG)`; with
209
+ `stream=True` the caller reads the body and closes the response.
210
+ - **Headers and time.** `headers` of the client and of the call are merged by
211
+ name (case-blind, a value of `None` removes the name) over
212
+ `User-Agent: <user_agent>` (default `syshttp/<version>`) and `Accept: */*`.
213
+ Every request gets `timeout=(10, 60)` seconds (`connect`, `read`) unless
214
+ the client or the call gives one; `timeout=None` waits without limit.
215
+ `timeout` bounds each wait; `deadline` (seconds, `None` for none) bounds the
216
+ whole request, redirects, retries and the reading of the body included, and
217
+ `TimeoutError` is raised once it has passed. A call's value replaces the
218
+ client's; a `deadline` that is not a number or `None` is `TypeError`, one
219
+ that is not positive `ValueError`. `repr()` of a client shows no header.
220
+ - **Redirects.** `GET`, `HEAD` and `OPTIONS` follow up to 30 redirects (a
221
+ `303` becomes a `GET`); `allow_redirects=False` returns the first answer. A
222
+ redirect whose `Location` is missing or cannot be requested is
223
+ `OSError(EIO)`. A redirect that changes the scheme, host or port drops
224
+ `Authorization`, `Cookie` and `Proxy-Authorization` for that hop and every
225
+ later one (the jar's cookies are chosen again for each hop), and nothing
226
+ else: every other header goes to the new host as well, so keep a secret in
227
+ one of those three. Every other method is sent without
228
+ following: a 307 or 308 (for `PROPFIND` also a 301, 302 or 303) to the same
229
+ scheme, host and port is sent once more with the same method, headers and
230
+ body, and any other 3xx, a redirect to another origin or a second redirect
231
+ is `OSError(EIO)` naming the status and the `Location` (userinfo, query and
232
+ fragment removed). `redirect(method, url, resp, ...)` sends `method` to where
233
+ `resp`, the answer to a request for `url` made with `allow_redirects=False`,
234
+ redirects to, with the same rules; cookies apply to every hop, the token,
235
+ hooks and retries belong to `request()`.
236
+ - **Routing.** Each request asks the proxy map (`proxymap`, any proxylib
237
+ `ProxyMap` or value `ProxyMap()` builds one from; default
238
+ `proxy.proxy_map(env=env)`) for the entries it may use and takes the first
239
+ that accepts a TCP connection (`proxylib.Probed`; with no entry reachable and
240
+ no `DIRECT`, `proxylib.NoReachableProxyError`). A direct entry connects to
241
+ the origin; an `http:` or `https:` proxy gets an absolute-form request for
242
+ an `http:` origin and a `CONNECT` for an `https:` one, an `https:` proxy
243
+ over TLS and the origin's TLS inside it; a `prefix+http(s)` gateway gets
244
+ `<gateway><path><URL>` with its credential as a header of that request, and
245
+ the response keeps the URL asked for; a plain `prefix+http` gateway is not
246
+ offered an `https:` origin, whose tokens and Secure cookies would cross it in
247
+ clear text: the entry is skipped like a SOCKS one, and with nothing else left
248
+ the request is `proxylib.UnsupportedProxyError`. A proxy's headers (its credential) go
249
+ to the proxy only: in the `CONNECT`, or on the plain-http request to it,
250
+ never through a tunnel. SOCKS entries are skipped; an answer with nothing
251
+ else is `proxylib.UnsupportedProxyError`. A map with no opinion for a URL
252
+ (`{}` for every URL) means a direct connection and the settings are not
253
+ read. Every hop of a redirect is routed again; a connection to a proxy this
254
+ thread already holds is reused without probing it. A `proxymap` of your own
255
+ adds no `SYSHTTP_PROXY_AUTH` credential. A malformed proxy setting
256
+ (`ConfigurationError`) is refused when the client is made.
257
+ - **TLS.** `https:` origins are verified against `ssl_context` when given,
258
+ used as given and never changed (its `verify_mode`, trust store, client
259
+ identity, ciphers and protocol versions decide; `verify` must stay `True`
260
+ beside it, else `ValueError`; a non-`SSLContext` is `TypeError`); else
261
+ against the CA bundle that `verify=<path>` or the `CA_BUNDLE` setting (read
262
+ at each new connection, through the prefixes) names, a context trusting only
263
+ that file (a file that cannot be read is `ConfigurationError` naming where
264
+ the path came from); else against the system's context
265
+ (`tls.os_ssl_context()`: the OS verifies, and a server that asks for a client
266
+ certificate gets one from the OS store). `verify=False` turns verification
267
+ off (a warning is logged under the `syshttp.client` logger when the client's
268
+ transport is made). requests' own `REQUESTS_CA_BUNDLE` / `CURL_CA_BUNDLE` are
269
+ not read. An `https:` proxy or gateway is spoken to with
270
+ `proxy_ssl_context`, else the system's context; the gateway never gets the
271
+ origin's `ssl_context` or client certificate. There is no per-request
272
+ `verify` or client certificate: build another client or load the identity
273
+ into the context.
274
+ - **Cookies.** A client keeps the cookies its responses set, per origin host
275
+ and never per proxy or gateway, on every hop: parsed as RFC 6265 reads them
276
+ (`cookies.set_cookie_specs`), sent as `cookies.cookie_applies` decides
277
+ (longer paths first), expired and deleted ones dropped. A `Cookie` header the
278
+ call or client names replaces the jar's for that request. They live in
279
+ memory for the client's life; `cookies=True` also keeps them in a file jar
280
+ under `jar.store_root(env=env)` by host, a string names one key for every
281
+ host, a jar object is used as given. A host's rows are loaded from the jar
282
+ the first time the host is asked for, and written back only when they differ
283
+ from what was loaded or written last; a jar that fails to read or write is
284
+ ignored. A by-host file gives its host (or group) only the rows whose domain
285
+ covers it; a named key holds every row. A `Set-Cookie` with a control
286
+ character (the tab apart) is ignored whole. A Secure cookie from plain http
287
+ (the loopback hosts apart, as curl has it) is ignored, and plain http cannot
288
+ set, replace or delete a stored Secure cookie of the same name whose domain
289
+ (either containing the other) and path match (RFC 6265bis 5.7). `cookie_hosts` (default `SYSHTTP_COOKIE_HOSTS`, read when the client
290
+ is made: groups split by `;`, hosts by `,`:
291
+ `app.example.com,app.example.net;wiki,wiki.corp.example`; `[]` for none)
292
+ declares groups of hosts that are one for cookies (`cookies.host_aliases`):
293
+ - a host-only cookie a member sets, and one whose `Domain` names a member,
294
+ belongs to the group's first host (stored there, kept in its jar file);
295
+ - a request to any member carries what a request to any member would,
296
+ `Secure` and the path still applying per request;
297
+ - a member deletes such a cookie as the first host would;
298
+ - a `Domain` outside the group is judged on the host that sent it, so a
299
+ group never lets a host set a cookie for a domain that is not its own.
300
+
301
+ Without groups the RFC 6265 rule stands: no host sets or deletes another
302
+ host's cookie. Hosts are names or IP addresses, without a port; a host in
303
+ two groups raises `ValueError`, a malformed variable `ConfigurationError`
304
+ naming it, a group given as one string `TypeError`.
305
+ - **Tokens.** `tokens=True` reads `Authorization` values from file jars under
306
+ `jar.store_root(env=env)` keyed by the origin's host (read at each request),
307
+ a string names one key for every host, a jar object is used as given. A
308
+ token is a per-request `Authorization`: `Bearer <token>`, or verbatim when it
309
+ already names a scheme, never a client header, so it follows no redirect to
310
+ another origin. An explicit `Authorization` header or `auth=` wins. A token
311
+ containing a CR, LF or NUL is `ConfigurationError` naming the jar key, never
312
+ the value.
313
+ - **Auth provider and hooks.** In `request()` (not `redirect()`),
314
+ `auth_provider.ensure(client, url)` runs first and its errors are swallowed;
315
+ then the `_PY` hooks (see "URL hooks") run on the caller's URL with the
316
+ keyword arguments the call gave (`data`, `headers`, `stream`,
317
+ `allow_redirects`, `timeout`, `deadline`, `auth`) as a mutable `kwargs`; one
318
+ that sets another name is `TypeError`. A hook may return a response, which
319
+ is used instead of sending. A request a provider or a hook makes through the
320
+ client runs neither again; the guard is per thread, so another thread's
321
+ requests still run them.
322
+ - **Retries.** A status in `status_forcelist` (default `TRANSIENT_STATUSES`),
323
+ or a failed connection (`ConnectionError`, not a timeout, not
324
+ `NoReachableProxyError`), is sent again up to `retries` times (default 3) for
325
+ `GET`, `HEAD`, `OPTIONS`, `PUT`, `DELETE`, `TRACE` and `PROPFIND` with a body
326
+ that can be sent twice. None waits before the first retry, then `backoff`
327
+ seconds doubled each time, never less than the reply's `Retry-After`, at most
328
+ 120 seconds; the `deadline` bounds the sum. A status that outlasts the
329
+ retries is returned, not raised; a connection that never succeeds raises
330
+ its last error. `retries=0` sends once.
331
+ - **Close.** `close()` (or leaving a `with` block) closes a transport the
332
+ client made, never one it was given; a closed client makes a new one when
333
+ used again.
334
+
335
+ ```python
336
+ class HTTPResponse:
337
+ status: int
338
+ reason: str
339
+ headers: Headers
340
+ url: str
341
+ def __init__(self, status, reason, headers, url, body=b"", *, decode=True): ...
342
+ def read(self, size=-1): ...
343
+ def iter_bytes(self, chunk_size=65536): ...
344
+ def buffered(self): ...
345
+ content: bytes # property
346
+ def text(self, encoding=None, errors="replace") -> str
347
+ def json(self) -> Any
348
+ def close(self): ...
349
+
350
+ class Headers: # read-only Mapping, names case-blind
351
+ def get_all(self, name): ...
352
+
353
+ class HTTPStatusError(Exception):
354
+ response: HTTPResponse
355
+
356
+ def raise_for_status(resp): ...
357
+ def read_limited(resp, what, limit=None): ...
358
+ def json_body(resp, what, limit=None): ...
359
+ def drop_body(resp, limit=None): ...
360
+ def peer_text(text, limit=120): ...
361
+ def split_userinfo(url): ...
362
+ def is_redirect(resp): ...
363
+ ```
364
+
365
+ - **`HTTPResponse`**: `status`, `reason`, `headers` (names compare without
366
+ case, a name sent twice reads as `"a, b"` and `headers.get_all()` lists them;
367
+ `repr()` shows names only), `url` (the URL requested, without fragment, an
368
+ empty path `/`; through a gateway still the URL asked for), `read(size=-1)`,
369
+ `iter_bytes(chunk_size)`, `buffered(limit=None)` (reads the rest of the body
370
+ into memory, closes the connection and returns the response, which stays
371
+ readable; more than `limit` decoded bytes, default `MAX_REPLY_BYTES`, is
372
+ `OSError(EFBIG)`), `close()` and a context manager. A `gzip` or `deflate` body is
373
+ decoded, in steps of at most 256 KiB, unless the request named an
374
+ `Accept-Encoding` of its own (`identity` counts as none); `headers` stay as
375
+ received, so `Content-Length` counts the encoded bytes. A body that ends
376
+ before the length the server stated, a chunked body cut short and a coded
377
+ body that is not valid are `OSError(EIO)`, never a shorter read.
378
+ `content` is the rest of the decoded body, read on first use (at most
379
+ `MAX_REPLY_BYTES`, else `OSError(EFBIG)` naming the URL as scheme, host,
380
+ port and path) and kept, so `content`, `text()`
381
+ and `json()` agree however often they are used; bytes already taken with
382
+ `read()` are not in it. `text()` decodes with `encoding`, else the
383
+ `charset` of `Content-Type`, else UTF-8, replacing what does not decode.
384
+ `json()` parses it (`None` for an empty body); a body that is not JSON is
385
+ `ResponseError` naming the status. A response built from `bytes` by a hook
386
+ or a fake transport has all of this.
387
+ - `raise_for_status(resp)` raises `HTTPStatusError` (holding `response`) for a
388
+ status of 400 or more. `read_limited(resp, what, limit=None)` is the whole
389
+ decoded body, at most `limit` bytes (default `MAX_REPLY_BYTES`, 8 MiB after
390
+ decoding; assign `syshttp.client.MAX_REPLY_BYTES` to allow more, it is read
391
+ each time): more is `OSError(EFBIG)` naming `what`, and the connection is
392
+ closed; `json_body()` is that, parsed (`ValueError` when it is not JSON).
393
+ `drop_body(resp, limit=None)` gives up the body of a reply nobody looks at: it
394
+ is read up to `limit` bytes so the connection can serve the next request,
395
+ and `resp` is closed. `peer_text(text, limit=120)` is text from the other end
396
+ (a reason phrase, a `Location`) as an error message may show it: control and
397
+ bidirectional characters escaped, cut to `limit`. `split_userinfo(url)` is
398
+ `(url_without_userinfo, (user, password) | None)`, the parts the octets their
399
+ percent-escapes stand for; `is_redirect(resp)` says whether a reply is a
400
+ 301, 302, 303, 307 or 308 that names a `Location`.
401
+ - **`fetch(url, method="GET", *, headers=None, data=None, json=None,
402
+ timeout=DEFAULT_TIMEOUT, client=None, transport=None, reauth=None,
403
+ **client_options)`** sends one request and returns the `HTTPResponse` with
404
+ its body read whole (`MAX_REPLY_BYTES` bounds it), whatever the status: take
405
+ `.status`, `.headers`, `.content`, `.text()` or `.json()` from it. The return
406
+ type never depends on an argument. `data` is bytes, text, a mapping
407
+ (form-encoded, `Content-Type: application/x-www-form-urlencoded`) or a binary
408
+ file; `json` is a value sent as JSON (`Content-Type: application/json`;
409
+ `None` sends nothing); both is `ValueError`. It sends through `client` when
410
+ given, left open, else through a new `HTTPClient(transport, **client_options)`
411
+ closed after the call: `transport` is any object implementing the transport
412
+ protocol (`None`: the `SYSHTTP_DEFAULT_TRANSPORT` setting's, else syshttp's
413
+ own) and `client_options` are the other keywords of `HTTPClient`. A `client`
414
+ beside `transport` or `client_options` is `ValueError`: a given client has
415
+ them already. `timeout` is for this call, whatever the client's own is.
416
+ `reauth` is called as `reauth(client, url)` (an object with
417
+ `ensure(client, url)`, an `AuthProvider`, through that) once after a 401 or
418
+ 403, and the request is sent one more time; its errors are the caller's, and
419
+ the body must be one that can be sent twice. `syshttp.fetch` is the same
420
+ function: the root imports the client only when it is asked for.
421
+
422
+ ### The transport protocol and the default transport (`syshttp.client`)
423
+
424
+ ```python
425
+ class Transport(Protocol):
426
+ def send(self, method, url, *, headers, body=None, timeout, stream=False) -> HTTPResponse: ...
427
+ def close(self) -> None: ...
428
+
429
+ class PreparedRequest(NamedTuple):
430
+ method: str; url: str; headers: dict[str, str]; body: Any
431
+ timeout: tuple[float | None, float | None]; decode: bool
432
+
433
+ def prepare_request(method, url, *, headers, body=None, timeout) -> PreparedRequest
434
+
435
+ class HTTPTransport:
436
+ def __init__(self, *, proxymap=None, env=None, ssl_context=None, proxy_ssl_context=None, verify=True)
437
+ def send(self, method, url, *, headers, body=None, timeout, stream=False) -> HTTPResponse
438
+ def close(self) -> None
439
+ ```
440
+
441
+ `HTTPClient(transport=...)` takes any object with `send()` (a transport that
442
+ calls `requests`, `httpx`, a fake in a test); the client owns redirects,
443
+ credentials, cookies, tokens, hooks, retries and the deadline above it, so a
444
+ transport does none of them. Without one the client builds the transport the
445
+ `SYSHTTP_DEFAULT_TRANSPORT` setting names (read through the prefixes when the
446
+ client is made): an object reference `module:attr` or `<file>.py:attr` (a
447
+ dotted `attr` is allowed), resolved as a `_PY` hook is (`hooks.load_object`).
448
+ The object is the transport when it has `send()`; a class or factory (anything
449
+ else callable) is called with no arguments to make one, which the client then
450
+ closes. A value that cannot be imported, resolved or called, or whose result
451
+ has no `send()`, is `ConfigurationError` naming the variable, never a silent
452
+ fallback; `proxymap`, `ssl_context`, `proxy_ssl_context` and `verify` beside a
453
+ setting, or beside a `transport` of your own, are refused. Precedence: an
454
+ explicit `transport=`, then the setting, then `HTTPTransport`. The setting
455
+ names code to import: never let an untrusted party set the environment of a
456
+ process using syshttp.
457
+
458
+ - **The contract.** `send()` makes exactly one request: it follows no
459
+ redirect, adds no credentials of the caller's and retries nothing but what is
460
+ said below. `body` is `None`, `bytes`, a binary file or an iterable of
461
+ `bytes` (never `str`: `TypeError`), sent under a `Content-Length` when its
462
+ length is known without reading it (`bytes`, a list or tuple of `bytes`, a
463
+ seekable file, a regular file), else `Transfer-Encoding: chunked`; a
464
+ `Content-Length` in `headers` is trusted (a longer body is cut there, a count
465
+ larger than a known length or not a number is `ValueError`, a body that runs
466
+ out first is `OSError(EIO)`). `timeout` is seconds, a `(connect, read)` pair
467
+ or `None`; the read half bounds each wait on the server. With `stream=False`
468
+ the body has been read when `send()` returns (a body that decodes to more
469
+ than `MAX_REPLY_BYTES` is `OSError(EFBIG)`); with `stream=True` it is read
470
+ from the response, which the caller closes. A transport raises only
471
+ `TimeoutError`, `ConnectionError`, `ssl.SSLError`, `ValueError` (before
472
+ anything is sent: a header or URL with a control character, a URL with
473
+ credentials, another scheme) or else `OSError(EIO)`; no third-party
474
+ exception and no URL or header value in its text. A reply below 200 is never
475
+ the one it returns. It builds the answer as `HTTPResponse(status, reason,
476
+ headers, url, body, decode=...)` with `body` as `bytes` or an object with
477
+ `read(amount)` and `close()`.
478
+ - **`prepare_request(method, url, *, headers, body=None, timeout)`** is the
479
+ arguments of `send()` as the contract reads them, so a transport that starts
480
+ with it refuses what `HTTPTransport` refuses and frames a body the same way:
481
+ `method` upper-cased, the fragment dropped from `url`, `headers` a `dict` of
482
+ text values with the `Content-Length` of a body whose length is known (`0`
483
+ for a `PATCH`, `POST` or `PUT` without one) and `Accept-Encoding: gzip,
484
+ deflate` unless the caller named an encoding, `body` an object that yields
485
+ exactly `Content-Length` bytes or an iterable to send chunked, `timeout` the
486
+ pair, `decode` what to pass as `HTTPResponse(..., decode=decode)`. It raises
487
+ `ValueError` for a method or header name that is not a token, a value or
488
+ URL with a control character, a URL with credentials or another scheme, a
489
+ `Content-Length` that is not a count of bytes or exceeds the body, and a
490
+ timeout that is not a positive finite number, `None` or a pair of them;
491
+ `TypeError` for a `str` body. No message quotes the URL or a header value.
492
+ - **What `HTTPTransport` does that another transport must do itself** (the
493
+ client does none of it): routing by the proxy map (`proxymap`, `env`: probing,
494
+ failover, proxies, `CONNECT`, gateways, a proxy's credentials), TLS
495
+ verification (`ssl_context`, `proxy_ssl_context`, `verify`, `CA_BUNDLE`, the
496
+ system's context), connection reuse (one idle connection per route and
497
+ thread, at most 8, closed when the thread ends or on `close()`; reused only
498
+ if the server has not closed it or sent bytes on it; an idempotent request
499
+ that fails before any byte of a reply on a reused connection is sent once
500
+ more on a new one, any other request reuses a connection only if it carried
501
+ a reply within the last second), skipping interim (1xx) replies (at most 8),
502
+ decoding `gzip` and `deflate`. A transport written on another library
503
+ therefore gets the library's own proxy and TLS behaviour, not syshttp's,
504
+ unless it implements them; it can still be used by `fetch(transport=...)`.
505
+ `HTTPTransport` adds to the contract's exceptions
506
+ `proxylib.NoReachableProxyError`, `proxylib.UnsupportedProxyError` and
507
+ `ConfigurationError`.
508
+
509
+ ## Exceptions (`syshttp.exceptions`)
510
+
511
+ ```python
512
+ class SysHTTPError(Exception)
513
+ class ConfigurationError(SysHTTPError, ValueError)
514
+ class HookError(SysHTTPError, RuntimeError)
515
+ class ResponseError(SysHTTPError, ValueError)
516
+ ```
517
+
518
+ - `SysHTTPError` is the base of what goes wrong at run time. proxylib's own
519
+ errors (`NoReachableProxyError`, `UnsupportedProxyError`, `ProxyParseError`)
520
+ pass through as they are.
521
+ - `ConfigurationError`: a setting or stored credential that cannot be used
522
+ (a malformed `SYSHTTP_PROXY_TYPE`, a regular expression that does not compile,
523
+ a hook that cannot be loaded, a token with a line break, a malformed
524
+ `SYSHTTP_PREFIX`). The message names the variable or key, never a secret.
525
+ - `HookError`: a `_PY` hook failed; the hook's own exception is `__cause__`.
526
+ - `ResponseError`: a body that is not the JSON `HTTPResponse.json()` was asked for.
527
+ - A caller's own mistake is a plain `ValueError` or `TypeError`, not one of
528
+ these: a malformed `prefixes=` argument, `verify=False` beside `ssl_context`,
529
+ a `client` beside `client_options` in `fetch()`, an unknown patch target.
530
+
531
+ ## Jars (`syshttp.jar`)
532
+
533
+ ```python
534
+ def store_root(*, env=None) -> Path
535
+ def default_root(*, env=None) -> Path
536
+ def safe_name(key: str) -> str
537
+ class FileCookieJar: def __init__(self, root: Path | None = None, *, env=None)
538
+ class FileTokenJar: def __init__(self, root: Path | None = None, *, env=None)
539
+ class MemoryCookieJar(dict)
540
+ class MemoryTokenJar(dict)
541
+ class CookieJar(Protocol); class TokenJar(Protocol) # __getitem__, __setitem__, get
542
+ ```
543
+
544
+ - `store_root()` returns `SYSHTTP_HOME` (a leading `~` expanded), else
545
+ `default_root()`: a directory `syshttp` under `%LOCALAPPDATA%` (Windows),
546
+ `~/Library/Application Support` (macOS) or `$XDG_STATE_HOME` /
547
+ `~/.local/state` (elsewhere).
548
+ - Cookies live in `<root>/cookies/<host>.txt` (Netscape), tokens in
549
+ `<root>/tokens/<host>.token`.
550
+ - Both are secrets and are written as such: mode `0600` (directories this
551
+ library creates `0700`; an existing directory keeps its mode), replaced
552
+ whole by a rename so a reader sees the old file or the new one, with LF line
553
+ ends on every platform. `cookies.save_netscape` does the same. The
554
+ constructors do nothing on disk: the directory is created at the first
555
+ write.
556
+ - `get()` returns the default on any read error. Token values are never
557
+ logged.
558
+ - A client saves a host's cookie file only when its cookies differ from
559
+ those last loaded or saved. A token file is read at every request.
560
+
561
+ ## Cookies (`syshttp.cookies`)
562
+
563
+ ```python
564
+ @dataclass(frozen=True)
565
+ class CookieSpec: domain: str; path: str; secure: bool; expires: int | None; name: str; value: str; http_only: bool = False; subdomains: bool = False
566
+ def load_netscape(path: Path) -> list[CookieSpec]
567
+ def save_netscape(session_cookies: Iterable[CookieLike], path: Path) -> None
568
+ def cookie_applies(cookie: CookieSpec, url: str, now: float | None = None) -> bool
569
+ def set_cookie_specs(values: Iterable[str], default_domain: str, now: float | None = None, *,
570
+ request_path: str = "/",
571
+ aliases: Mapping[str, str] | None = None) -> list[tuple[CookieSpec, bool]]
572
+ def domain_covers(domain: str, host: str) -> bool
573
+ def host_aliases(groups: Iterable[Iterable[str]]) -> dict[str, str]
574
+ HTTP_ONLY_PREFIX = "#HttpOnly_"
575
+ MAX_COOKIE_AGE = 400 * 24 * 3600
576
+ ```
577
+
578
+ - `load_netscape` returns `[]` for a missing file, reads `#HttpOnly_` rows,
579
+ keeps a row whose value is empty and treats an unreadable expiry as a
580
+ session cookie. Lines end at `\n` only (one trailing `\r` is taken off), so
581
+ a character `str.splitlines()` ends a line at (`\x0b`, `\x0c`, `\x85`)
582
+ cannot start a row inside a value; a row whose name or value holds a control
583
+ character is skipped. `save_netscape` writes `#HttpOnly_` rows as curl does,
584
+ and writes no row with a control character or a tab in a field.
585
+ - `cookie_applies` decides as curl does: the domain (a subdomain only with a
586
+ leading dot or the include-subdomains flag, and an IP only matches
587
+ itself), the RFC 6265 path match, `Secure` only over https or to loopback,
588
+ not expired (`0` / `None` is a session cookie).
589
+ - `set_cookie_specs` returns `(cookie, gone)` per `Set-Cookie` value, parsed
590
+ as RFC 6265 section 5.2 reads it: the name and value up to the first `;`
591
+ (a value may hold spaces, `=` or JSON), attribute names in any case,
592
+ unknown attributes (`Priority`, `Partitioned`) ignored, `Max-Age` beating
593
+ `Expires` in either order, an HTTP date in any of its three forms read as
594
+ UTC, and an expiry beyond 400 days cut to `MAX_COOKIE_AGE`. A header with
595
+ no `name=value` is skipped. `gone` marks a cookie the server expired (an
596
+ empty value, `Max-Age` of zero or less, a past date).
597
+ - `default_domain` is the host asked. A `Domain` attribute is kept only
598
+ when that host domain-matches it, on a set and on a deletion alike;
599
+ otherwise the cookie is skipped. It then gives `subdomains=True`. With no
600
+ `Domain`, or with one equal to an IP-literal host, the cookie is
601
+ host-only. A `Domain` that is a public suffix is skipped unless it
602
+ equals the host (`localhost`, `wiki`), which gives a host-only cookie.
603
+ - A public suffix is recognised without the Public Suffix List: a name
604
+ with no dot (a top level domain) or a registry under a two-letter country
605
+ code (`co.uk`, `com.au`). A shared suffix such as `github.io` is not
606
+ recognised, so a site there can set a `Domain` cookie for it.
607
+ - A cookie with no `Path` gets the directory of `request_path` (the
608
+ request's path; RFC 6265 section 5.1.4: `/app/login` is `/app`).
609
+ - `FileCookieJar` keeps `HttpOnly` and the include-subdomains flag. A
610
+ by-host file holds every cookie that applies to its host, one a host set
611
+ for a parent domain (`Domain=example.com` from `www.example.com`) included.
612
+ - `host_aliases(groups)` maps each host of each group (lower case, IPv6
613
+ brackets and a trailing dot removed) to the group's first host; a group of
614
+ one host is no alias. A host that is not a name or an IP address (a port,
615
+ a path, a blank) or that is in two groups raises `ValueError`; a group
616
+ given as one string, `TypeError`. `set_cookie_specs` takes the mapping as
617
+ `aliases`, and `HTTPClient` applies its `cookie_hosts` rules with it.
618
+
619
+ ## URL hooks (`syshttp.hooks`)
620
+
621
+ ```
622
+ SYSHTTP_HOOKS_<KEY>=<regex> re.search against the caller's URL
623
+ SYSHTTP_HOOKS_<KEY>_PY=<module:callable> syshttp calls callable(client, method, url, kwargs)
624
+ SYSHTTP_HOOKS_<KEY>_CURL=<command> a curl front end runs this in its place, its argv appended
625
+ ```
626
+
627
+ ```python
628
+ class Hook: key, pattern, curl, py, variable; matches(url) -> bool
629
+ def hooks_from_env(*, env=None) -> list[Hook]
630
+ def matching(url: str, *, kind: str | None = None, env=None) -> list[Hook]
631
+ def skipped(*, env=None) -> set[str]
632
+ def call_py_hooks(client, method, url, kwargs, *, env=None) -> HTTPResponse | None
633
+ def load_callable(spec: str) -> PyHook
634
+ def load_object(spec: str, what: str = "an object") -> Any
635
+ def split_command(value: str) -> list[str]
636
+ def curl_command(hook: Hook, python: str | None = None) -> list[str]
637
+ def wrapper_environment(hook: Hook, url: str, entry: str, *, env=None) -> dict[str, str]
638
+ ```
639
+
640
+ - Hooks run in KEY order. A `_PY` hook returning non-`None` is used as the
641
+ response and ends the chain; the response is an `HTTPResponse`. `kwargs`
642
+ holds the keyword arguments the `HTTPClient.request()` call gave and is
643
+ mutable (a name `request()` does not take is `TypeError`). A hook declared without
644
+ a `_PY` or `_CURL` target is ignored, and so is a target without a
645
+ pattern, or a blank one.
646
+ - KEYs are compared without regard to case (`Hook.key` is upper-case, as
647
+ Windows holds variable names): `HOOKS_mykey` and `HOOKS_MYKEY` are one
648
+ hook, and `HOOK_SKIP=mykey` suppresses it.
649
+ - `SYSHTTP_HOOK_SKIP` (a comma list of KEYs) suppresses hooks.
650
+ - `load_callable` accepts `module:attr` or `<file>.py:attr` (`attr` may be
651
+ dotted); `load_object` is the same lookup without the callable check, the
652
+ loader `SYSHTTP_DEFAULT_TRANSPORT` shares. A file is loaded once per process, by its resolved path, as a
653
+ `module` is imported once: its module-level state lives on between requests
654
+ and a later edit is not picked up. Two files named alike are two modules.
655
+ - A failing `_PY` hook raises `RuntimeError` naming the variable, the target,
656
+ the method and the URL as scheme, host, port and path (no userinfo, query
657
+ or fragment); the hook's own exception is chained and not rewritten.
658
+ - `curl_command` splits a shell-quoted command line. Backslashes are
659
+ literal, a bare existing path is one program, and a leading `.py` runs
660
+ under `python`.
661
+ - `wrapper_environment` returns the `SYSHTTP_HOOK_URL`, `SYSHTTP_HOOK_KEY`,
662
+ `SYSHTTP_HOOK_CURL` and `SYSHTTP_HOOK_SKIP` (this KEY added) variables a
663
+ `_CURL` wrapper run needs, each under the first prefix.
664
+
665
+ ## TLS and retry (`syshttp.tls`, `syshttp.retry`)
666
+
667
+ ```python
668
+ def new_os_context(cafile: str | None = None, capath: str | None = None, *,
669
+ client_certificates: bool = True) -> ssl.SSLContext
670
+ def os_ssl_context() -> ssl.SSLContext
671
+ OS_CONTEXT_MAX_AGE = 60
672
+ TRANSIENT_STATUSES = (408, 429, 500, 502, 503, 504)
673
+ def retry_after(headers: Iterable[tuple[str, str]], now: float | None = None) -> float | None
674
+ ```
675
+
676
+ - `new_os_context()` without `cafile`/`capath` is
677
+ `sslcontext.system_context()`. On Python 3.10+ it is a
678
+ `truststore.SSLContext`: the operating system verifies the chain
679
+ (CryptoAPI on Windows, the Security framework on macOS, OpenSSL over the
680
+ system store elsewhere), so OpenSSL's CRL flags have no effect on it. On
681
+ 3.9 OpenSSL verifies over the default store. A server that asks for a
682
+ client certificate gets one from the OS store (the Windows Personal store
683
+ and smart cards, the macOS keychains, the default PKCS#11 module), chosen
684
+ as a browser chooses (the remembered choice, the Edge/Chrome
685
+ `AutoSelectCertificateForUrls` policy, the only candidate), or none.
686
+ `client_certificates=False` never offers one: pass it before loading a
687
+ certificate of your own, which a store certificate would otherwise
688
+ replace. With `cafile`/`capath` it is a plain `ssl` context trusting only
689
+ them, offering no store certificate.
690
+ - `os_ssl_context()` is shared for `OS_CONTEXT_MAX_AGE` seconds, because
691
+ the clients that pool connections by context object (urllib3) would never
692
+ reuse one otherwise. It is what `syshttp.client` verifies with by default.
693
+ Never load a client certificate into it; use
694
+ `new_os_context(client_certificates=False)`.
695
+ - `retry_after` reads a number of seconds (the digits `0-9` only) or an HTTP
696
+ date, a date without a zone being UTC. It returns `None` when the header
697
+ is absent or unreadable.
698
+
699
+ ## Import-time patching (`syshttp.autopatch`)
700
+
701
+ ```python
702
+ SETTING = "SYSHTTP_IMPORTPATCH"
703
+ TARGETS = {"requests": "requests", "urllib": "urllib.request", "httpx": "httpx", "aiohttp": "aiohttp",
704
+ "pip": "pip._vendor.requests"}
705
+ def setting_name(*, env=None) -> str
706
+ def enabled(*, env=None) -> frozenset[str]
707
+ def importpatch(name: str, module: ModuleType) -> None # importpatch's entry point
708
+ def patch(targets: Iterable[str] | None = None) -> tuple[str, ...]
709
+ def unpatch() -> None
710
+ def patched() -> tuple[str, ...]
711
+ ```
712
+
713
+ - `SYSHTTP_IMPORTPATCH` patches client libraries as they are imported, in
714
+ any program run by an interpreter of the environment. It is read under the
715
+ prefixes like every setting (`SYSHTTP_PREFIX=AGENTS_`:
716
+ `AGENTS_SYSHTTP_IMPORTPATCH`; `setting_name()` is the variable found, else
717
+ the name under the first prefix). Values:
718
+ `1`/`true`/`yes`/`on` for every target, `0`/`false`/`no`/`off`, empty or
719
+ unset for none, else target names separated by `:` (`requests:httpx`;
720
+ `urllib.request` means `urllib`). An unknown name is
721
+ logged at WARNING and skipped. Read when the library is imported.
722
+ - The mechanism is importpatch (a dependency): its `.pth` hooks imports at
723
+ interpreter start, and syshttp's `importpatch` entry points name
724
+ `syshttp.autopatch` for `requests`, `urllib.request`, `httpx` and
725
+ `aiohttp`. `IMPORTPATCH_DISABLE=1` turns every importpatch hook off.
726
+ `patch(targets=None)` applies the patches from code whatever the setting
727
+ says (every installed target by default; a named one that is not installed
728
+ raises its `ImportError`, an unknown name `ValueError`); `unpatch()` undoes
729
+ them, though a client built while patched keeps what it was given.
730
+ - What a patched library does: routing is proxylib's integration for that
731
+ client (`proxylib.patch(proxy.proxy_map(), targets=[name])`, the newest
732
+ patch winning, as proxylib's own `PROXYLIB_IMPORTPATCH` would install it);
733
+ TLS is syshttp's: with verification on and no TLS argument of the
734
+ caller's, the system's context.
735
+ - **requests**: new `Session`s mount proxylib's `ProxyMapAdapter`:
736
+ entries tried in order. A proxy the caller gives (`proxies=` on the
737
+ call, or `session.proxies`) is used as given, without syshttp's
738
+ credential, and a session with `trust_env = False` is left to requests.
739
+ Every `HTTPAdapter` with `verify=True` and no context of its own pool
740
+ manager verifies with `tls.os_ssl_context()`.
741
+ - **urllib**: proxylib's global opener (installed when the program has
742
+ not installed its own) routes `urlopen`, through the first reachable
743
+ entry (`Probed`; none reachable: `URLError` whose `reason` is
744
+ `NoReachableProxyError`). A credential under another header than
745
+ `Proxy-Authorization` cannot go to an https proxy:
746
+ `proxylib.UnsupportedProxyError`. `ssl._create_default_https_context`
747
+ returns a system context, so `http.client` and every caller that gives
748
+ no context verify with it (one context of its own, shared 60 s:
749
+ http.client sets ALPN on it).
750
+ - **httpx**: `Client` / `AsyncClient` built without `transport`, `proxy`,
751
+ `proxies` or `mounts` get proxylib's `ProxyTransport` /
752
+ `AsyncProxyTransport`; `verify` left at `True` becomes the system
753
+ context (a fresh one offering no store certificate when `cert=` is
754
+ given).
755
+ - **aiohttp**: `ClientSession` requests without `proxy=` are routed by
756
+ proxylib, with failover. A URL-prefix gateway is not followed:
757
+ `proxylib.UnsupportedProxyError`. `ssl` left at `True` with no
758
+ `ssl_context` / `verify_ssl` / `fingerprint` becomes the system context.
759
+ - **pip** (`pip._vendor.requests`): every adapter pip mounts routes
760
+ through the map, as proxylib's `pip` target does; `pip --proxy` and
761
+ `--no-proxy-env` keep pip's own choice. pip keeps its own TLS context
762
+ (truststore).
763
+ - Not patched: urllib3 used directly, websocket libraries.
764
+ - proxylib's own `PROXYLIB_AUTOPROXY` names clients that use
765
+ `proxylib.auto_proxy()`; a client syshttp patches uses syshttp's map
766
+ instead (the newer patch wins).
767
+
768
+ ## Environment variables
769
+
770
+ | Variable | Sets | Default |
771
+ | --- | --- | --- |
772
+ | `SYSHTTP_PREFIX` | the prefixes every variable below is read under, `:`-separated, in order; read as written | none (the names as written) |
773
+ | `SYSHTTP_PROXY` | the proxy URL (`http://`, `https://`, or a bare `host:port`) | the request's scheme's `*_PROXY` variable, else `ALL_PROXY` (lowercase first); with none set, the system's configuration for the URL |
774
+ | `SYSHTTP_SYSTEM_PROXY` | whether the system's proxy configuration is consulted (boolean) | on |
775
+ | `SYSHTTP_PROXY_AUTH` | **secret**: `SYSHTTP_PROXY`'s credential, a header value sent verbatim | the URL's userinfo as Basic |
776
+ | `HTTP_PROXY_AUTH` | **secret**: the machine's proxy's credential (a `*_PROXY` variable's or the system's), as `Proxy-Authorization` | the URL's userinfo as Basic |
777
+ | `SYSHTTP_PROXY_AUTH_HEADER` | the header name `SYSHTTP_PROXY_AUTH` rides in | `Proxy-Authorization` |
778
+ | `SYSHTTP_PROXY_TYPE` | `connect` or `prefix[:/endpoint]` | `connect` |
779
+ | `SYSHTTP_HOME` | the jar root | the per-user state directory |
780
+ | `SYSHTTP_COOKIE_HOSTS` | groups of hosts that share cookies: hosts split by `,`, groups by `;` (`HTTPClient`) | none |
781
+ | `SYSHTTP_DEFAULT_TRANSPORT` | the transport of an `HTTPClient` (and `fetch()`) given none: `module:attr` or `<file>.py:attr`, an object with `send()` or something that makes one. It names code to import | syshttp's `HTTPTransport` |
782
+ | `SYSHTTP_IMPORTPATCH` | which client libraries are patched on import (`1` for all, or `:`-separated names: `requests`, `urllib`, `httpx`, `aiohttp`, `pip`) | none |
783
+ | `SYSHTTP_HOOKS_<KEY>` (+ `_PY`, `_CURL`) | a URL hook | none |
784
+ | `SYSHTTP_HOOK_SKIP` | KEYs of hooks not to run, comma list | none |
785
+ | `HTTP_PROXY`, `HTTPS_PROXY`, `FTP_PROXY`, `WS_PROXY`, `WSS_PROXY`, `ALL_PROXY` (either case, lowercase first) | the machine's proxy, per scheme | none |
786
+ | `NO_PROXY` / `no_proxy` | hosts that go direct | none |
787
+ | `CA_BUNDLE` | the CA bundle a client's `verify=True` uses (`REQUESTS_CA_BUNDLE` / `CURL_CA_BUNDLE` are not read) | the OS verifier |
788
+
789
+ Each of these is read under every prefix of `SYSHTTP_PREFIX`. A `_CURL`
790
+ hook's wrapper runs with `SYSHTTP_HOOK_URL`, `SYSHTTP_HOOK_KEY`,
791
+ `SYSHTTP_HOOK_CURL` and `SYSHTTP_HOOK_SKIP` set under the first prefix
792
+ (`hooks.wrapper_environment`); a curl front end reads them back.
793
+
794
+ Read as written, never prefixed: `LOCALAPPDATA` (Windows) and
795
+ `XDG_STATE_HOME` (Linux and other POSIX) for the per-user state directory;
796
+ importpatch's `IMPORTPATCH_DISABLE` (`1` turns every import-time patch off);
797
+ proxylib's own `PROXYLIB_IMPORTPATCH` and `PROXYLIB_AUTOPROXY`.
798
+
799
+ ## Gotchas
800
+
801
+ - With no proxy variable set, the system's configuration routes requests:
802
+ on a machine with a corporate PAC script, traffic goes where the browser's
803
+ would. `SYSHTTP_SYSTEM_PROXY=0` stops that.
804
+ - `SYSHTTP_PREFIX=MYTOOL_` reads `MYTOOL_<NAME>` only: `SYSHTTP_PROXY` and
805
+ the machine's `HTTPS_PROXY` are then ignored. Add the empty entry
806
+ (`MYTOOL_:`) to fall back to them.
807
+ - A malformed `SYSHTTP_PROXY_TYPE` is harmless until `SYSHTTP_PROXY` is set.
808
+ - Pass `root=` to keep jar files out of the user's state directory.
809
+ - A client certificate loaded into the context you pass is used for every
810
+ https host the client talks to, redirects included. Give a client an
811
+ identity only for hosts that should see it.
812
+ - `verify=False` logs a warning under the `syshttp.client` logger when the
813
+ transport is made, and changes no process-wide warning filter.
814
+ - A `_PY` hook and `SYSHTTP_DEFAULT_TRANSPORT` name code to import. Never let
815
+ an untrusted party set the environment of a process using syshttp.
816
+ - A client keeps the cookies its responses set for its whole life, as a
817
+ session does: build one per identity, or pass `cookies=` a jar of its own.