syshttp 0.1.0__py3-none-any.whl

syshttp/README.md

@@ -0,0 +1,177 @@
+# syshttp
+
+[![Version](https://img.shields.io/pypi/v/syshttp.svg)](https://pypi.org/project/syshttp/)
+[![Python versions](https://img.shields.io/pypi/pyversions/syshttp.svg)](https://pypi.org/project/syshttp/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/jose-pr/syshttp/blob/main/LICENSE)
+[![Docs](https://img.shields.io/badge/docs-latest-blue.svg)](https://jose-pr.github.io/syshttp/)
+[![CI](https://img.shields.io/github/actions/workflow/status/jose-pr/syshttp/test.yml)](https://github.com/jose-pr/syshttp/actions/workflows/test.yml)
+
+**HTTP plumbing that behaves the same in every client you build on it**:
+which proxy to use, with which credential, for which hosts; how a prefix
+gateway is spoken to; which cookies and tokens go with which origin; and TLS
+through the operating system's own certificate verifier. By default it
+routes the way the system does: the system's proxy configuration, PAC scripts
+and WPAD included, unless a proxy variable says otherwise. An HTTP client on the
+standard library puts all of them together; `requests` is needed by none of
+it. Documentation: <https://jose-pr.github.io/syshttp/>.
+
+## Features
+
+- **Routing is [proxylib](https://pypi.org/project/proxylib/)'s** —
+  `proxy.proxy_map()` turns the settings below into one proxylib proxy map,
+  and proxylib's client integrations send through it.
+- **The system's proxy, by default** — with no proxy variable set, the
+  system's configuration decides per URL: WinHTTP on Windows, CFNetwork on
+  macOS, libproxy or the desktop's settings elsewhere, PAC and WPAD included.
+  A list (`PROXY a; PROXY b; DIRECT`) is tried in order, the next entry when
+  one does not connect; with no proxy connecting and no DIRECT the request
+  fails rather than going direct.
+- **Proxies with credentials** — a proxy URL from `SYSHTTP_PROXY`, else the
+  usual `HTTPS_PROXY` / `HTTP_PROXY` / `ALL_PROXY` for the request's scheme.
+  The credential is a header value sent verbatim to that proxy only, so any
+  scheme works (`Basic`, `Bearer`, `Negotiate`). `NO_PROXY` is honoured
+  (CIDR included), and every redirect hop is decided again.
+- **Prefix gateways** — `SYSHTTP_PROXY_TYPE=prefix:/fetch/` sends
+  `GET https://api.example.com/x` to `<proxy>/fetch/https://api.example.com/x`
+  (the client, requests, httpx and urllib; the client sends an `https:` origin
+  through an `https` gateway only). Responses, redirects and cookies still use
+  the caller's URL.
+- **Prefixes of a tool's own** — `SYSHTTP_PREFIX=MYTOOL_` makes the package
+  read `MYTOOL_HTTPS_PROXY`, `MYTOOL_SYSHTTP_HOME` and so on instead of the
+  defaults; `MYTOOL_:` falls back to them, variable by variable. An
+  `HTTPEnv("MYTOOL_")` object keeps one configuration apart from another in
+  the same process.
+- **Cookie and token jars** — Netscape cookie files with curl's matching
+  rules, plus per-host token files. Both are chosen by the origin's host,
+  never by the proxy's or the gateway's.
+- **URL hooks** — `SYSHTTP_HOOKS_<KEY>=<regex>` plus `_PY=module:callable`
+  runs your code before matching requests: a login, a signature, a canned
+  response.
+- **The system's TLS client, or your own TLS context** — by default
+  [sslcontext](https://pypi.org/project/sslcontext/)'s system context: the
+  operating system verifies certificates (CryptoAPI, the Security framework),
+  with no `certifi` bundle involved, and a server that asks for a client
+  certificate gets one from the OS store (smart cards included), chosen as a
+  browser chooses. Or `HTTPClient(ssl_context=...)` for a given identity, a
+  cipher list or a pinned protocol version.
+- **An HTTP client on the standard library** — `HTTPClient` and `fetch()`:
+  retry with backoff, redirects that keep a credential with its origin, and
+  all of the above, on `http.client` and `ssl` alone. Its transport is
+  replaceable: any object with `send()` (a `requests` or `httpx` one, a fake
+  in a test), chosen per client or by `SYSHTTP_DEFAULT_TRANSPORT`.
+- **Other programs' clients, patched on import** — `SYSHTTP_IMPORTPATCH=1`
+  (or `requests:urllib:httpx:aiohttp:pip`) makes those libraries route and
+  verify the same way in any program of the environment, through
+  [importpatch](https://pypi.org/project/importpatch/); what a caller sets
+  itself still wins.
+
+## Installation
+
+```bash
+pip install syshttp
+```
+
+| Extra | Adds | Needed for |
+| --- | --- | --- |
+| `pac` | a JavaScript engine for proxylib | PAC scripts proxylib evaluates itself (Linux without libproxy) |
+
+syshttp depends on proxylib (the system proxy), sslcontext (the system's
+TLS client: the OS verifier through truststore on Python 3.10+, OpenSSL over
+the default store on 3.9) and importpatch (its `.pth` hooks imports at
+interpreter start; nothing is patched unless `SYSHTTP_IMPORTPATCH` says so).
+
+## Quick start
+
+Ask where a request would go, the way every client here does:
+
+```python
+import proxylib
+from syshttp import proxy
+
+for entry in proxylib.lookup(proxy.proxy_map(), "https://api.example.com/"):
+    print("DIRECT" if entry is None else entry.url)   # tried in this order
+```
+
+One request, routed and verified the way the system does (no extra to install):
+
+```python
+from syshttp import fetch
+
+response = fetch("https://api.example.com/thing")
+print(response.status, response.json())
+```
+
+A client with file-backed cookies and tokens, kept across requests:
+
+```python
+from syshttp.client import HTTPClient
+
+with HTTPClient(cookies=True, tokens=True) as client:
+    response = client.request("GET", "https://api.example.com/thing")
+```
+
+A separate configuration, isolated from the process environment:
+
+```python
+from syshttp import HTTPEnv, fetch
+
+env = HTTPEnv("MYTOOL_", environ={"MYTOOL_SYSHTTP_PROXY": "http://proxy.internal:3128"})
+response = fetch("https://api.example.com/thing", env=env)
+```
+
+## API overview
+
+| Module | Purpose |
+| --- | --- |
+| `syshttp.env` | `HTTPEnv`: the prefixes and mapping every setting is read from |
+| `syshttp.proxy` | `proxy_map()`: the settings as one proxylib proxy map |
+| `syshttp.client` | `HTTPClient` and `fetch()`: the HTTP client, its transport protocol and response |
+| `syshttp.jar` | file and memory cookie/token jars |
+| `syshttp.cookies` | Netscape cookie files and curl's cookie rules |
+| `syshttp.hooks` | URL hooks declared in the environment |
+| `syshttp.tls` | the OS verifier as an `SSLContext` |
+| `syshttp.autopatch` | requests, urllib, httpx, aiohttp and pip patched on import |
+| `syshttp.retry` | the transient statuses, `Retry-After` |
+
+## Settings
+
+Every setting is read as `<PREFIX><NAME>` for each prefix of
+`SYSHTTP_PREFIX` (`:`-separated; default none, the names as written). The
+full list is on the docs site:
+<https://jose-pr.github.io/syshttp/settings/>.
+
+| Variable | Sets |
+| --- | --- |
+| `SYSHTTP_PREFIX` | the prefixes every other setting is read under |
+| `SYSHTTP_PROXY` | the proxy URL; else `HTTPS_PROXY`, `HTTP_PROXY`, `ALL_PROXY`, else the system's |
+| `SYSHTTP_SYSTEM_PROXY` | `0` to stop consulting the system's proxy configuration |
+| `SYSHTTP_PROXY_AUTH` | the proxy credential, sent verbatim; else `HTTP_PROXY_AUTH` |
+| `SYSHTTP_PROXY_AUTH_HEADER` | the header it is sent in (default `Proxy-Authorization`) |
+| `SYSHTTP_PROXY_TYPE` | `connect` (default) or `prefix[:/endpoint]` |
+| `SYSHTTP_HOME` | the jar directory (default: the per-user state directory) |
+| `SYSHTTP_IMPORTPATCH` | `1`, or `:`-separated names: the client libraries patched on import |
+| `SYSHTTP_HOOKS_<KEY>` | a URL hook's pattern, with `_PY` / `_CURL` targets |
+| `SYSHTTP_DEFAULT_TRANSPORT` | `module:attr` or `<file>.py:attr`: the transport of a client given none (it names code to import) |
+| `CA_BUNDLE` | the CA bundle a client verifies with (default: the OS verifier) |
+
+## Development
+
+```bash
+dotagents pyvenv            # or any Python 3.9+ virtual environment
+.pyvenv/<tag>/Scripts/python -m pip install -e ".[dev]"
+.pyvenv/<tag>/Scripts/python -m pytest -q
+```
+
+### Releasing
+
+This project follows [Semantic Versioning](https://semver.org/) and keeps a
+[`CHANGELOG.md`](https://github.com/jose-pr/syshttp/blob/main/CHANGELOG.md).
+Pushing a tag matching `v*` triggers the release workflow: test gate → build
+(checking the tag names the version built) → a strict docs build as a gate →
+GitHub release → publish. The release workflow never deploys the docs site
+itself: for a final release its last job dispatches the docs workflow at the
+tag, which owns every Pages deploy.
+
+## License
+
+MIT — see [LICENSE](https://github.com/jose-pr/syshttp/blob/main/LICENSE).

syshttp/__init__.py

@@ -0,0 +1,71 @@
+"""syshttp -- HTTP plumbing that follows the system, and an HTTP client on the
+standard library that uses it.
+
+Public surface (the package's AGENTS.md documents each name):
+  env.HTTPEnv                          — where settings come from: <PREFIX><NAME> for each
+                                         prefix of SYSHTTP_PREFIX (default: none)
+  proxy.proxy_map                      — the routing as a proxylib ProxyMap: SYSHTTP_PROXY with
+                                         its credential, else the *_PROXY variables or the
+                                         system's configuration, NO_PROXY first (-x / -U /
+                                         --noproxy overrides included)
+  proxy.configured_proxy / proxy_type  — SYSHTTP_PROXY as a proxylib Proxy; SYSHTTP_PROXY_TYPE
+                                         (connect | prefix[:/endpoint])
+  client.HTTPClient / fetch            — the client: proxy map, system TLS, cookies, tokens,
+                                         hooks, retries; replaceable transport. ``fetch`` is
+                                         also ``syshttp.fetch``
+  cookies.load_netscape / save_netscape / cookie_applies / set_cookie_specs
+                                       — Netscape jars and curl's cookie rules
+  tls.os_ssl_context / new_os_context  — the system's TLS client (sslcontext) as an SSLContext
+  retry.TRANSIENT_STATUSES / retry_after — what counts as transient, Retry-After
+  jar.FileCookieJar / FileTokenJar     — file-backed jars under SYSHTTP_HOME
+  hooks                                — SYSHTTP_HOOKS_<KEY> URL hooks (a curl wrapper,
+                                         a syshttp callable) matched on the caller's URL
+  autopatch                            — SYSHTTP_IMPORTPATCH: requests, urllib, httpx, aiohttp,
+                                         pip patched on import to route and verify the same way
+
+Nothing here imports ``requests``: only ``autopatch`` patches it, in a program
+that imports it. Importing ``syshttp`` itself loads neither proxylib nor the
+client.
+"""
+
+from importlib.metadata import PackageNotFoundError, version
+from typing import Any
+
+from .env import HTTPEnv
+from .exceptions import ConfigurationError, HookError, ResponseError, SysHTTPError
+
+__all__ = [
+    "ConfigurationError",
+    "HTTPEnv",
+    "HookError",
+    "NoReachableProxyError",
+    "ResponseError",
+    "SysHTTPError",
+    "configured_proxy",
+    "fetch",
+    "proxy_map",
+    "proxy_type",
+]
+
+# Read from syshttp.proxy when first asked for: importing syshttp must not
+# import proxylib, whose own import of urllib.request runs syshttp's
+# import-time patch (syshttp.autopatch) while proxylib is half built.
+_FROM_PROXY = ("NoReachableProxyError", "configured_proxy", "proxy_map", "proxy_type")
+
+
+def __getattr__(name: str) -> Any:
+    if name in _FROM_PROXY:
+        from . import proxy
+
+        return getattr(proxy, name)
+    if name == "fetch":
+        from .client import fetch
+
+        return fetch
+    raise AttributeError("module %r has no attribute %r" % (__name__, name))
+
+
+try:
+    __version__ = version("syshttp")
+except PackageNotFoundError:  # a source tree on sys.path, not installed
+    __version__ = "0+unknown"

syshttp/_fileio.py

@@ -0,0 +1,78 @@
+"""Writing files that hold secrets: owner-only and replaced whole."""
+
+from __future__ import annotations
+
+import os
+import tempfile
+import time
+from pathlib import Path
+
+#: ``os.replace`` onto a file another handle has open fails with
+#: ``PermissionError`` on Windows until that handle closes, which a reader
+#: does within microseconds.
+_REPLACE_TRIES = 10
+_REPLACE_WAIT = 0.01
+
+
+def _make_directories(directory: Path) -> None:
+    """Create ``directory`` and its missing parents with mode ``0o700``; a
+    directory that exists is left as it is."""
+    missing = []
+    while not directory.exists() and directory != directory.parent:
+        missing.append(directory)
+        directory = directory.parent
+    for created in reversed(missing):
+        try:
+            created.mkdir(mode=0o700)
+        except FileExistsError:
+            continue
+        try:
+            created.chmod(0o700)
+        except OSError:  # a filesystem without modes
+            pass
+
+
+def read_text(path: Path) -> str:
+    """The UTF-8 text of ``path`` (undecodable bytes dropped). On Windows a
+    file being replaced refuses to open for an instant: that is retried."""
+    for attempt in range(_REPLACE_TRIES):
+        try:
+            return Path(path).read_text(encoding="utf-8", errors="ignore")
+        except PermissionError:
+            if attempt == _REPLACE_TRIES - 1:
+                raise
+            time.sleep(_REPLACE_WAIT)
+    raise AssertionError("unreachable")  # pragma: no cover
+
+
+def _replace(source: str, target: Path) -> None:
+    for attempt in range(_REPLACE_TRIES):
+        try:
+            os.replace(source, target)
+            return
+        except PermissionError:
+            if attempt == _REPLACE_TRIES - 1:
+                raise
+            time.sleep(_REPLACE_WAIT)
+
+
+def write_private(path: Path, data: bytes) -> None:
+    """Write ``data`` to ``path`` so that a reader sees the old file or the
+    new one, never part of either, and only the owner can read it (``0o600``;
+    missing parent directories are ``0o700``). The bytes are written as
+    given: no newline translation."""
+    path = Path(path)
+    _make_directories(path.parent)
+    fd, temporary = tempfile.mkstemp(dir=str(path.parent), prefix=path.name + ".", suffix=".tmp")
+    try:
+        with os.fdopen(fd, "wb") as handle:
+            handle.write(data)
+            handle.flush()
+            os.fsync(handle.fileno())
+        _replace(temporary, path)
+    except BaseException:
+        try:
+            os.unlink(temporary)
+        except OSError:
+            pass
+        raise

syshttp/_parse.py

@@ -0,0 +1,30 @@
+"""Readers for the numbers and dates that headers and cookie files carry."""
+
+from __future__ import annotations
+
+import email.utils
+from datetime import timezone
+from typing import Optional
+
+
+def ascii_int(value: str) -> Optional[int]:
+    """``value`` as a non-negative integer when it is only the digits
+    ``0-9`` (``str.isdigit`` also accepts ``"²"``); else ``None``."""
+    value = value.strip()
+    return int(value) if value.isascii() and value.isdigit() else None
+
+
+def http_date(value: str) -> Optional[float]:
+    """An HTTP date (IMF-fixdate, RFC 850 or asctime) as a unix time, or
+    ``None`` when unreadable. A date without a zone is UTC (RFC 9110
+    section 5.6.7)."""
+    try:
+        when = email.utils.parsedate_to_datetime(value)
+    except (TypeError, ValueError, IndexError, OverflowError):
+        return None
+    if when.tzinfo is None:
+        when = when.replace(tzinfo=timezone.utc)
+    try:
+        return when.timestamp()
+    except (ValueError, OverflowError, OSError):
+        return None

syshttp/auth.py

@@ -0,0 +1,17 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Protocol
+
+if TYPE_CHECKING:  # the client imports this module: types only
+    from .client import HTTPClient
+
+
+__all__ = ("AuthProvider",)
+
+
+class AuthProvider(Protocol):
+    """Something that makes ``client`` authenticated for ``url`` (a login,
+    a token refresh): ``HTTPClient`` calls it before each request, and
+    ``fetch(reauth=...)`` again after a 401 or 403."""
+
+    def ensure(self, client: "HTTPClient", url: str) -> None: ...