scrapely-python 0.1.36__tar.gz → 0.1.40__tar.gz

scrapely_python-0.1.40/CHANGELOG.md

@@ -0,0 +1,81 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [Unreleased]
+
+## [0.1.40] - 2026-10-28
+
+0.1.39 was never published to PyPI. This release includes the 0.1.39 change below, plus these fixes to `scrapely.http` from the fingerprint test on staging (2026-10-27):
+
+- **Accept-Language matches the exit IP.** Before, a client without a proxy country sent `en-US` on every identity: 8 of 20 residential sessions did so from Brazil, France, Spain, Mexico or Taiwan.
+  - **Residential platform proxy without a country:** each identity now picks an exit country, weighted like real web traffic, asks the proxy for it and sends that country's language.
+  - **Proxy with a country:** each identity gets one of the language setups common there (for example `de-DE,de;q=0.9` or `de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7`).
+  - **Several `locales`:** each identity uses one of them, not all of them in one header. With a residential proxy without a country, the exit follows the locale's country.
+  - **Unchanged:** no proxy, your own proxy URLs and the `DATACENTER` group (US exits only) keep `en-US,en;q=0.9`.
+  - `ProxyConfiguration.new_url()` takes an optional `country_code` for one URL (residential platform proxy without a country only). Sessions expose `accept_language` and `country`.
+- **No more `400` on `https://host?q`.** A URL with a query and no path is sent with the path `/`. wreq 0.13 sent `:path: ?q` over HTTP/2, which Cloudflare and nginx reject.
+- **Chrome 155 and 156.** The default profiles are Chrome 156, 155, 154 and 153, weighted 55 / 30 / 10 / 5. 155 and 156 use the Chrome 154 TLS and HTTP/2 profile, which is the same as real Chrome 156's, with their own User-Agent, `sec-ch-ua` and Accept (with `image/jxl`, which real 155 and 156 send).
+- **Firefox 153 ESR.** Added Firefox 153 (ESR) next to 152. Both match real Firefox 153.6 ESR and 152 builds byte for byte in the TLS ClientHello and HTTP/2 settings, and are now marked verified.
+
+## [0.1.39] - 2026-10-27 (not published)
+
+- A key-value store record write (`Actor.set_value`, `KeyValueStore.set_value`) is no longer lost to one failed answer. Each write now sends an `Idempotency-Key`, and when its answer is lost (a `502`/`504` page from the edge or load balancer, a dropped connection, a timeout) it is sent again with the same key; the API applies and counts it once. Before, the write was never re-sent, so one such page on the final `OUTPUT` write failed the whole run. An API that answers a keyed write without `X-Scrapely-Idempotency` (an older server) gets no re-sends, as before.
+
+## [0.1.38] - 2026-10-27
+
+0.1.37 was never published to PyPI. This release includes everything listed under 0.1.37 below, plus these changes to `scrapely.http`:
+
+- **Instant rotation.** A `429`, block page, captcha, challenge, `511`, `5xx` or network error now moves the request to a brand-new identity at once: a new proxy session, a new profile (a different one from the blocked identity's when there is a choice) and an empty cookie jar.
+  - **No waiting.** By default the client no longer waits: `Retry-After` is ignored, and there is no per-host backoff or slow mode.
+  - **Cap.** `max_rotations` (default 6) caps the rotations per request; `max_retries` is still accepted as its older name. With a fixed proxy URL, or no proxy, a block rotates once.
+- **Polite mode (opt in).** `polite=True` or `backoff=BackoffPolicy(...)` brings back `Retry-After` (capped at 60 s), the 1, 2, 4 … s per-host backoff, the PerimeterX same-identity retry and the circuit-breaker slow mode.
+- **Stats.** `stats()`, the summary line and `HTTP_STATS` count rotations: in total, per reason or vendor, and per host.
+
+## [0.1.37] - 2026-10-27 (not published)
+
+- Added `scrapely.http`, an HTTP client that looks like a real browser and handles blocks (`AsyncClient`, sync `Client`, `session()` for multi-step flows).
+  - **Fingerprint.** TLS, HTTP/2, User-Agent, client hints and header order all come from one current browser profile, listed in `scrapely/http/profiles.json`. The default desktop Chrome profiles send the same JA4, HTTP/2 settings and header order as real Chrome 156.
+  - **Identities.** An identity is a profile, a proxy session from `ProxyConfiguration.new_url(session_id)` and a cookie jar. They are pooled, concurrent requests never share one, and a blocked identity is retired as a whole.
+  - **Block detection.** Cloudflare, PerimeterX, DataDome, Akamai, `429`, `511` and `403` pages are detected, and so are blocks that your own `is_blocked` check reports. Beacons that real pages also carry are not counted as blocks.
+  - **Retries.** Blocked requests are retried on a fresh identity. (0.1.38 changed the waits; see above.)
+  - **Accept-Language.** It follows the proxy country, unless you set `locales`.
+  - **Headers you add** (referer, cookie, origin, content-type, `x-*` and so on) go where a real browser puts them. A User-Agent or client hint you pass is ignored, with one warning.
+  - **Options.** `fingerprint_options` takes snake_case keys, and also the camelCase `fingerprintOptions.fingerprintGeneratorOptions` form.
+  - **Stats.** When the client closes, it logs a one-line summary and writes the run's totals to the default key-value store as `HTTP_STATS`.
+  - **curl_cffi migration.** `AsyncSession` / `Session` accept `impersonate=`, `proxy=`, `proxies=`, `timeout=`, `headers=` and `verify=`, so switching from curl_cffi takes one line.
+- New dependency: `wreq>=0.13,<0.14` (Python 3.11+). If `wreq` can't be imported, the client falls back to `curl_cffi` when it is installed, and logs one warning.
+
+## [0.1.36] - 2026-10-26
+
+- Opening a named dataset, key-value store or request queue (`Actor.open_dataset('name')` and the like) retries when the API answers `503 storage-name-busy`. The API sends it when another request kept creating or renaming a storage of that name for over 15 s; nothing was changed, so the open is simply repeated. It used to fail the open.
+
+## [0.1.35] - 2026-10-26
+
+- Once `Actor.fail()` has run, the run stays failed. A later `Actor.exit()` keeps the failure's exit code and status message, so the usual `try: ... await Actor.fail(...) finally: await Actor.exit()` pattern now ends the run as FAILED. Before, the `exit()` in `finally` replaced the failure's exit code 1 with 0 and the run ended as SUCCEEDED with the failure message as its status. The same holds after `fail()` whose `SystemExit` was caught, after `fail(ExitOptions(exit=False))` (also inside `async with Actor(...)` and `Actor.main()`), and for a second `fail()`, which keeps the first failure's code and message.
+- `Actor.fail(ExitOptions(...))` without an exit code exits with 1. `ExitOptions` defaults `exit_code` to 0, which made such a failure exit with 0. A non-zero `exit_code` in the options, or an explicit `exit_code=` keyword, still wins.
+
+## [0.1.34] - 2026-10-23
+
+- Debug captures (`Actor.debug_capture`, `Actor.debug_capture_sync`) store parsed XML/HTML elements as their markup instead of a repr such as `<Element 'change' at 0x...>`: `xml.etree.ElementTree` elements and trees, lxml elements and trees, BeautifulSoup tags, selectolax nodes and parsel / Scrapy selectors, passed alone or inside a dict or list. The text after an element (its tail) is not included. Other objects are still stored as their `str()`. Captures are only active on the platform's own diagnostic runs (`SCRAPELY_DEBUG_CAPTURE=1`).
+
+## [0.1.33] - 2026-10-22
+
+- Debug captures (`Actor.debug_capture`, `Actor.debug_capture_sync`) keep payloads over 256 KiB whole. They are stored gzip-compressed and base64-encoded as `{key, payload, encoding: 'gzip-base64', length, truncated}` instead of being cut at 262,144 characters; up to 8 Mi characters are kept. A payload that does not compress enough keeps its first 256 KiB, now marked `truncated: True` with its full `length`. Dict and list payloads are dumped as compact JSON, and `bytes` payloads are stored as UTF-8 text (they used to be stored as their `repr`). Captures are only active on the platform's own diagnostic runs (`SCRAPELY_DEBUG_CAPTURE=1`).
+
+## [0.1.32] - 2026-10-20
+
+- Added `scrapely.scrapy`, the Scrapy integration: `run_scrapy_actor`, `initialize_logging`, `apply_apify_settings`, a scheduler backed by the run's request queue, a pipeline that pushes items to the default dataset, a proxy middleware, an HTTP cache storage and an extension that stops the crawl gracefully when the run is aborted. Queued Scrapy requests and cached responses are stored as JSON, never pickled, and a stored request only rebuilds as a `scrapy.Request` subclass. A request leaves the queue as handled only once Scrapy has finished it, so a migrated or aborted run does not lose or repeat it. Scrapy is imported only when these are used.
+
+## [0.1.31] - 2026-10-20
+
+- The storage client percent-encodes request ids and key-value store record keys as one URL path segment and refuses `''`, `.` and `..`. A request id such as `..` read from a queue used to send the call to another resource.
+
+## [0.1.30] - 2026-10-20
+
+- Removed `apify_margin_percentage` from the pricing info models; the API no longer returns it.
+- Added Apache-2.0 change notices to the storage clients, storage utils and log module derived from apify-sdk-python, and listed them in NOTICE.
+
+## [0.1.29] - 2026-10-19
+
+- Initial public release.

{scrapely_python-0.1.36 → scrapely_python-0.1.40}/PKG-INFO

@@ -1,8 +1,8 @@
 Metadata-Version: 2.4
 Name: scrapely-python
-Version: 0.1.36
+Version: 0.1.40
 Summary: Scrapely Python SDK - A toolkit for building Actors on the Scrapely platform
-Author-email: Scrapely <support@scrape.ly>
+Author-email: Scrapely <hello@scrape.ly>
 License-Expression: MIT AND Apache-2.0
 Project-URL: Homepage, https://scrape.ly
 Project-URL: Documentation, https://scrape.ly/docs/sdk-python
@@ -29,6 +29,7 @@ Requires-Dist: websockets>=12.0
 Requires-Dist: scrapely-client-python<2,>=1.0.2
 Requires-Dist: pydantic>=2.0.0
 Requires-Dist: cryptography>=42.0.0
+Requires-Dist: wreq<0.14,>=0.13; python_version >= "3.11"
 Provides-Extra: dev
 Requires-Dist: pytest>=8.0.0; extra == "dev"
 Requires-Dist: pytest-asyncio>=1.0.0; extra == "dev"
@@ -75,6 +76,21 @@ if __name__ == "__main__":
 
 `Actor.main()` initializes the Actor, awaits your callback, closes events and storage, and exits the process. For embedded applications and tests, use `async with Actor:` instead.
 
+## HTTP client
+
+```python
+from scrapely import Actor
+from scrapely.http import AsyncClient
+
+async with Actor:
+    proxy = await Actor.create_proxy_configuration(groups=["RESIDENTIAL"], country_code="DE")
+    async with AsyncClient(proxy_configuration=proxy) as http:
+        r = await http.get("https://www.example.com/")
+        data = (await http.get("https://www.example.com/api/items", kind="json")).json()
+```
+
+Every request uses a current browser profile: TLS, HTTP/2, User-Agent, client hints and header order all come from it. Each request runs on an identity, which is a profile, a proxy session and a cookie jar. When a site blocks an identity (429, block page, captcha), the client retires it and retries the request at once on a fresh one, with a new IP, a new profile and an empty jar. It doesn't wait; `polite=True` turns on Retry-After and backoff. Moving from curl_cffi takes one line: `from scrapely.http import AsyncSession`.
+
 ## Highlights
 
 - Local file-system storage and Scrapely API-backed datasets, key-value stores, and request queues
@@ -83,6 +99,8 @@ if __name__ == "__main__":
 - Automatic remote Chromium CDP wiring through `CDP_URL`
 - Actor start, call, abort, reboot, and metamorph operations
 - Ad-hoc webhooks, proxy configuration, and charging/pricing types
+- `scrapely.http`: an HTTP client with current browser fingerprints (TLS, HTTP/2, headers), identity rotation, block detection and retries; drop-in `AsyncSession` / `Session` for curl_cffi code
+- Scrapy integration in `scrapely.scrapy` (`run_scrapy_actor`, a request-queue scheduler, a dataset pipeline); install Scrapy separately
 - Legacy environment variable aliases and asynchronous client access
 
 See the [complete Python SDK guide](https://scrape.ly/docs/sdk-python) for lifecycle details, option objects, storage selection, events, CDP behavior, run control, environment variables, errors, and the full public export list.

{scrapely_python-0.1.36 → scrapely_python-0.1.40}/README.md

@@ -34,6 +34,21 @@ if __name__ == "__main__":
 
 `Actor.main()` initializes the Actor, awaits your callback, closes events and storage, and exits the process. For embedded applications and tests, use `async with Actor:` instead.
 
+## HTTP client
+
+```python
+from scrapely import Actor
+from scrapely.http import AsyncClient
+
+async with Actor:
+    proxy = await Actor.create_proxy_configuration(groups=["RESIDENTIAL"], country_code="DE")
+    async with AsyncClient(proxy_configuration=proxy) as http:
+        r = await http.get("https://www.example.com/")
+        data = (await http.get("https://www.example.com/api/items", kind="json")).json()
+```
+
+Every request uses a current browser profile: TLS, HTTP/2, User-Agent, client hints and header order all come from it. Each request runs on an identity, which is a profile, a proxy session and a cookie jar. When a site blocks an identity (429, block page, captcha), the client retires it and retries the request at once on a fresh one, with a new IP, a new profile and an empty jar. It doesn't wait; `polite=True` turns on Retry-After and backoff. Moving from curl_cffi takes one line: `from scrapely.http import AsyncSession`.
+
 ## Highlights
 
 - Local file-system storage and Scrapely API-backed datasets, key-value stores, and request queues
@@ -42,6 +57,8 @@ if __name__ == "__main__":
 - Automatic remote Chromium CDP wiring through `CDP_URL`
 - Actor start, call, abort, reboot, and metamorph operations
 - Ad-hoc webhooks, proxy configuration, and charging/pricing types
+- `scrapely.http`: an HTTP client with current browser fingerprints (TLS, HTTP/2, headers), identity rotation, block detection and retries; drop-in `AsyncSession` / `Session` for curl_cffi code
+- Scrapy integration in `scrapely.scrapy` (`run_scrapy_actor`, a request-queue scheduler, a dataset pipeline); install Scrapy separately
 - Legacy environment variable aliases and asynchronous client access
 
 See the [complete Python SDK guide](https://scrape.ly/docs/sdk-python) for lifecycle details, option objects, storage selection, events, CDP behavior, run control, environment variables, errors, and the full public export list.

{scrapely_python-0.1.36 → scrapely_python-0.1.40}/pyproject.toml

@@ -4,13 +4,13 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "scrapely-python"
-version = "0.1.36"
+version = "0.1.40"
 description = "Scrapely Python SDK - A toolkit for building Actors on the Scrapely platform"
 readme = "README.md"
 license = "MIT AND Apache-2.0"
 license-files = ["LICENSE", "LICENSE-APACHE", "NOTICE"]
 authors = [
-    {name = "Scrapely", email = "support@scrape.ly"}
+    {name = "Scrapely", email = "hello@scrape.ly"}
 ]
 classifiers = [
     "Development Status :: 3 - Alpha",
@@ -31,6 +31,7 @@ dependencies = [
     "scrapely-client-python>=1.0.2,<2",
     "pydantic>=2.0.0",
     "cryptography>=42.0.0",
+    "wreq>=0.13,<0.14; python_version>='3.11'",
 ]
 
 [project.optional-dependencies]
@@ -56,7 +57,7 @@ Changelog = "https://github.com/scrapely-app/scrapely-python/blob/main/CHANGELOG
 where = ["src"]
 
 [tool.setuptools.package-data]
-scrapely = ["py.typed"]
+scrapely = ["py.typed", "http/profiles.json"]
 
 [tool.black]
 line-length = 100
@@ -84,4 +85,6 @@ strict_equality = true
 
 [tool.pytest.ini_options]
 asyncio_mode = "auto"
-testpaths = ["tests"]
+testpaths = ["tests"]
+markers = ["network: needs internet access (run with -m network)"]
+addopts = "-m 'not network'"

{scrapely_python-0.1.36 → scrapely_python-0.1.40}/src/scrapely/__init__.py

@@ -227,7 +227,7 @@ if TYPE_CHECKING:  # Static analysers and IDEs see the lazy names as plain impor
         set_api_storage,
     )
 
-__version__ = '0.1.36'
+__version__ = '0.1.40'
 __all__ = [
     # Core
     'Actor',

{scrapely_python-0.1.36 → scrapely_python-0.1.40}/src/scrapely/charging.py

@@ -308,7 +308,8 @@ class ChargingManager:
 
     def _remaining_budget_usd(self) -> Decimal:
         """maxTotalChargeUsd minus this SDK's charges and the platform's other run spend."""
-        assert self._max_total_charge_usd is not None
+        if self._max_total_charge_usd is None:
+            raise RuntimeError('maxTotalChargeUsd is not set')
         return self._max_total_charge_usd - self._total_charged_usd - self._platform_other_usd
 
     def _sync_platform_budget(self) -> None:

{scrapely_python-0.1.36 → scrapely_python-0.1.40}/src/scrapely/events/event_manager.py

@@ -122,13 +122,15 @@ class ScrapelyEventManager(CrawleeEventManager):
         # Run tasks but don't await them
         ws_url = self._configuration.get('actor_events_ws_url')
         if ws_url:
-            # Append authentication parameters to the WebSocket URL
-            # This matches the Node SDK behavior in events-client.ts
+            # Authenticate with the token in a header and the runId in the URL (URLs end up in access
+            # logs). This matches the Node SDK behavior in events-client.ts
             token = self._configuration.get('token')
             run_id = self._configuration.get('actor_run_id')
+            headers: dict[str, str] = {}
             if token and run_id:
                 separator = '&' if '?' in ws_url else '?'
-                ws_url = f"{ws_url}{separator}{urlencode({'token': token, 'runId': run_id})}"
+                ws_url = f"{ws_url}{separator}{urlencode({'runId': run_id})}"
+                headers['Authorization'] = f'Bearer {token}'
                 logger.debug(f'Connecting to platform events websocket with runId={run_id}')
             else:
                 logger.warning(
@@ -139,7 +141,7 @@ class ScrapelyEventManager(CrawleeEventManager):
             # Connect in the background without blocking init, matching the Node SDK: a slow or failed
             # connection is retried and must not crash the Actor.
             self._process_platform_messages_task = asyncio.create_task(
-                self._process_platform_messages(ws_url)
+                self._process_platform_messages(ws_url, headers)
             )
         else:
             logger.debug(
@@ -170,7 +172,7 @@ class ScrapelyEventManager(CrawleeEventManager):
 
         await super().__aexit__(exc_type, exc_value, exc_traceback)
 
-    async def _process_platform_messages(self, ws_url: str) -> None:
+    async def _process_platform_messages(self, ws_url: str, headers: Optional[dict[str, str]] = None) -> None:
         """Process messages from the platform WebSocket with reconnect logic."""
         try:
             import websockets.asyncio.client
@@ -189,7 +191,7 @@ class ScrapelyEventManager(CrawleeEventManager):
         try:
             # Used as an async iterator, `connect` reconnects with exponential backoff on failed connection attempts,
             # including the first one (timeouts, network errors and 5xx responses are retried).
-            async for websocket in websockets.asyncio.client.connect(ws_url):
+            async for websocket in websockets.asyncio.client.connect(ws_url, additional_headers=headers or None):
                 self._platform_events_websocket = websocket
                 if self._connected_to_platform_websocket and not self._connected_to_platform_websocket.done():
                     self._connected_to_platform_websocket.set_result(True)

scrapely_python-0.1.40/src/scrapely/http/__init__.py

@@ -0,0 +1,57 @@
+"""scrapely.http: an HTTP client that looks like a real browser and handles blocks.
+
+Every request goes out with a current browser fingerprint whose TLS, HTTP/2,
+User-Agent, client hints and header order all come from one backend profile.
+Requests use identities (profile + proxy session + cookie jar) that are
+retired together when a site blocks them; the request is retried on a fresh
+identity at once, without waiting: the block was tied to the old identity.
+``polite=True`` turns on Retry-After, per-host backoff and a slow mode for
+actors that must be gentle with a site.
+
+Quick start::
+
+    from scrapely import Actor
+    from scrapely.http import AsyncClient
+
+    proxy = await Actor.create_proxy_configuration(groups=["RESIDENTIAL"], country_code="DE")
+    async with AsyncClient(proxy_configuration=proxy) as http:
+        r = await http.get("https://example.com/")
+        data = (await http.get("https://example.com/api/items", kind="json")).json()
+
+Migrating from curl_cffi is one line: ``from scrapely.http import AsyncSession``.
+"""
+
+from ._client import AsyncClient, AsyncHttpSession, Client, HttpSession
+from ._compat import AsyncSession, Session
+from ._detect import detect_block
+from ._options import FingerprintOptions
+from ._policy import BackoffPolicy
+from ._profiles import BrowserProfile, BrowserSpec, NoMatchingProfile, load_table
+from ._response import Headers, HTTPError, Response
+from ._stats import HttpStats
+from ._transport import TransportError
+
+# curl_cffi / requests-style name for transport errors.
+RequestsError = TransportError
+
+__all__ = [
+    'AsyncClient',
+    'AsyncHttpSession',
+    'AsyncSession',
+    'BackoffPolicy',
+    'BrowserProfile',
+    'BrowserSpec',
+    'Client',
+    'FingerprintOptions',
+    'HTTPError',
+    'Headers',
+    'HttpSession',
+    'HttpStats',
+    'NoMatchingProfile',
+    'RequestsError',
+    'Response',
+    'Session',
+    'TransportError',
+    'detect_block',
+    'load_table',
+]

scrapely_python-0.1.40/src/scrapely/http/_capture.py

@@ -0,0 +1,97 @@
+"""Read the headers a backend profile really sends, over loopback HTTP/1.1.
+
+Used by the profile-table tests and to regenerate the ``headers`` of each
+profile in ``profiles.json`` after a backend upgrade::
+
+    python -m scrapely.http._capture > /tmp/native.json
+
+No network access: a one-shot server on 127.0.0.1 records the request.
+"""
+
+from __future__ import annotations
+
+import json
+import socket
+import sys
+import threading
+from typing import Dict, List, Tuple
+
+
+class LoopbackServer:
+    """Accepts connections on 127.0.0.1 and records each request's header lines in order."""
+
+    def __init__(self, response: bytes = b'HTTP/1.1 200 OK\r\nContent-Length: 2\r\nConnection: close\r\n\r\nok') -> None:
+        self.response = response
+        self.requests: List[str] = []
+        self._sock = socket.socket()
+        self._sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+        self._sock.bind(('127.0.0.1', 0))
+        self._sock.listen(64)
+        self.port = self._sock.getsockname()[1]
+        threading.Thread(target=self._loop, daemon=True).start()
+
+    @property
+    def url(self) -> str:
+        return f'http://127.0.0.1:{self.port}'
+
+    def _loop(self) -> None:
+        while True:
+            try:
+                conn, _ = self._sock.accept()
+            except OSError:
+                return
+            threading.Thread(target=self._one, args=(conn,), daemon=True).start()
+
+    def _one(self, conn: socket.socket) -> None:
+        data = b''
+        try:
+            while b'\r\n\r\n' not in data:
+                chunk = conn.recv(65536)
+                if not chunk:
+                    break
+                data += chunk
+            self.requests.append(data.decode('latin-1'))
+            conn.sendall(self.response)
+        finally:
+            conn.close()
+
+    def close(self) -> None:
+        self._sock.close()
+
+
+def parse_headers(raw: str) -> List[Tuple[str, str]]:
+    head = raw.split('\r\n\r\n', 1)[0]
+    out = []
+    for line in head.split('\r\n')[1:]:
+        if ':' in line:
+            k, v = line.split(':', 1)
+            out.append((k.strip().lower(), v.strip()))
+    return out
+
+
+def native_headers(wreq_profile: str, wreq_platform: str) -> List[Tuple[str, str]]:
+    """Header (name, value) pairs wreq sends for a profile, in its native order (host excluded)."""
+    import wreq
+    from wreq.blocking import Client
+
+    srv = LoopbackServer()
+    try:
+        emu = wreq.Emulation(profile=getattr(wreq.Profile, wreq_profile), platform=getattr(wreq.Platform, wreq_platform))
+        Client(emulation=emu).get(srv.url + '/').text()
+        return [(k, v) for k, v in parse_headers(srv.requests[-1]) if k != 'host']
+    finally:
+        srv.close()
+
+
+def main() -> None:
+    from ._profiles import load_table
+
+    out: Dict[str, Dict[str, str]] = {}
+    for p in load_table().profiles:
+        out[p.id] = dict(native_headers(p.wreq_profile, p.wreq_platform))
+    json.dump(out, sys.stdout, indent=2)
+    sys.stdout.write('\n')
+
+
+if __name__ == '__main__':
+    main()