scrapely-python 0.1.35__tar.gz → 0.1.38__tar.gz

{scrapely_python-0.1.35 → scrapely_python-0.1.38}/CHANGELOG.md

@@ -2,6 +2,36 @@
 
 All notable changes to this project will be documented in this file.
 
+## [Unreleased]
+
+## [0.1.38] - 2026-10-27
+
+0.1.37 was never published to PyPI. This release includes everything listed under 0.1.37 below, plus these changes to `scrapely.http`:
+
+- **Instant rotation.** A `429`, block page, captcha, challenge, `511`, `5xx` or network error now moves the request to a brand-new identity at once: a new proxy session, a new profile (a different one from the blocked identity's when there is a choice) and an empty cookie jar.
+  - **No waiting.** By default the client no longer waits: `Retry-After` is ignored, and there is no per-host backoff or slow mode.
+  - **Cap.** `max_rotations` (default 6) caps the rotations per request; `max_retries` is still accepted as its older name. With a fixed proxy URL, or no proxy, a block rotates once.
+- **Polite mode (opt in).** `polite=True` or `backoff=BackoffPolicy(...)` brings back `Retry-After` (capped at 60 s), the 1, 2, 4 … s per-host backoff, the PerimeterX same-identity retry and the circuit-breaker slow mode.
+- **Stats.** `stats()`, the summary line and `HTTP_STATS` count rotations: in total, per reason or vendor, and per host.
+
+## [0.1.37] - 2026-10-27 (not published)
+
+- Added `scrapely.http`, an HTTP client that looks like a real browser and handles blocks (`AsyncClient`, sync `Client`, `session()` for multi-step flows).
+  - **Fingerprint.** TLS, HTTP/2, User-Agent, client hints and header order all come from one current browser profile, listed in `scrapely/http/profiles.json`. The default desktop Chrome profiles send the same JA4, HTTP/2 settings and header order as real Chrome 156.
+  - **Identities.** An identity is a profile, a proxy session from `ProxyConfiguration.new_url(session_id)` and a cookie jar. They are pooled, concurrent requests never share one, and a blocked identity is retired as a whole.
+  - **Block detection.** Cloudflare, PerimeterX, DataDome, Akamai, `429`, `511` and `403` pages are detected, and so are blocks that your own `is_blocked` check reports. Beacons that real pages also carry are not counted as blocks.
+  - **Retries.** Blocked requests are retried on a fresh identity. (0.1.38 changed the waits; see above.)
+  - **Accept-Language.** It follows the proxy country, unless you set `locales`.
+  - **Headers you add** (referer, cookie, origin, content-type, `x-*` and so on) go where a real browser puts them. A User-Agent or client hint you pass is ignored, with one warning.
+  - **Options.** `fingerprint_options` takes snake_case keys, and also the camelCase `fingerprintOptions.fingerprintGeneratorOptions` form.
+  - **Stats.** When the client closes, it logs a one-line summary and writes the run's totals to the default key-value store as `HTTP_STATS`.
+  - **curl_cffi migration.** `AsyncSession` / `Session` accept `impersonate=`, `proxy=`, `proxies=`, `timeout=`, `headers=` and `verify=`, so switching from curl_cffi takes one line.
+- New dependency: `wreq>=0.13,<0.14` (Python 3.11+). If `wreq` can't be imported, the client falls back to `curl_cffi` when it is installed, and logs one warning.
+
+## [0.1.36] - 2026-10-26
+
+- Opening a named dataset, key-value store or request queue (`Actor.open_dataset('name')` and the like) retries when the API answers `503 storage-name-busy`. The API sends it when another request kept creating or renaming a storage of that name for over 15 s; nothing was changed, so the open is simply repeated. It used to fail the open.
+
 ## [0.1.35] - 2026-10-26
 
 - Once `Actor.fail()` has run, the run stays failed. A later `Actor.exit()` keeps the failure's exit code and status message, so the usual `try: ... await Actor.fail(...) finally: await Actor.exit()` pattern now ends the run as FAILED. Before, the `exit()` in `finally` replaced the failure's exit code 1 with 0 and the run ended as SUCCEEDED with the failure message as its status. The same holds after `fail()` whose `SystemExit` was caught, after `fail(ExitOptions(exit=False))` (also inside `async with Actor(...)` and `Actor.main()`), and for a second `fail()`, which keeps the first failure's code and message.

{scrapely_python-0.1.35 → scrapely_python-0.1.38}/PKG-INFO

@@ -1,8 +1,8 @@
 Metadata-Version: 2.4
 Name: scrapely-python
-Version: 0.1.35
+Version: 0.1.38
 Summary: Scrapely Python SDK - A toolkit for building Actors on the Scrapely platform
-Author-email: Scrapely <support@scrape.ly>
+Author-email: Scrapely <hello@scrape.ly>
 License-Expression: MIT AND Apache-2.0
 Project-URL: Homepage, https://scrape.ly
 Project-URL: Documentation, https://scrape.ly/docs/sdk-python
@@ -29,6 +29,7 @@ Requires-Dist: websockets>=12.0
 Requires-Dist: scrapely-client-python<2,>=1.0.2
 Requires-Dist: pydantic>=2.0.0
 Requires-Dist: cryptography>=42.0.0
+Requires-Dist: wreq<0.14,>=0.13; python_version >= "3.11"
 Provides-Extra: dev
 Requires-Dist: pytest>=8.0.0; extra == "dev"
 Requires-Dist: pytest-asyncio>=1.0.0; extra == "dev"
@@ -75,6 +76,21 @@ if __name__ == "__main__":
 
 `Actor.main()` initializes the Actor, awaits your callback, closes events and storage, and exits the process. For embedded applications and tests, use `async with Actor:` instead.
 
+## HTTP client
+
+```python
+from scrapely import Actor
+from scrapely.http import AsyncClient
+
+async with Actor:
+    proxy = await Actor.create_proxy_configuration(groups=["RESIDENTIAL"], country_code="DE")
+    async with AsyncClient(proxy_configuration=proxy) as http:
+        r = await http.get("https://www.example.com/")
+        data = (await http.get("https://www.example.com/api/items", kind="json")).json()
+```
+
+Every request uses a current browser profile: TLS, HTTP/2, User-Agent, client hints and header order all come from it. Each request runs on an identity, which is a profile, a proxy session and a cookie jar. When a site blocks an identity (429, block page, captcha), the client retires it and retries the request at once on a fresh one, with a new IP, a new profile and an empty jar. It doesn't wait; `polite=True` turns on Retry-After and backoff. Moving from curl_cffi takes one line: `from scrapely.http import AsyncSession`.
+
 ## Highlights
 
 - Local file-system storage and Scrapely API-backed datasets, key-value stores, and request queues
@@ -83,6 +99,8 @@ if __name__ == "__main__":
 - Automatic remote Chromium CDP wiring through `CDP_URL`
 - Actor start, call, abort, reboot, and metamorph operations
 - Ad-hoc webhooks, proxy configuration, and charging/pricing types
+- `scrapely.http`: an HTTP client with current browser fingerprints (TLS, HTTP/2, headers), identity rotation, block detection and retries; drop-in `AsyncSession` / `Session` for curl_cffi code
+- Scrapy integration in `scrapely.scrapy` (`run_scrapy_actor`, a request-queue scheduler, a dataset pipeline); install Scrapy separately
 - Legacy environment variable aliases and asynchronous client access
 
 See the [complete Python SDK guide](https://scrape.ly/docs/sdk-python) for lifecycle details, option objects, storage selection, events, CDP behavior, run control, environment variables, errors, and the full public export list.

{scrapely_python-0.1.35 → scrapely_python-0.1.38}/README.md

@@ -34,6 +34,21 @@ if __name__ == "__main__":
 
 `Actor.main()` initializes the Actor, awaits your callback, closes events and storage, and exits the process. For embedded applications and tests, use `async with Actor:` instead.
 
+## HTTP client
+
+```python
+from scrapely import Actor
+from scrapely.http import AsyncClient
+
+async with Actor:
+    proxy = await Actor.create_proxy_configuration(groups=["RESIDENTIAL"], country_code="DE")
+    async with AsyncClient(proxy_configuration=proxy) as http:
+        r = await http.get("https://www.example.com/")
+        data = (await http.get("https://www.example.com/api/items", kind="json")).json()
+```
+
+Every request uses a current browser profile: TLS, HTTP/2, User-Agent, client hints and header order all come from it. Each request runs on an identity, which is a profile, a proxy session and a cookie jar. When a site blocks an identity (429, block page, captcha), the client retires it and retries the request at once on a fresh one, with a new IP, a new profile and an empty jar. It doesn't wait; `polite=True` turns on Retry-After and backoff. Moving from curl_cffi takes one line: `from scrapely.http import AsyncSession`.
+
 ## Highlights
 
 - Local file-system storage and Scrapely API-backed datasets, key-value stores, and request queues
@@ -42,6 +57,8 @@ if __name__ == "__main__":
 - Automatic remote Chromium CDP wiring through `CDP_URL`
 - Actor start, call, abort, reboot, and metamorph operations
 - Ad-hoc webhooks, proxy configuration, and charging/pricing types
+- `scrapely.http`: an HTTP client with current browser fingerprints (TLS, HTTP/2, headers), identity rotation, block detection and retries; drop-in `AsyncSession` / `Session` for curl_cffi code
+- Scrapy integration in `scrapely.scrapy` (`run_scrapy_actor`, a request-queue scheduler, a dataset pipeline); install Scrapy separately
 - Legacy environment variable aliases and asynchronous client access
 
 See the [complete Python SDK guide](https://scrape.ly/docs/sdk-python) for lifecycle details, option objects, storage selection, events, CDP behavior, run control, environment variables, errors, and the full public export list.

{scrapely_python-0.1.35 → scrapely_python-0.1.38}/pyproject.toml

@@ -4,13 +4,13 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "scrapely-python"
-version = "0.1.35"
+version = "0.1.38"
 description = "Scrapely Python SDK - A toolkit for building Actors on the Scrapely platform"
 readme = "README.md"
 license = "MIT AND Apache-2.0"
 license-files = ["LICENSE", "LICENSE-APACHE", "NOTICE"]
 authors = [
-    {name = "Scrapely", email = "support@scrape.ly"}
+    {name = "Scrapely", email = "hello@scrape.ly"}
 ]
 classifiers = [
     "Development Status :: 3 - Alpha",
@@ -31,6 +31,7 @@ dependencies = [
     "scrapely-client-python>=1.0.2,<2",
     "pydantic>=2.0.0",
     "cryptography>=42.0.0",
+    "wreq>=0.13,<0.14; python_version>='3.11'",
 ]
 
 [project.optional-dependencies]
@@ -56,7 +57,7 @@ Changelog = "https://github.com/scrapely-app/scrapely-python/blob/main/CHANGELOG
 where = ["src"]
 
 [tool.setuptools.package-data]
-scrapely = ["py.typed"]
+scrapely = ["py.typed", "http/profiles.json"]
 
 [tool.black]
 line-length = 100
@@ -84,4 +85,6 @@ strict_equality = true
 
 [tool.pytest.ini_options]
 asyncio_mode = "auto"
-testpaths = ["tests"]
+testpaths = ["tests"]
+markers = ["network: needs internet access (run with -m network)"]
+addopts = "-m 'not network'"

{scrapely_python-0.1.35 → scrapely_python-0.1.38}/src/scrapely/__init__.py

@@ -227,7 +227,7 @@ if TYPE_CHECKING:  # Static analysers and IDEs see the lazy names as plain impor
         set_api_storage,
     )
 
-__version__ = '0.1.35'
+__version__ = '0.1.38'
 __all__ = [
     # Core
     'Actor',

scrapely_python-0.1.38/src/scrapely/http/__init__.py

@@ -0,0 +1,57 @@
+"""scrapely.http: an HTTP client that looks like a real browser and handles blocks.
+
+Every request goes out with a current browser fingerprint whose TLS, HTTP/2,
+User-Agent, client hints and header order all come from one backend profile.
+Requests use identities (profile + proxy session + cookie jar) that are
+retired together when a site blocks them; the request is retried on a fresh
+identity at once, without waiting: the block was tied to the old identity.
+``polite=True`` turns on Retry-After, per-host backoff and a slow mode for
+actors that must be gentle with a site.
+
+Quick start::
+
+    from scrapely import Actor
+    from scrapely.http import AsyncClient
+
+    proxy = await Actor.create_proxy_configuration(groups=["RESIDENTIAL"], country_code="DE")
+    async with AsyncClient(proxy_configuration=proxy) as http:
+        r = await http.get("https://example.com/")
+        data = (await http.get("https://example.com/api/items", kind="json")).json()
+
+Migrating from curl_cffi is one line: ``from scrapely.http import AsyncSession``.
+"""
+
+from ._client import AsyncClient, AsyncHttpSession, Client, HttpSession
+from ._compat import AsyncSession, Session
+from ._detect import detect_block
+from ._options import FingerprintOptions
+from ._policy import BackoffPolicy
+from ._profiles import BrowserProfile, BrowserSpec, NoMatchingProfile, load_table
+from ._response import Headers, HTTPError, Response
+from ._stats import HttpStats
+from ._transport import TransportError
+
+# curl_cffi / requests-style name for transport errors.
+RequestsError = TransportError
+
+__all__ = [
+    'AsyncClient',
+    'AsyncHttpSession',
+    'AsyncSession',
+    'BackoffPolicy',
+    'BrowserProfile',
+    'BrowserSpec',
+    'Client',
+    'FingerprintOptions',
+    'HTTPError',
+    'Headers',
+    'HttpSession',
+    'HttpStats',
+    'NoMatchingProfile',
+    'RequestsError',
+    'Response',
+    'Session',
+    'TransportError',
+    'detect_block',
+    'load_table',
+]

scrapely_python-0.1.38/src/scrapely/http/_capture.py

@@ -0,0 +1,97 @@
+"""Read the headers a backend profile really sends, over loopback HTTP/1.1.
+
+Used by the profile-table tests and to regenerate the ``headers`` of each
+profile in ``profiles.json`` after a backend upgrade::
+
+    python -m scrapely.http._capture > /tmp/native.json
+
+No network access: a one-shot server on 127.0.0.1 records the request.
+"""
+
+from __future__ import annotations
+
+import json
+import socket
+import sys
+import threading
+from typing import Dict, List, Tuple
+
+
+class LoopbackServer:
+    """Accepts connections on 127.0.0.1 and records each request's header lines in order."""
+
+    def __init__(self, response: bytes = b'HTTP/1.1 200 OK\r\nContent-Length: 2\r\nConnection: close\r\n\r\nok') -> None:
+        self.response = response
+        self.requests: List[str] = []
+        self._sock = socket.socket()
+        self._sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+        self._sock.bind(('127.0.0.1', 0))
+        self._sock.listen(64)
+        self.port = self._sock.getsockname()[1]
+        threading.Thread(target=self._loop, daemon=True).start()
+
+    @property
+    def url(self) -> str:
+        return f'http://127.0.0.1:{self.port}'
+
+    def _loop(self) -> None:
+        while True:
+            try:
+                conn, _ = self._sock.accept()
+            except OSError:
+                return
+            threading.Thread(target=self._one, args=(conn,), daemon=True).start()
+
+    def _one(self, conn: socket.socket) -> None:
+        data = b''
+        try:
+            while b'\r\n\r\n' not in data:
+                chunk = conn.recv(65536)
+                if not chunk:
+                    break
+                data += chunk
+            self.requests.append(data.decode('latin-1'))
+            conn.sendall(self.response)
+        finally:
+            conn.close()
+
+    def close(self) -> None:
+        self._sock.close()
+
+
+def parse_headers(raw: str) -> List[Tuple[str, str]]:
+    head = raw.split('\r\n\r\n', 1)[0]
+    out = []
+    for line in head.split('\r\n')[1:]:
+        if ':' in line:
+            k, v = line.split(':', 1)
+            out.append((k.strip().lower(), v.strip()))
+    return out
+
+
+def native_headers(wreq_profile: str, wreq_platform: str) -> List[Tuple[str, str]]:
+    """Header (name, value) pairs wreq sends for a profile, in its native order (host excluded)."""
+    import wreq
+    from wreq.blocking import Client
+
+    srv = LoopbackServer()
+    try:
+        emu = wreq.Emulation(profile=getattr(wreq.Profile, wreq_profile), platform=getattr(wreq.Platform, wreq_platform))
+        Client(emulation=emu).get(srv.url + '/').text()
+        return [(k, v) for k, v in parse_headers(srv.requests[-1]) if k != 'host']
+    finally:
+        srv.close()
+
+
+def main() -> None:
+    from ._profiles import load_table
+
+    out: Dict[str, Dict[str, str]] = {}
+    for p in load_table().profiles:
+        out[p.id] = dict(native_headers(p.wreq_profile, p.wreq_platform))
+    json.dump(out, sys.stdout, indent=2)
+    sys.stdout.write('\n')
+
+
+if __name__ == '__main__':
+    main()