scrapely-python 0.1.41__tar.gz → 0.1.43__tar.gz

{scrapely_python-0.1.41 → scrapely_python-0.1.43}/CHANGELOG.md

@@ -4,6 +4,34 @@ All notable changes to this project will be documented in this file.
 
 ## [Unreleased]
 
+## [0.1.43] - 2026-11-05
+
+- **Python actor processes start without loading the SDK.** The base images import `scrapely.patch_browser_pool` (from `sitecustomize`) in every Python process. Until now that loaded the whole SDK at each interpreter start: `scrapely.actor`, charging, the `scrapely_client` models and Playwright. On `actor-python:3.11-0.1.42`, `python -c pass` took 0.89-0.95 s of CPU, which is 4.2 s of wall time at a 0.25-core quota. It now takes 0.016 s. Each import below gets the same start-up cut:
+  - `import scrapely` is lazy (PEP 562). Every public name and submodule attribute resolves on first use, so every `from scrapely import X` and `scrapely.X` that worked before still works.
+  - `scrapely_client` loads only when an API client is created (`Actor.apify_client`, `create_charging_client`). `USES_PUSH_DATA_INTERCEPTION` and `push_data_charging_context` are still the client's own objects.
+- **Browser patches are applied through an import hook.** `scrapely.patch_browser_pool` imports no browser library any more; it imports builtin modules only. Each patch now runs the moment its library finishes loading, before the import that loaded it returns. That covers the Crawlee BrowserPool / PlaywrightBrowserPlugin / PlaywrightCrawler, raw Playwright (`async_playwright` and `sync_playwright`, each when its module loads), Camoufox `AsyncNewBrowser` and pyppeteer `launch`. The patches behave exactly as before. The hook also:
+  - survives `importlib.util.find_spec()` availability checks and a failed first import;
+  - logs a warning when a patch fails instead of breaking the actor's import.
+- **Start-up measured on staging** (fresh builds of the same commit on the same base, old SDK vs 0.1.43; "boot" = container started to the SDK's "Initializing API storage" line; paired median): web-scraper -0.74 s, capterra-scraper (raw Playwright over the remote browser) -0.89 s, contact-info-scraper (512 MB) -0.51 s. All 88 scenario runs passed on both sides. Locally at 1 CPU, a full `from apify import Actor` + `Actor.init` + `push_data` + exit went from 1.19-1.36 s to 0.92-1.02 s of CPU.
+- Base images (released with this version): `sitecustomize` no longer imports `pathlib` (~70 ms per start), and the `apify` shim resolves names on first use instead of `from scrapely import *`.
+
+## [0.1.42] - 2026-11-02
+
+- **Sessions that keep their identity: `http.session(keep_identity=True)`.** A pinned `scrapely.http` session used to retire its identity on the first block (a 403, a challenge page, a network error). The session then moved to a new exit, a new profile and an empty cookie jar. That breaks every challenge solve: the challenge page, the solver's script and submission requests and the page fetched again must all share one identity. With `keep_identity=True`:
+  - a block comes back to the caller (`response.blocked` names it) and an error is raised, with no rotation;
+  - the next request goes out on the same profile, exit and jar, with the cookies the block set;
+  - `max_usage` / `max_age` do not retire it;
+  - `session.rotate()` moves the session to a fresh identity when you decide to.
+
+  The default (`keep_identity=False`) is unchanged. The option is available on `AsyncClient.session()` and `Client.session()`.
+- **`scrapely.solver.AkamaiSensor`: Akamai Bot Manager on your own session, the way the solver provider's examples do it.**
+  - `solve_sbsd(session, url, html)` handles a page that loads an SBSD bundle. It fetches the bundle on your session, gets the ledger from the solver (`/akamai/sbsd/generate-session`), then POSTs the ledger rows to the bundle path on your session.
+  - `solve_abck(session, url, html)` runs the `_abck` sensor over the solver's WebSocket session (`/akamai/session`, which the platform relays for runs). The solver generates each sensor; it is POSTed on your session with your client's own headers, and rounds continue until `_abck` is accepted (`~0~`).
+  - `solve()` does SBSD when the page carries a bundle, then `_abck`.
+
+  Use it with a `keep_identity=True` session (or one curl_cffi or httpx client), and set `profile_id` to the browser your client claims, e.g. `chrome-154-macos` for scrapely.http Chrome 154 on macOS. Submissions outside the target site are refused, and a refused session raises the platform's code (`VendorUnavailable`, `SolverRateLimited`, ...). Also new: `find_sensor_script()`, `find_sbsd_bundle()`, `abck_state()`, `SensorResult`.
+- **`AkamaiSensor.solve_pixel(session, url, html)`: Akamai's third channel, the pixel collector.** A page that loads `/akam/<n>/<hex>` (often next to an inline `bazadebezolkohpepadr=` value) expects that script to POST to `/akam/<n>/pixel_<hex>`. The collector is fetched on your session, the solver writes the POST the collector would send (`/akamai/pixel/generate`), and it is POSTed to the collector's own endpoint on your session. Also new: `find_pixel_collector()`, `pixel_target()`.
+
 ## [0.1.41] - 2026-10-29
 
 - **A refused charge names the limit that refused it.** When a charge is refused or cut short, the platform now says why (HTTP 402 `error.reason`, and `limitedBy` on a partial acceptance): the run's item limit (`maxItems`), its maximum total charge (`maxTotalChargeUsd`) or the account balance. The SDK used to call all of them the spending limit; a run stopped by its item limit logged "The spending limit of this run (maxTotalChargeUsd = $3.0) has been reached". Now:

{scrapely_python-0.1.41 → scrapely_python-0.1.43}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: scrapely-python
-Version: 0.1.41
+Version: 0.1.43
 Summary: Scrapely Python SDK - A toolkit for building Actors on the Scrapely platform
 Author-email: Scrapely <hello@scrape.ly>
 License-Expression: MIT AND Apache-2.0

{scrapely_python-0.1.41 → scrapely_python-0.1.43}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "scrapely-python"
-version = "0.1.41"
+version = "0.1.43"
 description = "Scrapely Python SDK - A toolkit for building Actors on the Scrapely platform"
 readme = "README.md"
 license = "MIT AND Apache-2.0"

{scrapely_python-0.1.41 → scrapely_python-0.1.43}/src/scrapely/__init__.py

@@ -10,95 +10,114 @@ Usage:
             await Actor.push_data({'url': actor_input['url']})
 """
 
-from typing import TYPE_CHECKING, Any
+from __future__ import annotations
+
+# ``typing`` is not imported at run time (see below): this package loads at
+# every interpreter start in actor images.
+TYPE_CHECKING = False
+if TYPE_CHECKING:
+    from typing import Any
 
 # Side-effect: patch browser launches (Crawlee BrowserPool, PlaywrightCrawler,
 # raw Playwright, Camoufox, pyppeteer) to use the remote CDP browser (CDP_URL).
 # Must run on package import so `from scrapely import Actor` enables the patch
-# before any browser is launched (Node SDK parity). It does not import Crawlee:
-# the Crawlee patches are applied when Crawlee's browser modules load.
+# before any browser is launched (Node SDK parity). It imports none of those
+# libraries: each patch is applied the moment its library finishes loading.
 from .patch_browser_pool import set_cdp_url  # noqa: F401
 
-# Crawlee is NOT imported here: it costs ~1 s per run on the platform and many
-# actors never touch it. Everything that needs Crawlee (the re-exported Crawlee
-# classes, storage clients, events, request loaders) is resolved on first
-# attribute access by ``__getattr__`` below; ``Actor`` imports what it needs on
-# first use. Every ``from scrapely import X`` that worked before still works.
-from .actor import Actor
-from .configuration import Configuration
-from .charging_client import (
-    USES_PUSH_DATA_INTERCEPTION,
-    create_charging_client,
-    push_data_charging_context,
-)
-from .proxy_configuration import ProxyConfiguration
-from .browser_configuration import BrowserConfiguration
-from .charging import (
-    ChargingManager,
-    ChargeUncertainError,
-    ChargeRejectedError,
-    has_default_dataset_item_event_in_pricing,
-    is_default_dataset_item_event,
-    merge_charge_results,
-    push_data_and_charge,
-    resolve_default_dataset_item_event_name,
-    APIFY_DEFAULT_DATASET_ITEM_EVENT,
-    SCRAPELY_DEFAULT_DATASET_ITEM_EVENT,
-    DEFAULT_DATASET_ITEM_EVENT,
-)
-
-from .types import (
-    ActorInput,
-    ActorPermissionLevel,
-    ActorRun,
-    ApifyEnv,
-    AbortOptions,
-    CallOptions,
-    ExitOptions,
-    InitOptions,
-    MetamorphOptions,
-    OpenStorageOptions,
-    ProxyInfo,
-    RebootOptions,
-    RecordOptions,
-    SetStatusMessageOptions,
-    StartOptions,
-    WebhookOptions,
-)
+# Nothing else is imported here. The base images' sitecustomize imports
+# ``scrapely.patch_browser_pool`` (and so this package) in EVERY Python process
+# of an actor image; loading the whole SDK there cost ~0.75 s per interpreter
+# start at 1 CPU (5.7 s at 0.25 CPU). Every public name is resolved on first
+# attribute access by ``__getattr__`` below and cached in the module namespace,
+# so every ``from scrapely import X`` that worked before still works.
 
-from .webhook import (
-    Webhook,
-    WebhookEventType,
-)
-from .pricing_types import (
-    PricingModel,
-    FreeActorPricingInfo,
-    FlatPricePerMonthActorPricingInfo,
-    PricePerDatasetItemActorPricingInfo,
-    PayPerEventActorPricingInfo,
-    PricingPerEvent,
-    ActorPricingInfo,
-    ActorChargeEvent,
-    ChargeResult,
-    ActorPricingInfoResult,
-)
-from .errors import (
-    ScrapelyClientError,
-    ScrapelyApiError,
-    InvalidResponseBodyError,
-    InvalidRequestError,
-    UnauthorizedError,
-    ForbiddenError,
-    NotFoundError,
-    ConflictError,
-    RateLimitError,
-    ServerError,
-    raise_for_status,
-)
-
-# Public name -> (module, attribute) for names that need Crawlee. Loaded on
-# first access and cached in the module namespace.
+# Public name -> (module, attribute). Loaded on first access.
 _LAZY_ATTRS = {
+    'Actor': ('scrapely.actor', 'Actor'),
+    'Configuration': ('scrapely.configuration', 'Configuration'),
+    'ProxyConfiguration': ('scrapely.proxy_configuration', 'ProxyConfiguration'),
+    'BrowserConfiguration': ('scrapely.browser_configuration', 'BrowserConfiguration'),
+    # Charging client
+    **{
+        name: ('scrapely.charging_client', name)
+        for name in ('USES_PUSH_DATA_INTERCEPTION', 'create_charging_client', 'push_data_charging_context')
+    },
+    # Charging
+    **{
+        name: ('scrapely.charging', name)
+        for name in (
+            'ChargingManager',
+            'ChargeUncertainError',
+            'ChargeRejectedError',
+            'has_default_dataset_item_event_in_pricing',
+            'is_default_dataset_item_event',
+            'merge_charge_results',
+            'push_data_and_charge',
+            'resolve_default_dataset_item_event_name',
+            'APIFY_DEFAULT_DATASET_ITEM_EVENT',
+            'SCRAPELY_DEFAULT_DATASET_ITEM_EVENT',
+            'DEFAULT_DATASET_ITEM_EVENT',
+        )
+    },
+    # Types
+    **{
+        name: ('scrapely.types', name)
+        for name in (
+            'ActorInput',
+            'ActorPermissionLevel',
+            'ActorRun',
+            'ApifyEnv',
+            'AbortOptions',
+            'CallOptions',
+            'ExitOptions',
+            'InitOptions',
+            'MetamorphOptions',
+            'OpenStorageOptions',
+            'ProxyInfo',
+            'RebootOptions',
+            'RecordOptions',
+            'SetStatusMessageOptions',
+            'StartOptions',
+            'WebhookOptions',
+        )
+    },
+    # Webhook
+    'Webhook': ('scrapely.webhook', 'Webhook'),
+    'WebhookEventType': ('scrapely.webhook', 'WebhookEventType'),
+    # Pricing
+    **{
+        name: ('scrapely.pricing_types', name)
+        for name in (
+            'PricingModel',
+            'FreeActorPricingInfo',
+            'FlatPricePerMonthActorPricingInfo',
+            'PricePerDatasetItemActorPricingInfo',
+            'PayPerEventActorPricingInfo',
+            'PricingPerEvent',
+            'ActorPricingInfo',
+            'ActorChargeEvent',
+            'ChargeResult',
+            'ActorPricingInfoResult',
+        )
+    },
+    # Errors
+    **{
+        name: ('scrapely.errors', name)
+        for name in (
+            'ScrapelyClientError',
+            'ScrapelyApiError',
+            'InvalidResponseBodyError',
+            'InvalidRequestError',
+            'UnauthorizedError',
+            'ForbiddenError',
+            'NotFoundError',
+            'ConflictError',
+            'RateLimitError',
+            'ServerError',
+            'raise_for_status',
+        )
+    },
     # Crawlee re-exports
     'Request': ('crawlee', 'Request'),
     'RequestLoader': ('crawlee.request_loaders', 'RequestLoader'),
@@ -150,15 +169,28 @@ _LAZY_ATTRS = {
 }
 
 # Submodules that used to be loaded (and so bound as attributes) by this
-# package's eager imports; ``scrapely.events`` etc. keep working without an
-# explicit ``import scrapely.events``.
+# package's eager imports; ``scrapely.actor``, ``scrapely.events`` etc. keep
+# working without an explicit ``import scrapely.<name>``.
 _LAZY_SUBMODULES = frozenset({
+    '_browser_locations',
+    'actor',
+    'browser_configuration',
+    'cdp_retry',
+    'charging',
+    'charging_client',
+    'configuration',
+    'crypto',
+    'errors',
     'events',
     'log',
+    'pricing_types',
+    'proxy_configuration',
     'request_loaders',
     'storage',
     'storage_clients',
     'storages',
+    'types',
+    'webhook',
 })
 
 
@@ -193,6 +225,40 @@ if TYPE_CHECKING:  # Static analysers and IDEs see the lazy names as plain impor
     from crawlee.storage_clients.models import DatasetItemsListPage
     from crawlee.storages import Dataset, KeyValueStore, RequestQueue
 
+    from .actor import Actor
+    from .browser_configuration import BrowserConfiguration
+    from .charging import (
+        APIFY_DEFAULT_DATASET_ITEM_EVENT,
+        DEFAULT_DATASET_ITEM_EVENT,
+        SCRAPELY_DEFAULT_DATASET_ITEM_EVENT,
+        ChargeRejectedError,
+        ChargeUncertainError,
+        ChargingManager,
+        has_default_dataset_item_event_in_pricing,
+        is_default_dataset_item_event,
+        merge_charge_results,
+        push_data_and_charge,
+        resolve_default_dataset_item_event_name,
+    )
+    from .charging_client import (
+        USES_PUSH_DATA_INTERCEPTION,
+        create_charging_client,
+        push_data_charging_context,
+    )
+    from .configuration import Configuration
+    from .errors import (
+        ConflictError,
+        ForbiddenError,
+        InvalidRequestError,
+        InvalidResponseBodyError,
+        NotFoundError,
+        RateLimitError,
+        ScrapelyApiError,
+        ScrapelyClientError,
+        ServerError,
+        UnauthorizedError,
+        raise_for_status,
+    )
     from .events import (
         AbortingEvent,
         ActorEventTypes,
@@ -216,6 +282,19 @@ if TYPE_CHECKING:  # Static analysers and IDEs see the lazy names as plain impor
         SystemInfoEventData,
         UnknownEvent,
     )
+    from .pricing_types import (
+        ActorChargeEvent,
+        ActorPricingInfo,
+        ActorPricingInfoResult,
+        ChargeResult,
+        FlatPricePerMonthActorPricingInfo,
+        FreeActorPricingInfo,
+        PayPerEventActorPricingInfo,
+        PricePerDatasetItemActorPricingInfo,
+        PricingModel,
+        PricingPerEvent,
+    )
+    from .proxy_configuration import ProxyConfiguration
     from .request_loaders import ScrapelyRequestList
     from .storage import (
         ApiStorageClient,
@@ -226,8 +305,27 @@ if TYPE_CHECKING:  # Static analysers and IDEs see the lazy names as plain impor
         StorageClient,
         set_api_storage,
     )
+    from .types import (
+        AbortOptions,
+        ActorInput,
+        ActorPermissionLevel,
+        ActorRun,
+        ApifyEnv,
+        CallOptions,
+        ExitOptions,
+        InitOptions,
+        MetamorphOptions,
+        OpenStorageOptions,
+        ProxyInfo,
+        RebootOptions,
+        RecordOptions,
+        SetStatusMessageOptions,
+        StartOptions,
+        WebhookOptions,
+    )
+    from .webhook import Webhook, WebhookEventType
 
-__version__ = '0.1.41'
+__version__ = '0.1.43'
 __all__ = [
     # Core
     'Actor',

{scrapely_python-0.1.41 → scrapely_python-0.1.43}/src/scrapely/charging_client.py

@@ -11,15 +11,29 @@ from __future__ import annotations
 
 from typing import TYPE_CHECKING, Any, Callable, Dict, Optional, Sequence
 
-from scrapely_client import ApifyClientAsync
+if TYPE_CHECKING:
+    from scrapely_client import USES_PUSH_DATA_INTERCEPTION, ApifyClientAsync, push_data_charging_context
+
+    from .charging import ChargingManager
 
+# ``scrapely_client`` (219 pydantic models) loads only when a client is created
+# or one of its names below is used: ``Actor`` imports this module, and most
+# runs never create an API client.
+#
 # Re-exported so the SDK and the client share one marker attribute and one
 # context var. Identity matters: the client's intercepted ``push_items()``
 # reads/writes these, and the SDK detects interception via the attribute.
-from scrapely_client import USES_PUSH_DATA_INTERCEPTION, push_data_charging_context
+_CLIENT_NAMES = frozenset({'ApifyClientAsync', 'USES_PUSH_DATA_INTERCEPTION', 'push_data_charging_context'})
 
-if TYPE_CHECKING:
-    from .charging import ChargingManager
+
+def __getattr__(name: str) -> Any:
+    if name in _CLIENT_NAMES:
+        import scrapely_client
+
+        value = getattr(scrapely_client, name)
+        globals()[name] = value
+        return value
+    raise AttributeError(f'module {__name__!r} has no attribute {name!r}')
 
 __all__ = [
     'USES_PUSH_DATA_INTERCEPTION',
@@ -81,7 +95,7 @@ def create_charging_client(
     charging_manager: Optional['ChargingManager'] = None,
     default_dataset_id: Optional[str] = None,
     **kwargs: Any,
-) -> ApifyClientAsync:
+) -> 'ApifyClientAsync':
     """Create an ``ApifyClientAsync`` configured with the SDK's charging hooks.
 
     Args:
@@ -101,6 +115,8 @@ def create_charging_client(
     if api_public_url:
         kwargs['api_public_url'] = api_public_url
 
+    from scrapely_client import ApifyClientAsync
+
     charging_hooks = _ChargingManagerHooks(charging_manager) if charging_manager is not None else None
 
     return ApifyClientAsync(

{scrapely_python-0.1.41 → scrapely_python-0.1.43}/src/scrapely/http/_client.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import asyncio
+import dataclasses
 import logging
 import random
 import threading
@@ -102,14 +103,30 @@ def _proxy_from_proxies(proxies: Any) -> Optional[str]:
 
 
 class _Pin:
-    """Holder of a pinned identity (a session)."""
+    """Holder of a pinned identity (a session).
 
-    __slots__ = ('identity', 'alock', 'tlock')
+    ``keep``: the identity is never retired or swapped by the client (no
+    rotation on a block or error, no usage/age expiry); only ``rotate()`` or
+    closing the session moves it to a new one.
+    """
+
+    __slots__ = ('identity', 'alock', 'tlock', 'keep')
 
-    def __init__(self) -> None:
+    def __init__(self, keep: bool = False) -> None:
         self.identity: Optional[Identity] = None
         self.alock: Optional[asyncio.Lock] = None
         self.tlock = threading.Lock()
+        self.keep = keep
+
+
+def _no_rotation(decision: Any) -> Any:
+    """The policy's decision with the rotation taken out (kept identities): stop here, retire nothing."""
+    try:
+        return dataclasses.replace(decision, retry=False, retire=False, delay=0.0)
+    except TypeError:
+        decision.retry = False
+        decision.retire = False
+        return decision
 
 
 class CookieStore:
@@ -522,7 +539,7 @@ class AsyncClient(_Base):
                 pin.alock = asyncio.Lock()
             async with pin.alock:
                 ident = pin.identity
-                if ident is not None and ident.expired(self._clock(), self.max_usage, self.max_age):
+                if ident is not None and not pin.keep and ident.expired(self._clock(), self.max_usage, self.max_age):
                     pin.identity = None
                     await self._retire(ident)
                     ident = None
@@ -558,6 +575,8 @@ class AsyncClient(_Base):
     async def _release(self, pin: Optional[_Pin], ident: Identity, retire: bool) -> None:
         ident.in_flight -= 1
         if pin is not None:
+            if pin.keep and pin.identity is ident:
+                return
             if retire or not self.sticky:
                 if pin.identity is ident:
                     pin.identity = None
@@ -658,12 +677,16 @@ class AsyncClient(_Base):
                 # Cancelled or failed in our own code: nothing says the identity is burnt.
                 await self._release(_pin, ident, retire=False)
                 raise
+            if decision.retry and not decision.same_identity and _pin is not None and _pin.keep:
+                # A kept identity is never swapped: the block or error goes back to the caller as is.
+                decision = _no_rotation(decision)
             if not decision.retry:
                 await self._release(_pin, ident, retire=decision.retire)
                 ok = resp is not None and reason is None and resp.status_code < 500
                 self._stats.record_request(ok)
                 if resp is None:
-                    assert error is not None
+                    if error is None:
+                        raise RuntimeError('scrapely.http: request ended with no response and no error')
                     raise error
                 resp.blocked = reason
                 return resp
@@ -703,14 +726,21 @@ class AsyncClient(_Base):
     async def options(self, url: str, **kwargs: Any) -> Response:
         return await self.request('OPTIONS', url, **kwargs)
 
-    def session(self) -> 'AsyncHttpSession':
+    def session(self, keep_identity: bool = False) -> 'AsyncHttpSession':
         """A pinned identity for multi-step flows (homepage → API with its cookies).
 
         All requests of the session use one profile, proxy session and cookie
         jar. On a block the whole identity is retired and the session continues
         on a fresh one (new IP, new profile, empty jar).
+
+        ``keep_identity=True`` keeps the identity and its jar on a block or an
+        error too: the blocked response comes back (``response.blocked`` names
+        the block) and the next request goes out on the same profile, exit and
+        cookies. That is what a challenge solve needs (the SBSD bundle, the
+        sensor submissions and the page fetches must share one identity).
+        ``rotate()`` moves the session to a fresh identity when you decide to.
         """
-        return AsyncHttpSession(self)
+        return AsyncHttpSession(self, keep_identity=keep_identity)
 
     async def aclose(self) -> None:
         if self._closed:
@@ -733,11 +763,21 @@ class AsyncClient(_Base):
 class AsyncHttpSession:
     """A pinned identity on an ``AsyncClient`` (see ``AsyncClient.session()``)."""
 
-    def __init__(self, client: AsyncClient) -> None:
+    def __init__(self, client: AsyncClient, keep_identity: bool = False) -> None:
         self._client = client
-        self._pin = _Pin()
+        self._pin = _Pin(keep=keep_identity)
         self.cookies = _SessionCookies(self)
 
+    @property
+    def keep_identity(self) -> bool:
+        return self._pin.keep
+
+    async def rotate(self) -> None:
+        """Retire the current identity; the next request opens a fresh one (new exit, profile and jar)."""
+        if self._pin.identity is not None:
+            ident, self._pin.identity = self._pin.identity, None
+            await self._client._retire(ident, force=ident.in_flight <= 0)
+
     @property
     def profile(self) -> Optional[str]:
         return self._pin.identity.profile.label if self._pin.identity else None
@@ -873,7 +913,7 @@ class Client(_Base):
         if pin is not None:
             with pin.tlock:
                 ident = pin.identity
-                if ident is not None and ident.expired(self._clock(), self.max_usage, self.max_age):
+                if ident is not None and not pin.keep and ident.expired(self._clock(), self.max_usage, self.max_age):
                     pin.identity = None
                     self._retire(ident)
                     ident = None
@@ -918,6 +958,8 @@ class Client(_Base):
         with self._lock:
             ident.in_flight -= 1
         if pin is not None:
+            if pin.keep and pin.identity is ident:
+                return
             if retire or not self.sticky:
                 with pin.tlock:
                     if pin.identity is ident:
@@ -1023,12 +1065,15 @@ class Client(_Base):
                 # Failed in our own code: nothing says the identity is burnt.
                 self._release(_pin, ident, retire=False)
                 raise
+            if decision.retry and not decision.same_identity and _pin is not None and _pin.keep:
+                decision = _no_rotation(decision)
             if not decision.retry:
                 self._release(_pin, ident, retire=decision.retire)
                 ok = resp is not None and reason is None and resp.status_code < 500
                 self._stats.record_request(ok)
                 if resp is None:
-                    assert error is not None
+                    if error is None:
+                        raise RuntimeError('scrapely.http: request ended with no response and no error')
                     raise error
                 resp.blocked = reason
                 return resp
@@ -1067,9 +1112,9 @@ class Client(_Base):
     def options(self, url: str, **kwargs: Any) -> Response:
         return self.request('OPTIONS', url, **kwargs)
 
-    def session(self) -> 'HttpSession':
+    def session(self, keep_identity: bool = False) -> 'HttpSession':
         """A pinned identity for multi-step flows (see ``AsyncClient.session``)."""
-        return HttpSession(self)
+        return HttpSession(self, keep_identity=keep_identity)
 
     def close(self) -> None:
         if self._closed:
@@ -1090,11 +1135,22 @@ class Client(_Base):
 class HttpSession:
     """A pinned identity on a sync ``Client`` (see ``Client.session()``)."""
 
-    def __init__(self, client: Client) -> None:
+    def __init__(self, client: Client, keep_identity: bool = False) -> None:
         self._client = client
-        self._pin = _Pin()
+        self._pin = _Pin(keep=keep_identity)
         self.cookies = _SessionCookies(self)
 
+    @property
+    def keep_identity(self) -> bool:
+        return self._pin.keep
+
+    def rotate(self) -> None:
+        """Retire the current identity; the next request opens a fresh one (new exit, profile and jar)."""
+        with self._pin.tlock:
+            ident, self._pin.identity = self._pin.identity, None
+        if ident is not None:
+            self._client._retire(ident, force=ident.in_flight <= 0)
+
     @property
     def profile(self) -> Optional[str]:
         return self._pin.identity.profile.label if self._pin.identity else None