apify 2.4.0b5__tar.gz → 2.5.0__tar.gz

{apify-2.4.0b5 → apify-2.5.0}/.github/workflows/run_code_checks.yaml

@@ -26,5 +26,5 @@ jobs:
   integration_tests:
     name: Integration tests
     needs: [lint_check, type_check, unit_tests]
-    uses: apify/workflows/.github/workflows/python_integration_tests.yaml@main
+    uses: apify/workflows/.github/workflows/python_integration_tests.yaml@fix-integration-tests-from-forks
     secrets: inherit

{apify-2.4.0b5 → apify-2.5.0}/CHANGELOG.md

@@ -2,6 +2,17 @@
 
 All notable changes to this project will be documented in this file.
 
+## [2.5.0](https://github.com/apify/apify-sdk-python/releases/tag/v2.5.0) (2025-03-27)
+
+### 🚀 Features
+
+- Implement Scrapy HTTP cache backend  ([#403](https://github.com/apify/apify-sdk-python/pull/403)) ([137e3c8](https://github.com/apify/apify-sdk-python/commit/137e3c8d5c6b28cf6935cfb742b5f072cd2e0a02)) by [@honzajavorek](https://github.com/honzajavorek)
+
+### 🐛 Bug Fixes
+
+- Fix calculation of CPU utilization from SystemInfo events ([#447](https://github.com/apify/apify-sdk-python/pull/447)) ([eb4c8e4](https://github.com/apify/apify-sdk-python/commit/eb4c8e4e498e23f573b9e2d4c7dbd8e2ecc277d9)) by [@janbuchar](https://github.com/janbuchar)
+
+
 ## [2.4.0](https://github.com/apify/apify-sdk-python/releases/tag/v2.4.0) (2025-03-07)
 
 ### 🚀 Features

{apify-2.4.0b5 → apify-2.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: apify
-Version: 2.4.0b5
+Version: 2.5.0
 Summary: Apify SDK for Python
 Project-URL: Homepage, https://docs.apify.com/sdk/python/
 Project-URL: Apify homepage, https://apify.com

{apify-2.4.0b5 → apify-2.5.0}/docs/02_guides/05_scrapy.mdx

@@ -40,6 +40,7 @@ The Apify SDK provides several custom components to support integration with the
 - [`apify.scrapy.ApifyScheduler`](https://docs.apify.com/sdk/python/reference/class/ApifyScheduler) - Replaces Scrapy's default [scheduler](https://docs.scrapy.org/en/latest/topics/scheduler.html) with one that uses Apify's [request queue](https://docs.apify.com/platform/storage/request-queue) for storing requests. It manages enqueuing, dequeuing, and maintaining the state and priority of requests.
 - [`apify.scrapy.ActorDatasetPushPipeline`](https://docs.apify.com/sdk/python/reference/class/ActorDatasetPushPipeline) - A Scrapy [item pipeline](https://docs.scrapy.org/en/latest/topics/item-pipeline.html) that pushes scraped items to Apify's [dataset](https://docs.apify.com/platform/storage/dataset). When enabled, every item produced by the spider is sent to the dataset.
 - [`apify.scrapy.ApifyHttpProxyMiddleware`](https://docs.apify.com/sdk/python/reference/class/ApifyHttpProxyMiddleware) - A Scrapy [middleware](https://docs.scrapy.org/en/latest/topics/downloader-middleware.html) that manages proxy configurations. This middleware replaces Scrapy's default `HttpProxyMiddleware` to facilitate the use of Apify's proxy service.
+- [`apify.scrapy.extensions.ApifyCacheStorage`](https://docs.apify.com/sdk/python/reference/class/ApifyCacheStorage) - A storage backend for Scrapy's built-in [HTTP cache middleware](https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#module-scrapy.downloadermiddlewares.httpcache). This backend uses Apify's [key-value store](https://docs.apify.com/platform/storage/key-value-store). Make sure to set `HTTPCACHE_ENABLED` and `HTTPCACHE_EXPIRATION_SECS` in your settings, or caching won't work.
 
 Additional helper functions in the [`apify.scrapy`](https://github.com/apify/apify-sdk-python/tree/master/src/apify/scrapy) subpackage include:
 
@@ -94,6 +95,12 @@ The following example demonstrates a Scrapy Actor that scrapes page titles and e
     </TabItem>
 </Tabs>
 
+## Dealing with ‘imminent migration to another host’
+
+Under some circumstances, the platform may decide to [migrate your Actor](https://docs.apify.com/academy/expert-scraping-with-apify/migrations-maintaining-state) from one piece of infrastructure to another while it's in progress. While [Crawlee](https://crawlee.dev/python)-based projects can pause and resume their work after a restart, achieving the same with a Scrapy-based project can be challenging.
+
+As a workaround for this issue (tracked as [apify/actor-templates#303](https://github.com/apify/actor-templates/issues/303)), turn on caching with `HTTPCACHE_ENABLED` and set `HTTPCACHE_EXPIRATION_SECS` to at least a few minutes—the exact value depends on your use case. If your Actor gets migrated and restarted, the subsequent run will hit the cache, making it fast and avoiding unnecessary resource consumption.
+
 ## Conclusion
 
 In this guide you learned how to use Scrapy in Apify Actors. You can now start building your own web scraping projects using Scrapy, the Apify SDK and host them on the Apify platform. See the [Actor templates](https://apify.com/templates/categories/python) to get started with your own scraping tasks. If you have questions or need assistance, feel free to reach out on our [GitHub](https://github.com/apify/apify-sdk-python) or join our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping!

{apify-2.4.0b5 → apify-2.5.0}/docs/02_guides/code/scrapy_project/src/settings.py

@@ -7,3 +7,5 @@ SPIDER_MODULES = ['src.spiders']
 TELNETCONSOLE_ENABLED = False
 # Do not change the Twisted reactor unless you really know what you are doing.
 TWISTED_REACTOR = 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'
+HTTPCACHE_ENABLED = True
+HTTPCACHE_EXPIRATION_SECS = 7200

{apify-2.4.0b5 → apify-2.5.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "apify"
-version = "2.4.0b5"
+version = "2.5.0"
 description = "Apify SDK for Python"
 authors = [{ name = "Apify Technologies s.r.o.", email = "support@apify.com" }]
 license = { file = "LICENSE" }
@@ -59,20 +59,18 @@ scrapy = ["scrapy>=2.11.0"]
 [dependency-groups]
 dev = [
     "build~=1.2.0",
-    "filelock~=3.17.0",
     "griffe~=1.6.0",
     "mypy~=1.15.0",
-    "pre-commit~=4.1.0",
+    "pre-commit~=4.2.0",
     "pydoc-markdown~=4.8.0",
-    "pytest~=8.3.0",
-    "pytest-asyncio~=0.25.0",
+    "pytest-asyncio~=0.26.0",
     "pytest-cov~=6.0.0",
     "pytest-only~=2.1.0",
-    "pytest-timeout~=2.3.0",
     "pytest-xdist~=3.6.0",
+    "pytest~=8.3.0",
     "respx~=0.22.0",
-    "ruff~=0.9.0",
-    "setuptools~=76.0.0",     # setuptools are used by pytest but not explicitly required
+    "ruff~=0.11.0",
+    "setuptools~=78.1.0",     # setuptools are used by pytest but not explicitly required
 ]
 
 [tool.hatch.build.targets.wheel]
@@ -93,6 +91,9 @@ ignore = [
     "D100",     # Missing docstring in public module
     "D104",     # Missing docstring in public package
     "D107",     # Missing docstring in `__init__`
+    "D203",     # One blank line required before class docstring
+    "D213",     # Multi-line docstring summary should start at the second line
+    "D413",     # Missing blank line after last section
     "EM",       # flake8-errmsg
     "G004",     # Logging statement uses f-string
     "ISC001",   # This rule may cause conflicts when used with the formatter
@@ -164,9 +165,6 @@ runtime-evaluated-base-classes = [
 [tool.ruff.lint.flake8-builtins]
 builtins-ignorelist = ["id"]
 
-[tool.ruff.lint.pydocstyle]
-convention = "google"
-
 [tool.ruff.lint.isort]
 known-local-folder = ["apify"]
 known-first-party = ["apify_client", "apify_shared", "crawlee"]

{apify-2.4.0b5 → apify-2.5.0}/src/apify/_actor.py

@@ -141,7 +141,7 @@ class _ActorType:
                 await self.exit()
 
     def __repr__(self) -> str:
-        if self is cast(Proxy, Actor).__wrapped__:
+        if self is cast('Proxy', Actor).__wrapped__:
             return '<apify.Actor>'
 
         return super().__repr__()
@@ -222,7 +222,7 @@ class _ActorType:
             self.log.warning('Repeated Actor initialization detected - this is non-standard usage, proceed with care')
 
         # Make sure that the currently initialized instance is also available through the global `Actor` proxy
-        cast(Proxy, Actor).__wrapped__ = self
+        cast('Proxy', Actor).__wrapped__ = self
 
         self._is_exiting = False
         self._was_final_persist_state_emitted = False
@@ -674,7 +674,7 @@ class _ActorType:
             elif isinstance(field.validation_alias, str):
                 aliases = [field.validation_alias]
             elif isinstance(field.validation_alias, AliasChoices):
-                aliases = cast(list[str], field.validation_alias.choices)
+                aliases = cast('list[str]', field.validation_alias.choices)
             else:
                 aliases = [field_name]
 
@@ -1138,7 +1138,7 @@ class _ActorType:
         return proxy_configuration
 
     def _get_default_exit_process(self) -> bool:
-        """Returns False for IPython, Pytest, and Scrapy environments, True otherwise."""
+        """Return False for IPython, Pytest, and Scrapy environments, True otherwise."""
         if is_running_in_ipython():
             self.log.debug('Running in IPython, setting default `exit_process` to False.')
             return False
@@ -1158,5 +1158,5 @@ class _ActorType:
         return True
 
 
-Actor = cast(_ActorType, Proxy(_ActorType))
+Actor = cast('_ActorType', Proxy(_ActorType))
 """The entry point of the SDK, through which all the Actor operations should be done."""

{apify-2.4.0b5 → apify-2.5.0}/src/apify/_configuration.py

@@ -5,8 +5,8 @@ from decimal import Decimal
 from logging import getLogger
 from typing import Annotated, Any
 
-from pydantic import AliasChoices, BeforeValidator, Field
-from typing_extensions import deprecated
+from pydantic import AliasChoices, BeforeValidator, Field, model_validator
+from typing_extensions import Self, deprecated
 
 from crawlee._utils.models import timedelta_ms
 from crawlee._utils.urls import validate_http_url
@@ -365,6 +365,19 @@ class Configuration(CrawleeConfiguration):
         ),
     ] = None
 
+    @model_validator(mode='after')
+    def disable_browser_sandbox_on_platform(self) -> Self:
+        """Disable the browser sandbox mode when running on the Apify platform.
+
+        Running in environment where `is_at_home` is True does not benefit from browser sandbox as it is already running
+        in a container. It can be on the contrary undesired as the process in the container might be running as root and
+        this will crash chromium that was started with browser sandbox mode.
+        """
+        if self.is_at_home and not self.disable_browser_sandbox:
+            self.disable_browser_sandbox = True
+            logger.warning('Actor is running on the Apify platform, `disable_browser_sandbox` was changed to True.')
+        return self
+
     @classmethod
     def get_global_configuration(cls) -> Configuration:
         """Retrieve the global instance of the configuration.

{apify-2.4.0b5 → apify-2.5.0}/src/apify/_platform_event_manager.py

@@ -48,11 +48,11 @@ class SystemInfoEventData(BaseModel):
     is_cpu_overloaded: Annotated[bool, Field(alias='isCpuOverloaded')]
     created_at: Annotated[datetime, Field(alias='createdAt')]
 
-    def to_crawlee_format(self) -> EventSystemInfoData:
+    def to_crawlee_format(self, dedicated_cpus: float) -> EventSystemInfoData:
         return EventSystemInfoData.model_validate(
             {
                 'cpu_info': {
-                    'used_ratio': self.cpu_current_usage / 100,
+                    'used_ratio': (self.cpu_current_usage / 100) / dedicated_cpus,
                     'created_at': self.created_at,
                 },
                 'memory_info': {
@@ -218,7 +218,7 @@ class PlatformEventManager(EventManager):
                             event=parsed_message.name,
                             event_data=parsed_message.data
                             if not isinstance(parsed_message.data, SystemInfoEventData)
-                            else parsed_message.data.to_crawlee_format(),
+                            else parsed_message.data.to_crawlee_format(self._config.dedicated_cpus or 1),
                         )
 
                         if parsed_message.name == Event.MIGRATING:

{apify-2.4.0b5 → apify-2.5.0}/src/apify/_utils.py

@@ -31,10 +31,16 @@ GroupName = Literal['Classes', 'Abstract classes', 'Interfaces', 'Data structure
 
 
 def docs_group(group_name: GroupName) -> Callable:  # noqa: ARG001
-    """Decorator to mark symbols for rendering and grouping in documentation.
+    """Mark a symbol for rendering and grouping in documentation.
 
-    This decorator is used purely for documentation purposes and does not alter the behavior
+    This decorator is used solely for documentation purposes and does not modify the behavior
     of the decorated callable.
+
+    Args:
+        group_name: The documentation group to which the symbol belongs.
+
+    Returns:
+        The original callable without modification.
     """
 
     def wrapper(func: Callable) -> Callable:
@@ -44,12 +50,16 @@ def docs_group(group_name: GroupName) -> Callable:  # noqa: ARG001
 
 
 def docs_name(symbol_name: str) -> Callable:  # noqa: ARG001
-    """Decorator for renaming symbols in documentation.
+    """Rename a symbol for documentation rendering.
 
-    This changes the rendered name of the symbol only in the rendered web documentation.
+    This decorator modifies only the displayed name of the symbol in the generated documentation
+    and does not affect its runtime behavior.
 
-    This decorator is used purely for documentation purposes and does not alter the behavior
-    of the decorated callable.
+    Args:
+        symbol_name: The name to be used in the documentation.
+
+    Returns:
+        The original callable without modification.
     """
 
     def wrapper(func: Callable) -> Callable:

{apify-2.4.0b5 → apify-2.5.0}/src/apify/scrapy/_async_thread.py

@@ -113,7 +113,7 @@ class AsyncThread:
         await asyncio.gather(*tasks, return_exceptions=True)
 
     def _force_exit_event_loop(self) -> None:
-        """Forcefully shut down the event loop and its thread."""
+        """Shut down the event loop and its thread forcefully."""
         try:
             logger.info('Forced shutdown of the event loop and its thread...')
             self._eventloop.call_soon_threadsafe(self._eventloop.stop)

apify-2.5.0/src/apify/scrapy/extensions/__init__.py

@@ -0,0 +1,3 @@
+from apify.scrapy.extensions._httpcache import ApifyCacheStorage
+
+__all__ = ['ApifyCacheStorage']

apify-2.5.0/src/apify/scrapy/extensions/_httpcache.py

@@ -0,0 +1,212 @@
+from __future__ import annotations
+
+import gzip
+import io
+import pickle
+import re
+import struct
+from logging import getLogger
+from time import time
+from typing import TYPE_CHECKING
+
+from scrapy.http.headers import Headers
+from scrapy.responsetypes import responsetypes
+
+from apify import Configuration
+from apify.apify_storage_client import ApifyStorageClient
+from apify.scrapy._async_thread import AsyncThread
+from apify.storages import KeyValueStore
+
+if TYPE_CHECKING:
+    from scrapy import Request, Spider
+    from scrapy.http.response import Response
+    from scrapy.settings import BaseSettings
+    from scrapy.utils.request import RequestFingerprinterProtocol
+
+logger = getLogger(__name__)
+
+
+class ApifyCacheStorage:
+    """A Scrapy cache storage that uses the Apify `KeyValueStore` to store responses.
+
+    It can be set as a storage for Scrapy's built-in `HttpCacheMiddleware`, which caches
+    responses to requests. See HTTPCache middleware settings (prefixed with `HTTPCACHE_`)
+    in the Scrapy documentation for more information. Requires the asyncio Twisted reactor
+    to be installed.
+    """
+
+    def __init__(self, settings: BaseSettings) -> None:
+        self._expiration_max_items = 100
+        self._expiration_secs: int = settings.getint('HTTPCACHE_EXPIRATION_SECS')
+        self._spider: Spider | None = None
+        self._kvs: KeyValueStore | None = None
+        self._fingerprinter: RequestFingerprinterProtocol | None = None
+        self._async_thread: AsyncThread | None = None
+
+    def open_spider(self, spider: Spider) -> None:
+        """Open the cache storage for a spider."""
+        logger.debug('Using Apify key value cache storage', extra={'spider': spider})
+        self._spider = spider
+        self._fingerprinter = spider.crawler.request_fingerprinter
+        kvs_name = get_kvs_name(spider.name)
+
+        async def open_kvs() -> KeyValueStore:
+            config = Configuration.get_global_configuration()
+            if config.is_at_home:
+                storage_client = ApifyStorageClient.from_config(config)
+                return await KeyValueStore.open(name=kvs_name, storage_client=storage_client)
+            return await KeyValueStore.open(name=kvs_name)
+
+        logger.debug("Starting background thread for cache storage's event loop")
+        self._async_thread = AsyncThread()
+        logger.debug(f"Opening cache storage's {kvs_name!r} key value store")
+        self._kvs = self._async_thread.run_coro(open_kvs())
+
+    def close_spider(self, _: Spider, current_time: int | None = None) -> None:
+        """Close the cache storage for a spider."""
+        if self._async_thread is None:
+            raise ValueError('Async thread not initialized')
+
+        logger.info(f'Cleaning up cache items (max {self._expiration_max_items})')
+        if self._expiration_secs > 0:
+            if current_time is None:
+                current_time = int(time())
+
+            async def expire_kvs() -> None:
+                if self._kvs is None:
+                    raise ValueError('Key value store not initialized')
+                i = 0
+                async for item in self._kvs.iterate_keys():
+                    value = await self._kvs.get_value(item.key)
+                    try:
+                        gzip_time = read_gzip_time(value)
+                    except Exception as e:
+                        logger.warning(f'Malformed cache item {item.key}: {e}')
+                        await self._kvs.set_value(item.key, None)
+                    else:
+                        if self._expiration_secs < current_time - gzip_time:
+                            logger.debug(f'Expired cache item {item.key}')
+                            await self._kvs.set_value(item.key, None)
+                        else:
+                            logger.debug(f'Valid cache item {item.key}')
+                    if i == self._expiration_max_items:
+                        break
+                    i += 1
+
+            self._async_thread.run_coro(expire_kvs())
+
+        logger.debug('Closing cache storage')
+        try:
+            self._async_thread.close()
+        except KeyboardInterrupt:
+            logger.warning('Shutdown interrupted by KeyboardInterrupt!')
+        except Exception:
+            logger.exception('Exception occurred while shutting down cache storage')
+        finally:
+            logger.debug('Cache storage closed')
+
+    def retrieve_response(self, _: Spider, request: Request, current_time: int | None = None) -> Response | None:
+        """Retrieve a response from the cache storage."""
+        if self._async_thread is None:
+            raise ValueError('Async thread not initialized')
+        if self._kvs is None:
+            raise ValueError('Key value store not initialized')
+        if self._fingerprinter is None:
+            raise ValueError('Request fingerprinter not initialized')
+
+        key = self._fingerprinter.fingerprint(request).hex()
+        value = self._async_thread.run_coro(self._kvs.get_value(key))
+
+        if value is None:
+            logger.debug('Cache miss', extra={'request': request})
+            return None
+
+        if current_time is None:
+            current_time = int(time())
+        if 0 < self._expiration_secs < current_time - read_gzip_time(value):
+            logger.debug('Cache expired', extra={'request': request})
+            return None
+
+        data = from_gzip(value)
+        url = data['url']
+        status = data['status']
+        headers = Headers(data['headers'])
+        body = data['body']
+        respcls = responsetypes.from_args(headers=headers, url=url, body=body)
+
+        logger.debug('Cache hit', extra={'request': request})
+        return respcls(url=url, headers=headers, status=status, body=body)
+
+    def store_response(self, _: Spider, request: Request, response: Response) -> None:
+        """Store a response in the cache storage."""
+        if self._async_thread is None:
+            raise ValueError('Async thread not initialized')
+        if self._kvs is None:
+            raise ValueError('Key value store not initialized')
+        if self._fingerprinter is None:
+            raise ValueError('Request fingerprinter not initialized')
+
+        key = self._fingerprinter.fingerprint(request).hex()
+        data = {
+            'status': response.status,
+            'url': response.url,
+            'headers': dict(response.headers),
+            'body': response.body,
+        }
+        value = to_gzip(data)
+        self._async_thread.run_coro(self._kvs.set_value(key, value))
+
+
+def to_gzip(data: dict, mtime: int | None = None) -> bytes:
+    """Dump a dictionary to a gzip-compressed byte stream."""
+    with io.BytesIO() as byte_stream:
+        with gzip.GzipFile(fileobj=byte_stream, mode='wb', mtime=mtime) as gzip_file:
+            pickle.dump(data, gzip_file, protocol=4)
+        return byte_stream.getvalue()
+
+
+def from_gzip(gzip_bytes: bytes) -> dict:
+    """Load a dictionary from a gzip-compressed byte stream."""
+    with io.BytesIO(gzip_bytes) as byte_stream, gzip.GzipFile(fileobj=byte_stream, mode='rb') as gzip_file:
+        data: dict = pickle.load(gzip_file)
+        return data
+
+
+def read_gzip_time(gzip_bytes: bytes) -> int:
+    """Read the modification time from a gzip-compressed byte stream without decompressing the data."""
+    header = gzip_bytes[:10]
+    header_components = struct.unpack('<HBBI2B', header)
+    mtime: int = header_components[3]
+    return mtime
+
+
+def get_kvs_name(spider_name: str, max_length: int = 60) -> str:
+    """Get the key value store name for a spider.
+
+    The key value store name is derived from the spider name by replacing all special characters
+    with hyphens and trimming leading and trailing hyphens. The resulting name is prefixed with
+    'httpcache-' and truncated to the maximum length.
+
+    The documentation
+    [about storages](https://docs.apify.com/platform/storage/usage#named-and-unnamed-storages)
+    mentions that names can be up to 63 characters long, so the default max length is set to 60.
+
+    Such naming isn't unique per spider, but should be sufficiently unique for most use cases.
+    The name of the key value store should indicate to which spider it belongs, e.g. in
+    the listing in the Apify's console.
+
+    Args:
+        spider_name: Value of the Spider instance's name attribute.
+        max_length: Maximum length of the key value store name.
+
+    Returns: Key value store name.
+
+    Raises:
+        ValueError: If the spider name contains only special characters.
+    """
+    slug = re.sub(r'[^a-zA-Z0-9-]', '-', spider_name)
+    slug = re.sub(r'-+', '-', slug)
+    slug = slug.strip('-')
+    if not slug:
+        raise ValueError(f'Unsupported spider name: {spider_name!r} (slug: {slug!r})')
+    return f'httpcache-{slug}'[:max_length]

{apify-2.4.0b5 → apify-2.5.0}/src/apify/scrapy/requests.py

@@ -93,7 +93,7 @@ def to_scrapy_request(apify_request: ApifyRequest, spider: Spider) -> ScrapyRequ
     Returns:
         The converted Scrapy request.
     """
-    if not isinstance(cast(Any, apify_request), ApifyRequest):
+    if not isinstance(cast('Any', apify_request), ApifyRequest):
         raise TypeError('apify_request must be a crawlee.ScrapyRequest instance')
 
     logger.debug(f'to_scrapy_request was called (apify_request={apify_request})...')

{apify-2.4.0b5 → apify-2.5.0}/src/apify/scrapy/utils.py

@@ -44,6 +44,9 @@ def apply_apify_settings(*, settings: Settings | None = None, proxy_config: dict
     settings['DOWNLOADER_MIDDLEWARES']['scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware'] = None
     settings['DOWNLOADER_MIDDLEWARES']['apify.scrapy.middlewares.ApifyHttpProxyMiddleware'] = 750
 
+    # Set the default HTTPCache middleware storage backend to ApifyCacheStorage
+    settings['HTTPCACHE_STORAGE'] = 'apify.scrapy.extensions.ApifyCacheStorage'
+
     # Store the proxy configuration
     settings['APIFY_PROXY_SETTINGS'] = proxy_config
 

{apify-2.4.0b5 → apify-2.5.0}/src/apify/storages/_request_list.py

@@ -51,7 +51,7 @@ class RequestList(CrawleeRequestList):
         request_list_sources_input: list[dict[str, Any]] | None = None,
         http_client: HttpClient | None = None,
     ) -> RequestList:
-        """Creates RequestList from Actor input requestListSources.
+        """Initialize a new instance from request list source input.
 
         Args:
             name: Name of the returned RequestList.
@@ -108,9 +108,10 @@ class RequestList(CrawleeRequestList):
 
     @staticmethod
     async def _fetch_requests_from_url(
-        remote_url_requests_inputs: list[_RequestsFromUrlInput], http_client: HttpClient
+        remote_url_requests_inputs: list[_RequestsFromUrlInput],
+        http_client: HttpClient,
     ) -> list[Request]:
-        """Crete list of requests from url.
+        """Create list of requests from url.
 
         Send GET requests to urls defined in each requests_from_url of remote_url_requests_inputs. Run extracting
         callback on each response body and use URL_NO_COMMAS_REGEX regex to find all links. Create list of Requests from
@@ -119,7 +120,11 @@ class RequestList(CrawleeRequestList):
         created_requests: list[Request] = []
 
         def create_requests_from_response(request_input: _RequestsFromUrlInput, task: Task) -> None:
-            """Callback to scrape response body with regexp and create Requests from matches."""
+            """Extract links from response body and use them to create `Request` objects.
+
+            Use the regular expression to find all matching links in the response body, then create `Request`
+            objects from these links and the provided input attributes.
+            """
             matches = re.finditer(URL_NO_COMMAS_REGEX, task.result().read().decode('utf-8'))
             created_requests.extend(
                 [

{apify-2.4.0b5 → apify-2.5.0}/tests/integration/test_actor_scrapy.py

@@ -41,7 +41,8 @@ async def test_actor_scrapy_title_spider(
 
     items = await actor.last_run().dataset().list_items()
 
-    assert items.count >= 10
+    # CLOSESPIDER_PAGECOUNT is set to 10 in the spider settings.
+    assert items.count >= 9
 
     for item in items.items:
         assert 'url' in item

{apify-2.4.0b5 → apify-2.5.0}/tests/unit/actor/test_actor_lifecycle.py

@@ -5,12 +5,11 @@ import contextlib
 import json
 import sys
 from datetime import datetime, timezone
-from typing import Any, Callable, cast
+from typing import TYPE_CHECKING, Any, Callable, cast
 from unittest.mock import AsyncMock, Mock
 
 import pytest
 import websockets.asyncio.server
-from lazy_object_proxy import Proxy
 
 from apify_shared.consts import ActorEnvVars, ApifyEnvVars
 from crawlee.events._types import Event, EventPersistStateData
@@ -19,12 +18,15 @@ import apify._actor
 from apify import Actor
 from apify._actor import _ActorType
 
+if TYPE_CHECKING:
+    from lazy_object_proxy import Proxy
+
 
 async def test_actor_properly_init_with_async() -> None:
     async with Actor:
-        assert cast(Proxy, apify._actor.Actor).__wrapped__ is not None
-        assert cast(Proxy, apify._actor.Actor).__wrapped__._is_initialized
-    assert not cast(Proxy, apify._actor.Actor).__wrapped__._is_initialized
+        assert cast('Proxy', apify._actor.Actor).__wrapped__ is not None
+        assert cast('Proxy', apify._actor.Actor).__wrapped__._is_initialized
+    assert not cast('Proxy', apify._actor.Actor).__wrapped__._is_initialized
 
 
 async def test_actor_init() -> None:

apify-2.5.0/tests/unit/actor/test_configuration.py

@@ -0,0 +1,21 @@
+import pytest
+
+from apify import Configuration
+
+
+@pytest.mark.parametrize(
+    ('is_at_home', 'disable_browser_sandbox_in', 'disable_browser_sandbox_out'),
+    [
+        (False, False, False),
+        (False, True, True),
+        (True, False, True),
+        (True, True, True),
+    ],
+)
+def test_disable_browser_sandbox(
+    *, is_at_home: bool, disable_browser_sandbox_in: bool, disable_browser_sandbox_out: bool
+) -> None:
+    assert (
+        Configuration(is_at_home=is_at_home, disable_browser_sandbox=disable_browser_sandbox_in).disable_browser_sandbox
+        == disable_browser_sandbox_out
+    )