crawlee 1.0.4b7__tar.gz → 1.0.5b1__tar.gz

{crawlee-1.0.4b7 → crawlee-1.0.5b1}/CHANGELOG.md

@@ -3,7 +3,11 @@
 All notable changes to this project will be documented in this file.
 
 <!-- git-cliff-unreleased-start -->
-## 1.0.4 - **not yet released**
+## 1.0.5 - **not yet released**
+
+
+<!-- git-cliff-unreleased-end -->
+## [1.0.4](https://github.com/apify/crawlee-python/releases/tag/v1.0.4) (2025-10-24)
 
 ### 🐛 Bug Fixes
 
@@ -11,9 +15,9 @@ All notable changes to this project will be documented in this file.
 - Exclude incorrect links before checking `robots.txt` ([#1502](https://github.com/apify/crawlee-python/pull/1502)) ([3273da5](https://github.com/apify/crawlee-python/commit/3273da5fee62ec9254666b376f382474c3532a56)) by [@Mantisus](https://github.com/Mantisus), closes [#1499](https://github.com/apify/crawlee-python/issues/1499)
 - Resolve compatibility issue between `SqlStorageClient` and `AdaptivePlaywrightCrawler` ([#1496](https://github.com/apify/crawlee-python/pull/1496)) ([ce172c4](https://github.com/apify/crawlee-python/commit/ce172c425a8643a1d4c919db4f5e5a6e47e91deb)) by [@Mantisus](https://github.com/Mantisus), closes [#1495](https://github.com/apify/crawlee-python/issues/1495)
 - Fix `BasicCrawler` statistics persistence ([#1490](https://github.com/apify/crawlee-python/pull/1490)) ([1eb1c19](https://github.com/apify/crawlee-python/commit/1eb1c19aa6f9dda4a0e3f7eda23f77a554f95076)) by [@Pijukatel](https://github.com/Pijukatel), closes [#1501](https://github.com/apify/crawlee-python/issues/1501)
+- Save context state in result for `AdaptivePlaywrightCrawler` after isolated processing in `SubCrawler` ([#1488](https://github.com/apify/crawlee-python/pull/1488)) ([62b7c70](https://github.com/apify/crawlee-python/commit/62b7c70b54085fc65a660062028014f4502beba9)) by [@Mantisus](https://github.com/Mantisus), closes [#1483](https://github.com/apify/crawlee-python/issues/1483)
 
 
-<!-- git-cliff-unreleased-end -->
 ## [1.0.3](https://github.com/apify/crawlee-python/releases/tag/v1.0.3) (2025-10-17)
 
 ### 🐛 Bug Fixes

{crawlee-1.0.4b7 → crawlee-1.0.5b1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: crawlee
-Version: 1.0.4b7
+Version: 1.0.5b1
 Summary: Crawlee for Python
 Project-URL: Apify Homepage, https://apify.com
 Project-URL: Changelog, https://crawlee.dev/python/docs/changelog

{crawlee-1.0.4b7 → crawlee-1.0.5b1}/docs/guides/architecture_overview.mdx

@@ -291,7 +291,7 @@ Request loaders provide a subset of <ApiLink to="class/RequestQueue">`RequestQue
 
 - <ApiLink to="class/RequestLoader">`RequestLoader`</ApiLink> - Base interface for read-only access to a stream of requests, with capabilities like fetching the next request, marking as handled, and status checking.
 - <ApiLink to="class/RequestList">`RequestList`</ApiLink> - Lightweight in-memory implementation of `RequestLoader` for managing static lists of URLs.
-- <ApiLink to="class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> - Specialized loader for reading URLs from XML sitemaps with filtering capabilities.
+- <ApiLink to="class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> - A specialized loader that reads URLs from XML and plain-text sitemaps following the [Sitemaps protocol](https://www.sitemaps.org/protocol.html) with filtering capabilities.
 
 ### Request managers
 

{crawlee-1.0.4b7 → crawlee-1.0.5b1}/docs/guides/request_loaders.mdx

@@ -31,7 +31,7 @@ The [`request_loaders`](https://github.com/apify/crawlee-python/tree/master/src/
 And specific request loader implementations:
 
 - <ApiLink to="class/RequestList">`RequestList`</ApiLink>: A lightweight implementation for managing a static list of URLs.
-- <ApiLink to="class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink>: A specialized loader that reads URLs from XML sitemaps with filtering capabilities.
+- <ApiLink to="class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink>: A specialized loader that reads URLs from XML and plain-text sitemaps following the [Sitemaps protocol](https://www.sitemaps.org/protocol.html) with filtering capabilities.
 
 Below is a class diagram that illustrates the relationships between these components and the <ApiLink to="class/RequestQueue">`RequestQueue`</ApiLink>:
 
@@ -130,7 +130,13 @@ To enable persistence, provide `persist_state_key` and optionally `persist_reque
 
 ### Sitemap request loader
 
-The <ApiLink to="class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> is a specialized request loader that reads URLs from XML sitemaps. It's particularly useful when you want to crawl a website systematically by following its sitemap structure. The loader supports filtering URLs using glob patterns and regular expressions, allowing you to include or exclude specific types of URLs. The <ApiLink to="class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> provides streaming processing of sitemaps, ensuring efficient memory usage without loading the entire sitemap into memory.
+The <ApiLink to="class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> is a specialized request loader that reads URLs from sitemaps following the [Sitemaps protocol](https://www.sitemaps.org/protocol.html). It supports both XML and plain text sitemap formats. It's particularly useful when you want to crawl a website systematically by following its sitemap structure.
+
+:::note
+The `SitemapRequestLoader` is designed specifically for sitemaps that follow the standard Sitemaps protocol. HTML pages containing links are not supported by this loader - those should be handled by regular crawlers using the `enqueue_links` functionality.
+:::
+
+The loader supports filtering URLs using glob patterns and regular expressions, allowing you to include or exclude specific types of URLs. The <ApiLink to="class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> provides streaming processing of sitemaps, ensuring efficient memory usage without loading the entire sitemap into memory.
 
 <RunnableCodeBlock className="language-python" language="python">
     {SitemapExample}

{crawlee-1.0.4b7 → crawlee-1.0.5b1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "crawlee"
-version = "1.0.4b7"
+version = "1.0.5b1"
 description = "Crawlee for Python"
 authors = [{ name = "Apify Technologies s.r.o.", email = "support@apify.com" }]
 license = { file = "LICENSE" }

{crawlee-1.0.4b7 → crawlee-1.0.5b1}/src/crawlee/crawlers/_adaptive_playwright/_adaptive_playwright_crawler.py

@@ -315,7 +315,7 @@ class AdaptivePlaywrightCrawler(
                 ),
                 logger=self._logger,
             )
-            return SubCrawlerRun(result=result)
+            return SubCrawlerRun(result=result, run_context=context_linked_to_result)
         except Exception as e:
             return SubCrawlerRun(exception=e)
 
@@ -371,7 +371,8 @@ class AdaptivePlaywrightCrawler(
                 self.track_http_only_request_handler_runs()
 
                 static_run = await self._crawl_one(rendering_type='static', context=context)
-                if static_run.result and self.result_checker(static_run.result):
+                if static_run.result and static_run.run_context and self.result_checker(static_run.result):
+                    self._update_context_from_copy(context, static_run.run_context)
                     self._context_result_map[context] = static_run.result
                     return
                 if static_run.exception:
@@ -402,13 +403,10 @@ class AdaptivePlaywrightCrawler(
         if pw_run.exception is not None:
             raise pw_run.exception
 
-        if pw_run.result:
-            self._context_result_map[context] = pw_run.result
-
+        if pw_run.result and pw_run.run_context:
             if should_detect_rendering_type:
                 detection_result: RenderingType
                 static_run = await self._crawl_one('static', context=context, state=old_state_copy)
-
                 if static_run.result and self.result_comparator(static_run.result, pw_run.result):
                     detection_result = 'static'
                 else:
@@ -417,6 +415,9 @@ class AdaptivePlaywrightCrawler(
                 context.log.debug(f'Detected rendering type {detection_result} for {context.request.url}')
                 self.rendering_type_predictor.store_result(context.request, detection_result)
 
+            self._update_context_from_copy(context, pw_run.run_context)
+            self._context_result_map[context] = pw_run.result
+
     def pre_navigation_hook(
         self,
         hook: Callable[[AdaptivePlaywrightPreNavCrawlingContext], Awaitable[None]] | None = None,
@@ -451,8 +452,32 @@ class AdaptivePlaywrightCrawler(
     def track_rendering_type_mispredictions(self) -> None:
         self.statistics.state.rendering_type_mispredictions += 1
 
+    def _update_context_from_copy(self, context: BasicCrawlingContext, context_copy: BasicCrawlingContext) -> None:
+        """Update mutable fields of `context` from `context_copy`.
+
+        Uses object.__setattr__ to bypass frozen dataclass restrictions,
+        allowing state synchronization after isolated crawler execution.
+        """
+        updating_attributes = {
+            'request': ('headers', 'user_data'),
+            'session': ('_user_data', '_usage_count', '_error_score', '_cookies'),
+        }
+
+        for attr, sub_attrs in updating_attributes.items():
+            original_sub_obj = getattr(context, attr)
+            copy_sub_obj = getattr(context_copy, attr)
+
+            # Check that both sub objects are not None
+            if original_sub_obj is None or copy_sub_obj is None:
+                continue
+
+            for sub_attr in sub_attrs:
+                new_value = getattr(copy_sub_obj, sub_attr)
+                object.__setattr__(original_sub_obj, sub_attr, new_value)
+
 
 @dataclass(frozen=True)
 class SubCrawlerRun:
     result: RequestHandlerRunResult | None = None
     exception: Exception | None = None
+    run_context: BasicCrawlingContext | None = None

{crawlee-1.0.4b7 → crawlee-1.0.5b1}/src/crawlee/request_loaders/_sitemap_request_loader.py

@@ -90,6 +90,11 @@ class SitemapRequestLoaderState(BaseModel):
 class SitemapRequestLoader(RequestLoader):
     """A request loader that reads URLs from sitemap(s).
 
+    The loader is designed to handle sitemaps that follow the format described in the Sitemaps protocol
+    (https://www.sitemaps.org/protocol.html). It supports both XML and plain text sitemap formats.
+    Note that HTML pages containing links are not supported - those should be handled by regular crawlers
+    and the `enqueue_links` functionality.
+
     The loader fetches and parses sitemaps in the background, allowing crawling to start
     before all URLs are loaded. It supports filtering URLs using glob and regex patterns.
 

{crawlee-1.0.4b7 → crawlee-1.0.5b1}/tests/unit/_utils/test_sitemap.py

@@ -1,9 +1,7 @@
 import base64
 import gzip
-import sys
 from datetime import datetime
 
-import pytest
 from yarl import URL
 
 from crawlee._utils.sitemap import Sitemap, SitemapUrl, parse_sitemap
@@ -95,8 +93,6 @@ async def test_gzipped_sitemap(server_url: URL, http_client: HttpClient) -> None
     assert set(sitemap.urls) == BASIC_RESULTS
 
 
-# TODO: Remove this skip when #1460 is resolved.
-@pytest.mark.skipif(sys.platform != 'linux', reason='Flaky with Curl on Windows, see #1460.')
 async def test_gzipped_sitemap_with_invalid_data(server_url: URL, http_client: HttpClient) -> None:
     """Test loading a invalid gzipped sitemap with correct type and .xml.gz url."""
     compress_data = compress_gzip(BASIC_SITEMAP)

{crawlee-1.0.4b7 → crawlee-1.0.5b1}/tests/unit/conftest.py

@@ -6,7 +6,7 @@ from __future__ import annotations
 import logging
 import os
 import warnings
-from typing import TYPE_CHECKING, cast
+from typing import TYPE_CHECKING, Any, cast
 
 import pytest
 from curl_cffi import CurlHttpVersion
@@ -205,9 +205,16 @@ def redirect_server_url(redirect_http_server: TestServer) -> URL:
         pytest.param('curl', id='curl'),
     ]
 )
-async def http_client(request: pytest.FixtureRequest) -> HttpClient:
+async def http_client(request: pytest.FixtureRequest) -> AsyncGenerator[HttpClient, None]:
+    class_client: type[HttpClient]
     if request.param == 'curl':
-        return CurlImpersonateHttpClient(http_version=CurlHttpVersion.V1_1)
-    if request.param == 'impit':
-        return ImpitHttpClient(http3=False)
-    return HttpxHttpClient(http2=False)
+        class_client = CurlImpersonateHttpClient
+        kwargs: dict[str, Any] = {'http_version': CurlHttpVersion.V1_1}
+    elif request.param == 'impit':
+        class_client = ImpitHttpClient
+        kwargs = {'http3': False}
+    else:
+        class_client = HttpxHttpClient
+        kwargs = {'http2': True}
+    async with class_client(**kwargs) as client:
+        yield client

{crawlee-1.0.4b7 → crawlee-1.0.5b1}/tests/unit/crawlers/_adaptive_playwright/test_adaptive_playwright_crawler.py

@@ -29,9 +29,10 @@ from crawlee.crawlers._adaptive_playwright._adaptive_playwright_crawler_statisti
 from crawlee.crawlers._adaptive_playwright._adaptive_playwright_crawling_context import (
     AdaptiveContextError,
 )
+from crawlee.sessions import SessionPool
 from crawlee.statistics import Statistics
 from crawlee.storage_clients import SqlStorageClient
-from crawlee.storages import KeyValueStore
+from crawlee.storages import KeyValueStore, RequestQueue
 
 if TYPE_CHECKING:
     from collections.abc import AsyncGenerator, Iterator
@@ -730,6 +731,84 @@ async def test_adaptive_context_query_non_existing_element(test_urls: list[str])
     mocked_h3_handler.assert_called_once_with(None)
 
 
+@pytest.mark.parametrize(
+    'test_input',
+    [
+        pytest.param(
+            TestInput(
+                expected_pw_count=0,
+                expected_static_count=2,
+                rendering_types=cycle(['static']),
+                detection_probability_recommendation=cycle([0]),
+            ),
+            id='Static only',
+        ),
+        pytest.param(
+            TestInput(
+                expected_pw_count=2,
+                expected_static_count=0,
+                rendering_types=cycle(['client only']),
+                detection_probability_recommendation=cycle([0]),
+            ),
+            id='Client only',
+        ),
+        pytest.param(
+            TestInput(
+                expected_pw_count=2,
+                expected_static_count=2,
+                rendering_types=cycle(['static', 'client only']),
+                detection_probability_recommendation=cycle([1]),
+            ),
+            id='Enforced rendering type detection',
+        ),
+    ],
+)
+async def test_change_context_state_after_handling(test_input: TestInput, server_url: URL) -> None:
+    """Test that context state is saved after handling the request."""
+    predictor = _SimpleRenderingTypePredictor(
+        rendering_types=test_input.rendering_types,
+        detection_probability_recommendation=test_input.detection_probability_recommendation,
+    )
+
+    request_queue = await RequestQueue.open(name='state-test')
+    used_session_id = None
+
+    async with SessionPool() as session_pool:
+        crawler = AdaptivePlaywrightCrawler.with_beautifulsoup_static_parser(
+            rendering_type_predictor=predictor,
+            session_pool=session_pool,
+            request_manager=request_queue,
+        )
+
+        @crawler.router.default_handler
+        async def request_handler(context: AdaptivePlaywrightCrawlingContext) -> None:
+            nonlocal used_session_id
+
+            if context.session is not None:
+                used_session_id = context.session.id
+                context.session.user_data['session_state'] = True
+
+            if isinstance(context.request.user_data['request_state'], list):
+                context.request.user_data['request_state'].append('handler')
+
+        request = Request.from_url(str(server_url), user_data={'request_state': ['initial']})
+
+        await crawler.run([request])
+
+        assert used_session_id is not None
+
+        session = await session_pool.get_session_by_id(used_session_id)
+        check_request = await request_queue.get_request(request.unique_key)
+
+        assert session is not None
+        assert check_request is not None
+        assert session.user_data.get('session_state') is True
+        # Check that request user data was updated in the handler and only onse.
+        assert check_request.user_data.get('request_state') == ['initial', 'handler']
+
+        await request_queue.drop()
+
+
 async def test_adaptive_playwright_crawler_with_sql_storage(test_urls: list[str], tmp_path: Path) -> None:
     """Tests that AdaptivePlaywrightCrawler can be initialized with SqlStorageClient."""
     storage_dir = tmp_path / 'test_table.db'

{crawlee-1.0.4b7 → crawlee-1.0.5b1}/tests/unit/http_clients/test_http_clients.py

@@ -1,7 +1,6 @@
 from __future__ import annotations
 
 import os
-import sys
 from typing import TYPE_CHECKING
 
 import pytest
@@ -164,10 +163,6 @@ async def test_send_request_allow_redirects_false(custom_http_client: HttpClient
 
 
 async def test_stream(http_client: HttpClient, server_url: URL) -> None:
-    # TODO: Remove this skip when #1494 is resolved.
-    if isinstance(http_client, CurlImpersonateHttpClient) and sys.platform != 'linux':
-        pytest.skip('Flaky with Curl on Windows, see #1494.')
-
     content_body: bytes = b''
 
     async with http_client.stream(str(server_url)) as response:

{crawlee-1.0.4b7 → crawlee-1.0.5b1}/tests/unit/storages/test_request_queue.py

@@ -449,8 +449,9 @@ async def test_add_requests_wait_for_all(
         # Immediately after adding, the total count may be less than 15 due to background processing
         assert await rq.get_total_count() <= 15
 
-        # Wait a 500 milliseconds for background tasks to complete
-        await asyncio.sleep(0.5)
+        # Wait for background tasks to complete
+        while await rq.get_total_count() < 15:  # noqa: ASYNC110
+            await asyncio.sleep(0.1)
 
     # Verify all requests were added
     assert await rq.get_total_count() == 15