crawlee 1.0.1b12__tar.gz → 1.0.2__tar.gz

{crawlee-1.0.1b12 → crawlee-1.0.2}/.github/workflows/build_and_deploy_docs.yaml

@@ -40,7 +40,7 @@ jobs:
           python-version: ${{ env.PYTHON_VERSION }}
 
       - name: Set up uv package manager
-        uses: astral-sh/setup-uv@v6
+        uses: astral-sh/setup-uv@v7
         with:
           python-version: ${{ env.PYTHON_VERSION }}
 

{crawlee-1.0.1b12 → crawlee-1.0.2}/.github/workflows/templates_e2e_tests.yaml

@@ -44,7 +44,7 @@ jobs:
         run: pipx install poetry
 
       - name: Set up uv package manager
-        uses: astral-sh/setup-uv@v6
+        uses: astral-sh/setup-uv@v7
         with:
           python-version: ${{ env.PYTHON_VERSION }}
 

{crawlee-1.0.1b12 → crawlee-1.0.2}/CHANGELOG.md

@@ -2,8 +2,16 @@
 
 All notable changes to this project will be documented in this file.
 
-<!-- git-cliff-unreleased-start -->
-## 1.0.1 - **not yet released**
+## [1.0.2](https://github.com/apify/crawlee-python/releases/tag/v1.0.2) (2025-10-08)
+
+### 🐛 Bug Fixes
+
+- Use Self type in the open() method of storage clients ([#1462](https://github.com/apify/crawlee-python/pull/1462)) ([4ec6f6c](https://github.com/apify/crawlee-python/commit/4ec6f6c08f81632197f602ff99151338b3eba6e7)) by [@janbuchar](https://github.com/janbuchar)
+- Add storages name validation ([#1457](https://github.com/apify/crawlee-python/pull/1457)) ([84de11a](https://github.com/apify/crawlee-python/commit/84de11a3a603503076f5b7df487c9abab68a9015)) by [@Mantisus](https://github.com/Mantisus), closes [#1434](https://github.com/apify/crawlee-python/issues/1434)
+- Pin pydantic version to &lt;2.12.0 to avoid compatibility issues ([#1467](https://github.com/apify/crawlee-python/pull/1467)) ([f11b86f](https://github.com/apify/crawlee-python/commit/f11b86f7ed57f98e83dc1b52f15f2017a919bf59)) by [@vdusek](https://github.com/vdusek)
+
+
+## [1.0.1](https://github.com/apify/crawlee-python/releases/tag/v1.0.1) (2025-10-06)
 
 ### 🐛 Bug Fixes
 
@@ -11,9 +19,11 @@ All notable changes to this project will be documented in this file.
 - Update templates to handle optional httpx client ([#1440](https://github.com/apify/crawlee-python/pull/1440)) ([c087efd](https://github.com/apify/crawlee-python/commit/c087efd39baedf46ca3e5cae1ddc1acd6396e6c1)) by [@Pijukatel](https://github.com/Pijukatel)
 
 
-<!-- git-cliff-unreleased-end -->
 ## [1.0.0](https://github.com/apify/crawlee-python/releases/tag/v1.0.0) (2025-09-29)
 
+- Check out the [Release blog post](https://crawlee.dev/blog/crawlee-for-python-v1) for more details.
+- Check out the [Upgrading guide](https://crawlee.dev/python/docs/upgrading/upgrading-to-v1) to ensure a smooth update.
+
 ### 🚀 Features
 
 - Add utility for load and parse Sitemap and `SitemapRequestLoader` ([#1169](https://github.com/apify/crawlee-python/pull/1169)) ([66599f8](https://github.com/apify/crawlee-python/commit/66599f8d085f3a8622e130019b6fdce2325737de)) by [@Mantisus](https://github.com/Mantisus), closes [#1161](https://github.com/apify/crawlee-python/issues/1161)
@@ -196,6 +206,9 @@ All notable changes to this project will be documented in this file.
 
 ## [0.6.0](https://github.com/apify/crawlee-python/releases/tag/v0.6.0) (2025-03-03)
 
+- Check out the [Release blog post](https://crawlee.dev/blog/crawlee-for-python-v06) for more details.
+- Check out the [Upgrading guide](https://crawlee.dev/python/docs/upgrading/upgrading-to-v0x#upgrading-to-v06) to ensure a smooth update.
+
 ### 🚀 Features
 
 - Integrate browserforge fingerprints ([#829](https://github.com/apify/crawlee-python/pull/829)) ([2b156b4](https://github.com/apify/crawlee-python/commit/2b156b4ba688f9111195422e6058dff30eb1f782)) by [@Pijukatel](https://github.com/Pijukatel), closes [#549](https://github.com/apify/crawlee-python/issues/549)
@@ -276,6 +289,9 @@ All notable changes to this project will be documented in this file.
 
 ## [0.5.0](https://github.com/apify/crawlee-python/releases/tag/v0.5.0) (2025-01-02)
 
+- Check out the [Release blog post](https://crawlee.dev/blog/crawlee-for-python-v05) for more details.
+- Check out the [Upgrading guide](https://crawlee.dev/python/docs/upgrading/upgrading-to-v0x#upgrading-to-v05) to ensure a smooth update.
+
 ### 🚀 Features
 
 - Add possibility to use None as no proxy in tiered proxies ([#760](https://github.com/apify/crawlee-python/pull/760)) ([0fbd017](https://github.com/apify/crawlee-python/commit/0fbd01723b9fe2e3410e0f358cab2f22848b08d0)) by [@Pijukatel](https://github.com/Pijukatel), closes [#687](https://github.com/apify/crawlee-python/issues/687)
@@ -367,6 +383,8 @@ All notable changes to this project will be documented in this file.
 
 ## [0.4.0](https://github.com/apify/crawlee-python/releases/tag/v0.4.0) (2024-11-01)
 
+- Check out the [Upgrading guide](https://crawlee.dev/python/docs/upgrading/upgrading-to-v0x#upgrading-to-v04) to ensure a smooth update.
+
 ### 🚀 Features
 
 - [**breaking**] Add headers in unique key computation ([#609](https://github.com/apify/crawlee-python/pull/609)) ([6c4746f](https://github.com/apify/crawlee-python/commit/6c4746fa8ff86952a812b32a1d70dc910e76b43e)) by [@Prathamesh010](https://github.com/Prathamesh010), closes [#548](https://github.com/apify/crawlee-python/issues/548)
@@ -476,6 +494,8 @@ All notable changes to this project will be documented in this file.
 
 ## [0.3.0](https://github.com/apify/crawlee-python/releases/tag/v0.3.0) (2024-08-27)
 
+- Check out the [Upgrading guide](https://crawlee.dev/python/docs/upgrading/upgrading-to-v0x#upgrading-to-v03) to ensure a smooth update.
+
 ### 🚀 Features
 
 - Implement ParselCrawler that adds support for Parsel ([#348](https://github.com/apify/crawlee-python/pull/348)) ([a3832e5](https://github.com/apify/crawlee-python/commit/a3832e527f022f32cce4a80055da3b7967b74522)) by [@asymness](https://github.com/asymness), closes [#335](https://github.com/apify/crawlee-python/issues/335)

{crawlee-1.0.1b12 → crawlee-1.0.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: crawlee
-Version: 1.0.1b12
+Version: 1.0.2
 Summary: Crawlee for Python
 Project-URL: Apify Homepage, https://apify.com
 Project-URL: Changelog, https://crawlee.dev/python/docs/changelog
@@ -232,7 +232,7 @@ Requires-Dist: more-itertools>=10.2.0
 Requires-Dist: protego>=0.5.0
 Requires-Dist: psutil>=6.0.0
 Requires-Dist: pydantic-settings!=2.7.0,!=2.7.1,!=2.8.0,>=2.2.0
-Requires-Dist: pydantic>=2.11.0
+Requires-Dist: pydantic<2.12.0,>=2.11.0
 Requires-Dist: pyee>=9.0.0
 Requires-Dist: tldextract>=5.1.0
 Requires-Dist: typing-extensions>=4.1.0

crawlee-1.0.2/docs/examples/code_examples/using_browser_profiles_chrome.py

@@ -0,0 +1,56 @@
+import asyncio
+import shutil
+from pathlib import Path
+from tempfile import TemporaryDirectory
+
+from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
+
+# Profile name to use (usually 'Default' for single profile setups)
+PROFILE_NAME = 'Default'
+
+# Paths to Chrome profiles in your system (example for Windows)
+# Use `chrome://version/` to find your profile path
+PROFILE_PATH = Path(Path.home(), 'AppData', 'Local', 'Google', 'Chrome', 'User Data')
+
+
+async def main() -> None:
+    # Create a temporary folder to copy the profile to
+    with TemporaryDirectory(prefix='crawlee-') as tmpdirname:
+        tmp_profile_dir = Path(tmpdirname)
+
+        # Copy the profile to a temporary folder
+        shutil.copytree(
+            PROFILE_PATH / PROFILE_NAME,
+            tmp_profile_dir / PROFILE_NAME,
+            dirs_exist_ok=True,
+        )
+
+        crawler = PlaywrightCrawler(
+            headless=False,
+            # Use chromium for Chrome compatibility
+            browser_type='chromium',
+            # Disable fingerprints to preserve profile identity
+            fingerprint_generator=None,
+            # Set user data directory to temp folder
+            user_data_dir=tmp_profile_dir,
+            browser_launch_options={
+                # Use installed Chrome browser
+                'channel': 'chrome',
+                # Slow down actions to mimic human behavior
+                'slow_mo': 200,
+                'args': [
+                    # Use the specified profile
+                    f'--profile-directory={PROFILE_NAME}',
+                ],
+            },
+        )
+
+        @crawler.router.default_handler
+        async def default_handler(context: PlaywrightCrawlingContext) -> None:
+            context.log.info(f'Visiting {context.request.url}')
+
+        await crawler.run(['https://crawlee.dev/'])
+
+
+if __name__ == '__main__':
+    asyncio.run(main())

crawlee-1.0.2/docs/examples/code_examples/using_browser_profiles_firefox.py

@@ -0,0 +1,42 @@
+import asyncio
+from pathlib import Path
+
+from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
+
+# Replace this with your actual Firefox profile name
+# Find it at about:profiles in Firefox
+PROFILE_NAME = 'your-profile-name-here'
+
+# Paths to Firefox profiles in your system (example for Windows)
+# Use `about:profiles` to find your profile path
+PROFILE_PATH = Path(
+    Path.home(), 'AppData', 'Roaming', 'Mozilla', 'Firefox', 'Profiles', PROFILE_NAME
+)
+
+
+async def main() -> None:
+    crawler = PlaywrightCrawler(
+        # Use Firefox browser type
+        browser_type='firefox',
+        # Disable fingerprints to use the profile as is
+        fingerprint_generator=None,
+        headless=False,
+        # Path to your Firefox profile
+        user_data_dir=PROFILE_PATH,
+        browser_launch_options={
+            'args': [
+                # Required to avoid version conflicts
+                '--allow-downgrade'
+            ]
+        },
+    )
+
+    @crawler.router.default_handler
+    async def default_handler(context: PlaywrightCrawlingContext) -> None:
+        context.log.info(f'Visiting {context.request.url}')
+
+    await crawler.run(['https://crawlee.dev/'])
+
+
+if __name__ == '__main__':
+    asyncio.run(main())

crawlee-1.0.2/docs/examples/using_browser_profile.mdx

@@ -0,0 +1,41 @@
+---
+id: using_browser_profile
+title: Using browser profile
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+import CodeBlock from '@theme/CodeBlock';
+
+import ChromeProfileExample from '!!raw-loader!./code_examples/using_browser_profiles_chrome.py';
+import FirefoxProfileExample from '!!raw-loader!./code_examples/using_browser_profiles_firefox.py';
+
+This example demonstrates how to run <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> using your local browser profile from [Chrome](https://www.google.com/intl/us/chrome/) or [Firefox](https://www.firefox.com/).
+
+Using browser profiles allows you to leverage existing login sessions, saved passwords, bookmarks, and other personalized browser data during crawling. This can be particularly useful for testing scenarios or when you need to access content that requires authentication.
+
+## Chrome browser
+
+To run <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> with your Chrome profile, you need to know the path to your profile files. You can find this information by entering `chrome://version/` as a URL in your Chrome browser. If you have multiple profiles, pay attention to the profile name - if you only have one profile, it's always `Default`.
+
+You also need to use the [`channel`](https://playwright.dev/python/docs/api/class-browsertype#browser-type-launch-option-channel) parameter in `browser_launch_options` to use the Chrome browser installed on your system instead of Playwright's Chromium.
+
+:::warning Profile access limitation
+Due to [Chrome's security policies](https://developer.chrome.com/blog/remote-debugging-port), automation cannot use your main browsing profile directly. The example copies your profile to a temporary location as a workaround.
+:::
+
+Make sure you don't have any running Chrome browser processes before running this code:
+
+<CodeBlock className="language-python" language="python">
+    {ChromeProfileExample}
+</CodeBlock>
+
+## Firefox browser
+
+To find the path to your Firefox profile, enter `about:profiles` as a URL in your Firefox browser. Unlike Chrome, you can use your standard profile path directly without copying it first.
+
+Make sure you don't have any running Firefox browser processes before running this code:
+
+<CodeBlock className="language-python" language="python">
+    {FirefoxProfileExample}
+</CodeBlock>

{crawlee-1.0.1b12 → crawlee-1.0.2}/docs/upgrading/upgrading_to_v1.md

@@ -333,3 +333,7 @@ async def main() -> None:
 
     await crawler.run(['https://crawlee.dev/'])
 ```
+
+### New storage naming restrictions
+
+We've introduced naming restrictions for storages to ensure compatibility with Apify Platform requirements and prevent potential conflicts. Storage names may include only letters (a–z, A–Z), digits (0–9), and hyphens (-), with hyphens allowed only in the middle of the name (for example, my-storage-1).

{crawlee-1.0.1b12 → crawlee-1.0.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "crawlee"
-version = "1.0.1b12"
+version = "1.0.2"
 description = "Crawlee for Python"
 authors = [{ name = "Apify Technologies s.r.o.", email = "support@apify.com" }]
 license = { file = "LICENSE" }
@@ -40,7 +40,7 @@ dependencies = [
     "protego>=0.5.0",
     "psutil>=6.0.0",
     "pydantic-settings>=2.2.0,!=2.7.0,!=2.7.1,!=2.8.0",
-    "pydantic>=2.11.0",
+    "pydantic>=2.11.0,<2.12.0",
     "pyee>=9.0.0",
     "tldextract>=5.1.0",
     "typing-extensions>=4.1.0",
@@ -107,7 +107,7 @@ dev = [
     "pytest-timeout~=2.4.0",
     "pytest-xdist~=3.8.0",
     "pytest~=8.4.0",
-    "ruff~=0.13.0",
+    "ruff~=0.14.0",
     "setuptools", # setuptools are used by pytest, but not explicitly required
     "types-beautifulsoup4~=4.12.0.20240229",
     "types-cachetools~=6.2.0.20250827",

{crawlee-1.0.1b12 → crawlee-1.0.2}/src/crawlee/storage_clients/_file_system/_dataset_client.py

@@ -9,7 +9,7 @@ from pathlib import Path
 from typing import TYPE_CHECKING, Any
 
 from pydantic import ValidationError
-from typing_extensions import override
+from typing_extensions import Self, override
 
 from crawlee._consts import METADATA_FILENAME
 from crawlee._utils.crypto import crypto_random_object_id
@@ -94,7 +94,7 @@ class FileSystemDatasetClient(DatasetClient):
         name: str | None,
         alias: str | None,
         configuration: Configuration,
-    ) -> FileSystemDatasetClient:
+    ) -> Self:
         """Open or create a file system dataset client.
 
         This method attempts to open an existing dataset from the file system. If a dataset with the specified ID

{crawlee-1.0.1b12 → crawlee-1.0.2}/src/crawlee/storage_clients/_file_system/_key_value_store_client.py

@@ -10,7 +10,7 @@ from pathlib import Path
 from typing import TYPE_CHECKING, Any
 
 from pydantic import ValidationError
-from typing_extensions import override
+from typing_extensions import Self, override
 
 from crawlee._consts import METADATA_FILENAME
 from crawlee._utils.crypto import crypto_random_object_id
@@ -93,7 +93,7 @@ class FileSystemKeyValueStoreClient(KeyValueStoreClient):
         name: str | None,
         alias: str | None,
         configuration: Configuration,
-    ) -> FileSystemKeyValueStoreClient:
+    ) -> Self:
         """Open or create a file system key-value store client.
 
         This method attempts to open an existing key-value store from the file system. If a KVS with the specified

{crawlee-1.0.1b12 → crawlee-1.0.2}/src/crawlee/storage_clients/_file_system/_request_queue_client.py

@@ -11,7 +11,7 @@ from pathlib import Path
 from typing import TYPE_CHECKING
 
 from pydantic import BaseModel, ValidationError
-from typing_extensions import override
+from typing_extensions import Self, override
 
 from crawlee import Request
 from crawlee._consts import METADATA_FILENAME
@@ -144,7 +144,7 @@ class FileSystemRequestQueueClient(RequestQueueClient):
         name: str | None,
         alias: str | None,
         configuration: Configuration,
-    ) -> FileSystemRequestQueueClient:
+    ) -> Self:
         """Open or create a file system request queue client.
 
         This method attempts to open an existing request queue from the file system. If a queue with the specified

{crawlee-1.0.1b12 → crawlee-1.0.2}/src/crawlee/storage_clients/_memory/_dataset_client.py

@@ -4,7 +4,7 @@ from datetime import datetime, timezone
 from logging import getLogger
 from typing import TYPE_CHECKING, Any
 
-from typing_extensions import override
+from typing_extensions import Self, override
 
 from crawlee._utils.crypto import crypto_random_object_id
 from crawlee._utils.raise_if_too_many_kwargs import raise_if_too_many_kwargs
@@ -55,7 +55,7 @@ class MemoryDatasetClient(DatasetClient):
         id: str | None,
         name: str | None,
         alias: str | None,
-    ) -> MemoryDatasetClient:
+    ) -> Self:
         """Open or create a new memory dataset client.
 
         This method creates a new in-memory dataset instance. Unlike persistent storage implementations, memory

{crawlee-1.0.1b12 → crawlee-1.0.2}/src/crawlee/storage_clients/_memory/_key_value_store_client.py

@@ -4,7 +4,7 @@ import sys
 from datetime import datetime, timezone
 from typing import TYPE_CHECKING, Any
 
-from typing_extensions import override
+from typing_extensions import Self, override
 
 from crawlee._utils.crypto import crypto_random_object_id
 from crawlee._utils.file import infer_mime_type
@@ -53,7 +53,7 @@ class MemoryKeyValueStoreClient(KeyValueStoreClient):
         id: str | None,
         name: str | None,
         alias: str | None,
-    ) -> MemoryKeyValueStoreClient:
+    ) -> Self:
         """Open or create a new memory key-value store client.
 
         This method creates a new in-memory key-value store instance. Unlike persistent storage implementations,

{crawlee-1.0.1b12 → crawlee-1.0.2}/src/crawlee/storage_clients/_memory/_request_queue_client.py

@@ -6,7 +6,7 @@ from datetime import datetime, timezone
 from logging import getLogger
 from typing import TYPE_CHECKING
 
-from typing_extensions import override
+from typing_extensions import Self, override
 
 from crawlee import Request
 from crawlee._utils.crypto import crypto_random_object_id
@@ -65,7 +65,7 @@ class MemoryRequestQueueClient(RequestQueueClient):
         id: str | None,
         name: str | None,
         alias: str | None,
-    ) -> MemoryRequestQueueClient:
+    ) -> Self:
         """Open or create a new memory request queue client.
 
         This method creates a new in-memory request queue instance. Unlike persistent storage implementations,

{crawlee-1.0.1b12 → crawlee-1.0.2}/src/crawlee/storage_clients/_sql/_dataset_client.py

@@ -4,7 +4,7 @@ from logging import getLogger
 from typing import TYPE_CHECKING, Any
 
 from sqlalchemy import Select, insert, select
-from typing_extensions import override
+from typing_extensions import Self, override
 
 from crawlee.storage_clients._base import DatasetClient
 from crawlee.storage_clients.models import DatasetItemsListPage, DatasetMetadata
@@ -78,7 +78,7 @@ class SqlDatasetClient(DatasetClient, SqlClientMixin):
         name: str | None,
         alias: str | None,
         storage_client: SqlStorageClient,
-    ) -> SqlDatasetClient:
+    ) -> Self:
         """Open an existing dataset or create a new one.
 
         Args:

{crawlee-1.0.1b12 → crawlee-1.0.2}/src/crawlee/storage_clients/_sql/_key_value_store_client.py

@@ -5,7 +5,7 @@ from logging import getLogger
 from typing import TYPE_CHECKING, Any
 
 from sqlalchemy import delete, select
-from typing_extensions import override
+from typing_extensions import Self, override
 
 from crawlee._utils.file import infer_mime_type
 from crawlee.storage_clients._base import KeyValueStoreClient
@@ -77,7 +77,7 @@ class SqlKeyValueStoreClient(KeyValueStoreClient, SqlClientMixin):
         name: str | None,
         alias: str | None,
         storage_client: SqlStorageClient,
-    ) -> SqlKeyValueStoreClient:
+    ) -> Self:
         """Open or create a SQL key-value store client.
 
         This method attempts to open an existing key-value store from the SQL database. If a KVS with the specified

{crawlee-1.0.1b12 → crawlee-1.0.2}/src/crawlee/storage_clients/_sql/_request_queue_client.py

@@ -10,7 +10,7 @@ from typing import TYPE_CHECKING, Any
 from sqlalchemy import func, or_, select, update
 from sqlalchemy.exc import SQLAlchemyError
 from sqlalchemy.orm import load_only
-from typing_extensions import NotRequired, override
+from typing_extensions import NotRequired, Self, override
 
 from crawlee import Request
 from crawlee._utils.crypto import crypto_random_object_id
@@ -119,7 +119,7 @@ class SqlRequestQueueClient(RequestQueueClient, SqlClientMixin):
         name: str | None,
         alias: str | None,
         storage_client: SqlStorageClient,
-    ) -> SqlRequestQueueClient:
+    ) -> Self:
         """Open an existing request queue or create a new one.
 
         This method first tries to find an existing queue by ID or name.

{crawlee-1.0.1b12 → crawlee-1.0.2}/src/crawlee/storages/_base.py

@@ -44,7 +44,9 @@ class Storage(ABC):
 
         Args:
             id: The storage ID.
-            name: The storage name (global scope, persists across runs).
+            name: The storage name (global scope, persists across runs). Name can only contain letters "a" through "z",
+                the digits "0" through "9", and the hyphen ("-") but only in the middle of the string
+                (e.g. "my-value-1").
             alias: The storage alias (run scope, creates unnamed storage).
             configuration: Configuration object used during the storage creation or restoration process.
             storage_client: Underlying storage client to use. If not provided, the default global storage client

{crawlee-1.0.1b12 → crawlee-1.0.2}/src/crawlee/storages/_dataset.py

@@ -12,6 +12,7 @@ from crawlee._utils.file import export_csv_to_stream, export_json_to_stream
 
 from ._base import Storage
 from ._key_value_store import KeyValueStore
+from ._utils import validate_storage_name
 
 if TYPE_CHECKING:
     from collections.abc import AsyncIterator
@@ -75,6 +76,8 @@ class Dataset(Storage):
             id: The unique identifier of the storage.
             name: The name of the storage, if available.
         """
+        validate_storage_name(name)
+
         self._client = client
         self._id = id
         self._name = name

{crawlee-1.0.1b12 → crawlee-1.0.2}/src/crawlee/storages/_key_value_store.py

@@ -15,6 +15,7 @@ from crawlee._utils.recoverable_state import RecoverableState
 from crawlee.storage_clients.models import KeyValueStoreMetadata
 
 from ._base import Storage
+from ._utils import validate_storage_name
 
 if TYPE_CHECKING:
     from collections.abc import AsyncIterator
@@ -84,6 +85,8 @@ class KeyValueStore(Storage):
             id: The unique identifier of the storage.
             name: The name of the storage, if available.
         """
+        validate_storage_name(name)
+
         self._client = client
         self._id = id
         self._name = name

{crawlee-1.0.1b12 → crawlee-1.0.2}/src/crawlee/storages/_request_queue.py

@@ -13,6 +13,7 @@ from crawlee._utils.wait import wait_for_all_tasks_for_finish
 from crawlee.request_loaders import RequestManager
 
 from ._base import Storage
+from ._utils import validate_storage_name
 
 if TYPE_CHECKING:
     from collections.abc import Sequence
@@ -80,6 +81,8 @@ class RequestQueue(Storage, RequestManager):
             id: The unique identifier of the storage.
             name: The name of the storage, if available.
         """
+        validate_storage_name(name)
+
         self._client = client
         self._id = id
         self._name = name

{crawlee-1.0.1b12 → crawlee-1.0.2}/src/crawlee/storages/_storage_instance_manager.py

@@ -8,6 +8,8 @@ from typing import TYPE_CHECKING, TypeVar
 from crawlee._utils.raise_if_too_many_kwargs import raise_if_too_many_kwargs
 from crawlee.storage_clients._base import DatasetClient, KeyValueStoreClient, RequestQueueClient
 
+from ._utils import validate_storage_name
+
 if TYPE_CHECKING:
     from ._base import Storage
 
@@ -90,7 +92,9 @@ class StorageInstanceManager:
         Args:
             cls: The storage class to instantiate.
             id: Storage ID.
-            name: Storage name. (global scope, persists across runs).
+            name: Storage name. (global scope, persists across runs). Name can only contain letters "a" through "z",
+                the digits "0" through "9", and the hyphen ("-") but only in the middle of the string
+                (e.g. "my-value-1").
             alias: Storage alias (run scope, creates unnamed storage).
             client_opener_coro: Coroutine to open the storage client when storage instance not found in cache.
             storage_client_cache_key: Additional optional key from storage client to differentiate cache entries.
@@ -146,6 +150,10 @@ class StorageInstanceManager:
                     f'Use a different name or drop the existing alias storage first.'
                 )
 
+            # Validate storage name
+            if name is not None:
+                validate_storage_name(name)
+
             # Create new instance
             client: KeyValueStoreClient | DatasetClient | RequestQueueClient
             client = await client_opener_coro

crawlee-1.0.2/src/crawlee/storages/_utils.py

@@ -0,0 +1,11 @@
+import re
+
+NAME_REGEX = re.compile(r'^([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9-]*[a-zA-Z0-9])$')
+
+
+def validate_storage_name(name: str | None) -> None:
+    if name and not NAME_REGEX.match(name):
+        raise ValueError(
+            f'Invalid storage name "{name}". Name can only contain letters "a" through "z", the digits "0" through'
+            '"9", and the hyphen ("-") but only in the middle of the string (e.g. "my-value-1")'
+        )

{crawlee-1.0.1b12 → crawlee-1.0.2}/tests/unit/_utils/test_sitemap.py

@@ -1,7 +1,9 @@
 import base64
 import gzip
+import os
 from datetime import datetime
 
+import pytest
 from yarl import URL
 
 from crawlee._utils.sitemap import Sitemap, SitemapUrl, parse_sitemap
@@ -104,6 +106,10 @@ async def test_gzipped_sitemap_with_invalid_data(server_url: URL, http_client: H
     assert sitemap.urls == []
 
 
+@pytest.mark.skipif(
+    os.name == 'nt',
+    reason='This test is flaky on Windows, see https://github.com/apify/crawlee-python/issues/1460.',
+)
 async def test_gz_sitemap_with_non_gzipped(server_url: URL, http_client: HttpClient) -> None:
     """Test loading a sitemap with gzip type and .xml.gz url, but without gzipped data."""
     sitemap_url = (server_url / 'sitemap.xml.gz').with_query(

{crawlee-1.0.1b12 → crawlee-1.0.2}/tests/unit/crawlers/_adaptive_playwright/test_adaptive_playwright_crawler.py

@@ -473,7 +473,7 @@ async def test_adaptive_crawler_exceptions_in_sub_crawlers(*, error_in_pw_crawle
 async def test_adaptive_playwright_crawler_statistics_in_init() -> None:
     """Tests that adaptive crawler uses created AdaptivePlaywrightCrawlerStatistics from inputted Statistics."""
     persistence_enabled = True
-    persist_state_kvs_name = 'some name'
+    persist_state_kvs_name = 'some-name'
     persist_state_key = 'come key'
     log_message = 'some message'
     periodic_message_logger = logging.getLogger('some logger')

{crawlee-1.0.1b12 → crawlee-1.0.2}/tests/unit/otel/test_crawler_instrumentor.py

@@ -1,8 +1,10 @@
 import io
 import json
+import os
 import re
 from unittest import mock
 
+import pytest
 from opentelemetry.sdk.resources import Resource
 from opentelemetry.sdk.trace import TracerProvider
 from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
@@ -14,6 +16,10 @@ from crawlee.otel.crawler_instrumentor import CrawlerInstrumentor
 from crawlee.storages import Dataset
 
 
+@pytest.mark.skipif(
+    os.name == 'nt',
+    reason='This test is flaky on Windows, see https://github.com/apify/crawlee-python/issues/1469.',
+)
 async def test_crawler_instrumentor_capability(server_url: URL) -> None:
     """Test OpenTelemetry instrumentation capability of the crawler.
 
@@ -38,7 +44,7 @@ async def test_crawler_instrumentor_capability(server_url: URL) -> None:
 
     # Generate first telemetry data from `Dataset` public methods.
     # `Dataset` is in `instrument_classes` argument, and thus it's public methods are instrumented.
-    dataset = await Dataset.open(name='test_dataset')
+    dataset = await Dataset.open(name='test-dataset')
     await dataset.drop()
 
     # Other traces will be from crawler run.

{crawlee-1.0.1b12 → crawlee-1.0.2}/tests/unit/sessions/test_session_pool.py

@@ -17,7 +17,7 @@ if TYPE_CHECKING:
     from collections.abc import AsyncGenerator
 
 MAX_POOL_SIZE = 3
-KVS_NAME = 'test_session_pool'
+KVS_NAME = 'test-session-pool'
 PERSIST_STATE_KEY = 'crawlee_session_pool_state'