scrapely-client-python 1.0.7__tar.gz → 1.0.9__tar.gz

{scrapely_client_python-1.0.7 → scrapely_client_python-1.0.9}/CHANGELOG.md

@@ -2,6 +2,16 @@
 
 All notable changes to this project will be documented in this file.
 
+## [1.0.9] - 2026-10-24
+
+- The `SCRAPELY_API_URL` environment variable sets the API URL when `api_url` is not passed to `ScrapelyClient`, `ScrapelyClientAsync` or `with_custom_http_client`; `api_url` still wins, and without either the client uses `https://api.scrape.ly`. An empty value is ignored. `api_public_url` is unchanged.
+- Version 1.0.3 defaulted to `https://api-staging.scrape.ly`; upgrade to 1.0.6 or later.
+
+## [1.0.8] - 2026-10-20
+
+- Request ids are percent-encoded as one URL path segment in `get_request`, `update_request`, `delete_request`, `prolong_request_lock` and `delete_request_lock` (sync and async); `''`, `.` and `..` raise `ValueError`. `update_request` checks the id against the API's grammar (1-128 letters, digits, `-`, `_`, `~`) and refuses a request without an id instead of sending `requests/None`. Resource ids (`client.dataset(id)`, versions, env vars) and automation run ids are encoded the same way. An id such as `..` used to send the call to another resource.
+- Models are generated from a committed, hash-checked copy of the OpenAPI specification; model docstrings updated, no behaviour change.
+
 ## [1.0.7] - 2026-10-20
 
 - License: README, CONTRIBUTING, the docs pages and code samples derived from apify-client-python, and `_resource_clients/__init__.py`, now carry an Apache-2.0 notice.

{scrapely_client_python-1.0.7 → scrapely_client_python-1.0.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: scrapely-client-python
-Version: 1.0.7
+Version: 1.0.9
 Summary: Scrapely API client for Python
 Keywords: scrapely,api,client,automation,crawling,scraping
 Author: TeraShift GmbH (Scrape.ly)
@@ -94,6 +94,8 @@ Description-Content-Type: text/markdown
 
 You'll need a Scrapely API token — find yours in the integrations section of the Scrapely dashboard. Pass it to the client and you're ready to go.
 
+The client sends requests to `https://api.scrape.ly` unless you pass another API origin as `api_url`. When `api_url` is not given, the `SCRAPELY_API_URL` environment variable sets it (for example `SCRAPELY_API_URL=https://api-staging.scrape.ly`). `api_public_url`, used for public storage links, keeps its own argument and default. The client does not read the token from the environment.
+
 ### Synchronous client
 
 ```python
@@ -190,6 +192,21 @@ for item in client.dataset('DATASET-ID').iterate_items(fields=['title', 'url']):
     process(item)
 ```
 
+### Create and run an automation
+
+```python
+automation = client.automations().create({
+    'title': 'Daily dentists',
+    'trigger': {'kind': 'schedule', 'cron': '0 9 * * *', 'timezone': 'Europe/Berlin'},
+    'steps': [{'kind': 'savedInput', 'savedInputId': 'SAVED-INPUT-ID'}],
+    'ai': {'mode': 'summarize', 'prompt': 'List the new practices with their phone numbers.'},
+    'deliveries': [{'channel': 'email'}],
+})
+run = client.automation(automation['id']).run()
+```
+
+Automations take and return the API's camelCase dictionaries (see the automations section of the API reference).
+
 ### Tune retries and timeouts
 
 ```python

{scrapely_client_python-1.0.7 → scrapely_client_python-1.0.9}/README.md

@@ -61,6 +61,8 @@
 
 You'll need a Scrapely API token — find yours in the integrations section of the Scrapely dashboard. Pass it to the client and you're ready to go.
 
+The client sends requests to `https://api.scrape.ly` unless you pass another API origin as `api_url`. When `api_url` is not given, the `SCRAPELY_API_URL` environment variable sets it (for example `SCRAPELY_API_URL=https://api-staging.scrape.ly`). `api_public_url`, used for public storage links, keeps its own argument and default. The client does not read the token from the environment.
+
 ### Synchronous client
 
 ```python
@@ -157,6 +159,21 @@ for item in client.dataset('DATASET-ID').iterate_items(fields=['title', 'url']):
     process(item)
 ```
 
+### Create and run an automation
+
+```python
+automation = client.automations().create({
+    'title': 'Daily dentists',
+    'trigger': {'kind': 'schedule', 'cron': '0 9 * * *', 'timezone': 'Europe/Berlin'},
+    'steps': [{'kind': 'savedInput', 'savedInputId': 'SAVED-INPUT-ID'}],
+    'ai': {'mode': 'summarize', 'prompt': 'List the new practices with their phone numbers.'},
+    'deliveries': [{'channel': 'email'}],
+})
+run = client.automation(automation['id']).run()
+```
+
+Automations take and return the API's camelCase dictionaries (see the automations section of the API reference).
+
 ### Tune retries and timeouts
 
 ```python

{scrapely_client_python-1.0.7 → scrapely_client_python-1.0.9}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "uv_build"
 
 [project]
 name = "scrapely-client-python"
-version = "1.0.7"
+version = "1.0.9"
 description = "Scrapely API client for Python"
 license = "Apache-2.0"
 license-files = [
@@ -236,7 +236,7 @@ context = 7
 
 [tool.datamodel-codegen]
 input_file_type = "openapi"
-output = "src/scrapely_client/_models.py"
+output = "tmp/generated/_models.py"
 target_python_version = "3.11"
 output_model_type = "pydantic_v2.BaseModel"
 use_schema_description = true
@@ -262,7 +262,8 @@ disable_timestamp = true
 keep_model_order = true
 
 [tool.scrapely.openapi-spec]
-version = "v2-2026-08-05T133145Z"
+version = "scrapely-v2-2026-10-20"
+sha256 = "0913387f7deb9064489d43ec9aa345439ce3cfa96b5f29748f46220a65c9fe5e"
 
 [tool.uv]
 exclude-newer = "24 hours"
@@ -305,10 +306,6 @@ shell = "uv run ruff format --check && uv run ruff check"
 [tool.poe.tasks.format]
 shell = "uv run ruff check --fix && uv run ruff format"
 
-[tool.poe.tasks.update-docs-theme]
-shell = "pnpm update @apify/docs-theme"
-cwd = "website"
-
 [tool.poe.tasks.build-docs]
 shell = "./build_api_reference.sh && pnpm install && uv run pnpm build"
 cwd = "website"
@@ -319,12 +316,15 @@ cwd = "website"
 
 [tool.poe.tasks.generate-models]
 shell = """
-uv run python -m scripts.openapi_spec fetch && uv run datamodel-codegen --input tmp/openapi.json --alias-generator to_camel && uv run datamodel-codegen --input tmp/openapi.json --output src/scrapely_client/_typeddicts.py --output-model-type typing.TypedDict --no-use-closed-typed-dict && python scripts/postprocess_generated_models.py && uv run python -m scripts.openapi_spec record-version
+uv run python -m scripts.openapi_spec verify && uv run python -m scripts.openapi_spec clean-output && uv run datamodel-codegen --input openapi/scrapely-api.json --output tmp/generated/_models.py --alias-generator to_camel && uv run datamodel-codegen --input openapi/scrapely-api.json --output tmp/generated/_typeddicts.py --output-model-type typing.TypedDict --no-use-closed-typed-dict && uv run python -m scripts.postprocess_generated_models
 """
 
+[tool.poe.tasks.accept-openapi-spec]
+cmd = "uv run python -m scripts.openapi_spec accept"
+
 [tool.poe.tasks.generate-models-from-file]
 shell = """
-uv run datamodel-codegen --input $input_file --alias-generator to_camel && uv run datamodel-codegen --input $input_file --output src/scrapely_client/_typeddicts.py --output-model-type typing.TypedDict --no-use-closed-typed-dict && python scripts/postprocess_generated_models.py && echo "Generated from $input_file - the version in [tool.scrapely.openapi-spec] still names the published specification, so do not commit these models."
+uv run python -m scripts.openapi_spec check-file "$input_file" && uv run python -m scripts.openapi_spec clean-output && uv run datamodel-codegen --input "$input_file" --output tmp/generated/_models.py --alias-generator to_camel && uv run datamodel-codegen --input "$input_file" --output tmp/generated/_typeddicts.py --output-model-type typing.TypedDict --no-use-closed-typed-dict && uv run python -m scripts.postprocess_generated_models && echo "Generated from $input_file, not from the committed specification - do not commit these models."
 """
 
 [[tool.poe.tasks.generate-models-from-file.args]]

{scrapely_client_python-1.0.7 → scrapely_client_python-1.0.9}/pyproject.toml.orig

@@ -4,7 +4,7 @@ build-backend = "uv_build"
 
 [project]
 name = "scrapely-client-python"
-version = "1.0.7"
+version = "1.0.9"
 description = "Scrapely API client for Python"
 authors = [{ name = "TeraShift GmbH (Scrape.ly)" }]
 license = "Apache-2.0"
@@ -224,7 +224,8 @@ context = 7
 # https://koxudaxi.github.io/datamodel-code-generator/
 [tool.datamodel-codegen]
 input_file_type = "openapi"
-output = "src/scrapely_client/_models.py"
+# Codegen never writes into the package: `scripts/postprocess_generated_models.py` checks this output and installs it.
+output = "tmp/generated/_models.py"
 target_python_version = "3.11"
 output_model_type = "pydantic_v2.BaseModel"
 use_schema_description = true
@@ -246,16 +247,12 @@ custom_file_header = "# generated by datamodel-codegen"
 disable_timestamp = true
 keep_model_order = true
 
-# Which published OpenAPI specification the committed `_models.py`, `_typeddicts.py`, and `_literals.py` were
-# generated from. A coarse marker, not a content identity: the specification is served latest-only, so the version
-# can't fetch that document back, and the docs repository bumps `components/version.yaml` in a follow-up `[skip ci]` commit,
-# so a deploy can publish new content under the old stamp.
-#
-# `poe generate-models` records whatever it downloaded, unconditionally; what keeps the value honest is the nightly
-# workflow committing nothing unless the models changed. A local regeneration that only moves this line is that
-# same case - drop it rather than committing it alone.
+# Identity of the reviewed OpenAPI specification in `openapi/scrapely-api.json`, the only input of
+# `poe generate-models`. `poe generate-models` refuses to run when the file doesn't match this hash; after reviewing
+# a change to the specification, record it with `poe accept-openapi-spec` (written by `scripts/openapi_spec.py`).
 [tool.scrapely.openapi-spec]
-version = "v2-2026-08-05T133145Z"
+version = "scrapely-v2-2026-10-20"
+sha256 = "0913387f7deb9064489d43ec9aa345439ce3cfa96b5f29748f46220a65c9fe5e"
 
 [tool.uv]
 # Minimal defense against supply-chain atatcks.
@@ -289,10 +286,6 @@ shell = "uv run ruff format --check && uv run ruff check"
 [tool.poe.tasks.format]
 shell = "uv run ruff check --fix && uv run ruff format"
 
-[tool.poe.tasks.update-docs-theme]
-shell = "pnpm update @apify/docs-theme"
-cwd = "website"
-
 [tool.poe.tasks.build-docs]
 shell = "./build_api_reference.sh && pnpm install && uv run pnpm build"
 cwd = "website"
@@ -301,35 +294,45 @@ cwd = "website"
 shell = "./build_api_reference.sh && pnpm install && uv run pnpm start"
 cwd = "website"
 
-# Both passes read one downloaded copy, so a specification redeployed mid-run can't leave the generated files
-# describing two different inputs. The version is recorded last, once generation succeeded.
+# Generates only from the committed specification, after checking it against the hash recorded above; nothing is
+# downloaded. Codegen writes into git-ignored `tmp/generated/`, and the post-processor writes the package files only
+# once all three contain declarations only.
 #
 # The `--alias-generator to_camel` flag lives on the `_models.py` command (not in `[tool.datamodel-codegen]`)
 # because datamodel-codegen only allows it for `pydantic_v2.BaseModel` output and would reject the TypedDict run.
 [tool.poe.tasks.generate-models]
 shell = """
-uv run python -m scripts.openapi_spec fetch \
-    && uv run datamodel-codegen --input tmp/openapi.json \
+uv run python -m scripts.openapi_spec verify \
+    && uv run python -m scripts.openapi_spec clean-output \
+    && uv run datamodel-codegen --input openapi/scrapely-api.json \
+        --output tmp/generated/_models.py \
         --alias-generator to_camel \
-    && uv run datamodel-codegen --input tmp/openapi.json \
-        --output src/scrapely_client/_typeddicts.py \
+    && uv run datamodel-codegen --input openapi/scrapely-api.json \
+        --output tmp/generated/_typeddicts.py \
         --output-model-type typing.TypedDict \
         --no-use-closed-typed-dict \
-    && python scripts/postprocess_generated_models.py \
-    && uv run python -m scripts.openapi_spec record-version
+    && uv run python -m scripts.postprocess_generated_models
 """
 
-# For a candidate specification that isn't published yet. The recorded version deliberately stays put - it names
-# the published specification - so models generated this way must not be committed, hence the warning.
+# Records the hash and version of `openapi/scrapely-api.json` after a reviewed change to it.
+[tool.poe.tasks.accept-openapi-spec]
+cmd = "uv run python -m scripts.openapi_spec accept"
+
+# For trying out a candidate specification from a local file. The recorded hash still names the committed
+# specification, so models generated this way must not be committed: update `openapi/scrapely-api.json`, accept
+# it, and run `generate-models` instead.
 [tool.poe.tasks.generate-models-from-file]
 shell = """
-uv run datamodel-codegen --input $input_file \
+uv run python -m scripts.openapi_spec check-file "$input_file" \
+    && uv run python -m scripts.openapi_spec clean-output \
+    && uv run datamodel-codegen --input "$input_file" \
+        --output tmp/generated/_models.py \
         --alias-generator to_camel \
-    && uv run datamodel-codegen --input $input_file \
-        --output src/scrapely_client/_typeddicts.py \
+    && uv run datamodel-codegen --input "$input_file" \
+        --output tmp/generated/_typeddicts.py \
         --output-model-type typing.TypedDict \
         --no-use-closed-typed-dict \
-    && python scripts/postprocess_generated_models.py \
-    && echo "Generated from $input_file - the version in [tool.scrapely.openapi-spec] still names the published specification, so do not commit these models."
+    && uv run python -m scripts.postprocess_generated_models \
+    && echo "Generated from $input_file, not from the committed specification - do not commit these models."
 """
 args = [{ name = "input-file", positional = true, required = true }]

{scrapely_client_python-1.0.7 → scrapely_client_python-1.0.9}/src/scrapely_client/_consts.py

@@ -7,6 +7,9 @@ from datetime import timedelta
 DEFAULT_API_URL = 'https://api.scrape.ly'
 """Default base URL for the Scrapely API."""
 
+API_URL_ENV_VAR = 'SCRAPELY_API_URL'
+"""Environment variable that sets the API URL when `api_url` is not passed to the client."""
+
 API_VERSION = 'v2'
 """Current Scrapely API version."""
 

{scrapely_client_python-1.0.7 → scrapely_client_python-1.0.9}/src/scrapely_client/_models.py

@@ -51,9 +51,7 @@ class ActVersion(BaseModel):
     version_number: Annotated[
         str | None, Field(examples=['0.0'], pattern='^([0-9]|[1-9][0-9])\\.([0-9]|[1-9][0-9])(\\.[0-9]+)?$')
     ] = None
-    git_repo_url: Annotated[
-        str | None, Field(examples=['https://github.com/example/my-actor.git#main:src'])
-    ] = None
+    git_repo_url: Annotated[str | None, Field(examples=['https://github.com/example/my-actor.git#main:src'])] = None
     """
     URL of the git repository, present when sourceType is GIT_REPO.
     """
@@ -151,9 +149,7 @@ class Actor(BaseModel):
     """
     Description of the Actor to display by search engines such as Google.
     """
-    picture_url: Annotated[
-        str | None, Field(examples=['https://images.scrape.ly/.../actor-picture.png'])
-    ] = None
+    picture_url: Annotated[str | None, Field(examples=['https://images.scrape.ly/.../actor-picture.png'])] = None
     """
     URL of the Actor's icon, displayed on the Actor's page in Scrapely Store and dashboard.
     """
@@ -224,7 +220,7 @@ class ActorChargeEvent(BaseModel):
 
 @docs_group('Models')
 class ActorDefinition(BaseModel):
-    """The definition of the Actor, the full specification of this field can be found in the upstream API docs."""
+    """The definition of the Actor, as declared in its Actor configuration file (for example `.actor/actor.json`)."""
 
     model_config = ConfigDict(
         extra='allow',
@@ -265,7 +261,7 @@ class ActorDefinition(BaseModel):
     """
     input: dict[str, Any] | None = None
     """
-    The input schema object, the full specification can be found in the upstream API docs
+    The input schema object.
     """
     changelog: str | None = None
     """
@@ -988,7 +984,7 @@ class Dataset(BaseModel):
         ),
     ] = None
     """
-    Defines the schema of items in your dataset, the full specification can be found in the upstream API docs
+    Defines the schema of items in your dataset.
     """
     console_url: Annotated[AnyUrl, Field(examples=['https://api.scrape.ly/storage/datasets/27TmTznX9YPeAYhkC'])]
     items_public_url: Annotated[
@@ -1290,7 +1286,11 @@ class EncodeAndSignData(BaseModel):
 
 @docs_group('Models')
 class DecodeAndVerifyRequest(EncodeAndSignData):
-    pass
+    model_config = ConfigDict(
+        extra='allow',
+        populate_by_name=True,
+        alias_generator=to_camel,
+    )
 
 
 @docs_group('Models')
@@ -1978,8 +1978,10 @@ class MonthlyUsage(BaseModel):
         alias_generator=to_camel,
     )
     usage_cycle: UsageCycle
-    # Scrapely emits raw byte counts (int) for the `DATA_TRANSFER_*_BYTES` keys
     monthly_service_usage: dict[str, UsageItem | int]
+    """
+    Scrapely emits raw byte counts (integers) for the `DATA_TRANSFER_*_BYTES` keys.
+    """
     daily_service_usages: list[DailyServiceUsages]
     total_usage_credits_usd_before_volume_discount: Annotated[float, Field(examples=[0.786143673840067])]
     total_usage_credits_usd_after_volume_discount: Annotated[float, Field(examples=[0.786143673840067])]
@@ -2133,9 +2135,14 @@ class ListOfStoreActors(PaginationResponse):
         alias_generator=to_camel,
     )
     items: list[StoreListActor]
-    # Scrapely's store envelope omits `desc` and `count`; tolerate their absence.
     desc: Annotated[bool, Field(examples=[False])] = False
+    """
+    Whether the results are sorted in descending order. The store listing may omit it.
+    """
     count: Annotated[int, Field(examples=[1], ge=0)] = 0
+    """
+    The number of items returned in this response. The store listing may omit it.
+    """
 
 
 @docs_group('Models')
@@ -2285,7 +2292,9 @@ class Profile(BaseModel):
     )
     bio: Annotated[str | None, Field(examples=['I started web scraping in 1985 using Altair BASIC.'])] = None
     name: Annotated[str | None, Field(examples=['Jane Doe'])] = None
-    picture_url: Annotated[AnyUrl | None, Field(examples=['https://api.scrape.ly/img/anonymous_user_picture.png'])] = None
+    picture_url: Annotated[AnyUrl | None, Field(examples=['https://api.scrape.ly/img/anonymous_user_picture.png'])] = (
+        None
+    )
     github_username: Annotated[str | None, Field(examples=['torvalds.'])] = None
     website_url: Annotated[AnyUrl | None, Field(examples=['http://www.example.com'])] = None
     twitter_username: Annotated[str | None, Field(examples=['@BillGates'])] = None
@@ -3414,7 +3423,7 @@ class Storages(BaseModel):
     )
     dataset: dict[str, Any] | None = None
     """
-    Defines the schema of items in your dataset, the full specification can be found in the upstream API docs
+    Defines the schema of items in your dataset.
     """
 
 
@@ -3426,8 +3435,10 @@ class StoreListActor(BaseModel):
         alias_generator=to_camel,
     )
     id: Annotated[str, Field(examples=['zdc3Pyhyz3m8vjDeM'])]
-    # Scrapely can return null titles; tolerate their absence.
     title: Annotated[str | None, Field(examples=['My Public Actor'])] = None
+    """
+    Scrapely can return a `null` title.
+    """
     name: Annotated[str, Field(examples=['my-public-actor'])]
     username: Annotated[str, Field(examples=['jane35'])]
     user_full_name: Annotated[str | None, Field(examples=['Jane H. Doe'])] = None
@@ -3752,7 +3763,7 @@ class UpdateLimitsRequest(BaseModel):
     """
     data_retention_days: Annotated[int | None, Field(examples=[90])] = None
     """
-    How many days unnamed storages (datasets, key-value stores and request queues without a name) are kept: Free 2 days (48 hours), Starter 31, Scale 60, Business 90. Runs follow their own retention: on the Free plan the latest 10 runs per Actor are kept and older ones are deleted after 7 days; on paid plans runs are deleted after 31 days. This value is set by your plan and cannot be changed through the API.
+    How many days unnamed storages (datasets, key-value stores and request queues without a name) are kept: Free 2 days (48 hours), Starter 31, Scale 60, Business 90. Runs and their logs are kept for the same time as their results; on the Free plan your latest 10 runs are kept with their results until you delete them. This value is set by your plan and cannot be changed through the API.
 
     """
 
@@ -3800,7 +3811,11 @@ class UpdateRunRequest(BaseModel):
 
 @docs_group('Models')
 class UpdateStoreRequest(UpdateDatasetRequest):
-    pass
+    model_config = ConfigDict(
+        extra='allow',
+        populate_by_name=True,
+        alias_generator=to_camel,
+    )
 
 
 @docs_group('Models')
@@ -3909,7 +3924,9 @@ class Version(BaseModel):
         populate_by_name=True,
         alias_generator=to_camel,
     )
-    version_number: Annotated[str, Field(examples=['0.0'], pattern='^([0-9]|[1-9][0-9])\\.([0-9]|[1-9][0-9])(\\.[0-9]+)?$')]
+    version_number: Annotated[
+        str, Field(examples=['0.0'], pattern='^([0-9]|[1-9][0-9])\\.([0-9]|[1-9][0-9])(\\.[0-9]+)?$')
+    ]
     """
     The version number of the Actor. `MAJOR.MINOR` or `MAJOR.MINOR.PATCH` (Scrapely staging uses triplets like `0.0.1`).
     """
@@ -4037,7 +4054,11 @@ class WebhookDispatch(BaseModel):
 
 @docs_group('Models')
 class WebhookDispatchResponse(TestWebhookResponse):
-    pass
+    model_config = ConfigDict(
+        extra='allow',
+        populate_by_name=True,
+        alias_generator=to_camel,
+    )
 
 
 @docs_group('Models')

{scrapely_client_python-1.0.7 → scrapely_client_python-1.0.9}/src/scrapely_client/_resource_clients/_resource_client.py

@@ -13,7 +13,7 @@ from scrapely_client._consts import DEFAULT_WAIT_FOR_FINISH, DEFAULT_WAIT_WHEN_J
 from scrapely_client._docs import docs_group
 from scrapely_client._logging import WithLogDetailsClient
 from scrapely_client._utils.errors import catch_not_found_for_resource_or_throw, catch_not_found_or_throw
-from scrapely_client._utils.http import response_to_dict, to_safe_id
+from scrapely_client._utils.http import response_to_dict, to_path_segment, to_safe_id
 from scrapely_client._utils.time import to_seconds
 from scrapely_client.errors import ApifyApiError
 
@@ -77,7 +77,8 @@ class ResourceClientBase(metaclass=WithLogDetailsClient):
         """Build the full resource URL from base URL, path, and optional ID."""
         url = f'{self._base_url}/{self._resource_path}'
         if self._resource_id is not None:
-            url = f'{url}/{to_safe_id(self._resource_id)}'
+            # One path segment whatever the id holds: `..`, `?` or `#` must not retarget the call.
+            url = f'{url}/{to_path_segment(to_safe_id(self._resource_id))}'
         return url
 
     @cached_property

{scrapely_client_python-1.0.7 → scrapely_client_python-1.0.9}/src/scrapely_client/_resource_clients/automation.py

@@ -3,12 +3,11 @@
 from __future__ import annotations
 
 from typing import TYPE_CHECKING, Any
-from urllib.parse import quote
 
 from scrapely_client._docs import docs_group
 from scrapely_client._resource_clients._resource_client import ResourceClient, ResourceClientAsync
 from scrapely_client._utils.errors import catch_not_found_or_throw
-from scrapely_client._utils.http import response_to_dict
+from scrapely_client._utils.http import response_to_dict, to_path_segment
 from scrapely_client.errors import ApifyApiError
 
 if TYPE_CHECKING:
@@ -134,7 +133,7 @@ class AutomationClient(ResourceClient):
         """
         try:
             response = self._http_client.call(
-                url=self._build_url(f'runs/{quote(run_id, safe="")}'),
+                url=self._build_url(f'runs/{to_path_segment(run_id)}'),
                 method='GET',
                 params=self._build_params(),
                 timeout=timeout,
@@ -290,7 +289,7 @@ class AutomationClientAsync(ResourceClientAsync):
         """
         try:
             response = await self._http_client.call(
-                url=self._build_url(f'runs/{quote(run_id, safe="")}'),
+                url=self._build_url(f'runs/{to_path_segment(run_id)}'),
                 method='GET',
                 params=self._build_params(),
                 timeout=timeout,

{scrapely_client_python-1.0.7 → scrapely_client_python-1.0.9}/src/scrapely_client/_resource_clients/request_queue.py

@@ -5,6 +5,7 @@ from __future__ import annotations
 import asyncio
 import json
 import math
+import re
 from collections.abc import Iterable
 from queue import Queue
 from typing import TYPE_CHECKING, Any, Literal
@@ -40,7 +41,7 @@ from scrapely_client._models import (
 from scrapely_client._pagination import DEFAULT_CHUNK_SIZE, get_cursor_iterator, get_cursor_iterator_async
 from scrapely_client._resource_clients._resource_client import ResourceClient, ResourceClientAsync
 from scrapely_client._utils.errors import catch_not_found_or_throw
-from scrapely_client._utils.http import response_to_dict
+from scrapely_client._utils.http import response_to_dict, to_path_segment
 from scrapely_client._utils.time import to_seconds
 from scrapely_client.errors import ApifyApiError
 
@@ -68,6 +69,32 @@ _MAX_PAYLOAD_SIZE_BYTES = 9 * 1024 * 1024
 _SAFETY_BUFFER_PERCENT = 0.01 / 100
 """Safety margin (0.01%) deducted from the maximum payload size when splitting requests into batches."""
 
+_REQUEST_ID_PATTERN = re.compile(r'[A-Za-z0-9_~-]{1,128}')
+"""Request ids the API accepts: letters, digits, `-`, `_` and `~`, at most 128 characters."""
+
+
+def _request_path(request_id: str, suffix: str = '') -> str:
+    """Build the path of one request, with the id percent-encoded as a single path segment.
+
+    Request ids can come from queue contents written by someone else. Unencoded, an id such as `..`,
+    `../../key-value-stores/<id>/records/INPUT` or `a?b` would point the authenticated call at another resource.
+    """
+    if not isinstance(request_id, str):
+        raise TypeError(f'Request id must be a string, not {type(request_id).__name__}.')
+    return f'requests/{to_path_segment(request_id)}{suffix}'
+
+
+def _stored_request_id(request: Request) -> str:
+    """Return the id of a request passed to `update_request`, after checking it against the API's id grammar."""
+    request_id = request.id
+    if request_id is None:
+        raise ValueError('The request has no id. update_request needs the id of a request stored in the queue.')
+    if not isinstance(request_id, str) or not _REQUEST_ID_PATTERN.fullmatch(request_id):
+        raise ValueError(
+            f'Invalid request id {request_id!r}: use 1-128 letters, digits, "-", "_" or "~", as returned by the API.'
+        )
+    return request_id
+
 
 def _serialize_requests(
     requests: list[RequestDraft] | list[RequestDraftDict] | list[RequestDraftCamelDict],
@@ -270,7 +297,7 @@ class RequestQueueClient(ResourceClient):
         """
         try:
             response = self._http_client.call(
-                url=self._build_url(f'requests/{request_id}'),
+                url=self._build_url(_request_path(request_id)),
                 method='GET',
                 params=self._build_params(),
                 timeout=timeout,
@@ -303,11 +330,12 @@ class RequestQueueClient(ResourceClient):
         """
         if not isinstance(request, Request):
             request = Request.model_validate(request)
+        request_id = _stored_request_id(request)
 
         request_params = self._build_params(forefront=forefront, clientKey=self.client_key)
 
         response = self._http_client.call(
-            url=self._build_url(f'requests/{request.id}'),
+            url=self._build_url(_request_path(request_id)),
             method='PUT',
             json=request.model_dump(by_alias=True, exclude_none=True),
             params=request_params,
@@ -330,7 +358,7 @@ class RequestQueueClient(ResourceClient):
         )
 
         self._http_client.call(
-            url=self._build_url(f'requests/{request_id}'),
+            url=self._build_url(_request_path(request_id)),
             method='DELETE',
             params=request_params,
             timeout=timeout,
@@ -360,7 +388,7 @@ class RequestQueueClient(ResourceClient):
         )
 
         response = self._http_client.call(
-            url=self._build_url(f'requests/{request_id}/lock'),
+            url=self._build_url(_request_path(request_id, '/lock')),
             method='PUT',
             params=request_params,
             timeout=timeout,
@@ -387,7 +415,7 @@ class RequestQueueClient(ResourceClient):
         request_params = self._build_params(clientKey=self.client_key, forefront=forefront)
 
         self._http_client.call(
-            url=self._build_url(f'requests/{request_id}/lock'),
+            url=self._build_url(_request_path(request_id, '/lock')),
             method='DELETE',
             params=request_params,
             timeout=timeout,
@@ -781,7 +809,7 @@ class RequestQueueClientAsync(ResourceClientAsync):
         """
         try:
             response = await self._http_client.call(
-                url=self._build_url(f'requests/{request_id}'),
+                url=self._build_url(_request_path(request_id)),
                 method='GET',
                 params=self._build_params(),
                 timeout=timeout,
@@ -812,11 +840,12 @@ class RequestQueueClientAsync(ResourceClientAsync):
         """
         if not isinstance(request, Request):
             request = Request.model_validate(request)
+        request_id = _stored_request_id(request)
 
         request_params = self._build_params(forefront=forefront, clientKey=self.client_key)
 
         response = await self._http_client.call(
-            url=self._build_url(f'requests/{request.id}'),
+            url=self._build_url(_request_path(request_id)),
             method='PUT',
             json=request.model_dump(by_alias=True, exclude_none=True),
             params=request_params,
@@ -837,7 +866,7 @@ class RequestQueueClientAsync(ResourceClientAsync):
         request_params = self._build_params(clientKey=self.client_key)
 
         await self._http_client.call(
-            url=self._build_url(f'requests/{request_id}'),
+            url=self._build_url(_request_path(request_id)),
             method='DELETE',
             params=request_params,
             timeout=timeout,
@@ -867,7 +896,7 @@ class RequestQueueClientAsync(ResourceClientAsync):
         )
 
         response = await self._http_client.call(
-            url=self._build_url(f'requests/{request_id}/lock'),
+            url=self._build_url(_request_path(request_id, '/lock')),
             method='PUT',
             params=request_params,
             timeout=timeout,
@@ -894,7 +923,7 @@ class RequestQueueClientAsync(ResourceClientAsync):
         request_params = self._build_params(clientKey=self.client_key, forefront=forefront)
 
         await self._http_client.call(
-            url=self._build_url(f'requests/{request_id}/lock'),
+            url=self._build_url(_request_path(request_id, '/lock')),
             method='DELETE',
             params=request_params,
             timeout=timeout,

{scrapely_client_python-1.0.7 → scrapely_client_python-1.0.9}/src/scrapely_client/_scrapely_client.py

@@ -2,12 +2,14 @@
 # Modified by TeraShift GmbH. See the LICENSE and NOTICE files.
 from __future__ import annotations
 
+import os
 from functools import cached_property
 from typing import TYPE_CHECKING
 
 from scrapely_client._charging import ChargingHooks, wrap_dataset_client_with_charging
 from scrapely_client._client_registry import ClientRegistry, ClientRegistryAsync
 from scrapely_client._consts import (
+    API_URL_ENV_VAR,
     API_VERSION,
     DEFAULT_API_URL,
     DEFAULT_MAX_RETRIES,
@@ -90,6 +92,13 @@ if TYPE_CHECKING:
     from scrapely_client.types import HttpCompressionAlgorithm
 
 
+def _resolve_api_url(api_url: str | None) -> str:
+    """Return `api_url` if given, else the `SCRAPELY_API_URL` environment variable, else the production API."""
+    if api_url is not None:
+        return api_url
+    return os.environ.get(API_URL_ENV_VAR, '').strip() or DEFAULT_API_URL
+
+
 @docs_group('Scrapely API clients')
 class ApifyClient:
     """Synchronous client for the Scrapely API.
@@ -124,7 +133,7 @@ class ApifyClient:
         self,
         token: str | None = None,
         *,
-        api_url: str = DEFAULT_API_URL,
+        api_url: str | None = None,
         api_public_url: str | None = DEFAULT_API_URL,
         max_retries: int = DEFAULT_MAX_RETRIES,
         min_delay_between_retries: timedelta = DEFAULT_MIN_DELAY_BETWEEN_RETRIES,
@@ -144,9 +153,9 @@ class ApifyClient:
         Args:
             token: The Scrapely API token. You can find your token on the
                 Integrations page in the Scrapely dashboard.
-            api_url: The URL of the Scrapely API server to connect to. Defaults to https://api.scrape.ly. It can
-                be an internal URL that is not globally accessible, in which case `api_public_url` should be set
-                as well.
+            api_url: The URL of the Scrapely API server to connect to. When not given, the `SCRAPELY_API_URL`
+                environment variable is used, then https://api.scrape.ly. It can be an internal URL that is not
+                globally accessible, in which case `api_public_url` should be set as well.
             api_public_url: The globally accessible URL of the Scrapely API server. Should be set only if `api_url`
                 is an internal URL that is not globally accessible. Defaults to https://api.scrape.ly.
             max_retries: How many times to retry a failed request at most.
@@ -165,7 +174,7 @@ class ApifyClient:
             default_dataset_id: ID of the actor run's default dataset, used to gate charging interception.
         """
         # We need to do this because of mocking in tests and default mutable arguments.
-        api_url = DEFAULT_API_URL if api_url is None else api_url
+        api_url = _resolve_api_url(api_url)
         api_public_url = DEFAULT_API_URL if api_public_url is None else api_public_url
 
         if headers:
@@ -234,7 +243,7 @@ class ApifyClient:
         cls,
         token: str | None = None,
         *,
-        api_url: str = DEFAULT_API_URL,
+        api_url: str | None = None,
         api_public_url: str | None = DEFAULT_API_URL,
         http_client: HttpClient,
     ) -> ApifyClient:
@@ -263,7 +272,8 @@ class ApifyClient:
         Args:
             token: The Scrapely API token. It is set as the `Authorization` header on the custom client,
                 unless the client already has one configured.
-            api_url: The URL of the Scrapely API server to connect to. Defaults to https://api.scrape.ly.
+            api_url: The URL of the Scrapely API server to connect to. When not given, the `SCRAPELY_API_URL`
+                environment variable is used, then https://api.scrape.ly.
             api_public_url: The globally accessible URL of the Scrapely API server. Defaults to https://api.scrape.ly.
             http_client: A custom HTTP client instance extending `HttpClient`.
         """
@@ -513,7 +523,7 @@ class ApifyClientAsync:
         self,
         token: str | None = None,
         *,
-        api_url: str = DEFAULT_API_URL,
+        api_url: str | None = None,
         api_public_url: str | None = DEFAULT_API_URL,
         max_retries: int = DEFAULT_MAX_RETRIES,
         min_delay_between_retries: timedelta = DEFAULT_MIN_DELAY_BETWEEN_RETRIES,
@@ -533,9 +543,9 @@ class ApifyClientAsync:
         Args:
             token: The Scrapely API token. You can find your token on the
                 Integrations page in the Scrapely dashboard.
-            api_url: The URL of the Scrapely API server to connect to. Defaults to https://api.scrape.ly. It can
-                be an internal URL that is not globally accessible, in which case `api_public_url` should be set
-                as well.
+            api_url: The URL of the Scrapely API server to connect to. When not given, the `SCRAPELY_API_URL`
+                environment variable is used, then https://api.scrape.ly. It can be an internal URL that is not
+                globally accessible, in which case `api_public_url` should be set as well.
             api_public_url: The globally accessible URL of the Scrapely API server. Should be set only if `api_url`
                 is an internal URL that is not globally accessible. Defaults to https://api.scrape.ly.
             max_retries: How many times to retry a failed request at most.
@@ -554,7 +564,7 @@ class ApifyClientAsync:
             default_dataset_id: ID of the actor run's default dataset, used to gate charging interception.
         """
         # We need to do this because of mocking in tests and default mutable arguments.
-        api_url = DEFAULT_API_URL if api_url is None else api_url
+        api_url = _resolve_api_url(api_url)
         api_public_url = DEFAULT_API_URL if api_public_url is None else api_public_url
 
         if headers:
@@ -623,7 +633,7 @@ class ApifyClientAsync:
         cls,
         token: str | None = None,
         *,
-        api_url: str = DEFAULT_API_URL,
+        api_url: str | None = None,
         api_public_url: str | None = DEFAULT_API_URL,
         http_client: HttpClientAsync,
     ) -> ApifyClientAsync:
@@ -652,7 +662,8 @@ class ApifyClientAsync:
         Args:
             token: The Scrapely API token. It is set as the `Authorization` header on the custom client,
                 unless the client already has one configured.
-            api_url: The URL of the Scrapely API server to connect to. Defaults to https://api.scrape.ly.
+            api_url: The URL of the Scrapely API server to connect to. When not given, the `SCRAPELY_API_URL`
+                environment variable is used, then https://api.scrape.ly.
             api_public_url: The globally accessible URL of the Scrapely API server. Defaults to https://api.scrape.ly.
             http_client: A custom HTTP client instance extending `HttpClientAsync`.
         """