pipelex-api 0.89.0__tar.gz → 0.91.0__tar.gz

{pipelex_api-0.89.0 → pipelex_api-0.91.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: pipelex-api
-Version: 0.89.0
+Version: 0.91.0
 Summary: Pipelex API
 Project-URL: Homepage, https://pipelex.com
 Project-URL: Repository, https://github.com/Pipelex/pipelex
@@ -16,7 +16,7 @@ Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: 3.14
 Requires-Python: <3.15,>=3.11
 Requires-Dist: fastapi<0.142,>=0.118.0
-Requires-Dist: pipelex[anthropic,bedrock,fal,google,google-genai,mistralai]==0.89.0
+Requires-Dist: pipelex[anthropic,bedrock,fal,google,google-genai,mistralai]==0.91.0
 Requires-Dist: pyjwt>=2.10.1
 Requires-Dist: uvicorn>=0.37.0
 Provides-Extra: dev
@@ -115,26 +115,27 @@ The API is now running at `http://localhost:8081`. To customize behavior (enable
 
 # 🧪 Run your first pipeline
 
-Once `/health` is green, send an inline pipeline definition and inputs to `/v1/execute`. The example below summarizes a string with a one-pipe MTHDS bundle — no files, no auth, copy-paste:
+Once `/health` is green, send a method and its inputs to `/v1/execute`. The method travels in `files`, a map from each file's path in the method's folder to that file's content. The example below summarizes a string with a method made of a single `.mthds` file, with no auth, ready to copy and paste:
 
 ```bash
 curl -s http://localhost:8081/v1/execute \
   -H "Content-Type: application/json" \
   -d '{
-    "pipe_code": "summarize",
-    "mthds_contents": ["domain = \"hello\"\nmain_pipe = \"summarize\"\n\n[pipe.summarize]\ntype = \"PipeLLM\"\ndescription = \"Summarize the input text in one sentence\"\ninputs = { text = \"Text\" }\noutput = \"Text\"\nprompt = \"Summarize in one sentence:\\n@text\"\n"],
+    "files": {
+      "summarize.mthds": "domain = \"hello\"\nmain_pipe = \"summarize\"\n\n[pipe.summarize]\ntype = \"PipeLLM\"\ndescription = \"Summarize the input text in one sentence\"\ninputs = { text = \"Text\" }\noutput = \"Text\"\nprompt = \"Summarize in one sentence:\\n@text\"\n"
+    },
     "inputs": { "text": "Pipelex turns plain-language pipeline definitions into reproducible AI workflows that run as HTTP endpoints." }
   }'
 ```
 
-You'll get back a JSON response with `state: "COMPLETED"` and the summary under `pipe_output.working_memory.root.<main_stuff_name>.content`.
+You'll get back a JSON response with `state: "COMPLETED"` and the summary under `pipe_output.working_memory.root.<main_stuff_name>.content`. The run starts at the method's `main_pipe`; to start at another pipe of the method, name it in `pipe_code`.
 
 **Passing files (PDFs, images) as inputs.** Use the `Document` concept and point it at any HTTP(S) URL:
 
 ```json
 {
   "pipe_code": "your_pipe",
-  "mthds_contents": ["...your MTHDS..."],
+  "files": { "your_method.mthds": "...your MTHDS..." },
   "inputs": {
     "cv": { "concept": "Document", "content": { "url": "https://example.com/resume.pdf" } }
   }
@@ -143,7 +144,7 @@ You'll get back a JSON response with `state: "COMPLETED"` and the summary under
 
 `Document` accepts public HTTP/HTTPS URLs, `pipelex-storage://` URIs, or base64 data URLs. For images, use the `Image` concept with the same `{ "url": "..." }` shape.
 
-For inline MTHDS in the request, `mthds_contents` is a JSON array of raw `.mthds` (TOML) file contents as strings — typically `[open("my_pipe.mthds").read()]` from a client. See the [Pipe Run page](https://docs.pipelex.com/latest/api-server/pipe-run/) for every supported input shape and the full `/execute` reference.
+**Sending the method.** `files` holds the whole method, as the MTHDS standard's method tree: each key is a file's path relative to the method's folder, such as `summarize.mthds` or `templates/invoice.docx`, and each value is that file's content. A text file, such as a `.mthds` bundle or a Python module, is sent as its UTF-8 text in a JSON string, and a file that is not text, such as an office template or an image, as `{"base64": "..."}`. This server also keeps a blob store, which `GET /v1/version` reports as `blob_store: true`, so a client can send a file as `{"sha256": "...", "size": n}` instead: it asks `POST /v1/blobs/missing` which of its files the server lacks, uploads each of those to the target the answer names, and then sends the method with a hash entry for each file, so a method that has not changed is not uploaded again. The older `mthds_contents` field, a JSON array of `.mthds` texts, is still accepted, but it is being retired together with `bundle_b64` and `resources`, and a later release refuses them, so new clients send `files`. See the [Pipe Run page](https://docs.pipelex.com/latest/api-server/pipe-run/) for every supported input shape, the rules of [a method sent as a tree](https://docs.pipelex.com/latest/api-server/pipe-run/#sending-a-method-as-a-tree) and of [sending files by hash](https://docs.pipelex.com/latest/api-server/pipe-run/#sending-files-by-hash), and the full `/execute` reference.
 
 # 📈 How to scale Pipelex
 

{pipelex_api-0.89.0 → pipelex_api-0.91.0}/README.md

@@ -69,26 +69,27 @@ The API is now running at `http://localhost:8081`. To customize behavior (enable
 
 # 🧪 Run your first pipeline
 
-Once `/health` is green, send an inline pipeline definition and inputs to `/v1/execute`. The example below summarizes a string with a one-pipe MTHDS bundle — no files, no auth, copy-paste:
+Once `/health` is green, send a method and its inputs to `/v1/execute`. The method travels in `files`, a map from each file's path in the method's folder to that file's content. The example below summarizes a string with a method made of a single `.mthds` file, with no auth, ready to copy and paste:
 
 ```bash
 curl -s http://localhost:8081/v1/execute \
   -H "Content-Type: application/json" \
   -d '{
-    "pipe_code": "summarize",
-    "mthds_contents": ["domain = \"hello\"\nmain_pipe = \"summarize\"\n\n[pipe.summarize]\ntype = \"PipeLLM\"\ndescription = \"Summarize the input text in one sentence\"\ninputs = { text = \"Text\" }\noutput = \"Text\"\nprompt = \"Summarize in one sentence:\\n@text\"\n"],
+    "files": {
+      "summarize.mthds": "domain = \"hello\"\nmain_pipe = \"summarize\"\n\n[pipe.summarize]\ntype = \"PipeLLM\"\ndescription = \"Summarize the input text in one sentence\"\ninputs = { text = \"Text\" }\noutput = \"Text\"\nprompt = \"Summarize in one sentence:\\n@text\"\n"
+    },
     "inputs": { "text": "Pipelex turns plain-language pipeline definitions into reproducible AI workflows that run as HTTP endpoints." }
   }'
 ```
 
-You'll get back a JSON response with `state: "COMPLETED"` and the summary under `pipe_output.working_memory.root.<main_stuff_name>.content`.
+You'll get back a JSON response with `state: "COMPLETED"` and the summary under `pipe_output.working_memory.root.<main_stuff_name>.content`. The run starts at the method's `main_pipe`; to start at another pipe of the method, name it in `pipe_code`.
 
 **Passing files (PDFs, images) as inputs.** Use the `Document` concept and point it at any HTTP(S) URL:
 
 ```json
 {
   "pipe_code": "your_pipe",
-  "mthds_contents": ["...your MTHDS..."],
+  "files": { "your_method.mthds": "...your MTHDS..." },
   "inputs": {
     "cv": { "concept": "Document", "content": { "url": "https://example.com/resume.pdf" } }
   }
@@ -97,7 +98,7 @@ You'll get back a JSON response with `state: "COMPLETED"` and the summary under
 
 `Document` accepts public HTTP/HTTPS URLs, `pipelex-storage://` URIs, or base64 data URLs. For images, use the `Image` concept with the same `{ "url": "..." }` shape.
 
-For inline MTHDS in the request, `mthds_contents` is a JSON array of raw `.mthds` (TOML) file contents as strings — typically `[open("my_pipe.mthds").read()]` from a client. See the [Pipe Run page](https://docs.pipelex.com/latest/api-server/pipe-run/) for every supported input shape and the full `/execute` reference.
+**Sending the method.** `files` holds the whole method, as the MTHDS standard's method tree: each key is a file's path relative to the method's folder, such as `summarize.mthds` or `templates/invoice.docx`, and each value is that file's content. A text file, such as a `.mthds` bundle or a Python module, is sent as its UTF-8 text in a JSON string, and a file that is not text, such as an office template or an image, as `{"base64": "..."}`. This server also keeps a blob store, which `GET /v1/version` reports as `blob_store: true`, so a client can send a file as `{"sha256": "...", "size": n}` instead: it asks `POST /v1/blobs/missing` which of its files the server lacks, uploads each of those to the target the answer names, and then sends the method with a hash entry for each file, so a method that has not changed is not uploaded again. The older `mthds_contents` field, a JSON array of `.mthds` texts, is still accepted, but it is being retired together with `bundle_b64` and `resources`, and a later release refuses them, so new clients send `files`. See the [Pipe Run page](https://docs.pipelex.com/latest/api-server/pipe-run/) for every supported input shape, the rules of [a method sent as a tree](https://docs.pipelex.com/latest/api-server/pipe-run/#sending-a-method-as-a-tree) and of [sending files by hash](https://docs.pipelex.com/latest/api-server/pipe-run/#sending-files-by-hash), and the full `/execute` reference.
 
 # 📈 How to scale Pipelex
 

{pipelex_api-0.89.0 → pipelex_api-0.91.0}/pipelex_api/api_config.py

@@ -30,8 +30,7 @@ from pipelex.runtime_bridge.orchestration_mode import DIRECT_ORCHESTRATION_MODE
 from pipelex.system.configuration.config_loader import config_manager
 from pydantic import BaseModel, ConfigDict
 
-from pipelex_api.error_types import ErrorType
-from pipelex_api.errors import raise_forbidden
+from pipelex_api.exceptions import ApiBootConfigError, OrchestrationOverrideForbiddenError
 
 API_CONFIG_NAME = "api"
 
@@ -79,7 +78,7 @@ def get_api_config() -> ApiConfig:
     The config is immutable for the life of the process (``PIPELEX_ENV`` is fixed
     at boot), so it is loaded once and cached. ``pipelex_api.main`` warms this at startup
     so a malformed ``api.toml`` / baked override fails the app fast — the same
-    fail-fast posture as ``ERROR_DISCLOSURE``. Tests that need a different mode
+    fail-fast posture as ``AUTH_MODE``. Tests that need a different mode
     patch this getter (or call :func:`resolve_orchestration_mode` with a hand-built
     config) rather than mutating the cache.
     """
@@ -94,7 +93,7 @@ def resolve_orchestration_mode(requested: str | None, *, config: ApiConfig) -> s
     override (``allow_request_orchestration_mode_override``). A caller-supplied
     token equal to the default is always honored (it changes nothing). A caller
     trying to FORCE a different mode on a runner whose policy forbids it is refused
-    with a 403 — so a locked-down Temporal runner can never be coerced into
+    with a 422 ``pipelex.deployment.orchestration_override_forbidden`` — so a locked-down Temporal runner can never be coerced into
     ``direct`` (whose whole point would be to bypass distributed execution), and
     vice versa. The token is a plain string compare; an *unregistered* token is
     not rejected here — that surfaces at dispatch as ``MissingOrchestratorError``.
@@ -107,11 +106,18 @@ def resolve_orchestration_mode(requested: str | None, *, config: ApiConfig) -> s
         f"This deployment does not allow overriding orchestration_mode per request "
         f"(configured mode '{config.orchestration_mode}', requested '{requested}')."
     )
-    raise_forbidden(msg, error_type=ErrorType.ORCHESTRATION_MODE_OVERRIDE_FORBIDDEN)
+    raise OrchestrationOverrideForbiddenError(msg, requested_mode=requested)
 
 
-class ApiBootConfigError(ValueError):
-    """Raised at startup when the deployment's orchestration config cannot boot coherently."""
+def is_callers_override(*, orchestration_mode: str, config: ApiConfig) -> bool:
+    """Whether a resolved orchestration mode is a per-request override the caller named, not the deployment's default.
+
+    :func:`resolve_orchestration_mode` returns the default unless the caller named a different mode the policy
+    allows, so a resolved mode other than the default is always the caller's. A refusal of such a mode, such as
+    one no installed plugin serves, is the caller's to change (``runtime.unsupported``), where the same refusal of
+    the default is the operator's (``pipelex.plugin.not_found``).
+    """
+    return orchestration_mode != config.orchestration_mode
 
 
 def resolve_boot_orchestrator(config: ApiConfig) -> str | None:

{pipelex_api-0.89.0 → pipelex_api-0.91.0}/pipelex_api/blob_store.py

@@ -36,14 +36,17 @@ From this the blob store answers without reading what it does not need:
 - `POST /v1/blobs/missing` lists a hash this runner does not hold at the size the sender states, an object held at
   another size included. The upload then tells the two cases apart, holding the bytes: bytes of the size held are the
   ones held, and nothing is written, while bytes of another size mean the object held is damaged, which only a storage
-  that does not publish writes whole could leave, a 500 `BlobStoreCorrupted`, since nothing writes over a blob and its
-  operator must remove the object.
+  that does not publish writes whole could leave, a 500 `pipelex.storage.damaged`, since nothing writes over a blob
+  and its operator must remove the object.
 - A run looks up each hash's stored size first. A hash held at another size than an entry states is a 422
-  `BlobMismatch`, never `MissingBlobs`, since uploading those bytes again would change nothing: the entry is wrong.
+  `request.invalid`, never `request.blobs_missing`, since uploading those bytes again would change nothing: the entry
+  is wrong.
   Only a blob held at the size its entries state is read, so the bytes read are bounded by the sizes the request
   states, which its census already held to the tree's limits, and each is checked against its hash once read. Bytes
-  that do not hash to their key mean the storage itself is damaged, which no caller can repair: a 500
-  `BlobStoreCorrupted`.
+  that do not hash to their key mean the object stored under it is damaged, which no caller can repair and no retry
+  cures: a 500 `pipelex.storage.damaged` naming the kind of storage, whose verdict is not to retry, and an error
+  record, "Stored blob bytes do not match their hash", naming the hash for the operator to remove the object, after
+  which the blob is uploaded again.
 
 Lookups and reads run a bounded number at a time, so one request naming many blobs never opens them all at once.
 """
@@ -54,15 +57,17 @@ from functools import partial
 from mthds.method_tree.digest import DIGEST_PREFIX, sha256_hex, tree_listing
 from mthds.method_tree.file_entry import BlobStore, InMemoryBlobStore
 from mthds.method_tree.models import MethodTree
+from pipelex import log
+from pipelex.config import get_config
 from pipelex.core.method_trees.exceptions import MethodFileUnavailableError
 from pipelex.core.method_trees.method_files import TreeListing
 from pipelex.runtime_hub import get_storage_provider
 from pipelex.tools.misc.async_utils import gather_bounded
 from pipelex.tools.storage.exceptions import StorageFileNotFoundError
+from pipelex.tools.storage.storage_config import StorageMethod
 from pipelex.tools.storage.storage_provider_abstract import PIPELEX_STORAGE_SCHEME, StorageProviderAbstract
 
-from pipelex_api.error_types import ErrorType
-from pipelex_api.errors import raise_internal_server_error
+from pipelex_api.exceptions import BlobStoreDamagedError
 
 BLOBS_PREFIX = "blobs"
 TREES_PREFIX = "trees"
@@ -72,19 +77,53 @@ _LISTING_CONTENT_TYPE = "text/plain; charset=utf-8"
 # request runs at a time.
 MAX_CONCURRENT_BLOB_LOOKUPS = 16
 MAX_CONCURRENT_BLOB_TRANSFERS = 4
+# The message of the error record a blob read whose bytes do not match their hash writes, for the operator to search.
+BLOB_DAMAGED_MESSAGE = "Stored blob bytes do not match their hash"
+
+
+# The kind of storage a `pipelex.storage.*` problem names for a storage plugin's own backend.
+PLUGIN_STORAGE_SCHEME = "plugin"
+
+
+def problem_scheme_of(*, storage_method: str) -> str:
+    """The kind of storage a `pipelex.storage.*` problem names for a configured storage method, as the registry spells it.
+
+    `plugin` for a storage plugin's own method, whose kind the registry cannot list.
+    """
+    match storage_method:
+        case StorageMethod.LOCAL:
+            return "local"
+        case StorageMethod.IN_MEMORY:
+            return "memory"
+        case StorageMethod.S3:
+            return "s3"
+        case StorageMethod.GCP:
+            return "gcs"
+        case _:
+            # An external storage plugin's method token: the configured method is an open string.
+            return PLUGIN_STORAGE_SCHEME
 
 
 class RunnerBlobStore:
     """The blobs and tree listings of one caller's scope, over the deployment's storage provider."""
 
-    def __init__(self, *, scope: str, storage: StorageProviderAbstract) -> None:
+    def __init__(self, *, scope: str, storage: StorageProviderAbstract, scheme: str) -> None:
+        """Build the blob store of one scope.
+
+        Args:
+            scope: The prefix of the scope's blobs and listings.
+            storage: The storage provider holding them.
+            scheme: The kind of storage the provider is, as a `pipelex.storage.*` problem names it (`problem_scheme_of`).
+        """
         self._scope = scope
         self._storage = storage
+        self._scheme = scheme
 
     @classmethod
     def make_for_scope(cls, *, scope: str) -> "RunnerBlobStore":
         """The blob store of `scope`, over the storage provider the runtime booted with."""
-        return cls(scope=scope, storage=get_storage_provider())
+        scheme = problem_scheme_of(storage_method=get_config().runtime.storage.method)
+        return cls(scope=scope, storage=get_storage_provider(), scheme=scheme)
 
     @property
     def scope(self) -> str:
@@ -119,15 +158,21 @@ class RunnerBlobStore:
     async def _read_verified(self, *, sha256: str, size: int) -> bytes:
         """The stored bytes of `sha256`, held at `size`, checked against their hash.
 
-        Raises (as a problem document): a 500 `BlobStoreCorrupted` for bytes that are not `size` bytes hashing to `sha256`.
+        Raises (as a problem document): a 500 `pipelex.storage.damaged` naming the kind of storage, for bytes that
+        are not `size` bytes hashing to `sha256`, after an error record naming the hash, the one place the operator
+        reads which damaged object to remove.
         """
         data = await self._storage.load(f"{PIPELEX_STORAGE_SCHEME}{self._blob_key(sha256=sha256)}")
         if len(data) != size or sha256_hex(data) != sha256:
+            log.error(
+                BLOB_DAMAGED_MESSAGE,
+                fields={"blob_sha256": sha256, "blob_size_bytes": size, "read_size_bytes": len(data), "storage_scheme": self._scheme},
+            )
             msg = (
-                f"The bytes this runner's blob store holds under {sha256} do not hash to it: its storage is damaged, "
-                "and its operator must remove that object for the blob to be uploaded again."
+                f"The bytes this runner's blob store holds under {sha256} do not hash to it: the storage holds a damaged "
+                "object, which its operator removes for the blob to be uploaded again."
             )
-            raise_internal_server_error(message=msg, error_type=ErrorType.BLOB_STORE_CORRUPTED)
+            raise BlobStoreDamagedError(msg, scheme=self._scheme)
         return data
 
     async def read_held(self, *, sizes: Mapping[str, int]) -> BlobStore:
@@ -149,8 +194,9 @@ class RunnerBlobStore:
         free, with its hash as its checksum: a writer that created it meanwhile wrote the same bytes, and the key
         holding an object is the blob held.
 
-        Raises (as a problem document): a 500 `BlobStoreCorrupted` when an object of another size is held under the
-        hash, which only damaged storage could hold and which this runner never writes over.
+        Raises (as a problem document): a 500 `pipelex.storage.damaged` naming the kind of storage when an object of
+        another size is held under the hash, which only damaged storage could hold and which this runner never writes
+        over, after the error record naming the hash that a read of damaged bytes writes.
         """
         sha256 = sha256_hex(data)
         await self._store_hashed_blob(sha256=sha256, data=data)
@@ -164,11 +210,15 @@ class RunnerBlobStore:
                 data=data, key=self._blob_key(sha256=sha256), content_type=_BLOB_CONTENT_TYPE, checksum_sha256_hex=sha256
             )
         elif held_size != len(data):
+            log.error(
+                BLOB_DAMAGED_MESSAGE,
+                fields={"blob_sha256": sha256, "blob_size_bytes": held_size, "read_size_bytes": len(data), "storage_scheme": self._scheme},
+            )
             msg = (
                 f"This runner's blob store holds {held_size} bytes under {sha256}, whose bytes are {len(data)} long: its "
                 "storage is damaged, and its operator must remove that object for the blob to be uploaded again."
             )
-            raise_internal_server_error(message=msg, error_type=ErrorType.BLOB_STORE_CORRUPTED)
+            raise BlobStoreDamagedError(msg, scheme=self._scheme)
 
     async def holds_tree(self, *, tree_digest: str) -> bool:
         """Whether the listing of the tree `tree_digest` names is held, which means every file of the tree is."""

{pipelex_api-0.89.0 → pipelex_api-0.91.0}/pipelex_api/blob_upload.py

@@ -5,8 +5,9 @@ must say that the upload is allowed, as a presigned URL does on an object store.
 names: its URL is its own `PUT /v1/blobs/{sha256}` with a query naming the blob store's scope, an expiry and an
 HMAC-SHA256 signature over the scope, the hash and the expiry. The upload route checks the signature and the expiry,
 writes under the scope the signature binds and nowhere else, and answers a missing, forged or expired signature with a
-403 `BlobUploadNotAuthorized`. The upload route therefore needs no bearer token: only a caller who passed the
-deployment's authentication on `POST /v1/blobs/missing` holds a target, for one hash, in its own scope, for a while.
+403 `auth.forbidden` whose `requirement` is a signed upload target. The upload route therefore needs no bearer token:
+only a caller who passed the deployment's authentication on `POST /v1/blobs/missing` holds a target, for one hash, in
+its own scope, for a while.
 
 The key is `BLOB_UPLOAD_SIGNING_KEY` when the deployment sets it, which it must when several instances serve one
 deployment, since a target named by one instance is checked by whichever instance receives the upload; otherwise each
@@ -28,8 +29,7 @@ from urllib.parse import urlencode, urlsplit
 from fastapi import Request
 from pipelex.system.environment import get_optional_env
 
-from pipelex_api.error_types import ErrorType
-from pipelex_api.errors import raise_internal_server_error
+from pipelex_api.exceptions import RunnerMisconfiguredError
 from pipelex_api.limits import BLOB_UPLOAD_URL_TTL_SECONDS
 
 PUBLIC_BASE_URL_ENV = "PUBLIC_BASE_URL"
@@ -62,8 +62,8 @@ def signed_upload_query(*, scope: str, sha256: str) -> str:
 def _public_base_url() -> str | None:
     """`PUBLIC_BASE_URL` without its trailing slash, or None when unset.
 
-    Raises (as a problem document): a 500 `ServerMisconfigured` for a value that is not an absolute http or https URL
-    without a query or a fragment.
+    Raises (as a problem document): a 500 `runtime.misconfigured` naming `PUBLIC_BASE_URL`, for a value that is not an
+    absolute http or https URL without a query or a fragment.
     """
     configured = get_optional_env(PUBLIC_BASE_URL_ENV)
     if not configured:
@@ -71,7 +71,7 @@ def _public_base_url() -> str | None:
     parts = urlsplit(configured)
     if parts.scheme not in {"http", "https"} or not parts.netloc or parts.query or parts.fragment:
         msg = f"Server configuration error: {PUBLIC_BASE_URL_ENV} must be an absolute http or https URL, such as https://runner.example.com."
-        raise_internal_server_error(message=msg, error_type=ErrorType.SERVER_MISCONFIGURED)
+        raise RunnerMisconfiguredError(msg, component=PUBLIC_BASE_URL_ENV)
     return configured.rstrip("/")
 
 

{pipelex_api-0.89.0 → pipelex_api-0.91.0}/pipelex_api/bundle.py

@@ -16,30 +16,31 @@ import binascii
 import zipfile
 from io import BytesIO
 
+from mthds.protocol.problem.registry import CoreCode, RequestMalformedReason
 from pipelex import log
+from pipelex.methods.package_reason import MethodPackageReason
 from pipelex.tools.log.error_fields import error_fields
 
-from pipelex_api.error_types import ErrorType
-from pipelex_api.errors import raise_bad_request, raise_payload_too_large, raise_validation_error
-from pipelex_api.limits import MAX_BUNDLE_FILES, MAX_BUNDLE_TOTAL_BYTES
+from pipelex_api.exceptions import RequestMethodError, RunnerRequestError
+from pipelex_api.limits import MAX_BUNDLE_FILES, MAX_BUNDLE_TOTAL_BYTES, base64_decoded_size
 
-# Base64 inflates by 4/3; a zip is compressed, so a bundle whose DECOMPRESSED content is within
-# the ceiling encodes to well under this. Bounding the base64 string BEFORE decoding stops a
-# ~100 MiB request-body (the only other bound) from being expanded into ~75 MiB of heap just to
-# be rejected later by the decompressed-size guard. Slack (+4) covers padding.
-_MAX_BUNDLE_B64_CHARS = MAX_BUNDLE_TOTAL_BYTES * 4 // 3 + 4
+# The request field the archive rides, which the refusals of its encoding name.
+_BUNDLE_FIELD = "bundle_b64"
 
 
 def _guard_count(count: int) -> None:
     if count == 0:
-        raise_validation_error(message="Bundle is empty (no files)", error_type=ErrorType.INVALID_BUNDLE)
+        msg = "Bundle is empty (no files)"
+        raise RunnerRequestError(msg, field=_BUNDLE_FIELD)
     if count > MAX_BUNDLE_FILES:
-        raise_payload_too_large(message=f"Bundle exceeds the {MAX_BUNDLE_FILES}-file limit (got {count})")
+        msg = f"Bundle exceeds the {MAX_BUNDLE_FILES}-file limit (got {count})"
+        raise RequestMethodError(msg).with_params(reason=MethodPackageReason.BUNDLE_TOO_MANY_FILES, limit=MAX_BUNDLE_FILES)
 
 
 def _guard_running_total(total_bytes: int) -> None:
     if total_bytes > MAX_BUNDLE_TOTAL_BYTES:
-        raise_payload_too_large(message=f"Bundle exceeds the {MAX_BUNDLE_TOTAL_BYTES // 1024} KiB decompressed-size limit")
+        msg = f"Bundle exceeds the {MAX_BUNDLE_TOTAL_BYTES // 1024} KiB decompressed-size limit"
+        raise RequestMethodError(msg).with_params(reason=MethodPackageReason.BUNDLE_TOO_LARGE, limit=MAX_BUNDLE_TOTAL_BYTES)
 
 
 def decode_bundle_archive(*, bundle_b64: str) -> list[tuple[str, bytes]]:
@@ -50,24 +51,45 @@ def decode_bundle_archive(*, bundle_b64: str) -> list[tuple[str, bytes]]:
     cheap length check on the still-encoded string runs FIRST, so an oversized payload is refused before it is buffered
     into memory as decoded bytes. A name the archive gives twice is returned twice, for the tree's reader to refuse.
 
-    Raises (as problem documents): an oversized archive (413 `PayloadTooLarge`), a value that is not base64 (400
-    `InvalidBase64`), and one that is not a zip archive or holds no file (422 `InvalidBundle`).
+    Raises (as problem documents): an archive past the size limit (413 `request.too_large`, its limit and its size in
+    bytes, the size read off the encoded length), a value that is
+    not base64 or not a zip archive (422 `request.malformed`), an archive holding no file (422 `request.invalid`), and
+    one past the limit on its files or on its decompressed size (422 `method.package_invalid`).
     """
-    if len(bundle_b64) > _MAX_BUNDLE_B64_CHARS:
-        raise_payload_too_large(message=f"bundle_b64 exceeds the {MAX_BUNDLE_TOTAL_BYTES // 1024} KiB compressed-size limit")
+    # A zip is compressed, so a bundle whose decompressed content is within the ceiling is an archive well under it. Its
+    # size is read off the encoded length, so a ~100 MiB request body, the only other bound, is refused before it is
+    # expanded into ~75 MiB of heap just to be refused later by the decompressed-size guard.
+    archive_bytes = base64_decoded_size(encoded=bundle_b64)
+    if archive_bytes > MAX_BUNDLE_TOTAL_BYTES:
+        msg = f"bundle_b64 exceeds the {MAX_BUNDLE_TOTAL_BYTES // 1024} KiB compressed-size limit"
+        raise (
+            RunnerRequestError(msg, field=_BUNDLE_FIELD)
+            .with_error_code(CoreCode.REQUEST_TOO_LARGE)
+            .with_params(limit=MAX_BUNDLE_TOTAL_BYTES, size=archive_bytes)
+        )
     try:
         raw = base64.b64decode(bundle_b64, validate=True)
     except (binascii.Error, ValueError) as decode_error:
         # DEBUG: the refusal below is a caller's mistake, which the error handler already logs; this keeps the cause.
         log.debug("A bundle's base64 could not be decoded", fields=error_fields(exc=decode_error))
-        raise_bad_request(message="bundle_b64 is not valid base64", error_type=ErrorType.INVALID_BASE64)
+        msg = "bundle_b64 is not valid base64"
+        raise (
+            RunnerRequestError(msg, field=_BUNDLE_FIELD)
+            .with_error_code(CoreCode.REQUEST_MALFORMED)
+            .with_params(reason=RequestMalformedReason.BAD_BASE64)
+        ) from decode_error
 
     try:
         archive = zipfile.ZipFile(BytesIO(raw))
     except zipfile.BadZipFile as zip_error:
         # DEBUG: the refusal below is a caller's mistake, which the error handler already logs; this keeps the cause.
         log.debug("A bundle is not a valid zip archive", fields=error_fields(exc=zip_error))
-        raise_validation_error(message="bundle_b64 is not a valid zip archive", error_type=ErrorType.INVALID_BUNDLE)
+        msg = "bundle_b64 is not a valid zip archive"
+        raise (
+            RunnerRequestError(msg, field=_BUNDLE_FIELD)
+            .with_error_code(CoreCode.REQUEST_MALFORMED)
+            .with_params(reason=RequestMalformedReason.BAD_ZIP)
+        ) from zip_error
 
     entries: list[tuple[str, bytes]] = []
     total_bytes = 0

pipelex_api-0.91.0/pipelex_api/bundle_resources.py

@@ -0,0 +1,95 @@
+"""Decode the files a request sends in `resources`, the files its bundle's steps read, such as a template file.
+
+`resources` is a JSON object mapping each file's bundle-relative path, the exact path a step names (a `PipeDocGen`
+step's `template_file`), to the file's bytes in standard base64. Beside `mthds_contents`, every bundle of the request is
+read as sitting in one directory, so one path holds one file, and the decoded files go to the runtime as its
+`submitted_files`, keyed by canonical path, which refuses a file none of the bundle's steps names once the bundle is
+parsed, and holds the rest as the files of the tree the bundle sits at the root of. Beside a method tree (`files` or
+`bundle_b64`), the decoded files join the tree at their paths from its root (`pipelex_api.method_tree_request`), where
+the tree's rule judges them as it judges the tree's own files.
+
+Refusals, in the order they are checked:
+
+- more `resources` than `MAX_RESOURCES_PER_REQUEST` → 422 `request.invalid`, naming the field;
+- a path that leaves the bundle (absolute, a drive, a `..` segment) → 422 `security.input_path_refused`, with the rule
+  it breaks, before any value is decoded, which also logs a security event, as a key of a method tree does;
+- a path of the wrong shape (a backslash, a colon, empty, or a second spelling of a path already submitted) → 422
+  `request.invalid`, naming the field;
+- a file whose size, read off its encoded length, exceeds `MAX_RESOURCE_FILE_BYTES` → 413 `request.too_large`, before
+  decoding;
+- a value that is not valid base64 → 422 `request.malformed`, as `bundle_b64` is;
+- a running total of decoded files over `MAX_RESOURCES_TOTAL_BYTES` → 413 `request.too_large`.
+
+Every `request.too_large` states its `limit` and its `size` in bytes, never in encoded characters.
+"""
+
+import base64
+import binascii
+from typing import NoReturn
+
+from mthds.protocol.problem.registry import CoreCode, RequestMalformedReason
+from pipelex import log
+from pipelex.core.bundle_resources.bundle_resource import SUBMITTED_FILES_FIELD, canonical_resource_path
+from pipelex.tools.log.error_fields import error_fields
+
+from pipelex_api.exceptions import RunnerRequestError
+from pipelex_api.limits import MAX_RESOURCE_FILE_BYTES, MAX_RESOURCES_PER_REQUEST, MAX_RESOURCES_TOTAL_BYTES, base64_decoded_size
+
+
+def refuse_two_files_under_one_path(*, path: str, where: str) -> NoReturn:
+    """Refuse two different files sent under one path, naming the path, with a 422 `request.invalid` on `resources`."""
+    msg = (
+        f"Two different files are named '{path}', {where}. A path names one file of a method: send one of them, or rename "
+        "one and the step that reads it."
+    )
+    raise RunnerRequestError(msg, field=SUBMITTED_FILES_FIELD)
+
+
+def _too_large(*, message: str, limit: int, size: int) -> RunnerRequestError:
+    """The refusal of submitted files past one of this runner's byte limits, a 413 `request.too_large`."""
+    return RunnerRequestError(message, field=SUBMITTED_FILES_FIELD).with_error_code(CoreCode.REQUEST_TOO_LARGE).with_params(limit=limit, size=size)
+
+
+def decode_bundle_resources(*, resources: dict[str, str] | None) -> dict[str, bytes] | None:
+    """The files `resources` sends, decoded and keyed by canonical path, or None when there are none."""
+    if not resources:
+        return None
+    if len(resources) > MAX_RESOURCES_PER_REQUEST:
+        # A count, not bytes, so not `request.too_large`, whose parameters are sizes in bytes: the field is refused.
+        msg = f"resources exceeds the {MAX_RESOURCES_PER_REQUEST}-file limit (got {len(resources)})"
+        raise RunnerRequestError(msg, field=SUBMITTED_FILES_FIELD)
+    # Every path first, so a second spelling of one file is refused as such before any value is decoded. A path that
+    # leaves the bundle is a security rule's refusal (`security.input_path_refused`), whose security event the error
+    # handler logs; one of the wrong shape is the request's (`request.invalid`), and logs none. Both propagate as raised.
+    sent_by_path: dict[str, tuple[str, str]] = {}
+    for path, encoded in resources.items():
+        canonical_path = canonical_resource_path(path=path)
+        if canonical_path in sent_by_path:
+            msg = f"Two files submitted with the bundle name the same file '{canonical_path}'."
+            raise RunnerRequestError(msg, field=SUBMITTED_FILES_FIELD)
+        sent_by_path[canonical_path] = (path, encoded)
+    files: dict[str, bytes] = {}
+    total_bytes = 0
+    for canonical_path, (path, encoded) in sent_by_path.items():
+        # Read off the encoded length, so an oversized file is refused before it is decoded into memory.
+        file_bytes = base64_decoded_size(encoded=encoded)
+        if file_bytes > MAX_RESOURCE_FILE_BYTES:
+            msg = f"The file '{path}' in resources exceeds the {MAX_RESOURCE_FILE_BYTES // 1024} KiB per-file limit"
+            raise _too_large(message=msg, limit=MAX_RESOURCE_FILE_BYTES, size=file_bytes)
+        try:
+            data = base64.b64decode(encoded, validate=True)
+        except (binascii.Error, ValueError) as decode_error:
+            # DEBUG: the refusal below is a caller's mistake, which the error handler already logs; this keeps the cause.
+            log.debug("A submitted file's base64 could not be decoded", fields=error_fields(exc=decode_error))
+            msg = f"The file '{path}' in resources is not valid base64"
+            raise (
+                RunnerRequestError(msg, field=SUBMITTED_FILES_FIELD)
+                .with_error_code(CoreCode.REQUEST_MALFORMED)
+                .with_params(reason=RequestMalformedReason.BAD_BASE64)
+            ) from decode_error
+        total_bytes += len(data)
+        if total_bytes > MAX_RESOURCES_TOTAL_BYTES:
+            msg = f"The files the bundle's steps read exceed the {MAX_RESOURCES_TOTAL_BYTES // 1024} KiB total-size limit"
+            raise _too_large(message=msg, limit=MAX_RESOURCES_TOTAL_BYTES, size=total_bytes)
+        files[canonical_path] = data
+    return files