pipelex-api 0.90.0__tar.gz → 0.91.0__tar.gz

{pipelex_api-0.90.0 → pipelex_api-0.91.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: pipelex-api
-Version: 0.90.0
+Version: 0.91.0
 Summary: Pipelex API
 Project-URL: Homepage, https://pipelex.com
 Project-URL: Repository, https://github.com/Pipelex/pipelex
@@ -16,7 +16,7 @@ Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: 3.14
 Requires-Python: <3.15,>=3.11
 Requires-Dist: fastapi<0.142,>=0.118.0
-Requires-Dist: pipelex[anthropic,bedrock,fal,google,google-genai,mistralai]==0.90.0
+Requires-Dist: pipelex[anthropic,bedrock,fal,google,google-genai,mistralai]==0.91.0
 Requires-Dist: pyjwt>=2.10.1
 Requires-Dist: uvicorn>=0.37.0
 Provides-Extra: dev
@@ -115,26 +115,27 @@ The API is now running at `http://localhost:8081`. To customize behavior (enable
 
 # 🧪 Run your first pipeline
 
-Once `/health` is green, send an inline pipeline definition and inputs to `/v1/execute`. The example below summarizes a string with a one-pipe MTHDS bundle — no files, no auth, copy-paste:
+Once `/health` is green, send a method and its inputs to `/v1/execute`. The method travels in `files`, a map from each file's path in the method's folder to that file's content. The example below summarizes a string with a method made of a single `.mthds` file, with no auth, ready to copy and paste:
 
 ```bash
 curl -s http://localhost:8081/v1/execute \
   -H "Content-Type: application/json" \
   -d '{
-    "pipe_code": "summarize",
-    "mthds_contents": ["domain = \"hello\"\nmain_pipe = \"summarize\"\n\n[pipe.summarize]\ntype = \"PipeLLM\"\ndescription = \"Summarize the input text in one sentence\"\ninputs = { text = \"Text\" }\noutput = \"Text\"\nprompt = \"Summarize in one sentence:\\n@text\"\n"],
+    "files": {
+      "summarize.mthds": "domain = \"hello\"\nmain_pipe = \"summarize\"\n\n[pipe.summarize]\ntype = \"PipeLLM\"\ndescription = \"Summarize the input text in one sentence\"\ninputs = { text = \"Text\" }\noutput = \"Text\"\nprompt = \"Summarize in one sentence:\\n@text\"\n"
+    },
     "inputs": { "text": "Pipelex turns plain-language pipeline definitions into reproducible AI workflows that run as HTTP endpoints." }
   }'
 ```
 
-You'll get back a JSON response with `state: "COMPLETED"` and the summary under `pipe_output.working_memory.root.<main_stuff_name>.content`.
+You'll get back a JSON response with `state: "COMPLETED"` and the summary under `pipe_output.working_memory.root.<main_stuff_name>.content`. The run starts at the method's `main_pipe`; to start at another pipe of the method, name it in `pipe_code`.
 
 **Passing files (PDFs, images) as inputs.** Use the `Document` concept and point it at any HTTP(S) URL:
 
 ```json
 {
   "pipe_code": "your_pipe",
-  "mthds_contents": ["...your MTHDS..."],
+  "files": { "your_method.mthds": "...your MTHDS..." },
   "inputs": {
     "cv": { "concept": "Document", "content": { "url": "https://example.com/resume.pdf" } }
   }
@@ -143,7 +144,7 @@ You'll get back a JSON response with `state: "COMPLETED"` and the summary under
 
 `Document` accepts public HTTP/HTTPS URLs, `pipelex-storage://` URIs, or base64 data URLs. For images, use the `Image` concept with the same `{ "url": "..." }` shape.
 
-For inline MTHDS in the request, `mthds_contents` is a JSON array of raw `.mthds` (TOML) file contents as strings — typically `[open("my_pipe.mthds").read()]` from a client. See the [Pipe Run page](https://docs.pipelex.com/latest/api-server/pipe-run/) for every supported input shape and the full `/execute` reference.
+**Sending the method.** `files` holds the whole method, as the MTHDS standard's method tree: each key is a file's path relative to the method's folder, such as `summarize.mthds` or `templates/invoice.docx`, and each value is that file's content. A text file, such as a `.mthds` bundle or a Python module, is sent as its UTF-8 text in a JSON string, and a file that is not text, such as an office template or an image, as `{"base64": "..."}`. This server also keeps a blob store, which `GET /v1/version` reports as `blob_store: true`, so a client can send a file as `{"sha256": "...", "size": n}` instead: it asks `POST /v1/blobs/missing` which of its files the server lacks, uploads each of those to the target the answer names, and then sends the method with a hash entry for each file, so a method that has not changed is not uploaded again. The older `mthds_contents` field, a JSON array of `.mthds` texts, is still accepted, but it is being retired together with `bundle_b64` and `resources`, and a later release refuses them, so new clients send `files`. See the [Pipe Run page](https://docs.pipelex.com/latest/api-server/pipe-run/) for every supported input shape, the rules of [a method sent as a tree](https://docs.pipelex.com/latest/api-server/pipe-run/#sending-a-method-as-a-tree) and of [sending files by hash](https://docs.pipelex.com/latest/api-server/pipe-run/#sending-files-by-hash), and the full `/execute` reference.
 
 # 📈 How to scale Pipelex
 

{pipelex_api-0.90.0 → pipelex_api-0.91.0}/README.md

@@ -69,26 +69,27 @@ The API is now running at `http://localhost:8081`. To customize behavior (enable
 
 # 🧪 Run your first pipeline
 
-Once `/health` is green, send an inline pipeline definition and inputs to `/v1/execute`. The example below summarizes a string with a one-pipe MTHDS bundle — no files, no auth, copy-paste:
+Once `/health` is green, send a method and its inputs to `/v1/execute`. The method travels in `files`, a map from each file's path in the method's folder to that file's content. The example below summarizes a string with a method made of a single `.mthds` file, with no auth, ready to copy and paste:
 
 ```bash
 curl -s http://localhost:8081/v1/execute \
   -H "Content-Type: application/json" \
   -d '{
-    "pipe_code": "summarize",
-    "mthds_contents": ["domain = \"hello\"\nmain_pipe = \"summarize\"\n\n[pipe.summarize]\ntype = \"PipeLLM\"\ndescription = \"Summarize the input text in one sentence\"\ninputs = { text = \"Text\" }\noutput = \"Text\"\nprompt = \"Summarize in one sentence:\\n@text\"\n"],
+    "files": {
+      "summarize.mthds": "domain = \"hello\"\nmain_pipe = \"summarize\"\n\n[pipe.summarize]\ntype = \"PipeLLM\"\ndescription = \"Summarize the input text in one sentence\"\ninputs = { text = \"Text\" }\noutput = \"Text\"\nprompt = \"Summarize in one sentence:\\n@text\"\n"
+    },
     "inputs": { "text": "Pipelex turns plain-language pipeline definitions into reproducible AI workflows that run as HTTP endpoints." }
   }'
 ```
 
-You'll get back a JSON response with `state: "COMPLETED"` and the summary under `pipe_output.working_memory.root.<main_stuff_name>.content`.
+You'll get back a JSON response with `state: "COMPLETED"` and the summary under `pipe_output.working_memory.root.<main_stuff_name>.content`. The run starts at the method's `main_pipe`; to start at another pipe of the method, name it in `pipe_code`.
 
 **Passing files (PDFs, images) as inputs.** Use the `Document` concept and point it at any HTTP(S) URL:
 
 ```json
 {
   "pipe_code": "your_pipe",
-  "mthds_contents": ["...your MTHDS..."],
+  "files": { "your_method.mthds": "...your MTHDS..." },
   "inputs": {
     "cv": { "concept": "Document", "content": { "url": "https://example.com/resume.pdf" } }
   }
@@ -97,7 +98,7 @@ You'll get back a JSON response with `state: "COMPLETED"` and the summary under
 
 `Document` accepts public HTTP/HTTPS URLs, `pipelex-storage://` URIs, or base64 data URLs. For images, use the `Image` concept with the same `{ "url": "..." }` shape.
 
-For inline MTHDS in the request, `mthds_contents` is a JSON array of raw `.mthds` (TOML) file contents as strings — typically `[open("my_pipe.mthds").read()]` from a client. See the [Pipe Run page](https://docs.pipelex.com/latest/api-server/pipe-run/) for every supported input shape and the full `/execute` reference.
+**Sending the method.** `files` holds the whole method, as the MTHDS standard's method tree: each key is a file's path relative to the method's folder, such as `summarize.mthds` or `templates/invoice.docx`, and each value is that file's content. A text file, such as a `.mthds` bundle or a Python module, is sent as its UTF-8 text in a JSON string, and a file that is not text, such as an office template or an image, as `{"base64": "..."}`. This server also keeps a blob store, which `GET /v1/version` reports as `blob_store: true`, so a client can send a file as `{"sha256": "...", "size": n}` instead: it asks `POST /v1/blobs/missing` which of its files the server lacks, uploads each of those to the target the answer names, and then sends the method with a hash entry for each file, so a method that has not changed is not uploaded again. The older `mthds_contents` field, a JSON array of `.mthds` texts, is still accepted, but it is being retired together with `bundle_b64` and `resources`, and a later release refuses them, so new clients send `files`. See the [Pipe Run page](https://docs.pipelex.com/latest/api-server/pipe-run/) for every supported input shape, the rules of [a method sent as a tree](https://docs.pipelex.com/latest/api-server/pipe-run/#sending-a-method-as-a-tree) and of [sending files by hash](https://docs.pipelex.com/latest/api-server/pipe-run/#sending-files-by-hash), and the full `/execute` reference.
 
 # 📈 How to scale Pipelex
 

{pipelex_api-0.90.0 → pipelex_api-0.91.0}/pipelex_api/method_tree_request.py

@@ -36,8 +36,8 @@ import base64
 import json
 import shutil
 import tempfile
-from collections.abc import Generator, Mapping
-from contextlib import ExitStack, contextmanager
+from collections.abc import AsyncGenerator, Generator, Mapping
+from contextlib import ExitStack, asynccontextmanager, contextmanager
 from pathlib import Path
 from typing import Any, NamedTuple, NoReturn, cast
 
@@ -56,6 +56,7 @@ from pipelex.cli.installed_methods import PROVENANCE_FILENAME
 from pipelex.config import is_pipe_func_sandbox_hosted
 from pipelex.core.bundle_resources.bundle_resource import SUBMITTED_FILES_FIELD, root_escape_of
 from pipelex.errors.registry.pipelex_codes import PipelexErrorCode
+from pipelex.libraries.method_load_threads import run_method_load
 from pipelex.libraries.written_trees import WRITTEN_METHOD_TREES, write_method_tree
 from pipelex.methods.package_reason import MethodPackageReason
 from pipelex.methods.structures_check import ensure_no_structured_content_in_library_sources, ensure_no_structured_content_python
@@ -778,6 +779,29 @@ class WrittenMethod(NamedTuple):
     tree: MethodTree
 
 
+def _write_method(*, request_method: RequestMethod, releases: ExitStack) -> WrittenMethod:
+    """Write a method's trees into a temporary directory and hold each as written, each release pushed on `releases`.
+
+    The removal of the directory is pushed before anything is written, and each holding as soon as it is taken, so
+    closing `releases` undoes exactly what was done, however far the writing got. Blocking work.
+    """
+    temp_dir = Path(tempfile.mkdtemp(prefix="pipelex-method-"))
+    releases.callback(shutil.rmtree, temp_dir, ignore_errors=True)
+    method_dir = temp_dir / "tree"
+    written = write_method_tree(contents=request_method.tree.contents, destination=method_dir, expected_digest=request_method.tree.tree.digest)
+    releases.enter_context(WRITTEN_METHOD_TREES.holding(tree=written))
+    methods_dirs: list[Path] = []
+    for index_dependency, (address, dependency) in enumerate(request_method.dependencies.items()):
+        methods_dir = temp_dir / f"methods-{index_dependency}"
+        dependency_dir = methods_dir / dependency_dir_name(address=address)
+        dependency_written = write_method_tree(contents=dependency.contents, destination=dependency_dir, expected_digest=dependency.tree.digest)
+        releases.enter_context(WRITTEN_METHOD_TREES.holding(tree=dependency_written))
+        if dependency.has_python and is_pipe_func_sandbox_hosted():
+            ensure_no_structured_content_python(package_dir=dependency_dir, package_address=address)
+        methods_dirs.append(methods_dir)
+    return WrittenMethod(method_dir=method_dir, methods_dirs=methods_dirs or None, tree=written)
+
+
 @contextmanager
 def written_method(*, request_method: RequestMethod) -> Generator[WrittenMethod, None, None]:
     """Write a method's trees into a temporary directory, hold each as written, and remove them all on the way out.
@@ -786,20 +810,25 @@ def written_method(*, request_method: RequestMethod) -> Generator[WrittenMethod,
     `<name>/` in a methods directory of its own, the layout the engine's `methods_dirs` take, so two dependencies sharing
     a name never meet. On a sandbox-hosted deployment, a dependency tree declaring a Python structure class is refused,
     naming it by its address, as a fetched package is (`MethodStructuresRefusedError`, a 422).
+
+    Writing and removing the files is blocking work, done on the calling thread: asynchronous code uses
+    `written_method_async`.
     """
-    with ExitStack() as stack:
-        temp_dir = Path(tempfile.mkdtemp(prefix="pipelex-method-"))
-        stack.callback(shutil.rmtree, temp_dir, ignore_errors=True)
-        method_dir = temp_dir / "tree"
-        written = write_method_tree(contents=request_method.tree.contents, destination=method_dir, expected_digest=request_method.tree.tree.digest)
-        stack.enter_context(WRITTEN_METHOD_TREES.holding(tree=written))
-        methods_dirs: list[Path] = []
-        for index_dependency, (address, dependency) in enumerate(request_method.dependencies.items()):
-            methods_dir = temp_dir / f"methods-{index_dependency}"
-            dependency_dir = methods_dir / dependency_dir_name(address=address)
-            dependency_written = write_method_tree(contents=dependency.contents, destination=dependency_dir, expected_digest=dependency.tree.digest)
-            stack.enter_context(WRITTEN_METHOD_TREES.holding(tree=dependency_written))
-            if dependency.has_python and is_pipe_func_sandbox_hosted():
-                ensure_no_structured_content_python(package_dir=dependency_dir, package_address=address)
-            methods_dirs.append(methods_dir)
-        yield WrittenMethod(method_dir=method_dir, methods_dirs=methods_dirs or None, tree=written)
+    with ExitStack() as releases:
+        yield _write_method(request_method=request_method, releases=releases)
+
+
+@asynccontextmanager
+async def written_method_async(*, request_method: RequestMethod) -> AsyncGenerator[WrittenMethod, None]:
+    """`written_method` for asynchronous code: the trees are written, and removed, in the method-load pool, off the event loop.
+
+    The directory is removed and its trees released whatever happens, a refusal while writing, a failure of the block
+    and a cancellation included. `run_method_load` lets a cancellation through only once its call has ended, or at once
+    when no thread has taken the call yet, so the removal never runs while the writing is still at work, and a
+    cancellation repeated during the removal waits for it too.
+    """
+    releases = ExitStack()
+    try:
+        yield await run_method_load(_write_method, request_method=request_method, releases=releases)
+    finally:
+        await run_method_load(releases.close)

{pipelex_api-0.90.0 → pipelex_api-0.91.0}/pipelex_api/routes/pipelex/crate_ops.py

@@ -22,7 +22,6 @@ worker thread (`project_resolved_method`), never on the event loop: a slow resol
 requests, and the current library, a context variable, is bound, read and cleared in that one thread's context.
 """
 
-import asyncio
 from collections.abc import Callable
 from enum import StrEnum
 from typing import Literal, NamedTuple, NoReturn, TypeVar
@@ -34,6 +33,7 @@ from mthds.protocol.problem.registry import CoreCode
 from pipelex.errors.problem import ProblemView
 from pipelex.interpreter_hub import clear_current_library, get_current_library_id_or_none, get_library_manager, get_required_entry_pipe
 from pipelex.libraries.library_crate import LibraryCrate
+from pipelex.libraries.method_load_threads import run_method_load
 from pipelex.libraries.pipe.exceptions import EntryPipeAmbiguousError, EntryPipeNotFoundError, PipeNotFoundError
 from pipelex.libraries.pipe.pipe_suggestions import MAIN_PIPE_DECLARATION
 from pipelex.methods.method_ref import looks_like_method_ref
@@ -160,11 +160,11 @@ async def project_resolved_method(
     """Resolve the method the request names into its normalized crate, and return what `project` makes of it.
 
     The method is read as `/validate` reads it (`select_request_method`), then written out as a run writes it and loaded
-    alone as the caller's own library directory, its dependency trees as the packages it ships, in a worker thread.
-    `project` runs in that thread with the freshly opened library loaded and current, so it can read live pipes, and the
-    written tree held; both are released when it returns or raises. A method that does not resolve returns its invalid
-    verdict instead, the `ValidateBundleError` carrying the structured validation items, with `project` never called and
-    nothing left loaded.
+    alone as the caller's own library directory, its dependency trees as the packages it ships, in a worker thread of the
+    method-load pool (`run_method_load`). `project` runs in that thread with the freshly opened library loaded and
+    current, so it can read live pipes, and the written tree held; both are released when it returns or raises. A
+    method that does not resolve returns its invalid verdict instead, the `ValidateBundleError` carrying the structured
+    validation items, with `project` never called and nothing left loaded.
 
     Raises (as problem documents): a 422 `request.unsupported` for a registry-form `method_ref`; the refusals of
     `select_request_method`; the address-form fetch and location failures (`MethodRefError` subclasses); and whatever
@@ -184,7 +184,7 @@ async def project_resolved_method(
     selected = await select_request_method(
         request=request, method_ref=request_data.method_ref, resources=request_data.resources, read_scope=request_data.read_scope
     )
-    return await asyncio.to_thread(
+    return await run_method_load(
         _project_written_method,
         request_method=selected.request_method,
         method_label=selected.method_label,

{pipelex_api-0.90.0 → pipelex_api-0.91.0}/pipelex_api/routes/pipelex/pipeline.py

@@ -16,14 +16,16 @@ from pipelex.core.pipes.pipe_output import PipeOutput
 from pipelex.errors.bundle_validation_failure import BundleValidationFailure
 from pipelex.errors.problem import ProblemView
 from pipelex.errors.registry.pipelex_codes import PipelexErrorCode
+from pipelex.libraries.method_load_threads import run_method_load
 from pipelex.libraries.method_requirements import ensure_served_method_requirements_met
+from pipelex.libraries.method_tree_listing import MethodTreeListing, scoped_tree_listing
 from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, StorageTarget, WebhookTarget
 from pipelex.pipe_run.exceptions import AsyncExecutionNotEnabledError
 from pipelex.pipe_run.pipe_run_protocol import PipeRunProtocol
 from pipelex.pipeline.exceptions import ValidateBundleError
 from pipelex.pipeline.pipeline_response import PipelexRunResultExecute, PipelexRunResultStart, RunState
 from pipelex.pipeline.pipeline_run_setup import pipeline_run_setup
-from pipelex.pipeline.runner import PipelexMTHDSProtocol
+from pipelex.pipeline.runner import PipelexMTHDSProtocol, refuse_method_files
 from pipelex.pipeline.validate_bundle_translation import translate_to_validate_bundle_error
 from pipelex.reporting.usage_records import apply_tokens_usage_wire_shape
 from pipelex.runtime_bridge.exceptions import MissingBundleValidatorError, MissingOrchestratorError
@@ -52,7 +54,7 @@ from pipelex_api.method_tree_request import (
     select_tree_entry,
     tree_documents_of,
     with_resources,
-    written_method,
+    written_method_async,
 )
 from pipelex_api.middleware import request_id_of
 from pipelex_api.openapi_responses import (
@@ -81,6 +83,7 @@ from pipelex_api.security import AuthMode, get_auth_mode
 if TYPE_CHECKING:
     from collections.abc import AsyncGenerator
 
+    from mthds.method_tree.file_entry import MethodFiles
     from mthds.protocol.pipe_output import VariableMultiplicity
     from mthds.protocol.pipeline_inputs import PipelineInputs
     from mthds.protocol.working_memory import WorkingMemoryAbstract
@@ -343,14 +346,17 @@ class ApiRunner(PipelexMTHDSProtocol):
     async def execute(
         self,
         pipe_code: str | None = None,
-        mthds_contents: list[str] | None = None,
+        files: MethodFiles | None = None,
         inputs: PipelineInputs | WorkingMemoryAbstract[Any] | None = None,
         output_name: str | None = None,
         output_multiplicity: VariableMultiplicity | None = None,
         dynamic_output_concept_ref: str | None = None,
         extra: dict[str, Any] | None = None,
+        *,
+        mthds_contents: list[str] | None = None,
         delivery_assignment: DeliveryAssignment | None = None,
         request_id: str | None = None,
+        tree_listing: MethodTreeListing | None = None,
         requested_orchestration_mode: str | None = None,
     ) -> PipelexRunResultExecute:
         """Execute a method synchronously, dispatching by the resolved `orchestration_mode`.
@@ -374,7 +380,14 @@ class ApiRunner(PipelexMTHDSProtocol):
         `RunMetadata.request_id` for log correlation, so it reaches the job a worker is handed.
         `requested_orchestration_mode` is the optional per-request backend override
         (`PipelineApiExtras.orchestration_mode`).
+
+        The method's trees are listed once for the request: the check of its requirements lists them through
+        `tree_listing`, or a listing of its own when none is given, and the run loads them from that same listing.
         """
+        # The route resolves a method sent as `files` into a written tree before it calls this runner, which takes it
+        # as its library directory, so the map reaches the runner only from in-process code, and is refused as the
+        # base runner refuses it, before any backend is looked up.
+        refuse_method_files(files=files)
         # Resolve the effective orchestration mode FIRST — a per-request override the deployment
         # policy forbids is refused (422) here, before any library load / run registration. Mirrors start().
         api_config = get_api_config()
@@ -382,7 +395,8 @@ class ApiRunner(PipelexMTHDSProtocol):
         orchestrator = get_orchestrator_registry().get_optional(mode=orchestration_mode)
         if orchestrator is None:
             raise _missing_orchestrator_error(mode=orchestration_mode, api_config=api_config)
-        await self._ensure_served_method_requirements_met()
+        listing = tree_listing if tree_listing is not None else MethodTreeListing()
+        await self._ensure_served_method_requirements_met(tree_listing=listing)
 
         # Dispatch the run through the mode-selected orchestrator by injecting it as this runner's
         # PipeRun, then delegate to the base execute, which owns the full run lifecycle. The
@@ -391,6 +405,7 @@ class ApiRunner(PipelexMTHDSProtocol):
         self._pipe_run = _OrchestratorPipeRun(orchestrator=orchestrator, orchestration_mode=orchestration_mode)
         return await super().execute(
             pipe_code=pipe_code,
+            files=files,
             mthds_contents=mthds_contents,
             inputs=inputs,
             output_name=output_name,
@@ -399,18 +414,21 @@ class ApiRunner(PipelexMTHDSProtocol):
             extra=extra,
             delivery_assignment=delivery_assignment,
             request_id=request_id,
+            tree_listing=listing,
         )
 
     @override
     async def start(
         self,
         pipe_code: str | None = None,
-        mthds_contents: list[str] | None = None,
+        files: MethodFiles | None = None,
         inputs: PipelineInputs | WorkingMemoryAbstract[Any] | None = None,
         output_name: str | None = None,
         output_multiplicity: VariableMultiplicity | None = None,
         dynamic_output_concept_ref: str | None = None,
         extra: dict[str, Any] | None = None,
+        *,
+        mthds_contents: list[str] | None = None,
         pipeline_run_id: str | None = None,
         callback_urls: list[str] | None = None,
         request_id: str | None = None,
@@ -451,6 +469,10 @@ class ApiRunner(PipelexMTHDSProtocol):
         (`PipelineApiExtras.orchestration_mode`); it is resolved against the deployment's
         `api.toml` policy and a forbidden override is refused with a 422.
         """
+        # The route resolves a method sent as `files` into a written tree before it calls this runner, which takes it
+        # as its library directory, so the map reaches the runner only from in-process code, and is refused as the
+        # base runner refuses it.
+        refuse_method_files(files=files)
         if extra:
             # Nothing on the wire reaches `extra`: the route parses the extras itself and passes them by name, so a
             # value here is a fault of the in-process code calling this runner, never the caller's.
@@ -472,7 +494,10 @@ class ApiRunner(PipelexMTHDSProtocol):
                 f"async-capable orchestration, and this deployment has none. Use /execute (synchronous) instead."
             )
             raise AsyncExecutionNotEnabledError(msg)
-        await self._ensure_served_method_requirements_met()
+        # The method's trees are listed once for the request: the requirements check lists them, and the job's load
+        # takes them from the same listing.
+        tree_listing = MethodTreeListing()
+        await self._ensure_served_method_requirements_met(tree_listing=tree_listing)
         created_at = get_current_iso_timestamp()
         pipelex_inputs: PipelineInputs | WorkingMemory | None = cast("PipelineInputs | WorkingMemory | None", inputs)
 
@@ -506,6 +531,7 @@ class ApiRunner(PipelexMTHDSProtocol):
             submitted_files=self.submitted_files,
             pipeline_run_id=pipeline_run_id,
             request_id=request_id,
+            tree_listing=tree_listing,
         )
 
         delivery_assignment = DeliveryAssignment(
@@ -554,13 +580,15 @@ class ApiRunner(PipelexMTHDSProtocol):
             crate_fingerprint=pipe_job.job_metadata.run_metadata.crate_fingerprint,
         )
 
-    async def _ensure_served_method_requirements_met(self) -> None:
+    async def _ensure_served_method_requirements_met(self, *, tree_listing: MethodTreeListing) -> None:
         """Refuse the request's method when its requirements.txt is not met, before anything loads or is dispatched.
 
         This runner's `library_dirs` are the directories it wrote the request's method tree into, the caller's own,
         whose load reports a requirements refusal as the caller's too; judging it here first refuses the run before
         any library opens or any run is registered, naming no path on this host. The tree is held as written, so its
-        listing is the tree the request sent. The listing reads and hashes files, so it runs off the event loop.
+        listing is the tree the request sent. The listing reads and hashes files, so it runs off the event loop, in the method-load pool,
+        and it is made through `tree_listing`, which the caller hands to the request's load, so the load reads the trees
+        checked here without listing them again.
 
         Raises:
             MethodRequirementsFileError: The method's requirements.txt holds a line the standard refuses.
@@ -571,7 +599,9 @@ class ApiRunner(PipelexMTHDSProtocol):
         if not self.library_dirs:
             return
         library_dirs = [Path(library_dir) for library_dir in self.library_dirs]
-        await asyncio.to_thread(ensure_served_method_requirements_met, library_dirs=library_dirs)
+        # In the method-load pool, whose cancellation waits for the check to end: the directories it lists are removed
+        # when the request's scope exits, which a client disconnect can make happen while the check still hashes them.
+        await run_method_load(ensure_served_method_requirements_met, library_dirs=library_dirs, tree_listing=tree_listing)
 
     async def validate_verdict(
         self,
@@ -621,6 +651,10 @@ class ApiRunner(PipelexMTHDSProtocol):
         arm hands the method's trees to its worker through. The route builds a validating runner with the
         `read_scope` its request names, `None` when it names none, and never with the caller's id a run
         would fall back to: nothing lists or erases what a validation would leave under that prefix.
+
+        The method's trees are listed once for the request: the requirements check lists them, and the validation is
+        called with that listing in scope (`scoped_tree_listing`), which the in-process validator loads the trees from,
+        while a validator dispatching to a worker never reads it.
         """
         # Resolve the effective mode FIRST — a per-request override the deployment policy forbids
         # is refused (422) here, before any validator dispatch / library load. Mirrors start().
@@ -631,33 +665,39 @@ class ApiRunner(PipelexMTHDSProtocol):
             raise _missing_bundle_validator_error(mode=orchestration_mode, api_config=api_config)
         library_dirs = [Path(library_dir) for library_dir in self.library_dirs] if self.library_dirs else None
         # The method's requirements.txt is judged here, on the runner, which holds the method's files and the package set
-        # its PipeFunc steps run with, before any dispatch: a dispatched validator never reads these directories. A
-        # refusal is the method's invalid verdict, as a run of it is refused before its files load.
+        # its PipeFunc steps run with, before any dispatch, so a refusal reaches the caller whichever validator runs. A
+        # worker a method tree's validation is dispatched to never reads these directories: it receives the method's
+        # trees through its method file store and judges their requirements again as it loads them. A refusal is the
+        # method's invalid verdict, as a run of it is refused before its files load.
+        tree_listing = MethodTreeListing()
         try:
             with translate_to_validate_bundle_error():
-                await self._ensure_served_method_requirements_met()
+                await self._ensure_served_method_requirements_met(tree_listing=tree_listing)
         except ValidateBundleError as invalid_method:
             return BundleValidationFailure(
                 problem=invalid_method.to_problem(view=ProblemView.OPERATOR),
                 validation_errors=invalid_method.validation_error_items(),
             )
-        verdict = await validator.validate_bundles(
-            mthds_contents=mthds_contents,
-            mthds_sources=mthds_sources,
-            allow_signatures=allow_signatures,
-            # A method directory is the whole method: no library directory loads beside it.
-            library_dirs=None if method_dir is not None else library_dirs,
-            method_dir=method_dir,
-            methods_dirs=self.methods_dirs if method_dir is not None else None,
-            # A validation is not a run, but its dry runs and its `pipe_dry_run` event are still
-            # done for the caller this runner was built for — the user and groups a run would state.
-            caller_identity=self.caller_identity,
-            graph_pipe_code=graph_pipe_code,
-            # The files submitted with the contents, which the validation's dry run reads, the template check included.
-            submitted_files=self.submitted_files,
-            # The scope the host named, under which a dispatched validator's worker reads the method's trees.
-            read_scope=self.read_scope,
-        )
+        # The validator seam takes no listing: the one the requirements check made is in scope around the call, so the
+        # in-process validator loads the trees it checked without listing them again.
+        with scoped_tree_listing(tree_listing=tree_listing):
+            verdict = await validator.validate_bundles(
+                mthds_contents=mthds_contents,
+                mthds_sources=mthds_sources,
+                allow_signatures=allow_signatures,
+                # A method directory is the whole method: no library directory loads beside it.
+                library_dirs=None if method_dir is not None else library_dirs,
+                method_dir=method_dir,
+                methods_dirs=self.methods_dirs if method_dir is not None else None,
+                # A validation is not a run, but its dry runs and its `pipe_dry_run` event are still
+                # done for the caller this runner was built for — the user and groups a run would state.
+                caller_identity=self.caller_identity,
+                graph_pipe_code=graph_pipe_code,
+                # The files submitted with the contents, which the validation's dry run reads, the template check included.
+                submitted_files=self.submitted_files,
+                # The scope the host named, under which a dispatched validator's worker reads the method's trees.
+                read_scope=self.read_scope,
+            )
         # The core seam types its valid arm at the protocol-level ValidationReport (a leaf type)
         # to stay import-acyclic in core; every registered validator in fact produces the canonical
         # PipelexValidationReport. Recover the precise type here — the single narrowing point — so
@@ -874,7 +914,7 @@ async def _request_method_of(*, run_request: RunRequest, body: bytes, blob_store
 async def _run_source(*, run_request: RunRequest, body: bytes, blob_scope: str) -> AsyncGenerator[_ResolvedRunSource, None]:
     """Resolve the request's run source: a method tree (`files`, `bundle_b64` or a `method_ref`), or the inline contents.
 
-    A method tree is written into a temporary directory and held as written, then handed to the runner as its one
+    A method tree is written into a temporary directory in a worker thread and held as written, then handed to the runner as its one
     library directory, the caller's own, with the dependency trees it carries as `methods_dirs`: the load reads the
     tree whole, its `.mthds` files as the method's bundles and every other file as one a step may read, such as a
     `PipeDocGen` template or a `PipeFunc` module, and names each by its tree path. A refusal while loading it is the
@@ -912,7 +952,7 @@ async def _run_source(*, run_request: RunRequest, body: bytes, blob_scope: str)
         )
         return
     pipe_code = run_request.pipe_code or select_tree_entry(tree=request_method.tree, method_label=method_label).pipe_ref
-    with written_method(request_method=request_method) as written:
+    async with written_method_async(request_method=request_method) as written:
         yield _ResolvedRunSource(
             mthds_contents=None,
             library_dirs=[str(written.method_dir)],

{pipelex_api-0.90.0 → pipelex_api-0.91.0}/pipelex_api/routes/pipelex/validate.py

@@ -21,7 +21,7 @@ from pipelex_api.bundle_resources import decode_bundle_resources
 from pipelex_api.exception_handlers import problem_response
 from pipelex_api.exceptions import RequestFieldConflictError
 from pipelex_api.json_body import JsonBodyRoute
-from pipelex_api.method_tree_request import RequestMethod, select_tree_entry, written_method
+from pipelex_api.method_tree_request import RequestMethod, select_tree_entry, written_method_async
 from pipelex_api.middleware import request_id_of
 from pipelex_api.openapi_responses import PROBLEM_403_READ_SCOPE, PROBLEM_404_METHOD_PACKAGE, PROBLEM_500_HELD_BLOBS, PROBLEM_501_RUN_POLICY
 from pipelex_api.problem_view import problem_view_of
@@ -79,7 +79,10 @@ def _resolve_validation_views(views: list[str]) -> set[ValidationView]:
 
 
 class ValidateRequest(MthdsContentsRequest, CallerAnalyticsGroupsMixin):
-    """The shared `mthds_contents` + `allow_signatures` payload, plus optional per-file sources.
+    """The method to validate, in `files`, by `method_ref` or as `mthds_contents`, plus `allow_signatures` and optional per-file sources.
+
+    `mthds_contents`, and the `mthds_sources` and `resources` that serve it, are being retired: a later release
+    refuses them, and a method is sent in `files`, whose bundles are named by their tree paths.
 
     `mthds_sources`, when provided, pairs each `mthds_contents[i]` with a logical source (e.g. the
     file's path relative to the submitted directory). The runner threads it onto
@@ -91,7 +94,8 @@ class ValidateRequest(MthdsContentsRequest, CallerAnalyticsGroupsMixin):
     mthds_sources: list[str] | None = Field(
         default=None,
         description=(
-            "Optional per-file sources, parallel to `mthds_contents`. When provided, each entry is threaded "
+            "Being retired with `mthds_contents`, and refused by a later release: the bundles of a method sent in `files` "
+            "are named by their tree paths. Optional per-file sources, parallel to `mthds_contents`. When provided, each entry is threaded "
             "onto the corresponding bundle's `source` so server-side validation errors carry a `source` pointing "
             "at the owning file. Must match `mthds_contents` in length when present."
         ),
@@ -366,7 +370,7 @@ async def validate_mthds(request: Request, request_data: ValidateRequest) -> JSO
     - **Either arm:** a member with no value is absent rather than `null`, at the arm's top level and in its
       problem items; the structured artifacts of the valid arm keep their own serialization.
     - **No verdict (non-2xx):** a malformed request body, an `mthds_sources` length mismatch, or
-      both/neither of `mthds_contents` / `method_ref` is a request-shape **422**, and so is a forbidden
+      more or fewer than one of `files` / `method_ref` / `mthds_contents` is a request-shape **422**, and so is a forbidden
       `orchestration_mode` override; a host-wiring programmer error or a genuine orchestrator fault is a
       **5xx**; auth is **401/403**. A `method_ref` **resolution failure** is also a no-verdict condition —
       never an `is_valid: false`: each answers the status its code declares, such as a **422** for a
@@ -425,7 +429,7 @@ async def validate_mthds(request: Request, request_data: ValidateRequest) -> JSO
         entry = _optional_tree_entry(request_method=request_method, method_label=method_label)
         tree_default_pipe_ref = entry.pipe_ref if entry is not None else None
         default_pipe_ref_known = True
-        with written_method(request_method=request_method) as written:
+        async with written_method_async(request_method=request_method) as written:
             verdict = await ApiRunner(
                 library_dirs=[str(written.method_dir)],
                 library_dirs_are_callers=True,

{pipelex_api-0.90.0 → pipelex_api-0.91.0}/pipelex_api/schemas/models.py

@@ -88,7 +88,7 @@ DEPENDENCIES_FIELD_DESCRIPTION = (
     "by. They are searched before the runner's installed copies, for this request alone. Accompanies `files`."
 )
 
-RESOURCES_FIELD_DESCRIPTION = (
+_RESOURCES_FIELD_FACTS = (
     "The files submitted with the bundle that its steps read, such as a PipeDocGen step's `template_file` (an "
     "`.xlsx`, `.docx`, `.pptx` or `.html` template): a JSON object mapping each file's bundle-relative path, exactly "
     "as the step names it, to the file's bytes in standard base64. A path must be relative and written with forward "
@@ -102,6 +102,21 @@ RESOURCES_FIELD_DESCRIPTION = (
     "names by its digest: no file's bytes travel with the run, each being read where a step uses it."
 )
 
+RESOURCES_FIELD_DESCRIPTION = (
+    f"{_RESOURCES_FIELD_FACTS} Being retired with `mthds_contents`, and refused by a later release: send these files in "
+    "the method's tree, in `files`."
+)
+"""`resources` on the routes that also take `mthds_contents`, the run routes and `/validate`, with which it is retired."""
+
+TREE_ROUTE_RESOURCES_FIELD_DESCRIPTION = (
+    f"{_RESOURCES_FIELD_FACTS} Being retired, and refused by a later release: a file the steps read travels inside the "
+    "method's tree, in `files`, at its path beside the bundle that names it."
+)
+"""`resources` on the routes that take a method only as `files` or `method_ref`: `/resolve`, `/codegen` and `/pipe-io`."""
+
+MTHDS_CONTENTS_RETIRING_NOTE = "Being retired, and refused by a later release: send the method in `files`."
+"""Opens the description of every request field that takes a method's bundles as `mthds_contents`."""
+
 
 class RunRequest(BaseModel):
     """Body of `POST /execute` — this server's typed request model.
@@ -124,7 +139,8 @@ class RunRequest(BaseModel):
 
     Attributes:
         pipe_code: Code of the pipe to execute.
-        mthds_contents: List of MTHDS bundle contents to load.
+        mthds_contents: List of MTHDS bundle contents to load. Being retired, with `bundle_b64`
+            and `resources`, in favor of `files`.
         inputs: Inputs in PipelineInputs format, as plain JSON — Pydantic validation is
             skipped to preserve the flexible format (strings, lists, objects); the runtime
             shapes each value against the entry pipe's declared input.
@@ -154,7 +170,14 @@ class RunRequest(BaseModel):
     model_config = ConfigDict(extra="allow")
 
     pipe_code: str | None = None
-    mthds_contents: list[str] | None = None
+    mthds_contents: list[str] | None = Field(
+        default=None,
+        description=(
+            f"{MTHDS_CONTENTS_RETIRING_NOTE} The method's `.mthds` files as raw strings, always an array, even for a single "
+            "file; with no `pipe_code`, a file must declare `main_pipe`. Mutually exclusive with `files`, `bundle_b64` and "
+            "`method_ref`."
+        ),
+    )
     inputs: Annotated[PipelineInputs | None, SkipValidation] = None
     output_name: str | None = None
     output_multiplicity: VariableMultiplicity | None = None
@@ -166,7 +189,8 @@ class RunRequest(BaseModel):
             "map of its entries, so the archive's names are judged by the method tree's rule as the keys of `files` are. "
             "The method packages it calls by address may travel with it under `.mthds/methods/<name>/`, each with its "
             "`METHODS.toml`; they are searched before the runner's installed copies, for this run alone. "
-            "Mutually exclusive with `files`. Custom `.py` is only honored on a sandbox-hosted deployment."
+            "Mutually exclusive with `files`. Custom `.py` is only honored on a sandbox-hosted deployment. Being retired "
+            "with `mthds_contents`, and refused by a later release: send the method in `files`."
         ),
     )
     # Documents the wire only: the map is read from the request's raw JSON text (`pipelex_api.method_tree_request`),
@@ -660,7 +684,7 @@ class MethodTreeRequest(BaseModel):
             "until a method registry exists). Exactly one of `files` / `method_ref`."
         ),
     )
-    resources: dict[str, str] | None = Field(default=None, description=RESOURCES_FIELD_DESCRIPTION)
+    resources: dict[str, str] | None = Field(default=None, description=TREE_ROUTE_RESOURCES_FIELD_DESCRIPTION)
     read_scope: str | None = Field(default=None, description=_TREE_READ_SCOPE_DESCRIPTION)
 
     @field_validator("read_scope")
@@ -735,7 +759,10 @@ class MthdsContentsRequest(BaseModel):
         default=None,
         min_length=1,
         max_length=MAX_MTHDS_FILES_PER_REQUEST,
-        description="MTHDS contents to load (always an array, even for a single file). Exactly one of `mthds_contents` / `files` / `method_ref`.",
+        description=(
+            f"{MTHDS_CONTENTS_RETIRING_NOTE} MTHDS contents to load (always an array, even for a single file). "
+            "Exactly one of `mthds_contents` / `files` / `method_ref`."
+        ),
     )
     # Documents the wire only: the route reads the map from the request's raw JSON text, never from this field.
     files: Annotated[dict[str, FileEntry] | None, SkipValidation] = Field(