letify 2.5.4__tar.gz → 2.6.1__tar.gz

{letify-2.5.4 → letify-2.6.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: letify
-Version: 2.5.4
+Version: 2.6.1
 Summary: Declarations that become infrastructure.
 Project-URL: Homepage, https://github.com/DarkPyonix/letify
 Project-URL: Documentation, https://github.com/DarkPyonix/letify/tree/main/docs

{letify-2.5.4 → letify-2.6.1}/docs/SPEC.md

@@ -316,7 +316,11 @@ Separately, and without needing a threshold at all, a refusal says how long ago
 
 The record therefore notes the process as well as the ref, `<username>/<slug>` and the pid separated by a tab, so a notebook a live process is still using is never deleted. A ref whose pid is still running on this machine is reported as in use and skipped. A ref with no pid, written by a letify older than this, cannot be checked and counts as an orphan, because the record is per machine and an older letify is not running.
 
-`letify stop` with no `--orphans` deletes nothing and says so, because a bare `stop` reads as ending everything. The command lists what it will delete and asks before deleting, `-y` answers yes, and `--alias` limits it to one account. Each ref is deleted with the same CLI call `stop` uses and dropped from the record only when the delete succeeds, so a failure leaves the ref to be tried again rather than losing track of it. The exit status is 1 when any delete failed.
+`letify stop` with no `--orphans` deletes nothing and says why rather than only that: a session whose
+creating process is still running ends when that process ends, because taking the session from under
+it leaves it calling a runtime that is gone. So the line names the way out, `send SIGTERM to the
+process that started it, which letify handles by closing its sessions`, and then what `--orphans` is
+for. A bare `stop` otherwise reads as ending everything. The command lists what it will delete and asks before deleting, `-y` answers yes, and `--alias` limits it to one account. Each ref is deleted with the same CLI call `stop` uses and dropped from the record only when the delete succeeds, so a failure leaves the ref to be tried again rather than losing track of it. The exit status is 1 when any delete failed.
 
 **Deleting the notebook, through the CLI only.** `stop` deletes the notebook with the official CLI, `kaggle kernels delete -y <username>/<slug>`, authenticated by the account's API token in `KAGGLE_API_TOKEN`; deletion of a notebook created for a cookie-authenticated interactive session is confirmed on the live account. The cookie is never asked, not even as a fallback, because deleting a notebook is not part of running or ending the interactive session. The call never fails the runtime, because the accelerator quota is already released by cancelling the run before it is tried. When it cannot run at all, for want of a token, of uv, or of the slug Kaggle did not return, or when the command itself fails, `stop` prints one warning line naming the notebook by its slug or id, never its cookie, so the user can delete it by hand on kaggle.com. The chain that starts the session, all on `https://www.kaggle.com/api/i/`, authenticated by the cookie:
 
@@ -1125,6 +1129,25 @@ So a blob above `transfer_parallel_mib`, 64 MiB by default, on a link whose meas
 
 The round trip is a condition because the gain depends on it. A single stream is window limited only when the window takes a long time to fill, so on a short link it already carries everything the link will carry and the extra connections are pure cost. Measured to a lab server over forward SSH at 0.25 ms, 256 MiB went at 36.1 MiB/s on one stream and 33.5 MiB/s on four, about 7% worse, which is the cost of opening the channels and their worker processes. The link reports what the probe measured, and a link with no measurement, such as a provider fallback, is treated as short and keeps one stream. A provider whose channel is not a connection it can open more of keeps one stream too: Modal's sandbox reads and writes through its own API, and a Kaggle session that never punched is carried by a kernel bridge, so neither has a second connection to offer. A Kaggle session that did punch is reached over SSH and does: each transfer stream asks the link for its own SSH command, which lands on a local forwarding port of its own and so makes the link punch again.
 
+The connections carry different blobs at the same time as well as different ranges of one, because a
+call whose blobs are each under the split threshold would otherwise send them one after another and
+reach one connection's ceiling however many files it has. Measured on a Kaggle runtime at a 190 ms
+round trip, 20 files of 10 MB each went at 5.0 MiB/s on one connection, which is the 0.9 MiB window
+over that round trip. So the sender runs `transfer_streams` senders over the pool rather than one, and
+the condition for using more than one connection is the link's round trip alone: the size threshold
+decides whether one blob is split, not whether blobs may overlap. A blob the worker asked for by
+`data_want` still goes next, because the queue is reordered before a sender takes from it.
+
+The connections are opened once per call and reused by every blob of it, not opened and closed per
+blob. A call that sends a hundred blobs over a punched link paid a hundred rounds of punching when
+each blob opened its own, which is both the cost the estimate below weighs and a hundred chances for
+a punch to be refused. They are closed when the call's transfer ends.
+
+Reuse is what makes the cost worth paying, so the estimate charges it only while the connections are
+not yet open. The first blob large enough to want them pays the punch and weighs it; every blob after
+it finds them open and weighs nothing, so a blob under the size threshold still goes on one
+connection, but a call's second large blob is split whenever splitting is faster at all.
+
 A connection that costs something to open has to pay for itself, so a provider reports `transfer_connection_cost_s`, the seconds one more connection takes. It is 0 for a provider that dials an address, and about 10 for one reached by a hole punch, because that is what a punch measures. The streams are opened only when the transfer is expected to finish sooner with them: the estimate uses the probe's upload rate, assumes the streams scale and that their connections are opened at the same time, and keeps one stream when `size / (rate x streams) + cost` is not less than `size / rate`. A measured case: a Kaggle runtime at a 44.7 ms round trip carried 128 MiB at 21.0 MiB/s on one stream and 9.5 MiB/s on four, 2.2 times worse, because three more punches cost more than the transfer saved. The same account at a 193 ms round trip carried 1.57 MiB/s on one raw SSH connection and 6.10 MiB/s on four, so the gain is real where the link is slow enough and the blob large enough for it to be worth 10 s.
 
 The connections have to be real ones. Every SSH command letify builds carries `ControlMaster=auto`, which multiplexes further sessions onto the first connection, so opening more channels that way would add streams inside one TCP connection and gain nothing. A transfer channel therefore sets `ControlPath=none` and `ControlMaster=no`, and a punched link redials for each, so each transfer channel is its own kernel connection.
@@ -1475,6 +1498,8 @@ For an ephemeral provider that archives its environment, the default managed Pyt
 
 Otherwise the sync step sets `UV_CACHE_DIR`, `UV_PYTHON_INSTALL_DIR` and `UV_TOOL_DIR` under `<workspace root>` for `uv sync` and `uv pip install`, unconditionally: not only when the provider's `persistence` is `persistent`, and not only when it sets no `env_root`. An `Env.vars` entry naming one of the three wins over this rule for that variable.
 
+`UV_PYTHON_INSTALL_DIR` is the exception: it follows the project directory whenever the provider sets an `env_root`, because the interpreter belongs with the `.venv` it serves and a workspace root is not always a filesystem one can be unpacked on. Modal's workspace root is a Modal volume, measured as supporting no hard link at all: a link inside it fails with `EPERM` and a link from it to the sandbox disk with `EXDEV`. So `uv sync --python 3.13` on a sandbox whose image carries 3.12 downloaded a managed interpreter into the volume and failed with `error: Operation not permitted (os error 1)`, which named neither the directory nor the reason. The archive case already placed it there, for its own reason, so this makes the two agree.
+
 On a runtime with a persistent workspace root the cache outlives a container rebuild along with the projects built from it. On an ephemeral provider the cache is filled once per runtime wherever it lives and discarded with it, but it is still written under the workspace root rather than the home directory, because the workspace root may be the only writable path.
 
 letify never deletes from the cache. The first sync on a runtime fills `<workspace root>/uv-cache` once, and a cache uv already had elsewhere is left in place. `uv cache prune` run with the same `UV_CACHE_DIR` removes entries no lock file needs any more; a file still hard linked from a `.venv` keeps its disk blocks until that `.venv` is removed as well.

{letify-2.5.4 → letify-2.6.1}/letify/__init__.py

@@ -66,7 +66,7 @@ distributed: Final = Host.distributed
 # The enum class stays importable from letify.declare.instance for letify's own use.
 del Host
 
-__version__ = "2.5.4"
+__version__ = "2.6.1"
 
 
 def fetch(tensor):  # type: ignore[no-untyped-def]

{letify-2.5.4 → letify-2.6.1}/letify/cli.py

@@ -755,8 +755,15 @@ def _dispatch(args: argparse.Namespace) -> int:
 
     if args.command == "stop":
         if not args.orphans:
-            # A bare `stop` reads as ending everything, so it ends nothing.
-            print("nothing was stopped. Pass --orphans to delete what a dead process left")
+            # Spec "Kaggle", stopping what a dead process left behind: a bare stop reads as
+            # ending everything, so it ends nothing, and says why rather than only that.
+            print(
+                "nothing was stopped. A session whose process is still running ends when "
+                "that process ends, because taking it away leaves that process calling a "
+                "runtime that is gone: send SIGTERM to the process that started it, which "
+                "letify handles by closing its sessions. Pass --orphans to delete what a "
+                "dead process left."
+            )
             return 0
         rows = let.orphans(args.alias)
         found = [(row["alias"], ref) for row in rows for ref in row.get("orphans", ())]

{letify-2.5.4 → letify-2.6.1}/letify/runtime/bootstrap.py

@@ -357,7 +357,9 @@ def sync_source(
     It raises ``RuntimeError`` with ``uv could not be installed on <name>`` or ``uv sync
     failed on <name>`` so the local side can name the step that failed.
     """
-    root = root or project_dir(DEFAULT_WORKSPACE_ROOT, env)
+    # Derived from the workspace root when it is given, so the two cannot disagree: the
+    # managed Python's directory is chosen by whether the project lies inside it.
+    root = root or project_dir(workspace or DEFAULT_WORKSPACE_ROOT, env)
     workspace = workspace or DEFAULT_WORKSPACE_ROOT
     encoded = {key: base64.b64encode(value).decode() for key, value in files.items()}
     sync_args = sync_command(env)[1:]
@@ -400,8 +402,15 @@ def sync_source(
     ]
     declared = dict(env.variables)
     directories = uv_env(workspace)
-    if archive:
-        # Keep an interpreter uv downloads inside the tree sent to the next VM.
+    inside = root.rstrip("/") == workspace.rstrip("/") or root.startswith(
+        workspace.rstrip("/") + "/"
+    )
+    if archive or not inside:
+        # Keep an interpreter uv downloads inside the tree sent to the next VM, and, when
+        # the project directory is off the workspace root, beside the .venv it serves: a
+        # workspace root is not always a filesystem an interpreter can be unpacked on.
+        # Spec "Remote paths letify writes": Modal's is a volume that carries no hard
+        # link, so unpacking one there fails with Operation not permitted.
         directories["UV_PYTHON_INSTALL_DIR"] = f"{root.rstrip('/')}/.letify-python"
     for key, value in directories.items():
         if key not in declared:

{letify-2.5.4 → letify-2.6.1}/letify/store/pathdata.py

@@ -18,6 +18,7 @@ import sys
 import threading
 import time
 import tomllib
+from collections.abc import Callable
 from dataclasses import dataclass, field
 from importlib import import_module
 from pathlib import Path
@@ -485,11 +486,16 @@ def _mib(size: int) -> str:
 
 
 class _Uploaded:
-    """Bytes sent so far, drawn as a progress line when the upload is large."""
+    """Bytes sent so far, drawn as a progress line when the upload is large.
+
+    Several senders advance this together once blobs overlap, spec "Several
+    connections at once", so ``base`` is kept under a lock rather than left to the GIL.
+    """
 
     def __init__(self, files: int, total: int) -> None:
         self.base = 0
         self.progress = None
+        self._lock = threading.Lock()
         if total >= PROGRESS_THRESHOLD:
             from ..install import _Progress
 
@@ -497,11 +503,14 @@ class _Uploaded:
 
     def update(self, done: int) -> None:
         if self.progress is not None:
-            self.progress.update(self.base + done)
+            with self._lock:
+                self.progress.update(self.base + done)
 
     def advance(self, size: int) -> None:
-        self.base += size
-        self.update(0)
+        with self._lock:
+            self.base += size
+            if self.progress is not None:
+                self.progress.update(self.base)
 
     def finish(self) -> None:
         if self.progress is not None:
@@ -628,6 +637,9 @@ class Stream:
         self._token: str | None = None
         #: The one stream every piece of this call travels on, opened at the first piece.
         self._pipeline: Any = None
+        #: The transfer connections of this call, opened lazily at the first large blob
+        #: and reused by every one after it. Spec "Several connections at once".
+        self._channels: _Channels | None = None
 
     # -- before the call -------------------------------------------------------
 
@@ -638,9 +650,31 @@ class Stream:
         started = time.monotonic()
         total = sum(self.plan.weight(digest)[1] for digest in wave)
         meter = _Uploaded(len(wave), total)
-        for digest in wave:
-            self._send(digest, meter)
-            meter.advance(self.plan.sizes.get(digest, 0))
+        senders = self._senders(len(wave))
+        if senders <= 1:
+            for digest in wave:
+                self._send(digest, meter)
+                meter.advance(self.plan.sizes.get(digest, 0))
+        else:
+            # Spec "Several connections at once": the condition for using more than one
+            # connection is the link's round trip alone, not the size threshold, so the
+            # first wave is sent with several senders too whenever the link allows it.
+            remaining = collections.deque(wave)
+            pop_lock = threading.Lock()
+
+            def pop() -> str | None:
+                with pop_lock:
+                    return remaining.popleft() if remaining else None
+
+            def worker(stop: threading.Event) -> None:
+                while not stop.is_set():
+                    digest = pop()
+                    if digest is None:
+                        return
+                    self._send(digest, meter, parallel=True)
+                    meter.advance(self.plan.sizes.get(digest, 0))
+
+            self._send_parallel(senders, worker)
         # Every piece answered, so the wave is placed when the call goes out.
         self._flush()
         meter.finish()
@@ -659,9 +693,26 @@ class Stream:
     def _run(self) -> None:
         started = time.monotonic()
         try:
-            while not self._cancelled.is_set():
-                if not self._pump():
-                    break
+            with self._lock:
+                left = len(self._queue)
+            senders = self._senders(left)
+            if senders <= 1:
+                while not self._cancelled.is_set():
+                    if not self._pump():
+                        break
+            else:
+                # Spec "Several connections at once": several senders over the pool,
+                # each reordering and taking from the same queue the single sender
+                # would have used, rather than one sender on the call's own stream.
+                def worker(stop: threading.Event) -> None:
+                    while not self._cancelled.is_set() and not stop.is_set():
+                        self.reorder()
+                        digest = self._take()
+                        if digest is None:
+                            return
+                        self._send(digest, None, parallel=True)
+
+                self._send_parallel(senders, worker)
             # The pieces still in flight are answered before the stream is released.
             self._flush()
         except Exception as exc:
@@ -741,6 +792,8 @@ class Stream:
         if (thread is None or not thread.is_alive()) and self._pipeline is not None:
             self._pipeline.close()
             self._pipeline = None
+        if self._channels is not None:
+            self._channels.close()
         self._drop_partials()
 
     def _drop_partials(self) -> None:
@@ -766,6 +819,14 @@ class Stream:
             self._pipeline = self.runtime.pipeline(window=WINDOW, timeout=3600)
         return self._pipeline
 
+    def _pool(self) -> _Channels:
+        """The call's transfer connections, opened lazily and reused by every blob."""
+        with self._lock:
+            if self._channels is None:
+                limit = int(getattr(self.runtime.provider, "transfer_streams", STREAMS) or 1)
+                self._channels = _Channels(self.runtime, limit)
+            return self._channels
+
     def _flush(self) -> None:
         """Read every outstanding reply, so a piece the worker refused fails here."""
         if self._pipeline is None:
@@ -775,9 +836,79 @@ class Stream:
         except RemoteError as exc:
             raise RuntimeFailure(f"{self.runtime.name}: data_put failed: {exc}") from exc
 
+    # -- several senders at once -------------------------------------------------
+
+    def _senders(self, available: int) -> int:
+        """How many blobs this call may have in flight at once, never above ``available``.
+
+        Spec "Several connections at once": the connections carry different blobs at the
+        same time as well as different ranges of one, and the condition for more than one
+        is the link's round trip alone. The size threshold decides whether one blob is
+        split, not whether blobs may overlap.
+        """
+        provider = getattr(self.runtime, "provider", None)
+        if not _has_transfer_channel(provider):
+            return 1
+        rtt = _measured_rtt(provider, self.runtime)
+        if rtt is None or rtt < PARALLEL_RTT_MS:
+            return 1
+        limit = int(getattr(provider, "transfer_streams", STREAMS) or 1)
+        return max(1, min(limit, available))
+
+    def _send_parallel(self, count: int, worker: Callable[[threading.Event], None]) -> None:
+        """Run ``worker(stop)`` on ``count`` threads; the first failure stops the rest.
+
+        One worker failing must not leave the others sending into a dead transfer, so
+        ``stop`` is set as soon as any of them raises, and the first failure is what
+        reaches the caller once every thread has joined.
+        """
+        failures: list[BaseException] = []
+        failures_lock = threading.Lock()
+        stop = threading.Event()
+
+        def guarded() -> None:
+            try:
+                worker(stop)
+            except BaseException as exc:
+                with failures_lock:
+                    failures.append(exc)
+                stop.set()
+
+        threads = [
+            threading.Thread(target=guarded, name=f"letify-data-{index}", daemon=True)
+            for index in range(count)
+        ]
+        for thread in threads:
+            thread.start()
+        for thread in threads:
+            thread.join()
+        if failures:
+            raise failures[0]
+
+    def _send_alone(self, action: Callable[[Any], None]) -> None:
+        """Run ``action(pipeline)`` on a pooled channel held for the call it makes.
+
+        Used when several senders share the call's pool: ``self._link()`` carries only
+        one sender's ordering at a time, so a blob that needs a pipeline for more than
+        one request borrows a channel of its own for as long as that takes. Spec
+        "Several connections at once".
+        """
+        channel = self._pool().take()
+        ok = False
+        try:
+            pipeline = channel.pipeline(window=WINDOW, timeout=3600)
+            action(pipeline)
+            pipeline.drain()
+            ok = True
+        finally:
+            if ok:
+                self._pool().give(channel)
+            else:
+                self._pool().drop(channel)
+
     # -- one blob ---------------------------------------------------------------
 
-    def _send(self, digest: str, meter: _Uploaded | None) -> None:
+    def _send(self, digest: str, meter: _Uploaded | None, *, parallel: bool = False) -> None:
         with self._lock:
             if digest in self._sent or self._cancelled.is_set():
                 return
@@ -787,10 +918,48 @@ class Stream:
         bucket = self.plan.bucket
         uploaded = True
         if bucket is None:
-            streams = _streams_for(self.runtime, size)
-            if streams > 1:
+            # The cost of opening the connections is weighed only while the pool is cold:
+            # the first blob large enough to want them pays the punch, every one after it
+            # finds them open. Spec "Several connections at once".
+            warm = self._channels is not None and self._channels.warm
+            streams = _streams_for(self.runtime, size, warm=warm)
+            if streams > 1 and parallel:
+                # self._link() carries only one sender's ordering at a time, so the
+                # data_begin/data_seal pair this blob needs borrows a pooled channel of
+                # its own instead, the same way the ranges already do.
+                self._send_alone(
+                    lambda pipeline: _put_parallel(
+                        self.runtime,
+                        pipeline,
+                        self.blobs,
+                        digest,
+                        local,
+                        size,
+                        meter,
+                        streams,
+                        self._pool(),
+                    )
+                )
+            elif streams > 1:
                 _put_parallel(
-                    self.runtime, self._link(), self.blobs, digest, local, size, meter, streams
+                    self.runtime,
+                    self._link(),
+                    self.blobs,
+                    digest,
+                    local,
+                    size,
+                    meter,
+                    streams,
+                    self._pool(),
+                )
+            elif parallel:
+                # Not split, but two senders cannot share self._link()'s ordering
+                # either, so this blob takes a pooled channel of its own for its
+                # whole send. Spec "Several connections at once".
+                self._send_alone(
+                    lambda pipeline: _put(
+                        self.runtime, pipeline, self.blobs, digest, local, size, meter
+                    )
                 )
             else:
                 _put(self.runtime, self._link(), self.blobs, digest, local, size, meter)
@@ -816,12 +985,12 @@ class Stream:
         files, total = self.plan.weight(digest)
         with self._lock:
             self.done.add(digest)
-        if self._thread is None:
-            self.before_files += files
-            self.before_bytes += total
-        else:
-            self.during_files += files
-            self.during_bytes += total
+            if self._thread is None:
+                self.before_files += files
+                self.before_bytes += total
+            else:
+                self.during_files += files
+                self.during_bytes += total
 
     @property
     def sent(self) -> int:
@@ -903,37 +1072,137 @@ def streams_for(
     return count
 
 
-def _streams_for(runtime: Any, size: int) -> int:
-    """``streams_for`` with the account's settings and the link's measured round trip."""
-    provider = getattr(runtime, "provider", None)
-    # On the class, not the instance: a provider's __getattr__ answers accelerator names
-    # and raises UnknownInstance for anything else, which hasattr does not catch.
-    if not hasattr(type(provider), "transfer_channel"):
-        # A provider whose channel is not a connection it can open more of, such as
-        # Modal's sandbox pipes or Kaggle's kernel bridge. Spec "Several connections at
-        # once".
-        return 1
-    rtt = None
-    try:
-        link = provider.link(runtime) if provider is not None else None
-        rtt = getattr(link, "rtt_ms", None)
-    except Exception:
-        rtt = None
-    rate = None
+def _has_transfer_channel(provider: Any) -> bool:
+    """Whether the provider's class can open another transfer connection.
+
+    On the class, not the instance: a provider's ``__getattr__`` answers accelerator
+    names and raises ``UnknownInstance`` for anything else, which ``hasattr`` does not
+    catch. A provider whose channel is not a connection it can open more of, such as
+    Modal's sandbox pipes or Kaggle's kernel bridge, answers False. Spec "Several
+    connections at once".
+    """
+    return hasattr(type(provider), "transfer_channel")
+
+
+def _measured_link(provider: Any, runtime: Any) -> Any:
+    """The provider's link for ``runtime``, or None where asking for it fails."""
     try:
-        rate = getattr(link, "upload_bps", None)
+        return provider.link(runtime) if provider is not None else None
     except Exception:
-        rate = None
+        return None
+
+
+def _measured_rtt(provider: Any, runtime: Any) -> float | None:
+    """The link's measured round trip in milliseconds, or None with no measurement."""
+    return getattr(_measured_link(provider, runtime), "rtt_ms", None)
+
+
+def _streams_for(runtime: Any, size: int, *, warm: bool = False) -> int:
+    """``streams_for`` with the account's settings and the link's measured round trip.
+
+    ``warm`` is True once the call's transfer connections are already open, so there is
+    nothing left to pay for opening them. Spec "Several connections at once": reuse is
+    what makes the cost worth paying, so the estimate charges it only while the pool is
+    cold.
+    """
+    provider = getattr(runtime, "provider", None)
+    if not _has_transfer_channel(provider):
+        return 1
+    link = _measured_link(provider, runtime)
+    rtt = getattr(link, "rtt_ms", None)
+    rate = getattr(link, "upload_bps", None)
+    cost_s = 0.0 if warm else float(getattr(provider, "transfer_connection_cost_s", 0.0) or 0.0)
     return streams_for(
         streams=int(getattr(provider, "transfer_streams", STREAMS) or 1),
         threshold_mib=int(getattr(provider, "transfer_parallel_mib", PARALLEL_MIB) or 0),
         rate_bps=rate,
-        cost_s=float(getattr(provider, "transfer_connection_cost_s", 0.0) or 0.0),
+        cost_s=cost_s,
         rtt_ms=rtt,
         size=size,
     )
 
 
+class _Channels:
+    """The transfer connections of one call, opened once and reused by every blob of it.
+
+    Spec "Several connections at once": the connections are opened once per call and
+    reused by every blob of it, not opened and closed per blob, which is both the cost the
+    estimate weighs and a chance for a punch to be refused each time it happens. Used from
+    several threads at once, one per range of a parallel put.
+    """
+
+    def __init__(self, runtime: Any, limit: int) -> None:
+        self._runtime = runtime
+        self._limit = max(1, int(limit))
+        self._free: list[Any] = []
+        self._opened = 0
+        self._closed = False
+        self._lock = threading.Lock()
+        self._condition = threading.Condition(self._lock)
+
+    @property
+    def warm(self) -> bool:
+        """True once at least one channel of this pool has been opened."""
+        with self._lock:
+            return self._opened > 0
+
+    def take(self) -> Any:
+        """A free channel, a newly opened one, or the next one ``give``n back.
+
+        Raises RuntimeFailure, naming the runtime, if the pool is already closed. Never
+        hangs past a ``close()`` that arrives while this call is waiting.
+        """
+        with self._condition:
+            while True:
+                if self._closed:
+                    raise RuntimeFailure(
+                        f"{self._runtime.name}: the transfer pool is closed"
+                    )
+                if self._free:
+                    return self._free.pop()
+                if self._opened < self._limit:
+                    self._opened += 1
+                    break
+                self._condition.wait()
+        try:
+            return self._runtime.provider.transfer_channel(self._runtime)
+        except BaseException:
+            # The slot was reserved before opening, and must be released on failure so a
+            # waiter is not left blocked on a channel that never came to exist.
+            with self._condition:
+                self._opened -= 1
+                self._condition.notify_all()
+            raise
+
+    def give(self, channel: Any) -> None:
+        """Return a channel to the free list, for the next ``take()`` to reuse."""
+        with self._condition:
+            if self._closed:
+                closed_already = True
+            else:
+                self._free.append(channel)
+                self._condition.notify_all()
+                closed_already = False
+        if closed_already:
+            channel.close()
+
+    def drop(self, channel: Any) -> None:
+        """A channel that failed: close it and free its slot for a replacement."""
+        channel.close()
+        with self._condition:
+            self._opened = max(0, self._opened - 1)
+            self._condition.notify_all()
+
+    def close(self) -> None:
+        """Close every channel this pool opened, and refuse any ``take()`` after it."""
+        with self._condition:
+            self._closed = True
+            free, self._free = self._free, []
+            self._condition.notify_all()
+        for channel in free:
+            channel.close()
+
+
 def _put_parallel(
     runtime: Any,
     pipeline: Any,
@@ -943,12 +1212,15 @@ def _put_parallel(
     size: int,
     meter: _Uploaded | None,
     streams: int,
+    channels: _Channels,
 ) -> None:
     """Send one file over several connections, each carrying one contiguous range.
 
     Spec "Several connections at once". The call's own stream makes the file and seals it,
     so a failure there is reported where every other data failure is. The ranges go out on
-    channels of their own, each its own kernel connection.
+    the call's pooled transfer connections, each its own kernel connection, taken from
+    ``channels`` and given back when the range finishes so the next blob of this call
+    reuses it rather than opening its own.
     """
     split = ranges(size, streams)
     try:
@@ -962,9 +1234,14 @@ def _put_parallel(
     lock = threading.Lock()
 
     def carry(index: int, offset: int, length: int) -> None:
-        channel = None
         try:
-            channel = runtime.provider.transfer_channel(runtime)
+            channel = channels.take()
+        except BaseException as exc:
+            with lock:
+                failures.append(exc)
+            return
+        ok = False
+        try:
             stream = channel.pipeline(window=WINDOW, timeout=3600)
             with open(local, "rb") as handle:
                 handle.seek(offset)
@@ -991,12 +1268,19 @@ def _put_parallel(
                             meter.update(sum(sent))
             stream.drain()
             stream.close()
+            ok = True
         except BaseException as exc:
             with lock:
                 failures.append(exc)
         finally:
-            if channel is not None:
-                channel.close()
+            # A range that finished cleanly gives the channel back for the next blob of
+            # this call to reuse. One that failed partway cannot be trusted to still be
+            # at a clean protocol boundary, so it is closed instead and the pool opens a
+            # replacement next time one is needed.
+            if ok:
+                channels.give(channel)
+            else:
+                channels.drop(channel)
 
     workers = [
         threading.Thread(target=carry, args=(index, offset, length), daemon=True)
@@ -1080,6 +1364,7 @@ class _Downloaded(_Uploaded):
     def __init__(self, files: int, total: int) -> None:
         self.base = 0
         self.progress = None
+        self._lock = threading.Lock()
         if total >= PROGRESS_THRESHOLD:
             from ..install import _Progress
 

{letify-2.5.4 → letify-2.6.1}/letify-core/Cargo.lock

@@ -138,7 +138,7 @@ dependencies = [
 
 [[package]]
 name = "letify-agent"
-version = "2.5.4"
+version = "2.6.1"
 dependencies = [
  "letify-wire",
  "libloading",
@@ -146,7 +146,7 @@ dependencies = [
 
 [[package]]
 name = "letify-driver"
-version = "2.5.4"
+version = "2.6.1"
 dependencies = [
  "letify-wire",
 ]
@@ -164,7 +164,7 @@ dependencies = [
 
 [[package]]
 name = "letify-wire"
-version = "2.5.4"
+version = "2.6.1"
 
 [[package]]
 name = "libc"