letify 2.8.0__tar.gz → 2.8.2__tar.gz

{letify-2.8.0 → letify-2.8.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: letify
-Version: 2.8.0
+Version: 2.8.2
 Summary: Declarations that become infrastructure.
 Project-URL: Homepage, https://github.com/DarkPyonix/letify
 Project-URL: Documentation, https://github.com/DarkPyonix/letify/tree/main/docs

{letify-2.8.0 → letify-2.8.2}/docs/SPEC.md

@@ -1275,7 +1275,9 @@ Letify keeps its files in a folder named `letify` at the root of that Drive, one
 
 An ephemeral provider uses the Drive store for its environment archive when the machine declares one and the account sets no `bucket`, which keeps precedence. A provider whose environment survives the session on its own disk or volume does not use it. An archive on Drive is an uncompressed tar, because compression was most of the cost: on a Kaggle runtime, packing a 5.5 GB environment as tar.gz and uploading it took about 12 minutes, against an upload of roughly 77 s for the uncompressed tree at the measured rate.
 
-Whether an archive is worth it is measured, not assumed, because a provider with a fast package index can build faster than it restores. On a Kaggle runtime the same 5.5 GB environment built in 35.4 s and restored from Drive in about 90 s. So after a sync letify packs only when the build took longer than the archive would take to restore, estimated as the project directory's size over 176 MiB/s, the slower Drive download measured, plus the same size over 170 MB/s, the extraction rate measured on a sandbox disk. The record beside the archive keeps the build seconds and, once a session has restored from it, the restore seconds that session measured; a later session restores only while the measured restore is shorter than the measured build, and syncs otherwise, saying `letify: <runtime>: syncing rather than restoring from Drive, because a restore took <s> s against a <s> s build`.
+Whether an archive is worth it is measured, not assumed, because a provider with a fast package index can build faster than it restores. On a Kaggle runtime the same 5.5 GB environment built in 35.4 s and restored from Drive in about 90 s. So after a sync letify packs only when the build took at least twice as long as the archive would take to restore, estimated as the project directory's size over 176 MiB/s, the slower Drive download measured, plus the same size over 170 MB/s, the extraction rate measured on a sandbox disk. The record beside the archive keeps the build seconds and, once a session has restored from it, the restore seconds that session measured; a later session restores only while the measured restore is shorter than the measured build, and syncs otherwise, saying `letify: <runtime>: syncing rather than restoring from Drive, because a restore took <s> s against a <s> s build`. The factor of two is there because packing has a cost of its own that the comparison leaves out: on a Kaggle runtime a 7.2 GB environment that built in 87.6 s against an estimated 81 s restore was packed, which would save about 6 s a session at the price of the pack.
+
+The pack to Drive runs on the worker beside the session's calls, not before them. After the interpreter check the session starts the pack as a background job and returns, so the first call starts at once; the worker answers calls on its main thread while a separate thread writes the tar and uploads it, and the archive's ref and build record are written only when the upload completes. The upload session needs the archive's digest and size, so the worker reports them once the tar is written, and a client thread that checks every 2 s creates the resumable session and hands its URL to the worker, which then uploads; the client records and reports the result at the start and end of a call and at session close. A session that ends first leaves no ref, and the next session syncs and may pack again. While it runs the pack says nothing more after `caching the environment, <size> GB`; when it ends it says `letify: <runtime>: cached the environment, <size> GB in <s> s (<rate> MiB/s)`, or the Drive failure line above, on standard error of whichever call or session close observes it first. Closing the session waits up to 60 s for a pack still running and then abandons it, saying `letify: <runtime>: not caching the environment, because the session closed <s> s into the pack`.
 
 Drive's own limits apply. The archive counts against the Google account's storage, which is the person's own plan. Google documents a daily upload limit of 750 GB per user for Google Workspace accounts, independent of the storage plan; an environment archive is a few gigabytes, so letify does not approach it. Standard use of the Drive API costs nothing, and Google documents that exceeding its request quota is planned to be charged to the project's billing account later in 2026; letify's requests are a handful per session against a quota of 325,000 units per user per minute.
 
@@ -1565,6 +1567,8 @@ Both run in two places with the same result:
 
 A body run on a one-shot channel, which has no stream back to the local process, cannot ask for a grant, so `stash` and `unstash` there raise `UnsupportedMode` naming the provider rather than reading the runtime's own configuration.
 
+The worker sets the environment variable `LETIFY_WORKER_PID` to its process id, so a process the body starts, such as one per GPU, inherits it without the grant hook. `stash` and `unstash` called there raise `UnsupportedMode` naming the handoff, saying the call must be made from the process letify runs the body in and not from a child process it started, and neither reads the configuration nor touches Drive.
+
 When the machine declares no Drive store, both raise `ConfigError` naming `drive_login` and `drive_project`. letify never falls back to another store or to the local disk. A Drive request that fails raises `RuntimeFailure` naming the Drive status and the name, and the function body decides whether to carry on.
 
 ## Environment
@@ -1908,12 +1912,29 @@ It is opt-in because it needs the user's machine to accept inbound SSH and needs
 
 The key is per session. letify generates a new ed25519 key pair for each session and appends the public key to the user's `~/.ssh/authorized_keys` with `restrict,port-forwarding,command="/bin/false"` and the comment `letify-session <alias>`. The private key reaches the remote side through the rendezvous. Closing the link removes the line. Before a new session's key is added, every line carrying the `letify-session` marker for that alias is removed, so a crashed session never leaves a key behind.
 
+#### Reverse SSH through a jump host <!-- id: reverse-ssh-jump -->
+
+> `reverse_ssh` takes an optional `jump`, for a user's machine whose SSH server the internet reaches only through another server.
+
+```toml
+[colab_pro_plus]
+reverse_ssh = { address = "168.188.127.175", port = 30008, user = "work", jump = { address = "jump.example.org", port = 20070, user = "root", key = "~/.ssh/id_jump" } }
+```
+
+With `jump`, `address` and `port` are the user's SSH server as the jump host reaches it, and the remote side connects to `<user>@<address>:<port>` through `-o ProxyCommand="ssh -o BatchMode=yes -o StrictHostKeyChecking=accept-new -i <session key> -W %h:%p -p <jump port> <jump user>@<jump address>"` and forwards its port back with `-R` as before. It is a ProxyCommand rather than `-J`, because the options given for the main hop on the command line do not reach a `-J` hop.
+
+The per-session key authorizes both hops. Besides the user's own `~/.ssh/authorized_keys`, letify appends the session's public key to the jump host's `~/.ssh/authorized_keys` with `restrict,port-forwarding,permitopen="<address>:<port>",command="/bin/false"` and the comment `letify-session <alias>`, so the key opens a connection to the user's SSH server and nothing else. letify reaches the jump host from the user's machine with `ssh -i <jump key> -p <jump port> <jump user>@<jump address>`, `BatchMode=yes`, and edits the file there with a shell command; `key` is optional and defaults to the SSH client's own keys. Closing the link removes that line, and before a new session's key is added every line carrying the `letify-session` marker for that alias is removed from the jump host too, as on the user's machine. The jump key never leaves the user's machine.
+
+A jump host that letify cannot log in to, or whose `authorized_keys` it cannot edit, fails the strategy with `reverse_ssh: the jump host <user>@<address>:<port> refused letify (<reason>)`, before anything is sent to the remote side.
+
 ### Colab <!-- id: colab-transport -->
 
 > Colab is a `Shell` whose rendezvous is `colab exec`. It has no forward SSH, and its fallback is `colab exec` with the Colab file API.
 
 The Colab CLI runs as `uv tool run --from google-colab-cli colab`, with `jupyter-kernel-client<1` pinned, because release 0.6.0 of the CLI calls an API that jupyter-kernel-client 1.0 removed. `colab new` and `colab stop` manage the session. `Colab.sessions()` reads `colab sessions` and returns the first word of each listing line. The CLI prints a session as `[<name>] <id> | Hardware: <hardware> | Variant: <variant>`, so a first word in square brackets gives the name inside them. A line starting with `[colab]` is a message from the CLI, such as `[colab] No active sessions found on server.`, and names no session, so an account with no session returns an empty list.
 
+**An assignment with no session.** `colab new` can fail after Colab has already assigned a machine, for example with `ReadTimeout` after 120 s, and the machine then bills (14.2 compute units per hour for an A100) with no local session state. `colab sessions` lists it as `[?] <endpoint> | ...`, and `colab stop -s <name>` cannot reach it. `Colab.create_session` therefore lists the account's assignments before `colab new`. When `colab new` raises, for any reason, it lists them again, releases every `?` assignment whose endpoint was not in the first list, and re-raises the original error. An assignment that existed before the attempt is never released. Each release prints one line on standard error naming the endpoint. When the first or the second listing fails, one line on standard error names the endpoint check and the error, and the original error is still the one raised. An assignment is released by endpoint with `Client.unassign` of the Colab CLI, run as `uv tool run --from google-colab-cli python -c <script> <endpoint>` with the account's environment, because the CLI has no command for it. `Colab.orphans()` returns the endpoints of the `?` rows of `colab sessions` and `Colab.stop_orphan(endpoint)` releases one, so `letify stop --orphans` covers Colab as it covers Kaggle.
+
 **Runtime tools.** A runtime whose rendezvous runs a Python program on the machine, Colab and Kaggle both, installs tailcat lazily when the tailcat strategy requests its endpoint, within that strategy's race thread. The install belongs to the rendezvous and not to one provider, so a provider that leaves it out is a defect rather than a choice: without it the tailcat request carries no path, the remote side runs a bare `tailcat`, and the strategy fails with `FileNotFoundError` on a runtime image that ships no such binary. Target construction performs no remote installation. TCP punch rendezvous never waits for tailcat preparation, even if installation fails or takes its full 180 second timeout. A locally available `tailcat` or configured `tailcat_binary` requires the Linux release pinned by `letify.install.TAILCAT_VERSION`, `TAILCAT_RELEASE` and `TAILCAT_SHA256`. The runtime selects its own architecture, verifies the archive SHA-256, extracts only a regular `tailcat` member into `~/.letify/tools/tailcat/<version>/tailcat`, and verifies the cached binary against its recorded SHA-256 on reuse. Only tailcat requests carry that absolute path. Installation errors name the tool and verification failure and fail that strategy. Tailcat remains eligible because throughput is measured per link rather than inferred from another session. The shared account throughput floor still rejects slow links.
 
 **Rendezvous scheduling.** Provider `colab exec` commands, including TCP punch rendezvous and tailcat preparation, do not acquire the wake lock. The lock protects concurrent wakes sharing CLI session state; rendezvous is a separate operation whose agreed dial time cannot wait behind a wake's 120 second timeout. Wakes acquire the account lock without waiting. If another wake holds it, the daemon records `wake_skipped` with reason `account_busy`, logs the skip and waits 60 seconds before trying again. Successful acquisition still holds the lock through the wake command so two wakes cannot collide. An already running wake cannot hold a lock needed by a connection.

{letify-2.8.0 → letify-2.8.2}/letify/__init__.py

@@ -68,7 +68,7 @@ distributed: Final = Host.distributed
 # The enum class stays importable from letify.declare.instance for letify's own use.
 del Host
 
-__version__ = "2.8.0"
+__version__ = "2.8.2"
 
 
 def fetch(tensor):  # type: ignore[no-untyped-def]

{letify-2.8.0 → letify-2.8.2}/letify/protocol/worker.py

@@ -1346,6 +1346,11 @@ _DRIVE_LOCK = threading.Condition()
 #: ``stash`` or ``unstash`` takes them.
 _DRIVE_ANSWERS = {}
 
+# A process the body starts inherits this and lacks ``_drive_ask``, which is how
+# ``stash`` and ``unstash`` tell it is a child of the body and refuse. Spec "Named handoffs
+# on the Drive store".
+os.environ["LETIFY_WORKER_PID"] = str(os.getpid())
+
 
 def _drive_ask(message):
     """Ask the client for a Drive grant. Nothing replies to this request.
@@ -2362,6 +2367,7 @@ def _op_push_blob(request):
 
     url = request.pop("url")
     headers = request.pop("headers", None) or {}
+    cancel = request.pop("cancel", None)
     path = _data_file(os.path.expanduser(request["dir"]), request["digest"])
     size = os.path.getsize(path)
     piece = int(request.get("piece") or (64 << 20))
@@ -2369,6 +2375,8 @@ def _op_push_blob(request):
     with open(path, "rb") as handle:
         offset = 0
         while offset < size:
+            if cancel is not None and cancel.is_set():
+                raise RuntimeError("the upload was abandoned")
             block = handle.read(piece)
             end = offset + len(block)
             span = "bytes %d-%d/%d" % (offset, end - 1, size)
@@ -2390,6 +2398,98 @@ def _op_push_blob(request):
     return {"ok": True, "value": {"id": remote_id, "size": size}}
 
 
+#: The one environment pack running beside the session's calls, or None. Spec "Project
+#: data / environment archive": the pack is a background job on the worker.
+_PACK_JOB = {}
+
+
+def _pack_push_run(job, request):
+    """Pack a directory, wait for the upload target, upload it, and remove the staging dir.
+
+    Runs on its own thread, so the worker's main thread keeps answering calls. The state
+    goes packing, packed, uploading, then done or failed. The staging directory is gone
+    before done or failed is reported.
+    """
+    import shutil
+
+    began = time.monotonic()
+    directory = os.path.expanduser(request["dir"])
+    result = None
+    try:
+        packed = _op_pack_blob(
+            {"path": request["path"], "dir": request["dir"], "gzip": False}
+        )["value"]
+        job["digest"], job["size"] = packed["digest"], packed["size"]
+        job["state"] = "packed"
+        job["ready"].wait()
+        target = job["target"]
+        if target is None:
+            raise RuntimeError("the pack was abandoned")
+        job["state"] = "uploading"
+        sent = _op_push_blob(
+            {
+                "dir": packed["dir"],
+                "digest": packed["digest"],
+                "url": target["url"],
+                "headers": target.get("headers") or {},
+                "piece": target.get("piece"),
+                "cancel": job["cancel"],
+            }
+        )["value"]
+        result = {"state": "done", "id": sent["id"], "size": sent["size"]}
+    except BaseException as exc:
+        result = {"state": "failed", "reason": str(exc) or type(exc).__name__}
+    finally:
+        shutil.rmtree(directory, ignore_errors=True)
+    result["seconds"] = time.monotonic() - began
+    job.update(result)
+
+
+def _op_pack_push_start(request):
+    """Start packing a directory on a thread of its own and return at once."""
+    import threading
+
+    current = _PACK_JOB.get("job")
+    if current is not None and current["state"] in ("packing", "packed", "uploading"):
+        return {"ok": False, "error": "a pack is already running", "traceback": ""}
+    job = {
+        "state": "packing",
+        "target": None,
+        "ready": threading.Event(),
+        "cancel": threading.Event(),
+    }
+    _PACK_JOB["job"] = job
+    threading.Thread(target=_pack_push_run, args=(job, request), daemon=True).start()
+    return {"ok": True, "value": None}
+
+
+def _op_pack_push_target(request):
+    """Hand the running pack its upload session, or abandon it when ``url`` is None."""
+    job = _PACK_JOB.get("job")
+    if job is None:
+        return {"ok": False, "error": "no pack is running", "traceback": ""}
+    if request.get("url") is None:
+        job["cancel"].set()
+        job["target"] = None
+    else:
+        job["target"] = {
+            "url": request["url"],
+            "headers": request.get("headers") or {},
+            "piece": request.get("piece"),
+        }
+    job["ready"].set()
+    return {"ok": True, "value": None}
+
+
+def _op_pack_push_state(request):
+    """Report the pack: its state, and the digest, size, file id, seconds or reason it has."""
+    job = _PACK_JOB.get("job")
+    if job is None:
+        return {"ok": True, "value": {"state": "none"}}
+    keys = ("state", "digest", "size", "id", "seconds", "reason")
+    return {"ok": True, "value": {key: job[key] for key in keys if key in job}}
+
+
 def _op_place_blob(request):
     """Move a received file blob to its destination path, and unpack it when asked.
 
@@ -2534,6 +2634,9 @@ _OPS = {
     "pack_dir": _op_pack_dir,
     "pack_blob": _op_pack_blob,
     "push_blob": _op_push_blob,
+    "pack_push_start": _op_pack_push_start,
+    "pack_push_target": _op_pack_push_target,
+    "pack_push_state": _op_pack_push_state,
     "place_blob": _op_place_blob,
     "exec": _op_exec,
     "eval": _op_eval,
@@ -2548,7 +2651,10 @@ _OPS = {
 #: Answered by the frame reader at once, so they work while a call runs.
 # ``data_failed`` is light because it must not queue behind the call it is about: the call
 # is blocked on a blob the sender will never send, so a queued answer would deadlock.
-_LIGHT = ("stat", "lease", "reduce_peers", "drive_grant", "data_failed")
+_LIGHT = (
+    "stat", "lease", "reduce_peers", "drive_grant", "data_failed",
+    "pack_push_state", "pack_push_target",
+)
 
 #: Answered by several replies, the last one marked. These send their own replies.
 _STREAMING = ("data_stream",)

{letify-2.8.0 → letify-2.8.2}/letify/providers/colab.py

@@ -121,6 +121,28 @@ def _access_token(stored: dict[str, Any]) -> str:
         return str(json.loads(response.read())["access_token"])
 
 
+#: Run in the CLI's own Python: release the server-side assignment at the endpoint in argv.
+#: ``colab stop`` cannot do this, because it needs the local session state an assignment
+#: made by a failed ``colab new`` never got. Spec "Colab", an assignment with no session.
+UNASSIGN_SCRIPT = (
+    "import sys\n"
+    "from colab_cli.common import state\n"
+    "state.client.unassign(sys.argv[1])\n"
+)
+
+
+def _parse_assignments(listing: str) -> list[tuple[str, str]]:
+    """The ``(name, endpoint)`` pairs in ``colab sessions`` output, ``?`` for no local state."""
+    found = []
+    for line in listing.splitlines():
+        head = line.strip().split("|")[0].split()
+        if len(head) >= 2 and head[0].startswith("[") and head[0].endswith("]"):
+            name = head[0][1:-1]
+            if name.lower() != "colab":
+                found.append((name, head[1]))
+    return found
+
+
 class Colab(Shell):
     """A Colab runtime for one Google account."""
 
@@ -316,6 +338,68 @@ class Colab(Shell):
                 names.append(word)
         return names
 
+    def _assignments(self) -> list[tuple[str, str]]:
+        """Every assignment the account holds, as ``(name, endpoint)``; ``?`` has no local state."""
+        return _parse_assignments(self._cli("sessions", timeout=120))
+
+    def _unassign(self, endpoint: str) -> None:
+        """Release one assignment by endpoint, through the CLI's own client library."""
+        uv = tools.find_uv()
+        if uv is None:
+            raise ProviderUnavailable(self.kind, tools.missing_uv_message())
+        command = [*tools.python_command(tools.COLAB, uv), "-c", UNASSIGN_SCRIPT, endpoint]
+        result = subprocess.run(
+            command, capture_output=True, text=True, timeout=180, env=self._env()
+        )
+        if result.returncode != 0:
+            raise RuntimeFailure(
+                f"unassigning {endpoint} exited {result.returncode}",
+                command=" ".join(command[:-3]),
+                stderr=(result.stderr or "").strip(),
+            )
+
+    def orphans(self) -> list[str]:
+        """Endpoints of assignments the account holds that no local session names."""
+        return [endpoint for name, endpoint in self._assignments() if name == "?"]
+
+    def stop_orphan(self, ref: str) -> bool:
+        """Unassign one endpoint `orphans` reported, and report whether it is gone."""
+        try:
+            self._unassign(ref)
+        except (RuntimeFailure, ProviderUnavailable, subprocess.TimeoutExpired):
+            return False
+        return True
+
+    def _release_leftovers(self, before: set[str]) -> None:
+        """Unassign each `?` assignment that appeared since ``before``, best effort."""
+        try:
+            leftovers = [
+                endpoint
+                for name, endpoint in self._assignments()
+                if name == "?" and endpoint not in before
+            ]
+        except (RuntimeFailure, ProviderUnavailable, subprocess.TimeoutExpired) as exc:
+            print(
+                f"letify: {self.alias}: the endpoint check after the failed 'colab new' "
+                f"could not list assignments ({exc}); an assignment may still be billing, "
+                "see 'letify stop --orphans'",
+                file=sys.stderr,
+            )
+            return
+        for endpoint in leftovers:
+            if self.stop_orphan(endpoint):
+                print(
+                    f"letify: {self.alias}: released {endpoint}, assigned by the failed "
+                    "'colab new'",
+                    file=sys.stderr,
+                )
+            else:
+                print(
+                    f"letify: {self.alias}: could not release {endpoint}, assigned by the "
+                    "failed 'colab new'; run 'letify stop --orphans'",
+                    file=sys.stderr,
+                )
+
     # -- the pipeline ----------------------------------------------------------
 
     def strategies(self) -> list[Strategy]:
@@ -388,7 +472,22 @@ class Colab(Shell):
             args += ["--gpu", ALIASES.get(instance.gpu, instance.gpu)]
         elif instance.tpu:
             args += ["--tpu", instance.tpu]
-        self._cli(*args, timeout=900)
+        try:
+            before = {endpoint for _, endpoint in self._assignments()}
+        except (RuntimeFailure, subprocess.TimeoutExpired) as exc:
+            before = None
+            print(
+                f"letify: {self.alias}: the endpoint check before 'colab new' could not "
+                f"list assignments ({exc}); a failed creation will not be cleaned up",
+                file=sys.stderr,
+            )
+        try:
+            self._cli(*args, timeout=900)
+        except BaseException:
+            # A timed out `colab new` can still leave a server-side assignment behind.
+            if before is not None:
+                self._release_leftovers(before)
+            raise
         try:
             self._start_keep_alive(name)
         except BaseException:

{letify-2.8.0 → letify-2.8.2}/letify/runtime/session.py

@@ -22,6 +22,7 @@ import datetime
 import json
 import posixpath
 import sys
+import threading
 import time
 from collections.abc import Callable, Iterator
 from dataclasses import dataclass, field
@@ -62,6 +63,16 @@ DRIVE_DOWNLOAD_BPS = 176 * 1024**2
 #: 170 MB/s on a sandbox disk. Spec "Materializing into a runtime".
 EXTRACT_BPS = 170_000_000
 
+#: How many times the estimated restore a build must have taken before the environment is
+#: packed to Drive. Spec "Materializing into a runtime": packing has a cost of its own.
+DRIVE_PACK_FACTOR = 2
+
+#: Seconds a closing session waits for a pack still running before it abandons it.
+PACK_CLOSE_WAIT_S = 60.0
+
+#: Seconds between the checks that hand a finished pack its upload session.
+PACK_POLL_S = 2.0
+
 
 def _against_the_build(size: int, rate: float, build_s: float) -> str:
     """The three numbers both cache decisions weigh, as they are reported.
@@ -87,6 +98,15 @@ def _sync_rather_than_restore(archive_bytes: int, rate: float, build_s: float) -
     )
 
 
+@dataclass
+class _DrivePack:
+    """A pack the worker is running for a Drive store, as this process tracks it."""
+
+    volume: Any
+    began: float = field(default_factory=time.monotonic)
+    target_sent: bool = False
+
+
 @dataclass
 class Runtime:
     """One live session on one provider."""
@@ -155,6 +175,11 @@ class Runtime:
     #: Set once ``shutdown`` has run, so a second discard does nothing.
     _shut_down: bool = field(default=False, repr=False)
 
+    #: The environment pack running on the worker, or None. Spec "Materializing into a
+    #: runtime": the pack to Drive runs beside the session's calls.
+    _pack: _DrivePack | None = field(default=None, repr=False)
+    _pack_lock: Any = field(default_factory=threading.RLock, repr=False)
+
     @property
     def key(self) -> str:
         """Pool key. Runtimes with equal keys are interchangeable."""
@@ -215,14 +240,12 @@ class Runtime:
                     + (f", {size / 1_000_000_000:.1f} GB" if size else "")
                 )
                 volume = self._environment_volumes()[0]
-                try:
+                if volume.store.backend.moves_on_runtime:
+                    self._start_drive_pack(volume, root)
+                else:
                     volume.cache_env_from(
                         self, self.env, root, platform=self.platform, build_s=self.env_build_s
                     )
-                except Exception as exc:
-                    if not volume.store.backend.moves_on_runtime:
-                        raise
-                    self._not_using_drive(exc, after="not caching the environment")
         self.keep_blobs_on_disk()
         for volume in self.volumes:
             self.attach(volume)
@@ -263,6 +286,7 @@ class Runtime:
         if self._shut_down:
             return
         self._shut_down = True
+        self._close_pack()
         timestamp = datetime.datetime.now(datetime.UTC).isoformat()
         print(
             f"{timestamp} letify: discarding {self.name} provider={self.provider.alias} "
@@ -371,9 +395,13 @@ class Runtime:
                 fn, args, kwargs, timeout=timeout, modules=self.env.ship_modules
             )
         args, kwargs = self._externalize(args, kwargs)
-        return self._call_with_data(
-            fn, args, kwargs, timeout, data_order, data_first_wave, replica
-        )
+        self._poll_pack()
+        try:
+            return self._call_with_data(
+                fn, args, kwargs, timeout, data_order, data_first_wave, replica
+            )
+        finally:
+            self._poll_pack()
 
     def _call_with_data(
         self,
@@ -695,46 +723,6 @@ class Runtime:
             )
         return digest, size
 
-    def pack_dir_to_store(self, path: str, backend: Any) -> tuple[str, int]:
-        """Pack a directory inside the runtime and have the runtime upload the archive.
-
-        Spec "Materializing into a runtime": the archive goes from the runtime's blob cache
-        to the backend without crossing this process. This process creates the upload
-        session, because it holds the credential, and the runtime is sent that session's
-        URL alone. Returns ``(digest, size)``. Needs a persistent channel, and raises
-        rather than routing the archive through this process.
-        """
-        if not self.persistent_channel:
-            raise RuntimeFailure("this channel keeps no worker that could upload the archive")
-        staging = f"{posixpath.dirname(path.rstrip('/'))}/.letify-pack"
-        value = self.request(
-            {"op": "pack_blob", "path": path, "dir": staging, "gzip": False}, timeout=3600
-        )
-        digest, size, directory = value["digest"], int(value["size"]), value["dir"]
-        try:
-            if not backend.has(digest):
-                target = backend.begin_upload(digest, size)
-                if target is None:
-                    raise RuntimeFailure(f"the {backend.name} backend offered no upload")
-                sent = self.request(
-                    {
-                        "op": "push_blob",
-                        "dir": directory,
-                        "digest": digest,
-                        "url": target["url"],
-                        "headers": dict(target.get("headers") or {}),  # type: ignore[call-overload]
-                        "piece": target.get("piece"),
-                    },
-                    timeout=3600,
-                )
-                backend.record_upload(digest, str(sent["id"]))
-        finally:
-            self.exec(
-                f"import shutil\nshutil.rmtree({directory!r}, ignore_errors=True)\n",
-                timeout=120,
-            )
-        return digest, size
-
     def put_path(
         self,
         local: Path,
@@ -937,24 +925,167 @@ class Runtime:
         return False
 
     def _worth_packing_to_drive(self, root: str) -> bool:
-        """Whether the build took longer than restoring the archive from Drive would.
+        """Whether the build took at least twice as long as restoring from Drive would.
 
         Spec "Materializing into a runtime": the archive goes from the runtime to Drive and
         never over this link, so the gate is the estimated restore, the project directory's
-        size over the Drive download rate plus the same size over the extraction rate.
+        size over the Drive download rate plus the same size over the extraction rate. The
+        build must take ``DRIVE_PACK_FACTOR`` times that, because the pack has a cost of its
+        own that the comparison leaves out.
         """
         size = self._tree_size(root)
         if size is None or self.env_build_s is None:
             return True
         restore_s = size / DRIVE_DOWNLOAD_BPS + size / EXTRACT_BPS
-        if self.env_build_s > restore_s:
+        if self.env_build_s >= DRIVE_PACK_FACTOR * restore_s:
             return True
         self._say(
             f"not caching the environment, because it built in {self.env_build_s:.0f} s "
-            f"and would take an estimated {restore_s:.0f} s to restore from Drive"
+            f"and would take an estimated {restore_s:.0f} s to restore from Drive, and a "
+            f"cache pays only when the build takes at least {DRIVE_PACK_FACTOR} times as long"
         )
         return False
 
+    # -- the pack that runs beside the calls ---------------------------------
+
+    def _start_drive_pack(self, volume: Volume, root: str) -> None:
+        """Have the worker pack the environment and upload it on a thread of its own.
+
+        Spec "Materializing into a runtime": this returns at once, so the first call does
+        not wait for the pack. The ref and the build record are written when a poll sees
+        the upload finish. A failure to start leaves nothing cached and says so.
+        """
+        staging = f"{posixpath.dirname(root.rstrip('/'))}/.letify-pack"
+        try:
+            if not self.persistent_channel:
+                raise RuntimeFailure("this channel keeps no worker that could upload the archive")
+            self.request(
+                {"op": "pack_push_start", "path": root, "dir": staging}, timeout=120
+            )
+        except Exception as exc:
+            self._not_using_drive(exc, after="not caching the environment")
+            return
+        self._pack = _DrivePack(volume)
+        threading.Thread(target=self._feed_pack, name="letify-pack", daemon=True).start()
+
+    def _feed_pack(self) -> None:
+        """Hand the pack its upload session soon after the archive is written.
+
+        A call can run for hours, and the upload cannot start until this process has made
+        the session, so this does only that. Reporting stays with the calls and the close.
+        """
+        while True:
+            time.sleep(PACK_POLL_S)
+            job = self._pack
+            if job is None or job.target_sent or self._shut_down:
+                return
+            self._poll_pack(report=False)
+
+    def _poll_pack(self, *, report: bool = True) -> bool:
+        """Ask the worker how the pack is doing and act on the answer.
+
+        A packed archive is given its upload session. A finished upload is recorded, and
+        reported when ``report`` is set. False when the worker could not be asked.
+        """
+        with self._pack_lock:
+            job = self._pack
+            if job is None or self.channel is None:
+                return True
+            try:
+                state = self.request({"op": "pack_push_state"}, timeout=30)
+            except Exception:
+                return False
+            kind = state.get("state")
+            backend = job.volume.store.backend
+            if kind == "packed" and not job.target_sent:
+                job.target_sent = True
+                try:
+                    if backend.has(state["digest"]):
+                        # Another session stored this archive first, so there is nothing to send.
+                        self._send_target(None)
+                        self._pack = None
+                        self._record_pack(job, state["digest"], int(state["size"]))
+                        return True
+                    target = backend.begin_upload(state["digest"], int(state["size"]))
+                    if target is None:
+                        raise RuntimeFailure(f"the {backend.name} backend offered no upload")
+                    self._send_target(target)
+                except Exception as exc:
+                    self._pack = None
+                    try:
+                        self._send_target(None)
+                    except Exception:
+                        pass
+                    self._not_using_drive(exc, after="not caching the environment")
+            elif kind == "done" and report:
+                self._pack = None
+                backend.record_upload(state["digest"], str(state["id"]))
+                self._record_pack(job, state["digest"], int(state["size"]))
+                seconds = float(state["seconds"])
+                size = int(state["size"])
+                self._say(
+                    f"cached the environment, {size / 1_000_000_000:.1f} GB in {seconds:.0f} s "
+                    f"({size / 1024**2 / max(seconds, 0.001):.1f} MiB/s)"
+                )
+            elif kind == "failed" and report:
+                self._pack = None
+                self._not_using_drive(
+                    RuntimeFailure(str(state.get("reason"))), after="not caching the environment"
+                )
+            elif kind == "none":
+                self._pack = None
+            return True
+
+    def _send_target(self, target: dict[str, Any] | None) -> None:
+        """Give the worker the upload session, or None to abandon the pack."""
+        payload: dict[str, Any] = {"op": "pack_push_target", "url": None}
+        if target is not None:
+            payload.update(
+                url=target["url"],
+                headers=dict(target.get("headers") or {}),  # type: ignore[call-overload]
+                piece=target.get("piece"),
+            )
+        self.request(payload, timeout=60)
+
+    def _record_pack(self, job: _DrivePack, digest: str, size: int) -> None:
+        """Write the ref and the build record, which only a finished upload earns."""
+        job.volume.record_env(
+            self.env, self.platform, digest, size, build_s=self.env_build_s
+        )
+
+    def _close_pack(self) -> None:
+        """Wait for a running pack, up to ``PACK_CLOSE_WAIT_S``, then abandon it.
+
+        Spec "Materializing into a runtime": an abandoned pack records nothing, so the
+        next session syncs and may pack again.
+        """
+        job = self._pack
+        if job is None:
+            return
+        deadline = time.monotonic() + PACK_CLOSE_WAIT_S
+        try:
+            while self._pack is not None:
+                if not self._poll_pack():
+                    self._pack = None
+                    return
+                if self._pack is None:
+                    return
+                if time.monotonic() >= deadline:
+                    break
+                time.sleep(0.2)
+            with self._pack_lock:
+                self._pack = None
+                try:
+                    self._send_target(None)
+                except Exception:
+                    pass
+            self._say(
+                "not caching the environment, because the session closed "
+                f"{time.monotonic() - job.began:.0f} s into the pack"
+            )
+        except Exception:
+            self._pack = None
+
     def _restore_from(self, volume: Volume, where: dict[str, Any]) -> bool:
         """Restore the environment from one volume's archive. True when the check passed.