letify 2.8.1__tar.gz → 2.8.3__tar.gz

{letify-2.8.1 → letify-2.8.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: letify
-Version: 2.8.1
+Version: 2.8.3
 Summary: Declarations that become infrastructure.
 Project-URL: Homepage, https://github.com/DarkPyonix/letify
 Project-URL: Documentation, https://github.com/DarkPyonix/letify/tree/main/docs

{letify-2.8.1 → letify-2.8.3}/docs/SPEC.md

@@ -1162,7 +1162,7 @@ The archive is `<workspace root>/env-archives/<env key>-<platform>.tar`, uncompr
 
 The client's inbound link is the weak one, so the store path is slower than the build in both directions. Compression costs more than it saves on the volume, so the archive is not compressed.
 
-After a successful sync and interpreter check, the runtime says `caching the environment on the workspace root, <size>`, runs tar from the project directory's parent into a temporary name beside the destination, and renames it into place, so a concurrent session never reads a half written archive. Nothing is sent to the client, and no measured rate gates the pack, because the transfer stays inside the provider.
+After a successful sync and interpreter check, the runtime says `caching the environment on the workspace root, <size>`, runs tar from the project directory's parent into a temporary name beside the destination, and renames it into place, so a concurrent session never reads a half written archive. The temporary name carries 16 random hex digits, `<archive>.<hex>.partial`, not the process id, because every Modal sandbox runs its worker under the same small pid, so two sessions packing the same environment at once wrote one file and one of them failed with tar exit status 2. A pack that fails for any reason removes its temporary file, says `letify: <runtime>: not caching the environment on the workspace root (<reason>)`, and the session goes on: the environment is already built and checked, so a failed cache never fails the session or its call. Nothing is sent to the client, and no measured rate gates the pack, because the transfer stays inside the provider.
 
 At the start of `install_env`, when that archive exists on the runtime, the runtime says `restoring the environment from the workspace root`, extracts it into the project directory's parent, and validates the interpreter with the same check an archive restore uses. The restore refuses a member whose path escapes the destination and allows the absolute symlinks a `.venv` needs. A directory an earlier session left under the same name is removed first. When validation fails, the runtime removes the extracted tree, syncs, and replaces the archive. `Runtime.env_source` is `"workspace"` after this restore. This path is taken before and instead of the store path, and such a provider never uses the client store or the bucket store for its environment.
 
@@ -1275,7 +1275,9 @@ Letify keeps its files in a folder named `letify` at the root of that Drive, one
 
 An ephemeral provider uses the Drive store for its environment archive when the machine declares one and the account sets no `bucket`, which keeps precedence. A provider whose environment survives the session on its own disk or volume does not use it. An archive on Drive is an uncompressed tar, because compression was most of the cost: on a Kaggle runtime, packing a 5.5 GB environment as tar.gz and uploading it took about 12 minutes, against an upload of roughly 77 s for the uncompressed tree at the measured rate.
 
-Whether an archive is worth it is measured, not assumed, because a provider with a fast package index can build faster than it restores. On a Kaggle runtime the same 5.5 GB environment built in 35.4 s and restored from Drive in about 90 s. So after a sync letify packs only when the build took longer than the archive would take to restore, estimated as the project directory's size over 176 MiB/s, the slower Drive download measured, plus the same size over 170 MB/s, the extraction rate measured on a sandbox disk. The record beside the archive keeps the build seconds and, once a session has restored from it, the restore seconds that session measured; a later session restores only while the measured restore is shorter than the measured build, and syncs otherwise, saying `letify: <runtime>: syncing rather than restoring from Drive, because a restore took <s> s against a <s> s build`.
+Whether an archive is worth it is measured, not assumed, because a provider with a fast package index can build faster than it restores. On a Kaggle runtime the same 5.5 GB environment built in 35.4 s and restored from Drive in about 90 s. So after a sync letify packs only when the build took at least twice as long as the archive would take to restore, estimated as the project directory's size over 176 MiB/s, the slower Drive download measured, plus the same size over 170 MB/s, the extraction rate measured on a sandbox disk. The record beside the archive keeps the build seconds and, once a session has restored from it, the restore seconds that session measured; a later session restores only while the measured restore is shorter than the measured build, and syncs otherwise, saying `letify: <runtime>: syncing rather than restoring from Drive, because a restore took <s> s against a <s> s build`. The factor of two is there because packing has a cost of its own that the comparison leaves out: on a Kaggle runtime a 7.2 GB environment that built in 87.6 s against an estimated 81 s restore was packed, which would save about 6 s a session at the price of the pack.
+
+The pack to Drive runs on the worker beside the session's calls, not before them. After the interpreter check the session starts the pack as a background job and returns, so the first call starts at once; the worker answers calls on its main thread while a separate thread writes the tar and uploads it, and the archive's ref and build record are written only when the upload completes. The upload session needs the archive's digest and size, so the worker reports them once the tar is written, and a client thread that checks every 2 s creates the resumable session and hands its URL to the worker, which then uploads; the client records and reports the result at the start and end of a call and at session close. A session that ends first leaves no ref, and the next session syncs and may pack again. While it runs the pack says nothing more after `caching the environment, <size> GB`; when it ends it says `letify: <runtime>: cached the environment, <size> GB in <s> s (<rate> MiB/s)`, or the Drive failure line above, on standard error of whichever call or session close observes it first. Closing the session waits up to 60 s for a pack still running and then abandons it, saying `letify: <runtime>: not caching the environment, because the session closed <s> s into the pack`.
 
 Drive's own limits apply. The archive counts against the Google account's storage, which is the person's own plan. Google documents a daily upload limit of 750 GB per user for Google Workspace accounts, independent of the storage plan; an environment archive is a few gigabytes, so letify does not approach it. Standard use of the Drive API costs nothing, and Google documents that exceeding its request quota is planned to be charged to the project's billing account later in 2026; letify's requests are a handful per session against a quota of 325,000 units per user per minute.
 
@@ -1910,11 +1912,26 @@ It is opt-in because it needs the user's machine to accept inbound SSH and needs
 
 The key is per session. letify generates a new ed25519 key pair for each session and appends the public key to the user's `~/.ssh/authorized_keys` with `restrict,port-forwarding,command="/bin/false"` and the comment `letify-session <alias>`. The private key reaches the remote side through the rendezvous. Closing the link removes the line. Before a new session's key is added, every line carrying the `letify-session` marker for that alias is removed, so a crashed session never leaves a key behind.
 
+#### Reverse SSH through a jump host <!-- id: reverse-ssh-jump -->
+
+> `reverse_ssh` takes an optional `jump`, for a user's machine whose SSH server the internet reaches only through another server.
+
+```toml
+[colab_pro_plus]
+reverse_ssh = { address = "168.188.127.175", port = 30008, user = "work", jump = { address = "jump.example.org", port = 20070, user = "root", key = "~/.ssh/id_jump" } }
+```
+
+With `jump`, `address` and `port` are the user's SSH server as the jump host reaches it, and the remote side connects to `<user>@<address>:<port>` through `-o ProxyCommand="ssh -o BatchMode=yes -o StrictHostKeyChecking=accept-new -i <session key> -W %h:%p -p <jump port> <jump user>@<jump address>"` and forwards its port back with `-R` as before. It is a ProxyCommand rather than `-J`, because the options given for the main hop on the command line do not reach a `-J` hop.
+
+The per-session key authorizes both hops. Besides the user's own `~/.ssh/authorized_keys`, letify appends the session's public key to the jump host's `~/.ssh/authorized_keys` with `restrict,port-forwarding,permitopen="<address>:<port>",command="/bin/false"` and the comment `letify-session <alias>`, so the key opens a connection to the user's SSH server and nothing else. letify reaches the jump host from the user's machine with `ssh -i <jump key> -p <jump port> <jump user>@<jump address>`, `BatchMode=yes`, and edits the file there with a shell command; `key` is optional and defaults to the SSH client's own keys. Closing the link removes that line, and before a new session's key is added every line carrying the `letify-session` marker for that alias is removed from the jump host too, as on the user's machine. The jump key never leaves the user's machine.
+
+A jump host that letify cannot log in to, or whose `authorized_keys` it cannot edit, fails the strategy with `reverse_ssh: the jump host <user>@<address>:<port> refused letify (<reason>)`, before anything is sent to the remote side.
+
 ### Colab <!-- id: colab-transport -->
 
 > Colab is a `Shell` whose rendezvous is `colab exec`. It has no forward SSH, and its fallback is `colab exec` with the Colab file API.
 
-The Colab CLI runs as `uv tool run --from google-colab-cli colab`, with `jupyter-kernel-client<1` pinned, because release 0.6.0 of the CLI calls an API that jupyter-kernel-client 1.0 removed. `colab new` and `colab stop` manage the session. `Colab.sessions()` reads `colab sessions` and returns the first word of each listing line. The CLI prints a session as `[<name>] <id> | Hardware: <hardware> | Variant: <variant>`, so a first word in square brackets gives the name inside them. A line starting with `[colab]` is a message from the CLI, such as `[colab] No active sessions found on server.`, and names no session, so an account with no session returns an empty list.
+The Colab CLI runs as `uv tool run --from google-colab-cli colab`, with `jupyter-kernel-client<1` pinned, because release 0.6.0 of the CLI calls an API that jupyter-kernel-client 1.0 removed. `colab new` and `colab stop` manage the session. `Colab.sessions()` reads `colab sessions` and returns the first word of each listing line. The CLI prints a session as `[<name>] <id> | Hardware: <hardware> | Variant: <variant>`, so a first word in square brackets gives the name inside them. A line starting with `[colab]` is a message from the CLI, such as `[colab] No active sessions found on server.`, and names no session, so an account with no session returns an empty list. `colab new` runs through the CLI's own Python with a request timeout of 600 s instead of google-auth's 120 s. google-auth binds `google.auth.transport.requests._DEFAULT_TIMEOUT` into `AuthorizedSession.request`'s signature when the module loads, so setting the module value alone changes nothing: the script sets it to 600 and wraps `AuthorizedSession.request` to default `timeout` to it, then calls the CLI's entry point `colab_cli.cli:main`. That is needed because the assignment request for an A100 can take longer than 120 s to answer and the CLI then fails with `ReadTimeout` while the server goes on assigning the machine. letify's own limit on `colab new` stays 900 s.
 
 **An assignment with no session.** `colab new` can fail after Colab has already assigned a machine, for example with `ReadTimeout` after 120 s, and the machine then bills (14.2 compute units per hour for an A100) with no local session state. `colab sessions` lists it as `[?] <endpoint> | ...`, and `colab stop -s <name>` cannot reach it. `Colab.create_session` therefore lists the account's assignments before `colab new`. When `colab new` raises, for any reason, it lists them again, releases every `?` assignment whose endpoint was not in the first list, and re-raises the original error. An assignment that existed before the attempt is never released. Each release prints one line on standard error naming the endpoint. When the first or the second listing fails, one line on standard error names the endpoint check and the error, and the original error is still the one raised. An assignment is released by endpoint with `Client.unassign` of the Colab CLI, run as `uv tool run --from google-colab-cli python -c <script> <endpoint>` with the account's environment, because the CLI has no command for it. `Colab.orphans()` returns the endpoints of the `?` rows of `colab sessions` and `Colab.stop_orphan(endpoint)` releases one, so `letify stop --orphans` covers Colab as it covers Kaggle.
 

{letify-2.8.1 → letify-2.8.3}/letify/__init__.py

@@ -68,7 +68,7 @@ distributed: Final = Host.distributed
 # The enum class stays importable from letify.declare.instance for letify's own use.
 del Host
 
-__version__ = "2.8.1"
+__version__ = "2.8.3"
 
 
 def fetch(tensor):  # type: ignore[no-untyped-def]

{letify-2.8.1 → letify-2.8.3}/letify/protocol/worker.py

@@ -2367,6 +2367,7 @@ def _op_push_blob(request):
 
     url = request.pop("url")
     headers = request.pop("headers", None) or {}
+    cancel = request.pop("cancel", None)
     path = _data_file(os.path.expanduser(request["dir"]), request["digest"])
     size = os.path.getsize(path)
     piece = int(request.get("piece") or (64 << 20))
@@ -2374,6 +2375,8 @@ def _op_push_blob(request):
     with open(path, "rb") as handle:
         offset = 0
         while offset < size:
+            if cancel is not None and cancel.is_set():
+                raise RuntimeError("the upload was abandoned")
             block = handle.read(piece)
             end = offset + len(block)
             span = "bytes %d-%d/%d" % (offset, end - 1, size)
@@ -2395,6 +2398,98 @@ def _op_push_blob(request):
     return {"ok": True, "value": {"id": remote_id, "size": size}}
 
 
+#: The one environment pack running beside the session's calls, or None. Spec "Project
+#: data / environment archive": the pack is a background job on the worker.
+_PACK_JOB = {}
+
+
+def _pack_push_run(job, request):
+    """Pack a directory, wait for the upload target, upload it, and remove the staging dir.
+
+    Runs on its own thread, so the worker's main thread keeps answering calls. The state
+    goes packing, packed, uploading, then done or failed. The staging directory is gone
+    before done or failed is reported.
+    """
+    import shutil
+
+    began = time.monotonic()
+    directory = os.path.expanduser(request["dir"])
+    result = None
+    try:
+        packed = _op_pack_blob(
+            {"path": request["path"], "dir": request["dir"], "gzip": False}
+        )["value"]
+        job["digest"], job["size"] = packed["digest"], packed["size"]
+        job["state"] = "packed"
+        job["ready"].wait()
+        target = job["target"]
+        if target is None:
+            raise RuntimeError("the pack was abandoned")
+        job["state"] = "uploading"
+        sent = _op_push_blob(
+            {
+                "dir": packed["dir"],
+                "digest": packed["digest"],
+                "url": target["url"],
+                "headers": target.get("headers") or {},
+                "piece": target.get("piece"),
+                "cancel": job["cancel"],
+            }
+        )["value"]
+        result = {"state": "done", "id": sent["id"], "size": sent["size"]}
+    except BaseException as exc:
+        result = {"state": "failed", "reason": str(exc) or type(exc).__name__}
+    finally:
+        shutil.rmtree(directory, ignore_errors=True)
+    result["seconds"] = time.monotonic() - began
+    job.update(result)
+
+
+def _op_pack_push_start(request):
+    """Start packing a directory on a thread of its own and return at once."""
+    import threading
+
+    current = _PACK_JOB.get("job")
+    if current is not None and current["state"] in ("packing", "packed", "uploading"):
+        return {"ok": False, "error": "a pack is already running", "traceback": ""}
+    job = {
+        "state": "packing",
+        "target": None,
+        "ready": threading.Event(),
+        "cancel": threading.Event(),
+    }
+    _PACK_JOB["job"] = job
+    threading.Thread(target=_pack_push_run, args=(job, request), daemon=True).start()
+    return {"ok": True, "value": None}
+
+
+def _op_pack_push_target(request):
+    """Hand the running pack its upload session, or abandon it when ``url`` is None."""
+    job = _PACK_JOB.get("job")
+    if job is None:
+        return {"ok": False, "error": "no pack is running", "traceback": ""}
+    if request.get("url") is None:
+        job["cancel"].set()
+        job["target"] = None
+    else:
+        job["target"] = {
+            "url": request["url"],
+            "headers": request.get("headers") or {},
+            "piece": request.get("piece"),
+        }
+    job["ready"].set()
+    return {"ok": True, "value": None}
+
+
+def _op_pack_push_state(request):
+    """Report the pack: its state, and the digest, size, file id, seconds or reason it has."""
+    job = _PACK_JOB.get("job")
+    if job is None:
+        return {"ok": True, "value": {"state": "none"}}
+    keys = ("state", "digest", "size", "id", "seconds", "reason")
+    return {"ok": True, "value": {key: job[key] for key in keys if key in job}}
+
+
 def _op_place_blob(request):
     """Move a received file blob to its destination path, and unpack it when asked.
 
@@ -2539,6 +2634,9 @@ _OPS = {
     "pack_dir": _op_pack_dir,
     "pack_blob": _op_pack_blob,
     "push_blob": _op_push_blob,
+    "pack_push_start": _op_pack_push_start,
+    "pack_push_target": _op_pack_push_target,
+    "pack_push_state": _op_pack_push_state,
     "place_blob": _op_place_blob,
     "exec": _op_exec,
     "eval": _op_eval,
@@ -2553,7 +2651,10 @@ _OPS = {
 #: Answered by the frame reader at once, so they work while a call runs.
 # ``data_failed`` is light because it must not queue behind the call it is about: the call
 # is blocked on a blob the sender will never send, so a queued answer would deadlock.
-_LIGHT = ("stat", "lease", "reduce_peers", "drive_grant", "data_failed")
+_LIGHT = (
+    "stat", "lease", "reduce_peers", "drive_grant", "data_failed",
+    "pack_push_state", "pack_push_target",
+)
 
 #: Answered by several replies, the last one marked. These send their own replies.
 _STREAMING = ("data_stream",)

{letify-2.8.1 → letify-2.8.3}/letify/providers/colab.py

@@ -131,6 +131,26 @@ UNASSIGN_SCRIPT = (
 )
 
 
+#: Run in the CLI's own Python: ``colab`` with google-auth's request timeout raised to 600 s.
+#: The CLI passes no timeout, so a slow A100 assignment fails at the 120 s default while the
+#: server goes on assigning. google-auth binds the 120 s default into the signature of
+#: ``AuthorizedSession.request`` when it is defined, so setting the module value alone changes
+#: nothing and the method is wrapped as well. Spec "Colab", the Colab CLI paragraph.
+NEW_SCRIPT = (
+    "import sys\n"
+    "import google.auth.transport.requests as requests\n"
+    "requests._DEFAULT_TIMEOUT = 600\n"
+    "_request = requests.AuthorizedSession.request\n"
+    "def _slow_request(self, *args, **kwargs):\n"
+    "    kwargs.setdefault('timeout', requests._DEFAULT_TIMEOUT)\n"
+    "    return _request(self, *args, **kwargs)\n"
+    "requests.AuthorizedSession.request = _slow_request\n"
+    "from colab_cli.cli import main\n"
+    "sys.argv = ['colab', *sys.argv[1:]]\n"
+    "sys.exit(main())\n"
+)
+
+
 def _parse_assignments(listing: str) -> list[tuple[str, str]]:
     """The ``(name, endpoint)`` pairs in ``colab sessions`` output, ``?`` for no local state."""
     found = []
@@ -275,8 +295,21 @@ class Colab(Shell):
 
     # -- the CLI -------------------------------------------------------------
 
-    def _cli(self, *args: str, timeout: float | None = None, stdin: str | None = None) -> str:
-        command = [*self._colab(), *args]
+    def _cli(
+        self,
+        *args: str,
+        timeout: float | None = None,
+        stdin: str | None = None,
+        script: str | None = None,
+    ) -> str:
+        """Run the CLI; with ``script``, run that code in the CLI's Python with ``args`` as argv."""
+        if script is None:
+            command = [*self._colab(), *args]
+        else:
+            uv = tools.find_uv()
+            if uv is None:
+                raise ProviderUnavailable(self.kind, tools.missing_uv_message())
+            command = [*tools.python_command(tools.COLAB, uv), "-c", script, *args]
         result = subprocess.run(
             command,
             input=stdin,
@@ -286,9 +319,10 @@ class Colab(Shell):
             env=self._env(),
         )
         if result.returncode != 0:
+            shown = ["<script>" if part == script else part for part in command]
             raise RuntimeFailure(
-                f"`{' '.join(command)}` exited {result.returncode}",
-                command=" ".join(command),
+                f"`{' '.join(shown)}` exited {result.returncode}",
+                command=" ".join(shown),
                 stderr=(result.stderr or "").strip(),
             )
         return result.stdout
@@ -482,7 +516,7 @@ class Colab(Shell):
                 file=sys.stderr,
             )
         try:
-            self._cli(*args, timeout=900)
+            self._cli(*args, timeout=900, script=NEW_SCRIPT)
         except BaseException:
             # A timed out `colab new` can still leave a server-side assignment behind.
             if before is not None:

{letify-2.8.1 → letify-2.8.3}/letify/runtime/bootstrap.py

@@ -281,13 +281,15 @@ def workspace_pack_source(root: str, archive: str) -> str:
     Spec "Volumes on a persistent runtime": runs ``tar`` from the directory's parent into a
     temporary name and renames it into place, so a concurrent session never reads a half
     written archive. Uncompressed, because gzip costs more than it saves on a local volume.
+    The temporary name carries random hex, not the process id, because every Modal sandbox
+    runs its worker under the same pid. A failed pack removes its temporary file.
     """
     return (
-        "import os, subprocess\n"
+        "import os, secrets, subprocess\n"
         f"_root = os.path.expanduser({root!r}).rstrip('/')\n"
         f"_archive = os.path.expanduser({archive!r})\n"
         "os.makedirs(os.path.dirname(_archive), exist_ok=True)\n"
-        "_partial = _archive + '.' + str(os.getpid()) + '.partial'\n"
+        "_partial = _archive + '.' + secrets.token_hex(8) + '.partial'\n"
         "try:\n"
         "    subprocess.run(['tar', '-cf', _partial, '-C', os.path.dirname(_root),\n"
         "                    os.path.basename(_root)], check=True, capture_output=True)\n"

{letify-2.8.1 → letify-2.8.3}/letify/runtime/session.py

@@ -22,6 +22,7 @@ import datetime
 import json
 import posixpath
 import sys
+import threading
 import time
 from collections.abc import Callable, Iterator
 from dataclasses import dataclass, field
@@ -62,6 +63,16 @@ DRIVE_DOWNLOAD_BPS = 176 * 1024**2
 #: 170 MB/s on a sandbox disk. Spec "Materializing into a runtime".
 EXTRACT_BPS = 170_000_000
 
+#: How many times the estimated restore a build must have taken before the environment is
+#: packed to Drive. Spec "Materializing into a runtime": packing has a cost of its own.
+DRIVE_PACK_FACTOR = 2
+
+#: Seconds a closing session waits for a pack still running before it abandons it.
+PACK_CLOSE_WAIT_S = 60.0
+
+#: Seconds between the checks that hand a finished pack its upload session.
+PACK_POLL_S = 2.0
+
 
 def _against_the_build(size: int, rate: float, build_s: float) -> str:
     """The three numbers both cache decisions weigh, as they are reported.
@@ -87,6 +98,15 @@ def _sync_rather_than_restore(archive_bytes: int, rate: float, build_s: float) -
     )
 
 
+@dataclass
+class _DrivePack:
+    """A pack the worker is running for a Drive store, as this process tracks it."""
+
+    volume: Any
+    began: float = field(default_factory=time.monotonic)
+    target_sent: bool = False
+
+
 @dataclass
 class Runtime:
     """One live session on one provider."""
@@ -155,6 +175,11 @@ class Runtime:
     #: Set once ``shutdown`` has run, so a second discard does nothing.
     _shut_down: bool = field(default=False, repr=False)
 
+    #: The environment pack running on the worker, or None. Spec "Materializing into a
+    #: runtime": the pack to Drive runs beside the session's calls.
+    _pack: _DrivePack | None = field(default=None, repr=False)
+    _pack_lock: Any = field(default_factory=threading.RLock, repr=False)
+
     @property
     def key(self) -> str:
         """Pool key. Runtimes with equal keys are interchangeable."""
@@ -215,14 +240,12 @@ class Runtime:
                     + (f", {size / 1_000_000_000:.1f} GB" if size else "")
                 )
                 volume = self._environment_volumes()[0]
-                try:
+                if volume.store.backend.moves_on_runtime:
+                    self._start_drive_pack(volume, root)
+                else:
                     volume.cache_env_from(
                         self, self.env, root, platform=self.platform, build_s=self.env_build_s
                     )
-                except Exception as exc:
-                    if not volume.store.backend.moves_on_runtime:
-                        raise
-                    self._not_using_drive(exc, after="not caching the environment")
         self.keep_blobs_on_disk()
         for volume in self.volumes:
             self.attach(volume)
@@ -263,6 +286,7 @@ class Runtime:
         if self._shut_down:
             return
         self._shut_down = True
+        self._close_pack()
         timestamp = datetime.datetime.now(datetime.UTC).isoformat()
         print(
             f"{timestamp} letify: discarding {self.name} provider={self.provider.alias} "
@@ -371,9 +395,13 @@ class Runtime:
                 fn, args, kwargs, timeout=timeout, modules=self.env.ship_modules
             )
         args, kwargs = self._externalize(args, kwargs)
-        return self._call_with_data(
-            fn, args, kwargs, timeout, data_order, data_first_wave, replica
-        )
+        self._poll_pack()
+        try:
+            return self._call_with_data(
+                fn, args, kwargs, timeout, data_order, data_first_wave, replica
+            )
+        finally:
+            self._poll_pack()
 
     def _call_with_data(
         self,
@@ -695,46 +723,6 @@ class Runtime:
             )
         return digest, size
 
-    def pack_dir_to_store(self, path: str, backend: Any) -> tuple[str, int]:
-        """Pack a directory inside the runtime and have the runtime upload the archive.
-
-        Spec "Materializing into a runtime": the archive goes from the runtime's blob cache
-        to the backend without crossing this process. This process creates the upload
-        session, because it holds the credential, and the runtime is sent that session's
-        URL alone. Returns ``(digest, size)``. Needs a persistent channel, and raises
-        rather than routing the archive through this process.
-        """
-        if not self.persistent_channel:
-            raise RuntimeFailure("this channel keeps no worker that could upload the archive")
-        staging = f"{posixpath.dirname(path.rstrip('/'))}/.letify-pack"
-        value = self.request(
-            {"op": "pack_blob", "path": path, "dir": staging, "gzip": False}, timeout=3600
-        )
-        digest, size, directory = value["digest"], int(value["size"]), value["dir"]
-        try:
-            if not backend.has(digest):
-                target = backend.begin_upload(digest, size)
-                if target is None:
-                    raise RuntimeFailure(f"the {backend.name} backend offered no upload")
-                sent = self.request(
-                    {
-                        "op": "push_blob",
-                        "dir": directory,
-                        "digest": digest,
-                        "url": target["url"],
-                        "headers": dict(target.get("headers") or {}),  # type: ignore[call-overload]
-                        "piece": target.get("piece"),
-                    },
-                    timeout=3600,
-                )
-                backend.record_upload(digest, str(sent["id"]))
-        finally:
-            self.exec(
-                f"import shutil\nshutil.rmtree({directory!r}, ignore_errors=True)\n",
-                timeout=120,
-            )
-        return digest, size
-
     def put_path(
         self,
         local: Path,
@@ -838,7 +826,16 @@ class Runtime:
             "caching the environment on the workspace root"
             + (f", {size / 1_000_000_000:.1f} GB" if size else "")
         )
-        self.eval(bootstrap.workspace_pack_source(root, self._workspace_archive()), timeout=3600)
+        source = bootstrap.workspace_pack_source(root, self._workspace_archive())
+        try:
+            self.eval(source, timeout=3600)
+        except Exception as error:
+            # The environment is built and checked, so a failed cache must not fail the call.
+            lines = str(error).strip().splitlines()
+            reason = f"{type(error).__name__}: {lines[0]}" if lines else type(error).__name__
+            if len(reason) > 200:
+                reason = reason[:197] + "..."
+            self._say(f"not caching the environment on the workspace root ({reason})")
 
     def _environment_volumes(self) -> tuple[Volume, ...]:
         """Explicit archive volumes, or the automatic cache where the project does not survive.
@@ -937,24 +934,167 @@ class Runtime:
         return False
 
     def _worth_packing_to_drive(self, root: str) -> bool:
-        """Whether the build took longer than restoring the archive from Drive would.
+        """Whether the build took at least twice as long as restoring from Drive would.
 
         Spec "Materializing into a runtime": the archive goes from the runtime to Drive and
         never over this link, so the gate is the estimated restore, the project directory's
-        size over the Drive download rate plus the same size over the extraction rate.
+        size over the Drive download rate plus the same size over the extraction rate. The
+        build must take ``DRIVE_PACK_FACTOR`` times that, because the pack has a cost of its
+        own that the comparison leaves out.
         """
         size = self._tree_size(root)
         if size is None or self.env_build_s is None:
             return True
         restore_s = size / DRIVE_DOWNLOAD_BPS + size / EXTRACT_BPS
-        if self.env_build_s > restore_s:
+        if self.env_build_s >= DRIVE_PACK_FACTOR * restore_s:
             return True
         self._say(
             f"not caching the environment, because it built in {self.env_build_s:.0f} s "
-            f"and would take an estimated {restore_s:.0f} s to restore from Drive"
+            f"and would take an estimated {restore_s:.0f} s to restore from Drive, and a "
+            f"cache pays only when the build takes at least {DRIVE_PACK_FACTOR} times as long"
         )
         return False
 
+    # -- the pack that runs beside the calls ---------------------------------
+
+    def _start_drive_pack(self, volume: Volume, root: str) -> None:
+        """Have the worker pack the environment and upload it on a thread of its own.
+
+        Spec "Materializing into a runtime": this returns at once, so the first call does
+        not wait for the pack. The ref and the build record are written when a poll sees
+        the upload finish. A failure to start leaves nothing cached and says so.
+        """
+        staging = f"{posixpath.dirname(root.rstrip('/'))}/.letify-pack"
+        try:
+            if not self.persistent_channel:
+                raise RuntimeFailure("this channel keeps no worker that could upload the archive")
+            self.request(
+                {"op": "pack_push_start", "path": root, "dir": staging}, timeout=120
+            )
+        except Exception as exc:
+            self._not_using_drive(exc, after="not caching the environment")
+            return
+        self._pack = _DrivePack(volume)
+        threading.Thread(target=self._feed_pack, name="letify-pack", daemon=True).start()
+
+    def _feed_pack(self) -> None:
+        """Hand the pack its upload session soon after the archive is written.
+
+        A call can run for hours, and the upload cannot start until this process has made
+        the session, so this does only that. Reporting stays with the calls and the close.
+        """
+        while True:
+            time.sleep(PACK_POLL_S)
+            job = self._pack
+            if job is None or job.target_sent or self._shut_down:
+                return
+            self._poll_pack(report=False)
+
+    def _poll_pack(self, *, report: bool = True) -> bool:
+        """Ask the worker how the pack is doing and act on the answer.
+
+        A packed archive is given its upload session. A finished upload is recorded, and
+        reported when ``report`` is set. False when the worker could not be asked.
+        """
+        with self._pack_lock:
+            job = self._pack
+            if job is None or self.channel is None:
+                return True
+            try:
+                state = self.request({"op": "pack_push_state"}, timeout=30)
+            except Exception:
+                return False
+            kind = state.get("state")
+            backend = job.volume.store.backend
+            if kind == "packed" and not job.target_sent:
+                job.target_sent = True
+                try:
+                    if backend.has(state["digest"]):
+                        # Another session stored this archive first, so there is nothing to send.
+                        self._send_target(None)
+                        self._pack = None
+                        self._record_pack(job, state["digest"], int(state["size"]))
+                        return True
+                    target = backend.begin_upload(state["digest"], int(state["size"]))
+                    if target is None:
+                        raise RuntimeFailure(f"the {backend.name} backend offered no upload")
+                    self._send_target(target)
+                except Exception as exc:
+                    self._pack = None
+                    try:
+                        self._send_target(None)
+                    except Exception:
+                        pass
+                    self._not_using_drive(exc, after="not caching the environment")
+            elif kind == "done" and report:
+                self._pack = None
+                backend.record_upload(state["digest"], str(state["id"]))
+                self._record_pack(job, state["digest"], int(state["size"]))
+                seconds = float(state["seconds"])
+                size = int(state["size"])
+                self._say(
+                    f"cached the environment, {size / 1_000_000_000:.1f} GB in {seconds:.0f} s "
+                    f"({size / 1024**2 / max(seconds, 0.001):.1f} MiB/s)"
+                )
+            elif kind == "failed" and report:
+                self._pack = None
+                self._not_using_drive(
+                    RuntimeFailure(str(state.get("reason"))), after="not caching the environment"
+                )
+            elif kind == "none":
+                self._pack = None
+            return True
+
+    def _send_target(self, target: dict[str, Any] | None) -> None:
+        """Give the worker the upload session, or None to abandon the pack."""
+        payload: dict[str, Any] = {"op": "pack_push_target", "url": None}
+        if target is not None:
+            payload.update(
+                url=target["url"],
+                headers=dict(target.get("headers") or {}),  # type: ignore[call-overload]
+                piece=target.get("piece"),
+            )
+        self.request(payload, timeout=60)
+
+    def _record_pack(self, job: _DrivePack, digest: str, size: int) -> None:
+        """Write the ref and the build record, which only a finished upload earns."""
+        job.volume.record_env(
+            self.env, self.platform, digest, size, build_s=self.env_build_s
+        )
+
+    def _close_pack(self) -> None:
+        """Wait for a running pack, up to ``PACK_CLOSE_WAIT_S``, then abandon it.
+
+        Spec "Materializing into a runtime": an abandoned pack records nothing, so the
+        next session syncs and may pack again.
+        """
+        job = self._pack
+        if job is None:
+            return
+        deadline = time.monotonic() + PACK_CLOSE_WAIT_S
+        try:
+            while self._pack is not None:
+                if not self._poll_pack():
+                    self._pack = None
+                    return
+                if self._pack is None:
+                    return
+                if time.monotonic() >= deadline:
+                    break
+                time.sleep(0.2)
+            with self._pack_lock:
+                self._pack = None
+                try:
+                    self._send_target(None)
+                except Exception:
+                    pass
+            self._say(
+                "not caching the environment, because the session closed "
+                f"{time.monotonic() - job.began:.0f} s into the pack"
+            )
+        except Exception:
+            self._pack = None
+
     def _restore_from(self, volume: Volume, where: dict[str, Any]) -> bool:
         """Restore the environment from one volume's archive. True when the check passed.