letify 2.8.1__tar.gz → 2.8.2__tar.gz

{letify-2.8.1 → letify-2.8.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: letify
-Version: 2.8.1
+Version: 2.8.2
 Summary: Declarations that become infrastructure.
 Project-URL: Homepage, https://github.com/DarkPyonix/letify
 Project-URL: Documentation, https://github.com/DarkPyonix/letify/tree/main/docs

{letify-2.8.1 → letify-2.8.2}/docs/SPEC.md

@@ -1275,7 +1275,9 @@ Letify keeps its files in a folder named `letify` at the root of that Drive, one
 
 An ephemeral provider uses the Drive store for its environment archive when the machine declares one and the account sets no `bucket`, which keeps precedence. A provider whose environment survives the session on its own disk or volume does not use it. An archive on Drive is an uncompressed tar, because compression was most of the cost: on a Kaggle runtime, packing a 5.5 GB environment as tar.gz and uploading it took about 12 minutes, against an upload of roughly 77 s for the uncompressed tree at the measured rate.
 
-Whether an archive is worth it is measured, not assumed, because a provider with a fast package index can build faster than it restores. On a Kaggle runtime the same 5.5 GB environment built in 35.4 s and restored from Drive in about 90 s. So after a sync letify packs only when the build took longer than the archive would take to restore, estimated as the project directory's size over 176 MiB/s, the slower Drive download measured, plus the same size over 170 MB/s, the extraction rate measured on a sandbox disk. The record beside the archive keeps the build seconds and, once a session has restored from it, the restore seconds that session measured; a later session restores only while the measured restore is shorter than the measured build, and syncs otherwise, saying `letify: <runtime>: syncing rather than restoring from Drive, because a restore took <s> s against a <s> s build`.
+Whether an archive is worth it is measured, not assumed, because a provider with a fast package index can build faster than it restores. On a Kaggle runtime the same 5.5 GB environment built in 35.4 s and restored from Drive in about 90 s. So after a sync letify packs only when the build took at least twice as long as the archive would take to restore, estimated as the project directory's size over 176 MiB/s, the slower Drive download measured, plus the same size over 170 MB/s, the extraction rate measured on a sandbox disk. The record beside the archive keeps the build seconds and, once a session has restored from it, the restore seconds that session measured; a later session restores only while the measured restore is shorter than the measured build, and syncs otherwise, saying `letify: <runtime>: syncing rather than restoring from Drive, because a restore took <s> s against a <s> s build`. The factor of two is there because packing has a cost of its own that the comparison leaves out: on a Kaggle runtime a 7.2 GB environment that built in 87.6 s against an estimated 81 s restore was packed, which would save about 6 s a session at the price of the pack.
+
+The pack to Drive runs on the worker beside the session's calls, not before them. After the interpreter check the session starts the pack as a background job and returns, so the first call starts at once; the worker answers calls on its main thread while a separate thread writes the tar and uploads it, and the archive's ref and build record are written only when the upload completes. The upload session needs the archive's digest and size, so the worker reports them once the tar is written, and a client thread that checks every 2 s creates the resumable session and hands its URL to the worker, which then uploads; the client records and reports the result at the start and end of a call and at session close. A session that ends first leaves no ref, and the next session syncs and may pack again. While it runs the pack says nothing more after `caching the environment, <size> GB`; when it ends it says `letify: <runtime>: cached the environment, <size> GB in <s> s (<rate> MiB/s)`, or the Drive failure line above, on standard error of whichever call or session close observes it first. Closing the session waits up to 60 s for a pack still running and then abandons it, saying `letify: <runtime>: not caching the environment, because the session closed <s> s into the pack`.
 
 Drive's own limits apply. The archive counts against the Google account's storage, which is the person's own plan. Google documents a daily upload limit of 750 GB per user for Google Workspace accounts, independent of the storage plan; an environment archive is a few gigabytes, so letify does not approach it. Standard use of the Drive API costs nothing, and Google documents that exceeding its request quota is planned to be charged to the project's billing account later in 2026; letify's requests are a handful per session against a quota of 325,000 units per user per minute.
 
@@ -1910,6 +1912,21 @@ It is opt-in because it needs the user's machine to accept inbound SSH and needs
 
 The key is per session. letify generates a new ed25519 key pair for each session and appends the public key to the user's `~/.ssh/authorized_keys` with `restrict,port-forwarding,command="/bin/false"` and the comment `letify-session <alias>`. The private key reaches the remote side through the rendezvous. Closing the link removes the line. Before a new session's key is added, every line carrying the `letify-session` marker for that alias is removed, so a crashed session never leaves a key behind.
 
+#### Reverse SSH through a jump host <!-- id: reverse-ssh-jump -->
+
+> `reverse_ssh` takes an optional `jump`, for a user's machine whose SSH server the internet reaches only through another server.
+
+```toml
+[colab_pro_plus]
+reverse_ssh = { address = "168.188.127.175", port = 30008, user = "work", jump = { address = "jump.example.org", port = 20070, user = "root", key = "~/.ssh/id_jump" } }
+```
+
+With `jump`, `address` and `port` are the user's SSH server as the jump host reaches it, and the remote side connects to `<user>@<address>:<port>` through `-o ProxyCommand="ssh -o BatchMode=yes -o StrictHostKeyChecking=accept-new -i <session key> -W %h:%p -p <jump port> <jump user>@<jump address>"` and forwards its port back with `-R` as before. It is a ProxyCommand rather than `-J`, because the options given for the main hop on the command line do not reach a `-J` hop.
+
+The per-session key authorizes both hops. Besides the user's own `~/.ssh/authorized_keys`, letify appends the session's public key to the jump host's `~/.ssh/authorized_keys` with `restrict,port-forwarding,permitopen="<address>:<port>",command="/bin/false"` and the comment `letify-session <alias>`, so the key opens a connection to the user's SSH server and nothing else. letify reaches the jump host from the user's machine with `ssh -i <jump key> -p <jump port> <jump user>@<jump address>`, `BatchMode=yes`, and edits the file there with a shell command; `key` is optional and defaults to the SSH client's own keys. Closing the link removes that line, and before a new session's key is added every line carrying the `letify-session` marker for that alias is removed from the jump host too, as on the user's machine. The jump key never leaves the user's machine.
+
+A jump host that letify cannot log in to, or whose `authorized_keys` it cannot edit, fails the strategy with `reverse_ssh: the jump host <user>@<address>:<port> refused letify (<reason>)`, before anything is sent to the remote side.
+
 ### Colab <!-- id: colab-transport -->
 
 > Colab is a `Shell` whose rendezvous is `colab exec`. It has no forward SSH, and its fallback is `colab exec` with the Colab file API.

{letify-2.8.1 → letify-2.8.2}/letify/__init__.py

@@ -68,7 +68,7 @@ distributed: Final = Host.distributed
 # The enum class stays importable from letify.declare.instance for letify's own use.
 del Host
 
-__version__ = "2.8.1"
+__version__ = "2.8.2"
 
 
 def fetch(tensor):  # type: ignore[no-untyped-def]

{letify-2.8.1 → letify-2.8.2}/letify/protocol/worker.py

@@ -2367,6 +2367,7 @@ def _op_push_blob(request):
 
     url = request.pop("url")
     headers = request.pop("headers", None) or {}
+    cancel = request.pop("cancel", None)
     path = _data_file(os.path.expanduser(request["dir"]), request["digest"])
     size = os.path.getsize(path)
     piece = int(request.get("piece") or (64 << 20))
@@ -2374,6 +2375,8 @@ def _op_push_blob(request):
     with open(path, "rb") as handle:
         offset = 0
         while offset < size:
+            if cancel is not None and cancel.is_set():
+                raise RuntimeError("the upload was abandoned")
             block = handle.read(piece)
             end = offset + len(block)
             span = "bytes %d-%d/%d" % (offset, end - 1, size)
@@ -2395,6 +2398,98 @@ def _op_push_blob(request):
     return {"ok": True, "value": {"id": remote_id, "size": size}}
 
 
+#: The one environment pack running beside the session's calls, or None. Spec "Project
+#: data / environment archive": the pack is a background job on the worker.
+_PACK_JOB = {}
+
+
+def _pack_push_run(job, request):
+    """Pack a directory, wait for the upload target, upload it, and remove the staging dir.
+
+    Runs on its own thread, so the worker's main thread keeps answering calls. The state
+    goes packing, packed, uploading, then done or failed. The staging directory is gone
+    before done or failed is reported.
+    """
+    import shutil
+
+    began = time.monotonic()
+    directory = os.path.expanduser(request["dir"])
+    result = None
+    try:
+        packed = _op_pack_blob(
+            {"path": request["path"], "dir": request["dir"], "gzip": False}
+        )["value"]
+        job["digest"], job["size"] = packed["digest"], packed["size"]
+        job["state"] = "packed"
+        job["ready"].wait()
+        target = job["target"]
+        if target is None:
+            raise RuntimeError("the pack was abandoned")
+        job["state"] = "uploading"
+        sent = _op_push_blob(
+            {
+                "dir": packed["dir"],
+                "digest": packed["digest"],
+                "url": target["url"],
+                "headers": target.get("headers") or {},
+                "piece": target.get("piece"),
+                "cancel": job["cancel"],
+            }
+        )["value"]
+        result = {"state": "done", "id": sent["id"], "size": sent["size"]}
+    except BaseException as exc:
+        result = {"state": "failed", "reason": str(exc) or type(exc).__name__}
+    finally:
+        shutil.rmtree(directory, ignore_errors=True)
+    result["seconds"] = time.monotonic() - began
+    job.update(result)
+
+
+def _op_pack_push_start(request):
+    """Start packing a directory on a thread of its own and return at once."""
+    import threading
+
+    current = _PACK_JOB.get("job")
+    if current is not None and current["state"] in ("packing", "packed", "uploading"):
+        return {"ok": False, "error": "a pack is already running", "traceback": ""}
+    job = {
+        "state": "packing",
+        "target": None,
+        "ready": threading.Event(),
+        "cancel": threading.Event(),
+    }
+    _PACK_JOB["job"] = job
+    threading.Thread(target=_pack_push_run, args=(job, request), daemon=True).start()
+    return {"ok": True, "value": None}
+
+
+def _op_pack_push_target(request):
+    """Hand the running pack its upload session, or abandon it when ``url`` is None."""
+    job = _PACK_JOB.get("job")
+    if job is None:
+        return {"ok": False, "error": "no pack is running", "traceback": ""}
+    if request.get("url") is None:
+        job["cancel"].set()
+        job["target"] = None
+    else:
+        job["target"] = {
+            "url": request["url"],
+            "headers": request.get("headers") or {},
+            "piece": request.get("piece"),
+        }
+    job["ready"].set()
+    return {"ok": True, "value": None}
+
+
+def _op_pack_push_state(request):
+    """Report the pack: its state, and the digest, size, file id, seconds or reason it has."""
+    job = _PACK_JOB.get("job")
+    if job is None:
+        return {"ok": True, "value": {"state": "none"}}
+    keys = ("state", "digest", "size", "id", "seconds", "reason")
+    return {"ok": True, "value": {key: job[key] for key in keys if key in job}}
+
+
 def _op_place_blob(request):
     """Move a received file blob to its destination path, and unpack it when asked.
 
@@ -2539,6 +2634,9 @@ _OPS = {
     "pack_dir": _op_pack_dir,
     "pack_blob": _op_pack_blob,
     "push_blob": _op_push_blob,
+    "pack_push_start": _op_pack_push_start,
+    "pack_push_target": _op_pack_push_target,
+    "pack_push_state": _op_pack_push_state,
     "place_blob": _op_place_blob,
     "exec": _op_exec,
     "eval": _op_eval,
@@ -2553,7 +2651,10 @@ _OPS = {
 #: Answered by the frame reader at once, so they work while a call runs.
 # ``data_failed`` is light because it must not queue behind the call it is about: the call
 # is blocked on a blob the sender will never send, so a queued answer would deadlock.
-_LIGHT = ("stat", "lease", "reduce_peers", "drive_grant", "data_failed")
+_LIGHT = (
+    "stat", "lease", "reduce_peers", "drive_grant", "data_failed",
+    "pack_push_state", "pack_push_target",
+)
 
 #: Answered by several replies, the last one marked. These send their own replies.
 _STREAMING = ("data_stream",)

{letify-2.8.1 → letify-2.8.2}/letify/runtime/session.py

@@ -22,6 +22,7 @@ import datetime
 import json
 import posixpath
 import sys
+import threading
 import time
 from collections.abc import Callable, Iterator
 from dataclasses import dataclass, field
@@ -62,6 +63,16 @@ DRIVE_DOWNLOAD_BPS = 176 * 1024**2
 #: 170 MB/s on a sandbox disk. Spec "Materializing into a runtime".
 EXTRACT_BPS = 170_000_000
 
+#: How many times the estimated restore a build must have taken before the environment is
+#: packed to Drive. Spec "Materializing into a runtime": packing has a cost of its own.
+DRIVE_PACK_FACTOR = 2
+
+#: Seconds a closing session waits for a pack still running before it abandons it.
+PACK_CLOSE_WAIT_S = 60.0
+
+#: Seconds between the checks that hand a finished pack its upload session.
+PACK_POLL_S = 2.0
+
 
 def _against_the_build(size: int, rate: float, build_s: float) -> str:
     """The three numbers both cache decisions weigh, as they are reported.
@@ -87,6 +98,15 @@ def _sync_rather_than_restore(archive_bytes: int, rate: float, build_s: float) -
     )
 
 
+@dataclass
+class _DrivePack:
+    """A pack the worker is running for a Drive store, as this process tracks it."""
+
+    volume: Any
+    began: float = field(default_factory=time.monotonic)
+    target_sent: bool = False
+
+
 @dataclass
 class Runtime:
     """One live session on one provider."""
@@ -155,6 +175,11 @@ class Runtime:
     #: Set once ``shutdown`` has run, so a second discard does nothing.
     _shut_down: bool = field(default=False, repr=False)
 
+    #: The environment pack running on the worker, or None. Spec "Materializing into a
+    #: runtime": the pack to Drive runs beside the session's calls.
+    _pack: _DrivePack | None = field(default=None, repr=False)
+    _pack_lock: Any = field(default_factory=threading.RLock, repr=False)
+
     @property
     def key(self) -> str:
         """Pool key. Runtimes with equal keys are interchangeable."""
@@ -215,14 +240,12 @@ class Runtime:
                     + (f", {size / 1_000_000_000:.1f} GB" if size else "")
                 )
                 volume = self._environment_volumes()[0]
-                try:
+                if volume.store.backend.moves_on_runtime:
+                    self._start_drive_pack(volume, root)
+                else:
                     volume.cache_env_from(
                         self, self.env, root, platform=self.platform, build_s=self.env_build_s
                     )
-                except Exception as exc:
-                    if not volume.store.backend.moves_on_runtime:
-                        raise
-                    self._not_using_drive(exc, after="not caching the environment")
         self.keep_blobs_on_disk()
         for volume in self.volumes:
             self.attach(volume)
@@ -263,6 +286,7 @@ class Runtime:
         if self._shut_down:
             return
         self._shut_down = True
+        self._close_pack()
         timestamp = datetime.datetime.now(datetime.UTC).isoformat()
         print(
             f"{timestamp} letify: discarding {self.name} provider={self.provider.alias} "
@@ -371,9 +395,13 @@ class Runtime:
                 fn, args, kwargs, timeout=timeout, modules=self.env.ship_modules
             )
         args, kwargs = self._externalize(args, kwargs)
-        return self._call_with_data(
-            fn, args, kwargs, timeout, data_order, data_first_wave, replica
-        )
+        self._poll_pack()
+        try:
+            return self._call_with_data(
+                fn, args, kwargs, timeout, data_order, data_first_wave, replica
+            )
+        finally:
+            self._poll_pack()
 
     def _call_with_data(
         self,
@@ -695,46 +723,6 @@ class Runtime:
             )
         return digest, size
 
-    def pack_dir_to_store(self, path: str, backend: Any) -> tuple[str, int]:
-        """Pack a directory inside the runtime and have the runtime upload the archive.
-
-        Spec "Materializing into a runtime": the archive goes from the runtime's blob cache
-        to the backend without crossing this process. This process creates the upload
-        session, because it holds the credential, and the runtime is sent that session's
-        URL alone. Returns ``(digest, size)``. Needs a persistent channel, and raises
-        rather than routing the archive through this process.
-        """
-        if not self.persistent_channel:
-            raise RuntimeFailure("this channel keeps no worker that could upload the archive")
-        staging = f"{posixpath.dirname(path.rstrip('/'))}/.letify-pack"
-        value = self.request(
-            {"op": "pack_blob", "path": path, "dir": staging, "gzip": False}, timeout=3600
-        )
-        digest, size, directory = value["digest"], int(value["size"]), value["dir"]
-        try:
-            if not backend.has(digest):
-                target = backend.begin_upload(digest, size)
-                if target is None:
-                    raise RuntimeFailure(f"the {backend.name} backend offered no upload")
-                sent = self.request(
-                    {
-                        "op": "push_blob",
-                        "dir": directory,
-                        "digest": digest,
-                        "url": target["url"],
-                        "headers": dict(target.get("headers") or {}),  # type: ignore[call-overload]
-                        "piece": target.get("piece"),
-                    },
-                    timeout=3600,
-                )
-                backend.record_upload(digest, str(sent["id"]))
-        finally:
-            self.exec(
-                f"import shutil\nshutil.rmtree({directory!r}, ignore_errors=True)\n",
-                timeout=120,
-            )
-        return digest, size
-
     def put_path(
         self,
         local: Path,
@@ -937,24 +925,167 @@ class Runtime:
         return False
 
     def _worth_packing_to_drive(self, root: str) -> bool:
-        """Whether the build took longer than restoring the archive from Drive would.
+        """Whether the build took at least twice as long as restoring from Drive would.
 
         Spec "Materializing into a runtime": the archive goes from the runtime to Drive and
         never over this link, so the gate is the estimated restore, the project directory's
-        size over the Drive download rate plus the same size over the extraction rate.
+        size over the Drive download rate plus the same size over the extraction rate. The
+        build must take ``DRIVE_PACK_FACTOR`` times that, because the pack has a cost of its
+        own that the comparison leaves out.
         """
         size = self._tree_size(root)
         if size is None or self.env_build_s is None:
             return True
         restore_s = size / DRIVE_DOWNLOAD_BPS + size / EXTRACT_BPS
-        if self.env_build_s > restore_s:
+        if self.env_build_s >= DRIVE_PACK_FACTOR * restore_s:
             return True
         self._say(
             f"not caching the environment, because it built in {self.env_build_s:.0f} s "
-            f"and would take an estimated {restore_s:.0f} s to restore from Drive"
+            f"and would take an estimated {restore_s:.0f} s to restore from Drive, and a "
+            f"cache pays only when the build takes at least {DRIVE_PACK_FACTOR} times as long"
         )
         return False
 
+    # -- the pack that runs beside the calls ---------------------------------
+
+    def _start_drive_pack(self, volume: Volume, root: str) -> None:
+        """Have the worker pack the environment and upload it on a thread of its own.
+
+        Spec "Materializing into a runtime": this returns at once, so the first call does
+        not wait for the pack. The ref and the build record are written when a poll sees
+        the upload finish. A failure to start leaves nothing cached and says so.
+        """
+        staging = f"{posixpath.dirname(root.rstrip('/'))}/.letify-pack"
+        try:
+            if not self.persistent_channel:
+                raise RuntimeFailure("this channel keeps no worker that could upload the archive")
+            self.request(
+                {"op": "pack_push_start", "path": root, "dir": staging}, timeout=120
+            )
+        except Exception as exc:
+            self._not_using_drive(exc, after="not caching the environment")
+            return
+        self._pack = _DrivePack(volume)
+        threading.Thread(target=self._feed_pack, name="letify-pack", daemon=True).start()
+
+    def _feed_pack(self) -> None:
+        """Hand the pack its upload session soon after the archive is written.
+
+        A call can run for hours, and the upload cannot start until this process has made
+        the session, so this does only that. Reporting stays with the calls and the close.
+        """
+        while True:
+            time.sleep(PACK_POLL_S)
+            job = self._pack
+            if job is None or job.target_sent or self._shut_down:
+                return
+            self._poll_pack(report=False)
+
+    def _poll_pack(self, *, report: bool = True) -> bool:
+        """Ask the worker how the pack is doing and act on the answer.
+
+        A packed archive is given its upload session. A finished upload is recorded, and
+        reported when ``report`` is set. False when the worker could not be asked.
+        """
+        with self._pack_lock:
+            job = self._pack
+            if job is None or self.channel is None:
+                return True
+            try:
+                state = self.request({"op": "pack_push_state"}, timeout=30)
+            except Exception:
+                return False
+            kind = state.get("state")
+            backend = job.volume.store.backend
+            if kind == "packed" and not job.target_sent:
+                job.target_sent = True
+                try:
+                    if backend.has(state["digest"]):
+                        # Another session stored this archive first, so there is nothing to send.
+                        self._send_target(None)
+                        self._pack = None
+                        self._record_pack(job, state["digest"], int(state["size"]))
+                        return True
+                    target = backend.begin_upload(state["digest"], int(state["size"]))
+                    if target is None:
+                        raise RuntimeFailure(f"the {backend.name} backend offered no upload")
+                    self._send_target(target)
+                except Exception as exc:
+                    self._pack = None
+                    try:
+                        self._send_target(None)
+                    except Exception:
+                        pass
+                    self._not_using_drive(exc, after="not caching the environment")
+            elif kind == "done" and report:
+                self._pack = None
+                backend.record_upload(state["digest"], str(state["id"]))
+                self._record_pack(job, state["digest"], int(state["size"]))
+                seconds = float(state["seconds"])
+                size = int(state["size"])
+                self._say(
+                    f"cached the environment, {size / 1_000_000_000:.1f} GB in {seconds:.0f} s "
+                    f"({size / 1024**2 / max(seconds, 0.001):.1f} MiB/s)"
+                )
+            elif kind == "failed" and report:
+                self._pack = None
+                self._not_using_drive(
+                    RuntimeFailure(str(state.get("reason"))), after="not caching the environment"
+                )
+            elif kind == "none":
+                self._pack = None
+            return True
+
+    def _send_target(self, target: dict[str, Any] | None) -> None:
+        """Give the worker the upload session, or None to abandon the pack."""
+        payload: dict[str, Any] = {"op": "pack_push_target", "url": None}
+        if target is not None:
+            payload.update(
+                url=target["url"],
+                headers=dict(target.get("headers") or {}),  # type: ignore[call-overload]
+                piece=target.get("piece"),
+            )
+        self.request(payload, timeout=60)
+
+    def _record_pack(self, job: _DrivePack, digest: str, size: int) -> None:
+        """Write the ref and the build record, which only a finished upload earns."""
+        job.volume.record_env(
+            self.env, self.platform, digest, size, build_s=self.env_build_s
+        )
+
+    def _close_pack(self) -> None:
+        """Wait for a running pack, up to ``PACK_CLOSE_WAIT_S``, then abandon it.
+
+        Spec "Materializing into a runtime": an abandoned pack records nothing, so the
+        next session syncs and may pack again.
+        """
+        job = self._pack
+        if job is None:
+            return
+        deadline = time.monotonic() + PACK_CLOSE_WAIT_S
+        try:
+            while self._pack is not None:
+                if not self._poll_pack():
+                    self._pack = None
+                    return
+                if self._pack is None:
+                    return
+                if time.monotonic() >= deadline:
+                    break
+                time.sleep(0.2)
+            with self._pack_lock:
+                self._pack = None
+                try:
+                    self._send_target(None)
+                except Exception:
+                    pass
+            self._say(
+                "not caching the environment, because the session closed "
+                f"{time.monotonic() - job.began:.0f} s into the pack"
+            )
+        except Exception:
+            self._pack = None
+
     def _restore_from(self, volume: Volume, where: dict[str, Any]) -> bool:
         """Restore the environment from one volume's archive. True when the check passed.
 

{letify-2.8.1 → letify-2.8.2}/letify/store/volume.py

@@ -24,7 +24,7 @@ from pathlib import Path
 from typing import TYPE_CHECKING, Any
 
 from .. import protocol
-from .cas import BlobInfo, Store
+from .cas import Store
 
 if TYPE_CHECKING:
     from ..declare.env import Env
@@ -236,22 +236,29 @@ class Volume:
         """
         runtime = _session(target)
         scratch = self.store.scratch_path() if self.store.streams else None
-        if self.store.backend.moves_on_runtime:
-            # The runtime uploads the archive itself, so it never crosses this process.
-            digest, size = runtime.pack_dir_to_store(path, self.store.backend)
-            info = BlobInfo(digest, size)
-        elif scratch is not None and runtime.persistent_channel:
+        if scratch is not None and runtime.persistent_channel:
             # Streamed: the archive lands in a file beside the store and is renamed in.
             digest, _size = runtime.pack_dir_to_file(path, scratch)
             info = self.store.adopt_file(scratch, digest)
         else:
             payload, _digest = runtime.pack_dir(path)
             info = self.store.put_bytes(payload)
-        self.store.point(self.env_ref(env, platform), info.digest)
-        if build_s is not None:
-            self.store.point(self.build_ref(env, platform), f"{build_s:.1f} {info.size}")
+        self.record_env(env, platform, info.digest, info.size, build_s=build_s)
         return info.digest
 
+    def record_env(
+        self, env: Env, platform: str | None, digest: str, size: int, *,
+        build_s: float | None = None,
+    ) -> None:
+        """Point the env ref at a stored archive and write the build record beside it.
+
+        A Drive archive is uploaded by the runtime in the background, so its ref is written
+        by the session that saw the upload finish, not by the pack that started it.
+        """
+        self.store.point(self.env_ref(env, platform), digest)
+        if build_s is not None:
+            self.store.point(self.build_ref(env, platform), f"{build_s:.1f} {size}")
+
     def build_ref(self, env: Env, platform: str | None = None) -> str:
         """The ref recording the build seconds and archive size beside the env ref."""
         return f"{self.env_ref(env, platform)}.build"

{letify-2.8.1 → letify-2.8.2}/letify/transport/nat.py

@@ -15,6 +15,7 @@ import os
 import re
 import secrets
 import select
+import shlex
 import socket
 import struct
 import subprocess
@@ -560,10 +561,23 @@ def begin(request: dict):
         descriptor = os.open(key, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
         with os.fdopen(descriptor, "w") as handle:
             handle.write(request["private_key"])
+        # With a jump host the same identity and host-key policy must reach both hops, and
+        # -J does not carry command-line options to the jump connection, so the hop is an
+        # explicit ProxyCommand that repeats them.
+        jump = request.get("jump")
+        hop: list[str] = []
+        if jump:
+            proxy = (
+                "ssh -o BatchMode=yes -o StrictHostKeyChecking=accept-new"
+                f" -i {shlex.quote(str(key))}"
+                f" -W %h:%p -p {int(jump.get('port', 22))}"
+                f" {shlex.quote(str(jump['user']) + '@' + str(jump['address']))}"
+            )
+            hop = ["-o", f"ProxyCommand={proxy}"]
         process = subprocess.Popen(
             [
                 "ssh", "-N", "-o", "BatchMode=yes", "-o", "ExitOnForwardFailure=yes",
-                "-o", "StrictHostKeyChecking=accept-new",
+                "-o", "StrictHostKeyChecking=accept-new", *hop,
                 "-R", f"0:127.0.0.1:{int(request.get('ssh_port', 22))}",
                 "-p", str(request["port"]), "-i", str(key),
                 f"{request['user']}@{request['address']}",