letify 2.8.4__tar.gz → 2.8.6__tar.gz

{letify-2.8.4 → letify-2.8.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: letify
-Version: 2.8.4
+Version: 2.8.6
 Summary: Declarations that become infrastructure.
 Project-URL: Homepage, https://github.com/DarkPyonix/letify
 Project-URL: Documentation, https://github.com/DarkPyonix/letify/tree/main/docs

{letify-2.8.4 → letify-2.8.6}/PROJECT.md

@@ -195,6 +195,7 @@ cache.resume(runtime, "run-1", path)   # put the newest one inside a session
 ```python
 letify.stash("ckpt", ckpt_dir)         # put a file or directory on the machine's Drive store
 letify.unstash("ckpt", ckpt_dir)       # put it back, or None when no handoff has that name
+letify.unstash("ckpt")                # no path: a dict of file path to bytes, nothing written to disk
 ```
 
 Both work in the local process and inside a shipped function, on any provider. They need `drive_login` and `drive_project` in the machine configuration and raise `ConfigError` without them. On a one-shot channel, which has no stream back to the local process, they raise `UnsupportedMode` naming the provider. `letify.fetch(tensor)` is a different call and still reads a tensor back.

{letify-2.8.4 → letify-2.8.6}/docs/SPEC.md

@@ -777,7 +777,7 @@ The protocol is one JSON object per line. letify sends `{"id": <int>, "op": <nam
 | Op | Fields | Value |
 |---|---|---|
 | `hello` | none | `{"modal": <installed Modal version>}` |
-| `create` | `app`, `args`, `packages`, `gpu`, `timeout`, `idle_timeout`, `volumes`, `ports` | `{"sandbox": <id>}`. Runs `app` as an ephemeral app on first use, builds `debian_slim` with `packages` installed, and starts `args` in a sandbox. Each port in `ports` is exposed with Modal `encrypted_ports` |
+| `create` | `app`, `args`, `packages`, `gpu`, `timeout`, `idle_timeout`, `memory`, `volumes`, `ports` | `{"sandbox": <id>}`. Runs `app` as an ephemeral app on first use, builds `debian_slim` with `packages` installed, and starts `args` in a sandbox. Each port in `ports` is exposed with Modal `encrypted_ports` |
 | `tunnel` | `sandbox`, `port` | `{"host": <text>, "port": <int>, "tls": <bool>}`. The address that reaches `port` inside the sandbox, from `Sandbox.tunnels()`. `tls` is true when the connection has to be made with TLS, which an encrypted port needs |
 | `write` | `sandbox`, `data` | `null`. `data` is base64 of at most 1 MiB. Writes the decoded bytes to the sandbox's standard input and drains it. Modal refuses a write that would buffer more than 2 MiB, so the channel splits a larger frame into `write` requests of 1 MiB, in order |
 | `read_until` | `sandbox`, `prefixes` | `{"lines": [...], "eof": <bool>}`. The sandbox's stdout lines up to and including the first that starts with one of `prefixes`, or every line left when the stream ends |
@@ -812,8 +812,14 @@ Modal also bounds a sandbox that nothing terminates. `create` passes `idle_timeo
 
 It passes `timeout`, the entry option `timeout`, as the sandbox's maximum lifetime, and the default is the longest Modal accepts, 86400 s, measured as accepted. It is not a shorter number letify picked, for the reason The declaration gives for `timeout` having no default: a deadline letify invented would end a two hour training run at whatever hour it guessed. The default was 3600 s, exactly one hour, and it ended two Modal sessions of a real training run at one hour each with nothing saying why, which is that sentence happening. An account that wants a hard cap sets `timeout` itself.
 
+It passes `memory`, the entry option `memory_gib` converted to MiB, as the sandbox's memory request, and passes nothing when the account does not set it, so Modal's default applies. A training process whose RAM grows past what the sandbox gets is killed with SIGKILL, which the channel sees as a worker that stopped without replying; an account whose runs need more sets `memory_gib`, for example `memory_gib = 64`. A value that is not a positive number raises `ConfigError` naming `memory_gib`.
+
 A sandbox that reaches that lifetime takes its worker with it, so the channel sees a worker that stopped without replying. Where the session's age is within a minute of the lifetime letify asked for, the failure says so instead of listing causes: `the Modal sandbox reached the <seconds> s lifetime letify asked for; raise timeout on the account to run longer`.
 
+Otherwise a Modal worker that stopped without replying is followed by one `poll` of its sandbox from a second adapter process, waiting at most 30 s for the exit code, and the failure names what the poll found instead of listing causes: `the Modal sandbox exited with code <n>` for a sandbox that ended, with `, which is SIGKILL, most often the container running out of memory` added for 137, and `the Modal sandbox is still running, so the worker process died inside it` for one still running. A poll that fails or times out leaves the message as it was. The message is what tells an out-of-memory kill from a preemption, which a training run that died twice in one day could not.
+
+A worker reached through a subprocess (`PersistentChannel`, for example `ssh` to a Colab or Kaggle machine) that exits before it says hello, or while a call is running, has its exit code named in the message text, so a user can tell a lost connection from a kill from a crash. A signal death is reported as the shell does, 128 plus the signal number. The message reads `the worker exited with code <n>` followed by a short meaning: `255` adds `: ssh could not reach or lost the machine`, `128+N` adds `: killed by signal N`, with `137` adding `, which is SIGKILL, most often out of memory`, and any other nonzero code adds nothing. A worker that never said hello reads `the worker exited before it was ready, with code <n>` followed by the same meaning. The last 2000 characters of the process's standard error follow under `--- last remote output ---`, and the startup failure also keeps the whole text as its `stderr` attribute. A channel whose provider can name a cause itself, such as Modal, keeps that cause. When the command's program is `ssh` or `ssh.exe`, `PersistentChannel` starts it as `ssh -E <file> ...`, with a fresh temporary file per channel placed right after the program. `ControlPersist` forks a master whose standard error is /dev/null, so its errors (`Timeout, server not responding`, `Connection reset`, `closed by remote host`, `Broken pipe`) reach only that file. A death message, both before hello and during a call, appends the last 2000 characters of the file under `--- ssh log ---` when it is not empty, after the standard error of the ssh client. Other commands are not changed. The file is deleted when the channel closes.
+
 ### Modal data channel <!-- id: modal-data-channel -->
 
 > Once the worker has said hello over the sandbox's standard input and output, the channel moves its frames to a TCP connection through a Modal encrypted port. Standard input and output stay the control path and the fallback.
@@ -1556,10 +1562,12 @@ A token is reused until 60 s before it expires. A service account key file is re
 
 A handoff is what a volume cannot be: a value written on one provider and read on another, such as a checkpoint written on Kaggle and resumed on `dept_gpu`. A volume belongs to one provider and stays attached to it, so letify keeps the two apart. `stash` and `unstash` take no provider and no volume.
 
-`stash(name, path)` packs `path`, a file or a directory, into an uncompressed tar whose members are relative to `path`'s parent, so the tar holds the one entry `path`'s base name. Its digest, letify's content hash as the blob store uses it, names the blob in the `letify` folder, and the ref `ref stash/<name>` holds that digest. It returns the digest once the upload has finished and the ref is written. Stashing a name again moves the ref, and the blob the ref named before is deleted unless another `stash/` ref still names it. An upload of the same digest that Drive already holds sends nothing. The tar keeps each member's modification time, so a file rewritten with the same bytes gives a new digest.
+`stash(name, path)` packs `path`, a file or a directory, into an uncompressed tar whose members are relative to `path`'s parent, so the tar holds the one entry `path`'s base name. Its digest, letify's content hash as the blob store uses it, names the blob in the `letify` folder, and the ref `ref stash/<name>` holds that digest. It returns the digest once the upload has finished and the ref is written. Stashing a name again moves the ref, and the blob the ref named before is deleted unless another `stash/` ref still names it. Whether another ref names it is decided from one listing of the `letify` folder: a ref holds the digest text and nothing else, so its Drive `md5Checksum` names its content, and no ref is downloaded. An upload of the same digest that Drive already holds sends nothing. The tar keeps each member's modification time, so a file rewritten with the same bytes gives a new digest.
 
 `unstash(name, path)` reads the ref, downloads the blob, checks its digest, and unpacks it so that the tar's one entry lands at `path`, replacing what `path` held. It returns `path` as a `pathlib.Path`. When no ref `stash/<name>` exists it returns `None` and changes nothing, so `if letify.unstash("ckpt", ckpt_dir) is None:` starts from scratch. A digest that does not match raises `RuntimeFailure` and leaves `path` untouched.
 
+`unstash(name)` with no path returns the handoff in memory and writes nothing to disk: a `dict[str, bytes]` that maps the path of each regular file in the tar, as stored (`ckpt/model.safetensors` for a directory handoff, `checkpoint.safetensors` for a one-file handoff), to that file's bytes. Directories are not entries. It returns `None` when no ref `stash/<name>` exists. Every byte of the download is hashed as it is read, and the dict is returned only after the digest matches; a digest that does not match raises `RuntimeFailure` and returns no dict. Peak memory is about the handoff's size, not twice it, because the download is read sequentially and each member's bytes are read from the response with one `read(size)` call into the bytes object the caller receives: no chunk list is joined, no archive is held whole, and tar headers (GNU long name records and pax records) are parsed as they arrive. It works in both places below with the same result.
+
 A name is 1 to 128 characters of letters, digits, `.`, `_` and `-`, and does not start with `.`; anything else raises `ValueError` naming the rule. Names are shared by every project on the machine's Drive store, so a project that wants its own space puts its name in the handoff name.
 
 The name `unstash` is not `fetch`, because `letify.fetch(tensor)` already reads a tensor back in `host=local`.
@@ -1567,7 +1575,7 @@ The name `unstash` is not `fetch`, because `letify.fetch(tensor)` already reads
 Both run in two places with the same result:
 
 - **In the local process**, the client moves the bytes itself with the `DriveBackend` of Google Drive is a store, not a volume.
-- **Inside a shipped function**, the worker moves the bytes between the runtime and Drive and they never cross the client's link. The worker asks the local process for a grant with a one-way request on a reserved stream, as `all_reduce` asks for a peer table, and the local process answers with the light op `drive_grant`: for `stash`, the existing digest or a resumable upload session URL for the archive's size, which the local process creates as Google Drive is a store, not a volume describes; for `unstash`, the blob's media URL with the short-lived request headers, or a reply that the name has no ref. After a `stash` upload the worker reports the Drive file id and the local process writes the ref and deletes the replaced blob. The worker holds no refresh token and writes no header to disk.
+- **Inside a shipped function**, the worker moves the bytes between the runtime and Drive and they never cross the client's link. The worker asks the local process for a grant with a one-way request on a reserved stream, as `all_reduce` asks for a peer table, and the local process answers with the light op `drive_grant`: for `stash`, the existing digest or a resumable upload session URL for the archive's size, which the local process creates as Google Drive is a store, not a volume describes; for `unstash`, the blob's media URL with the short-lived request headers, or a reply that the name has no ref. After a `stash` upload the worker reports the Drive file id and the local process writes the ref and answers. The replaced blob is deleted after that, on a thread of the local process, so the answer never waits for the deletion and the worker's grant timeout counts only the ref write. A deletion that fails prints `letify: drive handoff <name>: the replaced blob <digest> was not deleted: <error>` on stderr and leaves the blob; the worker's `stash` has already returned. A request the worker cannot send prints `letify: drive request <kind> <name> was not sent: <error>` on the worker's stderr, so a grant timeout that follows names its cause. The worker holds no refresh token and writes no header to disk. The local process prints one line on stderr for every grant it answers, `letify: drive grant <kind> <name>: answered in <s> s`, and when sending the answer fails, `letify: drive grant <kind> <name>: the answer was not delivered after <s> s: <error>`. A worker that waits out its grant timeout can then be matched to the local side: either no request arrived, or the answer did not reach it. When the call ends while a grant is still being made, for example a slow Drive call, the watcher waits 30 s for it and then prints `letify: drive grant <kind> <name>: still running after <s> s when the call ended; its result will not be reported`, so the missing `answered in` line has a stated cause.
 
 A body run on a one-shot channel, which has no stream back to the local process, cannot ask for a grant, so `stash` and `unstash` there raise `UnsupportedMode` naming the provider rather than reading the runtime's own configuration.
 
@@ -1723,7 +1731,7 @@ A `Shell` subclass differs from its parent only in its `Rendezvous` and its stra
 
 Strategies are raced rather than tried in turn, so a strategy that times out does not delay the others. Once the first strategy that can carry the probe connects, the pipeline waits a grace period of 2 s for lower ranked strategies before choosing.
 
-A strategy that cannot carry the probe, such as the provider fallback, does not start the grace period when it connects. It is held back until every strategy that can carry the probe has failed, or until the race timeout of 60 s passes with none of them connected. A rendezvous that takes seconds, such as `colab exec` before a TCP punch, therefore runs inside its strategy's own attempt time, not inside the grace period.
+A strategy that cannot carry the probe, such as the provider fallback, does not start the grace period when it connects. It is held back until every strategy that can carry the probe has failed, or until the race timeout of 180 s passes with none of them connected. A rendezvous that takes seconds, such as `colab exec` before a TCP punch, therefore runs inside its strategy's own attempt time, not inside the grace period.
 
 Every connected strategy is probed: 30 round trips, then a warm-up transfer that is discarded, then 2 s of transfer in each direction. The warm-up is `min(2.0, max(0.5, 8 x round trip))` seconds, long enough for TCP to leave slow start, and it runs as a transfer of its own on the same connection so the congestion window it opens carries into the measured one. Without it the probe charges a far link for its own ramp: the window is fixed in seconds while the ramp is fixed in round trips, so at a 190 ms round trip more than half the 2 s is spent ramping and the reported throughput is the ramp average, while at 47 ms it is a seventh. A live Kaggle runtime measured up 5.0 MiB/s at 190 ms and up 46.1 MiB/s at 47 ms on the same account, and the throughput times the round trip is 0.95 MiB against 2.18 MiB, which is a window that never finished opening rather than a path that cannot carry more. A strategy whose probe is below the account's floor, [Link floor](#link-floor), is rejected outright, before any comparison between strategies. Of what remains, a strategy whose throughput in either direction is below 25% of the fastest connected strategy in that direction is rejected. The lowest ranked strategy that remains is chosen. When only one strategy connects, it is chosen without comparison, but it is still held to the floor: a single connected strategy below the floor fails the connection rather than being used because nothing else was available.
 
@@ -1803,7 +1811,7 @@ A connection to the agent is told apart by its first bytes: `SSH-` is spliced to
 
 Both sides learn their public mapping from STUN servers reached over TCP on port 443, because networks that restrict outbound ports usually still allow 443. A punch starts at a time both sides agree on through the rendezvous. Each side connects from its bound port to the other's mapping and listens on the same port, so whichever direction's SYN arrives first completes the connection.
 
-The window has to cover the rendezvous, not just the agreed start. The start time is fixed before the rendezvous is sent, and the rendezvous takes as long as it takes: for a Colab runtime it is a `colab exec` that also starts an SSH server, bounded only by the 30 s connect timeout. The remote side reaches the agreed time while the answer is still travelling back, so the user's side begins later by the return trip. A window of 15 s therefore closed on the remote side before the user's side had begun whenever the rendezvous was slow, which is why the same account punched in 10.7 s one day and reported `did not connect in time` the next. The default window is 45 s, longer than the connect timeout, so the two sides overlap however long the rendezvous takes.
+The window has to cover the rendezvous, not just the agreed start. The start time is fixed before the rendezvous is sent, and the rendezvous takes as long as it takes: for a Colab runtime it is a `colab exec` that also starts an SSH server, bounded only by the 120 s connect timeout. The remote side reaches the agreed time while the answer is still travelling back, so the user's side begins later by the return trip. A window of 15 s therefore closed on the remote side before the user's side had begun whenever the rendezvous was slow, which is why the same account punched in 10.7 s one day and reported `did not connect in time` the next. The default window is 150 s, longer than the 120 s connect timeout, so the two sides overlap however long the rendezvous takes.
 
 The remote side also reports `punch_until`, the time on its own clock at which it stops, and the user's side stops then too rather than dialing alone for a window of its own. A punch that is already over fails at once and names that, instead of spending the window finding out.
 
@@ -1817,7 +1825,7 @@ A punch that ends with nothing connected names what happened to its dials, becau
 
 The dial is still repeated until the window closes even when it is refused, because the listener may yet receive the connection the peer is completing from its own side.
 
-The remote side's punch window is 45 s, or the request's `window` in seconds when it sets one. A punch that does not connect within it, or fails with a refused or unreachable connection, is expected, because the Tailcat link is already carrying the session. The agent then writes one line to stderr, `letify agent: TCP punch with <host>:<port> did not connect within <window> s; the Tailcat link is used instead`, or for a refused or unreachable peer `letify agent: TCP punch with <host>:<port> failed (<reason>); the Tailcat link is used instead`, and no traceback. Any other exception in the punch or the splice keeps its traceback.
+The remote side's punch window is 150 s, or the request's `window` in seconds when it sets one. A punch that does not connect within it, or fails with a refused or unreachable connection, is expected, because the Tailcat link is already carrying the session. The agent then writes one line to stderr, `letify agent: TCP punch with <host>:<port> did not connect within <window> s; the Tailcat link is used instead`, or for a refused or unreachable peer `letify agent: TCP punch with <host>:<port> failed (<reason>); the Tailcat link is used instead`, and no traceback. Any other exception in the punch or the splice keeps its traceback.
 
 Every SSH command letify builds sets `HostKeyAlias=letify-<alias>`, `StrictHostKeyChecking=accept-new` and `UserKnownHostsFile=~/.letify/accounts/<alias>/known_hosts`. Building the command creates that account directory with mode 0700 when it is missing, because `ssh` creates only `~/.ssh` and otherwise cannot record the key: it prints `Failed to add the host to the list of known hosts` on every connection and never pins the key. A host key that differs from the recorded one still fails the connection.
 
@@ -1953,7 +1961,7 @@ Colab limits outbound UDP to roughly 200 packets per second, so rank 3 is expect
 
 Kaggle reports 2.0 MiB/s, measured on a live account sending 64 MiB over the kernel bridge, and an account overrides it with `fallback_mib_per_s`. Without this a live session refused a punched link at down 8.1 MiB/s for being under the 10 MiB/s floor and then carried 26 GB over the 2 MiB/s bridge instead, which is the opposite of what the floor is for. Taking the fallback is also never silent: the reason is printed, because a session that quietly changed channel is one nobody can account for.
 
-A `channel = "exec"` entry skips the pipeline and uses the fallback directly. A Colab VM has no SSH server on the port letify splices to, so the rendezvous request asks the remote half to start one first. The remote half connects to `127.0.0.1:<ssh_port>`, and when no line starting with `SSH-` arrives within 3 s it installs `openssh-server` if `/usr/sbin/sshd` is missing, creates `/run/sshd`, runs `ssh-keygen -A` so the server has host keys, and starts `/usr/sbin/sshd -f /etc/ssh/sshd_config.letify -p <ssh_port> -o ListenAddress=127.0.0.1`. That file is the image's `/etc/ssh/sshd_config` with every `MaxStartups` line removed and `MaxStartups 64:30:128` as its first line, or that line alone when the image has no file. A Colab image's file sets `MaxStartups 1`, so the server dropped every connection that arrived while another was still authenticating, and a command-line `-o MaxStartups` does not override the file. Over `reverse_ssh` the transfer streams of a session arrive at once, so every one after the first ended with `Connection closed by 127.0.0.1 port <port>` and the call failed with `the worker exited before it was ready`. With the file, one 190.7 MiB blob crossed a reverse SSH link from a Colab CPU runtime in 5.4 s, 35.5 MiB/s. `ssh-keygen -A` creates only the keys that are missing, and it is needed because a Kaggle image ships `sshd` with no host key at all: the install step above is skipped since the package is already there, and a server with no host key accepts the connection and closes it before the banner, which the client reports as `Connection closed by 127.0.0.1 port <port>` with no reason. The port is explicit because a Colab image ships `sshd` configured for `127.0.0.1:2222`, so starting it with its own configuration leaves port 22 closed. An SSH server that already answers on the port is used as it is. The request carries the account's public key, `key` with `.pub` appended, which the remote side adds to `authorized_keys`. An account with no `key`, or whose `.pub` file is missing, has nothing the VM could authorize, so its Colab rendezvous is unavailable with the reason `no key`: `tcp_punch` and `tailcat` are skipped with that reason and the fallback carries the session. SSH over a punched or Tailcat link logs in as `root` unless `user` says otherwise.
+A `channel = "exec"` entry skips the pipeline and uses the fallback directly. A Colab VM has no SSH server on the port letify splices to, so the rendezvous request asks the remote half to start one first. The remote half connects to `127.0.0.1:<ssh_port>`, and when no line starting with `SSH-` arrives within 3 s it installs `openssh-server` if `/usr/sbin/sshd` is missing, creates `/run/sshd`, runs `ssh-keygen -A` so the server has host keys, and starts `/usr/sbin/sshd -f /etc/ssh/sshd_config.letify -p <ssh_port> -o ListenAddress=127.0.0.1`. That file is the image's `/etc/ssh/sshd_config` with every `MaxStartups` line removed and `MaxStartups 64:30:128` as its first line, or that line alone when the image has no file. A Colab image's file sets `MaxStartups 1`, so the server dropped every connection that arrived while another was still authenticating, and a command-line `-o MaxStartups` does not override the file. Over `reverse_ssh` the transfer streams of a session arrive at once, so every one after the first ended with `Connection closed by 127.0.0.1 port <port>` and the call failed with `the worker exited before it was ready`. With the file, one 190.7 MiB blob crossed a reverse SSH link from a Colab CPU runtime in 5.4 s, 35.5 MiB/s. `ssh-keygen -A` creates only the keys that are missing, and it is needed because a Kaggle image ships `sshd` with no host key at all: the install step above is skipped since the package is already there, and a server with no host key accepts the connection and closes it before the banner, which the client reports as `Connection closed by 127.0.0.1 port <port>` with no reason. The port is explicit because a Colab image ships `sshd` configured for `127.0.0.1:2222`, so starting it with its own configuration leaves port 22 closed. An SSH server that already answers on the port is used as it is. Two strategies rendezvous at once on a Colab VM, so the check, the file write and the start run under an exclusive `flock` on `/run/letify-sshd.lock`, and the check is repeated once the lock is held, so a caller that waited finds the first caller's server and returns. The file is written to a temporary file in `/etc/ssh` and moved onto `sshd_config.letify` with `os.replace`, so a server reading it sees the old file or the whole new one. The request carries the account's public key, `key` with `.pub` appended, which the remote side adds to `authorized_keys`. An account with no `key`, or whose `.pub` file is missing, has nothing the VM could authorize, so its Colab rendezvous is unavailable with the reason `no key`: `tcp_punch` and `tailcat` are skipped with that reason and the fallback carries the session. SSH over a punched or Tailcat link logs in as `root` unless `user` says otherwise.
 
 A Colab VM keeps its GPU driver libraries in `/usr/lib64-nvidia`, and only the notebook kernel's own environment names that directory in `LD_LIBRARY_PATH`. A login shell does not, so a worker started over SSH finds no `libnvidia-ml.so` and no `libcuda.so`: `nvidia-smi` fails and `torch.cuda.is_available()` is false on a session that holds a GPU. Every command letify runs over a Colab link, the worker and the `check` command included, is therefore prefixed with `LD_LIBRARY_PATH=/usr/lib64-nvidia${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}`, which keeps a path the machine already set. The `colab exec` fallback needs no prefix, because it runs inside the kernel.
 

{letify-2.8.4 → letify-2.8.6}/letify/__init__.py

@@ -29,7 +29,7 @@ from __future__ import annotations
 
 import os
 from pathlib import Path
-from typing import Final
+from typing import Final, overload
 
 from .declare.cache import session_cache
 from .declare.env import Env
@@ -68,7 +68,7 @@ distributed: Final = Host.distributed
 # The enum class stays importable from letify.declare.instance for letify's own use.
 del Host
 
-__version__ = "2.8.4"
+__version__ = "2.8.6"
 
 
 def fetch(tensor):  # type: ignore[no-untyped-def]
@@ -93,13 +93,29 @@ def stash(name: str, path: str | os.PathLike[str]) -> str:
     return stash_handoff(name, path)
 
 
-def unstash(name: str, path: str | os.PathLike[str]) -> Path | None:
+@overload
+def unstash(name: str) -> dict[str, bytes] | None: ...
+@overload
+def unstash(name: str, path: str | os.PathLike[str]) -> Path | None: ...
+def unstash(
+    name: str, path: str | os.PathLike[str] | None = None
+) -> Path | dict[str, bytes] | None:
     """Put the handoff ``name`` at ``path``, replacing what was there.
 
     Returns ``path``, or None when no handoff has that name, in which case nothing changed.
+
+    With no ``path`` the handoff is returned in memory and nothing is written to disk: a
+    ``dict[str, bytes]`` from the path of each regular file in the handoff, as stored
+    (``ckpt/model.safetensors``, or ``checkpoint.safetensors`` for a one-file handoff), to
+    its bytes. Directories are not entries, and None means no handoff has that name. The
+    dict is returned only after the digest matches; a mismatch raises ``RuntimeFailure``.
+    Peak memory is about the handoff's size, because members are read straight from the
+    download.
     """
     from .store.handoff import unstash as unstash_handoff
 
+    if path is None:
+        return unstash_handoff(name)
     return unstash_handoff(name, path)
 
 

{letify-2.8.4 → letify-2.8.6}/letify/protocol/worker.py

@@ -1380,8 +1380,14 @@ def _drive_ask(message):
     """
     try:
         _SENDER.message(REQUEST, DRIVE_STREAM, message)
-    except OSError:
-        pass
+    except OSError as exc:
+        # The stash or unstash then waits out its grant timeout; this line names why.
+        print(
+            f"letify: drive request {message.get('kind')} {message.get('name')} "
+            f"was not sent: {exc}",
+            file=sys.stderr,
+            flush=True,
+        )
 
 
 def _op_drive_grant(request):

{letify-2.8.4 → letify-2.8.6}/letify/providers/modal.py

@@ -31,7 +31,7 @@ import tempfile
 import threading
 import time
 import weakref
-from collections.abc import Mapping
+from collections.abc import Callable, Mapping
 from typing import IO, TYPE_CHECKING, Any
 
 from ..config import ProviderConfig
@@ -133,6 +133,22 @@ def lifetime_detail(*, age_s: float, lifetime_s: int) -> str | None:
     )
 
 
+def exit_cause(found: tuple[bool, int | None] | None) -> str | None:
+    """What a poll of a sandbox whose worker stopped says, or None when it is unknown.
+
+    Spec "Modal": the exit code tells an out of memory kill from a preemption.
+    """
+    if found is None:
+        return None
+    ended, code = found
+    if not ended:
+        return "the Modal sandbox is still running, so the worker process died inside it"
+    text = f"the Modal sandbox exited with code {code}"
+    if code == 137:
+        text += ", which is SIGKILL, most often the container running out of memory"
+    return text
+
+
 class VolumePathMissing(RuntimeFailure):
     """The adapter answered that a volume path does not exist."""
 
@@ -419,6 +435,21 @@ class Modal(Provider):
         value = self.config.option("idle_timeout", IDLE_TIMEOUT)
         return max(1, int(value))
 
+    @property
+    def sandbox_memory_mib(self) -> int | None:
+        """The sandbox's memory request in MiB from ``memory_gib``, or None for Modal's own.
+
+        Spec "Modal": a training process past what the sandbox gets is killed with SIGKILL.
+        """
+        value = self.config.option("memory_gib")
+        if value is None:
+            return None
+        if isinstance(value, bool) or not isinstance(value, (int, float)) or value <= 0:
+            raise ConfigError(
+                f"{self.alias}: memory_gib has to be a positive number of GiB, not {value!r}"
+            )
+        return int(value * 1024)
+
     def adapter(self) -> Adapter:
         """The adapter acting as this account, started on its first request."""
         if self._adapter is None or self._adapter.closed:
@@ -484,6 +515,7 @@ class Modal(Provider):
             gpu=self.wire_name(runtime.instance) or None,
             timeout=self.sandbox_timeout_s,
             idle_timeout=self.sandbox_idle_timeout_s,
+            memory=self.sandbox_memory_mib,
             ports=[] if data_port is None else [data_port],
         )
         sandbox = str(created["sandbox"])
@@ -491,17 +523,14 @@ class Modal(Provider):
         return SandboxChannel(
             adapter, sandbox, name=runtime.name, data_port=data_port, streams=streams,
             lifetime_s=self.sandbox_timeout_s,
+            probe=lambda: self.poll_sandbox(sandbox),
         )
 
-    def runtime_state(self, runtime: Runtime) -> RuntimeState:
-        """Poll the sandbox from a second adapter process, never the one carrying the call.
+    def poll_sandbox(self, sandbox: str) -> tuple[bool, int | None] | None:
+        """Poll a sandbox once from a second adapter: ``(ended, exit code)``, or None if unknown.
 
-        Spec "Lifetime", the lease. The adapter carrying the call may be blocked in a
-        request, so a short-lived one asks. Any failure answers unknown.
+        A short-lived adapter asks, because the one carrying the call may be blocked.
         """
-        sandbox = self._sandboxes.get(runtime.name)
-        if sandbox is None:
-            return RuntimeState()
         other = Adapter.for_account(self.alias)
         # A poll that hangs ends the second adapter, which makes the request raise.
         limit = threading.Timer(POLL_LIMIT, other.abort)
@@ -510,11 +539,26 @@ class Modal(Provider):
         try:
             code = other.request("poll", sandbox=sandbox)
         except (LetifyError, ValueError, OSError):
-            return RuntimeState()
+            return None
         finally:
             limit.cancel()
             other.close()
-        if code is None:
+        return (code is not None, code)
+
+    def runtime_state(self, runtime: Runtime) -> RuntimeState:
+        """Poll the sandbox from a second adapter process, never the one carrying the call.
+
+        Spec "Lifetime", the lease. The adapter carrying the call may be blocked in a
+        request, so a short-lived one asks. Any failure answers unknown.
+        """
+        sandbox = self._sandboxes.get(runtime.name)
+        if sandbox is None:
+            return RuntimeState()
+        found = self.poll_sandbox(sandbox)
+        if found is None:
+            return RuntimeState()
+        ended, code = found
+        if not ended:
             return RuntimeState("running")
         return RuntimeState(
             "ended", f"the Modal sandbox ended (exit code {code}) while the call was running"
@@ -686,8 +730,11 @@ class SandboxChannel(FramedChannel):
         data_port: int | None = None,
         streams: int = 1,
         lifetime_s: int = 0,
+        probe: Callable[[], tuple[bool, int | None] | None] | None = None,
     ):
         self.adapter = adapter
+        #: Polls the sandbox from a second adapter, see ``Modal.poll_sandbox``.
+        self.probe = probe
         self.sandbox = sandbox
         self.name = name
         self.data_port = data_port
@@ -716,6 +763,7 @@ class SandboxChannel(FramedChannel):
             wire.chunks_readinto(self._read_chunks),
             self._emit,
             death_detail=self._raw_text,
+            death_cause=self._exit_cause,
         )
         self._connection = self._stdio
         self._send_worker()
@@ -730,6 +778,17 @@ class SandboxChannel(FramedChannel):
         )
         return f"{said}\n{text}" if said else text
 
+    def _exit_cause(self) -> str | None:
+        """What one poll of the sandbox found, or None to keep the usual causes.
+
+        Spec "Modal": a lifetime death is already named by ``_raw_text``, so it is not polled.
+        """
+        if self.probe is None or lifetime_detail(
+            age_s=time.monotonic() - self.opened_at, lifetime_s=self.lifetime_s
+        ):
+            return None
+        return exit_cause(self.probe())
+
     def _send_worker(self) -> None:
         # After a reexec the data connection is gone, so the source goes back over stdio.
         if self._connection is not self._stdio:
@@ -821,7 +880,12 @@ class SandboxChannel(FramedChannel):
                 )
                 carrier.timeout = DATA_CONNECT_TIMEOUT
             connection = Connection(
-                self.name, carrier.send, carrier.recv_into, self._emit, death_detail=self._raw_text
+                self.name,
+                carrier.send,
+                carrier.recv_into,
+                self._emit,
+                death_detail=self._raw_text,
+                death_cause=self._exit_cause,
             )
             # The connect timeout still applies, so a missing hello fails here.
             connection.wait_hello()

{letify-2.8.4 → letify-2.8.6}/letify/providers/modal_adapter.py

@@ -109,6 +109,7 @@ class Adapter:
             gpu=request.get("gpu") or None,
             timeout=int(request["timeout"]),
             idle_timeout=int(request["idle_timeout"]) if request.get("idle_timeout") else None,
+            memory=int(request["memory"]) if request.get("memory") else None,
             volumes={
                 str(path): modal.Volume.from_name(str(name), create_if_missing=True)
                 for path, name in (request.get("volumes") or {}).items()

{letify-2.8.4 → letify-2.8.6}/letify/runtime/channel.py

@@ -27,6 +27,7 @@ import os
 import select
 import subprocess
 import sys
+import tempfile
 import threading
 import time
 from collections.abc import Callable, Iterator, Sequence
@@ -258,6 +259,7 @@ class Connection:
         emit: Callable[[str, bytes], None],
         *,
         death_detail: Callable[[], str] = lambda: "",
+        death_cause: Callable[[], str | None] = lambda: None,
         poll_fd: int | None = None,
     ):
         self.name = name
@@ -276,6 +278,8 @@ class Connection:
         self._receiver = wire.Receiver(readinto)
         self._emit = emit
         self._death_detail = death_detail
+        #: Names what ended the worker when the provider can tell, replacing the usual causes.
+        self._death_cause = death_cause
         #: Guards ``_reading`` and wakes waiters when a reply, a hello or a failure arrives.
         self._turn = threading.Condition()
         self._reading = False
@@ -383,10 +387,16 @@ class Connection:
             return
         if message is None:
             detail = self._death_detail()
+            cause = self._death_cause()
+            causes = (
+                f": {cause}."
+                if cause
+                else ". The usual causes are an out of memory kill, a preempted session, "
+                "or a crash below Python."
+            )
             message = (
                 f"{self.name}: the worker stopped without replying, so the process died "
-                f"before it finished. The usual causes are an out of memory kill, a "
-                f"preempted session, or a crash below Python.\n--- last remote output ---\n"
+                f"before it finished{causes}\n--- last remote output ---\n"
                 f"{self._tail.text()[-2000:]}{detail[-2000:]}"
             )
         with self._turn:
@@ -770,6 +780,27 @@ class FramedChannel(Channel):
         return connection.pipeline(window=window, timeout=timeout, kill=self._kill)
 
 
+def exit_text(code: int | None) -> str | None:
+    """Name a subprocess exit code with its short meaning, or None while it still runs.
+
+    The text reads ``code <n>`` plus a meaning, to follow ``the worker exited with``.
+
+    Spec "Modal": a negative code is a signal, reported as the shell's 128 plus N.
+    """
+    if code is None:
+        return None
+    if code < 0:
+        code = 128 - code
+    text = f"code {code}"
+    if code == 255:
+        text += ": ssh could not reach or lost the machine"
+    elif code > 128:
+        text += f": killed by signal {code - 128}"
+        if code == 137:
+            text += ", which is SIGKILL, most often out of memory"
+    return text
+
+
 class PersistentChannel(FramedChannel):
     """One worker process, kept alive behind a pipe pair.
 
@@ -789,6 +820,8 @@ class PersistentChannel(FramedChannel):
         self._start_lock = threading.Lock()
         self._stderr = _Tail()
         self._stderr_reader: threading.Thread | None = None
+        #: Where ``ssh -E`` appends the log of the client and its forked master.
+        self._ssh_log: str | None = None
 
     # -- lifecycle -----------------------------------------------------------
 
@@ -796,9 +829,10 @@ class PersistentChannel(FramedChannel):
         with self._start_lock:
             if self._process is not None:
                 return
+            command = self._with_ssh_log()
             try:
                 process = subprocess.Popen(
-                    self.command,
+                    command,
                     stdin=subprocess.PIPE,
                     stdout=subprocess.PIPE,
                     stderr=subprocess.PIPE,
@@ -806,6 +840,7 @@ class PersistentChannel(FramedChannel):
                     env=self.env,
                 )
             except OSError as exc:
+                self._drop_ssh_log()
                 raise RuntimeFailure(
                     f"{self.name}: could not start the worker: {exc}",
                     command=" ".join(self.command),
@@ -825,7 +860,8 @@ class PersistentChannel(FramedChannel):
                 wire.fd_writer(process.stdin.fileno()),
                 process.stdout.readinto,  # type: ignore[attr-defined]
                 self._emit,
-                death_detail=self._stderr_text,
+                death_detail=lambda: self._stderr_text() + self._ssh_log_section(),
+                death_cause=self._exit_cause,
                 poll_fd=process.stdout.fileno(),
             )
             self._send_worker()
@@ -835,6 +871,48 @@ class PersistentChannel(FramedChannel):
         if self._process is not None:
             self._process.kill()
 
+    def _with_ssh_log(self) -> list[str]:
+        """The command, with ``-E <file>`` after the program when that program is ssh.
+
+        Spec "Modal": ControlPersist forks a master whose standard error is /dev/null, so
+        the transport's own errors reach only this file.
+        """
+        self._drop_ssh_log()
+        if os.path.basename(self.command[0]) not in ("ssh", "ssh.exe"):
+            return self.command
+        handle, self._ssh_log = tempfile.mkstemp(prefix="letify-ssh-", suffix=".log")
+        os.close(handle)
+        return [self.command[0], "-E", self._ssh_log, *self.command[1:]]
+
+    def _drop_ssh_log(self) -> None:
+        log, self._ssh_log = self._ssh_log, None
+        if log is not None:
+            with contextlib.suppress(OSError):
+                os.unlink(log)
+
+    def _ssh_log_section(self) -> str:
+        """The last 2000 characters of the ssh log under a heading, or empty when it is empty."""
+        if self._ssh_log is None:
+            return ""
+        try:
+            with open(self._ssh_log, "rb") as handle:
+                text = handle.read().decode("utf-8", "replace").strip()
+        except OSError:
+            return ""
+        return f"\n--- ssh log ---\n{text[-2000:]}" if text else ""
+
+    def _exit_cause(self) -> str | None:
+        """What the exit code of the worker process says, once the process has exited."""
+        process = self._process
+        if process is None:
+            return None
+        try:
+            process.wait(timeout=2)
+        except subprocess.TimeoutExpired:
+            return None
+        code = exit_text(process.returncode)
+        return f"the worker exited with {code}" if code else None
+
     def _stderr_text(self) -> str:
         """The process's own standard error, complete once the process has exited."""
         process = self._process
@@ -850,10 +928,15 @@ class PersistentChannel(FramedChannel):
     def _startup_failure(self, expired: bool, cause: Exception) -> Exception:
         if expired:
             return super()._startup_failure(expired, cause)
+        stderr = self._stderr_text().strip()
+        code = exit_text(self._process.returncode) if self._process else None
+        reason = f", with {code}" if code else ""
         return RuntimeFailure(
-            f"{self.name}: the worker exited before it was ready",
+            f"{self.name}: the worker exited before it was ready{reason}\n"
+            "--- last remote output ---\n"
+            f"{stderr[-2000:]}{self._ssh_log_section()}",
             command=" ".join(self.command),
-            stderr=self._stderr_text().strip(),
+            stderr=stderr,
         )
 
     def close(self) -> None:
@@ -876,6 +959,7 @@ class PersistentChannel(FramedChannel):
         finally:
             if self._stderr_reader is not None:
                 self._stderr_reader.join(1)
+            self._drop_ssh_log()
 
     @property
     def alive(self) -> bool:
@@ -890,7 +974,8 @@ class PersistentChannel(FramedChannel):
         if process is None or connection is None:
             raise RuntimeLost(f"{self.name}: the channel is closed")
         if process.poll() is not None:
-            raise RuntimeLost(f"{self.name}: the worker exited with code {process.returncode}")
+            code = exit_text(process.returncode)
+            raise RuntimeLost(f"{self.name}: the worker exited with {code}")
         return connection.request(payload, timeout=timeout, kill=self._kill)
 
 

{letify-2.8.4 → letify-2.8.6}/letify/store/backends/drive.py

@@ -17,6 +17,7 @@ than one request per digest.
 
 from __future__ import annotations
 
+import hashlib
 import json
 import os
 import shutil
@@ -105,6 +106,24 @@ def download_to(url: str, headers: dict[str, str], destination: str | os.PathLik
     return os.path.getsize(destination)
 
 
+def open_stream(url: str, headers: dict[str, str]) -> Any:
+    """Open one Drive file for sequential reads and return the response.
+
+    The caller reads it and closes it. A failure to open raises ``RuntimeFailure`` as
+    ``download_to`` does.
+    """
+    request = urllib.request.Request(url, headers=headers)
+    try:
+        return urllib.request.urlopen(request, timeout=3600)
+    except urllib.error.HTTPError as exc:
+        detail = exc.read()[:300].decode(errors="replace")
+        raise RuntimeFailure(f"Drive download returned {exc.code}: {detail}") from exc
+    except (urllib.error.URLError, OSError) as exc:
+        raise RuntimeFailure(f"Drive download failed: {exc}") from exc
+    finally:
+        request = None  # type: ignore[assignment]
+
+
 class AccountTokenSource(TokenSource):
     """Access tokens minted from the refresh token in one account's ``token.json``."""
 
@@ -354,22 +373,31 @@ class DriveBackend(Backend):
         self._call("DELETE", url, what=f"deleting {digest}")
         self._ids.pop(digest, None)
 
-    def refs_with_prefix(self, prefix: str) -> dict[str, str]:
-        """Every ref whose name starts with ``prefix``, as name to digest, in one listing."""
-        found: dict[str, str] = {}
+    def ref_holds(self, prefix: str, digest: str) -> bool:
+        """Whether a ref whose name starts with ``prefix`` holds ``digest``, from one listing.
+
+        A ref holds the digest and nothing else, so the listing's ``md5Checksum`` of each ref
+        names its content and no ref is downloaded. An entry listed without a checksum is read.
+        """
         wanted = REF_PREFIX + prefix
+        checksum = hashlib.md5(digest.encode()).hexdigest()
         for entry in self._list(
-            f"{_quoted(self.folder())} in parents and trashed = false", "id,name"
+            f"{_quoted(self.folder())} in parents and trashed = false", "id,name,md5Checksum"
         ):
             name = str(entry["name"])
             if not name.startswith(wanted):
                 continue
+            listed = entry.get("md5Checksum")
+            if listed is not None:
+                if listed == checksum:
+                    return True
+                continue
             payload = self._call(
                 "GET", self._media_url(str(entry["id"])), what=f"reading ref {name[4:]}"
             )
-            if payload is not None:
-                found[name[len(REF_PREFIX) :]] = payload.decode().strip()
-        return found
+            if payload is not None and payload.decode().strip() == digest:
+                return True
+        return False
 
     # -- the backend -----------------------------------------------------------