letify 2.8.3__tar.gz → 2.8.5__tar.gz

{letify-2.8.3 → letify-2.8.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: letify
-Version: 2.8.3
+Version: 2.8.5
 Summary: Declarations that become infrastructure.
 Project-URL: Homepage, https://github.com/DarkPyonix/letify
 Project-URL: Documentation, https://github.com/DarkPyonix/letify/tree/main/docs

{letify-2.8.3 → letify-2.8.5}/docs/SPEC.md

@@ -777,7 +777,7 @@ The protocol is one JSON object per line. letify sends `{"id": <int>, "op": <nam
 | Op | Fields | Value |
 |---|---|---|
 | `hello` | none | `{"modal": <installed Modal version>}` |
-| `create` | `app`, `args`, `packages`, `gpu`, `timeout`, `idle_timeout`, `volumes`, `ports` | `{"sandbox": <id>}`. Runs `app` as an ephemeral app on first use, builds `debian_slim` with `packages` installed, and starts `args` in a sandbox. Each port in `ports` is exposed with Modal `encrypted_ports` |
+| `create` | `app`, `args`, `packages`, `gpu`, `timeout`, `idle_timeout`, `memory`, `volumes`, `ports` | `{"sandbox": <id>}`. Runs `app` as an ephemeral app on first use, builds `debian_slim` with `packages` installed, and starts `args` in a sandbox. Each port in `ports` is exposed with Modal `encrypted_ports` |
 | `tunnel` | `sandbox`, `port` | `{"host": <text>, "port": <int>, "tls": <bool>}`. The address that reaches `port` inside the sandbox, from `Sandbox.tunnels()`. `tls` is true when the connection has to be made with TLS, which an encrypted port needs |
 | `write` | `sandbox`, `data` | `null`. `data` is base64 of at most 1 MiB. Writes the decoded bytes to the sandbox's standard input and drains it. Modal refuses a write that would buffer more than 2 MiB, so the channel splits a larger frame into `write` requests of 1 MiB, in order |
 | `read_until` | `sandbox`, `prefixes` | `{"lines": [...], "eof": <bool>}`. The sandbox's stdout lines up to and including the first that starts with one of `prefixes`, or every line left when the stream ends |
@@ -812,8 +812,12 @@ Modal also bounds a sandbox that nothing terminates. `create` passes `idle_timeo
 
 It passes `timeout`, the entry option `timeout`, as the sandbox's maximum lifetime, and the default is the longest Modal accepts, 86400 s, measured as accepted. It is not a shorter number letify picked, for the reason The declaration gives for `timeout` having no default: a deadline letify invented would end a two hour training run at whatever hour it guessed. The default was 3600 s, exactly one hour, and it ended two Modal sessions of a real training run at one hour each with nothing saying why, which is that sentence happening. An account that wants a hard cap sets `timeout` itself.
 
+It passes `memory`, the entry option `memory_gib` converted to MiB, as the sandbox's memory request, and passes nothing when the account does not set it, so Modal's default applies. A training process whose RAM grows past what the sandbox gets is killed with SIGKILL, which the channel sees as a worker that stopped without replying; an account whose runs need more sets `memory_gib`, for example `memory_gib = 64`. A value that is not a positive number raises `ConfigError` naming `memory_gib`.
+
 A sandbox that reaches that lifetime takes its worker with it, so the channel sees a worker that stopped without replying. Where the session's age is within a minute of the lifetime letify asked for, the failure says so instead of listing causes: `the Modal sandbox reached the <seconds> s lifetime letify asked for; raise timeout on the account to run longer`.
 
+Otherwise a Modal worker that stopped without replying is followed by one `poll` of its sandbox from a second adapter process, waiting at most 30 s for the exit code, and the failure names what the poll found instead of listing causes: `the Modal sandbox exited with code <n>` for a sandbox that ended, with `, which is SIGKILL, most often the container running out of memory` added for 137, and `the Modal sandbox is still running, so the worker process died inside it` for one still running. A poll that fails or times out leaves the message as it was. The message is what tells an out-of-memory kill from a preemption, which a training run that died twice in one day could not.
+
 ### Modal data channel <!-- id: modal-data-channel -->
 
 > Once the worker has said hello over the sandbox's standard input and output, the channel moves its frames to a TCP connection through a Modal encrypted port. Standard input and output stay the control path and the fallback.
@@ -924,6 +928,8 @@ A top-level argument whose serialized size is 64 KiB or more is replaced by a `B
 
 A `bytes` argument is hashed as it is. Any other argument is pickled with protocol 5 and a `buffer_callback` through `codec.pickle_value_parts`, cloudpickle underneath rather than the standard library's `pickle.Pickler`, and the digest is taken over the pickle and then each out-of-band buffer in order, with no joined copy. The digest is blake3 with 16 byte output, or blake2b from the standard library where blake3 is missing.
 
+A digest names its own algorithm. A blake3 digest is the 32 hexadecimal characters of the 16 byte output, unchanged, so every digest already stored in a cache, a volume or Drive stays valid. A digest computed with blake2b because blake3 is missing is `b2-` followed by the 32 hexadecimal characters of the 16 byte blake2b output. Whoever checks a digest picks the algorithm from the digest, through `codec.hasher_for`, so a worker without blake3 and a client with it agree on every blob, archive, write-back file and handoff, and no handshake is needed. A side that cannot compute the algorithm of a digest it was given, a worker without blake3 handed a blake3 digest, does not check it and trusts the sender, and a name it computes itself uses its own algorithm. A placed file is compared by the algorithm of its recorded digest, so a worker without blake3 does not report an unchanged file as an output.
+
 cloudpickle rather than plain `pickle` because a `pickle.Pickler` writes a class or function defined in the caller's own run script as a name reference, `__main__.<name>`, and the worker's own `__main__` is not that script, so unpickling such a blob fails with `AttributeError: Can't get attribute '<name>' on <module '__main__' ...>`. cloudpickle ships such an object by value instead, the same as it already does for the call itself in `codec.dumps_call_parts`, so an argument too large to inline behaves the same as one that is not. The cost of cloudpickle's extra dispatch over plain `pickle` is in the noise: it only intercepts function, class and module objects, not the bulk of a large value, so a 256 MiB buffer or a multi-megabyte list of plain values serializes within a few percent of the same time either way, measured on this machine. There is accordingly no branch between the two picklers.
 
 The digest of an immutable argument is cached on the `Runtime` for the life of the session, so a repeated argument is not hashed again. Immutable means a `bytes` object, or an object that supports weak references and exposes a read-only buffer, such as a NumPy array with `writeable` set to `False`. A `bytes` entry holds a reference to its object so its id cannot be reused, and at most 16 such entries are kept, least recently used first out. Any other argument is pickled and hashed on every call, because it may have changed.
@@ -1088,7 +1094,7 @@ neither side holds it whole:
   returns `{"digest", "size", "dir"}`. The client receives the file with the stream a write-back
   uses, checks the digest as the pieces arrive into a scratch file beside the store, and renames it
   into the store with `Store.adopt_file`. The staging directory is removed afterwards. The digest
-  is the store's content digest (blake3 truncated to 16 bytes, blake2b without blake3).
+  is the store's content digest (blake3 truncated to 16 bytes, or `b2-` and blake2b where the worker has no blake3), and the client checks it with the algorithm the digest names.
 - Restore: a blob of more than one piece (8 MiB) that the store holds as a file is sent with
   `data_put` pieces into a staging directory beside the destination, then moved into place and
   unpacked by the worker op `place_blob`. Smaller blobs, and any blob from a store that does not
@@ -1114,6 +1120,8 @@ the kind this rule exists to catch, and the session that syncs instead both reco
 decides afresh whether the archive is worth keeping. The cost of that is one sync; the cost of
 trusting it is the transfer this rule was measured to avoid.
 
+A failure to cache the environment through the client's own store at boot, the pack, the stream back or the store write, never fails the session: it says `letify: <runtime>: not caching the environment (<reason>)` and the session goes on, the same as a failed pack on the workspace root.
+
 Packing is said, because it is a step that can take minutes and had nothing reporting it:
 `letify: <runtime>: caching the environment, 2.1 GB`. Skipping it is said too, because it is a
 decision letify made rather than a thing that did not
@@ -1949,7 +1957,7 @@ Colab limits outbound UDP to roughly 200 packets per second, so rank 3 is expect
 
 Kaggle reports 2.0 MiB/s, measured on a live account sending 64 MiB over the kernel bridge, and an account overrides it with `fallback_mib_per_s`. Without this a live session refused a punched link at down 8.1 MiB/s for being under the 10 MiB/s floor and then carried 26 GB over the 2 MiB/s bridge instead, which is the opposite of what the floor is for. Taking the fallback is also never silent: the reason is printed, because a session that quietly changed channel is one nobody can account for.
 
-A `channel = "exec"` entry skips the pipeline and uses the fallback directly. A Colab VM has no SSH server on the port letify splices to, so the rendezvous request asks the remote half to start one first. The remote half connects to `127.0.0.1:<ssh_port>`, and when no line starting with `SSH-` arrives within 3 s it installs `openssh-server` if `/usr/sbin/sshd` is missing, creates `/run/sshd`, runs `ssh-keygen -A` so the server has host keys, and starts `/usr/sbin/sshd -p <ssh_port> -o ListenAddress=127.0.0.1`. `ssh-keygen -A` creates only the keys that are missing, and it is needed because a Kaggle image ships `sshd` with no host key at all: the install step above is skipped since the package is already there, and a server with no host key accepts the connection and closes it before the banner, which the client reports as `Connection closed by 127.0.0.1 port <port>` with no reason. The port is explicit because a Colab image ships `sshd` configured for `127.0.0.1:2222`, so starting it with its own configuration leaves port 22 closed. An SSH server that already answers on the port is used as it is. The request carries the account's public key, `key` with `.pub` appended, which the remote side adds to `authorized_keys`. An account with no `key`, or whose `.pub` file is missing, has nothing the VM could authorize, so its Colab rendezvous is unavailable with the reason `no key`: `tcp_punch` and `tailcat` are skipped with that reason and the fallback carries the session. SSH over a punched or Tailcat link logs in as `root` unless `user` says otherwise.
+A `channel = "exec"` entry skips the pipeline and uses the fallback directly. A Colab VM has no SSH server on the port letify splices to, so the rendezvous request asks the remote half to start one first. The remote half connects to `127.0.0.1:<ssh_port>`, and when no line starting with `SSH-` arrives within 3 s it installs `openssh-server` if `/usr/sbin/sshd` is missing, creates `/run/sshd`, runs `ssh-keygen -A` so the server has host keys, and starts `/usr/sbin/sshd -f /etc/ssh/sshd_config.letify -p <ssh_port> -o ListenAddress=127.0.0.1`. That file is the image's `/etc/ssh/sshd_config` with every `MaxStartups` line removed and `MaxStartups 64:30:128` as its first line, or that line alone when the image has no file. A Colab image's file sets `MaxStartups 1`, so the server dropped every connection that arrived while another was still authenticating, and a command-line `-o MaxStartups` does not override the file. Over `reverse_ssh` the transfer streams of a session arrive at once, so every one after the first ended with `Connection closed by 127.0.0.1 port <port>` and the call failed with `the worker exited before it was ready`. With the file, one 190.7 MiB blob crossed a reverse SSH link from a Colab CPU runtime in 5.4 s, 35.5 MiB/s. `ssh-keygen -A` creates only the keys that are missing, and it is needed because a Kaggle image ships `sshd` with no host key at all: the install step above is skipped since the package is already there, and a server with no host key accepts the connection and closes it before the banner, which the client reports as `Connection closed by 127.0.0.1 port <port>` with no reason. The port is explicit because a Colab image ships `sshd` configured for `127.0.0.1:2222`, so starting it with its own configuration leaves port 22 closed. An SSH server that already answers on the port is used as it is. The request carries the account's public key, `key` with `.pub` appended, which the remote side adds to `authorized_keys`. An account with no `key`, or whose `.pub` file is missing, has nothing the VM could authorize, so its Colab rendezvous is unavailable with the reason `no key`: `tcp_punch` and `tailcat` are skipped with that reason and the fallback carries the session. SSH over a punched or Tailcat link logs in as `root` unless `user` says otherwise.
 
 A Colab VM keeps its GPU driver libraries in `/usr/lib64-nvidia`, and only the notebook kernel's own environment names that directory in `LD_LIBRARY_PATH`. A login shell does not, so a worker started over SSH finds no `libnvidia-ml.so` and no `libcuda.so`: `nvidia-smi` fails and `torch.cuda.is_available()` is false on a session that holds a GPU. Every command letify runs over a Colab link, the worker and the `check` command included, is therefore prefixed with `LD_LIBRARY_PATH=/usr/lib64-nvidia${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}`, which keeps a path the machine already set. The `colab exec` fallback needs no prefix, because it runs inside the kernel.
 

{letify-2.8.3 → letify-2.8.5}/letify/__init__.py

@@ -68,7 +68,7 @@ distributed: Final = Host.distributed
 # The enum class stays importable from letify.declare.instance for letify's own use.
 del Host
 
-__version__ = "2.8.3"
+__version__ = "2.8.5"
 
 
 def fetch(tensor):  # type: ignore[no-untyped-def]

{letify-2.8.3 → letify-2.8.5}/letify/protocol/codec.py

@@ -130,6 +130,10 @@ class _CallPickler(cloudpickle.Pickler):
         return super().reducer_override(obj)
 
 
+#: Prefix of a digest computed with blake2b because blake3 was missing. A blake3 digest has none.
+BLAKE2B_PREFIX = "b2-"
+
+
 def _hasher() -> Any:
     """A fresh incremental hasher: blake3 on every core, or blake2b where blake3 is missing."""
     try:
@@ -146,10 +150,28 @@ def new_hasher() -> Any:
     return _hasher()
 
 
+def hasher_for(digest: str) -> Any | None:
+    """A fresh hasher of the algorithm ``digest`` names, or None where it cannot be computed.
+
+    Spec "Argument addressing": a digest names its own algorithm, so a side checking one
+    follows the side that computed it. None means blake3 is missing here and the digest is
+    blake3, so the check cannot be made.
+    """
+    if digest.startswith(BLAKE2B_PREFIX):
+        import hashlib
+
+        return hashlib.blake2b(digest_size=16)
+    try:
+        import blake3
+    except ImportError:
+        return None
+    return blake3.blake3(max_threads=blake3.blake3.AUTO)
+
+
 def finish_digest(hasher: Any) -> str:
-    """The content address a hasher from ``new_hasher`` has accumulated."""
+    """The content address a hasher from ``new_hasher`` or ``hasher_for`` has accumulated."""
     if hasher.name == "blake2b":
-        return str(hasher.hexdigest())
+        return BLAKE2B_PREFIX + str(hasher.hexdigest())
     return str(hasher.hexdigest(length=16))
 
 
@@ -158,14 +180,13 @@ def digest_parts(parts: Iterable[Any]) -> str:
 
     blake3 is preferred because it hashes at several gigabytes per second, which
     keeps hashing orders of magnitude away from being the bottleneck on any real
-    network link. blake2b from the standard library is the fallback.
+    network link. blake2b from the standard library is the fallback, and its digests
+    carry ``BLAKE2B_PREFIX``.
     """
     hasher = _hasher()
     for part in parts:
         hasher.update(part)
-    if hasher.name == "blake2b":
-        return hasher.hexdigest()
-    return hasher.hexdigest(length=16)
+    return finish_digest(hasher)
 
 
 def digest_of(payload: bytes) -> str:
@@ -297,6 +318,7 @@ __all__ = [
     "dumps_call",
     "dumps_call_parts",
     "finish_digest",
+    "hasher_for",
     "new_hasher",
     "parse",
     "split_output",

{letify-2.8.3 → letify-2.8.5}/letify/protocol/worker.py

@@ -247,11 +247,9 @@ def _release_output():
 
 
 def _digest(payload):
-    try:
-        import blake3
-        return blake3.blake3(payload).hexdigest(length=16)
-    except ImportError:
-        return hashlib.blake2b(payload, digest_size=16).hexdigest()
+    hasher = _data_new_hasher()
+    hasher.update(payload)
+    return _data_finish(hasher)
 
 
 def _resolve(value):
@@ -394,20 +392,22 @@ def _letify_replica(replica):
     _replicas._WHO.rate_bps = replica.get("rate_bps")
 
 
-def _data_hash_file(path):
-    """The file digest of a file on the runtime: blake3 when present, else blake2b."""
-    hasher = _data_hasher()
+def _data_hash_file(path, like=None):
+    """The file digest of a file on the runtime: blake3 when present, else blake2b.
+
+    With ``like``, a digest the file may already be named by, the algorithm that digest
+    names is used when it can be, so the result is comparable with it.
+    """
+    hasher = _data_hasher(like) if like is not None else None
     if hasher is None:
-        hasher = hashlib.blake2b(digest_size=16)
+        hasher = _data_new_hasher()
     with open(path, "rb") as handle:
         while True:
             piece = handle.read(1 << 22)
             if not piece:
                 break
             hasher.update(piece)
-    if hasher.name == "blake2b":
-        return hasher.hexdigest()
-    return hasher.hexdigest(length=16)
+    return _data_finish(hasher)
 
 
 def _data_same_bytes(path, digest):
@@ -418,7 +418,7 @@ def _data_same_bytes(path, digest):
     Never raises; a file that cannot be read is not the file the digest promised.
     """
     try:
-        return _data_hash_file(path) == digest
+        return _data_hash_file(path, digest) == digest
     except OSError:
         return False
 
@@ -453,7 +453,7 @@ def _data_collect(data, placed):
             # The placed record ends with the cache stamp, which is not part of the comparison.
             if known is not None and known[1:4] == (info.st_ino, info.st_size, info.st_mtime_ns):
                 continue
-            digest = _data_hash_file(full)
+            digest = _data_hash_file(full, known[0] if known is not None else None)
             if known is not None and known[0] == digest:
                 continue
             final = _data_file(data["blobs"], digest)
@@ -667,7 +667,14 @@ def _data_file(root, digest):
     return os.path.join(root, digest[:2], digest)
 
 
-def _data_hasher():
+def _data_hasher(digest):
+    """A hasher of the algorithm ``digest`` names, or None where it cannot be computed here.
+
+    A digest with the ``b2-`` prefix is blake2b and one without is blake3, so a runtime
+    without blake3 cannot check a blake3 digest and trusts the sender.
+    """
+    if digest.startswith("b2-"):
+        return hashlib.blake2b(digest_size=16)
     try:
         import blake3
     except ImportError:
@@ -675,10 +682,23 @@ def _data_hasher():
     return blake3.blake3(max_threads=blake3.blake3.AUTO)
 
 
+def _data_new_hasher():
+    """The hasher this runtime names new content with: blake3 when present, else blake2b."""
+    hasher = _data_hasher("")
+    return hasher if hasher is not None else hashlib.blake2b(digest_size=16)
+
+
+def _data_finish(hasher):
+    """The digest a hasher has accumulated, with the prefix a blake2b digest carries."""
+    if hasher.name == "blake2b":
+        return "b2-" + hasher.hexdigest()
+    return hasher.hexdigest(length=16)
+
+
 def _data_commit(partial, final, digest, hasher):
     """Rename a received file into the cache once its digest matches, read-only."""
     if hasher is not None:
-        found = hasher.hexdigest(length=16)
+        found = _data_finish(hasher)
         if found != digest:
             os.remove(partial)
             raise ValueError("file blob %s arrived with digest %s" % (digest, found))
@@ -838,7 +858,7 @@ def _op_data_put(request):
         stale = _DATA_OPEN.pop(digest, None)
         if stale is not None:
             stale[0].close()
-        _DATA_OPEN[digest] = (open(partial, "wb"), _data_hasher())
+        _DATA_OPEN[digest] = (open(partial, "wb"), _data_hasher(digest))
     handle, hasher = _DATA_OPEN[digest]
     handle.write(chunk)
     if hasher is not None:
@@ -938,7 +958,7 @@ def _op_data_seal(request):
         raise ValueError(
             "file blob %s is short: %d of %d bytes arrived" % (digest, found, int(size))
         )
-    hasher = _data_hasher()
+    hasher = _data_hasher(digest)
     if hasher is not None:
         with open(partial, "rb") as handle:
             while True:
@@ -946,7 +966,7 @@ def _op_data_seal(request):
                 if not piece:
                     break
                 hasher.update(piece)
-        got = hasher.hexdigest(length=16)
+        got = _data_finish(hasher)
         if got != digest:
             os.remove(partial)
             raise ValueError("file blob %s arrived with digest %s" % (digest, got))
@@ -979,7 +999,7 @@ def _op_data_pull(request):
             partial = "%s.partial.%d.%d" % (final, os.getpid(), threading.get_ident())
             try:
                 os.makedirs(os.path.dirname(final), exist_ok=True)
-                hasher = _data_hasher()
+                hasher = _data_hasher(digest)
                 wanted = urllib.request.Request(url, headers=headers)
                 with urllib.request.urlopen(wanted, timeout=3600) as response, \
                         open(partial, "wb") as out:
@@ -2334,16 +2354,14 @@ def _op_pack_blob(request):
     directory = os.path.expanduser(request["dir"])
     os.makedirs(directory, exist_ok=True)
     partial = os.path.join(directory, ".pack.%d.partial" % os.getpid())
-    hasher = _data_hasher()
-    if hasher is None:
-        hasher = hashlib.blake2b(digest_size=16)
+    hasher = _data_new_hasher()
     try:
         with open(partial, "wb") as handle:
             hashed = _HashedFile(handle, hasher)
             mode = "w:gz" if request.get("gzip", True) else "w"
             with tarfile.open(fileobj=hashed, mode=mode) as archive:
                 archive.add(root, arcname=os.path.basename(root.rstrip("/")), recursive=True)
-        digest = hasher.hexdigest() if hasher.name == "blake2b" else hasher.hexdigest(length=16)
+        digest = _data_finish(hasher)
         final = _data_file(directory, digest)
         os.makedirs(os.path.dirname(final), exist_ok=True)
         os.replace(partial, final)

{letify-2.8.3 → letify-2.8.5}/letify/providers/modal.py

@@ -31,7 +31,7 @@ import tempfile
 import threading
 import time
 import weakref
-from collections.abc import Mapping
+from collections.abc import Callable, Mapping
 from typing import IO, TYPE_CHECKING, Any
 
 from ..config import ProviderConfig
@@ -133,6 +133,22 @@ def lifetime_detail(*, age_s: float, lifetime_s: int) -> str | None:
     )
 
 
+def exit_cause(found: tuple[bool, int | None] | None) -> str | None:
+    """What a poll of a sandbox whose worker stopped says, or None when it is unknown.
+
+    Spec "Modal": the exit code tells an out of memory kill from a preemption.
+    """
+    if found is None:
+        return None
+    ended, code = found
+    if not ended:
+        return "the Modal sandbox is still running, so the worker process died inside it"
+    text = f"the Modal sandbox exited with code {code}"
+    if code == 137:
+        text += ", which is SIGKILL, most often the container running out of memory"
+    return text
+
+
 class VolumePathMissing(RuntimeFailure):
     """The adapter answered that a volume path does not exist."""
 
@@ -419,6 +435,21 @@ class Modal(Provider):
         value = self.config.option("idle_timeout", IDLE_TIMEOUT)
         return max(1, int(value))
 
+    @property
+    def sandbox_memory_mib(self) -> int | None:
+        """The sandbox's memory request in MiB from ``memory_gib``, or None for Modal's own.
+
+        Spec "Modal": a training process past what the sandbox gets is killed with SIGKILL.
+        """
+        value = self.config.option("memory_gib")
+        if value is None:
+            return None
+        if isinstance(value, bool) or not isinstance(value, (int, float)) or value <= 0:
+            raise ConfigError(
+                f"{self.alias}: memory_gib has to be a positive number of GiB, not {value!r}"
+            )
+        return int(value * 1024)
+
     def adapter(self) -> Adapter:
         """The adapter acting as this account, started on its first request."""
         if self._adapter is None or self._adapter.closed:
@@ -484,6 +515,7 @@ class Modal(Provider):
             gpu=self.wire_name(runtime.instance) or None,
             timeout=self.sandbox_timeout_s,
             idle_timeout=self.sandbox_idle_timeout_s,
+            memory=self.sandbox_memory_mib,
             ports=[] if data_port is None else [data_port],
         )
         sandbox = str(created["sandbox"])
@@ -491,17 +523,14 @@ class Modal(Provider):
         return SandboxChannel(
             adapter, sandbox, name=runtime.name, data_port=data_port, streams=streams,
             lifetime_s=self.sandbox_timeout_s,
+            probe=lambda: self.poll_sandbox(sandbox),
         )
 
-    def runtime_state(self, runtime: Runtime) -> RuntimeState:
-        """Poll the sandbox from a second adapter process, never the one carrying the call.
+    def poll_sandbox(self, sandbox: str) -> tuple[bool, int | None] | None:
+        """Poll a sandbox once from a second adapter: ``(ended, exit code)``, or None if unknown.
 
-        Spec "Lifetime", the lease. The adapter carrying the call may be blocked in a
-        request, so a short-lived one asks. Any failure answers unknown.
+        A short-lived adapter asks, because the one carrying the call may be blocked.
         """
-        sandbox = self._sandboxes.get(runtime.name)
-        if sandbox is None:
-            return RuntimeState()
         other = Adapter.for_account(self.alias)
         # A poll that hangs ends the second adapter, which makes the request raise.
         limit = threading.Timer(POLL_LIMIT, other.abort)
@@ -510,11 +539,26 @@ class Modal(Provider):
         try:
             code = other.request("poll", sandbox=sandbox)
         except (LetifyError, ValueError, OSError):
-            return RuntimeState()
+            return None
         finally:
             limit.cancel()
             other.close()
-        if code is None:
+        return (code is not None, code)
+
+    def runtime_state(self, runtime: Runtime) -> RuntimeState:
+        """Poll the sandbox from a second adapter process, never the one carrying the call.
+
+        Spec "Lifetime", the lease. The adapter carrying the call may be blocked in a
+        request, so a short-lived one asks. Any failure answers unknown.
+        """
+        sandbox = self._sandboxes.get(runtime.name)
+        if sandbox is None:
+            return RuntimeState()
+        found = self.poll_sandbox(sandbox)
+        if found is None:
+            return RuntimeState()
+        ended, code = found
+        if not ended:
             return RuntimeState("running")
         return RuntimeState(
             "ended", f"the Modal sandbox ended (exit code {code}) while the call was running"
@@ -686,8 +730,11 @@ class SandboxChannel(FramedChannel):
         data_port: int | None = None,
         streams: int = 1,
         lifetime_s: int = 0,
+        probe: Callable[[], tuple[bool, int | None] | None] | None = None,
     ):
         self.adapter = adapter
+        #: Polls the sandbox from a second adapter, see ``Modal.poll_sandbox``.
+        self.probe = probe
         self.sandbox = sandbox
         self.name = name
         self.data_port = data_port
@@ -716,6 +763,7 @@ class SandboxChannel(FramedChannel):
             wire.chunks_readinto(self._read_chunks),
             self._emit,
             death_detail=self._raw_text,
+            death_cause=self._exit_cause,
         )
         self._connection = self._stdio
         self._send_worker()
@@ -730,6 +778,17 @@ class SandboxChannel(FramedChannel):
         )
         return f"{said}\n{text}" if said else text
 
+    def _exit_cause(self) -> str | None:
+        """What one poll of the sandbox found, or None to keep the usual causes.
+
+        Spec "Modal": a lifetime death is already named by ``_raw_text``, so it is not polled.
+        """
+        if self.probe is None or lifetime_detail(
+            age_s=time.monotonic() - self.opened_at, lifetime_s=self.lifetime_s
+        ):
+            return None
+        return exit_cause(self.probe())
+
     def _send_worker(self) -> None:
         # After a reexec the data connection is gone, so the source goes back over stdio.
         if self._connection is not self._stdio:
@@ -821,7 +880,12 @@ class SandboxChannel(FramedChannel):
                 )
                 carrier.timeout = DATA_CONNECT_TIMEOUT
             connection = Connection(
-                self.name, carrier.send, carrier.recv_into, self._emit, death_detail=self._raw_text
+                self.name,
+                carrier.send,
+                carrier.recv_into,
+                self._emit,
+                death_detail=self._raw_text,
+                death_cause=self._exit_cause,
             )
             # The connect timeout still applies, so a missing hello fails here.
             connection.wait_hello()

{letify-2.8.3 → letify-2.8.5}/letify/providers/modal_adapter.py

@@ -109,6 +109,7 @@ class Adapter:
             gpu=request.get("gpu") or None,
             timeout=int(request["timeout"]),
             idle_timeout=int(request["idle_timeout"]) if request.get("idle_timeout") else None,
+            memory=int(request["memory"]) if request.get("memory") else None,
             volumes={
                 str(path): modal.Volume.from_name(str(name), create_if_missing=True)
                 for path, name in (request.get("volumes") or {}).items()

{letify-2.8.3 → letify-2.8.5}/letify/runtime/channel.py

@@ -258,6 +258,7 @@ class Connection:
         emit: Callable[[str, bytes], None],
         *,
         death_detail: Callable[[], str] = lambda: "",
+        death_cause: Callable[[], str | None] = lambda: None,
         poll_fd: int | None = None,
     ):
         self.name = name
@@ -276,6 +277,8 @@ class Connection:
         self._receiver = wire.Receiver(readinto)
         self._emit = emit
         self._death_detail = death_detail
+        #: Names what ended the worker when the provider can tell, replacing the usual causes.
+        self._death_cause = death_cause
         #: Guards ``_reading`` and wakes waiters when a reply, a hello or a failure arrives.
         self._turn = threading.Condition()
         self._reading = False
@@ -383,10 +386,16 @@ class Connection:
             return
         if message is None:
             detail = self._death_detail()
+            cause = self._death_cause()
+            causes = (
+                f": {cause}."
+                if cause
+                else ". The usual causes are an out of memory kill, a preempted session, "
+                "or a crash below Python."
+            )
             message = (
                 f"{self.name}: the worker stopped without replying, so the process died "
-                f"before it finished. The usual causes are an out of memory kill, a "
-                f"preempted session, or a crash below Python.\n--- last remote output ---\n"
+                f"before it finished{causes}\n--- last remote output ---\n"
                 f"{self._tail.text()[-2000:]}{detail[-2000:]}"
             )
         with self._turn:

{letify-2.8.3 → letify-2.8.5}/letify/runtime/session.py

@@ -86,6 +86,13 @@ def _against_the_build(size: int, rate: float, build_s: float) -> str:
     )
 
 
+def _reason_of(error: BaseException) -> str:
+    """One line naming an error's type and first message line, for a status message."""
+    lines = str(error).strip().splitlines()
+    reason = f"{type(error).__name__}: {lines[0]}" if lines else type(error).__name__
+    return reason[:197] + "..." if len(reason) > 200 else reason
+
+
 def _sync_rather_than_restore(archive_bytes: int, rate: float, build_s: float) -> str:
     """The line explaining why a session syncs instead of restoring a cached archive.
 
@@ -243,9 +250,14 @@ class Runtime:
                 if volume.store.backend.moves_on_runtime:
                     self._start_drive_pack(volume, root)
                 else:
-                    volume.cache_env_from(
-                        self, self.env, root, platform=self.platform, build_s=self.env_build_s
-                    )
+                    try:
+                        volume.cache_env_from(
+                            self, self.env, root, platform=self.platform, build_s=self.env_build_s
+                        )
+                    except Exception as error:
+                        # The environment is built and checked, so a failed cache must not
+                        # fail the session. Spec "Materializing into a runtime".
+                        self._say(f"not caching the environment ({_reason_of(error)})")
         self.keep_blobs_on_disk()
         for volume in self.volumes:
             self.attach(volume)
@@ -831,11 +843,7 @@ class Runtime:
             self.eval(source, timeout=3600)
         except Exception as error:
             # The environment is built and checked, so a failed cache must not fail the call.
-            lines = str(error).strip().splitlines()
-            reason = f"{type(error).__name__}: {lines[0]}" if lines else type(error).__name__
-            if len(reason) > 200:
-                reason = reason[:197] + "..."
-            self._say(f"not caching the environment on the workspace root ({reason})")
+            self._say(f"not caching the environment on the workspace root ({_reason_of(error)})")
 
     def _environment_volumes(self) -> tuple[Volume, ...]:
         """Explicit archive volumes, or the automatic cache where the project does not survive.

{letify-2.8.3 → letify-2.8.5}/letify/store/handoff.py

@@ -248,7 +248,13 @@ def unstash(name: str, path: str | os.PathLike[str]) -> Path | None:
 
 
 def _verify(name: str, archive: Path, digest: str) -> None:
-    found = digest_of_file(archive)
+    hasher = codec.hasher_for(digest)
+    if hasher is None:
+        return
+    with open(archive, "rb") as handle:
+        while chunk := handle.read(1 << 20):
+            hasher.update(chunk)
+    found = codec.finish_digest(hasher)
     if found != digest:
         raise RuntimeFailure(
             f"handoff {name!r}: the blob downloaded from Drive has digest {found}, "

{letify-2.8.3 → letify-2.8.5}/letify/store/pathdata.py

@@ -139,9 +139,9 @@ def hash_file(path: Path, size: int) -> str:
         else:
             while chunk := handle.read(1 << 20):
                 hasher.update(chunk)
-    if hasher.name == "blake2b":
-        return hasher.hexdigest()
-    return hasher.hexdigest(length=16)
+    from ..protocol.codec import finish_digest
+
+    return finish_digest(hasher)
 
 
 class DigestCache:
@@ -1608,7 +1608,7 @@ def _get(
     Spec "Writing back". One writing thread takes the pieces, so receiving the next one
     overlaps writing and hashing the one before it and the link does not idle.
     """
-    from ..protocol.codec import _hasher
+    from ..protocol.codec import finish_digest, hasher_for
 
     # Spec "Writing back": a stream that fails after these were created would leave them
     # empty, and a resume that picks the newest directory by name then trusts a checkpoint
@@ -1617,7 +1617,8 @@ def _get(
     partial = target.with_name(
         f".{target.name}.letify-partial.{os.getpid()}.{threading.get_ident()}"
     )
-    hasher = _hasher()
+    # The algorithm the digest names, which is the worker's when its venv has no blake3.
+    hasher = hasher_for(digest)
     pieces: queue.Queue = queue.Queue(maxsize=WRITE_QUEUE)
     failure: list[BaseException] = []
 
@@ -1630,7 +1631,8 @@ def _get(
                     if piece is None:
                         return
                     handle.write(piece)
-                    hasher.update(piece)
+                    if hasher is not None:
+                        hasher.update(piece)
         except BaseException as exc:  # pragma: no cover - a local disk failure
             failure.append(exc)
             # Drained, so the receiving thread is never left waiting on a full queue.
@@ -1657,7 +1659,7 @@ def _get(
             raise RuntimeFailure(
                 f"{runtime.name}: file blob {digest} arrived with {received} bytes, not {size}"
             )
-        found = hasher.hexdigest() if hasher.name == "blake2b" else hasher.hexdigest(length=16)
+        found = finish_digest(hasher) if hasher is not None else digest
         if found != digest:
             raise RuntimeFailure(f"{runtime.name}: file blob {digest} arrived with digest {found}")
         os.replace(partial, target)