letify 2.5.2__tar.gz → 2.5.4__tar.gz

{letify-2.5.2 → letify-2.5.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: letify
-Version: 2.5.2
+Version: 2.5.4
 Summary: Declarations that become infrastructure.
 Project-URL: Homepage, https://github.com/DarkPyonix/letify
 Project-URL: Documentation, https://github.com/DarkPyonix/letify/tree/main/docs

{letify-2.5.2 → letify-2.5.4}/docs/SPEC.md

@@ -1098,6 +1098,15 @@ A detected path that exists is an input. A detected directory and a detected pat
 
 Any other path is pickled unchanged, so the body receives the local path as it was. Only `os.PathLike` objects are detected. A string that happens to name a local file is left alone, because no rule tells a path from any other string.
 
+One other path is refused rather than pickled unchanged: an `os.PathLike` naming a file or directory that exists on this machine and lies outside every allowed root. Passing it on leaves the body reading a path that is not there, which fails as `FileNotFoundError` on the runtime and names neither the cause nor the rule, and on a runtime where something does sit at that path it reads the wrong bytes instead. So the call raises `ValueError` before it is sent:
+
+```
+<path> is outside the roots letify may send data from. The project root is <root>. Add its
+directory to [tool.letify] data_roots in <root>/pyproject.toml, or move the file inside the root.
+```
+
+A path that exists here and is inside a root is data; a path that does not exist here is a runtime path and still travels unchanged, because that is how a body is told where to write.
+
 A directory is walked without following symbolic links to directories. Regular files and symbolic links to regular files are included. Entries named `.git`, `.venv` and `__pycache__` are skipped.
 
 #### Digests and the digest cache <!-- id: project-data-digests -->
@@ -1230,6 +1239,7 @@ A `stat` cannot take that snapshot first, because it is on a data loader's hot p
 #### When a blob does not arrive <!-- id: project-data-streaming-failures -->
 
 - **A blob that never arrives.** A `data_want` unanswered for `data_wait_timeout` seconds on the account, 600 by default, fails that open with `RuntimeFailure` naming the relative path and the digest. The call is not killed, because the body may handle it; the failure is infrastructure, so a retry of the call is allowed.
+- **A sender that stops.** The background sender raising leaves every blob still queued unsendable. letify says so once, `letify: data sending stopped after <n> of <n> files, <n> not sent: <error>`, and tells the runtime, which wakes every reader waiting on one of those blobs with `RuntimeFailure` naming the file and that reason, and fails a reader that arrives later at once. Waiting it out instead costs `data_wait_timeout` once for every blob still queued, which is how a hundred queued shards become an hour of silence.
 - **A client that disconnects mid stream.** The session ends as any lost channel ends it. Every blocked reader is woken with `RuntimeFailure`, the call ends with it, and no partial file is ever placed or read. A `.partial` file is removed when the worker next starts.
 - **A call that finishes before the background transfer.** The transfer is cancelled, both the channel sender and any bucket upload not yet started. A blob already committed to the cache stays and counts as held for the next call, and a partial one is removed. The write-back and the `data_evict` request follow the cancellation, in that order.
 - **A bucket-backed ephemeral provider.** A pull that fails is retried twice with the same downscoped headers, then a third time with headers the local process derives again, and a pull that still fails fails the open with `RuntimeFailure`. A `data_want` for a digest not yet uploaded moves that object to the front of the upload queue, and the worker pulls it when the local process sends its `data_pull`.
@@ -1290,6 +1300,10 @@ A call that detected data prints one line on standard error:
 
 The line is printed when the call has at least one input file, after the call's outcome is known, because what was sent during the call is not known before then. Sizes are in MiB with one decimal. `<the bucket>` is named when the bytes come from the bucket. `before the call` counts the first wave of The send order and the first wave, `during it` counts what the background sender delivered while the call ran, and the rate covers both. `first access waited` is the wall clock time in which at least one reader was blocked on a blob that had not arrived, and it is `0.0` when the send order was right.
 
+`already on` counts the blobs the runtime answered `data_have` with, not what is left after subtracting what was sent. The two differ whenever a blob was neither already there nor fully sent, and subtracting reports such a blob as having been there all along, which is the opposite of what happened. A blob the background sender was still carrying when the call returned is one of those, so the line ends with `, <files> files <size> dropped when the call ended` whenever there is one, and omits that clause when there is none. A dropped blob is not a failure: the call is over and its bytes are no longer needed, which is why the sender is cancelled. It is reported because a line that accounts for every detected file except the dropped ones cannot be read.
+
+Cancelling the sender removes the partial file of a blob it was writing, in the runtime's blob cache as well as a parallel put's staged ranges. A full sized partial left behind counts against the cache budget and is never completed, because the digest it would be sealed under is only written when every range has arrived.
+
 A call with an output location that returned prints a second line after the write-back:
 
 `letify: data wrote back <files> files <size> in <seconds> s (<rate> MiB/s), <files> files <size> during the call, <files> files <size> already on the client`
@@ -1356,7 +1370,7 @@ Sizes are in MiB with one decimal and `<budget>` in GiB with one decimal.
 
 `letify cache` shows the client digest cache, every local tree letify owns with its file count and size, and each provider's runtime file blob cache. `letify cache clear <target>` (or `letify cache clean <target>`) empties what is safe to clean. Credentials and configuration are never cleaned.
 
-`letify cache` first removes every digest cache entry whose file no longer exists, then prints the digest cache's entry count and how many were removed. It measures every local directory under `~/.letify`: `cache` (including digest cache, filesystem stores, and redirected CLI homes), `tools` (installed binaries), `tmp` (temporary session files), `ssh` (multiplexing control sockets), `runtime` (local provider workspace), and `accounts` (credentials, marked non-cleanable). For each provider in the configuration whose runtime disk persists between sessions, it starts a session on the provider's first instance, asks the worker with one `data_cache` request, and prints the alias, the blob count, the total size and the budget. A provider whose runtime disk does not persist is listed with `not kept between sessions` and no session is started. `--json` prints the same records under `digests`, `local` and `providers`.
+`letify cache` first removes every digest cache entry whose file no longer exists, then prints the digest cache's entry count and how many were removed. It measures every local directory under `~/.letify`: `cache` (including digest cache, filesystem stores, and redirected CLI homes), `tools` (installed binaries), `tmp` (temporary session files), `ssh` (multiplexing control sockets), `runtime` (local provider workspace), and `accounts` (credentials, marked non-cleanable). For each provider in the configuration whose runtime disk persists between sessions and already has a session running, it asks the worker with one `data_cache` request and prints the alias, the blob count, the total size and the budget. It starts a session only where starting one bills nothing, which is a provider whose runtime is this machine: a command that shows what is there must not create what is billed, and a configuration naming several providers would bill for all of them at once to print a listing. A provider with no session running is listed with `no session running`, and a provider whose runtime disk does not persist with `not kept between sessions`. Clearing a named provider still starts a session, because there the user named the provider and asked for the work. `--json` prints the same records under `digests`, `local` and `providers`.
 
 `letify cache clear <target>` (or `letify cache clean <target>`) clears caches:
 1. When `<target>` is a provider alias, it sends `data_cache` with `clear` set to that runtime. The worker removes committed blobs whose link count is 1, and the command prints `<alias>: removed <files> files <size>`.

{letify-2.5.2 → letify-2.5.4}/letify/__init__.py

@@ -66,7 +66,7 @@ distributed: Final = Host.distributed
 # The enum class stays importable from letify.declare.instance for letify's own use.
 del Host
 
-__version__ = "2.5.2"
+__version__ = "2.5.4"
 
 
 def fetch(tensor):  # type: ignore[no-untyped-def]

{letify-2.5.2 → letify-2.5.4}/letify/cli.py

@@ -322,8 +322,41 @@ def _setup_tool(args: argparse.Namespace) -> int:
     return 0
 
 
+def _running_runtime(let: Launcher, alias: str):
+    """A session this process already holds on that provider, or None.
+
+    Spec "The cache command": the listing asks only what is already running, because a
+    session it started would bill for the listing.
+    """
+    for runtime in let.pool.live:
+        if getattr(runtime.provider, "alias", None) == alias:
+            return runtime
+    return None
+
+
+def _data_cache_reply(let: Launcher, runtime, clear: bool) -> dict:
+    """One ``data_cache`` request to a session that is already up."""
+    from .store import pathdata
+
+    provider = runtime.provider
+    root = (runtime.workspace or provider.workspace_root).rstrip("/")
+    return runtime.request(
+        {
+            "op": "data_cache",
+            "dir": f"{root}/data/blobs",
+            "budget": pathdata.budget_bytes(provider),
+            "clear": clear,
+        },
+        timeout=600,
+    )
+
+
 def _data_cache_request(let: Launcher, alias: str, clear: bool) -> dict:
-    """Ask a session on the provider's first instance about its file blob cache."""
+    """Ask a session on the provider's first instance about its file blob cache.
+
+    This starts one when none is running, so only a named provider reaches it. Spec "The
+    cache command".
+    """
     from .declare import Env
     from .store import pathdata
 
@@ -462,7 +495,16 @@ def _cache(let: Launcher, args: argparse.Namespace) -> int:
             if not provider.persistent:
                 records.append({"alias": alias, "kept": False})
                 continue
-            result = _data_cache_request(let, alias, False)
+            # Spec "The cache command": a listing bills nothing, so it starts a session
+            # only where starting one is free, and otherwise asks one that is already up.
+            if provider.kind == "local":
+                result = _data_cache_request(let, alias, False)
+            else:
+                runtime = _running_runtime(let, alias)
+                if runtime is None:
+                    records.append({"alias": alias, "kept": True, "running": False})
+                    continue
+                result = _data_cache_reply(let, runtime, False)
         except LetifyError as exc:
             records.append({"alias": alias, "unavailable": str(exc)})
             continue
@@ -470,6 +512,7 @@ def _cache(let: Launcher, args: argparse.Namespace) -> int:
             {
                 "alias": alias,
                 "kept": True,
+                "running": True,
                 "files": result["files"],
                 "bytes": result["bytes"],
                 "budget": result["budget"],
@@ -498,6 +541,8 @@ def _cache(let: Launcher, args: argparse.Namespace) -> int:
             rows.append([row["alias"], f"unavailable: {row['unavailable']}", "", ""])
         elif not row["kept"]:
             rows.append([row["alias"], "not kept between sessions", "", ""])
+        elif not row.get("running"):
+            rows.append([row["alias"], "no session running", "", ""])
         else:
             rows.append(
                 [

{letify-2.5.2 → letify-2.5.4}/letify/protocol/worker.py

@@ -364,6 +364,8 @@ def _op_call(request):
         import shutil
         _data_unregister(state)
         _data_check_cache(data, state["placed"])
+        _DATA_FAILED.pop(data["dir"], None)
+        _DATA_FAILED.pop("", None)
         _DATA_POLL_SEEN.pop(data["dir"], None)
         _DATA_POLL_SENT.pop(data["dir"], None)
         shutil.rmtree(data["dir"], ignore_errors=True)
@@ -855,6 +857,40 @@ def _parallel_partial(root, digest):
     return "%s%s.parallel" % (_data_file(root, digest), _DATA_PARTIAL)
 
 
+def _op_data_drop(request):
+    """Remove the partials of a blob nobody is going to finish.
+
+    Spec "Data log line": a sender cancelled when the call ended leaves a partial that no
+    later request completes, because the digest it would be sealed under is written only
+    once every piece has arrived. A single put's partial is invisible to the budget scan,
+    so without this it stays for the life of the runtime.
+    """
+    digest = request["digest"]
+    final = _data_file(request["dir"], digest)
+    open_handle = _DATA_OPEN.pop(digest, None)
+    if open_handle is not None:
+        try:
+            open_handle[0].close()
+        except OSError:
+            pass
+    removed = 0
+    directory = os.path.dirname(final)
+    name = os.path.basename(final)
+    try:
+        entries = os.listdir(directory)
+    except OSError:
+        entries = []
+    for entry in entries:
+        if entry == name or not entry.startswith(name + "."):
+            continue
+        try:
+            os.remove(os.path.join(directory, entry))
+            removed += 1
+        except OSError:
+            pass
+    return {"ok": True, "value": removed}
+
+
 def _op_data_begin(request):
     """Make the file parallel senders will fill, of the size it will end up.
 
@@ -1013,6 +1049,9 @@ def _data_register(data):
         "linked": 0,
         "order": {},
         "arrived": 0,
+        # Spec "When a blob does not arrive": set when the local sender stops, so a
+        # reader fails at once with its reason rather than waiting out the timeout.
+        "failed": _DATA_FAILED.get(data["dir"]) or _DATA_FAILED.get(""),
         "wait": float(data.get("wait") or 600.0),
         "observe": bool(data.get("observe", True)),
         "prefetch": int(data.get("prefetch") or 64),
@@ -1204,6 +1243,32 @@ def _data_arrived(blobs, digest):
         _DATA_READY.notify_all()
 
 
+#: Reasons a sender stopped, by call directory, for a call that registers afterwards.
+_DATA_FAILED = {}
+
+
+def _op_data_failed(request):
+    """The local sender stopped, so the blobs it had left will never arrive.
+
+    Spec "When a blob does not arrive": every reader waiting on one of them is woken with
+    the reason, and a reader that arrives later fails on it at once, because waiting out
+    the per blob timeout costs that timeout once for every blob still queued.
+    """
+    reason = request.get("reason") or "the local process stopped sending"
+    where = request.get("dir") or ""
+    with _DATA_READY:
+        # Remembered by directory as well as set on the live call, because the sender can
+        # stop before the call request has been read, and the call then registers after
+        # this and would otherwise inherit nothing. Spec "When a blob does not arrive".
+        _DATA_FAILED[where] = reason
+        for state in _DATA_CALLS.values():
+            if where and state["dir"] != where:
+                continue
+            state["failed"] = reason
+        _DATA_READY.notify_all()
+    return {"ok": True, "value": True}
+
+
 def _data_want(digests, blocking=False):
     """Ask the local process for these blobs, in this order. Nothing replies to it."""
     if not digests:
@@ -1270,6 +1335,10 @@ def _data_wait(path):
         entry = state["pending"].get(path)
         if entry is None:
             return
+        if state["failed"]:
+            raise _DataFailure(
+                "%s was never sent: %s" % (os.path.basename(path), state["failed"])
+            )
         digest, size = entry
         index = state["order"].get(digest)
         if index is not None and index > state["arrived"] + _DATA_MISS_AHEAD:
@@ -1287,6 +1356,11 @@ def _data_wait(path):
     try:
         with _DATA_READY:
             while path in state["pending"] and state["dir"] in _DATA_CALLS:
+                if state["failed"]:
+                    raise _DataFailure(
+                        "%s was never sent: %s"
+                        % (os.path.basename(path), state["failed"])
+                    )
                 left = deadline - time.monotonic()
                 if left <= 0:
                     raise _DataFailure(
@@ -2254,6 +2328,8 @@ _OPS = {
     "data_have": _op_data_have,
     "data_put": _op_data_put,
     "data_begin": _op_data_begin,
+    "data_drop": _op_data_drop,
+    "data_failed": _op_data_failed,
     "data_put_at": _op_data_put_at,
     "data_seal": _op_data_seal,
     "data_pull": _op_data_pull,
@@ -2279,7 +2355,9 @@ _OPS = {
 }
 
 #: Answered by the frame reader at once, so they work while a call runs.
-_LIGHT = ("stat", "lease", "reduce_peers")
+# ``data_failed`` is light because it must not queue behind the call it is about: the call
+# is blocked on a blob the sender will never send, so a queued answer would deadlock.
+_LIGHT = ("stat", "lease", "reduce_peers", "data_failed")
 
 #: Answered by several replies, the last one marked. These send their own replies.
 _STREAMING = ("data_stream",)

{letify-2.5.2 → letify-2.5.4}/letify/runtime/session.py

@@ -419,7 +419,7 @@ class Runtime:
             needed = [d for d in plan.missing if d in ship_digests and d not in wave]
             if needed:
                 wave = needed + wave
-        stream = pathdata.Stream(self, blobs, plan)
+        stream = pathdata.Stream(self, blobs, plan, collector.call_dir)
         if not plan.missing:
             # Nothing to stream: the runtime holds every file, so the call places them all
             # before the body starts. It needs no manifest, no patch and no observation.

{letify-2.5.2 → letify-2.5.4}/letify/store/pathdata.py

@@ -23,7 +23,7 @@ from importlib import import_module
 from pathlib import Path
 from typing import TYPE_CHECKING, Any
 
-from ..errors import ConfigError, RemoteError, RuntimeFailure
+from ..errors import ConfigError, ProtocolError, RemoteError, RuntimeFailure
 
 if TYPE_CHECKING:
     from ..providers.base import Provider
@@ -370,6 +370,17 @@ class Collector:
         if resolved == project or resolved in project.parents:
             return None
         if not any(_inside(resolved, root) for root in roots):
+            if info is not None:
+                # Spec "Where the bytes come from": a file that exists here and cannot be
+                # sent would travel as a path the body cannot read, which fails on the
+                # runtime naming neither the cause nor this rule, or reads the wrong bytes
+                # where something does sit at that path.
+                raise ValueError(
+                    f"{resolved} is outside the roots letify may send data from. The "
+                    f"project root is {project}. Add its directory to [tool.letify] "
+                    f"data_roots in {project}/pyproject.toml, or move the file inside "
+                    f"the root."
+                )
             return None
         index = len(self.placed)
         placed = Placed(resolved, f"{self.call_dir}/{index}/{resolved.name}", is_dir)
@@ -597,10 +608,15 @@ class Stream:
     order after the call starts comes from the runtime rather than from the prediction.
     """
 
-    def __init__(self, runtime: Runtime, blobs: str, plan: Plan) -> None:
+    def __init__(self, runtime: Runtime, blobs: str, plan: Plan, call_dir: str = "") -> None:
         self.runtime = runtime
         self.blobs = blobs
         self.plan = plan
+        #: The call this stream feeds, so a stopped sender fails that call's readers only.
+        self.call_dir = call_dir
+        #: Digests whose send finished. ``_sent`` holds the ones it began, so a blob the
+        #: sender was carrying when the call ended is in one and not the other.
+        self.done: set[str] = set()
         self.before_files = self.before_bytes = 0
         self.during_files = self.during_bytes = 0
         self.before_seconds = self.during_seconds = 0.0
@@ -648,13 +664,35 @@ class Stream:
                     break
             # The pieces still in flight are answered before the stream is released.
             self._flush()
-        except Exception:
-            # The call owns the failure: a blob that never arrives fails the read that
-            # waits for it, with the message spec "When a blob does not arrive" names.
-            pass
+        except Exception as exc:
+            # Spec "When a blob does not arrive": the sender stopping leaves every blob
+            # still queued unsendable, so it is said once and the readers waiting on one
+            # are failed with the reason. Waiting it out costs the per blob timeout once
+            # for each, which is how 131 queued shards became a silent hour.
+            if not self._cancelled.is_set():
+                self._stopped(exc)
         finally:
             self.during_seconds = time.monotonic() - started
 
+    def _stopped(self, exc: BaseException) -> None:
+        """Say the sender stopped, and tell the runtime so its readers stop waiting."""
+        from .. import install
+
+        left = self.dropped
+        reason = f"{type(exc).__name__}: {exc}"
+        install.log(
+            f"data sending stopped after {len(self.done)} of "
+            f"{len(self.plan.missing)} files, {len(left)} not sent: {reason}"
+        )
+        try:
+            _worker(
+                self.runtime,
+                {"op": "data_failed", "dir": self.call_dir, "reason": reason},
+            )
+        except (RuntimeFailure, RemoteError, ProtocolError):
+            # The runtime is gone, which fails every reader on its own.
+            pass
+
     def _pump(self) -> bool:
         """Send the next blob of the queue. False once nothing is left to send."""
         self.reorder()
@@ -703,6 +741,22 @@ class Stream:
         if (thread is None or not thread.is_alive()) and self._pipeline is not None:
             self._pipeline.close()
             self._pipeline = None
+        self._drop_partials()
+
+    def _drop_partials(self) -> None:
+        """Remove the partial of every blob the sender was carrying. Spec "Data log line"."""
+        for digest in self.dropped:
+            try:
+                _worker(self.runtime, {"op": "data_drop", "dir": self.blobs, "digest": digest})
+            except (RuntimeFailure, RemoteError):
+                # The runtime is going away, which removes the partial with it.
+                return
+
+    @property
+    def dropped(self) -> list[str]:
+        """Blobs that were neither already on the runtime nor fully sent."""
+        with self._lock:
+            return [digest for digest in self.plan.missing if digest not in self.done]
 
     # -- the stream ---------------------------------------------------------------
 
@@ -760,6 +814,8 @@ class Stream:
         if not uploaded:
             return
         files, total = self.plan.weight(digest)
+        with self._lock:
+            self.done.add(digest)
         if self._thread is None:
             self.before_files += files
             self.before_bytes += total
@@ -780,19 +836,31 @@ class Stream:
 
         stats = stats or {}
         plan = self.plan
-        sent_files = self.before_files + self.during_files
         sent_bytes = self.before_bytes + self.during_bytes
         seconds = self.before_seconds + self.during_seconds
         rate = sent_bytes / (1 << 20) / seconds if seconds > 0 else 0.0
-        install.log(
+        # Spec "Data log line": counted from what the runtime answered data_have with,
+        # because subtracting what was sent reports a blob that was neither as having
+        # been there all along.
+        missing_weight = [plan.weight(digest) for digest in plan.missing]
+        already_files = plan.detected_files - sum(files for files, _b in missing_weight)
+        already_bytes = plan.detected_bytes - sum(total for _f, total in missing_weight)
+        dropped_weight = [plan.weight(digest) for digest in self.dropped]
+        dropped_files = sum(files for files, _b in dropped_weight)
+        dropped_bytes = sum(total for _f, total in dropped_weight)
+        line = (
             f"data {plan.detected_files} files {_mib(plan.detected_bytes)} detected, "
-            f"{plan.detected_files - sent_files} files "
-            f"{_mib(plan.detected_bytes - sent_bytes)} already on {plan.where}, "
+            f"{already_files} files {_mib(already_bytes)} already on {plan.where}, "
             f"sent {self.before_files} files {_mib(self.before_bytes)} before the call in "
             f"{self.before_seconds:.1f} s, {self.during_files} files "
             f"{_mib(self.during_bytes)} during it in {self.during_seconds:.1f} s "
             f"({rate:.1f} MiB/s), first access waited {float(stats.get('seconds', 0.0)):.1f} s"
         )
+        if dropped_files:
+            line += (
+                f", {dropped_files} files {_mib(dropped_bytes)} dropped when the call ended"
+            )
+        install.log(line)
         if stats.get("files"):
             install.log(
                 f"data waited for {stats['files']} files {_mib(int(stats.get('bytes', 0)))} "

{letify-2.5.2 → letify-2.5.4}/letify-core/Cargo.lock

@@ -138,7 +138,7 @@ dependencies = [
 
 [[package]]
 name = "letify-agent"
-version = "2.5.2"
+version = "2.5.4"
 dependencies = [
  "letify-wire",
  "libloading",
@@ -146,7 +146,7 @@ dependencies = [
 
 [[package]]
 name = "letify-driver"
-version = "2.5.2"
+version = "2.5.4"
 dependencies = [
  "letify-wire",
 ]
@@ -164,7 +164,7 @@ dependencies = [
 
 [[package]]
 name = "letify-wire"
-version = "2.5.2"
+version = "2.5.4"
 
 [[package]]
 name = "libc"

{letify-2.5.2 → letify-2.5.4}/letify-core/Cargo.toml

@@ -8,7 +8,7 @@ resolver = "3"
 members = ["letify-wire", "letify-driver", "letify-agent", "letify-quic"]
 
 [workspace.package]
-version = "2.5.2"
+version = "2.5.4"
 edition = "2024"
 license = "MIT"
 repository = "https://github.com/DarkPyonix/letify"

{letify-2.5.2 → letify-2.5.4}/letify-ext/package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "letify-status",
-  "version": "2.5.2",
+  "version": "2.5.4",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "letify-status",
-      "version": "2.5.2",
+      "version": "2.5.4",
       "license": "SEE LICENSE IN LICENSE",
       "devDependencies": {
         "@types/node": "^18.19.0",

{letify-2.5.2 → letify-2.5.4}/letify-ext/package.json

@@ -2,7 +2,7 @@
   "name": "letify-status",
   "displayName": "letify Status",
   "description": "letify quota and GPU activity in the status bar, read from the letify CLI.",
-  "version": "2.5.2",
+  "version": "2.5.4",
   "publisher": "darkpyonix",
   "license": "SEE LICENSE IN LICENSE",
   "repository": {

{letify-2.5.2 → letify-2.5.4}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "letify"
-version = "2.5.2"
+version = "2.5.4"
 description = "Declarations that become infrastructure."
 readme = "README.md"
 requires-python = ">=3.11"

{letify-2.5.2 → letify-2.5.4}/tests/test_cli.py

@@ -1606,3 +1606,61 @@ def test_stop_orphans_exits_one_when_a_delete_failed(isolated_home, capsys, monk
     )
     assert main(["stop", "--orphans", "-y"]) == 1
     assert "refused" in capsys.readouterr().out
+
+
+def test_the_cache_listing_starts_no_session(isolated_home, monkeypatch, capsys) -> None:
+    """Spec "The cache command": a command that shows what is there starts nothing.
+
+    KICT ran `letify cache` and it booted a Modal CPU session, which then failed its own
+    uv sync. A configuration naming several persistent providers would bill for all of
+    them to print a listing.
+    """
+    import letify
+    from letify.cli import main
+    from letify.runtime import pool
+
+    (isolated_home / ".letify" / "config.toml").write_text(
+        '[lab]\nkind = "shell"\naddress = "gpu.example.edu"\npersistent = true\n',
+        encoding="utf-8",
+    )
+    started: list[str] = []
+
+    real = pool.RuntimePool.acquire
+
+    def watch(self, instance, env, volumes=(), **kwargs):
+        provider = instance.provider
+        started.append(provider.alias)
+        assert provider.kind == "local", f"the listing started a billed session on {provider.alias}"
+        return real(self, instance, env, volumes, **kwargs)
+
+    monkeypatch.setattr(pool.RuntimePool, "acquire", watch)
+    assert main(["cache", "--json"]) == 0
+    assert all(alias == "local" for alias in started), started
+    out = capsys.readouterr().out
+    assert "providers" in out
+    assert letify  # the import is what makes the launcher reachable in this process
+
+
+def test_the_cache_listing_says_a_provider_has_no_session_running(
+    isolated_home, capsys
+) -> None:
+    """Spec "The cache command": a provider with no session is named, not skipped."""
+    import json
+
+    from letify.cli import main
+
+    (isolated_home / ".letify" / "config.toml").write_text(
+        '[lab]\nkind = "shell"\naddress = "gpu.example.edu"\npersistent = true\n',
+        encoding="utf-8",
+    )
+    assert main(["cache", "--json"]) == 0
+    record = json.loads(capsys.readouterr().out)
+    remote = [
+        row
+        for row in record["providers"]
+        if row.get("kept") and row["alias"] != "local"
+    ]
+    assert remote, "no billed persistent provider in the fixture, so this pins nothing"
+    for row in remote:
+        assert row.get("running") is False, row
+        assert "files" not in row, row

{letify-2.5.2 → letify-2.5.4}/tests/test_data.py

@@ -106,9 +106,12 @@ def test_a_path_that_does_not_exist_outside_the_roots_is_left_as_a_plain_path(
     assert not absent.exists()
 
 
-def test_a_path_outside_the_allowed_roots_is_left_as_a_plain_path(
-    let, cpu, project, tmp_path, capsys
-) -> None:
+def test_a_file_outside_the_allowed_roots_is_refused(let, cpu, project, tmp_path) -> None:
+    """Spec "Where the bytes come from": it used to travel as a plain path.
+
+    The body then read a path that is not on the runtime, which failed there as a
+    FileNotFoundError naming neither the cause nor the rule. KICT hit that twice.
+    """
     outside = tmp_path / "elsewhere.txt"
     outside.write_text("not project data", encoding="utf-8")
 
@@ -116,8 +119,9 @@ def test_a_path_outside_the_allowed_roots_is_left_as_a_plain_path(
     def where(path: Path) -> str:
         return str(path)
 
-    assert where(outside) == str(outside)
-    assert data_lines(capsys.readouterr().err) == []
+    with pytest.raises(ValueError) as failure:
+        where(outside)
+    assert "outside the roots letify may send data from" in str(failure.value)
 
 
 def test_a_data_root_named_in_pyproject_is_allowed(let, cpu, project, tmp_path) -> None:
@@ -780,3 +784,40 @@ def test_blobs_a_write_back_adds_keep_the_cache_within_its_budget(
     root = _blobs_root()
     assert sum(p.stat().st_size for p in root.glob("*/*")) <= 20 << 10
     assert len(evictions(capsys.readouterr().err)) == 1
+
+
+def test_a_file_outside_the_allowed_roots_is_refused_rather_than_sent_as_a_path(
+    let, cpu, project, tmp_path
+) -> None:
+    """Spec "Where the bytes come from": passing it on leaves the body reading nothing.
+
+    KICT hit this twice on Kaggle: a shard outside the project root travelled as a plain
+    path and the call died on the runtime with FileNotFoundError naming only the file.
+    """
+    outside = tmp_path / "outside" / "shard.pt"
+    outside.parent.mkdir(parents=True, exist_ok=True)
+    outside.write_bytes(b"x" * 4096)
+
+    @let.function(device=cpu, host="remote")
+    def read(path) -> int:
+        import os
+
+        return os.path.getsize(path)
+
+    with pytest.raises(ValueError) as failure:
+        read(path=outside)
+    message = str(failure.value)
+    assert str(outside) in message
+    assert "data_roots" in message
+    assert "outside the roots letify may send data from" in message
+
+
+def test_a_path_that_does_not_exist_here_still_travels_unchanged(let, cpu, project) -> None:
+    """Spec "Where the bytes come from": that is how a body is told where to write."""
+
+    @let.function(device=cpu, host="remote")
+    def name(path) -> str:
+        return str(path)
+
+    target = Path("/not/on/this/machine/out.pt")
+    assert name(path=target) == str(target)