letify 2.5.3__tar.gz → 2.6.0__tar.gz

{letify-2.5.3 → letify-2.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: letify
-Version: 2.5.3
+Version: 2.6.0
 Summary: Declarations that become infrastructure.
 Project-URL: Homepage, https://github.com/DarkPyonix/letify
 Project-URL: Documentation, https://github.com/DarkPyonix/letify/tree/main/docs

{letify-2.5.3 → letify-2.6.0}/docs/SPEC.md

@@ -316,7 +316,11 @@ Separately, and without needing a threshold at all, a refusal says how long ago
 
 The record therefore notes the process as well as the ref, `<username>/<slug>` and the pid separated by a tab, so a notebook a live process is still using is never deleted. A ref whose pid is still running on this machine is reported as in use and skipped. A ref with no pid, written by a letify older than this, cannot be checked and counts as an orphan, because the record is per machine and an older letify is not running.
 
-`letify stop` with no `--orphans` deletes nothing and says so, because a bare `stop` reads as ending everything. The command lists what it will delete and asks before deleting, `-y` answers yes, and `--alias` limits it to one account. Each ref is deleted with the same CLI call `stop` uses and dropped from the record only when the delete succeeds, so a failure leaves the ref to be tried again rather than losing track of it. The exit status is 1 when any delete failed.
+`letify stop` with no `--orphans` deletes nothing and says why rather than only that: a session whose
+creating process is still running ends when that process ends, because taking the session from under
+it leaves it calling a runtime that is gone. So the line names the way out, `send SIGTERM to the
+process that started it, which letify handles by closing its sessions`, and then what `--orphans` is
+for. A bare `stop` otherwise reads as ending everything. The command lists what it will delete and asks before deleting, `-y` answers yes, and `--alias` limits it to one account. Each ref is deleted with the same CLI call `stop` uses and dropped from the record only when the delete succeeds, so a failure leaves the ref to be tried again rather than losing track of it. The exit status is 1 when any delete failed.
 
 **Deleting the notebook, through the CLI only.** `stop` deletes the notebook with the official CLI, `kaggle kernels delete -y <username>/<slug>`, authenticated by the account's API token in `KAGGLE_API_TOKEN`; deletion of a notebook created for a cookie-authenticated interactive session is confirmed on the live account. The cookie is never asked, not even as a fallback, because deleting a notebook is not part of running or ending the interactive session. The call never fails the runtime, because the accelerator quota is already released by cancelling the run before it is tried. When it cannot run at all, for want of a token, of uv, or of the slug Kaggle did not return, or when the command itself fails, `stop` prints one warning line naming the notebook by its slug or id, never its cookie, so the user can delete it by hand on kaggle.com. The chain that starts the session, all on `https://www.kaggle.com/api/i/`, authenticated by the cookie:
 
@@ -1125,6 +1129,25 @@ So a blob above `transfer_parallel_mib`, 64 MiB by default, on a link whose meas
 
 The round trip is a condition because the gain depends on it. A single stream is window limited only when the window takes a long time to fill, so on a short link it already carries everything the link will carry and the extra connections are pure cost. Measured to a lab server over forward SSH at 0.25 ms, 256 MiB went at 36.1 MiB/s on one stream and 33.5 MiB/s on four, about 7% worse, which is the cost of opening the channels and their worker processes. The link reports what the probe measured, and a link with no measurement, such as a provider fallback, is treated as short and keeps one stream. A provider whose channel is not a connection it can open more of keeps one stream too: Modal's sandbox reads and writes through its own API, and a Kaggle session that never punched is carried by a kernel bridge, so neither has a second connection to offer. A Kaggle session that did punch is reached over SSH and does: each transfer stream asks the link for its own SSH command, which lands on a local forwarding port of its own and so makes the link punch again.
 
+The connections carry different blobs at the same time as well as different ranges of one, because a
+call whose blobs are each under the split threshold would otherwise send them one after another and
+reach one connection's ceiling however many files it has. Measured on a Kaggle runtime at a 190 ms
+round trip, 20 files of 10 MB each went at 5.0 MiB/s on one connection, which is the 0.9 MiB window
+over that round trip. So the sender runs `transfer_streams` senders over the pool rather than one, and
+the condition for using more than one connection is the link's round trip alone: the size threshold
+decides whether one blob is split, not whether blobs may overlap. A blob the worker asked for by
+`data_want` still goes next, because the queue is reordered before a sender takes from it.
+
+The connections are opened once per call and reused by every blob of it, not opened and closed per
+blob. A call that sends a hundred blobs over a punched link paid a hundred rounds of punching when
+each blob opened its own, which is both the cost the estimate below weighs and a hundred chances for
+a punch to be refused. They are closed when the call's transfer ends.
+
+Reuse is what makes the cost worth paying, so the estimate charges it only while the connections are
+not yet open. The first blob large enough to want them pays the punch and weighs it; every blob after
+it finds them open and weighs nothing, so a blob under the size threshold still goes on one
+connection, but a call's second large blob is split whenever splitting is faster at all.
+
 A connection that costs something to open has to pay for itself, so a provider reports `transfer_connection_cost_s`, the seconds one more connection takes. It is 0 for a provider that dials an address, and about 10 for one reached by a hole punch, because that is what a punch measures. The streams are opened only when the transfer is expected to finish sooner with them: the estimate uses the probe's upload rate, assumes the streams scale and that their connections are opened at the same time, and keeps one stream when `size / (rate x streams) + cost` is not less than `size / rate`. A measured case: a Kaggle runtime at a 44.7 ms round trip carried 128 MiB at 21.0 MiB/s on one stream and 9.5 MiB/s on four, 2.2 times worse, because three more punches cost more than the transfer saved. The same account at a 193 ms round trip carried 1.57 MiB/s on one raw SSH connection and 6.10 MiB/s on four, so the gain is real where the link is slow enough and the blob large enough for it to be worth 10 s.
 
 The connections have to be real ones. Every SSH command letify builds carries `ControlMaster=auto`, which multiplexes further sessions onto the first connection, so opening more channels that way would add streams inside one TCP connection and gain nothing. A transfer channel therefore sets `ControlPath=none` and `ControlMaster=no`, and a punched link redials for each, so each transfer channel is its own kernel connection.
@@ -1239,6 +1262,7 @@ A `stat` cannot take that snapshot first, because it is on a data loader's hot p
 #### When a blob does not arrive <!-- id: project-data-streaming-failures -->
 
 - **A blob that never arrives.** A `data_want` unanswered for `data_wait_timeout` seconds on the account, 600 by default, fails that open with `RuntimeFailure` naming the relative path and the digest. The call is not killed, because the body may handle it; the failure is infrastructure, so a retry of the call is allowed.
+- **A sender that stops.** The background sender raising leaves every blob still queued unsendable. letify says so once, `letify: data sending stopped after <n> of <n> files, <n> not sent: <error>`, and tells the runtime, which wakes every reader waiting on one of those blobs with `RuntimeFailure` naming the file and that reason, and fails a reader that arrives later at once. Waiting it out instead costs `data_wait_timeout` once for every blob still queued, which is how a hundred queued shards become an hour of silence.
 - **A client that disconnects mid stream.** The session ends as any lost channel ends it. Every blocked reader is woken with `RuntimeFailure`, the call ends with it, and no partial file is ever placed or read. A `.partial` file is removed when the worker next starts.
 - **A call that finishes before the background transfer.** The transfer is cancelled, both the channel sender and any bucket upload not yet started. A blob already committed to the cache stays and counts as held for the next call, and a partial one is removed. The write-back and the `data_evict` request follow the cancellation, in that order.
 - **A bucket-backed ephemeral provider.** A pull that fails is retried twice with the same downscoped headers, then a third time with headers the local process derives again, and a pull that still fails fails the open with `RuntimeFailure`. A `data_want` for a digest not yet uploaded moves that object to the front of the upload queue, and the worker pulls it when the local process sends its `data_pull`.

{letify-2.5.3 → letify-2.6.0}/letify/__init__.py

@@ -66,7 +66,7 @@ distributed: Final = Host.distributed
 # The enum class stays importable from letify.declare.instance for letify's own use.
 del Host
 
-__version__ = "2.5.3"
+__version__ = "2.6.0"
 
 
 def fetch(tensor):  # type: ignore[no-untyped-def]

{letify-2.5.3 → letify-2.6.0}/letify/cli.py

@@ -755,8 +755,15 @@ def _dispatch(args: argparse.Namespace) -> int:
 
     if args.command == "stop":
         if not args.orphans:
-            # A bare `stop` reads as ending everything, so it ends nothing.
-            print("nothing was stopped. Pass --orphans to delete what a dead process left")
+            # Spec "Kaggle", stopping what a dead process left behind: a bare stop reads as
+            # ending everything, so it ends nothing, and says why rather than only that.
+            print(
+                "nothing was stopped. A session whose process is still running ends when "
+                "that process ends, because taking it away leaves that process calling a "
+                "runtime that is gone: send SIGTERM to the process that started it, which "
+                "letify handles by closing its sessions. Pass --orphans to delete what a "
+                "dead process left."
+            )
             return 0
         rows = let.orphans(args.alias)
         found = [(row["alias"], ref) for row in rows for ref in row.get("orphans", ())]

{letify-2.5.3 → letify-2.6.0}/letify/protocol/worker.py

@@ -364,6 +364,8 @@ def _op_call(request):
         import shutil
         _data_unregister(state)
         _data_check_cache(data, state["placed"])
+        _DATA_FAILED.pop(data["dir"], None)
+        _DATA_FAILED.pop("", None)
         _DATA_POLL_SEEN.pop(data["dir"], None)
         _DATA_POLL_SENT.pop(data["dir"], None)
         shutil.rmtree(data["dir"], ignore_errors=True)
@@ -1047,6 +1049,9 @@ def _data_register(data):
         "linked": 0,
         "order": {},
         "arrived": 0,
+        # Spec "When a blob does not arrive": set when the local sender stops, so a
+        # reader fails at once with its reason rather than waiting out the timeout.
+        "failed": _DATA_FAILED.get(data["dir"]) or _DATA_FAILED.get(""),
         "wait": float(data.get("wait") or 600.0),
         "observe": bool(data.get("observe", True)),
         "prefetch": int(data.get("prefetch") or 64),
@@ -1238,6 +1243,32 @@ def _data_arrived(blobs, digest):
         _DATA_READY.notify_all()
 
 
+#: Reasons a sender stopped, by call directory, for a call that registers afterwards.
+_DATA_FAILED = {}
+
+
+def _op_data_failed(request):
+    """The local sender stopped, so the blobs it had left will never arrive.
+
+    Spec "When a blob does not arrive": every reader waiting on one of them is woken with
+    the reason, and a reader that arrives later fails on it at once, because waiting out
+    the per blob timeout costs that timeout once for every blob still queued.
+    """
+    reason = request.get("reason") or "the local process stopped sending"
+    where = request.get("dir") or ""
+    with _DATA_READY:
+        # Remembered by directory as well as set on the live call, because the sender can
+        # stop before the call request has been read, and the call then registers after
+        # this and would otherwise inherit nothing. Spec "When a blob does not arrive".
+        _DATA_FAILED[where] = reason
+        for state in _DATA_CALLS.values():
+            if where and state["dir"] != where:
+                continue
+            state["failed"] = reason
+        _DATA_READY.notify_all()
+    return {"ok": True, "value": True}
+
+
 def _data_want(digests, blocking=False):
     """Ask the local process for these blobs, in this order. Nothing replies to it."""
     if not digests:
@@ -1304,6 +1335,10 @@ def _data_wait(path):
         entry = state["pending"].get(path)
         if entry is None:
             return
+        if state["failed"]:
+            raise _DataFailure(
+                "%s was never sent: %s" % (os.path.basename(path), state["failed"])
+            )
         digest, size = entry
         index = state["order"].get(digest)
         if index is not None and index > state["arrived"] + _DATA_MISS_AHEAD:
@@ -1321,6 +1356,11 @@ def _data_wait(path):
     try:
         with _DATA_READY:
             while path in state["pending"] and state["dir"] in _DATA_CALLS:
+                if state["failed"]:
+                    raise _DataFailure(
+                        "%s was never sent: %s"
+                        % (os.path.basename(path), state["failed"])
+                    )
                 left = deadline - time.monotonic()
                 if left <= 0:
                     raise _DataFailure(
@@ -2289,6 +2329,7 @@ _OPS = {
     "data_put": _op_data_put,
     "data_begin": _op_data_begin,
     "data_drop": _op_data_drop,
+    "data_failed": _op_data_failed,
     "data_put_at": _op_data_put_at,
     "data_seal": _op_data_seal,
     "data_pull": _op_data_pull,
@@ -2314,7 +2355,9 @@ _OPS = {
 }
 
 #: Answered by the frame reader at once, so they work while a call runs.
-_LIGHT = ("stat", "lease", "reduce_peers")
+# ``data_failed`` is light because it must not queue behind the call it is about: the call
+# is blocked on a blob the sender will never send, so a queued answer would deadlock.
+_LIGHT = ("stat", "lease", "reduce_peers", "data_failed")
 
 #: Answered by several replies, the last one marked. These send their own replies.
 _STREAMING = ("data_stream",)

{letify-2.5.3 → letify-2.6.0}/letify/runtime/session.py

@@ -419,7 +419,7 @@ class Runtime:
             needed = [d for d in plan.missing if d in ship_digests and d not in wave]
             if needed:
                 wave = needed + wave
-        stream = pathdata.Stream(self, blobs, plan)
+        stream = pathdata.Stream(self, blobs, plan, collector.call_dir)
         if not plan.missing:
             # Nothing to stream: the runtime holds every file, so the call places them all
             # before the body starts. It needs no manifest, no patch and no observation.