letify 2.7.0__tar.gz → 2.7.1__tar.gz

{letify-2.7.0 → letify-2.7.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: letify
-Version: 2.7.0
+Version: 2.7.1
 Summary: Declarations that become infrastructure.
 Project-URL: Homepage, https://github.com/DarkPyonix/letify
 Project-URL: Documentation, https://github.com/DarkPyonix/letify/tree/main/docs

{letify-2.7.0 → letify-2.7.1}/docs/SPEC.md

@@ -1428,7 +1428,7 @@ A call with an output location that returned prints a second line after the writ
 
 `letify: data wrote back <files> files <size> in <seconds> s (<rate> MiB/s), <files> files <size> during the call, <files> files <size> already on the client`
 
-The first count and size are the total written back, the background write-back of Writing back and the final write-back combined, and the rate covers both. `during the call` is the part of that total the background write-back already delivered while the body was still running, counted as the call ran rather than only once it returned, so a long call's log line still shows separately how much of the output had already made it back before the call ended. `already on the client` is unchanged: files the local digest cache already held, fetched by neither path.
+The first count and size are the total written back, the background write-back of Writing back and the final write-back combined, and the rate covers both. The rate divides those bytes by the seconds spent carrying them, the background write-back's time inside its transfers plus the final write-back's own, and not by how long the background write-back was alive: it polls for the whole call, so dividing by its lifetime turned 1212.4 MiB carried in 19.5 s, about 62 MiB/s, into a reported 2.4 MiB/s. `<seconds>` is the final write-back's own time, as before. `during the call` is the part of that total the background write-back already delivered while the body was still running, counted as the call ran rather than only once it returned, so a long call's log line still shows separately how much of the output had already made it back before the call ended. `already on the client` is unchanged: files the local digest cache already held, fetched by neither path.
 
 #### Writing back <!-- id: project-data-write-back -->
 
@@ -1449,6 +1449,10 @@ A path written more than once during a call, such as a checkpoint overwritten on
 
 The final write-back, after the body returns, is unchanged in what it detects: the worker walks each output's runtime path, as it always has, and renames every changed file into the file blob cache as before. What changed is that it is now incremental. Before fetching a listed file, the local process checks it against what the background write-back already sent for that relative path during the call: a file whose listed digest equals the digest the background path already delivered is skipped outright, counted as already sent rather than fetched again, because fetching it a second time would be a duplicate transfer of bytes the local disk already has. Only a file the background path never sent, or sent an older digest for, travels on the final write-back's own `data_stream` requests.
 
+The final write-back starts only once the background write-back has stopped, including a file it was in the middle of carrying when the call returned, because the two decide from the same record of what was sent. The background write-back checks for the stop between files, so waiting for it costs at most the rest of one file. Starting early let the final write-back find a 402.2 MiB file already complete on the local disk while the background path had not yet recorded sending it, so the line counted it both as delivered during the call and as already on the client, and the final write-back then waited on the same output lock for the background transfer to finish, which is why that call reported 209.1 s.
+
+The progress line counts only the bytes this write-back fetches. A file skipped because the background path or the local digest cache already has it is left out of the line's total rather than added to its progress in an instant, which drew rates such as 31830.2 MiB/s.
+
 On the runtime, after the body returned and before the call directory is removed, the worker walks each output's runtime path as Which paths are data walks a directory, without following symbolic links. A file is unchanged when it is the hard link or copy the worker placed and its inode, size and modification time in nanoseconds equal those recorded when it was placed; it is not read. Any other regular file, a file Copy on write converted included, is hashed with the file digest, and a file whose digest equals the digest placed at that relative path is unchanged too. Each changed file is renamed into the file blob cache and made read-only, or dropped when the cache already holds its digest with its size. The worker keeps the list of relative path, digest and size per output under the call id until the local process asks for it with one `data_written` request, which removes it. The background poll above and the final `data_written` request keep separate bookkeeping on the worker, so one does not consume what the other needs.
 
 When the runtime path of an output that did not exist is a regular file, the local path receives that file. When it is a directory, the local path becomes a directory holding the changed files. When nothing was created there, nothing is written.

{letify-2.7.0 → letify-2.7.1}/letify/__init__.py

@@ -66,7 +66,7 @@ distributed: Final = Host.distributed
 # The enum class stays importable from letify.declare.instance for letify's own use.
 del Host
 
-__version__ = "2.7.0"
+__version__ = "2.7.1"
 
 
 def fetch(tensor):  # type: ignore[no-untyped-def]

{letify-2.7.0 → letify-2.7.1}/letify/store/pathdata.py

@@ -1435,6 +1435,9 @@ class WriteBack:
         self.blobs = blobs
         self.files = self.bytes = 0
         self.seconds = 0.0
+        #: Seconds spent inside transfers, which the wrote back rate divides by. Spec
+        #: "Data log line": ``seconds`` is how long this ran, which is the whole call.
+        self.carrying_seconds = 0.0
         #: The digest this path was last sent with, which keeps the final write-back from
         #: sending it again and a repeated poll from sending the same version twice.
         self.sent: dict[Path, str] = {}
@@ -1488,11 +1491,16 @@ class WriteBack:
                 if self.sent.get(target) == digest:
                     continue
                 with _output_lock(placed.local):
+                    began = time.monotonic()
                     _get(self.runtime, self.blobs, digest, target, int(size), _Uploaded(1, 0))
                     self.collector.cache.record(target, digest)
-                self.sent[target] = digest
-                self.files += 1
-                self.bytes += int(size)
+                    # Recorded before the lock is released, so the final write-back, which
+                    # decides under the same lock, never sees the file on disk unrecorded.
+                    # Spec "Writing back".
+                    self.carrying_seconds += time.monotonic() - began
+                    self.sent[target] = digest
+                    self.files += 1
+                    self.bytes += int(size)
 
     def cancel(self) -> None:
         """Stop polling, which the call's end does before the final write-back runs."""
@@ -1526,22 +1534,38 @@ def write_back(
             target = placed.local / rel if rel else placed.local
             files.append((target, str(digest), int(size)))
         plan.append((placed, files))
-    total = sum(size for _placed, files in plan for _t, _d, size in files)
-    meter = _Downloaded(sum(len(files) for _p, files in plan), total)
     sent_files = sent_bytes = kept_files = kept_bytes = 0
-    already = dict(background.sent) if background is not None else {}
+
+    def delivered(target: Path, digest: str) -> bool:
+        """Whether the background write-back already brought this very version.
+
+        Read under the output lock, never from a copy taken earlier: the background path
+        may still be finishing a file when the call returns, and it records the file
+        under the same lock. Spec "Writing back".
+        """
+        return background is not None and background.sent.get(target) == digest
+
+    # Decided before anything is fetched, so the progress line covers only the bytes this
+    # write-back carries. Spec "Writing back": a skip added its size at once.
+    fetch: list[tuple[Placed, list[tuple[Path, str, int]]]] = []
     for placed, files in plan:
+        wanted = []
         with _output_lock(placed.local):
             for target, digest, size in files:
-                if already.get(target) == digest:
-                    # The background write-back delivered this very version during the
-                    # call, so fetching it now would be a duplicate transfer.
-                    meter.advance(size)
+                if delivered(target, digest):
                     continue
                 if _same(cache, target, digest):
                     kept_files += 1
                     kept_bytes += size
-                    meter.advance(size)
+                    continue
+                wanted.append((target, digest, size))
+        fetch.append((placed, wanted))
+    total = sum(size for _placed, files in fetch for _t, _d, size in files)
+    meter = _Downloaded(sum(len(files) for _p, files in fetch), total)
+    for placed, files in fetch:
+        with _output_lock(placed.local):
+            for target, digest, size in files:
+                if delivered(target, digest):
                     continue
                 _get(runtime, blobs, digest, target, size, meter)
                 cache.record(target, digest)
@@ -1552,7 +1576,7 @@ def write_back(
     cache.save()
     during_files = background.files if background is not None else 0
     during_bytes = background.bytes if background is not None else 0
-    during_seconds = background.seconds if background is not None else 0.0
+    during_seconds = background.carrying_seconds if background is not None else 0.0
     elapsed = time.monotonic() - started
     total_files = sent_files + during_files
     total_bytes = sent_bytes + during_bytes

{letify-2.7.0 → letify-2.7.1}/letify-core/Cargo.lock

@@ -138,7 +138,7 @@ dependencies = [
 
 [[package]]
 name = "letify-agent"
-version = "2.7.0"
+version = "2.7.1"
 dependencies = [
  "letify-wire",
  "libloading",
@@ -146,7 +146,7 @@ dependencies = [
 
 [[package]]
 name = "letify-driver"
-version = "2.7.0"
+version = "2.7.1"
 dependencies = [
  "letify-wire",
 ]
@@ -164,7 +164,7 @@ dependencies = [
 
 [[package]]
 name = "letify-wire"
-version = "2.7.0"
+version = "2.7.1"
 
 [[package]]
 name = "libc"

{letify-2.7.0 → letify-2.7.1}/letify-core/Cargo.toml

@@ -8,7 +8,7 @@ resolver = "3"
 members = ["letify-wire", "letify-driver", "letify-agent", "letify-quic"]
 
 [workspace.package]
-version = "2.7.0"
+version = "2.7.1"
 edition = "2024"
 license = "MIT"
 repository = "https://github.com/DarkPyonix/letify"

{letify-2.7.0 → letify-2.7.1}/letify-ext/package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "letify-status",
-  "version": "2.7.0",
+  "version": "2.7.1",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "letify-status",
-      "version": "2.7.0",
+      "version": "2.7.1",
       "license": "SEE LICENSE IN LICENSE",
       "devDependencies": {
         "@types/node": "^18.19.0",

{letify-2.7.0 → letify-2.7.1}/letify-ext/package.json

@@ -2,7 +2,7 @@
   "name": "letify-status",
   "displayName": "letify Status",
   "description": "letify quota and GPU activity in the status bar, read from the letify CLI.",
-  "version": "2.7.0",
+  "version": "2.7.1",
   "publisher": "darkpyonix",
   "license": "SEE LICENSE IN LICENSE",
   "repository": {

{letify-2.7.0 → letify-2.7.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "letify"
-version = "2.7.0"
+version = "2.7.1"
 description = "Declarations that become infrastructure."
 readme = "README.md"
 requires-python = ">=3.11"

{letify-2.7.0 → letify-2.7.1}/tests/test_data_streaming.py

@@ -21,6 +21,7 @@ import tempfile
 import threading
 import time
 from pathlib import Path
+from typing import ClassVar
 
 import pytest
 
@@ -1820,3 +1821,116 @@ def test_a_large_blob_beside_the_call_draws_the_progress_line(
 
     assert read_it(path=big) == 70 << 20
     assert "uploading" in capsys.readouterr().err
+
+
+def test_the_wrote_back_rate_counts_the_seconds_spent_carrying(
+    launcher_from, project, capsys
+) -> None:
+    """Spec "Data log line": not how long the background write-back was alive.
+
+    It polls for the whole call, so dividing by its lifetime turned 1212.4 MiB carried in
+    19.5 s, about 62 MiB/s, into a reported 2.4 MiB/s on KICT's Modal run.
+    """
+    let = streaming(launcher_from)
+    root = project / "rate"
+    root.mkdir()
+
+    @let.function(device=let.providers.lab.CPU, host=letify.remote)
+    def train(directory: Path, held: float) -> str:
+        import time as clock
+
+        (directory / "checkpoint.bin").write_bytes(b"\x05" * (8 << 20))
+        clock.sleep(held)
+        return "done"
+
+    assert train(root, HELD_S) == "done"
+    line = wrote_back_line(capsys.readouterr().err)
+    size = re.search(r"wrote back \d+ files ([\d.]+) MiB", line)
+    rate = re.search(r"\(([\d.]+) MiB/s\)", line)
+    assert size and rate, line
+    # 8 MiB over a local channel takes a fraction of a second; the call held for HELD_S.
+    implied = float(size.group(1)) / max(float(rate.group(1)), 1e-9)
+    assert implied < HELD_S / 2, f"the rate implies {implied:.1f} s of carrying: {line}"
+
+
+def test_a_file_the_background_is_still_carrying_is_not_counted_twice(
+    launcher_from, project, capsys, monkeypatch
+) -> None:
+    """Spec "Writing back": the final write-back starts only once the background stopped.
+
+    KICT's line read "1 files 402.2 MiB during the call, 1 files 402.2 MiB already on the
+    client": the final write-back began while the background path was still carrying that
+    file, found it complete on the local disk, and counted it a second time. The wait
+    for the background used to give up after 60 s, so the test takes the wait away.
+    """
+    from letify.store import pathdata
+
+    # What a wait that gave up looks like: the stop is asked for and nothing waits for it.
+    monkeypatch.setattr(pathdata.WriteBack, "cancel", lambda self: self._cancelled.set())
+    real_get = pathdata._get
+
+    def slow_in_the_background(runtime, blobs, digest, target, size, meter):
+        if threading.current_thread().name == "letify-write-back":
+            time.sleep(3.0)
+        return real_get(runtime, blobs, digest, target, size, meter)
+
+    monkeypatch.setattr(pathdata, "_get", slow_in_the_background)
+    let = streaming(launcher_from)
+    root = project / "twice"
+    root.mkdir()
+
+    @let.function(device=let.providers.lab.CPU, host=letify.remote)
+    def train(directory: Path) -> str:
+        import time as clock
+
+        (directory / "checkpoint.bin").write_bytes(b"\x06" * (1 << 20))
+        # Long enough for a poll to find the file complete, short enough that the call
+        # returns while the background is still carrying it.
+        clock.sleep(1.5)
+        return "done"
+
+    assert train(root) == "done"
+    line = wrote_back_line(capsys.readouterr().err)
+    during = re.search(r"(\d+) files [\d.]+ MiB during the call", line)
+    kept = re.search(r"(\d+) files [\d.]+ MiB already on the client", line)
+    assert during and kept, line
+    assert not (int(during.group(1)) == 1 and int(kept.group(1)) == 1), line
+
+
+def test_skipped_files_do_not_enter_the_progress_line(project, monkeypatch) -> None:
+    """Spec "Writing back": a skip added its size at once and drew 31830.2 MiB/s."""
+    from letify.store import pathdata
+
+    totals: list[int] = []
+    real = pathdata._Downloaded
+
+    class Watched(real):  # type: ignore[misc, valid-type]
+        def __init__(self, files: int, total: int) -> None:
+            totals.append(total)
+            super().__init__(files, total)
+
+    monkeypatch.setattr(pathdata, "_Downloaded", Watched)
+    kept = project / "kept.bin"
+    kept.write_bytes(b"k" * 1000)
+
+    class Runtime:
+        name = "rt"
+
+        def request(self, payload, timeout=None):
+            raise AssertionError(f"nothing should be fetched: {payload.get('op')}")
+
+    class Placed:
+        local = project
+        runtime = "/rt/out"
+
+    class Collector:
+        call_dir = "/rt/call"
+        outputs: ClassVar[list] = [Placed]
+        cache = pathdata.DigestCache(project / "digests.json")
+
+    digest = Collector.cache.digest(kept)
+    monkeypatch.setattr(
+        pathdata, "_worker", lambda runtime, payload: {"/rt/out": [["kept.bin", digest, 1000]]}
+    )
+    pathdata.write_back(Runtime(), Collector(), "/rt/blobs")
+    assert totals == [0], totals