letify 2.6.4__tar.gz → 2.6.6__tar.gz

{letify-2.6.4 → letify-2.6.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: letify
-Version: 2.6.4
+Version: 2.6.6
 Summary: Declarations that become infrastructure.
 Project-URL: Homepage, https://github.com/DarkPyonix/letify
 Project-URL: Documentation, https://github.com/DarkPyonix/letify/tree/main/docs

{letify-2.6.4 → letify-2.6.6}/docs/SPEC.md

@@ -1063,6 +1063,16 @@ less time than the build did. The archive is smaller than the directory, so the
 overstates the transfer and errs toward building, which is the cheaper of the two to get wrong. The
 ref records the build seconds beside the archive.
 
+With no measured rate for the link, letify packs only when that directory is at most 1 GiB, and says
+why when it declines. Packing moves the archive as one payload that the worker and the client each
+hold whole, so a tree of several gigabytes is held twice in memory with nothing reporting progress:
+measured on a Modal A10, a 7.2 GB environment sat in that step for over 30 minutes while the session
+billed. Packing unconditionally was right while only an ephemeral provider packed, where the
+alternative was a certain full rebuild; a provider that keeps its workspace root and builds its venv
+elsewhere reaches the same branch, and there the alternative is a sync that may take two minutes.
+Making the pack stream rather than hold the archive whole is the fix that removes this limit, and
+until then the limit is what keeps the step from costing more than it saves.
+
 A later session restores only when the archive's size over the link measured for that session would
 take less than the recorded build. Otherwise it syncs, and says why:
 `letify: <runtime>: syncing rather than restoring, because 3.0 GB over 13.6 MiB/s is 210 s against a
@@ -1076,7 +1086,9 @@ the kind this rule exists to catch, and the session that syncs instead both reco
 decides afresh whether the archive is worth keeping. The cost of that is one sync; the cost of
 trusting it is the transfer this rule was measured to avoid.
 
-Skipping the pack is said too, because it is a decision letify made rather than a thing that did not
+Packing is said, because it is a step that can take minutes and had nothing reporting it:
+`letify: <runtime>: caching the environment, 2.1 GB`. Skipping it is said too, because it is a
+decision letify made rather than a thing that did not
 happen: `letify: <runtime>: not caching the environment, because 9.4 GB over 13.6 MiB/s is 659 s
 against a 34 s build`.
 
@@ -1353,6 +1365,14 @@ A call that detected data prints one line on standard error:
 
 The line is printed when the call has at least one input file, after the call's outcome is known, because what was sent during the call is not known before then. Sizes are in MiB with one decimal. `<the bucket>` is named when the bytes come from the bucket. `before the call` counts the first wave of The send order and the first wave, `during it` counts what the background sender delivered while the call ran, and the rate covers both. `first access waited` is the wall clock time in which at least one reader was blocked on a blob that had not arrived, and it is `0.0` when the send order was right.
 
+The bytes of a blob still on the way are reported too, because the rate counts only blobs that
+completed and a single large blob in flight therefore reads as `0.0 MiB/s` however fast it is moving.
+A line whose dropped clause names files ends with how far they got,
+`, 3 files 14985.8 MiB dropped when the call ended, 4512.0 MiB of them had arrived`, and the
+background sender draws the same `uploading` progress line the first wave draws, which it did not
+before: a blob sent beside the call reported nothing at all, so a transfer in progress and a transfer
+that had never started looked the same. Both cost a reader an afternoon.
+
 `already on` counts the blobs the runtime answered `data_have` with, not what is left after subtracting what was sent. The two differ whenever a blob was neither already there nor fully sent, and subtracting reports such a blob as having been there all along, which is the opposite of what happened. A blob the background sender was still carrying when the call returned is one of those, so the line ends with `, <files> files <size> dropped when the call ended` whenever there is one, and omits that clause when there is none. A dropped blob is not a failure: the call is over and its bytes are no longer needed, which is why the sender is cancelled. It is reported because a line that accounts for every detected file except the dropped ones cannot be read.
 
 Cancelling the sender removes the partial file of a blob it was writing, in the runtime's blob cache as well as a parallel put's staged ranges. A full sized partial left behind counts against the cache budget and is never completed, because the digest it would be sealed under is only written when every range has arrived.

{letify-2.6.4 → letify-2.6.6}/letify/__init__.py

@@ -66,7 +66,7 @@ distributed: Final = Host.distributed
 # The enum class stays importable from letify.declare.instance for letify's own use.
 del Host
 
-__version__ = "2.6.4"
+__version__ = "2.6.6"
 
 
 def fetch(tensor):  # type: ignore[no-untyped-def]

{letify-2.6.4 → letify-2.6.6}/letify/runtime/session.py

@@ -47,6 +47,11 @@ if TYPE_CHECKING:
 #: The total size of argument blob files kept under a persistent workspace root, 32 GiB.
 BLOB_DISK_LIMIT = 32 * 1024**3
 
+#: The largest project directory letify packs into an environment archive when it has no
+#: measured rate to judge the transfer by. Spec "Materializing into a runtime": the pack
+#: moves the archive as one payload the worker and the client each hold whole.
+PACK_WITHOUT_A_RATE = 1024**3
+
 
 def _against_the_build(size: int, rate: float, build_s: float) -> str:
     """The three numbers both cache decisions weigh, as they are reported.
@@ -185,6 +190,11 @@ class Runtime:
                 self.provider.env_root or self.workspace or self.provider.workspace_root, self.env
             )
             if self._worth_packing(root):
+                size = self._tree_size(root)
+                self._say(
+                    "caching the environment"
+                    + (f", {size / 1_000_000_000:.1f} GB" if size else "")
+                )
                 self._environment_volumes()[0].cache_env_from(
                     self, self.env, root, platform=self.platform, build_s=self.env_build_s
                 )
@@ -687,25 +697,38 @@ class Runtime:
             return None
         return float(rate)
 
+    def _tree_size(self, root: str) -> int | None:
+        """The byte size of the project directory on the runtime, or None when unknown."""
+        from . import bootstrap
+
+        try:
+            size = self.eval(bootstrap.tree_size_source(root), timeout=120)
+        except Exception:
+            return None
+        return size if isinstance(size, int) and size > 0 else None
+
     def _worth_packing(self, root: str) -> bool:
         """Whether moving the project directory over this session's link beats the build.
 
         Spec "Materializing into a runtime": the directory is larger than the archive it
-        would become, so this overstates the transfer and errs toward building. A session
-        with no measured rate packs unconditionally, as it always did.
+        would become, so this overstates the transfer and errs toward building. With no
+        measured rate it packs only what letify is willing to hold in memory, because the
+        pack moves the archive as one payload the worker and the client each hold whole.
         """
         if self.env_build_s is None:
             return True
         rate = self._measured_rate()
+        size = self._tree_size(root)
         if rate is None:
-            return True
-        from . import bootstrap
-
-        try:
-            size = self.eval(bootstrap.tree_size_source(root), timeout=120)
-        except Exception:
-            return True
-        if not isinstance(size, int) or size <= 0:
+            if size is None or size <= PACK_WITHOUT_A_RATE:
+                return True
+            self._say(
+                f"not caching the environment, because {size / 1_000_000_000:.1f} GB is over "
+                f"the {PACK_WITHOUT_A_RATE / 1024**3:.1f} GiB letify will hold in memory and "
+                f"this link was never measured"
+            )
+            return False
+        if size is None:
             return True
         if (size / rate) < self.env_build_s:
             return True

{letify-2.6.4 → letify-2.6.6}/letify/store/pathdata.py

@@ -485,6 +485,32 @@ def _mib(size: int) -> str:
     return f"{size / (1 << 20):.1f} MiB"
 
 
+class _Carried:
+    """One blob's progress, recorded on the stream and forwarded to the real meter.
+
+    Spec "Data log line": the bytes of a blob still on the way are reported, and only the
+    blob's own sender knows how far it got.
+    """
+
+    def __init__(self, stream: Any, digest: str, meter: Any) -> None:
+        self._stream = stream
+        self._digest = digest
+        self._meter = meter
+
+    def update(self, done: int) -> None:
+        self._stream.carried[self._digest] = done
+        if self._meter is not None:
+            self._meter.update(done)
+
+    def advance(self, size: int) -> None:
+        if self._meter is not None:
+            self._meter.advance(size)
+
+    def finish(self) -> None:
+        if self._meter is not None:
+            self._meter.finish()
+
+
 class _Uploaded:
     """Bytes sent so far, drawn as a progress line when the upload is large.
 
@@ -626,6 +652,10 @@ class Stream:
         #: Digests whose send finished. ``_sent`` holds the ones it began, so a blob the
         #: sender was carrying when the call ended is in one and not the other.
         self.done: set[str] = set()
+        #: Bytes of each blob handed to the runtime, including one still on the way.
+        self.carried: dict[str, int] = {}
+        #: The meter for blobs sent beside the call, spec "Data log line".
+        self._meter: _Uploaded | None = None
         self.before_files = self.before_bytes = 0
         self.during_files = self.during_bytes = 0
         self.before_seconds = self.during_seconds = 0.0
@@ -750,9 +780,22 @@ class Stream:
         digest = self._take()
         if digest is None:
             return False
-        self._send(digest, None)
+        self._send(digest, self._beside())
         return True
 
+    def _beside(self) -> _Uploaded:
+        """The meter for blobs sent beside the call, made at the first one.
+
+        Spec "Data log line": without one, a blob sent beside the call reported nothing,
+        so a transfer in progress and one that never started looked the same.
+        """
+        with self._lock:
+            if self._meter is None:
+                left = [d for d in self.plan.missing if d not in self.done]
+                total = sum(self.plan.weight(digest)[1] for digest in left)
+                self._meter = _Uploaded(len(left), total)
+            return self._meter
+
     def _take(self) -> str | None:
         with self._lock:
             while self._queue:
@@ -916,6 +959,7 @@ class Stream:
         size = self.plan.sizes.get(digest, 0)
         local = self.plan.files[digest]
         bucket = self.plan.bucket
+        meter = _Carried(self, digest, meter)
         uploaded = True
         if bucket is None:
             # The cost of opening the connections is weighed only while the pool is cold:
@@ -1014,9 +1058,13 @@ class Stream:
         missing_weight = [plan.weight(digest) for digest in plan.missing]
         already_files = plan.detected_files - sum(files for files, _b in missing_weight)
         already_bytes = plan.detected_bytes - sum(total for _f, total in missing_weight)
-        dropped_weight = [plan.weight(digest) for digest in self.dropped]
+        dropped = self.dropped
+        dropped_weight = [plan.weight(digest) for digest in dropped]
         dropped_files = sum(files for files, _b in dropped_weight)
         dropped_bytes = sum(total for _f, total in dropped_weight)
+        # Spec "Data log line": the rate counts only blobs that completed, so a large one
+        # in flight reads as 0.0 MiB/s however fast it is moving.
+        arrived = sum(self.carried.get(digest, 0) for digest in dropped)
         line = (
             f"data {plan.detected_files} files {_mib(plan.detected_bytes)} detected, "
             f"{already_files} files {_mib(already_bytes)} already on {plan.where}, "
@@ -1029,6 +1077,8 @@ class Stream:
             line += (
                 f", {dropped_files} files {_mib(dropped_bytes)} dropped when the call ended"
             )
+            if arrived:
+                line += f", {_mib(arrived)} of them had arrived"
         install.log(line)
         if stats.get("files"):
             install.log(

{letify-2.6.4 → letify-2.6.6}/letify-core/Cargo.lock

@@ -138,7 +138,7 @@ dependencies = [
 
 [[package]]
 name = "letify-agent"
-version = "2.6.4"
+version = "2.6.6"
 dependencies = [
  "letify-wire",
  "libloading",
@@ -146,7 +146,7 @@ dependencies = [
 
 [[package]]
 name = "letify-driver"
-version = "2.6.4"
+version = "2.6.6"
 dependencies = [
  "letify-wire",
 ]
@@ -164,7 +164,7 @@ dependencies = [
 
 [[package]]
 name = "letify-wire"
-version = "2.6.4"
+version = "2.6.6"
 
 [[package]]
 name = "libc"

{letify-2.6.4 → letify-2.6.6}/letify-core/Cargo.toml

@@ -8,7 +8,7 @@ resolver = "3"
 members = ["letify-wire", "letify-driver", "letify-agent", "letify-quic"]
 
 [workspace.package]
-version = "2.6.4"
+version = "2.6.6"
 edition = "2024"
 license = "MIT"
 repository = "https://github.com/DarkPyonix/letify"

{letify-2.6.4 → letify-2.6.6}/letify-ext/package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "letify-status",
-  "version": "2.6.4",
+  "version": "2.6.6",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "letify-status",
-      "version": "2.6.4",
+      "version": "2.6.6",
       "license": "SEE LICENSE IN LICENSE",
       "devDependencies": {
         "@types/node": "^18.19.0",

{letify-2.6.4 → letify-2.6.6}/letify-ext/package.json

@@ -2,7 +2,7 @@
   "name": "letify-status",
   "displayName": "letify Status",
   "description": "letify quota and GPU activity in the status bar, read from the letify CLI.",
-  "version": "2.6.4",
+  "version": "2.6.6",
   "publisher": "darkpyonix",
   "license": "SEE LICENSE IN LICENSE",
   "repository": {

{letify-2.6.4 → letify-2.6.6}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "letify"
-version = "2.6.4"
+version = "2.6.6"
 description = "Declarations that become infrastructure."
 readme = "README.md"
 requires-python = ">=3.11"

{letify-2.6.4 → letify-2.6.6}/tests/test_data_streaming.py

@@ -1757,3 +1757,66 @@ def test_a_blob_that_never_starts_still_times_out_and_says_how_far_it_got(
     message = str(failure.value)
     assert "did not arrive" in message, message
     assert "arrived" in message, message
+
+
+def test_a_blob_sent_beside_the_call_reports_its_bytes(
+    let, cpu, project, capsys, monkeypatch
+) -> None:
+    """Spec "Data log line": a transfer in progress read the same as one never started.
+
+    KICT's 11.3 GB table was moving the whole time and the line said 0.0 MiB/s, because
+    the rate counts only blobs that completed and the background sender was given no
+    meter at all.
+    """
+    from letify.store import pathdata
+
+    big = project / "beside.bin"
+    big.write_bytes(b"q" * (3 << 20))
+    monkeypatch.setattr(pathdata, "FIRST_WAVE_MIB", 0.0)
+    monkeypatch.setattr(pathdata, "FIRST_WAVE_FILES", 0)
+    monkeypatch.setattr(pathdata, "WAIT_TIMEOUT_S", 4.0)
+    def stalled(runtime, pipeline, blobs, digest, local, size, meter):
+        # A third of the blob is reported as handed over and then the send stops, which
+        # is the shape of a blob the sender was carrying when the call ended.
+        meter.update(1 << 20)
+        raise RuntimeFailure("the transfer stream failed")
+
+    monkeypatch.setattr(pathdata, "_put", stalled)
+
+    @let.function(device=cpu, host=letify.remote, retries=0)
+    def peek(path) -> int:
+        return len(path.read_bytes())
+
+    with pytest.raises(RuntimeFailure):
+        peek(path=big)
+    lines = [
+        text
+        for text in capsys.readouterr().err.splitlines()
+        if text.startswith("letify: data ") and "detected," in text
+    ]
+    assert lines
+    assert "dropped when the call ended" in lines[0], lines[0]
+    assert "had arrived" in lines[0], lines[0]
+
+
+def test_a_large_blob_beside_the_call_draws_the_progress_line(
+    let, cpu, project, capsys, monkeypatch
+) -> None:
+    """Spec "Data log line": the background sender draws the same line the wave draws.
+
+    KICT reported no 'uploading' line at all while 11.3 GB moved beside a call, because
+    the background sender was given no meter.
+    """
+    from letify.store import pathdata
+
+    big = project / "wide.bin"
+    big.write_bytes(b"w" * (70 << 20))
+    monkeypatch.setattr(pathdata, "FIRST_WAVE_MIB", 0.0)
+    monkeypatch.setattr(pathdata, "FIRST_WAVE_FILES", 0)
+
+    @let.function(device=cpu, host=letify.remote)
+    def read_it(path) -> int:
+        return len(path.read_bytes())
+
+    assert read_it(path=big) == 70 << 20
+    assert "uploading" in capsys.readouterr().err

{letify-2.6.4 → letify-2.6.6}/tests/test_runtime.py

@@ -2593,3 +2593,57 @@ def test_a_persistent_provider_that_builds_its_venv_elsewhere_still_archives(
     finally:
         runtime.shutdown()
     assert volume.cached_env(env, platform), "nothing was archived, so the next session resyncs"
+
+
+def test_a_large_environment_is_not_packed_when_the_link_was_never_measured(
+    uv_project: Path, tmp_path: Path, monkeypatch, capsys
+) -> None:
+    """Spec "Materializing into a runtime": the pack holds the archive whole, twice.
+
+    2.6.3 made a provider that keeps its workspace root archive like an ephemeral one, and
+    that branch packed unconditionally with no measured rate. On a Modal A10 a 7.2 GB
+    environment sat in the pack for over 30 minutes, silently, while the session billed.
+    """
+    from letify.runtime import session as sessionmod
+
+    provider = provider_of(PreparingLocal, "lab", persistent=False)
+    volume = provider.volume(
+        "cache", backend="filesystem", root=str(tmp_path / "store"), mount=str(tmp_path / "mount")
+    )
+    env = Env()
+    instance = Instance(provider, gpu=None)._placed("remote")
+    # No measured rate, as a Modal sandbox has none, and a tree over the limit.
+    monkeypatch.setattr(sessionmod.Runtime, "_measured_rate", lambda self: None)
+    monkeypatch.setattr(
+        sessionmod.Runtime, "_tree_size", lambda self, root: 8 * 1024**3
+    )
+    runtime = provider.start(instance, env, name="lab-1", volumes=(volume,))
+    try:
+        platform = runtime.platform
+    finally:
+        runtime.shutdown()
+    assert volume.cached_env(env, platform) is None
+    err = capsys.readouterr().err
+    assert "over the 1.0 GiB letify will hold in memory" in err, err
+
+
+def test_a_small_environment_is_still_packed_and_says_so(
+    uv_project: Path, tmp_path: Path, monkeypatch, capsys
+) -> None:
+    """Spec "Materializing into a runtime": the pack can take minutes, so it is said."""
+    from letify.runtime import session as sessionmod
+
+    provider = provider_of(PreparingLocal, "lab", persistent=False)
+    volume = provider.volume(
+        "cache", backend="filesystem", root=str(tmp_path / "store"), mount=str(tmp_path / "mount")
+    )
+    env = Env()
+    instance = Instance(provider, gpu=None)._placed("remote")
+    monkeypatch.setattr(sessionmod.Runtime, "_measured_rate", lambda self: None)
+    runtime = provider.start(instance, env, name="lab-1", volumes=(volume,))
+    try:
+        platform = runtime.platform
+    finally:
+        runtime.shutdown()
+    assert volume.cached_env(env, platform)
+    assert "caching the environment" in capsys.readouterr().err