letify 2.6.3__tar.gz → 2.6.5__tar.gz

{letify-2.6.3 → letify-2.6.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: letify
-Version: 2.6.3
+Version: 2.6.5
 Summary: Declarations that become infrastructure.
 Project-URL: Homepage, https://github.com/DarkPyonix/letify
 Project-URL: Documentation, https://github.com/DarkPyonix/letify/tree/main/docs

{letify-2.6.3 → letify-2.6.5}/docs/SPEC.md

@@ -807,7 +807,11 @@ A `read_until` blocks until the sandbox prints a line, and the adapter answers o
 - `Modal.stop` calls `abort()` when the adapter is out of step or its lock is still held after 5 s. The call timeout watchdog calls it through `SandboxChannel._kill`, so a call past its `timeout` ends its sandbox too.
 - The adapter's `terminate` for an id it did not create finds the sandbox with `modal.Sandbox.from_id` and terminates it. A sandbox that is already gone is not an error.
 
-Modal also bounds a sandbox that nothing terminates. `create` passes `timeout`, the entry option `timeout`, 3600 s by default, as the sandbox's maximum lifetime. It passes `idle_timeout`, the entry option `idle_timeout`, 600 s by default, after which Modal terminates a sandbox that is idle.
+Modal also bounds a sandbox that nothing terminates. `create` passes `idle_timeout`, the entry option `idle_timeout`, 600 s by default, after which Modal terminates a sandbox that is idle. That is the bound that stops an abandoned sandbox billing, because an abandoned one is idle by definition once its worker has no one to answer.
+
+It passes `timeout`, the entry option `timeout`, as the sandbox's maximum lifetime, and the default is the longest Modal accepts, 86400 s, measured as accepted. It is not a shorter number letify picked, for the reason The declaration gives for `timeout` having no default: a deadline letify invented would end a two hour training run at whatever hour it guessed. The default was 3600 s, exactly one hour, and it ended two Modal sessions of a real training run at one hour each with nothing saying why, which is that sentence happening. An account that wants a hard cap sets `timeout` itself.
+
+A sandbox that reaches that lifetime takes its worker with it, so the channel sees a worker that stopped without replying. Where the session's age is within a minute of the lifetime letify asked for, the failure says so instead of listing causes: `the Modal sandbox reached the <seconds> s lifetime letify asked for; raise timeout on the account to run longer`.
 
 ### Modal data channel <!-- id: modal-data-channel -->
 
@@ -1059,6 +1063,16 @@ less time than the build did. The archive is smaller than the directory, so the
 overstates the transfer and errs toward building, which is the cheaper of the two to get wrong. The
 ref records the build seconds beside the archive.
 
+With no measured rate for the link, letify packs only when that directory is at most 1 GiB, and says
+why when it declines. Packing moves the archive as one payload that the worker and the client each
+hold whole, so a tree of several gigabytes is held twice in memory with nothing reporting progress:
+measured on a Modal A10, a 7.2 GB environment sat in that step for over 30 minutes while the session
+billed. Packing unconditionally was right while only an ephemeral provider packed, where the
+alternative was a certain full rebuild; a provider that keeps its workspace root and builds its venv
+elsewhere reaches the same branch, and there the alternative is a sync that may take two minutes.
+Making the pack stream rather than hold the archive whole is the fix that removes this limit, and
+until then the limit is what keeps the step from costing more than it saves.
+
 A later session restores only when the archive's size over the link measured for that session would
 take less than the recorded build. Otherwise it syncs, and says why:
 `letify: <runtime>: syncing rather than restoring, because 3.0 GB over 13.6 MiB/s is 210 s against a
@@ -1072,7 +1086,9 @@ the kind this rule exists to catch, and the session that syncs instead both reco
 decides afresh whether the archive is worth keeping. The cost of that is one sync; the cost of
 trusting it is the transfer this rule was measured to avoid.
 
-Skipping the pack is said too, because it is a decision letify made rather than a thing that did not
+Packing is said, because it is a step that can take minutes and had nothing reporting it:
+`letify: <runtime>: caching the environment, 2.1 GB`. Skipping it is said too, because it is a
+decision letify made rather than a thing that did not
 happen: `letify: <runtime>: not caching the environment, because 9.4 GB over 13.6 MiB/s is 659 s
 against a 34 s build`.
 

{letify-2.6.3 → letify-2.6.5}/letify/__init__.py

@@ -66,7 +66,7 @@ distributed: Final = Host.distributed
 # The enum class stays importable from letify.declare.instance for letify's own use.
 del Host
 
-__version__ = "2.6.3"
+__version__ = "2.6.5"
 
 
 def fetch(tensor):  # type: ignore[no-untyped-def]

{letify-2.6.3 → letify-2.6.5}/letify/providers/modal.py

@@ -102,6 +102,26 @@ STOP_WAIT = 5.0
 #: ``idle_timeout``.
 IDLE_TIMEOUT = 600
 
+#: The sandbox's maximum lifetime, the longest Modal accepts, measured as accepted.
+#: Spec "Modal": not a shorter number letify picked, for the reason The declaration gives
+#: for ``timeout`` having no default. The previous 3600 s ended two sessions of a real
+#: training run at one hour each with nothing saying why.
+SANDBOX_TIMEOUT = 86400
+
+
+def lifetime_detail(*, age_s: float, lifetime_s: int) -> str | None:
+    """What to say when a sandbox died at the lifetime letify asked for, or None.
+
+    Spec "Modal": the channel otherwise reports a worker that stopped without replying and
+    lists causes, none of which is the deadline letify set itself.
+    """
+    if lifetime_s <= 0 or age_s < lifetime_s - 60.0:
+        return None
+    return (
+        f"the Modal sandbox reached the {lifetime_s} s lifetime letify asked for; "
+        f"raise timeout on the account to run longer"
+    )
+
 
 class VolumePathMissing(RuntimeFailure):
     """The adapter answered that a volume path does not exist."""
@@ -377,6 +397,18 @@ class Modal(Provider):
         self._sandboxes: dict[str, str] = {}
         self._adapter: Adapter | None = None
 
+    @property
+    def sandbox_timeout_s(self) -> int:
+        """The sandbox's maximum lifetime, from the account or the longest Modal accepts."""
+        value = self.config.option("timeout", SANDBOX_TIMEOUT)
+        return max(1, int(value))
+
+    @property
+    def sandbox_idle_timeout_s(self) -> int:
+        """Seconds a sandbox may be idle before Modal terminates it. Spec "Modal"."""
+        value = self.config.option("idle_timeout", IDLE_TIMEOUT)
+        return max(1, int(value))
+
     def adapter(self) -> Adapter:
         """The adapter acting as this account, started on its first request."""
         if self._adapter is None or self._adapter.closed:
@@ -440,14 +472,15 @@ class Modal(Provider):
             args=["python3", "-u", "-c", BOOTSTRAP],
             packages=list(WORKER_PACKAGES),
             gpu=self.wire_name(runtime.instance) or None,
-            timeout=int(self.config.option("timeout", 3600)),
-            idle_timeout=int(self.config.option("idle_timeout", IDLE_TIMEOUT)),
+            timeout=self.sandbox_timeout_s,
+            idle_timeout=self.sandbox_idle_timeout_s,
             ports=[] if data_port is None else [data_port],
         )
         sandbox = str(created["sandbox"])
         self._sandboxes[runtime.name] = sandbox
         return SandboxChannel(
-            adapter, sandbox, name=runtime.name, data_port=data_port, streams=streams
+            adapter, sandbox, name=runtime.name, data_port=data_port, streams=streams,
+            lifetime_s=self.sandbox_timeout_s,
         )
 
     def stop(self, runtime: Runtime) -> None:
@@ -598,11 +631,15 @@ class SandboxChannel(FramedChannel):
         name: str,
         data_port: int | None = None,
         streams: int = 1,
+        lifetime_s: int = 0,
     ):
         self.adapter = adapter
         self.sandbox = sandbox
         self.name = name
         self.data_port = data_port
+        #: The sandbox's maximum lifetime, so a death at it is named rather than guessed.
+        self.lifetime_s = lifetime_s
+        self.opened_at = time.monotonic()
         #: TCP connections the data channel carries its frames over.
         self.streams = streams
         self._sockets: list[socket.socket] = []
@@ -628,7 +665,13 @@ class SandboxChannel(FramedChannel):
         self._await_ready()
 
     def _raw_text(self) -> str:
-        return bytes(self._raw[-2000:]).decode("utf-8", "replace")
+        text = bytes(self._raw[-2000:]).decode("utf-8", "replace")
+        # Spec "Modal": a sandbox that reached the lifetime letify asked for takes its
+        # worker with it, and the channel otherwise lists causes, none of them this one.
+        said = lifetime_detail(
+            age_s=time.monotonic() - self.opened_at, lifetime_s=self.lifetime_s
+        )
+        return f"{said}\n{text}" if said else text
 
     def _send_worker(self) -> None:
         # After a reexec the data connection is gone, so the source goes back over stdio.

{letify-2.6.3 → letify-2.6.5}/letify/runtime/session.py

@@ -47,6 +47,11 @@ if TYPE_CHECKING:
 #: The total size of argument blob files kept under a persistent workspace root, 32 GiB.
 BLOB_DISK_LIMIT = 32 * 1024**3
 
+#: The largest project directory letify packs into an environment archive when it has no
+#: measured rate to judge the transfer by. Spec "Materializing into a runtime": the pack
+#: moves the archive as one payload the worker and the client each hold whole.
+PACK_WITHOUT_A_RATE = 1024**3
+
 
 def _against_the_build(size: int, rate: float, build_s: float) -> str:
     """The three numbers both cache decisions weigh, as they are reported.
@@ -185,6 +190,11 @@ class Runtime:
                 self.provider.env_root or self.workspace or self.provider.workspace_root, self.env
             )
             if self._worth_packing(root):
+                size = self._tree_size(root)
+                self._say(
+                    "caching the environment"
+                    + (f", {size / 1_000_000_000:.1f} GB" if size else "")
+                )
                 self._environment_volumes()[0].cache_env_from(
                     self, self.env, root, platform=self.platform, build_s=self.env_build_s
                 )
@@ -687,25 +697,38 @@ class Runtime:
             return None
         return float(rate)
 
+    def _tree_size(self, root: str) -> int | None:
+        """The byte size of the project directory on the runtime, or None when unknown."""
+        from . import bootstrap
+
+        try:
+            size = self.eval(bootstrap.tree_size_source(root), timeout=120)
+        except Exception:
+            return None
+        return size if isinstance(size, int) and size > 0 else None
+
     def _worth_packing(self, root: str) -> bool:
         """Whether moving the project directory over this session's link beats the build.
 
         Spec "Materializing into a runtime": the directory is larger than the archive it
-        would become, so this overstates the transfer and errs toward building. A session
-        with no measured rate packs unconditionally, as it always did.
+        would become, so this overstates the transfer and errs toward building. With no
+        measured rate it packs only what letify is willing to hold in memory, because the
+        pack moves the archive as one payload the worker and the client each hold whole.
         """
         if self.env_build_s is None:
             return True
         rate = self._measured_rate()
+        size = self._tree_size(root)
         if rate is None:
-            return True
-        from . import bootstrap
-
-        try:
-            size = self.eval(bootstrap.tree_size_source(root), timeout=120)
-        except Exception:
-            return True
-        if not isinstance(size, int) or size <= 0:
+            if size is None or size <= PACK_WITHOUT_A_RATE:
+                return True
+            self._say(
+                f"not caching the environment, because {size / 1_000_000_000:.1f} GB is over "
+                f"the {PACK_WITHOUT_A_RATE / 1024**3:.1f} GiB letify will hold in memory and "
+                f"this link was never measured"
+            )
+            return False
+        if size is None:
             return True
         if (size / rate) < self.env_build_s:
             return True

{letify-2.6.3 → letify-2.6.5}/letify-core/Cargo.lock

@@ -138,7 +138,7 @@ dependencies = [
 
 [[package]]
 name = "letify-agent"
-version = "2.6.3"
+version = "2.6.5"
 dependencies = [
  "letify-wire",
  "libloading",
@@ -146,7 +146,7 @@ dependencies = [
 
 [[package]]
 name = "letify-driver"
-version = "2.6.3"
+version = "2.6.5"
 dependencies = [
  "letify-wire",
 ]
@@ -164,7 +164,7 @@ dependencies = [
 
 [[package]]
 name = "letify-wire"
-version = "2.6.3"
+version = "2.6.5"
 
 [[package]]
 name = "libc"

{letify-2.6.3 → letify-2.6.5}/letify-core/Cargo.toml

@@ -8,7 +8,7 @@ resolver = "3"
 members = ["letify-wire", "letify-driver", "letify-agent", "letify-quic"]
 
 [workspace.package]
-version = "2.6.3"
+version = "2.6.5"
 edition = "2024"
 license = "MIT"
 repository = "https://github.com/DarkPyonix/letify"

{letify-2.6.3 → letify-2.6.5}/letify-ext/package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "letify-status",
-  "version": "2.6.3",
+  "version": "2.6.5",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "letify-status",
-      "version": "2.6.3",
+      "version": "2.6.5",
       "license": "SEE LICENSE IN LICENSE",
       "devDependencies": {
         "@types/node": "^18.19.0",

{letify-2.6.3 → letify-2.6.5}/letify-ext/package.json

@@ -2,7 +2,7 @@
   "name": "letify-status",
   "displayName": "letify Status",
   "description": "letify quota and GPU activity in the status bar, read from the letify CLI.",
-  "version": "2.6.3",
+  "version": "2.6.5",
   "publisher": "darkpyonix",
   "license": "SEE LICENSE IN LICENSE",
   "repository": {

{letify-2.6.3 → letify-2.6.5}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "letify"
-version = "2.6.3"
+version = "2.6.5"
 description = "Declarations that become infrastructure."
 readme = "README.md"
 requires-python = ">=3.11"

{letify-2.6.3 → letify-2.6.5}/tests/test_providers.py

@@ -2218,7 +2218,9 @@ def test_a_modal_sandbox_is_created_with_a_lifetime_and_an_idle_limit(
     provider.open_channel(runtime)
     try:
         [created] = fake_modal.requests("create")
-        assert created["timeout"] == 3600
+        # Spec "Modal": the lifetime is the longest Modal accepts, not an hour letify
+        # picked, because a deadline letify invented ends the user's work at its guess.
+        assert created["timeout"] == 86400
         assert created["idle_timeout"] == 600
     finally:
         provider.stop(runtime)
@@ -2910,3 +2912,32 @@ def test_colab_punch_rendezvous_completes_during_tailcat_installation(
         release.set()
         installing.join(2)
     assert endpoints == [("fake", 22)]
+
+
+def test_a_modal_sandbox_is_not_given_an_hour_long_deadline(config_file) -> None:
+    """Spec "Modal": a deadline letify invented would end a two hour run at its guess.
+
+    The default was 3600 s, exactly one hour, and it ended two Modal sessions of a real
+    training run at one hour each with nothing saying why. The declaration's own `timeout`
+    has no default for this reason, and the sandbox's lifetime is the same question.
+    """
+    from conftest import provider_of
+
+    from letify.providers import Modal
+
+    provider = provider_of(Modal, "lab")
+    assert provider.sandbox_timeout_s == 86400, "letify picked a deadline again"
+    # An idle sandbox is what has to be bounded, and that bound is unchanged.
+    assert provider.sandbox_idle_timeout_s == 600
+    # An account that wants a cap still sets one.
+    assert provider_of(Modal, "lab", timeout=7200).sandbox_timeout_s == 7200
+
+
+def test_a_sandbox_that_reached_its_lifetime_says_so(config_file) -> None:
+    """Spec "Modal": the channel otherwise lists causes, none of them this one."""
+    from letify.providers.modal import lifetime_detail
+
+    assert lifetime_detail(age_s=3599.5, lifetime_s=3600) is not None
+    assert "3600" in lifetime_detail(age_s=3599.5, lifetime_s=3600)
+    assert "raise timeout" in lifetime_detail(age_s=3601.0, lifetime_s=3600)
+    assert lifetime_detail(age_s=120.0, lifetime_s=3600) is None

{letify-2.6.3 → letify-2.6.5}/tests/test_runtime.py

@@ -2593,3 +2593,57 @@ def test_a_persistent_provider_that_builds_its_venv_elsewhere_still_archives(
     finally:
         runtime.shutdown()
     assert volume.cached_env(env, platform), "nothing was archived, so the next session resyncs"
+
+
+def test_a_large_environment_is_not_packed_when_the_link_was_never_measured(
+    uv_project: Path, tmp_path: Path, monkeypatch, capsys
+) -> None:
+    """Spec "Materializing into a runtime": the pack holds the archive whole, twice.
+
+    2.6.3 made a provider that keeps its workspace root archive like an ephemeral one, and
+    that branch packed unconditionally with no measured rate. On a Modal A10 a 7.2 GB
+    environment sat in the pack for over 30 minutes, silently, while the session billed.
+    """
+    from letify.runtime import session as sessionmod
+
+    provider = provider_of(PreparingLocal, "lab", persistent=False)
+    volume = provider.volume(
+        "cache", backend="filesystem", root=str(tmp_path / "store"), mount=str(tmp_path / "mount")
+    )
+    env = Env()
+    instance = Instance(provider, gpu=None)._placed("remote")
+    # No measured rate, as a Modal sandbox has none, and a tree over the limit.
+    monkeypatch.setattr(sessionmod.Runtime, "_measured_rate", lambda self: None)
+    monkeypatch.setattr(
+        sessionmod.Runtime, "_tree_size", lambda self, root: 8 * 1024**3
+    )
+    runtime = provider.start(instance, env, name="lab-1", volumes=(volume,))
+    try:
+        platform = runtime.platform
+    finally:
+        runtime.shutdown()
+    assert volume.cached_env(env, platform) is None
+    err = capsys.readouterr().err
+    assert "over the 1.0 GiB letify will hold in memory" in err, err
+
+
+def test_a_small_environment_is_still_packed_and_says_so(
+    uv_project: Path, tmp_path: Path, monkeypatch, capsys
+) -> None:
+    """Spec "Materializing into a runtime": the pack can take minutes, so it is said."""
+    from letify.runtime import session as sessionmod
+
+    provider = provider_of(PreparingLocal, "lab", persistent=False)
+    volume = provider.volume(
+        "cache", backend="filesystem", root=str(tmp_path / "store"), mount=str(tmp_path / "mount")
+    )
+    env = Env()
+    instance = Instance(provider, gpu=None)._placed("remote")
+    monkeypatch.setattr(sessionmod.Runtime, "_measured_rate", lambda self: None)
+    runtime = provider.start(instance, env, name="lab-1", volumes=(volume,))
+    try:
+        platform = runtime.platform
+    finally:
+        runtime.shutdown()
+    assert volume.cached_env(env, platform)
+    assert "caching the environment" in capsys.readouterr().err