letify 2.10.1__tar.gz → 2.10.2__tar.gz

{letify-2.10.1 → letify-2.10.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: letify
-Version: 2.10.1
+Version: 2.10.2
 Summary: Declarations that become infrastructure.
 Project-URL: Homepage, https://github.com/DarkPyonix/letify
 Project-URL: Documentation, https://github.com/DarkPyonix/letify/tree/main/docs

{letify-2.10.1 → letify-2.10.2}/docs/SPEC.md

@@ -1089,7 +1089,7 @@ A call has exactly one owner at a time, and the session still ends with the call
 
 1. **The registry keeps an orphan.** The record also holds `project`, the project root the call was sent from, and `lease_deadline`, the Unix time at which the runtime's lease ends unless renewed, rewritten at each renewal. It also holds what a second process needs to reach the call and finish it: `door`, the path of the worker's socket below; `config`, the project `config.toml` that names the account; and `call_dir`, `blobs` and `outputs`, the runtime paths and the output locations to write back, each location stored relative to `project` when it lies inside it. No secret is among them. A record whose owner pid is gone is no longer removed when read while its `lease_deadline` is in the future: `letify runs` lists it with `owner` `gone` and the seconds left, and it can be taken over. After the deadline it is removed as stale.
 2. **The worker keeps a second door.** While a call runs, the worker listens on a Unix socket on the runtime, `<workspace root>/.letify/run-<run id>.sock`, mode 0600. When that path is longer than 100 bytes, the limit of a Unix socket address, the worker and the relay reach the socket through `/proc/self/fd/<n>`, a descriptor of its directory, so it stays at that path; where `/proc` is missing the worker reports that the run has no second door. A channel that closes while a call runs, with no shutdown frame before it, does not end the worker: the body keeps running until the lease deadline passes. A shutdown frame is different: it is the owner ending the run on purpose, as [A deliberate shutdown](#run-shutdown) describes. Output the body makes while no channel is open is kept, the last 256 KiB per stream, and is sent to the next owner first. A body that returns while no channel is open keeps its result for a new owner until the lease deadline; on a provider with no lease the worker ends then, as before.
-3. **The new owner connects.** `letify takeover <run>` reads the record, opens a channel to the same runtime through the record's provider and runtime name, and starts a relay to that socket instead of a new worker. It then sends `adopt` with the run id. The worker answers once with the call's state, which is `running` or `finished`, and from then on sends the call's output and its reply on the new channel, the reply as the last piece of the same answer. The old channel, if still open, is told `moved` on the call's stream and answers every later request on it with `moved`. A door takes one connection at a time and no request but `adopt` before it, and a second owner can adopt after the first one's connection has closed.
+3. **The new owner connects.** `letify takeover <run>` reads the record, opens a channel to the same runtime through the record's provider and runtime name, and starts a relay to that socket instead of a new worker. It then sends `adopt` with the run id. The worker answers once with the call's state, which is `running` or `finished`, and from then on sends the call's output and its reply on the new channel, the reply as the last piece of the same answer. The old channel, if still open, is told `moved` on the call's stream, after the worker has switched the call's output to the new channel, so everything the body prints once the old owner knows of the move reaches the new owner. It answers every later request on it with `moved`. A door takes one connection at a time and no request but `adopt` before it, and a second owner can adopt after the first one's connection has closed. The adoption is atomic with respect to the old channel: if the old channel closes while the adoption is under way, the worker decides what that close means only after the adoption has finished, sees that the call has moved, and does nothing. The new owner's channel is then never marked as having lost its input, and no exit hook runs for the close.
 4. **The old owner lets go.** When the old owner is alive, takeover first asks it through its control socket to release the run. Its waiting call raises `OwnershipMoved` naming the run and the new owner's pid, it stops renewing that runtime's lease, and it drops the runtime from its pool without stopping it. A `keep_alive` block in the old owner no longer holds that session. An old owner that does not answer within 5 s is treated as gone.
 5. **The call ends at the new owner.** `letify takeover` prints the body's output as it arrives. When the call returns it prints the `repr` of the value, or with `--result PATH` writes the value with pickle to `PATH`, and exits 0. When the body raises it prints the traceback and exits 1. With `--json` the last line is one object with `run`, `ok`, `value_repr` or `error`, and `result_path`. The call's outputs are written back under the record's `project`, not the current directory. The session is then ended.
 6. **The same from Python.** `letify.takeover(run)` does the same in the calling process and returns the value or raises the body's exception, as a call does. It holds no handle: it returns once the call has ended and the session is gone. A darkpyonix kernel uses this to adopt a run started elsewhere.

{letify-2.10.1 → letify-2.10.2}/letify/__init__.py

@@ -70,7 +70,7 @@ distributed: Final = Host.distributed
 # The enum class stays importable from letify.declare.instance for letify's own use.
 del Host
 
-__version__ = "2.10.1"
+__version__ = "2.10.2"
 
 
 def fetch(tensor):  # type: ignore[no-untyped-def]

{letify-2.10.1 → letify-2.10.2}/letify/protocol/worker.py

@@ -3549,11 +3549,12 @@ def _reader_ended(door, shutdown=False):
     Spec "A deliberate shutdown": a shutdown frame on the channel that owns a running call
     ends the body and the worker. End of input alone leaves the body for a takeover.
     """
-    if door is None and _TAKE["moved"]:
-        # The call was adopted away; this channel closing is the old owner leaving.
-        return
     during = False
     with _TAKE_COND:
+        if door is None and _TAKE["moved"]:
+            # The call was adopted away; this channel closing is the old owner leaving. Read
+            # under the lock, because an adoption in progress holds it until it has moved.
+            return
         if door is None or (_TAKE["owner"] is door and _TAKE["live"]):
             during = _TAKE["run"] is not None
             if door is not None or during:
@@ -3702,12 +3703,6 @@ def _door_adopt(door, stream, request):
         moved = _moved_outcome()
         if _TAKE["owner"] is not None:
             moved["last"] = True
-        # Told on a thread, because a former owner that has stopped reading can block the write.
-        told = threading.Thread(
-            target=_answer, args=(former, former_stream, moved), daemon=True
-        )
-        told.start()
-        told.join(2.0)
         state = {
             "run": run,
             "state": "finished" if _TAKE["finished"] else "running",
@@ -3728,6 +3723,14 @@ def _door_adopt(door, stream, request):
         )
         _SENDER = door.sender
         _TAKE_COND.notify_all()
+        # Told last, after the output has been switched, so that nothing the body prints from
+        # the moment the former owner learns of the move is sent to a channel that has stopped
+        # reading. A thread, because a former owner that has stopped reading can block the write.
+        told = threading.Thread(
+            target=_answer, args=(former, former_stream, moved), daemon=True
+        )
+        told.start()
+        told.join(2.0)
 
 
 def _deliver_call(stream, outcome):

{letify-2.10.1 → letify-2.10.2}/letify-core/Cargo.lock

@@ -138,11 +138,11 @@ dependencies = [
 
 [[package]]
 name = "letify"
-version = "2.10.1"
+version = "2.10.2"
 
 [[package]]
 name = "letify-quic"
-version = "2.10.1"
+version = "2.10.2"
 dependencies = [
  "quinn",
  "rcgen",

{letify-2.10.1 → letify-2.10.2}/letify-core/Cargo.toml

@@ -8,7 +8,7 @@ resolver = "3"
 members = ["letify", "letify-quic"]
 
 [workspace.package]
-version = "2.10.1"
+version = "2.10.2"
 edition = "2024"
 license = "MIT"
 repository = "https://github.com/DarkPyonix/letify"

{letify-2.10.1 → letify-2.10.2}/letify-ext/package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "letify-status",
-  "version": "2.10.1",
+  "version": "2.10.2",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "letify-status",
-      "version": "2.10.1",
+      "version": "2.10.2",
       "license": "SEE LICENSE IN LICENSE",
       "devDependencies": {
         "@types/node": "^18.19.0",

{letify-2.10.1 → letify-2.10.2}/letify-ext/package.json

@@ -2,7 +2,7 @@
   "name": "letify-status",
   "displayName": "letify Status",
   "description": "letify quota and GPU activity in the status bar, read from the letify CLI.",
-  "version": "2.10.1",
+  "version": "2.10.2",
   "publisher": "darkpyonix",
   "license": "SEE LICENSE IN LICENSE",
   "repository": {

{letify-2.10.1 → letify-2.10.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "letify"
-version = "2.10.1"
+version = "2.10.2"
 description = "Declarations that become infrastructure."
 readme = "README.md"
 requires-python = ">=3.11"

{letify-2.10.1 → letify-2.10.2}/tests/test_run_takeover.py

@@ -289,6 +289,8 @@ def test_the_new_owner_takes_the_record_and_removes_it_when_the_call_ends(world,
     until(lambda: any(r["pid"] == os.getpid() for r in control.list_runs()), "the new record")
     row = next(r for r in control.list_runs() if r["pid"] == os.getpid())
     assert row["run"] == owner.run
+    # The body returns only once the old owner has let go, which is after the adoption.
+    owner.answer()
     owner.finish_body()
     taking.join()
     assert control.list_runs(orphans=True) == []
@@ -742,3 +744,111 @@ def test_detaching_a_persistent_channel_sends_no_shutdown(monkeypatch) -> None:
     channel.detach()
     assert wire.SHUTDOWN not in sent
     assert sent == []
+
+
+#: The old owner's channel closes while the worker is still adopting the call to the new owner.
+#: The adoption holds the call's lock, so the close has to wait for it and then see the move.
+RACE_PROBE = textwrap.dedent(
+    """
+    import json, sys, threading, time
+    from pathlib import Path
+
+    from letify.protocol import worker
+
+    out = sys.argv[1]
+    text = worker.SOURCE.rstrip()
+    assert text.endswith("_serve()")
+    space = {"__name__": "probe"}
+    exec(compile(text[: -len("_serve()")], "worker", "exec"), space)
+    hooks = []
+    space["_run_exit_hooks"] = lambda reason: hooks.append(reason)
+    take, cond = space["_TAKE"], space["_TAKE_COND"]
+    take.update(run="abcdef012345", live=True, moved=False)
+    new_owner = object()
+    with cond:
+        closing = threading.Thread(target=space["_reader_ended"], args=(None, False))
+        closing.start()
+        time.sleep(0.5)
+        # The adoption finishes under the lock, after the close began.
+        take.update(owner=new_owner, moved=True, live=True, ended_input=False)
+    closing.join(30)
+    Path(out).write_text(json.dumps({
+        "live": take["live"], "ended_input": take["ended_input"], "hooks": hooks,
+    }))
+    """
+)
+
+
+def test_the_old_owners_channel_closing_during_the_adoption_leaves_the_new_owner_live(
+    tmp_path,
+) -> None:
+    script = tmp_path / "race.py"
+    script.write_text(RACE_PROBE, encoding="utf-8")
+    out = tmp_path / "race.out"
+    root = str(Path(letify.__file__).resolve().parent.parent)
+    subprocess.run(
+        [sys.executable, str(script), str(out)],
+        check=True, timeout=60, env={**os.environ, "PYTHONPATH": root},
+        stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
+    )
+    assert json.loads(out.read_text(encoding="utf-8")) == {
+        "live": True, "ended_input": False, "hooks": [],
+    }
+
+
+#: Adopts a call with stand-in channels and records whether the worker already sends output
+#: to the new owner at the moment the old owner is told the call has moved.
+ORDER_PROBE = textwrap.dedent(
+    """
+    import json, sys
+    from pathlib import Path
+
+    from letify.protocol import worker
+
+    out = sys.argv[1]
+    text = worker.SOURCE.rstrip()
+    assert text.endswith("_serve()")
+    space = {"__name__": "probe"}
+    exec(compile(text[: -len("_serve()")], "worker", "exec"), space)
+
+
+    class Channel:
+        def message(self, *args):
+            pass
+
+        def frame(self, *args):
+            pass
+
+
+    class Connection:
+        def settimeout(self, value):
+            pass
+
+
+    class Door:
+        sender = Channel()
+        connection = Connection()
+
+
+    door, former = Door(), Channel()
+    seen = []
+    space["_answer"] = lambda sender, stream, outcome: seen.append(space["_SENDER"] is door.sender)
+    space["_SENDER"] = former
+    space["_TAKE"].update(run="abcdef012345", live=True, stream=3)
+    space["_door_adopt"](door, 7, {"run": "abcdef012345", "owner": 1234})
+    Path(out).write_text(json.dumps(seen))
+    """
+)
+
+
+def test_output_after_the_old_owner_is_told_the_call_moved_goes_to_the_new_owner(tmp_path) -> None:
+    script = tmp_path / "order.py"
+    script.write_text(ORDER_PROBE, encoding="utf-8")
+    out = tmp_path / "order.out"
+    root = str(Path(letify.__file__).resolve().parent.parent)
+    subprocess.run(
+        [sys.executable, str(script), str(out)],
+        check=True, timeout=60, env={**os.environ, "PYTHONPATH": root},
+        stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
+    )
+    assert json.loads(out.read_text(encoding="utf-8")) == [True]