letify 2.8.7__tar.gz → 2.9.0__tar.gz

{letify-2.8.7 → letify-2.9.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: letify
-Version: 2.8.7
+Version: 2.9.0
 Summary: Declarations that become infrastructure.
 Project-URL: Homepage, https://github.com/DarkPyonix/letify
 Project-URL: Documentation, https://github.com/DarkPyonix/letify/tree/main/docs

{letify-2.8.7 → letify-2.9.0}/PROJECT.md

@@ -190,6 +190,22 @@ cache.fetch_checkpoint("run-1", target)
 cache.resume(runtime, "run-1", path)   # put the newest one inside a session
 ```
 
+### Exit hooks
+
+```python
+letify.on_exit(fn)                    # inside a body: run fn if the worker ends before the body does
+```
+
+The hook runs on `SIGTERM`, when the lease deadline passes and when the channel closes during the call, newest first, within 120 s in total. Outside a body it raises `UnsupportedMode`. `letify runs`, `interrupt` and `exec` reach a running call from the command line; see [Controlling a running call](docs/SPEC.md#run-control).
+
+### Taking over a running call
+
+```python
+value = letify.takeover(run)         # become the owner of a running call, return its value
+```
+
+`letify.takeover(run)` and `letify takeover <run>` make the calling process the owner of a call another process started: it renews the lease, receives the output and the result, writes the call's outputs back under the project the call was sent from, and ends the session. The previous owner, if it is alive, is asked to let go and its waiting call raises `OwnershipMoved`. A provider whose runtime a second process cannot reach raises `UnsupportedMode` naming it; see [Taking over a running call](docs/SPEC.md#run-takeover).
+
 ### Handoffs
 
 ```python
@@ -225,6 +241,7 @@ Both work in the local process and inside a shipped function, on any provider. T
 | `ProtocolError` | The remote process died before producing a result | yes |
 | `RemoteError` | The shipped function raised; carries the remote traceback | no |
 | `UnsupportedMode` | The requested mode cannot work here | no |
+| `OwnershipMoved` | The call you were waiting on was taken over by another process, which now receives its result | no |
 | `InsufficientDevices` | The devices a call asks for cannot be allocated; names what holds them | no |
 
 ### Command line
@@ -237,6 +254,10 @@ letify devices                # accelerators each provider offers
 letify status                 # live sessions and what they are costing
 letify usage [alias]          # what is left on each account
 letify utilization [alias]    # how busy each instance's accelerator is
+letify runs                   # calls running from this machine, with their run ids
+letify interrupt <run>        # raise KeyboardInterrupt in a running call's body
+letify exec <run> CODE        # run source beside a running call (-f FILE, --timeout S)
+letify takeover <run>         # become the owner of a running call (--result PATH, --json)
 letify check <alias>          # confirm a machine answers
 letify efficiency 0.5 3 150   # expected share of a direct run
 letify stubs                  # write typings/letify_providers.pyi for editor completion

{letify-2.8.7 → letify-2.9.0}/docs/SPEC.md

@@ -818,7 +818,7 @@ A sandbox that reaches that lifetime takes its worker with it, so the channel se
 
 Otherwise a Modal worker that stopped without replying is followed by one `poll` of its sandbox from a second adapter process, waiting at most 30 s for the exit code, and the failure names what the poll found instead of listing causes: `the Modal sandbox exited with code <n>` for a sandbox that ended, with `, which is SIGKILL, most often the container running out of memory` added for 137, and `the Modal sandbox is still running, so the worker process died inside it` for one still running. A poll that fails or times out leaves the message as it was. The message is what tells an out-of-memory kill from a preemption, which a training run that died twice in one day could not.
 
-A worker reached through a subprocess (`PersistentChannel`, for example `ssh` to a Colab or Kaggle machine) that exits before it says hello, or while a call is running, has its exit code named in the message text, so a user can tell a lost connection from a kill from a crash. A signal death is reported as the shell does, 128 plus the signal number. The message reads `the worker exited with code <n>` followed by a short meaning: `255` adds `: ssh could not reach or lost the machine`, `128+N` adds `: killed by signal N`, with `137` adding `, which is SIGKILL, most often out of memory`, and any other nonzero code adds nothing. A worker that never said hello reads `the worker exited before it was ready, with code <n>` followed by the same meaning. The last 2000 characters of the process's standard error follow under `--- last remote output ---`, and the startup failure also keeps the whole text as its `stderr` attribute. A channel whose provider can name a cause itself, such as Modal, keeps that cause. When the command's program is `ssh` or `ssh.exe`, `PersistentChannel` starts it as `ssh -E <file> ...`, with a fresh temporary file per channel placed right after the program. `ControlPersist` forks a master whose standard error is /dev/null, so its errors (`Timeout, server not responding`, `Connection reset`, `closed by remote host`, `Broken pipe`) reach only that file. A death message, both before hello and during a call, appends the last 2000 characters of the file under `--- ssh log ---` when it is not empty, after the standard error of the ssh client. Other commands are not changed. The file is deleted when the channel closes.
+A worker reached through a subprocess (`PersistentChannel`, for example `ssh` to a Colab or Kaggle machine) that exits before it says hello, or while a call is running, has its exit code named in the message text, so a user can tell a lost connection from a kill from a crash. A signal death is reported as the shell does, 128 plus the signal number. The message reads `the worker exited with code <n>` followed by a short meaning: `255` adds `: ssh could not reach or lost the machine`, `128+N` adds `: killed by signal N`, with `137` adding `, which is SIGKILL, most often out of memory`, and any other nonzero code adds nothing. A worker that never said hello reads `the worker exited before it was ready, with code <n>` followed by the same meaning. The last 2000 characters of the process's standard error follow under `--- last remote output ---`, and the startup failure also keeps the whole text as its `stderr` attribute. A channel whose provider can name a cause itself, such as Modal, keeps that cause. When the command's program is `ssh` or `ssh.exe`, `PersistentChannel` starts it as `ssh -E <file> ...`, with a fresh temporary file per channel placed right after the program. The ssh of a worker channel is never a client of a shared connection (see "SSH authentication"), so its own transport errors (`Connection to <host> closed by remote host.`, `Timeout, server not responding`, `Broken pipe`) reach its standard error and this file. A death message, both before hello and during a call, appends the last 2000 characters of the file under `--- ssh log ---` when it is not empty, after the standard error of the ssh client. Other commands are not changed. The file is deleted when the channel closes.
 
 ### Modal data channel <!-- id: modal-data-channel -->
 
@@ -1034,6 +1034,60 @@ Each runtime also reports `price_type`, as [Price type](#elice-price-type) descr
 
 `status()` describes this process only. A session started by a different process is not in it, since the pool lives in the process that owns it. What a machine itself is doing is a different question, answered by `letify utilization`.
 
+## Controlling a running call <!-- id: run-control -->
+
+> A running call can be interrupted, inspected and changed from outside, the way a local process can, without the declaration growing a handle.
+
+A call on a remote runtime is a program running somewhere else. On a laptop the person who started a training loop can press Ctrl-C, attach a debugger or read a variable; letify gives the same reach to a call, so a run that turns out to save no checkpoint can still be saved. None of this adds a value to user code: a call is addressed from the command line by a run id, the way a local process is addressed by its pid, and the session lifetime in [Lifetime](#lifetime) is unchanged.
+
+### Run ids and the registry
+
+Every call on a runtime gets a run id, 12 lowercase hex characters, when it is sent. The owning process writes `~/.letify/runs/<run id>.json` before the call is sent and removes it when the call returns or raises. The record holds `run`, `function` (the qualified name), `runtime`, `provider` (the account alias), `started_at` (Unix seconds), `pid` (the owning process) and `socket`, the owner's control socket. The directory is created with mode 0700 and each file with mode 0600.
+
+Each owning process listens on one Unix socket, `~/.letify/runs/<pid>.sock`, created on the first call and removed at exit. A command reaches a run by reading its record and connecting to that socket; the owner forwards the request to the runtime over the call's own channel. When `~/.letify/runs/<pid>.sock` would exceed 100 bytes, the limit of a Unix socket path, the socket is created as `run-<pid>.sock` in the SSH control directory of [Declared filesystem paths](#declared-paths) instead, and `socket` in the record names it. A record whose `pid` no longer exists is stale: it is removed when read and the run is reported as not found. On Windows there is no control socket, and the commands below say so.
+
+The request travels as one JSON object per line on the socket and the answer as one JSON object per line, so a tool that is not letify can drive it.
+
+### Commands
+
+Every command takes `--json`, which prints one JSON object instead of text. The exit status is 0 on success, 2 for a usage error, 3 when the run is not found, and 1 for any other failure.
+
+- `letify runs` lists the live runs of this machine: run id, function, runtime, provider, age and owner pid.
+- `letify interrupt <run>` raises `KeyboardInterrupt` in the call's body on the runtime, in the thread that runs it, as Ctrl-C does in a local process. The body's `except` and `finally` blocks run, so the call ends the way an interrupted local function ends: its outputs are written back as for any call that raises, and the caller sees the exception. A body that catches it keeps running.
+- `letify exec <run> CODE` or `letify exec <run> -f FILE` runs Python source inside the worker process, on a thread of its own, while the body keeps running. The source sees these names:
+  - `frame`, the innermost frame of the body's thread whose code is not in the standard library, site-packages or letify, so it is the user's own loop;
+  - `frames`, every such frame from the outermost to the innermost;
+  - `L`, a mapping over `frame.f_locals` that records the names the source assigns, so the answer can say whether each was applied, and `G`, `frame.f_globals`.
+
+  When the source is a single expression the answer carries the `repr` of its value. The answer always carries what the source printed on stdout and stderr, and a traceback when it raised. `--timeout S`, 60 by default, bounds the wait for the answer, not the thread. The body is not paused, so the source sees the body's objects as they are while it runs. Mutating an object, such as `L["config"].save_every = 100`, reaches the running loop on every Python. Assigning a name, such as `L["save_every"] = 100`, reaches the running frame only on Python 3.13 and later, where `f_locals` writes through; on an earlier Python the answer lists the assigned names and says that they were not applied. A file written by the source lives on the runtime: calling `letify.stash(name, path)` in the source moves it to the Drive store, as from the body, so a checkpoint can be rescued from a runtime that may not survive.
+
+A command against a run on a provider whose worker has no persistent channel, the one-shot channels, fails with `UnsupportedMode` naming the provider.
+
+### Taking over a running call <!-- id: run-takeover -->
+
+> `letify takeover <run>` makes the process that runs it the owner of a running call and its session, so a training run outlives a stuck or dying owner without anything outliving its owner.
+
+A call has exactly one owner at a time, and the session still ends with the call that needed it. Taking over moves that ownership to another process; it does not let a call run with no owner. The new owner renews the lease, receives the body's output and the call's result, writes the call's outputs back, and ends the session when the call ends, as the first owner would have.
+
+1. **The registry keeps an orphan.** The record also holds `project`, the project root the call was sent from, and `lease_deadline`, the Unix time at which the runtime's lease ends unless renewed, rewritten at each renewal. It also holds what a second process needs to reach the call and finish it: `door`, the path of the worker's socket below; `config`, the project `config.toml` that names the account; and `call_dir`, `blobs` and `outputs`, the runtime paths and the output locations to write back, each location stored relative to `project` when it lies inside it. No secret is among them. A record whose owner pid is gone is no longer removed when read while its `lease_deadline` is in the future: `letify runs` lists it with `owner` `gone` and the seconds left, and it can be taken over. After the deadline it is removed as stale.
+2. **The worker keeps a second door.** While a call runs, the worker listens on a Unix socket on the runtime, `<workspace root>/.letify/run-<run id>.sock`, mode 0600. When that path is longer than 100 bytes, the limit of a Unix socket address, the worker and the relay reach the socket through `/proc/self/fd/<n>`, a descriptor of its directory, so it stays at that path; where `/proc` is missing the worker reports that the run has no second door. A channel closing while a call runs does not end the worker: the body keeps running, as it does today, until the lease deadline passes. Output the body makes while no channel is open is kept, the last 256 KiB per stream, and is sent to the next owner first. A body that returns while no channel is open keeps its result for a new owner until the lease deadline; on a provider with no lease the worker ends then, as before.
+3. **The new owner connects.** `letify takeover <run>` reads the record, opens a channel to the same runtime through the record's provider and runtime name, and starts a relay to that socket instead of a new worker. It then sends `adopt` with the run id. The worker answers once with the call's state, which is `running` or `finished`, and from then on sends the call's output and its reply on the new channel, the reply as the last piece of the same answer. The old channel, if still open, is told `moved` on the call's stream and answers every later request on it with `moved`. A door takes one connection at a time and no request but `adopt` before it, and a second owner can adopt after the first one's connection has closed.
+4. **The old owner lets go.** When the old owner is alive, takeover first asks it through its control socket to release the run. Its waiting call raises `OwnershipMoved` naming the run and the new owner's pid, it stops renewing that runtime's lease, and it drops the runtime from its pool without stopping it. A `keep_alive` block in the old owner no longer holds that session. An old owner that does not answer within 5 s is treated as gone.
+5. **The call ends at the new owner.** `letify takeover` prints the body's output as it arrives. When the call returns it prints the `repr` of the value, or with `--result PATH` writes the value with pickle to `PATH`, and exits 0. When the body raises it prints the traceback and exits 1. With `--json` the last line is one object with `run`, `ok`, `value_repr` or `error`, and `result_path`. The call's outputs are written back under the record's `project`, not the current directory. The session is then ended.
+6. **The same from Python.** `letify.takeover(run)` does the same in the calling process and returns the value or raises the body's exception, as a call does. It holds no handle: it returns once the call has ended and the session is gone. A darkpyonix kernel uses this to adopt a run started elsewhere.
+
+A provider whose runtime cannot be reached by a second process refuses with `UnsupportedMode` naming it. On the `Local` provider the runtime is this machine, so the new owner connects to the socket directly, but the worker is a child of the owning process and dies with it: a `Local` run can be taken over only from an owner that is still alive.
+
+### Ctrl-C in the owning process
+
+The first `KeyboardInterrupt` that reaches a thread waiting for a call's reply sends the same interrupt as `letify interrupt` and keeps waiting, so the body's handlers run and the outputs come back before the call raises. The waiting thread prints `letify: interrupting <run>; press Ctrl-C again to stop waiting`. A second `KeyboardInterrupt` during that wait stops waiting at once and propagates, as before this section existed. No signal handler is installed for this, so a notebook kernel's interrupt behaves the same way.
+
+### Exit hooks
+
+`letify.on_exit(fn)`, called inside a body, registers `fn` to run when the worker is about to end while that body is still running, for a reason other than the body returning or raising. The reasons are the worker receiving `SIGTERM` (a preempted session, a provider stopping it), the lease deadline passing, and the channel closing while a call runs. Hooks run in the reverse order of registration, each at most once, on the body's process, with 120 s in total; a hook that raises is reported on stderr and the next one runs. Hooks registered by a call are dropped when that call ends. `SIGKILL`, the out of memory killer and a power cut run no code at all, so no hook runs for them; checkpoints are what covers those.
+
+Outside a body, `letify.on_exit` raises `UnsupportedMode`, because a local process has `atexit` already.
+
 ## Storage
 
 > A volume is a content addressed blob store on whichever backend a provider has. Each provider holds one volume per project, filled automatically from what a call uses, so a declaration never names one.
@@ -2102,6 +2156,7 @@ Local filesystem modifications are consolidated under the single root `~/.letify
 | `~/.letify/cache/tools/<alias>/` | Redirected `HOME` and `USERPROFILE` for provider CLIs (`colab`, `kaggle`). Mode 0700. Keeps CLI temporary files, packages, and `.config/` out of the credential directory. The CLI finds its token via a symbolic link pointing to `~/.letify/accounts/<alias>/token.json` (or copy fallback on systems without symlink support) | `LETIFY_HOME` |
 | `~/.letify/tmp/` | Temporary directory for ephemeral keys during reverse SSH negotiation and adapter stderr files. Mode 0700 | `LETIFY_HOME` |
 | `~/.letify/ssh/` | OpenSSH multiplexing control sockets (`ControlPath`). Mode 0700 | `XDG_RUNTIME_DIR` (moved to `$XDG_RUNTIME_DIR/letify`), `LETIFY_HOME` (with `XDG_RUNTIME_DIR`: `$XDG_RUNTIME_DIR/letify-<h>`; without it: `$LETIFY_HOME/ssh`) |
+| `~/.letify/runs/` | One record `<run id>.json` per running call and one control socket `<pid>.sock` per owning process, as [Controlling a running call](#run-control) and [Taking over a running call](#run-takeover) describe. Mode 0700 directory, mode 0600 records | `LETIFY_HOME` |
 | `~/.letify/runtime/` | Local provider workspace root for calls and local volumes. Holds `data/blobs`, `data/calls`, `tmp`. Falls back to reading `~/.letify-runtime` if it exists | `LETIFY_HOME` |
 
 **Documented local exceptions:**
@@ -2127,6 +2182,7 @@ Every remote write is constrained to the designated workspace root (`~/.letify-r
 | `<workspace root>/blobs` | Argument blobs on persistent providers | account `workspace` setting |
 | `<workspace root>/modules` | Shipped module sources placed on the runtime and added to sys.path | account `workspace` setting |
 | `<workspace root>/tmp/letify-<hex>` | Ephemeral reverse SSH key directory on remote instance | account `workspace` setting |
+| `<workspace root>/.letify/run-<run id>.sock` | The second door of a running call, mode 0600 in a mode 0700 directory, removed when the call's reply has been delivered. [Taking over a running call](#run-takeover) | account `workspace` setting |
 
 **Documented remote exceptions:**
 
@@ -2247,7 +2303,7 @@ So `letify login shell` sets up key authentication and treats the password as a
 4. The connection is confirmed with `BatchMode=yes`, which proves the key works before the alias is declared rather than at the first call.
 5. The machine's GPUs are detected over that confirmed connection, as described under Recording devices at login.
 
-Every SSH command letify builds also carries `-o ControlMaster=auto`, `-o ControlPersist=60` and `-o ControlPath=<control directory>/<tag>-%C`, so the worker channel, the busy card check and every later command to the same machine share one authenticated connection, instead of paying about 0.24 s for a new one each. On Windows, where OpenSSH does not implement connection multiplexing, the three options are left out and each command opens its own connection.
+Every SSH command letify builds also carries `-o ControlMaster=auto`, `-o ControlPersist=60` and `-o ControlPath=<control directory>/<tag>-%C`, so the worker channel, the busy card check and every later command to the same machine share one authenticated connection, instead of paying about 0.24 s for a new one each. The worker channel's ssh and every transfer stream's ssh are the exception: `sshopts.unshared` removes the three options and puts `-o ControlMaster=no -o ControlPath=none` after the program, so each runs as its own connection. A detached master has no standard error and writes no `-E` file, so a client of it that loses the connection exits 255 and prints nothing; a connection of its own prints the cause. A command over a punched link (`PunchedLink`) carries none of the three options, because its local forwarding port is new for every command, so no two commands could share a connection and each would leave a master holding a punched socket for 60 s. On Windows, where OpenSSH does not implement connection multiplexing, the three options are left out and each command opens its own connection.
 
 Control sockets live in a designated directory checked before use, constrained by the Unix domain socket path limit of 108 bytes on Linux and 104 bytes on macOS, including the terminating byte:
 

{letify-2.8.7 → letify-2.9.0}/docs/guide/05-concurrency.md

@@ -132,6 +132,53 @@ Confirm the final candidate on the bigger card.
 
 > ⚠️ This does not apply when you need a specific capability. NVFP4 needs Blackwell tensor cores, which on Colab means `G4` and nothing else. There is no cheaper substitute.
 
+## Reaching into a running call
+
+A call is a program running somewhere else, and you can still do to it what you would do to a local process. Every call gets a 12 character run id while it runs. `letify runs` lists the live calls of the machine:
+
+```bash
+letify runs
+letify interrupt 3f9a0c61d2b4                   # raise KeyboardInterrupt in the body, as Ctrl-C does
+letify exec 3f9a0c61d2b4 "L['step']"            # read a local variable of the running loop
+letify exec 3f9a0c61d2b4 "L['config'].save_every = 100"
+letify exec 3f9a0c61d2b4 -f rescue.py --timeout 300
+```
+
+`interrupt` runs the body's `except` and `finally` blocks, so a loop that saves in a `finally` saves. Pressing Ctrl-C in the process that made the call does the same once, then waits for the outputs. A second Ctrl-C stops waiting.
+
+`exec` runs your source on a thread of the worker while the body keeps running. It sees `frame`, `frames`, `L` (the locals of `frame`) and `G` (its globals), where `frame` is the innermost frame of your own code. A single expression answers with its `repr`. Changing an object, as in `L["config"].save_every = 100`, reaches the loop on every Python. Assigning a name, as in `L["step"] = 0`, reaches it only on Python 3.13 and later, and the answer says so on an earlier one. A file the source writes is on the runtime, so `import letify; letify.stash("ckpt", "model.pt")` in the source moves it to the Drive store before the runtime goes away.
+
+To save state when the runtime is taken away, register a hook inside the body:
+
+```python
+@let.function(device=colab.G4, host=letify.remote)
+def train():
+    letify.on_exit(lambda: letify.stash("ckpt", "model.pt"))
+    ...
+```
+
+The hook runs when the worker receives `SIGTERM`, when its lease runs out, or when the channel closes during the call. It does not run for `SIGKILL` or the out of memory killer, so keep writing checkpoints as well. Hooks run newest first and share 120 seconds.
+
+None of this works on a provider whose worker has only a one-shot channel, and the commands say which provider. There is no control socket on Windows. See the spec section "Controlling a running call".
+
+### Handing a call to another process
+
+A call has one owner: the process that sent it, which renews the runtime's lease, receives the output and the result, and ends the session. When that process is stuck, or you want to close the notebook that started it, move the call to another process:
+
+```bash
+letify takeover 3f9a0c61d2b4                    # prints the body's output, then the repr of the value
+letify takeover 3f9a0c61d2b4 --result out.pkl   # writes the value with pickle instead
+letify takeover 3f9a0c61d2b4 --json             # the last line is one JSON object: run, ok, value_repr or error, result_path
+```
+
+The old owner is asked to let go. Its waiting call raises `OwnershipMoved`, which names the run and the new owner and is never retried. It stops renewing the lease and leaves the session alone, even inside a `keep_alive` block. If it does not answer within 5 seconds it is treated as gone. The new owner prints the output from then on, writes the files the call produced under the project the call was sent from, not the current directory, and ends the session when the call ends. The exit status is 0 when the call returned, 1 when its body raised (the traceback is printed) or the run could not be taken, and 3 when no such run exists.
+
+From Python, `value = letify.takeover("3f9a0c61d2b4")` does the same in the calling process and raises the body's exception as a call does.
+
+If the owner died, `letify runs` still lists the call while the runtime's lease has time left, with `gone` as its owner and the seconds left. Take it over before the lease ends, or the worker ends with it. Output the body made while nobody listened is kept (the last 256 KiB) and printed first.
+
+A provider whose runtime a second process cannot reach raises `UnsupportedMode` naming it. Local and Shell can be taken over. A Local worker is a child of its owner and has no lease, so a Local call can be taken over only while its owner is alive. Kaggle, Modal and Elice cannot be taken over, and Colab can only on its default SSH channel. See the spec section "Taking over a running call".
+
 ## Practical patterns
 
 **Run the configurations, collect a table.**

{letify-2.8.7 → letify-2.9.0}/docs/guide/ko/05-/353/217/231/354/213/234-/354/213/244/355/226/211.md

@@ -128,6 +128,53 @@ L4 세 개가 A100 하나보다 시간당 싸면서, A100 하나가 조합 하
 
 > ⚠️ 특정 기능이 필요할 때는 해당되지 않습니다. NVFP4는 블랙웰 텐서 코어가 필요하고, Colab에서는 `G4`뿐입니다. 더 싼 대체가 없습니다.
 
+## 실행 중인 호출에 손대기
+
+호출은 다른 곳에서 돌아가는 프로그램이지만, 로컬 프로세스에 하던 일을 그대로 할 수 있습니다. 호출은 실행되는 동안 12자리 run id를 갖습니다. `letify runs`는 이 머신에서 살아 있는 호출을 보여 줍니다.
+
+```bash
+letify runs
+letify interrupt 3f9a0c61d2b4                   # Ctrl-C처럼 본문에 KeyboardInterrupt를 일으킴
+letify exec 3f9a0c61d2b4 "L['step']"            # 돌고 있는 루프의 지역 변수 읽기
+letify exec 3f9a0c61d2b4 "L['config'].save_every = 100"
+letify exec 3f9a0c61d2b4 -f rescue.py --timeout 300
+```
+
+`interrupt`는 본문의 `except`와 `finally` 블록을 실행하므로, `finally`에서 저장하는 루프는 저장하고 끝납니다. 호출을 만든 프로세스에서 Ctrl-C를 한 번 누르면 같은 일을 한 뒤 출력이 돌아오기를 기다립니다. 두 번째 Ctrl-C는 기다리기를 멈춥니다.
+
+`exec`는 본문이 계속 도는 동안 워커의 별도 스레드에서 소스를 실행합니다. 소스에서는 `frame`, `frames`, `L`(`frame`의 지역 변수), `G`(전역 변수)가 보이며, `frame`은 사용자 코드의 가장 안쪽 프레임입니다. 식 하나이면 그 값의 `repr`을 답으로 돌려줍니다. `L["config"].save_every = 100`처럼 객체를 바꾸면 어느 파이썬에서나 루프에 반영됩니다. `L["step"] = 0`처럼 이름에 대입하면 파이썬 3.13 이상에서만 반영되고, 더 낮은 버전에서는 답에 반영되지 않았다고 나옵니다. 소스가 쓴 파일은 런타임에 있으므로, 소스에서 `import letify; letify.stash("ckpt", "model.pt")`를 부르면 런타임이 사라지기 전에 Drive 저장소로 옮길 수 있습니다.
+
+런타임을 빼앗길 때 상태를 저장하려면 본문 안에서 훅을 등록합니다.
+
+```python
+@let.function(device=colab.G4, host=letify.remote)
+def train():
+    letify.on_exit(lambda: letify.stash("ckpt", "model.pt"))
+    ...
+```
+
+훅은 워커가 `SIGTERM`을 받을 때, 리스 기한이 지날 때, 호출 중에 채널이 닫힐 때 실행됩니다. `SIGKILL`이나 메모리 부족 종료에는 실행되지 않으므로 체크포인트도 계속 쓰세요. 훅은 나중에 등록한 것부터 실행되며 합쳐서 120초를 씁니다.
+
+워커에 일회성 채널만 있는 프로바이더에서는 이 기능을 쓸 수 없고, 명령이 그 프로바이더 이름을 알려 줍니다. Windows에는 제어 소켓이 없습니다. 스펙의 "Controlling a running call" 절을 참고하세요.
+
+### 호출을 다른 프로세스에 넘기기
+
+호출의 소유자는 하나입니다. 호출을 보낸 프로세스가 런타임의 리스를 갱신하고, 출력과 결과를 받고, 세션을 끝냅니다. 그 프로세스가 멈췄거나 호출을 시작한 노트북을 닫고 싶으면 호출을 다른 프로세스로 옮깁니다.
+
+```bash
+letify takeover 3f9a0c61d2b4                    # 본문의 출력을 보여 준 뒤 반환값의 repr를 출력
+letify takeover 3f9a0c61d2b4 --result out.pkl   # 반환값을 pickle로 파일에 저장
+letify takeover 3f9a0c61d2b4 --json             # 마지막 줄이 JSON 객체 하나: run, ok, value_repr 또는 error, result_path
+```
+
+이전 소유자에게 놓아 달라고 요청합니다. 기다리던 이전 소유자의 호출은 `OwnershipMoved`를 일으키고, 이 예외는 run id와 새 소유자를 알려 주며 재시도되지 않습니다. 이전 소유자는 리스 갱신을 멈추고, `keep_alive` 블록 안에서도 세션을 건드리지 않습니다. 5초 안에 답하지 않으면 사라진 것으로 봅니다. 새 소유자는 그 시점부터 출력을 보여 주고, 호출이 만든 파일을 현재 디렉터리가 아니라 호출을 보낸 프로젝트 아래에 쓰고, 호출이 끝나면 세션을 끝냅니다. 종료 코드는 호출이 반환하면 0, 본문이 예외를 일으키거나(트레이스백을 출력합니다) 호출을 넘겨받지 못하면 1, 그런 run이 없으면 3입니다.
+
+파이썬에서는 `value = letify.takeover("3f9a0c61d2b4")`가 호출 프로세스에서 같은 일을 하며, 호출처럼 본문의 예외를 그대로 일으킵니다.
+
+소유자가 죽었더라도 런타임의 리스가 남아 있는 동안은 `letify runs`가 그 호출을 소유자 `gone`과 남은 초와 함께 보여 줍니다. 리스가 끝나기 전에 넘겨받지 않으면 워커도 함께 끝납니다. 아무도 듣지 않는 동안 본문이 낸 출력은 마지막 256 KiB까지 보관했다가 먼저 출력합니다.
+
+다른 프로세스가 런타임에 닿을 수 없는 프로바이더에서는 그 이름과 함께 `UnsupportedMode`가 일어납니다. Local과 Shell은 넘겨받을 수 있습니다. Local의 워커는 소유자의 자식 프로세스이고 리스가 없으므로, Local 호출은 소유자가 살아 있는 동안에만 넘겨받을 수 있습니다. Kaggle, Modal, Elice는 넘겨받을 수 없고, Colab은 기본 SSH 채널에서만 가능합니다. 스펙의 "Taking over a running call" 절을 참고하세요.
+
 ## 실용 패턴
 
 **설정을 돌려 표로 모으기.**

{letify-2.8.7 → letify-2.9.0}/letify/__init__.py

@@ -33,6 +33,7 @@ from typing import Final, overload
 
 from .declare.cache import session_cache
 from .declare.env import Env
+from .declare.exit import on_exit
 from .declare.function import Function
 from .declare.instance import AnyInstance, Host, Instance
 from .declare.reduce import all_reduce, mean, sum
@@ -43,6 +44,7 @@ from .errors import (
     InsufficientDevices,
     InterpreterMismatch,
     LetifyError,
+    OwnershipMoved,
     ProtocolError,
     ProviderUnavailable,
     ReductionFailed,
@@ -68,7 +70,7 @@ distributed: Final = Host.distributed
 # The enum class stays importable from letify.declare.instance for letify's own use.
 del Host
 
-__version__ = "2.8.7"
+__version__ = "2.9.0"
 
 
 def fetch(tensor):  # type: ignore[no-untyped-def]
@@ -93,6 +95,22 @@ def stash(name: str, path: str | os.PathLike[str]) -> str:
     return stash_handoff(name, path)
 
 
+def takeover(run: str, *, launcher: Launcher | None = None) -> object:
+    """Become the owner of the running call ``run`` and return its value.
+
+    Spec "Taking over a running call". The previous owner, if it is alive, is asked to let go
+    and its waiting call raises ``OwnershipMoved``. This process then renews the runtime's
+    lease, prints the body's output as it arrives, writes the call's outputs back under the
+    project the call was sent from, ends the session, and returns the value or raises the
+    body's exception as a call does. It holds no handle and returns only once the call has
+    ended and the session is gone. ``launcher`` supplies the accounts when this process
+    already has one; otherwise they are read from the configuration the run was started with.
+    """
+    from .runtime.takeover import take
+
+    return take(run, launcher=launcher)
+
+
 @overload
 def unstash(name: str) -> dict[str, bytes] | None: ...
 @overload
@@ -131,6 +149,7 @@ __all__ = [
     "InterpreterMismatch",
     "Launcher",
     "LetifyError",
+    "OwnershipMoved",
     "ProtocolError",
     "ProviderUnavailable",
     "Providers",
@@ -151,11 +170,13 @@ __all__ = [
     "local",
     "local_steps",
     "mean",
+    "on_exit",
     "rank",
     "remote",
     "session_cache",
     "stash",
     "sum",
+    "takeover",
     "unstash",
     "world_size",
 ]

{letify-2.8.7 → letify-2.9.0}/letify/cli.py

@@ -12,6 +12,8 @@ import json
 import os
 import shutil
 import sys
+import time
+from collections.abc import Callable
 from pathlib import Path
 
 from . import __version__, render
@@ -191,6 +193,39 @@ def build_parser() -> argparse.ArgumentParser:
     efficiency.add_argument("round_trip_ms", type=float, help="network round trip")
     efficiency.add_argument("--json", action="store_true", help="print the record unformatted")
 
+    runs_parser = sub.add_parser("runs", help="list the calls running from this machine")
+    runs_parser.add_argument("--json", action="store_true", help="print the records unformatted")
+    interrupt_parser = sub.add_parser(
+        "interrupt", help="raise KeyboardInterrupt in a running call's body"
+    )
+    interrupt_parser.add_argument("run", help="a run id, as letify runs lists it")
+    interrupt_parser.add_argument(
+        "--json", action="store_true", help="print the record unformatted"
+    )
+    exec_parser = sub.add_parser("exec", help="run Python source inside a running call's worker")
+    exec_parser.add_argument("run", help="a run id, as letify runs lists it")
+    exec_parser.add_argument("code", nargs="?", help="the source to run")
+    exec_parser.add_argument("-f", "--file", help="read the source from this file instead")
+    exec_parser.add_argument(
+        "--timeout",
+        type=float,
+        default=60.0,
+        metavar="S",
+        help="seconds to wait for the answer; the source keeps running after that",
+    )
+    exec_parser.add_argument("--json", action="store_true", help="print the record unformatted")
+
+    takeover_parser = sub.add_parser(
+        "takeover", help="become the owner of a running call and see it through"
+    )
+    takeover_parser.add_argument("run", help="a run id, as letify runs lists it")
+    takeover_parser.add_argument(
+        "--result", metavar="PATH", help="write the call's value to PATH with pickle"
+    )
+    takeover_parser.add_argument(
+        "--json", action="store_true", help="end with one JSON object about the call"
+    )
+
     setup_parser = sub.add_parser(
         "setup", help="install tailcat or eci from its publisher ahead of time"
     )
@@ -593,6 +628,164 @@ def _confirmed(question: str) -> bool:
     return answer in ("y", "yes")
 
 
+def _run_control(args: argparse.Namespace) -> int:
+    """``letify runs``, ``interrupt`` and ``exec``: spec "Controlling a running call".
+
+    Exit status 0 on success, 2 for a usage error, 3 when the run is not found and 1 for
+    any other failure.
+    """
+    from .errors import RunNotFound, UnsupportedMode
+    from .runtime import control
+
+    def refuse(message: str, status: int, kind: str = "Error") -> int:
+        if args.json:
+            _json({"ok": False, "type": kind, "error": message})
+        else:
+            _fail(message)
+        return status
+
+    if args.command == "runs":
+        if not control.available():
+            return refuse("run control is not available on Windows", 1, "UnsupportedMode")
+        rows = control.list_runs(orphans=True)
+        if args.json:
+            return _json(rows)
+        if not rows:
+            print("no calls are running")
+            return 0
+        now = time.time()
+        table = [("RUN", "FUNCTION", "RUNTIME", "PROVIDER", "AGE", "OWNER")] + [
+            (
+                str(row["run"]),
+                str(row.get("function", "")),
+                str(row.get("runtime", "")),
+                str(row.get("provider", "")),
+                _age(now - float(row.get("started_at", now))),
+                f"gone, {_age(row['seconds_left'])} left"
+                if row.get("owner") == "gone"
+                else str(row.get("pid", "")),
+            )
+            for row in rows
+        ]
+        widths = [max(len(line[column]) for line in table) for column in range(6)]
+        for line in table:
+            cells = (cell.ljust(width) for cell, width in zip(line, widths, strict=True))
+            print("  ".join(cells).rstrip())
+        return 0
+
+    if args.command == "takeover":
+        return _takeover(args, refuse)
+
+    command: dict[str, object] = {"cmd": args.command}
+    wait = 90.0
+    if args.command == "exec":
+        if (args.code is None) == (args.file is None):
+            return refuse("give the source as CODE or with -f FILE, not both and not neither", 2)
+        if args.timeout <= 0:
+            return refuse("--timeout must be positive", 2)
+        try:
+            source = args.code if args.code is not None else Path(args.file).read_text("utf-8")
+        except (OSError, UnicodeDecodeError) as exc:
+            return refuse(f"cannot read {args.file}: {exc}", 1)
+        command.update({"source": source, "timeout": args.timeout})
+        wait = args.timeout + 30
+    try:
+        answer = control.ask(args.run, command, wait=wait)
+    except RunNotFound as exc:
+        return refuse(str(exc), 3, "RunNotFound")
+    except UnsupportedMode as exc:
+        return refuse(str(exc), 1, "UnsupportedMode")
+    failed = not answer.get("ok")
+    if args.command == "exec" and answer.get("error"):
+        failed = True
+    if args.json:
+        _json(answer)
+        return 1 if failed else 0
+    if not answer.get("ok"):
+        kind = answer.get("type") or "Error"
+        reason = answer.get("error") or answer.get("reason") or "failed"
+        _fail(f"{kind}: {reason}" if kind != "Error" else str(reason))
+        return 1
+    if args.command == "interrupt":
+        if not answer.get("interrupted"):
+            _fail(f"{args.run} was not interrupted: {answer.get('reason', 'unknown reason')}")
+            return 1
+        _say("ok", f"{args.run} interrupted")
+        return 0
+    sys.stdout.write(str(answer.get("stdout") or ""))
+    sys.stderr.write(str(answer.get("stderr") or ""))
+    if answer.get("error"):
+        sys.stderr.write(str(answer["error"]))
+        return 1
+    if answer.get("has_value"):
+        print(answer["value"])
+    assigned = answer.get("assigned") or []
+    if assigned and not answer.get("applied"):
+        _fail(
+            "this Python does not write assignments to f_locals back into the running "
+            f"frame, so {', '.join(map(str, assigned))} was not applied"
+        )
+    return 0
+
+
+def _takeover(args: argparse.Namespace, refuse: Callable[..., int]) -> int:
+    """``letify takeover``: spec "Taking over a running call", item 5.
+
+    Exit status 0 when the call returned, 1 when its body raised or it could not be taken, 2
+    for a usage error and 3 when no such run is waiting for an owner.
+    """
+    import pickle
+
+    from .errors import RemoteError, RunNotFound, UnsupportedMode
+    from .runtime import control
+    from .runtime.takeover import take
+
+    if not control.available():
+        return refuse("run control is not available on Windows", 1, "UnsupportedMode")
+    try:
+        value = take(args.run)
+    except RunNotFound as exc:
+        return refuse(str(exc), 3, "RunNotFound")
+    except UnsupportedMode as exc:
+        return refuse(str(exc), 1, "UnsupportedMode")
+    except RemoteError as exc:
+        sys.stderr.write(f"{exc}\n")
+        if args.json:
+            _json_line({"run": args.run, "ok": False, "error": str(exc)})
+        return 1
+    result = None
+    if args.result:
+        result = str(args.result)
+        try:
+            Path(result).write_bytes(pickle.dumps(value))
+        except (OSError, pickle.PicklingError, TypeError) as exc:
+            return refuse(f"the value of {args.run} cannot be written to {result}: {exc}", 1)
+    if args.json:
+        _json_line(
+            {"run": args.run, "ok": True, "value_repr": repr(value), "result_path": result}
+        )
+    elif result is None:
+        print(repr(value))
+    else:
+        _say("ok", f"{args.run} returned; its value is in {result}")
+    return 0
+
+
+def _json_line(value: object) -> None:
+    """One JSON object on one line, which is how a program reads the end of a stream."""
+    print(json.dumps(value))
+    sys.stdout.flush()
+
+
+def _age(seconds: float) -> str:
+    seconds = max(0, int(seconds))
+    if seconds < 60:
+        return f"{seconds}s"
+    if seconds < 3600:
+        return f"{seconds // 60}m{seconds % 60:02d}s"
+    return f"{seconds // 3600}h{seconds % 3600 // 60:02d}m"
+
+
 def _json(value: object) -> int:
     print(json.dumps(value, indent=2))
     return 0
@@ -620,6 +813,10 @@ def _dispatch(args: argparse.Namespace) -> int:
     if args.command == "setup":
         return _setup_tool(args)
 
+    if args.command in ("runs", "interrupt", "exec", "takeover"):
+        # Talks to the process that owns the run, so no Launcher is built.
+        return _run_control(args)
+
     if args.command == "client":
         # Runs on the remote machine, which has no accounts, so no Launcher is built.
         return _client_shell_connect(args)

{letify-2.8.7 → letify-2.9.0}/letify/config/__init__.py

@@ -56,6 +56,8 @@ def load(path: str | Path | None = None, *, home: bool = True) -> Config:
 
     home_entries = _read_entries(home_path, config, require_kind=True) if home else {}
     project_entries = _read_entries(project_path, config, require_kind=False)
+    if project_path.is_file():
+        config.project_file = project_path.resolve()
 
     counter = 0
 

{letify-2.8.7 → letify-2.9.0}/letify/config/schema.py

@@ -42,6 +42,9 @@ class Config:
     providers: dict[str, ProviderConfig] = field(default_factory=dict)
     defaults: dict[str, Any] = field(default_factory=dict)
     sources: list[Path] = field(default_factory=list)
+    #: The project's own ``config.toml`` when it exists, which a run's record names so another
+    #: process can find the same accounts. Spec "Taking over a running call".
+    project_file: Path | None = None
 
     @property
     def order(self) -> list[str]:

letify-2.9.0/letify/declare/exit.py

@@ -0,0 +1,50 @@
+"""``letify.on_exit``: a function to run when a body's worker is about to end.
+
+Owns the public registration call and the errors for where it cannot work. It does not own
+when hooks run, which is the worker's (``letify.protocol.worker``), nor how they are ordered
+and bounded, which spec "Controlling a running call", Exit hooks, states.
+"""
+
+from __future__ import annotations
+
+import os
+import sys
+from collections.abc import Callable
+
+from ..errors import UnsupportedMode
+
+
+def on_exit(fn: Callable[[], object]) -> None:
+    """Run ``fn`` if the worker ends while this body is still running.
+
+    Call it inside a body. ``fn`` runs, with no arguments, when the worker receives
+    ``SIGTERM``, when its lease deadline passes, or when the channel closes during the call.
+    Hooks run newest first, each once, within 120 s in total, and are dropped when the call
+    ends. ``SIGKILL`` and the out of memory killer run no hook. Outside a body this raises
+    ``UnsupportedMode``, because a local process has ``atexit``.
+    """
+    if not callable(fn):
+        raise TypeError(f"letify.on_exit needs a callable, got {type(fn).__name__}")
+    main = sys.modules.get("__main__")
+    register = getattr(main, "_exit_register", None)
+    if register is None:
+        worker_pid = os.environ.get("LETIFY_WORKER_PID")
+        if worker_pid is not None and worker_pid != str(os.getpid()):
+            raise UnsupportedMode(
+                "letify.on_exit must be called in the process letify runs the body in, not in "
+                f"a child process it started (a child of worker process {worker_pid})."
+            )
+        if os.environ.get("LETIFY_ONE_SHOT") is not None:
+            raise UnsupportedMode(
+                "letify.on_exit needs a persistent channel, and this body runs on a one-shot "
+                "channel, which ends its process when the call does."
+            )
+        raise UnsupportedMode(
+            "letify.on_exit registers a hook inside a body running on a runtime. "
+            "In a local process use atexit."
+        )
+    if not register(fn):
+        raise UnsupportedMode(
+            "letify.on_exit was called while no body is running, so there is no call to "
+            "attach the hook to. In a local process use atexit."
+        )