letify 2.9.0__tar.gz → 2.10.0__tar.gz

{letify-2.9.0 → letify-2.10.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: letify
-Version: 2.9.0
+Version: 2.10.0
 Summary: Declarations that become infrastructure.
 Project-URL: Homepage, https://github.com/DarkPyonix/letify
 Project-URL: Documentation, https://github.com/DarkPyonix/letify/tree/main/docs
@@ -492,6 +492,8 @@ letify setup tailcat          # install tailcat or eci from its publisher ahead
 letify providers              # who is declared, storage, channel kind
 letify devices                # what each one offers
 letify status                 # what is running right now
+letify sessions               # what each account holds, workers on a shell host
+letify stop --orphans         # end what a dead process left behind
 letify usage                  # what is left on each account
 letify utilization            # how busy each instance's GPU is
 letify check lab              # does this machine answer?

{letify-2.9.0 → letify-2.10.0}/README.md

@@ -467,6 +467,8 @@ letify setup tailcat          # install tailcat or eci from its publisher ahead
 letify providers              # who is declared, storage, channel kind
 letify devices                # what each one offers
 letify status                 # what is running right now
+letify sessions               # what each account holds, workers on a shell host
+letify stop --orphans         # end what a dead process left behind
 letify usage                  # what is left on each account
 letify utilization            # how busy each instance's GPU is
 letify check lab              # does this machine answer?

{letify-2.9.0 → letify-2.10.0}/docs/SPEC.md

@@ -320,7 +320,7 @@ letify creates one notebook per run, with `CreateKernelWithSettings`, and never
 
 `Kaggle.sessions()` returns the recorded refs that the account still has, checked against the official CLI, `kaggle kernels list --mine --csv`, authenticated by the account's API token in `KAGGLE_API_TOKEN` and never by the cookie, which keeps the boundary this section states. A ref the record names and the account no longer has was deleted elsewhere, and is dropped from the record rather than reported. This matters because a process that dies before `stop` leaves its notebook and its run behind, and that notebook holds weekly accelerator quota with nothing pointing at it: `letify sessions` is how it is found and `kaggle kernels delete` how it is removed. Discovery needs the API token and uv, and the listing is empty rather than failing when either is missing, when the command exits non-zero, or when its output does not parse, because reading what exists must not fail a command that is asking what exists. A notebook created before letify kept this record is not in it and so is not reported.
 
-Whether an account can be asked at all is the provider's own answer, `discovers_sessions`, rather than the launcher naming a provider kind. `colab` and `kaggle` answer yes; every other kind reports that discovery is not supported, which is what `letify sessions` prints for it.
+Whether an account can be asked at all is the provider's own answer, `discovers_sessions`, rather than the launcher naming a provider kind. `colab`, `kaggle` and `shell` answer yes, the last as [Workers on a shell host](#shell-workers) describes. `tunnel` and `elice` answer no, because reaching their machine runs the connection pipeline or allocates a machine, and listing must do neither; every other kind reports that discovery is not supported, which is what `letify sessions` prints for it.
 
 **Not making sessions faster than Kaggle allows.** Creating and ending interactive sessions in quick succession makes Kaggle refuse the next run, and nothing letify does recovers it except waiting. The refusal has no message: `CommitAndRun` answers without a `kernelRunId`, which letify reports as `Kaggle did not start a session run`.
 
@@ -364,6 +364,24 @@ The accelerators letify offers are the fixed set `discover` returns, `CPU` and `
 
 `Shell` and its subclasses default to ephemeral because a machine's disk policy is not knowable in advance. Assuming ephemeral costs time, since letify rebuilds the environment each runtime and the work still succeeds; assuming persistent fails outright when the disk turns out to be wiped. A configuration entry overrides it with `persistent = true`.
 
+### Provider session discovery <!-- id: provider-session-discovery -->
+
+> `letify sessions` lists what an account already holds, and `letify stop` ends what a dead process left behind, for every provider that can be asked.
+
+Whether a provider can be asked is its own answer, `discovers_sessions`. `sessions()` returns one readable line per session, `orphans()` returns the refs of those whose owner is gone, and `stop_orphan(ref)` ends one and reports whether it is gone. Kaggle and Colab answer through their accounts, as their sections describe. A provider that cannot be asked prints `session discovery is not supported`.
+
+#### Workers on a shell host <!-- id: shell-workers -->
+
+> A `Shell` machine answers by listing the letify workers running on it, because the machine has no account to ask.
+
+1. **The worker records itself.** After the environment is built and the worker has moved to the project interpreter, the client sends the op `register` with the workspace root, the runtime name, the pool key and its own host name and pid. The worker writes `<workspace root>/.letify/workers/<worker pid>.json` and removes it when it ends, including when it ends through the lease or `SIGTERM`. The directory has mode 0700 and the file mode 0600, and the file is replaced whole so a reader never sees half of it.
+2. **The record.** `pid`, `proc_start` (the process start time in clock ticks since boot, field 22 of `/proc/<pid>/stat`, or null where `/proc` is missing), `started` (UTC, ISO 8601), `runtime`, `key`, `devices` (the worker's `CUDA_VISIBLE_DEVICES`, or null when unset), `owner` (`host` and `pid` of the local process that opened the session), `run` (the run id of the call being served, or null) and `lease_deadline` (Unix seconds on the host, rewritten at each lease renewal, or null when no lease is armed). `run` is set when the second door opens and cleared when the call ends. A provider whose class sets `records_workers` to false writes no record. That is every provider but `Shell` and its subclasses reached over a persistent channel.
+3. **Listing.** `Shell.sessions()` runs one program on the host over the account's link. It reads the records, drops each whose pid is not a live worker and deletes that file, and prints the rest. A pid is a live worker when a process with that pid runs and its start time equals `proc_start`, which keeps a reused pid from reading as a worker; with no `proc_start` in the record, when the pid answers signal 0. Each line reads `pid <pid>  gpu <indices or none>  started <time>  runtime <name>  key <key>  run <id or none>  owner <host>:<pid> <alive|gone|remote>`. An unreachable host raises, as every other command on the account does, rather than listing nothing.
+4. **Owner state.** `alive` and `gone` are judged only for an owner whose host is this machine's host name: `gone` when that pid does not run. An owner on another host is `remote` and is never judged, because its liveness cannot be read from here.
+5. **Orphans.** A worker is an orphan when its `lease_deadline` has passed on the host, or when its owner is `gone` and no call is running (`run` is null). A worker whose owner is `gone` while a call runs and whose lease is in the future is not an orphan: it can be taken over, see [Taking over a running call](#run-takeover), and stopping it would end a training run that can still be saved. `letify stop --orphans` never touches a worker whose owner is `alive` or `remote` unless its lease has passed.
+6. **Stopping one.** `Shell.stop_orphan(pid)` and `letify stop --alias <alias> --worker <pid>` run one program on the host that reads the record for that pid, refuses a pid with no record or no live worker, sends `SIGTERM`, waits up to 30 s (`STOP_GRACE_S`) for the worker to end, then sends `SIGKILL`, and answers whether the pid is gone. A pid with no record is refused, so the command cannot end a process that is not a letify worker. `--worker` stops what is named whatever its owner state, so it asks for confirmation unless `-y` is given.
+7. **Colab keeps its own answers.** `Colab` inherits `Shell` for its link, but sets `records_workers` to false and lists, orphans and stops through its CLI as before.
+
 ### Remaining usage
 
 > Every provider is asked what is left on its account, reads the answer from the service itself, and says why when it cannot.
@@ -1070,7 +1088,7 @@ A command against a run on a provider whose worker has no persistent channel, th
 A call has exactly one owner at a time, and the session still ends with the call that needed it. Taking over moves that ownership to another process; it does not let a call run with no owner. The new owner renews the lease, receives the body's output and the call's result, writes the call's outputs back, and ends the session when the call ends, as the first owner would have.
 
 1. **The registry keeps an orphan.** The record also holds `project`, the project root the call was sent from, and `lease_deadline`, the Unix time at which the runtime's lease ends unless renewed, rewritten at each renewal. It also holds what a second process needs to reach the call and finish it: `door`, the path of the worker's socket below; `config`, the project `config.toml` that names the account; and `call_dir`, `blobs` and `outputs`, the runtime paths and the output locations to write back, each location stored relative to `project` when it lies inside it. No secret is among them. A record whose owner pid is gone is no longer removed when read while its `lease_deadline` is in the future: `letify runs` lists it with `owner` `gone` and the seconds left, and it can be taken over. After the deadline it is removed as stale.
-2. **The worker keeps a second door.** While a call runs, the worker listens on a Unix socket on the runtime, `<workspace root>/.letify/run-<run id>.sock`, mode 0600. When that path is longer than 100 bytes, the limit of a Unix socket address, the worker and the relay reach the socket through `/proc/self/fd/<n>`, a descriptor of its directory, so it stays at that path; where `/proc` is missing the worker reports that the run has no second door. A channel closing while a call runs does not end the worker: the body keeps running, as it does today, until the lease deadline passes. Output the body makes while no channel is open is kept, the last 256 KiB per stream, and is sent to the next owner first. A body that returns while no channel is open keeps its result for a new owner until the lease deadline; on a provider with no lease the worker ends then, as before.
+2. **The worker keeps a second door.** While a call runs, the worker listens on a Unix socket on the runtime, `<workspace root>/.letify/run-<run id>.sock`, mode 0600. When that path is longer than 100 bytes, the limit of a Unix socket address, the worker and the relay reach the socket through `/proc/self/fd/<n>`, a descriptor of its directory, so it stays at that path; where `/proc` is missing the worker reports that the run has no second door. A channel that closes while a call runs, with no shutdown frame before it, does not end the worker: the body keeps running until the lease deadline passes. A shutdown frame is different: it is the owner ending the run on purpose, as [A deliberate shutdown](#run-shutdown) describes. Output the body makes while no channel is open is kept, the last 256 KiB per stream, and is sent to the next owner first. A body that returns while no channel is open keeps its result for a new owner until the lease deadline; on a provider with no lease the worker ends then, as before.
 3. **The new owner connects.** `letify takeover <run>` reads the record, opens a channel to the same runtime through the record's provider and runtime name, and starts a relay to that socket instead of a new worker. It then sends `adopt` with the run id. The worker answers once with the call's state, which is `running` or `finished`, and from then on sends the call's output and its reply on the new channel, the reply as the last piece of the same answer. The old channel, if still open, is told `moved` on the call's stream and answers every later request on it with `moved`. A door takes one connection at a time and no request but `adopt` before it, and a second owner can adopt after the first one's connection has closed.
 4. **The old owner lets go.** When the old owner is alive, takeover first asks it through its control socket to release the run. Its waiting call raises `OwnershipMoved` naming the run and the new owner's pid, it stops renewing that runtime's lease, and it drops the runtime from its pool without stopping it. A `keep_alive` block in the old owner no longer holds that session. An old owner that does not answer within 5 s is treated as gone.
 5. **The call ends at the new owner.** `letify takeover` prints the body's output as it arrives. When the call returns it prints the `repr` of the value, or with `--result PATH` writes the value with pickle to `PATH`, and exits 0. When the body raises it prints the traceback and exits 1. With `--json` the last line is one object with `run`, `ok`, `value_repr` or `error`, and `result_path`. The call's outputs are written back under the record's `project`, not the current directory. The session is then ended.
@@ -1078,13 +1096,24 @@ A call has exactly one owner at a time, and the session still ends with the call
 
 A provider whose runtime cannot be reached by a second process refuses with `UnsupportedMode` naming it. On the `Local` provider the runtime is this machine, so the new owner connects to the socket directly, but the worker is a child of the owning process and dies with it: a `Local` run can be taken over only from an owner that is still alive.
 
+### A deliberate shutdown <!-- id: run-shutdown -->
+
+> A shutdown frame from the channel that owns a running call ends the body and the worker, because a lost owner can be replaced and an owner that left on purpose cannot.
+
+`Pool.shutdown`, a session ending and a process exiting normally close a persistent channel by sending a shutdown frame before closing its input. The worker tells that from a lost owner:
+
+1. **A shutdown frame on the owning channel while a body runs** runs the exit hooks with the reason "the owner shut down", then ends the process with status 128 + `SIGTERM`, as a received `SIGTERM` does. No lease wait and no takeover follow. Before it exits, the worker ends the processes the body started (data loader workers, subprocesses). It builds the tree of descendants of its own pid from `/proc/<pid>/stat`, sends each `SIGTERM`, waits up to 5 s (`CHILD_GRACE_S`) for them to end, and sends `SIGKILL` to the survivors. The tree is walked rather than the process group, because a worker started over SSH is usually not a group leader (the remote shell leads the session). Where `/proc` is missing, the worker sends `SIGTERM` to its process group only when it leads that group, and signals no one else.
+2. **End of input without a shutdown frame** keeps the behaviour of [Taking over a running call](#run-takeover): the body runs on until the lease deadline.
+3. **A shutdown frame while no body runs** ends the worker, as before.
+4. **A shutdown frame on a channel the call moved away from** does not end the call. The channel that gave the call up closes with `detach`, which sends no shutdown frame, and a shutdown that does arrive on it is ignored.
+
 ### Ctrl-C in the owning process
 
 The first `KeyboardInterrupt` that reaches a thread waiting for a call's reply sends the same interrupt as `letify interrupt` and keeps waiting, so the body's handlers run and the outputs come back before the call raises. The waiting thread prints `letify: interrupting <run>; press Ctrl-C again to stop waiting`. A second `KeyboardInterrupt` during that wait stops waiting at once and propagates, as before this section existed. No signal handler is installed for this, so a notebook kernel's interrupt behaves the same way.
 
 ### Exit hooks
 
-`letify.on_exit(fn)`, called inside a body, registers `fn` to run when the worker is about to end while that body is still running, for a reason other than the body returning or raising. The reasons are the worker receiving `SIGTERM` (a preempted session, a provider stopping it), the lease deadline passing, and the channel closing while a call runs. Hooks run in the reverse order of registration, each at most once, on the body's process, with 120 s in total; a hook that raises is reported on stderr and the next one runs. Hooks registered by a call are dropped when that call ends. `SIGKILL`, the out of memory killer and a power cut run no code at all, so no hook runs for them; checkpoints are what covers those.
+`letify.on_exit(fn)`, called inside a body, registers `fn` to run when the worker is about to end while that body is still running, for a reason other than the body returning or raising. The reasons are the worker receiving `SIGTERM` (a preempted session, a provider stopping it), the lease deadline passing, the channel closing while a call runs, and the owner shutting down. Hooks run in the reverse order of registration, each at most once, on the body's process, with 120 s in total; a hook that raises is reported on stderr and the next one runs. Hooks registered by a call are dropped when that call ends. `SIGKILL`, the out of memory killer and a power cut run no code at all, so no hook runs for them; checkpoints are what covers those.
 
 Outside a body, `letify.on_exit` raises `UnsupportedMode`, because a local process has `atexit` already.
 
@@ -1304,7 +1333,7 @@ connection, but a call's second large blob is split whenever splitting is faster
 
 A connection that costs something to open has to pay for itself, so a provider reports `transfer_connection_cost_s`, the seconds one more connection takes. It is 0 for a provider that dials an address, and about 10 for one reached by a hole punch, because that is what a punch measures. The streams are opened only when the transfer is expected to finish sooner with them: the estimate uses the probe's upload rate, assumes the streams scale and that their connections are opened at the same time, and keeps one stream when `size / (rate x streams) + cost` is not less than `size / rate`. A measured case: a Kaggle runtime at a 44.7 ms round trip carried 128 MiB at 21.0 MiB/s on one stream and 9.5 MiB/s on four, 2.2 times worse, because three more punches cost more than the transfer saved. The same account at a 193 ms round trip carried 1.57 MiB/s on one raw SSH connection and 6.10 MiB/s on four, so the gain is real where the link is slow enough and the blob large enough for it to be worth 10 s.
 
-The connections have to be real ones. Every SSH command letify builds carries `ControlMaster=auto`, which multiplexes further sessions onto the first connection, so opening more channels that way would add streams inside one TCP connection and gain nothing. A transfer channel therefore sets `ControlPath=none` and `ControlMaster=no`, and a punched link redials for each, so each transfer channel is its own kernel connection.
+The connections have to be real ones. Every SSH command letify builds carries `ControlMaster=auto`, which multiplexes further sessions onto the first connection, so opening more channels that way would add streams inside one TCP connection and gain nothing. A transfer channel therefore sets `ControlPath=none` and `ControlMaster=no`, and a punched link gives each its own socket, so each transfer channel is its own kernel connection. A punched link holds the spare sockets of its punch, `PUNCH_SOCKETS` of them in all, [Rendezvous](#rendezvous), and hands one to each SSH command before it redials. The worker channel and the first transfer channels therefore cost no rendezvous, and `transfer_connection_cost_s` applies only to the streams beyond the spares.
 
 The worker's `data_put` appends to one handle per digest and hashes as it goes, which assumes the pieces arrive in order on one stream. Parallel senders cannot use it, so they use two other requests:
 
@@ -1883,7 +1912,9 @@ The remote side's punch window is 150 s, or the request's `window` in seconds wh
 
 Every SSH command letify builds sets `HostKeyAlias=letify-<alias>`, `StrictHostKeyChecking=accept-new` and `UserKnownHostsFile=~/.letify/accounts/<alias>/known_hosts`. Building the command creates that account directory with mode 0700 when it is missing, because `ssh` creates only `~/.ssh` and otherwise cannot record the key: it prints `Failed to add the host to the list of known hosts` on every connection and never pins the key. A host key that differs from the recorded one still fails the connection.
 
-A punched connection first answers the probe, then on request is spliced to the remote machine's SSH server, and SSH runs over a local forwarding port. The splice copies with `os.splice` through a pipe where the platform has it, which is Linux, so the bytes move between the two sockets inside the kernel. Every byte of a punched session crosses it, and the machine on the other end is often a two core VM, so a read into Python and a write back out costs real throughput there. A platform without `os.splice`, or a call it refuses, falls back to the read and write loop. A second SSH connection over the same link punches again.
+A punched connection first answers the probe, then on request is spliced to the remote machine's SSH server, and SSH runs over a local forwarding port. The splice copies with `os.splice` through a pipe where the platform has it, which is Linux, so the bytes move between the two sockets inside the kernel. Every byte of a punched session crosses it, and the machine on the other end is often a two core VM, so a read into Python and a write back out costs real throughput there. A platform without `os.splice`, or a call it refuses, falls back to the read and write loop. A second SSH connection over the same link takes a spare socket from the same punch, and punches again only when the spares are used up. Next section.
+
+A punch opens several sockets in one rendezvous, because each extra rendezvous costs about 10 s of lead time and one TCP flow over a long path carries a fraction of the path. The request names `count`, 1 when it is absent and at most 8, `PUNCH_SOCKETS` (4) when the strategy asks. Each side binds `count` local ports, learns a STUN mapping for each, and the request and the answer carry them as `mappings`, in order, with `mapping` still holding the first. Pair `i` is the user's port `i` and the remote's port `i`. It uses the token `token + bytes([i])`, so a hello written for one pair is refused by every other, and all pairs punch in the same window, each on its own thread. A pair that does not connect is absent, and the punch fails only when no pair connected. The remote side serves every connected socket as it serves one, each on its own thread, and its detached process stays until the last of them has ended, because the first one, which carries the probe, often ends long before a spare in use does. Every connected socket has `SO_KEEPALIVE` with `TCP_KEEPIDLE` 20 s and `TCP_KEEPINTVL` 10 s where the platform has them, so the NAT mapping of a socket waiting for its turn is not dropped. The strategy logs `tcp_punch connected <k> of <n> sockets in <seconds> s`. [PR #75](https://github.com/DarkPyonix/letify/pull/75) records the measurements.
 
 A local port that letify binds for forwarding is chosen by the operating system, never fixed, because Windows reserves port ranges that vary by machine. The remote end of a reverse forward is chosen the same way, with `ssh -R 0:`.
 
@@ -2453,11 +2484,28 @@ The two `Tensor` methods are replaced on the class and the `torch.accelerator` f
 
 ### Compilation <!-- id: forwarding-compile -->
 
-> Under `host="local"`, `torch.compile` returns the function or module it is given, unchanged, and warns once that it runs eagerly, so a compiled training loop runs and computes what eager code computes.
+> Under `host="local"`, `torch.compile` returns the function or module it is given, unchanged. When step compilation is enabled it also asks the runtime to compile each captured step with Inductor, so the same Python code gets fused kernels without a CUDA driver in this process. Step compilation is off by default.
+
+**Switch.** Step compilation is enabled by the environment variable `LETIFY_COMPILE_STEPS` when it is set, else by `compile_steps = true` at the top level of `~/.letify/config.toml`, else it is off. For the variable, `1`, `true`, `yes` and `on` enable it and `0`, `false`, `no` and `off` disable it, in any case; any other value disables it. The client reads the switch once, when it is created, as it does `auto_fetch`. The default is off because it measured no gain: on a Colab T4, 15.0 ms per step eager against 16.0 ms compiled, because under `host="local"` the step time is bound by the client's dispatch, not by the runtime GPU.
+
+**Client.** Inductor cannot run here: before tracing it creates a CUDA tensor in this process to set up a device context, which needs a driver this process does not have. Dynamo with any backend also traces into `RemoteTensor` dispatch, which is letify's own Python. So while forwarding is active, `torch.compile(model, ...)` with any arguments returns `model` itself, and `torch.compile(...)` used as a decorator factory returns a decorator that returns the function itself. `Module.compile(...)` does nothing else. `torch.compile` and `Module.compile` are restored when forwarding ends and undone in a forked process, as the rest of the mapping is.
+
+While step compilation is off, `torch.compile` and `Module.compile` send nothing, the executor replays every step as [Step capture](#forwarding-step-capture) describes, and the first call in a process emits a `UserWarning` naming `host="local"`, saying that the model runs eagerly, and naming `LETIFY_COMPILE_STEPS=1` and `compile_steps = true` as the way to compile captured steps on the runtime. While it is on, the first call in a session that does not pass `disable=True` and has the default `backend` sends the request `letify.compile` with its `mode` and `options`, `None` and `{}` when absent. A later call in the session sends nothing. The first such call in a process also emits a `UserWarning` naming `host="local"` and saying that compilation happens on the runtime per captured step. Until the request arrives, and when nobody calls `torch.compile`, the runtime replays every step as [Step capture](#forwarding-step-capture) describes, unchanged.
+
+**Compiled segments.** Once compilation is enabled, the worker compiles a segment of a step: the operators `start` to `stop - 1` of one step entry. A segment is identified by `(step, start, stop)`, so a partial entry is its own segment. Its compiled function returns every tensor that some entry of the segment kept, and an entry stores only the ones in its own `keep`; an entry whose `keep` has an offset not yet returned is another variant. A segment that has any of these stays eager for the rest of the session: an operator whose name contains a random-number token (`rand`, `dropout`, `bernoulli`, `normal`, `multinomial`, `uniform`, `exponential`, `random`), because Inductor draws from its own generator; a compilation failure; or a fifth variant of its step.
+
+- **Sightings.** The first entry of a segment runs eagerly and records its scalars. On the second, a scalar slot whose value differs from the first and whose type is `int` or `float` is varying, and every other slot is constant. The second entry also starts the compilation, and runs eagerly.
+- **Function.** The compiled function takes, in this order, the distinct tensors the segment reads that it does not create (the entry's `externals` and the handles created before `start`, each tensor once however often it is read), the CPU tensors the segment reads as blobs, and one 0-d CPU tensor for each varying scalar, `float64` for a float and `int64` for an int. It replays the operators exactly as the **Worker** paragraph of [Step capture](#forwarding-step-capture) does, over a table of its own, reading a varying scalar with `aten._local_scalar_dense` so that an optimizer's learning rate or bias correction is a runtime value, not a constant. It returns the survivors: the tensors the segment creates that are in `keep` or not released inside the segment, in creation order. An in-place update of an input is written back to that input.
+- **Tracing.** The function is traced with `make_fx` in its symbolic mode over fake inputs of the observed shapes, and compiled with `torch._inductor.compile_fx.compile_fx` against fake inputs of static shapes. Inductor's `keep_output_stride` keeps its default.
+- **Background.** Tracing and compiling run in a thread of the worker. Until the compiled function is ready, entries of the segment run eagerly.
+- **Guard.** An entry runs compiled when its constant scalars equal those compiled in, the pattern of repeated tensors among the inputs is the same, every input has the shape, strides, dtype and device compiled in, and no two inputs share storage. Otherwise it runs eagerly. A mismatch of the constants, the pattern or the inputs starts a new compilation of the segment with the changed scalar slots varying, which is another variant. A step has at most 4 variants in the session.
+- **Run.** A compiled entry stores each survivor that is in its `keep` or not released inside the segment under its handle, `first handle` plus its offset, drops the handles created before `start` that the eager replay would release, and creates no other handle.
+
+**Mode and options.** `options` is applied as `torch._inductor.config.patch(options)` around the compilation, so a key Inductor does not know is a compilation failure. `mode="max-autotune"` and `mode="max-autotune-no-cudagraphs"` set `max_autotune`. Every other mode, `"reduce-overhead"` included, compiles as the default mode does, because the worker does not use CUDA graphs.
 
-Inductor, the default backend, cannot compile here: before tracing it creates a CUDA tensor in this process to set up a device context, which needs a driver this process does not have. Dynamo with any backend also traces into `RemoteTensor` dispatch, which is letify's own Python, and recompiles it for every operator. A replayed step already sends the whole step as one entry, as [Step capture](#forwarding-step-capture) describes, so compiling on the client has nothing left to batch.
+**Failure.** A compilation that fails marks the segment eager and queues a note `letify: step <step> could not be compiled: <reason>; it runs eagerly`. The note travels with the next reply and the client prints it on standard error once. A compiled entry that raises while running is reported as [Failure semantics](#forwarding-failure) describes, naming `compiled step <step>`, the segment is marked eager, and the entry is not run again.
 
-While forwarding is active, `torch.compile(model, ...)` with any arguments returns `model` itself, and `torch.compile(...)` used as a decorator factory returns a decorator that returns the function itself. `Module.compile(...)` does nothing. The first such call in a process emits a `UserWarning` naming `host="local"` and eager execution. `torch.compile` and `Module.compile` are restored when forwarding ends and undone in a forked process, as the rest of the mapping is.
+**Request.** `letify.compile_stats` with `wait` true joins the compilations in progress, then returns counters: `compiled`, the successful compilations; `failed`, the failed ones; `runs`, the entries that ran compiled; `eager`, the segments marked eager.
 
 `torch.cuda.get_rng_state` and `torch.cuda.set_rng_state` stay refused, so code that saves and restores the generator state names the gap instead of silently restoring a state that is not the runtime's.
 
@@ -2582,7 +2630,7 @@ A step is what a training loop repeats: forward, backward and the optimizer upda
 
 **Early release.** A tensor a repetition creates is released on the runtime right after the last operator of the step that reads it, when nothing else can read it, rather than with the release list after the whole step. When a step is registered, the client and the worker both compute each created handle's release position from the wiring: the last position whose operator takes that offset as an argument, or the position that created it when no later operator of the step reads it. A step entry carries `keep`, the offsets whose release position lies in `start` to `stop - 1` and that must not be released early. The client computes `keep` when it takes the entry into a batch: an offset is kept when a `RemoteTensor` still names its handle, which is when its handle has not been released locally, or when an entry after it in that batch, an entry left in the queue, or the unfinished repetition reads the handle. A saved tensor autograd holds for backward is named by a `RemoteTensor`, so it is released only after the backward operator that reads it. A tensor the loop keeps across steps, such as the loss or a gradient, is kept, and released by the release list as before.
 
-**Worker.** The worker executes operators `start` to `stop - 1` of the step in order, reading an argument's handle as `first handle + offset` or from `externals`, and stores each new output under the next handle counted from `first handle`. After each operator it drops the handles whose release position is that operator, other than those in `keep`, from its table. Each operator is the same call an eager entry makes, so a failure is recorded and reported as [Failure semantics](#forwarding-failure) describes, naming the operator. The worker does not use CUDA graphs, because the handles bound to a step, such as the batch, change between repetitions and a graph needs fixed input memory.
+**Worker.** The worker executes operators `start` to `stop - 1` of the step in order, reading an argument's handle as `first handle + offset` or from `externals`, and stores each new output under the next handle counted from `first handle`. After each operator it drops the handles whose release position is that operator, other than those in `keep`, from its table. Each operator is the same call an eager entry makes, so a failure is recorded and reported as [Failure semantics](#forwarding-failure) describes, naming the operator. When compilation is enabled, a segment may run as one compiled function, as [Compilation](#forwarding-compile) describes. The worker does not use CUDA graphs, because the handles bound to a step, such as the batch, change between repetitions and a graph needs fixed input memory.
 
 ### Handles <!-- id: forwarding-handles -->
 

{letify-2.9.0 → letify-2.10.0}/docs/guide/03-execution-modes.md

@@ -72,6 +72,8 @@ A read does not have to be a synchronization. `loss.detach().to("cpu", non_block
 
 You do not have to write either of those for a logged number. Under `host="local"`, `loss.item()` already returns a deferred value: the read is queued and the loop keeps going, and the value is waited for where it is used, such as in the `print` a step later. `tolist()`, `float(tensor)` and `int(tensor)` work the same way. `bool(tensor)` still waits, because control flow needs the value at once. The value a call returns is always a real `float`, `int` or `list`. Put `auto_fetch = false` in the account entry, or `LETIFY_AUTO_FETCH=0` in the environment, to turn this off. At the end of a call letify prints how many reads it deferred and how many had to wait, so a loop that gains nothing from it is visible.
 
+`torch.compile` and `Module.compile` return the model unchanged under `host="local"` and warn once that it runs eagerly. To have the runtime compile each captured step with Inductor instead, set `LETIFY_COMPILE_STEPS=1` in the environment or `compile_steps = true` at the top level of `~/.letify/config.toml`. It is off by default because it measured no gain on a Colab T4: 15.0 ms per step eager against 16.0 ms compiled, since the step time is bound by the client's dispatch.
+
 Other things that force a synchronization: `nonzero()`, boolean mask indexing, `unique()`, and mixture of experts routing, which reads per-expert token counts back to the host. Any operation whose output shape the host has to know is a synchronization.
 
 ### Measure it instead of guessing

{letify-2.9.0 → letify-2.10.0}/docs/guide/05-concurrency.md

@@ -157,7 +157,7 @@ def train():
     ...
 ```
 
-The hook runs when the worker receives `SIGTERM`, when its lease runs out, or when the channel closes during the call. It does not run for `SIGKILL` or the out of memory killer, so keep writing checkpoints as well. Hooks run newest first and share 120 seconds.
+The hook runs when the worker receives `SIGTERM`, when its lease runs out, when the channel closes during the call, or when the owner shuts down. A shutdown by the owner (a pool shutdown, or a client that received `SIGTERM`) runs the hooks and then ends the body and the worker. A channel that closes with no shutdown leaves the body running until the lease deadline, so another process can take it over. It does not run for `SIGKILL` or the out of memory killer, so keep writing checkpoints as well. Hooks run newest first and share 120 seconds.
 
 None of this works on a provider whose worker has only a one-shot channel, and the commands say which provider. There is no control socket on Windows. See the spec section "Controlling a running call".
 

{letify-2.9.0 → letify-2.10.0}/docs/guide/07-troubleshooting.md

@@ -138,6 +138,16 @@ and letify cannot know when that process ends.
 
 On a shared machine this usually means a colleague is computing on those cards. Check with `letify utilization`, then retry later or narrow `indices` in the devices table.
 
+**Cards held by a worker you cannot see.** A worker left behind when the local process died keeps its cards until its lease ends, 300 s after the last renewal. On a `Shell` machine, list the workers and stop one:
+
+```bash
+letify sessions
+letify stop --orphans --alias lab
+letify stop --alias lab --worker 4242
+```
+
+`letify sessions` prints one line per worker: pid, GPU indices, start time, runtime name, pool key, run id and the local process that owns it, marked `alive`, `gone` or `remote` (owned from another machine, so not judged). `--orphans` stops the workers whose lease has passed, and those whose owner is gone while no call runs. It never stops a worker whose owner is alive. A worker whose owner is gone but whose call still runs is listed by `letify runs` and can be taken over, so `--orphans` leaves it alone. `--worker` stops the one pid you name, whatever its owner, and refuses a pid that is not a letify worker. A stop sends `SIGTERM`, waits 30 s, then sends `SIGKILL`.
+
 ---
 
 ## `RemoteError`

{letify-2.9.0 → letify-2.10.0}/docs/guide/ko/03-/354/213/244/355/226/211-/353/260/251/354/213/235.md

@@ -72,6 +72,8 @@ Hugging Face 학습 스텝은 기본 설정에서 `k ≈ 3`입니다.
 
 로깅하는 숫자 하나 때문에 위의 두 가지를 직접 쓸 필요는 없습니다. `host="local"`에서 `loss.item()`은 이미 지연 값을 돌려줍니다. 읽기는 큐에 들어가고 루프는 계속 진행되며, 값은 실제로 쓰이는 곳, 예를 들어 한 스텝 뒤의 `print`에서 기다립니다. `tolist()`, `float(tensor)`, `int(tensor)`도 같습니다. `bool(tensor)`는 제어 흐름이 값을 바로 필요로 하므로 그대로 기다립니다. 호출이 돌려주는 값은 언제나 실제 `float`, `int`, `list`입니다. 이 동작을 끄려면 계정 항목에 `auto_fetch = false`를 넣거나 환경 변수 `LETIFY_AUTO_FETCH=0`을 설정하세요. 호출이 끝나면 letify가 몇 개의 읽기를 지연했고 몇 개가 기다려야 했는지 한 줄로 출력하므로, 이 동작에서 얻는 것이 없는 루프는 바로 드러납니다.
 
+`host="local"`에서 `torch.compile`과 `Module.compile`은 모델을 그대로 돌려주고, 모델이 eager로 실행된다는 경고를 한 번 출력합니다. 런타임이 캡처된 각 스텝을 Inductor로 컴파일하게 하려면 환경 변수 `LETIFY_COMPILE_STEPS=1`을 설정하거나 `~/.letify/config.toml` 최상위에 `compile_steps = true`를 넣으세요. Colab T4에서 이득이 측정되지 않아 기본값은 꺼짐입니다. 스텝당 eager 15.0 ms, 컴파일 16.0 ms였고, 스텝 시간이 클라이언트의 디스패치에 묶여 있기 때문입니다.
+
 동기화를 강제하는 다른 것들도 있습니다. `nonzero()`, 불리언 마스크 인덱싱, `unique()`, 그리고 전문가별 토큰 수를 호스트로 읽어오는 MoE 라우팅입니다. 출력 크기를 호스트가 알아야 하는 연산은 모두 동기화입니다.
 
 ### 추측하지 말고 측정하세요

{letify-2.9.0 → letify-2.10.0}/docs/guide/ko/05-/353/217/231/354/213/234-/354/213/244/355/226/211.md

@@ -153,7 +153,7 @@ def train():
     ...
 ```
 
-훅은 워커가 `SIGTERM`을 받을 때, 리스 기한이 지날 때, 호출 중에 채널이 닫힐 때 실행됩니다. `SIGKILL`이나 메모리 부족 종료에는 실행되지 않으므로 체크포인트도 계속 쓰세요. 훅은 나중에 등록한 것부터 실행되며 합쳐서 120초를 씁니다.
+훅은 워커가 `SIGTERM`을 받을 때, 리스 기한이 지날 때, 호출 중에 채널이 닫힐 때, 소유자가 종료를 요청할 때 실행됩니다. 소유자가 종료를 요청하면(풀 종료, `SIGTERM`을 받은 클라이언트) 훅을 실행한 뒤 본문과 워커가 함께 끝납니다. 종료 요청 없이 채널만 끊기면 본문은 리스 기한까지 계속 실행되어 다른 프로세스가 이어받을 수 있습니다. `SIGKILL`이나 메모리 부족 종료에는 실행되지 않으므로 체크포인트도 계속 쓰세요. 훅은 나중에 등록한 것부터 실행되며 합쳐서 120초를 씁니다.
 
 워커에 일회성 채널만 있는 프로바이더에서는 이 기능을 쓸 수 없고, 명령이 그 프로바이더 이름을 알려 줍니다. Windows에는 제어 소켓이 없습니다. 스펙의 "Controlling a running call" 절을 참고하세요.
 

{letify-2.9.0 → letify-2.10.0}/docs/guide/ko/07-/353/254/270/354/240/234-/355/225/264/352/262/260.md

@@ -138,6 +138,16 @@ and letify cannot know when that process ends.
 
 공유 머신이라면 보통 동료가 그 카드로 계산 중이라는 뜻입니다. `letify utilization`으로 확인한 뒤 나중에 다시 시도하거나, devices 테이블의 `indices`를 좁히세요.
 
+**보이지 않는 워커가 잡고 있는 카드.** 로컬 프로세스가 죽으면서 남은 워커는 리스가 끝날 때까지, 마지막 갱신 후 300초 동안 카드를 잡고 있습니다. `Shell` 머신에서는 워커를 나열하고 하나를 멈출 수 있습니다.
+
+```bash
+letify sessions
+letify stop --orphans --alias lab
+letify stop --alias lab --worker 4242
+```
+
+`letify sessions`는 워커마다 한 줄을 출력합니다. pid, GPU 번호, 시작 시각, 런타임 이름, 풀 키, 실행 id, 그리고 소유한 로컬 프로세스와 그 상태(`alive`, `gone`, `remote`)입니다. `remote`는 다른 머신이 소유한 경우라 판정하지 않습니다. `--orphans`는 리스가 지난 워커와, 소유자가 사라졌고 실행 중인 호출이 없는 워커를 멈춥니다. 소유자가 살아 있는 워커는 멈추지 않습니다. 소유자는 사라졌지만 호출이 아직 도는 워커는 `letify runs`에 나오고 인수할 수 있으므로 `--orphans`가 건드리지 않습니다. `--worker`는 지정한 pid 하나를 소유자와 관계없이 멈추고, letify 워커가 아닌 pid는 거부합니다. 멈출 때는 `SIGTERM`을 보내고 30초 기다린 뒤 `SIGKILL`을 보냅니다.
+
 ---
 
 ## `RemoteError`

{letify-2.9.0 → letify-2.10.0}/docs/locales/README_ko.md

@@ -467,6 +467,8 @@ letify setup tailcat  # 배포처에서 tailcat 또는 eci를 미리 설치
 letify providers      # 선언된 프로바이더, 저장소 수명, 기본 배치
 letify devices           # 각자 제공하는 GPU
 letify status         # 지금 돌고 있는 것
+letify sessions       # 계정이 잡고 있는 것, shell 호스트의 워커
+letify stop --orphans # 죽은 프로세스가 남긴 것 정리
 letify usage          # 계정마다 남은 사용량
 letify utilization    # 인스턴스별 GPU가 얼마나 바쁜가
 letify check lab      # 이 머신이 응답하나?

{letify-2.9.0 → letify-2.10.0}/letify/__init__.py

@@ -70,7 +70,7 @@ distributed: Final = Host.distributed
 # The enum class stays importable from letify.declare.instance for letify's own use.
 del Host
 
-__version__ = "2.9.0"
+__version__ = "2.10.0"
 
 
 def fetch(tensor):  # type: ignore[no-untyped-def]

{letify-2.9.0 → letify-2.10.0}/letify/cli.py

@@ -49,6 +49,11 @@ def build_parser() -> argparse.ArgumentParser:
         help="delete the sessions whose creating process is gone",
     )
     stop.add_argument("--alias", help="only this account")
+    stop.add_argument(
+        "--worker",
+        metavar="PID",
+        help="stop the letify worker with this pid on the machine --alias names",
+    )
     stop.add_argument("-y", "--yes", action="store_true", help="do not ask")
     stop.add_argument("--json", action="store_true", help="print the records unformatted")
 
@@ -947,6 +952,21 @@ def _dispatch(args: argparse.Namespace) -> int:
         return 0
 
     if args.command == "stop":
+        if args.worker is not None:
+            if not args.alias:
+                print("letify: --worker needs --alias, the machine it runs on", file=sys.stderr)
+                return 2
+            print(f"worker {args.worker} on {args.alias} gets SIGTERM, then SIGKILL after 30 s")
+            if not args.yes and not _confirmed("Stop it?"):
+                print("nothing was stopped")
+                return 0
+            row = let.stop_worker(args.alias, args.worker)
+            if args.json:
+                _json(row)
+                return 0 if row.get("stopped") else 1
+            mark = "stopped" if row.get("stopped") else f"failed{_detail(row)}"
+            print(f"{row['alias']}  {row['ref']}  {mark}")
+            return 0 if row.get("stopped") else 1
         if not args.orphans:
             # Spec "Kaggle", stopping what a dead process left behind: a bare stop reads as
             # ending everything, so it ends nothing, and says why rather than only that.

{letify-2.9.0 → letify-2.10.0}/letify/launcher.py

@@ -517,6 +517,18 @@ class Launcher:
                 rows.append({"alias": name, "orphans": [], "unavailable": str(exc)})
         return rows
 
+    def stop_worker(self, alias: str, pid: str) -> dict[str, Any]:
+        """Stop one named worker on a provider that keeps worker records.
+
+        Spec "Workers on a shell host", stopping one. The provider refuses a pid that no
+        worker record names.
+        """
+        provider = self.provider(alias)
+        try:
+            return {"alias": alias, "ref": pid, "stopped": provider.stop_worker(pid)}
+        except LetifyError as exc:
+            return {"alias": alias, "ref": pid, "stopped": False, "error": str(exc)}
+
     def stop_orphans(self, alias: str | None = None) -> list[dict[str, Any]]:
         """Delete what ``orphans`` reported, one at a time, reporting each outcome."""
         done: list[dict[str, Any]] = []

{letify-2.9.0 → letify-2.10.0}/letify/protocol/worker.py

@@ -377,6 +377,7 @@ def _reply(stream, outcome):
     try:
         _send_outcome(_SENDER, stream, outcome)
     except OSError:
+        _drop_record()
         os._exit(0)
 
 
@@ -2677,6 +2678,7 @@ def _op_lease(request):
     """
     state = globals().setdefault("_LEASE", {})
     state["deadline"] = time.time() + float(request["grace"])
+    _record_update(lease_deadline=state["deadline"])
 
     def watch():
         while True:
@@ -2691,6 +2693,82 @@ def _op_lease(request):
     return {"ok": True, "value": state["deadline"]}
 
 
+# -- the worker's record on its host ---------------------------------------------------
+# Spec "Workers on a shell host": a file under the workspace root saying who this worker is,
+# written when the client registers it and removed when the worker ends.
+
+_RECORD = {"path": None, "data": None, "lock": threading.Lock()}
+
+
+def _proc_start():
+    """This process's start time in clock ticks since boot, or None where /proc is missing."""
+    try:
+        with open("/proc/self/stat") as handle:
+            return int(handle.read().rsplit(")", 1)[1].split()[19])
+    except (OSError, ValueError, IndexError):
+        return None
+
+
+def _write_record():
+    """Replace the record file whole, so a reader never sees half of it."""
+    import json
+    path, data = _RECORD["path"], _RECORD["data"]
+    if path is None:
+        return
+    partial = "%s.%d.part" % (path, os.getpid())
+    try:
+        descriptor = os.open(partial, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
+        with os.fdopen(descriptor, "w") as handle:
+            json.dump(data, handle)
+        os.replace(partial, path)
+    except OSError as exc:
+        print("letify: the worker record was not written: %s" % exc, file=sys.stderr)
+
+
+def _record_update(**fields):
+    """Change fields of the record and write it again. Nothing when no record is kept."""
+    with _RECORD["lock"]:
+        if _RECORD["path"] is None:
+            return
+        _RECORD["data"].update(fields)
+        _write_record()
+
+
+def _drop_record():
+    """Remove the record. Called before every way this process ends."""
+    with _RECORD["lock"]:
+        path, _RECORD["path"] = _RECORD["path"], None
+    if path is not None:
+        try:
+            os.unlink(path)
+        except OSError:
+            pass
+
+
+def _op_register(request):
+    """Write ``<workspace>/.letify/workers/<pid>.json`` and keep it current until exit."""
+    root = os.path.expanduser(request["workspace"])
+    directory = os.path.join(root, ".letify", "workers")
+    os.makedirs(directory, exist_ok=True)
+    os.chmod(directory, 0o700)
+    data = {
+        "pid": os.getpid(),
+        "proc_start": _proc_start(),
+        "started": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+        "runtime": request["runtime"],
+        "key": request["key"],
+        "devices": os.environ.get("CUDA_VISIBLE_DEVICES"),
+        "owner": request["owner"],
+        "run": _TAKE["run"],
+        "lease_deadline": globals().get("_LEASE", {}).get("deadline"),
+    }
+    with _RECORD["lock"]:
+        _RECORD["path"] = os.path.join(directory, "%d.json" % os.getpid())
+        _RECORD["data"] = data
+        _write_record()
+    return {"ok": True, "value": None}
+
+
 # -- controlling a running call --------------------------------------------------------
 # Spec "Controlling a running call": an interrupt, source run beside the body, and exit hooks.
 
@@ -2761,9 +2839,86 @@ def _run_exit_hooks(reason):
 
 def _exit_with_hooks(code, reason="the lease ended"):
     _run_exit_hooks(reason)
+    _drop_record()
     os._exit(code)
 
 
+#: Seconds the body's descendants get to end after SIGTERM before they are killed.
+CHILD_GRACE_S = 5.0
+
+
+def _descendants():
+    """Pids of every descendant of this process, from /proc, or None where /proc is missing."""
+    try:
+        names = [n for n in os.listdir("/proc") if n.isdigit()]
+    except OSError:
+        return None
+    children = {}
+    for name in names:
+        try:
+            with open("/proc/%s/stat" % name) as handle:
+                fields = handle.read().rsplit(")", 1)[1].split()
+        except (OSError, IndexError):
+            continue
+        children.setdefault(int(fields[1]), []).append(int(name))
+    found, queue = [], [os.getpid()]
+    while queue:
+        for pid in children.get(queue.pop(), ()):
+            found.append(pid)
+            queue.append(pid)
+    return found
+
+
+def _alive(pid):
+    try:
+        with open("/proc/%d/stat" % pid) as handle:
+            return handle.read().rsplit(")", 1)[1].split()[0] != "Z"
+    except (OSError, IndexError):
+        return False
+
+
+def _end_descendants():
+    """SIGTERM the processes the body started, wait CHILD_GRACE_S, then SIGKILL survivors."""
+    import signal
+    pids = _descendants()
+    if pids is None:
+        # No /proc: the group is signalled only when this worker leads it.
+        if hasattr(os, "killpg") and os.getpgrp() == os.getpid():
+            try:
+                os.killpg(os.getpgrp(), signal.SIGTERM)
+                time.sleep(2.0)
+            except OSError:
+                pass
+        return
+    for pid in pids:
+        try:
+            os.kill(pid, signal.SIGTERM)
+        except OSError:
+            pass
+    end = time.monotonic() + CHILD_GRACE_S
+    while time.monotonic() < end and any(_alive(pid) for pid in pids):
+        time.sleep(0.05)
+    for pid in pids:
+        if _alive(pid):
+            try:
+                os.kill(pid, signal.SIGKILL)
+            except OSError:
+                pass
+
+
+def _end_by_owner():
+    """The owner shut down on purpose: run the hooks, then end the body and the worker.
+
+    Spec "A deliberate shutdown". The exit status is the one a received SIGTERM gives. The
+    processes the body started are ended first.
+    """
+    import signal
+    _run_exit_hooks("the owner shut down")
+    _end_descendants()
+    _drop_record()
+    os._exit(128 + signal.SIGTERM)
+
+
 def _on_sigint(signum, frame):
     """Raise KeyboardInterrupt, as the default handler does, except for a stale interrupt.
 
@@ -2800,6 +2955,7 @@ def _watch_sigterm(fd):
             return
         if signal.SIGTERM in data and signal.getsignal(signal.SIGTERM) is _on_sigterm:
             _run_exit_hooks("SIGTERM")
+            _drop_record()
             os._exit(128 + signal.SIGTERM)
 
 
@@ -3089,6 +3245,7 @@ _OPS = {
     "device": _op_device,
     "stat": _op_stat,
     "lease": _op_lease,
+    "register": _op_register,
     "interrupt": _op_interrupt,
     "inspect_exec": _op_inspect_exec,
     "reduce_peers": _op_reduce_peers,
@@ -3310,6 +3467,7 @@ def _read(readinto=None, door=None):
     channel the call's output and reply go to.
     """
     receiver = Receiver(readinto or sys.stdin.buffer.readinto)
+    shutdown = False
     while True:
         try:
             event = receiver.next_event()
@@ -3323,6 +3481,7 @@ def _read(readinto=None, door=None):
             continue
         kind, stream, value = event
         if kind == SHUTDOWN:
+            shutdown = True
             break
         if kind != REQUEST:
             continue
@@ -3381,11 +3540,15 @@ def _read(readinto=None, door=None):
         _JOBS.put((stream, request))
         if op == "reexec":
             return
-    _reader_ended(door)
+    _reader_ended(door, shutdown)
+
 
+def _reader_ended(door, shutdown=False):
+    """A channel's reader stopped: it was closed, or sent a shutdown.
 
-def _reader_ended(door):
-    """A channel's reader stopped: it was closed, or sent a shutdown."""
+    Spec "A deliberate shutdown": a shutdown frame on the channel that owns a running call
+    ends the body and the worker. End of input alone leaves the body for a takeover.
+    """
     if door is None and _TAKE["moved"]:
         # The call was adopted away; this channel closing is the old owner leaving.
         return
@@ -3404,6 +3567,9 @@ def _reader_ended(door):
     inbox = _DEVICE.get("inbox")
     if inbox is not None:
         inbox.put(None)
+    if shutdown and during:
+        _end_by_owner()
+        return
     # Spec "Exit hooks": a channel that closes while a body runs is a reason to end, and
     # the hooks run before the main thread is told there is nothing more to serve. Spec
     # "Taking over a running call": the body keeps running, and the worker ends when the
@@ -3456,6 +3622,7 @@ def _orphan_output(kind, data):
 
 def _door_open(take):
     """Start listening on the call's door. A door that cannot open leaves the call as it was."""
+    _record_update(run=take["run"])
     with _TAKE_COND:
         _TAKE.update(
             run=take["run"], live=True, ended_input=False, finished=False, outcome=None,
@@ -3592,7 +3759,9 @@ def _deliver_call(stream, outcome):
         listener, path = _TAKE["listener"], _TAKE["path"]
         _TAKE.update(run=None, listener=None, path=None, finished=False, outcome=None)
     _door_close(listener, path)
+    _record_update(run=None)
     if failed and owner is None:
+        _drop_record()
         os._exit(0)
     if ended:
         _JOBS.put(None)
@@ -3633,6 +3802,7 @@ def _serve():
         _run(stream, request, True)
         request = None
     _release_output()
+    _drop_record()
 
 
 _serve()