microsandbox-rb 0.17.0 → 0.17.2

data/DESIGN.md

@@ -50,7 +50,7 @@ multi-threaded tokio runtime via `runtime.block_on(fut)`.
 
 ### GVL release
 
-`block_on` is wrapped in `nogvl` (`rb_thread_call_without_gvl`) so the Ruby
+`block_on` is wrapped in `nogvl` (`rb_thread_call_without_gvl2`) so the Ruby
 Global VM Lock is **released** while a sandbox operation is in flight. Without
 this, a long-running `exec` would freeze every other Ruby thread/fiber in the
 process (a real problem under Rails/Puma). The `nogvl` closure runs pure Rust
@@ -59,21 +59,81 @@ captured and re-raised *after* the GVL is re-acquired rather than unwinding
 across the C frame (which would be UB).
 
 **The *calling* thread is not preemptible during the call.** `nogvl` passes a
-null unblock-function to `rb_thread_call_without_gvl`, and Ruby only checks
-pending interrupts *before* and *after* the GVL-released region — so while a
-native call blocks, the thread that issued it cannot be interrupted:
-`Timeout::timeout` (which relies on `Thread#raise`), `Thread#kill`, and `SIGINT`
-(Ctrl-C) are all deferred until the call returns on its own. The GVL release
-keeps *other* threads live; it does **not** make the calling thread
-cancelable. This is harmless for the bounded calls (`exec`/`shell` with a
-`timeout:`, `AgentClient` with a `timeout:`) because the deadline fires inside
-the future, but the unbounded/streaming paths — `exec`/`shell` with
-`timeout: nil`, `ExecHandle#recv`/`#wait`, `log_stream`/`metrics_stream` `recv`
-(especially `follow: true`), `Sandbox#wait`/`SandboxHandle#wait_until_stopped`,
-and `AgentClient#request` with no timeout — can block their caller indefinitely
-if the guest wedges or a relay drops. **Bound such calls with the explicit
-`timeout:` knobs rather than wrapping them in `Timeout::timeout`**, which will
-not fire while the native call is in flight.
+null unblock-function, so while a native call blocks, the thread that issued it
+cannot be interrupted: `Timeout::timeout` (which relies on `Thread#raise`),
+`Thread#kill`, and `SIGINT` (Ctrl-C) are all deferred until the call returns on
+its own. The GVL release keeps *other* threads live; it does **not** make the
+calling thread cancelable. This is harmless for the bounded calls
+(`exec`/`shell` with a `timeout:`, `AgentClient` with a `timeout:`) because the
+deadline fires inside the future, but the unbounded/streaming paths —
+`exec`/`shell` with `timeout: nil`, `ExecHandle#recv`/`#wait`,
+`log_stream`/`metrics_stream` `recv` (especially `follow: true`),
+`Sandbox#wait`/`SandboxHandle#wait_until_stopped`, and `AgentClient#request`
+with no timeout — can block their caller indefinitely if the guest wedges or a
+relay drops. **Bound such calls with the explicit `timeout:` knobs rather than
+wrapping them in `Timeout::timeout`**, which will not fire while the native
+call is in flight.
+
+### Interrupts never lose a result
+
+An interrupt that arrives during a native call is delivered **after the call
+returns**, and the call's result is not lost. Ruby delivers `Thread#raise`,
+`Thread#kill` and `SIGINT` with a `longjmp`, which skips Rust frames without
+running destructors; if that happened while the value a call had just produced
+(a booted sandbox, a guest exec session, megabytes of output) lived only in a
+Rust frame, it would be neither returned to Ruby nor dropped. The plain
+`rb_thread_call_without_gvl` does exactly that — it checks interrupts once more
+after the callback, before returning — which is why it is not used. Two rules
+in `runtime.rs` replace it:
+
+1. `nogvl` checks interrupts itself, under `rb_protect`, **only before the work
+   starts**: once up front, and again whenever `rb_thread_call_without_gvl2`
+   declined to run the callback because an interrupt was pending at entry. A
+   raise or kill there is an ordinary `Err` that unwinds the Rust frames
+   normally — nothing has happened yet. After the callback has run, nothing is
+   checked. (`nogvl` and `block_on` therefore return `Result`, and every call
+   site propagates it with `?`. A bound method that needs two awaits does both
+   in one `block_on`, so no value sits between two entry checks.)
+2. Between `block_on` returning and the bound method returning its value, the
+   binding does not call into Ruby: no `funcall` (the VM checks interrupts
+   after every one), only C-level Hash/Array/String construction and
+   `TypedData` wrapping. The class lookup magnus would otherwise do with
+   `const_get` on a type's first wrap is resolved at extension load instead.
+
+So the next interrupt checkpoint after a native call is the Ruby VM's own,
+reached once the result is a Ruby object. What happens to it then:
+
+- **`Sandbox.create`, `Sandbox.connect_or_create`, `Sandbox.start`, and
+  `PullSession#sandbox`** mask interrupts (`Thread.handle_interrupt(Object =>
+  :never)`) around the native call plus the wrap, and an `ensure` **stops the
+  sandbox** when an exception or a kill cuts in before the caller has the
+  object — but only a sandbox the call owns (`Sandbox#owns_lifecycle?`, the
+  same predicate as `connect_or_create`'s block form): one that was already
+  running for someone else is left running. A `SIGINT` cannot be masked this
+  way; it can arrive before the wrapper is assigned, and then the next case
+  applies.
+- **Every other wrapper** (`ExecHandle`, streams, SSH clients, write sinks, …)
+  and every plain result simply becomes garbage; GC frees the wrapper and the
+  Rust value's `Drop` closes the session or connection.
+
+The calls themselves remain uninterruptible, so `Timeout.timeout { Sandbox.create(...) }`
+still waits for the create to finish — and now also for the stop that cleans
+it up — before `Timeout::Error` surfaces. That stop is best-effort (a failure
+is swallowed so the interrupt is what the caller sees; GC then tears the
+sandbox down).
+
+`Thread.handle_interrupt(X => :on_blocking)` treats a native call as the
+blocking point at its *start*: the bridge raises an exception that is already
+deferred before the call begins, and one that arrives during the call waits
+for the thread's next blocking operation. The four acquisition methods are
+the exception: their own `Object => :never` mask sits on top of the caller's,
+so a deferred exception is not raised at the native call's entry there — the
+sandbox is returned and the exception is delivered at the next blocking
+operation after the acquisition.
+
+`spec/fake_runtime/interrupt_safety_spec.rb` pins all of this against the real
+extension, driving it with a fake `msb` (`spec/support/fake_runtime/`) whose
+log shows whether a sandbox was stopped, dropped, or left attached.
 
 ## Error mapping
 
@@ -185,7 +245,7 @@ extension.
 `ext/microsandbox/Cargo.toml` depends on the core crate via a **pinned git tag**
 (`microsandbox` / `microsandbox-network`, pinned to the same tag as
 `Microsandbox::RUNTIME_VERSION` — currently `v0.6.18`), so the gem builds anywhere
-— CI, `rake-compiler-dock` release containers, and end-user source installs —
+— CI and end-user source installs —
 without an adjacent checkout. For fast local development against a sibling
 microsandbox checkout, copy `.cargo/config.toml.example` to `.cargo/config.toml`
 (gitignored); its `paths` override builds against the local crates instead of
@@ -195,25 +255,13 @@ git. The override must never be committed — it would break container builds.
 
 * **Source gem**: compiles the extension via `extconf.rb` (rb-sys
   `create_rust_makefile`); requires a Rust toolchain (MSRV below).
-* **Precompiled platform gems**: built best-effort by the `cross-gems` job
-  (`.github/workflows/release.yml`) with `oxidize-rb/cross-gem-action`
-  (`rake-compiler-dock`) per `Gem::Platform`, shipping multi-ABI
-  `lib/microsandbox/<ruby_abi>/` native artifacts — the same model Node uses with
-  per-platform packages. End users then install with no Rust toolchain, and the
-  host runtime comes from the `microsandbox-rb-binaries` gem — or, without it, is
-  fetched on first use (see above). The guest `agentd` is baked into
-  the extension by *target* arch (`filesystem/build.rs` uses
-  `CARGO_CFG_TARGET_ARCH` + `include_bytes!`), so it cross-compiles correctly;
-  the real cross work is linking the *target* native libs — `libcap-ng` on Linux
-  (via Debian multiarch for `aarch64-linux`) and the Hypervisor + Security
-  frameworks on macOS (via osxcross; `arm64-darwin` is the platform still to
-  confirm — if osxcross can't link them, move it to a native `macos-14` runner).
-  The job is gated to `workflow_dispatch` and **not** auto-published on tags:
-  since CI can't boot a microVM to prove a built gem actually works, gems are
-  promoted to the publish path manually after per-platform validation. Published
-  to RubyGems via Trusted Publishing (OIDC). See [Releasing](README.md#releasing).
+* **Precompiled platform gems**: not built or published. Every platform
+  installs the source gem, which compiles the extension via `rb_sys` at install
+  time; the host runtime comes from the `microsandbox-rb-binaries` gem — or,
+  without it, is fetched on first use (see above). The companion binaries gems
+  below are unaffected. See [docs/releasing.md](docs/releasing.md).
 * **Runtime binaries gems** (`microsandbox-rb-binaries`): a distinct artifact
-  from the precompiled *extension* gems above — no Ruby code beyond a small
+  from the SDK gem — no Ruby code beyond a small
   locator module, just the verified upstream `msb` + `libkrunfw` for one
   platform. CI's `binaries` job vendors and builds all three platforms on every
   run and smoke-tests the host one; the `package` job installs the source gem
@@ -232,42 +280,11 @@ git. The override must never be committed — it would break container builds.
 
 ## Implemented surface (v1) vs roadmap
 
-**Implemented:** sandbox lifecycle (`create`/`start`/`get`/`list`/`list_with`/
-`remove`; the live `Sandbox` exposes `stop`/`stop_and_wait`/`kill`/`drain`/
-`wait` (→ `ExitStatus`) / `status` / `detach` / `owns_lifecycle?`, while the
-controllable `SandboxHandle` from `get`/`list` carries the fine-grained
-`stop_with_timeout`/`request_stop`/`request_kill`/`request_drain`/
-`wait_until_stopped` (→ `SandboxStopResult`) controls — the v0.5.8 live-vs-handle
-split that mirrors the official SDKs), backend routing (`set_default_backend`/
-`with_backend`/`default_backend_kind`), block form), `exec`/`shell` with collected `ExecOutput`,
-**streaming** `exec_stream`/`shell_stream` (`ExecHandle` is `Enumerable` over
-`ExecEvent`s, with stdin sink + signal/kill/resize), the full guest filesystem
-API (`fs.read`/`write`/`list`/`mkdir`/`remove`/`stat`/…), `metrics`,
-`Microsandbox.all_sandbox_metrics`, **streaming `metrics_stream`/`log_stream`**
-(`Enumerable` over `Metrics`/`LogEntry`), `logs`,
-**OCI image-cache management** (`Image.get`/`list`/`inspect`/`remove`/`prune`),
-**named volumes** (`Volume.create`/`get`/`list`/`remove` + `volumes:` mounts),
-**snapshots** (`Snapshot.create`/`get`/`list`/`remove`/`verify`/`export`/`import`
-+ `from_snapshot:` boot), **rootfs patches** (`Patch.text`/`file`/`append`/
-`copy_file`/`copy_dir`/`symlink`/`mkdir`/`remove` via `create(patches:)`),
-**custom per-rule network policies** (`NetworkPolicy`/`Rule`/`Destination` —
-CIDR/IP/domain/suffix/group allow-deny rules with per-direction defaults and
-bulk domain denials, alongside the presets), interactive **`attach`/
-`attach_shell`** (host-TTY coupled — raw mode + SIGWINCH), **SSH**
-(`Sandbox#ssh` → `SshClient`/`SftpClient`/`SshServer`), the **raw agent client**
-(`AgentClient` → `AgentStream`/`AgentFrame`),
-`version`/`install`/`installed?`/`ensure_runtime!`, **registry auth**
-(`registry_auth`/`registry_insecure`/`registry_ca_certs` on `create`, for
-private/authenticated registries), and the typed error hierarchy.
-
-Create options now cover `image`, `cpus`, `memory`, `oci_upper_size`, `env`,
-`workdir`, `shell`, `user`, `hostname`, `labels`, `scripts`, `entrypoint`,
-`ports`/`ports_udp`, `volumes`, `patches`, `network` (policy presets
-`public_only`/`none`/`allow_all`/`non_local`, or a custom `NetworkPolicy`/Hash),
-`log_level`, `quiet_logs`, `security`, `max_duration`, `idle_timeout`, `rlimits`,
-`pull_policy`, `registry_auth`/`registry_insecure`/`registry_ca_certs`,
-`secrets`, `from_snapshot`, `detached`, and `replace`/`replace_with_timeout`.
-`exec`/`shell` add per-call `rlimits`.
+The authoritative, up-to-date scope statement — every implemented API family,
+the full create-option list, and the handful of upstream knobs not yet exposed —
+lives in [docs/surface.md](docs/surface.md), with the typed counterpart in
+`sig/microsandbox.rbs`. It is maintained there (and only there) so this file
+and the README cannot drift from each other.
 
 ## Verification
 
@@ -300,7 +317,8 @@ create options, full mount options (tmpfs/disk + stat-virtualization/
 host-permissions), and snapshot inspection (`open`/`list_dir`/`reindex`).
 
 A few **secondary** upstream knobs remain unexposed (a genuine binding gap, not
-upstream-gated — they exist at the pinned `v0.6.2` runtime): per-published-port host
+upstream-gated — they exist at the pinned runtime,
+`Microsandbox::RUNTIME_VERSION`): per-published-port host
 **bind address** (ports always bind loopback), network **interface overrides**,
 and inline **named-volume create-mode** (pre-create with `Volume.create`, then
 mount with `{ named: }`). These slot in module-by-module exactly as the existing