kern-sandbox 0.1.42 → 0.2.0

package/README.md

@@ -133,10 +133,10 @@ LangChain tool and the MCP server already use it.
 |---|---|
 | `timeout` | the call exceeded `timeoutS`; the binding killed the box |
 | `escape_blocked` | a syscall was blocked by the seccomp filter (SIGSYS) |
-| `oom` | the box was SIGKILLed and a `memoryMb` cap was **in force**: a breached `memory.max` is the cgroup OOM-killer (`memory.oom.group=1` kills the whole box). kern reports whether the cap actually bound on an unforgeable per-box channel (2nd byte of `KERN_STARTED_FD`), so this is an *enforced-cap* OOM |
-| `killed` | a SIGKILL **not** attributed to a cgroup OOM: no `memoryMb` cap was set, or kern reported the cap did not bind here (no cgroup delegation), so it is host pressure / an external kill. Older kern (no enforcement byte) falls back to `oom` when a cap was set |
-
+| `oom` | kern reported that the kernel's OOM killer took the box against its own memory cap: a breached `memory.max` takes the whole box, since kern sets `memory.oom.group=1`. Reported on a channel the code in the box cannot write (a third byte on kern's own descriptor), so it is an observation of the kernel's counter rather than a guess from the exit code |
+| `killed` | the box was SIGKILLed with **no** OOM reported against its cap: an external kill (`kern stop`, a signal, the host running out of memory), or a cap that did not bind here (no cgroup delegation, which the message names). A `memoryMb` cap being set is not, by itself, evidence that memory is what killed the box |
 | `exec_failed` | the box started but the command did not exist inside it. `runCode(code, {language:"node"})` on an image with no `node` is the ordinary way to reach it; the message names the binary AND the image, because the remedy is a different `language` or a different `image`. The `language` enum is a convenience, not a promise about the image: the default `python:3.12-slim` carries `python` and `bash`. A shell's own `command not found` inside your script stays an ordinary non-zero exit |
+| `startup_failed` | returned as data, not thrown, in one case: your `timeoutS` fired while kern was still BUILDING the box, so the code never ran. kern reports on a separate descriptor whether it reached your workload, which is what tells this from a slow cell. A host path that blocks does it: a bind source on a dead NFS export, a FUSE mount whose daemon is gone. A longer timeout does not help |
 
 **An enforced `pids` cap produces no fault, and that is deliberate.** When `pids` binds, the refused
 `fork` returns `EAGAIN`. Code that catches it exits 0, so the call reports `fault: null, success:

package/index.js

@@ -36,7 +36,7 @@ const crypto = require("crypto");
 const zlib = require("zlib");
 const { spawn, spawnSync } = require("child_process");
 
-const VERSION = "0.1.42";
+const VERSION = "0.2.0";
 
 const DEFAULT_IMAGE = "python:3.12-slim";
 const WORKSPACE = "/workspace"; // where the persistent workspace is mounted inside every box
@@ -415,6 +415,12 @@ while True:
 const EXIT_SIGKILL = 137; // SIGKILL: timeout backstop or OOM (indistinguishable without cgroup)
 const EXIT_SIGSYS = 159; // SIGSYS: a seccomp-denied syscall = a blocked escape attempt
 const EXIT_SIGTERM = 143; // SIGTERM: kern's --timeout backstop reaping the box
+// The signal NUMBERS behind those codes, for kern's 4th started-byte. Spelled out rather than derived
+// from the code, because `128 + N` is the convention being checked and deriving N from it would make the
+// check circular.
+const SIG_KILL = 9;
+const SIG_TERM = 15;
+const SIG_SYS = 31;
 
 // Per-call kwargs that DEFAULT to the Sandbox value: UNSET means "inherit the constructor's", whereas
 // an explicit `null` means "disable" (used for onStdout/onStderr overrides).
@@ -562,7 +568,59 @@ function sandboxFault(type, message) {
   return { type, message };
 }
 
-/** Locate `kern`: $KERN_BIN if set, else the first `kern` on $PATH. */
+/** Binaries already identified as kern, keyed by identity and not by path: a `kern` REPLACED between two
+ * calls is a different program and gets checked again. */
+const VERIFIED_KERN = new Set();
+
+/** Refuse a binary that does not IDENTIFY ITSELF as kern. Throws `SandboxError` if it does not.
+ *
+ * MEASURED, and found by an external reviewer running the positive control this project wrote for him:
+ * with `KERN_BIN=/bin/true` a call returned `success: true, exitCode: 0, fault: null` and an empty
+ * stdout. The code never ran and the caller was told it had. Any `kern` earlier in `PATH` that is not
+ * kern does this: a leftover wrapper, a shim, a no-op. An agent loop reads `success` and every
+ * conclusion after that is about a program that never executed.
+ *
+ * POSITIVE IDENTIFICATION, not inference from a missing signal: a kern old enough to predate
+ * `KERN_STARTED_FD` writes no bytes either, and refusing it would punish an old binary rather than a
+ * fake one. `kern --version` prints `kern <version>`, a prefix kern's own suite asserts.
+ *
+ * Memoised per binary identity, so it costs one `--version` (measured at 0.9 ms) per distinct binary
+ * per process and nothing afterwards. Fail-closed: unrunnable, slow or unrecognised is refused, because
+ * an unverifiable runtime is the case this exists for. Mirrors `_verify_is_kern`. */
+function verifyIsKern(bin) {
+  let key;
+  try {
+    const st = fs.statSync(bin);
+    key = [fs.realpathSync(bin), st.dev, st.ino, st.size, st.mtimeMs].join("|");
+  } catch (e) {
+    throw new SandboxError(`could not stat the kern binary at '${bin}': ${e.message}`);
+  }
+  if (VERIFIED_KERN.has(key)) return;
+  const hint =
+    "If this is not the kern you meant, set $KERN_BIN to the right path. To install kern:\n" +
+    "    curl -fsSL https://raw.githubusercontent.com/getkern/kern/main/install.sh | sh";
+  // Generous on purpose: a loaded machine must not be told its kern is fake. `--version` writes one
+  // line, so a binary that cannot answer in ten seconds is not one to trust with a box.
+  const out = spawnSync(bin, ["--version"], { encoding: "utf8", timeout: 10000 });
+  if (out.error && out.error.code === "ETIMEDOUT")
+    throw new SandboxError(
+      `'${bin}' did not answer \`--version\` within 10s, so it cannot be identified as kern. ${hint}`,
+    );
+  if (out.error) throw new SandboxError(`could not run '${bin} --version': ${out.error.message}. ${hint}`);
+  const first = String(out.stdout || "").trim().split("\n")[0] || "";
+  if (out.status !== 0 || !first.startsWith("kern ")) {
+    const shown = first ? JSON.stringify(first.slice(0, 120)) : "(no output)";
+    throw new SandboxError(
+      `'${bin}' is not kern: \`${bin} --version\` exited ${out.status} and printed ${shown}, where kern ` +
+        "prints a line beginning 'kern '. Refusing to run code, because a binary that is not kern would " +
+        `return an EMPTY, SUCCESSFUL result for every call and the code would never run. ${hint}`,
+    );
+  }
+  VERIFIED_KERN.add(key);
+}
+
+/** Locate `kern`: $KERN_BIN if set, else the first `kern` on $PATH. The result is also IDENTIFIED as
+ * kern (see `verifyIsKern`): being executable and being named `kern` are not the same as being kern. */
 function findKern() {
   const env = process.env.KERN_BIN;
   if (env) {
@@ -572,6 +630,7 @@ function findKern() {
     } catch {
       throw new SandboxError(`$KERN_BIN='${env}' is not an executable file`);
     }
+    verifyIsKern(env);
     return env;
   }
   const exts = [""];
@@ -581,7 +640,10 @@ function findKern() {
       const cand = path.join(d, "kern" + ext);
       try {
         fs.accessSync(cand, fs.constants.X_OK);
-        if (fs.statSync(cand).isFile()) return cand;
+        if (fs.statSync(cand).isFile()) {
+          verifyIsKern(cand);
+          return cand;
+        }
       } catch {
         /* keep looking */
       }
@@ -594,12 +656,21 @@ function findKern() {
     throw new SandboxError(
       "the `kern` binary was not found on PATH, and this is macOS: kern is Linux-only " +
         "(no namespaces, no cgroups on a Mac), so there is no macOS build to find. " +
-        "Run inside a Linux VM (colima, Lima, OrbStack, UTM), where kern installs normally, " +
-        "or set $KERN_BIN to a kern reachable from here. https://github.com/getkern/kern",
+        "Run inside a Linux VM (colima, Lima, OrbStack, UTM) and install it there with:\n" +
+        "    curl -fsSL https://raw.githubusercontent.com/getkern/kern/main/install.sh | sh\n" +
+        "or set $KERN_BIN to a kern reachable from here.",
     );
+  // THE COMMAND, NOT A LINK. `npm install kern-sandbox` does NOT bring the binary: this package is a
+  // wrapper around a process it does not ship, and the moment a user meets that fact is this error.
+  // It used to answer with a repository URL, which asks someone one paste away from working to go
+  // and read a page first. The same sentence the Python binding gives, deliberately: two wrappers
+  // around one runtime must not disagree about how to get it, and the installer line is the one the
+  // project's README leads with.
   throw new SandboxError(
-    "the `kern` binary was not found on PATH - install it " +
-      "(https://github.com/getkern/kern) or set $KERN_BIN",
+    "the `kern` binary was not found on PATH. `npm install kern-sandbox` installs this wrapper, " +
+      "not the runtime it drives - install kern with:\n" +
+      "    curl -fsSL https://raw.githubusercontent.com/getkern/kern/main/install.sh | sh\n" +
+      "or point $KERN_BIN at a kern you already have.",
   );
 }
 
@@ -875,7 +946,74 @@ function isKernDiagnostic(line) {
   return KERN_DIAGNOSTICS.some((p) => s.startsWith(p));
 }
 
+/** The sentence kern prints when it has READ the kernel's OOM counter for this box's own cgroup. A
+ * contract between two programs: if kern rewords it, this stops recognising a real OOM and starts
+ * reporting `killed` - wrong in the safe direction, still wrong. Mirrors `_KERN_OOM_MARKER`. */
+const KERN_OOM_MARKER = "killed by the kernel's OOM killer";
 
+/** kern's KERN_STARTED_FD payload, as `{ boxStarted, capSignal, oomSignal, workloadSignal }`.
+ *
+ * THE WIRE FORMAT IS SPELLED HERE AND NOWHERE ELSE, because it grew twice in one day (the OOM outcome,
+ * then the workload's signal) and each time a reader with its own copy of the layout was left behind.
+ * Three places in this file read those bytes; they now all read them through this.
+ *
+ *   byte 0 = the box started (kern reached its `Ok` arm with a code that is not 125)
+ *   byte 1 = the memory-cap enforcement signal: 0 undetermined, 1 enforced, 2 requested but not enforced
+ *   byte 2 = the OOM outcome: 1 iff the kernel's OOM killer fired against this box's own cgroup
+ *   byte 3 = the signal that terminated the workload, 0 if it exited on its own
+ *
+ * A SHORT buffer is an OLDER kern, not a malformed one: every absent byte reads as its "undetermined"
+ * value (0, and `null` for the signal, which must stay distinguishable from "nothing killed it").
+ * Mirrors `_parse_started_bytes`. */
+function parseStartedBytes(buf) {
+  const b = buf || Buffer.alloc(0);
+  return {
+    boxStarted: b.length >= 1 && b[0] === 1,
+    capSignal: b.length >= 2 ? b[1] : 0,
+    oomSignal: b.length >= 3 ? b[2] : 0,
+    workloadSignal: b.length >= 4 ? b[3] : null,
+  };
+}
+
+/** The byte kern writes to `KERN_ALIVE_FD` the moment it accepts the descriptor, before any box setup.
+ * It is what tells "the setup has not finished" from "this binary does not speak the protocol": an older
+ * kern never writes to that pipe AND never closes it, so the pipe is open and silent in both cases.
+ * Mirrors `_ALIVE_ACK`. */
+const ALIVE_ACK = 0x41; // 'A'
+
+/** Where kern was when the deadline fired, read off the `KERN_ALIVE_FD` pipe. Spelled once so a caller
+ * cannot invent a fourth answer. Mirrors the `_ALIVE_*` constants in the Python binding. */
+const ALIVE_PAST_SETUP = "past-setup"; // the workload ran (EOF at execvp), or kern reported setup failed
+const ALIVE_IN_SETUP = "in-setup"; // kern acknowledged the channel and is still BUILDING the box
+const ALIVE_UNKNOWN = "unknown"; // nothing on the pipe: a kern that predates this channel
+
+/** True iff KERN said the kernel's OOM killer took this box against its own memory cap.
+ *
+ * The ONE definition of "this was an OOM", used by all three death paths (the one-shot exit-code
+ * classifier, the resident kernel's death, and a pool box that died) so they cannot drift into
+ * disagreeing about the same box. It is an OBSERVATION - kern reads `memory.events` and says so -
+ * where the SDK can only infer, and the inference it replaced was MEASURED wrong in both directions:
+ * `kern stop` during a cell was reported `oom`, while a real OOM on the resident kernel was reported
+ * as a box that failed to start (and raised).
+ *
+ * Anchored on kern's `kern:` line prefix, which the real line carries (measured verbatim: `kern: the
+ * workload was killed by the kernel's OOM killer against this box's own memory cap.`), MINUS the
+ * benign diagnostics - `kern: note: <quoting the sentence>` is kern TALKING about an OOM, not
+ * reporting one.
+ *
+ * THE FALLBACK, NOT THE AUTHORITY. Against a kern that writes the 3rd KERN_STARTED_FD byte the verdict
+ * comes from there instead, on a pipe the workload never holds. This covers an older binary, and it is
+ * forgeable in exactly one direction: a workload that writes the whole prefixed sentence itself turns its
+ * own `killed` into `oom`. Both are sandbox faults, and timeout / blocked-escape are decided by exit code
+ * before any text is read, so the worst case is a caller misleading itself about its own kill.
+ * Mirrors `_kern_reported_oom`. */
+function kernReportedOom(stderr) {
+  for (const line of String(stderr || "").split("\n")) {
+    const s = line.replace(/^\s+/, "");
+    if (s.startsWith("kern:") && !isKernDiagnostic(s) && s.includes(KERN_OOM_MARKER)) return true;
+  }
+  return false;
+}
 
 function looksLikeStartupFailure(stderr) {
   const markers = [
@@ -889,9 +1027,13 @@ function looksLikeStartupFailure(stderr) {
     "error: oci:",
     "error: image:",
   ];
+  // The OOM sentence is skipped for a sharper reason than the benign notes: it is a report about a box
+  // that RAN, and it is `kern:`-prefixed, so it used to satisfy this predicate. MEASURED, that is how a
+  // real OOM on a resident kernel came back as `startup_failed` and was THROWN instead of returning an
+  // `oom` fault.
   for (const line of stderr.split("\n")) {
     const s = line.replace(/^\s+/, "");
-    if (isKernDiagnostic(s)) continue;
+    if (isKernDiagnostic(s) || kernReportedOom(s)) continue;
     if (s.includes("sandbox setup failed") || markers.some((m) => s.startsWith(m))) return true;
   }
   return false;
@@ -1415,32 +1557,71 @@ class Sandbox {
     // OLD kern never writes it, `boxStarted` stays false, and the stderr heuristic stands (backward
     // compatible).
     childEnv.KERN_STARTED_FD = "3";
+    // A SECOND, LIVE channel, because the first one is post-mortem. kern writes KERN_STARTED_FD at the
+    // box's TEARDOWN, so when OUR deadline fires we kill kern before that write and learn nothing: a
+    // workload that was slow and a kern whose SETUP blocked are the same overrun. fd 4 carries kern's
+    // readiness pipe: the ack byte on acceptance, EOF when the workload `execvp`s (the box child marks it
+    // FD_CLOEXEC), one byte if setup or exec failed, and nothing at all from a kern that predates it.
+    childEnv.KERN_ALIVE_FD = "4";
 
     const started = process.hrtime.bigint();
     return new Promise((resolve, reject) => {
       let child;
       let boxStarted = false;
       let capSignal = 0; // 2nd started byte: 0 undetermined/old-kern, 1 memory cap enforced, 2 not enforced
+      let oomSignal = 0; // 3rd started byte: 1 = the kernel OOM-killed this box's own cgroup, 0 = it did not
+      let workloadSignal = null; // 4th started byte: the signal that killed the workload, 0 = it exited
       try {
         // detached: own process group, so we can signal the box + kern as a unit (killpg).
         // The 4th stdio slot is fd 3: the child (kern) writes the started byte, the parent reads it.
         child = spawn(argv[0], argv.slice(1), {
           env: childEnv,
           detached: true,
-          stdio: ["ignore", "pipe", "pipe", "pipe"],
+          stdio: ["ignore", "pipe", "pipe", "pipe", "pipe"],
         });
       } catch (e) {
         this._removeEnvFile(name);
         return reject(new SandboxError(`could not spawn the box: ${e.message}`));
       }
 
+      // The alive channel's state, updated as the pipe speaks. Read at the deadline, BEFORE the kill:
+      // the teardown closes every write end, so afterwards the pipe reads EOF whatever the box was doing
+      // and the question answers itself wrongly.
+      let aliveState = ALIVE_UNKNOWN;
+      const aliveCh = child.stdio[4];
+      if (aliveCh) {
+        aliveCh.on("data", (b) => {
+          if (aliveState === ALIVE_PAST_SETUP) return;
+          for (const byte of b) {
+            if (byte === ALIVE_ACK) {
+              if (aliveState === ALIVE_UNKNOWN) aliveState = ALIVE_IN_SETUP;
+            } else {
+              aliveState = ALIVE_PAST_SETUP; // a byte beyond the ack: kern said setup failed
+              return;
+            }
+          }
+        });
+        // EOF: the box child's FD_CLOEXEC closed it at `execvp`, so the workload ran.
+        aliveCh.on("end", () => {
+          aliveState = ALIVE_PAST_SETUP;
+        });
+        aliveCh.on("error", () => {});
+      }
       const startedCh = child.stdio[3];
       if (startedCh) {
         // Byte 0 (0x01) = the box started; stream end with no byte = never started / old kern. Byte 1
-        // (a NEWER kern only, same atomic write) = the memory-cap enforcement signal; absent = 0.
+        // (a NEWER kern only, same atomic write) = the memory-cap enforcement signal; absent = 0. Byte 2
+        // (a NEWER kern still) = the OOM OUTCOME: 1 iff the kernel's OOM killer fired against this box's
+        // OWN cgroup. Enforcement is not an outcome, and this byte is the only place the outcome arrives
+        // on a channel the workload cannot write.
+        // ACCUMULATED rather than read off one chunk: kern's write is atomic, so all four bytes arrive
+        // together in practice, but a stream that split them would silently cost us the last one - and a
+        // lost OOM byte reads as "no OOM", a lost signal byte as "nothing killed it", both wrong answers
+        // arrived at invisibly.
+        let sig = Buffer.alloc(0);
         startedCh.on("data", (b) => {
-          if (b.length && b[0] === 1) boxStarted = true;
-          if (b.length >= 2) capSignal = b[1];
+          sig = Buffer.concat([sig, b]);
+          ({ boxStarted, capSignal, oomSignal, workloadSignal } = parseStartedBytes(sig));
         });
         startedCh.on("error", () => {});
       }
@@ -1467,7 +1648,9 @@ class Sandbox {
         const stdout = out.buffer().toString("utf8");
         const stderr = err.buffer().toString("utf8");
         const rc = toRc(code, signal);
-        let fault = this._classify(rc, signal, stderr, timedOut, timeoutS, capSignal);
+        let fault = this._classify(
+          rc, signal, stderr, timedOut, timeoutS, capSignal, oomSignal, aliveState, workloadSignal,
+        );
         const execFail = execFailureBinary(stderr);
         if (execFail !== null && rc !== 0) {
           // BEFORE the suppression below, which would erase it: the box started, so that branch
@@ -1565,37 +1748,75 @@ class Sandbox {
     }
   }
 
-  _classify(rc, signal, stderr, timedOut, timeoutS, capSignal = 0) {
+  _classify(
+    rc, signal, stderr, timedOut, timeoutS, capSignal = 0, oomSignal = 0, aliveState = ALIVE_UNKNOWN,
+    workloadSignal = null,
+  ) {
     // ORDER IS A SECURITY PROPERTY: deterministic-by-exit-code classes are decided BEFORE the stderr
     // heuristic, because stderr is a channel the workload controls.
-    if (timedOut)
+    if (timedOut) {
+      // AND THE DEADLINE ALONE DOES NOT SAY WHOSE FAULT IT WAS. `ALIVE_IN_SETUP` is kern's own answer,
+      // read off the readiness pipe while kern was still alive: the box was still being BUILT, so the
+      // code never ran and calling this a `timeout` would tell the caller their workload was slow. That
+      // class was measured with a FIFO volume source (404 seconds in `wait_for_partner`), and the shapes
+      // behind it - an `lstat` on a dead NFS mount, a FUSE whose daemon is gone - are not FIFOs and
+      // cannot be refused by type. Every other state keeps the old verdict, `ALIVE_UNKNOWN` (an older
+      // kern) included: absence of evidence is not evidence.
+      if (aliveState === ALIVE_IN_SETUP)
+        return sandboxFault(
+          "startup_failed",
+          `the box never started: kern was still setting it up when the ${timeoutS ?? this.timeoutS}s ` +
+            "deadline fired, so the code never ran. A host path that blocks is what does this - a bind " +
+            "source on a dead NFS or a FUSE mount whose daemon is gone, an image layer on a stalled " +
+            "disk - and the remedy is that path, not a longer timeout",
+        );
       return sandboxFault(
         "timeout",
         `exceeded the ${timeoutS ?? this.timeoutS}s time limit (killed by the binding)`,
       );
-    if (rc === EXIT_SIGSYS || signal === "SIGSYS")
+    }
+    // THE EXIT CODE IS THE RIGHT THING TO PROPAGATE AND THE WRONG THING TO CLASSIFY FROM: kern reports
+    // the workload's status as `128 + N`, so a workload the kernel killed and one that called `exit(137)`
+    // are the same number. MEASURED through the Python binding: `sys.exit(137)` came back `killed` with a
+    // message about an external kill that never happened, and `sys.exit(159)` came back `escape_blocked`,
+    // a security event a cell could fabricate in one line. kern's 4th started-byte carries the signal.
+    // `null` (an older kern, or a kern our teardown killed first) keeps the old exit-code reading:
+    // absence of evidence is not evidence. `signal === "SIG..."` is a different question, kern ITSELF
+    // being signalled, and stays as it was.
+    const killedBy = (n) => workloadSignal === null || workloadSignal === n;
+    if ((rc === EXIT_SIGSYS && killedBy(SIG_SYS)) || signal === "SIGSYS")
       return sandboxFault("escape_blocked", "a syscall was blocked by the seccomp filter (SIGSYS)");
-    if (rc === EXIT_SIGKILL || signal === "SIGKILL") {
-      // A memory-capped box SIGKILLed is the cgroup OOM-killer - what a breached memory.max does (kern
-      // sets memory.oom.group=1, so the whole box dies at once). `capSignal` is kern's UNFORGEABLE
-      // enforcement byte (2nd byte of KERN_STARTED_FD, not the workload's stderr): 1 = enforced, 2 =
-      // requested but NOT enforced here, 0 = undetermined (old kern / no --memory). Claim `oom` when a
-      // --memory cap was set AND kern did not report it unenforced (capSignal !== 2): enforced (1) is a
-      // certain cgroup OOM, undetermined (0) keeps the pre-signal heuristic. When kern reports the cap did
-      // not bind (2), a SIGKILL cannot be attributed to the box's cgroup - keep the honest `killed`.
-      if (this.memoryMb !== null && capSignal !== 2)
+    if ((rc === EXIT_SIGKILL && killedBy(SIG_KILL)) || signal === "SIGKILL") {
+      // Only kern's own OOM sentence buys the `oom` label (`kernReportedOom`, the one definition shared
+      // with the resident-kernel and pool death paths).
+      //
+      // WHAT THIS REPLACED, because the replaced version read as sound: a SIGKILL of a memory-capped box
+      // was called the cgroup OOM-killer, which is what a breached memory.max does (kern sets
+      // memory.oom.group=1, so the whole box goes at once). MEASURED: `kern stop` during a cell returns
+      // 137, so it came back `oom`, and an agent branching on the fault would retry with MORE MEMORY a
+      // kill that had nothing to do with memory. A confident wrong answer is worse than no answer.
+      if (oomSignal === 1 || kernReportedOom(stderr))
         return sandboxFault(
           "oom",
           "the box exceeded its memory cap and was OOM-killed (SIGKILL, exit 137)" + this._scratchNote(),
         );
+      // `capSignal` (kern's UNFORGEABLE enforcement byte: 1 = enforced, 2 = requested but NOT enforced
+      // here, 0 = undetermined) no longer decides the TYPE - a SIGKILL on a capped box is not evidence of
+      // an OOM, whatever the byte says - and a 2 still earns its own sentence, because "your cap was not
+      // in force here" is the one thing the caller cannot find out for itself.
       if (capSignal === 2)
         return sandboxFault(
           "killed",
-          "the box was SIGKILLed, but its memory cap was not enforced here (no cgroup delegation), so it is not attributed to a cgroup OOM",
+          "the box was SIGKILLed, and its memory cap was not enforced here (no cgroup delegation), so no memory limit was in force to attribute it to",
+        );
+      if (this.memoryMb !== null)
+        return sandboxFault(
+          "killed",
+          "the box was SIGKILLed and the kernel reported no OOM against its memory cap: this is an external kill (`kern stop`, a signal, or the host's own OOM killer), not the box exceeding its own memory",
         );
       return sandboxFault("killed", "the box was killed (SIGKILL); no memory cap was set to attribute it to OOM");
     }
-    if (rc === EXIT_SIGTERM || signal === "SIGTERM")
+    if ((rc === EXIT_SIGTERM && killedBy(SIG_TERM)) || signal === "SIGTERM")
       return sandboxFault("timeout", "the box exceeded its time limit (reaped by kern's timeout backstop)");
     // Box-not-started: a non-zero exit whose stderr carries kern's OWN setup markers (printed by the
     // PARENT before the box runs). kern's box-not-started paths BOTH exit 125 AND print a `kern:` marker,
@@ -2155,10 +2376,12 @@ class Kernel {
     this._waiters = []; // FIFO of { resolve, timer }; one reply per request keeps them in order
     this._stderr = Buffer.alloc(0);
     this._dead = false;
-    // kern's memory-cap enforcement byte (2nd byte of KERN_STARTED_FD). For a RESIDENT box kern writes
-    // it only at box teardown (a cell kills the kernel), so it arrives ~concurrent with the death we
-    // detect on stdout; read once, bounded, on death (`_readCapSignal`). 0 = undetermined / old kern.
-    this._capSignal = 0;
+    // kern's KERN_STARTED_FD bytes for a RESIDENT box: the enforcement byte (2nd) and the OOM-outcome
+    // byte (3rd). kern writes them only at box teardown (a cell kills the kernel), so they arrive
+    // ~concurrent with the death we detect on stdout; read once, bounded, on death (`_readCapSignal`).
+    // Kept as the raw buffer because the bytes arrive in ONE atomic write and a stream is free to deliver
+    // it in pieces. Absent bytes read as 0 = undetermined / old kern.
+    this._startedSig = Buffer.alloc(0);
   }
 
   async _open() {
@@ -2185,7 +2408,7 @@ class Kernel {
     });
     const startedCh = this._child.stdio[3];
     if (startedCh) {
-      startedCh.on("data", (b) => { if (b.length >= 2) this._capSignal = b[1]; });
+      startedCh.on("data", (b) => { this._startedSig = Buffer.concat([this._startedSig, b]); });
       startedCh.on("error", () => {});
     }
     this._child.on("error", () => { this._dead = true; this._flush(null); });
@@ -2286,7 +2509,7 @@ class Kernel {
       return this._teardownResult("killed", `the kernel reply exceeded the ${this._cap}-byte cap`, started);
     if (reply === null) {
       const err = this._stderr.toString("utf8");
-      const [kind, dflt] = this._kernelDeathFault(err, await this._readCapSignal());
+      const [kind, dflt] = this._kernelDeathFault(err, ...(await this._readCapSignal()));
       return this._teardownResult(kind, err.trim() || dflt, started);
     }
     return this._resultFromReply(reply, started);
@@ -2333,41 +2556,53 @@ class Kernel {
     });
   }
 
-  /** Why the resident kernel box died mid-cell, as `[type, defaultMessage]`. A kern setup marker on
-   * stderr means it never came up (startup_failed). Otherwise this is the runCode counterpart of the
-   * one-shot _classify SIGKILL branch - a kernel death has no per-cell exit code. `capSignal` is kern's
-   * unforgeable enforcement byte (0 = old kern / undetermined, 1 = memory cap enforced, 2 = requested
-   * but NOT enforced): with a memoryMb cap AND not-reported-unenforced (`!== 2`), the cgroup OOM-killer
-   * is the cause -> `oom`; when kern reports the cap did not bind (2), the kill is not attributable to
-   * the box's cgroup -> `killed`; uncapped is also `killed`. */
-  _kernelDeathFault(err, capSignal = 0) {
+  /** Why the resident kernel box died mid-cell, as `[type, defaultMessage]`. The runCode counterpart of
+   * the one-shot _classify SIGKILL branch: a kernel death has no per-cell exit code, so the whole verdict
+   * is made here from what kern wrote.
+   *
+   * ORDER, and it was measured wrong before: kern's OOM sentence is asked about FIRST, because it is
+   * `kern:`-prefixed and so was also matching the box-did-not-start heuristic below. A real OOM on a
+   * resident kernel therefore came back `startup_failed`, which `_teardownResult` THROWS - so the
+   * flagship path could not produce an `oom` fault at all, while an external `kern stop` DID produce one
+   * from the memoryMb inference. Two defects pointing opposite ways.
+   *
+   * `capSignal` is kern's unforgeable enforcement byte (0 = old kern / undetermined, 1 = cap enforced, 2 =
+   * requested but NOT enforced). It no longer decides the TYPE, and a 2 still earns a sentence, because
+   * "your cap was not in force here" is the one thing the caller cannot find out for itself. */
+  _kernelDeathFault(err, capSignal = 0, oomSignal = 0) {
+    if (oomSignal === 1 || kernReportedOom(err)) return ["oom", "the kernel box exceeded its memory cap and was OOM-killed"];
     if (looksLikeStartupFailure(err)) return ["startup_failed", "the kernel box failed to start"];
-    if (this._sbx.memoryMb !== null && capSignal !== 2)
-      return ["oom", "the kernel box was OOM-killed (it exceeded its memory cap)"];
     if (capSignal === 2)
       return [
         "killed",
-        "the kernel box was SIGKILLed, but its memory cap was not enforced here (no cgroup delegation), so it is not attributed to a cgroup OOM",
+        "the kernel box was killed, and its memory cap was not enforced here (no cgroup delegation), so no memory limit was in force to attribute it to",
+      ];
+    if (this._sbx.memoryMb !== null)
+      return [
+        "killed",
+        "the kernel box was killed and the kernel reported no OOM against its memory cap: an external kill (`kern stop`, a signal, or the host running out of memory), not the box exceeding its own memory",
       ];
     return ["killed", "the kernel box exited"];
   }
 
-  /** kern's memory-cap enforcement byte for the resident box, read ONCE on kernel death. kern writes
-   * the two-byte KERN_STARTED_FD signal only at box teardown (a resident box exits when a cell kills it),
-   * ~concurrent with the death detected on stdout. The fd-3 `data` handler in `_open` records the byte
-   * as it arrives; this awaits a BOUNDED window (the fd's own `end`, or 1 s) so the read is deterministic
-   * rather than a race, then returns the byte (0 = EOF / old kern / not yet -> the memoryMb heuristic). */
+  /** kern's enforcement and OOM-outcome bytes for the resident box, read ONCE on kernel death, as
+   * `[capSignal, oomSignal]`. kern writes the KERN_STARTED_FD signal only at box teardown (a resident box
+   * exits when a cell kills it), ~concurrent with the death detected on stdout. The fd-3 `data` handler in
+   * `_open` accumulates the bytes as they arrive; this awaits a BOUNDED window (the fd's own `end`, or
+   * 1 s) so the read is deterministic rather than a race. `[0, 0]` on EOF / an old kern / not yet, which
+   * falls back to kern's stderr sentence. */
   async _readCapSignal() {
     const ch = this._child && this._child.stdio && this._child.stdio[3];
-    if (!ch) return 0;
-    if (this._capSignal === 0 && !ch.destroyed) {
+    if (!ch) return [0, 0];
+    if (this._startedSig.length < 3 && !ch.destroyed) {
       await new Promise((res) => {
         const t = setTimeout(res, 1000);
         ch.once("end", () => { clearTimeout(t); res(); });
         ch.once("error", () => { clearTimeout(t); res(); });
       });
     }
-    return this._capSignal;
+    const { capSignal, oomSignal } = parseStartedBytes(this._startedSig);
+    return [capSignal, oomSignal];
   }
 
   _teardownResult(type, message, started) {
@@ -2503,7 +2738,7 @@ class WarmBox {
     this._born = Date.now();
     this._spent = false;
     this._rc = null;
-    this._capSignal = 0;
+    this._startedSig = Buffer.alloc(0); // KERN_STARTED_FD bytes: [started, cap enforcement, OOM outcome]
     this._stderr = Buffer.alloc(0);
     this._chunks = [];
     this._total = 0;
@@ -2545,7 +2780,7 @@ class WarmBox {
     }
     const startedCh = this._child.stdio[3];
     if (startedCh) {
-      startedCh.on("data", (b) => { if (b.length >= 2) this._capSignal = b[1]; });
+      startedCh.on("data", (b) => { this._startedSig = Buffer.concat([this._startedSig, b]); });
       startedCh.on("error", () => {});
     }
     this._child.on("error", () => { this._dead = true; this._flush(null); });
@@ -2730,18 +2965,26 @@ class WarmBox {
         fault: { type: "timeout", message: msg || "the code exceeded its deadline" },
       });
     }
-    const capSignal = this._capSignal;
+    const { capSignal, oomSignal } = parseStartedBytes(this._startedSig);
     this.retire();
-    if (looksLikeStartupFailure(err)) throw new SandboxError(err.trim() || "the box failed to start");
     let type = "killed";
     let dflt = "the box exited before the code finished";
-    if (this._sbx.memoryMb !== null && this._sbx.memoryMb !== undefined && capSignal !== 2) {
+    // Same order, and for the same measured reason, as `_kernelDeathFault`: kern's OOM sentence carries
+    // the `kern:` prefix that `looksLikeStartupFailure` matches on, so asking about the start SECOND is
+    // what keeps a pool box's OOM from being thrown as a box that never came up.
+    if (oomSignal === 1 || kernReportedOom(err)) {
       type = "oom";
-      dflt = "the box was OOM-killed (it exceeded its memory cap)";
+      dflt = "the box exceeded its memory cap and was OOM-killed";
+    } else if (looksLikeStartupFailure(err)) {
+      throw new SandboxError(err.trim() || "the box failed to start");
     } else if (capSignal === 2) {
       dflt =
-        "the box was SIGKILLed, but its memory cap was not enforced here (no cgroup delegation), " +
-        "so it is not attributed to a cgroup OOM";
+        "the box was killed, and its memory cap was not enforced here (no cgroup delegation), " +
+        "so no memory limit was in force to attribute it to";
+    } else if (this._sbx.memoryMb !== null && this._sbx.memoryMb !== undefined) {
+      dflt =
+        "the box was killed and the kernel reported no OOM against its memory cap: an external kill " +
+        "(`kern stop`, a signal, or the host running out of memory), not the box exceeding its own memory";
     }
     return this._result("", "", this._exitCode(), started, before, {
       fault: { type, message: err.trim() || dflt },

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "kern-sandbox",
-  "version": "0.1.42",
+  "version": "0.2.0",
   "description": "kern is a fast, rootless sandbox and virtual resource runtime for any workload, including untrusted and AI-generated code; kern-sandbox is its Node/TypeScript binding. Run untrusted or agent-generated code (Python/JS/Bash) in a real, kernel-enforced box in single-digit milliseconds, with no cloud, no account and no VM.",
   "keywords": [
     "sandbox",