start-command 0.38.0 → 0.39.1

package/CHANGELOG.md

@@ -1,5 +1,33 @@
 # start-command
 
+## 0.39.1
+
+### Patch Changes
+
+- 55c57b0: Stop the Docker cgroup sampler when the counters can never be read (#247). On hosts whose containers have no `memory.events` (the `memory` controller is not delegated, e.g. a root `cgroup.subtree_control` of `cpuset cpu pids`) and whose local cgroup is hidden, the sampler started one `docker exec` and one `docker inspect -f {{.State.Running}}` every interval for the whole life of the task without ever recording a sample. Exit 3 (unreadable counters) from the long-lived stream or from the per-round read now records the reason and stops sampling, as a missing `sh` (exit 126/127) already did; the post-mortem reports `Memory:     unavailable (local cgroup unavailable; Docker exec exit=3: private cgroup v2 counters unreadable)` instead of the last round's error. Other failed rounds are still retried, but each one doubles the wait, up to 60 s (or the interval, if longer); a sample restores the interval. The completion watcher (`docker wait`) still finishes the run.
+
+  Attached `--isolated screen` runs no longer report a failed command as exit 0 when `screen -dmS` returns before `screen -ls` lists the new session. A session missing from `screen -ls` now counts as finished only once its exit-code file is written, or after 20 polls (about 2 s) without the session.
+
+## 0.39.0
+
+### Minor Changes
+
+- CI/CD no longer reports false results (#234). A rejected `npm publish` or any failing release command now fails the release instead of printing success, and the published version is confirmed on the registry: npm gets up to 20 minutes to show it, and a "cannot publish over" conflict is read as "already landed" rather than retried. A release push that loses the race for `main` is rebased and retried. Releases wait for every check their stage requires. A failing Rust test is reported as a failed test rather than as low coverage. `bun test -t <pattern>` keeps the 30s default timeout. Snapshot-resume tests in both languages use a private snapshot lock. The link checker fails on every broken live link, re-checks transient 429/5xx answers, persists its cache and uploads its report. ESLint warnings fail CI and the pre-commit hook. The changeset PR is opened with `RELEASE_PR_TOKEN` so its checks run. Step outputs reach `run:` blocks through `env`, tool versions are pinned, path-filtered workflows re-run when their scripts change, and the npm-publish tests no longer reach the real npm on macOS.
+
+  Support Docker `--memory-reservation` with sizes, daemon percentages and percentage ranges. Validate the soft cap against the effective hard memory limit, record it in status, and preserve or override it across resumes and recovery.
+
+  Reduce Docker cgroup sampler polling with a configurable five-second default, one namespace lookup per sampler, and local cgroup liveness checks. Set `START_COMMAND_CGROUP_SAMPLE_INTERVAL_SECONDS` to a positive whole number of seconds to override the default for attached and detached runs.
+
+  Main pushes publish again (#240). Since #234 the Release job was skipped on every push to `main`: at job level the wildcard `needs.*.result` list also saw the deliberately skipped changeset check behind `test`/`coverage`. Release now compares each needed job by name, and the JavaScript Stage prints every term of the release condition and fails a main push whose release was skipped although all of its needs succeeded.
+
+  Attribute a Docker daemon restart without the host journal (#241). With a mounted `docker.sock` (Docker-outside-of-Docker) or a remote daemon, an exit-137 task is now reported as `killed (docker daemon restart)` when another container finished within the 15 s shutdown window with exit 137/143, or was started again after it, and was not OOM-killed. `exitEvidence.daemonRestartSource` records whether the host journal or the Docker API supported the attribution, and `--status` collects the evidence itself when the detached watcher died with the daemon before it could log any.
+
+  Sample Docker cgroup counters through one long-lived `docker exec -i` per run when the cgroup is only readable through the daemon (a mounted `docker.sock`). The host paces it with one tick per interval on stdin, so the container runs only shell builtins and the stream ends with the sampler; the per-round exec stays as a fallback. Docker is asked for `{{.State.Running}}` only after a round without a sample, and `START_COMMAND_CGROUP_SAMPLE_INTERVAL_SECONDS=off` disables sampling.
+
+  `$ --status` and `$ --list` cost far less with many records (#243). The store is decoded once per content within a process, and a JSON copy of the decoded records next to `executions.lino` (tagged with the store's SHA-256) lets the next process skip the decode entirely. One `docker inspect --type container` now answers every docker probe of a query instead of one or two processes per record, and records whose container is already marked removed are no longer probed. Output is unchanged; probes the batch cannot answer (an old or stubbed `docker`) run one by one as before.
+
+  `--init` / `--no-init` run Docker's bundled tini (`docker run --init`) as PID 1 for docker isolation, so orphaned processes of a long-running task are reaped instead of piling up as zombies (#244). The default stays off; `START_COMMAND_DOCKER_INIT=1` turns it on for every docker launch on a host, and `--no-init` overrides that. The choice is stored with the record, shown as an `init` row in the start block and in `--status`, and carried into `--resume` snapshots and stacked launches; records from before keep running without it. Exit codes and OOM evidence are unchanged, but `--stop` (`docker stop`) now ends a command without a SIGTERM handler with 143 at once instead of 137 after the grace period.
+
 ## 0.38.0
 
 ### Minor Changes

package/README.md

@@ -73,9 +73,11 @@ Released into the public domain under the [Unlicense](../LICENSE).
 
 ### Docker resource and recovery controls
 
-Launch with `--memory 64m`, `--memory-swap 64m` and `--cpus 1`, or use daemon
+Launch with `--memory 64m`, `--memory-swap 64m`, `--memory-reservation 32m` and `--cpus 1`, or use daemon
 percentages and uniform ranges such as `--memory '70%-80%'`. Memory-swap defaults
-to memory. `--cpu-penalty` enables delayed caps and reports state through status.
+to memory. Reservation is a soft cap and must be below the hard memory cap when
+one is set. It accepts the same sizes and percentages, is shown in `--status`,
+and survives resumes. `--cpu-penalty` enables delayed caps and reports state through status.
 Manual `--resume <id> --memory 128m` updates before starting;
 `--on-kill-resume-memory` changes memory only after a fresh qualifying OOM.
 See the [full resource and recovery guide](../README.md#docker-launch-limits-recovery-limits-and-cpu-penalty)

package/bunfig.toml

@@ -1,3 +1,7 @@
 # Bun configuration
 # Tests use repository naming without conventional suffixes, so package scripts pass
 # the test files explicitly instead of relying on Bun's default discovery.
+
+[test]
+# Every test process gets a private Docker snapshot lock (issue #234).
+preload = ["./test/setup/private-snapshot-lock.js"]

package/eslint.config.mjs

@@ -25,6 +25,7 @@ export default [
         process: 'readonly',
         Buffer: 'readonly',
         fetch: 'readonly',
+        structuredClone: 'readonly',
         Bun: 'readonly',
       },
     },

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "start-command",
-  "version": "0.38.0",
+  "version": "0.39.1",
   "description": "Gamification of coding, execute any command with ability to auto-report issues on GitHub",
   "main": "src/bin/cli.js",
   "exports": {
@@ -12,12 +12,12 @@
   },
   "scripts": {
     "test": "bun ../scripts/run-js-tests.mjs",
-    "lint": "eslint .",
+    "lint": "eslint . --max-warnings 0",
     "lint:fix": "eslint . --fix",
     "format": "prettier --write .",
     "format:check": "prettier --check .",
     "check:file-size": "bun ../scripts/check-file-size.mjs",
-    "lint:scripts": "cd .. && eslint .",
+    "lint:scripts": "cd .. && eslint . --max-warnings 0",
     "format:check:scripts": "cd .. && prettier --check \"scripts/**/*.{js,mjs,cjs}\"",
     "check": "bun run lint && bun run lint:scripts && bun run format:check && bun run format:check:scripts && bun run check:file-size",
     "prepare": "cd .. && husky .husky || true",
@@ -58,7 +58,7 @@
   },
   "lint-staged": {
     "*.{js,mjs,cjs}": [
-      "eslint --fix --max-warnings 6",
+      "eslint --fix --max-warnings 0 --no-warn-ignored",
       "prettier --write",
       "prettier --check"
     ],

package/src/lib/args-parser.js

@@ -14,6 +14,7 @@
  * --mount <mount-spec>             Docker --mount spec (repeatable, docker only)
  * --env, -e <KEY=VALUE>            Environment variable for docker container (repeatable, docker only)
  * --privileged                     Run docker container in privileged mode (docker only)
+ * --init, --no-init                Run an init process (tini) as PID 1 to reap orphans (docker only; default START_COMMAND_DOCKER_INIT)
  * --network <name>                 Connect docker container to a named network (repeatable, docker only)
  * --network-alias <alias>          Add an alias to the first network (repeatable, docker only)
  * --endpoint <endpoint>            SSH endpoint (required for ssh isolation, e.g., user@host)
@@ -47,6 +48,7 @@ const dockerNetworkOptions = require('./docker-network-options');
 const dockerRecoveryOptions = require('./docker-recovery-options');
 const cpuPenalty = require('./cpu-penalty');
 const dockerResourceOptions = require('./docker-resource-options');
+const dockerInitOption = require('./docker-init-option');
 const snapshotResumeOptions = require('./snapshot-resume-options');
 const launchIdentityOptions = require('./launch-identity-options');
 const { parseSequence, isSequence } = require('./sequence-parser');
@@ -199,6 +201,7 @@ function parseArgs(args) {
     labels: [], // Caller attribution, applied to every Docker level
     help: false,
     privileged: false, // Run docker container in privileged mode
+    init: null, // docker run --init; null until resolved from START_COMMAND_DOCKER_INIT (issue #244)
     network: null, // First Docker network name (compatibility accessor)
     networks: [], // Ordered Docker network names
     networkAliases: [], // Docker network-scoped aliases
@@ -430,6 +433,11 @@ function parseOption(args, index, options) {
     return 1;
   }
 
+  // --init / --no-init (for docker, issue #244)
+  if (dockerInitOption.parse(args, index, options)) {
+    return 1;
+  }
+
   const networkOption = dockerNetworkOptions.parse(args, index, options);
   if (networkOption) {
     return networkOption;
@@ -803,6 +811,7 @@ function validateOptions(options) {
 
   dockerRecoveryOptions.validate(options);
   dockerResourceOptions.validate(options);
+  dockerInitOption.validate(options);
   cpuPenalty.validate(options);
 
   // Session name is only valid with isolation

package/src/lib/cgroup-memory.js

@@ -27,8 +27,17 @@
 
 const { shellQuote } = require('./isolation-log-utils');
 
-/** Seconds between two samples of a running container's cgroup. */
-const CGROUP_SAMPLE_INTERVAL_SECONDS = 1;
+/** Default seconds between samples; the watcher environment may override it. */
+const CGROUP_SAMPLE_INTERVAL_SECONDS = 5;
+
+/**
+ * Longest wait between rounds that keep failing (issue #247), unless the
+ * interval itself is longer. Each round without a sample doubles the wait.
+ */
+const CGROUP_SAMPLE_BACKOFF_MAX_SECONDS = 60;
+
+/** Exit status of both in-container reads for unreadable counters. */
+const CGROUP_UNREADABLE_EXIT = 3;
 
 /** Shell variables used by the sampler snippets. */
 const CGROUP_SHELL_VARS = {
@@ -74,43 +83,125 @@ function buildCgroupFunctionsSnippet() {
   ].join('; ');
 }
 
+/**
+ * In-container loop of the long-lived sampler exec (issue #242): `$1` is the
+ * cgroup directory. The host paces it with one newline per interval on stdin
+ * (`docker exec -i`), so it runs only shell builtins: the task's own CPU
+ * limit pays for no `cat`, `sed` or `sleep` processes. Stdin reaches EOF when
+ * the exec's client is gone, so the loop never outlives the sampler; dockerd
+ * would accept its output even then. Exit 3: unreadable counters (no
+ * delegated `memory` controller), which no later round can change.
+ * @returns {string} Shell script for `sh -c`
+ */
+function buildCgroupStreamScript() {
+  return [
+    'while read -r __scx_tick; do __scx_oom=; __scx_kill=; __scx_max=-; __scx_peak=-',
+    `[ -r "$1/memory.events" ] || exit ${CGROUP_UNREADABLE_EXIT}`,
+    'while read -r __scx_key __scx_value; do case "$__scx_key" in ' +
+      'oom) __scx_oom=$__scx_value;; oom_kill) __scx_kill=$__scx_value;; esac; ' +
+      'done < "$1/memory.events"',
+    `case "$__scx_oom:$__scx_kill" in *[!0-9:]*|:*|*:) exit ${CGROUP_UNREADABLE_EXIT};; esac`,
+    '[ ! -r "$1/memory.max" ] || read -r __scx_max < "$1/memory.max"',
+    '[ ! -r "$1/memory.peak" ] || read -r __scx_peak < "$1/memory.peak"',
+    'printf \'%s %s %s %s %s\\n\' "${__scx_max:--}" "${__scx_peak:--}" ' +
+      '"$__scx_oom" "$__scx_kill" "$1" || exit 0; done',
+  ].join('; ');
+}
+
+/** `$0` of the long-lived exec, visible in `ps` inside the task container. */
+const CGROUP_STREAM_NAME = 'start-command-cgroup-stream';
+
+/** Interval value that turns the sampler off (issue #242). */
+const CGROUP_SAMPLING_OFF_REASON =
+  'sampling disabled by START_COMMAND_CGROUP_SAMPLE_INTERVAL_SECONDS=off';
+
 /**
  * Shell fragment starting the background sampler. It writes each sample to a
  * temp file with `mv`, so the watcher never reads a half-written line, and
- * ends by itself once the cgroup is gone.
+ * ends by itself once the container is gone.
+ *
+ * Where the cgroup is readable locally it reads it directly. Otherwise, with
+ * a private cgroup namespace (a mounted docker.sock), one long-lived
+ * `docker exec -i` answers one host tick per interval (issue #242); a
+ * successful read proves the container runs, so Docker is asked for liveness
+ * only after a failed round or when the stream ends. Without a usable stream it falls back
+ * to one `docker exec` per round; without `sh` in the image (exit 126/127) or
+ * without any readable cgroup (exit 3 from either exec, issue #247) it
+ * records the reason and stops. Other failed rounds are retried, each one
+ * doubling the wait up to CGROUP_SAMPLE_BACKOFF_MAX_SECONDS; a sample restores
+ * the interval. `START_COMMAND_CGROUP_SAMPLE_INTERVAL_SECONDS=off` disables
+ * sampling.
  * @param {string} containerName - Container to sample
  * @returns {string} Shell command
  */
 function buildCgroupSamplerStartSnippet(containerName) {
   const v = CGROUP_SHELL_VARS;
   const name = shellQuote(containerName);
-  const remoteRead = '__start_command_cgroup_read /sys/fs/cgroup';
+  const remoteRead = `__start_command_cgroup_read /sys/fs/cgroup || exit ${CGROUP_UNREADABLE_EXIT}`;
+  // Exit statuses no later round can change: no `sh`, or no counters.
+  const permanent = `${CGROUP_UNREADABLE_EXIT}|126|127`;
+  const backoff = CGROUP_SAMPLE_BACKOFF_MAX_SECONDS;
+  const unavailable = (detail) =>
+    `printf '%s\\n' "local cgroup unavailable; ${detail}" > "$${v.file}.reason"`;
+  const execReason = unavailable(
+    'Docker exec exit=$__scs_exit: ${__scs_detail:-private cgroup v2 counters unreadable}'
+  );
   return [
     buildCgroupFunctionsSnippet(),
     `${v.file}="\${TMPDIR:-/tmp}/start-command-cgroup.$$"`,
     `rm -f "$${v.file}" "$${v.file}.tmp" "$${v.file}.reason"`,
     `__start_command_memory_limit=$(docker inspect -f '{{.HostConfig.Memory}}' ${name} 2>/dev/null)`,
     `__start_command_sample_name=${name}`,
-    // The first counter is a baseline, including after watcher restart. Only
-    // changes between adjacent successful samples supply exit-time evidence.
-    `( export START_COMMAND_CGROUP_SAMPLING=1; __scs_previous=''; __scs_previous_at=0; __scs_changed=0; ` +
-      `__scs_dir=$(__start_command_cgroup_dir ${name}); while :; do ` +
-      '__scs_line=""; if [ -n "$__scs_dir" ]; then __scs_line=$(__start_command_cgroup_read "$__scs_dir"); fi; ' +
-      `if [ -z "$__scs_line" ]; then __scs_ns=$(docker inspect -f '{{.HostConfig.CgroupnsMode}}' ${name} 2>/dev/null); ` +
-      `if [ "$__scs_ns" = private ]; then ` +
-      `__scs_remote=$(docker exec ${name} sh -c ${shellQuote(`${buildCgroupFunctionsSnippet()}; ${remoteRead}`)} 2>&1); __scs_exit=$?; ` +
-      'if [ "$__scs_exit" = 0 ]; then __scs_line=$__scs_remote; ' +
-      'else __scs_detail=$(printf "%s" "$__scs_remote" | sed -n "1p" | cut -c 1-240); ' +
-      `printf '%s\n' "local cgroup unavailable; Docker exec exit=$__scs_exit: \${__scs_detail:-private cgroup v2 counters unreadable}" > "$${v.file}.reason"; fi; ` +
-      `else printf '%s\n' "local cgroup unavailable; container cgroup namespace=\${__scs_ns:-unavailable}, private required" > "$${v.file}.reason"; fi; fi; ` +
-      '__scs_now=$(date +%s); if [ -n "$__scs_line" ]; then ' +
-      '__scs_kill=$(printf "%s" "$__scs_line" | cut -d " " -f 4); ' +
-      'if [ -n "$__scs_previous" ] && [ "$__scs_kill" -gt "$__scs_previous" ] 2>/dev/null && [ "$((__scs_now-__scs_previous_at))" -le 3 ]; then __scs_changed=$__scs_now; fi; ' +
-      '__scs_previous=$__scs_kill; __scs_previous_at=$__scs_now; ' +
-      '__scs_fields=$(printf "%s" "$__scs_line" | cut -d " " -f 1-4); ' +
-      `printf '%s %s %s %s\\n' "$__scs_fields" "$__scs_changed" "$__scs_now" "$__scs_dir" > "$${v.file}.tmp" && mv -f "$${v.file}.tmp" "$${v.file}"; fi; ` +
-      `if [ "$(docker inspect -f '{{.State.Running}}' ${name} 2>/dev/null)" = false ]; then break; fi; ` +
-      `sleep ${CGROUP_SAMPLE_INTERVAL_SECONDS}; done ) >/dev/null 2>&1 & ${v.sampler}=$!`,
+    `( export START_COMMAND_CGROUP_SAMPLING=1; ` +
+      // Validate in the watcher, so attached and detached launches inherit the
+      // same setting. Never allow a malformed/zero sleep to create a busy loop.
+      `__scs_interval=\${START_COMMAND_CGROUP_SAMPLE_INTERVAL_SECONDS:-${CGROUP_SAMPLE_INTERVAL_SECONDS}}; ` +
+      `case "$__scs_interval" in [Oo][Ff][Ff]) printf '%s\\n' '${CGROUP_SAMPLING_OFF_REASON}' > "$${v.file}.reason"; exit 0;; ` +
+      `*[!0-9]*|'') __scs_interval=${CGROUP_SAMPLE_INTERVAL_SECONDS};; esac; ` +
+      `[ "$__scs_interval" -gt 0 ] 2>/dev/null || __scs_interval=${CGROUP_SAMPLE_INTERVAL_SECONDS}; ` +
+      // Store one `<max> <peak> <oom> <oom_kill>` line. The previous sample is
+      // read back from the file, so the stream's reader subshell can record
+      // too. The first counter is a baseline, including after watcher restart;
+      // only changes between adjacent samples supply exit-time evidence.
+      `__scs_record() { set -f; set -- $1; set +f; case "$3:$4" in *[!0-9:]*|:*|*:) return 1;; esac; ` +
+      `__scs_now=$(date +%s); __scs_previous=''; __scs_changed=0; __scs_previous_at=0; ` +
+      `[ ! -r "$${v.file}" ] || read -r __scs_p __scs_p __scs_p __scs_previous __scs_changed __scs_previous_at __scs_p < "$${v.file}"; ` +
+      'if [ -n "$__scs_previous" ] && [ "$4" -gt "$__scs_previous" ] 2>/dev/null && [ "$((__scs_now-__scs_previous_at))" -le 3 ]; then __scs_changed=$__scs_now; fi; ' +
+      `printf '%s %s %s %s %s %s %s\\n' "$1" "$2" "$3" "$4" "$__scs_changed" "$__scs_now" "$__scs_dir" > "$${v.file}.tmp" && mv -f "$${v.file}.tmp" "$${v.file}"; }; ` +
+      `__scs_dir=$(__start_command_cgroup_dir ${name}); ` +
+      // The namespace is fixed at container creation, including when reads fail.
+      `__scs_ns=$(docker inspect -f '{{.HostConfig.CgroupnsMode}}' ${name} 2>/dev/null); __scs_stream=1; __scs_reader=''; ` +
+      `__scs_delay=$__scs_interval; ` +
+      `while :; do __scs_ok=''; __scs_sampled=''; if [ -n "$__scs_dir" ]; then ` +
+      '__scs_line=$(__start_command_cgroup_read "$__scs_dir") && __scs_record "$__scs_line" && __scs_ok=1; ' +
+      `elif [ "$__scs_ns" != private ]; then ${unavailable('container cgroup namespace=${__scs_ns:-unavailable}, private required')}; break; ` +
+      `elif [ "$__scs_stream" = 1 ]; then ` +
+      // The reader ends with the stream; stopping the sampler stops it too.
+      `trap 'kill "$__scs_reader" 2>/dev/null; wait "$__scs_reader" 2>/dev/null; exit 0' TERM; ` +
+      // The ticker runs in the background, so a failed exec is reported at
+      // once; it ends on its first write after the exec is gone.
+      `{ { while :; do echo || exit 0; sleep "$__scs_interval" || exit 0; done 2>/dev/null & } | ` +
+      `docker exec -i ${name} sh -c ${shellQuote(buildCgroupStreamScript())} ${CGROUP_STREAM_NAME} /sys/fs/cgroup 2>&1; printf '__scs_exit %s\\n' "$?"; } | ` +
+      `{ __scs_got=''; __scs_exit=''; __scs_detail=''; while IFS= read -r __scs_line; do case "$__scs_line" in ` +
+      `'__scs_exit '*) __scs_exit=\${__scs_line#__scs_exit };; ` +
+      `*) if __scs_record "$__scs_line"; then __scs_got=1; elif [ -z "$__scs_detail" ]; then __scs_detail=$(printf '%s' "$__scs_line" | cut -c 1-240); fi;; esac; done; ` +
+      `[ -z "$__scs_got" ] || exit 0; ${execReason}; ` +
+      `case "$__scs_exit" in ''|0|*[!0-9]*) exit 1;; esac; exit "$__scs_exit"; } & ` +
+      `__scs_reader=$!; wait "$__scs_reader"; __scs_exit=$?; trap - TERM; __scs_reader=''; ` +
+      // No shell in the image or no counters: a per-round exec cannot
+      // succeed either.
+      `case "$__scs_exit" in 0) __scs_sampled=1;; ${permanent}) break;; *) __scs_stream=0;; esac; ` +
+      `else __scs_remote=$(docker exec ${name} sh -c ${shellQuote(`${buildCgroupFunctionsSnippet()}; ${remoteRead}`)} 2>&1); __scs_exit=$?; ` +
+      'if [ "$__scs_exit" = 0 ] && __scs_record "$__scs_remote"; then __scs_ok=1; ' +
+      `else __scs_detail=$(printf "%s" "$__scs_remote" | sed -n "1p" | cut -c 1-240); ${execReason}; ` +
+      `case "$__scs_exit" in ${permanent}) break;; esac; fi; fi; ` +
+      // A successful read needs no Docker liveness probe, and the completion
+      // watcher still stops us after docker wait.
+      `if [ -z "$__scs_ok" ] && [ "$(docker inspect -f '{{.State.Running}}' ${name} 2>/dev/null)" = false ]; then break; fi; ` +
+      // Back off while rounds keep failing, never below the interval.
+      `if [ -n "$__scs_ok$__scs_sampled" ]; then __scs_delay=$__scs_interval; else __scs_delay=$((__scs_delay*2)); ` +
+      `[ "$__scs_delay" -le ${backoff} ] || __scs_delay=${backoff}; [ "$__scs_delay" -ge "$__scs_interval" ] || __scs_delay=$__scs_interval; fi; ` +
+      `sleep "$__scs_delay"; done ) >/dev/null 2>&1 & ${v.sampler}=$!`,
   ].join('; ');
 }
 
@@ -304,14 +395,19 @@ function formatCgroupMemory(memory) {
 }
 
 module.exports = {
+  CGROUP_SAMPLE_BACKOFF_MAX_SECONDS,
   CGROUP_SAMPLE_INTERVAL_SECONDS,
+  CGROUP_SAMPLING_OFF_REASON,
   CGROUP_SHELL_VARS,
+  CGROUP_STREAM_NAME,
+  CGROUP_UNREADABLE_EXIT,
   OOM_SCOPE,
   OOM_SCOPE_NOTES,
   buildCgroupFunctionsSnippet,
   buildCgroupMemoryLogSnippet,
   buildCgroupSamplerStartSnippet,
   buildCgroupSamplerStopSnippet,
+  buildCgroupStreamScript,
   describeCgroupOomScope,
   formatCgroupMemory,
   formatCgroupMemoryLogLine,

package/src/lib/command-builder.js

@@ -60,6 +60,7 @@ function buildNextLevelCommand(options, command) {
     for (const [key, flag] of [
       ['memory', '--memory'],
       ['memorySwap', '--memory-swap'],
+      ['memoryReservation', '--memory-reservation'],
       ['cpus', '--cpus'],
       ['onKillResumeMemory', '--on-kill-resume-memory'],
     ]) {
@@ -112,6 +113,9 @@ function buildNextLevelCommand(options, command) {
     if (options.privileged) {
       parts.push('--privileged');
     }
+    if (options.init) {
+      parts.push('--init');
+    }
   }
 
   // Pass through global flags

package/src/lib/container-removal.js

@@ -13,6 +13,7 @@
  */
 
 const { spawnSync } = require('child_process');
+const { prefetchedInspect } = require('./docker-inspect-batch');
 
 /** `containerState` of a record whose container no longer exists. */
 const CONTAINER_STATE_REMOVED = 'removed';
@@ -50,11 +51,9 @@ function isDockerNotFound(result) {
  *   the inspect failed; `notFound` says the failure was "no such container"
  */
 function dockerInspect(dockerCommand, spawnOptions, format, target) {
-  const result = spawnSync(
-    dockerCommand,
-    ['inspect', '-f', format, target],
-    spawnOptions
-  );
+  const result =
+    prefetchedInspect(format, target) ||
+    spawnSync(dockerCommand, ['inspect', '-f', format, target], spawnOptions);
   if (result.error || result.status !== 0 || !result.stdout) {
     return { stdout: null, notFound: isDockerNotFound(result) };
   }

package/src/lib/detached-finalize.js

@@ -170,6 +170,7 @@ function finalizeDetachedExecution(options = {}) {
   );
   record.options.exitEvidence = {
     daemonRestart: logged.daemonRestart,
+    ...evidence.sourceOf(logged),
     mainOom:
       !logged.daemonRestart &&
       (logged.mainOom ||

package/src/lib/docker-init-option.js

@@ -0,0 +1,65 @@
+/**
+ * Parse and validate the opt-in init process for docker isolation
+ * (issue #244):
+ *   --init      run `docker run --init`, so Docker's bundled tini is PID 1
+ *               and reaps orphaned processes
+ *   --no-init   keep the command as PID 1, overriding the environment default
+ *
+ * Without either flag, START_COMMAND_DOCKER_INIT=1 turns it on for every
+ * docker launch on the host. The default stays off: a command that is its own
+ * supervisor keeps today's behaviour, and recovery works the same for
+ * containers started with or without it.
+ */
+
+const DOCKER_INIT_ENV = 'START_COMMAND_DOCKER_INIT';
+
+/**
+ * Whether the environment asks for an init process by default.
+ * @param {object} [env] - Environment
+ * @returns {boolean}
+ */
+function dockerInitFromEnv(env = process.env) {
+  const value = String(env[DOCKER_INIT_ENV] || '')
+    .trim()
+    .toLowerCase();
+  return ['1', 'true', 'yes', 'on'].includes(value);
+}
+
+function parseDockerInitOption(args, index, options) {
+  const arg = args[index];
+  if (arg === '--init' || arg === '--no-init') {
+    options.init = arg === '--init';
+    return 1;
+  }
+  return 0;
+}
+
+/**
+ * Resolve `options.init` to a boolean: the flag if one was given (`null`
+ * means none), otherwise the environment default for launches whose stack
+ * includes docker. Idempotent, because `validateOptions` may run again on
+ * options it already resolved: `--no-init` (or a resolved `false`) without
+ * docker is the default anyway and is accepted.
+ * @param {object} options - Parsed options
+ * @param {object} [env] - Environment
+ * @throws {Error} If `--init` is given without docker isolation
+ */
+function validateDockerInitOption(options, env = process.env) {
+  const stack = options.isolatedStack || [options.isolated];
+  const hasDocker = options.isolated !== null && stack.includes('docker');
+  if (options.init === true && !hasDocker) {
+    throw new Error(
+      '--init option is only valid when isolation stack includes docker'
+    );
+  }
+  if (typeof options.init !== 'boolean') {
+    options.init = hasDocker && dockerInitFromEnv(env);
+  }
+}
+
+module.exports = {
+  DOCKER_INIT_ENV,
+  dockerInitFromEnv,
+  parse: parseDockerInitOption,
+  validate: validateDockerInitOption,
+};

package/src/lib/docker-inspect-batch.js

@@ -0,0 +1,176 @@
+/**
+ * One `docker inspect` for every container a status query probes (issue #243).
+ *
+ * `--list` used to start one or two `docker inspect` processes per record
+ * (the live state, the process IDs, the removal check), one after another,
+ * and `--status` up to three for its one record. While a query runs, this
+ * module asks Docker once for every field those probes read, and answers the
+ * per-record `docker inspect -f <format> <target>` calls from that answer.
+ *
+ * Docker prints one line per found container, in argument order, and one
+ * `No such container: <target>` line on stderr per missing one. Each line
+ * is matched back to its target by name or ID prefix (Docker's own lookup
+ * order), so the answer does not depend on the line order. A target the
+ * batch cannot account for (an old or stubbed docker, a daemon error) is
+ * not answered, and its per-record probe runs as before.
+ */
+
+const { spawnSync } = require('child_process');
+
+/** Fields of the batched inspect, in output order. */
+const BATCH_FIELDS = [
+  '.Name',
+  '.Id',
+  '.State.Pid',
+  '.State.Running',
+  '.State.ExitCode',
+  '.State.OOMKilled',
+  '.State.StartedAt',
+  '.State.FinishedAt',
+];
+const BATCH_FORMAT = BATCH_FIELDS.map((field) => `{{${field}}}`).join(' ');
+const NOT_FOUND_LINE = /no such (?:object|container):\s*(\S+)/gi;
+const PLACEHOLDER = /\{\{\s*(\.[A-Za-z.]+)\s*\}\}/g;
+
+/** Answers of the query in progress: target -> fields, or null (missing). */
+let prefetched = null;
+
+/**
+ * Containers the status probes of these records would inspect. Records
+ * whose container is already marked removed are skipped: they are never
+ * probed (issue #243).
+ * @param {Object[]} records - Execution records
+ * @returns {string[]} Unique names and IDs
+ */
+function dockerProbeTargets(records) {
+  const targets = new Set();
+  for (const record of records) {
+    const opts = (record && record.options) || {};
+    if (opts.isolated !== 'docker' || record.containerState === 'removed') {
+      continue;
+    }
+    for (const target of [opts.sessionName, opts.containerId]) {
+      if (typeof target === 'string' && target) {
+        targets.add(target);
+      }
+    }
+  }
+  return [...targets];
+}
+
+/**
+ * Match a batched inspect answer to its targets.
+ * @param {string[]} targets - Names and IDs passed to docker
+ * @param {string} stdout - One line of `BATCH_FORMAT` per found container
+ * @param {string} stderr - Docker's errors
+ * @returns {Map<string, Object|null>} Fields per answered target, null for
+ *   a target Docker reported missing; unanswered targets are absent
+ */
+function parseBatchInspect(targets, stdout, stderr) {
+  const rows = String(stdout || '')
+    .split('\n')
+    .map((line) => line.trim().split(/\s+/))
+    .filter((parts) => parts.length === BATCH_FIELDS.length)
+    .map((parts) =>
+      Object.fromEntries(BATCH_FIELDS.map((field, i) => [field, parts[i]]))
+    );
+  const missing = new Set(
+    [...String(stderr || '').matchAll(NOT_FOUND_LINE)].map((m) => m[1])
+  );
+  const answers = new Map();
+  for (const target of targets) {
+    // Docker's own "no such container" wins over a prefix of another
+    // target's ID.
+    const row = missing.has(target)
+      ? null
+      : rows.find((r) => r['.Name'] === `/${target}`) ||
+        rows.find((r) => r['.Id'].startsWith(target));
+    if (row !== undefined) {
+      answers.set(target, row);
+    }
+  }
+  return answers;
+}
+
+/**
+ * Run the batched inspect.
+ * @returns {Map<string, Object|null>} See `parseBatchInspect`
+ */
+function runBatchInspect(targets, dockerCommand, spawnOptions) {
+  if (targets.length === 0) {
+    return new Map();
+  }
+  const result = spawnSync(
+    dockerCommand,
+    ['inspect', '--type', 'container', '-f', BATCH_FORMAT, ...targets],
+    spawnOptions
+  );
+  if (result.error) {
+    return new Map();
+  }
+  return parseBatchInspect(targets, result.stdout, result.stderr);
+}
+
+/**
+ * Run `fn` with the docker probes of `records` answered by one inspect.
+ * Nested calls reuse the outer answer.
+ * @param {Object[]} records - Records the query will probe
+ * @param {{dockerCommand: string, spawnOptions: Object}} docker - How to run docker
+ * @param {function(): *} fn - The query
+ * @returns {*} What `fn` returns
+ */
+function withDockerInspectBatch(records, docker, fn) {
+  if (prefetched) {
+    return fn();
+  }
+  prefetched = runBatchInspect(
+    dockerProbeTargets(records),
+    docker.dockerCommand,
+    docker.spawnOptions
+  );
+  try {
+    return fn();
+  } finally {
+    prefetched = null;
+  }
+}
+
+/**
+ * Answer `docker inspect -f <format> <target>` from the batch.
+ * @param {string} format - Go template of the per-record probe
+ * @param {string} target - Container name or ID
+ * @returns {{success: boolean, status: number, stdout: string,
+ *   stderr: string}|null} A spawn-like result, or null when the batch has
+ *   no answer and the probe must run
+ */
+function prefetchedInspect(format, target) {
+  if (!prefetched || !prefetched.has(target)) {
+    return null;
+  }
+  const row = prefetched.get(target);
+  if (row === null) {
+    return {
+      success: false,
+      status: 1,
+      stdout: '',
+      stderr: `Error response from daemon: No such container: ${target}\n`,
+    };
+  }
+  let known = true;
+  const stdout = format.replace(PLACEHOLDER, (_, field) => {
+    known = known && field in row;
+    return row[field];
+  });
+  if (!known || stdout.includes('{{')) {
+    return null;
+  }
+  return { success: true, status: 0, stdout: `${stdout}\n`, stderr: '' };
+}
+
+module.exports = {
+  BATCH_FORMAT,
+  dockerProbeTargets,
+  parseBatchInspect,
+  prefetchedInspect,
+  withDockerInspectBatch,
+};