cdx-manager 0.25.2 → 0.25.4

package/README.md

@@ -4,7 +4,7 @@
 
 <br clear="left"/>
 
-[![License](https://img.shields.io/badge/license-MIT-4C8BF5)](LICENSE) ![Version](https://img.shields.io/badge/version-v0.25.2-4C8BF5) ![Python](https://img.shields.io/badge/python-3.9%2B-3776AB?logo=python&logoColor=white)
+[![License](https://img.shields.io/badge/license-MIT-4C8BF5)](LICENSE) ![Version](https://img.shields.io/badge/version-v0.25.4-4C8BF5) ![Python](https://img.shields.io/badge/python-3.9%2B-3776AB?logo=python&logoColor=white)
 
 **The right AI coding account, when you need it.** CDX Manager keeps your Codex and Claude accounts in separate profiles, shows which ones have quota, and helps you pick the next usable session from your terminal.
 
@@ -88,3 +88,108 @@ Recovery does not require `--failover`, which still controls account quota exhau
 - [Tray companion testing](docs/tray-local-testing.md) — local development loops.
 
 Contributions are welcome; see [CONTRIBUTING.md](CONTRIBUTING.md). CDX Manager is [MIT licensed](LICENSE).
+
+### Run Codex on an SSH worker
+
+```sh
+cdx run work --remote agent@worker --cwd /srv/project --prompt 'Run the checks' --json
+cdx run work --remote worker --cwd /srv/project --prompt-file task.txt --detach --label checks --budget 2 --json
+cdx run work --remote Spire --remote-prefix 'wsl.exe -d Ubuntu --' --cwd /srv/project --prompt 'Run the checks' --json
+```
+
+The controller remains the authority for the profile, quota, history, costs and
+run registry. Use `run-status`, `run-report` and `run-tail` on the controller with
+the returned run ID. Permission, model, reasoning effort, prompt, timeout, budget,
+label and detached execution use the existing run options. `--cwd` names an
+absolute directory on the worker; it need not exist on the controller. Files are
+not synchronized. Use `--project` to explicitly attribute a remote checkout to a
+project. Remote runs do not retry or fail over automatically, including when
+`--failover` is supplied.
+
+Requirements: a POSIX controller with `ssh` and Codex on PATH, and a Linux worker
+with Python 3, memory-backed `/dev/shm`, and exactly the same `codex --version`.
+Configure SSH keys, host trust and aliases in advance; the connection uses batch
+mode and bounded connection/keepalive timeouts. Provider refusal, unreachable
+workers and version mismatches fail before sending credentials. Only Codex OAuth
+profiles are supported. Remote continuation is refused because the temporary home
+does not preserve conversation rollouts. Codex's native permission sandbox still
+applies to ordinary remote runs.
+
+For SSH servers landing in Windows `cmd.exe` (including Spire and Citadel), pass
+`--remote-prefix 'wsl.exe -d Ubuntu --'` to enter Linux before starting Python.
+The prefix accepts plain command words, options and paths without spaces or shell
+metacharacters; it is forwarded to detached runs. POSIX targets need no prefix.
+Only a short, double-quoted Python bootstrap is sent in the SSH command line;
+the worker and shared sandbox source are framed on stdin before the payload.
+Neither source size nor prompt size consumes cmd.exe's 8191-character command limit.
+
+`--remote` supports `--isolate-workspace` using the same local isolation mechanism:
+bubblewrap, pasta (passt), nftables, Python, the checkout/toolchain allowlist,
+and an actual namespace/network setup using a dummy credential. Missing dependencies,
+namespace denial or invalid checkouts fail before sending real credentials, without
+an unisolated fallback.
+
+For **every isolated Codex run**, local or remote, the sandbox mounts only a
+run-private, mode-600 `auth.json` on verified memory-backed storage, with an empty
+`refresh_token`. It never mounts the controller profile's full authentication file.
+The controller refreshes under the authentication lock at the thresholds below
+before preparing the snapshot, then releases the lock for the run. The temporary
+home is removed after success, failure, launch errors and stop. Non-isolated local
+runs and interactive login/refresh retain their normal profile authentication.
+
+**Accepted residual risk (operator decision, 2026-11-01):** agent tools inside the
+sandbox can read the copied access token, which can remain valid for up to ten
+days. A read-only mount prevents modification, not disclosure. The rotating refresh
+token never enters the sandbox or leaves the controller. Codex was measured to run
+with an empty refresh token on 2026-11-01.
+
+A long isolated Tower mission keeps its original credential snapshot even when
+scheduled controller refresh updates the real profile. There is no authentication
+writeback or token renewal inside the sandbox. A run that outlives its access token
+fails with a clear temporary-credential expiry error and must start a new run.
+
+The controller takes the profile's authentication lock and refreshes through its
+local [Codex app-server](https://learn.chatgpt.com/docs/app-server) when `last_refresh` is older than seven days or the access
+token has less than 24 hours left. It verifies freshness after refreshing and
+fails closed if freshness or expiration cannot be established. Only
+`access_token`, `id_token`, `account_id` and `last_refresh` travel, with an empty
+`refresh_token`. The rotating refresh token never leaves the controller. The
+credential travels exclusively over SSH stdin, and is absent from command
+arguments, environment variables, run records and output. The agent receives
+`/dev/null` as stdin, not the bootstrap script or credential transport.
+
+Each worker run gets a private mode-700 home under
+`/dev/shm/cdx-remote-<uid>/run-*`, with mode-600 `auth.json` and file-only credential storage. Only non-secret scalar
+preferences (`model`, `model_reasoning_effort`, `service_tier`, `personality`,
+`model_context_window`, `model_auto_compact_token_limit`) from the top level of
+`config.toml` are copied. Hooks, custom providers, environment settings and other
+controller-local configuration are not copied. The home is removed on completion,
+failure, cancellation and SSH stdin closure. Stop a detached run by sending
+SIGTERM to the controller PID returned at launch. Uncatchable worker termination
+can leave a home behind; the next launch sweeps homes whose worker lock is no
+longer held. No persistent worker authentication is installed or refreshed.
+
+A profile occupied by a remote run is busy for local and remote runs using the
+same controller through the run registry. The authentication lock is held only
+while refreshing and validating the credential snapshot, then released before SSH
+launch. Scheduled `cdx auth refresh` and quota probes can acquire it during a long
+mission; they no longer see a mission-long `locked`/`auth_refresh_indeterminate`
+result. A probe overlapping the short snapshot can still report contention.
+Controller renewal does not extend the worker's copied access token; quota remains
+one controller-side account snapshot, never a worker copy. Multiple independent controllers sharing copied OAuth profiles are not
+supported. Costs and budget checks use measured worker usage and observed rollout
+models, using the same exact-thread reader and 128 MiB rollout limit as local
+isolated runs. Content-free evidence travels in supervisor frames separated from
+provider stdout; raw rollouts and credentials are never returned. Live rollout
+counters can stop an over-budget run before `turn.completed`. Unknown prices and
+delayed provider reporting retain the usual budget limitations. With `--budget`,
+measured but unpriceable usage still fails with `budget_unverifiable` (exit 125),
+even after the provider finishes, preserving the documented fail-closed guarantee.
+Provider output remains in the run artifacts; without a budget, a completed run
+keeps its provider status and unknown cost. A run that outlives the access credential fails with
+`remote_credential_expired`, requiring a new controller launch.
+
+The empty-refresh-token behavior was measured with Codex CLI 0.166.0. Real-host
+acceptance should verify that the installed matching CLI does not attempt a
+refresh with the temporary home, that the controller token chain stays unchanged
+for a fresh credential, and that cleanup works after disconnect and SIGTERM.

package/VERSION

@@ -1 +1 @@
-0.25.2
+0.25.4

package/changelogs/CHANGELOGS_0_25_3.md

@@ -0,0 +1,20 @@
+# CDX Manager 0.25.3
+
+Release date: 2026-11-01
+
+This patch release adds remote Codex execution over SSH and strengthens credential isolation for local and remote sandboxes.
+
+## Features
+
+- Run Codex on a Linux SSH worker with temporary credentials while keeping profiles, quota, history, costs, and run tracking on the controller. Support detached execution and clean up the temporary worker home after the run. (#78)
+
+## Fixes and security
+
+- Keep rotating refresh tokens on the controller and out of local and remote sandboxes. Use temporary access-token snapshots on verified memory-backed storage, refresh and validate credentials before launch, and fail explicitly when a snapshot expires. (#81)
+- Upgrade cryptography and pytest to address security alerts while preserving Python 3.9 compatibility through a version-specific pytest requirement. (#80)
+- Fix remote bootstrapping through Windows SSH shells and WSL without exceeding the command-line length limit. Check isolation prerequisites before sending credentials and refuse unsafe isolation without an unisolated fallback. (#79)
+
+## Maintenance
+
+- Record tray companion checksums for v0.25.2. (`4f810c3801418bc0ad8a480333cc0bdfda39383e`)
+- Record release archive checksums for v0.25.2. (`b33eba1894d9cc68d395133848beccae38589a68`)

package/changelogs/CHANGELOGS_0_25_4.md

@@ -0,0 +1,14 @@
+# CDX Manager 0.25.4
+
+Release date: 2026-11-01
+
+This patch release measures remote Codex run costs from observed model evidence and applies budgets to those estimates.
+
+## Fixes
+
+- Attribute remote run costs to the model observed in Codex usage evidence rather than the requested model. Preserve that evidence in run tracking, stop runs when measured usage exceeds the budget, and report unverifiable budgets explicitly when reliable model evidence or pricing is unavailable. (#83)
+
+## Maintenance
+
+- Record tray companion checksums for v0.25.3. (`222d5127314bab4200bb13b1b364b3ab5253fb1b`)
+- Record release archive checksums for v0.25.3. (`9bac8bf616345e4348507038dc392a51eb0a360f`)

package/changelogs/CHANGELOGS_UNRELEASED.md

@@ -1,7 +1,42 @@
 # Unreleased
 
+## Dependency security
+
+- Require cryptography >=50.0.0 for encrypted auth bundles and remove the vulnerable 47.0.0 lock entry. Python 3.9 remains supported from 3.9.2, as cryptography 50 excludes 3.9.0 and 3.9.1.
+- Require pytest >=9.0.3 in the development extra on Python >=3.10; retain the existing 9.2.0 lock entry. pytest is not a runtime dependency. Python 3.9 still uses 8.4.2, the last compatible release, and its medium-severity alert remains unresolved pending an upstream backport or a decision to drop Python 3.9 development support.
+- Check lock consistency and run the CI test matrix against locked development dependencies without upgrading unrelated packages.
+
 ## Run costs and budgets
 
 - Add nullable `cost_usd` to JSON history entries using the same observed usage, model pricing, and price overrides as stats.
 - Add `cdx run --budget USD` to stop on measured estimated spend with exit code 125, `budget_exceeded`, and `error.spent_usd` in the final JSON. Share the budget across recovery attempts and failover; support detached runs and fresh budgets for continued runs without charging earlier turns.
 - Document unknown usage/prices and delayed provider reporting, including possible budget overshoot. Cover pricing, unknown costs, validation, live process termination, Claude transcript accumulation, continuation boundaries, and detached supervision in tests.
+
+## Remote run cost observations
+
+- Use the local exact-thread observation reader, usage normalization and 128 MiB rollout limit on workers; return content-free model/counter evidence before volatile storage cleanup and bind it to controller run/attempt artifacts.
+- Separate supervisor evidence frames from provider stdout, and check live rollout measurements before `turn.completed` so measured remote budget excess still stops the provider. Never price from requested models or controller-side rollouts.
+- Preserve the documented fail-closed budget guarantee: measured but unpriceable remote usage exits 125 with `budget_unverifiable`, including after provider completion. Keep provider output in artifacts; completed runs without a budget retain their actual status and unknown cost. Credential snapshots, redaction, cleanup and workspace isolation remain unchanged.
+- Cover priced remote/isolated runs, Windows transport, rollouts over 16 MiB, live budget stops, missing/inconsistent/mixed model evidence, unknown prices and completion without pricing using the fake transport.
+
+## Remote Codex runs
+
+- Add `cdx run --remote SSH_TARGET` with controller-owned supervision, registry, history, output, costs, budgets and detached execution.
+- Refresh under the controller authentication lock at the seven-day/24-hour thresholds, then send a temporary OAuth credential over SSH stdin with an empty refresh token. Refuse rotating credentials, other providers and CLI version mismatches.
+- Keep worker credentials in a private memory-backed home, clean up after success, failure and stop, and sweep dead homes on the next launch. Keep agent stdin separate from bootstrap input and report credential expiration without retrying.
+- Cover transport and credential isolation, thresholds, cleanup, sweep, unreachable workers, provider refusal and registry reports with a fake SSH worker.
+
+## Remote worker transport and isolation preflight
+
+- Add `--remote-prefix` for Windows SSH landings such as `wsl.exe -d Ubuntu --`, including detached runs. Keep the POSIX default and use a short double-quoted bootstrap on argv; send framed worker/sandbox source and then credentials/payload over stdin.
+- Probe the local bubblewrap/pasta/nftables isolation mechanism on the worker with dummy credentials, including namespace/network startup. Refuse missing dependencies or setup failures without falling back to an unisolated run.
+- Enable remote isolated Codex execution after a successful isolation probe, following the operator decision of 2026-11-01. Agent tools can read the copied access token (up to ten days of validity); missing dependencies still refuse without fallback.
+- Release the controller authentication lock after preparing the credential snapshot so scheduled authentication refresh can run during long remote missions. Keep profile mission exclusion in the run registry and the worker credential's original expiry.
+- Test cmd.exe length/quoting enforcement, Windows prefixes, large stdin payloads, POSIX transport, sanitized credential snapshots, sandbox construction/dependencies and concurrent scheduled refresh with synthetic credentials.
+
+## Isolated Codex credentials
+
+- Replace the local full-profile auth mount with a mode-600 run-private snapshot on verified memory-backed storage, containing an empty refresh token. Refresh on the controller under the auth lock at the existing seven-day/24-hour thresholds before launch.
+- Keep snapshots independent of controller refresh during long missions, prohibit authentication writeback, enforce their original expiry, and remove temporary storage after success, failure and stop. Non-isolated local authentication remains unchanged.
+- Reject rotating credentials at the sandbox boundary. Test snapshot contents, permissions, cleanup, refresh lock ownership, concurrent controller refresh, remote isolation through fake SSH and missing dependency refusal.
+- The separately maintained remote-ops `scripts/workspace_sandbox.py` requires the same credential guard and run-private lifecycle integration; it is not shipped by this repository.

package/checksums/release-archives.json

@@ -705,6 +705,28 @@
         "x86_64-pc-windows-msvc": "183ef3664489888a356ad9a6096b68c913d433c513d6d10e9890e65667e1e417",
         "x86_64-unknown-linux-musl": "5660cfd08c0580db43dee5664fe95fc4cd528a6495c40167e34c122d3c4fcd4a"
       }
+    },
+    "v0.25.2": {
+      "github_tarball_sha256": "8ed02d4ce31ca6aa9669396675ee1d5b6669e80f33418c00c2a210306ea19e0d",
+      "github_zip_sha256": "0e8770a0e18c6c99bd5d845c8c54d90eb33fbf71720be89d87396eaee1632cfe",
+      "tray_assets": {
+        "aarch64-apple-darwin": "3c0cc113b70670e0f1bd9eff4589cddd8a5e3f76d7b53d7a1d66a2529f5e6e84",
+        "aarch64-unknown-linux-musl": "ff978672276fdfa8f22af8d3dadb63e0d26fa40c6f10d48f834073b08c620efd",
+        "x86_64-apple-darwin": "42323cbce340238ada8d986192d198a82e37d5958fb25b56cbaf8dba347da05c",
+        "x86_64-pc-windows-msvc": "ffcd4deb8b3843771659d2c6b92b003822b288329ddcf3b9a13f0f9820fa3df1",
+        "x86_64-unknown-linux-musl": "2cefeea0aad5bb0acfcbc99146ffacc5c5da9386f8338e4a01541e97ccf71e07"
+      }
+    },
+    "v0.25.3": {
+      "github_tarball_sha256": "34114563f0edf4d8e13d9410d6a37e4c315571a8410de607b8bc541032561a21",
+      "github_zip_sha256": "58cabe7d9d48c314e568c317893b460ca8c8853434a68ee3397e23999f823ddf",
+      "tray_assets": {
+        "aarch64-apple-darwin": "14da8f42193e78c1bfca5ac046a6a23c7ab52b4155ac9eb213bd9af09cd5e6cc",
+        "aarch64-unknown-linux-musl": "47b6988f5f361403e2c8e0c2ff5951cb758af3cde3fc6bb02f7eb718042c3027",
+        "x86_64-apple-darwin": "aac73ff0abdb76a957ebd7c143ff3ace46d6df7b08f13db922b28e76dcbc82d0",
+        "x86_64-pc-windows-msvc": "c81623ffb94b11ae7b7b28be123b49393240bb0edd31c69969ca07d7eab8dff7",
+        "x86_64-unknown-linux-musl": "6892112587c1d8c6e805a90148ec8deb6c04b5927c7ed0c18f19479d521c0ced"
+      }
     }
   }
 }

package/docs/cli.md

@@ -254,7 +254,7 @@ At or above the limit, cdx terminates the provider process tree, skips further r
 
 With `--detach`, the launch still exits successfully and includes `budget_usd`; cdx always uses its own child supervisor for budgeted runs, including when native background delegation is enabled. Inspect `cdx run-status RUN_ID --json` or `cdx run-report RUN_ID --json` for the child's final payload and budget error. `--continue-run` starts a fresh budget for the new run, including in detached mode.
 
-Conservative assumption: this is a stop based on observed, priceable usage, not a prepaid billing limit. Provider updates can arrive late or only at completion, and termination can take time, so actual spend can exceed the amount. Before the first measurement, `budget_status` is `usage_not_measured`; absence is not zero spend. Once usage is measured, missing continuation boundaries, mixed/unobserved models, and unknown prices stop further execution/recovery with exit **125**, `error.source: "cdx"`, `error.code: "budget_unverifiable"`, `budget_status: "unverifiable"`, and `cost_usd: null`. This is a measurement failure, not a claim that the dollar limit was exceeded or enforced. Supply a `CDX_TOKEN_PRICES` override for a model missing from the table. Existing profile-level `cdx set --budget` settings remain the provider-native Claude cap and apply independently.
+Conservative assumption: this is a stop based on observed, priceable usage, not a prepaid billing limit. Provider updates can arrive late or only at completion, and termination can take time, so actual spend can exceed the amount. Before the first measurement, `budget_status` is `usage_not_measured`; absence is not zero spend. Once usage is measured, missing continuation boundaries, mixed/unobserved models, and unknown prices stop further execution/recovery with exit **125**, `error.source: "cdx"`, `error.code: "budget_unverifiable"`, `budget_status: "unverifiable"`, and `cost_usd: null`. This is a measurement failure, not a claim that the dollar limit was exceeded or enforced. This fail-closed guarantee also applies to a remote agent that has already completed: the supervisor reports `failed`/125 while retaining its final output and artifacts. Without `--budget`, unknown pricing does not replace the provider's completion status. Supply a `CDX_TOKEN_PRICES` override for a model missing from the table. Existing profile-level `cdx set --budget` settings remain the provider-native Claude cap and apply independently.
 
 Every usage record, whichever path produced it, uses one set of definitions:
 
@@ -292,7 +292,7 @@ The figure uses standard public list prices, excluding Fast/priority, Batch/Flex
 
 Completed results, registry records, and history JSON include `cost_status` (`estimated` or `unknown`) and nullable `cost_reason`: `usage_not_measured`, `model_not_observed`, `model_price_unknown`, `attempt_usage_unavailable`, or `mixed_models_without_counters`. Stats include `unpriced_reasons` counts alongside `priced_runs` and `unpriced_models`. These are list-price estimates, not billing receipts. `budget_status` is `not_requested`, `usage_not_measured`, `estimated`, `exceeded`, or `unverifiable`.
 
-New headless attempts have distinct `attempt_id` values and artifact paths, including retries on the same profile. Final payloads and registry reports expose `attempts` with their individual model evidence, counters, and estimates; top-level usage describes the final attempt, while a budget-exceeded spend includes all attempts. New isolated Codex attempts mount only a fresh sessions directory from volatile `/dev/shm` storage. The supervisor reads the exact `thread.started` rollout while it exists and at closeout, requiring matching session identity, turn timestamps within this attempt, and token counters consistent with stdout. It exports `usage_evidence` containing only the stdout attempt identity, thread id, timestamp bounds, turn ids, observed models, and normalized counters; live rollout counters can establish the first measurement before `turn.completed`. No requested model, agent-authored cost report, current profile setting, prompt, message, or tool output is used as model evidence. The raw temporary sessions are removed at closeout, including cancellation and startup failure; they are never copied into persistent run artifacts. Host profiles are not mounted. Mixed serving models stay unpriced without attributable counters. Isolated continuation without comparable launch boundaries stays unmeasured rather than charging an older conversation to the new attempt.
+New headless attempts have distinct `attempt_id` values and artifact paths, including retries on the same profile. Final payloads and registry reports expose `attempts` with their individual model evidence, counters, and estimates; top-level usage describes the final attempt, while a budget-exceeded spend includes all attempts. New isolated Codex attempts mount only a fresh sessions directory from volatile `/dev/shm` storage. The supervisor reads the exact `thread.started` rollout while it exists and at closeout, requiring matching session identity, turn timestamps within this attempt, and token counters consistent with stdout. It exports `usage_evidence` containing only the stdout attempt identity, thread id, timestamp bounds, turn ids, observed models, and normalized counters; live rollout counters can establish the first measurement before `turn.completed`. No requested model, agent-authored cost report, current profile setting, prompt, message, or tool output is used as model evidence. The raw temporary sessions are removed at closeout, including cancellation and startup failure; they are never copied into persistent run artifacts. Host profiles are not mounted. Remote Codex runs ship the same observation reader and normalization code over the existing stdin bootstrap. The worker validates the exact fresh-home thread, timestamp bounds and matching counters with the local 128 MiB limit, and returns only normalized evidence through supervisor frames, separate from provider stdout. The controller attaches its own stdout path, run id and attempt id before live budget checks and final persistence; it never consults a controller profile rollout for remote usage. Worker stdout capture stays in the existing volatile private home and is deleted with it. Credentials, refresh handling and workspace mounts are unchanged. Live rollout counters can establish remote usage before `turn.completed`. Mixed serving models stay unpriced without attributable counters. Isolated continuation without comparable launch boundaries stays unmeasured rather than charging an older conversation to the new attempt.
 
 Completed run JSON and registry reports expose `usage_model` separately from the requested `model`. Pricing uses the observed model from provider output; Codex can also supply it through the exact rollout identified by `thread.started.thread_id`. Configured defaults alone never establish serving-model identity, and missing or mixed model evidence stays unpriced. `--by model` groups Claude's `[1m]` context variant with its base model while preserving the original model string and per-run pricing. New runs record this evidence automatically. To recover missing models in existing history, run:
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "cdx-manager",
-  "version": "0.25.2",
+  "version": "0.25.4",
   "description": "Terminal session manager for Codex and Claude accounts.",
   "license": "MIT",
   "author": "Alexandre Agostini",

package/pyproject.toml

@@ -4,10 +4,10 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "cdx-manager"
-version = "0.25.2"
+version = "0.25.4"
 description = "Terminal session manager for Codex and Claude accounts."
 readme = "README.md"
-requires-python = ">=3.9"
+requires-python = ">=3.9.2"
 license = "MIT"
 authors = [
   { name = "Alexandre Agostini" }
@@ -26,11 +26,13 @@ classifiers = [
   "Topic :: Software Development",
   "Topic :: Terminals",
 ]
-dependencies = ["cryptography>=42"]
+dependencies = ["cryptography>=50.0.0"]
 
 [project.optional-dependencies]
 dev = [
-  "pytest>=8",
+  # pytest 9 requires Python 3.10; 8.4.2 is the last Python 3.9 release.
+  "pytest>=8.4.2; python_version < '3.10'",
+  "pytest>=9.0.3; python_version >= '3.10'",
   "pytest-cov>=5",
   "ruff>=0.6",
 ]
@@ -70,6 +72,7 @@ show_missing = true
 skip_empty = true
 
 [tool.ruff]
+extend-exclude = [".jarvis-input"]
 target-version = "py39"
 line-length = 140
 

package/src/cli.py

@@ -245,7 +245,7 @@ def _print_help(use_color=False):
         f"  {_style('cdx reset <name> [--yes] [--json]', '36', use_color)}",
         f"  {_style('cdx auth refresh <name|all> [--json]', '36', use_color)}",
         f"  {_style('cdx select --provider PROVIDER [--min-reasoning-effort minimal|low|medium|high|xhigh] [--min-power minimal|low|medium|high|xhigh] [--require-ready] [--refresh] --json', '36', use_color)}",
-        f"  {_style('cdx run [session] --cwd PATH [--project PATH] (--prompt-file PATH|--prompt TEXT|--prompt-file -) [--provider PROVIDER] [--model MODEL] [--kind assistant|code-review|retro] [--reasoning-effort minimal|low|medium|high|xhigh] [--power minimal|low|medium|high|xhigh] [--permission review|default|auto|full|workspace-write|read-only|danger-full-access] [--timeout-seconds N] [--label TEXT] [--max-parallel N] [--continue-run RUN_ID] [--detach] [--failover] [--refresh] --json', '36', use_color)}",
+        f"  {_style('cdx run [session] [--remote SSH_TARGET] --cwd PATH [--project PATH] (--prompt-file PATH|--prompt TEXT|--prompt-file -) [--provider PROVIDER] [--model MODEL] [--kind assistant|code-review|retro] [--reasoning-effort minimal|low|medium|high|xhigh] [--power minimal|low|medium|high|xhigh] [--permission review|default|auto|full|workspace-write|read-only|danger-full-access] [--timeout-seconds N] [--label TEXT] [--max-parallel N] [--continue-run RUN_ID] [--detach] [--failover] [--refresh] --json', '36', use_color)}",
         f"  {_style('cdx runs [--limit N] [--since 7d|today|DATE] [--running] --json', '36', use_color)}",
         f"  {_style('cdx run-status <run_id> --json', '36', use_color)}",
         f"  {_style('cdx run-report <run_id> --json', '36', use_color)}",

package/src/cli_args.py

@@ -48,7 +48,7 @@ CAN_RESUME_USAGE = "Usage: cdx can-resume <name> [--conversation <id>] [--json]"
 SESSIONS_USAGE = "Usage: cdx sessions [--all] --json"
 SELECT_USAGE = "Usage: cdx select --provider PROVIDER [--min-reasoning-effort minimal|low|medium|high|xhigh] [--min-power minimal|low|medium|high|xhigh] [--require-ready] [--refresh] --json"
 NEXT_USAGE = "Usage: cdx next [--json] [--refresh]"
-RUN_USAGE = "Usage: cdx run [session] (--cwd|--dir) PATH [--project PATH] (--prompt-file PATH|--prompt TEXT|--prompt-file -) [--provider PROVIDER] [--model MODEL] [--kind assistant|code-review|retro] [--reasoning-effort minimal|low|medium|high|xhigh] [--power minimal|low|medium|high|xhigh] [--permission review|default|auto|full|workspace-write|read-only|danger-full-access] [--network-access] [--isolate-workspace] [--timeout-seconds N] [--budget USD] [--label TEXT] [--max-parallel N] [--continue-run RUN_ID] [--detach] [--failover] [--refresh] --json"
+RUN_USAGE = "Usage: cdx run [session] [--remote SSH_TARGET] [--remote-prefix COMMAND] (--cwd|--dir) PATH [--project PATH] (--prompt-file PATH|--prompt TEXT|--prompt-file -) [--provider PROVIDER] [--model MODEL] [--kind assistant|code-review|retro] [--reasoning-effort minimal|low|medium|high|xhigh] [--power minimal|low|medium|high|xhigh] [--permission review|default|auto|full|workspace-write|read-only|danger-full-access] [--network-access] [--isolate-workspace] [--timeout-seconds N] [--budget USD] [--label TEXT] [--max-parallel N] [--continue-run RUN_ID] [--detach] [--failover] [--refresh] --json"
 RUN_JSON_REQUIRED = "cdx run: --json is required."
 RUN_TARGET_REQUIRED = "cdx run: specify a session name or --provider PROVIDER."
 RUN_SESSION_PROVIDER_CONFLICT = "cdx run: cannot specify both a session name and --provider."
@@ -546,6 +546,13 @@ def cdx_schema():
             "run": [
                 "invalid_cwd",
                 "session_disabled",
+                "session_busy",
+                "auth_lock_failed",
+                "remote_unavailable",
+                "remote_version_mismatch",
+                "remote_provider_unsupported",
+                "remote_credential_expired",
+                "remote_isolation_unavailable",
                 "no_suitable_session",
                 "provider_cli_not_found",
                 "provider_start_failed",
@@ -673,6 +680,8 @@ def _parse_max_parallel(value):
 
 def _parse_run_args(args):
     parsed = _parse_run_flag_args(args, {
+        "--remote": {"key": "remote", "type": "str", "default": None},
+        "--remote-prefix": {"key": "remote_prefix", "type": "str", "default": None},
         "--project": {"key": "project", "type": "str", "default": None},
         "--cwd": {"key": "cwd", "type": "str", "default": None},
         "--dir": {"key": "cwd", "type": "str", "default": None},
@@ -732,9 +741,17 @@ def _parse_run_args(args):
     if parsed["continue_run"] is not None and not parsed["continue_run"].strip():
         raise CdxArgumentError("cdx run: --continue-run requires a nonempty run id.",
                                code=ARG_CODE_INVALID_VALUE, arguments=["--continue-run"])
+    if parsed["remote"] is not None and not parsed["remote"].strip():
+        raise CdxArgumentError("cdx run: --remote requires a nonempty SSH target.",
+                               code=ARG_CODE_INVALID_VALUE, arguments=["--remote"])
+    if parsed["remote_prefix"] is not None and (not parsed["remote"] or not parsed["remote_prefix"].strip()):
+        raise CdxArgumentError("cdx run: --remote-prefix requires --remote and a nonempty command.",
+                               code=ARG_CODE_INVALID_VALUE, arguments=["--remote-prefix"])
     effort = _parse_run_effort(parsed["reasoning_effort"], parsed["power"])
     return {
         "name": name,
+        "remote": parsed["remote"],
+        "remote_prefix": parsed["remote_prefix"],
         "provider": parsed["provider"],
         "cwd": parsed["cwd"],
         "project": parsed["project"],

package/src/codex_usage.py

@@ -18,7 +18,7 @@ CODEX_AUTH_LOCK_NAME = ".cdx-auth.lock"
 
 
 @contextlib.contextmanager
-def codex_auth_lock(auth_home, blocking=False, timeout_seconds=None, retry_interval_seconds=0.05):
+def codex_auth_lock(auth_home, blocking=False, timeout_seconds=None, retry_interval_seconds=0.05, strict=False):
     """Serialize codex token refreshes per CODEX_HOME.
 
     Codex rotates its OAuth refresh_token on refresh and invalidates the old
@@ -28,6 +28,8 @@ def codex_auth_lock(auth_home, blocking=False, timeout_seconds=None, retry_inter
     the lock is held, False when a non-blocking acquire found it already taken.
     ponytail: POSIX flock only; on Windows (no fcntl) this is a no-op.
     """
+    if strict and (not auth_home or fcntl is None):
+        raise OSError("A POSIX authentication lock is required")
     if not auth_home or fcntl is None:
         yield True
         return
@@ -35,6 +37,8 @@ def codex_auth_lock(auth_home, blocking=False, timeout_seconds=None, retry_inter
         os.makedirs(auth_home, exist_ok=True)
         handle = open(os.path.join(auth_home, CODEX_AUTH_LOCK_NAME), "w")
     except OSError:
+        if strict:
+            raise
         yield True
         return
     deadline = None if timeout_seconds is None else time.monotonic() + timeout_seconds

package/src/commands/runs.py

@@ -48,6 +48,7 @@ from ..provider_runtime import (
     _run_headless_provider_command,
     with_requested_conversation,
 )
+from ..remote_run import preflight, run_remote
 from ..run_command import read_run_prompt, run_cdx_error_code, run_launch_payload, run_result_payload
 from ..run_failover import (
     CAPACITY_RETRY_DELAYS,
@@ -203,6 +204,8 @@ def _detached_child_argv(parsed, prompt_path, session_name, cwd):
     # from the package root, so a relative "--cwd ." would resolve there.
     argv = ["run", session_name, "--cwd", cwd, "--prompt-file", prompt_path]
     for flag, key in (
+        ("--remote", "remote"),
+        ("--remote-prefix", "remote_prefix"),
         ("--project", "project"),
         ("--budget", "budget"),
         ("--continue-run", "continue_run"),
@@ -546,7 +549,7 @@ def _env_max_runs(ctx):
 
 
 def _observe_headless_usage(session, run_info, baseline=None):
-    isolated = (session.get("launch") or {}).get("isolate_workspace")
+    isolated = bool(run_info.get("remote")) or (session.get("launch") or {}).get("isolate_workspace")
     evidence = run_info.get("usage_evidence")
     observed_models = []
     usage, model = extract_run_observation(session.get("provider"), run_info.get("stdout_path"),
@@ -581,7 +584,7 @@ def _observe_headless_usage(session, run_info, baseline=None):
 
 def _budget_observation(session, paths, baseline):
     observation = _observe_headless_usage(session, paths, baseline if session["provider"] == "codex" else None)
-    if (session.get("launch") or {}).get("isolate_workspace"):
+    if paths.get("remote") or (session.get("launch") or {}).get("isolate_workspace"):
         return observation
     identifier = (extract_run_conversation_id(session["provider"], paths.get("stdout_path"))
                   or ((session.get("conversation") or {}).get("id")
@@ -710,8 +713,18 @@ def handle_run(rest, ctx):
             session = selected["session"]
             selection_row = selected.get("row") or {}
 
+        remote_version = None
+        if parsed.get("remote"):
+            if session["provider"] != "codex":
+                raise CdxError("--remote is only available for Codex profiles.", code="remote_provider_unsupported")
+            if parsed.get("continue_run"):
+                raise CdxError("--remote requires a fresh run without --continue-run.")
+            remote_version = preflight(
+                parsed["remote"], {**os.environ, **(ctx.get("env") or {})},
+                remote_prefix=parsed.get("remote_prefix"), isolate_workspace=parsed.get("isolate_workspace", False),
+                cwd=parsed["cwd"], network=parsed.get("network_access", False), writable=parsed.get("permission") != "review")
         cwd = os.path.abspath(parsed["cwd"])
-        if not os.path.isdir(cwd):
+        if not parsed.get("remote") and not os.path.isdir(cwd):
             raise CdxError(f"Invalid cwd: {parsed['cwd']}", code="invalid_cwd")
         from ..project_identity import resolve_project
 
@@ -753,17 +766,18 @@ def handle_run(rest, ctx):
                 **launch_updates,
             },
         }
-        _ensure_session_authentication(
-            run_session,
-            ctx["service"],
-            spawn=ctx.get("spawn"),
-            spawn_sync=ctx.get("spawn_sync"),
-            env_override=ctx.get("env"),
-            stdin_is_tty=False,
-            behavior="launch",
-            signal_emitter=ctx.get("signal_emitter"),
-            trust_local_credentials=False,
-        )
+        if not parsed.get("remote"):
+            _ensure_session_authentication(
+                run_session,
+                ctx["service"],
+                spawn=ctx.get("spawn"),
+                spawn_sync=ctx.get("spawn_sync"),
+                env_override=ctx.get("env"),
+                stdin_is_tty=False,
+                behavior="launch",
+                signal_emitter=ctx.get("signal_emitter"),
+                trust_local_credentials=False,
+            )
         registry = _run_registry(ctx)
         # A detached child is handed its parent's run_id so both agree on the
         # identity the launch payload already reported to the caller.
@@ -788,6 +802,7 @@ def handle_run(rest, ctx):
             },
             label=parsed.get("label"),
             max_parallel=max_parallel,
+            remote=parsed.get("remote"),
             parent_run_id=parsed.get("continue_run"),
             conversation_id=(run_session.get("conversation") or {}).get("id") if run_session["provider"] == "claude" or parsed.get("continue_run") else None,
             launch=run_session.get("launch"),
@@ -870,7 +885,11 @@ def handle_run(rest, ctx):
                         budget_observation = observation
                 return attempt_cost is not None and already_spent + attempt_cost >= parsed["budget"]
 
-            run_info = _run_headless_provider_command(
+            runner = _run_headless_provider_command
+            if parsed.get("remote"):
+                def runner(*args, **kwargs):
+                    return run_remote(*args, target=parsed["remote"], version=remote_version, remote_prefix=parsed.get("remote_prefix"), **kwargs)
+            run_info = runner(
                 run_session,
                 cwd=cwd,
                 env_override=ctx.get("env"),
@@ -918,7 +937,7 @@ def handle_run(rest, ctx):
                 **run_info,
                 **project,
             })
-            if run_info.get("budget_exceeded") or run_info.get("budget_unverifiable") or parsed.get("continue_run"):
+            if parsed.get("remote") or run_info.get("budget_exceeded") or run_info.get("budget_unverifiable") or parsed.get("continue_run"):
                 break
             if looks_model_at_capacity(run_session.get("provider"), run_info):
                 event = {"reason": "model_at_capacity", "attempt": attempt,

package/src/provider_runtime.py

@@ -1208,16 +1208,26 @@ def _read_permission_denials(stdout_path):
 
 def _run_headless_provider_command(session, cwd=None, env_override=None, initial_prompt=None,
                                    timeout_seconds=None, spawn=None, run_id=None, budget_check=None, attempt=1):
-    import tempfile
+    from .remote_run import remote_auth_lock, temporary_auth
+    from .workspace_sandbox import private_auth_home
 
     isolated_codex = session.get("provider") == PROVIDER_CODEX and (session.get("launch") or {}).get("isolate_workspace")
     if isolated_codex:
-        # Volatile storage only: raw rollouts must not survive closeout, including
-        # cancellation/start failures. Only this attempt's sessions are mounted.
-        with tempfile.TemporaryDirectory(prefix="cdx-observation-", dir="/dev/shm" if sys.platform.startswith("linux") else None) as directory:
-            os.mkdir(os.path.join(directory, "sessions"), mode=0o700)
-            return _run_headless_provider_attempt(session, cwd, env_override, initial_prompt,
-                                                  timeout_seconds, spawn, run_id, budget_check, directory, attempt)
+        handlers = _route_termination_to_interrupt()
+        try:
+            with remote_auth_lock(_get_auth_home(session)):
+                auth = temporary_auth(_get_auth_home(session))
+            with private_auth_home(auth) as directory:
+                return _run_headless_provider_attempt(session, cwd, env_override, initial_prompt,
+                                                      timeout_seconds, spawn, run_id, budget_check, directory, attempt)
+        except KeyboardInterrupt as cause:
+            error = CdxError('Isolated Codex run was cancelled.', 130)
+            error.cancelled = True
+            raise error from cause
+        except (OSError, RuntimeError) as error:
+            raise CdxError(str(error)) from error
+        finally:
+            _restore_signal_handlers(handlers)
     return _run_headless_provider_attempt(session, cwd, env_override, initial_prompt,
                                           timeout_seconds, spawn, run_id, budget_check, attempt=attempt)
 
@@ -1266,7 +1276,7 @@ def _run_headless_provider_attempt(session, cwd=None, env_override=None, initial
                 [spec["command"], *spec["args"]], Path(cwd),
                 network=launch.get("network_access", False),
                 writable=launch.get("permission") != "review",
-                source=spec["options"]["env"], auth_home=_get_auth_home(session),
+                source=spec["options"]["env"], auth_home=observation_home or _get_auth_home(session),
                 provider=session.get("provider"),
                 **({"observation_home": os.path.join(observation_home, "sessions")} if observation_home else {}))
         except (OSError, RuntimeError) as error:
@@ -1291,6 +1301,12 @@ def _run_headless_provider_attempt(session, cwd=None, env_override=None, initial
     timed_out = False
     cancelled = False
     budget_exceeded = False
+    credential_expired = False
+    expires_at = None
+    if observation_home:
+        from .remote_run import token_expiry
+        with open(os.path.join(observation_home, 'auth.json')) as handle:
+            expires_at = token_expiry(json.load(handle)['tokens']['access_token'])
     with open(paths["stdout_path"], "w", encoding="utf-8", errors="replace") as stdout_file, \
             open(paths["stderr_path"], "w", encoding="utf-8", errors="replace") as stderr_file:
         options = {k: v for k, v in spec.get("options", {}).items() if k not in ("stdio", "stdout", "stderr")}
@@ -1319,11 +1335,15 @@ def _run_headless_provider_attempt(session, cwd=None, env_override=None, initial
         # this supervisor and keep its run "running" forever.
         previous_handlers = _route_termination_to_interrupt()
         try:
-            if budget_check is not None:
+            if budget_check is not None or expires_at is not None:
                 deadline = time.monotonic() + timeout_seconds if timeout_seconds is not None else None
                 while True:
                     snapshot()
-                    if budget_check(paths):
+                    if expires_at is not None and time.time() >= expires_at:
+                        credential_expired = True
+                        _terminate_child_tree(child)
+                        break
+                    if budget_check and budget_check(paths):
                         budget_exceeded = True
                         _terminate_child_tree(child)
                         break
@@ -1333,7 +1353,7 @@ def _run_headless_provider_attempt(session, cwd=None, env_override=None, initial
                     try:
                         child.wait(timeout=min(0.25, remaining) if remaining is not None else 0.25)
                         snapshot()
-                        budget_exceeded = budget_check(paths)
+                        budget_exceeded = bool(budget_check and budget_check(paths))
                         break
                     except subprocess.TimeoutExpired:
                         continue
@@ -1357,6 +1377,11 @@ def _run_headless_provider_attempt(session, cwd=None, env_override=None, initial
             snapshot()
             _restore_signal_handlers(previous_handlers)
 
+    if credential_expired or (expires_at is not None and time.time() >= expires_at):
+        _combine_headless_transcript(paths)
+        error = CdxError('Isolated temporary credential expired; start a new run from the controller.', code='remote_credential_expired')
+        error.run_info = _headless_run_info(paths, spec, start_time, child.returncode)
+        raise error
     if cancelled:
         _combine_headless_transcript(paths)
         returncode = getattr(child, "returncode", None)

package/src/remote_run.py

@@ -0,0 +1,272 @@
+"""Controller-side temporary Codex credentials and SSH transport."""
+
+import base64
+import json
+import math
+import os
+import pathlib
+import re
+import subprocess
+import threading
+import time
+from contextlib import contextmanager
+from datetime import datetime, timezone
+
+from .codex_usage import _request_codex_app_server, codex_auth_lock
+from .errors import CdxError
+
+
+def token_expiry(token):
+    try:
+        part = token.split('.')[1]
+        expiry = float(json.loads(base64.urlsafe_b64decode(part + '=' * (-len(part) % 4)))['exp'])
+        if not math.isfinite(expiry) or expiry <= 0:
+            raise ValueError('Invalid expiration')
+        return expiry
+    except (ValueError, KeyError, IndexError, TypeError) as error:
+        raise CdxError('Cannot determine remote access token expiration; log in on the controller.') from error
+
+
+def needs_refresh(auth, now):
+    try:
+        refreshed = datetime.fromisoformat(auth['last_refresh'].replace('Z', '+00:00')).timestamp()
+    except (ValueError, KeyError, TypeError):
+        return True
+    return now - refreshed > 7 * 86400 or token_expiry(auth['tokens']['access_token']) - now < 86400
+
+
+def refresh_auth(home):
+    result = _request_codex_app_server(home, 'account/read', {'refreshToken': True}, 'refresh_failed', timeout=30)
+    if not result.get('ok'):
+        raise CdxError('Controller Codex credential refresh failed; log in on the controller.')
+
+
+def temporary_auth(home, now=None):
+    now = time.time() if now is None else now
+    try:
+        auth = json.loads((pathlib.Path(home) / 'auth.json').read_text())
+        if needs_refresh(auth, now):
+            refresh_auth(home)
+            auth = json.loads((pathlib.Path(home) / 'auth.json').read_text())
+            if needs_refresh(auth, now):
+                raise CdxError('Controller refresh did not produce a fresh remote credential.')
+        tokens = {key: auth['tokens'][key] for key in ('access_token', 'id_token', 'account_id')}
+        tokens['refresh_token'] = ''
+        result = {'tokens': tokens, 'last_refresh': auth['last_refresh']}
+        rotating = auth['tokens'].get('refresh_token')
+        if result['tokens']['refresh_token'] != '' or (rotating and rotating in json.dumps(result)):
+            raise CdxError('Refusing to send a refresh token to a worker.')
+        return result
+    except (OSError, ValueError, KeyError, TypeError) as error:
+        raise CdxError('Cannot read controller Codex OAuth credentials; log in on the controller.') from error
+
+
+def nonsecret_config(home):
+    # Copy only known scalar preferences. Provider URLs, hooks, env and arbitrary
+    # config entries can contain secrets or refer to controller-only programs.
+    path = pathlib.Path(home) / 'config.toml'
+    if not path.exists():
+        return ''
+    allowed = {'model', 'model_reasoning_effort', 'service_tier', 'personality',
+               'model_context_window', 'model_auto_compact_token_limit'}
+    lines = []
+    for line in path.read_text().splitlines():
+        if line.lstrip().startswith('['):
+            break
+        key, sep, value = line.partition('=')
+        if sep and key.strip() in allowed and re.fullmatch(r'\s*("[a-zA-Z0-9_. /-]+"|[0-9]+)\s*', value):
+            lines.append(line)
+    return '\n'.join(lines) + '\n'
+
+
+def worker_input():
+    # One JSON line frames the source, leaving the next line for the payload.
+    directory = pathlib.Path(__file__).parent
+    source = 'SANDBOX_SOURCE = ' + repr((directory / 'workspace_sandbox.py').read_text()) + '\n'
+    from .interactive_usage import MAX_TRANSCRIPT_BYTES, MAX_TRANSCRIPT_SCAN_SECONDS
+    modules = {name: (directory / (name + '.py')).read_text()
+               for name in ('errors', 'run_usage', 'run_observation')}
+    source += 'OBSERVATION_MODULES = ' + repr(modules) + '\n'
+    source += f'OBSERVATION_LIMITS = {MAX_TRANSCRIPT_BYTES, MAX_TRANSCRIPT_SCAN_SECONDS!r}\n'
+    source += (directory / 'remote_worker.py').read_text()
+    return json.dumps(source).encode() + b'\n'
+
+
+def ssh_command(target, mode, remote_prefix=None):
+    if not re.fullmatch(r'[a-zA-Z0-9_@.:[\]-]+', target) or target.startswith('-'):
+        raise CdxError('Invalid --remote SSH target; use a host alias or user@host.')
+    prefix = (remote_prefix or '').strip()
+    if prefix and not re.fullmatch(r'[a-zA-Z0-9_./:=+ -]+', prefix):
+        raise CdxError('Invalid --remote-prefix; use a plain command prefix such as wsl.exe -d Ubuntu --.')
+    if mode not in ('probe', 'run'):
+        raise CdxError('Invalid remote worker mode.')
+    # Double quotes work in both POSIX sh and cmd.exe. The bootstrap contains
+    # no embedded double quotes, shell expansions, or worker source in argv.
+    command = (prefix + ' ' if prefix else '') + 'python3 -u -c "import sys,json;exec(json.loads(sys.stdin.buffer.readline()))" ' + mode
+    if len(command) > 8000:
+        raise CdxError('Remote command prefix exceeds the Windows command-line limit.')
+    return ['ssh', '-T', '-o', 'BatchMode=yes', '-o', 'ConnectTimeout=5', '-o', 'ConnectionAttempts=1',
+            '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=1', target, command]
+
+
+def preflight(target, env=None, *, remote_prefix=None, isolate_workspace=False, cwd=None, network=False, writable=True):
+    payload = {'isolate_workspace': isolate_workspace, 'cwd': cwd, 'network': network, 'writable': writable}
+    try:
+        local = subprocess.check_output(['codex', '--version'], env=env, stderr=subprocess.DEVNULL, timeout=10).decode().strip()
+        result = subprocess.run(ssh_command(target, 'probe', remote_prefix), env=env,
+                                input=worker_input() + json.dumps(payload).encode() + b'\n',
+                                capture_output=True, timeout=30)
+    except (OSError, subprocess.TimeoutExpired) as error:
+        raise CdxError('Remote worker unreachable or Codex CLI unavailable.', code='remote_unavailable') from error
+    if result.returncode:
+        message = ('Remote workspace isolation unavailable; requires bubblewrap, pasta, nftables, working namespaces and a valid checkout.'
+                   if isolate_workspace else 'Remote worker unreachable or missing Python 3, Codex or memory-backed /dev/shm.')
+        raise CdxError(message, code='remote_unavailable')
+    if local != result.stdout.decode().strip():
+        raise CdxError('Remote Codex CLI version mismatch; install the controller version on the worker.', code='remote_version_mismatch')
+    return local
+
+
+@contextmanager
+def remote_auth_lock(home):
+    try:
+        with codex_auth_lock(home, strict=True) as acquired:
+            if not acquired:
+                raise CdxError('Codex profile is busy on the controller or a worker.', code='session_busy')
+            yield
+    except OSError as error:
+        raise CdxError('Unable to hold the controller authentication lock.', code='auth_lock_failed') from error
+
+
+def run_remote(session, target, version, cwd=None, env_override=None, initial_prompt=None,
+               timeout_seconds=None, run_id=None, attempt=1, budget_check=None, remote_prefix=None, **_):
+    from .provider_runtime import (
+        _build_headless_launch_spec,
+        _combine_headless_transcript,
+        _headless_artifact_paths,
+        _headless_run_info,
+        _restore_signal_handlers,
+        _route_termination_to_interrupt,
+    )
+
+    home = session['authHome']
+    with remote_auth_lock(home):
+        auth = temporary_auth(home)
+        if auth['tokens'].get('refresh_token') != '':
+            raise CdxError('Refusing to send a refresh token to a worker.')
+        spec = _build_headless_launch_spec(session, cwd, env_override, initial_prompt)
+        if (session.get('launch') or {}).get('network_access'):
+            index = spec['args'].index('--') if '--' in spec['args'] else len(spec['args'])
+            spec['args'][index:index] = ['-c', 'sandbox_workspace_write.network_access=true']
+        config = nonsecret_config(home)
+        launch_text = json.dumps(spec['args']) + config
+        if any(value and value in launch_text for value in auth['tokens'].values()):
+            raise CdxError('Refusing a credential embedded in the remote prompt or configuration.')
+        payload = {'auth': auth, 'config': config, 'args': spec['args'], 'cwd': cwd,
+                   'version': version, 'expires_at': token_expiry(auth['tokens']['access_token']),
+                   'isolate_workspace': bool((session.get('launch') or {}).get('isolate_workspace')),
+                   'network': bool((session.get('launch') or {}).get('network_access')),
+                   'writable': (session.get('launch') or {}).get('permission') != 'review'}
+        rotating = json.loads((pathlib.Path(home) / 'auth.json').read_text())['tokens'].get('refresh_token')
+        if rotating and rotating in json.dumps(payload):
+            raise CdxError('Refusing to send a refresh token to a worker.')
+    paths = _headless_artifact_paths(session, run_id, attempt)
+    paths['remote'] = target
+    start = datetime.now(timezone.utc)
+    child = receiver = out = err = None
+    cancelled = timed_out = budget_exceeded = credential_expired = False
+    handlers = _route_termination_to_interrupt()
+    observation_ready = threading.Event()
+    observation_lock = threading.Lock()
+
+    def receive(source, output):
+        for line in source:
+            try:
+                frame = json.loads(line)
+                if frame.get('stream') == 'stdout':
+                    with observation_lock:
+                        observation_ready.clear()
+                        output.write(frame['data'])
+                        output.flush()
+                elif frame.get('stream') == 'observation':
+                    evidence = frame['evidence']
+                    evidence.update(stdout_path=paths['stdout_path'], run_id=paths['run_id'], attempt_id=paths.get('attempt_id'))
+                    with observation_lock:
+                        paths['usage_evidence'] = evidence
+                        observation_ready.set()
+            except (ValueError, TypeError, KeyError, AttributeError):
+                continue
+
+    try:
+        out = open(paths['stdout_path'], 'w', encoding='utf-8')
+        err = open(paths['stderr_path'], 'w', encoding='utf-8')
+        child = subprocess.Popen(ssh_command(target, 'run', remote_prefix), env={**os.environ, **(env_override or {})},
+                                 stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=err)
+        receiver = threading.Thread(target=receive, args=(child.stdout, out))
+        receiver.start()
+        child.stdin.write(worker_input() + json.dumps(payload).encode() + b'\n')
+        child.stdin.flush()
+        deadline = time.monotonic() + timeout_seconds if timeout_seconds else None
+        while child.poll() is None:
+            if time.time() >= payload['expires_at']:
+                credential_expired = True
+                break
+            with observation_lock:
+                if budget_check and observation_ready.is_set():
+                    observation_ready.clear()
+                    exceeded = budget_check(paths)
+                else:
+                    exceeded = False
+            if exceeded:
+                budget_exceeded = True
+                break
+            if deadline and time.monotonic() >= deadline:
+                timed_out = True
+                break
+            time.sleep(0.1)
+    except KeyboardInterrupt:
+        cancelled = True
+    except (OSError, ValueError) as error:
+        raise CdxError('Remote SSH transport failed.', code='remote_unavailable') from error
+    finally:
+        if child:
+            # EOF makes the worker stop its process group and remove the home.
+            try:
+                child.stdin.close()
+            except (OSError, BrokenPipeError):
+                pass
+            try:
+                child.wait(timeout=10)
+            except subprocess.TimeoutExpired:
+                child.terminate()
+                try:
+                    child.wait(timeout=2)
+                except subprocess.TimeoutExpired:
+                    child.kill()
+                    child.wait()
+        if receiver:
+            receiver.join()
+        if out:
+            out.close()
+        if err:
+            err.close()
+        _restore_signal_handlers(handlers)
+        _combine_headless_transcript(paths)
+    info = {**_headless_run_info(paths, spec, start, child.returncode), 'remote': target,
+            'timed_out': timed_out, 'budget_exceeded': budget_exceeded}
+    if cancelled:
+        error = CdxError('Remote run was cancelled.', 130)
+        error.cancelled = True
+        error.run_info = {**info, 'returncode': 130}
+        raise error
+    if credential_expired or time.time() >= payload['expires_at'] or child.returncode == 123:
+        error = CdxError('Remote temporary credential expired; start a new run from the controller.', code='remote_credential_expired')
+        error.run_info = info
+        raise error
+    if child.returncode == 255 and not (timed_out or budget_exceeded):
+        error = CdxError('Remote SSH connection failed or was lost.', code='remote_unavailable')
+        error.run_info = info
+        raise error
+    if timed_out or budget_exceeded:
+        info['returncode'] = 125 if budget_exceeded else 124
+    return info

package/src/remote_worker.py

@@ -0,0 +1,248 @@
+"""Self-contained Linux worker, framed over SSH stdin before the run payload."""
+
+import fcntl
+import json
+import os
+import pathlib
+import shutil
+import signal
+import stat
+import subprocess
+import sys
+import tempfile
+import threading
+import time
+import types
+from datetime import datetime, timezone
+
+# Supplied by the controller before this module is executed.
+SANDBOX_SOURCE = globals().get('SANDBOX_SOURCE', '')
+OBSERVATION_MODULES = globals().get('OBSERVATION_MODULES', {})
+OBSERVATION_LIMITS = globals().get('OBSERVATION_LIMITS', (128 * 1024 * 1024, 1))
+
+
+def observation_runtime():
+    # Load only controller-owned readers, with no worker installation required.
+    package = types.ModuleType('cdx_observation')
+    package.__path__ = []
+    sys.modules[package.__name__] = package
+    for name, source in OBSERVATION_MODULES.items():
+        module = types.ModuleType(package.__name__ + '.' + name)
+        module.__package__ = package.__name__
+        sys.modules[module.__name__] = module
+        exec(compile(source, name + '.py', 'exec'), module.__dict__)
+    return sys.modules[package.__name__ + '.run_usage'], sys.modules[package.__name__ + '.run_observation']
+
+
+def version():
+    return subprocess.check_output(["codex", "--version"], stderr=subprocess.DEVNULL, timeout=10).decode().strip()
+
+
+def storage():
+    mounts = [row.split() for row in pathlib.Path('/proc/mounts').read_text().splitlines()]
+    location = pathlib.Path('/dev/shm').resolve()
+    enclosing = [row for row in mounts if pathlib.Path(row[1]) == location or pathlib.Path(row[1]) in location.parents]
+    if not enclosing or max(enclosing, key=lambda row: len(row[1]))[2] != 'tmpfs':
+        raise RuntimeError("Remote execution requires memory-backed /dev/shm")
+    root = pathlib.Path('/dev/shm') / f'cdx-remote-{os.getuid()}'
+    root.mkdir(mode=0o700, exist_ok=True)
+    info = root.lstat()
+    if not stat.S_ISDIR(info.st_mode) or info.st_uid != os.getuid() or stat.S_IMODE(info.st_mode) != 0o700:
+        raise RuntimeError("Unsafe remote credential directory")
+    return root
+
+
+def allocate(root):
+    # Serialize sweep and allocation so another launcher cannot sweep a new home.
+    with open(root / '.sweep', 'a') as sweep:
+        fcntl.flock(sweep, fcntl.LOCK_EX)
+        for path in root.glob('run-*'):
+            if path.is_symlink() or not path.is_dir():
+                continue
+            try:
+                with open(path / '.live', 'a') as live:
+                    fcntl.flock(live, fcntl.LOCK_EX | fcntl.LOCK_NB)
+                    shutil.rmtree(path)
+            except (BlockingIOError, FileNotFoundError):
+                continue
+        home = pathlib.Path(tempfile.mkdtemp(prefix='run-', dir=root))
+        live = open(home / '.live', 'a')
+        fcntl.flock(live, fcntl.LOCK_EX)
+    return home, live
+
+
+def isolation_probe_command(payload, home, env):
+    # Execute exactly the local confinement implementation, shipped over stdin.
+    namespace = {'__name__': 'cdx_workspace_sandbox'}
+    exec(compile(SANDBOX_SOURCE, 'workspace_sandbox.py', 'exec'), namespace)
+    return namespace['sandbox_command'](
+        ['codex', '--version'], pathlib.Path(payload['cwd']), network=payload['network'],
+        writable=payload['writable'], source=env, auth_home=str(home), provider='codex',
+        observation_home=str(home / 'sessions'))
+
+
+def probe(root, payload):
+    if not payload.get('isolate_workspace'):
+        print(version())
+        return 0
+    # Exercise mounts, namespaces and the nft setup with a dummy home before
+    # the controller reads or sends a credential. Binary presence is insufficient.
+    with tempfile.TemporaryDirectory(prefix='probe-', dir=root) as directory:
+        home = pathlib.Path(directory)
+        (home / 'auth.json').write_text(json.dumps({'tokens': {'refresh_token': ''}}))
+        os.chmod(home / 'auth.json', 0o600)
+        (home / 'sessions').mkdir()
+        command, env = isolation_probe_command(payload, home, dict(os.environ))
+        result = subprocess.run(command, env=env, stdin=subprocess.DEVNULL,
+                                capture_output=True, timeout=20)
+        if result.returncode:
+            raise RuntimeError('Remote workspace isolation unavailable')
+        print(result.stdout.decode().strip())
+        return 0
+
+
+def main():
+    root = storage()
+    payload = json.loads(sys.stdin.buffer.readline())
+    if sys.argv[-1] == 'probe':
+        return probe(root, payload)
+    auth = payload['auth']
+    if auth['tokens'].get('refresh_token') != '':
+        raise RuntimeError('Refusing a rotating remote credential')
+    if version() != payload['version']:
+        raise RuntimeError('Remote Codex CLI version mismatch')
+    child = None
+    stopped = threading.Event()
+
+    def stop(*_):
+        stopped.set()
+        if child is not None and child.poll() is None:
+            try:
+                os.killpg(child.pid, signal.SIGTERM)
+                child.wait(timeout=3)
+            except subprocess.TimeoutExpired:
+                os.killpg(child.pid, signal.SIGKILL)
+                child.wait()
+            except ProcessLookupError:
+                pass
+
+    def disconnected():
+        while os.read(0, 1):
+            pass
+        stop()
+
+    signal.signal(signal.SIGTERM, stop)
+    signal.signal(signal.SIGHUP, stop)
+    signal.signal(signal.SIGINT, stop)
+    home, live = allocate(root)
+    try:
+        if stopped.is_set():
+            return 130
+        os.umask(0o077)
+        (home / 'auth.json').write_text(json.dumps(auth))
+        os.chmod(home / 'auth.json', 0o600)
+        (home / 'config.toml').write_text('cli_auth_credentials_store = "file"\n' + payload['config'])
+        env = {key: value for key, value in os.environ.items()
+               if key not in ('OPENAI_API_KEY', 'OPENAI_ACCESS_TOKEN', 'CODEX_ACCESS_TOKEN', 'CODEX_API_KEY', 'CODEX_HOME')}
+        env['CODEX_HOME'] = str(home)
+        command = ['codex', *payload['args']]
+        if payload.get('isolate_workspace'):
+            (home / 'sessions').mkdir(mode=0o700)
+            args = payload['args'][:]
+            while '-s' in args:
+                index = args.index('-s')
+                del args[index:index + 2]
+            if '--dangerously-bypass-approvals-and-sandbox' not in args:
+                index = args.index('--') if '--' in args else len(args)
+                args[index:index] = ['--dangerously-bypass-approvals-and-sandbox']
+            namespace = {'__name__': 'cdx_workspace_sandbox'}
+            exec(compile(SANDBOX_SOURCE, 'workspace_sandbox.py', 'exec'), namespace)
+            command, env = namespace['sandbox_command'](
+                ['codex', *args], pathlib.Path(payload['cwd']), network=payload['network'],
+                writable=payload['writable'], source=env, auth_home=str(home), provider='codex',
+                observation_home=str(home / 'sessions'))
+        started_at = datetime.now(timezone.utc).isoformat()
+        child = subprocess.Popen(command, cwd=payload['cwd'], env=env,
+                                 stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+                                 start_new_session=True)
+        if stopped.is_set():
+            stop()
+        threading.Thread(target=disconnected, daemon=True).start()
+        secrets = [value for value in auth['tokens'].values() if isinstance(value, str) and value]
+
+        output_lock = threading.Lock()
+        usage_reader, observation_reader = observation_runtime()
+        stdout_path = home / 'stdout.jsonl'
+        stdout_path.touch(mode=0o600)
+
+        def snapshot():
+            # Read the same exact-thread evidence as the local supervisor before
+            # volatile storage is removed. Only normalized metadata is exported.
+            with output_lock:
+                usage = usage_reader.extract_run_usage('codex', stdout_path)
+                identifier = usage_reader.extract_run_conversation_id('codex', stdout_path)
+                evidence = {'source': 'isolated_codex_rollout', 'thread_id': identifier,
+                            'started_at': started_at, 'usage': usage, 'models': [], 'turns': []}
+                if identifier:
+                    paths = []
+                    deadline = time.monotonic() + OBSERVATION_LIMITS[1]
+                    for directory, subdirs, names in os.walk(home / 'sessions'):
+                        if time.monotonic() > deadline:
+                            paths = []
+                            break
+                        subdirs[:] = [name for name in subdirs if name != 'subagents']
+                        paths.extend(pathlib.Path(directory) / name for name in names
+                                     if name.startswith('rollout-') and name.endswith(f'-{identifier}.jsonl'))
+                        if len(paths) > 1:
+                            break
+                    if len(paths) == 1:
+                        evidence = observation_reader.codex_rollout_evidence(home, paths[0], evidence, OBSERVATION_LIMITS[0])
+                event = json.dumps({'stream': 'observation', 'evidence': evidence})
+                for secret in secrets:
+                    event = event.replace(secret, '[REDACTED]')
+                print(event, flush=True)
+
+        def forward(source, target):
+            for line in iter(source.readline, b''):
+                for secret in secrets:
+                    line = line.replace(secret.encode(), b'[REDACTED]')
+                with output_lock:
+                    if target is sys.stdout:
+                        with stdout_path.open('ab') as capture:
+                            capture.write(line)
+                        # Frame provider output so it cannot forge supervisor evidence.
+                        print(json.dumps({'stream': 'stdout', 'data': line.decode('utf-8', errors='replace')}), flush=True)
+                    else:
+                        target.buffer.write(line)
+                        target.buffer.flush()
+
+        threads = [threading.Thread(target=forward, args=pair) for pair in
+                   ((child.stdout, sys.stdout), (child.stderr, sys.stderr))]
+        for thread in threads:
+            thread.start()
+        while child.poll() is None:
+            try:
+                child.wait(timeout=0.25)
+            except subprocess.TimeoutExpired:
+                snapshot()
+        result = child.returncode
+        for thread in threads:
+            thread.join()
+        snapshot()
+        if time.time() >= payload['expires_at']:
+            print('Remote temporary credential expired; start a new run from the controller.', file=sys.stderr)
+            return 123
+        return 130 if stopped.is_set() else result
+    finally:
+        stop()
+        shutil.rmtree(home)
+        live.close()
+
+
+if __name__ == '__main__':
+    try:
+        sys.exit(main())
+    except Exception:
+        # Never render exception values: input and provider errors may contain credentials.
+        print('Remote worker setup or execution failed; check Python, Codex, cwd, /dev/shm and isolation dependencies.', file=sys.stderr)
+        sys.exit(126)

package/src/run_command.py

@@ -169,6 +169,7 @@ def run_launch_payload(api_schema_version, parsed, session, artifacts, cwd=None,
         "ok": True,
         "action": "run",
         "launcher": "cdx",
+        **({"remote": parsed["remote"]} if parsed.get("remote") else {}),
         "project": parsed.get("project"),
         "project_source": parsed.get("project_source", "unknown"),
         "parent_run_id": parsed.get("continue_run"),
@@ -221,6 +222,7 @@ def run_result_payload(api_schema_version, ok, parsed, session, run_info=None,
         "ok": bool(ok),
         "action": "run",
         "launcher": "cdx",
+        **({"remote": parsed["remote"]} if parsed.get("remote") else {}),
         "project": parsed.get("project"),
         "project_source": parsed.get("project_source", "unknown"),
         "parent_run_id": parsed.get("continue_run"),

package/src/run_observation.py

@@ -6,8 +6,6 @@ import stat
 from datetime import datetime, timezone
 from pathlib import Path
 
-from .history_models import _exact_transcript
-from .interactive_usage import MAX_TRANSCRIPT_BYTES
 from .run_usage import _usage_from_dict, extract_run_conversation_id, extract_run_usage
 
 
@@ -31,6 +29,9 @@ def codex_attempt_evidence(auth_home, stdout_path, started_at):
     Do not import claimed costs, requested models, messages, or arbitrary files
     from the workspace. The exported object is never writable by the sandbox.
     """
+    from .history_models import _exact_transcript
+    from .interactive_usage import MAX_TRANSCRIPT_BYTES
+
     usage = extract_run_usage("codex", stdout_path)
     identifier = extract_run_conversation_id("codex", stdout_path)
     evidence = {"source": "isolated_codex_rollout", "stdout_path": str(stdout_path),
@@ -41,6 +42,14 @@ def codex_attempt_evidence(auth_home, stdout_path, started_at):
     path, error = _exact_transcript("codex", auth_home, identifier)
     if error or not path:
         return evidence
+    return codex_rollout_evidence(auth_home, path, evidence, MAX_TRANSCRIPT_BYTES)
+
+
+def codex_rollout_evidence(auth_home, path, evidence, max_bytes):
+    """Validate exact-attempt rollout evidence identically on local and remote hosts."""
+    usage = evidence["usage"]
+    identifier = evidence["thread_id"]
+    started_at = evidence["started_at"]
     models = set()
     turns = []
     identity = False
@@ -52,8 +61,8 @@ def codex_attempt_evidence(auth_home, stdout_path, started_at):
         with os.fdopen(descriptor, encoding="utf-8") as handle:
             if not stat.S_ISREG(os.fstat(handle.fileno()).st_mode):
                 return evidence
-            text = handle.read(MAX_TRANSCRIPT_BYTES + 1)
-        if len(text) > MAX_TRANSCRIPT_BYTES:
+            text = handle.read(max_bytes + 1)
+        if len(text) > max_bytes:
             return evidence
         for line in text.splitlines():
             record = json.loads(line)

package/src/run_registry.py

@@ -244,15 +244,21 @@ class RunRegistry:
             _check_capacity(_read_registry(self.path), max_parallel)
 
     def start(self, run_id, *, kind, session, provider, model, cwd, artifacts=None, label=None, max_parallel=None,
-              parent_run_id=None, conversation_id=None, launch=None, reasoning_effort=None, project=None, project_source="unknown"):
+              parent_run_id=None, conversation_id=None, launch=None, reasoning_effort=None, project=None, project_source="unknown", remote=None):
         with _registry_lock(self.path):
             data = _read_registry(self.path)
+            _refresh_stale_runs(data)
+            for run in data["runs"]:
+                if (run.get("run_id") != run_id and run.get("status") == "running"
+                        and run.get("session") == session and (remote or run.get("remote"))):
+                    raise CdxError(f"Session is busy on the controller or a worker: {session}", code="session_busy")
             data["runs"] = [run for run in data["runs"] if run.get("run_id") != run_id]
             if max_parallel:
                 # Counted under the lock and without this run's own record, which a
                 # detached child writes a second time under the same id.
                 _check_capacity(data, max_parallel)
             record = _base_record(run_id, kind=kind, session=session, provider=provider, model=model, cwd=cwd, artifacts=artifacts)
+            record["remote"] = remote
             record.update(parent_run_id=parent_run_id, conversation_id=conversation_id,
                           launch=dict(launch or {}), reasoning_effort=reasoning_effort,
                           project=project, project_source=project_source)

package/src/workspace_sandbox.py

@@ -5,8 +5,11 @@ Provider authentication is the only credential deliberately passed to the provid
 """
 from __future__ import annotations
 
+import json
 import os
 import shutil
+import tempfile
+from contextlib import contextmanager
 from pathlib import Path
 
 CACHE = Path.home() / ".cdx" / "shared-cache"
@@ -47,6 +50,24 @@ sys.exit(status)
 """
 
 
+@contextmanager
+def private_auth_home(auth):
+    """Keep a sanitized credential in verified memory-backed, private storage."""
+    location = Path('/dev/shm').resolve()
+    mounts = [row.split() for row in Path('/proc/mounts').read_text().splitlines()]
+    enclosing = [row for row in mounts if Path(row[1]) == location or Path(row[1]) in location.parents]
+    if not enclosing or max(enclosing, key=lambda row: len(row[1]))[2] != 'tmpfs':
+        raise RuntimeError('Isolated Codex runs require memory-backed /dev/shm')
+    if auth.get('tokens', {}).get('refresh_token') != '':
+        raise RuntimeError('Refusing a rotating sandbox credential')
+    with tempfile.TemporaryDirectory(prefix='cdx-observation-', dir=location) as directory:
+        path = Path(directory) / 'auth.json'
+        with open(path, 'x', opener=lambda name, flags: os.open(name, flags, 0o600)) as handle:
+            json.dump(auth, handle)
+        (Path(directory) / 'sessions').mkdir(mode=0o700)
+        yield directory
+
+
 def clean_env(source: dict) -> dict:
     allowed = ("PATH", "LANG", "LC_ALL", "TERM", "TZ")
     env = {key: source[key] for key in allowed if key in source}
@@ -126,6 +147,9 @@ def sandbox_command(command: list[str], tree: Path, *, network: bool, writable:
         raise RuntimeError("Expected an isolated git worktree")
     env = clean_env(source)
     if provider == "codex":
+        auth_path = Path(auth_home) / "auth.json"
+        if json.loads(auth_path.read_text()).get('tokens', {}).get('refresh_token') != '':
+            raise RuntimeError('Refusing a rotating sandbox credential')
         env["CODEX_HOME"] = "/home/agent/.codex"
         if observation_home:
             # Fresh, volatile attempt storage; no host profile is exposed.