cdx-manager 0.24.6 → 0.24.8

package/README.md

@@ -4,7 +4,7 @@
 
 <br clear="left"/>
 
-[![License](https://img.shields.io/badge/license-MIT-4C8BF5)](LICENSE) ![Version](https://img.shields.io/badge/version-v0.24.6-4C8BF5) ![Python](https://img.shields.io/badge/python-3.9%2B-3776AB?logo=python&logoColor=white)
+[![License](https://img.shields.io/badge/license-MIT-4C8BF5)](LICENSE) ![Version](https://img.shields.io/badge/version-v0.24.8-4C8BF5) ![Python](https://img.shields.io/badge/python-3.9%2B-3776AB?logo=python&logoColor=white)
 
 **The right AI coding account, when you need it.** CDX Manager keeps your Codex and Claude accounts in separate profiles, shows which ones have quota, and helps you pick the next usable session from your terminal.
 
@@ -42,6 +42,14 @@ cdx work
 
 CDX checks authentication before interactive launches. Handoff keeps the source transcript in place and asks the receiving agent to verify the conversation and workspace before acting. No account switch or prompt submission happens just because you run `cdx next`.
 
+## Codex model capacity recovery
+
+`cdx run` (including `--detach`) automatically recovers from Codex's "Selected model is at capacity" failure. It retries the same prompt, profile, and model after 1 and 2 seconds, then tries the profile's `fallback_model` list in configured order, removing duplicates. Configure alternatives with `cdx set work --fallback-model MODEL_A,MODEL_B`.
+
+After exhausting those models, CDX selects another ready Codex profile using the normal ranking, including profile priority. Disabled, logged-out, `never`, and `ask-first` profiles are excluded from automatic selection. To choose an `ask-first` profile explicitly, name it in a new command (`cdx run NAME ...`). Each successor uses `--model` if supplied, otherwise its own configured/default model, followed by its own fallback list. Each model gets at most three attempts; at most four profile switches are allowed. Settings are not changed. Each attempt retains the command's timeout; the backoff adds at most three seconds per model.
+
+Recovery does not require `--failover`, which still controls account quota exhaustion separately. Other provider failures and timeouts do not trigger capacity recovery. Attempts restart the prompt in the same workspace; files from earlier attempts remain there. Each retry, model fallback, profile fallback, and exhausted budget is logged to stderr and persisted in `recovery_events` in the run registry and final JSON (`cdx run-status` / `cdx run-report`). Events include archived attempt output paths. Recovery keeps one run ID and reports the final effective model. If no eligible alternative remains or the profile switch budget is reached, the run fails with `capacity_recovery_exhausted`.
+
 ## Documentation
 
 - [Installation and first steps](docs/getting-started.md) — prerequisites, npm/pipx/uv, Windows, configuration.

package/VERSION

@@ -1 +1 @@
-0.24.6
+0.24.8

package/changelogs/CHANGELOGS_0_24_7.md

@@ -0,0 +1,22 @@
+# CDX Manager 0.24.7
+
+Release date: 2026-10-29
+
+This patch fixes command execution in isolated Codex workspace runs. (#65)
+
+## Workspace runs
+
+- Drop Codex's nested sandbox when `--isolate-workspace` is enabled, avoiding the user namespace mapping failure that prevented commands from running. The outer workspace sandbox remains the isolation boundary. (#65)
+- Remove Codex's sandbox mode and network overrides inside the isolated workspace and pass `--dangerously-bypass-approvals-and-sandbox`; the outer sandbox controls network access and keeps review runs read-only through its workspace mount. (#65)
+
+## Codex capacity recovery
+
+- Retry headless Codex model capacity failures after 1 and 2 seconds before trying configured fallback models, then ready Codex profiles in the existing priority-aware order. This also applies to detached runs and does not require `--failover`.
+- Exclude disabled, logged-out, `never`, and `ask-first` profiles from automatic fallback. Bound recovery to three attempts per model and four profile switches, with a distinct `capacity_recovery_exhausted` outcome.
+- Log each recovery action to stderr and structured run data, preserve failed attempt output, and report the effective model without changing profile settings.
+- Cover capacity detection, retry and profile budgets, fallback order, selection policies, and backoff cancellation with tests.
+
+## Maintenance
+
+- Record tray companion checksums for v0.24.6. (`87aba5d2cffdfb1a96e386abcdabe54df9ee4808`)
+- Record release archive checksums for v0.24.6. (`6b92094143ea28bf0cc096aef88d1015565aba18`)

package/changelogs/CHANGELOGS_0_24_8.md

@@ -0,0 +1,16 @@
+# CDX Manager 0.24.8
+
+Release date: 2026-10-30
+
+This patch improves recovery from Codex model capacity failures. (#67)
+
+## Codex capacity recovery
+
+- Retry headless Codex capacity failures after 1 and 2 seconds before trying configured fallback models, then ready Codex profiles in priority-aware order. Recovery also applies to detached runs and does not require `--failover`. (#67)
+- Exclude disabled, logged-out, `never`, and `ask-first` profiles from automatic fallback. Limit recovery to three attempts per model and four profile switches, reporting `capacity_recovery_exhausted` when no eligible alternative remains or the budget is exhausted. (#67)
+- Log recovery actions to stderr and structured run data, preserve failed attempt output, and report the effective model without changing profile settings. (#67)
+
+## Maintenance
+
+- Record tray companion checksums for v0.24.7. (`16938c0d22de4589f5b4757fd7ff55e848127e7d`)
+- Record release archive checksums for v0.24.7. (`a169c3eee41fcbc97477ca4d74bfc8f77c1330d5`)

package/checksums/release-archives.json

@@ -639,6 +639,28 @@
         "x86_64-pc-windows-msvc": "74124b2e40a58a58b55724ef453c4b543a83eefab9a5f92685b239c5d97e7f67",
         "x86_64-unknown-linux-musl": "089e552a4dfae4080a2e76cedb6fe72fc8013a9cd08cf3c8b333f916198da57c"
       }
+    },
+    "v0.24.6": {
+      "github_tarball_sha256": "4b61861ae26fc7904e751571b65d0783c16dcf35d3b485379434469e27511fb4",
+      "github_zip_sha256": "6c5cfa2139eb53a20a8eca8624437a06b18392b0df7774d9c7c64a74b8066f59",
+      "tray_assets": {
+        "aarch64-apple-darwin": "7200f658e3413b2614a7ffa5d03e1dac173fc1a653dc659d92f2bef40f2bf5d4",
+        "aarch64-unknown-linux-musl": "b892a9a3bf009e5069dc798ab910b20e77ac9d03b9b575a80cb19af1dc13533a",
+        "x86_64-apple-darwin": "d1e1b619feab8eae1e2a25e17942d8d690fc1e6184f294599a15858a1f0c146e",
+        "x86_64-pc-windows-msvc": "0d125d8e0f7d9880a2ed9e0a794964f3d69ef3353b90a1d594b17f9392a0a0af",
+        "x86_64-unknown-linux-musl": "d6c369ace10a6d373366cd12e4e6cd7f2fd278fbce2fd9a13c5bf51047847441"
+      }
+    },
+    "v0.24.7": {
+      "github_tarball_sha256": "170df61319347dc133442c6750c31812a55a3e8a002fae60bd4822771839750f",
+      "github_zip_sha256": "fa3eda26dbfec1af36e25f72a0fcf01a18fa064dc442ffc39b64c7bb2ab7e1ae",
+      "tray_assets": {
+        "aarch64-apple-darwin": "f775ae9e140f1147368c4658a27e6f0751585a5b001b633fcfccfe252b85721e",
+        "aarch64-unknown-linux-musl": "528d61bb9dc5c395f74aad98d35f8cebea958ba2561fd15302846eb8bc036343",
+        "x86_64-apple-darwin": "e85cd5d2cb2f5dd1a2dd89ff9b532432cffd86175e709d147ff4595551d4ff50",
+        "x86_64-pc-windows-msvc": "41df35daa664802b87c052d6a9f634b240eb4e4a915868f1dd629b6734eab76a",
+        "x86_64-unknown-linux-musl": "2492e137e5c883fa16f502843a867e0325744dfc66d7d38be2c00eb362425af6"
+      }
     }
   }
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "cdx-manager",
-  "version": "0.24.6",
+  "version": "0.24.8",
   "description": "Terminal session manager for Codex and Claude accounts.",
   "license": "MIT",
   "author": "Alexandre Agostini",

package/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "cdx-manager"
-version = "0.24.6"
+version = "0.24.8"
 description = "Terminal session manager for Codex and Claude accounts."
 readme = "README.md"
 requires-python = ">=3.9"

package/src/cli_args.py

@@ -543,6 +543,7 @@ def cdx_schema():
                 "too_many_runs",
                 "failover_exhausted",
                 "failover_limit_reached",
+                "capacity_recovery_exhausted",
                 "invalid_request",
                 "cdx_error",
             ],

package/src/commands/runs.py

@@ -5,8 +5,10 @@ Split out of cli_commands.py. Moved verbatim; re-exported by cli_commands.
 
 import importlib.util
 import os
+import shutil
 import subprocess
 import sys
+import time
 
 from ..cli_args import (
     RUN_REPORT_USAGE,
@@ -39,10 +41,18 @@ from ..provider_runtime import (
     _build_launch_spec,
     _ensure_session_authentication,
     _headless_artifact_paths,
+    _restore_signal_handlers,
+    _route_termination_to_interrupt,
     _run_headless_provider_command,
 )
 from ..run_command import read_run_prompt, run_cdx_error_code, run_launch_payload, run_result_payload
-from ..run_failover import MAX_FAILOVER_TRANSITIONS, failover_reason
+from ..run_failover import (
+    CAPACITY_RETRY_DELAYS,
+    MAX_FAILOVER_TRANSITIONS,
+    capacity_models,
+    failover_reason,
+    looks_model_at_capacity,
+)
 from ..run_registry import RunRegistry, build_code_review_report, final_message
 from ..run_usage import extract_run_observation
 from ..session_ranking import FACTOR_DESCRIPTIONS, rank_sessions, selection_policy
@@ -676,7 +686,13 @@ def handle_run(rest, ctx):
 
         attempted = [run_session["name"]]
         failover_error_code = None
+        recovery_events = []
+        attempt = 0
+        models = capacity_models(run_session)
+        model_index = 0
+        capacity_retries = 0
         while True:
+            attempt += 1
             run_info = _run_headless_provider_command(
                 run_session,
                 cwd=cwd,
@@ -690,16 +706,70 @@ def handle_run(rest, ctx):
                 run_session.get("provider"), run_info.get("stdout_path"),
                 run_session.get("authHome"),
             )
-            run_info = {**run_info, "usage": usage, "usage_model": usage_model}
+            run_info = {**run_info, "usage": usage, "usage_model": usage_model,
+                        "effective_model": (run_session.get("launch") or {}).get("model"),
+                        "recovery_events": recovery_events}
             ctx["service"]["record_launch_history"](run_session["name"], {
                 "status": "success" if run_info.get("returncode") == 0 else "failed",
                 "action": "run",
                 "run_id": run_id,
-                "attempt": len(attempted),
+                "attempt": attempt,
                 "cwd": cwd,
                 "exit_code": run_info.get("returncode"),
                 **run_info,
             })
+            if looks_model_at_capacity(run_session.get("provider"), run_info):
+                event = {"reason": "model_at_capacity", "attempt": attempt,
+                         "session": run_session["name"], "model": models[model_index]}
+                if capacity_retries < len(CAPACITY_RETRY_DELAYS):
+                    delay = CAPACITY_RETRY_DELAYS[capacity_retries]
+                    capacity_retries += 1
+                    event.update(action="retry", retry=capacity_retries, delay_seconds=delay)
+                elif model_index + 1 < len(models):
+                    model_index += 1
+                    capacity_retries = 0
+                    run_session["launch"]["model"] = models[model_index]
+                    event.update(action="fallback_model", target_model=models[model_index])
+                else:
+                    successor = None if len(attempted) > MAX_FAILOVER_TRANSITIONS else _select_headless_session(
+                        ctx, "codex", min_reasoning_effort=parsed["reasoning_effort"],
+                        require_ready=True, exclude=attempted,
+                    )
+                    if not successor:
+                        failover_error_code = "capacity_recovery_exhausted"
+                        event.update(action="exhausted")
+                    else:
+                        selection_row = successor.get("row") or {}
+                        next_session = successor["session"]
+                        run_session = {**next_session, "launch": {**(next_session.get("launch") or {}), **launch_updates}}
+                        # Each new profile starts with the requested model, or its own default.
+                        models = capacity_models(run_session)
+                        model_index = capacity_retries = 0
+                        attempted.append(run_session["name"])
+                        event.update(action="fallback_profile", target_session=run_session["name"], target_model=models[0])
+                _record_capacity_recovery(ctx, registry, run_id, event, run_info, recovery_events)
+                if failover_error_code:
+                    break
+                if event["action"] == "retry":
+                    previous_handlers = _route_termination_to_interrupt()
+                    try:
+                        time.sleep(event["delay_seconds"])
+                    except KeyboardInterrupt as interrupt:
+                        error = CdxError("Codex capacity recovery was cancelled.", 130)
+                        error.cancelled = True
+                        error.history_recorded = True
+                        error.run_info = {**run_info, "returncode": 130}
+                        raise error from interrupt
+                    finally:
+                        _restore_signal_handlers(previous_handlers)
+                elif event["action"] == "fallback_profile":
+                    _ensure_session_authentication(
+                        run_session, ctx["service"], spawn=ctx.get("spawn"), spawn_sync=ctx.get("spawn_sync"),
+                        env_override=ctx.get("env"), stdin_is_tty=False, behavior="launch",
+                        signal_emitter=ctx.get("signal_emitter"), trust_local_credentials=False,
+                    )
+                    registry.migrate(run_id, session=run_session["name"], provider="codex", reason="model_at_capacity")
+                continue
             if run_info.get("returncode") == 0 or not parsed.get("failover"):
                 break
             reason = failover_reason(
@@ -731,6 +801,8 @@ def handle_run(rest, ctx):
                 "launch": {**(successor["session"].get("launch") or {}), **launch_updates},
             }
             attempted.append(run_session["name"])
+            models = capacity_models(run_session)
+            model_index = capacity_retries = 0
             # A different account means different credentials; an unauthenticated
             # successor would fail for a reason unrelated to the task.
             _ensure_session_authentication(
@@ -769,7 +841,9 @@ def handle_run(rest, ctx):
             )
             _write_json(ctx, final_payload)
             return 0
-        if failover_error_code == "failover_exhausted":
+        if failover_error_code == "capacity_recovery_exhausted":
+            message = "Codex model capacity recovery exhausted its retry, model, or profile budget."
+        elif failover_error_code == "failover_exhausted":
             message = f"Every eligible session was rate limited: {', '.join(attempted)}."
         elif failover_error_code == "failover_limit_reached":
             message = f"Failover stopped after {MAX_FAILOVER_TRANSITIONS} transitions: {', '.join(attempted)}."
@@ -806,7 +880,9 @@ def handle_run(rest, ctx):
     except CdxError as error:
         run_info = getattr(error, "run_info", None)
         cancelled = bool(getattr(error, "cancelled", False))
-        if run_info and run_id:
+        if run_info is not None:
+            run_info = {**run_info, "recovery_events": locals().get("recovery_events", [])}
+        if run_info and run_id and not getattr(error, "history_recorded", False):
             usage, usage_model = extract_run_observation(
                 run_session.get("provider"), run_info.get("stdout_path"),
                 run_session.get("authHome"),
@@ -816,7 +892,7 @@ def handle_run(rest, ctx):
                 "status": "failed",
                 "action": "run",
                 "run_id": run_id,
-                "attempt": len(attempted) if "attempted" in locals() else 1,
+                "attempt": locals().get("attempt", 1),
                 "cwd": cwd,
                 "exit_code": error.exit_code,
                 **run_info,
@@ -825,13 +901,14 @@ def handle_run(rest, ctx):
             API_SCHEMA_VERSION,
             False,
             locals().get("parsed", {}) or {},
-            locals().get("session"),
+            locals().get("run_session", locals().get("session")),
             run_info=run_info,
             error=error,
             error_source="provider" if cancelled else "cdx",
             error_code="run_cancelled" if cancelled else run_cdx_error_code(error),
             selection_row=selection_row,
         )
+        final_payload["recovery_events"] = locals().get("recovery_events", [])
         if registry and run_id:
             registry.finish(
                 run_id,
@@ -845,6 +922,22 @@ def handle_run(rest, ctx):
         _write_json(ctx, final_payload)
         return error.exit_code
 
+def _record_capacity_recovery(ctx, registry, run_id, event, run_info, events):
+    # Subsequent attempts reuse the run's paths; retain the failed output first.
+    artifacts = {}
+    for key in ("stdout_path", "stderr_path", "transcript_path"):
+        path = run_info.get(key)
+        if path and os.path.isfile(path):
+            archived = f"{path}.attempt-{event['attempt']}"
+            shutil.copyfile(path, archived)
+            artifacts[key] = archived
+    event["artifacts"] = artifacts
+    events.append(event)
+    registry.record_recovery(run_id, event)
+    details = ", ".join(f"{key}={value}" for key, value in event.items() if key != "artifacts")
+    ctx["err"](f"cdx capacity recovery: {details}\n")
+
+
 def _select_headless_session(
     ctx, provider, min_reasoning_effort=None, require_ready=False, force_refresh=False, exclude=None,
 ):

package/src/provider_runtime.py

@@ -1212,7 +1212,10 @@ def _run_headless_provider_command(session, cwd=None, env_override=None, initial
     )
     launch = session.get("launch") or {}
     if launch.get("network_access"):
-        if session.get("provider") == PROVIDER_CODEX:
+        if session.get("provider") == PROVIDER_CODEX and launch.get("isolate_workspace"):
+            # The outer sandbox owns the network; Codex's own sandbox is dropped below.
+            pass
+        elif session.get("provider") == PROVIDER_CODEX:
             # Keep the workspace sandbox; append overrides before the prompt separator.
             index = spec["args"].index("--") if "--" in spec["args"] else len(spec["args"])
             spec["args"][index:index] = ["-c", 'sandbox_workspace_write.network_access=true', '-c', 'sandbox_workspace_write.writable_roots=["/cache", "/home/agent"]']
@@ -1226,6 +1229,17 @@ def _run_headless_provider_command(session, cwd=None, env_override=None, initial
             # The outer OS sandbox is the permission boundary for unattended Bash.
             index = spec["args"].index("--") if "--" in spec["args"] else len(spec["args"])
             spec["args"][index:index] = ["--permission-mode", "bypassPermissions"]
+        elif session.get("provider") == PROVIDER_CODEX:
+            # Codex's bubblewrap sandbox cannot nest inside ours (no uid map), so the
+            # outer box is the boundary: drop `-s <mode>` and bypass the inner one.
+            # A review run stays read-only through the outer mount.
+            args = spec["args"]
+            while "-s" in args:
+                index = args.index("-s")
+                del args[index:index + 2]
+            if "--dangerously-bypass-approvals-and-sandbox" not in args:
+                index = args.index("--") if "--" in args else len(args)
+                args[index:index] = ["--dangerously-bypass-approvals-and-sandbox"]
         try:
             argv, child_env = sandbox_command(
                 [spec["command"], *spec["args"]], Path(cwd),

package/src/run_command.py

@@ -210,7 +210,8 @@ def run_result_payload(api_schema_version, ok, parsed, session, run_info=None,
         "launcher": "cdx",
         "session": session.get("name") if session else None,
         "provider": session.get("provider") if session else parsed.get("provider"),
-        "model": parsed.get("model") or ((session.get("launch") or {}).get("model") if session else None),
+        "model": run_info.get("effective_model", parsed.get("model") or ((session.get("launch") or {}).get("model") if session else None)),
+        "recovery_events": run_info.get("recovery_events", []),
         "reasoning_effort": run_payload_reasoning_effort(parsed, session),
         "power": parsed.get("power") or ((session.get("launch") or {}).get("power") if session else None),
         "cwd": os.path.abspath(parsed.get("cwd") or os.getcwd()),

package/src/run_failover.py

@@ -1,6 +1,6 @@
-"""Deciding whether a finished run stopped because its account ran out.
+"""Classify finished runs for account exhaustion or model capacity recovery.
 
-Two independent signals have to agree before cdx moves a task to another
+For account exhaustion, two independent signals must agree before cdx moves a task to another
 account, because the two ways of being wrong are not symmetric. Missing a rate
 limit costs nothing new: the run fails exactly as it does today. Inventing one
 migrates a healthy run off a working account, discards its progress, and
@@ -22,6 +22,7 @@ codebase reads provider output for this purpose.
 
 import json
 import os
+import re
 
 from .config import PROVIDER_CLAUDE, PROVIDER_CODEX
 
@@ -235,3 +236,32 @@ def failover_reason(provider, run_info, status_row):
 
 def should_fail_over(provider, run_info, status_row):
     return failover_reason(provider, run_info, status_row) is not None
+
+
+CAPACITY_RETRY_DELAYS = (1, 2)
+_CAPACITY_MESSAGE = re.compile(r"(?:selected model|(?:the )?model(?: [\w.-]+)?) is at capacity\b", re.IGNORECASE)
+
+
+def looks_model_at_capacity(provider, run_info):
+    """Recognize Codex failures, excluding successful output and assistant prose."""
+    if provider != PROVIDER_CODEX or run_info.get("timed_out") or run_info.get("returncode") in (None, 0):
+        return False
+    for key in ("stdout_path", "stderr_path"):
+        text = _read_tail(run_info.get(key))
+        for record in _structured_records(provider, text):
+            if any(_CAPACITY_MESSAGE.search(message) for message in _error_texts(provider, record)):
+                return True
+        # Some Codex versions print this diagnostic as a standalone plain line.
+        for line in text.splitlines():
+            line = line.strip()
+            if _CAPACITY_MESSAGE.match(re.sub(r"^error:\s*", "", line, flags=re.IGNORECASE)):
+                return True
+    return False
+
+
+def capacity_models(session):
+    """Configured alternatives in order, without retrying the current model."""
+    launch = session.get("launch") or {}
+    models = [launch.get("model")]
+    models.extend(str(launch.get("fallback_model") or "").split(","))
+    return list(dict.fromkeys(model.strip() if model else None for model in models if model is None or model.strip()))

package/src/run_registry.py

@@ -321,6 +321,16 @@ class RunRegistry:
                 return run
             return None
 
+    def record_recovery(self, run_id, event):
+        """Publish retries and fallbacks while a supervised run is still active."""
+        with _registry_lock(self.path):
+            data = _read_registry(self.path)
+            for run in data["runs"]:
+                if run.get("run_id") == run_id:
+                    run.setdefault("recovery_events", []).append({"at": utc_now_iso(), **event})
+                    _write_registry(self.path, data)
+                    return
+
     def finish(self, run_id, *, status, final_payload=None, run_info=None, error=None, task_report=None):
         with _registry_lock(self.path):
             return self._finish_locked(run_id, status=status, final_payload=final_payload, run_info=run_info, error=error, task_report=task_report)
@@ -359,6 +369,7 @@ class RunRegistry:
                 # used to store whichever shape the producing reader emitted.
                 run["usage"] = coerce_usage(final_payload.get("usage"))
                 run["usage_model"] = final_payload.get("usage_model")
+                run["model"] = final_payload.get("model")
                 run["final_payload"] = final_payload
             if error:
                 run["error"] = error