@dtechvision/fabrik-runtime 0.1.0 → 0.1.1

package/README.md

@@ -1,70 +1,59 @@
 # @dtechvision/fabrik-runtime
 
-Shared TypeScript utilities for Fabrik workflow pods.
+TypeScript helpers for Fabrik/Smithers workflows.
 
-- **Credential pool** — read from mounted `/etc/fabrik/credentials`, rotate on failure, notify operators
-- **Codex auth rotation** — rotate among `auth.json` / `*.auth.json` credentials for Codex-backed workflows
-- **K8s jobs** — dispatch child verification jobs from a running workflow
-- **JJ shell** — deterministic JJ/Git snapshot, bookmark push, workspace prep
+## Scope
 
-## Import Surface
+Use this package for server-side workflow code.
 
-Workflows should import from `@dtechvision/fabrik-runtime/...`.
+Do not use it for:
+- browser apps
+- generic Node libraries
 
+Published entrypoints:
+- `@dtechvision/fabrik-runtime`
 - `@dtechvision/fabrik-runtime/credential-pool`
 - `@dtechvision/fabrik-runtime/codex-auth`
 - `@dtechvision/fabrik-runtime/jj-shell`
 - `@dtechvision/fabrik-runtime/k8s-jobs`
 
-For in-cluster Fabrik runs, the Smithers runtime image ships this package in its `node_modules`.
-For local workflow development in another repo, add the package as a dependency from a release or local path.
-
-## Installation
+## Requirements
 
-Install from npm:
-
-```bash
-bun add @dtechvision/fabrik-runtime
-```
+- ESM-capable runtime
+- TypeScript source consumption
+- Bun/Smithers-style workflow execution is the primary target
 
-or:
+Module-specific requirements:
 
-```bash
-npm install @dtechvision/fabrik-runtime
-```
+| Module | Requirement |
+|---|---|
+| `credential-pool` | mounted credential files for file-pool features |
+| `codex-auth` | Codex auth files in the credential pool layout |
+| `jj-shell` | `jj` and `git` in `PATH` |
+| `k8s-jobs` | Kubernetes runtime access |
 
-Smithers workflows also need their normal workflow dependencies in the consuming repo:
+## Install
 
 ```bash
-bun add smithers-orchestrator zod
+bun add @dtechvision/fabrik-runtime smithers-orchestrator zod
 ```
 
-or:
+or
 
 ```bash
-npm install smithers-orchestrator zod
+npm install @dtechvision/fabrik-runtime smithers-orchestrator zod
 ```
 
-Package releases follow the same `v*` tag version as the Fabrik CLI release flow.
-
-## Smithers Integration
-
-Use the package from ordinary Smithers workflows:
+## Quickstart
 
 ```ts
 /** @jsxImportSource smithers-orchestrator */
 import { createSmithers, Task, Workflow } from "smithers-orchestrator";
 import { z } from "zod";
 import { withCodexAuthPoolEnv } from "@dtechvision/fabrik-runtime/codex-auth";
-import { prepareWorkspaces } from "@dtechvision/fabrik-runtime/jj-shell";
 
 const { smithers, outputs } = createSmithers(
-  {
-    report: z.object({
-      codexHomeSet: z.boolean(),
-      jjHelpersLoaded: z.boolean(),
-    }),
-  },
+  { report: z.object({ codexHomeSet: z.boolean() }) },
   { dbPath: process.env.SMITHERS_DB_PATH ?? ".smithers/runtime-check.db" },
 );
 
@@ -73,42 +62,30 @@ export default smithers(() => (
     <Task id="verify" output={outputs.report}>
       {async () => {
         const env = withCodexAuthPoolEnv({});
-        return {
-          codexHomeSet: typeof env.CODEX_HOME === "string" && env.CODEX_HOME.length > 0,
-          jjHelpersLoaded: typeof prepareWorkspaces === "function",
-        };
+        return { codexHomeSet: typeof env.CODEX_HOME === "string" && env.CODEX_HOME.length > 0 };
       }}
     </Task>
   </Workflow>
 ));
 ```
 
-Run it locally with Smithers from a repo that has installed:
-
-- `@dtechvision/fabrik-runtime`
-- `smithers-orchestrator`
-- `zod`
-
-Then:
+Run:
 
 ```bash
 bunx smithers run path/to/workflow.tsx --run-id runtime-package-check
 ```
 
-The workflow file should live in the consuming project tree so normal Node/Bun package resolution can find the installed dependencies.
+## Common use
 
-## Credentials
-
-Operators manage `fabrik-credentials` in `fabrik-system` via kubectl. The CLI mirrors it into the run namespace at dispatch time. The secret is directory-mounted (no subPath) at `/etc/fabrik/credentials/` so running pods observe file replacements.
+Read a credential into `process.env`:
 
 ```ts
 import { injectCredentialEnv } from "@dtechvision/fabrik-runtime/credential-pool";
 
-// Reads /etc/fabrik/credentials/ANTHROPIC_API_KEY → process.env.ANTHROPIC_API_KEY
 injectCredentialEnv("ANTHROPIC_API_KEY");
 ```
 
-For file-pool rotation (e.g. multiple Codex auth files):
+Rotate across credential files:
 
 ```ts
 import { CredentialFilePool } from "@dtechvision/fabrik-runtime/credential-pool";
@@ -120,63 +97,49 @@ const pool = new CredentialFilePool({
   activeFilename: "auth.json",
   agent: "codex",
 });
-pool.init();
 
-// On auth failure:
+pool.init();
 const rotated = await pool.handleError(err);
 ```
 
-For Codex-specific rotation, use the higher-level helper:
+Create a Codex agent with auth rotation:
 
 ```ts
-import { createCodexAgentWithPool } from "@dtechvision/fabrik-runtime/codex-auth";
+import {
+  CodexAuthBlockedError,
+  createCodexAgentWithPool,
+} from "@dtechvision/fabrik-runtime/codex-auth";
 
 const codex = createCodexAgentWithPool({
   model: "gpt-5",
   cwd: process.cwd(),
   env: {},
 });
-```
 
-## Local Verification
-
-Runtime package tests:
-
-```bash
-cd src/fabrik-runtime
-bun test ./src
-```
-
-Repo-wide CLI and workflow verification:
-
-```bash
-make verify-cli
-make verify-cli-k3d
-```
-
-Focused runtime-package k3d import verification:
-
-```bash
-cd src/fabrik-cli
-FABRIK_K3D_E2E=1 FABRIK_K3D_CLUSTER=dev-single \
-  go test ./internal/run -run TestK3dWorkflowRuntimePackageImports -timeout 10m -v
+try {
+  await codex.generate({ prompt: "Hello" });
+} catch (err) {
+  if (err instanceof CodexAuthBlockedError) {
+    // resumable auth exhaustion; restore credentials and resume the run
+    console.log(err.details); // { total, failed, remaining, activeAuthName, failedAuths }
+  }
+  throw err;
+}
 ```
 
-The complex sample in [examples/complex/README.md](/Users/samuel/git/local-isolated-ralph/examples/complex/README.md) shows how workflow code consumes the package surface in practice.
+Read the auth home directory at runtime (lazy, respects `CODEX_AUTH_HOME` env var):
 
-Local Smithers CLI verification:
+```ts
+import { getCodexAuthHome } from "@dtechvision/fabrik-runtime/codex-auth";
 
-```bash
-bunx smithers run path/to/workflow.tsx --run-id runtime-package-check
+const home = getCodexAuthHome(); // e.g. /tmp/codex-auth-pool
 ```
 
-The expected result is a successful run whose output reports:
-
-- `codexHomeSet: true`
-- `jjHelpersLoaded: true`
-
-## Precedence
+## Notes
 
-1. Fabrik runtime metadata (`SMITHERS_*`, `FABRIK_*`, `KUBERNETES_*`)
-2. Project env (`fabrik-env-<project>-<env>`) via `envFrom`
-3. Shared credentials (`fabrik-credentials`) via file mount
+- Each `RotatingCodexAgent` instance owns its own pool state. Multiple agents in the same process do not interfere with each other.
+- Auth failures are tracked by file path + content hash, so replacing a credential file on disk clears its failure history.
+- Codex auth rotation emits OTEL metrics/events when an OpenTelemetry SDK is configured (events are attached to the active span, not standalone).
+- Fabrik runtime images may already ship this package.
+- Package versions follow the same `v*` tag line as Fabrik releases.
+- Prefer aligned Fabrik image and package versions when both are in use.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@dtechvision/fabrik-runtime",
-  "version": "0.1.0",
+  "version": "0.1.1",
   "type": "module",
   "main": "src/index.ts",
   "types": "src/index.ts",
@@ -15,6 +15,7 @@
     "src/index.ts",
     "src/credential-pool.ts",
     "src/codex-auth.ts",
+    "src/codex-auth-telemetry.ts",
     "src/k8s-jobs.ts",
     "src/jj-shell.ts",
     "README.md"
@@ -32,6 +33,7 @@
     "test": "bun test ./src"
   },
   "dependencies": {
+    "@opentelemetry/api": "1.9.0",
     "smithers-orchestrator": "0.9.1"
   },
   "devDependencies": {

package/src/codex-auth-telemetry.ts

@@ -0,0 +1,153 @@
+import * as otel from "@opentelemetry/api";
+import type { FailureKind } from "./credential-pool";
+
+export type CodexAuthBlockedDetails = {
+  total: number;
+  failed: number;
+  remaining: number;
+  activeAuthName: string | null;
+  failedAuths: Array<{ authName: string; kind: FailureKind }>;
+};
+
+export type CodexAuthTelemetryContext = {
+  runId?: string;
+  namespace?: string;
+};
+
+type CodexAuthTelemetrySink = {
+  counter(name: string, value: number, attrs?: Record<string, unknown>): void;
+  gauge(name: string, value: number, attrs?: Record<string, unknown>): void;
+  event(name: string, attrs?: Record<string, unknown>): void;
+};
+
+const createOtelSink = (): CodexAuthTelemetrySink => {
+  const meter = otel.metrics.getMeter("@dtechvision/fabrik-runtime/codex-auth");
+
+  const counters = new Map<string, ReturnType<typeof meter.createCounter>>();
+  const gauges = new Map<string, ReturnType<typeof meter.createGauge>>();
+
+  const getCounter = (name: string) => {
+    let counter = counters.get(name);
+    if (!counter) {
+      counter = meter.createCounter(name);
+      counters.set(name, counter);
+    }
+    return counter;
+  };
+
+  const getGauge = (name: string) => {
+    let gauge = gauges.get(name);
+    if (!gauge) {
+      gauge = meter.createGauge(name);
+      gauges.set(name, gauge);
+    }
+    return gauge;
+  };
+
+  return {
+    counter(name, value, attrs) {
+      getCounter(name).add(value, attrs);
+    },
+    gauge(name, value, attrs) {
+      getGauge(name).record(value, attrs);
+    },
+    event(name, attrs) {
+      const span = otel.trace.getActiveSpan();
+      if (span) {
+        span.addEvent(name, attrs);
+      }
+    },
+  };
+};
+
+let sink: CodexAuthTelemetrySink = createOtelSink();
+
+export function __setCodexAuthTelemetrySinkForTests(next: CodexAuthTelemetrySink): void {
+  sink = next;
+}
+
+export function __resetCodexAuthTelemetrySinkForTests(): void {
+  sink = createOtelSink();
+}
+
+const baseAttrs = (ctx: CodexAuthTelemetryContext) => ({
+  ...(ctx.runId ? { run_id: ctx.runId } : {}),
+  ...(ctx.namespace ? { kubernetes_namespace: ctx.namespace } : {}),
+});
+
+const failureKindCounts = (details: CodexAuthBlockedDetails) => ({
+  usage_limit: details.failedAuths.filter((entry) => entry.kind === "usage_limit").length,
+  refresh_token_reused: details.failedAuths.filter((entry) => entry.kind === "refresh_token_reused").length,
+  auth_invalid: details.failedAuths.filter((entry) => entry.kind === "auth_invalid").length,
+});
+
+export function recordCodexAuthPoolSnapshot(
+  details: CodexAuthBlockedDetails,
+  ctx: CodexAuthTelemetryContext,
+): void {
+  const attrs = baseAttrs(ctx);
+  const counts = failureKindCounts(details);
+  sink.gauge("fabrik.codex_auth.pool.total", details.total, attrs);
+  sink.gauge("fabrik.codex_auth.pool.failed", details.failed, attrs);
+  sink.gauge("fabrik.codex_auth.pool.remaining", details.remaining, attrs);
+  sink.gauge("fabrik.codex_auth.pool.failed_usage_limit", counts.usage_limit, attrs);
+  sink.gauge(
+    "fabrik.codex_auth.pool.failed_refresh_token_reused",
+    counts.refresh_token_reused,
+    attrs,
+  );
+  sink.gauge("fabrik.codex_auth.pool.failed_auth_invalid", counts.auth_invalid, attrs);
+}
+
+export function recordCodexAuthFailure(
+  failure: { authName: string; kind: FailureKind },
+  details: CodexAuthBlockedDetails,
+  ctx: CodexAuthTelemetryContext,
+): void {
+  const attrs = {
+    ...baseAttrs(ctx),
+    auth_name: failure.authName,
+    failure_kind: failure.kind,
+  };
+  sink.counter("fabrik.codex_auth.failure_total", 1, attrs);
+  sink.event("codex.auth.failure", {
+    ...attrs,
+    total: details.total,
+    failed: details.failed,
+    remaining: details.remaining,
+  });
+}
+
+export function recordCodexAuthRotation(
+  rotation: { fromAuthName?: string; toAuthName: string; reason: string },
+  details: CodexAuthBlockedDetails,
+  ctx: CodexAuthTelemetryContext,
+): void {
+  const attrs = {
+    ...baseAttrs(ctx),
+    ...(rotation.fromAuthName ? { from_auth_name: rotation.fromAuthName } : {}),
+    to_auth_name: rotation.toAuthName,
+    reason: rotation.reason,
+  };
+  sink.counter("fabrik.codex_auth.rotation_total", 1, attrs);
+  sink.event("codex.auth.rotation", {
+    ...attrs,
+    total: details.total,
+    failed: details.failed,
+    remaining: details.remaining,
+  });
+}
+
+export function recordCodexAuthExhausted(
+  details: CodexAuthBlockedDetails,
+  ctx: CodexAuthTelemetryContext,
+): void {
+  const attrs = baseAttrs(ctx);
+  sink.counter("fabrik.codex_auth.exhausted_total", 1, attrs);
+  sink.event("codex.auth.exhausted", {
+    ...attrs,
+    total: details.total,
+    failed: details.failed,
+    remaining: details.remaining,
+  });
+}

package/src/codex-auth.ts

@@ -7,9 +7,23 @@ import {
   getCredentialMountPath,
   type FailureKind,
 } from "./credential-pool";
+import {
+  recordCodexAuthExhausted,
+  recordCodexAuthFailure,
+  recordCodexAuthPoolSnapshot,
+  recordCodexAuthRotation,
+  type CodexAuthBlockedDetails,
+} from "./codex-auth-telemetry";
+
+export type { CodexAuthBlockedDetails } from "./codex-auth-telemetry";
 
 const DEFAULT_CODEX_DIR = resolve(process.env.HOME ?? "", ".codex");
 
+// Resolution order:
+// 1. CODEX_AUTH_SOURCE_DIR — test/dev override
+// 2. FABRIK_SHARED_CREDENTIALS_DIR — production (set by fabrik-cli dispatch)
+// 3. credential mount path — fallback if directory exists
+// 4. ~/.codex — local dev default
 function getCodexAuthSourceDir(): string {
   const sourceDir =
     process.env.CODEX_AUTH_SOURCE_DIR ??
@@ -18,18 +32,20 @@ function getCodexAuthSourceDir(): string {
   return resolve(sourceDir);
 }
 
-export const CODEX_AUTH_HOME = resolve(
-  process.env.CODEX_AUTH_HOME ?? resolve(tmpdir(), "codex-auth-pool"),
-);
+export function getCodexAuthHome(): string {
+  return resolve(process.env.CODEX_AUTH_HOME ?? resolve(tmpdir(), "codex-auth-pool"));
+}
+
 
-const NOTIFY_WEBHOOK_URL = process.env.CODEX_AUTH_NOTIFY_WEBHOOK_URL?.trim() ?? "";
-const NOTIFY_CLUSTER = process.env.KUBERNETES_NAMESPACE?.trim() ?? "";
-const NOTIFY_RUN_ID = process.env.SMITHERS_RUN_ID?.trim() ?? "";
+const getNotifyWebhookUrl = () => process.env.CODEX_AUTH_NOTIFY_WEBHOOK_URL?.trim() ?? "";
+const getNotifyCluster = () => process.env.KUBERNETES_NAMESPACE?.trim() ?? "";
+const getNotifyRunID = () => process.env.SMITHERS_RUN_ID?.trim() ?? "";
 
-const AUTH_ROTATE_PATTERN =
-  /no last agent message|usage limit|quota|rate limit|insufficient (?:credits|balance|quota)|payment required|billing|exceeded.*(quota|limit)|not signed in|please run 'codex login'|unauthorized|authentication required|authentication failed|forbidden|invalid (?:api key|token|credentials)|expired (?:token|credentials)/i;
-const AUTH_REFRESH_REUSED_PATTERN =
-  /refresh_token_reused|refresh token has already been used|could not be refreshed because your refresh token was already used/i;
+type AuthEntry = {
+  path: string;
+  authName: string;
+  contents: string;
+};
 
 const listAuthFiles = (): string[] => {
   const sourceDir = getCodexAuthSourceDir();
@@ -40,83 +56,63 @@ const listAuthFiles = (): string[] => {
     .sort();
 };
 
-const ensureCodexHome = () => {
-  if (!existsSync(CODEX_AUTH_HOME)) {
-    mkdirSync(CODEX_AUTH_HOME, { recursive: true });
+const ensureDir = (dir: string) => {
+  if (!existsSync(dir)) {
+    mkdirSync(dir, { recursive: true });
   }
 };
 
-let authPool = listAuthFiles();
-let authIndex = 0;
-let activeAuth = "";
-const authFailures = new Map<string, FailureKind>();
+export class CodexAuthBlockedError extends Error {
+  readonly code = "CODEX_AUTH_BLOCKED" as const;
+  readonly reason = "auth_pool_exhausted" as const;
+  readonly details: CodexAuthBlockedDetails;
+  readonly runId?: string;
+  readonly namespace?: string;
 
-export function resetCodexAuthStateForTests(): void {
-  authPool = [];
-  authIndex = 0;
-  activeAuth = "";
-  authFailures.clear();
+  constructor(args: {
+    message?: string;
+    details: CodexAuthBlockedDetails;
+    runId?: string;
+    namespace?: string;
+    cause?: unknown;
+  }) {
+    super(args.message ?? "Codex auth pool exhausted", { cause: args.cause });
+    this.name = "CodexAuthBlockedError";
+    this.details = args.details;
+    this.runId = args.runId;
+    this.namespace = args.namespace;
+  }
 }
 
-const setActiveAuth = (authPath: string, reason: string) => {
-  ensureCodexHome();
-  const authContents = readFileSync(authPath, "utf8");
-  writeFileSync(resolve(CODEX_AUTH_HOME, "auth.json"), authContents, "utf8");
-  const previous = activeAuth ? ` from ${basename(activeAuth)}` : "";
-  activeAuth = authPath;
-  console.error(
-    `[fabrik-runtime] codex auth rotation${previous} -> ${basename(authPath)} (${reason})`,
-  );
-};
-
-const initAuthPool = () => {
-  ensureCodexHome();
-  authPool = listAuthFiles();
-  if (authPool.length === 0 || activeAuth) return;
-  const defaultAuth = resolve(getCodexAuthSourceDir(), "auth.json");
-  if (existsSync(defaultAuth)) {
-    setActiveAuth(defaultAuth, "initial");
-    return;
-  }
-  setActiveAuth(authPool[0]!, "initial");
-};
+const telemetryContext = () => ({
+  runId: getNotifyRunID() || undefined,
+  namespace: getNotifyCluster() || undefined,
+});
 
-const logAuthSummary = () => {
-  const total = authPool.length;
-  const failed = [...authFailures.entries()].map(
-    ([path, status]) => `${basename(path)}:${status}`,
-  );
-  const failedCount = authFailures.size;
-  const remaining = Math.max(total - failedCount, 0);
-  const active = activeAuth ? basename(activeAuth) : "none";
-  console.error(
-    `[fabrik-runtime] codex auth pool summary: total=${total} failed=${failedCount} remaining=${remaining} active=${active}`,
+const writeBlockerArtifact = (details: CodexAuthBlockedDetails) => {
+  const smithersHome = resolve(process.env.SMITHERS_HOME ?? ".");
+  const blockerPath = resolve(smithersHome, ".smithers", "blockers", "codex-auth.json");
+  mkdirSync(resolve(blockerPath, ".."), { recursive: true });
+  writeFileSync(
+    blockerPath,
+    JSON.stringify(
+      {
+        kind: "auth_pool_exhausted",
+        resumable: true,
+        runId: getNotifyRunID() || undefined,
+        namespace: getNotifyCluster() || undefined,
+        details,
+      },
+      null,
+      2,
+    ),
+    "utf8",
   );
-  if (failed.length > 0) {
-    console.error(`[fabrik-runtime] failed auths: ${failed.join(", ")}`);
-  }
-};
-
-const rotateAuth = (reason: string): boolean => {
-  authPool = listAuthFiles();
-  if (authPool.length === 0) return false;
-  for (let i = 0; i < authPool.length; i += 1) {
-    const next = authPool[authIndex % authPool.length];
-    authIndex += 1;
-    if (next && next !== activeAuth && !authFailures.has(next)) {
-      setActiveAuth(next, reason);
-      logAuthSummary();
-      return true;
-    }
-  }
-  console.error("[fabrik-runtime] no codex auth left to rotate to");
-  logAuthSummary();
-  return false;
 };
 
 export const withCodexAuthPoolEnv = (env: Record<string, string>) => ({
   ...env,
-  CODEX_HOME: CODEX_AUTH_HOME,
+  CODEX_HOME: getCodexAuthHome(),
 });
 
 export type AuthFailureKind = FailureKind;
@@ -142,9 +138,10 @@ const notifyAuthFailure = async (
   if (onAuthFailure) {
     await onAuthFailure(event);
   }
-  if (!NOTIFY_WEBHOOK_URL) return;
+  const webhookUrl = getNotifyWebhookUrl();
+  if (!webhookUrl) return;
   try {
-    const response = await fetch(NOTIFY_WEBHOOK_URL, {
+    const response = await fetch(webhookUrl, {
       method: "POST",
       headers: { "content-type": "application/json" },
       body: JSON.stringify(event),
@@ -175,6 +172,10 @@ export const createCodexAgentWithPool = (
 export class RotatingCodexAgent {
   private readonly inner: CodexAgent;
   private readonly onAuthFailure?: RotatingCodexAgentOptions["onAuthFailure"];
+  private authPool: AuthEntry[] = [];
+  private authIndex = 0;
+  private activeAuth: AuthEntry | null = null;
+  private readonly authFailures = new Map<string, FailureKind>();
 
   constructor(inner: CodexAgent, opts: RotatingCodexAgentOptions = {}) {
     this.inner = inner;
@@ -189,9 +190,136 @@ export class RotatingCodexAgent {
     return this.inner.tools;
   }
 
+  private failureKey(entry: AuthEntry): string {
+    return `${entry.path}:${entry.contents.length}:${Bun.hash(entry.contents)}`;
+  }
+
+  private scanAuthPool(): AuthEntry[] {
+    return listAuthFiles().map((path) => ({
+      path,
+      authName: basename(path),
+      contents: readFileSync(path, "utf8"),
+    }));
+  }
+
+  private refreshAuthPool(): void {
+    this.authPool = this.scanAuthPool();
+    if (!this.activeAuth) return;
+    const nextActive = this.authPool.find((entry) => entry.path === this.activeAuth?.path) ?? null;
+    this.activeAuth = nextActive;
+  }
+
+  private getBlockedDetails(): CodexAuthBlockedDetails {
+    this.refreshAuthPool();
+    const failedAuths = this.authPool
+      .filter((entry) => this.authFailures.has(this.failureKey(entry)))
+      .map((entry) => ({
+        authName: entry.authName,
+        kind: this.authFailures.get(this.failureKey(entry))!,
+      }));
+    return {
+      total: this.authPool.length,
+      failed: failedAuths.length,
+      remaining: Math.max(this.authPool.length - failedAuths.length, 0),
+      activeAuthName: this.activeAuth?.authName ?? null,
+      failedAuths,
+    };
+  }
+
+  private logAuthSummary(): void {
+    const details = this.getBlockedDetails();
+    recordCodexAuthPoolSnapshot(details, telemetryContext());
+    const failed = details.failedAuths.map(({ authName, kind }) => `${authName}:${kind}`);
+    const active = details.activeAuthName ?? "none";
+    console.error(
+      `[fabrik-runtime] codex auth pool summary: total=${details.total} failed=${details.failed} remaining=${details.remaining} active=${active}`,
+    );
+    if (failed.length > 0) {
+      console.error(`[fabrik-runtime] failed auths: ${failed.join(", ")}`);
+    }
+  }
+
+  /** Write the active auth file to the codex auth home dir without emitting telemetry or logs. */
+  private syncActiveAuthFile(entry: AuthEntry): void {
+    const home = getCodexAuthHome();
+    ensureDir(home);
+    writeFileSync(resolve(home, "auth.json"), entry.contents, "utf8");
+  }
+
+  /** Activate a credential: write file, update state, emit telemetry + log. */
+  private setActiveAuth(entry: AuthEntry, reason: string): void {
+    this.syncActiveAuthFile(entry);
+    const previousAuth = this.activeAuth?.authName;
+    const previous = previousAuth ? ` from ${previousAuth}` : "";
+    this.activeAuth = entry;
+    recordCodexAuthRotation(
+      {
+        fromAuthName: previousAuth,
+        toAuthName: entry.authName,
+        reason,
+      },
+      this.getBlockedDetails(),
+      telemetryContext(),
+    );
+    console.error(
+      `[fabrik-runtime] codex auth rotation${previous} -> ${entry.authName} (${reason})`,
+    );
+  }
+
+  private ensureActiveAuth(): void {
+    const home = getCodexAuthHome();
+    ensureDir(home);
+    this.refreshAuthPool();
+    if (this.authPool.length === 0) return;
+    const currentFailed =
+      this.activeAuth && this.authFailures.has(this.failureKey(this.activeAuth));
+    if (this.activeAuth && !currentFailed) {
+      // Re-sync file if contents changed on disk (operator rotated credentials),
+      // or if the file doesn't exist yet. No telemetry — this is not a rotation.
+      const authFile = resolve(home, "auth.json");
+      const onDisk = existsSync(authFile) ? readFileSync(authFile, "utf8") : null;
+      if (onDisk !== this.activeAuth.contents) {
+        this.syncActiveAuthFile(this.activeAuth);
+      }
+      return;
+    }
+    const defaultAuth = resolve(getCodexAuthSourceDir(), "auth.json");
+    const initial =
+      this.authPool.find((entry) => entry.path === defaultAuth && !this.authFailures.has(this.failureKey(entry))) ??
+      this.authPool.find((entry) => !this.authFailures.has(this.failureKey(entry)));
+    if (!initial) {
+      this.activeAuth = this.authPool.find((entry) => entry.path === this.activeAuth?.path) ?? null;
+      return;
+    }
+    const reason = this.activeAuth ? "refresh" : "initial";
+    this.setActiveAuth(initial, reason);
+    this.authIndex = this.authPool.findIndex((entry) => entry.path === initial.path) + 1;
+  }
+
+  private rotateAuth(reason: string): boolean {
+    this.refreshAuthPool();
+    if (this.authPool.length === 0) return false;
+    for (let i = 0; i < this.authPool.length; i += 1) {
+      const next = this.authPool[this.authIndex % this.authPool.length];
+      this.authIndex += 1;
+      if (
+        next &&
+        next.path !== this.activeAuth?.path &&
+        !this.authFailures.has(this.failureKey(next))
+      ) {
+        this.setActiveAuth(next, reason);
+        this.logAuthSummary();
+        return true;
+      }
+    }
+    console.error("[fabrik-runtime] no codex auth left to rotate to");
+    this.logAuthSummary();
+    return false;
+  }
+
   async generate(args: Parameters<CodexAgent["generate"]>[0]) {
-    initAuthPool();
-    const attempts = Math.max(authPool.length, 1);
+    this.ensureActiveAuth();
+    const attempts = Math.max(this.authPool.length, 1);
     let lastError: unknown = null;
     for (let i = 0; i < attempts; i += 1) {
       try {
@@ -199,33 +327,56 @@ export class RotatingCodexAgent {
       } catch (err) {
         lastError = err;
         const message = err instanceof Error ? err.message : String(err);
-        if (!AUTH_ROTATE_PATTERN.test(message)) {
+        const kind = classifyFailure(message);
+        if (kind === "unknown") {
           throw err;
         }
-        if (activeAuth) {
-          const kind = classifyFailure(message);
-          authFailures.set(activeAuth, kind);
-          if (AUTH_REFRESH_REUSED_PATTERN.test(message)) {
+        if (this.activeAuth) {
+          this.authFailures.set(this.failureKey(this.activeAuth), kind);
+          recordCodexAuthFailure(
+            { authName: this.activeAuth.authName, kind },
+            this.getBlockedDetails(),
+            telemetryContext(),
+          );
+          if (kind === "refresh_token_reused") {
             console.error("[fabrik-runtime] codex refresh token reused; re-auth required");
           }
           await notifyAuthFailure(
             {
-              authPath: activeAuth,
-              authName: basename(activeAuth),
+              authPath: this.activeAuth.path,
+              authName: this.activeAuth.authName,
               reason: "codex generate failed and rotation was requested",
               kind,
               message,
-              clusterNamespace: NOTIFY_CLUSTER || undefined,
-              runId: NOTIFY_RUN_ID || undefined,
+              clusterNamespace: getNotifyCluster() || undefined,
+              runId: getNotifyRunID() || undefined,
             },
             this.onAuthFailure,
           );
         }
-        if (!rotateAuth("codex auth / usage failure")) {
-          break;
+        if (!this.rotateAuth("codex auth / usage failure")) {
+          const details = this.getBlockedDetails();
+          recordCodexAuthExhausted(details, telemetryContext());
+          writeBlockerArtifact(details);
+          throw new CodexAuthBlockedError({
+            message: "Codex auth pool exhausted",
+            details,
+            runId: getNotifyRunID() || undefined,
+            namespace: getNotifyCluster() || undefined,
+            cause: err,
+          });
         }
       }
     }
-    throw lastError ?? new Error("Codex auth pool exhausted");
+    const details = this.getBlockedDetails();
+    recordCodexAuthExhausted(details, telemetryContext());
+    writeBlockerArtifact(details);
+    throw new CodexAuthBlockedError({
+      message: "Codex auth pool exhausted",
+      details,
+      runId: getNotifyRunID() || undefined,
+      namespace: getNotifyCluster() || undefined,
+      cause: lastError,
+    });
   }
 }

package/src/index.ts

@@ -25,13 +25,15 @@ export {
 } from "./credential-pool";
 
 export {
-  CODEX_AUTH_HOME,
+  getCodexAuthHome,
   withCodexAuthPoolEnv,
   createCodexAgentWithPool,
   RotatingCodexAgent,
+  CodexAuthBlockedError,
   type AuthFailureKind,
   type AuthFailureEvent,
   type RotatingCodexAgentOptions,
+  type CodexAuthBlockedDetails,
 } from "./codex-auth";
 
 export {

package/src/jj-shell.ts

@@ -22,6 +22,12 @@ type JjResult = {
   exitCode: number;
 };
 
+const PUSH_RETRY_LIMIT = 3;
+const STALE_REF_PATTERNS = [
+  "unexpectedly moved on the remote",
+  "reason: stale info",
+];
+
 async function jj(args: string[], cwd: string): Promise<JjResult> {
   const result = await $`jj ${args}`.cwd(cwd).nothrow().quiet();
   return {
@@ -32,6 +38,47 @@ async function jj(args: string[], cwd: string): Promise<JjResult> {
   };
 }
 
+function isStaleRefPushFailure(result: JjResult): boolean {
+  const text = `${result.stdout}\n${result.stderr}`;
+  return STALE_REF_PATTERNS.some((pattern) => text.includes(pattern));
+}
+
+function summarizeJjResult(label: string, result: JjResult): string {
+  const details = [result.stdout, result.stderr].filter(Boolean).join(" | ");
+  return `${label} exit=${result.exitCode}${details ? ` ${details}` : ""}`;
+}
+
+async function hasConflicts(workspacePath: string): Promise<boolean> {
+  const conflicts = await jj(["log", "-r", "conflicts()", "--no-graph", "-T", "commit_id"], workspacePath);
+  if (!conflicts.ok) {
+    return true;
+  }
+  return conflicts.stdout.trim().length > 0;
+}
+
+async function trackRemoteBookmark(
+  workspacePath: string,
+  bookmarkName: string,
+): Promise<JjResult> {
+  return jj(["bookmark", "track", `glob:${bookmarkName}`, "--remote", "origin"], workspacePath);
+}
+
+async function setBookmarkToTarget(
+  workspacePath: string,
+  bookmarkName: string,
+  targetRev: string,
+): Promise<JjResult> {
+  const move = await jj(
+    ["bookmark", "set", bookmarkName, "-r", targetRev, "--allow-backwards"],
+    workspacePath,
+  );
+  if (move.ok) {
+    return move;
+  }
+
+  return jj(["bookmark", "create", "-r", targetRev, bookmarkName], workspacePath);
+}
+
 export async function prepareWorkspaces(
   repoRoot: string,
   workspacesDir: string,
@@ -137,76 +184,131 @@ export async function pushBookmark(
   ticketId: string,
 ): Promise<ReportOutput> {
   const targetRev = "@-";
-  const track = await jj(
-    ["bookmark", "track", bookmarkName, "--remote", "origin"],
-    workspacePath,
-  );
+  const track = await trackRemoteBookmark(workspacePath, bookmarkName);
   const trackSummary =
     track.ok || track.stderr === ""
       ? ""
       : ` Tracking remote bookmark reported: ${track.stderr}`;
 
-  const targetCommit = await jj(
-    ["log", "-r", targetRev, "--no-graph", "-T", "commit_id"],
-    workspacePath,
-  );
-  if (!targetCommit.ok || !targetCommit.stdout) {
-    return {
-      ticketId,
-      status: "blocked",
-      summary: `Failed to resolve target revision for bookmark push: ${targetCommit.stderr}`,
-    };
-  }
+  let lastAttemptSummary = "";
 
-  const move = await jj(
-    ["bookmark", "set", bookmarkName, "-r", targetRev, "--allow-backwards"],
-    workspacePath,
-  );
+  for (let attempt = 1; attempt <= PUSH_RETRY_LIMIT; attempt += 1) {
+    if (attempt > 1) {
+      const fetch = await jj(["git", "fetch"], workspacePath);
+      if (!fetch.ok) {
+        return {
+          ticketId,
+          status: "blocked",
+          summary:
+            `Bookmark push retry ${attempt}/${PUSH_RETRY_LIMIT} failed during fetch: ${fetch.stderr || fetch.stdout}.` +
+            trackSummary +
+            (lastAttemptSummary ? ` Last push state: ${lastAttemptSummary}` : ""),
+        };
+      }
+
+      const retryTrack = await trackRemoteBookmark(workspacePath, bookmarkName);
+      if (!retryTrack.ok && retryTrack.stderr !== "") {
+        lastAttemptSummary = `${lastAttemptSummary} ${summarizeJjResult("track", retryTrack)}`.trim();
+      }
 
-  if (!move.ok) {
-    const create = await jj(
-      ["bookmark", "create", "-r", targetRev, bookmarkName],
+      const rebase = await jj(
+        ["rebase", "-s", `roots(${bookmarkName}@origin..${targetRev})`, "-d", `${bookmarkName}@origin`],
+        workspacePath,
+      );
+      if (!rebase.ok) {
+        return {
+          ticketId,
+          status: "blocked",
+          summary:
+            `Bookmark push retry ${attempt}/${PUSH_RETRY_LIMIT} failed during rebase: ${rebase.stderr || rebase.stdout}.` +
+            trackSummary +
+            (lastAttemptSummary ? ` Last push state: ${lastAttemptSummary}` : ""),
+        };
+      }
+
+      if (await hasConflicts(workspacePath)) {
+        return {
+          ticketId,
+          status: "blocked",
+          summary:
+            `Bookmark push retry ${attempt}/${PUSH_RETRY_LIMIT} stopped after rebase conflict on '${bookmarkName}'.` +
+            ` ${summarizeJjResult("rebase", rebase)}` +
+            trackSummary,
+        };
+      }
+    }
+
+    const targetCommit = await jj(
+      ["log", "-r", targetRev, "--no-graph", "-T", "commit_id"],
       workspacePath,
     );
-    if (!create.ok) {
+    if (!targetCommit.ok || !targetCommit.stdout) {
       return {
         ticketId,
         status: "blocked",
-        summary: `Failed to set bookmark '${bookmarkName}': ${create.stderr}`,
+        summary: `Failed to resolve target revision for bookmark push: ${targetCommit.stderr}`,
       };
     }
-  }
 
-  const push = await jj(
-    ["git", "push", "--bookmark", bookmarkName],
-    workspacePath,
-  );
-  if (!push.ok) {
-    return {
-      ticketId,
-      status: "blocked",
-      summary: `Bookmark set but push failed: ${push.stderr}${trackSummary}`,
-    };
-  }
+    const move = await setBookmarkToTarget(workspacePath, bookmarkName, targetRev);
+    if (!move.ok) {
+      return {
+        ticketId,
+        status: "blocked",
+        summary: `Failed to set bookmark '${bookmarkName}': ${move.stderr}`,
+      };
+    }
+
+    const push = await jj(
+      ["git", "push", "--bookmark", bookmarkName],
+      workspacePath,
+    );
+    if (!push.ok) {
+      lastAttemptSummary = [
+        `attempt ${attempt}/${PUSH_RETRY_LIMIT}`,
+        summarizeJjResult("push", push),
+      ].join(" ");
+
+      if (isStaleRefPushFailure(push) && attempt < PUSH_RETRY_LIMIT) {
+        continue;
+      }
+
+      const exhausted = isStaleRefPushFailure(push) && attempt === PUSH_RETRY_LIMIT;
+      return {
+        ticketId,
+        status: "blocked",
+        summary:
+          `${exhausted ? `Bookmark push retries exhausted for '${bookmarkName}'.` : "Bookmark set but push failed:"} ${push.stderr || push.stdout}` +
+          trackSummary +
+          (lastAttemptSummary ? ` Last attempt: ${lastAttemptSummary}` : ""),
+      };
+    }
+
+    const remote = await $`git ls-remote origin refs/heads/${bookmarkName}`
+      .cwd(workspacePath)
+      .nothrow()
+      .quiet();
+    const remoteCommit = remote.stdout.toString().trim().split(/\s+/)[0] ?? "";
+    if (remote.exitCode !== 0 || remoteCommit !== targetCommit.stdout) {
+      return {
+        ticketId,
+        status: "blocked",
+        summary:
+          `Bookmark push returned success but remote ${bookmarkName} is ${remoteCommit || "missing"} instead of ${targetCommit.stdout}.` +
+          trackSummary,
+      };
+    }
 
-  const remote = await $`git ls-remote origin refs/heads/${bookmarkName}`
-    .cwd(workspacePath)
-    .nothrow()
-    .quiet();
-  const remoteCommit = remote.stdout.toString().trim().split(/\s+/)[0] ?? "";
-  if (remote.exitCode !== 0 || remoteCommit !== targetCommit.stdout) {
     return {
       ticketId,
-      status: "blocked",
-      summary:
-        `Bookmark push returned success but remote ${bookmarkName} is ${remoteCommit || "missing"} instead of ${targetCommit.stdout}.` +
-        trackSummary,
+      status: "done",
+      summary: `Pushed bookmark '${bookmarkName}' to origin at ${targetCommit.stdout}.${trackSummary}`,
     };
   }
 
   return {
     ticketId,
-    status: "done",
-    summary: `Pushed bookmark '${bookmarkName}' to origin at ${targetCommit.stdout}.${trackSummary}`,
+    status: "blocked",
+    summary: `Bookmark push retries exhausted for '${bookmarkName}'.${trackSummary}${lastAttemptSummary ? ` Last attempt: ${lastAttemptSummary}` : ""}`,
   };
 }