agents-relay 1.0.94 → 1.0.96

package/README.md

@@ -70,7 +70,7 @@ Planner-created tasks use stable IDs and the existing parent/subtask tree, so
 retries and daemon restarts do not create duplicate work.
 Task producers must follow `skills/agents-relay/contracts/task-job-contract.md`; that shared skill contract defines story size, lifecycle events, review, and auto-merge semantics.
 
-Worker/model routing is code-driven: Relay gathers authoritative worker/model/resource facts, applies hard gates, then can call the generic typed `decide(request, contract)` boundary for judgment among legal candidates. The optional Laya implementation is a transient CLI subprocess per decision (no resident model/service), with deterministic fallback and `off|shadow|active` rollout. See [Worker Router, Model Router, and typed decisions](docs/worker-model-routers.md).
+Worker/model routing is code-driven: Relay gathers authoritative worker/model/resource facts, applies hard gates, then can call the generic typed `decide(request, contract)` boundary for judgment among legal candidates. The optional Laya implementation is a transient CLI subprocess per decision (no resident model/service), with active Laya decisions and deterministic fallback on decision failure. See [Worker Router, Model Router, and typed decisions](docs/worker-model-routers.md).
 All task producers use the same story-size contract: `S`≈5 minutes, `M`≈15 minutes, and `L`≈45 minutes of normal-reasoning execution, with `M` as the default. `high` reasoning doubles that overall window. ChatGPT Browser Worker start-ack timeout is derived as one third of the effective overall timeout, clamped between 1 and 5 minutes. Orchestrators and planners should choose `storySize`, not raw timeout milliseconds; `--timeout` remains only as an explicit compatibility/debug override.
 
 Autonomous jobs can also be scheduled without a Relay database. The job marker
@@ -187,3 +187,9 @@ GitHub PR state is authoritative for terminal lifecycle: a merged PR reconciles
 `adapter: orchestrator` is reserved for synchronous self-work by the exact orchestrator turn that creates the Task. Relay deliberately does not auto-start these Tasks during reconciliation. The creator must perform the work before ending the turn and terminalize the same Task with `npx agents-relay task complete --task-id <id> --summary <summary> ...`. If work should continue asynchronously or in a later turn, use a real worker adapter instead.
 
 For work that was completed outside the relay but must be represented truthfully in durable history, use `npx agents-relay record --task-id <id> --summary <summary> --commit <sha>`. Recorded tasks use the `orchestrator` adapter and terminal `SUCCEEDED` state; they do not pretend a shell/model worker executed the work. This is a repair/backfill mechanism—normal work should be created as a durable Task before execution. For managed jobs, the GitHub PR body is the canonical Job surface: hidden Job metadata marker plus the visible Markdown objective.
+
+### Automatic task troubleshooting
+
+When an ACTIVE Job contains a current-run `FAILED` or `BLOCKED` task, reconciliation creates one P0-priority `troubleshooter` recovery task for that exact failed attempt. The recovery task is routed to a local Codex-capable worker with repository/tool access and runs under the bundled Troubleshooter agent contract. Troubleshooter diagnosis is event-first: it must inspect the failed task's task-scoped lifecycle event history (including ordering, duplicate/missing/late events, source and phase data) before classifying root cause or retrying.
+
+The Troubleshooter diagnoses before retrying. A task-local or transient blocker is repaired narrowly; a shared worker/runtime/tool defect is repaired through the owning repository's P0/self-job workflow. Only after the blocker is removed does it retry the original task and verify durable progress. Historical failures from older scheduled autonomous runs do not spawn recovery work, and deterministic reconciliation does not duplicate a Troubleshooter task for the same failed attempt.

package/dist/decision.js

@@ -84,7 +84,7 @@ export class LayaCliDecisionEngine {
     timeoutMs;
     env;
     constructor(options = {}) {
-        this.command = options.command ?? process.env.AGENTS_RELAY_LAYA_COMMAND ?? 'uvx';
+        this.command = options.command ?? 'uvx';
         this.args = options.args ?? layaArgsFromEnv();
         this.timeoutMs = options.timeoutMs ?? Number(process.env.AGENTS_RELAY_LAYA_TIMEOUT_MS ?? 30_000);
         this.env = options.env;
@@ -168,8 +168,8 @@ function layaArgsFromEnv() {
         throw new Error('AGENTS_RELAY_LAYA_ARGS must be a JSON string array');
     }
 }
-export function decisionMode(value = process.env.AGENTS_RELAY_DECISION_MODE) {
-    return value === 'shadow' || value === 'active' ? value : 'off';
+export function decisionMode() {
+    return 'active';
 }
 export async function chooseWithDecision(engine, mode, request, contract, fallback) {
     validateDecisionRequest(request, contract);

package/dist/execution-context.js

@@ -283,5 +283,5 @@ Runtime capabilities:
 ${runtimes}
 
 Delegation contract:
-Nested work MUST be created as an Agents Relay durable child task with parentTaskId lineage. Do not directly spawn arbitrary child agents. Leaf workers do not inherit top-level XChat orchestration duties merely because xchat-orchestrator is visible.`;
+${context.role === 'troubleshooter' ? 'This is an orchestrator-side recovery role. Follow the selected Troubleshooter contract for P0/self-job repair authority and use durable Agents Relay Jobs/Tasks for delegated work.' : 'Nested work MUST be created as an Agents Relay durable child task with parentTaskId lineage. Do not directly spawn arbitrary child agents. Leaf workers do not inherit top-level XChat orchestration duties merely because xchat-orchestrator is visible.'}`;
 }

package/dist/reconciler.js

@@ -11,6 +11,7 @@ import { ReconcilerReviewBase } from './reconciler-review.js';
 import { executionEventSource } from './reconciler-runtime.js';
 export { executionEventSource } from './reconciler-runtime.js';
 import { CallbackWorkflowEffectHandler } from './workflow/effects.js';
+import { pendingTroubleshooterTasks } from './troubleshooter.js';
 const PRIORITY_RANK = { P0: 0, P1: 1, P2: 2, P3: 3 };
 export function compareTaskPriority(a, b, jobPriority = 'P2') { const ar = PRIORITY_RANK[a.priority ?? jobPriority]; const br = PRIORITY_RANK[b.priority ?? jobPriority]; return ar - br || a.createdAt.localeCompare(b.createdAt) || a.id.localeCompare(b.id); }
 export async function managedWorkerInput(job, task, context) {
@@ -131,6 +132,10 @@ export class Reconciler extends ReconcilerReviewBase {
                 return job;
             await this.recoverLostExecutions(job, now);
             await this.recoverPlannerChildren(job);
+            for (const recovery of pendingTroubleshooterTasks(await this.store.load(jobId))) {
+                await this.store.appendTask(recovery);
+                await this.emit(eventFor(job.id, recovery.id, null, "troubleshooter.created", "queued", "Troubleshooter task " + recovery.id + " created for failed/blocked work", "orchestrator", { agentName: "troubleshooter" }));
+            }
             await this.routeExecutableTasks(await this.store.load(jobId));
             for (const task of job.tasks)
                 if (task.state === 'CANCELLED') {
@@ -313,7 +318,7 @@ export class Reconciler extends ReconcilerReviewBase {
         }
         let executionContext;
         try {
-            const role = task.agentName === 'reviewer' ? 'reviewer' : task.parentTaskId ? 'nested-worker' : 'worker';
+            const role = task.agentName === 'reviewer' ? 'reviewer' : task.agentName === 'troubleshooter' ? 'troubleshooter' : task.parentTaskId ? 'nested-worker' : 'worker';
             executionContext = await this.resolvedContext(job, task, role);
         }
         catch (error) {

package/dist/troubleshooter.js

@@ -0,0 +1,76 @@
+import { taskOverallTimeoutMs } from './types.js';
+function slug(value) {
+    return value.replace(/[^A-Za-z0-9_-]+/g, '-').replace(/^-+|-+$/g, '').slice(0, 72) || 'task';
+}
+export function troubleshooterTaskId(task) {
+    return `troubleshoot-${slug(task.id)}-a${task.attempt}`;
+}
+function failureInput(job, task) {
+    return `Recover the failed/blocked task below and make it move again if technically safe.
+
+Original repository: ${job.repository}
+Original PR: #${job.prNumber}
+Original job: ${job.id}
+Original task: ${task.id}
+Original task URL: ${task.taskUrl ?? 'unknown'}
+Original state: ${task.state}
+Original attempt: ${task.attempt}
+Original worker: ${task.workerSelection?.workerId ?? task.adapter}
+Original routing: ${task.routing ? JSON.stringify(task.routing) : 'unrouted'}
+Failure reason: ${task.failureReason ?? task.error ?? 'unknown'}
+
+MANDATORY EVENT EVIDENCE BEFORE DIAGNOSIS:
+Retrieve this task's lifecycle event history before classifying the root cause or retrying:
+GET http://127.0.0.1:8787/api/events?job=${encodeURIComponent(job.id)}&task=${encodeURIComponent(task.id)}&limit=200
+Inspect event order, type/status, source, data/phase, duplicate events, missing expected lifecycle events, late events, and execution identifiers. If the event stream points to delivery/runtime failure, inspect the corresponding Relay/worker/tool logs too. Do not treat the final task.error string as the complete root cause.
+
+Follow the selected Troubleshooter agent contract. Diagnose root cause from durable task state + event history + relevant runtime logs before retry. If the blocker is a shared worker/runtime/tool defect, create and directly complete a P0/self-job in the owning repository, including required release/deploy verification. Retry this exact original task only after the blocker is removed, reconcile the original job, and verify durable progress.`;
+}
+function belongsToCurrentRun(job, task) {
+    const slot = job.schedule?.currentRunSlot;
+    return !slot || task.createdAt >= slot;
+}
+export function pendingTroubleshooterTasks(job) {
+    const existing = new Set(job.tasks.filter(task => task.agentName === 'troubleshooter').map(task => task.id));
+    const recoveries = [];
+    for (const task of job.tasks) {
+        if (!['FAILED', 'BLOCKED'].includes(task.state) || task.kind === 'planner' || task.agentName === 'troubleshooter' || !belongsToCurrentRun(job, task))
+            continue;
+        const id = troubleshooterTaskId(task);
+        if (existing.has(id))
+            continue;
+        const now = new Date().toISOString();
+        recoveries.push({
+            jobId: job.id,
+            id,
+            kind: 'work',
+            priority: 'P0',
+            storySize: 'M',
+            projectName: job.title,
+            agentName: 'troubleshooter',
+            parentTaskId: null,
+            dependencies: [],
+            capabilities: ['troubleshoot', 'recovery'],
+            workerRequirements: { capabilities: ['model', 'local_repo', 'iterative_tools'], runtime: 'codex' },
+            adapter: 'codex',
+            input: failureInput(job, task),
+            output: { kind: 'task_pr' },
+            continuation: null,
+            continuationDeliveredAt: null,
+            state: 'QUEUED',
+            attempt: 0,
+            maxAttempts: 1,
+            leaseOwner: null,
+            leaseExpiresAt: null,
+            executionId: null,
+            threadId: null,
+            result: null,
+            plannerResult: null,
+            error: null,
+            timeoutMs: taskOverallTimeoutMs('M'),
+            createdAt: now,
+            updatedAt: now,
+        });
+    }
+    return recoveries;
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agents-relay",
-  "version": "1.0.94",
+  "version": "1.0.96",
   "description": "Durable async agent jobs coordinated through GitHub pull requests",
   "type": "module",
   "bin": {

package/skills/agents-relay/SKILL.md

@@ -111,3 +111,9 @@ Planner-created child Tasks remain descriptive agent work and must include worke
 GitHub PR/job/task markers remain durable workflow state. Events are the authoritative execution-state transition signal for model-backed tasks. Runtime events, process exits, browser submission receipts, and conversation text are evidence only.
 
 Keep secrets, credentials, private prompts, and large private payloads out of durable markers and events.
+
+### Automatic task troubleshooting
+
+When an ACTIVE Job contains a current-run `FAILED` or `BLOCKED` task, reconciliation creates one P0-priority `troubleshooter` recovery task for that exact failed attempt. The recovery task is routed to a local Codex-capable worker with repository/tool access and runs under the bundled Troubleshooter agent contract. Troubleshooter diagnosis is event-first: it must inspect the failed task's task-scoped lifecycle event history (including ordering, duplicate/missing/late events, source and phase data) before classifying root cause or retrying.
+
+The Troubleshooter diagnoses before retrying. A task-local or transient blocker is repaired narrowly; a shared worker/runtime/tool defect is repaired through the owning repository's P0/self-job workflow. Only after the blocker is removed does it retry the original task and verify durable progress. Historical failures from older scheduled autonomous runs do not spawn recovery work, and deterministic reconciliation does not duplicate a Troubleshooter task for the same failed attempt.

package/skills/agents-relay/agents/troubleshooter.agent.md

@@ -0,0 +1,52 @@
+---
+name: troubleshooter
+description: Recover failed or blocked Agents Relay tasks by diagnosing the root cause, repairing recoverable blockers, and retrying the original task only after the blocker is removed.
+capabilities: [troubleshoot, recovery, orchestration, local_repo, iterative_tools]
+runtime: codex
+delegation: [p0-self-job, relay-task-retry]
+restrictions: [preserve-failure-evidence, no-blind-retry, no-human-blocker-bypass]
+---
+
+# Troubleshooter Agent
+
+You are the recovery orchestrator for a failed or blocked Agents Relay task.
+
+Your goal is not to explain the failure. Your goal is to make the original task move again when it is safe and technically possible.
+
+Read and follow the shared Task / Job contract and orchestrator P0/self-job workflow before acting.
+
+## Recovery loop
+
+1. Retrieve the durable original Job and failed/blocked Task from GitHub.
+2. **Mandatory before diagnosis:** retrieve the failed Task's task-scoped lifecycle event history. When the local Relay dashboard is available, use `GET http://127.0.0.1:8787/api/events?job=<job_id>&task=<task_id>&limit=200`. Inspect event order, type/status, source, data/phase, execution IDs, duplicates, missing expected lifecycle events, and late events. Then inspect task error/failureReason, worker selection, routing, attempt count, task input/output contract, PR state, and relevant runtime/service logs.
+3. Classify the blocker:
+   - transient execution/delivery/lease failure;
+   - task-local implementation or validation failure;
+   - shared worker/runtime/tool/Agents Relay defect;
+   - genuine external/human blocker.
+4. Diagnose the smallest root cause from the combined durable Task + event sequence + logs. The final `task.error` is only one piece of evidence. Never retry first and investigate later.
+5. Repair:
+   - For a transient failure, verify the transient condition is gone.
+   - For task-local failure, make the smallest durable repair in the original Job/PR when that is within scope.
+   - For a shared platform/runtime defect, become the active orchestrator for a P0/self-job in the repository that owns the broken component. Create the durable P0 Job and its orchestrator-owned Task, fix it directly, validate, merge, and complete any required publish/deploy reconciliation.
+   - For a genuine human/external blocker, do not fake progress. Publish task.blocked with the exact action required.
+6. Only after the blocker is removed, run `agents-relay task retry` for the original Task and reconcile its Job.
+7. Verify durable evidence that the original Task moved: at minimum QUEUED after retry, preferably a new execution / task.started / observable progress.
+8. Publish this Troubleshooter Task's terminal event only after the recovery outcome is durable.
+
+## Rules
+
+- Never erase or rewrite the original failure evidence.
+- Never blindly retry a deterministic failure. Event-history inspection is a hard precondition for retry; if event history cannot be retrieved, record that observability blocker explicitly rather than pretending diagnosis is complete.
+- Never create a second P0 Job for the same root cause when a suitable active one already exists; reuse/adopt/reconcile it.
+- P0 fixes must follow the P0/self-job contract: durable Job first, orchestrator-owned Task immediately, direct execution by this recovery orchestrator, no reviewer gate unless explicitly required.
+- Repository PR creation/adoption/repair still goes through Agents Relay.
+- Keep fixes narrowly scoped; do not use a failed application task as permission for broad platform refactoring.
+- If a P0 fix publishes a package or changes a managed service, verify the released artifact and dependent service health before retrying the original Task.
+- Do not mark yourself successful merely because a repair PR was opened. Recovery succeeds when the original Task is actually retryable and has been retried.
+
+## Example
+
+A final review fails because the ChatGPT browser worker reports `Temporary Chat toggle was not uniquely observed`.
+
+That is not a defect in the reviewed PR. Inspect the browser-worker/browser-harness path, repair that shared runtime through a P0/self-job in its owning repository, validate/deploy the fix, then retry the original final-review Task and verify its new execution begins.