agents-relay 2.0.151 → 2.0.153
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -1
- package/dist/cli-help.js +2 -1
- package/dist/managed-worker-input.js +1 -1
- package/dist/pool.js +1 -1
- package/dist/reconciler-runtime.js +42 -38
- package/dist/reconciler.js +10 -4
- package/dist/task-context.js +1 -1
- package/dist/task-inactivity-watchdog.js +45 -0
- package/dist/workflow/contracts.js +1 -0
- package/dist/workflow/durable-transition.js +26 -0
- package/package.json +2 -1
- package/skills/agents-relay/SKILL.md +2 -2
- package/skills/agents-relay/contracts/event-contract.md +3 -1
- package/skills/agents-relay/contracts/task-job-contract.md +2 -0
- package/workflow/task-lifecycle.mmd +1 -0
- package/dist/terminal-event-watchdog.js +0 -11
package/README.md
CHANGED
|
@@ -138,7 +138,7 @@ retries and daemon restarts do not create duplicate work.
|
|
|
138
138
|
Task producers must follow `skills/agents-relay/contracts/task-job-contract.md`; that shared skill contract defines story size, lifecycle events, review, and auto-merge semantics.
|
|
139
139
|
|
|
140
140
|
Execution routing is code-driven: Relay builds complete executable worker+model routes from authoritative runtime/model/resource facts, merges profile defaults with the user's explicit `[choices]`, deduplicates them, removes only impossible routes via hard gates, then calls the generic typed `decide(request, contract)` boundary exactly once with one merged choice list and the `[routing-instruction]` text forwarded verbatim. Task/resource/capability signals and the routing instruction are soft decision facts; decision failure falls back deterministically to a usable Codex default route. See [Execution Router and typed decisions](docs/worker-model-routers.md).
|
|
141
|
-
All task producers use the same story-size contract: `S`≈5 minutes, `M`≈15 minutes, and `L`≈45 minutes of
|
|
141
|
+
All task producers use the same story-size contract: `S`≈5 minutes, `M`≈15 minutes, and `L`≈45 minutes of inactivity, with `M` as the default. `high` reasoning doubles that inactivity window. Accepted `task.started` and newer `task.progress` events from the exact current execution reset it. ChatGPT Browser Worker start-ack timeout is derived as one third of the effective inactivity timeout, clamped between 1 and 5 minutes. Orchestrators and planners should choose `storySize`, not raw timeout milliseconds; `--timeout` remains only as an explicit compatibility/debug override.
|
|
142
142
|
|
|
143
143
|
Autonomous jobs can also be scheduled. The Job marker stores a five-field cron expression, IANA timezone, next run, and missed-run/concurrency policy. The daemon evaluates the SQLite-backed managed Job view and validates durable schedule facts before claiming a due slot. A repository heartbeat invokes `tick` at 10-minute resolution. See
|
|
144
144
|
[Scheduled autonomous jobs](docs/scheduled-autonomous-jobs.md) for the exact
|
package/dist/cli-help.js
CHANGED
|
@@ -135,7 +135,8 @@ Example:
|
|
|
135
135
|
'task update': `Update mutable durable Task fields in place.
|
|
136
136
|
|
|
137
137
|
Required: --task-id ID and the normal Job storage options.
|
|
138
|
-
Mutable while not RUNNING/SUCCEEDED/CANCELLED: --title, --adapter, --priority, --story-size, --input-file PATH|- or --input MARKDOWN, --project, --agent, --agent-graph-file PATH|-, --parent, --deps, --capabilities, --timeout, --max-attempts, --provider, --model, --profile, --reasoning, --cwd, --chatgpt-project, --chatgpt-tab-close-policy, --draft, --queue.
|
|
138
|
+
Mutable while not RUNNING/SUCCEEDED/CANCELLED: --title, --adapter, --priority, --story-size, --input-file PATH|- or --input MARKDOWN, --project, --agent, --agent-graph-file PATH|-, --parent, --deps, --capabilities, --timeout (inactivity window), --max-attempts, --provider, --model, --profile, --reasoning, --cwd, --chatgpt-project, --chatgpt-tab-close-policy, --draft, --queue.
|
|
139
|
+
Accepted task.started and newer current-execution task.progress events reset --timeout.
|
|
139
140
|
Use --clear-routing to remove routing metadata.
|
|
140
141
|
|
|
141
142
|
Example:
|
|
@@ -70,7 +70,7 @@ As your FIRST execution action, before reading files, editing code, running comm
|
|
|
70
70
|
you MUST publish exactly one canonical task.started event for this exact Job/Task.
|
|
71
71
|
Immediately after task.started, BEFORE reading task context or doing any implementation/review/recovery work, read the full authoritative Relay event contract at absolute path /Users/chengli/Workspace/agents-relay/skills/agents-relay/contracts/event-contract.md. If it cannot be read, publish task.failed with the blocker and stop. Do not rely only on this prompt summary or a relative path.
|
|
72
72
|
|
|
73
|
-
|
|
73
|
+
Publish task.progress only for meaningful work milestones. It is nonterminal; do not fabricate progress. Relay's Task.timeoutMs is an inactivity timeout reset only by accepted task.started or newer task.progress events for this exact execution.
|
|
74
74
|
Before returning or allowing the worker process to exit, you MUST publish exactly one canonical terminal event:
|
|
75
75
|
- task.completed only after the declared output is durable;
|
|
76
76
|
- task.failed for implementation, tool, command, test, patch, commit, push, runtime, or other execution failure that prevents completion;
|
package/dist/pool.js
CHANGED
|
@@ -232,7 +232,7 @@ export class RepositoryWorkerPool {
|
|
|
232
232
|
return;
|
|
233
233
|
if (routedWebhook)
|
|
234
234
|
return;
|
|
235
|
-
const lifecycle = ['task.started', 'task.completed', 'task.failed', 'task.blocked', 'task.unblocked', 'task.cancelled', 'review.started', 'review.approved', 'review.changes_required', 'review.blocked'].includes(event.type);
|
|
235
|
+
const lifecycle = ['task.started', 'task.progress', 'task.completed', 'task.failed', 'task.blocked', 'task.unblocked', 'task.cancelled', 'review.started', 'review.approved', 'review.changes_required', 'review.blocked'].includes(event.type);
|
|
236
236
|
// GitHub transport is observational. It must never wake lifecycle execution.
|
|
237
237
|
if (event.type === 'github.webhook')
|
|
238
238
|
return;
|
|
@@ -9,12 +9,12 @@ import { eventFor } from './events.js';
|
|
|
9
9
|
import { WorkflowEffectExecutor } from './workflow/effects.js';
|
|
10
10
|
import { ReconcileCoordinator } from './reconcile-coordinator.js';
|
|
11
11
|
import { acceptedReportedTag, eventDataWithoutTag } from './task-tag.js';
|
|
12
|
-
import { rearmTerminalEventDeadline } from './terminal-event-watchdog.js';
|
|
13
12
|
import { createWorkerStartAck } from './worker-start-ack.js';
|
|
14
13
|
import { monitorExecutionProcess } from './execution-process-tracker.js';
|
|
15
14
|
import { HostInteractiveInvocationCoordinator, hostInvocationPorts } from './interactive-invocation.js';
|
|
16
15
|
import { resumeInteractiveTask } from './interactive-resume.js';
|
|
17
16
|
import { codexFallbackCompatible, persistChatGptPlatformFailure } from './chatgpt-platform-fallback.js';
|
|
17
|
+
import { acceptTaskProgress, armInactivityWatchdog, inactivityDeadline, isCorrelatedWorkerEvent, isRecoverableWatchdogFailure } from './task-inactivity-watchdog.js';
|
|
18
18
|
const activeExecutionOwners = new Map();
|
|
19
19
|
import { withExecutionBoundary } from './execution-boundary.js';
|
|
20
20
|
export { withExecutionBoundary } from './execution-boundary.js';
|
|
@@ -84,6 +84,8 @@ export class ReconcilerRuntimeBase {
|
|
|
84
84
|
return activeOwner.applyEvent(event);
|
|
85
85
|
if (event.type === 'task.started')
|
|
86
86
|
return this.markWorkerStarted(event);
|
|
87
|
+
if (event.type === 'task.progress')
|
|
88
|
+
return this.markWorkerProgress(event);
|
|
87
89
|
if (event.type === 'task.completed') {
|
|
88
90
|
const task = (await this.store.load(event.job_id)).tasks.find(item => item.id === event.task_id);
|
|
89
91
|
const normalized = task?.agentName === 'reviewer' ? normalizeReviewerTerminalEvent(event) : null;
|
|
@@ -176,14 +178,8 @@ export class ReconcilerRuntimeBase {
|
|
|
176
178
|
owner.clearExecution(taskId, executionId);
|
|
177
179
|
return true;
|
|
178
180
|
}
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
if (!live || live.executionId !== executionId)
|
|
182
|
-
return;
|
|
183
|
-
const timer = this.terminalTimers.get(taskId);
|
|
184
|
-
if (timer)
|
|
185
|
-
clearTimeout(timer);
|
|
186
|
-
this.terminalTimers.delete(taskId);
|
|
181
|
+
armInactivityWatchdog(task, executionId) {
|
|
182
|
+
armInactivityWatchdog(task, executionId, this.terminalTimers, this.options.postExitTerminalGraceMs ?? 15_000, () => this.detach(this.timeoutTask(task.jobId, task.id, executionId), task.jobId, task.id, 'worker inactivity timeout'));
|
|
187
183
|
}
|
|
188
184
|
clearExecution(taskId, executionId) {
|
|
189
185
|
const live = this.live.get(taskId);
|
|
@@ -206,16 +202,19 @@ export class ReconcilerRuntimeBase {
|
|
|
206
202
|
eligible: (job, task) => {
|
|
207
203
|
if (!task.executionId || eventSourceId(event) !== executionEventSource(job.id, task.id, task.executionId))
|
|
208
204
|
return false;
|
|
205
|
+
if (!Number.isFinite(Date.parse(event.timestamp)) || Date.parse(event.timestamp) > Date.now())
|
|
206
|
+
return false;
|
|
209
207
|
const claimedExecution = task.executionId !== null && task.leaseOwner !== null;
|
|
210
208
|
const exactLiveExecution = live && event.job_id === live.jobId && task.executionId === live.executionId;
|
|
211
|
-
return Boolean(exactLiveExecution || claimedExecution ||
|
|
209
|
+
return Boolean(exactLiveExecution || claimedExecution || isRecoverableWatchdogFailure(task));
|
|
212
210
|
},
|
|
213
211
|
facts: (_job, task) => ({
|
|
214
|
-
recoverableExecution:
|
|
212
|
+
recoverableExecution: isRecoverableWatchdogFailure(task)
|
|
215
213
|
&& Boolean(task.executionId && eventSourceId(event) === executionEventSource(event.job_id, task.id, task.executionId)),
|
|
216
214
|
}),
|
|
217
215
|
mutate: (_job, task) => {
|
|
218
|
-
task.updatedAt = new Date().toISOString();
|
|
216
|
+
task.updatedAt = task.lastProgressAt = new Date().toISOString();
|
|
217
|
+
task.lastProgressEventAt = event.timestamp;
|
|
219
218
|
},
|
|
220
219
|
});
|
|
221
220
|
if (!transition?.decision.accepted)
|
|
@@ -227,21 +226,11 @@ export class ReconcilerRuntimeBase {
|
|
|
227
226
|
}
|
|
228
227
|
if (live?.executionId === transition.task.executionId)
|
|
229
228
|
live.workerStarted = true;
|
|
229
|
+
this.armInactivityWatchdog(transition.task, transition.task.executionId);
|
|
230
230
|
return { applied: true, reconcile: transition.from === 'FAILED', reason: transition.from === 'FAILED' ? 'task.started.recovery' : 'task.started', reconcileMode: transition.from === 'FAILED' ? 'detached' : undefined };
|
|
231
231
|
}
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
return true;
|
|
235
|
-
const error = task.error ?? '';
|
|
236
|
-
return error.includes('worker_start_timeout')
|
|
237
|
-
|| error.includes('timed out waiting for the mandatory terminal worker event')
|
|
238
|
-
|| error.includes('exited before publishing task.started')
|
|
239
|
-
|| error.includes('exited after task.started without publishing a terminal task event')
|
|
240
|
-
|| error.includes('Worker delivery failed after task.started');
|
|
241
|
-
}
|
|
242
|
-
isCorrelatedWorkerEvent(task, event) {
|
|
243
|
-
return task.executionId !== null && eventSourceId(event) === executionEventSource(event.job_id, task.id, task.executionId);
|
|
244
|
-
}
|
|
232
|
+
async markWorkerProgress(event) { const task = await acceptTaskProgress(this.taskTransitions, event); if (!task?.executionId)
|
|
233
|
+
return { applied: false, reconcile: false, reason: 'task.progress.ignored' }; this.armInactivityWatchdog(task, task.executionId); return { applied: true, reconcile: false, reason: 'task.progress' }; }
|
|
245
234
|
finalizeReviewExecution(jobId, task, event, afterCommit = true) {
|
|
246
235
|
if (!task.executionId)
|
|
247
236
|
return true;
|
|
@@ -331,29 +320,38 @@ export class ReconcilerRuntimeBase {
|
|
|
331
320
|
const task = job.tasks.find(item => item.id === taskId);
|
|
332
321
|
if (!task)
|
|
333
322
|
return;
|
|
323
|
+
if (task.adapter !== 'chatgpt') {
|
|
324
|
+
const exited = await this.taskTransitions.updateFact(jobId, taskId, current => current.executionId === executionId && ['RUNNING', 'QUEUED'].includes(current.state), current => { current.processExitedAt = new Date().toISOString(); });
|
|
325
|
+
if (exited)
|
|
326
|
+
this.armInactivityWatchdog(exited, executionId);
|
|
327
|
+
}
|
|
334
328
|
const correlated = task.executionId === executionId || task.executionHistory?.some(item => item.id === executionId);
|
|
335
329
|
if (!correlated)
|
|
336
330
|
return;
|
|
337
331
|
if (task.adapter !== 'chatgpt' && (task.executionId !== executionId || terminalExecutionClaimed(jobId, taskId, executionId)))
|
|
338
332
|
return;
|
|
339
|
-
if (task.adapter !== 'chatgpt' && task.executionId === executionId && live?.executionId === executionId)
|
|
340
|
-
rearmTerminalEventDeadline(this.terminalTimers, taskId, task.timeoutMs - (Date.now() - live.launchBarrier), this.options.postExitTerminalGraceMs ?? 15_000, () => this.detach(this.timeoutTask(jobId, taskId, executionId), jobId, taskId, 'post-exit terminal event timeout'));
|
|
341
333
|
await this.emit(eventFor(jobId, taskId, task.parentTaskId, task.adapter === 'chatgpt' ? 'task.process.async_exited' : 'task.process.exited', live?.execution.processExit ? (live.execution.processExit.exitCode === 0 ? 'succeeded' : 'failed') : (error ? 'failed' : 'succeeded'), result?.summary ?? (live?.execution.processExit?.terminationReason === 'parent-shutdown' ? 'Worker interrupted by daemon shutdown' : error ? `Worker exited: ${error}${live?.execution.processExit ? ` (exit=${live.execution.processExit.exitCode ?? 'null'}${live.execution.processExit.signal ? ` signal=${live.execution.processExit.signal}` : ''})` : ''}` : 'Worker runtime exited'), 'orchestrator', { authoritative: false, waitingForTerminalEvent: true, exitCode: live?.execution.processExit?.exitCode ?? null,
|
|
342
334
|
signal: live?.execution.processExit?.signal ?? null, terminationReason: live?.execution.processExit?.terminationReason ?? null, stderr: live?.execution.processExit?.stderr ?? '', processError: error, executionId, pid: live?.execution.pid ?? null }, { id: executionEventSource(jobId, taskId, executionId), agent: 'relay', component: 'agents-relay' }));
|
|
343
335
|
}
|
|
344
336
|
async timeoutTask(jobId, taskId, executionId) {
|
|
345
|
-
const
|
|
346
|
-
if (
|
|
347
|
-
|
|
348
|
-
|
|
337
|
+
const current = (await this.store.load(jobId)).tasks.find(task => task.id === taskId);
|
|
338
|
+
if (!current || current.executionId !== executionId || !['QUEUED', 'RUNNING'].includes(current.state))
|
|
339
|
+
return;
|
|
340
|
+
const deadline = inactivityDeadline(current, executionId, this.options.postExitTerminalGraceMs ?? 15_000);
|
|
341
|
+
if (deadline > Date.now()) {
|
|
342
|
+
this.armInactivityWatchdog(current, executionId);
|
|
343
|
+
return;
|
|
349
344
|
}
|
|
345
|
+
const live = this.live.get(taskId);
|
|
346
|
+
// Do not cancel before the serialized watchdog transition rechecks the
|
|
347
|
+
// deadline: a concurrent accepted progress event may have extended it.
|
|
350
348
|
const exit = live?.executionId === executionId && live.processSettled ? live.execution.processExit : undefined;
|
|
351
349
|
const rawError = live?.executionId === executionId && live.processSettled ? (live.processError || exit?.stderr || '').trim() : '';
|
|
352
350
|
const failedExit = Boolean(exit && (exit.exitCode !== 0 || exit.signal));
|
|
353
351
|
const failureReason = failedExit ? classifyWorkerRejection(rawError) : 'terminal_event_timeout';
|
|
354
|
-
const failureMessage = failedExit ? (rawError || `Worker exited with code ${exit?.exitCode ?? 'null'}${exit?.signal ? `, signal ${exit.signal}` : ''}`) : `Task ${taskId}
|
|
352
|
+
const failureMessage = failedExit ? (rawError || `Worker exited with code ${exit?.exitCode ?? 'null'}${exit?.signal ? `, signal ${exit.signal}` : ''}`) : `Task ${taskId} exceeded its inactivity timeout without a terminal event`;
|
|
355
353
|
const transition = await this.taskTransitions.apply(jobId, taskId, 'watchdog.failed', {
|
|
356
|
-
eligible: (_job, task) => task.executionId === executionId,
|
|
354
|
+
eligible: (_job, task) => task.executionId === executionId && inactivityDeadline(task, executionId, this.options.postExitTerminalGraceMs ?? 15_000) <= Date.now(),
|
|
357
355
|
mutate: (_job, task) => {
|
|
358
356
|
this.clearExecution(task.id, executionId);
|
|
359
357
|
task.error = failureMessage;
|
|
@@ -363,9 +361,15 @@ export class ReconcilerRuntimeBase {
|
|
|
363
361
|
},
|
|
364
362
|
});
|
|
365
363
|
if (!transition?.decision.accepted) {
|
|
366
|
-
this.
|
|
364
|
+
const latest = (await this.store.load(jobId)).tasks.find(task => task.id === taskId);
|
|
365
|
+
if (latest?.executionId === executionId && ['QUEUED', 'RUNNING'].includes(latest.state))
|
|
366
|
+
this.armInactivityWatchdog(latest, executionId);
|
|
367
367
|
return;
|
|
368
368
|
}
|
|
369
|
+
if (live?.executionId === executionId) {
|
|
370
|
+
live.controller.abort();
|
|
371
|
+
live.execution.cancel();
|
|
372
|
+
}
|
|
369
373
|
let reroutedTask = null;
|
|
370
374
|
if (reroutableWorkerRejections.has(failureReason) && transition.task.attempt < transition.task.maxAttempts) {
|
|
371
375
|
const current = (await this.store.load(jobId)).tasks.find(task => task.id === taskId);
|
|
@@ -410,19 +414,19 @@ export class ReconcilerRuntimeBase {
|
|
|
410
414
|
if (event.type === 'task.completed' && task.interactiveInvocation?.status !== undefined && task.interactiveInvocation.status !== 'approved')
|
|
411
415
|
return false; // current interactive revision requires durable approval
|
|
412
416
|
const modelBacked = task.adapter === 'codex' || task.adapter === 'chatgpt';
|
|
413
|
-
if (!trustedInternal && modelBacked && !
|
|
417
|
+
if (!trustedInternal && modelBacked && !isCorrelatedWorkerEvent(task, event)) {
|
|
414
418
|
const eventExecutionId = executionIdFromSource(eventSourceId(event), _job.id, task.id);
|
|
415
419
|
const prior = eventExecutionId ? task.executionHistory?.find(item => item.id === eventExecutionId) : undefined;
|
|
416
420
|
if (!prior || prior.attempt !== task.attempt)
|
|
417
421
|
return false;
|
|
418
422
|
}
|
|
419
423
|
const terminalWaiter = this.terminalWaiters.get(task.id);
|
|
420
|
-
return !terminalWaiter || Boolean(task.executionId && terminalWaiter.executionId === task.executionId &&
|
|
424
|
+
return !terminalWaiter || Boolean(task.executionId && terminalWaiter.executionId === task.executionId && isCorrelatedWorkerEvent(task, event));
|
|
421
425
|
},
|
|
422
426
|
facts: (_job, task) => {
|
|
423
427
|
const processExitedAfterPublishedTerminal = task.failureReason === 'worker_exit_without_terminal_event';
|
|
424
|
-
const freshEnough = processExitedAfterPublishedTerminal ||
|
|
425
|
-
return { recoverableExecution:
|
|
428
|
+
const freshEnough = processExitedAfterPublishedTerminal || isCorrelatedWorkerEvent(task, event);
|
|
429
|
+
return { recoverableExecution: isRecoverableWatchdogFailure(task) && freshEnough };
|
|
426
430
|
},
|
|
427
431
|
mutate: (job, task) => {
|
|
428
432
|
const modelBacked = task.adapter === 'codex' || task.adapter === 'chatgpt';
|
package/dist/reconciler.js
CHANGED
|
@@ -180,6 +180,9 @@ export class Reconciler extends ReconcilerReviewBase {
|
|
|
180
180
|
const taskStatesBeforeSchedule = new Map(beforeSchedule.tasks.map(task => [task.id, task.state]));
|
|
181
181
|
const scheduleBefore = JSON.stringify(beforeSchedule.schedule ?? null);
|
|
182
182
|
let scheduled = schedule(beforeSchedule);
|
|
183
|
+
for (const task of scheduled.tasks)
|
|
184
|
+
if (task.executionId && ['QUEUED', 'RUNNING'].includes(task.state))
|
|
185
|
+
this.armInactivityWatchdog(task, task.executionId);
|
|
183
186
|
for (const task of scheduled.tasks) {
|
|
184
187
|
if (taskStatesBeforeSchedule.get(task.id) !== task.state)
|
|
185
188
|
await this.store.saveTask(task);
|
|
@@ -366,6 +369,9 @@ export class Reconciler extends ReconcilerReviewBase {
|
|
|
366
369
|
freshTask.leaseOwner = this.options.owner;
|
|
367
370
|
freshTask.leaseExpiresAt = new Date(startedAt + leaseMs).toISOString();
|
|
368
371
|
freshTask.executionId = randomUUID();
|
|
372
|
+
delete freshTask.lastProgressAt;
|
|
373
|
+
delete freshTask.lastProgressEventAt;
|
|
374
|
+
delete freshTask.processExitedAt;
|
|
369
375
|
freshTask.executionHistory = [...(freshTask.executionHistory ?? []), {
|
|
370
376
|
id: freshTask.executionId,
|
|
371
377
|
attempt: freshTask.attempt,
|
|
@@ -396,6 +402,9 @@ export class Reconciler extends ReconcilerReviewBase {
|
|
|
396
402
|
durable.leaseOwner = this.options.owner;
|
|
397
403
|
durable.leaseExpiresAt = new Date(startedAt + leaseMs).toISOString();
|
|
398
404
|
durable.executionId = randomUUID();
|
|
405
|
+
delete durable.lastProgressAt;
|
|
406
|
+
delete durable.lastProgressEventAt;
|
|
407
|
+
delete durable.processExitedAt;
|
|
399
408
|
durable.executionHistory = [...(durable.executionHistory ?? []), {
|
|
400
409
|
id: durable.executionId, attempt: durable.attempt, startedAt: new Date(startedAt).toISOString(),
|
|
401
410
|
}].slice(-8);
|
|
@@ -514,10 +523,7 @@ export class Reconciler extends ReconcilerReviewBase {
|
|
|
514
523
|
transition: { graph: 'task-lifecycle', from: 'QUEUED', to: 'QUEUED', actions: ['effect.launchTask'] },
|
|
515
524
|
data: { adapter: adapter.id, attempt: task.attempt, model: task.routing?.model ?? null },
|
|
516
525
|
});
|
|
517
|
-
|
|
518
|
-
this.detach(this.timeoutTask(job.id, task.id, durableExecutionId), job.id, task.id, 'worker terminal-event timeout');
|
|
519
|
-
}, Math.max(1, task.timeoutMs));
|
|
520
|
-
this.terminalTimers.set(task.id, terminalTimer);
|
|
526
|
+
this.armInactivityWatchdog(task, durableExecutionId);
|
|
521
527
|
this.detach(execution.promise.then(result => this.runtimeSettled(job.id, task.id, durableExecutionId, result, null), error => error instanceof StartAckTimeout
|
|
522
528
|
? this.failLaunch(job.id, task.id, durableExecutionId, error, 'start-ack')
|
|
523
529
|
: error instanceof LaunchFailure || error instanceof PreSubmissionRejection
|
package/dist/task-context.js
CHANGED
|
@@ -45,5 +45,5 @@ THEN read the entire authoritative event contract from /Users/chengli/Workspace/
|
|
|
45
45
|
THEN retrieve the authoritative Task Context:
|
|
46
46
|
agents-relay task context --repo ${repository} --id ${jobId} --task-id ${taskId} --execution-id ${executionId}
|
|
47
47
|
Verify the execution ID. Read Task input, dependencies, Plan, project context, and output requirements from that response. Never guess or create a new Job. Continue in this thread; Relay owns completion and review.
|
|
48
|
-
Follow the file's lifecycle contract, including progress
|
|
48
|
+
Follow the file's lifecycle contract, including optional meaningful progress and exactly one task.completed or task.failed before exiting. Accepted current-execution task.started and newer task.progress events reset Task.timeoutMs; duplicate, stale, or out-of-order progress does not. If context is unavailable publish task.failed with the blocker.`;
|
|
49
49
|
}
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
import { executionEventSource, eventSourceId } from './execution-event-identity.js';
|
|
2
|
+
export function isRecoverableWatchdogFailure(task) {
|
|
3
|
+
if (['worker_start_timeout', 'terminal_event_timeout', 'worker_exit_before_start', 'worker_exit_without_terminal_event', 'worker_delivery_after_start'].includes(task.failureReason ?? ''))
|
|
4
|
+
return true;
|
|
5
|
+
const error = task.error ?? '';
|
|
6
|
+
return error.includes('worker_start_timeout') || error.includes('timed out waiting for the mandatory terminal worker event') || error.includes('exceeded its inactivity timeout')
|
|
7
|
+
|| error.includes('exited before publishing task.started') || error.includes('exited after task.started without publishing a terminal task event')
|
|
8
|
+
|| error.includes('Worker delivery failed after task.started');
|
|
9
|
+
}
|
|
10
|
+
export function isCorrelatedWorkerEvent(task, event) {
|
|
11
|
+
return task.executionId !== null && eventSourceId(event) === executionEventSource(event.job_id, task.id, task.executionId);
|
|
12
|
+
}
|
|
13
|
+
export function inactivityDeadline(task, executionId, postExitGraceMs) {
|
|
14
|
+
const progressAt = Date.parse(task.lastProgressAt ?? task.executionHistory?.find(item => item.id === executionId)?.startedAt ?? task.updatedAt);
|
|
15
|
+
const timeoutAt = (Number.isFinite(progressAt) ? progressAt : Date.now()) + Math.max(1, task.timeoutMs);
|
|
16
|
+
const exitAt = task.adapter !== 'chatgpt' && task.processExitedAt ? Date.parse(task.processExitedAt) : NaN;
|
|
17
|
+
return Number.isFinite(exitAt) ? Math.min(timeoutAt, exitAt + postExitGraceMs) : timeoutAt;
|
|
18
|
+
}
|
|
19
|
+
export function armInactivityWatchdog(task, executionId, timers, postExitGraceMs, expire) {
|
|
20
|
+
const existing = timers.get(task.id);
|
|
21
|
+
if (existing)
|
|
22
|
+
clearTimeout(existing);
|
|
23
|
+
timers.delete(task.id);
|
|
24
|
+
if (!task.jobId || task.executionId !== executionId || !['QUEUED', 'RUNNING'].includes(task.state))
|
|
25
|
+
return;
|
|
26
|
+
const timer = setTimeout(expire, Math.max(1, inactivityDeadline(task, executionId, postExitGraceMs) - Date.now()));
|
|
27
|
+
timer.unref();
|
|
28
|
+
timers.set(task.id, timer);
|
|
29
|
+
}
|
|
30
|
+
export async function acceptTaskProgress(transitions, event) {
|
|
31
|
+
const parsedAt = Date.parse(event.timestamp);
|
|
32
|
+
const receivedAt = Date.now();
|
|
33
|
+
const transition = await transitions.apply(event.job_id, event.task_id, 'task.progress', {
|
|
34
|
+
eligible: (job, task) => task.state === 'RUNNING'
|
|
35
|
+
&& Boolean(task.executionId && eventSourceId(event) === executionEventSource(job.id, task.id, task.executionId))
|
|
36
|
+
&& Number.isFinite(parsedAt) && parsedAt <= receivedAt
|
|
37
|
+
&& (!task.lastProgressEventAt || parsedAt > Date.parse(task.lastProgressEventAt)),
|
|
38
|
+
mutate: (_job, task) => {
|
|
39
|
+
task.lastProgressAt = new Date(receivedAt).toISOString();
|
|
40
|
+
task.lastProgressEventAt = event.timestamp;
|
|
41
|
+
task.updatedAt = task.lastProgressAt;
|
|
42
|
+
},
|
|
43
|
+
});
|
|
44
|
+
return transition?.decision.accepted ? transition.task : null;
|
|
45
|
+
}
|
|
@@ -52,6 +52,32 @@ export class DurableTaskTransitionRuntime {
|
|
|
52
52
|
return { job, task, from, decision };
|
|
53
53
|
});
|
|
54
54
|
}
|
|
55
|
+
/** Persist a non-lifecycle fact without racing the serialized terminal transitions. */
|
|
56
|
+
async updateFact(jobId, taskId, eligible, mutate) {
|
|
57
|
+
return this.withLock(`${jobId}/${taskId}`, async () => {
|
|
58
|
+
let accepted = false;
|
|
59
|
+
const apply = (job) => {
|
|
60
|
+
const task = job.tasks.find(item => item.id === taskId);
|
|
61
|
+
if (!task || !eligible(task))
|
|
62
|
+
return job;
|
|
63
|
+
const previous = Date.parse(task.updatedAt);
|
|
64
|
+
mutate(task);
|
|
65
|
+
task.updatedAt = new Date(Math.max(Date.now(), Number.isFinite(previous) ? previous + 1 : 0)).toISOString();
|
|
66
|
+
accepted = true;
|
|
67
|
+
return job;
|
|
68
|
+
};
|
|
69
|
+
if (this.store.mutateJob) {
|
|
70
|
+
const updated = await this.store.mutateJob(jobId, apply);
|
|
71
|
+
return accepted ? updated.tasks.find(item => item.id === taskId) ?? null : null;
|
|
72
|
+
}
|
|
73
|
+
const job = apply(await this.store.load(jobId));
|
|
74
|
+
if (!accepted)
|
|
75
|
+
return null;
|
|
76
|
+
const task = job.tasks.find(item => item.id === taskId);
|
|
77
|
+
await this.store.saveTask(task);
|
|
78
|
+
return task;
|
|
79
|
+
});
|
|
80
|
+
}
|
|
55
81
|
async withLock(key, work) {
|
|
56
82
|
const previous = this.locks.get(key);
|
|
57
83
|
let release;
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "agents-relay",
|
|
3
|
-
"version": "2.0.
|
|
3
|
+
"version": "2.0.153",
|
|
4
4
|
"description": "Durable async agent jobs coordinated through GitHub pull requests",
|
|
5
5
|
"repository": {
|
|
6
6
|
"type": "git",
|
|
@@ -27,6 +27,7 @@
|
|
|
27
27
|
"test:verification:matrix": "node test/verification/recovery-report.mjs /tmp/agents-relay-job481-evidence",
|
|
28
28
|
"test:verification:l2": "npm run build && node --test test/verification/l2/*.test.mjs",
|
|
29
29
|
"test:verification:l2:live": "node test/verification/l2/live-provider.mjs",
|
|
30
|
+
"test:verification:l2:live-codex": "npm run build && node test/verification/l2/live-codex-agents.mjs",
|
|
30
31
|
"test:l2:agents": "npm run build && node test/l2-agent-prompts/evaluate.mjs"
|
|
31
32
|
},
|
|
32
33
|
"devDependencies": {
|
|
@@ -56,9 +56,9 @@ Agents Relay maps those terminal events to durable state:
|
|
|
56
56
|
- `task.failed` → `FAILED`
|
|
57
57
|
Relay may authoritatively move a Task to `BLOCKED` for an external condition and may resume it only through an authorized unblock operation. A human response is an append-only Task Comment and resumes the blocked Troubleshooter, not the failed original Task. Task input remains stable; recovery agents receive current authoritative state and relevant comments as execution context.
|
|
58
58
|
|
|
59
|
-
A model-backed task must not launch without an event bus. If no
|
|
59
|
+
A model-backed task must not launch without an event bus. If no accepted `task.started` or newer current-execution `task.progress` arrives within `Task.timeoutMs`, Relay records an inactivity timeout failure. A Codex process exit or ChatGPT submission receipt alone never changes the task to a terminal state.
|
|
60
60
|
|
|
61
|
-
Worker process exit is observational only. Relay does not perform an exit-time terminal check or synthesize task failure from process exit;
|
|
61
|
+
Worker process exit is observational only. Relay does not perform an exit-time terminal check or synthesize task failure from process exit; for synchronous workers it caps the remaining inactivity wait at the shorter post-exit terminal grace.
|
|
62
62
|
|
|
63
63
|
## Managed GitHub jobs
|
|
64
64
|
|
|
@@ -4,7 +4,9 @@
|
|
|
4
4
|
|
|
5
5
|
## Producer contract
|
|
6
6
|
|
|
7
|
-
Every Implementation, Reviewer, and Troubleshooter execution publishes `task.started`, optional `task.progress
|
|
7
|
+
Every Implementation, Reviewer, and Troubleshooter execution publishes `task.started`, optional `task.progress`, then **exactly one** agent terminal event:
|
|
8
|
+
|
|
9
|
+
`task.started` and `task.progress` reset the Task inactivity timeout only when Relay accepts the event for the exact current execution. Progress is nonterminal, must be newer than the last accepted progress event, and cannot extend a terminal or stale execution.
|
|
8
10
|
|
|
9
11
|
- `task.completed`: assigned responsibility finished and its role-specific output is durable, regardless of whether a reviewed artifact passed.
|
|
10
12
|
- `task.failed`: assigned responsibility could not be fulfilled; include failure phase, reason, and evidence in `data`.
|
|
@@ -22,6 +22,8 @@ Ordinary `job create` requires standalone objective Markdown. An external displa
|
|
|
22
22
|
Any task producer — orchestrator, planner, or nested task-producing agent — must follow this contract.
|
|
23
23
|
Task producers choose semantic task properties such as objective, dependencies, worker requirements, priority, and `storySize`. They do not normally choose an adapter, provider/model, or raw timeout milliseconds.
|
|
24
24
|
|
|
25
|
+
`Task.timeoutMs` is the inactivity window. Only an accepted `task.started` or newer `task.progress` event for the exact current execution resets it; duplicate, stale, and out-of-order progress does not.
|
|
26
|
+
|
|
25
27
|
### Planning graph and semantic Task tree
|
|
26
28
|
|
|
27
29
|
The planning graph expresses execution flow. The Task tree expresses semantic ownership. `dependencies` answer which durable outcomes must succeed before a Task can run; `parentTaskId` answers which durable artifact or milestone owns the Task. These are independent relationships.
|
|
@@ -151,6 +151,7 @@ leaseFailQueuedClearExecution --> failed
|
|
|
151
151
|
queued -->|on: cancel| cancelled
|
|
152
152
|
|
|
153
153
|
running -->|on: task.blocked| blocked
|
|
154
|
+
running -->|on: task.progress| running
|
|
154
155
|
running -->|on: review.approved| reviewApproveRunningClearError
|
|
155
156
|
reviewApproveRunningClearError --> reviewApproveRunningClearReason
|
|
156
157
|
reviewApproveRunningClearReason --> succeeded
|
|
@@ -1,11 +0,0 @@
|
|
|
1
|
-
/**
|
|
2
|
-
* Replace the overall worker deadline with a shorter post-exit deadline.
|
|
3
|
-
* Execution outcomes remain owned by the authoritative task lifecycle graph.
|
|
4
|
-
* Never extend the original overall timeout.
|
|
5
|
-
*/
|
|
6
|
-
export function rearmTerminalEventDeadline(timers, taskId, overallRemainingMs, graceMs, expire) {
|
|
7
|
-
const existing = timers.get(taskId);
|
|
8
|
-
if (existing)
|
|
9
|
-
clearTimeout(existing);
|
|
10
|
-
timers.set(taskId, setTimeout(expire, Math.max(1, Math.min(overallRemainingMs, graceMs))));
|
|
11
|
-
}
|