agents-relay 2.0.152 → 2.0.154
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -1
- package/dist/cli-help.js +2 -1
- package/dist/dashboard-html.js +26 -5
- package/dist/dashboard.js +5 -0
- package/dist/managed-worker-input.js +1 -1
- package/dist/pool.js +1 -1
- package/dist/reconciler-runtime.js +42 -38
- package/dist/reconciler.js +10 -4
- package/dist/task-context.js +1 -1
- package/dist/task-inactivity-watchdog.js +45 -0
- package/dist/troubleshooter-qa-api.js +61 -0
- package/dist/workflow/contracts.js +1 -0
- package/dist/workflow/durable-transition.js +26 -0
- package/package.json +1 -1
- package/skills/agents-relay/SKILL.md +2 -2
- package/skills/agents-relay/agents/troubleshooter.agent.md +2 -0
- package/skills/agents-relay/contracts/event-contract.md +3 -1
- package/skills/agents-relay/contracts/task-job-contract.md +2 -0
- package/skills/agents-relay/troubleshooter-qa.md +25 -0
- package/workflow/task-lifecycle.mmd +1 -0
- package/dist/terminal-event-watchdog.js +0 -11
package/README.md
CHANGED
|
@@ -138,7 +138,7 @@ retries and daemon restarts do not create duplicate work.
|
|
|
138
138
|
Task producers must follow `skills/agents-relay/contracts/task-job-contract.md`; that shared skill contract defines story size, lifecycle events, review, and auto-merge semantics.
|
|
139
139
|
|
|
140
140
|
Execution routing is code-driven: Relay builds complete executable worker+model routes from authoritative runtime/model/resource facts, merges profile defaults with the user's explicit `[choices]`, deduplicates them, removes only impossible routes via hard gates, then calls the generic typed `decide(request, contract)` boundary exactly once with one merged choice list and the `[routing-instruction]` text forwarded verbatim. Task/resource/capability signals and the routing instruction are soft decision facts; decision failure falls back deterministically to a usable Codex default route. See [Execution Router and typed decisions](docs/worker-model-routers.md).
|
|
141
|
-
All task producers use the same story-size contract: `S`≈5 minutes, `M`≈15 minutes, and `L`≈45 minutes of
|
|
141
|
+
All task producers use the same story-size contract: `S`≈5 minutes, `M`≈15 minutes, and `L`≈45 minutes of inactivity, with `M` as the default. `high` reasoning doubles that inactivity window. Accepted `task.started` and newer `task.progress` events from the exact current execution reset it. ChatGPT Browser Worker start-ack timeout is derived as one third of the effective inactivity timeout, clamped between 1 and 5 minutes. Orchestrators and planners should choose `storySize`, not raw timeout milliseconds; `--timeout` remains only as an explicit compatibility/debug override.
|
|
142
142
|
|
|
143
143
|
Autonomous jobs can also be scheduled. The Job marker stores a five-field cron expression, IANA timezone, next run, and missed-run/concurrency policy. The daemon evaluates the SQLite-backed managed Job view and validates durable schedule facts before claiming a due slot. A repository heartbeat invokes `tick` at 10-minute resolution. See
|
|
144
144
|
[Scheduled autonomous jobs](docs/scheduled-autonomous-jobs.md) for the exact
|
package/dist/cli-help.js
CHANGED
|
@@ -135,7 +135,8 @@ Example:
|
|
|
135
135
|
'task update': `Update mutable durable Task fields in place.
|
|
136
136
|
|
|
137
137
|
Required: --task-id ID and the normal Job storage options.
|
|
138
|
-
Mutable while not RUNNING/SUCCEEDED/CANCELLED: --title, --adapter, --priority, --story-size, --input-file PATH|- or --input MARKDOWN, --project, --agent, --agent-graph-file PATH|-, --parent, --deps, --capabilities, --timeout, --max-attempts, --provider, --model, --profile, --reasoning, --cwd, --chatgpt-project, --chatgpt-tab-close-policy, --draft, --queue.
|
|
138
|
+
Mutable while not RUNNING/SUCCEEDED/CANCELLED: --title, --adapter, --priority, --story-size, --input-file PATH|- or --input MARKDOWN, --project, --agent, --agent-graph-file PATH|-, --parent, --deps, --capabilities, --timeout (inactivity window), --max-attempts, --provider, --model, --profile, --reasoning, --cwd, --chatgpt-project, --chatgpt-tab-close-policy, --draft, --queue.
|
|
139
|
+
Accepted task.started and newer current-execution task.progress events reset --timeout.
|
|
139
140
|
Use --clear-routing to remove routing metadata.
|
|
140
141
|
|
|
141
142
|
Example:
|
package/dist/dashboard-html.js
CHANGED
|
@@ -34,6 +34,7 @@ ${dashboardStyles}
|
|
|
34
34
|
<button id="jobs-tab" type="button" role="tab" aria-selected="true" aria-controls="jobs-view">Jobs</button>
|
|
35
35
|
<button id="usage-tab" type="button" role="tab" aria-selected="false" aria-controls="usage-view">Usage</button>
|
|
36
36
|
<button id="rules-tab" type="button" role="tab" aria-selected="false" aria-controls="rules-view">Router Rules</button>
|
|
37
|
+
<button id="qa-tab" type="button" role="tab" aria-selected="false" aria-controls="qa-view">Troubleshooter Q&A</button>
|
|
37
38
|
<button id="shortcuts-tab" type="button" role="tab" aria-selected="false" aria-controls="shortcuts-view">Shortcuts</button>
|
|
38
39
|
<button id="graphs-tab" type="button" role="tab" aria-selected="false" aria-controls="graphs-view">Workflow</button>
|
|
39
40
|
</nav>
|
|
@@ -83,6 +84,14 @@ ${dashboardStyles}
|
|
|
83
84
|
</section>
|
|
84
85
|
<dialog id="graph-dialog" class="graph-dialog" aria-label="Workflow graph viewer"><div class="graph-dialog-header"><strong id="graph-dialog-title">Workflow graph</strong><div class="graph-navigation"><button type="button" id="graph-zoom-out" class="refresh-button" aria-label="Zoom out">−</button><span id="graph-zoom-level" aria-live="polite">100%</span><button type="button" id="graph-zoom-in" class="refresh-button" aria-label="Zoom in">+</button><button type="button" id="graph-zoom-reset" class="refresh-button">Fit</button><button type="button" id="graph-dialog-close" class="refresh-button">Close</button></div></div><div id="graph-dialog-viewport" class="graph-dialog-viewport"></div></dialog>
|
|
85
86
|
<section id="rules-view" role="tabpanel" aria-labelledby="rules-tab" hidden><div class="panel"><div class="panel-head"><h2>Router Rules</h2><span id="rules-meta" class="subtle"></span></div><div class="panel-body"><div class="router-rules-workspace"><div class="router-rules-pane rules-editor"><h3>Rules Editor</h3><p class="rules-help">Execution Router rules with an explicit <code>[choices]</code> section (provider: model [reasoning] entries, additive to profile defaults) and a <code>[routing-instruction]</code> section forwarded verbatim to Laya. Relay deduplicates, applies hard gates, and sends one merged choice list; impossible routes remain excluded by code and Codex stays the deterministic fallback.</p><textarea id="rules-text" class="rules-textarea" spellcheck="true" aria-label="Execution router rules" placeholder="[choices]\nopenai: sol medium, luna high\nzai: glm-5.3-flash high\n\n[routing-instruction]\nPrefer zai until its quota resets; otherwise use the openai default."></textarea><div class="rules-footer"><span id="rules-status" class="subtle" aria-live="polite"></span><div><button id="rules-test" class="rules-save" type="button">Test Router</button> <button id="rules-save" class="rules-save" type="button">Save rules</button></div></div></div><div class="router-rules-pane"><div class="panel-head"><h3>Audit Log</h3><button id="audit-refresh" type="button">Refresh</button></div><div id="audit-log" class="audit-log" aria-live="polite"><p class="empty">Loading latest routing decisions…</p></div></div></div><div class="router-test-panel"><h3>Test Router Result</h3><div id="router-test-results" class="router-test-results" aria-live="polite"><p class="empty">Select a Job and Task, then run Test Router.</p></div></div></div></div></section>
|
|
87
|
+
<section id="qa-view" role="tabpanel" aria-labelledby="qa-tab" hidden>
|
|
88
|
+
<div class="panel"><div class="panel-head"><h2>Troubleshooter Q&A</h2><span class="subtle">skills/agents-relay/troubleshooter-qa.md</span></div>
|
|
89
|
+
<div class="panel-body"><div class="rules-editor">
|
|
90
|
+
<p class="rules-help">Edit Markdown directly. Troubleshooter reads this file as guidance, not workflow policy.</p>
|
|
91
|
+
<textarea id="qa-text" class="rules-textarea" spellcheck="true" aria-label="Troubleshooter Q&A Markdown"></textarea>
|
|
92
|
+
<div class="rules-footer"><span id="qa-status" class="subtle" aria-live="polite"></span><div><button id="qa-reload" class="rules-save" type="button">Reload</button> <button id="qa-save" class="rules-save" type="button">Save Q&A</button></div></div>
|
|
93
|
+
</div></div></div>
|
|
94
|
+
</section>
|
|
86
95
|
<section id="shortcuts-view" role="tabpanel" aria-labelledby="shortcuts-tab" hidden>
|
|
87
96
|
<div class="panel">
|
|
88
97
|
<div class="panel-head"><h2>ChatGPT Shortcuts</h2><span class="subtle">One per line · %1 = current repo/pr</span></div>
|
|
@@ -117,6 +126,13 @@ const usageTab=document.querySelector('#usage-tab');
|
|
|
117
126
|
const graphsTab=document.querySelector('#graphs-tab');
|
|
118
127
|
const rulesTab=document.querySelector('#rules-tab');
|
|
119
128
|
const shortcutsTab=document.querySelector('#shortcuts-tab');
|
|
129
|
+
const qaTab=document.querySelector('#qa-tab');
|
|
130
|
+
const qaView=document.querySelector('#qa-view');
|
|
131
|
+
const qaText=document.querySelector('#qa-text');
|
|
132
|
+
const qaStatus=document.querySelector('#qa-status');
|
|
133
|
+
const qaSave=document.querySelector('#qa-save');
|
|
134
|
+
let qaRevision='';
|
|
135
|
+
|
|
120
136
|
const jobsView=document.querySelector('#jobs-view');
|
|
121
137
|
const usageView=document.querySelector('#usage-view');
|
|
122
138
|
const graphsView=document.querySelector('#graphs-view');
|
|
@@ -318,7 +334,9 @@ async function testRouter(){rulesTest.disabled=true;routerTestResults.replaceChi
|
|
|
318
334
|
async function saveRules(){rulesSave.disabled=true;rulesStatus.textContent='Saving…';try{const response=await fetch('/api/execution-router-rules',{method:'PUT',headers:{'content-type':'text/plain; charset=utf-8'},body:rulesText.value});if(!response.ok)throw new Error(response.status+' '+await response.text());const rules=await response.json();rulesMeta.textContent='Saved · '+rules.hash.slice(0,12);rulesStatus.textContent='Saved. New routing decisions will load these rules.';}catch(error){rulesStatus.textContent='Save failed: '+error.message;}finally{rulesSave.disabled=false;}}
|
|
319
335
|
async function drawShortcuts(){shortcutsStatus.textContent='';shortcutsText.value=chatShortcutText();}
|
|
320
336
|
function saveShortcuts(){const value=shortcutsText.value.trim();if(value)localStorage.setItem(CHAT_SHORTCUTS_KEY,value);else localStorage.removeItem(CHAT_SHORTCUTS_KEY);shortcutsText.value=chatShortcutText();shortcutsStatus.textContent='Saved.';}
|
|
321
|
-
function
|
|
337
|
+
async function drawQa(){qaStatus.textContent='Loading…';try{const result=await loadJSON('/api/troubleshooter-qa');qaText.value=result.text;qaRevision=result.hash;qaStatus.textContent='Loaded';}catch(error){qaStatus.textContent='Load failed: '+error.message;}}
|
|
338
|
+
async function saveQa(){qaSave.disabled=true;qaStatus.textContent='Saving…';try{const response=await fetch('/api/troubleshooter-qa',{method:'PUT',headers:{'content-type':'application/json'},body:JSON.stringify({text:qaText.value,hash:qaRevision})});if(!response.ok)throw new Error(response.status+' '+await response.text());const saved=await response.json();qaRevision=saved.hash;qaStatus.textContent='Saved to Markdown file';}catch(error){qaStatus.textContent='Save failed: '+error.message;}finally{qaSave.disabled=false;}}
|
|
339
|
+
function showView(view){const isJobs=view==='jobs';const isUsage=view==='usage';const isGraphs=view==='graphs';const isRules=view==='rules';const isShortcuts=view==='shortcuts';const isQa=view==='qa';qaView.hidden=!isQa;qaTab.setAttribute('aria-selected',String(isQa));jobsView.hidden=!isJobs;jobsWorkspace.hidden=!isJobs;usageView.hidden=!isUsage;graphsView.hidden=!isGraphs;rulesView.hidden=!isRules;shortcutsView.hidden=!isShortcuts;jobsTab.setAttribute('aria-selected',String(isJobs));usageTab.setAttribute('aria-selected',String(isUsage));graphsTab.setAttribute('aria-selected',String(isGraphs));rulesTab.setAttribute('aria-selected',String(isRules));shortcutsTab.setAttribute('aria-selected',String(isShortcuts));if(isJobs)void refresh();else if(isUsage)void drawUsage();else if(isGraphs)void drawGraphs();else if(isRules)void drawRules();else if(isQa)void drawQa();else void drawShortcuts();}
|
|
322
340
|
jobsTab.addEventListener('click',()=>{location.hash=selectedJob?'#/jobs/'+encodeURIComponent(selectedJob):'#/jobs';showView('jobs');});
|
|
323
341
|
jobsRefresh.addEventListener('click',()=>void refresh(true));
|
|
324
342
|
eventsAllTasks.addEventListener('click',()=>{eventTaskId=null;clearEvents();renderEventScope();void loadEventHistory();});
|
|
@@ -337,15 +355,18 @@ document.querySelector('#graph-entry').addEventListener('click',()=>navigateGrap
|
|
|
337
355
|
const graphDialog=document.querySelector('#graph-dialog');const graphViewport=document.querySelector('#graph-dialog-viewport');let graphZoom=1;const graphCanvasHome=graphCanvas.parentNode;function setGraphZoom(value){graphZoom=Math.min(4,Math.max(.35,value));const svg=graphCanvas.querySelector('svg');if(svg){svg.style.width=(Number(svg.getAttribute('width'))||Math.max(900,svg.viewBox.baseVal.width))*graphZoom+'px';svg.style.maxWidth='none';}document.querySelector('#graph-zoom-level').textContent=Math.round(graphZoom*100)+'%';}document.querySelector('#graph-maximize').addEventListener('click',()=>{graphDialog.showModal();graphViewport.append(graphCanvas);document.querySelector('#graph-dialog-title').textContent=graphTitle.textContent;setGraphZoom(1);});function closeGraphViewer(){graphCanvasHome.insertBefore(graphCanvas,graphCanvasHome.querySelector('.graph-source-details'));setGraphZoom(1);}graphDialog.addEventListener('close',closeGraphViewer);document.querySelector('#graph-dialog-close').addEventListener('click',()=>graphDialog.close());document.querySelector('#graph-zoom-in').addEventListener('click',()=>setGraphZoom(graphZoom*1.25));document.querySelector('#graph-zoom-out').addEventListener('click',()=>setGraphZoom(graphZoom/1.25));document.querySelector('#graph-zoom-reset').addEventListener('click',()=>setGraphZoom(1));graphViewport.addEventListener('wheel',event=>{if(!event.ctrlKey&&!event.metaKey)return;event.preventDefault();setGraphZoom(graphZoom*(event.deltaY<0?1.15:1/1.15));},{passive:false});let drag=null;graphViewport.addEventListener('pointerdown',event=>{if(event.target.closest('.drillable'))return;drag={x:event.clientX,y:event.clientY,left:graphViewport.scrollLeft,top:graphViewport.scrollTop};graphViewport.setPointerCapture(event.pointerId);});graphViewport.addEventListener('pointermove',event=>{if(!drag)return;graphViewport.scrollLeft=drag.left-(event.clientX-drag.x);graphViewport.scrollTop=drag.top-(event.clientY-drag.y);});graphViewport.addEventListener('pointerup',()=>drag=null);graphViewport.addEventListener('pointercancel',()=>drag=null);
|
|
338
356
|
document.querySelector('#graph-back').addEventListener('click',()=>history.back());
|
|
339
357
|
document.querySelector('#graph-forward').addEventListener('click',()=>history.forward());
|
|
340
|
-
window.addEventListener('popstate',()=>{if(graphFromHash()){renderGraphNavigation();showView('graphs');}else if(location.hash==='#/usage')showView('usage');else if(location.hash==='#/execution-router-rules')showView('rules');else if(location.hash==='#/shortcuts')showView('shortcuts');else showView('jobs');});
|
|
341
|
-
window.addEventListener('hashchange',()=>{if(graphFromHash()){renderGraphNavigation();showView('graphs');}else if(location.hash==='#/usage')showView('usage');else if(location.hash==='#/execution-router-rules')showView('rules');else if(location.hash==='#/shortcuts')showView('shortcuts');else showView('jobs');});
|
|
358
|
+
window.addEventListener('popstate',()=>{if(graphFromHash()){renderGraphNavigation();showView('graphs');}else if(location.hash==='#/usage')showView('usage');else if(location.hash==='#/execution-router-rules')showView('rules');else if(location.hash==='#/shortcuts')showView('shortcuts');else if(location.hash==='#/troubleshooter-qa')showView('qa');else showView('jobs');});
|
|
359
|
+
window.addEventListener('hashchange',()=>{if(graphFromHash()){renderGraphNavigation();showView('graphs');}else if(location.hash==='#/usage')showView('usage');else if(location.hash==='#/execution-router-rules')showView('rules');else if(location.hash==='#/shortcuts')showView('shortcuts');else if(location.hash==='#/troubleshooter-qa')showView('qa');else showView('jobs');});
|
|
342
360
|
rulesTab.addEventListener('click',()=>{location.hash='#/execution-router-rules';showView('rules');});
|
|
361
|
+
qaTab.addEventListener('click',()=>{location.hash='#/troubleshooter-qa';showView('qa');});
|
|
362
|
+
qaSave.addEventListener('click',()=>void saveQa());
|
|
363
|
+
document.querySelector('#qa-reload').addEventListener('click',()=>{if(!qaText.value||confirm('Discard unsaved Q&A edits and reload?'))void drawQa();});
|
|
343
364
|
shortcutsTab.addEventListener('click',()=>{location.hash='#/shortcuts';showView('shortcuts');});
|
|
344
365
|
rulesSave.addEventListener('click',()=>void saveRules());
|
|
345
366
|
rulesTest.addEventListener('click',()=>void testRouter());
|
|
346
367
|
auditRefresh.addEventListener('click',()=>void drawAudit());
|
|
347
368
|
shortcutsSave.addEventListener('click',saveShortcuts);
|
|
348
|
-
window.addEventListener('keydown',event=>{if(event.key==='Tab'&&event.altKey){event.preventDefault();const views=['jobs','usage','graphs','rules','shortcuts'];const current=location.hash==='#/usage'?'usage':graphFromHash()?'graphs':location.hash==='#/execution-router-rules'?'rules':location.hash==='#/shortcuts'?'shortcuts':'jobs';const next=views[(views.indexOf(current)+1)%views.length];location.hash=next==='usage'?'#/usage':next==='graphs'?'#/graphs':next==='rules'?'#/execution-router-rules':next==='shortcuts'?'#/shortcuts':(selectedJob?'#/jobs/'+encodeURIComponent(selectedJob):'#/jobs');showView(next);}});
|
|
369
|
+
window.addEventListener('keydown',event=>{if(event.key==='Tab'&&event.altKey){event.preventDefault();const views=['jobs','usage','graphs','rules','qa','shortcuts'];const current=location.hash==='#/usage'?'usage':graphFromHash()?'graphs':location.hash==='#/execution-router-rules'?'rules':location.hash==='#/shortcuts'?'shortcuts':location.hash==='#/troubleshooter-qa'?'qa':'jobs';const next=views[(views.indexOf(current)+1)%views.length];location.hash=next==='usage'?'#/usage':next==='graphs'?'#/graphs':next==='rules'?'#/execution-router-rules':next==='shortcuts'?'#/shortcuts':next==='qa'?'#/troubleshooter-qa':(selectedJob?'#/jobs/'+encodeURIComponent(selectedJob):'#/jobs');showView(next);}});
|
|
349
370
|
function parseMessage(message){try{return JSON.parse(message.data);}catch{return null;}}
|
|
350
371
|
function isFreshEnough(incoming,current){const a=Date.parse(incoming||''),b=Date.parse(current||'');return !Number.isFinite(b)||!Number.isFinite(a)||a>=b;}
|
|
351
372
|
function applyStatePatch(patch){if(!patch?.jobId)return false;const job=jobs.find(candidate=>candidate.id===patch.jobId);if(!job)return false;if(patch.jobState&&isFreshEnough(patch.jobUpdatedAt,job.updatedAt)){job['state']=patch.jobState;if(patch.jobUpdatedAt)job.updatedAt=patch.jobUpdatedAt;}if(patch.taskId&&patch.taskState){const task=(job.tasks||[]).find(candidate=>candidate.id===patch.taskId);if(task&&isFreshEnough(patch.taskUpdatedAt,task.updatedAt)){task['state']=patch.taskState;if(patch.taskUpdatedAt)task.updatedAt=patch.taskUpdatedAt;}}return true;}
|
|
@@ -355,7 +376,7 @@ stream.onopen=()=>{connection.textContent='';connection.setAttribute('aria-label
|
|
|
355
376
|
stream.onerror=()=>{connection.textContent='';connection.setAttribute('aria-label','Event stream reconnecting');connection.className='status-pill warn';};
|
|
356
377
|
stream.onmessage=message=>{const payload=parseMessage(message);if(payload?.job){upsertFreshJob(payload.job);renderAll();void drawRuntimeSummary();}};
|
|
357
378
|
stream.addEventListener('relay',message=>{const payload=parseMessage(message);if(payload?.job)upsertFreshJob(payload.job);else if(payload?.statePatch)applyStatePatch(payload.statePatch);if(payload?.event)addEvent(payload.event);renderAll();void drawRuntimeSummary();});
|
|
358
|
-
if(location.hash==='#/usage')showView('usage');else if(graphFromHash())showView('graphs');else if(location.hash==='#/execution-router-rules')showView('rules');else if(location.hash==='#/shortcuts')showView('shortcuts');else showView('jobs');
|
|
379
|
+
if(location.hash==='#/usage')showView('usage');else if(graphFromHash())showView('graphs');else if(location.hash==='#/execution-router-rules')showView('rules');else if(location.hash==='#/shortcuts')showView('shortcuts');else if(location.hash==='#/troubleshooter-qa')showView('qa');else showView('jobs');
|
|
359
380
|
setInterval(()=>{if(location.hash==='#/usage')void drawUsage();},30000);
|
|
360
381
|
setInterval(()=>void refreshVersion(),30000);
|
|
361
382
|
async function refreshSelected(){const job=selected();if(!job){await refresh();return;}selectedJobRefresh.disabled=true;selectedJobRefresh.textContent='Refreshing…';try{const params=new URLSearchParams({id:job.id,repo:job.repository,pr:String(job.prNumber)});const updated=await loadJSON('/api/job?'+params.toString());const index=jobs.findIndex(candidate=>jobSelectionKey(candidate)===jobSelectionKey(job));if(index>=0)jobs[index]=updated;else jobs.push(updated);selectedJob=jobSelectionKey(updated);rememberSelection();renderAll();connection.textContent='';connection.setAttribute('aria-label','Live durable state connected');connection.className='status-pill ok';}catch(error){connection.textContent='';connection.setAttribute('aria-label','Durable state unavailable');connection.className='status-pill bad';}finally{selectedJobRefresh.disabled=false;selectedJobRefresh.textContent='Refresh';}}
|
package/dist/dashboard.js
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
import { inspectWorkflowSymbol } from './dashboard-workflow-inspection.js';
|
|
2
2
|
import { createServer } from 'node:http';
|
|
3
3
|
import { subscribeLocalCommits } from './local-commit-events.js';
|
|
4
|
+
import { handleTroubleshooterQa } from './troubleshooter-qa-api.js';
|
|
4
5
|
import { readFile, readdir } from 'node:fs/promises';
|
|
5
6
|
import { join } from 'node:path';
|
|
6
7
|
import { fileURLToPath } from 'node:url';
|
|
@@ -192,6 +193,10 @@ export function serveDashboard(store, port = 8787, bus, refreshMs = 300000, jobI
|
|
|
192
193
|
}
|
|
193
194
|
return;
|
|
194
195
|
}
|
|
196
|
+
if (req.url === '/api/troubleshooter-qa') {
|
|
197
|
+
await handleTroubleshooterQa(req, res);
|
|
198
|
+
return;
|
|
199
|
+
}
|
|
195
200
|
if (req.url === '/api/graphs') {
|
|
196
201
|
try {
|
|
197
202
|
const graphs = await dashboardWorkflowGraphs();
|
|
@@ -70,7 +70,7 @@ As your FIRST execution action, before reading files, editing code, running comm
|
|
|
70
70
|
you MUST publish exactly one canonical task.started event for this exact Job/Task.
|
|
71
71
|
Immediately after task.started, BEFORE reading task context or doing any implementation/review/recovery work, read the full authoritative Relay event contract at absolute path /Users/chengli/Workspace/agents-relay/skills/agents-relay/contracts/event-contract.md. If it cannot be read, publish task.failed with the blocker and stop. Do not rely only on this prompt summary or a relative path.
|
|
72
72
|
|
|
73
|
-
|
|
73
|
+
Publish task.progress only for meaningful work milestones. It is nonterminal; do not fabricate progress. Relay's Task.timeoutMs is an inactivity timeout reset only by accepted task.started or newer task.progress events for this exact execution.
|
|
74
74
|
Before returning or allowing the worker process to exit, you MUST publish exactly one canonical terminal event:
|
|
75
75
|
- task.completed only after the declared output is durable;
|
|
76
76
|
- task.failed for implementation, tool, command, test, patch, commit, push, runtime, or other execution failure that prevents completion;
|
package/dist/pool.js
CHANGED
|
@@ -232,7 +232,7 @@ export class RepositoryWorkerPool {
|
|
|
232
232
|
return;
|
|
233
233
|
if (routedWebhook)
|
|
234
234
|
return;
|
|
235
|
-
const lifecycle = ['task.started', 'task.completed', 'task.failed', 'task.blocked', 'task.unblocked', 'task.cancelled', 'review.started', 'review.approved', 'review.changes_required', 'review.blocked'].includes(event.type);
|
|
235
|
+
const lifecycle = ['task.started', 'task.progress', 'task.completed', 'task.failed', 'task.blocked', 'task.unblocked', 'task.cancelled', 'review.started', 'review.approved', 'review.changes_required', 'review.blocked'].includes(event.type);
|
|
236
236
|
// GitHub transport is observational. It must never wake lifecycle execution.
|
|
237
237
|
if (event.type === 'github.webhook')
|
|
238
238
|
return;
|
|
@@ -9,12 +9,12 @@ import { eventFor } from './events.js';
|
|
|
9
9
|
import { WorkflowEffectExecutor } from './workflow/effects.js';
|
|
10
10
|
import { ReconcileCoordinator } from './reconcile-coordinator.js';
|
|
11
11
|
import { acceptedReportedTag, eventDataWithoutTag } from './task-tag.js';
|
|
12
|
-
import { rearmTerminalEventDeadline } from './terminal-event-watchdog.js';
|
|
13
12
|
import { createWorkerStartAck } from './worker-start-ack.js';
|
|
14
13
|
import { monitorExecutionProcess } from './execution-process-tracker.js';
|
|
15
14
|
import { HostInteractiveInvocationCoordinator, hostInvocationPorts } from './interactive-invocation.js';
|
|
16
15
|
import { resumeInteractiveTask } from './interactive-resume.js';
|
|
17
16
|
import { codexFallbackCompatible, persistChatGptPlatformFailure } from './chatgpt-platform-fallback.js';
|
|
17
|
+
import { acceptTaskProgress, armInactivityWatchdog, inactivityDeadline, isCorrelatedWorkerEvent, isRecoverableWatchdogFailure } from './task-inactivity-watchdog.js';
|
|
18
18
|
const activeExecutionOwners = new Map();
|
|
19
19
|
import { withExecutionBoundary } from './execution-boundary.js';
|
|
20
20
|
export { withExecutionBoundary } from './execution-boundary.js';
|
|
@@ -84,6 +84,8 @@ export class ReconcilerRuntimeBase {
|
|
|
84
84
|
return activeOwner.applyEvent(event);
|
|
85
85
|
if (event.type === 'task.started')
|
|
86
86
|
return this.markWorkerStarted(event);
|
|
87
|
+
if (event.type === 'task.progress')
|
|
88
|
+
return this.markWorkerProgress(event);
|
|
87
89
|
if (event.type === 'task.completed') {
|
|
88
90
|
const task = (await this.store.load(event.job_id)).tasks.find(item => item.id === event.task_id);
|
|
89
91
|
const normalized = task?.agentName === 'reviewer' ? normalizeReviewerTerminalEvent(event) : null;
|
|
@@ -176,14 +178,8 @@ export class ReconcilerRuntimeBase {
|
|
|
176
178
|
owner.clearExecution(taskId, executionId);
|
|
177
179
|
return true;
|
|
178
180
|
}
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
if (!live || live.executionId !== executionId)
|
|
182
|
-
return;
|
|
183
|
-
const timer = this.terminalTimers.get(taskId);
|
|
184
|
-
if (timer)
|
|
185
|
-
clearTimeout(timer);
|
|
186
|
-
this.terminalTimers.delete(taskId);
|
|
181
|
+
armInactivityWatchdog(task, executionId) {
|
|
182
|
+
armInactivityWatchdog(task, executionId, this.terminalTimers, this.options.postExitTerminalGraceMs ?? 15_000, () => this.detach(this.timeoutTask(task.jobId, task.id, executionId), task.jobId, task.id, 'worker inactivity timeout'));
|
|
187
183
|
}
|
|
188
184
|
clearExecution(taskId, executionId) {
|
|
189
185
|
const live = this.live.get(taskId);
|
|
@@ -206,16 +202,19 @@ export class ReconcilerRuntimeBase {
|
|
|
206
202
|
eligible: (job, task) => {
|
|
207
203
|
if (!task.executionId || eventSourceId(event) !== executionEventSource(job.id, task.id, task.executionId))
|
|
208
204
|
return false;
|
|
205
|
+
if (!Number.isFinite(Date.parse(event.timestamp)) || Date.parse(event.timestamp) > Date.now())
|
|
206
|
+
return false;
|
|
209
207
|
const claimedExecution = task.executionId !== null && task.leaseOwner !== null;
|
|
210
208
|
const exactLiveExecution = live && event.job_id === live.jobId && task.executionId === live.executionId;
|
|
211
|
-
return Boolean(exactLiveExecution || claimedExecution ||
|
|
209
|
+
return Boolean(exactLiveExecution || claimedExecution || isRecoverableWatchdogFailure(task));
|
|
212
210
|
},
|
|
213
211
|
facts: (_job, task) => ({
|
|
214
|
-
recoverableExecution:
|
|
212
|
+
recoverableExecution: isRecoverableWatchdogFailure(task)
|
|
215
213
|
&& Boolean(task.executionId && eventSourceId(event) === executionEventSource(event.job_id, task.id, task.executionId)),
|
|
216
214
|
}),
|
|
217
215
|
mutate: (_job, task) => {
|
|
218
|
-
task.updatedAt = new Date().toISOString();
|
|
216
|
+
task.updatedAt = task.lastProgressAt = new Date().toISOString();
|
|
217
|
+
task.lastProgressEventAt = event.timestamp;
|
|
219
218
|
},
|
|
220
219
|
});
|
|
221
220
|
if (!transition?.decision.accepted)
|
|
@@ -227,21 +226,11 @@ export class ReconcilerRuntimeBase {
|
|
|
227
226
|
}
|
|
228
227
|
if (live?.executionId === transition.task.executionId)
|
|
229
228
|
live.workerStarted = true;
|
|
229
|
+
this.armInactivityWatchdog(transition.task, transition.task.executionId);
|
|
230
230
|
return { applied: true, reconcile: transition.from === 'FAILED', reason: transition.from === 'FAILED' ? 'task.started.recovery' : 'task.started', reconcileMode: transition.from === 'FAILED' ? 'detached' : undefined };
|
|
231
231
|
}
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
return true;
|
|
235
|
-
const error = task.error ?? '';
|
|
236
|
-
return error.includes('worker_start_timeout')
|
|
237
|
-
|| error.includes('timed out waiting for the mandatory terminal worker event')
|
|
238
|
-
|| error.includes('exited before publishing task.started')
|
|
239
|
-
|| error.includes('exited after task.started without publishing a terminal task event')
|
|
240
|
-
|| error.includes('Worker delivery failed after task.started');
|
|
241
|
-
}
|
|
242
|
-
isCorrelatedWorkerEvent(task, event) {
|
|
243
|
-
return task.executionId !== null && eventSourceId(event) === executionEventSource(event.job_id, task.id, task.executionId);
|
|
244
|
-
}
|
|
232
|
+
async markWorkerProgress(event) { const task = await acceptTaskProgress(this.taskTransitions, event); if (!task?.executionId)
|
|
233
|
+
return { applied: false, reconcile: false, reason: 'task.progress.ignored' }; this.armInactivityWatchdog(task, task.executionId); return { applied: true, reconcile: false, reason: 'task.progress' }; }
|
|
245
234
|
finalizeReviewExecution(jobId, task, event, afterCommit = true) {
|
|
246
235
|
if (!task.executionId)
|
|
247
236
|
return true;
|
|
@@ -331,29 +320,38 @@ export class ReconcilerRuntimeBase {
|
|
|
331
320
|
const task = job.tasks.find(item => item.id === taskId);
|
|
332
321
|
if (!task)
|
|
333
322
|
return;
|
|
323
|
+
if (task.adapter !== 'chatgpt') {
|
|
324
|
+
const exited = await this.taskTransitions.updateFact(jobId, taskId, current => current.executionId === executionId && ['RUNNING', 'QUEUED'].includes(current.state), current => { current.processExitedAt = new Date().toISOString(); });
|
|
325
|
+
if (exited)
|
|
326
|
+
this.armInactivityWatchdog(exited, executionId);
|
|
327
|
+
}
|
|
334
328
|
const correlated = task.executionId === executionId || task.executionHistory?.some(item => item.id === executionId);
|
|
335
329
|
if (!correlated)
|
|
336
330
|
return;
|
|
337
331
|
if (task.adapter !== 'chatgpt' && (task.executionId !== executionId || terminalExecutionClaimed(jobId, taskId, executionId)))
|
|
338
332
|
return;
|
|
339
|
-
if (task.adapter !== 'chatgpt' && task.executionId === executionId && live?.executionId === executionId)
|
|
340
|
-
rearmTerminalEventDeadline(this.terminalTimers, taskId, task.timeoutMs - (Date.now() - live.launchBarrier), this.options.postExitTerminalGraceMs ?? 15_000, () => this.detach(this.timeoutTask(jobId, taskId, executionId), jobId, taskId, 'post-exit terminal event timeout'));
|
|
341
333
|
await this.emit(eventFor(jobId, taskId, task.parentTaskId, task.adapter === 'chatgpt' ? 'task.process.async_exited' : 'task.process.exited', live?.execution.processExit ? (live.execution.processExit.exitCode === 0 ? 'succeeded' : 'failed') : (error ? 'failed' : 'succeeded'), result?.summary ?? (live?.execution.processExit?.terminationReason === 'parent-shutdown' ? 'Worker interrupted by daemon shutdown' : error ? `Worker exited: ${error}${live?.execution.processExit ? ` (exit=${live.execution.processExit.exitCode ?? 'null'}${live.execution.processExit.signal ? ` signal=${live.execution.processExit.signal}` : ''})` : ''}` : 'Worker runtime exited'), 'orchestrator', { authoritative: false, waitingForTerminalEvent: true, exitCode: live?.execution.processExit?.exitCode ?? null,
|
|
342
334
|
signal: live?.execution.processExit?.signal ?? null, terminationReason: live?.execution.processExit?.terminationReason ?? null, stderr: live?.execution.processExit?.stderr ?? '', processError: error, executionId, pid: live?.execution.pid ?? null }, { id: executionEventSource(jobId, taskId, executionId), agent: 'relay', component: 'agents-relay' }));
|
|
343
335
|
}
|
|
344
336
|
async timeoutTask(jobId, taskId, executionId) {
|
|
345
|
-
const
|
|
346
|
-
if (
|
|
347
|
-
|
|
348
|
-
|
|
337
|
+
const current = (await this.store.load(jobId)).tasks.find(task => task.id === taskId);
|
|
338
|
+
if (!current || current.executionId !== executionId || !['QUEUED', 'RUNNING'].includes(current.state))
|
|
339
|
+
return;
|
|
340
|
+
const deadline = inactivityDeadline(current, executionId, this.options.postExitTerminalGraceMs ?? 15_000);
|
|
341
|
+
if (deadline > Date.now()) {
|
|
342
|
+
this.armInactivityWatchdog(current, executionId);
|
|
343
|
+
return;
|
|
349
344
|
}
|
|
345
|
+
const live = this.live.get(taskId);
|
|
346
|
+
// Do not cancel before the serialized watchdog transition rechecks the
|
|
347
|
+
// deadline: a concurrent accepted progress event may have extended it.
|
|
350
348
|
const exit = live?.executionId === executionId && live.processSettled ? live.execution.processExit : undefined;
|
|
351
349
|
const rawError = live?.executionId === executionId && live.processSettled ? (live.processError || exit?.stderr || '').trim() : '';
|
|
352
350
|
const failedExit = Boolean(exit && (exit.exitCode !== 0 || exit.signal));
|
|
353
351
|
const failureReason = failedExit ? classifyWorkerRejection(rawError) : 'terminal_event_timeout';
|
|
354
|
-
const failureMessage = failedExit ? (rawError || `Worker exited with code ${exit?.exitCode ?? 'null'}${exit?.signal ? `, signal ${exit.signal}` : ''}`) : `Task ${taskId}
|
|
352
|
+
const failureMessage = failedExit ? (rawError || `Worker exited with code ${exit?.exitCode ?? 'null'}${exit?.signal ? `, signal ${exit.signal}` : ''}`) : `Task ${taskId} exceeded its inactivity timeout without a terminal event`;
|
|
355
353
|
const transition = await this.taskTransitions.apply(jobId, taskId, 'watchdog.failed', {
|
|
356
|
-
eligible: (_job, task) => task.executionId === executionId,
|
|
354
|
+
eligible: (_job, task) => task.executionId === executionId && inactivityDeadline(task, executionId, this.options.postExitTerminalGraceMs ?? 15_000) <= Date.now(),
|
|
357
355
|
mutate: (_job, task) => {
|
|
358
356
|
this.clearExecution(task.id, executionId);
|
|
359
357
|
task.error = failureMessage;
|
|
@@ -363,9 +361,15 @@ export class ReconcilerRuntimeBase {
|
|
|
363
361
|
},
|
|
364
362
|
});
|
|
365
363
|
if (!transition?.decision.accepted) {
|
|
366
|
-
this.
|
|
364
|
+
const latest = (await this.store.load(jobId)).tasks.find(task => task.id === taskId);
|
|
365
|
+
if (latest?.executionId === executionId && ['QUEUED', 'RUNNING'].includes(latest.state))
|
|
366
|
+
this.armInactivityWatchdog(latest, executionId);
|
|
367
367
|
return;
|
|
368
368
|
}
|
|
369
|
+
if (live?.executionId === executionId) {
|
|
370
|
+
live.controller.abort();
|
|
371
|
+
live.execution.cancel();
|
|
372
|
+
}
|
|
369
373
|
let reroutedTask = null;
|
|
370
374
|
if (reroutableWorkerRejections.has(failureReason) && transition.task.attempt < transition.task.maxAttempts) {
|
|
371
375
|
const current = (await this.store.load(jobId)).tasks.find(task => task.id === taskId);
|
|
@@ -410,19 +414,19 @@ export class ReconcilerRuntimeBase {
|
|
|
410
414
|
if (event.type === 'task.completed' && task.interactiveInvocation?.status !== undefined && task.interactiveInvocation.status !== 'approved')
|
|
411
415
|
return false; // current interactive revision requires durable approval
|
|
412
416
|
const modelBacked = task.adapter === 'codex' || task.adapter === 'chatgpt';
|
|
413
|
-
if (!trustedInternal && modelBacked && !
|
|
417
|
+
if (!trustedInternal && modelBacked && !isCorrelatedWorkerEvent(task, event)) {
|
|
414
418
|
const eventExecutionId = executionIdFromSource(eventSourceId(event), _job.id, task.id);
|
|
415
419
|
const prior = eventExecutionId ? task.executionHistory?.find(item => item.id === eventExecutionId) : undefined;
|
|
416
420
|
if (!prior || prior.attempt !== task.attempt)
|
|
417
421
|
return false;
|
|
418
422
|
}
|
|
419
423
|
const terminalWaiter = this.terminalWaiters.get(task.id);
|
|
420
|
-
return !terminalWaiter || Boolean(task.executionId && terminalWaiter.executionId === task.executionId &&
|
|
424
|
+
return !terminalWaiter || Boolean(task.executionId && terminalWaiter.executionId === task.executionId && isCorrelatedWorkerEvent(task, event));
|
|
421
425
|
},
|
|
422
426
|
facts: (_job, task) => {
|
|
423
427
|
const processExitedAfterPublishedTerminal = task.failureReason === 'worker_exit_without_terminal_event';
|
|
424
|
-
const freshEnough = processExitedAfterPublishedTerminal ||
|
|
425
|
-
return { recoverableExecution:
|
|
428
|
+
const freshEnough = processExitedAfterPublishedTerminal || isCorrelatedWorkerEvent(task, event);
|
|
429
|
+
return { recoverableExecution: isRecoverableWatchdogFailure(task) && freshEnough };
|
|
426
430
|
},
|
|
427
431
|
mutate: (job, task) => {
|
|
428
432
|
const modelBacked = task.adapter === 'codex' || task.adapter === 'chatgpt';
|
package/dist/reconciler.js
CHANGED
|
@@ -180,6 +180,9 @@ export class Reconciler extends ReconcilerReviewBase {
|
|
|
180
180
|
const taskStatesBeforeSchedule = new Map(beforeSchedule.tasks.map(task => [task.id, task.state]));
|
|
181
181
|
const scheduleBefore = JSON.stringify(beforeSchedule.schedule ?? null);
|
|
182
182
|
let scheduled = schedule(beforeSchedule);
|
|
183
|
+
for (const task of scheduled.tasks)
|
|
184
|
+
if (task.executionId && ['QUEUED', 'RUNNING'].includes(task.state))
|
|
185
|
+
this.armInactivityWatchdog(task, task.executionId);
|
|
183
186
|
for (const task of scheduled.tasks) {
|
|
184
187
|
if (taskStatesBeforeSchedule.get(task.id) !== task.state)
|
|
185
188
|
await this.store.saveTask(task);
|
|
@@ -366,6 +369,9 @@ export class Reconciler extends ReconcilerReviewBase {
|
|
|
366
369
|
freshTask.leaseOwner = this.options.owner;
|
|
367
370
|
freshTask.leaseExpiresAt = new Date(startedAt + leaseMs).toISOString();
|
|
368
371
|
freshTask.executionId = randomUUID();
|
|
372
|
+
delete freshTask.lastProgressAt;
|
|
373
|
+
delete freshTask.lastProgressEventAt;
|
|
374
|
+
delete freshTask.processExitedAt;
|
|
369
375
|
freshTask.executionHistory = [...(freshTask.executionHistory ?? []), {
|
|
370
376
|
id: freshTask.executionId,
|
|
371
377
|
attempt: freshTask.attempt,
|
|
@@ -396,6 +402,9 @@ export class Reconciler extends ReconcilerReviewBase {
|
|
|
396
402
|
durable.leaseOwner = this.options.owner;
|
|
397
403
|
durable.leaseExpiresAt = new Date(startedAt + leaseMs).toISOString();
|
|
398
404
|
durable.executionId = randomUUID();
|
|
405
|
+
delete durable.lastProgressAt;
|
|
406
|
+
delete durable.lastProgressEventAt;
|
|
407
|
+
delete durable.processExitedAt;
|
|
399
408
|
durable.executionHistory = [...(durable.executionHistory ?? []), {
|
|
400
409
|
id: durable.executionId, attempt: durable.attempt, startedAt: new Date(startedAt).toISOString(),
|
|
401
410
|
}].slice(-8);
|
|
@@ -514,10 +523,7 @@ export class Reconciler extends ReconcilerReviewBase {
|
|
|
514
523
|
transition: { graph: 'task-lifecycle', from: 'QUEUED', to: 'QUEUED', actions: ['effect.launchTask'] },
|
|
515
524
|
data: { adapter: adapter.id, attempt: task.attempt, model: task.routing?.model ?? null },
|
|
516
525
|
});
|
|
517
|
-
|
|
518
|
-
this.detach(this.timeoutTask(job.id, task.id, durableExecutionId), job.id, task.id, 'worker terminal-event timeout');
|
|
519
|
-
}, Math.max(1, task.timeoutMs));
|
|
520
|
-
this.terminalTimers.set(task.id, terminalTimer);
|
|
526
|
+
this.armInactivityWatchdog(task, durableExecutionId);
|
|
521
527
|
this.detach(execution.promise.then(result => this.runtimeSettled(job.id, task.id, durableExecutionId, result, null), error => error instanceof StartAckTimeout
|
|
522
528
|
? this.failLaunch(job.id, task.id, durableExecutionId, error, 'start-ack')
|
|
523
529
|
: error instanceof LaunchFailure || error instanceof PreSubmissionRejection
|
package/dist/task-context.js
CHANGED
|
@@ -45,5 +45,5 @@ THEN read the entire authoritative event contract from /Users/chengli/Workspace/
|
|
|
45
45
|
THEN retrieve the authoritative Task Context:
|
|
46
46
|
agents-relay task context --repo ${repository} --id ${jobId} --task-id ${taskId} --execution-id ${executionId}
|
|
47
47
|
Verify the execution ID. Read Task input, dependencies, Plan, project context, and output requirements from that response. Never guess or create a new Job. Continue in this thread; Relay owns completion and review.
|
|
48
|
-
Follow the file's lifecycle contract, including progress
|
|
48
|
+
Follow the file's lifecycle contract, including optional meaningful progress and exactly one task.completed or task.failed before exiting. Accepted current-execution task.started and newer task.progress events reset Task.timeoutMs; duplicate, stale, or out-of-order progress does not. If context is unavailable publish task.failed with the blocker.`;
|
|
49
49
|
}
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
import { executionEventSource, eventSourceId } from './execution-event-identity.js';
|
|
2
|
+
export function isRecoverableWatchdogFailure(task) {
|
|
3
|
+
if (['worker_start_timeout', 'terminal_event_timeout', 'worker_exit_before_start', 'worker_exit_without_terminal_event', 'worker_delivery_after_start'].includes(task.failureReason ?? ''))
|
|
4
|
+
return true;
|
|
5
|
+
const error = task.error ?? '';
|
|
6
|
+
return error.includes('worker_start_timeout') || error.includes('timed out waiting for the mandatory terminal worker event') || error.includes('exceeded its inactivity timeout')
|
|
7
|
+
|| error.includes('exited before publishing task.started') || error.includes('exited after task.started without publishing a terminal task event')
|
|
8
|
+
|| error.includes('Worker delivery failed after task.started');
|
|
9
|
+
}
|
|
10
|
+
export function isCorrelatedWorkerEvent(task, event) {
|
|
11
|
+
return task.executionId !== null && eventSourceId(event) === executionEventSource(event.job_id, task.id, task.executionId);
|
|
12
|
+
}
|
|
13
|
+
export function inactivityDeadline(task, executionId, postExitGraceMs) {
|
|
14
|
+
const progressAt = Date.parse(task.lastProgressAt ?? task.executionHistory?.find(item => item.id === executionId)?.startedAt ?? task.updatedAt);
|
|
15
|
+
const timeoutAt = (Number.isFinite(progressAt) ? progressAt : Date.now()) + Math.max(1, task.timeoutMs);
|
|
16
|
+
const exitAt = task.adapter !== 'chatgpt' && task.processExitedAt ? Date.parse(task.processExitedAt) : NaN;
|
|
17
|
+
return Number.isFinite(exitAt) ? Math.min(timeoutAt, exitAt + postExitGraceMs) : timeoutAt;
|
|
18
|
+
}
|
|
19
|
+
export function armInactivityWatchdog(task, executionId, timers, postExitGraceMs, expire) {
|
|
20
|
+
const existing = timers.get(task.id);
|
|
21
|
+
if (existing)
|
|
22
|
+
clearTimeout(existing);
|
|
23
|
+
timers.delete(task.id);
|
|
24
|
+
if (!task.jobId || task.executionId !== executionId || !['QUEUED', 'RUNNING'].includes(task.state))
|
|
25
|
+
return;
|
|
26
|
+
const timer = setTimeout(expire, Math.max(1, inactivityDeadline(task, executionId, postExitGraceMs) - Date.now()));
|
|
27
|
+
timer.unref();
|
|
28
|
+
timers.set(task.id, timer);
|
|
29
|
+
}
|
|
30
|
+
export async function acceptTaskProgress(transitions, event) {
|
|
31
|
+
const parsedAt = Date.parse(event.timestamp);
|
|
32
|
+
const receivedAt = Date.now();
|
|
33
|
+
const transition = await transitions.apply(event.job_id, event.task_id, 'task.progress', {
|
|
34
|
+
eligible: (job, task) => task.state === 'RUNNING'
|
|
35
|
+
&& Boolean(task.executionId && eventSourceId(event) === executionEventSource(job.id, task.id, task.executionId))
|
|
36
|
+
&& Number.isFinite(parsedAt) && parsedAt <= receivedAt
|
|
37
|
+
&& (!task.lastProgressEventAt || parsedAt > Date.parse(task.lastProgressEventAt)),
|
|
38
|
+
mutate: (_job, task) => {
|
|
39
|
+
task.lastProgressAt = new Date(receivedAt).toISOString();
|
|
40
|
+
task.lastProgressEventAt = event.timestamp;
|
|
41
|
+
task.updatedAt = task.lastProgressAt;
|
|
42
|
+
},
|
|
43
|
+
});
|
|
44
|
+
return transition?.decision.accepted ? transition.task : null;
|
|
45
|
+
}
|
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
import { readFile, writeFile } from 'node:fs/promises';
|
|
2
|
+
import { createHash } from 'node:crypto';
|
|
3
|
+
import { fileURLToPath } from 'node:url';
|
|
4
|
+
const FILE = fileURLToPath(new URL('../skills/agents-relay/troubleshooter-qa.md', import.meta.url));
|
|
5
|
+
const qaHash = (content) => createHash('sha256').update(content).digest('hex');
|
|
6
|
+
export async function handleTroubleshooterQa(req, res) {
|
|
7
|
+
try {
|
|
8
|
+
const current = await readFile(FILE, 'utf8');
|
|
9
|
+
if (req.method === 'GET') {
|
|
10
|
+
res.writeHead(200, { 'content-type': 'application/json', 'cache-control': 'no-store' });
|
|
11
|
+
res.end(JSON.stringify({ text: current, hash: qaHash(current) }));
|
|
12
|
+
return;
|
|
13
|
+
}
|
|
14
|
+
if (req.method === 'PUT') {
|
|
15
|
+
if (req.headers['content-type']?.split(';')[0] !== 'application/json') {
|
|
16
|
+
res.writeHead(415);
|
|
17
|
+
res.end('Expected JSON');
|
|
18
|
+
return;
|
|
19
|
+
}
|
|
20
|
+
const chunks = [];
|
|
21
|
+
let size = 0;
|
|
22
|
+
for await (const chunk of req) {
|
|
23
|
+
const buffer = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk);
|
|
24
|
+
size += buffer.length;
|
|
25
|
+
if (size > 128 * 1024) {
|
|
26
|
+
res.writeHead(413);
|
|
27
|
+
res.end('Q&A too large');
|
|
28
|
+
return;
|
|
29
|
+
}
|
|
30
|
+
chunks.push(buffer);
|
|
31
|
+
}
|
|
32
|
+
const body = JSON.parse(Buffer.concat(chunks).toString('utf8'));
|
|
33
|
+
if (!body || typeof body !== 'object' || !('text' in body) || !('hash' in body) || typeof body.text !== 'string' || typeof body.hash !== 'string') {
|
|
34
|
+
res.writeHead(400);
|
|
35
|
+
res.end('Invalid Q&A payload');
|
|
36
|
+
return;
|
|
37
|
+
}
|
|
38
|
+
if (body.hash !== qaHash(current)) {
|
|
39
|
+
res.writeHead(409);
|
|
40
|
+
res.end('Q&A changed on disk; reload before saving');
|
|
41
|
+
return;
|
|
42
|
+
}
|
|
43
|
+
if (Buffer.byteLength(body.text) > 128 * 1024) {
|
|
44
|
+
res.writeHead(413);
|
|
45
|
+
res.end('Q&A too large');
|
|
46
|
+
return;
|
|
47
|
+
}
|
|
48
|
+
await writeFile(FILE, body.text, 'utf8');
|
|
49
|
+
res.writeHead(200, { 'content-type': 'application/json', 'cache-control': 'no-store' });
|
|
50
|
+
res.end(JSON.stringify({ hash: qaHash(body.text) }));
|
|
51
|
+
return;
|
|
52
|
+
}
|
|
53
|
+
res.writeHead(405, { allow: 'GET, PUT' });
|
|
54
|
+
res.end('Method not allowed');
|
|
55
|
+
}
|
|
56
|
+
catch (error) {
|
|
57
|
+
res.writeHead(500);
|
|
58
|
+
res.end(error instanceof Error ? error.message : String(error));
|
|
59
|
+
}
|
|
60
|
+
return;
|
|
61
|
+
}
|
|
@@ -52,6 +52,32 @@ export class DurableTaskTransitionRuntime {
|
|
|
52
52
|
return { job, task, from, decision };
|
|
53
53
|
});
|
|
54
54
|
}
|
|
55
|
+
/** Persist a non-lifecycle fact without racing the serialized terminal transitions. */
|
|
56
|
+
async updateFact(jobId, taskId, eligible, mutate) {
|
|
57
|
+
return this.withLock(`${jobId}/${taskId}`, async () => {
|
|
58
|
+
let accepted = false;
|
|
59
|
+
const apply = (job) => {
|
|
60
|
+
const task = job.tasks.find(item => item.id === taskId);
|
|
61
|
+
if (!task || !eligible(task))
|
|
62
|
+
return job;
|
|
63
|
+
const previous = Date.parse(task.updatedAt);
|
|
64
|
+
mutate(task);
|
|
65
|
+
task.updatedAt = new Date(Math.max(Date.now(), Number.isFinite(previous) ? previous + 1 : 0)).toISOString();
|
|
66
|
+
accepted = true;
|
|
67
|
+
return job;
|
|
68
|
+
};
|
|
69
|
+
if (this.store.mutateJob) {
|
|
70
|
+
const updated = await this.store.mutateJob(jobId, apply);
|
|
71
|
+
return accepted ? updated.tasks.find(item => item.id === taskId) ?? null : null;
|
|
72
|
+
}
|
|
73
|
+
const job = apply(await this.store.load(jobId));
|
|
74
|
+
if (!accepted)
|
|
75
|
+
return null;
|
|
76
|
+
const task = job.tasks.find(item => item.id === taskId);
|
|
77
|
+
await this.store.saveTask(task);
|
|
78
|
+
return task;
|
|
79
|
+
});
|
|
80
|
+
}
|
|
55
81
|
async withLock(key, work) {
|
|
56
82
|
const previous = this.locks.get(key);
|
|
57
83
|
let release;
|
package/package.json
CHANGED
|
@@ -56,9 +56,9 @@ Agents Relay maps those terminal events to durable state:
|
|
|
56
56
|
- `task.failed` → `FAILED`
|
|
57
57
|
Relay may authoritatively move a Task to `BLOCKED` for an external condition and may resume it only through an authorized unblock operation. A human response is an append-only Task Comment and resumes the blocked Troubleshooter, not the failed original Task. Task input remains stable; recovery agents receive current authoritative state and relevant comments as execution context.
|
|
58
58
|
|
|
59
|
-
A model-backed task must not launch without an event bus. If no
|
|
59
|
+
A model-backed task must not launch without an event bus. If no accepted `task.started` or newer current-execution `task.progress` arrives within `Task.timeoutMs`, Relay records an inactivity timeout failure. A Codex process exit or ChatGPT submission receipt alone never changes the task to a terminal state.
|
|
60
60
|
|
|
61
|
-
Worker process exit is observational only. Relay does not perform an exit-time terminal check or synthesize task failure from process exit;
|
|
61
|
+
Worker process exit is observational only. Relay does not perform an exit-time terminal check or synthesize task failure from process exit; for synchronous workers it caps the remaining inactivity wait at the shorter post-exit terminal grace.
|
|
62
62
|
|
|
63
63
|
## Managed GitHub jobs
|
|
64
64
|
|
|
@@ -15,6 +15,8 @@ Your goal is not to explain the failure. Your goal is to make the original task
|
|
|
15
15
|
|
|
16
16
|
Read and follow the shared Task / Job contract and orchestrator P0/self-job workflow before acting.
|
|
17
17
|
|
|
18
|
+
Read `skills/agents-relay/troubleshooter-qa.md` for user-editable best practices before diagnosis. Treat its answers as guidance only, below current evidence and authoritative contracts. After a verified recovery, add or refine a concise Q&A entry when it provides reusable knowledge; never include secrets or unverified claims.
|
|
19
|
+
|
|
18
20
|
## Recovery loop
|
|
19
21
|
|
|
20
22
|
1. Retrieve the durable original Job and Task from the authoritative Relay store with `agents-relay task context`; GitHub is only an optional projection and is never lifecycle authority.
|
|
@@ -4,7 +4,9 @@
|
|
|
4
4
|
|
|
5
5
|
## Producer contract
|
|
6
6
|
|
|
7
|
-
Every Implementation, Reviewer, and Troubleshooter execution publishes `task.started`, optional `task.progress
|
|
7
|
+
Every Implementation, Reviewer, and Troubleshooter execution publishes `task.started`, optional `task.progress`, then **exactly one** agent terminal event:
|
|
8
|
+
|
|
9
|
+
`task.started` and `task.progress` reset the Task inactivity timeout only when Relay accepts the event for the exact current execution. Progress is nonterminal, must be newer than the last accepted progress event, and cannot extend a terminal or stale execution.
|
|
8
10
|
|
|
9
11
|
- `task.completed`: assigned responsibility finished and its role-specific output is durable, regardless of whether a reviewed artifact passed.
|
|
10
12
|
- `task.failed`: assigned responsibility could not be fulfilled; include failure phase, reason, and evidence in `data`.
|
|
@@ -22,6 +22,8 @@ Ordinary `job create` requires standalone objective Markdown. An external displa
|
|
|
22
22
|
Any task producer — orchestrator, planner, or nested task-producing agent — must follow this contract.
|
|
23
23
|
Task producers choose semantic task properties such as objective, dependencies, worker requirements, priority, and `storySize`. They do not normally choose an adapter, provider/model, or raw timeout milliseconds.
|
|
24
24
|
|
|
25
|
+
`Task.timeoutMs` is the inactivity window. Only an accepted `task.started` or newer `task.progress` event for the exact current execution resets it; duplicate, stale, and out-of-order progress does not.
|
|
26
|
+
|
|
25
27
|
### Planning graph and semantic Task tree
|
|
26
28
|
|
|
27
29
|
The planning graph expresses execution flow. The Task tree expresses semantic ownership. `dependencies` answer which durable outcomes must succeed before a Task can run; `parentTaskId` answers which durable artifact or milestone owns the Task. These are independent relationships.
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# Troubleshooter Q&A
|
|
2
|
+
|
|
3
|
+
A simple, user-editable reference for recurring failures and proven recovery practices.
|
|
4
|
+
|
|
5
|
+
Troubleshooter: read this file before diagnosis, but always inspect current task events and logs. Answers are guidance, not authority to retry, unblock, change routing, or change workflow state. The canonical workflow graphs and contracts remain authoritative.
|
|
6
|
+
|
|
7
|
+
Users can add or edit entries directly. After a recovery is verified, the Troubleshooter may add a short entry or improve an existing answer. Do not write secrets, tokens, or private log contents here. Keep unverified hypotheses clearly labeled.
|
|
8
|
+
|
|
9
|
+
## Q: Browser Workspace validation says COMPOSER_NOT_READY. Does that mean the user is logged out?
|
|
10
|
+
|
|
11
|
+
**A:** No. This code reports UI readiness, not authentication. Use the platform's validated session and inspect worker authentication and delivery evidence separately. Do not infer login failure from a disabled Send button or DOM snapshot.
|
|
12
|
+
|
|
13
|
+
## Q: A worker fails or times out. Should Troubleshooter immediately retry?
|
|
14
|
+
|
|
15
|
+
**A:** No. Read the original task's correlated lifecycle events, attempts, routing, and relevant logs. Diagnose the actual cause first. Only recover the original Task after verifying that the blocker is resolved, using the canonical recovery path.
|
|
16
|
+
|
|
17
|
+
## Q: A shared runtime or tool defect blocks several tasks. What should Troubleshooter do?
|
|
18
|
+
|
|
19
|
+
**A:** Reuse an existing blocker Job when possible. Otherwise create the owning P0/self-job through Agents Relay. Verify the repair and any required release or deployment before recovering dependent tasks. For deferred repair, register the durable blocker relation instead of keeping a worker alive.
|
|
20
|
+
|
|
21
|
+
## Q: When can a BLOCKED task resume?
|
|
22
|
+
|
|
23
|
+
**A:** Only after an authoritative unblock operation or verified blocker-Job resolution. A Q&A answer cannot unblock a task.
|
|
24
|
+
|
|
25
|
+
<!-- Add more entries below using: ## Q: ... / **A:** ... -->
|
|
@@ -151,6 +151,7 @@ leaseFailQueuedClearExecution --> failed
|
|
|
151
151
|
queued -->|on: cancel| cancelled
|
|
152
152
|
|
|
153
153
|
running -->|on: task.blocked| blocked
|
|
154
|
+
running -->|on: task.progress| running
|
|
154
155
|
running -->|on: review.approved| reviewApproveRunningClearError
|
|
155
156
|
reviewApproveRunningClearError --> reviewApproveRunningClearReason
|
|
156
157
|
reviewApproveRunningClearReason --> succeeded
|
|
@@ -1,11 +0,0 @@
|
|
|
1
|
-
/**
|
|
2
|
-
* Replace the overall worker deadline with a shorter post-exit deadline.
|
|
3
|
-
* Execution outcomes remain owned by the authoritative task lifecycle graph.
|
|
4
|
-
* Never extend the original overall timeout.
|
|
5
|
-
*/
|
|
6
|
-
export function rearmTerminalEventDeadline(timers, taskId, overallRemainingMs, graceMs, expire) {
|
|
7
|
-
const existing = timers.get(taskId);
|
|
8
|
-
if (existing)
|
|
9
|
-
clearTimeout(existing);
|
|
10
|
-
timers.set(taskId, setTimeout(expire, Math.max(1, Math.min(overallRemainingMs, graceMs))));
|
|
11
|
-
}
|