agents-relay 2.0.195 → 2.0.197
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/adapters.js +10 -6
- package/dist/agent-content-resolver.js +28 -1
- package/dist/agent-prompt-builder.js +73 -3
- package/dist/browser-worker-runtime.js +40 -22
- package/dist/cli-help.js +20 -5
- package/dist/cli-prompt.js +132 -0
- package/dist/cli-shutdown.js +3 -0
- package/dist/cli-support.js +47 -5
- package/dist/cli.js +19 -30
- package/dist/events-bus-runtime.js +58 -8
- package/dist/local-job-registry.js +9 -5
- package/dist/managed-worker-input.js +48 -74
- package/dist/planner.js +16 -10
- package/dist/reconciler.js +3 -7
- package/dist/troubleshooter.js +2 -2
- package/dist/z-adapter.js +16 -8
- package/package.json +3 -1
- package/skills/agents-relay/SKILL.md +25 -111
- package/skills/agents-relay/agents/implementation.agent.md +1 -2
- package/skills/agents-relay/agents/planner.agent.md +12 -12
- package/skills/agents-relay/agents/reviewer.agent.md +1 -1
- package/skills/agents-relay/agents/troubleshooter.agent.md +6 -6
- package/skills/agents-relay/contracts/event-contract.md +6 -3
- package/skills/agents-relay/contracts/orchestrator-workflows.md +6 -4
- package/skills/agents-relay/contracts/task-job-contract.md +23 -20
- package/skills/agents-relay/contracts/task-recovery-flows.md +15 -35
- package/skills/agents-relay/evals/README.md +15 -0
- package/skills/agents-relay/evals/capabilities.json +229 -0
|
@@ -1,36 +1,24 @@
|
|
|
1
|
-
# Task Failure Recovery and Human Escalation —
|
|
1
|
+
# Task Failure Recovery and Human Escalation — Current Contract
|
|
2
2
|
|
|
3
|
-
**Status:** Current recovery contract.
|
|
3
|
+
**Status:** Current recovery contract. Query Task state, recovery status, and comments through the matching Relay CLI.
|
|
4
4
|
|
|
5
5
|
## Principles and ownership
|
|
6
6
|
|
|
7
7
|
- **Events Bus is neutral.** Its generic event envelope consists of event identity, type, time, and opaque client-defined data (plus any transport metadata needed by the bus). It must not understand Relay Jobs, Tasks, roles, or verdicts.
|
|
8
|
-
- **One agent Task event contract** applies to Implementation, Reviewer, and Troubleshooter: optional progress
|
|
8
|
+
- **One agent Task event contract** applies to Implementation, Reviewer, and Troubleshooter: `task.started`, optional `task.progress`, then exactly one agent terminal event, `task.completed` or `task.failed`. Role-specific results belong in `data`. A review with findings is a *completed* review, not a failed agent execution.
|
|
9
9
|
- **`task.blocked` and `task.unblocked` are authoritative Relay lifecycle transitions**, not ordinary agent terminal events. A human may authorize unblocking a Troubleshooter but does not directly unblock the failed original Task.
|
|
10
10
|
- **Troubleshooter owns diagnosis and recovery decisions.** It chooses technical repair/retry, blocker-Job dependency, or human escalation. There is no separate Escalator agent.
|
|
11
|
-
- **Relay owns transitions and dispatch.** Troubleshooter requests authorized operations
|
|
11
|
+
- **Relay owns transitions and dispatch.** Troubleshooter requests authorized operations through Relay CLI, then checks the CLI response and fresh Task Context. Event transport acceptance alone is not confirmation of a Task transition.
|
|
12
12
|
- Preserve the original failure, exact execution correlation, evidence, and idempotency throughout retries and resumes.
|
|
13
13
|
|
|
14
|
-
##
|
|
14
|
+
## External recovery interface
|
|
15
15
|
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
T["Troubleshooter"] --> U
|
|
23
|
-
U --> B["Neutral Events Bus"]
|
|
24
|
-
B --> C["Relay event consumer"]
|
|
25
|
-
C --> S["Validate execution identity and role-specific data"]
|
|
26
|
-
S --> W["Relay workflow engine / reconciler"]
|
|
27
|
-
W --> D[("Durable Task and Job state")]
|
|
28
|
-
W --> R["Execution / recovery / human-interaction operations"]
|
|
29
|
-
```
|
|
30
|
-
|
|
31
|
-
- Agents publish `task.completed` or `task.failed` as terminal execution outcomes; Reviewer verdicts and Troubleshooter recovery evidence are results inside `data`.
|
|
32
|
-
- `task.blocked` / `task.unblocked` describe Relay-authorized state changes, not a third agent terminal outcome.
|
|
33
|
-
- Transport does not interpret verdicts, execution IDs, blocker relations, or recovery decisions.
|
|
16
|
+
Task agents publish their required Task events via the canonical Neo Events Bus
|
|
17
|
+
protocol. All Relay status reads and authorized changes use the Prompt-pinned
|
|
18
|
+
Relay CLI (`task context`, `task show`, `task events`, `task recover`,
|
|
19
|
+
`task escalate`, or other documented commands as appropriate). Event transport
|
|
20
|
+
acceptance does not confirm a Task transition. Never access Relay backing
|
|
21
|
+
storage, PR markers, or private HTTP endpoints to infer or change state.
|
|
34
22
|
|
|
35
23
|
## Overall flow — Task outcome and recovery branches
|
|
36
24
|
|
|
@@ -78,7 +66,7 @@ flowchart TD
|
|
|
78
66
|
```
|
|
79
67
|
|
|
80
68
|
1. Original agent reports `task.failed` with useful diagnostic `data`; Relay durably records `FAILED`.
|
|
81
|
-
2.
|
|
69
|
+
2. Relay assigns a correlated Troubleshooter Task. Troubleshooter examines durable Task history and technical evidence before choosing a resolution.
|
|
82
70
|
3. For a technical remedy, Troubleshooter repairs/validates, requests authorized retry, confirms the original Task requeued, and completes **its own** Task.
|
|
83
71
|
4. For a blocker Job, Troubleshooter registers a durable dependency, then completes **its own** Task; the dependency mechanism later wakes the original Task after verified resolution.
|
|
84
72
|
5. Only a **real user action** calls for human escalation (Flow 2). Missing credentials that can be found or configured automatically, crashes, and retries are Troubleshooter work, not escalation.
|
|
@@ -127,21 +115,13 @@ flowchart TD
|
|
|
127
115
|
|
|
128
116
|
`BLOCKED` is a durable Task state and `task.blocked` may be published by Relay as a state-change fact; it is **not** an agent-selected terminal outcome. Human unblocking resumes the blocked Troubleshooter Task rather than terminating or re-creating its recovery responsibility.
|
|
129
117
|
|
|
130
|
-
## Required follow-up implementation alignment
|
|
131
|
-
|
|
132
|
-
- Align `skills/agents-relay/SKILL.md` and role instructions with the two agent terminal events; remove agent-directed `task.blocked` emission.
|
|
133
|
-
- Align escalation graph, CLI, reconciler, and unblock commands so human intervention blocks **Troubleshooter**, not the original failed Task; preserve the original failure and correlated recovery context.
|
|
134
|
-
- Ensure the blocked Troubleshooter can resume after an authorized human unblock rather than being treated as a finished terminal execution.
|
|
135
|
-
- Add integrated checks: technical retry, blocker-Job handoff, human request/block, human unblock/resume, failed verification/re-escalation, stale/duplicate responses, and correct original-Task retry.
|
|
136
|
-
|
|
137
118
|
## Durable recovery history
|
|
138
119
|
|
|
139
120
|
`Task.input` remains the stable objective and original evidence. `Task.resolution` and
|
|
140
121
|
`Task.escalation` remain the current authoritative states; comments are append-only
|
|
141
|
-
history, not
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
`resolutionHistory` field.
|
|
122
|
+
history, not current state. Use `task context` through the matching Relay CLI
|
|
123
|
+
to read the current recovery state and history; do not read internal storage
|
|
124
|
+
or invent a second recovery-history source.
|
|
145
125
|
|
|
146
126
|
Human responses are comments. Resuming a Troubleshooter must not rewrite its
|
|
147
127
|
input or append a stale “Required resolution condition” as new objective text.
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# Agents Relay Skill capability evaluations
|
|
2
|
+
|
|
3
|
+
These cases evaluate **agent decisions**, not whether the Relay implementation functions. They are derived from the user capabilities: route direct vs durable work, create/adopt, plan, inspect/review, recover/escalate, execute with a pinned context, and manage services.
|
|
4
|
+
|
|
5
|
+
Use `capabilities.json` as a provider-neutral case set. For each case, supply the active `SKILL.md` and `request` to the target agent, ask for a **dry-run** decision (no CLI side effects), and normalize its answer to:
|
|
6
|
+
|
|
7
|
+
```json
|
|
8
|
+
{"decision":"direct|create|adopt|inspect|plan|task_create|review|recover|escalate|service|worker_context","action":"intended CLI subcommand or direct operation","verification":"required evidence","avoid":"unsafe assumption to avoid"}
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
Check `expected.decision` and, when present, `expected.command` / `expected.flags` deterministically. An answer fails when it recommends any `forbidden` behavior. Use a qualitative rubric only if an answer is ambiguous despite having correct structured fields. Cross-provider execution may use any authenticated harness but must not silently change case expectations.
|
|
12
|
+
|
|
13
|
+
`test/agents-relay-skill-evals.test.js` checks **fixture integrity and coverage only**, not model compliance. The generated-prompt L4 tests check prompt references and runtime bootstrap, not the model's semantic decisions. Do not report these as live-provider behavioral passes.
|
|
14
|
+
|
|
15
|
+
Separately, real integration tests must verify CLI create/show, generated role prompts on each supported adapter, event correlation and start/progress/terminal state, reviewer changes-required, planner apply receipt, recovery, and failed/unknown outcomes. Never run real integration tests with the canonical user Job store or create remote PRs without explicit authorization.
|
|
@@ -0,0 +1,229 @@
|
|
|
1
|
+
{
|
|
2
|
+
"schema": "agents-relay-skill-capabilities-v1",
|
|
3
|
+
"instruction": "Use SKILL.md with the scenario. Return the intended action, CLI subcommand or direct tool, what must be checked, and what must not be assumed. Do not perform side effects.",
|
|
4
|
+
"output_schema": {
|
|
5
|
+
"decision": "direct|create|adopt|inspect|plan|task_create|review|recover|escalate|service|worker_context",
|
|
6
|
+
"action": "string",
|
|
7
|
+
"verification": "string",
|
|
8
|
+
"avoid": "string"
|
|
9
|
+
},
|
|
10
|
+
"cases": [
|
|
11
|
+
{
|
|
12
|
+
"id": "direct-one-shot",
|
|
13
|
+
"capability": "route",
|
|
14
|
+
"request": "Fix a typo in one local note in this current conversation; no job or delegation needed.",
|
|
15
|
+
"expected": {
|
|
16
|
+
"decision": "direct"
|
|
17
|
+
},
|
|
18
|
+
"forbidden": [
|
|
19
|
+
"job create"
|
|
20
|
+
]
|
|
21
|
+
},
|
|
22
|
+
{
|
|
23
|
+
"id": "managed-multiphase",
|
|
24
|
+
"capability": "route",
|
|
25
|
+
"request": "Implement an API change with independent review, a recovery path, and durable progress across sessions.",
|
|
26
|
+
"expected": {
|
|
27
|
+
"decision": "create",
|
|
28
|
+
"command": "job create",
|
|
29
|
+
"flags": [
|
|
30
|
+
"--objective-file",
|
|
31
|
+
"--initial-plan"
|
|
32
|
+
]
|
|
33
|
+
},
|
|
34
|
+
"forbidden": [
|
|
35
|
+
"raw GitHub PR"
|
|
36
|
+
]
|
|
37
|
+
},
|
|
38
|
+
{
|
|
39
|
+
"id": "create-only",
|
|
40
|
+
"capability": "create",
|
|
41
|
+
"request": "Store this full objective as a new Relay Job, but do not start planning or executing.",
|
|
42
|
+
"expected": {
|
|
43
|
+
"decision": "create",
|
|
44
|
+
"command": "job create",
|
|
45
|
+
"flags": [
|
|
46
|
+
"--objective-file"
|
|
47
|
+
]
|
|
48
|
+
},
|
|
49
|
+
"forbidden": [
|
|
50
|
+
"--initial-plan",
|
|
51
|
+
"task create"
|
|
52
|
+
]
|
|
53
|
+
},
|
|
54
|
+
{
|
|
55
|
+
"id": "existing-pr",
|
|
56
|
+
"capability": "create",
|
|
57
|
+
"request": "Adopt an existing GitHub PR into Relay, preserving its history.",
|
|
58
|
+
"expected": {
|
|
59
|
+
"decision": "adopt",
|
|
60
|
+
"command": "job adopt"
|
|
61
|
+
},
|
|
62
|
+
"forbidden": [
|
|
63
|
+
"job create",
|
|
64
|
+
"raw GitHub PR create"
|
|
65
|
+
]
|
|
66
|
+
},
|
|
67
|
+
{
|
|
68
|
+
"id": "local-job",
|
|
69
|
+
"capability": "create",
|
|
70
|
+
"request": "Coordinate a durable local-only file workflow without GitHub authentication.",
|
|
71
|
+
"expected": {
|
|
72
|
+
"decision": "create",
|
|
73
|
+
"command": "job create",
|
|
74
|
+
"flags": [
|
|
75
|
+
"--no-github"
|
|
76
|
+
]
|
|
77
|
+
},
|
|
78
|
+
"forbidden": [
|
|
79
|
+
"GitHub authentication required"
|
|
80
|
+
]
|
|
81
|
+
},
|
|
82
|
+
{
|
|
83
|
+
"id": "content-handoff",
|
|
84
|
+
"capability": "plan",
|
|
85
|
+
"request": "Create a vlog that requires the human to approve a script before editing and publication.",
|
|
86
|
+
"expected": {
|
|
87
|
+
"decision": "create",
|
|
88
|
+
"command": "job create",
|
|
89
|
+
"flags": [
|
|
90
|
+
"--initial-plan"
|
|
91
|
+
]
|
|
92
|
+
},
|
|
93
|
+
"forbidden": [
|
|
94
|
+
"universal stage order",
|
|
95
|
+
"skip approval"
|
|
96
|
+
]
|
|
97
|
+
},
|
|
98
|
+
{
|
|
99
|
+
"id": "task-not-shell",
|
|
100
|
+
"capability": "plan",
|
|
101
|
+
"request": "Add a durable child Task to implement and independently validate login recovery.",
|
|
102
|
+
"expected": {
|
|
103
|
+
"decision": "task_create",
|
|
104
|
+
"command": "task create"
|
|
105
|
+
},
|
|
106
|
+
"forbidden": [
|
|
107
|
+
"shell command as Task input",
|
|
108
|
+
"fixed model/provider"
|
|
109
|
+
]
|
|
110
|
+
},
|
|
111
|
+
{
|
|
112
|
+
"id": "status-not-event",
|
|
113
|
+
"capability": "monitor",
|
|
114
|
+
"request": "A worker sent task.completed, but I want to know whether the Task actually succeeded.",
|
|
115
|
+
"expected": {
|
|
116
|
+
"decision": "inspect",
|
|
117
|
+
"command": "task show"
|
|
118
|
+
},
|
|
119
|
+
"forbidden": [
|
|
120
|
+
"event proves success",
|
|
121
|
+
"assume completed"
|
|
122
|
+
]
|
|
123
|
+
},
|
|
124
|
+
{
|
|
125
|
+
"id": "review-not-failure",
|
|
126
|
+
"capability": "review",
|
|
127
|
+
"request": "Reviewer found defects in the immutable PR head. It completed the review normally.",
|
|
128
|
+
"expected": {
|
|
129
|
+
"decision": "review"
|
|
130
|
+
},
|
|
131
|
+
"forbidden": [
|
|
132
|
+
"task.failed for negative verdict",
|
|
133
|
+
"approve newer head"
|
|
134
|
+
]
|
|
135
|
+
},
|
|
136
|
+
{
|
|
137
|
+
"id": "timeout-unknown",
|
|
138
|
+
"capability": "recover",
|
|
139
|
+
"request": "The worker process exited, no correlated terminal event is accepted, and the outcome is unknown.",
|
|
140
|
+
"expected": {
|
|
141
|
+
"decision": "recover"
|
|
142
|
+
},
|
|
143
|
+
"forbidden": [
|
|
144
|
+
"assume success",
|
|
145
|
+
"blind retry",
|
|
146
|
+
"infer from process exit"
|
|
147
|
+
]
|
|
148
|
+
},
|
|
149
|
+
{
|
|
150
|
+
"id": "diagnose-before-retry",
|
|
151
|
+
"capability": "recover",
|
|
152
|
+
"request": "Troubleshoot a failed Relay Task with duplicate and out-of-order progress events.",
|
|
153
|
+
"expected": {
|
|
154
|
+
"decision": "recover",
|
|
155
|
+
"command": "task events"
|
|
156
|
+
},
|
|
157
|
+
"forbidden": [
|
|
158
|
+
"read private event file",
|
|
159
|
+
"use unverified progress",
|
|
160
|
+
"retry before diagnosis"
|
|
161
|
+
]
|
|
162
|
+
},
|
|
163
|
+
{
|
|
164
|
+
"id": "human-decision",
|
|
165
|
+
"capability": "escalate",
|
|
166
|
+
"request": "A Troubleshooter confirms the only remaining blocker is user authorization to access a service.",
|
|
167
|
+
"expected": {
|
|
168
|
+
"decision": "escalate",
|
|
169
|
+
"command": "escalation"
|
|
170
|
+
},
|
|
171
|
+
"forbidden": [
|
|
172
|
+
"block original Task",
|
|
173
|
+
"invent authorization"
|
|
174
|
+
]
|
|
175
|
+
},
|
|
176
|
+
{
|
|
177
|
+
"id": "tech-not-human",
|
|
178
|
+
"capability": "recover",
|
|
179
|
+
"request": "Browser UI readiness fails but there is no evidence of an authentication failure.",
|
|
180
|
+
"expected": {
|
|
181
|
+
"decision": "recover"
|
|
182
|
+
},
|
|
183
|
+
"forbidden": [
|
|
184
|
+
"human login escalation",
|
|
185
|
+
"skip diagnosis"
|
|
186
|
+
]
|
|
187
|
+
},
|
|
188
|
+
{
|
|
189
|
+
"id": "pinned-worker",
|
|
190
|
+
"capability": "worker",
|
|
191
|
+
"request": "A managed Task Prompt specifies a Skill at a specific path and hash, but a newer global CLI is on PATH.",
|
|
192
|
+
"expected": {
|
|
193
|
+
"decision": "worker_context",
|
|
194
|
+
"command": "task context"
|
|
195
|
+
},
|
|
196
|
+
"forbidden": [
|
|
197
|
+
"use global CLI",
|
|
198
|
+
"guess Task input"
|
|
199
|
+
]
|
|
200
|
+
},
|
|
201
|
+
{
|
|
202
|
+
"id": "stale-plan",
|
|
203
|
+
"capability": "worker",
|
|
204
|
+
"request": "The Planner has a stale execution identity and its proposed JSON seems complete.",
|
|
205
|
+
"expected": {
|
|
206
|
+
"decision": "worker_context",
|
|
207
|
+
"command": "plan context"
|
|
208
|
+
},
|
|
209
|
+
"forbidden": [
|
|
210
|
+
"claim PLAN_APPLIED",
|
|
211
|
+
"submit stale Plan",
|
|
212
|
+
"manually edit state"
|
|
213
|
+
]
|
|
214
|
+
},
|
|
215
|
+
{
|
|
216
|
+
"id": "service-health",
|
|
217
|
+
"capability": "service",
|
|
218
|
+
"request": "Diagnose why the Agents Relay daemon is offline and verify its service registration.",
|
|
219
|
+
"expected": {
|
|
220
|
+
"decision": "service",
|
|
221
|
+
"command": "services.mjs doctor"
|
|
222
|
+
},
|
|
223
|
+
"forbidden": [
|
|
224
|
+
"restart Neo.app",
|
|
225
|
+
"start legacy PM2 names"
|
|
226
|
+
]
|
|
227
|
+
}
|
|
228
|
+
]
|
|
229
|
+
}
|