@agent-native/core 0.119.5 → 0.119.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/corpus/README.md +2 -2
- package/corpus/core/CHANGELOG.md +6 -0
- package/corpus/core/package.json +1 -1
- package/corpus/core/src/agent/durable-background.ts +1 -0
- package/corpus/core/src/deploy/build.ts +144 -2
- package/corpus/core/src/deploy/workspace-deploy.ts +123 -1
- package/corpus/core/src/integrations/integration-durable-dispatch-config.ts +22 -0
- package/corpus/core/src/integrations/integration-durable-dispatch.ts +190 -0
- package/corpus/core/src/integrations/pending-tasks-retry-job.ts +175 -87
- package/corpus/core/src/integrations/pending-tasks-store.ts +114 -11
- package/corpus/core/src/integrations/plugin.ts +113 -60
- package/corpus/core/src/integrations/task-queue-stats.ts +68 -15
- package/corpus/core/src/integrations/webhook-handler.ts +21 -46
- package/corpus/core/src/server/auth.ts +6 -0
- package/corpus/core/src/templates/default/.agents/skills/integration-webhooks/SKILL.md +63 -22
- package/corpus/core/src/templates/headless/.agents/skills/integration-webhooks/SKILL.md +63 -22
- package/corpus/core/src/templates/workspace-core/.agents/skills/integration-webhooks/SKILL.md +63 -22
- package/corpus/templates/dispatch/changelog/2026-07-21-recover-stalled-slack-agent-work.md +6 -0
- package/dist/agent/durable-background.d.ts +1 -0
- package/dist/agent/durable-background.d.ts.map +1 -1
- package/dist/agent/durable-background.js +1 -0
- package/dist/agent/durable-background.js.map +1 -1
- package/dist/collab/routes.d.ts +1 -1
- package/dist/collab/struct-routes.d.ts +1 -1
- package/dist/deploy/build.d.ts +3 -0
- package/dist/deploy/build.d.ts.map +1 -1
- package/dist/deploy/build.js +106 -3
- package/dist/deploy/build.js.map +1 -1
- package/dist/deploy/workspace-deploy.d.ts.map +1 -1
- package/dist/deploy/workspace-deploy.js +103 -2
- package/dist/deploy/workspace-deploy.js.map +1 -1
- package/dist/file-upload/actions/upload-image.d.ts +1 -1
- package/dist/integrations/integration-durable-dispatch-config.d.ts +7 -0
- package/dist/integrations/integration-durable-dispatch-config.d.ts.map +1 -0
- package/dist/integrations/integration-durable-dispatch-config.js +16 -0
- package/dist/integrations/integration-durable-dispatch-config.js.map +1 -0
- package/dist/integrations/integration-durable-dispatch.d.ts +23 -0
- package/dist/integrations/integration-durable-dispatch.d.ts.map +1 -0
- package/dist/integrations/integration-durable-dispatch.js +115 -0
- package/dist/integrations/integration-durable-dispatch.js.map +1 -0
- package/dist/integrations/pending-tasks-retry-job.d.ts +13 -1
- package/dist/integrations/pending-tasks-retry-job.d.ts.map +1 -1
- package/dist/integrations/pending-tasks-retry-job.js +133 -74
- package/dist/integrations/pending-tasks-retry-job.js.map +1 -1
- package/dist/integrations/pending-tasks-store.d.ts +14 -2
- package/dist/integrations/pending-tasks-store.d.ts.map +1 -1
- package/dist/integrations/pending-tasks-store.js +75 -11
- package/dist/integrations/pending-tasks-store.js.map +1 -1
- package/dist/integrations/plugin.d.ts.map +1 -1
- package/dist/integrations/plugin.js +88 -55
- package/dist/integrations/plugin.js.map +1 -1
- package/dist/integrations/task-queue-stats.d.ts +15 -3
- package/dist/integrations/task-queue-stats.d.ts.map +1 -1
- package/dist/integrations/task-queue-stats.js +45 -15
- package/dist/integrations/task-queue-stats.js.map +1 -1
- package/dist/integrations/webhook-handler.d.ts +1 -1
- package/dist/integrations/webhook-handler.d.ts.map +1 -1
- package/dist/integrations/webhook-handler.js +18 -42
- package/dist/integrations/webhook-handler.js.map +1 -1
- package/dist/notifications/routes.d.ts +3 -3
- package/dist/observability/routes.d.ts +5 -5
- package/dist/progress/routes.d.ts +1 -1
- package/dist/provider-api/actions/custom-provider-registration.d.ts +5 -5
- package/dist/provider-api/actions/provider-api.d.ts +4 -4
- package/dist/resources/handlers.d.ts +1 -1
- package/dist/secrets/routes.d.ts +3 -3
- package/dist/server/auth.d.ts.map +1 -1
- package/dist/server/auth.js +5 -0
- package/dist/server/auth.js.map +1 -1
- package/dist/server/realtime-token.d.ts +1 -1
- package/dist/server/transcribe-voice.d.ts +1 -1
- package/dist/templates/default/.agents/skills/integration-webhooks/SKILL.md +63 -22
- package/dist/templates/headless/.agents/skills/integration-webhooks/SKILL.md +63 -22
- package/dist/templates/workspace-core/.agents/skills/integration-webhooks/SKILL.md +63 -22
- package/package.json +1 -1
- package/src/agent/durable-background.ts +1 -0
- package/src/deploy/build.ts +144 -2
- package/src/deploy/workspace-deploy.ts +123 -1
- package/src/integrations/integration-durable-dispatch-config.ts +22 -0
- package/src/integrations/integration-durable-dispatch.ts +190 -0
- package/src/integrations/pending-tasks-retry-job.ts +175 -87
- package/src/integrations/pending-tasks-store.ts +114 -11
- package/src/integrations/plugin.ts +113 -60
- package/src/integrations/task-queue-stats.ts +68 -15
- package/src/integrations/webhook-handler.ts +21 -46
- package/src/server/auth.ts +6 -0
- package/src/templates/default/.agents/skills/integration-webhooks/SKILL.md +63 -22
- package/src/templates/headless/.agents/skills/integration-webhooks/SKILL.md +63 -22
- package/src/templates/workspace-core/.agents/skills/integration-webhooks/SKILL.md +63 -22
|
@@ -16,10 +16,10 @@ metadata:
|
|
|
16
16
|
|
|
17
17
|
Integration webhooks (Slack, Telegram, WhatsApp, email, Google Docs, etc.) must
|
|
18
18
|
**enqueue work to SQL and return 200 immediately**, then process the work in a
|
|
19
|
-
**separate fresh function execution** kicked off by a self-fired HTTP POST.
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
19
|
+
**separate fresh function execution** kicked off by a self-fired HTTP POST. SQL
|
|
20
|
+
is always the source of truth. Long-lived hosts use an in-process retry loop as
|
|
21
|
+
a best-effort safety net; serverless deployments must use an external durable
|
|
22
|
+
sweep when recovery cannot depend on a process staying alive.
|
|
23
23
|
|
|
24
24
|
Do not run agent loops inside the webhook handler itself. Do not rely on
|
|
25
25
|
fire-and-forget `Promise`s after `return`ing from a serverless handler — they get
|
|
@@ -38,8 +38,9 @@ Past attempts that don't work cross-host:
|
|
|
38
38
|
freeze the execution context the moment the response goes out. The promise
|
|
39
39
|
is silently killed, the user gets no reply, and there's no error in the
|
|
40
40
|
logs.
|
|
41
|
-
- **
|
|
42
|
-
|
|
41
|
+
- **A platform background function as the queue** — host-specific execution is
|
|
42
|
+
useful as a delivery mechanism, but it cannot replace the SQL row or atomic
|
|
43
|
+
claim shared by every host.
|
|
43
44
|
- **Cloudflare `event.waitUntil()`** — CF Workers only, not portable.
|
|
44
45
|
- **Vercel Fluid / `after()`** — Vercel-only, gated behind specific runtimes.
|
|
45
46
|
- **A long-lived in-process queue** — fine on a single Node box, but on
|
|
@@ -62,8 +63,9 @@ and a recurring job is the safety net.
|
|
|
62
63
|
2. INSERT INTO integration_pending_tasks
|
|
63
64
|
(status='pending', payload=...)
|
|
64
65
|
│
|
|
65
|
-
3.
|
|
66
|
-
— fire
|
|
66
|
+
3. dispatch POST /integrations/process-task
|
|
67
|
+
— portable self-fire by default
|
|
68
|
+
— acknowledged background handoff when enabled
|
|
67
69
|
│
|
|
68
70
|
4. return 200 to platform ◄───────────┘
|
|
69
71
|
|
|
@@ -79,7 +81,7 @@ and a recurring job is the safety net.
|
|
|
79
81
|
|
|
80
82
|
|
|
81
83
|
┌──────────────────────────────────────────────┐
|
|
82
|
-
│
|
|
84
|
+
│ Recovery sweep (every 60s) — safety net │
|
|
83
85
|
│ Re-fires processor for tasks stuck in │
|
|
84
86
|
│ 'pending' or 'processing' beyond timeout. │
|
|
85
87
|
│ Caps retries at 3 then marks 'failed'. │
|
|
@@ -98,6 +100,7 @@ loop.
|
|
|
98
100
|
| `packages/core/src/integrations/webhook-handler.ts` | Verifies signature, parses, enqueues task, fires processor |
|
|
99
101
|
| `packages/core/src/integrations/pending-tasks-store.ts` | SQL queue: `insertPendingTask`, `claimPendingTask`, `markTaskCompleted`, `markTaskFailed` |
|
|
100
102
|
| `packages/core/src/integrations/pending-tasks-retry-job.ts` | Recurring retry sweep (`startPendingTasksRetryJob`, `retryStuckPendingTasks`) |
|
|
103
|
+
| `packages/core/src/integrations/integration-durable-dispatch.ts` | Default-off acknowledged dispatch, scoped rollout, and outcome recording |
|
|
101
104
|
| `packages/core/src/integrations/types.ts` | `PlatformAdapter`, `IncomingMessage`, `OutgoingMessage` |
|
|
102
105
|
| `packages/core/src/integrations/adapters/{slack,telegram,whatsapp,email,google-docs}.ts` | One adapter per platform |
|
|
103
106
|
|
|
@@ -108,7 +111,8 @@ All under `/_agent-native/integrations/`:
|
|
|
108
111
|
| Method | Path | Purpose |
|
|
109
112
|
| ------ | -------------------------- | ------------------------------------------------------------- |
|
|
110
113
|
| POST | `/:platform/webhook` | Platform pings this. Verifies, enqueues, returns 200 quickly. |
|
|
111
|
-
| POST | `/
|
|
114
|
+
| POST | `/process-task` | Processor target. Claims a task and runs the agent loop. |
|
|
115
|
+
| POST | `/retry-stuck-tasks` | Signed, bounded recovery sweep for durable schedulers. |
|
|
112
116
|
| GET | `/status` | All integrations status (settings UI). |
|
|
113
117
|
| GET | `/:platform/status` | One platform's status. |
|
|
114
118
|
| POST | `/:platform/enable` | Enable an integration. |
|
|
@@ -129,6 +133,10 @@ CREATE TABLE IF NOT EXISTS integration_pending_tasks (
|
|
|
129
133
|
org_id TEXT,
|
|
130
134
|
status TEXT NOT NULL, -- pending | processing | completed | failed
|
|
131
135
|
attempts INTEGER NOT NULL DEFAULT 0,
|
|
136
|
+
dispatch_attempts INTEGER NOT NULL DEFAULT 0,
|
|
137
|
+
last_dispatch_at INTEGER,
|
|
138
|
+
last_dispatch_outcome TEXT,
|
|
139
|
+
dispatch_scope TEXT, -- persisted channel scope for recovery
|
|
132
140
|
error_message TEXT,
|
|
133
141
|
created_at INTEGER NOT NULL,
|
|
134
142
|
updated_at INTEGER NOT NULL,
|
|
@@ -136,6 +144,8 @@ CREATE TABLE IF NOT EXISTS integration_pending_tasks (
|
|
|
136
144
|
);
|
|
137
145
|
CREATE INDEX IF NOT EXISTS idx_pending_tasks_status_created
|
|
138
146
|
ON integration_pending_tasks(status, created_at);
|
|
147
|
+
CREATE INDEX IF NOT EXISTS idx_pending_tasks_dispatch_scope
|
|
148
|
+
ON integration_pending_tasks(platform, dispatch_scope);
|
|
139
149
|
```
|
|
140
150
|
|
|
141
151
|
The store layer creates this lazily on first use via `ensureTable()` and uses
|
|
@@ -143,10 +153,38 @@ The store layer creates this lazily on first use via `ensureTable()` and uses
|
|
|
143
153
|
|
|
144
154
|
`claimPendingTask` is the critical concurrency primitive: it atomically flips
|
|
145
155
|
`pending` → `processing` and increments `attempts`, returning `null` if another
|
|
146
|
-
worker beat us to it. Both the initial
|
|
156
|
+
worker beat us to it. Both the initial dispatch and every recovery sweep
|
|
147
157
|
funnel through the same processor endpoint, and `claimPendingTask` is what
|
|
148
158
|
prevents the same task from being processed twice.
|
|
149
159
|
|
|
160
|
+
### Netlify durable recovery
|
|
161
|
+
|
|
162
|
+
The portable self-dispatch remains the default. To emit and use Netlify's
|
|
163
|
+
acknowledged background worker plus a one-minute scheduled recovery function,
|
|
164
|
+
set `AGENT_INTEGRATION_DURABLE_DISPATCH=true` at build and runtime. A production
|
|
165
|
+
rollout can be narrowed with the comma-separated
|
|
166
|
+
`AGENT_INTEGRATION_DURABLE_DISPATCH_SCOPES` allowlist. Supported values are
|
|
167
|
+
`<platform>:*`, `<platform>:<external-thread-id>`, and, for adapters that expose
|
|
168
|
+
one, `<platform>:<channel-id>` (for example `slack:C123`).
|
|
169
|
+
Channel-scoped handoffs persist that scope separately from the provider thread
|
|
170
|
+
identity so a later bounded recovery sweep applies the same allowlist decision.
|
|
171
|
+
|
|
172
|
+
Workspace deploys emit this pair only for the Dispatch control-plane app, which
|
|
173
|
+
owns workspace messaging integrations. Standalone templates emit their own
|
|
174
|
+
pair when the same flag is enabled.
|
|
175
|
+
|
|
176
|
+
The background handoff waits only for Netlify's enqueue acknowledgement, not
|
|
177
|
+
for the agent loop. A failed or non-2xx acknowledgement falls back to the
|
|
178
|
+
portable processor. The scheduled function calls the HMAC-authenticated,
|
|
179
|
+
bounded `/retry-stuck-tasks` route; its atomic compare-and-set update prevents
|
|
180
|
+
overlapping sweeps from dispatching the same stale row twice.
|
|
181
|
+
Rows acknowledged by the 15-minute background worker use a 16-minute stale
|
|
182
|
+
cutoff, while portable synchronous dispatches retain the shorter serverless
|
|
183
|
+
cutoff. Apply that lease predicate before `LIMIT` so healthy long runs cannot
|
|
184
|
+
hide recoverable work. The background processor writes its lease marker in the
|
|
185
|
+
same atomic update that claims the row; best-effort dispatch telemetry is not
|
|
186
|
+
trusted for replay safety.
|
|
187
|
+
|
|
150
188
|
## Adding a New Platform Adapter
|
|
151
189
|
|
|
152
190
|
1. **Implement `PlatformAdapter`** in `packages/core/src/integrations/adapters/<platform>.ts`:
|
|
@@ -245,9 +283,10 @@ so a normal long-running reply is safe.
|
|
|
245
283
|
|
|
246
284
|
## Cross-Platform Considerations
|
|
247
285
|
|
|
248
|
-
- **
|
|
249
|
-
|
|
250
|
-
|
|
286
|
+
- **Portable correctness, optional host acceleration.** Every host uses SQL,
|
|
287
|
+
HMAC-authenticated processor routes, and atomic claims. A host-specific
|
|
288
|
+
background function or scheduler may provide a more reliable wake-up, but
|
|
289
|
+
disabling it must preserve the portable path.
|
|
251
290
|
- **No assumed runtime.** The processor endpoint is a normal H3 handler under
|
|
252
291
|
`/_agent-native/`. It runs wherever the rest of the framework runs.
|
|
253
292
|
- **No persistent in-memory state.** The dedup map in the webhook handler is
|
|
@@ -262,11 +301,12 @@ so a normal long-running reply is safe.
|
|
|
262
301
|
|
|
263
302
|
## Why Fire-and-Forget on Serverless Is Unreliable
|
|
264
303
|
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
complete before the function freezes.
|
|
268
|
-
|
|
269
|
-
|
|
304
|
+
The portable webhook path gives the outbound processor request a short head
|
|
305
|
+
start but does not wait for the response body, so that initial dispatch is
|
|
306
|
+
**not** guaranteed to complete before the function freezes. On a long-lived
|
|
307
|
+
host, the in-process retry loop is a best-effort safety net. On Netlify with
|
|
308
|
+
durable dispatch enabled, the initial handoff is acknowledged and the external
|
|
309
|
+
scheduled function supplies the recovery wake-up for cases where:
|
|
270
310
|
|
|
271
311
|
- The serverless platform froze the handler before the outbound `fetch`
|
|
272
312
|
flushed its bytes.
|
|
@@ -278,8 +318,9 @@ Tasks stuck in `pending` for >90s or `processing` for >5min get re-fired up to
|
|
|
278
318
|
3 times. After 3 attempts they're marked `failed` permanently so we stop
|
|
279
319
|
spamming the processor.
|
|
280
320
|
|
|
281
|
-
**Never assume the initial
|
|
282
|
-
|
|
321
|
+
**Never assume the initial dispatch succeeded.** Rely on the SQL queue, atomic
|
|
322
|
+
claim, and a recovery mechanism whose lifetime is independent of the original
|
|
323
|
+
request. An in-process timer alone is not that mechanism on serverless.
|
|
283
324
|
|
|
284
325
|
## Debugging Checklist
|
|
285
326
|
|
|
@@ -292,7 +333,7 @@ queue + retry job for at-least-once delivery.
|
|
|
292
333
|
4. **Status?** `pending` means the processor never picked it up — check that
|
|
293
334
|
`_process-task` is reachable from the box itself (the self-fetch must work
|
|
294
335
|
over the public URL). `processing` for over 5 minutes means the processor
|
|
295
|
-
died mid-run —
|
|
336
|
+
died mid-run — a live recovery sweep will pick it up.
|
|
296
337
|
5. **Failed?** Check `error_message` and `attempts`. After 3 attempts the row
|
|
297
338
|
is parked at `failed` and won't be retried.
|
|
298
339
|
6. **Reply not delivered?** The processor likely succeeded but
|
|
@@ -16,10 +16,10 @@ metadata:
|
|
|
16
16
|
|
|
17
17
|
Integration webhooks (Slack, Telegram, WhatsApp, email, Google Docs, etc.) must
|
|
18
18
|
**enqueue work to SQL and return 200 immediately**, then process the work in a
|
|
19
|
-
**separate fresh function execution** kicked off by a self-fired HTTP POST.
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
19
|
+
**separate fresh function execution** kicked off by a self-fired HTTP POST. SQL
|
|
20
|
+
is always the source of truth. Long-lived hosts use an in-process retry loop as
|
|
21
|
+
a best-effort safety net; serverless deployments must use an external durable
|
|
22
|
+
sweep when recovery cannot depend on a process staying alive.
|
|
23
23
|
|
|
24
24
|
Do not run agent loops inside the webhook handler itself. Do not rely on
|
|
25
25
|
fire-and-forget `Promise`s after `return`ing from a serverless handler — they get
|
|
@@ -38,8 +38,9 @@ Past attempts that don't work cross-host:
|
|
|
38
38
|
freeze the execution context the moment the response goes out. The promise
|
|
39
39
|
is silently killed, the user gets no reply, and there's no error in the
|
|
40
40
|
logs.
|
|
41
|
-
- **
|
|
42
|
-
|
|
41
|
+
- **A platform background function as the queue** — host-specific execution is
|
|
42
|
+
useful as a delivery mechanism, but it cannot replace the SQL row or atomic
|
|
43
|
+
claim shared by every host.
|
|
43
44
|
- **Cloudflare `event.waitUntil()`** — CF Workers only, not portable.
|
|
44
45
|
- **Vercel Fluid / `after()`** — Vercel-only, gated behind specific runtimes.
|
|
45
46
|
- **A long-lived in-process queue** — fine on a single Node box, but on
|
|
@@ -62,8 +63,9 @@ and a recurring job is the safety net.
|
|
|
62
63
|
2. INSERT INTO integration_pending_tasks
|
|
63
64
|
(status='pending', payload=...)
|
|
64
65
|
│
|
|
65
|
-
3.
|
|
66
|
-
— fire
|
|
66
|
+
3. dispatch POST /integrations/process-task
|
|
67
|
+
— portable self-fire by default
|
|
68
|
+
— acknowledged background handoff when enabled
|
|
67
69
|
│
|
|
68
70
|
4. return 200 to platform ◄───────────┘
|
|
69
71
|
|
|
@@ -79,7 +81,7 @@ and a recurring job is the safety net.
|
|
|
79
81
|
|
|
80
82
|
|
|
81
83
|
┌──────────────────────────────────────────────┐
|
|
82
|
-
│
|
|
84
|
+
│ Recovery sweep (every 60s) — safety net │
|
|
83
85
|
│ Re-fires processor for tasks stuck in │
|
|
84
86
|
│ 'pending' or 'processing' beyond timeout. │
|
|
85
87
|
│ Caps retries at 3 then marks 'failed'. │
|
|
@@ -98,6 +100,7 @@ loop.
|
|
|
98
100
|
| `packages/core/src/integrations/webhook-handler.ts` | Verifies signature, parses, enqueues task, fires processor |
|
|
99
101
|
| `packages/core/src/integrations/pending-tasks-store.ts` | SQL queue: `insertPendingTask`, `claimPendingTask`, `markTaskCompleted`, `markTaskFailed` |
|
|
100
102
|
| `packages/core/src/integrations/pending-tasks-retry-job.ts` | Recurring retry sweep (`startPendingTasksRetryJob`, `retryStuckPendingTasks`) |
|
|
103
|
+
| `packages/core/src/integrations/integration-durable-dispatch.ts` | Default-off acknowledged dispatch, scoped rollout, and outcome recording |
|
|
101
104
|
| `packages/core/src/integrations/types.ts` | `PlatformAdapter`, `IncomingMessage`, `OutgoingMessage` |
|
|
102
105
|
| `packages/core/src/integrations/adapters/{slack,telegram,whatsapp,email,google-docs}.ts` | One adapter per platform |
|
|
103
106
|
|
|
@@ -108,7 +111,8 @@ All under `/_agent-native/integrations/`:
|
|
|
108
111
|
| Method | Path | Purpose |
|
|
109
112
|
| ------ | -------------------------- | ------------------------------------------------------------- |
|
|
110
113
|
| POST | `/:platform/webhook` | Platform pings this. Verifies, enqueues, returns 200 quickly. |
|
|
111
|
-
| POST | `/
|
|
114
|
+
| POST | `/process-task` | Processor target. Claims a task and runs the agent loop. |
|
|
115
|
+
| POST | `/retry-stuck-tasks` | Signed, bounded recovery sweep for durable schedulers. |
|
|
112
116
|
| GET | `/status` | All integrations status (settings UI). |
|
|
113
117
|
| GET | `/:platform/status` | One platform's status. |
|
|
114
118
|
| POST | `/:platform/enable` | Enable an integration. |
|
|
@@ -129,6 +133,10 @@ CREATE TABLE IF NOT EXISTS integration_pending_tasks (
|
|
|
129
133
|
org_id TEXT,
|
|
130
134
|
status TEXT NOT NULL, -- pending | processing | completed | failed
|
|
131
135
|
attempts INTEGER NOT NULL DEFAULT 0,
|
|
136
|
+
dispatch_attempts INTEGER NOT NULL DEFAULT 0,
|
|
137
|
+
last_dispatch_at INTEGER,
|
|
138
|
+
last_dispatch_outcome TEXT,
|
|
139
|
+
dispatch_scope TEXT, -- persisted channel scope for recovery
|
|
132
140
|
error_message TEXT,
|
|
133
141
|
created_at INTEGER NOT NULL,
|
|
134
142
|
updated_at INTEGER NOT NULL,
|
|
@@ -136,6 +144,8 @@ CREATE TABLE IF NOT EXISTS integration_pending_tasks (
|
|
|
136
144
|
);
|
|
137
145
|
CREATE INDEX IF NOT EXISTS idx_pending_tasks_status_created
|
|
138
146
|
ON integration_pending_tasks(status, created_at);
|
|
147
|
+
CREATE INDEX IF NOT EXISTS idx_pending_tasks_dispatch_scope
|
|
148
|
+
ON integration_pending_tasks(platform, dispatch_scope);
|
|
139
149
|
```
|
|
140
150
|
|
|
141
151
|
The store layer creates this lazily on first use via `ensureTable()` and uses
|
|
@@ -143,10 +153,38 @@ The store layer creates this lazily on first use via `ensureTable()` and uses
|
|
|
143
153
|
|
|
144
154
|
`claimPendingTask` is the critical concurrency primitive: it atomically flips
|
|
145
155
|
`pending` → `processing` and increments `attempts`, returning `null` if another
|
|
146
|
-
worker beat us to it. Both the initial
|
|
156
|
+
worker beat us to it. Both the initial dispatch and every recovery sweep
|
|
147
157
|
funnel through the same processor endpoint, and `claimPendingTask` is what
|
|
148
158
|
prevents the same task from being processed twice.
|
|
149
159
|
|
|
160
|
+
### Netlify durable recovery
|
|
161
|
+
|
|
162
|
+
The portable self-dispatch remains the default. To emit and use Netlify's
|
|
163
|
+
acknowledged background worker plus a one-minute scheduled recovery function,
|
|
164
|
+
set `AGENT_INTEGRATION_DURABLE_DISPATCH=true` at build and runtime. A production
|
|
165
|
+
rollout can be narrowed with the comma-separated
|
|
166
|
+
`AGENT_INTEGRATION_DURABLE_DISPATCH_SCOPES` allowlist. Supported values are
|
|
167
|
+
`<platform>:*`, `<platform>:<external-thread-id>`, and, for adapters that expose
|
|
168
|
+
one, `<platform>:<channel-id>` (for example `slack:C123`).
|
|
169
|
+
Channel-scoped handoffs persist that scope separately from the provider thread
|
|
170
|
+
identity so a later bounded recovery sweep applies the same allowlist decision.
|
|
171
|
+
|
|
172
|
+
Workspace deploys emit this pair only for the Dispatch control-plane app, which
|
|
173
|
+
owns workspace messaging integrations. Standalone templates emit their own
|
|
174
|
+
pair when the same flag is enabled.
|
|
175
|
+
|
|
176
|
+
The background handoff waits only for Netlify's enqueue acknowledgement, not
|
|
177
|
+
for the agent loop. A failed or non-2xx acknowledgement falls back to the
|
|
178
|
+
portable processor. The scheduled function calls the HMAC-authenticated,
|
|
179
|
+
bounded `/retry-stuck-tasks` route; its atomic compare-and-set update prevents
|
|
180
|
+
overlapping sweeps from dispatching the same stale row twice.
|
|
181
|
+
Rows acknowledged by the 15-minute background worker use a 16-minute stale
|
|
182
|
+
cutoff, while portable synchronous dispatches retain the shorter serverless
|
|
183
|
+
cutoff. Apply that lease predicate before `LIMIT` so healthy long runs cannot
|
|
184
|
+
hide recoverable work. The background processor writes its lease marker in the
|
|
185
|
+
same atomic update that claims the row; best-effort dispatch telemetry is not
|
|
186
|
+
trusted for replay safety.
|
|
187
|
+
|
|
150
188
|
## Adding a New Platform Adapter
|
|
151
189
|
|
|
152
190
|
1. **Implement `PlatformAdapter`** in `packages/core/src/integrations/adapters/<platform>.ts`:
|
|
@@ -245,9 +283,10 @@ so a normal long-running reply is safe.
|
|
|
245
283
|
|
|
246
284
|
## Cross-Platform Considerations
|
|
247
285
|
|
|
248
|
-
- **
|
|
249
|
-
|
|
250
|
-
|
|
286
|
+
- **Portable correctness, optional host acceleration.** Every host uses SQL,
|
|
287
|
+
HMAC-authenticated processor routes, and atomic claims. A host-specific
|
|
288
|
+
background function or scheduler may provide a more reliable wake-up, but
|
|
289
|
+
disabling it must preserve the portable path.
|
|
251
290
|
- **No assumed runtime.** The processor endpoint is a normal H3 handler under
|
|
252
291
|
`/_agent-native/`. It runs wherever the rest of the framework runs.
|
|
253
292
|
- **No persistent in-memory state.** The dedup map in the webhook handler is
|
|
@@ -262,11 +301,12 @@ so a normal long-running reply is safe.
|
|
|
262
301
|
|
|
263
302
|
## Why Fire-and-Forget on Serverless Is Unreliable
|
|
264
303
|
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
complete before the function freezes.
|
|
268
|
-
|
|
269
|
-
|
|
304
|
+
The portable webhook path gives the outbound processor request a short head
|
|
305
|
+
start but does not wait for the response body, so that initial dispatch is
|
|
306
|
+
**not** guaranteed to complete before the function freezes. On a long-lived
|
|
307
|
+
host, the in-process retry loop is a best-effort safety net. On Netlify with
|
|
308
|
+
durable dispatch enabled, the initial handoff is acknowledged and the external
|
|
309
|
+
scheduled function supplies the recovery wake-up for cases where:
|
|
270
310
|
|
|
271
311
|
- The serverless platform froze the handler before the outbound `fetch`
|
|
272
312
|
flushed its bytes.
|
|
@@ -278,8 +318,9 @@ Tasks stuck in `pending` for >90s or `processing` for >5min get re-fired up to
|
|
|
278
318
|
3 times. After 3 attempts they're marked `failed` permanently so we stop
|
|
279
319
|
spamming the processor.
|
|
280
320
|
|
|
281
|
-
**Never assume the initial
|
|
282
|
-
|
|
321
|
+
**Never assume the initial dispatch succeeded.** Rely on the SQL queue, atomic
|
|
322
|
+
claim, and a recovery mechanism whose lifetime is independent of the original
|
|
323
|
+
request. An in-process timer alone is not that mechanism on serverless.
|
|
283
324
|
|
|
284
325
|
## Debugging Checklist
|
|
285
326
|
|
|
@@ -292,7 +333,7 @@ queue + retry job for at-least-once delivery.
|
|
|
292
333
|
4. **Status?** `pending` means the processor never picked it up — check that
|
|
293
334
|
`_process-task` is reachable from the box itself (the self-fetch must work
|
|
294
335
|
over the public URL). `processing` for over 5 minutes means the processor
|
|
295
|
-
died mid-run —
|
|
336
|
+
died mid-run — a live recovery sweep will pick it up.
|
|
296
337
|
5. **Failed?** Check `error_message` and `attempts`. After 3 attempts the row
|
|
297
338
|
is parked at `failed` and won't be retried.
|
|
298
339
|
6. **Reply not delivered?** The processor likely succeeded but
|
|
@@ -16,10 +16,10 @@ metadata:
|
|
|
16
16
|
|
|
17
17
|
Integration webhooks (Slack, Telegram, WhatsApp, email, Google Docs, etc.) must
|
|
18
18
|
**enqueue work to SQL and return 200 immediately**, then process the work in a
|
|
19
|
-
**separate fresh function execution** kicked off by a self-fired HTTP POST.
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
19
|
+
**separate fresh function execution** kicked off by a self-fired HTTP POST. SQL
|
|
20
|
+
is always the source of truth. Long-lived hosts use an in-process retry loop as
|
|
21
|
+
a best-effort safety net; serverless deployments must use an external durable
|
|
22
|
+
sweep when recovery cannot depend on a process staying alive.
|
|
23
23
|
|
|
24
24
|
Do not run agent loops inside the webhook handler itself. Do not rely on
|
|
25
25
|
fire-and-forget `Promise`s after `return`ing from a serverless handler — they get
|
|
@@ -38,8 +38,9 @@ Past attempts that don't work cross-host:
|
|
|
38
38
|
freeze the execution context the moment the response goes out. The promise
|
|
39
39
|
is silently killed, the user gets no reply, and there's no error in the
|
|
40
40
|
logs.
|
|
41
|
-
- **
|
|
42
|
-
|
|
41
|
+
- **A platform background function as the queue** — host-specific execution is
|
|
42
|
+
useful as a delivery mechanism, but it cannot replace the SQL row or atomic
|
|
43
|
+
claim shared by every host.
|
|
43
44
|
- **Cloudflare `event.waitUntil()`** — CF Workers only, not portable.
|
|
44
45
|
- **Vercel Fluid / `after()`** — Vercel-only, gated behind specific runtimes.
|
|
45
46
|
- **A long-lived in-process queue** — fine on a single Node box, but on
|
|
@@ -62,8 +63,9 @@ and a recurring job is the safety net.
|
|
|
62
63
|
2. INSERT INTO integration_pending_tasks
|
|
63
64
|
(status='pending', payload=...)
|
|
64
65
|
│
|
|
65
|
-
3.
|
|
66
|
-
— fire
|
|
66
|
+
3. dispatch POST /integrations/process-task
|
|
67
|
+
— portable self-fire by default
|
|
68
|
+
— acknowledged background handoff when enabled
|
|
67
69
|
│
|
|
68
70
|
4. return 200 to platform ◄───────────┘
|
|
69
71
|
|
|
@@ -79,7 +81,7 @@ and a recurring job is the safety net.
|
|
|
79
81
|
|
|
80
82
|
|
|
81
83
|
┌──────────────────────────────────────────────┐
|
|
82
|
-
│
|
|
84
|
+
│ Recovery sweep (every 60s) — safety net │
|
|
83
85
|
│ Re-fires processor for tasks stuck in │
|
|
84
86
|
│ 'pending' or 'processing' beyond timeout. │
|
|
85
87
|
│ Caps retries at 3 then marks 'failed'. │
|
|
@@ -98,6 +100,7 @@ loop.
|
|
|
98
100
|
| `packages/core/src/integrations/webhook-handler.ts` | Verifies signature, parses, enqueues task, fires processor |
|
|
99
101
|
| `packages/core/src/integrations/pending-tasks-store.ts` | SQL queue: `insertPendingTask`, `claimPendingTask`, `markTaskCompleted`, `markTaskFailed` |
|
|
100
102
|
| `packages/core/src/integrations/pending-tasks-retry-job.ts` | Recurring retry sweep (`startPendingTasksRetryJob`, `retryStuckPendingTasks`) |
|
|
103
|
+
| `packages/core/src/integrations/integration-durable-dispatch.ts` | Default-off acknowledged dispatch, scoped rollout, and outcome recording |
|
|
101
104
|
| `packages/core/src/integrations/types.ts` | `PlatformAdapter`, `IncomingMessage`, `OutgoingMessage` |
|
|
102
105
|
| `packages/core/src/integrations/adapters/{slack,telegram,whatsapp,email,google-docs}.ts` | One adapter per platform |
|
|
103
106
|
|
|
@@ -108,7 +111,8 @@ All under `/_agent-native/integrations/`:
|
|
|
108
111
|
| Method | Path | Purpose |
|
|
109
112
|
| ------ | -------------------------- | ------------------------------------------------------------- |
|
|
110
113
|
| POST | `/:platform/webhook` | Platform pings this. Verifies, enqueues, returns 200 quickly. |
|
|
111
|
-
| POST | `/
|
|
114
|
+
| POST | `/process-task` | Processor target. Claims a task and runs the agent loop. |
|
|
115
|
+
| POST | `/retry-stuck-tasks` | Signed, bounded recovery sweep for durable schedulers. |
|
|
112
116
|
| GET | `/status` | All integrations status (settings UI). |
|
|
113
117
|
| GET | `/:platform/status` | One platform's status. |
|
|
114
118
|
| POST | `/:platform/enable` | Enable an integration. |
|
|
@@ -129,6 +133,10 @@ CREATE TABLE IF NOT EXISTS integration_pending_tasks (
|
|
|
129
133
|
org_id TEXT,
|
|
130
134
|
status TEXT NOT NULL, -- pending | processing | completed | failed
|
|
131
135
|
attempts INTEGER NOT NULL DEFAULT 0,
|
|
136
|
+
dispatch_attempts INTEGER NOT NULL DEFAULT 0,
|
|
137
|
+
last_dispatch_at INTEGER,
|
|
138
|
+
last_dispatch_outcome TEXT,
|
|
139
|
+
dispatch_scope TEXT, -- persisted channel scope for recovery
|
|
132
140
|
error_message TEXT,
|
|
133
141
|
created_at INTEGER NOT NULL,
|
|
134
142
|
updated_at INTEGER NOT NULL,
|
|
@@ -136,6 +144,8 @@ CREATE TABLE IF NOT EXISTS integration_pending_tasks (
|
|
|
136
144
|
);
|
|
137
145
|
CREATE INDEX IF NOT EXISTS idx_pending_tasks_status_created
|
|
138
146
|
ON integration_pending_tasks(status, created_at);
|
|
147
|
+
CREATE INDEX IF NOT EXISTS idx_pending_tasks_dispatch_scope
|
|
148
|
+
ON integration_pending_tasks(platform, dispatch_scope);
|
|
139
149
|
```
|
|
140
150
|
|
|
141
151
|
The store layer creates this lazily on first use via `ensureTable()` and uses
|
|
@@ -143,10 +153,38 @@ The store layer creates this lazily on first use via `ensureTable()` and uses
|
|
|
143
153
|
|
|
144
154
|
`claimPendingTask` is the critical concurrency primitive: it atomically flips
|
|
145
155
|
`pending` → `processing` and increments `attempts`, returning `null` if another
|
|
146
|
-
worker beat us to it. Both the initial
|
|
156
|
+
worker beat us to it. Both the initial dispatch and every recovery sweep
|
|
147
157
|
funnel through the same processor endpoint, and `claimPendingTask` is what
|
|
148
158
|
prevents the same task from being processed twice.
|
|
149
159
|
|
|
160
|
+
### Netlify durable recovery
|
|
161
|
+
|
|
162
|
+
The portable self-dispatch remains the default. To emit and use Netlify's
|
|
163
|
+
acknowledged background worker plus a one-minute scheduled recovery function,
|
|
164
|
+
set `AGENT_INTEGRATION_DURABLE_DISPATCH=true` at build and runtime. A production
|
|
165
|
+
rollout can be narrowed with the comma-separated
|
|
166
|
+
`AGENT_INTEGRATION_DURABLE_DISPATCH_SCOPES` allowlist. Supported values are
|
|
167
|
+
`<platform>:*`, `<platform>:<external-thread-id>`, and, for adapters that expose
|
|
168
|
+
one, `<platform>:<channel-id>` (for example `slack:C123`).
|
|
169
|
+
Channel-scoped handoffs persist that scope separately from the provider thread
|
|
170
|
+
identity so a later bounded recovery sweep applies the same allowlist decision.
|
|
171
|
+
|
|
172
|
+
Workspace deploys emit this pair only for the Dispatch control-plane app, which
|
|
173
|
+
owns workspace messaging integrations. Standalone templates emit their own
|
|
174
|
+
pair when the same flag is enabled.
|
|
175
|
+
|
|
176
|
+
The background handoff waits only for Netlify's enqueue acknowledgement, not
|
|
177
|
+
for the agent loop. A failed or non-2xx acknowledgement falls back to the
|
|
178
|
+
portable processor. The scheduled function calls the HMAC-authenticated,
|
|
179
|
+
bounded `/retry-stuck-tasks` route; its atomic compare-and-set update prevents
|
|
180
|
+
overlapping sweeps from dispatching the same stale row twice.
|
|
181
|
+
Rows acknowledged by the 15-minute background worker use a 16-minute stale
|
|
182
|
+
cutoff, while portable synchronous dispatches retain the shorter serverless
|
|
183
|
+
cutoff. Apply that lease predicate before `LIMIT` so healthy long runs cannot
|
|
184
|
+
hide recoverable work. The background processor writes its lease marker in the
|
|
185
|
+
same atomic update that claims the row; best-effort dispatch telemetry is not
|
|
186
|
+
trusted for replay safety.
|
|
187
|
+
|
|
150
188
|
## Adding a New Platform Adapter
|
|
151
189
|
|
|
152
190
|
1. **Implement `PlatformAdapter`** in `packages/core/src/integrations/adapters/<platform>.ts`:
|
|
@@ -245,9 +283,10 @@ so a normal long-running reply is safe.
|
|
|
245
283
|
|
|
246
284
|
## Cross-Platform Considerations
|
|
247
285
|
|
|
248
|
-
- **
|
|
249
|
-
|
|
250
|
-
|
|
286
|
+
- **Portable correctness, optional host acceleration.** Every host uses SQL,
|
|
287
|
+
HMAC-authenticated processor routes, and atomic claims. A host-specific
|
|
288
|
+
background function or scheduler may provide a more reliable wake-up, but
|
|
289
|
+
disabling it must preserve the portable path.
|
|
251
290
|
- **No assumed runtime.** The processor endpoint is a normal H3 handler under
|
|
252
291
|
`/_agent-native/`. It runs wherever the rest of the framework runs.
|
|
253
292
|
- **No persistent in-memory state.** The dedup map in the webhook handler is
|
|
@@ -262,11 +301,12 @@ so a normal long-running reply is safe.
|
|
|
262
301
|
|
|
263
302
|
## Why Fire-and-Forget on Serverless Is Unreliable
|
|
264
303
|
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
complete before the function freezes.
|
|
268
|
-
|
|
269
|
-
|
|
304
|
+
The portable webhook path gives the outbound processor request a short head
|
|
305
|
+
start but does not wait for the response body, so that initial dispatch is
|
|
306
|
+
**not** guaranteed to complete before the function freezes. On a long-lived
|
|
307
|
+
host, the in-process retry loop is a best-effort safety net. On Netlify with
|
|
308
|
+
durable dispatch enabled, the initial handoff is acknowledged and the external
|
|
309
|
+
scheduled function supplies the recovery wake-up for cases where:
|
|
270
310
|
|
|
271
311
|
- The serverless platform froze the handler before the outbound `fetch`
|
|
272
312
|
flushed its bytes.
|
|
@@ -278,8 +318,9 @@ Tasks stuck in `pending` for >90s or `processing` for >5min get re-fired up to
|
|
|
278
318
|
3 times. After 3 attempts they're marked `failed` permanently so we stop
|
|
279
319
|
spamming the processor.
|
|
280
320
|
|
|
281
|
-
**Never assume the initial
|
|
282
|
-
|
|
321
|
+
**Never assume the initial dispatch succeeded.** Rely on the SQL queue, atomic
|
|
322
|
+
claim, and a recovery mechanism whose lifetime is independent of the original
|
|
323
|
+
request. An in-process timer alone is not that mechanism on serverless.
|
|
283
324
|
|
|
284
325
|
## Debugging Checklist
|
|
285
326
|
|
|
@@ -292,7 +333,7 @@ queue + retry job for at-least-once delivery.
|
|
|
292
333
|
4. **Status?** `pending` means the processor never picked it up — check that
|
|
293
334
|
`_process-task` is reachable from the box itself (the self-fetch must work
|
|
294
335
|
over the public URL). `processing` for over 5 minutes means the processor
|
|
295
|
-
died mid-run —
|
|
336
|
+
died mid-run — a live recovery sweep will pick it up.
|
|
296
337
|
5. **Failed?** Check `error_message` and `attempts`. After 3 attempts the row
|
|
297
338
|
is parked at `failed` and won't be retried.
|
|
298
339
|
6. **Reply not delivered?** The processor likely succeeded but
|