@sgtbeatdown/pi-tasks 0.1.14 → 0.1.15
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/protocol-reference.md +7 -3
- package/package.json +1 -1
- package/src/extension.ts +22 -19
- package/src/task-core.ts +200 -91
- package/src/task-inbox.ts +75 -61
- package/src/task-store.ts +61 -10
- package/src/task-work-scope.ts +63 -0
|
@@ -28,7 +28,7 @@ The `agent_task_send` schema is exactly `to`, `task`, and optional `timeoutMs`.
|
|
|
28
28
|
|
|
29
29
|
## history versus live state
|
|
30
30
|
|
|
31
|
-
- Pi requests task-processing turns through its safe `deliverAs: "followUp"` queue. Pi session messages and tool calls/results preserve historical context. Non-waking receipts, parent ACKs
|
|
31
|
+
- Pi requests task-processing turns through its safe `deliverAs: "followUp"` queue. Pi session messages and tool calls/results preserve historical context. Non-waking receipts, parent ACKs, late-terminal facts and confirmed receiver-authored conversation echoes are archived as `pi-tasks-event-record` custom entries before transport ACK, without model insertion or wake. Received model-visible task events include the complete structured event in `pi-tasks-event` details, including results—not just a rendered summary.
|
|
32
32
|
- Active tasks, outbox intentions, individual ACK checkpoints, deduplication and delivery-blocked evidence are RAM-owned. Each endpoint has lower-only limits of 16,384 records/32MiB encoded state. Admission fails before exceeding those bounds; they are not a process-RSS guarantee.
|
|
33
33
|
- `agent_task_send` confirms relay acceptance, **not execution**. `receiver_recorded` confirms receiver RAM receipt, **not durable persistence**. Pi insertion and wake acceptance are separate evidence, never proof of model execution.
|
|
34
34
|
- Session history is not an operational journal. A new process/lifecycle starts empty with a fresh endpoint/generation; it does not reconstruct tasks, replay envelopes, reuse old ACKs or reopen a worker gate from old history.
|
|
@@ -38,9 +38,13 @@ The `agent_task_send` schema is exactly `to`, `task`, and optional `timeoutMs`.
|
|
|
38
38
|
|
|
39
39
|
`createConfiguredTaskCore()` owns its transport and `createTaskStore()` RAM state. Startup registers; polling runs every five seconds and at agent settlement. Shutdown fences new work, aborts requests, drains active calls and discards state. Late callbacks cannot mutate a successor lifetime.
|
|
40
40
|
|
|
41
|
-
Background refresh records protocol facts in RAM even while Pi is busy or has queued messages. A valid completion within the fetched receive page can become canonical before its notification is presented. Model-visible insertion and the separate follow-up wake still wait for idle Pi with no queued messages; visible relay mail remains replayable until the required structured insertion/wake evidence is present. Intake alone does not incorporate an assignment or authorize worker tools. Parent acknowledgment before terminal presentation retains the event but suppresses its redundant wake.
|
|
41
|
+
Background refresh records protocol facts in RAM even while Pi is busy or has queued messages. A valid completion within the fetched receive page can become canonical before its notification is presented. Model-visible insertion and the separate follow-up wake still wait for idle Pi with no queued messages; visible relay mail remains replayable until the required structured insertion/wake evidence is present. Intake alone does not incorporate an assignment or authorize worker tools. Parent acknowledgment before terminal presentation retains the event but suppresses its redundant wake. Origin-authored conversations do not echo to the origin. Receiver-authored conversation copies remain canonical replicas, but local outbound intent identity makes their presentation archive-only; payload author claims are not provenance. Missing or unmatched intent identity leaves peer conversations actionable. This correlation is RAM-only and is never recovered from session history.
|
|
42
42
|
|
|
43
|
-
|
|
43
|
+
The normal 5s polling cadence and relay byte limits are unchanged. A single-flight background refresh shares one work scope through lease connection, ACK repair, timeout fan-out, receive and all inbox receipts: at most 56 scoped sends plus 8 rotating old-work sends, 48 ACK attempts and one initial receive page of up to 50 items, with a 2s overall deadline. Failed attempts consume the shared counts; the same send/ACK identity is not attempted twice in that scope. Separate 250ms retry/ACK-repair slices stop admitting another attempt, not an already admitted request: each admitted request gets the remaining background window. These are scheduling bounds, not throughput guarantees. Foreground send/message/done retain the configured transport request deadline (default 12s) and caller cancellation instead of inheriting the background cutoff. Any speculative old-work retry after a foreground send uses a finite 2s background window, starting only when that share is reached.
|
|
44
|
+
|
|
45
|
+
Each refresh deliberately fetches only one initial page, even when `hasMore` is true or the ACK frontier advances. Extra-page saturation draining (#27) is deferred, not fixed or optimized; byte limits can make this page shorter than 50 items, and shared send/ACK counts can defer work within it. The normal refresh reserves initial protocol intake ahead of requested-ACK repair; page intake also precedes its own repair pass. Speculative old-work sends wait until all current-page intake and available presentation/evidence/ACK work completes or deliberately yields. Inbox delivery holds that gate across receive and nested fanout/receipt flushes, releasing it in finalization even on empty, held/busy or failed passes when budget remains. Direct core receive suppresses speculation during page materialization and retains its bounded after-intake retry opportunity; an inbox-owned gate remains closed until presentation yields. After required canonical fan-out acceptance, both new and replayed raw intents stage authorized ACK requests in the existing checkpoint rather than retrying inline. The shared after-intake repair pass uses the same ACK budget, admission slice and overall background deadline; remaining raw ACK work reports local deferral on budget/slice/cooldown exhaustion. Failed or pending required fan-out still stops intake at that delivery. Retryable/slow repair still reports failure and retains uncertain requested IDs, but cannot prevent that initial real receive. A failed repair yields before presentation. Ignored-abort ACK responses cannot later advance local authority. Held visible mail, missing insertion/wake evidence, backpressure or exhaustion yields without reading ahead. A full held prefix can still prevent intake of a newer intent beyond that page; general backlog scanning is not provided.
|
|
46
|
+
|
|
47
|
+
Scoped operations attempt their own immutable IDs and cannot report acceptance for omitted or partial fan-out. Concurrent same-ID sends join existing delivery rather than retransmitting. Pending retry IDs rotate incrementally in RAM on attempts/actual skips, including failures and cooldown skips, without copying payloads or rotating an unattempted suffix past the retry deadline. A foreground packet's capacity refusal does not suppress smaller old packets to the same target; same-cycle target suppression starts only when speculative retry itself encounters capacity. Known capacity defers that envelope/ACK for at least 5s (or the validated longer `retryAfterMs`), without a retry timer. `TASK_DELIVERY_DEFERRED` is a retryable local budget/cooldown outcome, not relay acceptance or permanent quarantine. Raw intent replays retain their canonical fan-out reservation and are not ACKed while required sends remain pending; quarantined required sends report their blocked outcome instead of false acceptance. The exported `TaskStore` contract requires indexed lookup, fair rotation and cooldown capabilities, supplied by `createTaskStore()`; there is no legacy snapshot/fixed-prefix fallback. `TaskCore.receive()` retains its array return and the documented transport seam is unchanged.
|
|
44
48
|
|
|
45
49
|
The default extension automatically replaces a lifetime only after a structured `RELAY_RESET` or `RELAY_REBIND_REQUIRED`. It fences and discards the unusable RAM state, records each active task in session history as locally failed with `RELAY_STATE_LOST` and an explicitly unknown remote outcome, then binds a fresh empty endpoint. The loss record does not wake the model by itself. Transient availability, authentication, capacity, abort, profile and malformed-response failures never trigger replacement. Historical tasks are not adopted into the successor lifetime.
|
|
46
50
|
|
package/package.json
CHANGED
package/src/extension.ts
CHANGED
|
@@ -4,6 +4,7 @@ import { Text } from "@earendil-works/pi-tui";
|
|
|
4
4
|
import { Type } from "typebox";
|
|
5
5
|
|
|
6
6
|
import { abortableSleep } from "./abortable-sleep";
|
|
7
|
+
import { createTaskWorkScope } from "./task-work-scope";
|
|
7
8
|
import { deliverTaskInbox, incorporatedTaskEvents } from "./task-inbox";
|
|
8
9
|
import { TaskDeliveryEvidenceState, TaskDeliveryStage, TaskOutboxDeliveryError, TaskProtocolError, isBlockedOutboxDeliveryCode } from "./task-protocol";
|
|
9
10
|
import { createConfiguredTaskCore } from "./configured-task-core";
|
|
@@ -113,18 +114,21 @@ export function registerAgentTaskTools(pi: ExtensionAPI, core: TaskCore | undefi
|
|
|
113
114
|
const epoch = lifecycleEpoch;
|
|
114
115
|
const isCurrent = (): boolean => inboxContext === context && lifecycleEpoch === epoch;
|
|
115
116
|
if (!context) return undefined;
|
|
117
|
+
const work = createTaskWorkScope(signal);
|
|
118
|
+
work.retryAllowed = false; // Reserve initial intake ahead of speculative backlog work and ACK repair.
|
|
119
|
+
signal = work.signal;
|
|
116
120
|
const activeCore = await configuredCore(signal);
|
|
117
121
|
if (!isCurrent()) return undefined;
|
|
118
122
|
let outboxError: unknown;
|
|
119
123
|
try {
|
|
120
|
-
await activeCore.
|
|
124
|
+
await activeCore.connect(signal, work);
|
|
121
125
|
} catch (error) {
|
|
122
126
|
if (isRelayLoss(error)) return error;
|
|
123
127
|
outboxError = error;
|
|
124
128
|
}
|
|
125
129
|
if (!isCurrent()) return undefined;
|
|
126
130
|
try {
|
|
127
|
-
await activeCore.evaluateTimeouts(signal);
|
|
131
|
+
await activeCore.evaluateTimeouts(signal, work);
|
|
128
132
|
} catch (error) {
|
|
129
133
|
if (isRelayLoss(error)) return error;
|
|
130
134
|
outboxError ??= error;
|
|
@@ -132,20 +136,26 @@ export function registerAgentTaskTools(pi: ExtensionAPI, core: TaskCore | undefi
|
|
|
132
136
|
if (!isCurrent()) return undefined;
|
|
133
137
|
const guardedCore: TaskCore = {
|
|
134
138
|
...activeCore,
|
|
135
|
-
async receive(receiveSignal) {
|
|
139
|
+
async receive(receiveSignal, receiveWork) {
|
|
136
140
|
if (!isCurrent()) return [];
|
|
137
|
-
const deliveries = await activeCore.receive(receiveSignal);
|
|
141
|
+
const deliveries = await activeCore.receive(receiveSignal, receiveWork);
|
|
138
142
|
return isCurrent() ? deliveries : [];
|
|
139
143
|
},
|
|
140
|
-
async acknowledgeRelayDelivery(cursor, acknowledgementSignal) {
|
|
141
|
-
if (isCurrent()) await activeCore.acknowledgeRelayDelivery(cursor, acknowledgementSignal);
|
|
144
|
+
async acknowledgeRelayDelivery(cursor, acknowledgementSignal, acknowledgementWork) {
|
|
145
|
+
if (isCurrent()) await activeCore.acknowledgeRelayDelivery(cursor, acknowledgementSignal, acknowledgementWork);
|
|
142
146
|
},
|
|
143
|
-
async recordInsertion(input, insertionSignal) {
|
|
144
|
-
if (isCurrent()) await activeCore.recordInsertion(input, insertionSignal);
|
|
147
|
+
async recordInsertion(input, insertionSignal, insertionWork) {
|
|
148
|
+
if (isCurrent()) await activeCore.recordInsertion(input, insertionSignal, insertionWork);
|
|
145
149
|
},
|
|
146
|
-
|
|
150
|
+
async submitIntent(input, intentSignal, intentWork) {
|
|
151
|
+
if (isCurrent()) await activeCore.submitIntent(input, intentSignal, intentWork);
|
|
152
|
+
},
|
|
153
|
+
async flushOutbox(flushSignal, flushWork) {
|
|
154
|
+
if (isCurrent()) await activeCore.flushOutbox(flushSignal, flushWork);
|
|
155
|
+
},
|
|
156
|
+
recordWakeRequest(input, wakeWork) {
|
|
147
157
|
if (!isCurrent()) throw new TaskProtocolError("RELAY_CLOSED", "task lifecycle was replaced", { retryable: false });
|
|
148
|
-
const request = activeCore.recordWakeRequest(input);
|
|
158
|
+
const request = activeCore.recordWakeRequest(input, wakeWork);
|
|
149
159
|
return { flush: async (flushSignal?: AbortSignal): Promise<void> => {
|
|
150
160
|
if (!isCurrent()) throw new TaskProtocolError("RELAY_CLOSED", "task lifecycle was replaced", { retryable: false });
|
|
151
161
|
await request.flush(flushSignal);
|
|
@@ -159,7 +169,7 @@ export function registerAgentTaskTools(pi: ExtensionAPI, core: TaskCore | undefi
|
|
|
159
169
|
isIdle: (): boolean => isCurrent() && context.isIdle(),
|
|
160
170
|
hasPendingMessages: (): boolean => !isCurrent() || context.hasPendingMessages(),
|
|
161
171
|
sessionManager: context.sessionManager,
|
|
162
|
-
}, signal);
|
|
172
|
+
}, signal, work);
|
|
163
173
|
return outboxError ?? inboxOutboxError;
|
|
164
174
|
});
|
|
165
175
|
const startBackgroundRefresh = (context: ExtensionContext, epoch: number): void => {
|
|
@@ -220,18 +230,11 @@ export function registerAgentTaskTools(pi: ExtensionAPI, core: TaskCore | undefi
|
|
|
220
230
|
let activeCore: (TaskCore & Partial<Pick<OwnedTaskCore, "close">>) | undefined;
|
|
221
231
|
try {
|
|
222
232
|
activeCore = await configuredCore();
|
|
223
|
-
let connectionError: unknown;
|
|
224
|
-
try {
|
|
225
|
-
await activeCore.connect();
|
|
226
|
-
} catch (error) {
|
|
227
|
-
connectionError = error;
|
|
228
|
-
}
|
|
229
233
|
if (!isCurrent()) return;
|
|
230
|
-
if (isRelayLoss(connectionError) && await recoverRelayLoss(context, epoch, activeCore)) return;
|
|
231
234
|
const outboxError = await refreshInbox();
|
|
232
235
|
if (!isCurrent()) return;
|
|
233
236
|
if (isRelayLoss(outboxError) && await recoverRelayLoss(context, epoch, activeCore)) return;
|
|
234
|
-
const lifecycleError = outboxError
|
|
237
|
+
const lifecycleError = outboxError;
|
|
235
238
|
const status = lifecycleError === undefined ? undefined : outboxFailureStatus(lifecycleError);
|
|
236
239
|
context.ui.setStatus("pi-tasks", status === undefined ? undefined : context.ui.theme.fg("warning", status));
|
|
237
240
|
if (lifecycleError === undefined) announceFreshBinding(context);
|
package/src/task-core.ts
CHANGED
|
@@ -21,10 +21,13 @@ import type {
|
|
|
21
21
|
TaskRelay,
|
|
22
22
|
TaskSnapshot,
|
|
23
23
|
} from "./task-protocol";
|
|
24
|
-
import type { TaskStore } from "./task-store";
|
|
24
|
+
import type { TaskStore, OutboxRecord } from "./task-store";
|
|
25
|
+
import { TASK_WORK_LIMITS, capacityRetryAt, createTaskWorkScope, deferredTaskDelivery } from "./task-work-scope";
|
|
26
|
+
import type { TaskWorkScope } from "./task-work-scope";
|
|
25
27
|
|
|
26
|
-
const RECEIVE_PAGE_SIZE =
|
|
28
|
+
const RECEIVE_PAGE_SIZE = TASK_WORK_LIMITS.pageItems;
|
|
27
29
|
const DELIVERY_EVIDENCE_OPERATION = "delivery_evidence";
|
|
30
|
+
const RECEIVED_INTENT_OPERATION = "received_intent";
|
|
28
31
|
const TERMINAL_STATUSES = new Set<TaskRecord["status"]>(["completed", "failed", "cancelled", "timed_out"]);
|
|
29
32
|
const TERMINAL_EVENTS = new Set(["task.completed", "task.failed", "task.cancelled", "task.timed_out"]);
|
|
30
33
|
const CONVERSATIONAL_EVENTS = new Set(["task.information", "task.question", "task.answer"]);
|
|
@@ -68,19 +71,21 @@ export interface WakeRequest {
|
|
|
68
71
|
|
|
69
72
|
export interface TaskCore {
|
|
70
73
|
readonly endpoint: TaskEndpoint;
|
|
71
|
-
connect(signal?: AbortSignal): Promise<void>;
|
|
72
|
-
createTask(input: CreateTaskInput, signal?: AbortSignal): Promise<{ readonly taskId: string }>;
|
|
74
|
+
connect(signal?: AbortSignal, work?: TaskWorkScope): Promise<void>;
|
|
75
|
+
createTask(input: CreateTaskInput, signal?: AbortSignal, work?: TaskWorkScope): Promise<{ readonly taskId: string }>;
|
|
73
76
|
getTask(taskId: string): TaskSnapshot | undefined;
|
|
74
77
|
listTasks(): readonly TaskSnapshot[];
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
78
|
+
/** Correlate a receiver conversation with a locally recorded outbound intent, never session history or payload author claims. */
|
|
79
|
+
isSelfAuthoredConversation(event: TaskEvent): boolean;
|
|
80
|
+
flushOutbox(signal?: AbortSignal, work?: TaskWorkScope): Promise<void>;
|
|
81
|
+
receive(signal?: AbortSignal, work?: TaskWorkScope): Promise<readonly RelayDelivery[]>;
|
|
82
|
+
acknowledgeRelayDelivery(cursor: string, signal?: AbortSignal, work?: TaskWorkScope): Promise<void>;
|
|
83
|
+
submitIntent(input: SubmitIntentInput, signal?: AbortSignal, work?: TaskWorkScope): Promise<void>;
|
|
84
|
+
submitIntentWithOutcome?(input: SubmitIntentInput, signal?: AbortSignal, work?: TaskWorkScope): Promise<SubmitIntentOutcome>;
|
|
85
|
+
recordInsertion(input: { readonly taskId: string; readonly eventId: string }, signal?: AbortSignal, work?: TaskWorkScope): Promise<void>;
|
|
86
|
+
recordWakeRequest(input: { readonly taskId: string; readonly eventId: string }, work?: TaskWorkScope): WakeRequest;
|
|
87
|
+
evaluateTimeouts(signal?: AbortSignal, work?: TaskWorkScope): Promise<void>;
|
|
88
|
+
acknowledgeParent(taskId: string, signal?: AbortSignal, work?: TaskWorkScope): Promise<void>;
|
|
84
89
|
}
|
|
85
90
|
|
|
86
91
|
type ReceivedEnvelope =
|
|
@@ -91,20 +96,34 @@ type ReceivedEnvelope =
|
|
|
91
96
|
export function createTaskCore(options: TaskCoreOptions): TaskCore {
|
|
92
97
|
const clock = options.clock ?? { now: (): number => Date.now() };
|
|
93
98
|
const ids = options.ids ?? (() => crypto.randomUUID());
|
|
94
|
-
const
|
|
99
|
+
const flush = createOutboxFlusher(options);
|
|
100
|
+
const acknowledgeWithSignal = async (cursor: string, signal: AbortSignal, work: TaskWorkScope): Promise<void> => {
|
|
95
101
|
if (!decimal(cursor)) throw new TaskProtocolError("INVALID_CURSOR", "relay cursor must be a non-negative integer");
|
|
96
102
|
const envelopeId = options.store.pendingRelayEnvelopeId(options.endpoint, cursor);
|
|
97
103
|
if (envelopeId === undefined) throw new TaskProtocolError("INVALID_CURSOR", "relay delivery is not pending in this endpoint scope");
|
|
104
|
+
const retryAt = options.store.relayAcknowledgementRetryAt(options.endpoint, cursor);
|
|
105
|
+
if (retryAt > Date.now() || !work.claimAcknowledgement(cursor)) throw deferredTaskDelivery({ cursor, retryAfterMs: Math.max(0, retryAt - Date.now()) });
|
|
98
106
|
options.store.transaction(() => options.store.requestRelayAcknowledgement(options.endpoint, cursor));
|
|
99
|
-
|
|
100
|
-
|
|
107
|
+
try {
|
|
108
|
+
await waitForRelayOperation(options.relay.acknowledgeDelivery({ endpoint: options.endpoint, cursor, envelopeId }, signal), signal);
|
|
109
|
+
throwIfAborted(signal);
|
|
110
|
+
options.store.transaction(() => options.store.acknowledgeRelayCursor(options.endpoint, cursor));
|
|
111
|
+
} catch (error) {
|
|
112
|
+
if (signal.aborted) throw error;
|
|
113
|
+
const retryAt = capacityRetryAt(error);
|
|
114
|
+
if (retryAt !== undefined) options.store.transaction(() => options.store.requestRelayAcknowledgement(options.endpoint, cursor, retryAt));
|
|
115
|
+
throw error;
|
|
116
|
+
}
|
|
101
117
|
};
|
|
102
|
-
const
|
|
103
|
-
|
|
104
|
-
|
|
118
|
+
const acknowledgeRelayDelivery = (cursor: string, signal?: AbortSignal, work = createTaskWorkScope(signal)): Promise<void> => acknowledgeWithSignal(cursor, work.signal, work);
|
|
119
|
+
const retryRequestedAcknowledgements = async (work: TaskWorkScope): Promise<void> => {
|
|
120
|
+
for (const cursor of options.store.requestedRelayAcknowledgements(options.endpoint, TASK_WORK_LIMITS.acknowledgements, Date.now())) {
|
|
121
|
+
if (!work.canAcknowledge() || !work.canRepairAcknowledgement()) break;
|
|
122
|
+
if (!work.attemptedAcknowledgements.has(cursor) && options.store.pendingRelayEnvelopeId(options.endpoint, cursor) !== undefined) await acknowledgeWithSignal(cursor, work.backgroundSignal(), work);
|
|
105
123
|
}
|
|
106
124
|
};
|
|
107
|
-
const submitIntentWithOutcome = async (input: SubmitIntentInput, signal?: AbortSignal): Promise<SubmitIntentOutcome> => {
|
|
125
|
+
const submitIntentWithOutcome = async (input: SubmitIntentInput, signal?: AbortSignal, work = createTaskWorkScope(signal, "foreground")): Promise<SubmitIntentOutcome> => {
|
|
126
|
+
signal = work.signal;
|
|
108
127
|
if (!RECEIVER_INTENT_TYPES.has(input.type) || !isRecord(input.payload)) throw new TaskProtocolError("INVALID_INTENT", "intent type or payload is invalid");
|
|
109
128
|
let persistedEnvelopeIds: readonly string[] = [];
|
|
110
129
|
let outcome: SubmitIntentOutcome | undefined;
|
|
@@ -120,10 +139,10 @@ export function createTaskCore(options: TaskCoreOptions): TaskCore {
|
|
|
120
139
|
outcome = persisted.outcome;
|
|
121
140
|
});
|
|
122
141
|
if (evidence !== undefined) {
|
|
123
|
-
await flushDeliveryEvidence(options, signal, input.taskId, persistedEnvelopeIds, evidence);
|
|
142
|
+
await flushDeliveryEvidence(options, signal, input.taskId, persistedEnvelopeIds, evidence, work, flush);
|
|
124
143
|
} else {
|
|
125
144
|
try {
|
|
126
|
-
await flush(
|
|
145
|
+
await flush(signal, new Set(persistedEnvelopeIds), work);
|
|
127
146
|
} catch (error) {
|
|
128
147
|
if (originCancellation) throwIfCancellationDeliveryBlocked(options, input.taskId, persistedEnvelopeIds);
|
|
129
148
|
throw error;
|
|
@@ -137,15 +156,19 @@ export function createTaskCore(options: TaskCoreOptions): TaskCore {
|
|
|
137
156
|
|
|
138
157
|
return {
|
|
139
158
|
endpoint: options.endpoint,
|
|
140
|
-
async connect(signal) {
|
|
159
|
+
async connect(signal, work = createTaskWorkScope(signal)) {
|
|
160
|
+
signal = work.signal;
|
|
141
161
|
const connection = await options.relay.connect({ endpoint: options.endpoint, protocolVersion: TASK_PROTOCOL_VERSION, receiveCursor: options.store.getReceiveCursor() }, signal);
|
|
142
162
|
if (!sameEndpoint(connection.endpoint, options.endpoint)) throw new TaskProtocolError("INVALID_CONNECTION", "relay connected a different endpoint");
|
|
143
163
|
throwIfAborted(signal);
|
|
144
|
-
|
|
164
|
+
// A reserved refresh repairs only after the initial page has been intaken.
|
|
165
|
+
if (work.retryAllowed) await retryRequestedAcknowledgements(work);
|
|
145
166
|
},
|
|
146
|
-
async createTask(input, signal) {
|
|
167
|
+
async createTask(input, signal, work = createTaskWorkScope(signal, "foreground")) {
|
|
168
|
+
signal = work.signal;
|
|
147
169
|
if (input.task.length === 0 || !Number.isInteger(input.timeoutMs) || input.timeoutMs < 1) throw new TaskProtocolError("INVALID_TASK", "task and a positive timeout are required");
|
|
148
170
|
const target = await options.relay.resolve({ relay: input.target.relay, reference: input.target.id }, signal);
|
|
171
|
+
throwIfAborted(signal);
|
|
149
172
|
if (!isEndpoint(target)) throw new TaskProtocolError("INVALID_TARGET", "relay resolved an invalid task endpoint");
|
|
150
173
|
const now = clock.now();
|
|
151
174
|
const taskId = ids();
|
|
@@ -161,7 +184,7 @@ export function createTaskCore(options: TaskCoreOptions): TaskCore {
|
|
|
161
184
|
options.store.putOutbox(assignment);
|
|
162
185
|
});
|
|
163
186
|
try {
|
|
164
|
-
await flush(
|
|
187
|
+
await flush(signal, new Set([assignment.envelopeId]), work);
|
|
165
188
|
} catch (error) {
|
|
166
189
|
throw operationError(error, taskId);
|
|
167
190
|
}
|
|
@@ -169,44 +192,72 @@ export function createTaskCore(options: TaskCoreOptions): TaskCore {
|
|
|
169
192
|
},
|
|
170
193
|
getTask(taskId) { return options.store.getTask(taskId); },
|
|
171
194
|
listTasks() { return options.store.listTasks(); },
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
const
|
|
178
|
-
if (!
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
if (
|
|
199
|
-
|
|
200
|
-
|
|
195
|
+
isSelfAuthoredConversation(event) {
|
|
196
|
+
if (!CONVERSATIONAL_EVENTS.has(event.type) || typeof event.payload.intentId !== "string") return false;
|
|
197
|
+
const task = options.store.getTask(event.taskId);
|
|
198
|
+
if (!task || !sameEndpoint(options.endpoint, task.target) || sameEndpoint(options.endpoint, task.origin)
|
|
199
|
+
|| !sameEndpoint(event.source, task.origin) || !sameEndpoint(event.target, task.target)) return false;
|
|
200
|
+
const localEnvelope = options.store.getIntentEnvelope(event.payload.intentId);
|
|
201
|
+
if (!localEnvelope || localEnvelope.kind !== TaskEnvelopeKind.intent || localEnvelope.taskId !== task.taskId
|
|
202
|
+
|| !sameEndpoint(localEnvelope.source, options.endpoint) || !sameEndpoint(localEnvelope.target, task.origin)) return false;
|
|
203
|
+
const intent = parsePayload(localEnvelope);
|
|
204
|
+
return isTaskIntent(intent) && intent.taskId === event.taskId && intent.type === event.type && intent.intentId === event.payload.intentId;
|
|
205
|
+
},
|
|
206
|
+
async flushOutbox(signal, work = createTaskWorkScope(signal)) { await flush(work.signal, undefined, work); },
|
|
207
|
+
async receive(signal, work = createTaskWorkScope(signal)) {
|
|
208
|
+
signal = work.signal;
|
|
209
|
+
const retryAllowed = work.retryAllowed;
|
|
210
|
+
work.retryAllowed = false; // Nested fanout must finish this page before unrelated retries.
|
|
211
|
+
try {
|
|
212
|
+
const cursor = options.store.getReceiveCursor();
|
|
213
|
+
const page = await options.relay.receive({ endpoint: options.endpoint, cursor, limit: RECEIVE_PAGE_SIZE }, signal);
|
|
214
|
+
throwIfAborted(signal);
|
|
215
|
+
if (!isInboxPage(page) || page.deliveries.length > RECEIVE_PAGE_SIZE) throw new TaskProtocolError("INVALID_INBOX", "relay returned an invalid inbox page");
|
|
216
|
+
let previous = BigInt(cursor);
|
|
217
|
+
for (const delivery of page.deliveries) {
|
|
218
|
+
if (!isDelivery(delivery) || !sameEndpoint(delivery.envelope.target, options.endpoint) || BigInt(delivery.cursor) <= previous) throw new TaskProtocolError("INVALID_DELIVERY", "relay returned an invalid or unordered delivery");
|
|
219
|
+
previous = BigInt(delivery.cursor);
|
|
220
|
+
}
|
|
221
|
+
if (page.nextCursor !== (page.deliveries.at(-1)?.cursor ?? cursor) || (!page.deliveries.length && page.hasMore)) throw new TaskProtocolError("INVALID_INBOX", "relay page cursor does not describe its deliveries");
|
|
222
|
+
// Track the entire observed prefix before an intent can ACK a later
|
|
223
|
+
// delivery. Keep only pending IDs plus a high-water mark, durably.
|
|
224
|
+
options.store.transaction(() => options.store.trackRelayDeliveries(options.endpoint, page.deliveries.map(delivery => ({ cursor: delivery.cursor, envelopeId: delivery.envelope.envelopeId }))));
|
|
225
|
+
const visibleDeliveries: RelayDelivery[] = [];
|
|
226
|
+
for (const delivery of page.deliveries) {
|
|
227
|
+
if (options.store.pendingRelayEnvelopeId(options.endpoint, delivery.cursor) === undefined) continue;
|
|
228
|
+
const received = validateReceivedEnvelope(options, delivery.envelope);
|
|
229
|
+
let persistedEnvelopeIds: readonly string[] = [];
|
|
230
|
+
options.store.transaction(() => {
|
|
231
|
+
if (!options.store.persistInbox(delivery.envelope, delivery.cursor) && received.kind !== "intent") return;
|
|
232
|
+
persistedEnvelopeIds = persistReceivedEnvelope(options, clock.now, ids, received, delivery.envelope.envelopeId);
|
|
233
|
+
});
|
|
234
|
+
if (persistedEnvelopeIds.length > 0) await flush(signal, new Set(persistedEnvelopeIds), work);
|
|
235
|
+
if (received.kind === "intent") {
|
|
236
|
+
// Accepted fanout authorizes ACK, but raw repair must not gate later
|
|
237
|
+
// facts in this fetched page. The checkpoint owns the staged request.
|
|
238
|
+
options.store.transaction(() => options.store.requestRelayAcknowledgement(options.endpoint, delivery.cursor,
|
|
239
|
+
options.store.relayAcknowledgementRetryAt(options.endpoint, delivery.cursor)));
|
|
240
|
+
continue;
|
|
241
|
+
}
|
|
242
|
+
visibleDeliveries.push(delivery);
|
|
201
243
|
}
|
|
202
|
-
|
|
244
|
+
// Repair failure remains visible, but cannot gate this page's protocol intake.
|
|
245
|
+
await retryRequestedAcknowledgements(work);
|
|
246
|
+
const pendingRaw = page.deliveries.find(delivery => delivery.envelope.kind === "intent" && options.store.pendingRelayEnvelopeId(options.endpoint, delivery.cursor) !== undefined);
|
|
247
|
+
if (pendingRaw) throw deferredTaskDelivery({ cursor: pendingRaw.cursor,
|
|
248
|
+
retryAfterMs: Math.max(0, options.store.relayAcknowledgementRetryAt(options.endpoint, pendingRaw.cursor) - Date.now()) });
|
|
249
|
+
return visibleDeliveries;
|
|
250
|
+
} finally {
|
|
251
|
+
work.retryAllowed = retryAllowed;
|
|
252
|
+
// Inbox delivery owns the later retry share; direct receive retains its own.
|
|
253
|
+
if (retryAllowed && !work.signal.aborted) await flush(work.signal, IGNORE_OUTBOX_FAILURES, work);
|
|
203
254
|
}
|
|
204
|
-
return visibleDeliveries;
|
|
205
255
|
},
|
|
206
256
|
acknowledgeRelayDelivery,
|
|
207
|
-
async submitIntent(input, signal) { await submitIntentWithOutcome(input, signal); },
|
|
257
|
+
async submitIntent(input, signal, work) { await submitIntentWithOutcome(input, signal, work); },
|
|
208
258
|
submitIntentWithOutcome,
|
|
209
|
-
async recordInsertion(input, signal) {
|
|
259
|
+
async recordInsertion(input, signal, work = createTaskWorkScope(signal)) {
|
|
260
|
+
signal = work.signal;
|
|
210
261
|
const task = requiredTask(options.store, input.taskId);
|
|
211
262
|
const evidence: DeliveryEvidence = { eventId: input.eventId, stage: TaskDeliveryStage.piInserted, state: TaskDeliveryEvidenceState.confirmed };
|
|
212
263
|
let persistedEnvelopeIds: readonly string[] = [];
|
|
@@ -218,9 +269,9 @@ export function createTaskCore(options: TaskCoreOptions): TaskCore {
|
|
|
218
269
|
payload: { eventId: input.eventId, stage: evidence.stage, state: evidence.state },
|
|
219
270
|
}, deliveryEvidenceOperation(evidence)).envelopeIds;
|
|
220
271
|
});
|
|
221
|
-
await flushDeliveryEvidence(options, signal, input.taskId, persistedEnvelopeIds, evidence);
|
|
272
|
+
await flushDeliveryEvidence(options, signal, input.taskId, persistedEnvelopeIds, evidence, work, flush);
|
|
222
273
|
},
|
|
223
|
-
recordWakeRequest(input) {
|
|
274
|
+
recordWakeRequest(input, work) {
|
|
224
275
|
const task = requiredTask(options.store, input.taskId);
|
|
225
276
|
const evidence: DeliveryEvidence = { eventId: input.eventId, stage: TaskDeliveryStage.wakeRequested, state: TaskDeliveryEvidenceState.confirmed };
|
|
226
277
|
let persistedEnvelopeIds: readonly string[] = [];
|
|
@@ -232,10 +283,12 @@ export function createTaskCore(options: TaskCoreOptions): TaskCore {
|
|
|
232
283
|
}, deliveryEvidenceOperation(evidence)).envelopeIds;
|
|
233
284
|
});
|
|
234
285
|
return { flush: async (signal?: AbortSignal): Promise<void> => {
|
|
235
|
-
|
|
286
|
+
const activeWork = work ?? createTaskWorkScope(signal);
|
|
287
|
+
await flushDeliveryEvidence(options, activeWork.signal, input.taskId, persistedEnvelopeIds, evidence, activeWork, flush);
|
|
236
288
|
} };
|
|
237
289
|
},
|
|
238
|
-
async evaluateTimeouts(signal) {
|
|
290
|
+
async evaluateTimeouts(signal, work = createTaskWorkScope(signal)) {
|
|
291
|
+
signal = work.signal;
|
|
239
292
|
const persistedEnvelopeIds: string[] = [];
|
|
240
293
|
for (const candidate of options.store.listTasks()) {
|
|
241
294
|
if (!sameEndpoint(candidate.origin, options.endpoint) || TERMINAL_STATUSES.has(candidate.status) || candidate.expiresAt > clock.now()) continue;
|
|
@@ -245,9 +298,10 @@ export function createTaskCore(options: TaskCoreOptions): TaskCore {
|
|
|
245
298
|
persistedEnvelopeIds.push(...canonicalize(options, clock.now, ids, task, { intentId: ids(), taskId: task.taskId, type: "task.cancelled", payload: {} }, "task.timed_out").envelopeIds);
|
|
246
299
|
});
|
|
247
300
|
}
|
|
248
|
-
if (persistedEnvelopeIds.length > 0) await flush(
|
|
301
|
+
if (persistedEnvelopeIds.length > 0) await flush(signal, new Set(persistedEnvelopeIds), work);
|
|
249
302
|
},
|
|
250
|
-
async acknowledgeParent(taskId, signal) {
|
|
303
|
+
async acknowledgeParent(taskId, signal, work = createTaskWorkScope(signal)) {
|
|
304
|
+
signal = work.signal;
|
|
251
305
|
let persistedEnvelopeIds: readonly string[] = [];
|
|
252
306
|
options.store.transaction(() => {
|
|
253
307
|
const task = requiredTask(options.store, taskId);
|
|
@@ -255,7 +309,7 @@ export function createTaskCore(options: TaskCoreOptions): TaskCore {
|
|
|
255
309
|
if (!TERMINAL_STATUSES.has(task.status)) throw new TaskProtocolError("TASK_NOT_TERMINAL", "only a terminal task may be acknowledged", { retryable: false });
|
|
256
310
|
persistedEnvelopeIds = canonicalize(options, clock.now, ids, task, { intentId: ids(), taskId, type: "task.information", payload: {} }, "task.parent_acknowledged", PARENT_ACKNOWLEDGMENT_OPERATION).envelopeIds;
|
|
257
311
|
});
|
|
258
|
-
await flush(
|
|
312
|
+
await flush(signal, new Set(persistedEnvelopeIds), work);
|
|
259
313
|
},
|
|
260
314
|
};
|
|
261
315
|
}
|
|
@@ -305,13 +359,13 @@ function validateCanonicalEvent(options: TaskCoreOptions, envelope: RelayEnvelop
|
|
|
305
359
|
return { kind: "canonical_event", task, event: payload };
|
|
306
360
|
}
|
|
307
361
|
|
|
308
|
-
function persistReceivedEnvelope(options: TaskCoreOptions, now: () => number, ids: () => string, received: ReceivedEnvelope): readonly string[] {
|
|
362
|
+
function persistReceivedEnvelope(options: TaskCoreOptions, now: () => number, ids: () => string, received: ReceivedEnvelope, envelopeId: string): readonly string[] {
|
|
309
363
|
if (received.kind === "assignment") {
|
|
310
364
|
options.store.putTask(received.task);
|
|
311
365
|
options.store.appendEvent(received.event);
|
|
312
366
|
return [];
|
|
313
367
|
}
|
|
314
|
-
if (received.kind === "intent") return canonicalize(options, now, ids, received.task, received.intent, received.intent.type).envelopeIds;
|
|
368
|
+
if (received.kind === "intent") return canonicalize(options, now, ids, received.task, received.intent, received.intent.type, `${RECEIVED_INTENT_OPERATION}:${envelopeId}`).envelopeIds;
|
|
315
369
|
if (options.store.appendEvent(received.event) && TERMINAL_EVENTS.has(received.event.type)) options.store.setStatus(received.task.taskId, statusFor(received.event.type));
|
|
316
370
|
return [];
|
|
317
371
|
}
|
|
@@ -391,7 +445,7 @@ function canonicalize(options: TaskCoreOptions, now: () => number, ids: () => st
|
|
|
391
445
|
const terminal = TERMINAL_EVENTS.has(requestedType);
|
|
392
446
|
const type = terminal && TERMINAL_STATUSES.has(task.status) ? "task.late_terminal" : requestedType;
|
|
393
447
|
const sequence = String(task.events.length + 1);
|
|
394
|
-
const canonical = event(task, ids(), type, sequence, options.endpoint, task.target, now(), { intentId: intent.intentId
|
|
448
|
+
const canonical = event(task, ids(), type, sequence, options.endpoint, task.target, now(), { ...intent.payload, intentId: intent.intentId });
|
|
395
449
|
const suppressSelfTargetConversationEcho = !echoCanonicalEventToOrigin && sameEndpoint(task.origin, task.target);
|
|
396
450
|
const targetEnvelope = suppressSelfTargetConversationEcho ? undefined : envelope(ids(), options.endpoint, task.target, task.taskId, TaskEnvelopeKind.canonicalEvent, canonical, canonical.occurredAt);
|
|
397
451
|
const originEnvelope = !echoCanonicalEventToOrigin || sameEndpoint(task.origin, task.target) ? undefined : envelope(ids(), options.endpoint, task.origin, task.taskId, TaskEnvelopeKind.canonicalEvent, canonical, canonical.occurredAt);
|
|
@@ -407,9 +461,9 @@ function canonicalize(options: TaskCoreOptions, now: () => number, ids: () => st
|
|
|
407
461
|
return { envelopeIds: persistedEnvelopeIds, eventType: type, reused: false };
|
|
408
462
|
}
|
|
409
463
|
|
|
410
|
-
async function flushDeliveryEvidence(options: TaskCoreOptions, signal: AbortSignal | undefined, taskId: string, envelopeIds: readonly string[], evidence: DeliveryEvidence): Promise<void> {
|
|
464
|
+
async function flushDeliveryEvidence(options: TaskCoreOptions, signal: AbortSignal | undefined, taskId: string, envelopeIds: readonly string[], evidence: DeliveryEvidence, work: TaskWorkScope, flush: OutboxFlusher): Promise<void> {
|
|
411
465
|
try {
|
|
412
|
-
if (envelopeIds.length > 0) await flush(
|
|
466
|
+
if (envelopeIds.length > 0) await flush(signal, new Set(envelopeIds), work);
|
|
413
467
|
} catch (error) {
|
|
414
468
|
throwIfDeliveryEvidenceBlocked(options.store, taskId, envelopeIds, evidence);
|
|
415
469
|
throw error;
|
|
@@ -417,29 +471,84 @@ async function flushDeliveryEvidence(options: TaskCoreOptions, signal: AbortSign
|
|
|
417
471
|
throwIfDeliveryEvidenceBlocked(options.store, taskId, envelopeIds, evidence);
|
|
418
472
|
}
|
|
419
473
|
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
|
|
423
|
-
|
|
424
|
-
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
|
|
428
|
-
|
|
429
|
-
|
|
430
|
-
|
|
431
|
-
|
|
432
|
-
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
|
|
474
|
+
type OutboxFlusher = (signal: AbortSignal | undefined, reportedEnvelopeIds: ReadonlySet<string> | undefined, work: TaskWorkScope) => Promise<void>;
|
|
475
|
+
|
|
476
|
+
function createOutboxFlusher(options: TaskCoreOptions): OutboxFlusher {
|
|
477
|
+
const inFlight = new Map<string, Promise<void>>();
|
|
478
|
+
const send = async (record: OutboxRecord, retry: boolean, work: TaskWorkScope): Promise<void> => {
|
|
479
|
+
const id = record.envelope.envelopeId;
|
|
480
|
+
const joined = inFlight.get(id);
|
|
481
|
+
if (joined) { await waitForRelayOperation(joined, work.signal); return; }
|
|
482
|
+
const retryAt = options.store.outboxRetryAt(id);
|
|
483
|
+
if (retryAt > Date.now() || !work.claimSend(id, retry)) throw deferredTaskDelivery({ envelopeId: id, retryAfterMs: Math.max(0, retryAt - Date.now()) });
|
|
484
|
+
const signal = retry ? work.backgroundSignal() : work.signal;
|
|
485
|
+
const delivery = (async (): Promise<void> => {
|
|
486
|
+
try {
|
|
487
|
+
await waitForRelayOperation(options.relay.send(record.envelope, signal), signal);
|
|
488
|
+
throwIfAborted(signal);
|
|
489
|
+
options.store.transaction(() => options.store.markOutboxAccepted(id));
|
|
490
|
+
} catch (error) {
|
|
491
|
+
if (signal.aborted) throw error;
|
|
492
|
+
const retryAt = capacityRetryAt(error);
|
|
493
|
+
if (retryAt !== undefined) {
|
|
494
|
+
options.store.postponeOutbox(id, retryAt);
|
|
495
|
+
// A large foreground refusal does not prove smaller old mail cannot fit.
|
|
496
|
+
if (retry) work.blockedRetryTargets.add(JSON.stringify(record.envelope.target));
|
|
497
|
+
}
|
|
498
|
+
if (error instanceof TaskProtocolError && isBlockedOutboxDeliveryCode(error.code) && !error.retryable) options.store.transaction(() => {
|
|
499
|
+
options.store.quarantineOutbox(id, { errorCode: error.code, reason: error.message, details: error.details ?? {}, quarantinedAt: Date.now() });
|
|
436
500
|
});
|
|
501
|
+
throw error;
|
|
502
|
+
}
|
|
503
|
+
})();
|
|
504
|
+
inFlight.set(id, delivery);
|
|
505
|
+
try { await delivery; } finally { if (inFlight.get(id) === delivery) inFlight.delete(id); }
|
|
506
|
+
};
|
|
507
|
+
return async (signal, reportedEnvelopeIds, work) => {
|
|
508
|
+
throwIfAborted(signal);
|
|
509
|
+
let failure: unknown;
|
|
510
|
+
if (reportedEnvelopeIds) for (const id of reportedEnvelopeIds) {
|
|
511
|
+
const record = options.store.getOutboxRecord(id);
|
|
512
|
+
if (record?.state === "accepted") continue;
|
|
513
|
+
if (!record) {
|
|
514
|
+
const quarantined = options.store.quarantinedOutbox().find(item => item.envelope.envelopeId === id);
|
|
515
|
+
failure ??= quarantined ? new TaskProtocolError(quarantined.errorCode, quarantined.reason, { retryable: false, details: quarantined.details }) : deferredTaskDelivery({ envelopeId: id });
|
|
516
|
+
continue;
|
|
517
|
+
}
|
|
518
|
+
try { await send(record, false, work); } catch (error) {
|
|
519
|
+
if (work.signal.aborted) throw error;
|
|
520
|
+
failure ??= error;
|
|
521
|
+
if (error instanceof TaskProtocolError && error.code === "TASK_DELIVERY_DEFERRED") break;
|
|
437
522
|
}
|
|
438
|
-
failures.push({ envelopeId: record.envelope.envelopeId, error });
|
|
439
523
|
}
|
|
440
|
-
|
|
441
|
-
|
|
442
|
-
|
|
524
|
+
if (work.retryAllowed && !work.retryStarted) {
|
|
525
|
+
work.retryStarted = true;
|
|
526
|
+
let attempts = 0;
|
|
527
|
+
for (let inspected = 0; inspected < TASK_WORK_LIMITS.retryInspections && attempts < TASK_WORK_LIMITS.retries; inspected++) {
|
|
528
|
+
if (!options.store.hasPendingOutbox() || !work.canRetry()) break;
|
|
529
|
+
// Advance one attempted/actually skipped ID, never an unattempted batch suffix.
|
|
530
|
+
const record = options.store.selectPendingOutbox(1, Date.now(), 1)[0];
|
|
531
|
+
if (!record) continue;
|
|
532
|
+
const id = record.envelope.envelopeId;
|
|
533
|
+
if (reportedEnvelopeIds?.has(id) || work.attemptedSends.has(id) || inFlight.has(id) || work.blockedRetryTargets.has(JSON.stringify(record.envelope.target))) continue;
|
|
534
|
+
attempts++;
|
|
535
|
+
try { await send(record, true, work); } catch (error) {
|
|
536
|
+
if (work.signal.aborted) throw error;
|
|
537
|
+
if (reportedEnvelopeIds === undefined) failure ??= error;
|
|
538
|
+
}
|
|
539
|
+
}
|
|
540
|
+
}
|
|
541
|
+
if (failure === undefined && reportedEnvelopeIds === undefined && options.store.hasPendingOutbox()) failure = deferredTaskDelivery();
|
|
542
|
+
if (failure !== undefined) throw outboxDeliveryError(failure);
|
|
543
|
+
};
|
|
544
|
+
}
|
|
545
|
+
|
|
546
|
+
async function waitForRelayOperation(delivery: Promise<unknown>, signal: AbortSignal): Promise<void> {
|
|
547
|
+
throwIfAborted(signal);
|
|
548
|
+
let abort!: () => void;
|
|
549
|
+
const cancelled = new Promise<never>((_, reject) => { abort = () => reject(new TaskProtocolError("ABORTED", "task operation aborted", { details: { mayHaveBeenDelivered: true } })); signal.addEventListener("abort", abort, { once: true }); });
|
|
550
|
+
try { await Promise.race([delivery, cancelled]); throwIfAborted(signal); }
|
|
551
|
+
finally { signal.removeEventListener("abort", abort); }
|
|
443
552
|
}
|
|
444
553
|
|
|
445
554
|
function outboxDeliveryError(error: unknown): TaskOutboxDeliveryError {
|
package/src/task-inbox.ts
CHANGED
|
@@ -1,4 +1,6 @@
|
|
|
1
1
|
import type { TaskCore } from "./task-core";
|
|
2
|
+
import { createTaskWorkScope } from "./task-work-scope";
|
|
3
|
+
import type { TaskWorkScope } from "./task-work-scope";
|
|
2
4
|
import { TaskDeliveryEvidenceState, TaskDeliveryStage, TaskEnvelopeKind, TaskOutboxDeliveryError, TaskProtocolError, isBlockedOutboxDeliveryCode } from "./task-protocol";
|
|
3
5
|
import type { RelayDelivery, TaskEvent } from "./task-protocol";
|
|
4
6
|
|
|
@@ -34,78 +36,90 @@ type DeliveryEvidencePayload =
|
|
|
34
36
|
| { readonly stage: typeof TaskDeliveryStage.piInsertion; readonly state: typeof TaskDeliveryEvidenceState.blocked; readonly retryable: true };
|
|
35
37
|
|
|
36
38
|
/** Persists model-visible Pi evidence before advancing the relay cursor, then starts one separate turn. */
|
|
37
|
-
export async function deliverTaskInbox(pi: InboxPi, core: TaskCore, context: InboxContext, signal?: AbortSignal): Promise<TaskOutboxDeliveryError | undefined> {
|
|
38
|
-
|
|
39
|
+
export async function deliverTaskInbox(pi: InboxPi, core: TaskCore, context: InboxContext, signal?: AbortSignal, work = createTaskWorkScope(signal)): Promise<TaskOutboxDeliveryError | undefined> {
|
|
40
|
+
signal = work.signal;
|
|
41
|
+
work.retryAllowed = false; // Reserve intake, presentation and evidence/ACK ahead of old work.
|
|
39
42
|
let outboxError: TaskOutboxDeliveryError | undefined;
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
if (!incorporated && isModelVisible(event.type)) {
|
|
62
|
-
if (!context.isIdle()) return outboxError;
|
|
63
|
-
pi.sendMessage({
|
|
64
|
-
customType: TASK_EVENT_CUSTOM_TYPE,
|
|
65
|
-
content: renderTaskEvent(event),
|
|
66
|
-
display: true,
|
|
67
|
-
details: { ...eventDetails, event },
|
|
68
|
-
}, { triggerTurn: false });
|
|
69
|
-
incorporated = incorporatedEvents(context.sessionManager.getEntries()).has(eventKey);
|
|
70
|
-
if (!incorporated) {
|
|
71
|
-
const evidenceError = await recordDeliveryEvidence(core, eventDetails, { stage: TaskDeliveryStage.piInsertion, state: TaskDeliveryEvidenceState.blocked, retryable: true }, signal);
|
|
43
|
+
try {
|
|
44
|
+
const deliveries = await core.receive(signal, work);
|
|
45
|
+
for (const delivery of deliveries) {
|
|
46
|
+
if (delivery.envelope.kind === TaskEnvelopeKind.intent) {
|
|
47
|
+
await core.acknowledgeRelayDelivery(delivery.cursor, signal, work);
|
|
48
|
+
pi.appendEntry(TASK_CURSOR_CUSTOM_TYPE, { cursor: delivery.cursor });
|
|
49
|
+
continue;
|
|
50
|
+
}
|
|
51
|
+
const event = inboxEvent(delivery);
|
|
52
|
+
if (!isKnownEvent(event.type)) throw new TaskProtocolError("UNKNOWN_EVENT", `unknown task inbox event type: ${event.type}`);
|
|
53
|
+
if (context.hasPendingMessages()) break;
|
|
54
|
+
const eventDetails = { taskId: event.taskId, eventId: event.eventId };
|
|
55
|
+
const eventKey = key(event.taskId, event.eventId);
|
|
56
|
+
const modelVisible = isModelVisible(event.type) && !core.isSelfAuthoredConversation(event);
|
|
57
|
+
if (!modelVisible && !recordedEventKeys(context.sessionManager.getEntries()).has(eventKey)) {
|
|
58
|
+
// Archive control facts and confirmed self-authored conversations without model insertion or wake.
|
|
59
|
+
// This is evidence only: no task state or delivery queue is reconstructed.
|
|
60
|
+
pi.appendEntry(TASK_RECORD_CUSTOM_TYPE, { ...eventDetails, event });
|
|
61
|
+
}
|
|
62
|
+
if (event.type === "task.created") {
|
|
63
|
+
const evidenceError = await recordDeliveryEvidence(core, eventDetails, { stage: TaskDeliveryStage.receiverRecorded, state: TaskDeliveryEvidenceState.confirmed }, signal, work);
|
|
72
64
|
outboxError ??= evidenceError;
|
|
73
|
-
return outboxError;
|
|
74
65
|
}
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
outboxError ??= insertionError;
|
|
79
|
-
let wakeAccepted = taskMessageKeys(context.sessionManager.getEntries(), TASK_WAKE_CUSTOM_TYPE).has(eventKey);
|
|
80
|
-
const wakeSuppressed = !wakeAccepted && hasAcknowledgedLocalOriginTerminal(core, event);
|
|
81
|
-
if (!wakeAccepted && !wakeSuppressed) {
|
|
82
|
-
if (!context.isIdle()) return outboxError;
|
|
83
|
-
// This synchronous local write closes the ACK/wake race; do not await relay I/O before Pi emits the wake.
|
|
84
|
-
const wakeRequest = core.recordWakeRequest(eventDetails);
|
|
66
|
+
let incorporated = incorporatedEvents(context.sessionManager.getEntries()).has(eventKey);
|
|
67
|
+
if (!incorporated && modelVisible) {
|
|
68
|
+
if (!context.isIdle()) break;
|
|
85
69
|
pi.sendMessage({
|
|
86
|
-
customType:
|
|
87
|
-
content:
|
|
88
|
-
display:
|
|
89
|
-
details: eventDetails,
|
|
90
|
-
}, { triggerTurn:
|
|
91
|
-
|
|
92
|
-
if (!
|
|
93
|
-
|
|
94
|
-
|
|
70
|
+
customType: TASK_EVENT_CUSTOM_TYPE,
|
|
71
|
+
content: renderTaskEvent(event),
|
|
72
|
+
display: true,
|
|
73
|
+
details: { ...eventDetails, event },
|
|
74
|
+
}, { triggerTurn: false });
|
|
75
|
+
incorporated = incorporatedEvents(context.sessionManager.getEntries()).has(eventKey);
|
|
76
|
+
if (!incorporated) {
|
|
77
|
+
const evidenceError = await recordDeliveryEvidence(core, eventDetails, { stage: TaskDeliveryStage.piInsertion, state: TaskDeliveryEvidenceState.blocked, retryable: true }, signal, work);
|
|
78
|
+
outboxError ??= evidenceError;
|
|
79
|
+
break;
|
|
80
|
+
}
|
|
95
81
|
}
|
|
96
|
-
if (
|
|
97
|
-
const
|
|
98
|
-
outboxError ??=
|
|
82
|
+
if (modelVisible) {
|
|
83
|
+
const insertionError = await settleDeliveryEvidence(() => core.recordInsertion(eventDetails, signal, work));
|
|
84
|
+
outboxError ??= insertionError;
|
|
85
|
+
let wakeAccepted = taskMessageKeys(context.sessionManager.getEntries(), TASK_WAKE_CUSTOM_TYPE).has(eventKey);
|
|
86
|
+
const wakeSuppressed = !wakeAccepted && hasAcknowledgedLocalOriginTerminal(core, event);
|
|
87
|
+
if (!wakeAccepted && !wakeSuppressed) {
|
|
88
|
+
if (!context.isIdle()) break;
|
|
89
|
+
// This synchronous local write closes the ACK/wake race; do not await relay I/O before Pi emits the wake.
|
|
90
|
+
const wakeRequest = core.recordWakeRequest(eventDetails, work);
|
|
91
|
+
pi.sendMessage({
|
|
92
|
+
customType: TASK_WAKE_CUSTOM_TYPE,
|
|
93
|
+
content: "Process the pending Pi task event.",
|
|
94
|
+
display: false,
|
|
95
|
+
details: eventDetails,
|
|
96
|
+
}, { triggerTurn: true, deliverAs: "followUp" });
|
|
97
|
+
wakeAccepted = taskMessageKeys(context.sessionManager.getEntries(), TASK_WAKE_CUSTOM_TYPE).has(eventKey);
|
|
98
|
+
if (!wakeAccepted) break;
|
|
99
|
+
const wakeRequestError = await settleDeliveryEvidence(() => wakeRequest.flush(signal));
|
|
100
|
+
outboxError ??= wakeRequestError;
|
|
101
|
+
}
|
|
102
|
+
if (wakeAccepted) {
|
|
103
|
+
const wakeAcceptanceError = await recordDeliveryEvidence(core, eventDetails, { stage: TaskDeliveryStage.wakeAccepted, state: TaskDeliveryEvidenceState.confirmed }, signal, work);
|
|
104
|
+
outboxError ??= wakeAcceptanceError;
|
|
105
|
+
}
|
|
99
106
|
}
|
|
107
|
+
await core.acknowledgeRelayDelivery(delivery.cursor, signal, work);
|
|
108
|
+
pi.appendEntry(TASK_CURSOR_CUSTOM_TYPE, { cursor: delivery.cursor });
|
|
109
|
+
}
|
|
110
|
+
} finally {
|
|
111
|
+
// Completed, held or failed presentation yields the remaining bounded share to old work.
|
|
112
|
+
work.retryAllowed = true;
|
|
113
|
+
if (!work.signal.aborted) try { await core.flushOutbox(work.signal, work); } catch (error) {
|
|
114
|
+
if (!(error instanceof TaskOutboxDeliveryError)) throw error;
|
|
115
|
+
outboxError ??= error;
|
|
100
116
|
}
|
|
101
|
-
await core.acknowledgeRelayDelivery(delivery.cursor, signal);
|
|
102
|
-
pi.appendEntry(TASK_CURSOR_CUSTOM_TYPE, { cursor: delivery.cursor });
|
|
103
117
|
}
|
|
104
118
|
return outboxError;
|
|
105
119
|
}
|
|
106
120
|
|
|
107
|
-
async function recordDeliveryEvidence(core: TaskCore, event: TaskEventDetails, evidence: DeliveryEvidencePayload, signal
|
|
108
|
-
return settleDeliveryEvidence(() => core.submitIntent({ taskId: event.taskId, type: "task.delivery_receipt", payload: { eventId: event.eventId, ...evidence } }, signal));
|
|
121
|
+
async function recordDeliveryEvidence(core: TaskCore, event: TaskEventDetails, evidence: DeliveryEvidencePayload, signal: AbortSignal, work: TaskWorkScope): Promise<TaskOutboxDeliveryError | undefined> {
|
|
122
|
+
return settleDeliveryEvidence(() => core.submitIntent({ taskId: event.taskId, type: "task.delivery_receipt", payload: { eventId: event.eventId, ...evidence } }, signal, work));
|
|
109
123
|
}
|
|
110
124
|
|
|
111
125
|
async function settleDeliveryEvidence(operation: () => Promise<void>): Promise<TaskOutboxDeliveryError | undefined> {
|
package/src/task-store.ts
CHANGED
|
@@ -1,3 +1,4 @@
|
|
|
1
|
+
import { TASK_WORK_LIMITS } from "./task-work-scope";
|
|
1
2
|
import { TERMINAL_INTENT_OPERATION, TaskProtocolError } from "./task-protocol";
|
|
2
3
|
import type { RelayTransportBinding, RelayEnvelope, TaskEndpoint, TaskEvent, TaskRecord, TaskSnapshot, TerminalDeliveryState, TerminalTaskIntentType } from "./task-protocol";
|
|
3
4
|
|
|
@@ -25,6 +26,12 @@ export interface TaskStore {
|
|
|
25
26
|
appendEvent(event: TaskEvent): boolean;
|
|
26
27
|
putOutbox(envelope: RelayEnvelope): void;
|
|
27
28
|
outbox(state: OutboxRecord["state"]): readonly OutboxRecord[];
|
|
29
|
+
/** Indexed lookup, fair rotation and cooldown are required store capabilities. */
|
|
30
|
+
getOutboxRecord(envelopeId: string): OutboxRecord | undefined;
|
|
31
|
+
hasPendingOutbox(): boolean;
|
|
32
|
+
selectPendingOutbox(limit: number, now: number, maxInspections?: number): readonly OutboxRecord[];
|
|
33
|
+
outboxRetryAt(envelopeId: string): number;
|
|
34
|
+
postponeOutbox(envelopeId: string, until: number): void;
|
|
28
35
|
markOutboxAccepted(envelopeId: string): void;
|
|
29
36
|
quarantineOutbox(envelopeId: string, input: OutboxQuarantineInput): void;
|
|
30
37
|
quarantinedOutbox(): readonly OutboxQuarantineRecord[];
|
|
@@ -33,10 +40,13 @@ export interface TaskStore {
|
|
|
33
40
|
setReceiveCursor(cursor: string): void;
|
|
34
41
|
trackRelayDeliveries(endpoint: TaskEndpoint, deliveries: readonly { readonly cursor: string; readonly envelopeId: string }[]): void;
|
|
35
42
|
pendingRelayEnvelopeId(endpoint: TaskEndpoint, cursor: string): string | undefined;
|
|
36
|
-
requestRelayAcknowledgement(endpoint: TaskEndpoint, cursor: string): void;
|
|
37
|
-
requestedRelayAcknowledgements(endpoint: TaskEndpoint): readonly string[];
|
|
43
|
+
requestRelayAcknowledgement(endpoint: TaskEndpoint, cursor: string, retryAt?: number): void;
|
|
44
|
+
requestedRelayAcknowledgements(endpoint: TaskEndpoint, limit?: number, now?: number): readonly string[];
|
|
45
|
+
relayAcknowledgementRetryAt(endpoint: TaskEndpoint, cursor: string): number;
|
|
38
46
|
acknowledgeRelayCursor(endpoint: TaskEndpoint, cursor: string): void;
|
|
39
47
|
putIntent(intentId: string, taskId: string, envelopeId: string): void;
|
|
48
|
+
/** Locally submitted intent only; retained through acceptance or quarantine in this RAM lifetime. */
|
|
49
|
+
getIntentEnvelope(intentId: string): RelayEnvelope | undefined;
|
|
40
50
|
putInsertionReceipt(taskId: string, eventId: string): boolean;
|
|
41
51
|
reserveTaskOperation(input: TaskOperationRecord): { readonly created: boolean; readonly record: TaskOperationRecord };
|
|
42
52
|
getEndpointGeneration(): string | undefined;
|
|
@@ -52,7 +62,7 @@ export interface TaskStore {
|
|
|
52
62
|
const LIMITS = { entries: 16_384, bytes: 32 * 1024 * 1024 };
|
|
53
63
|
type Cell = { readonly text: string; readonly bytes: number; readonly order: number };
|
|
54
64
|
type Undo = { readonly table: Map<string, Cell>; readonly key: string; readonly prior: Cell | undefined };
|
|
55
|
-
interface Checkpoint { frontier: string; highWater: string; pending: Record<string, string>; requested: Record<string,
|
|
65
|
+
interface Checkpoint { frontier: string; highWater: string; pending: Record<string, string>; requested: Record<string, number>; }
|
|
56
66
|
|
|
57
67
|
/** One endpoint lifetime. No SQLite, files, migrations, startup reads or replay. */
|
|
58
68
|
export function createTaskStore(options: TaskStoreOptions = {}): TaskStore {
|
|
@@ -61,6 +71,14 @@ export function createTaskStore(options: TaskStoreOptions = {}): TaskStore {
|
|
|
61
71
|
if (!Number.isInteger(maxEntries) || maxEntries < 1 || maxEntries > LIMITS.entries || !Number.isInteger(maxBytes) || maxBytes < 1 || maxBytes > LIMITS.bytes) throw new TypeError("invalid task memory limit");
|
|
62
72
|
const tables = new Map<string, Map<string, Cell>>();
|
|
63
73
|
const frames: Map<string, Undo>[] = [];
|
|
74
|
+
// ID-only retry rotation, bounded by the existing pending outbox; no payload copy.
|
|
75
|
+
const pendingOutbox = new Map<string, number>();
|
|
76
|
+
const syncPendingOutbox = (id: string) => {
|
|
77
|
+
const row = tables.get("outbox")?.get(id);
|
|
78
|
+
if (row && (JSON.parse(row.text) as OutboxRecord).state === "pending") {
|
|
79
|
+
if (!pendingOutbox.has(id)) pendingOutbox.set(id, 0);
|
|
80
|
+
} else pendingOutbox.delete(id);
|
|
81
|
+
};
|
|
64
82
|
let bytes = 0, entries = 0, sequence = 0, closed = false;
|
|
65
83
|
const remember = (name: string, key: string, target: Map<string, Cell>, prior: Cell | undefined) => {
|
|
66
84
|
const frame = frames.at(-1), identity = JSON.stringify([name, key]);
|
|
@@ -116,7 +134,11 @@ export function createTaskStore(options: TaskStoreOptions = {}): TaskStore {
|
|
|
116
134
|
const result = operation();
|
|
117
135
|
if (result instanceof Promise) throw new TypeError("task transactions must be synchronous");
|
|
118
136
|
const parent = frames.at(-2);
|
|
119
|
-
if (parent)
|
|
137
|
+
if (parent) {
|
|
138
|
+
for (const [key, item] of frame) if (!parent.has(key)) parent.set(key, item);
|
|
139
|
+
} else {
|
|
140
|
+
for (const item of frame.values()) if (item.table === tables.get("outbox")) syncPendingOutbox(item.key);
|
|
141
|
+
}
|
|
120
142
|
return result;
|
|
121
143
|
} catch (error) {
|
|
122
144
|
for (const item of [...frame.values()].reverse()) { if (item.prior) item.table.set(item.key, item.prior); else item.table.delete(item.key); }
|
|
@@ -133,9 +155,28 @@ export function createTaskStore(options: TaskStoreOptions = {}): TaskStore {
|
|
|
133
155
|
},
|
|
134
156
|
setStatus(id, status) { const task = get<Omit<TaskRecord, "events">>("tasks", id); if (task) put("tasks", id, { ...task, status }); },
|
|
135
157
|
appendEvent(event) { requireTask(event.taskId); return put("events", event.eventId, event, true); },
|
|
136
|
-
putOutbox(envelope) { put("outbox", envelope.envelopeId, { envelope, state: "pending" }, true); },
|
|
158
|
+
putOutbox(envelope) { put("outbox", envelope.envelopeId, { envelope, state: "pending" }, true); if (!frames.length) syncPendingOutbox(envelope.envelopeId); },
|
|
137
159
|
outbox(status) { return rows<OutboxRecord>("outbox").filter(item => item.state === status); },
|
|
138
|
-
|
|
160
|
+
getOutboxRecord(id) { return get<OutboxRecord>("outbox", id); },
|
|
161
|
+
hasPendingOutbox() { check(); return pendingOutbox.size > 0; },
|
|
162
|
+
selectPendingOutbox(limit, now, maxInspections = TASK_WORK_LIMITS.retryInspections) {
|
|
163
|
+
check();
|
|
164
|
+
const selected: OutboxRecord[] = [];
|
|
165
|
+
// Also bound inspection when the queue is cooling down. Rotate even skipped IDs.
|
|
166
|
+
const inspections = Math.min(pendingOutbox.size, TASK_WORK_LIMITS.retryInspections, maxInspections);
|
|
167
|
+
for (let i = 0; i < inspections && selected.length < limit; i++) {
|
|
168
|
+
const [id, retryAt] = pendingOutbox.entries().next().value!;
|
|
169
|
+
pendingOutbox.delete(id); pendingOutbox.set(id, retryAt);
|
|
170
|
+
if (retryAt <= now) {
|
|
171
|
+
const record = get<OutboxRecord>("outbox", id);
|
|
172
|
+
if (record?.state === "pending") selected.push(record);
|
|
173
|
+
}
|
|
174
|
+
}
|
|
175
|
+
return selected;
|
|
176
|
+
},
|
|
177
|
+
outboxRetryAt(id) { check(); return pendingOutbox.get(id) ?? 0; },
|
|
178
|
+
postponeOutbox(id, until) { check(); if (pendingOutbox.has(id)) pendingOutbox.set(id, until); },
|
|
179
|
+
markOutboxAccepted(id) { const prior = get<OutboxRecord>("outbox", id); if (prior) put("outbox", id, { ...prior, state: "accepted" }); if (!frames.length) syncPendingOutbox(id); },
|
|
139
180
|
quarantineOutbox(id, input) {
|
|
140
181
|
store.transaction(() => { const prior = get<OutboxRecord>("outbox", id); if (!prior) return;
|
|
141
182
|
put("quarantine", id, { ...input, envelope: prior.envelope, priorState: prior.state }, true); remove("outbox", id);
|
|
@@ -156,12 +197,16 @@ export function createTaskStore(options: TaskStoreOptions = {}): TaskStore {
|
|
|
156
197
|
writeCheckpoint(endpoint, value);
|
|
157
198
|
},
|
|
158
199
|
pendingRelayEnvelopeId(endpoint, cursor) { return checkpoint(endpoint).pending[cursor]; },
|
|
159
|
-
requestRelayAcknowledgement(endpoint, cursor) {
|
|
200
|
+
requestRelayAcknowledgement(endpoint, cursor, retryAt = 0) {
|
|
160
201
|
const value = checkpoint(endpoint);
|
|
161
202
|
if (value.pending[cursor] === undefined) throw new TaskProtocolError("INVALID_CURSOR", "relay delivery is not pending in this endpoint scope");
|
|
162
|
-
value.requested[cursor] =
|
|
203
|
+
value.requested[cursor] = retryAt; writeCheckpoint(endpoint, value);
|
|
163
204
|
},
|
|
164
|
-
requestedRelayAcknowledgements(endpoint
|
|
205
|
+
requestedRelayAcknowledgements(endpoint, limit = Number.MAX_SAFE_INTEGER, now = Date.now()) {
|
|
206
|
+
const value = checkpoint(endpoint);
|
|
207
|
+
return Object.keys(value.requested).filter(cursor => value.requested[cursor]! <= now).slice(0, limit);
|
|
208
|
+
},
|
|
209
|
+
relayAcknowledgementRetryAt(endpoint, cursor) { return checkpoint(endpoint).requested[cursor] ?? 0; },
|
|
165
210
|
acknowledgeRelayCursor(endpoint, cursor) {
|
|
166
211
|
store.transaction(() => {
|
|
167
212
|
const value = checkpoint(endpoint); if (value.pending[cursor] === undefined) return;
|
|
@@ -173,6 +218,12 @@ export function createTaskStore(options: TaskStoreOptions = {}): TaskStore {
|
|
|
173
218
|
});
|
|
174
219
|
},
|
|
175
220
|
putIntent(intentId, taskId, envelopeId) { put("intents", intentId, { taskId, envelopeId }, true); },
|
|
221
|
+
getIntentEnvelope(intentId) {
|
|
222
|
+
const intent = get<{ taskId: string; envelopeId: string }>("intents", intentId);
|
|
223
|
+
if (!intent) return undefined;
|
|
224
|
+
const envelope = (get<OutboxRecord>("outbox", intent.envelopeId) ?? get<OutboxQuarantineRecord>("quarantine", intent.envelopeId))?.envelope;
|
|
225
|
+
return envelope?.taskId === intent.taskId ? envelope : undefined;
|
|
226
|
+
},
|
|
176
227
|
putInsertionReceipt(taskId, eventId) { requireTask(taskId); return put("insertions", pair(taskId, eventId), true, true); },
|
|
177
228
|
reserveTaskOperation(input) { requireTask(input.taskId); const key = pair(input.taskId, input.operation); const created = put("operations", key, input, true); return { created, record: get<TaskOperationRecord>("operations", key)! }; },
|
|
178
229
|
getEndpointGeneration() { return state<string>("generation"); },
|
|
@@ -181,7 +232,7 @@ export function createTaskStore(options: TaskStoreOptions = {}): TaskStore {
|
|
|
181
232
|
setEndpointBinding(value) { put("state", "endpoint", value); },
|
|
182
233
|
getRelayTransportBinding() { return state<RelayTransportBinding>("transport"); },
|
|
183
234
|
setRelayTransportBinding(value) { put("state", "transport", value); },
|
|
184
|
-
clear() { check(); if (frames.length) throw new Error("cannot clear task state during a transaction"); tables.clear(); bytes = entries = sequence = 0; },
|
|
235
|
+
clear() { check(); if (frames.length) throw new Error("cannot clear task state during a transaction"); tables.clear(); pendingOutbox.clear(); bytes = entries = sequence = 0; },
|
|
185
236
|
close() { if (closed) return; store.clear(); closed = true; },
|
|
186
237
|
};
|
|
187
238
|
return store;
|
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
import { TaskProtocolError } from "./task-protocol";
|
|
2
|
+
|
|
3
|
+
export const TASK_WORK_LIMITS = { scopedSends: 56, retries: 8, acknowledgements: 48, pageItems: 50,
|
|
4
|
+
deadlineMs: 2_000, retrySliceMs: 250, ackRepairSliceMs: 250, capacityCooldownMs: 5_000, retryInspections: 64 } as const;
|
|
5
|
+
|
|
6
|
+
/** Shared counts for nested receipts/fanout; background I/O also has one finite window. */
|
|
7
|
+
export interface TaskWorkScope {
|
|
8
|
+
readonly signal: AbortSignal;
|
|
9
|
+
readonly attemptedSends: Set<string>;
|
|
10
|
+
readonly attemptedAcknowledgements: Set<string>;
|
|
11
|
+
readonly blockedRetryTargets: Set<string>;
|
|
12
|
+
retryStarted: boolean;
|
|
13
|
+
retryAllowed: boolean;
|
|
14
|
+
claimSend(envelopeId: string, retry: boolean): boolean;
|
|
15
|
+
claimAcknowledgement(cursor: string): boolean;
|
|
16
|
+
canAcknowledge(): boolean;
|
|
17
|
+
canRetry(): boolean;
|
|
18
|
+
canRepairAcknowledgement(): boolean;
|
|
19
|
+
backgroundSignal(): AbortSignal;
|
|
20
|
+
}
|
|
21
|
+
|
|
22
|
+
export function createTaskWorkScope(signal?: AbortSignal, mode: "background" | "foreground" = "background"): TaskWorkScope {
|
|
23
|
+
const caller = signal ?? new AbortController().signal;
|
|
24
|
+
let background: AbortSignal | undefined;
|
|
25
|
+
const backgroundSignal = () => background ??= AbortSignal.any([caller, AbortSignal.timeout(TASK_WORK_LIMITS.deadlineMs)]);
|
|
26
|
+
// Foreground requests retain the transport/caller timeout. Their speculative
|
|
27
|
+
// retries start a bounded background window only after required sends finish.
|
|
28
|
+
const active = mode === "background" ? backgroundSignal() : caller;
|
|
29
|
+
const attemptedSends = new Set<string>(), attemptedAcknowledgements = new Set<string>(), blockedRetryTargets = new Set<string>();
|
|
30
|
+
let scopedSends = 0, retrySends = 0, acknowledgements = 0;
|
|
31
|
+
let retryUntil: number | undefined, repairUntil: number | undefined;
|
|
32
|
+
return {
|
|
33
|
+
signal: active, attemptedSends, attemptedAcknowledgements, blockedRetryTargets, retryStarted: false, retryAllowed: true,
|
|
34
|
+
backgroundSignal,
|
|
35
|
+
claimSend(id, retry) {
|
|
36
|
+
if ((retry ? backgroundSignal() : active).aborted || attemptedSends.has(id) || (retry ? retrySends >= TASK_WORK_LIMITS.retries : scopedSends >= TASK_WORK_LIMITS.scopedSends)) return false;
|
|
37
|
+
attemptedSends.add(id);
|
|
38
|
+
if (retry) retrySends++; else scopedSends++;
|
|
39
|
+
return true;
|
|
40
|
+
},
|
|
41
|
+
claimAcknowledgement(cursor) {
|
|
42
|
+
if (active.aborted || attemptedAcknowledgements.has(cursor) || acknowledgements >= TASK_WORK_LIMITS.acknowledgements) return false;
|
|
43
|
+
attemptedAcknowledgements.add(cursor); acknowledgements++;
|
|
44
|
+
return true;
|
|
45
|
+
},
|
|
46
|
+
canAcknowledge() { return !active.aborted && acknowledgements < TASK_WORK_LIMITS.acknowledgements; },
|
|
47
|
+
// Slices stop admission, not completion: an admitted request gets the
|
|
48
|
+
// remaining background window, including cancellation and late fences.
|
|
49
|
+
canRetry() { return !backgroundSignal().aborted && performance.now() < (retryUntil ??= performance.now() + TASK_WORK_LIMITS.retrySliceMs); },
|
|
50
|
+
canRepairAcknowledgement() { return !backgroundSignal().aborted && performance.now() < (repairUntil ??= performance.now() + TASK_WORK_LIMITS.ackRepairSliceMs); },
|
|
51
|
+
};
|
|
52
|
+
}
|
|
53
|
+
|
|
54
|
+
export function deferredTaskDelivery(details?: Record<string, unknown>): TaskProtocolError {
|
|
55
|
+
return new TaskProtocolError("TASK_DELIVERY_DEFERRED", "task delivery deferred by the local work budget or retry cooldown", { retryable: true, details });
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
/** Known admission backpressure gets at least one normal polling interval, without another timer. */
|
|
59
|
+
export function capacityRetryAt(error: unknown): number | undefined {
|
|
60
|
+
if (!(error instanceof TaskProtocolError) || error.code !== "RELAY_CAPACITY" || !error.retryable) return undefined;
|
|
61
|
+
const hint = error.details?.retryAfterMs;
|
|
62
|
+
return Date.now() + Math.max(TASK_WORK_LIMITS.capacityCooldownMs, typeof hint === "number" && Number.isFinite(hint) && hint >= 0 ? hint : 0);
|
|
63
|
+
}
|