@friggframework/core 2.0.0--canary.657.b14221f.0 → 2.0.0--canary.658.a5177aa.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +47 -55
- package/core/CLAUDE.md +0 -19
- package/core/Worker.js +5 -336
- package/handlers/backend-utils.js +0 -45
- package/integrations/integration-base.js +37 -1
- package/integrations/repositories/integration-repository-documentdb.js +25 -6
- package/integrations/repositories/integration-repository-interface.js +22 -5
- package/integrations/repositories/integration-repository-mongo.js +32 -12
- package/integrations/repositories/integration-repository-postgres.js +33 -12
- package/integrations/repositories/message-item-shared.js +24 -0
- package/integrations/tests/doubles/test-integration-repository.js +15 -2
- package/integrations/use-cases/record-rate-limit-message.js +85 -0
- package/integrations/use-cases/update-integration-messages.js +8 -6
- package/modules/module.js +30 -0
- package/modules/requester/oauth-2.js +0 -6
- package/modules/requester/requester.js +21 -0
- package/package.json +5 -6
- package/queues/queuer-util.js +1 -1
- package/types/core/index.d.ts +0 -20
- package/types/integrations/index.d.ts +25 -4
- package/types/module-plugin/index.d.ts +3 -0
- package/queues/queue-deferral.js +0 -79
package/CLAUDE.md
CHANGED
|
@@ -546,61 +546,6 @@ The single source of the max receive count is
|
|
|
546
546
|
devtools integration builder uses it for the queue's `RedrivePolicy` and sets
|
|
547
547
|
it as `FRIGG_QUEUE_MAX_RECEIVE_COUNT` on the queue worker function.
|
|
548
548
|
|
|
549
|
-
**Rate-limit deferral** (ADR-049): when a queue handler throws an error with
|
|
550
|
-
`isRateLimited` and a `retryAt` (a `RateLimitError` from the Requester, or one a
|
|
551
|
-
module throws), `Worker.run` does not give the message back to SQS for a 30
|
|
552
|
-
minute redelivery that spends one of its three receives. It puts the message
|
|
553
|
-
back so it runs at `retryAt`:
|
|
554
|
-
|
|
555
|
-
| Wait | What the worker does | Record |
|
|
556
|
-
|---|---|---|
|
|
557
|
-
| Up to 900 s | Sends the same body again with `DelaySeconds`, then acknowledges the old message | handled |
|
|
558
|
-
| Over 900 s, and `SCHEDULER_ROLE_ARN` is set | Creates a one-time EventBridge schedule at `retryAt` that sends the body to the queue, then acknowledges | handled |
|
|
559
|
-
| Over 900 s, no scheduler | Extends the message's visibility timeout to `retryAt` (12 h at most). This uses one receive | reported failed |
|
|
560
|
-
|
|
561
|
-
- The new message starts at receive count 1. A deferred body carries
|
|
562
|
-
`_frigg: { deferrals, firstDeferredAt }`; `_frigg` is reserved and the handler
|
|
563
|
-
never sees it.
|
|
564
|
-
- Two caps bound it: `FRIGG_QUEUE_MAX_DEFERRALS` (default 10) and
|
|
565
|
-
`FRIGG_QUEUE_MAX_DEFERRED_MS` (default 24 h, from the first deferral to the
|
|
566
|
-
retry time). Past a cap, with no `retryAt`, with no event source ARN or on a
|
|
567
|
-
FIFO queue, the record fails like any retryable error and `delivery`
|
|
568
|
-
applies.
|
|
569
|
-
- The mock scheduler is never used for a deferral: a schedule kept in memory
|
|
570
|
-
would be lost with the acknowledged message.
|
|
571
|
-
- With a `processId` in the message, the worker writes
|
|
572
|
-
`Process.context.rateLimit = { status: 'WAITING' | 'EXHAUSTED', mechanism,
|
|
573
|
-
retryAt, reason, module, deferrals, updatedAt }`. `EXHAUSTED` means a cap ended
|
|
574
|
-
the deferrals, or a visibility change came on the last delivery. The value is
|
|
575
|
-
one per process, so it holds the latest known wait: a new wait replaces it only
|
|
576
|
-
when its `retryAt` is later, a `WAITING` value never hides an `EXHAUSTED` one,
|
|
577
|
-
and after a deferred or redelivered message ran the worker sets it to `null`
|
|
578
|
-
only when it is `WAITING` and its `retryAt` is past or within 5 s.
|
|
579
|
-
- Delivery stays at least once. The new message is sent before the old one is
|
|
580
|
-
acknowledged, so a crash between the two can run the work twice.
|
|
581
|
-
- Handlers check `delivery.isLastAttempt` first, then rethrow. The delay and
|
|
582
|
-
schedule tiers send a new message, so they never reach the last attempt. The
|
|
583
|
-
visibility tier and a cap use a receive: on the last one the worker writes an
|
|
584
|
-
ERROR (`frigg.worker.record_lost_rate_limited`), and the handler has this one
|
|
585
|
-
chance to end the run.
|
|
586
|
-
|
|
587
|
-
```javascript
|
|
588
|
-
async processBatch({ data, delivery }) {
|
|
589
|
-
try {
|
|
590
|
-
await this.syncPage(data);
|
|
591
|
-
} catch (error) {
|
|
592
|
-
if (delivery?.isLastAttempt) return this.failRun(data.processId, error);
|
|
593
|
-
throw error; // a RateLimitError too: core puts the message back at retryAt
|
|
594
|
-
}
|
|
595
|
-
}
|
|
596
|
-
```
|
|
597
|
-
|
|
598
|
-
Records: `frigg.worker.record_deferred` (`mechanism` `delay` or `schedule`),
|
|
599
|
-
`record_visibility_extended`, `record_deferral_capped`,
|
|
600
|
-
`record_lost_rate_limited` and `rate_limit_state_failed`. A queue in a stack
|
|
601
|
-
that does not own it (`ownership.queue: 'external'`) needs its own
|
|
602
|
-
`sqs:ChangeMessageVisibility` and scheduler access.
|
|
603
|
-
|
|
604
549
|
### 8. Error Handling (`/errors`)
|
|
605
550
|
|
|
606
551
|
**Purpose**: Standardized error types with proper HTTP semantics.
|
|
@@ -623,6 +568,53 @@ if (!userId) {
|
|
|
623
568
|
}
|
|
624
569
|
```
|
|
625
570
|
|
|
571
|
+
**Rate-limit message** (ADR-049): when a wait is too long to sleep, the
|
|
572
|
+
Requester tells its delegate (`RATE_LIMITED`, with the `RateLimitError`) and
|
|
573
|
+
then throws. The notification takes the path of `CREDENTIAL_INVALIDATED`:
|
|
574
|
+
Requester, `Module`, `IntegrationBase`.
|
|
575
|
+
|
|
576
|
+
- The `Module` sends `{ moduleName, reason, retryAt, policy, statusCode, links }`.
|
|
577
|
+
`links` is `static rateLimit.userHints[reason].links`, or `[]`. The message,
|
|
578
|
+
url, body and headers of the error are never sent: the message reaches end
|
|
579
|
+
users.
|
|
580
|
+
- `IntegrationBase` writes one `WARN` (`integration.<name>.rate_limited`), and
|
|
581
|
+
`RecordRateLimitMessage` appends one item to the stored `warnings`. It
|
|
582
|
+
changes no status. A rate limit is not an `ERROR`, and an `ERROR` status
|
|
583
|
+
makes the queue worker drop every webhook.
|
|
584
|
+
- A report for the same module whose reset time is within 60 s of a stored one
|
|
585
|
+
is skipped. The check reads the stored `warnings` with
|
|
586
|
+
`findIntegrationMessages`, not `this.messages`, which is per instance and
|
|
587
|
+
empty on Postgres. It is a read then a write, so workers that report at the
|
|
588
|
+
same instant can still store a few copies.
|
|
589
|
+
- Every hop is best effort. A failure is logged, and the caller still gets the
|
|
590
|
+
`RateLimitError`.
|
|
591
|
+
|
|
592
|
+
```javascript
|
|
593
|
+
{
|
|
594
|
+
title: 'Rate limit reached',
|
|
595
|
+
message: 'The hubspot API rate limit was reached and resets at 2026-09-28 14:31 UTC.',
|
|
596
|
+
timestamp: 1790000000000,
|
|
597
|
+
code: 'RATE_LIMITED',
|
|
598
|
+
module: 'hubspot',
|
|
599
|
+
reason: 'daily',
|
|
600
|
+
retryAt: '2026-09-28T14:30:15.000Z',
|
|
601
|
+
actions: [
|
|
602
|
+
{ type: 'RETRY_WHEN_READY' },
|
|
603
|
+
{ type: 'LINK', label: 'API usage limits', url: 'https://example.com/limits' },
|
|
604
|
+
],
|
|
605
|
+
}
|
|
606
|
+
```
|
|
607
|
+
|
|
608
|
+
The text is fixed. It holds the module name and the reset time in UTC, rounded
|
|
609
|
+
up to the minute, and nothing from the request.
|
|
610
|
+
`updateIntegrationMessages(id, type, title, body, timestamp)` still works, and
|
|
611
|
+
it also takes one item object, `updateIntegrationMessages(id, type, item)`,
|
|
612
|
+
whose keys are stored as they are. Every call appends.
|
|
613
|
+
`findIntegrationMessages(id, type)` reads the stored items of one type. The
|
|
614
|
+
`messages` of the record from `findIntegrationById` is not filled on Postgres
|
|
615
|
+
and Mongo, because the schema keeps the messages in the `errors`, `warnings`,
|
|
616
|
+
`info` and `logs` columns.
|
|
617
|
+
|
|
626
618
|
### 9. Logging System (`/logs`)
|
|
627
619
|
|
|
628
620
|
**Purpose**: One redacted JSON record per line to stdout (ADR-048). See `docs/guides/LOGGING.md`.
|
package/core/CLAUDE.md
CHANGED
|
@@ -79,25 +79,6 @@ await worker.send({
|
|
|
79
79
|
}, delaySeconds);
|
|
80
80
|
```
|
|
81
81
|
|
|
82
|
-
**Rate-limit deferral** (ADR-049): `run()` handles an error with `isRateLimited`
|
|
83
|
-
and a `retryAt` through `defer(record, body, error, delivery)`, which returns
|
|
84
|
-
`{ outcome }`:
|
|
85
|
-
- `acked`: the message was sent again with `DelaySeconds` (wait up to 900 s) or
|
|
86
|
-
scheduled with the one-time scheduler (longer wait). The record is handled.
|
|
87
|
-
- `failed`: the visibility timeout was extended to `retryAt`, or a cap
|
|
88
|
-
(`FRIGG_QUEUE_MAX_DEFERRALS`, `FRIGG_QUEUE_MAX_DEFERRED_MS`) ended the
|
|
89
|
-
deferrals. The record is reported failed.
|
|
90
|
-
- `skipped`: no `retryAt`, no `eventSourceARN`, a FIFO queue, or a failed send.
|
|
91
|
-
The record fails as any error does, and `record_failed` carries
|
|
92
|
-
`deferral: { skipped }`.
|
|
93
|
-
|
|
94
|
-
Subclasses override two no-op hooks to keep run state: `recordRateLimitWait(body,
|
|
95
|
-
error, state)` runs after the message was put back, and `clearRateLimitWait(body)`
|
|
96
|
-
runs after a deferred or redelivered message succeeded. A failing hook logs
|
|
97
|
-
`frigg.worker.rate_limit_state_failed` and never changes the outcome. The queue
|
|
98
|
-
worker of an integration (`createQueueWorker`) writes `Process.context.rateLimit`
|
|
99
|
-
in them. See `packages/core/CLAUDE.md` section 7.
|
|
100
|
-
|
|
101
82
|
### Delegate Pattern System (`Delegate.js:3-27`)
|
|
102
83
|
|
|
103
84
|
**Purpose**: Observer/delegation pattern for decoupled component communication
|
package/core/Worker.js
CHANGED
|
@@ -1,39 +1,14 @@
|
|
|
1
|
-
const {
|
|
2
|
-
const {
|
|
3
|
-
SQSClient,
|
|
4
|
-
GetQueueUrlCommand,
|
|
5
|
-
SendMessageCommand,
|
|
6
|
-
ChangeMessageVisibilityCommand,
|
|
7
|
-
} = require('@aws-sdk/client-sqs');
|
|
1
|
+
const { SQSClient, GetQueueUrlCommand, SendMessageCommand } = require('@aws-sdk/client-sqs');
|
|
8
2
|
const _ = require('lodash');
|
|
9
3
|
const { RequiredPropertyError } = require('../errors');
|
|
10
4
|
const { get } = require('../assertions');
|
|
11
5
|
const { readQueueDelivery } = require('../queues/queue-delivery');
|
|
12
|
-
const { awsConfigOptions } = require('../queues/queuer-util');
|
|
13
|
-
const {
|
|
14
|
-
MAX_DELAY_SECONDS,
|
|
15
|
-
MAX_VISIBILITY_TIMEOUT_SECONDS,
|
|
16
|
-
isDeferralCapped,
|
|
17
|
-
nextDeferral,
|
|
18
|
-
readDeferral,
|
|
19
|
-
readDeferralLimits,
|
|
20
|
-
withDeferral,
|
|
21
|
-
} = require('../queues/queue-deferral');
|
|
22
6
|
const { runMessageScope } = require('./invocation-scope');
|
|
23
|
-
const { getLogger
|
|
7
|
+
const { getLogger } = require('../logs');
|
|
24
8
|
|
|
25
|
-
const sqs = new SQSClient({
|
|
26
|
-
region: process.env.AWS_REGION,
|
|
27
|
-
...awsConfigOptions(),
|
|
28
|
-
});
|
|
9
|
+
const sqs = new SQSClient({ region: process.env.AWS_REGION });
|
|
29
10
|
const log = getLogger('frigg.worker');
|
|
30
11
|
|
|
31
|
-
const validDate = (value) =>
|
|
32
|
-
value instanceof Date && !Number.isNaN(value.getTime()) ? value : null;
|
|
33
|
-
const queueNameOf = (arn) =>
|
|
34
|
-
typeof arn === 'string' && arn ? arn.split(':').pop() : undefined;
|
|
35
|
-
const causeOf = (error) => serializeError(error).message;
|
|
36
|
-
|
|
37
12
|
class Worker {
|
|
38
13
|
async getQueueURL(params) {
|
|
39
14
|
// Passing params in because there will be multiple QueueNames
|
|
@@ -58,31 +33,12 @@ class Worker {
|
|
|
58
33
|
// messageId, receiveCount and the event come from the scope.
|
|
59
34
|
log.debug('Record started', { eventName: 'frigg.worker.record_started' });
|
|
60
35
|
|
|
61
|
-
const delivery = readQueueDelivery(record);
|
|
62
|
-
let runParams;
|
|
63
36
|
try {
|
|
64
|
-
runParams = JSON.parse(record.body);
|
|
37
|
+
const runParams = JSON.parse(record.body);
|
|
65
38
|
this._validateParams(runParams);
|
|
66
|
-
await this._run(runParams, context,
|
|
67
|
-
await this._clearDeferredState(runParams, delivery);
|
|
39
|
+
await this._run(runParams, context, readQueueDelivery(record));
|
|
68
40
|
log.debug('Record succeeded', { eventName: 'frigg.worker.record_succeeded' });
|
|
69
41
|
} catch (error) {
|
|
70
|
-
let deferral;
|
|
71
|
-
if (error.isRateLimited && runParams) {
|
|
72
|
-
deferral = await this.defer(
|
|
73
|
-
record,
|
|
74
|
-
runParams,
|
|
75
|
-
error,
|
|
76
|
-
delivery
|
|
77
|
-
);
|
|
78
|
-
if (deferral.outcome === 'acked') return;
|
|
79
|
-
if (deferral.outcome === 'failed') {
|
|
80
|
-
batchItemFailures.push({
|
|
81
|
-
itemIdentifier: record.messageId,
|
|
82
|
-
});
|
|
83
|
-
return;
|
|
84
|
-
}
|
|
85
|
-
}
|
|
86
42
|
if (error.isHaltError) {
|
|
87
43
|
// HaltError means "discard this message, don't retry".
|
|
88
44
|
// Treat as success so SQS deletes it from the queue.
|
|
@@ -99,14 +55,6 @@ class Worker {
|
|
|
99
55
|
log.warn('Record failed, returned for retry', {
|
|
100
56
|
eventName: 'frigg.worker.record_failed',
|
|
101
57
|
error,
|
|
102
|
-
...(deferral && {
|
|
103
|
-
deferral: {
|
|
104
|
-
skipped: deferral.skipped,
|
|
105
|
-
...(deferral.cause && {
|
|
106
|
-
cause: causeOf(deferral.cause),
|
|
107
|
-
}),
|
|
108
|
-
},
|
|
109
|
-
}),
|
|
110
58
|
});
|
|
111
59
|
batchItemFailures.push({ itemIdentifier: record.messageId });
|
|
112
60
|
}
|
|
@@ -127,285 +75,6 @@ class Worker {
|
|
|
127
75
|
// parameters
|
|
128
76
|
}
|
|
129
77
|
|
|
130
|
-
/**
|
|
131
|
-
* Puts a rate-limited message back so it runs at `error.retryAt`, without
|
|
132
|
-
* spending a receive. The message is sent again with a delay of up to 900 s,
|
|
133
|
-
* or scheduled with the one-time scheduler, or has its visibility timeout
|
|
134
|
-
* extended. The new message goes out before the old one is acknowledged, so
|
|
135
|
-
* delivery stays at least once.
|
|
136
|
-
*
|
|
137
|
-
* @returns {Promise<{outcome: 'acked'|'failed'|'skipped', skipped?: string, cause?: Error}>}
|
|
138
|
-
* `acked`: the message is handled. `failed`: report the record failed and
|
|
139
|
-
* let SQS redeliver it. `skipped`: nothing was done; fail the record as
|
|
140
|
-
* any other error.
|
|
141
|
-
*/
|
|
142
|
-
async defer(record, body, error, delivery) {
|
|
143
|
-
const retryAt = validDate(error.retryAt);
|
|
144
|
-
if (!retryAt) return { outcome: 'skipped', skipped: 'no_retry_at' };
|
|
145
|
-
const queueName = queueNameOf(record.eventSourceARN);
|
|
146
|
-
if (!queueName || queueName.endsWith('.fifo')) {
|
|
147
|
-
return { outcome: 'skipped', skipped: 'no_event_source' };
|
|
148
|
-
}
|
|
149
|
-
|
|
150
|
-
const now = Date.now();
|
|
151
|
-
const waitMs = Math.max(0, retryAt.getTime() - now);
|
|
152
|
-
const deferral = nextDeferral(body, now);
|
|
153
|
-
const details = {
|
|
154
|
-
waitMs,
|
|
155
|
-
retryAt: retryAt.toISOString(),
|
|
156
|
-
deferrals: deferral.deferrals,
|
|
157
|
-
firstDeferredAt: deferral.firstDeferredAt,
|
|
158
|
-
reason: error.reason,
|
|
159
|
-
module: error.module,
|
|
160
|
-
statusCode: error.statusCode,
|
|
161
|
-
};
|
|
162
|
-
const state = (status, mechanism) => ({
|
|
163
|
-
status,
|
|
164
|
-
mechanism,
|
|
165
|
-
deferrals: deferral.deferrals,
|
|
166
|
-
retryAt,
|
|
167
|
-
});
|
|
168
|
-
|
|
169
|
-
const limits = readDeferralLimits();
|
|
170
|
-
if (isDeferralCapped({ ...deferral, retryAt }, limits)) {
|
|
171
|
-
log.warn('Rate-limit deferral capped', {
|
|
172
|
-
eventName: 'frigg.worker.record_deferral_capped',
|
|
173
|
-
...details,
|
|
174
|
-
maxDeferrals: limits.maxDeferrals,
|
|
175
|
-
maxDeferredMs: limits.maxDeferredMs,
|
|
176
|
-
deferredMs:
|
|
177
|
-
retryAt.getTime() - Date.parse(deferral.firstDeferredAt),
|
|
178
|
-
error,
|
|
179
|
-
});
|
|
180
|
-
await this._recordWait(body, error, state('EXHAUSTED', 'none'));
|
|
181
|
-
this._logIfLost(delivery, error, details);
|
|
182
|
-
return { outcome: 'failed' };
|
|
183
|
-
}
|
|
184
|
-
|
|
185
|
-
const tier = {
|
|
186
|
-
record,
|
|
187
|
-
queueName,
|
|
188
|
-
body,
|
|
189
|
-
next: withDeferral(body, deferral),
|
|
190
|
-
error,
|
|
191
|
-
delivery,
|
|
192
|
-
retryAt,
|
|
193
|
-
waitMs,
|
|
194
|
-
details,
|
|
195
|
-
state,
|
|
196
|
-
};
|
|
197
|
-
if (waitMs <= MAX_DELAY_SECONDS * 1000)
|
|
198
|
-
return this._deferWithDelay(tier);
|
|
199
|
-
const scheduled = await this._deferWithSchedule(tier);
|
|
200
|
-
if (scheduled?.outcome) return scheduled;
|
|
201
|
-
return this._deferWithVisibility({
|
|
202
|
-
...tier,
|
|
203
|
-
scheduleError: scheduled?.scheduleError,
|
|
204
|
-
});
|
|
205
|
-
}
|
|
206
|
-
|
|
207
|
-
async _deferWithDelay({
|
|
208
|
-
queueName,
|
|
209
|
-
body,
|
|
210
|
-
next,
|
|
211
|
-
error,
|
|
212
|
-
waitMs,
|
|
213
|
-
details,
|
|
214
|
-
state,
|
|
215
|
-
}) {
|
|
216
|
-
const delaySeconds = Math.ceil(waitMs / 1000);
|
|
217
|
-
try {
|
|
218
|
-
const queueUrl = await this._queueUrl(queueName);
|
|
219
|
-
await this.send({ ...next, QueueUrl: queueUrl }, delaySeconds);
|
|
220
|
-
} catch (cause) {
|
|
221
|
-
return { outcome: 'skipped', skipped: 'send_failed', cause };
|
|
222
|
-
}
|
|
223
|
-
await this._recordWait(body, error, state('WAITING', 'delay'));
|
|
224
|
-
log.warn('Record deferred', {
|
|
225
|
-
eventName: 'frigg.worker.record_deferred',
|
|
226
|
-
mechanism: 'delay',
|
|
227
|
-
delaySeconds,
|
|
228
|
-
...details,
|
|
229
|
-
error,
|
|
230
|
-
});
|
|
231
|
-
return { outcome: 'acked' };
|
|
232
|
-
}
|
|
233
|
-
|
|
234
|
-
/** `acked`, `{ scheduleError }` when the schedule failed, null with no scheduler. */
|
|
235
|
-
async _deferWithSchedule({
|
|
236
|
-
record,
|
|
237
|
-
body,
|
|
238
|
-
next,
|
|
239
|
-
error,
|
|
240
|
-
retryAt,
|
|
241
|
-
details,
|
|
242
|
-
state,
|
|
243
|
-
}) {
|
|
244
|
-
const scheduler = this.getSchedulerService();
|
|
245
|
-
if (!scheduler) return null;
|
|
246
|
-
const scheduleName = `frigg-defer-${record.messageId || randomUUID()}`;
|
|
247
|
-
try {
|
|
248
|
-
await scheduler.scheduleOneTime({
|
|
249
|
-
scheduleName,
|
|
250
|
-
scheduleAt: retryAt,
|
|
251
|
-
queueResourceId: record.eventSourceARN,
|
|
252
|
-
payload: next,
|
|
253
|
-
});
|
|
254
|
-
} catch (cause) {
|
|
255
|
-
if (cause?.name !== 'ConflictException')
|
|
256
|
-
return { scheduleError: cause };
|
|
257
|
-
}
|
|
258
|
-
await this._recordWait(body, error, state('WAITING', 'schedule'));
|
|
259
|
-
log.warn('Record deferred', {
|
|
260
|
-
eventName: 'frigg.worker.record_deferred',
|
|
261
|
-
mechanism: 'schedule',
|
|
262
|
-
scheduleName,
|
|
263
|
-
...details,
|
|
264
|
-
error,
|
|
265
|
-
});
|
|
266
|
-
return { outcome: 'acked' };
|
|
267
|
-
}
|
|
268
|
-
|
|
269
|
-
async _deferWithVisibility({
|
|
270
|
-
record,
|
|
271
|
-
queueName,
|
|
272
|
-
body,
|
|
273
|
-
error,
|
|
274
|
-
delivery,
|
|
275
|
-
waitMs,
|
|
276
|
-
details,
|
|
277
|
-
state,
|
|
278
|
-
scheduleError,
|
|
279
|
-
}) {
|
|
280
|
-
const visibilityTimeout = Math.min(
|
|
281
|
-
Math.ceil(waitMs / 1000),
|
|
282
|
-
MAX_VISIBILITY_TIMEOUT_SECONDS
|
|
283
|
-
);
|
|
284
|
-
try {
|
|
285
|
-
const queueUrl = await this._queueUrl(queueName);
|
|
286
|
-
await sqs.send(
|
|
287
|
-
new ChangeMessageVisibilityCommand({
|
|
288
|
-
QueueUrl: queueUrl,
|
|
289
|
-
ReceiptHandle: record.receiptHandle,
|
|
290
|
-
VisibilityTimeout: visibilityTimeout,
|
|
291
|
-
})
|
|
292
|
-
);
|
|
293
|
-
} catch (cause) {
|
|
294
|
-
return { outcome: 'skipped', skipped: 'visibility_failed', cause };
|
|
295
|
-
}
|
|
296
|
-
await this._recordWait(
|
|
297
|
-
body,
|
|
298
|
-
error,
|
|
299
|
-
state(
|
|
300
|
-
delivery.isLastAttempt ? 'EXHAUSTED' : 'WAITING',
|
|
301
|
-
'visibility'
|
|
302
|
-
)
|
|
303
|
-
);
|
|
304
|
-
log.warn('Record visibility extended', {
|
|
305
|
-
eventName: 'frigg.worker.record_visibility_extended',
|
|
306
|
-
mechanism: 'visibility',
|
|
307
|
-
visibilityTimeout,
|
|
308
|
-
...details,
|
|
309
|
-
...(scheduleError && { scheduleError: causeOf(scheduleError) }),
|
|
310
|
-
error,
|
|
311
|
-
});
|
|
312
|
-
this._logIfLost(delivery, error, details);
|
|
313
|
-
return { outcome: 'failed' };
|
|
314
|
-
}
|
|
315
|
-
|
|
316
|
-
/**
|
|
317
|
-
* The one-time scheduler, or null when the stack has none. The mock
|
|
318
|
-
* scheduler is never used: a schedule kept in memory would be lost with
|
|
319
|
-
* the message.
|
|
320
|
-
*/
|
|
321
|
-
getSchedulerService() {
|
|
322
|
-
if (this._schedulerService !== undefined) return this._schedulerService;
|
|
323
|
-
this._schedulerService = null;
|
|
324
|
-
if (
|
|
325
|
-
process.env.SCHEDULER_ROLE_ARN &&
|
|
326
|
-
process.env.SCHEDULER_PROVIDER !== 'mock'
|
|
327
|
-
) {
|
|
328
|
-
const {
|
|
329
|
-
createSchedulerService,
|
|
330
|
-
SCHEDULER_PROVIDERS,
|
|
331
|
-
} = require('../infrastructure/scheduler/scheduler-service-factory');
|
|
332
|
-
this._schedulerService = createSchedulerService({
|
|
333
|
-
provider: SCHEDULER_PROVIDERS.EVENTBRIDGE,
|
|
334
|
-
});
|
|
335
|
-
}
|
|
336
|
-
return this._schedulerService;
|
|
337
|
-
}
|
|
338
|
-
|
|
339
|
-
/**
|
|
340
|
-
* Hook: the message was put back for a rate limit. Runs after it was sent.
|
|
341
|
-
* @param {Object} body The parsed message body.
|
|
342
|
-
* @param {Error} error The RateLimitError.
|
|
343
|
-
* @param {{status: 'WAITING'|'EXHAUSTED', mechanism: string, deferrals: number, retryAt: Date}} state
|
|
344
|
-
*/
|
|
345
|
-
async recordRateLimitWait() {
|
|
346
|
-
// A subclass keeps the run state; nothing to do by default.
|
|
347
|
-
}
|
|
348
|
-
|
|
349
|
-
/**
|
|
350
|
-
* Hook: a message that was put back for a rate limit ran without error.
|
|
351
|
-
* @param {Object} body The parsed message body.
|
|
352
|
-
*/
|
|
353
|
-
async clearRateLimitWait() {
|
|
354
|
-
// A subclass keeps the run state; nothing to do by default.
|
|
355
|
-
}
|
|
356
|
-
|
|
357
|
-
_queueUrl(queueName) {
|
|
358
|
-
this._queueUrls ??= new Map();
|
|
359
|
-
if (!this._queueUrls.has(queueName)) {
|
|
360
|
-
const lookup = this.getQueueURL({ QueueName: queueName }).catch(
|
|
361
|
-
(error) => {
|
|
362
|
-
this._queueUrls.delete(queueName);
|
|
363
|
-
throw error;
|
|
364
|
-
}
|
|
365
|
-
);
|
|
366
|
-
this._queueUrls.set(queueName, lookup);
|
|
367
|
-
}
|
|
368
|
-
return this._queueUrls.get(queueName);
|
|
369
|
-
}
|
|
370
|
-
|
|
371
|
-
async _recordWait(body, error, state) {
|
|
372
|
-
await this._runStateHook('record', () =>
|
|
373
|
-
this.recordRateLimitWait(body, error, state)
|
|
374
|
-
);
|
|
375
|
-
}
|
|
376
|
-
|
|
377
|
-
async _clearDeferredState(body, delivery) {
|
|
378
|
-
const redelivered = delivery.receiveCount > 1;
|
|
379
|
-
if (readDeferral(body).deferrals === 0 && !redelivered) return;
|
|
380
|
-
await this._runStateHook('clear', () => this.clearRateLimitWait(body));
|
|
381
|
-
}
|
|
382
|
-
|
|
383
|
-
async _runStateHook(operation, hook) {
|
|
384
|
-
try {
|
|
385
|
-
await hook();
|
|
386
|
-
} catch (error) {
|
|
387
|
-
log.warn('Rate-limit run state not written', {
|
|
388
|
-
eventName: 'frigg.worker.rate_limit_state_failed',
|
|
389
|
-
operation,
|
|
390
|
-
error,
|
|
391
|
-
});
|
|
392
|
-
}
|
|
393
|
-
}
|
|
394
|
-
|
|
395
|
-
_logIfLost(delivery, error, details) {
|
|
396
|
-
if (!delivery.isLastAttempt) return;
|
|
397
|
-
log.error(
|
|
398
|
-
'Rate-limited record lost: the message goes to the dead-letter queue next',
|
|
399
|
-
{
|
|
400
|
-
eventName: 'frigg.worker.record_lost_rate_limited',
|
|
401
|
-
retryAt: details.retryAt,
|
|
402
|
-
reason: error.reason,
|
|
403
|
-
module: error.module,
|
|
404
|
-
error,
|
|
405
|
-
}
|
|
406
|
-
);
|
|
407
|
-
}
|
|
408
|
-
|
|
409
78
|
// returns the message id
|
|
410
79
|
async send(params, delay = 0) {
|
|
411
80
|
this._validateParams(params);
|
|
@@ -22,15 +22,6 @@ const {
|
|
|
22
22
|
} = require('../integrations/utils/map-integration-dto');
|
|
23
23
|
const { processNotFound } = require('../integrations/use-cases/process-errors');
|
|
24
24
|
|
|
25
|
-
const CLEAR_RATE_LIMIT_TOLERANCE_MS = 5_000;
|
|
26
|
-
|
|
27
|
-
const keepsCurrentRateLimit = (current, next) => {
|
|
28
|
-
if (!current) return false;
|
|
29
|
-
if (next.status === 'EXHAUSTED') return false;
|
|
30
|
-
if (current.status === 'EXHAUSTED') return true;
|
|
31
|
-
return Date.parse(current.retryAt) > next.retryAt.getTime();
|
|
32
|
-
};
|
|
33
|
-
|
|
34
25
|
const loadRouterFromObject = (IntegrationClass, routerObject) => {
|
|
35
26
|
const router = Router();
|
|
36
27
|
const { path, method, event } = routerObject;
|
|
@@ -174,42 +165,6 @@ const createQueueWorker = (integrationClass) => {
|
|
|
174
165
|
const integrationName = integrationClass.Definition.name;
|
|
175
166
|
|
|
176
167
|
class QueueWorker extends Worker {
|
|
177
|
-
async recordRateLimitWait(body, error, state) {
|
|
178
|
-
const processId = body.data?.processId;
|
|
179
|
-
if (!processId) return;
|
|
180
|
-
const processRepository = createProcessRepository();
|
|
181
|
-
const process = await processRepository.findById(processId);
|
|
182
|
-
if (keepsCurrentRateLimit(process?.context?.rateLimit, state))
|
|
183
|
-
return;
|
|
184
|
-
await processRepository.applyProcessUpdate(processId, {
|
|
185
|
-
set: {
|
|
186
|
-
'context.rateLimit': {
|
|
187
|
-
status: state.status,
|
|
188
|
-
mechanism: state.mechanism,
|
|
189
|
-
retryAt: state.retryAt.toISOString(),
|
|
190
|
-
reason: error.reason,
|
|
191
|
-
module: error.module,
|
|
192
|
-
deferrals: state.deferrals,
|
|
193
|
-
updatedAt: new Date().toISOString(),
|
|
194
|
-
},
|
|
195
|
-
},
|
|
196
|
-
});
|
|
197
|
-
}
|
|
198
|
-
|
|
199
|
-
async clearRateLimitWait(body) {
|
|
200
|
-
const processId = body?.data?.processId;
|
|
201
|
-
if (!processId) return;
|
|
202
|
-
const processRepository = createProcessRepository();
|
|
203
|
-
const process = await processRepository.findById(processId);
|
|
204
|
-
const current = process?.context?.rateLimit;
|
|
205
|
-
if (current?.status !== 'WAITING') return;
|
|
206
|
-
const remainingMs = Date.parse(current.retryAt) - Date.now();
|
|
207
|
-
if (remainingMs > CLEAR_RATE_LIMIT_TOLERANCE_MS) return;
|
|
208
|
-
await processRepository.applyProcessUpdate(processId, {
|
|
209
|
-
set: { 'context.rateLimit': null },
|
|
210
|
-
});
|
|
211
|
-
}
|
|
212
|
-
|
|
213
168
|
async _run(params, context, delivery) {
|
|
214
169
|
const logCtx = {
|
|
215
170
|
integration: integrationName,
|
|
@@ -17,6 +17,9 @@ const {
|
|
|
17
17
|
const {
|
|
18
18
|
UpdateIntegrationConfig,
|
|
19
19
|
} = require('./use-cases/update-integration-config');
|
|
20
|
+
const {
|
|
21
|
+
RecordRateLimitMessage,
|
|
22
|
+
} = require('./use-cases/record-rate-limit-message');
|
|
20
23
|
const { validateExtensionBinding } = require('./extension');
|
|
21
24
|
const { getTelemetry } = require('../telemetry/telemetry-runtime');
|
|
22
25
|
const { instrumentHandler } = require('../telemetry/instrument-handler');
|
|
@@ -59,6 +62,9 @@ class IntegrationBase {
|
|
|
59
62
|
updateIntegrationConfig = new UpdateIntegrationConfig({
|
|
60
63
|
integrationRepository: this.integrationRepository,
|
|
61
64
|
});
|
|
65
|
+
recordRateLimitMessage = new RecordRateLimitMessage({
|
|
66
|
+
integrationRepository: this.integrationRepository,
|
|
67
|
+
});
|
|
62
68
|
|
|
63
69
|
// this.telemetry.count('records.synced', n, { entity: 'contact' })
|
|
64
70
|
telemetry = bindTelemetryContext(getTelemetry(), () =>
|
|
@@ -865,7 +871,9 @@ class IntegrationBase {
|
|
|
865
871
|
* something integration-level needs attention. Today this catches the
|
|
866
872
|
* `CREDENTIAL_INVALIDATED` event Module fires from `markCredentialsInvalid`
|
|
867
873
|
* and flips this integration's status to ERROR so the queue worker
|
|
868
|
-
* stops processing further webhooks until the user re-authorizes.
|
|
874
|
+
* stops processing further webhooks until the user re-authorizes. It also
|
|
875
|
+
* catches `RATE_LIMITED`, which records one warning for the user and
|
|
876
|
+
* changes no status: a rate limit is not an error.
|
|
869
877
|
*
|
|
870
878
|
* Modules are wired to this delegate in `_appendModules()`, which runs
|
|
871
879
|
* during `setIntegrationRecord()` — this covers every construction path
|
|
@@ -908,6 +916,17 @@ class IntegrationBase {
|
|
|
908
916
|
});
|
|
909
917
|
await this.persistStatus('ENABLED');
|
|
910
918
|
}
|
|
919
|
+
|
|
920
|
+
if (delegateString === 'RATE_LIMITED') {
|
|
921
|
+
const moduleName = object?.moduleName ?? notifier?.name;
|
|
922
|
+
this.logger.warn('Module reported a rate limit', {
|
|
923
|
+
eventName: `${this.logger.name}.rate_limited`,
|
|
924
|
+
moduleName,
|
|
925
|
+
reason: object?.reason,
|
|
926
|
+
retryAt: object?.retryAt,
|
|
927
|
+
});
|
|
928
|
+
await this._recordRateLimitMessage({ ...object, moduleName });
|
|
929
|
+
}
|
|
911
930
|
}
|
|
912
931
|
|
|
913
932
|
/**
|
|
@@ -933,6 +952,23 @@ class IntegrationBase {
|
|
|
933
952
|
});
|
|
934
953
|
}
|
|
935
954
|
}
|
|
955
|
+
|
|
956
|
+
/**
|
|
957
|
+
* Best-effort: the module's request is about to throw a RateLimitError,
|
|
958
|
+
* and a failed warning must not change that. A rate limit is not an ERROR,
|
|
959
|
+
* so this never touches the integration status.
|
|
960
|
+
* @param {Object} payload - The RATE_LIMITED payload of the module.
|
|
961
|
+
*/
|
|
962
|
+
async _recordRateLimitMessage(payload) {
|
|
963
|
+
try {
|
|
964
|
+
await this.recordRateLimitMessage.execute(this.id, payload);
|
|
965
|
+
} catch (error) {
|
|
966
|
+
this.logger.error('Failed to record rate limit message', {
|
|
967
|
+
eventName: `${this.logger.name}.rate_limit_message_record_failed`,
|
|
968
|
+
error,
|
|
969
|
+
});
|
|
970
|
+
}
|
|
971
|
+
}
|
|
936
972
|
}
|
|
937
973
|
|
|
938
974
|
module.exports = { IntegrationBase };
|