backend-manager 5.2.12 → 5.2.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +17 -0
- package/docs/consent.md +5 -5
- package/docs/testing.md +3 -1
- package/package.json +3 -3
- package/src/manager/libraries/email/data/disposable-domains.json +13 -0
- package/src/manager/libraries/email/marketing/index.js +5 -0
- package/src/manager/libraries/infer-contact.js +5 -5
- package/src/manager/routes/marketing/webhook/post.js +12 -72
- package/src/manager/routes/marketing/webhook/processors/beehiiv.js +3 -2
- package/src/manager/routes/marketing/webhook/processors/sendgrid.js +4 -3
- package/src/test/test-accounts.js +1 -1
- package/templates/_.env +2 -3
- package/test/routes/marketing/webhook.js +38 -50
package/CHANGELOG.md
CHANGED
|
@@ -14,6 +14,23 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
|
|
14
14
|
- `Fixed` for any bug fixes.
|
|
15
15
|
- `Security` in case of vulnerabilities.
|
|
16
16
|
|
|
17
|
+
# [5.2.13] - 2026-05-28
|
|
18
|
+
|
|
19
|
+
### Removed
|
|
20
|
+
|
|
21
|
+
- **Dropped the `marketing-webhooks` Firestore idempotency ledger.** The marketing-webhook dispatcher (`src/manager/routes/marketing/webhook/post.js`) no longer reads/writes `marketing-webhooks/{eventId}` docs for dedup. Both handler side effects — writing `consent.marketing.status = 'revoked'` and the cross-provider `mailer.remove()` — are naturally idempotent, so a provider retry or duplicate parent fan-out re-runs to the same end state with no extra side effects. (This is the key difference from `payments-webhooks`, where dedup is load-bearing because payment side effects are NOT idempotent.) Events with no `eventId` are now processed instead of skipped, since dedup is no longer required. Removed `marketing-webhooks` from the test runner's pre-test cleanup list (`src/test/test-accounts.js`). Consumers with an existing `marketing-webhooks` collection can safely delete it — nothing reads or writes it anymore.
|
|
22
|
+
|
|
23
|
+
### Fixed
|
|
24
|
+
|
|
25
|
+
- **`libraries/infer-contact.js`: guard the optional `assistant` arg.** All log/error calls now use `assistant?.` so `inferContact(email)` works without an assistant. Previously threw a `TypeError` when `BACKEND_MANAGER_OPENAI_API_KEY` was unset and no assistant was passed.
|
|
26
|
+
- **`libraries/email/marketing/index.js`: gate `Marketing.remove()` behind test mode.** `remove()` now short-circuits with `if (assistant.isTesting() && !process.env.TEST_EXTENDED_MODE) return {}`, matching `add()` and `sync()`. Closes a gap where auth `onDelete` (fired ~44× at test startup during user cleanup) could hit the live SendGrid/Beehiiv remove APIs during a normal test run. The gate lives at the library SSOT so every caller (onDelete, webhook processors, contact-delete route) inherits it.
|
|
27
|
+
|
|
28
|
+
### Changed
|
|
29
|
+
|
|
30
|
+
- **Dependency bumps**: `@anthropic-ai/claude-agent-sdk` ^0.3.152 → ^0.3.153, `stripe` ^22.1.1 → ^22.2.0.
|
|
31
|
+
- **`templates/_.env`**: consolidate OpenAI/Anthropic keys under an "AI" section and add `CLAUDE_CODE_OAUTH_TOKEN`.
|
|
32
|
+
- **Marketing webhook tests** (`test/routes/marketing/webhook.js`): renamed `*-duplicate-event-skipped` → `*-reprocessed-idempotently` (assert re-delivery reprocesses and the user stays revoked); `sendgrid-event-without-eventId-*` now asserts the event is processed; beehiiv idempotency variant skips when no publication is configured. Updated `docs/consent.md` and `docs/testing.md` to describe the no-ledger design.
|
|
33
|
+
|
|
17
34
|
# [5.2.12] - 2026-05-27
|
|
18
35
|
|
|
19
36
|
### Changed
|
package/docs/consent.md
CHANGED
|
@@ -121,9 +121,9 @@ POST /backend-manager/marketing/webhook?provider=beehiiv&key=<BACKEND_MANAGER_WE
|
|
|
121
121
|
The dispatcher loads `processors/{provider}.js`, parses the event(s), and for each event:
|
|
122
122
|
|
|
123
123
|
1. Checks `isSupported(eventType)` — filters out non-revoke events like `delivered` / `open`.
|
|
124
|
-
2.
|
|
125
|
-
|
|
126
|
-
|
|
124
|
+
2. Calls `handleEvent({ Manager, assistant, parsed })` on the processor.
|
|
125
|
+
|
|
126
|
+
There is **no idempotency ledger**. Both handler side effects — writing `consent.marketing.status = 'revoked'` and calling `mailer.remove()` — are idempotent, so a provider retry (or a duplicate fan-out from the parent) re-runs to the same end state with no extra side effects. This is the key difference from `payments-webhooks`, where dedup is load-bearing because payment side effects are not idempotent.
|
|
127
127
|
|
|
128
128
|
Each processor's `handleEvent` does the same shape of work:
|
|
129
129
|
|
|
@@ -159,7 +159,7 @@ The parent forwarder:
|
|
|
159
159
|
2. Reads the `brands` collection from the parent's own Firestore.
|
|
160
160
|
3. For each brand: derives the child API URL by inserting `api.` into the brand's URL (`https://somiibo.com` → `https://api.somiibo.com/backend-manager/marketing/webhook?provider=X&key=Y`).
|
|
161
161
|
4. POSTs the raw provider body to every child in parallel via `Promise.allSettled`.
|
|
162
|
-
5. Returns 200 even if some children fail — child
|
|
162
|
+
5. Returns 200 even if some children fail — idempotent child handlers make provider retries (and re-fans) safe.
|
|
163
163
|
|
|
164
164
|
### Why fan-out instead of central processing
|
|
165
165
|
|
|
@@ -167,7 +167,7 @@ Each brand has its own Firebase project, so its `users` collection is separate.
|
|
|
167
167
|
|
|
168
168
|
- **Correct per-brand updates** — only brands where the user actually has an account update their user docs.
|
|
169
169
|
- **Failure isolation** — one child being down doesn't block updates on the others.
|
|
170
|
-
- **
|
|
170
|
+
- **Idempotent handlers** — re-processing the same event (provider retry or re-fan) produces the same end state, so no dedup ledger is needed.
|
|
171
171
|
- **No new schema** — no need for the parent to maintain a brand → publication map; each child filters on its own.
|
|
172
172
|
|
|
173
173
|
### Why self IS in the fan-out
|
package/docs/testing.md
CHANGED
|
@@ -19,7 +19,7 @@ What the runner wipes pre-test (in [src/test/test-accounts.js](../src/test/test-
|
|
|
19
19
|
|
|
20
20
|
1. **`meta/stats`** doc ensured (required for on-create batch writes).
|
|
21
21
|
2. **`users/_test-*`** Firebase Auth users + Firestore docs (delete).
|
|
22
|
-
3. **Mixed Firestore collections** — `payments-orders`, `payments-webhooks`, `payments-intents`, `payments-disputes
|
|
22
|
+
3. **Mixed Firestore collections** — `payments-orders`, `payments-webhooks`, `payments-intents`, `payments-disputes`. Two-pass cleanup per collection:
|
|
23
23
|
- Pass 1 — owner-keyed: `where('owner', 'in', [...testUids])` (batched at 30 uids per `in` query).
|
|
24
24
|
- Pass 2 — id-keyed: any doc whose ID starts with `_test-` (catches ownerless test docs like dispute alerts and raw test webhooks).
|
|
25
25
|
4. **Test-only Firestore collections** — `_test`, `_test_query` — wiped in full.
|
|
@@ -52,6 +52,8 @@ The rule: **never put cleanup at the END of a test file or suite for the purpose
|
|
|
52
52
|
|
|
53
53
|
Several routes/handlers skip external API calls (SendGrid, Beehiiv, Stripe webhooks, dispute handlers, marketing libraries) when `process.env.TEST_EXTENDED_MODE` is unset, so unit tests don't fire real emails or webhook side effects. Set the flag to opt **in** to those side effects for a full end-to-end run.
|
|
54
54
|
|
|
55
|
+
The marketing library gates at the SSOT level: `Marketing.add()`, `Marketing.sync()`, and `Marketing.remove()` each short-circuit with `if (assistant.isTesting() && !process.env.TEST_EXTENDED_MODE) return {}` before touching any provider. Callers (auth `onDelete`, webhook processors, contact-delete route) inherit the gate for free — do NOT rely on a per-caller guard for provider safety; add the gate to the library method itself when introducing a new provider-touching method.
|
|
56
|
+
|
|
55
57
|
**Live sync — no env coordination across terminals.** The flag flows automatically from the test command to the running emulator via a small shared state file at `<projectRoot>/.temp/test-mode.json`. The test command writes the file pre-flight; the emulator's function workers watch it via `fs.watch` and mutate their own `process.env.TEST_EXTENDED_MODE` in place. Effect: you only need to set the flag on **the test command**. The emulator follows.
|
|
56
58
|
|
|
57
59
|
```bash
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "backend-manager",
|
|
3
|
-
"version": "5.2.
|
|
3
|
+
"version": "5.2.13",
|
|
4
4
|
"description": "Quick tools for developing Firebase functions",
|
|
5
5
|
"main": "src/manager/index.js",
|
|
6
6
|
"bin": {
|
|
@@ -49,7 +49,7 @@
|
|
|
49
49
|
}
|
|
50
50
|
},
|
|
51
51
|
"dependencies": {
|
|
52
|
-
"@anthropic-ai/claude-agent-sdk": "^0.3.
|
|
52
|
+
"@anthropic-ai/claude-agent-sdk": "^0.3.153",
|
|
53
53
|
"@anthropic-ai/sdk": "^0.99.0",
|
|
54
54
|
"@firebase/rules-unit-testing": "^5.0.1",
|
|
55
55
|
"@google-cloud/firestore": "^7.11.6",
|
|
@@ -86,7 +86,7 @@
|
|
|
86
86
|
"pushid": "^1.0.0",
|
|
87
87
|
"sanitize-html": "^2.17.4",
|
|
88
88
|
"sharp": "^0.34.5",
|
|
89
|
-
"stripe": "^22.
|
|
89
|
+
"stripe": "^22.2.0",
|
|
90
90
|
"uid-generator": "^2.0.0",
|
|
91
91
|
"uuid": "^14.0.0",
|
|
92
92
|
"wonderful-fetch": "^2.0.5",
|
|
@@ -120,6 +120,7 @@
|
|
|
120
120
|
"24mail.top",
|
|
121
121
|
"25u.com",
|
|
122
122
|
"2anom.com",
|
|
123
|
+
"2bd.net",
|
|
123
124
|
"2chmail.net",
|
|
124
125
|
"2ether.net",
|
|
125
126
|
"2fdgdfgdfgdf.tk",
|
|
@@ -335,6 +336,7 @@
|
|
|
335
336
|
"air2token.com",
|
|
336
337
|
"airmailbox.website",
|
|
337
338
|
"airsworld.net",
|
|
339
|
+
"aisubpro.click",
|
|
338
340
|
"aiworldx.com",
|
|
339
341
|
"aixnv.com",
|
|
340
342
|
"ajaxapp.net",
|
|
@@ -1183,6 +1185,7 @@
|
|
|
1183
1185
|
"dim-coin.com",
|
|
1184
1186
|
"dimalk.com",
|
|
1185
1187
|
"dingbone.com",
|
|
1188
|
+
"dino.icu",
|
|
1186
1189
|
"diolang.com",
|
|
1187
1190
|
"directmail24.net",
|
|
1188
1191
|
"dis.hopto.org",
|
|
@@ -2054,6 +2057,7 @@
|
|
|
2054
2057
|
"gowikitv.com",
|
|
2055
2058
|
"grandmamail.com",
|
|
2056
2059
|
"grandmasmail.com",
|
|
2060
|
+
"graphicenda.site",
|
|
2057
2061
|
"grassdev.com",
|
|
2058
2062
|
"gravityengine.cc",
|
|
2059
2063
|
"great-host.in",
|
|
@@ -2709,6 +2713,7 @@
|
|
|
2709
2713
|
"laste.ml",
|
|
2710
2714
|
"lastmail.co",
|
|
2711
2715
|
"lastmail.com",
|
|
2716
|
+
"launders.money",
|
|
2712
2717
|
"lawlita.com",
|
|
2713
2718
|
"laxex.ru",
|
|
2714
2719
|
"laxex.store",
|
|
@@ -2757,11 +2762,13 @@
|
|
|
2757
2762
|
"link2mail.net",
|
|
2758
2763
|
"linkedintuts2016.pw",
|
|
2759
2764
|
"linkmail.info",
|
|
2765
|
+
"linkpc.net",
|
|
2760
2766
|
"linlshe.com",
|
|
2761
2767
|
"linshiyou.com",
|
|
2762
2768
|
"linshiyouxiang.net",
|
|
2763
2769
|
"linuxmail.so",
|
|
2764
2770
|
"liocbrco.com",
|
|
2771
|
+
"liopsacac.shop",
|
|
2765
2772
|
"lista.cc",
|
|
2766
2773
|
"litedrop.com",
|
|
2767
2774
|
"liveradio.tk",
|
|
@@ -3418,6 +3425,7 @@
|
|
|
3418
3425
|
"newfilm24.ru",
|
|
3419
3426
|
"newideasfornewpeople.info",
|
|
3420
3427
|
"newmail.top",
|
|
3428
|
+
"newmano.store",
|
|
3421
3429
|
"newyork.io.vn",
|
|
3422
3430
|
"next.ovh",
|
|
3423
3431
|
"nextmail.info",
|
|
@@ -3498,6 +3506,7 @@
|
|
|
3498
3506
|
"nospam4.us",
|
|
3499
3507
|
"nospamfor.us",
|
|
3500
3508
|
"nospamthanks.info",
|
|
3509
|
+
"nosubcriber.shop",
|
|
3501
3510
|
"notboxletters.com",
|
|
3502
3511
|
"nothingtoseehere.ca",
|
|
3503
3512
|
"notif.me",
|
|
@@ -3697,6 +3706,7 @@
|
|
|
3697
3706
|
"payspun.com",
|
|
3698
3707
|
"pazard.com",
|
|
3699
3708
|
"pazuric.com",
|
|
3709
|
+
"pbhak.dev",
|
|
3700
3710
|
"pckage.com",
|
|
3701
3711
|
"pdf-cutter.com",
|
|
3702
3712
|
"pe.hu",
|
|
@@ -3957,6 +3967,7 @@
|
|
|
3957
3967
|
"rapidefr.fr.nf",
|
|
3958
3968
|
"rapt.be",
|
|
3959
3969
|
"raqid.com",
|
|
3970
|
+
"ravavo.bond",
|
|
3960
3971
|
"rawr.foo",
|
|
3961
3972
|
"rax.la",
|
|
3962
3973
|
"raxtest.com",
|
|
@@ -5114,6 +5125,7 @@
|
|
|
5114
5125
|
"vuket.org",
|
|
5115
5126
|
"vulca.sbs",
|
|
5116
5127
|
"vusra.com",
|
|
5128
|
+
"vutrugay.org",
|
|
5117
5129
|
"vvatxiy.com",
|
|
5118
5130
|
"vwhins.com",
|
|
5119
5131
|
"vxsolar.com",
|
|
@@ -5243,6 +5255,7 @@
|
|
|
5243
5255
|
"womp-wo.mp",
|
|
5244
5256
|
"woofidog.fr.nf",
|
|
5245
5257
|
"woopros.com",
|
|
5258
|
+
"work.gd",
|
|
5246
5259
|
"workingtall.com",
|
|
5247
5260
|
"worldlylife.store",
|
|
5248
5261
|
"worldspace.link",
|
|
@@ -230,6 +230,11 @@ Marketing.prototype.remove = async function (email) {
|
|
|
230
230
|
return {};
|
|
231
231
|
}
|
|
232
232
|
|
|
233
|
+
if (assistant.isTesting() && !process.env.TEST_EXTENDED_MODE) {
|
|
234
|
+
assistant.log('Marketing.remove(): Skipping providers (testing mode)');
|
|
235
|
+
return {};
|
|
236
|
+
}
|
|
237
|
+
|
|
233
238
|
assistant.log('Marketing.remove():', { email });
|
|
234
239
|
|
|
235
240
|
const results = {};
|
|
@@ -26,9 +26,9 @@ async function inferContact(email, assistant) {
|
|
|
26
26
|
if (aiResult) {
|
|
27
27
|
return aiResult;
|
|
28
28
|
}
|
|
29
|
-
assistant
|
|
29
|
+
assistant?.log(`inferContact: AI returned null for ${email} — falling back to empty result`);
|
|
30
30
|
} else {
|
|
31
|
-
assistant
|
|
31
|
+
assistant?.log(`inferContact: BACKEND_MANAGER_OPENAI_API_KEY not set — skipping AI inference for ${email}`);
|
|
32
32
|
}
|
|
33
33
|
|
|
34
34
|
return { firstName: '', lastName: '', company: '', confidence: 0, method: 'none' };
|
|
@@ -68,14 +68,14 @@ async function inferContactWithAI(email, assistant) {
|
|
|
68
68
|
method: 'ai',
|
|
69
69
|
};
|
|
70
70
|
if (!inferred.firstName && !inferred.lastName && !inferred.company) {
|
|
71
|
-
assistant
|
|
71
|
+
assistant?.log(`inferContactWithAI: AI parsed response had ALL fields empty for ${email}. Raw:`, parsed);
|
|
72
72
|
}
|
|
73
73
|
return inferred;
|
|
74
74
|
}
|
|
75
75
|
|
|
76
|
-
assistant
|
|
76
|
+
assistant?.log(`inferContactWithAI: AI response missing firstName for ${email}. Raw result:`, result);
|
|
77
77
|
} catch (e) {
|
|
78
|
-
assistant
|
|
78
|
+
assistant?.error('inferContactWithAI: Failed:', e);
|
|
79
79
|
}
|
|
80
80
|
|
|
81
81
|
return null;
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
* 2. Optionally rejects mismatched brand via ?brand= filter
|
|
7
7
|
* 3. Loads the matching processor module from ./processors/{provider}.js
|
|
8
8
|
* 4. Parses the webhook payload into one or more normalized events
|
|
9
|
-
* 5. For each event:
|
|
9
|
+
* 5. For each supported event: dispatch to the processor's handler
|
|
10
10
|
* 6. Returns 200 immediately so the provider doesn't retry
|
|
11
11
|
*
|
|
12
12
|
* Each processor module defines:
|
|
@@ -14,15 +14,13 @@
|
|
|
14
14
|
* - isSupported(type) — returns true if this event should be processed
|
|
15
15
|
* - handleEvent(ctx) — does the work for one event (user doc + cross-provider sync)
|
|
16
16
|
*
|
|
17
|
-
*
|
|
18
|
-
*
|
|
19
|
-
*
|
|
17
|
+
* No idempotency ledger: every supported handler (revoke consent + cross-provider
|
|
18
|
+
* remove) is naturally idempotent — re-processing a provider retry produces the same
|
|
19
|
+
* end state with no extra side effects, so duplicate suppression buys nothing.
|
|
20
20
|
*/
|
|
21
21
|
const path = require('path');
|
|
22
|
-
const powertools = require('node-powertools');
|
|
23
22
|
|
|
24
|
-
module.exports = async ({ assistant, Manager
|
|
25
|
-
const { admin } = libraries;
|
|
23
|
+
module.exports = async ({ assistant, Manager }) => {
|
|
26
24
|
const query = assistant.request.query;
|
|
27
25
|
|
|
28
26
|
const provider = query.provider;
|
|
@@ -78,7 +76,7 @@ module.exports = async ({ assistant, Manager, libraries }) => {
|
|
|
78
76
|
// Use Promise.allSettled so we return success only after all events have been
|
|
79
77
|
// attempted.
|
|
80
78
|
const results = await Promise.allSettled(
|
|
81
|
-
events.map((event) => processOneEvent({ Manager, assistant,
|
|
79
|
+
events.map((event) => processOneEvent({ Manager, assistant, provider, event, processorModule }))
|
|
82
80
|
);
|
|
83
81
|
|
|
84
82
|
let processed = 0;
|
|
@@ -100,81 +98,23 @@ module.exports = async ({ assistant, Manager, libraries }) => {
|
|
|
100
98
|
};
|
|
101
99
|
|
|
102
100
|
/**
|
|
103
|
-
* Process a single event
|
|
101
|
+
* Process a single event: support check, then dispatch to the processor's handler.
|
|
102
|
+
* Handlers are idempotent, so provider retries re-run safely with no dedup ledger.
|
|
104
103
|
* Returns { processed: bool, skipped?: string, error?: any }.
|
|
105
104
|
*/
|
|
106
|
-
async function processOneEvent({ Manager, assistant,
|
|
107
|
-
const {
|
|
108
|
-
|
|
109
|
-
// No eventId means we can't dedupe — skip rather than risk double-processing
|
|
110
|
-
if (!eventId) {
|
|
111
|
-
assistant.log(`marketing webhook: ${provider} event missing eventId (type=${eventType}), skipping`);
|
|
112
|
-
return { processed: false, skipped: 'missing-event-id' };
|
|
113
|
-
}
|
|
105
|
+
async function processOneEvent({ Manager, assistant, provider, event, processorModule }) {
|
|
106
|
+
const { eventType } = event;
|
|
114
107
|
|
|
115
108
|
// Filter by supported event types
|
|
116
109
|
if (processorModule.isSupported && !processorModule.isSupported(eventType)) {
|
|
117
110
|
return { processed: false, skipped: 'unsupported-event-type' };
|
|
118
111
|
}
|
|
119
112
|
|
|
120
|
-
// Idempotency: skip if we've already processed this event
|
|
121
|
-
const idempotencyRef = admin.firestore().doc(`marketing-webhooks/${eventId}`);
|
|
122
|
-
const existingDoc = await idempotencyRef.get();
|
|
123
|
-
|
|
124
|
-
if (existingDoc.exists) {
|
|
125
|
-
const existingStatus = existingDoc.data()?.status;
|
|
126
|
-
if (existingStatus !== 'failed') {
|
|
127
|
-
assistant.log(`marketing webhook: ${provider} duplicate event ${eventId} (status=${existingStatus}), skipping`);
|
|
128
|
-
return { processed: false, skipped: 'duplicate' };
|
|
129
|
-
}
|
|
130
|
-
assistant.log(`marketing webhook: ${provider} retrying previously failed event ${eventId}`);
|
|
131
|
-
}
|
|
132
|
-
|
|
133
|
-
// Build the audit doc
|
|
134
|
-
const now = powertools.timestamp(new Date(), { output: 'string' });
|
|
135
|
-
const nowUNIX = powertools.timestamp(now, { output: 'unix' });
|
|
136
|
-
|
|
137
|
-
// Write 'pending' state before dispatching so concurrent deliveries see the lock
|
|
138
|
-
await idempotencyRef.set({
|
|
139
|
-
id: eventId,
|
|
140
|
-
provider,
|
|
141
|
-
status: 'pending',
|
|
142
|
-
raw: event.raw || null,
|
|
143
|
-
event: {
|
|
144
|
-
type: eventType,
|
|
145
|
-
email: event.email || null,
|
|
146
|
-
timestamp: event.timestamp || null,
|
|
147
|
-
},
|
|
148
|
-
error: null,
|
|
149
|
-
metadata: {
|
|
150
|
-
created: { timestamp: now, timestampUNIX: nowUNIX },
|
|
151
|
-
completed: { timestamp: null, timestampUNIX: null },
|
|
152
|
-
},
|
|
153
|
-
});
|
|
154
|
-
|
|
155
|
-
// Dispatch to the processor's event handler
|
|
156
|
-
let handlerResult;
|
|
157
113
|
try {
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
// Mark completed
|
|
161
|
-
await idempotencyRef.set({
|
|
162
|
-
status: 'completed',
|
|
163
|
-
result: handlerResult || null,
|
|
164
|
-
metadata: {
|
|
165
|
-
completed: { timestamp: powertools.timestamp(new Date(), { output: 'string' }), timestampUNIX: powertools.timestamp(new Date(), { output: 'unix' }) },
|
|
166
|
-
},
|
|
167
|
-
}, { merge: true });
|
|
168
|
-
|
|
114
|
+
await processorModule.handleEvent({ Manager, assistant, parsed: event });
|
|
169
115
|
return { processed: true };
|
|
170
116
|
} catch (e) {
|
|
171
|
-
assistant.error(`marketing webhook: handler failed for ${provider} event ${eventId}:`, e);
|
|
172
|
-
|
|
173
|
-
await idempotencyRef.set({
|
|
174
|
-
status: 'failed',
|
|
175
|
-
error: { message: e.message || String(e), stack: e.stack || null },
|
|
176
|
-
}, { merge: true }).catch(() => {});
|
|
177
|
-
|
|
117
|
+
assistant.error(`marketing webhook: handler failed for ${provider} event ${event.eventId} (${eventType}):`, e);
|
|
178
118
|
return { processed: false, error: e };
|
|
179
119
|
}
|
|
180
120
|
}
|
|
@@ -19,7 +19,8 @@
|
|
|
19
19
|
* - getPublicationId() reads from config.marketing.beehiiv.publicationId
|
|
20
20
|
* or fuzzy-matches by brand name against the Beehiiv API.
|
|
21
21
|
*
|
|
22
|
-
*
|
|
22
|
+
* No idempotency ledger — the revoke + cross-provider remove are idempotent, so a
|
|
23
|
+
* provider retry re-runs safely with the same end state.
|
|
23
24
|
*/
|
|
24
25
|
|
|
25
26
|
const REVOKE_EVENT_TYPES = new Set([
|
|
@@ -85,7 +86,7 @@ function isSupported(eventType) {
|
|
|
85
86
|
}
|
|
86
87
|
|
|
87
88
|
/**
|
|
88
|
-
* Process a single parsed event. Called by the dispatcher
|
|
89
|
+
* Process a single parsed event. Called by the dispatcher for each supported event.
|
|
89
90
|
* Returns a result object summarizing what happened.
|
|
90
91
|
*/
|
|
91
92
|
async function handleEvent({ Manager, assistant, parsed }) {
|
|
@@ -19,7 +19,8 @@
|
|
|
19
19
|
* GROUPS.marketing (25928) is account-global across all brands, an unsub from group 25928
|
|
20
20
|
* legitimately removes the user from marketing across the entire SendGrid account.
|
|
21
21
|
*
|
|
22
|
-
*
|
|
22
|
+
* No idempotency ledger — the revoke + cross-provider remove are idempotent, so a
|
|
23
|
+
* provider retry re-runs safely with the same end state.
|
|
23
24
|
*/
|
|
24
25
|
|
|
25
26
|
const REVOKE_EVENT_TYPES = new Set([
|
|
@@ -61,7 +62,7 @@ function parseWebhook(req) {
|
|
|
61
62
|
}
|
|
62
63
|
|
|
63
64
|
return events.map((event) => {
|
|
64
|
-
// sg_event_id is SendGrid's per-event unique ID,
|
|
65
|
+
// sg_event_id is SendGrid's per-event unique ID, retained for log context.
|
|
65
66
|
// smtp-id is another stable identifier we fall back to.
|
|
66
67
|
const eventId = event.sg_event_id || event['smtp-id'] || event.smtpId || null;
|
|
67
68
|
const eventType = event.event;
|
|
@@ -88,7 +89,7 @@ function isSupported(eventType) {
|
|
|
88
89
|
}
|
|
89
90
|
|
|
90
91
|
/**
|
|
91
|
-
* Process a single parsed event. Called by the dispatcher
|
|
92
|
+
* Process a single parsed event. Called by the dispatcher for each supported event.
|
|
92
93
|
*
|
|
93
94
|
* Returns a result object summarizing what happened (for logging/response).
|
|
94
95
|
*/
|
|
@@ -694,7 +694,7 @@ async function deleteTestUsers(admin) {
|
|
|
694
694
|
// identified solely by an `_test-` doc id prefix (id-keyed). All must be wiped
|
|
695
695
|
// at the start of every run so a test that died mid-execution leaves no
|
|
696
696
|
// ghosts. New collections that participate in tests MUST be added here too.
|
|
697
|
-
const testDataCollections = ['payments-orders', 'payments-webhooks', 'payments-intents', 'payments-disputes'
|
|
697
|
+
const testDataCollections = ['payments-orders', 'payments-webhooks', 'payments-intents', 'payments-disputes'];
|
|
698
698
|
// Collections that exist solely for tests — wipe in full. All docs in these
|
|
699
699
|
// collections come from tests, so a single recursive delete handles cleanup.
|
|
700
700
|
const testOnlyCollections = ['_test', '_test_query'];
|
package/templates/_.env
CHANGED
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
* - Auth via ?key= query param
|
|
9
9
|
* - Provider validation
|
|
10
10
|
* - Brand filter (ignore mismatched brand)
|
|
11
|
-
* -
|
|
11
|
+
* - Idempotent re-delivery (handlers re-run safely; no dedup ledger)
|
|
12
12
|
*
|
|
13
13
|
* SendGrid processor tests:
|
|
14
14
|
* - Various event types (group_unsubscribe, unsubscribe, spamreport, bounce, dropped)
|
|
@@ -120,12 +120,6 @@ module.exports = {
|
|
|
120
120
|
assert.equal(userDoc?.consent?.marketing?.status, 'revoked', 'marketing.status should be revoked');
|
|
121
121
|
assert.equal(userDoc?.consent?.marketing?.revokedAt?.source, 'sendgrid', 'revokedAt.source should be sendgrid');
|
|
122
122
|
assert.equal(userDoc?.consent?.marketing?.revokedAt?.timestampUNIX, eventTimestamp, 'revokedAt.timestampUNIX should match event timestamp');
|
|
123
|
-
|
|
124
|
-
// Idempotency doc should exist with status=completed
|
|
125
|
-
const webhookDoc = await firestore.get(`marketing-webhooks/${eventId}`);
|
|
126
|
-
assert.ok(webhookDoc, 'Idempotency doc should exist');
|
|
127
|
-
assert.equal(webhookDoc?.status, 'completed', 'Idempotency doc should be marked completed');
|
|
128
|
-
assert.equal(webhookDoc?.provider, 'sendgrid', 'Idempotency doc should record provider');
|
|
129
123
|
},
|
|
130
124
|
},
|
|
131
125
|
|
|
@@ -244,9 +238,9 @@ module.exports = {
|
|
|
244
238
|
[sgEvent({ id: eventId, type: 'group_unsubscribe', email: '_test.never-existed@example.com' })]
|
|
245
239
|
);
|
|
246
240
|
|
|
247
|
-
// Dispatcher still
|
|
248
|
-
//
|
|
249
|
-
//
|
|
241
|
+
// Dispatcher still runs the handler (which returns handled:false). From the
|
|
242
|
+
// dispatcher's POV this counts as 'processed=1' since the handler didn't throw.
|
|
243
|
+
// The handler's internal "user-not-found" branch is silent by design.
|
|
250
244
|
assert.isSuccess(response, 'Should accept unknown-email gracefully');
|
|
251
245
|
assert.propertyEquals(response, 'data.failed', 0, 'No failures for unknown email');
|
|
252
246
|
},
|
|
@@ -277,28 +271,19 @@ module.exports = {
|
|
|
277
271
|
assert.propertyEquals(response, 'data.processed', 2, '2 supported events should be processed');
|
|
278
272
|
assert.propertyEquals(response, 'data.skipped', 1, '1 unsupported event should be skipped');
|
|
279
273
|
|
|
280
|
-
// Each processed event gets its own idempotency doc
|
|
281
|
-
const doc1 = await firestore.get(`marketing-webhooks/${e1}`);
|
|
282
|
-
const doc3 = await firestore.get(`marketing-webhooks/${e3}`);
|
|
283
|
-
assert.ok(doc1 && doc1.status === 'completed', 'First event idempotency doc completed');
|
|
284
|
-
assert.ok(doc3 && doc3.status === 'completed', 'Third event idempotency doc completed');
|
|
285
|
-
|
|
286
|
-
// The skipped event should NOT have an idempotency doc (we filter by isSupported before writing)
|
|
287
|
-
const doc2 = await firestore.get(`marketing-webhooks/${e2}`);
|
|
288
|
-
assert.ok(!doc2, 'Unsupported event should NOT have an idempotency doc');
|
|
289
|
-
|
|
290
274
|
// User doc should be revoked
|
|
291
275
|
const userDoc = await firestore.get(`users/${uid}`);
|
|
292
276
|
assert.equal(userDoc?.consent?.marketing?.status, 'revoked');
|
|
293
277
|
},
|
|
294
278
|
},
|
|
295
279
|
|
|
296
|
-
// ───
|
|
280
|
+
// ─── Idempotent re-delivery (no dedup ledger) ───
|
|
297
281
|
|
|
298
282
|
{
|
|
299
|
-
name: 'sendgrid-duplicate-event-
|
|
283
|
+
name: 'sendgrid-duplicate-event-reprocessed-idempotently',
|
|
300
284
|
auth: 'none',
|
|
301
285
|
async run({ http, firestore, assert, accounts }) {
|
|
286
|
+
const uid = accounts.basic.uid;
|
|
302
287
|
const email = accounts.basic.email;
|
|
303
288
|
const eventId = sgEventId('duplicate');
|
|
304
289
|
|
|
@@ -310,27 +295,27 @@ module.exports = {
|
|
|
310
295
|
assert.isSuccess(response1);
|
|
311
296
|
assert.propertyEquals(response1, 'data.processed', 1, 'First delivery should process');
|
|
312
297
|
|
|
313
|
-
// Second delivery — same eventId
|
|
298
|
+
// Second delivery — same eventId. With no dedup ledger the handler runs
|
|
299
|
+
// again, but the revoke is idempotent so the end state is unchanged.
|
|
314
300
|
const response2 = await http.as('none').post(
|
|
315
301
|
`marketing/webhook?provider=sendgrid&key=${process.env.BACKEND_MANAGER_WEBHOOK_KEY}`,
|
|
316
302
|
[sgEvent({ id: eventId, type: 'group_unsubscribe', email })]
|
|
317
303
|
);
|
|
318
304
|
assert.isSuccess(response2);
|
|
319
|
-
assert.propertyEquals(response2, 'data.processed',
|
|
320
|
-
assert.propertyEquals(response2, 'data.skipped', 1, 'Duplicate should be skipped');
|
|
305
|
+
assert.propertyEquals(response2, 'data.processed', 1, 'Re-delivery reprocesses (idempotent), not skipped');
|
|
321
306
|
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
assert.equal(webhookDoc?.status, 'completed');
|
|
307
|
+
const userDoc = await firestore.get(`users/${uid}`);
|
|
308
|
+
assert.equal(userDoc?.consent?.marketing?.status, 'revoked', 'User remains revoked after re-delivery');
|
|
325
309
|
},
|
|
326
310
|
},
|
|
327
311
|
|
|
328
|
-
// ─── Missing event ID ───
|
|
312
|
+
// ─── Missing event ID — still processed (no dedup requirement) ───
|
|
329
313
|
|
|
330
314
|
{
|
|
331
|
-
name: 'sendgrid-event-without-eventId-
|
|
315
|
+
name: 'sendgrid-event-without-eventId-processed',
|
|
332
316
|
auth: 'none',
|
|
333
|
-
async run({ http, assert, accounts }) {
|
|
317
|
+
async run({ http, firestore, assert, accounts }) {
|
|
318
|
+
const uid = accounts.basic.uid;
|
|
334
319
|
const email = accounts.basic.email;
|
|
335
320
|
|
|
336
321
|
const response = await http.as('none').post(
|
|
@@ -339,8 +324,10 @@ module.exports = {
|
|
|
339
324
|
);
|
|
340
325
|
|
|
341
326
|
assert.isSuccess(response, 'Should accept the request');
|
|
342
|
-
assert.propertyEquals(response, 'data.processed',
|
|
343
|
-
|
|
327
|
+
assert.propertyEquals(response, 'data.processed', 1, 'Event without eventId is still processed (no dedup needed)');
|
|
328
|
+
|
|
329
|
+
const userDoc = await firestore.get(`users/${uid}`);
|
|
330
|
+
assert.equal(userDoc?.consent?.marketing?.status, 'revoked', 'User should be revoked');
|
|
344
331
|
},
|
|
345
332
|
},
|
|
346
333
|
|
|
@@ -351,14 +338,16 @@ module.exports = {
|
|
|
351
338
|
{
|
|
352
339
|
name: 'beehiiv-subscription-unsubscribed-writes-consent',
|
|
353
340
|
auth: 'none',
|
|
354
|
-
async run({ http, firestore, assert, accounts, config }) {
|
|
341
|
+
async run({ http, firestore, assert, accounts, config, skip }) {
|
|
355
342
|
const uid = accounts.basic.uid;
|
|
356
343
|
const email = accounts.basic.email;
|
|
357
344
|
const eventId = `_test-bh-unsub-${Date.now()}`;
|
|
358
345
|
const eventISO = new Date().toISOString();
|
|
359
346
|
const publicationId = config.marketing?.beehiiv?.publicationId;
|
|
360
347
|
|
|
361
|
-
|
|
348
|
+
if (!publicationId) {
|
|
349
|
+
return skip('No Beehiiv publication ID configured for this brand');
|
|
350
|
+
}
|
|
362
351
|
|
|
363
352
|
const response = await http.as('none').post(
|
|
364
353
|
`marketing/webhook?provider=beehiiv&key=${process.env.BACKEND_MANAGER_WEBHOOK_KEY}`,
|
|
@@ -378,12 +367,6 @@ module.exports = {
|
|
|
378
367
|
assert.equal(userDoc?.consent?.marketing?.status, 'revoked', 'marketing.status should be revoked');
|
|
379
368
|
assert.equal(userDoc?.consent?.marketing?.revokedAt?.source, 'beehiiv', 'revokedAt.source should be beehiiv');
|
|
380
369
|
assert.ok(userDoc?.consent?.marketing?.revokedAt?.timestamp, 'revokedAt.timestamp should be set');
|
|
381
|
-
|
|
382
|
-
// Idempotency doc should exist
|
|
383
|
-
const webhookDoc = await firestore.get(`marketing-webhooks/${eventId}`);
|
|
384
|
-
assert.ok(webhookDoc, 'Idempotency doc should exist');
|
|
385
|
-
assert.equal(webhookDoc?.status, 'completed');
|
|
386
|
-
assert.equal(webhookDoc?.provider, 'beehiiv');
|
|
387
370
|
},
|
|
388
371
|
},
|
|
389
372
|
|
|
@@ -472,8 +455,8 @@ module.exports = {
|
|
|
472
455
|
);
|
|
473
456
|
|
|
474
457
|
// The dispatcher counts this as 'processed' from its POV (the handler
|
|
475
|
-
// ran without error
|
|
476
|
-
//
|
|
458
|
+
// ran without error), but the handler returned
|
|
459
|
+
// { handled: false, reason: 'publication-mismatch' }.
|
|
477
460
|
// What matters: the user doc should NOT have been mutated.
|
|
478
461
|
assert.isSuccess(response, 'Pub-mismatch event should be accepted gracefully');
|
|
479
462
|
|
|
@@ -542,13 +525,18 @@ module.exports = {
|
|
|
542
525
|
},
|
|
543
526
|
|
|
544
527
|
{
|
|
545
|
-
name: 'beehiiv-duplicate-event-
|
|
528
|
+
name: 'beehiiv-duplicate-event-reprocessed-idempotently',
|
|
546
529
|
auth: 'none',
|
|
547
|
-
async run({ http, firestore, assert, accounts, config }) {
|
|
530
|
+
async run({ http, firestore, assert, accounts, config, skip }) {
|
|
531
|
+
const uid = accounts.basic.uid;
|
|
548
532
|
const email = accounts.basic.email;
|
|
549
533
|
const publicationId = config.marketing?.beehiiv?.publicationId;
|
|
550
534
|
const eventId = `_test-bh-dup-${Date.now()}`;
|
|
551
535
|
|
|
536
|
+
if (!publicationId) {
|
|
537
|
+
return skip('No Beehiiv publication ID configured for this brand');
|
|
538
|
+
}
|
|
539
|
+
|
|
552
540
|
const payload = {
|
|
553
541
|
id: eventId,
|
|
554
542
|
event: 'subscription.unsubscribed',
|
|
@@ -565,17 +553,17 @@ module.exports = {
|
|
|
565
553
|
assert.isSuccess(r1);
|
|
566
554
|
assert.propertyEquals(r1, 'data.processed', 1, 'First delivery should process');
|
|
567
555
|
|
|
568
|
-
// Second delivery — same id
|
|
556
|
+
// Second delivery — same id. No dedup ledger, so it reprocesses; the
|
|
557
|
+
// revoke is idempotent so the end state is unchanged.
|
|
569
558
|
const r2 = await http.as('none').post(
|
|
570
559
|
`marketing/webhook?provider=beehiiv&key=${process.env.BACKEND_MANAGER_WEBHOOK_KEY}`,
|
|
571
560
|
payload
|
|
572
561
|
);
|
|
573
562
|
assert.isSuccess(r2);
|
|
574
|
-
assert.propertyEquals(r2, 'data.processed',
|
|
575
|
-
assert.propertyEquals(r2, 'data.skipped', 1, 'Duplicate should be skipped');
|
|
563
|
+
assert.propertyEquals(r2, 'data.processed', 1, 'Re-delivery reprocesses (idempotent), not skipped');
|
|
576
564
|
|
|
577
|
-
const
|
|
578
|
-
assert.equal(
|
|
565
|
+
const userDoc = await firestore.get(`users/${uid}`);
|
|
566
|
+
assert.equal(userDoc?.consent?.marketing?.status, 'revoked', 'User remains revoked after re-delivery');
|
|
579
567
|
},
|
|
580
568
|
},
|
|
581
569
|
],
|