@sdflc/backend-helpers 1.0.140 → 1.0.142
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +421 -119
- package/build/EmergencyAlerter.d.ts +35 -0
- package/build/EmergencyAlerter.js +134 -7
- package/build/esm/EmergencyAlerter.d.ts +35 -0
- package/build/esm/EmergencyAlerter.js +132 -6
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# @sdflc/backend-helpers
|
|
2
2
|
|
|
3
|
-
A comprehensive backend framework for Node.js applications providing database access, authentication, caching, IP-based rate limiting, GraphQL resolver helpers, REST controllers, CLI utilities, and
|
|
3
|
+
A comprehensive backend framework for Node.js applications providing database access, authentication, caching, IP-based rate limiting, GraphQL resolver helpers, REST controllers, CLI utilities, migration tools, ready-made gateways to outside services (SendGrid, Telegram, Twilio, Slack, Discord, signed HTTP webhooks), and emergency alerts a service raises by itself.
|
|
4
4
|
|
|
5
5
|
---
|
|
6
6
|
|
|
@@ -27,13 +27,26 @@ A comprehensive backend framework for Node.js applications providing database ac
|
|
|
27
27
|
- [BaseRestApiCrudController](#baserestapicrudcontroller)
|
|
28
28
|
- [BaseExternalApiGateway](#baseexternalapigateway)
|
|
29
29
|
- [Gateways to outside services](#gateways-to-outside-services)
|
|
30
|
+
- [SendGridGw](#sendgridgw)
|
|
31
|
+
- [TelegramGw](#telegramgw)
|
|
32
|
+
- [TwilioGw](#twiliogw)
|
|
33
|
+
- [SlackWebhookGw](#slackwebhookgw)
|
|
34
|
+
- [DiscordWebhookGw](#discordwebhookgw)
|
|
35
|
+
- [HttpWebhookGw](#httpwebhookgw)
|
|
36
|
+
- [Outbound URL safety](#outbound-url-safety)
|
|
37
|
+
- [Testing a gateway](#testing-a-gateway)
|
|
38
|
+
- [Emergency alerts](#emergency-alerts)
|
|
39
|
+
- [Wiring it into a service](#wiring-it-into-a-service)
|
|
40
|
+
- [The settings fromConfig reads](#the-settings-fromconfig-reads)
|
|
41
|
+
- [Building it yourself](#building-it-yourself)
|
|
42
|
+
- [What it answers](#what-it-answers)
|
|
43
|
+
- [The rules it keeps](#the-rules-it-keeps)
|
|
30
44
|
- [IP rate limiting](#ip-rate-limiting)
|
|
31
45
|
- [How it works](#how-it-works)
|
|
32
46
|
- [Setup](#setup)
|
|
33
47
|
- [Global tier](#global-tier)
|
|
34
48
|
- [Core-level rules](#core-level-rules)
|
|
35
49
|
- [Action-level rules](#action-level-rules)
|
|
36
|
-
- [Per-call override](#per-call-override)
|
|
37
50
|
- [Disabling rate limiting](#disabling-rate-limiting)
|
|
38
51
|
- [Resetting a counter after success](#resetting-a-counter-after-success)
|
|
39
52
|
- [Custom limiter](#custom-limiter)
|
|
@@ -57,7 +70,6 @@ A comprehensive backend framework for Node.js applications providing database ac
|
|
|
57
70
|
- [Utility functions](#utility-functions)
|
|
58
71
|
- [downloadFile](#downloadfile)
|
|
59
72
|
- [passwordHelpers](#passwordhelpers)
|
|
60
|
-
- [Emergency alerts](#emergency-alerts)
|
|
61
73
|
- [Constants](#constants)
|
|
62
74
|
- [Interfaces reference](#interfaces-reference)
|
|
63
75
|
|
|
@@ -73,7 +85,7 @@ npm install @sdflc/backend-helpers
|
|
|
73
85
|
|
|
74
86
|
## Requirements
|
|
75
87
|
|
|
76
|
-
- Node.js >= 22
|
|
88
|
+
- Node.js >= 22. The gateways use the built-in `fetch`, `AbortSignal.timeout` and `net.BlockList`
|
|
77
89
|
- TypeScript >= 5.7
|
|
78
90
|
- PostgreSQL or MSSQL (via Knex)
|
|
79
91
|
|
|
@@ -93,17 +105,21 @@ Business layer → Core classes (BaseCore subclasses)
|
|
|
93
105
|
validation, IP rate limiting, enforce business rules,
|
|
94
106
|
and normalise responses into a consistent OpResult shape.
|
|
95
107
|
|
|
96
|
-
Data layer → Gateway classes
|
|
97
|
-
|
|
98
|
-
queries with joins and filters
|
|
99
|
-
|
|
108
|
+
Data layer → Gateway classes
|
|
109
|
+
BaseGateway subclasses perform CRUD operations on database
|
|
110
|
+
tables and run complex queries with joins and filters.
|
|
111
|
+
BaseExternalApiGateway subclasses talk to outside services
|
|
112
|
+
(email, messaging, webhooks, payment providers).
|
|
100
113
|
```
|
|
101
114
|
|
|
102
115
|
Each inbound request — whether from an HTTP client, a GraphQL operation, or a CLI command — creates a fresh **ApiContext** that carries the database connection, all gateway instances, all core instances, and request-scoped data (userId, spaceId, headers, etc.).
|
|
103
116
|
|
|
104
117
|
**Key invariants:**
|
|
105
118
|
|
|
106
|
-
-
|
|
119
|
+
- Database gateways (`BaseGateway`) and cores are instantiated **per-request** and must never be reused across requests
|
|
120
|
+
- Gateways to outside services (`BaseExternalApiGateway`) hold no request state: create one once (or per request, if it
|
|
121
|
+
sits in your gateway barrel) and `configure()` it whenever its settings change. `configure` **replaces** the settings,
|
|
122
|
+
so nothing of one tenant carries over to the next
|
|
107
123
|
- The data layer (gateways) has no knowledge of HTTP, GraphQL, or authentication — it only handles data
|
|
108
124
|
- The business layer (cores) has no knowledge of transport details — it receives plain args and returns OpResults
|
|
109
125
|
- The application layer (controllers/resolvers/CLI) has no direct database access — it goes through cores
|
|
@@ -1163,58 +1179,382 @@ acme.configure({ apiKey: await keyOf(spaceId) }); // per tenant, whenever
|
|
|
1163
1179
|
const result = await acme.ping();
|
|
1164
1180
|
```
|
|
1165
1181
|
|
|
1166
|
-
|
|
1167
|
-
|
|
1182
|
+
What a subclass overrides, and what it uses:
|
|
1183
|
+
|
|
1184
|
+
| Member | Purpose |
|
|
1185
|
+
|---|---|
|
|
1186
|
+
| `validateSettings(settings)` | Override. What's missing or malformed, in words (never a value). Drives `isConfigured()` / `getConfigProblems()` |
|
|
1187
|
+
| `secretValues()` | Override. The values scrubbed out of everything the gateway returns (keys, tokens, secret URLs) |
|
|
1188
|
+
| `errorMessageOf(reply)` | Override. The refusal's words, from the service's own error shape |
|
|
1189
|
+
| `shouldRetry(result)` | Override. Refuse hopeless retries (default: network, timeout, 429, 5xx) |
|
|
1190
|
+
| `retryAfterMs(result)` | Override. How long the service asked to wait (default: the `Retry-After` header) |
|
|
1191
|
+
| `onConfigured()` | Override. Called after every `configure` (to rebuild an SDK client) |
|
|
1192
|
+
| `settings` | Use. The current settings (they hold secrets: never log or return them) |
|
|
1193
|
+
| `request(req)` / `requestWithRetries(req)` | Use. One HTTP attempt, or attempts with retries. `req`: `{ url, method?, headers?, json?, body?, timeoutMs?, secrets? }`; `secrets` are per-request values to scrub |
|
|
1194
|
+
| `notConfigured()` | Use. The standard refusal while the gateway isn't set up |
|
|
1195
|
+
| `scrub(text, extra?)` / `scrubValue(value, extra?)` | Use. Replace every secret with `***` |
|
|
1196
|
+
|
|
1197
|
+
ⓘ `setCredentials()` / `onCredentialsSet()` still work for gateways written before `configure()`.
|
|
1168
1198
|
|
|
1169
1199
|
---
|
|
1170
1200
|
|
|
1171
1201
|
## Gateways to outside services
|
|
1172
1202
|
|
|
1173
|
-
Ready-made gateways on `BaseExternalApiGateway`
|
|
1203
|
+
Ready-made gateways on [`BaseExternalApiGateway`](#baseexternalapigateway): no SDKs, one REST call each, an `OpResult`
|
|
1204
|
+
out, never a throw. Create one **once**, then `configure()` it whenever the settings change (per brand, per tenant, per
|
|
1205
|
+
request). `configure()` **replaces** the settings; it never merges them.
|
|
1206
|
+
|
|
1207
|
+
| Gateway | `configure({...})` | Operation | OK data |
|
|
1208
|
+
|---|---|---|---|
|
|
1209
|
+
| [`SendGridGw`](#sendgridgw) | `apiKey`, `fromEmail?`, `fromName?` | `sendEmail(email)` | `{ messageId }` |
|
|
1210
|
+
| [`TelegramGw`](#telegramgw) | `botToken`, `chatId?`, `parseMode?` | `sendMessage(message)` | `{ messageId }` |
|
|
1211
|
+
| [`TwilioGw`](#twiliogw) | `accountSid`, `authToken`, `from?`, `messagingServiceSid?` | `sendMessage(message)` | `{ sid, status }` |
|
|
1212
|
+
| [`SlackWebhookGw`](#slackwebhookgw) | `webhookUrl` | `postMessage(message)` | `{ messageId: null }` |
|
|
1213
|
+
| [`DiscordWebhookGw`](#discordwebhookgw) | `webhookUrl`, `username?`, `avatarUrl?` | `postMessage(message)` | `{ messageId }` |
|
|
1214
|
+
| [`HttpWebhookGw`](#httpwebhookgw) | `url`, `method?`, `headers?`, `signingSecret?` | `send(request)` | `{ status, body }` |
|
|
1174
1215
|
|
|
1175
|
-
|
|
1216
|
+
Every operation answers the same way:
|
|
1217
|
+
|
|
1218
|
+
| Outcome | `OpResult` code | Retried? |
|
|
1176
1219
|
|---|---|---|
|
|
1177
|
-
|
|
|
1178
|
-
|
|
|
1179
|
-
|
|
|
1180
|
-
|
|
|
1181
|
-
|
|
|
1182
|
-
|
|
|
1220
|
+
| Delivered | `OK`, with the data above | — |
|
|
1221
|
+
| Not configured, or the message itself is wrong (no recipient, too long, a bad number…) | `VALIDATION_FAILED` (nothing is sent) | no |
|
|
1222
|
+
| The service refused it | the code for its HTTP status: `UNAUTHORIZED` 401, `FORBIDDEN` 403, `NOT_FOUND` 404, `CONFLICT` 409, `VALIDATION_FAILED` other 4xx | no |
|
|
1223
|
+
| Throttled | `TOO_MANY_REQUESTS` 429 | yes, after its `Retry-After` |
|
|
1224
|
+
| The service failed | `FAILED` 5xx | yes |
|
|
1225
|
+
| No answer | `NETWORK_ERROR`, or `TIMEOUT` (also 408) | yes (email and SMS: not after a timeout) |
|
|
1226
|
+
|
|
1227
|
+
The message is the service's own words (`result.getErrorSummary('')`). The data carries `{ status, headers, body }` of
|
|
1228
|
+
its reply, scrubbed of every secret.
|
|
1183
1229
|
|
|
1184
1230
|
```typescript
|
|
1185
|
-
import { SendGridGw
|
|
1231
|
+
import { SendGridGw } from '@sdflc/backend-helpers';
|
|
1232
|
+
|
|
1233
|
+
const mail = new SendGridGw(); // once
|
|
1234
|
+
mail.configure({ apiKey: brand.sendgridKey, fromEmail: brand.fromEmail, fromName: brand.name }); // per brand, whenever
|
|
1186
1235
|
|
|
1187
|
-
const mail = new SendGridGw().configure({ apiKey, fromEmail: 'hello@brand.com', fromName: 'Brand' });
|
|
1188
1236
|
const sent = await mail.sendEmail({ to: 'guest@example.com', subject: 'Welcome', html: '<p>Hi</p>' });
|
|
1189
1237
|
|
|
1190
1238
|
if (sent.didFail()) {
|
|
1191
|
-
logger.error(sent.getErrorSummary(''));
|
|
1239
|
+
logger.error(`welcome email: ${sent.getErrorSummary('')}`); // the service's words, no key in them
|
|
1192
1240
|
}
|
|
1193
1241
|
```
|
|
1194
1242
|
|
|
1195
|
-
|
|
1196
|
-
|
|
1197
|
-
|
|
1198
|
-
|
|
1199
|
-
|
|
1200
|
-
|
|
1201
|
-
|
|
1202
|
-
|
|
1203
|
-
|
|
1204
|
-
|
|
1205
|
-
|
|
1206
|
-
|
|
1207
|
-
|
|
1208
|
-
|
|
1209
|
-
|
|
1210
|
-
|
|
1211
|
-
|
|
1212
|
-
|
|
1213
|
-
|
|
1214
|
-
|
|
1215
|
-
|
|
1216
|
-
|
|
1217
|
-
|
|
1243
|
+
### SendGridGw
|
|
1244
|
+
|
|
1245
|
+
Email through SendGrid's v3 Mail Send API: plain text, HTML, or a dynamic template.
|
|
1246
|
+
|
|
1247
|
+
```typescript
|
|
1248
|
+
const mail = new SendGridGw().configure({ apiKey, fromEmail: 'hello@brand.com', fromName: 'Brand' });
|
|
1249
|
+
|
|
1250
|
+
await mail.sendEmail({
|
|
1251
|
+
to: ['a@example.com', { email: 'b@example.com', name: 'B' }], // a SendGridAddress, or several
|
|
1252
|
+
cc: 'c@example.com',
|
|
1253
|
+
replyTo: 'support@brand.com',
|
|
1254
|
+
subject: 'Your ticket',
|
|
1255
|
+
text: 'Plain version',
|
|
1256
|
+
html: '<p>Rich version</p>',
|
|
1257
|
+
categories: ['tickets'],
|
|
1258
|
+
customArgs: { orderId: 'o-1' },
|
|
1259
|
+
});
|
|
1260
|
+
|
|
1261
|
+
await mail.sendEmail({ to: 'a@example.com', templateId: 'd-123', dynamicTemplateData: { name: 'A' } }); // no subject needed
|
|
1262
|
+
```
|
|
1263
|
+
|
|
1264
|
+
- The From (`from`, else the configured `fromEmail`) must be on a domain authenticated in that SendGrid account;
|
|
1265
|
+
SendGrid refuses anything else.
|
|
1266
|
+
- A malformed recipient is **refused**, never silently dropped.
|
|
1267
|
+
- Not retried after a timeout: SendGrid may well have accepted the mail, and a retry would send it twice.
|
|
1268
|
+
|
|
1269
|
+
### TelegramGw
|
|
1270
|
+
|
|
1271
|
+
Messages through the Telegram Bot API.
|
|
1272
|
+
|
|
1273
|
+
```typescript
|
|
1274
|
+
const telegram = new TelegramGw().configure({ botToken: '123456:ABC…', chatId: '-1001234567890' });
|
|
1275
|
+
|
|
1276
|
+
await telegram.sendMessage({ text: '<b>Deployed</b> v1.2' }); // HTML by default
|
|
1277
|
+
await telegram.sendMessage({ text: 'plain', parseMode: null, chatId: '@ops_channel' }); // another chat, no markup
|
|
1278
|
+
await telegram.sendMessage({ text: 'Open', inlineKeyboard: [[{ text: 'Dashboard', url: 'https://…' }]] });
|
|
1279
|
+
```
|
|
1280
|
+
|
|
1281
|
+
- A chat is a numeric id (negative for a group) or `@channelname`; `isTelegramChatId(value)` checks one.
|
|
1282
|
+
- A text over 4096 characters is **refused, not cut**. It's counted the way Telegram counts it (`telegramVisibleLength`:
|
|
1283
|
+
tags are free, an entity is one character). Cutting finished HTML can split an entity, and Telegram then refuses the
|
|
1284
|
+
whole message, so cut your text **before** adding markup.
|
|
1285
|
+
- A 200 with `ok: false` is a failure. A permanent refusal (chat not found, bot blocked, broken markup) is not retried.
|
|
1286
|
+
Telegram's own `retry_after` is honoured.
|
|
1287
|
+
- The bot token sits in the request URL. It is scrubbed out of everything returned.
|
|
1288
|
+
|
|
1289
|
+
### TwilioGw
|
|
1290
|
+
|
|
1291
|
+
SMS, or WhatsApp with `whatsapp:+…` addresses, through Twilio's Messages API.
|
|
1292
|
+
|
|
1293
|
+
```typescript
|
|
1294
|
+
const twilio = new TwilioGw().configure({ accountSid: 'AC…', authToken, from: '+15551234567' });
|
|
1295
|
+
// or configure({ accountSid, authToken, messagingServiceSid: 'MG…' }) to send through a messaging service
|
|
1296
|
+
|
|
1297
|
+
const result = await twilio.sendMessage({ to: '+15557654321', body: 'Your code is 123456', statusCallback: 'https://…' });
|
|
1298
|
+
// OK → [{ sid: 'SM…', status: 'queued' }]
|
|
1299
|
+
```
|
|
1300
|
+
|
|
1301
|
+
- Numbers must be E.164. A body over 1600 characters is refused before anything is sent.
|
|
1302
|
+
- Not retried after a timeout (it may have gone out, and a retry texts the person twice).
|
|
1303
|
+
- The auth token (and the Basic credentials built from it) is scrubbed out of everything returned.
|
|
1304
|
+
|
|
1305
|
+
### SlackWebhookGw
|
|
1306
|
+
|
|
1307
|
+
Posts to a Slack channel through an incoming webhook.
|
|
1308
|
+
|
|
1309
|
+
```typescript
|
|
1310
|
+
const slack = new SlackWebhookGw().configure({ webhookUrl: 'https://hooks.slack.com/services/…' });
|
|
1311
|
+
|
|
1312
|
+
await slack.postMessage({ text: 'Deployed', blocks: [/* Block Kit */] });
|
|
1313
|
+
```
|
|
1314
|
+
|
|
1315
|
+
- The webhook URL **is** the credential; it is scrubbed out of everything returned. Only `https://hooks.slack.com/…` is
|
|
1316
|
+
accepted.
|
|
1317
|
+
- Slack's webhook answers no id, so the OK data is `{ messageId: null }`, the same shape as the others.
|
|
1318
|
+
|
|
1319
|
+
### DiscordWebhookGw
|
|
1320
|
+
|
|
1321
|
+
Posts to a Discord channel through a webhook.
|
|
1322
|
+
|
|
1323
|
+
```typescript
|
|
1324
|
+
const discord = new DiscordWebhookGw().configure({ webhookUrl: 'https://discord.com/api/webhooks/…', username: 'Alerts' });
|
|
1325
|
+
|
|
1326
|
+
await discord.postMessage({ content: 'Deployed', embeds: [{ title: 'v1.2', color: 0x5865f2 }] });
|
|
1327
|
+
```
|
|
1328
|
+
|
|
1329
|
+
- Content is at most 2000 characters, with at most 10 embeds.
|
|
1330
|
+
- **Nobody is pinged** unless `allowedMentions` says so: `@everyone` in a text stays text.
|
|
1331
|
+
- An unknown webhook or a bad webhook token is not retried. The webhook URL is scrubbed out of everything returned.
|
|
1332
|
+
|
|
1333
|
+
### HttpWebhookGw
|
|
1334
|
+
|
|
1335
|
+
Sends JSON to any URL, for anything without a gateway of its own, optionally **signed**.
|
|
1336
|
+
|
|
1337
|
+
```typescript
|
|
1338
|
+
const hook = new HttpWebhookGw(); // once: the address protection is on
|
|
1339
|
+
hook.configure({ url: tenant.webhookUrl, headers: { 'X-Api-Key': tenant.apiKey }, signingSecret: tenant.signingSecret });
|
|
1340
|
+
|
|
1341
|
+
const result = await hook.send({ payload: { event: 'order.paid', id }, headers: { 'X-Trace': traceId } });
|
|
1342
|
+
// OK → [{ status: 200, body: … }] (the receiver's answer, scrubbed)
|
|
1343
|
+
```
|
|
1344
|
+
|
|
1345
|
+
- **The URL a user typed must not reach your own network** (see [Outbound URL safety](#outbound-url-safety)). It is
|
|
1346
|
+
checked before the request, and the address the socket actually connects to is checked again at connect time.
|
|
1347
|
+
- `allowPrivateNetwork: true` is a **constructor** option, never a setting: `new HttpWebhookGw({ allowPrivateNetwork: true })`.
|
|
1348
|
+
Use it only for a URL **you** configure (an internal service). Settings often come from a tenant's own row, and a
|
|
1349
|
+
tenant must not be able to switch the protection off.
|
|
1350
|
+
- Every send carries `X-Webhook-Id`, which stays the same on its retries so a receiver can drop a duplicate.
|
|
1351
|
+
- With `signingSecret` it also carries `X-Webhook-Timestamp` (unix seconds) and
|
|
1352
|
+
`X-Webhook-Signature: sha256=<hex HMAC-SHA256 of "<timestamp>.<body>">`. A configured or per-request header can never
|
|
1353
|
+
replace these.
|
|
1354
|
+
- A header with a line break, or a payload that can't be sent as JSON (a function), is refused.
|
|
1355
|
+
- Every header value is scrubbed out of everything returned, configured or per-request, whole and word by word.
|
|
1356
|
+
|
|
1357
|
+
A receiver checks a signed request like this:
|
|
1358
|
+
|
|
1359
|
+
```typescript
|
|
1360
|
+
import { timingSafeEqual } from 'crypto';
|
|
1361
|
+
import { webhookSignatureOf } from '@sdflc/backend-helpers';
|
|
1362
|
+
|
|
1363
|
+
const timestamp = Number(req.headers['x-webhook-timestamp']);
|
|
1364
|
+
const expected = Buffer.from(webhookSignatureOf(secret, timestamp, rawBody)); // the RAW body, before JSON parsing
|
|
1365
|
+
const given = Buffer.from(String(req.headers['x-webhook-signature'] ?? ''));
|
|
1366
|
+
const fresh = Math.abs(Date.now() / 1000 - timestamp) < 300;
|
|
1367
|
+
const genuine = fresh && expected.length === given.length && timingSafeEqual(expected, given);
|
|
1368
|
+
```
|
|
1369
|
+
|
|
1370
|
+
### Outbound URL safety
|
|
1371
|
+
|
|
1372
|
+
For any code that sends a request to a URL a **user** gave (webhooks, callbacks, "import from URL"):
|
|
1373
|
+
|
|
1374
|
+
- **`checkExternalUrl(url, { protocols?, lookup?, field? })`**: `OK` when the URL may be requested, else
|
|
1375
|
+
`VALIDATION_FAILED` saying why. It refuses:
|
|
1376
|
+
- anything but http(s);
|
|
1377
|
+
- a user name or password in the URL;
|
|
1378
|
+
- `localhost`, `*.localhost` and `*.internal` names, and the cloud metadata host names;
|
|
1379
|
+
- any host that is, or resolves to, a blocked address. **Every** address it resolves to is checked: a name with one
|
|
1380
|
+
private address among public ones is refused.
|
|
1381
|
+
- **`isBlockedAddress(ip)`**: true for the addresses a user-given URL must never reach, however they're written. IPv4:
|
|
1382
|
+
this-network, private, carrier-grade NAT, loopback, link-local (169.254.169.254, the metadata service), protocol
|
|
1383
|
+
assignments, test networks, the 6to4 relay, benchmarking, multicast, reserved. IPv6: unspecified, loopback,
|
|
1384
|
+
IPv4-compatible, IPv4-mapped (checked as IPv4), NAT64 (both prefixes), 6to4, discard-only, documentation,
|
|
1385
|
+
unique-local `fc00::/7`, link-local, multicast.
|
|
1386
|
+
- **`createGuardedFetch({ dnsLookup?, isBlocked? })`**: a `fetch` that checks the address **at the moment the socket
|
|
1387
|
+
connects**. A check made only beforehand loses to DNS rebinding (a server that answers a public address first and an
|
|
1388
|
+
internal one to the second lookup). It also checks an IP written in the URL directly, never follows redirects, reads
|
|
1389
|
+
at most 64 KB, and ends at its signal. `HttpWebhookGw` uses it by default.
|
|
1390
|
+
|
|
1391
|
+
### Testing a gateway
|
|
1392
|
+
|
|
1393
|
+
Every gateway takes a `fetch` in its constructor props, so a test runs the real gateway against a stub and nothing leaves
|
|
1394
|
+
the machine:
|
|
1395
|
+
|
|
1396
|
+
```typescript
|
|
1397
|
+
const calls: any[] = [];
|
|
1398
|
+
const fetch = async (url: string, init: any) => {
|
|
1399
|
+
calls.push({ url, body: JSON.parse(init.body) });
|
|
1400
|
+
|
|
1401
|
+
return { status: 202, headers: { forEach: () => null }, text: async () => '' };
|
|
1402
|
+
};
|
|
1403
|
+
|
|
1404
|
+
const mail = new SendGridGw({ fetch, retries: 0 }).configure({ apiKey: 'SG.test' });
|
|
1405
|
+
const result = await mail.sendEmail({ to: 'a@b.test', from: 'x@b.test', subject: 's', text: 't' });
|
|
1406
|
+
// result.didSucceed() === true; calls[0].body.personalizations[0].to[0].email === 'a@b.test'
|
|
1407
|
+
```
|
|
1408
|
+
|
|
1409
|
+
`HttpWebhookGw` also takes `lookup` (the pre-check's resolver) and `dnsLookup` (the connect-time resolver).
|
|
1410
|
+
|
|
1411
|
+
---
|
|
1412
|
+
|
|
1413
|
+
## Emergency alerts
|
|
1414
|
+
|
|
1415
|
+
A service raises an alarm **by itself**, by email and/or Telegram, when it starts with missing configuration or
|
|
1416
|
+
crashes. `EmergencyAlerter` decides **what** is said, **to whom** and **how often**. It sends through the gateways above
|
|
1417
|
+
(`SendGridGw`, `TelegramGw`) and contains no transport code itself. Each alert names the app, the service and the
|
|
1418
|
+
environment (`[partylabz · ms_storage · production]`), because several apps can run the same services side by side on
|
|
1419
|
+
one machine.
|
|
1420
|
+
|
|
1421
|
+
### Wiring it into a service
|
|
1422
|
+
|
|
1423
|
+
```typescript
|
|
1424
|
+
// src/index.ts
|
|
1425
|
+
import { EmergencyAlerter, CriticalConfigItem } from '@sdflc/backend-helpers';
|
|
1426
|
+
|
|
1427
|
+
// What this service starts WITHOUT (optional to the config loader) but cannot fully work without.
|
|
1428
|
+
// APP_NAME is always checked as well. `key` (the built-config name) is derived from `env`; pass it only when it differs.
|
|
1429
|
+
const CRITICAL_CONFIG: CriticalConfigItem[] = [
|
|
1430
|
+
{ env: 'INTERSERVICE_API_KEY', effect: 'Every call from our other services is refused.' },
|
|
1431
|
+
];
|
|
1432
|
+
|
|
1433
|
+
const alerter = EmergencyAlerter.fromConfig(config, 'ms_storage', { log: (m) => logger.error(m) });
|
|
1434
|
+
|
|
1435
|
+
alerter.installCrashHandlers(process); // uncaught exception / unhandled rejection → alert, then exit(1)
|
|
1436
|
+
|
|
1437
|
+
// The whole start. `start` builds the app and answers the HTTP server to listen on (for Express + Apollo, the
|
|
1438
|
+
// http.Server configureApp creates — listening on it is what makes Apollo's drain plugin work).
|
|
1439
|
+
void alerter.startService({
|
|
1440
|
+
start: () => configureApp(express()),
|
|
1441
|
+
port: config.port,
|
|
1442
|
+
config, // checked AFTER the server listens: the service still starts, and says loudly what is missing
|
|
1443
|
+
critical: CRITICAL_CONFIG,
|
|
1444
|
+
afterListen: () => {
|
|
1445
|
+
logger.log(`Server started on port ${config.port}`);
|
|
1446
|
+
startBackgroundJobs();
|
|
1447
|
+
},
|
|
1448
|
+
});
|
|
1449
|
+
```
|
|
1450
|
+
|
|
1451
|
+
`startService(opts)` answers `[{ status: 'listening', port }]`, and never throws:
|
|
1452
|
+
|
|
1453
|
+
- **A start that fails** — `start` throws, the port is in use, the port is not a port — is alerted (keyed
|
|
1454
|
+
`crash:startup:<fingerprint>`, so one failure does not silence a different one), then `exit(1)`.
|
|
1455
|
+
- **A start that hangs** is alerted once after `readyTimeoutMs` (default `START_READY_TIMEOUT_MS`, a minute) and keeps
|
|
1456
|
+
waiting; when it listens at last, that is said too. ⚠ This is the common one: Redis's client retries a refused
|
|
1457
|
+
connection for ever, so an unreachable Redis (or a wrong `REDIS_PASSWORD`) leaves the port closed with nothing but a
|
|
1458
|
+
log line every half second.
|
|
1459
|
+
- **`afterListen` failing** is alerted, never fatal — the service is already answering.
|
|
1460
|
+
- Options: `start`, `port`, `config?` + `critical?` (omit `config` to skip the check), `afterListen?`, `readyTimeoutMs?`,
|
|
1461
|
+
`exit?` (default `process.exit`).
|
|
1462
|
+
|
|
1463
|
+
Anything else worth waking someone for:
|
|
1464
|
+
|
|
1465
|
+
```typescript
|
|
1466
|
+
const result = await alerter.send({ key: 'queue:stalled', subject: 'The queue stopped draining', lines: ['…'] });
|
|
1467
|
+
```
|
|
1468
|
+
|
|
1469
|
+
### The settings fromConfig reads
|
|
1470
|
+
|
|
1471
|
+
From the built config (`buildConfig`). All of them are optional, and each channel needs all of its own. Leave them
|
|
1472
|
+
commented out in `.env.example` so a service without them still starts.
|
|
1473
|
+
|
|
1474
|
+
| Env | Meaning |
|
|
1475
|
+
|---|---|
|
|
1476
|
+
| `APP_NAME` | The app this service belongs to. Alerts say `unknown-app` without it, and `checkConfig` reports it as missing |
|
|
1477
|
+
| `EMERGENCY_SENDGRID_API_KEY` | Email: a SendGrid key allowed to send mail |
|
|
1478
|
+
| `EMERGENCY_EMAIL_FROM` | Email: a sender on a domain authenticated in that SendGrid account. Required: a guessed one is refused |
|
|
1479
|
+
| `EMERGENCY_EMAIL_TO` | Email: recipients, comma-separated |
|
|
1480
|
+
| `TELEGRAM_BOT_ID` | Telegram: the bot token `<id>:<secret>` from @BotFather |
|
|
1481
|
+
| `TELEGRAM_CHAT_ID` | Telegram: the chat id (negative for a group) or `@channel`. The bot must be in it |
|
|
1482
|
+
| `EMERGENCY_EMAIL_MIN_INTERVAL_MINUTES` | The same alert at most once per this many minutes, on every channel. Default 60; 0 = every time |
|
|
1483
|
+
|
|
1484
|
+
`fromConfig(config, serviceName, { log?, redact?, memory?, now?, hostname? })` gives each provider a 4 s timeout and one
|
|
1485
|
+
short retry, so both channels together fit the crash wait. It says once, through `log`, when a channel was set up but
|
|
1486
|
+
can't be used (for example a malformed token, or an email missing its From).
|
|
1487
|
+
|
|
1488
|
+
### Building it yourself
|
|
1489
|
+
|
|
1490
|
+
Credentials live in the gateways; who to alert lives in the alerter.
|
|
1491
|
+
|
|
1492
|
+
```typescript
|
|
1493
|
+
const alerter = new EmergencyAlerter({
|
|
1494
|
+
email: new SendGridGw({ timeoutMs: 4000, retries: 1 }).configure({ apiKey }),
|
|
1495
|
+
telegram: new TelegramGw({ timeoutMs: 4000, retries: 1 }).configure({ botToken }),
|
|
1496
|
+
settings: {
|
|
1497
|
+
appName: 'partylabz',
|
|
1498
|
+
serviceName: 'ms_storage',
|
|
1499
|
+
environment: 'production',
|
|
1500
|
+
minIntervalMinutes: 60,
|
|
1501
|
+
emailTo: ['ops@brand.com'],
|
|
1502
|
+
emailFrom: 'alerts@brand.com',
|
|
1503
|
+
telegramChatId: '-100123',
|
|
1504
|
+
},
|
|
1505
|
+
log: (m) => logger.error(m),
|
|
1506
|
+
redact: (text) => text.replace(/password=\S+/g, 'password=***'),
|
|
1507
|
+
});
|
|
1508
|
+
```
|
|
1509
|
+
|
|
1510
|
+
`configure(settings)` **replaces** the settings, and `useEmail(gateway | null)` / `useTelegram(gateway | null)` swap a
|
|
1511
|
+
channel's gateway.
|
|
1512
|
+
|
|
1513
|
+
### What it answers
|
|
1514
|
+
|
|
1515
|
+
Every method returns an `OpResult` and never throws.
|
|
1516
|
+
|
|
1517
|
+
| Call | OK | Not OK |
|
|
1518
|
+
|---|---|---|
|
|
1519
|
+
| `send(alert)` | `[{ status: 'sent', channels: { email, telegram } }]` (each `'sent'`, `'failed'` or `'off'`), or `[{ status: 'recently-sent' }]` | `VALIDATION_FAILED` `[{ status: 'unconfigured' }]` when no channel can be used; `FAILED` `[{ status: 'failed', channels }]` when no channel delivered it, with each channel's error merged in |
|
|
1520
|
+
| `checkConfig(config, items)` | `[]` when nothing is missing | otherwise whatever `send` answered for the missing-configuration alert |
|
|
1521
|
+
| `startService(opts)` | `[{ status: 'listening', port }]` | the start failed — alerted, then `exit(1)` — with the error |
|
|
1522
|
+
|
|
1523
|
+
Also available:
|
|
1524
|
+
- `getLabel()`: `app · service · environment`.
|
|
1525
|
+
- `getChannelProblems()`: why each channel can't be used, in words, never a value.
|
|
1526
|
+
- `getMinIntervalMs()`.
|
|
1527
|
+
- `emailOf(alert, host, at)` and `telegramTextOf(alert, host, at)`: exactly what would be sent.
|
|
1528
|
+
- The pure helpers `missingCriticalConfig(config, items)`, `missingConfigAlert(config, items)`, `emergencyOneLine(value, max)`,
|
|
1529
|
+
`emergencyIntervalText(ms)`, `emergencyFileMemory(app, service, dir?)` and `redactKnownSecrets(text)`.
|
|
1530
|
+
|
|
1531
|
+
### The rules it keeps
|
|
1532
|
+
|
|
1533
|
+
- **Never throws, never blocks for long.** Even a `log` or `redact` that throws can't make it throw. A crash waits for its
|
|
1534
|
+
alert at most 10 s, then exits, and **always** exits, whatever was thrown (even a value that can't be turned into
|
|
1535
|
+
text).
|
|
1536
|
+
- **One alert per key per interval, across all channels.** It's remembered in a file under the OS temp folder named after
|
|
1537
|
+
the app and service, so a crash loop sends one message rather than one per restart.
|
|
1538
|
+
- Sends of the same key **at the same time** go out once.
|
|
1539
|
+
- A send that no channel delivered is not remembered, so the next one tries again.
|
|
1540
|
+
- A remembered time in the future (a clock moved back) doesn't silence anything.
|
|
1541
|
+
- **A crash's key names which crash** (`crash:<kind>:<fingerprint of the message and top frame>`). The same crash in a
|
|
1542
|
+
loop is sent once, and a different crash is never silenced by it.
|
|
1543
|
+
- **A channel set up wrongly is said once, at the start**, never silently skipped.
|
|
1544
|
+
- **What leaves is redacted.** `fromConfig` removes every config value whose **name** looks secret (password, secret,
|
|
1545
|
+
token, key…) from every subject, line and log line, because an error message can carry SQL with its values. And
|
|
1546
|
+
EVERY alerter — under any custom `redact` — removes what LOOKS like a secret wherever it came from
|
|
1547
|
+
(`redactKnownSecrets`): SendGrid keys, Stripe secret/restricted keys and webhook secrets, Telegram bot tokens, JWTs,
|
|
1548
|
+
Azure `AccountKey=`/`SharedAccess…=`, `Bearer`/`Basic` values, credentials inside a URL, and `password=`/`token=`/
|
|
1549
|
+
`api_key=`-style pairs — a per-tenant secret read from a database row is known to no config. Pass `redact` for more.
|
|
1550
|
+
- **Subjects are one line**, because a crash's subject is the error's message, which often spans lines. The Telegram text
|
|
1551
|
+
is cut **before** it's escaped, to whatever the heading and footer leave, so a long stack trace neither splits an
|
|
1552
|
+
entity nor overruns Telegram's limit.
|
|
1553
|
+
- **It goes straight to the providers**, never through another service of yours. It has to work when the rest doesn't.
|
|
1554
|
+
- **Never put a secret, a token or a database row in an alert yourself.** It leaves your infrastructure.
|
|
1555
|
+
|
|
1556
|
+
Proofs: `npm run build && npm run prove`. They cover the gateways and the alerter, each against a stub `fetch`, plus a
|
|
1557
|
+
real local server for the connect-time address check.
|
|
1218
1558
|
|
|
1219
1559
|
---
|
|
1220
1560
|
|
|
@@ -2067,79 +2407,6 @@ const isCorrect = await checkPassword(plaintextPassword, storedHash);
|
|
|
2067
2407
|
|
|
2068
2408
|
---
|
|
2069
2409
|
|
|
2070
|
-
## Emergency alerts
|
|
2071
|
-
|
|
2072
|
-
A service raises an alarm **by itself** — by email and/or Telegram — when it starts with missing configuration or
|
|
2073
|
-
crashes. `EmergencyAlerter` decides **what** is said, **to whom** and **how often**; it sends through the gateways above
|
|
2074
|
-
(`SendGridGw`, `TelegramGw`). Each alert names the app, the service and the environment
|
|
2075
|
-
(`[partylabz · ms_storage · production]`), because several apps can run the same services side by side on one machine.
|
|
2076
|
-
|
|
2077
|
-
```typescript
|
|
2078
|
-
import { EmergencyAlerter, CriticalConfigItem } from '@sdflc/backend-helpers';
|
|
2079
|
-
|
|
2080
|
-
// What this service starts WITHOUT (optional to the config loader) but cannot fully work without.
|
|
2081
|
-
// APP_NAME is always checked as well; `key` (the built-config name) is derived from `env`.
|
|
2082
|
-
const CRITICAL_CONFIG: CriticalConfigItem[] = [
|
|
2083
|
-
{ env: 'INTERSERVICE_API_KEY', effect: 'Every call from our other services is refused.' },
|
|
2084
|
-
];
|
|
2085
|
-
|
|
2086
|
-
const alerter = EmergencyAlerter.fromConfig(config, 'ms_storage', { log: (m) => logger.error(m) });
|
|
2087
|
-
|
|
2088
|
-
alerter.installCrashHandlers(process); // uncaught exception / unhandled rejection → alert, then exit(1)
|
|
2089
|
-
|
|
2090
|
-
httpServer.listen(port, async () => {
|
|
2091
|
-
await alerter.checkConfig(config, CRITICAL_CONFIG); // the service still starts; it says loudly what is missing
|
|
2092
|
-
});
|
|
2093
|
-
|
|
2094
|
-
const result = await alerter.send({ key: 'queue:stalled', subject: 'The queue stopped draining', lines: ['…'] });
|
|
2095
|
-
// OK → data [{ status: 'sent', channels: { email: 'sent', telegram: 'failed' } }] or [{ status: 'recently-sent' }]
|
|
2096
|
-
// VALIDATION_FAILED → no channel can be used; FAILED → no channel delivered it (each channel's error merged in)
|
|
2097
|
-
```
|
|
2098
|
-
|
|
2099
|
-
Or build it yourself — credentials live in the gateways, the addressing in the alerter:
|
|
2100
|
-
|
|
2101
|
-
```typescript
|
|
2102
|
-
const alerter = new EmergencyAlerter({
|
|
2103
|
-
email: new SendGridGw().configure({ apiKey }),
|
|
2104
|
-
telegram: new TelegramGw().configure({ botToken }),
|
|
2105
|
-
settings: { appName: 'partylabz', serviceName: 'ms_storage', environment: 'production',
|
|
2106
|
-
emailTo: ['ops@brand.com'], emailFrom: 'alerts@brand.com', telegramChatId: '-100123' },
|
|
2107
|
-
});
|
|
2108
|
-
```
|
|
2109
|
-
|
|
2110
|
-
Settings `fromConfig` reads (all optional; each channel needs all of its own):
|
|
2111
|
-
|
|
2112
|
-
| Env | Meaning |
|
|
2113
|
-
|---|---|
|
|
2114
|
-
| `APP_NAME` | The app this service belongs to; alerts say `unknown-app` without it (and `checkConfig` reports it) |
|
|
2115
|
-
| `EMERGENCY_SENDGRID_API_KEY` | Email: a SendGrid key allowed to send mail |
|
|
2116
|
-
| `EMERGENCY_EMAIL_FROM` | Email: a sender on a domain authenticated in that SendGrid account (required — a guessed one is refused) |
|
|
2117
|
-
| `EMERGENCY_EMAIL_TO` | Email: recipients, comma-separated |
|
|
2118
|
-
| `TELEGRAM_BOT_ID` | Telegram: the bot token `<id>:<secret>` from @BotFather |
|
|
2119
|
-
| `TELEGRAM_CHAT_ID` | Telegram: the chat id (negative for a group) or `@channel` |
|
|
2120
|
-
| `EMERGENCY_EMAIL_MIN_INTERVAL_MINUTES` | The same alert at most once per this many minutes, on every channel (default 60; 0 = always) |
|
|
2121
|
-
|
|
2122
|
-
Rules it keeps:
|
|
2123
|
-
|
|
2124
|
-
- **Never throws, never blocks for long.** `fromConfig` gives each provider a 4 s timeout (one short retry); a crash
|
|
2125
|
-
waits for its alert at most 10 s, then exits — and **always** exits, whatever was thrown.
|
|
2126
|
-
- **A channel set up wrongly is said once, at the start** — never silently skipped (`getChannelProblems()`).
|
|
2127
|
-
- **What leaves is redacted:** `fromConfig` removes every config value whose NAME looks secret (password, secret,
|
|
2128
|
-
token, key…) from every subject, line and log line — an error message can carry SQL with its values. Pass `redact`
|
|
2129
|
-
for more.
|
|
2130
|
-
- **A crash's key names which crash** (its message and top frame): one crash never silences a different one.
|
|
2131
|
-
- **One alert per key per interval across all channels**, remembered in a file under the OS temp folder named after
|
|
2132
|
-
the app and service — a crash loop sends one message, not one per restart. Sends of the same key **at the same time**
|
|
2133
|
-
go out once; a send no channel delivered is not remembered, so the next one tries again.
|
|
2134
|
-
- **Goes straight to the providers**, never through another service — it has to work when the rest does not.
|
|
2135
|
-
- **Subjects are one line**; the Telegram text is cut **before** it is escaped, to what the heading and footer leave,
|
|
2136
|
-
so a long stack trace neither splits an entity nor overruns Telegram's limit.
|
|
2137
|
-
- **Never put a secret, a token or a database row in an alert** — it leaves your infrastructure.
|
|
2138
|
-
|
|
2139
|
-
Proofs: `npm run build && npm run prove` (the gateways and the alerter, each against a stub `fetch`).
|
|
2140
|
-
|
|
2141
|
-
---
|
|
2142
|
-
|
|
2143
2410
|
## Constants
|
|
2144
2411
|
|
|
2145
2412
|
```typescript
|
|
@@ -2177,6 +2444,23 @@ ORDER_BYS.ASC; // 'asc'
|
|
|
2177
2444
|
ORDER_BYS.DESC; // 'desc'
|
|
2178
2445
|
```
|
|
2179
2446
|
|
|
2447
|
+
Gateways and emergency alerts:
|
|
2448
|
+
|
|
2449
|
+
```typescript
|
|
2450
|
+
MAX_REPLY_BYTES; // 65536: the most of a reply a gateway reads
|
|
2451
|
+
MAX_RETRY_WAIT_MS; // 10000: a longer Retry-After is not waited for (the failure comes back at once)
|
|
2452
|
+
TELEGRAM_MAX_MESSAGE_LENGTH; // 4096
|
|
2453
|
+
TWILIO_MAX_BODY_LENGTH; // 1600
|
|
2454
|
+
DISCORD_MAX_CONTENT_LENGTH; // 2000
|
|
2455
|
+
DISCORD_MAX_EMBEDS; // 10
|
|
2456
|
+
WEBHOOK_ID_HEADER; // 'X-Webhook-Id'
|
|
2457
|
+
WEBHOOK_TIMESTAMP_HEADER; // 'X-Webhook-Timestamp'
|
|
2458
|
+
WEBHOOK_SIGNATURE_HEADER; // 'X-Webhook-Signature'
|
|
2459
|
+
UNKNOWN_APP_NAME; // 'unknown-app': what an alert says without APP_NAME
|
|
2460
|
+
APP_NAME_CONFIG; // the CriticalConfigItem every checkConfig includes
|
|
2461
|
+
START_READY_TIMEOUT_MS; // 60000: startService alerts when the server is not listening after this long
|
|
2462
|
+
```
|
|
2463
|
+
|
|
2180
2464
|
---
|
|
2181
2465
|
|
|
2182
2466
|
## Interfaces reference
|
|
@@ -2188,7 +2472,25 @@ ORDER_BYS.DESC; // 'desc'
|
|
|
2188
2472
|
| `BaseCorePropsInterface` | Constructor props for `BaseCore` subclasses |
|
|
2189
2473
|
| `BaseCoreRateLimitOnActionInterface` | Per-action rate-limit rule map used in `BaseCorePropsInterface` |
|
|
2190
2474
|
| `BaseGatewayPropsInterface` | Constructor props for `BaseGateway` subclasses — includes `statusField`, `nonCreatableFields`, `nonUpdatableFields`, and the filter rule arrays |
|
|
2191
|
-
| `BaseExternalApiGatewayPropsInterface` | Constructor props for `BaseExternalApiGateway`
|
|
2475
|
+
| `BaseExternalApiGatewayPropsInterface` | Constructor props for `BaseExternalApiGateway`: `settings`, `timeoutMs`, `retries`, `retryDelaysMs`, `fetch`, `logger`, `baseUrl`, `credentials` (legacy) |
|
|
2476
|
+
| `ExternalApiFetch` | The HTTP client shape a gateway calls (the platform `fetch`, or a test's stub) |
|
|
2477
|
+
| `ExternalApiRequest` | One outbound request for `request()` / `requestWithRetries()`: `{ url, method?, headers?, json?, body?, timeoutMs?, secrets? }` |
|
|
2478
|
+
| `ExternalApiReply` | What a request answered, in its OpResult data: `{ status, headers, body }`, scrubbed and capped |
|
|
2479
|
+
| `SendGridSettings` / `SendGridEmail` / `SendGridAddress` | `SendGridGw` settings, the email, an address (`'a@b.com'` or `{ email, name? }`) |
|
|
2480
|
+
| `TelegramSettings` / `TelegramMessage` | `TelegramGw` settings and message |
|
|
2481
|
+
| `TwilioSettings` / `TwilioMessage` | `TwilioGw` settings and message |
|
|
2482
|
+
| `SlackWebhookSettings` / `SlackMessage` | `SlackWebhookGw` settings and message |
|
|
2483
|
+
| `DiscordWebhookSettings` / `DiscordMessage` / `DiscordEmbed` | `DiscordWebhookGw` settings, message and embed |
|
|
2484
|
+
| `HttpWebhookSettings` / `HttpWebhookSend` / `HttpWebhookGwProps` | `HttpWebhookGw` settings, one send, and its constructor props (`allowPrivateNetwork`, `lookup`, `dnsLookup`) |
|
|
2485
|
+
| `ExternalUrlOptions` | Options for `checkExternalUrl()`: `{ protocols?, lookup?, field? }` |
|
|
2486
|
+
| `GuardedFetchOptions` | Options for `createGuardedFetch()`: `{ dnsLookup?, isBlocked? }` |
|
|
2487
|
+
| `EmergencyAlert` | One alert: `{ key, subject, lines }` |
|
|
2488
|
+
| `EmergencyAlerterSettings` | `EmergencyAlerter` settings: `appName`, `serviceName`, `environment`, `minIntervalMinutes`, `emailTo`, `emailFrom`, `telegramChatId` |
|
|
2489
|
+
| `EmergencyAlerterProps` | `EmergencyAlerter` constructor props: `settings`, `email`, `telegram`, `log`, `redact`, `memory`, `now`, `hostname` |
|
|
2490
|
+
| `EmergencyAlertMemory` | Where sent alerts are remembered: `{ read, write }` (default: a file under the OS temp folder, per app and service) |
|
|
2491
|
+
| `CriticalConfigItem` | A setting a service cannot fully work without: `{ env, key?, effect }` |
|
|
2492
|
+
| `EmergencyStartOptions` | `startService()` options: `start`, `port`, `config?`, `critical?`, `afterListen?`, `readyTimeoutMs?`, `exit?` |
|
|
2493
|
+
| `EmergencyStartServer` | What `startService()` listens on: `{ listen, once, removeListener }` — an `http.Server` |
|
|
2192
2494
|
| `BaseRestApiCrudControllerPropsInterface` | Constructor props for `BaseRestApiCrudController` |
|
|
2193
2495
|
| `BuildDefaultResolvers` | Arguments for `buildDefaultResolvers()` |
|
|
2194
2496
|
| `BuildListQueryArgsInterface` | Arguments passed to `buildListQuery()` overrides: `{ filter, params, fieldsToUse, useCache }` |
|
|
@@ -1,6 +1,28 @@
|
|
|
1
1
|
import { OpResult } from '@sdflc/api-helpers';
|
|
2
2
|
import { SendGridGw, SendGridEmail } from './gateways/SendGridGw';
|
|
3
3
|
import { TelegramGw } from './gateways/TelegramGw';
|
|
4
|
+
/** What `startService` listens on — an `http.Server` (for Express + Apollo, the one `configureApp` creates). */
|
|
5
|
+
export interface EmergencyStartServer {
|
|
6
|
+
listen: (port: number, callback: () => void) => any;
|
|
7
|
+
once: (event: 'error', handler: (error: any) => void) => any;
|
|
8
|
+
removeListener: (event: 'error', handler: (error: any) => void) => any;
|
|
9
|
+
}
|
|
10
|
+
export interface EmergencyStartOptions {
|
|
11
|
+
/** Builds the app and answers the server to listen on. Everything slow (Redis, the database, Apollo) happens here. */
|
|
12
|
+
start: () => Promise<EmergencyStartServer> | EmergencyStartServer;
|
|
13
|
+
port: number | string;
|
|
14
|
+
/** The service config — checked against `critical` (and APP_NAME) AFTER the server listens. Omit to skip the check. */
|
|
15
|
+
config?: any;
|
|
16
|
+
critical?: CriticalConfigItem[];
|
|
17
|
+
/** Runs once the server listens: a startup log, background jobs, the service's own checks. A failure is alerted, never fatal. */
|
|
18
|
+
afterListen?: () => Promise<unknown> | unknown;
|
|
19
|
+
/** Still not listening this long after the start began → one alert; the start keeps waiting. Default 60 s. */
|
|
20
|
+
readyTimeoutMs?: number;
|
|
21
|
+
/** Called with 1 when the start fails, after its alert. Default `process.exit`. */
|
|
22
|
+
exit?: (code: number) => any;
|
|
23
|
+
}
|
|
24
|
+
/** The default wait before "still not listening" is alerted. */
|
|
25
|
+
export declare const START_READY_TIMEOUT_MS = 60000;
|
|
4
26
|
/** What an alert says when APP_NAME is not set. */
|
|
5
27
|
export declare const UNKNOWN_APP_NAME = "unknown-app";
|
|
6
28
|
export interface EmergencyAlert {
|
|
@@ -74,6 +96,8 @@ export declare const missingCriticalConfig: (config: any, items?: CriticalConfig
|
|
|
74
96
|
}>;
|
|
75
97
|
/** The alert for missing configuration, or null when nothing is missing. Pure. */
|
|
76
98
|
export declare const missingConfigAlert: (config: any, items?: CriticalConfigItem[]) => EmergencyAlert | null;
|
|
99
|
+
/** `text` with everything that looks like a secret replaced by `***`. Pure; never throws. */
|
|
100
|
+
export declare const redactKnownSecrets: (text: string) => string;
|
|
77
101
|
/** The memory in a file under the OS temp folder, one per app AND service (several may share a temp folder). */
|
|
78
102
|
export declare const emergencyFileMemory: (appName: string, serviceName: string, dir?: string) => EmergencyAlertMemory;
|
|
79
103
|
declare class EmergencyAlerter {
|
|
@@ -126,6 +150,17 @@ declare class EmergencyAlerter {
|
|
|
126
150
|
* when nothing is missing; otherwise what `send` answered. Never throws.
|
|
127
151
|
*/
|
|
128
152
|
checkConfig(config: any, items?: CriticalConfigItem[]): Promise<OpResult>;
|
|
153
|
+
/**
|
|
154
|
+
* The whole start of a service. Listens on the server `start` answers; then runs `afterListen` and checks the
|
|
155
|
+
* configuration (a missing value is alerted, never fatal — the service answers what it can). OK with
|
|
156
|
+
* `{ status: 'listening', port }`. When the start throws, or the port cannot be listened on, it alerts (keyed by WHICH
|
|
157
|
+
* failure) and exits 1. Never throws.
|
|
158
|
+
*
|
|
159
|
+
* ⚠ WHY THE TIMER: a dependency that never answers — Redis's client retries a refused connection for ever — hangs the
|
|
160
|
+
* start with the port closed and nothing but a log line every half second. Not listening after `readyTimeoutMs` is
|
|
161
|
+
* alerted once; the start keeps waiting, and when it listens at last that is said too.
|
|
162
|
+
*/
|
|
163
|
+
startService(opts: EmergencyStartOptions): Promise<OpResult>;
|
|
129
164
|
/**
|
|
130
165
|
* Alerts on an uncaught exception or an unhandled rejection, then exits 1 — the process is in an unknown state after an
|
|
131
166
|
* uncaught error, and a restart is the recovery (what Node would have done without the handler). The exit waits for the
|
|
@@ -3,7 +3,7 @@ var __importDefault = (this && this.__importDefault) || function (mod) {
|
|
|
3
3
|
return (mod && mod.__esModule) ? mod : { "default": mod };
|
|
4
4
|
};
|
|
5
5
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
6
|
-
exports.EmergencyAlerter = exports.emergencyFileMemory = exports.missingConfigAlert = exports.missingCriticalConfig = exports.emergencyIntervalText = exports.emergencyOneLine = exports.APP_NAME_CONFIG = exports.UNKNOWN_APP_NAME = void 0;
|
|
6
|
+
exports.EmergencyAlerter = exports.emergencyFileMemory = exports.redactKnownSecrets = exports.missingConfigAlert = exports.missingCriticalConfig = exports.emergencyIntervalText = exports.emergencyOneLine = exports.APP_NAME_CONFIG = exports.UNKNOWN_APP_NAME = exports.START_READY_TIMEOUT_MS = void 0;
|
|
7
7
|
// ./src/EmergencyAlerter.ts
|
|
8
8
|
//
|
|
9
9
|
// An EMERGENCY alert a service sends BY ITSELF — for what nobody would otherwise notice until somebody trips over it: a
|
|
@@ -12,7 +12,11 @@ exports.EmergencyAlerter = exports.emergencyFileMemory = exports.missingConfigAl
|
|
|
12
12
|
//
|
|
13
13
|
// const alerter = EmergencyAlerter.fromConfig(config, 'ms_storage', { log: (m) => logger.error(m) });
|
|
14
14
|
// alerter.installCrashHandlers(process);
|
|
15
|
-
//
|
|
15
|
+
// void alerter.startService({ start: () => configureApp(express()), port: config.port, config, critical: CRITICAL_CONFIG });
|
|
16
|
+
//
|
|
17
|
+
// `startService` is the whole start: listen on the server `start` answers, check the configuration AFTER listening (a
|
|
18
|
+
// missing value never stops the start), alert when the server is still not listening after a while (a Redis or database
|
|
19
|
+
// that never answers hangs the start without a word otherwise), and alert + exit 1 when the start fails.
|
|
16
20
|
//
|
|
17
21
|
// Every alert names its app, service and environment — `[partylabz · ms_storage · production]` — because several apps
|
|
18
22
|
// run the same services side by side on one machine, and "ms_storage crashed" alone does not say whose.
|
|
@@ -37,11 +41,14 @@ const crypto_1 = require("crypto");
|
|
|
37
41
|
const fs_1 = __importDefault(require("fs"));
|
|
38
42
|
const os_1 = __importDefault(require("os"));
|
|
39
43
|
const path_1 = __importDefault(require("path"));
|
|
44
|
+
const timers_1 = require("timers");
|
|
40
45
|
const promises_1 = require("timers/promises");
|
|
41
46
|
const lodash_1 = require("lodash");
|
|
42
47
|
const api_helpers_1 = require("@sdflc/api-helpers");
|
|
43
48
|
const SendGridGw_1 = require("./gateways/SendGridGw");
|
|
44
49
|
const TelegramGw_1 = require("./gateways/TelegramGw");
|
|
50
|
+
/** The default wait before "still not listening" is alerted. */
|
|
51
|
+
exports.START_READY_TIMEOUT_MS = 60_000;
|
|
45
52
|
/** What an alert says when APP_NAME is not set. */
|
|
46
53
|
exports.UNKNOWN_APP_NAME = 'unknown-app';
|
|
47
54
|
/** Checked in every service: without it no alert can say which app it comes from. */
|
|
@@ -113,6 +120,37 @@ const escapedPrefix = (text, budget) => {
|
|
|
113
120
|
const keep = low > 0 && code >= 0xd800 && code <= 0xdbff ? low - 1 : low;
|
|
114
121
|
return escapeHtml(text.slice(0, keep));
|
|
115
122
|
};
|
|
123
|
+
/**
|
|
124
|
+
* Secrets that look like secrets whatever variable held them — provider keys, tokens, credentials inside URLs, connection
|
|
125
|
+
* strings and `name=value` pairs. Alerts carry error messages and stacks, which can quote a per-tenant secret read from a
|
|
126
|
+
* database row (that no config scrub knows about). Applied to EVERY alert, under any custom `redact`.
|
|
127
|
+
*/
|
|
128
|
+
const KNOWN_SECRET_PATTERNS = [
|
|
129
|
+
[/\bSG\.[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}/g, 'SG.***'], // SendGrid
|
|
130
|
+
[/\b(sk|rk)_(live|test)_[A-Za-z0-9]{8,}/g, '$1_$2_***'], // Stripe secret / restricted
|
|
131
|
+
[/\bwhsec_[A-Za-z0-9]{8,}/g, 'whsec_***'], // Stripe webhook secret
|
|
132
|
+
[/\b\d{6,12}:[A-Za-z0-9_-]{30,}/g, '***:***'], // Telegram bot token
|
|
133
|
+
[/\beyJ[A-Za-z0-9_-]{5,}\.[A-Za-z0-9_-]{5,}\.[A-Za-z0-9_-]{5,}/g, '***.***.***'], // JWT
|
|
134
|
+
[/\b(AccountKey|SharedAccessKey|SharedAccessSignature)=[^;\s"']+/gi, '$1=***'], // Azure connection strings
|
|
135
|
+
[/\b(Bearer|Basic)\s+[A-Za-z0-9._~+/-]{8,}=*/gi, '$1 ***'], // Authorization values
|
|
136
|
+
[/(:\/\/[^/\s:@]+:)[^@\s/]+@/g, '$1***@'], // credentials inside a URL
|
|
137
|
+
[/\b(password|passwd|pwd|secret|token|api[_-]?key|apikey)(["']?\s*[:=]\s*["']?)[^\s"',;&]{6,}/gi, '$1$2***'], // name=value
|
|
138
|
+
];
|
|
139
|
+
/** `text` with everything that looks like a secret replaced by `***`. Pure; never throws. */
|
|
140
|
+
const redactKnownSecrets = (text) => {
|
|
141
|
+
try {
|
|
142
|
+
return KNOWN_SECRET_PATTERNS.reduce((out, [pattern, replacement]) => out.replace(pattern, replacement), String(text));
|
|
143
|
+
}
|
|
144
|
+
catch {
|
|
145
|
+
return '[redacted]';
|
|
146
|
+
}
|
|
147
|
+
};
|
|
148
|
+
exports.redactKnownSecrets = redactKnownSecrets;
|
|
149
|
+
/** Which failure this is — its message and top frame — so one failure does not silence a different one. Pure. */
|
|
150
|
+
const failureFingerprintOf = (message, stack) => (0, crypto_1.createHash)('sha256')
|
|
151
|
+
.update(`${message}\n${stack[1] ?? ''}`)
|
|
152
|
+
.digest('hex')
|
|
153
|
+
.slice(0, 12);
|
|
116
154
|
/** The memory in a file under the OS temp folder, one per app AND service (several may share a temp folder). */
|
|
117
155
|
const emergencyFileMemory = (appName, serviceName, dir = os_1.default.tmpdir()) => {
|
|
118
156
|
const safe = (name) => name.replace(/[^A-Za-z0-9_.-]/g, '_');
|
|
@@ -164,9 +202,10 @@ class EmergencyAlerter {
|
|
|
164
202
|
}
|
|
165
203
|
};
|
|
166
204
|
const redact = props.redact ?? ((text) => text);
|
|
205
|
+
// ⚠ Known secret shapes are removed ALWAYS, after any custom redact: a custom hook narrows nothing.
|
|
167
206
|
this.redact = (text) => {
|
|
168
207
|
try {
|
|
169
|
-
return String(redact(String(text ?? '')));
|
|
208
|
+
return (0, exports.redactKnownSecrets)(String(redact(String(text ?? ''))));
|
|
170
209
|
}
|
|
171
210
|
catch {
|
|
172
211
|
return '[redacted: the redact hook failed]';
|
|
@@ -413,6 +452,97 @@ class EmergencyAlerter {
|
|
|
413
452
|
const alert = (0, exports.missingConfigAlert)(config, items);
|
|
414
453
|
return alert ? this.send(alert) : api_helpers_1.OpResult.ok([]);
|
|
415
454
|
}
|
|
455
|
+
/**
|
|
456
|
+
* The whole start of a service. Listens on the server `start` answers; then runs `afterListen` and checks the
|
|
457
|
+
* configuration (a missing value is alerted, never fatal — the service answers what it can). OK with
|
|
458
|
+
* `{ status: 'listening', port }`. When the start throws, or the port cannot be listened on, it alerts (keyed by WHICH
|
|
459
|
+
* failure) and exits 1. Never throws.
|
|
460
|
+
*
|
|
461
|
+
* ⚠ WHY THE TIMER: a dependency that never answers — Redis's client retries a refused connection for ever — hangs the
|
|
462
|
+
* start with the port closed and nothing but a log line every half second. Not listening after `readyTimeoutMs` is
|
|
463
|
+
* alerted once; the start keeps waiting, and when it listens at last that is said too.
|
|
464
|
+
*/
|
|
465
|
+
async startService(opts) {
|
|
466
|
+
const exit = opts?.exit ?? ((code) => process.exit(code));
|
|
467
|
+
const readyMs = Number.isFinite(Number(opts?.readyTimeoutMs))
|
|
468
|
+
? Number(opts.readyTimeoutMs)
|
|
469
|
+
: exports.START_READY_TIMEOUT_MS;
|
|
470
|
+
const port = Number(opts?.port);
|
|
471
|
+
let listening = false;
|
|
472
|
+
let lateAlerted = false;
|
|
473
|
+
const startedAt = Date.now();
|
|
474
|
+
// ⚠ Not unref'd: a start that hangs on nothing at all must still live long enough to say so.
|
|
475
|
+
const timer = (0, timers_1.setTimeout)(() => {
|
|
476
|
+
if (listening) {
|
|
477
|
+
return;
|
|
478
|
+
}
|
|
479
|
+
lateAlerted = true;
|
|
480
|
+
void this.send({
|
|
481
|
+
key: 'startup:not-listening',
|
|
482
|
+
subject: `Not listening ${(0, exports.emergencyIntervalText)(readyMs)} after the start`,
|
|
483
|
+
lines: [
|
|
484
|
+
`The start is still waiting, and port ${opts?.port} is closed: nothing is answered.`,
|
|
485
|
+
'It waits on something that does not answer — Redis, the database, or another service. It keeps waiting.',
|
|
486
|
+
],
|
|
487
|
+
});
|
|
488
|
+
}, readyMs);
|
|
489
|
+
try {
|
|
490
|
+
if (!Number.isInteger(port) || port < 0 || port > 65535) {
|
|
491
|
+
throw new Error(`The port '${opts?.port}' is not a port`);
|
|
492
|
+
}
|
|
493
|
+
const server = await opts.start();
|
|
494
|
+
await new Promise((resolve, reject) => {
|
|
495
|
+
const onError = (error) => reject(error);
|
|
496
|
+
server.once('error', onError);
|
|
497
|
+
server.listen(port, () => {
|
|
498
|
+
server.removeListener('error', onError);
|
|
499
|
+
resolve();
|
|
500
|
+
});
|
|
501
|
+
});
|
|
502
|
+
listening = true;
|
|
503
|
+
(0, timers_1.clearTimeout)(timer);
|
|
504
|
+
if (lateAlerted) {
|
|
505
|
+
void this.send({
|
|
506
|
+
key: 'startup:listening-late',
|
|
507
|
+
subject: `Listening at last, ${(0, exports.emergencyIntervalText)(Date.now() - startedAt)} after the start`,
|
|
508
|
+
lines: ['What it waited on answered. Nothing to do unless it happens again.'],
|
|
509
|
+
});
|
|
510
|
+
}
|
|
511
|
+
}
|
|
512
|
+
catch (e) {
|
|
513
|
+
(0, timers_1.clearTimeout)(timer);
|
|
514
|
+
const { message, stack } = describeError(e);
|
|
515
|
+
await this.send({
|
|
516
|
+
key: `crash:startup:${failureFingerprintOf(message, stack)}`,
|
|
517
|
+
subject: `Could not start: ${(0, exports.emergencyOneLine)(message, 120)}`,
|
|
518
|
+
lines: [message, '', ...stack],
|
|
519
|
+
});
|
|
520
|
+
try {
|
|
521
|
+
exit(1);
|
|
522
|
+
}
|
|
523
|
+
catch {
|
|
524
|
+
// An exit that throws leaves nothing more to do.
|
|
525
|
+
}
|
|
526
|
+
return api_helpers_1.OpResult.fromException(e);
|
|
527
|
+
}
|
|
528
|
+
if (opts.afterListen) {
|
|
529
|
+
try {
|
|
530
|
+
await opts.afterListen();
|
|
531
|
+
}
|
|
532
|
+
catch (e) {
|
|
533
|
+
const { message, stack } = describeError(e);
|
|
534
|
+
await this.send({
|
|
535
|
+
key: `startup:after-listen:${failureFingerprintOf(message, stack)}`,
|
|
536
|
+
subject: `Started, but a startup step failed: ${(0, exports.emergencyOneLine)(message, 120)}`,
|
|
537
|
+
lines: [message, '', ...stack],
|
|
538
|
+
});
|
|
539
|
+
}
|
|
540
|
+
}
|
|
541
|
+
if (opts.config !== undefined) {
|
|
542
|
+
await this.checkConfig(opts.config, opts.critical ?? []);
|
|
543
|
+
}
|
|
544
|
+
return api_helpers_1.OpResult.ok([{ status: 'listening', port }]);
|
|
545
|
+
}
|
|
416
546
|
/**
|
|
417
547
|
* Alerts on an uncaught exception or an unhandled rejection, then exits 1 — the process is in an unknown state after an
|
|
418
548
|
* uncaught error, and a restart is the recovery (what Node would have done without the handler). The exit waits for the
|
|
@@ -429,10 +559,7 @@ class EmergencyAlerter {
|
|
|
429
559
|
try {
|
|
430
560
|
const { message, stack } = describeError(error);
|
|
431
561
|
// ⚠ The key names WHICH crash (its message and top frame): one crash must not silence a different one for an hour.
|
|
432
|
-
const fingerprint = (
|
|
433
|
-
.update(`${message}\n${stack[1] ?? ''}`)
|
|
434
|
-
.digest('hex')
|
|
435
|
-
.slice(0, 12);
|
|
562
|
+
const fingerprint = failureFingerprintOf(message, stack);
|
|
436
563
|
const alert = {
|
|
437
564
|
key: `crash:${kind}:${fingerprint}`,
|
|
438
565
|
subject: `Crashed (${kind}): ${(0, exports.emergencyOneLine)(message, 120)}`,
|
|
@@ -1,6 +1,28 @@
|
|
|
1
1
|
import { OpResult } from '@sdflc/api-helpers';
|
|
2
2
|
import { SendGridGw, SendGridEmail } from './gateways/SendGridGw';
|
|
3
3
|
import { TelegramGw } from './gateways/TelegramGw';
|
|
4
|
+
/** What `startService` listens on — an `http.Server` (for Express + Apollo, the one `configureApp` creates). */
|
|
5
|
+
export interface EmergencyStartServer {
|
|
6
|
+
listen: (port: number, callback: () => void) => any;
|
|
7
|
+
once: (event: 'error', handler: (error: any) => void) => any;
|
|
8
|
+
removeListener: (event: 'error', handler: (error: any) => void) => any;
|
|
9
|
+
}
|
|
10
|
+
export interface EmergencyStartOptions {
|
|
11
|
+
/** Builds the app and answers the server to listen on. Everything slow (Redis, the database, Apollo) happens here. */
|
|
12
|
+
start: () => Promise<EmergencyStartServer> | EmergencyStartServer;
|
|
13
|
+
port: number | string;
|
|
14
|
+
/** The service config — checked against `critical` (and APP_NAME) AFTER the server listens. Omit to skip the check. */
|
|
15
|
+
config?: any;
|
|
16
|
+
critical?: CriticalConfigItem[];
|
|
17
|
+
/** Runs once the server listens: a startup log, background jobs, the service's own checks. A failure is alerted, never fatal. */
|
|
18
|
+
afterListen?: () => Promise<unknown> | unknown;
|
|
19
|
+
/** Still not listening this long after the start began → one alert; the start keeps waiting. Default 60 s. */
|
|
20
|
+
readyTimeoutMs?: number;
|
|
21
|
+
/** Called with 1 when the start fails, after its alert. Default `process.exit`. */
|
|
22
|
+
exit?: (code: number) => any;
|
|
23
|
+
}
|
|
24
|
+
/** The default wait before "still not listening" is alerted. */
|
|
25
|
+
export declare const START_READY_TIMEOUT_MS = 60000;
|
|
4
26
|
/** What an alert says when APP_NAME is not set. */
|
|
5
27
|
export declare const UNKNOWN_APP_NAME = "unknown-app";
|
|
6
28
|
export interface EmergencyAlert {
|
|
@@ -74,6 +96,8 @@ export declare const missingCriticalConfig: (config: any, items?: CriticalConfig
|
|
|
74
96
|
}>;
|
|
75
97
|
/** The alert for missing configuration, or null when nothing is missing. Pure. */
|
|
76
98
|
export declare const missingConfigAlert: (config: any, items?: CriticalConfigItem[]) => EmergencyAlert | null;
|
|
99
|
+
/** `text` with everything that looks like a secret replaced by `***`. Pure; never throws. */
|
|
100
|
+
export declare const redactKnownSecrets: (text: string) => string;
|
|
77
101
|
/** The memory in a file under the OS temp folder, one per app AND service (several may share a temp folder). */
|
|
78
102
|
export declare const emergencyFileMemory: (appName: string, serviceName: string, dir?: string) => EmergencyAlertMemory;
|
|
79
103
|
declare class EmergencyAlerter {
|
|
@@ -126,6 +150,17 @@ declare class EmergencyAlerter {
|
|
|
126
150
|
* when nothing is missing; otherwise what `send` answered. Never throws.
|
|
127
151
|
*/
|
|
128
152
|
checkConfig(config: any, items?: CriticalConfigItem[]): Promise<OpResult>;
|
|
153
|
+
/**
|
|
154
|
+
* The whole start of a service. Listens on the server `start` answers; then runs `afterListen` and checks the
|
|
155
|
+
* configuration (a missing value is alerted, never fatal — the service answers what it can). OK with
|
|
156
|
+
* `{ status: 'listening', port }`. When the start throws, or the port cannot be listened on, it alerts (keyed by WHICH
|
|
157
|
+
* failure) and exits 1. Never throws.
|
|
158
|
+
*
|
|
159
|
+
* ⚠ WHY THE TIMER: a dependency that never answers — Redis's client retries a refused connection for ever — hangs the
|
|
160
|
+
* start with the port closed and nothing but a log line every half second. Not listening after `readyTimeoutMs` is
|
|
161
|
+
* alerted once; the start keeps waiting, and when it listens at last that is said too.
|
|
162
|
+
*/
|
|
163
|
+
startService(opts: EmergencyStartOptions): Promise<OpResult>;
|
|
129
164
|
/**
|
|
130
165
|
* Alerts on an uncaught exception or an unhandled rejection, then exits 1 — the process is in an unknown state after an
|
|
131
166
|
* uncaught error, and a restart is the recovery (what Node would have done without the handler). The exit waits for the
|
|
@@ -6,7 +6,11 @@
|
|
|
6
6
|
//
|
|
7
7
|
// const alerter = EmergencyAlerter.fromConfig(config, 'ms_storage', { log: (m) => logger.error(m) });
|
|
8
8
|
// alerter.installCrashHandlers(process);
|
|
9
|
-
//
|
|
9
|
+
// void alerter.startService({ start: () => configureApp(express()), port: config.port, config, critical: CRITICAL_CONFIG });
|
|
10
|
+
//
|
|
11
|
+
// `startService` is the whole start: listen on the server `start` answers, check the configuration AFTER listening (a
|
|
12
|
+
// missing value never stops the start), alert when the server is still not listening after a while (a Redis or database
|
|
13
|
+
// that never answers hangs the start without a word otherwise), and alert + exit 1 when the start fails.
|
|
10
14
|
//
|
|
11
15
|
// Every alert names its app, service and environment — `[partylabz · ms_storage · production]` — because several apps
|
|
12
16
|
// run the same services side by side on one machine, and "ms_storage crashed" alone does not say whose.
|
|
@@ -31,11 +35,14 @@ import { createHash } from 'crypto';
|
|
|
31
35
|
import fs from 'fs';
|
|
32
36
|
import os from 'os';
|
|
33
37
|
import path from 'path';
|
|
38
|
+
import { clearTimeout, setTimeout as setTimer } from 'timers';
|
|
34
39
|
import { setTimeout as sleep } from 'timers/promises';
|
|
35
40
|
import { camelCase } from 'lodash';
|
|
36
41
|
import { OP_RESULT_CODES, OpResult } from '@sdflc/api-helpers';
|
|
37
42
|
import { SendGridGw } from './gateways/SendGridGw';
|
|
38
43
|
import { TelegramGw, TELEGRAM_MAX_MESSAGE_LENGTH, isTelegramChatId } from './gateways/TelegramGw';
|
|
44
|
+
/** The default wait before "still not listening" is alerted. */
|
|
45
|
+
export const START_READY_TIMEOUT_MS = 60_000;
|
|
39
46
|
/** What an alert says when APP_NAME is not set. */
|
|
40
47
|
export const UNKNOWN_APP_NAME = 'unknown-app';
|
|
41
48
|
/** Checked in every service: without it no alert can say which app it comes from. */
|
|
@@ -103,6 +110,36 @@ const escapedPrefix = (text, budget) => {
|
|
|
103
110
|
const keep = low > 0 && code >= 0xd800 && code <= 0xdbff ? low - 1 : low;
|
|
104
111
|
return escapeHtml(text.slice(0, keep));
|
|
105
112
|
};
|
|
113
|
+
/**
|
|
114
|
+
* Secrets that look like secrets whatever variable held them — provider keys, tokens, credentials inside URLs, connection
|
|
115
|
+
* strings and `name=value` pairs. Alerts carry error messages and stacks, which can quote a per-tenant secret read from a
|
|
116
|
+
* database row (that no config scrub knows about). Applied to EVERY alert, under any custom `redact`.
|
|
117
|
+
*/
|
|
118
|
+
const KNOWN_SECRET_PATTERNS = [
|
|
119
|
+
[/\bSG\.[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}/g, 'SG.***'], // SendGrid
|
|
120
|
+
[/\b(sk|rk)_(live|test)_[A-Za-z0-9]{8,}/g, '$1_$2_***'], // Stripe secret / restricted
|
|
121
|
+
[/\bwhsec_[A-Za-z0-9]{8,}/g, 'whsec_***'], // Stripe webhook secret
|
|
122
|
+
[/\b\d{6,12}:[A-Za-z0-9_-]{30,}/g, '***:***'], // Telegram bot token
|
|
123
|
+
[/\beyJ[A-Za-z0-9_-]{5,}\.[A-Za-z0-9_-]{5,}\.[A-Za-z0-9_-]{5,}/g, '***.***.***'], // JWT
|
|
124
|
+
[/\b(AccountKey|SharedAccessKey|SharedAccessSignature)=[^;\s"']+/gi, '$1=***'], // Azure connection strings
|
|
125
|
+
[/\b(Bearer|Basic)\s+[A-Za-z0-9._~+/-]{8,}=*/gi, '$1 ***'], // Authorization values
|
|
126
|
+
[/(:\/\/[^/\s:@]+:)[^@\s/]+@/g, '$1***@'], // credentials inside a URL
|
|
127
|
+
[/\b(password|passwd|pwd|secret|token|api[_-]?key|apikey)(["']?\s*[:=]\s*["']?)[^\s"',;&]{6,}/gi, '$1$2***'], // name=value
|
|
128
|
+
];
|
|
129
|
+
/** `text` with everything that looks like a secret replaced by `***`. Pure; never throws. */
|
|
130
|
+
export const redactKnownSecrets = (text) => {
|
|
131
|
+
try {
|
|
132
|
+
return KNOWN_SECRET_PATTERNS.reduce((out, [pattern, replacement]) => out.replace(pattern, replacement), String(text));
|
|
133
|
+
}
|
|
134
|
+
catch {
|
|
135
|
+
return '[redacted]';
|
|
136
|
+
}
|
|
137
|
+
};
|
|
138
|
+
/** Which failure this is — its message and top frame — so one failure does not silence a different one. Pure. */
|
|
139
|
+
const failureFingerprintOf = (message, stack) => createHash('sha256')
|
|
140
|
+
.update(`${message}\n${stack[1] ?? ''}`)
|
|
141
|
+
.digest('hex')
|
|
142
|
+
.slice(0, 12);
|
|
106
143
|
/** The memory in a file under the OS temp folder, one per app AND service (several may share a temp folder). */
|
|
107
144
|
export const emergencyFileMemory = (appName, serviceName, dir = os.tmpdir()) => {
|
|
108
145
|
const safe = (name) => name.replace(/[^A-Za-z0-9_.-]/g, '_');
|
|
@@ -153,9 +190,10 @@ class EmergencyAlerter {
|
|
|
153
190
|
}
|
|
154
191
|
};
|
|
155
192
|
const redact = props.redact ?? ((text) => text);
|
|
193
|
+
// ⚠ Known secret shapes are removed ALWAYS, after any custom redact: a custom hook narrows nothing.
|
|
156
194
|
this.redact = (text) => {
|
|
157
195
|
try {
|
|
158
|
-
return String(redact(String(text ?? '')));
|
|
196
|
+
return redactKnownSecrets(String(redact(String(text ?? ''))));
|
|
159
197
|
}
|
|
160
198
|
catch {
|
|
161
199
|
return '[redacted: the redact hook failed]';
|
|
@@ -402,6 +440,97 @@ class EmergencyAlerter {
|
|
|
402
440
|
const alert = missingConfigAlert(config, items);
|
|
403
441
|
return alert ? this.send(alert) : OpResult.ok([]);
|
|
404
442
|
}
|
|
443
|
+
/**
|
|
444
|
+
* The whole start of a service. Listens on the server `start` answers; then runs `afterListen` and checks the
|
|
445
|
+
* configuration (a missing value is alerted, never fatal — the service answers what it can). OK with
|
|
446
|
+
* `{ status: 'listening', port }`. When the start throws, or the port cannot be listened on, it alerts (keyed by WHICH
|
|
447
|
+
* failure) and exits 1. Never throws.
|
|
448
|
+
*
|
|
449
|
+
* ⚠ WHY THE TIMER: a dependency that never answers — Redis's client retries a refused connection for ever — hangs the
|
|
450
|
+
* start with the port closed and nothing but a log line every half second. Not listening after `readyTimeoutMs` is
|
|
451
|
+
* alerted once; the start keeps waiting, and when it listens at last that is said too.
|
|
452
|
+
*/
|
|
453
|
+
async startService(opts) {
|
|
454
|
+
const exit = opts?.exit ?? ((code) => process.exit(code));
|
|
455
|
+
const readyMs = Number.isFinite(Number(opts?.readyTimeoutMs))
|
|
456
|
+
? Number(opts.readyTimeoutMs)
|
|
457
|
+
: START_READY_TIMEOUT_MS;
|
|
458
|
+
const port = Number(opts?.port);
|
|
459
|
+
let listening = false;
|
|
460
|
+
let lateAlerted = false;
|
|
461
|
+
const startedAt = Date.now();
|
|
462
|
+
// ⚠ Not unref'd: a start that hangs on nothing at all must still live long enough to say so.
|
|
463
|
+
const timer = setTimer(() => {
|
|
464
|
+
if (listening) {
|
|
465
|
+
return;
|
|
466
|
+
}
|
|
467
|
+
lateAlerted = true;
|
|
468
|
+
void this.send({
|
|
469
|
+
key: 'startup:not-listening',
|
|
470
|
+
subject: `Not listening ${emergencyIntervalText(readyMs)} after the start`,
|
|
471
|
+
lines: [
|
|
472
|
+
`The start is still waiting, and port ${opts?.port} is closed: nothing is answered.`,
|
|
473
|
+
'It waits on something that does not answer — Redis, the database, or another service. It keeps waiting.',
|
|
474
|
+
],
|
|
475
|
+
});
|
|
476
|
+
}, readyMs);
|
|
477
|
+
try {
|
|
478
|
+
if (!Number.isInteger(port) || port < 0 || port > 65535) {
|
|
479
|
+
throw new Error(`The port '${opts?.port}' is not a port`);
|
|
480
|
+
}
|
|
481
|
+
const server = await opts.start();
|
|
482
|
+
await new Promise((resolve, reject) => {
|
|
483
|
+
const onError = (error) => reject(error);
|
|
484
|
+
server.once('error', onError);
|
|
485
|
+
server.listen(port, () => {
|
|
486
|
+
server.removeListener('error', onError);
|
|
487
|
+
resolve();
|
|
488
|
+
});
|
|
489
|
+
});
|
|
490
|
+
listening = true;
|
|
491
|
+
clearTimeout(timer);
|
|
492
|
+
if (lateAlerted) {
|
|
493
|
+
void this.send({
|
|
494
|
+
key: 'startup:listening-late',
|
|
495
|
+
subject: `Listening at last, ${emergencyIntervalText(Date.now() - startedAt)} after the start`,
|
|
496
|
+
lines: ['What it waited on answered. Nothing to do unless it happens again.'],
|
|
497
|
+
});
|
|
498
|
+
}
|
|
499
|
+
}
|
|
500
|
+
catch (e) {
|
|
501
|
+
clearTimeout(timer);
|
|
502
|
+
const { message, stack } = describeError(e);
|
|
503
|
+
await this.send({
|
|
504
|
+
key: `crash:startup:${failureFingerprintOf(message, stack)}`,
|
|
505
|
+
subject: `Could not start: ${emergencyOneLine(message, 120)}`,
|
|
506
|
+
lines: [message, '', ...stack],
|
|
507
|
+
});
|
|
508
|
+
try {
|
|
509
|
+
exit(1);
|
|
510
|
+
}
|
|
511
|
+
catch {
|
|
512
|
+
// An exit that throws leaves nothing more to do.
|
|
513
|
+
}
|
|
514
|
+
return OpResult.fromException(e);
|
|
515
|
+
}
|
|
516
|
+
if (opts.afterListen) {
|
|
517
|
+
try {
|
|
518
|
+
await opts.afterListen();
|
|
519
|
+
}
|
|
520
|
+
catch (e) {
|
|
521
|
+
const { message, stack } = describeError(e);
|
|
522
|
+
await this.send({
|
|
523
|
+
key: `startup:after-listen:${failureFingerprintOf(message, stack)}`,
|
|
524
|
+
subject: `Started, but a startup step failed: ${emergencyOneLine(message, 120)}`,
|
|
525
|
+
lines: [message, '', ...stack],
|
|
526
|
+
});
|
|
527
|
+
}
|
|
528
|
+
}
|
|
529
|
+
if (opts.config !== undefined) {
|
|
530
|
+
await this.checkConfig(opts.config, opts.critical ?? []);
|
|
531
|
+
}
|
|
532
|
+
return OpResult.ok([{ status: 'listening', port }]);
|
|
533
|
+
}
|
|
405
534
|
/**
|
|
406
535
|
* Alerts on an uncaught exception or an unhandled rejection, then exits 1 — the process is in an unknown state after an
|
|
407
536
|
* uncaught error, and a restart is the recovery (what Node would have done without the handler). The exit waits for the
|
|
@@ -418,10 +547,7 @@ class EmergencyAlerter {
|
|
|
418
547
|
try {
|
|
419
548
|
const { message, stack } = describeError(error);
|
|
420
549
|
// ⚠ The key names WHICH crash (its message and top frame): one crash must not silence a different one for an hour.
|
|
421
|
-
const fingerprint =
|
|
422
|
-
.update(`${message}\n${stack[1] ?? ''}`)
|
|
423
|
-
.digest('hex')
|
|
424
|
-
.slice(0, 12);
|
|
550
|
+
const fingerprint = failureFingerprintOf(message, stack);
|
|
425
551
|
const alert = {
|
|
426
552
|
key: `crash:${kind}:${fingerprint}`,
|
|
427
553
|
subject: `Crashed (${kind}): ${emergencyOneLine(message, 120)}`,
|