@data-fair/lib-node 2.16.0 → 2.16.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +137 -0
- package/events-queue.d.ts +2 -2
- package/events-queue.js +9 -2
- package/events-queue.js.map +1 -1
- package/events-queue.test.js +22 -0
- package/events-queue.test.js.map +1 -1
- package/package.json +1 -2
package/README.md
ADDED
|
@@ -0,0 +1,137 @@
|
|
|
1
|
+
# @data-fair/lib-node
|
|
2
|
+
|
|
3
|
+
Server-side Node.js utilities shared by the services of the data-fair stack (data-fair, processings, events, catalogs…): HTTP clients protected against SSRF, Prometheus observability, MongoDB helpers, upgrade scripts, websocket emitter and client.
|
|
4
|
+
|
|
5
|
+
## Install
|
|
6
|
+
|
|
7
|
+
```sh
|
|
8
|
+
npm i @data-fair/lib-node
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
Peer dependencies, all declared optional: install the ones required by the modules you import.
|
|
12
|
+
|
|
13
|
+
| Peer dependency | Needed by |
|
|
14
|
+
| --- | --- |
|
|
15
|
+
| `prom-client@15` | `observer.js`, `http-agents.js` and everything built on them (`axios.js`, `events-queue.js`…) |
|
|
16
|
+
| `mongodb@6` | `mongo.js`, `locks.js`, `upgrade-scripts.js` |
|
|
17
|
+
| `ws@8` | `ws-client.js` |
|
|
18
|
+
| `tough-cookie@5` | `axios-with-cookies.js`, `axios-auth.js` |
|
|
19
|
+
|
|
20
|
+
There is no root entry point: import each module by its path.
|
|
21
|
+
|
|
22
|
+
## Modules
|
|
23
|
+
|
|
24
|
+
| Import path | Description |
|
|
25
|
+
| --- | --- |
|
|
26
|
+
| `@data-fair/lib-node/axios.js` | Axios instances with shorter error stack traces and a data-fair `User-Agent`: `axiosInstance` (default export, public), `privateAxiosInstance`, `axiosBuilder()`. |
|
|
27
|
+
| `@data-fair/lib-node/http-agents.js` | Keepalive HTTP(S) agents with cached DNS lookups and Prometheus metrics: `httpAgent` / `httpsAgent` (public, SSRF protected), `privateHttpAgent` / `privateHttpsAgent`. |
|
|
28
|
+
| `@data-fair/lib-node/ssrf.js` | SSRF checks for clients that don't use the agents above: `resolvePublicAddress()`, `publicLookup`, `SsrfError`. |
|
|
29
|
+
| `@data-fair/lib-node/axios-with-cookies.js` | Axios instance with a `tough-cookie` cookie jar. |
|
|
30
|
+
| `@data-fair/lib-node/axios-auth.js` | Axios instance logged into simple-directory with email/password. Integration tests only. |
|
|
31
|
+
| `@data-fair/lib-node/observer.js` | Mini web server exposing Prometheus metrics and CPU/heap profiles, plus the shared `internalError()` counter. |
|
|
32
|
+
| `@data-fair/lib-node/mongo.js` | `Mongo` connection wrapper (default export: a singleton) with declarative named index management (`configure()`). |
|
|
33
|
+
| `@data-fair/lib-node/locks.js` | Distributed locks stored in a MongoDB `locks` collection with a TTL (default export: a singleton). |
|
|
34
|
+
| `@data-fair/lib-node/upgrade-scripts.js` | Runner for the database migration scripts of a service. |
|
|
35
|
+
| `@data-fair/lib-node/events-queue.js` | Queue of events and notifications pushed to the events service. |
|
|
36
|
+
| `@data-fair/lib-node/session.js` | `SessionHandler`: reads and verifies simple-directory session cookies (JWKS). Used by `@data-fair/lib-express/session.js` and Nuxt server code. |
|
|
37
|
+
| `@data-fair/lib-node/ws-emitter.js` | Publishes websocket messages through a capped MongoDB collection, read by `@data-fair/lib-express/ws-server.js`. |
|
|
38
|
+
| `@data-fair/lib-node/ws-client.js` | Websocket client with auto-reconnect and channel subscriptions (`WsClient`, `DataFairWsClient`). |
|
|
39
|
+
| `@data-fair/lib-node/fs.js` | `ensureDir()`, `emptyDir()`, `exists()`. |
|
|
40
|
+
| `@data-fair/lib-node/validation.js` | Re-export of `@data-fair/lib-validation`. |
|
|
41
|
+
| `@data-fair/lib-node/test-spies.js` | Test helpers: turns `// @test:spy(...)` comments into events that tests can `waitFor()`. |
|
|
42
|
+
|
|
43
|
+
For sessions see the [`data-fair-session` skill](https://github.com/data-fair/lib/tree/main/skills/data-fair-session), for websockets the [`data-fair-ws` skill](https://github.com/data-fair/lib/tree/main/skills/data-fair-ws).
|
|
44
|
+
|
|
45
|
+
## Outgoing HTTP requests and SSRF
|
|
46
|
+
|
|
47
|
+
The default axios instance and the public agents refuse to connect to non public addresses (loopback, private ranges, link-local including the cloud metadata service, etc.). Use them for every URL that a user can influence.
|
|
48
|
+
|
|
49
|
+
`privateAxiosInstance` and the private agents have no such restriction. Use them only for URLs that come from the configuration of the service (other services of the same infrastructure).
|
|
50
|
+
|
|
51
|
+
```ts
|
|
52
|
+
import axios, { privateAxiosInstance } from '@data-fair/lib-node/axios.js'
|
|
53
|
+
|
|
54
|
+
await axios.get(userProvidedUrl)
|
|
55
|
+
await privateAxiosInstance.get(config.privateDirectoryUrl + '/api/...')
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
- `SSRF_PUBLIC_IPS`: comma separated IPs / CIDR ranges to consider public, for example an intranet data source of a self-hosted instance, or `127.0.0.1,::1` in a development environment.
|
|
59
|
+
- `SSRF_PRIVATE_IPS`: comma separated IPs / CIDR ranges to consider private even if they are public addresses, for example the database servers.
|
|
60
|
+
|
|
61
|
+
If `HTTP_PROXY`, `HTTPS_PROXY` or `ALL_PROXY` (or their lowercase forms) is defined, the protection of the agents is disabled (they connect to the proxy, not to the target) and a warning is logged at startup.
|
|
62
|
+
|
|
63
|
+
For clients that don't use these agents (ftp, sftp, raw sockets, SDKs…), `@data-fair/lib-node/ssrf.js` exports `resolvePublicAddress(host)` (checks every address of the host and returns the one to connect to) and `publicLookup` (a `dns.lookup` compatible function for any `lookup` option). They apply the same rules and are not disabled by a proxy env variable. `publicLookup` is not called for IP literals: check those with `resolvePublicAddress`.
|
|
64
|
+
|
|
65
|
+
See the [`data-fair-security` skill](https://github.com/data-fair/lib/tree/main/skills/data-fair-security) for the stack-wide rules.
|
|
66
|
+
|
|
67
|
+
## Observer
|
|
68
|
+
|
|
69
|
+
A very light web server for observability, on port 9090 by default:
|
|
70
|
+
|
|
71
|
+
- `GET /metrics`: Prometheus metrics of this process,
|
|
72
|
+
- `GET /service-metrics` (or `/global-metrics`): metrics registered in `servicePromRegistry`,
|
|
73
|
+
- `GET /cpu-profile`, `/heap-profile`, `/heap-profile-all` (optional `?duration=` in ms, default 2000): sampled profiles,
|
|
74
|
+
- `GET /heap-snapshot`: full retained heap snapshot (blocks the event loop while V8 walks the heap).
|
|
75
|
+
|
|
76
|
+
```ts
|
|
77
|
+
import { startObserver, stopObserver, internalError } from '@data-fair/lib-node/observer.js'
|
|
78
|
+
|
|
79
|
+
await startObserver() // or startObserver(port)
|
|
80
|
+
// ...
|
|
81
|
+
await stopObserver()
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
`internalError(errorCode, error, ...optionalParams)` increments the shared `df_internal_error` counter and logs the error. Use it for anomalies that should trigger alerts, not for client errors:
|
|
85
|
+
|
|
86
|
+
```ts
|
|
87
|
+
internalError('http', 'failure while serving http request', err)
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
Process metrics are plain `prom-client` metrics. A service-level metric (its value depends on a shared state, like the database, not on the activity of the current process) is registered in `servicePromRegistry`:
|
|
91
|
+
|
|
92
|
+
```ts
|
|
93
|
+
import { Gauge } from 'prom-client'
|
|
94
|
+
import { servicePromRegistry } from '@data-fair/lib-node/observer.js'
|
|
95
|
+
|
|
96
|
+
new Gauge({
|
|
97
|
+
name: 'df_my_gauge',
|
|
98
|
+
help: '...',
|
|
99
|
+
registers: [servicePromRegistry],
|
|
100
|
+
async collect () {
|
|
101
|
+
this.set(await db.collection('my-collection').estimatedDocumentCount())
|
|
102
|
+
}
|
|
103
|
+
})
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
## Upgrade scripts
|
|
107
|
+
|
|
108
|
+
Scripts live in `upgrade/<last-released-version>/` directories of the service. The folder name must be the version of the **last release** of the service at the time the script is written, never an anticipated future version.
|
|
109
|
+
|
|
110
|
+
At startup the runner reads the version stored in the `services` Mongo collection, runs every script whose folder version is `>=` that stored version, then stores the current `package.json` version. So a script runs once on the production upgrade and re-runs on every staging deploy until the next release ships.
|
|
111
|
+
|
|
112
|
+
**Scripts must be idempotent**: each script should run once, but this cannot be strictly ensured (concurrent pods, crashes, manual re-runs, misnamed folders), so running it again must be a no-op, not a failure.
|
|
113
|
+
|
|
114
|
+
A script is a module whose default export follows the `UpgradeScript` interface:
|
|
115
|
+
|
|
116
|
+
```ts
|
|
117
|
+
import type { UpgradeScript } from '@data-fair/lib-node/upgrade-scripts.js'
|
|
118
|
+
|
|
119
|
+
export default {
|
|
120
|
+
description: 'Add a default status to items',
|
|
121
|
+
async exec (db, debug) {
|
|
122
|
+
await db.collection('items').updateMany({ status: { $exists: false } }, { $set: { status: 'active' } })
|
|
123
|
+
}
|
|
124
|
+
} as UpgradeScript
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
Run them during service initialization (the runner takes the `upgrade` lock, so only one process runs them):
|
|
128
|
+
|
|
129
|
+
```ts
|
|
130
|
+
import upgradeScripts from '@data-fair/lib-node/upgrade-scripts.js'
|
|
131
|
+
import locks from '@data-fair/lib-node/locks.js'
|
|
132
|
+
|
|
133
|
+
await locks.start(db)
|
|
134
|
+
await upgradeScripts(db, locks) // optional: basePath = './', isFresh callback
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
The [`upgrade-scripts` skill](https://github.com/data-fair/lib/tree/main/skills/upgrade-scripts) covers the folder-naming rule, idempotency patterns and fresh installs in detail.
|
package/events-queue.d.ts
CHANGED
|
@@ -2,8 +2,8 @@ import { type Event } from '@data-fair/lib-common-types/event/index.js';
|
|
|
2
2
|
import { type Notification } from '@data-fair/lib-common-types/notification/index.js';
|
|
3
3
|
import { SessionState } from '@data-fair/lib-common-types/session/index.js';
|
|
4
4
|
export type EventsQueueOptions = {
|
|
5
|
-
eventsUrl
|
|
6
|
-
eventsSecret
|
|
5
|
+
eventsUrl?: string | null;
|
|
6
|
+
eventsSecret?: string | null;
|
|
7
7
|
inactive?: boolean;
|
|
8
8
|
};
|
|
9
9
|
export type PushEvent = Omit<Event, 'date'>;
|
package/events-queue.js
CHANGED
|
@@ -17,8 +17,15 @@ export class EventsQueue {
|
|
|
17
17
|
start = async (options) => {
|
|
18
18
|
if (this._options)
|
|
19
19
|
return; // already initialized
|
|
20
|
-
|
|
21
|
-
|
|
20
|
+
// events is an optional service: without its url it is not deployed and events are dropped
|
|
21
|
+
let inactive = !!options.inactive || !options.eventsUrl;
|
|
22
|
+
if (!inactive && !options.eventsSecret) {
|
|
23
|
+
internalError('events-queue', 'the events service url is set but its secret is missing, events are dropped');
|
|
24
|
+
inactive = true;
|
|
25
|
+
}
|
|
26
|
+
this._options = { ...options, inactive };
|
|
27
|
+
if (!inactive)
|
|
28
|
+
this.loop();
|
|
22
29
|
};
|
|
23
30
|
stop = async () => {
|
|
24
31
|
this.stopped = true;
|
package/events-queue.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"events-queue.js","sourceRoot":"","sources":["events-queue.ts"],"names":[],"mappings":"AAEA,OAAO,EAAE,aAAa,EAAE,MAAM,iCAAiC,CAAA;AAC/D,uFAAuF;AACvF,OAAO,EAAE,oBAAoB,IAAI,KAAK,EAAE,MAAM,YAAY,CAAA;AAC1D,OAAO,KAAK,MAAM,OAAO,CAAA;AAGzB,MAAM,KAAK,GAAG,KAAK,CAAC,cAAc,CAAC,CAAA;AAWnC,MAAM,OAAO,WAAW;IACtB,QAAQ,CAAgC;IACxC,WAAW,GAAY,EAAE,CAAA;IACzB,kBAAkB,GAAmB,EAAE,CAAA;IACvC,OAAO,GAAG,KAAK,CAAA;IACf,mBAAmB,CAA2B;IAE9C,OAAO;QACL,IAAI,CAAC,IAAI,CAAC,QAAQ;YAAE,MAAM,IAAI,KAAK,CAAC,kCAAkC,CAAC,CAAA;QACvE,OAAO,IAAI,CAAC,QAAQ,CAAA;IACtB,CAAC;IAED,KAAK,GAAG,KAAK,EAAE,OAA2B,EAAE,EAAE;QAC5C,IAAI,IAAI,CAAC,QAAQ;YAAE,OAAM,CAAC,sBAAsB;QAChD,IAAI,CAAC,QAAQ,GAAG,OAAO,CAAA;
|
|
1
|
+
{"version":3,"file":"events-queue.js","sourceRoot":"","sources":["events-queue.ts"],"names":[],"mappings":"AAEA,OAAO,EAAE,aAAa,EAAE,MAAM,iCAAiC,CAAA;AAC/D,uFAAuF;AACvF,OAAO,EAAE,oBAAoB,IAAI,KAAK,EAAE,MAAM,YAAY,CAAA;AAC1D,OAAO,KAAK,MAAM,OAAO,CAAA;AAGzB,MAAM,KAAK,GAAG,KAAK,CAAC,cAAc,CAAC,CAAA;AAWnC,MAAM,OAAO,WAAW;IACtB,QAAQ,CAAgC;IACxC,WAAW,GAAY,EAAE,CAAA;IACzB,kBAAkB,GAAmB,EAAE,CAAA;IACvC,OAAO,GAAG,KAAK,CAAA;IACf,mBAAmB,CAA2B;IAE9C,OAAO;QACL,IAAI,CAAC,IAAI,CAAC,QAAQ;YAAE,MAAM,IAAI,KAAK,CAAC,kCAAkC,CAAC,CAAA;QACvE,OAAO,IAAI,CAAC,QAAQ,CAAA;IACtB,CAAC;IAED,KAAK,GAAG,KAAK,EAAE,OAA2B,EAAE,EAAE;QAC5C,IAAI,IAAI,CAAC,QAAQ;YAAE,OAAM,CAAC,sBAAsB;QAChD,2FAA2F;QAC3F,IAAI,QAAQ,GAAG,CAAC,CAAC,OAAO,CAAC,QAAQ,IAAI,CAAC,OAAO,CAAC,SAAS,CAAA;QACvD,IAAI,CAAC,QAAQ,IAAI,CAAC,OAAO,CAAC,YAAY,EAAE,CAAC;YACvC,aAAa,CAAC,cAAc,EAAE,6EAA6E,CAAC,CAAA;YAC5G,QAAQ,GAAG,IAAI,CAAA;QACjB,CAAC;QACD,IAAI,CAAC,QAAQ,GAAG,EAAE,GAAG,OAAO,EAAE,QAAQ,EAAE,CAAA;QACxC,IAAI,CAAC,QAAQ;YAAE,IAAI,CAAC,IAAI,EAAE,CAAA;IAC5B,CAAC,CAAA;IAED,IAAI,GAAG,KAAK,IAAI,EAAE;QAChB,IAAI,CAAC,OAAO,GAAG,IAAI,CAAA;QACnB,IAAI,IAAI,CAAC,mBAAmB;YAAE,MAAM,IAAI,CAAC,mBAAmB,CAAA;QAC5D,MAAM,IAAI,CAAC,KAAK,EAAE,CAAA;IACpB,CAAC,CAAA;IAED,KAAK,CAAC,IAAI;QACR,OAAO,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;YACrB,MAAM,IAAI,OAAO,CAAC,OAAO,CAAC,EAAE,CAAC,UAAU,CAAC,OAAO,EAAE,IAAI,CAAC,CAAC,CAAA;YACvD,IAAI,CAAC,mBAAmB,GAAG,IAAI,CAAC,KAAK,EAAE,CAAA;YACvC,MAAM,IAAI,CAAC,mBAAmB,CAAA;YAC9B,IAAI,CAAC,mBAAmB,GAAG,SAAS,CAAA;QACtC,CAAC;IACH,CAAC;IAED,KAAK,CAAC,KAAK;QACT,IAAI,IAAI,CAAC,WAAW,CAAC,MAAM,EAAE,CAAC;YAC5B,MAAM,MAAM,GAAG,IAAI,CAAC,WAAW,CAAA;YAC/B,IAAI,CAAC,WAAW,GAAG,EAAE,CAAA;YACrB,KAAK,CAAC,cAAc,EAAE,MAAM,CAAC,MAAM,CAAC,CAAA;YACpC,IAAI,CAAC;gBACH,MAAM,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC,SAAS,GAAG,aAAa,EAAE,MAAM,EAAE,EAAE,OAAO,EAAE,EAAE,cAAc,EAAE,IAAI,CAAC,OAAO,EAAE,CAAC,YAAY,EAAE,EAAE,CAAC,CAAA;YAClI,CAAC;YAAC,OAAO,GAAQ,EAAE,CAAC;gBAClB,aAAa,CAAC,yBAAyB,EAAE,GAAG,CAAC,CAAA;gBAC7C,cAAc;gBACd,IAAI,GAAG,CAAC,MAAM,IAAI,GAAG,IAAI,MAAM,CAAC,MAAM,GAAG,IAAI;oBAAE,IAAI,CAAC,WAAW,GAAG,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC,WAAW,CAAC,CAAA;YACnG,CAAC;QACH,CAAC;QACD,IAAI,IAAI,CAAC,kBAAkB,CAAC,MAAM,EAAE,CAAC;YACnC,MAAM,aAAa,GAAG,IAAI,CAAC,kBAAkB,CAAA;YAC7C,IAAI,CAAC,kBAAkB,GAAG,EAAE,CAAA;YAC5B,KAAK,CAAC,qBAAqB,EAAE,aAAa,CAAC,MAAM,CAAC,CAAA;YAClD,KAAK,MAAM,YAAY,IAAI,aAAa,EAAE,CAAC;gBACzC,IAAI,CAAC;oBACH,MAAM,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC,SAAS,GAAG,oBAAoB,EAAE,YAAY,EAAE,EAAE,OAAO,EAAE,EAAE,cAAc,EAAE,IAAI,CAAC,OAAO,EAAE,CAAC,YAAY,EAAE,EAAE,CAAC,CAAA;gBAC/I,CAAC;gBAAC,OAAO,GAAQ,EAAE,CAAC;oBAClB,aAAa,CAAC,yBAAyB,EAAE,GAAG,CAAC,CAAA;oBAC7C,cAAc;oBACd,IAAI,GAAG,CAAC,MAAM,IAAI,GAAG,IAAI,CAAC,aAAa,CAAC,MAAM,GAAG,IAAI,CAAC,kBAAkB,CAAC,MAAM,CAAC,GAAG,IAAI,EAAE,CAAC;wBACxF,IAAI,CAAC,kBAAkB,CAAC,OAAO,CAAC,YAAY,CAAC,CAAA;oBAC/C,CAAC;gBACH,CAAC;YACH,CAAC;QACH,CAAC;IACH,CAAC;IAED,SAAS,CAAE,KAAgB,EAAE,YAA2B;QACtD,MAAM,OAAO,GAAG,IAAI,CAAC,OAAO,EAAE,CAAA;QAC9B,IAAI,OAAO,CAAC,QAAQ;YAAE,OAAM;QAC5B,IAAI,IAAI,CAAC,OAAO;YAAE,MAAM,IAAI,KAAK,CAAC,+BAA+B,CAAC,CAAC;QAClE,KAAe,CAAC,IAAI,GAAG,IAAI,IAAI,EAAE,CAAC,WAAW,EAAE,CAAA;QAChD,KAAK,CAAC,WAAW,EAAE,KAAK,CAAC,CAAA;QACzB,IAAI,CAAC,KAAK,CAAC,UAAU,IAAI,YAAY,EAAE,IAAI,IAAI,YAAY,CAAC,OAAO,EAAE,CAAC;YACpE,IAAI,YAAY,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;gBAC9B,KAAK,CAAC,UAAU,GAAG;oBACjB,IAAI,EAAE,EAAE,GAAG,YAAY,CAAC,IAAI,CAAC,OAAO,EAAE,KAAK,EAAE,IAAI,EAAE;iBACpD,CAAA;YACH,CAAC;iBAAM,IAAI,YAAY,CAAC,IAAI,CAAC,SAAS,IAAI,CAAC,YAAY,CAAC,OAAO,CAAC,IAAI,KAAK,KAAK,CAAC,MAAM,EAAE,IAAI,IAAI,YAAY,CAAC,OAAO,CAAC,EAAE,KAAK,KAAK,CAAC,MAAM,EAAE,EAAE,CAAC,EAAE,CAAC;gBAC7I,KAAK,CAAC,UAAU,GAAG;oBACjB,IAAI,EAAE,EAAE,EAAE,EAAE,YAAY,CAAC,IAAI,CAAC,EAAE,EAAE,IAAI,EAAE,YAAY,CAAC,IAAI,CAAC,IAAI,EAAE,KAAK,EAAE,IAAI,EAAE;iBAC9E,CAAA;YACH,CAAC;iBAAM,CAAC;gBACN,KAAK,CAAC,UAAU,GAAG;oBACjB,IAAI,EAAE,EAAE,EAAE,EAAE,YAAY,CAAC,IAAI,CAAC,EAAE,EAAE,IAAI,EAAE,YAAY,CAAC,IAAI,CAAC,IAAI,EAAE;iBACjE,CAAA;gBACD,IAAI,YAAY,CAAC,YAAY,EAAE,CAAC;oBAC9B,KAAK,CAAC,UAAU,CAAC,YAAY,GAAG;wBAC9B,EAAE,EAAE,YAAY,CAAC,YAAY,CAAC,EAAE;wBAChC,IAAI,EAAE,YAAY,CAAC,YAAY,CAAC,IAAI;wBACpC,UAAU,EAAE,YAAY,CAAC,YAAY,CAAC,UAAU;wBAChD,cAAc,EAAE,YAAY,CAAC,YAAY,CAAC,cAAc;qBACzD,CAAA;gBACH,CAAC;YACH,CAAC;QACH,CAAC;QACD,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,KAAc,CAAC,CAAA;IACvC,CAAC;IAED,gBAAgB,CAAE,YAA8B;QAC9C,MAAM,OAAO,GAAG,IAAI,CAAC,OAAO,EAAE,CAAA;QAC9B,IAAI,OAAO,CAAC,QAAQ;YAAE,OAAM;QAC5B,IAAI,IAAI,CAAC,OAAO;YAAE,MAAM,IAAI,KAAK,CAAC,sCAAsC,CAAC,CAAC;QACzE,YAA6B,CAAC,IAAI,GAAG,IAAI,IAAI,EAAE,CAAC,WAAW,EAAE,CAAA;QAC9D,KAAK,CAAC,kBAAkB,EAAE,YAAY,CAAC,CAAA;QACvC,IAAI,CAAC,kBAAkB,CAAC,IAAI,CAAC,YAA4B,CAAC,CAAA;IAC5D,CAAC;CACF;AAED,MAAM,WAAW,GAAG,IAAI,WAAW,EAAE,CAAA;AAErC,eAAe,WAAW,CAAA","sourcesContent":["import { type Event } from '@data-fair/lib-common-types/event/index.js'\nimport { type Notification } from '@data-fair/lib-common-types/notification/index.js'\nimport { internalError } from '@data-fair/lib-node/observer.js'\n// the events service is part of our own infrastructure (configuration, not user input)\nimport { privateAxiosInstance as axios } from './axios.js'\nimport Debug from 'debug'\nimport { SessionState } from '@data-fair/lib-common-types/session/index.js'\n\nconst debug = Debug('events-queue')\n\nexport type EventsQueueOptions = {\n eventsUrl?: string | null,\n eventsSecret?: string | null,\n inactive?: boolean\n}\n\nexport type PushEvent = Omit<Event, 'date'>\nexport type PushNotification = Omit<Notification, 'date'>\n\nexport class EventsQueue {\n _options: EventsQueueOptions | undefined\n eventsQueue: Event[] = []\n notificationsQueue: Notification[] = []\n stopped = false\n currentDrainPromise: Promise<void> | undefined\n\n options () {\n if (!this._options) throw new Error('events queue was not initialized')\n return this._options\n }\n\n start = async (options: EventsQueueOptions) => {\n if (this._options) return // already initialized\n // events is an optional service: without its url it is not deployed and events are dropped\n let inactive = !!options.inactive || !options.eventsUrl\n if (!inactive && !options.eventsSecret) {\n internalError('events-queue', 'the events service url is set but its secret is missing, events are dropped')\n inactive = true\n }\n this._options = { ...options, inactive }\n if (!inactive) this.loop()\n }\n\n stop = async () => {\n this.stopped = true\n if (this.currentDrainPromise) await this.currentDrainPromise\n await this.drain()\n }\n\n async loop () {\n while (!this.stopped) {\n await new Promise(resolve => setTimeout(resolve, 3000))\n this.currentDrainPromise = this.drain()\n await this.currentDrainPromise\n this.currentDrainPromise = undefined\n }\n }\n\n async drain () {\n if (this.eventsQueue.length) {\n const events = this.eventsQueue\n this.eventsQueue = []\n debug('drain events', events.length)\n try {\n await axios.post(this.options().eventsUrl + '/api/events', events, { headers: { 'x-secret-key': this.options().eventsSecret } })\n } catch (err: any) {\n internalError('events-queue-push-event', err)\n // retry later\n if (err.status >= 500 && events.length < 1000) this.eventsQueue = events.concat(this.eventsQueue)\n }\n }\n if (this.notificationsQueue.length) {\n const notifications = this.notificationsQueue\n this.notificationsQueue = []\n debug('drain notifications', notifications.length)\n for (const notification of notifications) {\n try {\n await axios.post(this.options().eventsUrl + '/api/notifications', notification, { headers: { 'x-secret-key': this.options().eventsSecret } })\n } catch (err: any) {\n internalError('events-queue-push-notif', err)\n // retry later\n if (err.status >= 500 && (notifications.length + this.notificationsQueue.length) < 1000) {\n this.notificationsQueue.unshift(notification)\n }\n }\n }\n }\n }\n\n pushEvent (event: PushEvent, sessionState?: SessionState) {\n const options = this.options()\n if (options.inactive) return\n if (this.stopped) throw new Error('events queue has been stopped');\n (event as Event).date = new Date().toISOString()\n debug('pushEvent', event)\n if (!event.originator && sessionState?.user && sessionState.account) {\n if (sessionState.user.asAdmin) {\n event.originator = {\n user: { ...sessionState.user.asAdmin, admin: true }\n }\n } else if (sessionState.user.adminMode && (sessionState.account.type !== event.sender?.type || sessionState.account.id !== event.sender?.id)) {\n event.originator = {\n user: { id: sessionState.user.id, name: sessionState.user.name, admin: true }\n }\n } else {\n event.originator = {\n user: { id: sessionState.user.id, name: sessionState.user.name }\n }\n if (sessionState.organization) {\n event.originator.organization = {\n id: sessionState.organization.id,\n name: sessionState.organization.name,\n department: sessionState.organization.department,\n departmentName: sessionState.organization.departmentName\n }\n }\n }\n }\n this.eventsQueue.push(event as Event)\n }\n\n pushNotification (notification: PushNotification) {\n const options = this.options()\n if (options.inactive) return\n if (this.stopped) throw new Error('notifications queue has been stopped');\n (notification as Notification).date = new Date().toISOString()\n debug('pushNotification', notification)\n this.notificationsQueue.push(notification as Notification)\n }\n}\n\nconst eventsQueue = new EventsQueue()\n\nexport default eventsQueue\n"]}
|
package/events-queue.test.js
CHANGED
|
@@ -31,5 +31,27 @@ describe('events queue', () => {
|
|
|
31
31
|
assert.equal(received[0].url, '/api/events');
|
|
32
32
|
assert.equal(received[0].body[0].title, 'test event');
|
|
33
33
|
});
|
|
34
|
+
const event = { title: 'dropped', topic: { key: 'test' }, sender: { type: 'user', id: 'user1' } };
|
|
35
|
+
it('should drop events silently when the events service has no url', async () => {
|
|
36
|
+
const queue = new EventsQueue();
|
|
37
|
+
await queue.start({ eventsUrl: '', eventsSecret: 'secret' });
|
|
38
|
+
assert.equal(queue.options().inactive, true);
|
|
39
|
+
queue.pushEvent(event);
|
|
40
|
+
queue.pushNotification({ title: 'dropped', topic: { key: 'test' } });
|
|
41
|
+
assert.equal(queue.eventsQueue.length, 0);
|
|
42
|
+
assert.equal(queue.notificationsQueue.length, 0);
|
|
43
|
+
await queue.stop();
|
|
44
|
+
});
|
|
45
|
+
it('should drop events when the url is set but the secret is missing', async () => {
|
|
46
|
+
const queue = new EventsQueue();
|
|
47
|
+
await queue.start({ eventsUrl: `http://127.0.0.1:${port}` });
|
|
48
|
+
assert.equal(queue.options().inactive, true);
|
|
49
|
+
queue.pushEvent(event);
|
|
50
|
+
assert.equal(queue.eventsQueue.length, 0);
|
|
51
|
+
await queue.stop();
|
|
52
|
+
});
|
|
53
|
+
it('should still refuse events before start', () => {
|
|
54
|
+
assert.throws(() => new EventsQueue().pushEvent(event), /not initialized/);
|
|
55
|
+
});
|
|
34
56
|
});
|
|
35
57
|
//# sourceMappingURL=events-queue.test.js.map
|
package/events-queue.test.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"events-queue.test.js","sourceRoot":"","sources":["events-queue.test.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,EAAE,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,WAAW,CAAA;AACvD,OAAO,EAAE,MAAM,IAAI,MAAM,EAAE,MAAM,QAAQ,CAAA;AACzC,OAAO,IAAI,MAAM,WAAW,CAAA;AAE5B,OAAO,EAAE,WAAW,EAAE,MAAM,mBAAmB,CAAA;AAE/C,QAAQ,CAAC,cAAc,EAAE,GAAG,EAAE;IAC5B,IAAI,MAAmB,CAAA;IACvB,IAAI,IAAY,CAAA;IAChB,MAAM,QAAQ,GAAkC,EAAE,CAAA;IAElD,MAAM,CAAC,KAAK,IAAI,EAAE;QAChB,MAAM,GAAG,IAAI,CAAC,YAAY,CAAC,CAAC,GAAG,EAAE,GAAG,EAAE,EAAE;YACtC,IAAI,IAAI,GAAG,EAAE,CAAA;YACb,GAAG,CAAC,EAAE,CAAC,MAAM,EAAE,KAAK,CAAC,EAAE,GAAG,IAAI,IAAI,KAAK,CAAA,CAAC,CAAC,CAAC,CAAA;YAC1C,GAAG,CAAC,EAAE,CAAC,KAAK,EAAE,GAAG,EAAE;gBACjB,QAAQ,CAAC,IAAI,CAAC,EAAE,GAAG,EAAE,GAAG,CAAC,GAAG,EAAE,IAAI,EAAE,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC,CAAA;gBACvD,GAAG,CAAC,GAAG,EAAE,CAAA;YACX,CAAC,CAAC,CAAA;QACJ,CAAC,CAAC,CAAA;QACF,MAAM,IAAI,OAAO,CAAO,OAAO,CAAC,EAAE,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,EAAE,WAAW,EAAE,OAAO,CAAC,CAAC,CAAA;QAC1E,IAAI,GAAI,MAAM,CAAC,OAAO,EAAkB,CAAC,IAAI,CAAA;IAC/C,CAAC,CAAC,CAAA;IAEF,KAAK,CAAC,GAAG,EAAE,GAAG,MAAM,CAAC,KAAK,EAAE,CAAA,CAAC,CAAC,CAAC,CAAA;IAE/B,mGAAmG;IACnG,qEAAqE;IACrE,EAAE,CAAC,gDAAgD,EAAE,KAAK,IAAI,EAAE;QAC9D,MAAM,CAAC,EAAE,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,eAAe,EAAE,2CAA2C,CAAC,CAAA;QACpF,MAAM,KAAK,GAAG,IAAI,WAAW,EAAE,CAAA;QAC/B,KAAK,CAAC,QAAQ,GAAG,EAAE,SAAS,EAAE,oBAAoB,IAAI,EAAE,EAAE,YAAY,EAAE,QAAQ,EAAE,CAAA;QAClF,KAAK,CAAC,SAAS,CAAC,EAAE,KAAK,EAAE,YAAY,EAAE,KAAK,EAAE,EAAE,GAAG,EAAE,MAAM,EAAE,EAAE,MAAM,EAAE,EAAE,IAAI,EAAE,MAAM,EAAE,EAAE,EAAE,OAAO,EAAE,EAAS,CAAC,CAAA;QAC9G,MAAM,KAAK,CAAC,KAAK,EAAE,CAAA;QACnB,MAAM,CAAC,KAAK,CAAC,QAAQ,CAAC,MAAM,EAAE,CAAC,CAAC,CAAA;QAChC,MAAM,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,GAAG,EAAE,aAAa,CAAC,CAAA;QAC5C,MAAM,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,KAAK,EAAE,YAAY,CAAC,CAAA;IACvD,CAAC,CAAC,CAAA;AACJ,CAAC,CAAC,CAAA","sourcesContent":["import { describe, it, before, after } from 'node:test'\nimport { strict as assert } from 'assert'\nimport http from 'node:http'\nimport type { AddressInfo } from 'node:net'\nimport { EventsQueue } from './events-queue.js'\n\ndescribe('events queue', () => {\n let server: http.Server\n let port: number\n const received: { url?: string, body: any }[] = []\n\n before(async () => {\n server = http.createServer((req, res) => {\n let body = ''\n req.on('data', chunk => { body += chunk })\n req.on('end', () => {\n received.push({ url: req.url, body: JSON.parse(body) })\n res.end()\n })\n })\n await new Promise<void>(resolve => server.listen(0, '127.0.0.1', resolve))\n port = (server.address() as AddressInfo).port\n })\n\n after(() => { server.close() })\n\n // the events service is configured, not user provided: it must stay reachable on a private address\n // even though the default http agents refuse those (SSRF protection)\n it('should push events to a private events service', async () => {\n assert.ok(!process.env.SSRF_PUBLIC_IPS, 'the test must run without SSRF_PUBLIC_IPS')\n const queue = new EventsQueue()\n queue._options = { eventsUrl: `http://127.0.0.1:${port}`, eventsSecret: 'secret' }\n queue.pushEvent({ title: 'test event', topic: { key: 'test' }, sender: { type: 'user', id: 'user1' } } as any)\n await queue.drain()\n assert.equal(received.length, 1)\n assert.equal(received[0].url, '/api/events')\n assert.equal(received[0].body[0].title, 'test event')\n })\n})\n"]}
|
|
1
|
+
{"version":3,"file":"events-queue.test.js","sourceRoot":"","sources":["events-queue.test.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,EAAE,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,WAAW,CAAA;AACvD,OAAO,EAAE,MAAM,IAAI,MAAM,EAAE,MAAM,QAAQ,CAAA;AACzC,OAAO,IAAI,MAAM,WAAW,CAAA;AAE5B,OAAO,EAAE,WAAW,EAAE,MAAM,mBAAmB,CAAA;AAE/C,QAAQ,CAAC,cAAc,EAAE,GAAG,EAAE;IAC5B,IAAI,MAAmB,CAAA;IACvB,IAAI,IAAY,CAAA;IAChB,MAAM,QAAQ,GAAkC,EAAE,CAAA;IAElD,MAAM,CAAC,KAAK,IAAI,EAAE;QAChB,MAAM,GAAG,IAAI,CAAC,YAAY,CAAC,CAAC,GAAG,EAAE,GAAG,EAAE,EAAE;YACtC,IAAI,IAAI,GAAG,EAAE,CAAA;YACb,GAAG,CAAC,EAAE,CAAC,MAAM,EAAE,KAAK,CAAC,EAAE,GAAG,IAAI,IAAI,KAAK,CAAA,CAAC,CAAC,CAAC,CAAA;YAC1C,GAAG,CAAC,EAAE,CAAC,KAAK,EAAE,GAAG,EAAE;gBACjB,QAAQ,CAAC,IAAI,CAAC,EAAE,GAAG,EAAE,GAAG,CAAC,GAAG,EAAE,IAAI,EAAE,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC,CAAA;gBACvD,GAAG,CAAC,GAAG,EAAE,CAAA;YACX,CAAC,CAAC,CAAA;QACJ,CAAC,CAAC,CAAA;QACF,MAAM,IAAI,OAAO,CAAO,OAAO,CAAC,EAAE,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,EAAE,WAAW,EAAE,OAAO,CAAC,CAAC,CAAA;QAC1E,IAAI,GAAI,MAAM,CAAC,OAAO,EAAkB,CAAC,IAAI,CAAA;IAC/C,CAAC,CAAC,CAAA;IAEF,KAAK,CAAC,GAAG,EAAE,GAAG,MAAM,CAAC,KAAK,EAAE,CAAA,CAAC,CAAC,CAAC,CAAA;IAE/B,mGAAmG;IACnG,qEAAqE;IACrE,EAAE,CAAC,gDAAgD,EAAE,KAAK,IAAI,EAAE;QAC9D,MAAM,CAAC,EAAE,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,eAAe,EAAE,2CAA2C,CAAC,CAAA;QACpF,MAAM,KAAK,GAAG,IAAI,WAAW,EAAE,CAAA;QAC/B,KAAK,CAAC,QAAQ,GAAG,EAAE,SAAS,EAAE,oBAAoB,IAAI,EAAE,EAAE,YAAY,EAAE,QAAQ,EAAE,CAAA;QAClF,KAAK,CAAC,SAAS,CAAC,EAAE,KAAK,EAAE,YAAY,EAAE,KAAK,EAAE,EAAE,GAAG,EAAE,MAAM,EAAE,EAAE,MAAM,EAAE,EAAE,IAAI,EAAE,MAAM,EAAE,EAAE,EAAE,OAAO,EAAE,EAAS,CAAC,CAAA;QAC9G,MAAM,KAAK,CAAC,KAAK,EAAE,CAAA;QACnB,MAAM,CAAC,KAAK,CAAC,QAAQ,CAAC,MAAM,EAAE,CAAC,CAAC,CAAA;QAChC,MAAM,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,GAAG,EAAE,aAAa,CAAC,CAAA;QAC5C,MAAM,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,KAAK,EAAE,YAAY,CAAC,CAAA;IACvD,CAAC,CAAC,CAAA;IAEF,MAAM,KAAK,GAAG,EAAE,KAAK,EAAE,SAAS,EAAE,KAAK,EAAE,EAAE,GAAG,EAAE,MAAM,EAAE,EAAE,MAAM,EAAE,EAAE,IAAI,EAAE,MAAM,EAAE,EAAE,EAAE,OAAO,EAAE,EAAS,CAAA;IAExG,EAAE,CAAC,gEAAgE,EAAE,KAAK,IAAI,EAAE;QAC9E,MAAM,KAAK,GAAG,IAAI,WAAW,EAAE,CAAA;QAC/B,MAAM,KAAK,CAAC,KAAK,CAAC,EAAE,SAAS,EAAE,EAAE,EAAE,YAAY,EAAE,QAAQ,EAAE,CAAC,CAAA;QAC5D,MAAM,CAAC,KAAK,CAAC,KAAK,CAAC,OAAO,EAAE,CAAC,QAAQ,EAAE,IAAI,CAAC,CAAA;QAC5C,KAAK,CAAC,SAAS,CAAC,KAAK,CAAC,CAAA;QACtB,KAAK,CAAC,gBAAgB,CAAC,EAAE,KAAK,EAAE,SAAS,EAAE,KAAK,EAAE,EAAE,GAAG,EAAE,MAAM,EAAE,EAAS,CAAC,CAAA;QAC3E,MAAM,CAAC,KAAK,CAAC,KAAK,CAAC,WAAW,CAAC,MAAM,EAAE,CAAC,CAAC,CAAA;QACzC,MAAM,CAAC,KAAK,CAAC,KAAK,CAAC,kBAAkB,CAAC,MAAM,EAAE,CAAC,CAAC,CAAA;QAChD,MAAM,KAAK,CAAC,IAAI,EAAE,CAAA;IACpB,CAAC,CAAC,CAAA;IAEF,EAAE,CAAC,kEAAkE,EAAE,KAAK,IAAI,EAAE;QAChF,MAAM,KAAK,GAAG,IAAI,WAAW,EAAE,CAAA;QAC/B,MAAM,KAAK,CAAC,KAAK,CAAC,EAAE,SAAS,EAAE,oBAAoB,IAAI,EAAE,EAAE,CAAC,CAAA;QAC5D,MAAM,CAAC,KAAK,CAAC,KAAK,CAAC,OAAO,EAAE,CAAC,QAAQ,EAAE,IAAI,CAAC,CAAA;QAC5C,KAAK,CAAC,SAAS,CAAC,KAAK,CAAC,CAAA;QACtB,MAAM,CAAC,KAAK,CAAC,KAAK,CAAC,WAAW,CAAC,MAAM,EAAE,CAAC,CAAC,CAAA;QACzC,MAAM,KAAK,CAAC,IAAI,EAAE,CAAA;IACpB,CAAC,CAAC,CAAA;IAEF,EAAE,CAAC,yCAAyC,EAAE,GAAG,EAAE;QACjD,MAAM,CAAC,MAAM,CAAC,GAAG,EAAE,CAAC,IAAI,WAAW,EAAE,CAAC,SAAS,CAAC,KAAK,CAAC,EAAE,iBAAiB,CAAC,CAAA;IAC5E,CAAC,CAAC,CAAA;AACJ,CAAC,CAAC,CAAA","sourcesContent":["import { describe, it, before, after } from 'node:test'\nimport { strict as assert } from 'assert'\nimport http from 'node:http'\nimport type { AddressInfo } from 'node:net'\nimport { EventsQueue } from './events-queue.js'\n\ndescribe('events queue', () => {\n let server: http.Server\n let port: number\n const received: { url?: string, body: any }[] = []\n\n before(async () => {\n server = http.createServer((req, res) => {\n let body = ''\n req.on('data', chunk => { body += chunk })\n req.on('end', () => {\n received.push({ url: req.url, body: JSON.parse(body) })\n res.end()\n })\n })\n await new Promise<void>(resolve => server.listen(0, '127.0.0.1', resolve))\n port = (server.address() as AddressInfo).port\n })\n\n after(() => { server.close() })\n\n // the events service is configured, not user provided: it must stay reachable on a private address\n // even though the default http agents refuse those (SSRF protection)\n it('should push events to a private events service', async () => {\n assert.ok(!process.env.SSRF_PUBLIC_IPS, 'the test must run without SSRF_PUBLIC_IPS')\n const queue = new EventsQueue()\n queue._options = { eventsUrl: `http://127.0.0.1:${port}`, eventsSecret: 'secret' }\n queue.pushEvent({ title: 'test event', topic: { key: 'test' }, sender: { type: 'user', id: 'user1' } } as any)\n await queue.drain()\n assert.equal(received.length, 1)\n assert.equal(received[0].url, '/api/events')\n assert.equal(received[0].body[0].title, 'test event')\n })\n\n const event = { title: 'dropped', topic: { key: 'test' }, sender: { type: 'user', id: 'user1' } } as any\n\n it('should drop events silently when the events service has no url', async () => {\n const queue = new EventsQueue()\n await queue.start({ eventsUrl: '', eventsSecret: 'secret' })\n assert.equal(queue.options().inactive, true)\n queue.pushEvent(event)\n queue.pushNotification({ title: 'dropped', topic: { key: 'test' } } as any)\n assert.equal(queue.eventsQueue.length, 0)\n assert.equal(queue.notificationsQueue.length, 0)\n await queue.stop()\n })\n\n it('should drop events when the url is set but the secret is missing', async () => {\n const queue = new EventsQueue()\n await queue.start({ eventsUrl: `http://127.0.0.1:${port}` })\n assert.equal(queue.options().inactive, true)\n queue.pushEvent(event)\n assert.equal(queue.eventsQueue.length, 0)\n await queue.stop()\n })\n\n it('should still refuse events before start', () => {\n assert.throws(() => new EventsQueue().pushEvent(event), /not initialized/)\n })\n})\n"]}
|