@winkjs/composer 0.7.0 → 0.8.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +9 -9
- package/package.json +8 -6
- package/src/core/emitter-manager/mqtt/constants.js +19 -3
- package/src/core/emitter-manager/mqtt/emitter.js +4 -3
- package/src/core/emitter-manager/mqtt/index.js +3 -1
- package/src/core/emitter-manager/mqtt/resolve-config.js +6 -6
- package/src/core/env-vars.js +20 -38
- package/src/core/storage-manager/questdb/index.js +11 -58
- package/src/core/storage-manager/questdb/resolve-options.js +21 -158
package/README.md
CHANGED
|
@@ -38,7 +38,7 @@ The Quick Start below builds one such pipeline end to end.
|
|
|
38
38
|
|
|
39
39
|
## Quick Start
|
|
40
40
|
|
|
41
|
-
Requires Node.js 22 or newer.
|
|
41
|
+
Requires Node.js 22 or newer. Each release is certified on the newest long-term support line of Node.js, 26 today, and tested on 22 and 24.
|
|
42
42
|
|
|
43
43
|
The Quick Start flow replays a CSV feed of pump motor temperatures. It cleans each reading, flags a hot motor, confirms the heat is not a blip, and prints an alert. One command scaffolds it as a runnable project:
|
|
44
44
|
|
|
@@ -106,15 +106,15 @@ The example above replays a CSV file. A live deployment reads from an MQTT broke
|
|
|
106
106
|
|
|
107
107
|
The numbers below come from a pure compute benchmark — every step a live message takes through an 8-node flow, from arrival to final output. **Storage and MQTT I/O are excluded** — these are compute ceilings, not full-deployment numbers.
|
|
108
108
|
|
|
109
|
-
| Configuration
|
|
110
|
-
|
|
111
|
-
|
|
|
112
|
-
| MacBook Pro (M4 Max)
|
|
113
|
-
| Tracking 10K assets
|
|
109
|
+
| Configuration | Throughput |
|
|
110
|
+
|-------------------------------|-------------------------|
|
|
111
|
+
| Revolution Pi Connect 5 (CM5) | 200K+ messages/second |
|
|
112
|
+
| MacBook Pro (M4 Max) | ~1.1M messages/second |
|
|
113
|
+
| Tracking 10K assets | ~500K messages/second |
|
|
114
114
|
|
|
115
|
-
|
|
115
|
+
The first row was measured on a Revolution Pi Connect 5, an industrial computer built on the Raspberry Pi Compute Module 5. The other two rows were measured on a MacBook Pro (M4 Max). All three ran under Node.js 26. Every figure is single-core.
|
|
116
116
|
|
|
117
|
-
Two configurations produced these numbers. The first two rows interleave 10 asset pipelines in random order — 4.5 million messages (10 pipelines × 900 data points × 500 iterations). The third row runs the same flow with 10,000 assets — 10,000 isolated states alive at once, 9 million messages,
|
|
117
|
+
Two configurations produced these numbers. The first two rows interleave 10 asset pipelines in random order — 4.5 million messages (10 pipelines × 900 data points × 500 iterations). The third row runs the same flow with 10,000 assets — 10,000 isolated states alive at once, 9 million messages, about 150 MB of heap. Pipelines are created dynamically as each asset first appears; timing uses `process.hrtime.bigint()`. The same pipeline [runs in your browser](https://composer.winkjs.org/docs/benchmark) — browser performance varies from native Node.js due to JIT differences.
|
|
118
118
|
|
|
119
119
|
Reproduce them from a repo clone. `node benchmark/compare.js 10 500` reports the median of three rounds. `node benchmark/run-benchmark.js static 10000 1` gives the 10K-asset row.
|
|
120
120
|
|
|
@@ -123,7 +123,7 @@ Reproduce them from a repo clone. `node benchmark/compare.js 10 500` reports the
|
|
|
123
123
|
|
|
124
124
|
Write a flow once and run it anywhere Node.js does — an industrial-grade Raspberry Pi, a production server, a Kubernetes cluster. Each asset's state is isolated, so a fault in one stays in one. Messages [queue locally](docs/handbook/resilience.md) when the network drops and drain cleanly on reconnect. Shutdown is ordered and deterministic: sources close first, storage last. A misconfigured flow fails when you define it, not in production — unknown options, output collisions, and bad triggers are all caught at definition time.
|
|
125
125
|
|
|
126
|
-
The test suite holds over
|
|
126
|
+
The test suite holds over 7,000 tests behind a 100% coverage gate on statements, branches, functions, and lines. Integration tests run against real Mosquitto and QuestDB services, not mocks. Every npm release carries SLSA provenance — a public, verifiable link from the package back to the exact source commit and the build that produced it. `npm audit signatures` verifies it.
|
|
127
127
|
|
|
128
128
|
The documentation also relates winkComposer's concepts to two industrial standards: [ISO 13374](https://composer.winkjs.org/docs/reference/iso-13374-mapping) for condition monitoring, the [NIST AI RMF](https://composer.winkjs.org/docs/reference/nist-ai-rmf-mapping) for AI risk management.
|
|
129
129
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@winkjs/composer",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.8.1",
|
|
4
4
|
"description": "Composable Streaming Intelligence — turn streaming data into actionable insights, from edge to cloud",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"streaming",
|
|
@@ -61,23 +61,25 @@
|
|
|
61
61
|
"access": "public"
|
|
62
62
|
},
|
|
63
63
|
"devDependencies": {
|
|
64
|
-
"c8": "^
|
|
64
|
+
"c8": "^12.0.0",
|
|
65
65
|
"chai": "^5.2.0",
|
|
66
66
|
"classic-level": "^3.0.0",
|
|
67
67
|
"eslint": "^9.21.0",
|
|
68
68
|
"eslint-plugin-security": "^4.0.0",
|
|
69
|
-
"mocha": "^
|
|
69
|
+
"mocha": "^12.0.3",
|
|
70
70
|
"sinon": "^19.0.2"
|
|
71
71
|
},
|
|
72
72
|
"dependencies": {
|
|
73
73
|
"@questdb/nodejs-client": "~4.2.0",
|
|
74
|
-
"glob": "^
|
|
75
|
-
"mqtt": "^5.
|
|
74
|
+
"glob": "^14.0.3",
|
|
75
|
+
"mqtt": "^5.16.1",
|
|
76
76
|
"msgpackr": "^1.11.10",
|
|
77
77
|
"pg": "^8.20.0"
|
|
78
78
|
},
|
|
79
79
|
"overrides": {
|
|
80
80
|
"serialize-javascript": "^7.1.0",
|
|
81
|
-
"diff": "^8.0.3"
|
|
81
|
+
"diff": "^8.0.3",
|
|
82
|
+
"minimatch": "^11.0.3",
|
|
83
|
+
"glob": "^14.0.3"
|
|
82
84
|
}
|
|
83
85
|
}
|
|
@@ -61,11 +61,27 @@ export const QUEUE_CRITICAL_THRESHOLD = 0.8;
|
|
|
61
61
|
*
|
|
62
62
|
* The emitter uses the client's UniqueMessageIdProvider, which never
|
|
63
63
|
* reissues an id that is in use. So the unacked window must never
|
|
64
|
-
* hold more messages than the id space has ids.
|
|
65
|
-
*
|
|
64
|
+
* hold more messages than the id space has ids. The id space alone
|
|
65
|
+
* would allow about 60,000.
|
|
66
|
+
*
|
|
67
|
+
* A second, tighter limit sets the value. libuv 1.52 and later (the
|
|
68
|
+
* version inside Node 24 and Node 26) refuses one socket write call
|
|
69
|
+
* that carries more than 1,048,576 buffer pieces. The call fails
|
|
70
|
+
* with `write EINVAL` and the connection drops. Source:
|
|
71
|
+
* `uv__check_before_write` in libuv `src/unix/stream.c`, added by
|
|
72
|
+
* libuv PR #5014. mqtt-packet corks the socket, so every publish
|
|
73
|
+
* accepted inside one event-loop turn leaves in ONE write call. One
|
|
74
|
+
* MQTT 5 publish with this emitter's properties is 29 pieces. The
|
|
75
|
+
* accepted window is 0.9 × maxQueueSize, and the whole window must
|
|
76
|
+
* fit one call: 33,000 × 0.9 × 29 = 861,300 pieces, 82% of the
|
|
77
|
+
* limit. Even the full cap fits: 33,000 × 29 = 957,000, 91%.
|
|
78
|
+
* mqtt.js resends a stored backlog one packet per acknowledgement,
|
|
79
|
+
* but it releases the publishes queued during that resend in one
|
|
80
|
+
* turn, so the window bounds that path too. The constant keeps its
|
|
81
|
+
* original name. The id space is still the first of its two reasons.
|
|
66
82
|
* @type {number}
|
|
67
83
|
*/
|
|
68
|
-
export const MQTT_INFLIGHT_ID_LIMIT =
|
|
84
|
+
export const MQTT_INFLIGHT_ID_LIMIT = 33000;
|
|
69
85
|
|
|
70
86
|
// Message expiry intervals (seconds), indexed by the caller's
|
|
71
87
|
// `options.type`. A dictionary indexed by an external key has no
|
|
@@ -105,9 +105,10 @@
|
|
|
105
105
|
* - Pre-flight reject fires at `pressure >= STORAGE_PRESSURE_LIMIT` (0.9),
|
|
106
106
|
* leaving 10% headroom for in-flight drain on shutdown and a clean band
|
|
107
107
|
* between the yellow-health threshold (0.66) and exhaustion (1.0).
|
|
108
|
-
* - The cap is `maxQueueSize`, clamped to
|
|
109
|
-
*
|
|
110
|
-
*
|
|
108
|
+
* - The cap is `maxQueueSize`, clamped to MQTT_INFLIGHT_ID_LIMIT
|
|
109
|
+
* (33,000). Two limits set that ceiling: the 16-bit packet-id space,
|
|
110
|
+
* and the libuv cap on buffer pieces per socket write. constants.js
|
|
111
|
+
* carries both with their sources.
|
|
111
112
|
* - The old byte-axis limit (`maxQueueBytes`) died with the disk store;
|
|
112
113
|
* the count cap bounds memory at cap × payload size.
|
|
113
114
|
*
|
|
@@ -65,7 +65,9 @@
|
|
|
65
65
|
*
|
|
66
66
|
* `brokerUrl` falls back to `MQTT_BROKER_URL`. The reconnect period
|
|
67
67
|
* gains a random share of up to 20% per client (`utils/jitter`).
|
|
68
|
-
* The cap is clamped to
|
|
68
|
+
* The cap is clamped to 33,000, the in-flight ceiling. constants.js
|
|
69
|
+
* carries its two reasons: the packet-id space and the libuv write
|
|
70
|
+
* limit.
|
|
69
71
|
*
|
|
70
72
|
* Related decisions beyond ADR-018 and ADR-021. ADR-022: this adapter
|
|
71
73
|
* writes the `winkDedupId` the source dedups on. ADR-023: `publishNow`
|
|
@@ -213,11 +213,11 @@ const resolveGrace = function ( config ) {
|
|
|
213
213
|
}; // resolveGrace()
|
|
214
214
|
|
|
215
215
|
/**
|
|
216
|
-
* Resolves the unacked cap
|
|
217
|
-
*
|
|
218
|
-
*
|
|
219
|
-
*
|
|
220
|
-
*
|
|
216
|
+
* Resolves the unacked cap, clamped to MQTT_INFLIGHT_ID_LIMIT. Two
|
|
217
|
+
* limits set that ceiling: the 16-bit packet-id space, and the libuv
|
|
218
|
+
* cap on buffer pieces per socket write. constants.js carries both
|
|
219
|
+
* with their sources. The clamp prints the same warning the LevelDB
|
|
220
|
+
* store applied.
|
|
221
221
|
*
|
|
222
222
|
* @param {Object} config - emitter config
|
|
223
223
|
* @returns {number} the effective cap
|
|
@@ -227,7 +227,7 @@ const resolveQueueSize = function ( config ) {
|
|
|
227
227
|
if ( requestedQueueSize > MQTT_INFLIGHT_ID_LIMIT ) {
|
|
228
228
|
logger.warn(
|
|
229
229
|
`winkComposer/mqttEmitter: maxQueueSize ${requestedQueueSize} exceeds the MQTT ` +
|
|
230
|
-
`
|
|
230
|
+
`in-flight ceiling — clamped to ${MQTT_INFLIGHT_ID_LIMIT}`
|
|
231
231
|
);
|
|
232
232
|
}
|
|
233
233
|
return Math.min( requestedQueueSize, MQTT_INFLIGHT_ID_LIMIT );
|
package/src/core/env-vars.js
CHANGED
|
@@ -99,23 +99,7 @@ const ENV_VARS = {
|
|
|
99
99
|
// outright, so a deployment that sets it stops here, at import.
|
|
100
100
|
questdbIlpUrl: ( process.env.QUESTDB_ILP_URL ?? '127.0.0.1:9000' ).trim(),
|
|
101
101
|
questdbPgUrl: ( process.env.QUESTDB_PG_URL ?? '127.0.0.1:8812' ).trim(),
|
|
102
|
-
// The five legacy flush variables (deprecated, ADR-029; removed in
|
|
103
|
-
// 0.8.0). Each carries a value only when set, so the adapter's
|
|
104
|
-
// DEPRECATED_OPTION line names only what an operator actually set.
|
|
105
|
-
// QUESTDB_IDLE_FLUSH_CHECK_MS maps to QUESTDB_FLUSH_INTERVAL_MS and
|
|
106
|
-
// QUESTDB_AUTO_FLUSH_ROWS to QUESTDB_FLUSH_ROWS; the other three
|
|
107
|
-
// are accepted and ignored.
|
|
108
|
-
questdbFlushMode: process.env.QUESTDB_FLUSH_MODE ?
|
|
109
|
-
process.env.QUESTDB_FLUSH_MODE.trim() : undefined,
|
|
110
|
-
questdbIdleFlushAfterMs: process.env.QUESTDB_IDLE_FLUSH_AFTER_MS ?
|
|
111
|
-
parseInt( process.env.QUESTDB_IDLE_FLUSH_AFTER_MS, 10 ) : undefined,
|
|
112
|
-
questdbIdleFlushCheckMs: process.env.QUESTDB_IDLE_FLUSH_CHECK_MS ?
|
|
113
|
-
parseInt( process.env.QUESTDB_IDLE_FLUSH_CHECK_MS, 10 ) : undefined,
|
|
114
102
|
// Optional QuestDB settings (undefined if not set)
|
|
115
|
-
questdbAutoFlushRows: process.env.QUESTDB_AUTO_FLUSH_ROWS ?
|
|
116
|
-
parseInt( process.env.QUESTDB_AUTO_FLUSH_ROWS, 10 ) : undefined,
|
|
117
|
-
questdbAutoFlushIntervalMs: process.env.QUESTDB_AUTO_FLUSH_INTERVAL_MS ?
|
|
118
|
-
parseInt( process.env.QUESTDB_AUTO_FLUSH_INTERVAL_MS, 10 ) : undefined,
|
|
119
103
|
questdbMaxBufSize: process.env.QUESTDB_MAX_BUF_SIZE ?
|
|
120
104
|
parseInt( process.env.QUESTDB_MAX_BUF_SIZE, 10 ) : undefined,
|
|
121
105
|
questdbRetryTimeout: process.env.QUESTDB_RETRY_TIMEOUT ?
|
|
@@ -218,15 +202,6 @@ const validators = {
|
|
|
218
202
|
return null;
|
|
219
203
|
},
|
|
220
204
|
|
|
221
|
-
questdbFlushMode: function ( value ) {
|
|
222
|
-
if ( value === undefined ) return null;
|
|
223
|
-
const validModes = [ 'auto', 'manual' ];
|
|
224
|
-
if ( !validModes.includes( value ) ) {
|
|
225
|
-
return `Must be one of ${validModes.join( ', ' )}, got: "${value}"`;
|
|
226
|
-
}
|
|
227
|
-
return null;
|
|
228
|
-
},
|
|
229
|
-
|
|
230
205
|
questdbStdlibHttp: function ( value ) {
|
|
231
206
|
if ( value === undefined ) return null;
|
|
232
207
|
const validWords = [ 'on', 'off' ];
|
|
@@ -251,14 +226,6 @@ const validators = {
|
|
|
251
226
|
return null;
|
|
252
227
|
},
|
|
253
228
|
|
|
254
|
-
nonNegativeIntOrUndefined: function ( value, originalEnv ) {
|
|
255
|
-
if ( value === undefined ) return null;
|
|
256
|
-
if ( isNaN( value ) || value < 0 ) {
|
|
257
|
-
return `Must be non-negative integer, got: "${originalEnv}"`;
|
|
258
|
-
}
|
|
259
|
-
return null;
|
|
260
|
-
},
|
|
261
|
-
|
|
262
229
|
nonNegativeNumberOrInfinity: function ( value, originalEnv ) {
|
|
263
230
|
if ( isNaN( value ) || value < 0 ) {
|
|
264
231
|
return `Must be a non-negative number or Infinity, got: "${originalEnv}"`;
|
|
@@ -313,11 +280,6 @@ const validationConfig = [
|
|
|
313
280
|
// QuestDB Configuration
|
|
314
281
|
{ field: 'questdbIlpUrl', validator: validators.hostPort, label: 'QUESTDB_ILP_URL' },
|
|
315
282
|
{ field: 'questdbPgUrl', validator: validators.hostPort, label: 'QUESTDB_PG_URL' },
|
|
316
|
-
{ field: 'questdbFlushMode', validator: validators.questdbFlushMode, label: 'QUESTDB_FLUSH_MODE' },
|
|
317
|
-
{ field: 'questdbIdleFlushAfterMs', validator: validators.nonNegativeIntOrUndefined, originalEnv: 'QUESTDB_IDLE_FLUSH_AFTER_MS' },
|
|
318
|
-
{ field: 'questdbIdleFlushCheckMs', validator: validators.positiveIntOrUndefined, originalEnv: 'QUESTDB_IDLE_FLUSH_CHECK_MS' },
|
|
319
|
-
{ field: 'questdbAutoFlushRows', validator: validators.positiveIntOrUndefined, originalEnv: 'QUESTDB_AUTO_FLUSH_ROWS' },
|
|
320
|
-
{ field: 'questdbAutoFlushIntervalMs', validator: validators.positiveIntOrUndefined, originalEnv: 'QUESTDB_AUTO_FLUSH_INTERVAL_MS' },
|
|
321
283
|
{ field: 'questdbMaxBufSize', validator: validators.positiveIntOrUndefined, originalEnv: 'QUESTDB_MAX_BUF_SIZE' },
|
|
322
284
|
{ field: 'questdbRetryTimeout', validator: validators.positiveIntOrUndefined, originalEnv: 'QUESTDB_RETRY_TIMEOUT' },
|
|
323
285
|
{ field: 'questdbFlushRows', validator: validators.positiveIntOrUndefined, originalEnv: 'QUESTDB_FLUSH_ROWS' },
|
|
@@ -333,12 +295,32 @@ const validationConfig = [
|
|
|
333
295
|
// questdbPassword: no validation — allows empty for passwordless auth
|
|
334
296
|
];
|
|
335
297
|
|
|
298
|
+
// The five QuestDB flush variables 0.7.0 deprecated and 0.8.0 removed
|
|
299
|
+
// (ADR-029 item 10). A deployment that still sets one stops here, at
|
|
300
|
+
// import, and the failure line names what to do instead. Ignoring the
|
|
301
|
+
// variable would let an operator believe a setting took effect.
|
|
302
|
+
const REMOVED_ENV_VARS = [
|
|
303
|
+
{ name: 'QUESTDB_FLUSH_MODE', action: 'delete it, composer owns every flush' },
|
|
304
|
+
{ name: 'QUESTDB_IDLE_FLUSH_AFTER_MS', action: 'delete it, composer owns every flush' },
|
|
305
|
+
{ name: 'QUESTDB_IDLE_FLUSH_CHECK_MS', action: 'use QUESTDB_FLUSH_INTERVAL_MS' },
|
|
306
|
+
{ name: 'QUESTDB_AUTO_FLUSH_ROWS', action: 'use QUESTDB_FLUSH_ROWS' },
|
|
307
|
+
{ name: 'QUESTDB_AUTO_FLUSH_INTERVAL_MS', action: 'delete it, composer owns every flush' }
|
|
308
|
+
];
|
|
309
|
+
|
|
336
310
|
// ============================================================================
|
|
337
311
|
// VALIDATION RUNNER
|
|
338
312
|
// ============================================================================
|
|
339
313
|
const validate = function () {
|
|
340
314
|
const errors = [];
|
|
341
315
|
|
|
316
|
+
// A removed variable is refused by name, even when empty, so a
|
|
317
|
+
// leftover line in an env file is found.
|
|
318
|
+
for ( const removed of REMOVED_ENV_VARS ) {
|
|
319
|
+
if ( process.env[ removed.name ] !== undefined ) {
|
|
320
|
+
errors.push( `${removed.name}: Removed in 0.8.0; ${removed.action}` );
|
|
321
|
+
}
|
|
322
|
+
}
|
|
323
|
+
|
|
342
324
|
// Run each validator
|
|
343
325
|
for ( const config of validationConfig ) {
|
|
344
326
|
const value = ENV_VARS[ config.field ];
|
|
@@ -116,12 +116,13 @@
|
|
|
116
116
|
* `consecutiveFlushFailures`, `lastFlushAt`, and `lastFlushError`.
|
|
117
117
|
* `flush-engine.js` documents the derivation.
|
|
118
118
|
*
|
|
119
|
-
*
|
|
120
|
-
*
|
|
121
|
-
*
|
|
122
|
-
*
|
|
123
|
-
* variables
|
|
124
|
-
*
|
|
119
|
+
* The five legacy flush keys of 0.7.0 (`flushMode`, `idleFlushAfterMs`,
|
|
120
|
+
* `idleFlushCheckMs`, `autoFlushRows`, `autoFlushIntervalMs`) are gone
|
|
121
|
+
* since 0.8.0 (ADR-029 item 10). The schema refuses each one as an
|
|
122
|
+
* unknown key at flow definition, with `INVALID_CONFIG`. `env-vars.js`
|
|
123
|
+
* refuses their `QUESTDB_*` variables at import. `autoFlushRows`
|
|
124
|
+
* became `flushRows` and `idleFlushCheckMs` became `flushIntervalMs`.
|
|
125
|
+
* The other three have no replacement: composer owns every flush.
|
|
125
126
|
*
|
|
126
127
|
* `err.code` vocabulary (per-adapter; ADR-018 has each adapter document
|
|
127
128
|
* its own codes in this header):
|
|
@@ -258,8 +259,6 @@
|
|
|
258
259
|
* interval. Remediation: the endpoint is not taking rows; read the
|
|
259
260
|
* `CIRCUIT_OPEN` and `DELIVERY_HEALTH` lines beside it, and raise
|
|
260
261
|
* `bufferCeilingRows` when a longer outage must be ridden through.
|
|
261
|
-
* - `DEPRECATED_OPTION` — one `logger.warn` line at setup naming the
|
|
262
|
-
* legacy keys in use (see Deprecated options above).
|
|
263
262
|
* - `ADDRESS_IS_NAME` — `ilpUrl` or `pgUrl` is a name other than
|
|
264
263
|
* `localhost`. One `logger.warn` line per field at setup, before any
|
|
265
264
|
* socket opens (ADR-030). A name is allowed and the adapter
|
|
@@ -356,13 +355,12 @@ import { Sender } from '@questdb/nodejs-client';
|
|
|
356
355
|
import pg from 'pg';
|
|
357
356
|
|
|
358
357
|
import { ENV_VARS } from '../../env-vars.js';
|
|
359
|
-
import { logger } from '../../logger/index.js';
|
|
360
358
|
import { validators } from '../../utils/validate/index.js';
|
|
361
359
|
import { probeAddress, describeProbe } from '../../utils/address/probe.js';
|
|
362
360
|
import { buildPersistPlans } from './persist-plan.js';
|
|
363
361
|
import { ensureTables } from './ensure-tables.js';
|
|
364
362
|
import { assertColumnFacts } from './assert-columns.js';
|
|
365
|
-
import { resolveOptions
|
|
363
|
+
import { resolveOptions } from './resolve-options.js';
|
|
366
364
|
import {
|
|
367
365
|
isAllowedIlpUrl,
|
|
368
366
|
isAllowedPgUrl,
|
|
@@ -505,9 +503,8 @@ const openTransport = async function ( { SenderClass, senderConfig, ilpUrl, stdl
|
|
|
505
503
|
* Create QuestDB storage adapter.
|
|
506
504
|
*
|
|
507
505
|
* The options are resolved by `resolve-options.js` (config over
|
|
508
|
-
* environment over default
|
|
509
|
-
*
|
|
510
|
-
* the rest.
|
|
506
|
+
* environment over default). Only the keys that reach the engine are
|
|
507
|
+
* listed here; the resolver documents the rest.
|
|
511
508
|
*
|
|
512
509
|
* @param {Object} assetClass - Asset class definition with columns and insightTypes
|
|
513
510
|
* @param {string} tablePrefix - Prefix for table names (typically assetClass.name)
|
|
@@ -536,17 +533,12 @@ const openTransport = async function ( { SenderClass, senderConfig, ilpUrl, stdl
|
|
|
536
533
|
const createQuestDBStorage = async function ( assetClass, tablePrefix, options, deps = {} ) {
|
|
537
534
|
// Options become settings here, once (ADR-029). A ceiling below the
|
|
538
535
|
// threshold fails setup inside the resolver with INVALID_CONFIG.
|
|
539
|
-
const
|
|
536
|
+
const settings = resolveOptions( options, ENV_VARS );
|
|
540
537
|
const {
|
|
541
538
|
ilpUrl, pgUrl, stdlibHttp, requestTimeout, retryTimeout, initBufSize, maxBufSize,
|
|
542
539
|
partitionBy, onWarning, onDeliveryFailure
|
|
543
540
|
} = settings;
|
|
544
541
|
|
|
545
|
-
// One line names every legacy key in use and what happened to it.
|
|
546
|
-
if ( deprecations.length > 0 ) {
|
|
547
|
-
logger.warn( deprecationMessage( deprecations ) );
|
|
548
|
-
}
|
|
549
|
-
|
|
550
542
|
// Runtime validation — required from either DSL config or ENV_VARS.
|
|
551
543
|
// Per ADR-018, setup-time throws carry classified err.code.
|
|
552
544
|
if ( !ilpUrl ) {
|
|
@@ -716,11 +708,6 @@ const configSchema = {
|
|
|
716
708
|
'ilpUrl',
|
|
717
709
|
'pgUrl',
|
|
718
710
|
'tablePrefix',
|
|
719
|
-
'flushMode',
|
|
720
|
-
'idleFlushAfterMs',
|
|
721
|
-
'idleFlushCheckMs',
|
|
722
|
-
'autoFlushRows',
|
|
723
|
-
'autoFlushIntervalMs',
|
|
724
711
|
'flushRows',
|
|
725
712
|
'flushIntervalMs',
|
|
726
713
|
'bufferCeilingRows',
|
|
@@ -760,40 +747,6 @@ const configSchema = {
|
|
|
760
747
|
error: 'tablePrefix must use letters, digits, _ and $ only, and not start with a digit ' +
|
|
761
748
|
'(defaults to assetClass.name when omitted)'
|
|
762
749
|
},
|
|
763
|
-
// The five legacy keys (deprecated, ADR-029; removed in 0.8.0). They
|
|
764
|
-
// keep their old validation so an existing flow still passes the
|
|
765
|
-
// schema. The option resolver maps or ignores them and setup prints
|
|
766
|
-
// one DEPRECATED_OPTION line.
|
|
767
|
-
flushMode: {
|
|
768
|
-
type: 'string',
|
|
769
|
-
required: false,
|
|
770
|
-
validator: validators.oneOf( [ 'auto', 'manual' ] ),
|
|
771
|
-
error: 'flushMode must be "auto" or "manual" (deprecated: composer owns every flush)'
|
|
772
|
-
},
|
|
773
|
-
idleFlushAfterMs: {
|
|
774
|
-
type: 'number',
|
|
775
|
-
required: false,
|
|
776
|
-
validator: validators.positiveInteger,
|
|
777
|
-
error: 'idleFlushAfterMs must be a positive integer (deprecated: ignored)'
|
|
778
|
-
},
|
|
779
|
-
idleFlushCheckMs: {
|
|
780
|
-
type: 'number',
|
|
781
|
-
required: false,
|
|
782
|
-
validator: validators.positiveInteger,
|
|
783
|
-
error: 'idleFlushCheckMs must be a positive integer (deprecated: use flushIntervalMs)'
|
|
784
|
-
},
|
|
785
|
-
autoFlushRows: {
|
|
786
|
-
type: 'number',
|
|
787
|
-
required: false,
|
|
788
|
-
validator: validators.positiveInteger,
|
|
789
|
-
error: 'autoFlushRows must be a positive integer (deprecated: use flushRows)'
|
|
790
|
-
},
|
|
791
|
-
autoFlushIntervalMs: {
|
|
792
|
-
type: 'number',
|
|
793
|
-
required: false,
|
|
794
|
-
validator: validators.positiveInteger,
|
|
795
|
-
error: 'autoFlushIntervalMs must be a positive integer (deprecated: ignored)'
|
|
796
|
-
},
|
|
797
750
|
// The flush settings composer owns (ADR-029). The relation between
|
|
798
751
|
// the ceiling and the threshold is checked by the option resolver
|
|
799
752
|
// at setup, where both values are known.
|
|
@@ -36,23 +36,15 @@
|
|
|
36
36
|
* 50000-row catch-up send gets about 9 minutes. When the operator
|
|
37
37
|
* sets `flushDeadlineMs`, that fixed value is used for every flush.
|
|
38
38
|
*
|
|
39
|
-
* Precedence, highest first (ADR-018 §10
|
|
40
|
-
*
|
|
39
|
+
* Precedence, highest first (ADR-018 §10): the key in the config
|
|
40
|
+
* (`flushRows`), then the environment variable (`QUESTDB_FLUSH_ROWS`),
|
|
41
|
+
* then the default.
|
|
41
42
|
*
|
|
42
|
-
*
|
|
43
|
-
*
|
|
44
|
-
*
|
|
45
|
-
*
|
|
46
|
-
*
|
|
47
|
-
*
|
|
48
|
-
* Two legacy keys map to a new one: `autoFlushRows` to `flushRows`, and
|
|
49
|
-
* `idleFlushCheckMs` to `flushIntervalMs`. The other three,
|
|
50
|
-
* `flushMode`, `idleFlushAfterMs` and `autoFlushIntervalMs`, have no
|
|
51
|
-
* meaning once composer owns every flush (ADR-029). They are accepted
|
|
52
|
-
* and ignored. Every supplied legacy key is named in the deprecation
|
|
53
|
-
* report, with what happened to it, so the adapter can print one
|
|
54
|
-
* `DEPRECATED_OPTION` line at setup. All five keys are removed in
|
|
55
|
-
* 0.8.0.
|
|
43
|
+
* The five legacy keys of 0.7.0 (`flushMode`, `idleFlushAfterMs`,
|
|
44
|
+
* `idleFlushCheckMs`, `autoFlushRows`, `autoFlushIntervalMs`) are gone
|
|
45
|
+
* since 0.8.0 (ADR-029 item 10). The adapter's schema refuses each one
|
|
46
|
+
* as an unknown key, and `env-vars.js` refuses their `QUESTDB_*`
|
|
47
|
+
* variables at import. Nothing here looks for them.
|
|
56
48
|
*
|
|
57
49
|
* The one relation enforced here: the ceiling is never below twice
|
|
58
50
|
* the threshold. Rows in flight count against the ceiling, so a
|
|
@@ -170,105 +162,10 @@ const CLIENT_MAX_RETRY_BACKOFF_MS = 1000;
|
|
|
170
162
|
*/
|
|
171
163
|
const FLUSH_DEADLINE_MARGIN_MS = 5000;
|
|
172
164
|
|
|
173
|
-
/**
|
|
174
|
-
* The release that removes the five legacy keys.
|
|
175
|
-
* @type {string}
|
|
176
|
-
*/
|
|
177
|
-
const REMOVAL_RELEASE = '0.8.0';
|
|
178
|
-
|
|
179
|
-
/**
|
|
180
|
-
* The five legacy keys in report order. `envField` is the property on
|
|
181
|
-
* the `ENV_VARS` object, `envVar` the variable name an operator sees.
|
|
182
|
-
* `target` names the new key a legacy value maps to; a key without one
|
|
183
|
-
* is accepted and ignored.
|
|
184
|
-
* @type {Array<{key: string, envField: string, envVar: string, target?: string}>}
|
|
185
|
-
*/
|
|
186
|
-
const LEGACY_KEYS = [
|
|
187
|
-
{ key: 'flushMode', envField: 'questdbFlushMode', envVar: 'QUESTDB_FLUSH_MODE' },
|
|
188
|
-
{ key: 'idleFlushAfterMs', envField: 'questdbIdleFlushAfterMs', envVar: 'QUESTDB_IDLE_FLUSH_AFTER_MS' },
|
|
189
|
-
{
|
|
190
|
-
key: 'idleFlushCheckMs',
|
|
191
|
-
envField: 'questdbIdleFlushCheckMs',
|
|
192
|
-
envVar: 'QUESTDB_IDLE_FLUSH_CHECK_MS',
|
|
193
|
-
target: 'flushIntervalMs'
|
|
194
|
-
},
|
|
195
|
-
{ key: 'autoFlushRows', envField: 'questdbAutoFlushRows', envVar: 'QUESTDB_AUTO_FLUSH_ROWS', target: 'flushRows' },
|
|
196
|
-
{ key: 'autoFlushIntervalMs', envField: 'questdbAutoFlushIntervalMs', envVar: 'QUESTDB_AUTO_FLUSH_INTERVAL_MS' }
|
|
197
|
-
];
|
|
198
|
-
|
|
199
165
|
// ============================================================================
|
|
200
166
|
// RESOLUTION
|
|
201
167
|
// ============================================================================
|
|
202
168
|
|
|
203
|
-
/**
|
|
204
|
-
* Resolves one setting that has a legacy alias, walking the five layers
|
|
205
|
-
* in precedence order. Returns the value and which layer supplied it,
|
|
206
|
-
* so the deprecation report can say whether a legacy value took effect.
|
|
207
|
-
*
|
|
208
|
-
* @param {Object} options - The raw storage options
|
|
209
|
-
* @param {Object} envVars - The `ENV_VARS` object
|
|
210
|
-
* @param {Object} spec - `{ newKey, legacyKey, newEnvField, legacyEnvField, fallback }`
|
|
211
|
-
* @returns {{value: *, winner: string}} The value and its layer
|
|
212
|
-
*/
|
|
213
|
-
const resolveWithAlias = function ( options, envVars, spec ) {
|
|
214
|
-
if ( options[ spec.newKey ] !== undefined ) {
|
|
215
|
-
return { value: options[ spec.newKey ], winner: 'config' };
|
|
216
|
-
}
|
|
217
|
-
if ( options[ spec.legacyKey ] !== undefined ) {
|
|
218
|
-
return { value: options[ spec.legacyKey ], winner: 'legacyConfig' };
|
|
219
|
-
}
|
|
220
|
-
if ( envVars[ spec.newEnvField ] !== undefined ) {
|
|
221
|
-
return { value: envVars[ spec.newEnvField ], winner: 'env' };
|
|
222
|
-
}
|
|
223
|
-
if ( envVars[ spec.legacyEnvField ] !== undefined ) {
|
|
224
|
-
return { value: envVars[ spec.legacyEnvField ], winner: 'legacyEnv' };
|
|
225
|
-
}
|
|
226
|
-
return { value: spec.fallback, winner: 'default' };
|
|
227
|
-
}; // resolveWithAlias()
|
|
228
|
-
|
|
229
|
-
/**
|
|
230
|
-
* Describes one supplied legacy key for the report. A key with a target
|
|
231
|
-
* is `mapped` when its layer is the one that won that target, and
|
|
232
|
-
* `ignored` otherwise. A key without a target is always `ignored`.
|
|
233
|
-
*
|
|
234
|
-
* @param {string} name - The config key or the environment variable name
|
|
235
|
-
* @param {Object} legacy - The `LEGACY_KEYS` entry
|
|
236
|
-
* @param {Object} winners - Winning layer per target setting
|
|
237
|
-
* @param {string} layer - `'legacyConfig'` or `'legacyEnv'`
|
|
238
|
-
* @returns {{name: string, effect: string, target?: string}} One report entry
|
|
239
|
-
*/
|
|
240
|
-
const describeLegacy = function ( name, legacy, winners, layer ) {
|
|
241
|
-
if ( ( legacy.target !== undefined ) && ( winners[ legacy.target ] === layer ) ) {
|
|
242
|
-
return { name, effect: 'mapped', target: legacy.target };
|
|
243
|
-
}
|
|
244
|
-
return { name, effect: 'ignored' };
|
|
245
|
-
}; // describeLegacy()
|
|
246
|
-
|
|
247
|
-
/**
|
|
248
|
-
* Builds the deprecation report: every supplied legacy config key in
|
|
249
|
-
* order, then every supplied legacy environment variable in the same
|
|
250
|
-
* order.
|
|
251
|
-
*
|
|
252
|
-
* @param {Object} options - The raw storage options
|
|
253
|
-
* @param {Object} envVars - The `ENV_VARS` object
|
|
254
|
-
* @param {Object} winners - Winning layer per target setting
|
|
255
|
-
* @returns {Array<Object>} The report entries, possibly empty
|
|
256
|
-
*/
|
|
257
|
-
const collectDeprecations = function ( options, envVars, winners ) {
|
|
258
|
-
const report = [];
|
|
259
|
-
for ( const legacy of LEGACY_KEYS ) {
|
|
260
|
-
if ( options[ legacy.key ] !== undefined ) {
|
|
261
|
-
report.push( describeLegacy( legacy.key, legacy, winners, 'legacyConfig' ) );
|
|
262
|
-
}
|
|
263
|
-
}
|
|
264
|
-
for ( const legacy of LEGACY_KEYS ) {
|
|
265
|
-
if ( envVars[ legacy.envField ] !== undefined ) {
|
|
266
|
-
report.push( describeLegacy( legacy.envVar, legacy, winners, 'legacyEnv' ) );
|
|
267
|
-
}
|
|
268
|
-
}
|
|
269
|
-
return report;
|
|
270
|
-
}; // collectDeprecations()
|
|
271
|
-
|
|
272
169
|
/**
|
|
273
170
|
* Maps the words of `QUESTDB_STDLIB_HTTP` to a boolean. The words are
|
|
274
171
|
* the client's own for `stdlib_http`, `on` and `off`, so an operator
|
|
@@ -286,55 +183,41 @@ const onOffToBoolean = function ( word ) {
|
|
|
286
183
|
}; // onOffToBoolean()
|
|
287
184
|
|
|
288
185
|
/**
|
|
289
|
-
* Resolves the storage options into the adapter's settings
|
|
290
|
-
*
|
|
291
|
-
*
|
|
292
|
-
* `flushDeadlineFor`.
|
|
186
|
+
* Resolves the storage options into the adapter's settings.
|
|
187
|
+
* `flushDeadlineMs` is undefined unless the operator set it; the engine
|
|
188
|
+
* then derives each flush's deadline with `flushDeadlineFor`.
|
|
293
189
|
*
|
|
294
190
|
* @param {Object} options - The raw storage options from `.storage()` config
|
|
295
191
|
* @param {Object} envVars - The `ENV_VARS` object (injected, for purity)
|
|
296
|
-
* @returns {
|
|
192
|
+
* @returns {Object} The resolved settings
|
|
297
193
|
* @throws {Error} INVALID_CONFIG when the ceiling is below twice the threshold
|
|
298
194
|
*/
|
|
299
195
|
const resolveOptions = function ( options, envVars ) {
|
|
300
|
-
const
|
|
301
|
-
|
|
302
|
-
legacyKey: 'autoFlushRows',
|
|
303
|
-
newEnvField: 'questdbFlushRows',
|
|
304
|
-
legacyEnvField: 'questdbAutoFlushRows',
|
|
305
|
-
fallback: DEFAULT_FLUSH_ROWS
|
|
306
|
-
} );
|
|
307
|
-
const interval = resolveWithAlias( options, envVars, {
|
|
308
|
-
newKey: 'flushIntervalMs',
|
|
309
|
-
legacyKey: 'idleFlushCheckMs',
|
|
310
|
-
newEnvField: 'questdbFlushIntervalMs',
|
|
311
|
-
legacyEnvField: 'questdbIdleFlushCheckMs',
|
|
312
|
-
fallback: DEFAULT_FLUSH_INTERVAL_MS
|
|
313
|
-
} );
|
|
314
|
-
|
|
196
|
+
const flushRows = options.flushRows ?? envVars.questdbFlushRows ?? DEFAULT_FLUSH_ROWS;
|
|
197
|
+
const flushIntervalMs = options.flushIntervalMs ?? envVars.questdbFlushIntervalMs ?? DEFAULT_FLUSH_INTERVAL_MS;
|
|
315
198
|
const bufferCeilingRows = options.bufferCeilingRows ??
|
|
316
199
|
envVars.questdbBufferCeilingRows ??
|
|
317
|
-
(
|
|
200
|
+
( flushRows * DEFAULT_CEILING_MULTIPLIER );
|
|
318
201
|
|
|
319
202
|
// Rows in flight count against the ceiling. A ceiling equal to the
|
|
320
203
|
// threshold would refuse every row that arrives during a
|
|
321
204
|
// row-triggered flush, because the batch in flight fills it. Twice
|
|
322
205
|
// the threshold is the least that holds one batch in flight and
|
|
323
206
|
// one batch buffering.
|
|
324
|
-
if ( bufferCeilingRows < ( 2 *
|
|
207
|
+
if ( bufferCeilingRows < ( 2 * flushRows ) ) {
|
|
325
208
|
const err = new Error(
|
|
326
|
-
`winkComposer/questdb: bufferCeilingRows ${bufferCeilingRows} is below twice flushRows ${
|
|
209
|
+
`winkComposer/questdb: bufferCeilingRows ${bufferCeilingRows} is below twice flushRows ${flushRows} [INVALID_CONFIG]: ` +
|
|
327
210
|
'the ceiling must hold one batch in flight and one batch buffering; raise bufferCeilingRows or lower flushRows'
|
|
328
211
|
);
|
|
329
212
|
err.code = 'INVALID_CONFIG';
|
|
330
213
|
throw err;
|
|
331
214
|
}
|
|
332
215
|
|
|
333
|
-
|
|
216
|
+
return {
|
|
334
217
|
ilpUrl: options.ilpUrl ?? envVars.questdbIlpUrl,
|
|
335
218
|
pgUrl: options.pgUrl ?? envVars.questdbPgUrl,
|
|
336
|
-
flushRows
|
|
337
|
-
flushIntervalMs
|
|
219
|
+
flushRows,
|
|
220
|
+
flushIntervalMs,
|
|
338
221
|
bufferCeilingRows,
|
|
339
222
|
flushDeadlineMs: options.flushDeadlineMs ?? envVars.questdbFlushDeadlineMs,
|
|
340
223
|
stdlibHttp: options.stdlibHttp ?? onOffToBoolean( envVars.questdbStdlibHttp ) ?? DEFAULT_STDLIB_HTTP,
|
|
@@ -346,9 +229,6 @@ const resolveOptions = function ( options, envVars ) {
|
|
|
346
229
|
onWarning: options.onWarning,
|
|
347
230
|
onDeliveryFailure: options.onDeliveryFailure
|
|
348
231
|
};
|
|
349
|
-
|
|
350
|
-
const winners = { flushRows: rows.winner, flushIntervalMs: interval.winner };
|
|
351
|
-
return { settings, deprecations: collectDeprecations( options, envVars, winners ) };
|
|
352
232
|
}; // resolveOptions()
|
|
353
233
|
|
|
354
234
|
/**
|
|
@@ -379,25 +259,8 @@ const flushDeadlineFor = function ( rows, settings ) {
|
|
|
379
259
|
return retryTimeout + ( 2 * attemptMs ) + CLIENT_MAX_RETRY_BACKOFF_MS + FLUSH_DEADLINE_MARGIN_MS;
|
|
380
260
|
}; // flushDeadlineFor()
|
|
381
261
|
|
|
382
|
-
/**
|
|
383
|
-
* Formats the deprecation report as the one console line the adapter
|
|
384
|
-
* prints at setup, in the ADR-028 message grammar.
|
|
385
|
-
*
|
|
386
|
-
* @param {Array<Object>} deprecations - A non-empty report from `resolveOptions`
|
|
387
|
-
* @returns {string} The console line
|
|
388
|
-
*/
|
|
389
|
-
const deprecationMessage = function ( deprecations ) {
|
|
390
|
-
const parts = deprecations.map( function ( entry ) {
|
|
391
|
-
return ( entry.effect === 'mapped' ) ?
|
|
392
|
-
`${entry.name} maps to ${entry.target}` :
|
|
393
|
-
`${entry.name} is ignored`;
|
|
394
|
-
} );
|
|
395
|
-
return 'winkComposer/questdb: deprecated storage options in use [DEPRECATED_OPTION]: ' +
|
|
396
|
-
`${parts.join( '; ' )}; all five deprecated keys are removed in ${REMOVAL_RELEASE}`;
|
|
397
|
-
}; // deprecationMessage()
|
|
398
|
-
|
|
399
262
|
// ============================================================================
|
|
400
263
|
// EXPORTS
|
|
401
264
|
// ============================================================================
|
|
402
265
|
|
|
403
|
-
export { resolveOptions, flushDeadlineFor,
|
|
266
|
+
export { resolveOptions, flushDeadlineFor, CLIENT_MAX_RETRY_BACKOFF_MS };
|