@ultimat3/realtime 15.0.0 → 17.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +57 -0
- package/package.json +3 -3
- package/src/change-buffer.ts +13 -4
- package/src/changefeed.ts +2 -1
- package/src/channel.ts +11 -3
- package/src/client.ts +6 -2
- package/src/live-definition.ts +2 -1
- package/src/live-query.ts +6 -2
- package/src/nats-lib-client.ts +17 -5
- package/src/nats-transport.ts +18 -3
- package/src/pg-replication.ts +11 -4
- package/src/pg-wire.ts +14 -2
- package/src/presence.ts +6 -3
- package/src/socket.ts +18 -2
- package/src/subscription-book.ts +24 -2
- package/src/sync-listen.ts +2 -2
- package/src/sync-node-bounds.ts +129 -0
- package/src/sync-node.ts +19 -39
- package/src/sync-upgrade.ts +16 -1
- package/src/thundering-herd.ts +10 -1
- package/src/transport-env.ts +6 -1
package/CLAUDE.md
CHANGED
|
@@ -63,6 +63,63 @@ Tier 3 package. Channels, live queries, local-first sync. One protocol for all t
|
|
|
63
63
|
for a hand-written definition and `live-definition.test.ts` proves it for a real declared
|
|
64
64
|
`query({ live: true })` — the second one matters, because a rule that only holds for test fakes
|
|
65
65
|
is a rule no declaration can reach.
|
|
66
|
+
- **Every numeric option is refused when it is not a FINITE number, `As of 2026-08-26`.**
|
|
67
|
+
`@ultimat3/core`'s `finite-option.ts` holds the one refusal, `finiteOption()`. `??` guards
|
|
68
|
+
nullish and `NaN` is not, so `Number(process.env.X)` on an unset variable reaches the bound
|
|
69
|
+
intact, and `Math.max`/`Math.min`/`Math.floor` propagate rather than validate — `AcceptBudget`
|
|
70
|
+
was `Math.max(1, options.perSecond)` and admitted every accept, because `NaN < 1` is false.
|
|
71
|
+
- **"Pinned at zero" is a claim about the RATCHET, not about this package, and the two came apart
|
|
72
|
+
once (`As of 2026-08-26`).** `bun run finite-bounds` is a count of options defaulted with `??`
|
|
73
|
+
and never screened; `realtime` is absent from `scripts/lib/finite-bounds-pins.ts`, which is that
|
|
74
|
+
count reading zero. It read zero while **two** options were unscreened, because the rule's own
|
|
75
|
+
header names the shape it cannot see: an option with **no `??` default**. Both had one spelling
|
|
76
|
+
— a value the caller either supplies or does not, forwarded or compared as-is.
|
|
77
|
+
`SubscriptionBook`'s `maxPerTenant` (`count >= NaN` is false, so the only cap spanning the
|
|
78
|
+
sockets of one tenant was OFF; measured, 5,000 subscribes admitted under `maxPerTenant: NaN`,
|
|
79
|
+
and it sat two lines under the screened `maxPerSocket`), and `openNatsClient`'s
|
|
80
|
+
`maxReconnectAttempts`, handed to the library unscreened — measured, the dial then never
|
|
81
|
+
returns. It also cannot see WHERE a screen runs, which is the bullet below. So: a package's
|
|
82
|
+
numeric options are audited by reading its option interfaces, and the ratchet is the floor under
|
|
83
|
+
that, never the proof of it. `SyncGrant.expiresAt` is deliberately NOT on that list and is the
|
|
84
|
+
one number left with the same shape: it is DATA an app's `authenticate` returns, not a
|
|
85
|
+
configuration option, and `expiresAt: NaN` makes `GrantBook.expired()` skip that grant forever —
|
|
86
|
+
the same outcome as omitting it, which is already a supported spelling.
|
|
87
|
+
- **A ceiling is refused where the object is BUILT, never inside a callback the runtime invokes
|
|
88
|
+
per connection (`As of 2026-08-26`).** `maxBufferedBytes`, `maxDroppedFrames`,
|
|
89
|
+
`maxFramesPerSecond` and `frameBurst` were screened only in `SyncSocket`'s constructor, which
|
|
90
|
+
`sync-node` runs inside `websocket.open`, which Bun runs SYNCHRONOUSLY inside `server.upgrade`.
|
|
91
|
+
Measured: `createSyncNode` did not throw, `/healthz` and `/readyz` both answered, `ready` was
|
|
92
|
+
true, and every upgrade threw `X_INVARIANT` with the node holding zero sockets — a node that
|
|
93
|
+
boots green and refuses every client, whose cause is one frame inside the runtime. They are
|
|
94
|
+
`socketCeilings()` in `sync-node-bounds.ts` now, called once by `createSyncNode`, and the
|
|
95
|
+
per-socket screen STAYS beside it: `SyncSocket` is exported and an app may build one directly,
|
|
96
|
+
which is the layered repair `finite-bounds`' own header prescribes for a check in another file.
|
|
97
|
+
`undefined` stays `undefined` — `SyncSocket` owns those four defaults, and a second spelling of
|
|
98
|
+
one is a number that can drift from it.
|
|
99
|
+
- **An upgrade that THROWS gives the grant back, not only one that answers `false`
|
|
100
|
+
(`As of 2026-08-26`).** `handleUpgrade` records the grant before `server.upgrade` because `open`
|
|
101
|
+
reads it, and released it on the `false` branch alone — so a throw out of `open` left one
|
|
102
|
+
`GrantBook` entry per connection ATTEMPT, unreapable: `sweepGrants` only visits a grant carrying
|
|
103
|
+
an `expiresAt`, which `authenticate: async () => ({ actor })` does not produce. Measured, 20
|
|
104
|
+
failing upgrades left 20 grants. The `try` around `server.upgrade` rethrows untouched — the throw
|
|
105
|
+
is the operator's diagnosis and that line owes it the release, not a verdict.
|
|
106
|
+
- **Every ceiling is refused when it is not a FINITE number, `As of 2026-08-26`.**
|
|
107
|
+
`AcceptBudget`'s `perSecond`/`burst`, `SyncSocket`'s `maxBufferedBytes`/`maxDroppedFrames` and
|
|
108
|
+
`SocketRegistry`'s `idleTimeoutMs` throw `X_INVARIANT` at construction on `NaN` and `±Infinity`.
|
|
109
|
+
Every comparison against a non-finite bound is FALSE, so each guard did not loosen — it switched
|
|
110
|
+
off, silently. Measured: `tryAccept()` asks `tokens < 1`, so `perSecond: NaN` admitted every
|
|
111
|
+
accept, herd included, on the node path AND on the per-socket frame flood budget;
|
|
112
|
+
`maxBufferedBytes: NaN` made `send()` answer TRUE with 10 MB queued, so a discarded frame was
|
|
113
|
+
neither counted in `channel_frames_dropped_total` nor desync-marked — the delivery-accounting
|
|
114
|
+
guarantee this file states, voidable through one option; `idleTimeoutMs: NaN` left a socket idle
|
|
115
|
+
for 10,000,000 ms out of `idle()`. `??` guards only nullish and `Math.max` is a clamp, not a
|
|
116
|
+
validator: `Number(process.env.X)` on an unset variable reaches the comparison intact.
|
|
117
|
+
- **A SQLSTATE off the wire is DATA, so `pg-wire.ts`'s `FIXES` is read with `Object.hasOwn`.**
|
|
118
|
+
`FIXES['constructor']` answered the `Object` function — not nullish, so `?? GENERIC_FIX` never
|
|
119
|
+
fired — and `UltimateError` then ran `singleLine(fn)`: a `TypeError` out of the constructor of
|
|
120
|
+
the error that exists to explain the failure, so the caller lost `X_REPLICATION_FAILED`, its
|
|
121
|
+
cause and its fix at once. This was `realtime`'s one `proto-index` pin, and the pin's stated
|
|
122
|
+
reason ("keyed by a replication op this package declares") described a different read.
|
|
66
123
|
- **A name nothing registered is `X_LIVE_QUERY_UNKNOWN`, never `X_PROTOCOL_VERSION`.** The frame
|
|
67
124
|
parsed and the version matched — one string in it names nothing — so "x build && redeploy the
|
|
68
125
|
client" is the one instruction that cannot work: a rebuilt client spells the typo the same way,
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ultimat3/realtime",
|
|
3
|
-
"version": "
|
|
3
|
+
"version": "17.0.0",
|
|
4
4
|
"description": "Three-tier realtime: channels, live queries, local-first sync — one protocol, one mutator shape",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
|
@@ -36,8 +36,8 @@
|
|
|
36
36
|
"test": "bun test"
|
|
37
37
|
},
|
|
38
38
|
"dependencies": {
|
|
39
|
-
"@ultimat3/core": "
|
|
40
|
-
"@ultimat3/query": "
|
|
39
|
+
"@ultimat3/core": "17.0.0",
|
|
40
|
+
"@ultimat3/query": "17.0.0",
|
|
41
41
|
"nats": "2.29.3"
|
|
42
42
|
}
|
|
43
43
|
}
|
package/src/change-buffer.ts
CHANGED
|
@@ -13,6 +13,7 @@
|
|
|
13
13
|
// node fills from the change stream it already subscribes to, which is a `ResumeSource` shape
|
|
14
14
|
// change, not a placement change.
|
|
15
15
|
|
|
16
|
+
import { finiteOption } from '@ultimat3/core';
|
|
16
17
|
import type { ResumeSource } from './cursor';
|
|
17
18
|
import type { RowPatch } from './json';
|
|
18
19
|
|
|
@@ -64,10 +65,18 @@ export class RingChangeBuffer implements ResumeSource {
|
|
|
64
65
|
#bytes = 0;
|
|
65
66
|
|
|
66
67
|
constructor(options: ChangeBufferOptions = {}) {
|
|
67
|
-
this.#capacity = options.capacity ?? 1024;
|
|
68
|
-
this.#maxQueries = options.maxQueries ?? 4096;
|
|
69
|
-
this.#maxBytesPerQuery =
|
|
70
|
-
|
|
68
|
+
this.#capacity = finiteOption('ChangeBuffer', 'capacity', options.capacity ?? 1024);
|
|
69
|
+
this.#maxQueries = finiteOption('ChangeBuffer', 'maxQueries', options.maxQueries ?? 4096);
|
|
70
|
+
this.#maxBytesPerQuery = finiteOption(
|
|
71
|
+
'ChangeBuffer',
|
|
72
|
+
'maxBytesPerQuery',
|
|
73
|
+
options.maxBytesPerQuery ?? DEFAULT_MAX_BUFFER_BYTES_PER_QUERY,
|
|
74
|
+
);
|
|
75
|
+
this.#maxBytes = finiteOption(
|
|
76
|
+
'ChangeBuffer',
|
|
77
|
+
'maxBytes',
|
|
78
|
+
options.maxBytes ?? DEFAULT_MAX_BUFFER_BYTES,
|
|
79
|
+
);
|
|
71
80
|
}
|
|
72
81
|
|
|
73
82
|
/** Retained bytes across every query on this node. The number the ceiling is about. */
|
package/src/changefeed.ts
CHANGED
|
@@ -3,6 +3,7 @@
|
|
|
3
3
|
// replicator, and the fanout never learn which one they are attached to.
|
|
4
4
|
|
|
5
5
|
import type { Clock } from '@ultimat3/core';
|
|
6
|
+
import { finiteOption } from '@ultimat3/core';
|
|
6
7
|
import { ReplicationFailedError } from './errors';
|
|
7
8
|
import type { Row } from './json';
|
|
8
9
|
import { PgReplicationStream, type ReplicationStreamStats } from './pg-replication';
|
|
@@ -72,7 +73,7 @@ export class InMemoryChangeFeed implements ChangeFeed {
|
|
|
72
73
|
#lastLsn: string | null = null;
|
|
73
74
|
|
|
74
75
|
constructor(options: InMemoryChangeFeedOptions = {}) {
|
|
75
|
-
this.#retain = options.retain ?? 1024;
|
|
76
|
+
this.#retain = finiteOption('the change feed', 'retain', options.retain ?? 1024);
|
|
76
77
|
}
|
|
77
78
|
|
|
78
79
|
async start(options: ChangeFeedStartOptions): Promise<void> {
|
package/src/channel.ts
CHANGED
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
// append-only stream, so tier 1 needs no frame of its own. That is why climbing the ladder is a
|
|
5
5
|
// config change: the client's frame handler is the same code at every rung.
|
|
6
6
|
|
|
7
|
-
import { type Actor, logger, renderThrowable } from '@ultimat3/core';
|
|
7
|
+
import { type Actor, finiteOption, logger, renderThrowable } from '@ultimat3/core';
|
|
8
8
|
import { formatLsn } from './changefeed';
|
|
9
9
|
import {
|
|
10
10
|
isPolicyDenial,
|
|
@@ -101,8 +101,16 @@ export class ChannelHub {
|
|
|
101
101
|
constructor(options: ChannelHubOptions) {
|
|
102
102
|
this.#transport = options.transport;
|
|
103
103
|
this.#sockets = options.sockets;
|
|
104
|
-
this.#maxTopicsPerSocket =
|
|
105
|
-
|
|
104
|
+
this.#maxTopicsPerSocket = finiteOption(
|
|
105
|
+
'ChannelHub',
|
|
106
|
+
'maxTopicsPerSocket',
|
|
107
|
+
options.maxTopicsPerSocket ?? 64,
|
|
108
|
+
);
|
|
109
|
+
this.#maxTopicsPerNode = finiteOption(
|
|
110
|
+
'ChannelHub',
|
|
111
|
+
'maxTopicsPerNode',
|
|
112
|
+
options.maxTopicsPerNode ?? DEFAULT_MAX_TOPICS_PER_NODE,
|
|
113
|
+
);
|
|
106
114
|
}
|
|
107
115
|
|
|
108
116
|
/** Sockets this node will deliver `name` to. The metric the fanout reads. */
|
package/src/client.ts
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
// serves all three tiers: `useLive` is tier 2, and a `store` + `queue` makes the same call tier 3
|
|
4
4
|
// with nothing about the subscription changing — that is the ladder's whole promise.
|
|
5
5
|
|
|
6
|
-
import { type Clock, systemClock, uuid } from '@ultimat3/core';
|
|
6
|
+
import { type Clock, finiteOption, systemClock, uuid } from '@ultimat3/core';
|
|
7
7
|
import type { Topic } from './channel';
|
|
8
8
|
import type {
|
|
9
9
|
ClientSocket,
|
|
@@ -110,7 +110,11 @@ export class LiveClient<T extends TableMap = TableMap> {
|
|
|
110
110
|
this.#connected = connected;
|
|
111
111
|
this.#setConnected = setConnected;
|
|
112
112
|
this.#heartbeat = new Heartbeat({
|
|
113
|
-
intervalMs:
|
|
113
|
+
intervalMs: finiteOption(
|
|
114
|
+
'the sync client',
|
|
115
|
+
'heartbeatMs',
|
|
116
|
+
options.heartbeatMs ?? DEFAULT_HEARTBEAT_MS,
|
|
117
|
+
),
|
|
114
118
|
schedule: options.scheduler ?? timeoutScheduler,
|
|
115
119
|
now: () => this.#clock.now().getTime(),
|
|
116
120
|
beat: () => this.#beat(),
|
package/src/live-definition.ts
CHANGED
|
@@ -9,6 +9,7 @@
|
|
|
9
9
|
// every time. Collapsing the second onto the first is privilege escalation with a cache hit rate.
|
|
10
10
|
|
|
11
11
|
import type { Ctx } from '@ultimat3/core';
|
|
12
|
+
import { finiteOption } from '@ultimat3/core';
|
|
12
13
|
import { type AnyQuery, queryHash, queryName } from '@ultimat3/query';
|
|
13
14
|
import { LiveRowUnidentifiedError } from './errors';
|
|
14
15
|
import { isRow, type JsonValue, type Row } from './json';
|
|
@@ -120,7 +121,7 @@ export function liveQueryDefinition(
|
|
|
120
121
|
},
|
|
121
122
|
};
|
|
122
123
|
windows.set(qid, built);
|
|
123
|
-
evictOldest(windows, options.maxWindows ?? 256);
|
|
124
|
+
evictOldest(windows, finiteOption('live()', 'maxWindows', options.maxWindows ?? 256));
|
|
124
125
|
return built;
|
|
125
126
|
};
|
|
126
127
|
|
package/src/live-query.ts
CHANGED
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
// policy is never sent to that actor — it arrives as a `delete` if they hold it, and is dropped
|
|
7
7
|
// otherwise.
|
|
8
8
|
|
|
9
|
-
import { type Actor, type Clock, systemClock, uuid } from '@ultimat3/core';
|
|
9
|
+
import { type Actor, type Clock, finiteOption, systemClock, uuid } from '@ultimat3/core';
|
|
10
10
|
import { queryHash } from '@ultimat3/query';
|
|
11
11
|
import type { ChangeEvent } from './changefeed';
|
|
12
12
|
import {
|
|
@@ -76,7 +76,11 @@ export class LiveQueryRegistry {
|
|
|
76
76
|
this.#options = options;
|
|
77
77
|
this.#clock = options.clock ?? systemClock;
|
|
78
78
|
this.#gate = new SubscriberGate(options);
|
|
79
|
-
this.#maxEntries =
|
|
79
|
+
this.#maxEntries = finiteOption(
|
|
80
|
+
'the live-query registry',
|
|
81
|
+
'maxEntries',
|
|
82
|
+
options.maxEntries ?? DEFAULT_MAX_ENTRIES,
|
|
83
|
+
);
|
|
80
84
|
// The book owns the caps because it is the only thing that can answer them in O(1).
|
|
81
85
|
this.#book = new SubscriptionBook(options);
|
|
82
86
|
this.#fanout = { gate: this.#gate, source: options.source, clock: this.#clock };
|
package/src/nats-lib-client.ts
CHANGED
|
@@ -14,7 +14,7 @@
|
|
|
14
14
|
// This header said "every failure leaves here as an `UltimateError`" until 2026-08, which a reader
|
|
15
15
|
// took as a guarantee it never was.
|
|
16
16
|
|
|
17
|
-
import { renderThrowable } from '@ultimat3/core';
|
|
17
|
+
import { finiteOption, renderThrowable } from '@ultimat3/core';
|
|
18
18
|
import { connect, Events, headers, Match, type Msg, type MsgHdrs, type NatsConnection } from 'nats';
|
|
19
19
|
import { TransportUnavailableError } from './errors';
|
|
20
20
|
import {
|
|
@@ -76,7 +76,11 @@ class LibNatsClient implements NatsClient {
|
|
|
76
76
|
constructor(connection: NatsConnection, target: NatsTarget, options: NatsClientOptions) {
|
|
77
77
|
this.#connection = connection;
|
|
78
78
|
this.#target = target;
|
|
79
|
-
this.#timeoutMs =
|
|
79
|
+
this.#timeoutMs = finiteOption(
|
|
80
|
+
'the nats client',
|
|
81
|
+
'requestTimeoutMs',
|
|
82
|
+
options.requestTimeoutMs ?? DEFAULT_REQUEST_TIMEOUT_MS,
|
|
83
|
+
);
|
|
80
84
|
this.#report = options.onError ?? ((): void => undefined);
|
|
81
85
|
void this.#watch(options);
|
|
82
86
|
}
|
|
@@ -192,14 +196,22 @@ class LibNatsClient implements NatsClient {
|
|
|
192
196
|
*/
|
|
193
197
|
export const openNatsClient = async (options: NatsClientOptions): Promise<NatsClient> => {
|
|
194
198
|
const target = parseNatsUrl(options.url);
|
|
199
|
+
// Screened BEFORE the try, and before the dial. Two reasons, and the second is why it is not a
|
|
200
|
+
// line further down: the value is FORWARDED to the library rather than compared here, so there is
|
|
201
|
+
// no `??` for `bun run finite-bounds` to see — the same conditional-spread shape that hid the
|
|
202
|
+
// sync node's four socket ceilings; and inside the `try`, the catch below would re-render a
|
|
203
|
+
// misconfiguration as `X_TRANSPORT_UNAVAILABLE`, which is a bus outage nobody can fix by looking
|
|
204
|
+
// at the bus. `-1` stays legal: that is the library's "reconnect forever".
|
|
205
|
+
const maxReconnectAttempts =
|
|
206
|
+
options.maxReconnectAttempts === undefined
|
|
207
|
+
? undefined
|
|
208
|
+
: finiteOption('the nats client', 'maxReconnectAttempts', options.maxReconnectAttempts);
|
|
195
209
|
try {
|
|
196
210
|
const connection = await connect({
|
|
197
211
|
servers: [`${target.host}:${target.port}`],
|
|
198
212
|
name: options.name ?? 'ultimate',
|
|
199
213
|
waitOnFirstConnect: true,
|
|
200
|
-
...(
|
|
201
|
-
? {}
|
|
202
|
-
: { maxReconnectAttempts: options.maxReconnectAttempts }),
|
|
214
|
+
...(maxReconnectAttempts === undefined ? {} : { maxReconnectAttempts }),
|
|
203
215
|
...(options.reconnectDelay === undefined
|
|
204
216
|
? {}
|
|
205
217
|
: { reconnectDelayHandler: options.reconnectDelay }),
|
package/src/nats-transport.ts
CHANGED
|
@@ -3,7 +3,14 @@
|
|
|
3
3
|
// re-establishing subscriptions, which is what makes a `sync` node stateless: a lost connection is
|
|
4
4
|
// re-dialled and re-subscribed underneath the caller, and this file keeps no socket state at all.
|
|
5
5
|
|
|
6
|
-
import {
|
|
6
|
+
import {
|
|
7
|
+
type Clock,
|
|
8
|
+
finiteOption,
|
|
9
|
+
isUltimateError,
|
|
10
|
+
logger,
|
|
11
|
+
renderThrowable,
|
|
12
|
+
systemClock,
|
|
13
|
+
} from '@ultimat3/core';
|
|
7
14
|
import { TransportUnavailableError } from './errors';
|
|
8
15
|
import type { Transport, TransportHandler, TransportSet, TransportSubscription } from './fanout';
|
|
9
16
|
import type { NatsClient, NatsConnect } from './nats-client';
|
|
@@ -56,7 +63,11 @@ export class NatsTransport implements Transport {
|
|
|
56
63
|
this.#options = options;
|
|
57
64
|
this.#connect = options.connect ?? openNatsClient;
|
|
58
65
|
this.#backoff = options.backoff ?? defaultBackoff;
|
|
59
|
-
this.#attempts =
|
|
66
|
+
this.#attempts = finiteOption(
|
|
67
|
+
'createNatsTransport',
|
|
68
|
+
'maxReconnectAttempts',
|
|
69
|
+
options.maxReconnectAttempts ?? DEFAULT_ATTEMPTS,
|
|
70
|
+
);
|
|
60
71
|
this.#rng = options.rng ?? Math.random;
|
|
61
72
|
this.shared = new NatsKvSet({
|
|
62
73
|
client: () => this.#ensure(),
|
|
@@ -172,7 +183,11 @@ export class NatsTransport implements Transport {
|
|
|
172
183
|
return ensureKvBucket(
|
|
173
184
|
client,
|
|
174
185
|
this.#options.bucket,
|
|
175
|
-
|
|
186
|
+
finiteOption(
|
|
187
|
+
'createNatsTransport',
|
|
188
|
+
'presenceTtlMs',
|
|
189
|
+
this.#options.presenceTtlMs ?? DEFAULT_PRESENCE_TTL_MS,
|
|
190
|
+
),
|
|
176
191
|
);
|
|
177
192
|
}
|
|
178
193
|
|
package/src/pg-replication.ts
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
// the framing and the pgoutput decode live next door; what is decided here is *ordering*, because
|
|
4
4
|
// the lsn is the only authority the pipeline has.
|
|
5
5
|
|
|
6
|
-
import { type Clock, logger, renderThrowable, systemClock } from '@ultimat3/core';
|
|
6
|
+
import { type Clock, finiteOption, logger, renderThrowable, systemClock } from '@ultimat3/core';
|
|
7
7
|
import type { ChangeEvent, ChangeOp, PgLogicalReplicationOptions } from './changefeed';
|
|
8
8
|
import { ReplicationProtocolError } from './errors';
|
|
9
9
|
import { isRow, type Row } from './json';
|
|
@@ -184,9 +184,16 @@ export class PgReplicationStream {
|
|
|
184
184
|
// supervisor that reads it never sees the replicator come back.
|
|
185
185
|
this.#failure = null;
|
|
186
186
|
this.#confirmFailures = 0;
|
|
187
|
-
this.#timer = setInterval(
|
|
188
|
-
|
|
189
|
-
|
|
187
|
+
this.#timer = setInterval(
|
|
188
|
+
() => {
|
|
189
|
+
void this.#confirmOnTimer();
|
|
190
|
+
},
|
|
191
|
+
finiteOption(
|
|
192
|
+
'the replication stream',
|
|
193
|
+
'statusIntervalMs',
|
|
194
|
+
this.#options.statusIntervalMs ?? DEFAULT_STATUS_INTERVAL_MS,
|
|
195
|
+
),
|
|
196
|
+
);
|
|
190
197
|
// A pending timer must not be what keeps `x dev` alive after the app is done with it.
|
|
191
198
|
this.#timer.unref?.();
|
|
192
199
|
this.#pump = this.#drain(connection, handlers);
|
package/src/pg-wire.ts
CHANGED
|
@@ -195,13 +195,25 @@ export const FIXES: Readonly<Record<string, string>> = {
|
|
|
195
195
|
'0A000': 'set wal_level = logical in postgresql.conf and restart the server',
|
|
196
196
|
};
|
|
197
197
|
|
|
198
|
-
/**
|
|
198
|
+
/** What a SQLSTATE this table has no entry for is answered with. */
|
|
199
|
+
const GENERIC_FIX = 'x doctor db — the postgres message above names the object to change';
|
|
200
|
+
|
|
201
|
+
/**
|
|
202
|
+
* An `ErrorResponse` becomes the one error class whose `fix` names the command to run.
|
|
203
|
+
*
|
|
204
|
+
* `Object.hasOwn`, because `code` is `fields['C']` — read off the WIRE, so it is data and this
|
|
205
|
+
* table is keyed by it. `FIXES['constructor']` answered the `Object` function, which is not
|
|
206
|
+
* nullish, so `?? GENERIC_FIX` never fired and `UltimateError` ran `singleLine(fn)`: a `TypeError`
|
|
207
|
+
* out of the constructor of the error that exists to explain the failure, so the caller lost
|
|
208
|
+
* `X_REPLICATION_FAILED`, its cause and its fix at once. `packages/schema/src/errors.ts` shipped
|
|
209
|
+
* the same shape and `scripts/proto-index.ts` was written for it.
|
|
210
|
+
*/
|
|
199
211
|
export const serverError = (stage: string, body: Uint8Array): ReplicationFailedError => {
|
|
200
212
|
const fields = responseFields(body);
|
|
201
213
|
const code = fields['C'] ?? '';
|
|
202
214
|
return new ReplicationFailedError({
|
|
203
215
|
stage,
|
|
204
216
|
detail: describeFields(fields),
|
|
205
|
-
fix: FIXES[code] ??
|
|
217
|
+
fix: Object.hasOwn(FIXES, code) ? (FIXES[code] ?? GENERIC_FIX) : GENERIC_FIX,
|
|
206
218
|
});
|
|
207
219
|
};
|
package/src/presence.ts
CHANGED
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
// simply stop heartbeating and expire, and every other node already sees the same set. Ephemeral
|
|
5
5
|
// state is never modelled as rows — that rule is what keeps presence off the write path entirely.
|
|
6
6
|
|
|
7
|
-
import { type Clock, systemClock, uuid } from '@ultimat3/core';
|
|
7
|
+
import { type Clock, finiteOption, systemClock, uuid } from '@ultimat3/core';
|
|
8
8
|
import type { ChannelHub, Topic } from './channel';
|
|
9
9
|
import type { Transport } from './fanout';
|
|
10
10
|
import type { JsonObject } from './json';
|
|
@@ -66,8 +66,11 @@ export class PresenceRegistry {
|
|
|
66
66
|
this.#transport = options.transport;
|
|
67
67
|
this.#hub = options.hub;
|
|
68
68
|
this.#clock = options.clock ?? systemClock;
|
|
69
|
-
this.#ttlMs = options.ttlMs ?? 30_000;
|
|
70
|
-
this.#maxMembers = Math.max(
|
|
69
|
+
this.#ttlMs = finiteOption('presence', 'ttlMs', options.ttlMs ?? 30_000);
|
|
70
|
+
this.#maxMembers = Math.max(
|
|
71
|
+
1,
|
|
72
|
+
finiteOption('presence', 'maxMembers', options.maxMembers ?? DEFAULT_MAX_PRESENCE_MEMBERS),
|
|
73
|
+
);
|
|
71
74
|
this.#nodeId = options.nodeId ?? uuid();
|
|
72
75
|
}
|
|
73
76
|
|
package/src/socket.ts
CHANGED
|
@@ -10,6 +10,7 @@ import {
|
|
|
10
10
|
type Clock,
|
|
11
11
|
type Counter,
|
|
12
12
|
counter,
|
|
13
|
+
finiteOption,
|
|
13
14
|
logger,
|
|
14
15
|
recordConnection,
|
|
15
16
|
systemClock,
|
|
@@ -102,6 +103,14 @@ export function actorIdOf(actor: Actor | null): string | null {
|
|
|
102
103
|
return actor === null ? null : actor.id;
|
|
103
104
|
}
|
|
104
105
|
|
|
106
|
+
/**
|
|
107
|
+
* MEASURED, both directions: with `maxBufferedBytes: NaN`, `getBufferedAmount() > NaN` is false and
|
|
108
|
+
* `send()` answered TRUE with 10 MB already buffered — backpressure never trips, the caller is told
|
|
109
|
+
* the frame left, and the drop reaches neither `channel_frames_dropped_total` nor the desync mark;
|
|
110
|
+
* with `idleTimeoutMs: NaN`, `idleFor(now) > NaN` is false and a socket idle for 10,000,000 ms is
|
|
111
|
+
* not in `idle()`, so a wedged client holds its grant, its subscriptions and its topic membership
|
|
112
|
+
* forever. `@ultimat3/core`'s `finiteOption` is the one refusal; `bun run finite-bounds` is the ratchet over it.
|
|
113
|
+
*/
|
|
105
114
|
export class SyncSocket {
|
|
106
115
|
readonly id: string;
|
|
107
116
|
readonly clientBuildId: string;
|
|
@@ -150,10 +159,16 @@ export class SyncSocket {
|
|
|
150
159
|
this.serverBuildId = options.serverBuildId;
|
|
151
160
|
this.actor = options.actor ?? null;
|
|
152
161
|
this.#maxBufferedBytes = options.maxBufferedBytes ?? DEFAULT_MAX_BUFFERED_BYTES;
|
|
162
|
+
finiteOption('SyncSocket', 'maxBufferedBytes', this.#maxBufferedBytes);
|
|
153
163
|
this.#maxDroppedFrames = options.maxDroppedFrames ?? 32;
|
|
164
|
+
finiteOption('SyncSocket', 'maxDroppedFrames', this.#maxDroppedFrames);
|
|
154
165
|
this.frameBudget = new AcceptBudget({
|
|
155
|
-
perSecond:
|
|
156
|
-
|
|
166
|
+
perSecond: finiteOption(
|
|
167
|
+
'SyncSocket',
|
|
168
|
+
'maxFramesPerSecond',
|
|
169
|
+
options.maxFramesPerSecond ?? DEFAULT_MAX_FRAMES_PER_SECOND,
|
|
170
|
+
),
|
|
171
|
+
burst: finiteOption('SyncSocket', 'frameBurst', options.frameBurst ?? DEFAULT_FRAME_BURST),
|
|
157
172
|
clock: this.#clock,
|
|
158
173
|
});
|
|
159
174
|
// Two clocks on purpose: `openedAt` is an instant a human reads, `lastSeenMonotonicMs` is the
|
|
@@ -287,6 +302,7 @@ export class SocketRegistry {
|
|
|
287
302
|
constructor(options: SocketRegistryOptions = {}) {
|
|
288
303
|
this.#clock = options.clock ?? systemClock;
|
|
289
304
|
this.#idleTimeoutMs = options.idleTimeoutMs ?? DEFAULT_IDLE_TIMEOUT_MS;
|
|
305
|
+
finiteOption('SocketRegistry', 'idleTimeoutMs', this.#idleTimeoutMs);
|
|
290
306
|
}
|
|
291
307
|
|
|
292
308
|
/**
|
package/src/subscription-book.ts
CHANGED
|
@@ -4,6 +4,7 @@
|
|
|
4
4
|
// the only thing that knows what exists. Every question it answers is indexed, never scanned.
|
|
5
5
|
|
|
6
6
|
import type { Actor } from '@ultimat3/core';
|
|
7
|
+
import { finiteOption } from '@ultimat3/core';
|
|
7
8
|
import { SubscriptionIdTakenError, SubscriptionLimitError } from './errors';
|
|
8
9
|
import type { LiveSubscription } from './live-contract';
|
|
9
10
|
import type { SyncSocket } from './socket';
|
|
@@ -36,6 +37,8 @@ export interface SubscriptionCaps {
|
|
|
36
37
|
/** Sockets may open this many live queries before `X_SUBSCRIPTION_LIMIT`. */
|
|
37
38
|
export const DEFAULT_MAX_PER_SOCKET = 128;
|
|
38
39
|
|
|
40
|
+
const SUBJECT = 'the subscription caps';
|
|
41
|
+
|
|
39
42
|
/**
|
|
40
43
|
* Every live subscription on this node, keyed by `(socket, sid)`.
|
|
41
44
|
*
|
|
@@ -70,9 +73,28 @@ export class SubscriptionBook {
|
|
|
70
73
|
/** The same claims counted per tenant, because that cap spans sockets and a lane cannot see it. */
|
|
71
74
|
readonly #claimedPerTenant = new Map<string, number>();
|
|
72
75
|
readonly #caps: SubscriptionCaps;
|
|
76
|
+
/**
|
|
77
|
+
* BOTH caps screened here, once, rather than on the subscribe path — and `maxPerTenant` was not
|
|
78
|
+
* screened at all: it has no `??` default, which is the one shape `bun run finite-bounds` states
|
|
79
|
+
* in its own header that it cannot see. `count >= NaN` is false, so the only cap that spans the
|
|
80
|
+
* sockets of one tenant was off in silence; measured, a book built with `maxPerTenant: NaN`
|
|
81
|
+
* admitted 5,000 subscribes for one tenant. `undefined` stays `undefined`, because there the
|
|
82
|
+
* caller is saying "no per-tenant cap" rather than handing a number that is not one.
|
|
83
|
+
*/
|
|
84
|
+
readonly #maxPerSocket: number;
|
|
85
|
+
readonly #maxPerTenant: number | undefined;
|
|
73
86
|
|
|
74
87
|
constructor(caps: SubscriptionCaps = {}) {
|
|
75
88
|
this.#caps = caps;
|
|
89
|
+
this.#maxPerSocket = finiteOption(
|
|
90
|
+
SUBJECT,
|
|
91
|
+
'maxPerSocket',
|
|
92
|
+
caps.maxPerSocket ?? DEFAULT_MAX_PER_SOCKET,
|
|
93
|
+
);
|
|
94
|
+
this.#maxPerTenant =
|
|
95
|
+
caps.maxPerTenant === undefined
|
|
96
|
+
? undefined
|
|
97
|
+
: finiteOption(SUBJECT, 'maxPerTenant', caps.maxPerTenant);
|
|
76
98
|
}
|
|
77
99
|
|
|
78
100
|
get(socketId: string, sid: string): LiveSubscription | undefined {
|
|
@@ -160,7 +182,7 @@ export class SubscriptionBook {
|
|
|
160
182
|
* node to an entry, a matcher and a read.
|
|
161
183
|
*/
|
|
162
184
|
assertCapacity(socket: SyncSocket): void {
|
|
163
|
-
const perSocket = this.#
|
|
185
|
+
const perSocket = this.#maxPerSocket;
|
|
164
186
|
const claimed = this.#claimedBySocket.get(socket.id)?.size ?? 0;
|
|
165
187
|
if (socket.queries.size + claimed >= perSocket) {
|
|
166
188
|
throw new SubscriptionLimitError({
|
|
@@ -170,7 +192,7 @@ export class SubscriptionBook {
|
|
|
170
192
|
knob: 'maxPerSocket',
|
|
171
193
|
});
|
|
172
194
|
}
|
|
173
|
-
const perTenant = this.#
|
|
195
|
+
const perTenant = this.#maxPerTenant;
|
|
174
196
|
const tenant = this.#tenantFor(socket);
|
|
175
197
|
if (perTenant === undefined || tenant === null) return;
|
|
176
198
|
if (this.tenantCount(tenant) + (this.#claimedPerTenant.get(tenant) ?? 0) >= perTenant) {
|
package/src/sync-listen.ts
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
// so the node itself stays testable with no server: this file is the only place realtime calls
|
|
3
3
|
// `Bun.serve`, and the only thing in the package that knows a port exists.
|
|
4
4
|
|
|
5
|
-
import { markListening, onShutdown } from '@ultimat3/core';
|
|
5
|
+
import { finiteOption, markListening, onShutdown } from '@ultimat3/core';
|
|
6
6
|
import type { SyncNode } from './sync-node';
|
|
7
7
|
|
|
8
8
|
export interface ListenOptions {
|
|
@@ -21,7 +21,7 @@ export interface SyncListener {
|
|
|
21
21
|
*/
|
|
22
22
|
export function listenSyncNode(node: SyncNode, options: ListenOptions = {}): SyncListener {
|
|
23
23
|
const server = Bun.serve({
|
|
24
|
-
port: options.port ?? 3001,
|
|
24
|
+
port: finiteOption('listenSyncNode', 'port', options.port ?? 3001),
|
|
25
25
|
fetch: node.fetch,
|
|
26
26
|
websocket: node.websocket,
|
|
27
27
|
});
|
|
@@ -0,0 +1,129 @@
|
|
|
1
|
+
// Every numeric option `listenSyncNode` accepts, read and REFUSED in one place — split out of
|
|
2
|
+
// `sync-node.ts`, which is at the 500-line ceiling `x verify`'s `filesize` step enforces.
|
|
3
|
+
//
|
|
4
|
+
// WHY A REFUSAL AND NOT A CLAMP: `@ultimat3/core`'s `finite-option.ts` carries the argument in full. The short version is
|
|
5
|
+
// that `??` guards nullish, `NaN` is not nullish, and every comparison against a non-finite bound
|
|
6
|
+
// reads false — so `maxConnections: NaN` is a node that accepts without limit and says nothing.
|
|
7
|
+
|
|
8
|
+
import { finiteOption } from '@ultimat3/core';
|
|
9
|
+
|
|
10
|
+
const SUBJECT = 'the sync node';
|
|
11
|
+
|
|
12
|
+
/**
|
|
13
|
+
* How often an expired grant is re-decided. A third of the shortest TTL worth issuing: a grant is
|
|
14
|
+
* re-checked on the pass after it expires, so the window a revoked actor keeps its socket is this
|
|
15
|
+
* interval and not its token's lifetime.
|
|
16
|
+
*/
|
|
17
|
+
export const DEFAULT_REAUTH_INTERVAL_MS = 30_000;
|
|
18
|
+
|
|
19
|
+
/**
|
|
20
|
+
* Concurrent sockets one node will hold. The accept budget bounds the accept RATE and nothing
|
|
21
|
+
* bounded the COUNT: at the 500/s that budget permits, an attacker holding each socket open with
|
|
22
|
+
* one keepalive frame a minute reaches 1.8M sockets an hour, each carrying a `GrantBook` entry.
|
|
23
|
+
*
|
|
24
|
+
* The number clears the 50,000 real clients this repo has measured on one node
|
|
25
|
+
* (`scripts/bench/restart-bench.ts`) with room to spare, because a ceiling that refuses a proven
|
|
26
|
+
* workload is an outage the framework caused.
|
|
27
|
+
*/
|
|
28
|
+
export const DEFAULT_MAX_CONNECTIONS = 250_000;
|
|
29
|
+
|
|
30
|
+
/**
|
|
31
|
+
* Inbound bytes one frame may carry. Bun's own default is 16 MiB, which one authenticated socket
|
|
32
|
+
* can push continuously; a `subscribe` frame carrying a full 512-id cursor is under 32 KiB.
|
|
33
|
+
*/
|
|
34
|
+
export const DEFAULT_MAX_FRAME_BYTES = 256 * 1024;
|
|
35
|
+
|
|
36
|
+
export const DEFAULT_DRAIN_SPREAD_MS = 30_000;
|
|
37
|
+
export const DEFAULT_DRAIN_GRACE_MS = 5_000;
|
|
38
|
+
|
|
39
|
+
/** The subset of `SyncNodeOptions` this module reads. Structural, so the full type satisfies it. */
|
|
40
|
+
export interface SyncNodeNumericOptions {
|
|
41
|
+
readonly maxConnections?: number | undefined;
|
|
42
|
+
readonly reauthenticateIntervalMs?: number | undefined;
|
|
43
|
+
readonly maxFrameBytes?: number | undefined;
|
|
44
|
+
readonly drainSpreadMs?: number | undefined;
|
|
45
|
+
}
|
|
46
|
+
|
|
47
|
+
export interface SyncNodeBounds {
|
|
48
|
+
readonly maxConnections: number;
|
|
49
|
+
readonly reauthenticateIntervalMs: number;
|
|
50
|
+
readonly maxFrameBytes: number;
|
|
51
|
+
readonly drainSpreadMs: number;
|
|
52
|
+
}
|
|
53
|
+
|
|
54
|
+
export function syncNodeBounds(options: SyncNodeNumericOptions): SyncNodeBounds {
|
|
55
|
+
return {
|
|
56
|
+
maxConnections: finiteOption(
|
|
57
|
+
SUBJECT,
|
|
58
|
+
'maxConnections',
|
|
59
|
+
options.maxConnections ?? DEFAULT_MAX_CONNECTIONS,
|
|
60
|
+
),
|
|
61
|
+
reauthenticateIntervalMs: finiteOption(
|
|
62
|
+
SUBJECT,
|
|
63
|
+
'reauthenticateIntervalMs',
|
|
64
|
+
options.reauthenticateIntervalMs ?? DEFAULT_REAUTH_INTERVAL_MS,
|
|
65
|
+
),
|
|
66
|
+
maxFrameBytes: finiteOption(
|
|
67
|
+
SUBJECT,
|
|
68
|
+
'maxFrameBytes',
|
|
69
|
+
options.maxFrameBytes ?? DEFAULT_MAX_FRAME_BYTES,
|
|
70
|
+
),
|
|
71
|
+
drainSpreadMs: finiteOption(
|
|
72
|
+
SUBJECT,
|
|
73
|
+
'drainSpreadMs',
|
|
74
|
+
options.drainSpreadMs ?? DEFAULT_DRAIN_SPREAD_MS,
|
|
75
|
+
),
|
|
76
|
+
};
|
|
77
|
+
}
|
|
78
|
+
|
|
79
|
+
/**
|
|
80
|
+
* The four ceilings this node forwards to every socket it builds, and the only options it accepts
|
|
81
|
+
* that `SyncSocket` — not this module — owns the default of.
|
|
82
|
+
*
|
|
83
|
+
* They were screened only in `SyncSocket`'s constructor, which Bun runs inside `websocket.open`,
|
|
84
|
+
* which Bun runs SYNCHRONOUSLY inside `server.upgrade`. Measured: `createSyncNode` did not throw,
|
|
85
|
+
* `/healthz` and `/readyz` both answered, `ready` was true, and every upgrade threw `X_INVARIANT`
|
|
86
|
+
* with the node holding zero sockets — a misconfiguration that fails per connection instead of at
|
|
87
|
+
* boot, and each of those throws leaked the grant `handleUpgrade` records before the upgrade.
|
|
88
|
+
*
|
|
89
|
+
* `undefined` stays `undefined` rather than acquiring a default here: `SyncSocket` owns those four
|
|
90
|
+
* numbers, and a second spelling of one is a number that can drift from the one it copies. Nothing
|
|
91
|
+
* narrower than "finite" either — `maxDroppedFrames: 0` is "close on the first drop",
|
|
92
|
+
* `maxFramesPerSecond: 0` is clamped to 1 by `AcceptBudget`'s own documented floor, and a screen
|
|
93
|
+
* that refused either would turn a working deployment into a boot failure.
|
|
94
|
+
*
|
|
95
|
+
* The per-socket screen STAYS, because `SyncSocket` is exported and an app may build one directly
|
|
96
|
+
* — the layered form `finite-bounds`' own header prescribes for a repair in a different file.
|
|
97
|
+
*/
|
|
98
|
+
export interface SocketCeilings {
|
|
99
|
+
// `?: number` and never `?: number | undefined`, unlike `SyncNodeNumericOptions` above: this one
|
|
100
|
+
// is SPREAD into `SyncSocketOptions`, and under `exactOptionalPropertyTypes` an explicit
|
|
101
|
+
// `undefined` is a different type from an absent key. The build error is the enforcement — a
|
|
102
|
+
// screened ceiling that arrived as `undefined` would silently take the socket's default.
|
|
103
|
+
readonly maxFramesPerSecond?: number;
|
|
104
|
+
readonly frameBurst?: number;
|
|
105
|
+
readonly maxBufferedBytes?: number;
|
|
106
|
+
readonly maxDroppedFrames?: number;
|
|
107
|
+
}
|
|
108
|
+
|
|
109
|
+
export function socketCeilings(options: SocketCeilings): SocketCeilings {
|
|
110
|
+
const { maxFramesPerSecond, frameBurst, maxBufferedBytes, maxDroppedFrames } = options;
|
|
111
|
+
return {
|
|
112
|
+
...(maxFramesPerSecond === undefined
|
|
113
|
+
? {}
|
|
114
|
+
: { maxFramesPerSecond: finiteOption(SUBJECT, 'maxFramesPerSecond', maxFramesPerSecond) }),
|
|
115
|
+
...(frameBurst === undefined
|
|
116
|
+
? {}
|
|
117
|
+
: { frameBurst: finiteOption(SUBJECT, 'frameBurst', frameBurst) }),
|
|
118
|
+
...(maxBufferedBytes === undefined
|
|
119
|
+
? {}
|
|
120
|
+
: { maxBufferedBytes: finiteOption(SUBJECT, 'maxBufferedBytes', maxBufferedBytes) }),
|
|
121
|
+
...(maxDroppedFrames === undefined
|
|
122
|
+
? {}
|
|
123
|
+
: { maxDroppedFrames: finiteOption(SUBJECT, 'maxDroppedFrames', maxDroppedFrames) }),
|
|
124
|
+
};
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
/** Per CALL, not per node: `drain({ graceMs })` is an argument, so it is refused where it arrives. */
|
|
128
|
+
export const drainGraceMs = (graceMs: number | undefined): number =>
|
|
129
|
+
finiteOption('the sync node drain', 'graceMs', graceMs ?? DEFAULT_DRAIN_GRACE_MS);
|
package/src/sync-node.ts
CHANGED
|
@@ -23,6 +23,7 @@ import {
|
|
|
23
23
|
} from './socket';
|
|
24
24
|
import { GrantBook, type SyncAuthenticator, sweepGrants } from './sync-auth';
|
|
25
25
|
import { ackRefOf, createFrameRouter, type MutationHandler } from './sync-frames';
|
|
26
|
+
import { drainGraceMs, socketCeilings, syncNodeBounds } from './sync-node-bounds';
|
|
26
27
|
import { decode, type Frame, PROTOCOL_VERSION, toWireError } from './sync-protocol';
|
|
27
28
|
import { handleUpgrade, type UpgradeTarget, type WsData } from './sync-upgrade';
|
|
28
29
|
import {
|
|
@@ -38,29 +39,13 @@ export type { UpgradeTarget, WsData } from './sync-upgrade';
|
|
|
38
39
|
|
|
39
40
|
export type SyncWs = WsLike & { readonly data: WsData };
|
|
40
41
|
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
/**
|
|
49
|
-
* Concurrent sockets one node will hold. The accept budget bounds the accept RATE and nothing
|
|
50
|
-
* bounded the COUNT: at the 500/s that budget permits, an attacker holding each socket open with
|
|
51
|
-
* one keepalive frame a minute reaches 1.8M sockets an hour, each carrying a `GrantBook` entry.
|
|
52
|
-
*
|
|
53
|
-
* The number clears the 50,000 real clients this repo has measured on one node
|
|
54
|
-
* (`scripts/bench/restart-bench.ts`) with room to spare, because a ceiling that refuses a proven
|
|
55
|
-
* workload is an outage the framework caused.
|
|
56
|
-
*/
|
|
57
|
-
export const DEFAULT_MAX_CONNECTIONS = 250_000;
|
|
58
|
-
|
|
59
|
-
/**
|
|
60
|
-
* Inbound bytes one frame may carry. Bun's own default is 16 MiB, which one authenticated socket
|
|
61
|
-
* can push continuously; a `subscribe` frame carrying a full 512-id cursor is under 32 KiB.
|
|
62
|
-
*/
|
|
63
|
-
export const DEFAULT_MAX_FRAME_BYTES = 256 * 1024;
|
|
42
|
+
// Moved to `sync-node-bounds.ts` with the refusals that read them, and re-exported here because
|
|
43
|
+
// `server.ts` publishes all three and a moved constant must not become a moved import path.
|
|
44
|
+
export {
|
|
45
|
+
DEFAULT_MAX_CONNECTIONS,
|
|
46
|
+
DEFAULT_MAX_FRAME_BYTES,
|
|
47
|
+
DEFAULT_REAUTH_INTERVAL_MS,
|
|
48
|
+
} from './sync-node-bounds';
|
|
64
49
|
|
|
65
50
|
export interface SyncNodeOptions {
|
|
66
51
|
readonly hub: ChannelHub;
|
|
@@ -153,7 +138,11 @@ export function createSyncNode(options: SyncNodeOptions): SyncNode {
|
|
|
153
138
|
});
|
|
154
139
|
const clock = options.clock ?? systemClock;
|
|
155
140
|
const accept = options.accept ?? new AcceptBudget({ perSecond: 500, burst: 2000, clock });
|
|
156
|
-
const
|
|
141
|
+
const bounds = syncNodeBounds(options);
|
|
142
|
+
// Screened at construction, once, and reused per socket: the four ceilings below are read inside
|
|
143
|
+
// `websocket.open`, which Bun runs synchronously inside `server.upgrade`, so refusing them there
|
|
144
|
+
// is a node that boots clean and throws out of every upgrade. `sync-node-bounds.ts` carries why.
|
|
145
|
+
const ceilings = socketCeilings(options);
|
|
157
146
|
const path = options.path ?? '/_x/sync';
|
|
158
147
|
const presence = options.presence;
|
|
159
148
|
const grants = new GrantBook();
|
|
@@ -318,7 +307,7 @@ export function createSyncNode(options: SyncNodeOptions): SyncNode {
|
|
|
318
307
|
if (options.authenticate) {
|
|
319
308
|
reauthing = setInterval(
|
|
320
309
|
() => detach(reauthenticate(), 'sync.reauthenticate'),
|
|
321
|
-
|
|
310
|
+
bounds.reauthenticateIntervalMs,
|
|
322
311
|
);
|
|
323
312
|
reauthing.unref();
|
|
324
313
|
} else {
|
|
@@ -348,7 +337,7 @@ export function createSyncNode(options: SyncNodeOptions): SyncNode {
|
|
|
348
337
|
{
|
|
349
338
|
path,
|
|
350
339
|
buildId: options.buildId,
|
|
351
|
-
maxConnections,
|
|
340
|
+
maxConnections: bounds.maxConnections,
|
|
352
341
|
accept,
|
|
353
342
|
rng: options.rng ?? Math.random,
|
|
354
343
|
// Read per call, never captured: `ready` and the socket count both move while a request
|
|
@@ -377,7 +366,7 @@ export function createSyncNode(options: SyncNodeOptions): SyncNode {
|
|
|
377
366
|
// one filtered `send` per socket through `SocketRegistry.deliver`, which is the only path
|
|
378
367
|
// that can count the frame it dropped — a flag configuring a mechanism nothing uses reads
|
|
379
368
|
// as a live one to the next person who has to decide how delivery works.
|
|
380
|
-
maxPayloadLength:
|
|
369
|
+
maxPayloadLength: bounds.maxFrameBytes,
|
|
381
370
|
sendPings: true,
|
|
382
371
|
|
|
383
372
|
open(ws: SyncWs): void {
|
|
@@ -391,16 +380,7 @@ export function createSyncNode(options: SyncNodeOptions): SyncNode {
|
|
|
391
380
|
serverBuildId: options.buildId,
|
|
392
381
|
actor: grants.get(ws.data.socketId)?.actor ?? null,
|
|
393
382
|
clock,
|
|
394
|
-
...
|
|
395
|
-
? {}
|
|
396
|
-
: { maxFramesPerSecond: options.maxFramesPerSecond }),
|
|
397
|
-
...(options.frameBurst === undefined ? {} : { frameBurst: options.frameBurst }),
|
|
398
|
-
...(options.maxBufferedBytes === undefined
|
|
399
|
-
? {}
|
|
400
|
-
: { maxBufferedBytes: options.maxBufferedBytes }),
|
|
401
|
-
...(options.maxDroppedFrames === undefined
|
|
402
|
-
? {}
|
|
403
|
-
: { maxDroppedFrames: options.maxDroppedFrames }),
|
|
383
|
+
...ceilings,
|
|
404
384
|
});
|
|
405
385
|
sockets.add(socket);
|
|
406
386
|
},
|
|
@@ -452,7 +432,7 @@ export function createSyncNode(options: SyncNodeOptions): SyncNode {
|
|
|
452
432
|
ready = false;
|
|
453
433
|
const ids = [...sockets.all()].map((socket) => socket.id);
|
|
454
434
|
const spread = drainPlan(ids, {
|
|
455
|
-
spreadMs:
|
|
435
|
+
spreadMs: bounds.drainSpreadMs,
|
|
456
436
|
...(options.rng ? { rng: options.rng } : {}),
|
|
457
437
|
});
|
|
458
438
|
// The answer is read, not assumed: this frame IS the socket's slot, so a client that never
|
|
@@ -467,7 +447,7 @@ export function createSyncNode(options: SyncNodeOptions): SyncNode {
|
|
|
467
447
|
if (notified < plan.length) {
|
|
468
448
|
logger.warn('sync.drain_frames_dropped', { sockets: plan.length, notified });
|
|
469
449
|
}
|
|
470
|
-
const graceMs = drainOptions.graceMs
|
|
450
|
+
const graceMs = drainGraceMs(drainOptions.graceMs);
|
|
471
451
|
if (graceMs > 0) await new Promise((resolve) => setTimeout(resolve, graceMs));
|
|
472
452
|
// Through `evict`, never `sockets.remove` + `grants.delete`: those are three of `teardown`'s
|
|
473
453
|
// five steps, and the two they skip are the ones the rest of the fleet can see. A drained
|
package/src/sync-upgrade.ts
CHANGED
|
@@ -124,7 +124,22 @@ export async function handleUpgrade(
|
|
|
124
124
|
// not return until it has, so a grant recorded on the next line is one the socket was already
|
|
125
125
|
// built without.
|
|
126
126
|
if (grant) deps.onGranted(data.socketId, grant);
|
|
127
|
-
|
|
127
|
+
let upgraded: boolean;
|
|
128
|
+
try {
|
|
129
|
+
upgraded = server.upgrade(request, { data });
|
|
130
|
+
} catch (error) {
|
|
131
|
+
// The other exit that never opens a socket, and the one the `false` branch below hid. Bun runs
|
|
132
|
+
// `websocket.open` synchronously inside `upgrade`, so ANYTHING that throws in there — a socket
|
|
133
|
+
// refusing its own ceiling, an app-supplied registry, an `open` a later change adds work to —
|
|
134
|
+
// comes out here, with no `close` callback behind it. Unreleased, that is one `GrantBook` entry
|
|
135
|
+
// per connection ATTEMPT, and an unreapable one: `sweepGrants` only visits a grant carrying an
|
|
136
|
+
// `expiresAt`, which `authenticate: async () => ({ actor })` does not produce. Measured, 20
|
|
137
|
+
// failing upgrades left 20 grants. Rethrown untouched — the throw is the operator's diagnosis,
|
|
138
|
+
// and this line owes it the release, not a verdict.
|
|
139
|
+
deps.onUngranted(data.socketId);
|
|
140
|
+
throw error;
|
|
141
|
+
}
|
|
142
|
+
if (!upgraded) {
|
|
128
143
|
deps.onUngranted(data.socketId);
|
|
129
144
|
return new Response('expected websocket', { status: 426 });
|
|
130
145
|
}
|
package/src/thundering-herd.ts
CHANGED
|
@@ -9,6 +9,7 @@
|
|
|
9
9
|
import {
|
|
10
10
|
type Clock,
|
|
11
11
|
backoffDelay as coreBackoffDelay,
|
|
12
|
+
finiteOption,
|
|
12
13
|
type JitterMode,
|
|
13
14
|
type Random,
|
|
14
15
|
systemClock,
|
|
@@ -111,7 +112,7 @@ export function drainPlan(
|
|
|
111
112
|
socketIds: readonly string[],
|
|
112
113
|
options: DrainPlanOptions = {},
|
|
113
114
|
): DrainPlanEntry[] {
|
|
114
|
-
const spreadMs = options.spreadMs ?? 30_000;
|
|
115
|
+
const spreadMs = finiteOption('drainPlan', 'spreadMs', options.spreadMs ?? 30_000);
|
|
115
116
|
const rng = options.rng ?? Math.random;
|
|
116
117
|
const total = socketIds.length;
|
|
117
118
|
if (total === 0) return [];
|
|
@@ -146,6 +147,14 @@ export class AcceptBudget {
|
|
|
146
147
|
#lastRefill: number;
|
|
147
148
|
|
|
148
149
|
constructor(options: AcceptBudgetOptions) {
|
|
150
|
+
// `Math.max(1, …)` is a CLAMP, not a validator, and it propagates every non-finite value it is
|
|
151
|
+
// handed. Measured: `perSecond: NaN` makes `#tokens` NaN, `tryAccept` asks `#tokens < 1`,
|
|
152
|
+
// `NaN < 1` is false — so the bucket admits every accept, forever, and `retryAfterMs()`
|
|
153
|
+
// answers NaN, which `JSON.stringify` writes into the `reconnect` frame as `null`. `Infinity`
|
|
154
|
+
// is the same failure spelled differently: a budget that never refuses is not a budget. A
|
|
155
|
+
// finite clamp is monotone and safe, so 0 and negatives keep their floor of 1.
|
|
156
|
+
finiteOption('AcceptBudget', 'perSecond', options.perSecond);
|
|
157
|
+
finiteOption('AcceptBudget', 'burst', options.burst ?? options.perSecond);
|
|
149
158
|
this.#perSecond = Math.max(1, options.perSecond);
|
|
150
159
|
this.#burst = Math.max(1, options.burst ?? options.perSecond);
|
|
151
160
|
this.#clock = options.clock ?? systemClock;
|
package/src/transport-env.ts
CHANGED
|
@@ -5,6 +5,7 @@
|
|
|
5
5
|
// two sides, and a caller that had to pass each one separately could quietly set them apart.
|
|
6
6
|
|
|
7
7
|
import type { Clock } from '@ultimat3/core';
|
|
8
|
+
import { finiteOption } from '@ultimat3/core';
|
|
8
9
|
import type { Transport } from './fanout';
|
|
9
10
|
import { InProcessTransport } from './fanout';
|
|
10
11
|
import type { NatsConnect } from './nats-client';
|
|
@@ -67,7 +68,11 @@ export function selectTransport(
|
|
|
67
68
|
env: TransportEnvironment,
|
|
68
69
|
options: SelectTransportOptions = {},
|
|
69
70
|
): TransportSelection {
|
|
70
|
-
const presenceTtlMs =
|
|
71
|
+
const presenceTtlMs = finiteOption(
|
|
72
|
+
'the transport env',
|
|
73
|
+
'presenceTtlMs',
|
|
74
|
+
options.presenceTtlMs ?? DEFAULT_PRESENCE_TTL_MS,
|
|
75
|
+
);
|
|
71
76
|
const url = nonEmpty(env['NATS_URL']);
|
|
72
77
|
|
|
73
78
|
if (url === undefined) {
|