@homie-rocks/studio 0.46.5 → 0.46.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +25 -0
- package/README.md +12 -1
- package/lib/crowd-cloud.mjs +75 -17
- package/lib/crowd-diagnostics.mjs +1 -0
- package/lib/crowd-load.mjs +15 -3
- package/lib/crowd-probe/README.md +58 -0
- package/lib/crowd-probe/client.mjs +3 -0
- package/lib/crowd-probe/reference-work.mjs +45 -0
- package/lib/crowd-probe/run-trial.sh +207 -0
- package/lib/crowd-probe/worker.mjs +37 -10
- package/lib/crowd-summary.mjs +74 -7
- package/lib/crowd-worker.mjs +12 -4
- package/lib/template-history.json +42 -0
- package/package.json +1 -1
- package/rules/host.ts +4 -1
- package/worker/build-id.mjs +1 -1
- package/worker/gate.mjs +72 -25
- package/worker/index.mjs +26 -8
- package/worker/office.mjs +2 -2
- package/worker/room-store.mjs +71 -11
- package/worker/room.mjs +16 -3
- package/worker/version.mjs +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -17,6 +17,31 @@ To bring a studio up to date, tell Claude: "Upgrade my studio to the newest Homi
|
|
|
17
17
|
`npx -y @homie-rocks/studio@latest upgrade`, which shows what's new since the version your studio pins (from this
|
|
18
18
|
file) and what the upgrade would change, and changes nothing until you agree.
|
|
19
19
|
|
|
20
|
+
## [0.46.6] - 2026-10-11
|
|
21
|
+
|
|
22
|
+
**Plugin 0.46.6** · Big rooms: what one room carries, measured.
|
|
23
|
+
|
|
24
|
+
What one room object carries on Cloudflare, measured in six trials and stated: with a light game at 20 ticks a second, 300 players with margin, 600 at the edge, 800 to 1,000 only by luck.
|
|
25
|
+
|
|
26
|
+
### Changed
|
|
27
|
+
|
|
28
|
+
- A big match room's once-a-second checkpoint no longer holds its frames. Cloudflare holds whatever an object sends after a storage write until the write is confirmed, so one or two worlds a second reached every Gate a tick late, at every size. The checkpoint is now written with `allowUnconfirmed`; a save an admission asks for is confirmed as before and replaces it. A lasting world's writes are unchanged. At 300 players the gap every Gate saw once a second is gone, and a Gate's round trip to the room fell from 40 ms to 18.
|
|
29
|
+
- A big room costs less of its one thread than on 0.46.3 (a fifth less; a quarter less when inputs arrive late): it no longer builds the whole snapshot as text for nobody, does not encode an ASCII message to measure it, and reads the round trip it tells the rules, for a player behind a Gate, once a second from its ten-second window, not once an input.
|
|
30
|
+
- A player is sent fewer frames only when behind and not catching up. A connection that stalls for a second and then delivers what it held catches up by itself; 0.46.2 slowed every player of a healthy room of 600 when their one home link hiccuped.
|
|
31
|
+
- A Gate nobody is on gives its channel to the room back after ten seconds; the next player to arrive opens another. An open channel kept a big room and its Gates in memory after everybody had left.
|
|
32
|
+
- Each Gate reports the share of worlds over 100, 200 and 400 ms old and the players more than a second behind, beside the worst of each second.
|
|
33
|
+
|
|
34
|
+
### Added
|
|
35
|
+
|
|
36
|
+
- Measuring a game's capacity on your own Cloudflare account (`lib/crowd-probe/README.md`): the in-Cloudflare probe offers compression as a browser does, reports every thirty seconds so a request cut short has still said what it saw, can play slow players among healthy ones (`--slow-every`), and times a fixed piece of work against your computer with the stopwatch outside Cloudflare (`--calibrate`). `lib/crowd-probe/run-trial.sh` runs a whole trial, each in-Cloudflare step three times, and writes one summary. The home driver can stop every Kth player's socket reading (`--slow-every`). Slow players' own figures are kept apart from everybody else's.
|
|
37
|
+
|
|
38
|
+
### Upgrade notes
|
|
39
|
+
|
|
40
|
+
- 1,000 players in one room is not promised; past what one room carries a world is split across rooms. Rooms of 64 seats or fewer and games a player's browser hosts are not on any of this path and behave as on 0.46.3.
|
|
41
|
+
- **What one room carries** is in the README ("Room size and delivery") and, with the measurements and the ways past one room, in `docs/rooms-capacity-findings.md`. Homie still sets no limit on `players.max`: choose it with your own game's numbers.
|
|
42
|
+
- Nothing an object measures about itself by its own clock is to be trusted on Cloudflare: a Worker's clock stands still while its code runs and reads a timer's due time when the timer fires. Each Gate's 20 ms lateness timer (added early in this release) read 0 in every step of every trial, fallen ones too. Read a Gate's round trip to the room and the share of worlds over 200 ms old.
|
|
43
|
+
- `HOMIE_GATE_PLAYERS` on a studio's Worker sets the players a Gate holds, for a trial that is finding the right number. A studio does not set it.
|
|
44
|
+
- The local crowd reproduction (eight workerd processes, two Chrome players and the load driver on one computer) is no longer a check. Run it by hand with `npm run test:crowd`; `docs/crowd-transport-0462.md` says what it can and cannot show.
|
|
20
45
|
## [0.46.5] - 2026-10-11
|
|
21
46
|
|
|
22
47
|
**Plugin 0.46.5**
|
package/README.md
CHANGED
|
@@ -222,7 +222,18 @@ For example, a 300-player room can choose spatial delivery explicitly:
|
|
|
222
222
|
The studio chooses the radius, precision and distant update rate for its game. The
|
|
223
223
|
controlled body stays precise. Whole-room state remains the default. The build runs a
|
|
224
224
|
local capacity trial; its numbers are not Cloudflare capacity guarantees. Test the actual
|
|
225
|
-
mechanic at the chosen size.
|
|
225
|
+
mechanic at the chosen size.
|
|
226
|
+
|
|
227
|
+
**What one room carries.** A room's rules run in one object on one thread, whatever its
|
|
228
|
+
`players.max`. Measured on Cloudflare in October 2026 with a light game at 20 ticks a
|
|
229
|
+
second (every player in one dense arena, ten inputs a second each): 300 players with
|
|
230
|
+
margin (clean in every run), 600 at the edge (four runs of six), and 800 to 1,000 only by
|
|
231
|
+
luck (five runs of twenty-five). A game with heavier rules carries fewer. Homie sets no
|
|
232
|
+
limit on `players.max`; it is the studio's to choose with its own game's numbers:
|
|
233
|
+
`lib/crowd-probe/README.md` is how to measure them on your own Cloudflare account. Beyond
|
|
234
|
+
what one room carries, a world is split across rooms (several rooms of a game today; one
|
|
235
|
+
world of many areas is on the roadmap). `docs/rooms-capacity-findings.md` in the Homie
|
|
236
|
+
repository has the measurements. Rules builds derive their revision from rules and view;
|
|
226
237
|
do not manually bump `netplay.version`. Optional `netplay.params` still selects play-page
|
|
227
238
|
query parameters passed to the view, beside `debug` and `q`.
|
|
228
239
|
|
package/lib/crowd-cloud.mjs
CHANGED
|
@@ -3,7 +3,8 @@
|
|
|
3
3
|
* One crowd step played from inside Cloudflare, through the in-Cloudflare probe (lib/crowd-probe/README.md).
|
|
4
4
|
*
|
|
5
5
|
* node lib/crowd-cloud.mjs --probe <probe address> --key <PROBE_KEY> --url <studio> --n 300
|
|
6
|
-
* [--seconds 150] [--room <name>] [--out <dir>] [--ramp-ms 50] [--game crowd-circuit]
|
|
6
|
+
* [--seconds 150] [--room <name>] [--out <dir>] [--ramp-ms 50] [--game crowd-circuit] [--compress on] [--slow-every 10]
|
|
7
|
+
* node lib/crowd-cloud.mjs --probe <probe address> --key <PROBE_KEY> --calibrate <file>
|
|
7
8
|
*
|
|
8
9
|
* It opens a request to the probe for every five players, holds them open for the step, and writes report.json in
|
|
9
10
|
* the shape crowd-load.mjs writes (what applies: there is no browser and no home connection in this one).
|
|
@@ -12,12 +13,44 @@ import { mkdir, writeFile } from 'node:fs/promises';
|
|
|
12
13
|
import { resolve } from 'node:path';
|
|
13
14
|
import { fileURLToPath } from 'node:url';
|
|
14
15
|
import { Tally } from './crowd-probe/client.mjs';
|
|
16
|
+
import { referenceWork } from './crowd-probe/reference-work.mjs';
|
|
15
17
|
|
|
16
18
|
const pick = (sorted, q) => (sorted.length ? sorted[Math.min(sorted.length - 1, Math.floor(sorted.length * q))] : null);
|
|
17
19
|
|
|
18
20
|
/** Every probe request's last line, added up into one report. */
|
|
19
|
-
|
|
20
|
-
|
|
21
|
+
/**
|
|
22
|
+
* How fast Cloudflare runs the reference work against this computer. A Worker cannot time its own work, so the
|
|
23
|
+
* stopwatch is here: a request that does the work `rounds` times, less a request that does none, each the quickest
|
|
24
|
+
* of several (the least delayed by the connection), divided by the rounds. This process times the same call.
|
|
25
|
+
* `ratio` is Cloudflare's milliseconds over this computer's: what a millisecond of test/crowd-room-cost.mjs costs there.
|
|
26
|
+
*/
|
|
27
|
+
export async function calibrate({ probe, key, requests = 6, rounds = 10, fetchFn = fetch, now = () => performance.now() }) {
|
|
28
|
+
referenceWork(4);
|
|
29
|
+
const local = [];
|
|
30
|
+
for (let i = 0; i < 5; i += 1) { const at = now(); referenceWork(); local.push(Math.round((now() - at) * 100) / 100); }
|
|
31
|
+
const check = referenceWork(), mine = Math.min(...local);
|
|
32
|
+
const ask = async (n) => { const at = now(); const answer = await (await fetchFn(`${probe}/calibrate?rounds=${n}&i=${Math.round(at)}`, { headers: { 'x-probe-key': key }, signal: AbortSignal.timeout(120000) })).json(); return { ms: Math.round((now() - at) * 100) / 100, ...answer }; };
|
|
33
|
+
const empty = [], worked = [];
|
|
34
|
+
for (let i = 0; i < requests; i += 1) {
|
|
35
|
+
try { empty.push(await ask(0)); } catch { /* one reading fewer */ }
|
|
36
|
+
try { const answer = await ask(rounds); if (answer.ok && answer.rounds === rounds) worked.push(answer); } catch { /* one reading fewer */ }
|
|
37
|
+
}
|
|
38
|
+
const base = Math.min(...empty.map((e) => e.ms)), each = worked.map((w) => Math.round((w.ms - base) / rounds * 100) / 100).sort((a, b) => a - b);
|
|
39
|
+
const of = (ms) => (Number.isFinite(ms) && mine > 0 ? Math.round(ms / mine * 100) / 100 : null);
|
|
40
|
+
return { at: new Date().toISOString(), rounds, localMs: local, emptyRequestMs: empty.map((e) => e.ms), workedRequestMs: worked.map((w) => w.ms), colos: [...new Set(worked.map((w) => w.colo).filter(Boolean))],
|
|
41
|
+
// What the Worker's own clock said the same rounds took: next to the stopwatch's figure, what a clock inside is worth.
|
|
42
|
+
workerOwnClockMs: worked.map((w) => w.own),
|
|
43
|
+
cloudMsARound: { min: each[0] ?? null, median: pick(each, 0.5), max: each.at(-1) ?? null },
|
|
44
|
+
ratio: each.length && Number.isFinite(base) ? { min: of(each[0]), median: of(pick(each, 0.5)), max: of(each.at(-1)) } : null,
|
|
45
|
+
same: worked.every((w) => w.check === check) };
|
|
46
|
+
}
|
|
47
|
+
|
|
48
|
+
export function cloudReport({ origin, room, n, seconds, started, measureAt, endAt, finals, failures = [], toolkit = null, compress = false, lost = 0, roomHz = 20 }) {
|
|
49
|
+
// A probe's clock is a Worker's: it reads a timer's due time and runs slow when the Worker runs late, so a rate
|
|
50
|
+
// timed by it reads high (the fifth trial's rows of 25 to 32 "ticks a second" from a 20-tick room). Frames for
|
|
51
|
+
// each tick the room advanced need no clock: times the room's rate, that is frames a second. (Not for a slow
|
|
52
|
+
// player, whose ticks are the ones it has got to: its rate stays by the clock.)
|
|
53
|
+
const players = finals.flatMap((f) => f.players).map((p) => (Number.isFinite(p.framesPerTick) && !p.slow ? { ...p, clockDeliveredHz: p.deliveredHz, deliveredHz: p.framesPerTick * roomHz } : p)), tallies = { ack: new Tally(), gaps: new Tally(), loops: new Tally() };
|
|
21
54
|
for (const f of finals) for (const key of Object.keys(tallies)) tallies[key].merge(f.tallies?.[key]);
|
|
22
55
|
const of = (key) => players.map((p) => p[key]).filter(Number.isFinite).sort((a, b) => a - b);
|
|
23
56
|
const range = (key) => { const a = of(key); return { count: a.length, min: a[0] ?? null, median: pick(a, 0.5), p50: pick(a, 0.5), max: a.at(-1) ?? null }; };
|
|
@@ -36,10 +69,25 @@ export function cloudReport({ origin, room, n, seconds, started, measureAt, endA
|
|
|
36
69
|
inputAckMs: tallies.ack.json(), receiveGapMs: tallies.gaps.json(),
|
|
37
70
|
// From the tick a player held when it stamped an input to the tick that input reached the rules.
|
|
38
71
|
inputLoopMs: tallies.loops.json(),
|
|
39
|
-
tickHz
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
72
|
+
// `tickHz` is by the probe's own clock (see above): above the room's rate, it says the probe ran late.
|
|
73
|
+
tickHz: range('tickHz'), tickHzClock: 'probe', deliveredHz: range('deliveredHz'), framesPerTick: range('framesPerTick'),
|
|
74
|
+
payload: { upBytes: sum('upBytes'), downBytes: sum('downBytes'), inputFrames: sum('inputs'), downFrames: sum('frames'), wireDownBytes: 0, compressedSockets: players.filter((p) => p.compressed).length },
|
|
75
|
+
// Sockets that never leave Cloudflare cross no home connection; with --compress on the studio's Gates compress
|
|
76
|
+
// for them as for a browser (the bytes saved cannot be seen from inside a Worker).
|
|
77
|
+
compression: { offered: Boolean(compress), sockets: players.filter((p) => p.compressed).length, of: players.length, ratio: null },
|
|
78
|
+
// Probe requests whose last word never came: their players were in the room, and are counted from each
|
|
79
|
+
// request's last thirty-second report where there was one.
|
|
80
|
+
lostReports: lost, partialReports: finals.filter((f) => f.t === 'partial').length,
|
|
81
|
+
// --slow-every K: every Kth player takes six frames a second. `healthy` is everybody else: what the slow
|
|
82
|
+
// ones cost them is the difference from the same step without any.
|
|
83
|
+
slowPlayers: players.filter((p) => p.slow).length,
|
|
84
|
+
healthy: (() => {
|
|
85
|
+
const a = players.filter((p) => !p.slow), list = (key) => a.map((p) => p[key]).filter(Number.isFinite).sort((x, y) => x - y);
|
|
86
|
+
const hz = list('deliveredHz'), gap = list('maxStateGapMs');
|
|
87
|
+
return { players: a.length, deliveredHz: { min: hz[0] ?? null, p05: pick(hz, 0.05), median: pick(hz, 0.5) }, maxStateGapMs: { median: pick(gap, 0.5), max: gap.at(-1) ?? null } };
|
|
88
|
+
})(),
|
|
89
|
+
slowInputAckMs: (() => { const t = new Tally(); for (const f of finals) t.merge(f.slowAck); return t.json(); })(),
|
|
90
|
+
slow: (() => { const a = players.filter((p) => p.slow), hz = a.map((p) => p.deliveredHz).filter(Number.isFinite).sort((x, y) => x - y); return a.length ? { players: a.length, deliveredHz: { min: hz[0] ?? null, median: pick(hz, 0.5), max: hz.at(-1) ?? null } } : null; })(),
|
|
43
91
|
inputLead: { ticksMedian: pick(leads, 0.5), ticksMax: leads.at(-1) ?? null, lateShare: sum('reports') ? Math.round(sum('late') / sum('reports') * 1000) / 1000 : null },
|
|
44
92
|
driver: { requests: finals.length, failedRequests: failures.length, workerLoopP99Ms: [], peakTimerLagMs: finals.map((f) => f.worstTimerLagMs) },
|
|
45
93
|
errors: [...errors].map(([text, count]) => ({ text, count })),
|
|
@@ -50,9 +98,17 @@ export function cloudReport({ origin, room, n, seconds, started, measureAt, endA
|
|
|
50
98
|
if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
|
51
99
|
const opts = {};
|
|
52
100
|
for (let i = 2; i < process.argv.length; i += 2) { if (!process.argv[i].startsWith('--') || process.argv[i + 1] === undefined) throw Error('Use --key value'); opts[process.argv[i].slice(2)] = process.argv[i + 1]; }
|
|
53
|
-
const probe = String(opts.probe ?? '').replace(/\/$/, ''), key = opts.key, origin = new URL(opts.url).origin, n = Number(opts.n ?? 50), seconds = Number(opts.seconds ?? 150);
|
|
54
|
-
const rampMs = Number(opts['ramp-ms'] ?? 50), game = opts.game ?? 'crowd-circuit', room = opts.room ?? `cloud-${n}-${Date.now().toString(36)}`, out = resolve(opts.out ?? `reports/crowd-circuit/${room}`);
|
|
55
|
-
if (
|
|
101
|
+
const probe = String(opts.probe ?? '').replace(/\/$/, ''), key = opts.key, origin = opts.url ? new URL(opts.url).origin : null, n = Number(opts.n ?? 50), seconds = Number(opts.seconds ?? 150);
|
|
102
|
+
const compress = opts.compress === 'on', rampMs = Number(opts['ramp-ms'] ?? 50), game = opts.game ?? 'crowd-circuit', room = opts.room ?? `cloud-${n}-${Date.now().toString(36)}`, out = resolve(opts.out ?? `reports/crowd-circuit/${room}`);
|
|
103
|
+
if (opts.calibrate) {
|
|
104
|
+
// `--calibrate <file>`: only the reference work, here and in the probe; no players.
|
|
105
|
+
const result = await calibrate({ probe, key });
|
|
106
|
+
await writeFile(resolve(opts.calibrate), JSON.stringify(result, null, 2));
|
|
107
|
+
console.log(JSON.stringify({ phase: 'calibrated', localMs: Math.min(...result.localMs), cloudMsARound: result.cloudMsARound, ratio: result.ratio, workerOwnClockMs: result.workerOwnClockMs }));
|
|
108
|
+
process.exit(0);
|
|
109
|
+
}
|
|
110
|
+
const slowEvery = Math.max(0, Math.floor(Number(opts['slow-every'] ?? 0)) || 0);
|
|
111
|
+
if (!probe || !key || !origin || !Number.isInteger(n) || n < 1 || n > 1000 || !(seconds >= 1 && seconds <= 600)) throw Error('--probe, --key, --url, --n 1..1000, --seconds 1..600');
|
|
56
112
|
await mkdir(out, { recursive: true });
|
|
57
113
|
const hello = await fetch(`${probe}/hello`, { headers: { 'x-probe-key': key }, signal: AbortSignal.timeout(15000) });
|
|
58
114
|
if (!hello.ok) throw Error(`The probe did not answer (${hello.status}): is it deployed, and is the key its PROBE_KEY?`);
|
|
@@ -62,26 +118,28 @@ if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.ur
|
|
|
62
118
|
// (fifteen seconds after the last has joined) and when to stop.
|
|
63
119
|
const started = Date.now(), joinAt = started + 5000, measureAt = joinAt + n * rampMs + 15000, endAt = measureAt + seconds * 1000;
|
|
64
120
|
console.log(JSON.stringify({ phase: 'start', generator: 'cloudflare', origin, room, n, seconds, requests: Math.ceil(n / 5), toolkit }));
|
|
65
|
-
const finals = [], failures = []; let lastNote = 0; const progress = new Map();
|
|
121
|
+
const finals = [], failures = []; let lastNote = 0, lost = 0; const progress = new Map();
|
|
66
122
|
await Promise.all(Array.from({ length: Math.ceil(n / 5) }, async (_, r) => {
|
|
67
|
-
const from = r * 5, count = Math.min(5, n - from);
|
|
123
|
+
const from = r * 5, count = Math.min(5, n - from); let partial = null;
|
|
124
|
+
// Not all at once: a thousand players are two hundred requests.
|
|
125
|
+
await new Promise((done) => setTimeout(done, r * 10));
|
|
68
126
|
try {
|
|
69
|
-
const res = await fetch(`${probe}/play?origin=${encodeURIComponent(origin)}&game=${encodeURIComponent(game)}&room=${encodeURIComponent(room)}&from=${from}&count=${count}&joinAt=${joinAt}&rampMs=${rampMs}&measureAt=${measureAt}&endAt=${endAt}`, { headers: { 'x-probe-key': key }, signal: AbortSignal.timeout(endAt - Date.now() +
|
|
127
|
+
const res = await fetch(`${probe}/play?origin=${encodeURIComponent(origin)}&game=${encodeURIComponent(game)}&room=${encodeURIComponent(room)}&from=${from}&count=${count}&joinAt=${joinAt}&rampMs=${rampMs}&measureAt=${measureAt}&endAt=${endAt}${compress ? '&compress=1' : ''}${slowEvery ? `&slowEvery=${slowEvery}` : ''}`, { headers: { 'x-probe-key': key }, signal: AbortSignal.timeout(endAt - Date.now() + 30000) });
|
|
70
128
|
if (!res.ok || !res.body) throw Error(`probe HTTP ${res.status}`);
|
|
71
|
-
let text = '', final = null;
|
|
129
|
+
let text = '', final = null; partial = null;
|
|
72
130
|
for await (const chunk of res.body.pipeThrough(new TextDecoderStream())) {
|
|
73
131
|
text += chunk;
|
|
74
132
|
for (let at = text.indexOf('\n'); at >= 0; at = text.indexOf('\n')) {
|
|
75
133
|
const row = JSON.parse(text.slice(0, at)); text = text.slice(at + 1);
|
|
76
|
-
if (row.t === 'final') final = row; else if (row.t === 'failed') throw Error(row.error); else if (row.t === 'sample') progress.set(from, row);
|
|
134
|
+
if (row.t === 'final') final = row; else if (row.t === 'partial') partial = row; else if (row.t === 'failed') throw Error(row.error); else if (row.t === 'sample') progress.set(from, row);
|
|
77
135
|
}
|
|
78
136
|
if (Date.now() - lastNote > 10000) { lastNote = Date.now(); const rows = [...progress.values()]; console.log(JSON.stringify({ phase: rows.some((x) => x.measuring) ? 'steady' : 'ramp', joined: rows.reduce((a, x) => a + x.joined, 0), open: rows.reduce((a, x) => a + x.open, 0) })); }
|
|
79
137
|
}
|
|
80
138
|
if (!final) throw Error('the probe ended without its report');
|
|
81
139
|
finals.push(final);
|
|
82
|
-
} catch (error) { failures.push(`probe request ${from}: ${String(error?.message ?? error).slice(0, 160)}`); }
|
|
140
|
+
} catch (error) { failures.push(`probe request ${from}: ${String(error?.message ?? error).slice(0, 160)}`); if (partial) finals.push(partial); else lost += 1; }
|
|
83
141
|
}));
|
|
84
|
-
const report = cloudReport({ origin, room, n, seconds, started, measureAt, endAt, finals, failures, toolkit });
|
|
142
|
+
const report = cloudReport({ origin, room, n, seconds, started, measureAt, endAt, finals, failures, toolkit, compress, lost });
|
|
85
143
|
await writeFile(`${out}/report.json`, JSON.stringify(report, null, 2));
|
|
86
144
|
console.log(JSON.stringify({ phase: 'done', out: `${out}/report.json`, joined: report.joined, connectedAtEnd: report.connectedAtEnd, ack: report.inputAckMs, loop: report.inputLoopMs, deliveredHz: report.deliveredHz, lead: report.inputLead, failedRequests: failures.length }));
|
|
87
145
|
if (!finals.length) process.exitCode = 1;
|
package/lib/crowd-load.mjs
CHANGED
|
@@ -31,7 +31,8 @@ for (const k of Object.keys(opts))
|
|
|
31
31
|
'churn',
|
|
32
32
|
'churn-at',
|
|
33
33
|
'compress',
|
|
34
|
-
'strict-chrome'
|
|
34
|
+
'strict-chrome',
|
|
35
|
+
'slow-every'
|
|
35
36
|
].includes(k)
|
|
36
37
|
)
|
|
37
38
|
throw Error('Unknown option --' + k);
|
|
@@ -127,7 +128,8 @@ function makeWorker(index) {
|
|
|
127
128
|
room,
|
|
128
129
|
index,
|
|
129
130
|
workers: workerCount,
|
|
130
|
-
compress: opts.compress !== 'off'
|
|
131
|
+
compress: opts.compress !== 'off',
|
|
132
|
+
slowEvery: Number(opts['slow-every'] ?? 0)
|
|
131
133
|
}
|
|
132
134
|
});
|
|
133
135
|
let sequence = 0;
|
|
@@ -474,7 +476,7 @@ try {
|
|
|
474
476
|
);
|
|
475
477
|
loop.disable();
|
|
476
478
|
const histograms = {};
|
|
477
|
-
for (const key of ['first', 'welcome', 'ack', 'gaps', 'loops']) {
|
|
479
|
+
for (const key of ['first', 'welcome', 'ack', 'slowAck', 'gaps', 'loops']) {
|
|
478
480
|
const h = new Histogram();
|
|
479
481
|
for (const f of finals) h.merge(f.histograms[key]);
|
|
480
482
|
histograms[key] = h.json();
|
|
@@ -518,6 +520,8 @@ try {
|
|
|
518
520
|
firstStateMs: histograms.first,
|
|
519
521
|
welcomeMs: histograms.welcome,
|
|
520
522
|
inputAckMs: histograms.ack,
|
|
523
|
+
// With --slow-every: the slow players' own acknowledgements, kept out of everybody else's.
|
|
524
|
+
slowInputAckMs: histograms.slowAck,
|
|
521
525
|
receiveGapMs: histograms.gaps,
|
|
522
526
|
inputLoopMs: histograms.loops,
|
|
523
527
|
ackSkipped: sum('ackSkipped'),
|
|
@@ -541,6 +545,14 @@ try {
|
|
|
541
545
|
wireDownBytes: sum('wireDownBytes'),
|
|
542
546
|
compressedSockets: sum('compressed')
|
|
543
547
|
},
|
|
548
|
+
// --slow-every K: every Kth simulated player reads for one second in five (so every Gate holds some). `healthy` is everybody else: what the
|
|
549
|
+
// slow ones cost them is the difference from a run without any.
|
|
550
|
+
slowPlayers: players.filter((p) => p.slow).length,
|
|
551
|
+
healthy: (() => {
|
|
552
|
+
const a = players.filter((p) => !p.slow && !p.retired), of = (key) => a.map((p) => p[key]).filter(Number.isFinite).sort((x, y) => x - y);
|
|
553
|
+
const hz = of('deliveredHz'), gap = of('maxStateGapMs');
|
|
554
|
+
return { players: a.length, deliveredHz: { min: hz[0] ?? null, median: hz[Math.floor(hz.length / 2)] ?? null }, maxStateGapMs: { median: gap[Math.floor(gap.length / 2)] ?? null, max: gap.at(-1) ?? null } };
|
|
555
|
+
})(),
|
|
544
556
|
// Samples in which a Chrome player was not a connected replica of this room.
|
|
545
557
|
chromeTrouble,
|
|
546
558
|
// Whether the server agreed to permessage-deflate on these sockets, as a
|
|
@@ -48,6 +48,64 @@ node node_modules/@homie-rocks/studio/lib/crowd-cloud.mjs \
|
|
|
48
48
|
It opens one request to the probe for every five players, each held open for the step, adds up what they
|
|
49
49
|
measured, and writes `report.json` in the shape `crowd-load.mjs` writes, so `crowd-summary.mjs` reads both.
|
|
50
50
|
|
|
51
|
+
## Slow players among them
|
|
52
|
+
|
|
53
|
+
`--slow-every 10` makes every tenth player a slow one: it takes what it is sent at six frames a second, in order,
|
|
54
|
+
as a connection that carries no more does, and says which tick it holds. The report then has `healthy` (everybody
|
|
55
|
+
else) beside `slow`: what a slow player costs the players on the same Gate is the difference from the same step
|
|
56
|
+
without any. A Worker cannot slow a socket's reads, so this is a slow player and not a slow socket; for a socket
|
|
57
|
+
that stops reading, `crowd-load.mjs --slow-every 10` from a computer pauses every tenth player's socket for four
|
|
58
|
+
seconds in five.
|
|
59
|
+
|
|
60
|
+
## How fast Cloudflare runs, against your computer
|
|
61
|
+
|
|
62
|
+
```sh
|
|
63
|
+
node node_modules/@homie-rocks/studio/lib/crowd-cloud.mjs \
|
|
64
|
+
--probe https://homie-crowd-probe.<account>.workers.dev --key $KEY --calibrate reports/calibration.json
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
A Worker's clock stands still while its code runs, so nothing timed on a computer says by itself what the same
|
|
68
|
+
work costs on Cloudflare, and a Worker cannot time it either. This runs one fixed piece of work
|
|
69
|
+
(`reference-work.mjs`) in the probe, ten rounds a request, with the stopwatch on your computer (a request that
|
|
70
|
+
does the work less one that does none, the quickest of several), and the same work on your computer, and writes
|
|
71
|
+
the ratio. `crowd-summary.mjs` prints it under a run's tables.
|
|
72
|
+
|
|
73
|
+
## A whole trial
|
|
74
|
+
|
|
75
|
+
`run-trial.sh` in this folder runs a plan of steps against a deployed studio: each from your computer and from
|
|
76
|
+
inside Cloudflare, the in-Cloudflare steps three times each, one with slow players, the room's own figures beside
|
|
77
|
+
every step, the calibration before and after, and one `summary.md`. It deploys nothing.
|
|
78
|
+
|
|
79
|
+
```sh
|
|
80
|
+
cd <your studio>
|
|
81
|
+
URL=https://<studio> WORKER=<its worker's name> PROBE_URL=https://homie-crowd-probe.<account>.workers.dev PROBE_KEY=$KEY \
|
|
82
|
+
bash node_modules/@homie-rocks/studio/lib/crowd-probe/run-trial.sh
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
`PLAN=$'300 cloud 3\n600 cloud 3'` chooses the steps ("players mode runs"; the modes are in the script).
|
|
86
|
+
|
|
87
|
+
## What a room carries, and how to read a trial
|
|
88
|
+
|
|
89
|
+
One room's rules run in one object on one thread. With the crowd-circuit fixture (light rules, 20 ticks a second)
|
|
90
|
+
that was 300 players with margin on Cloudflare, 600 at the edge, and 800 to 1,000 only by luck: the same step gave
|
|
91
|
+
10 frames a second or 20, minutes apart. A game with heavier rules carries fewer.
|
|
92
|
+
|
|
93
|
+
- Run the size you want and twice that size, three times each, at a busy hour. A size is carried when every run
|
|
94
|
+
is clean and twice the size is not falling apart. One clean run proves nothing: at its edge a room is clean or
|
|
95
|
+
not by what else its machine is doing.
|
|
96
|
+
- Read **a Gate's round trip to the room** ("Gate rtt"), **the share of worlds over 200 ms old** and **"server
|
|
97
|
+
late"** (inputs that reached the rules after their tick). They are measured from outside the room.
|
|
98
|
+
- Do not read the room's word about itself. On Cloudflare a Worker's clock stands still while its code runs and
|
|
99
|
+
reads a timer's due time when the timer fires, so a room that is seconds behind counts no late ticks. For the
|
|
100
|
+
same reason a probe that runs late reports rates that read high; the report's delivered frames a second are
|
|
101
|
+
counted against the room's ticks, not a clock.
|
|
102
|
+
- Cloudflare's count of the room's processor time is not in a `wrangler tail` (it reads under 1% for a room that
|
|
103
|
+
is out of time). It is in the GraphQL analytics API (`durableObjectsPeriodicGroups`), which needs an API token
|
|
104
|
+
that may read account analytics.
|
|
105
|
+
- `node packages/studio/test/crowd-room-cost.mjs` in the Homie repository says what a tick of the room costs on a
|
|
106
|
+
computer, divided into rules, input and worlds. With the calibration above it compares one game with another;
|
|
107
|
+
it does not predict where a room falls.
|
|
108
|
+
|
|
51
109
|
## Remove
|
|
52
110
|
|
|
53
111
|
```sh
|
|
@@ -90,6 +90,9 @@ export function crowdClient({ index, game, send, tallies, period = 50, lead = 3
|
|
|
90
90
|
const span = c.lastAt > c.firstAt ? c.lastAt - c.firstAt : 0;
|
|
91
91
|
return { i: index, seat: c.seat, firstStateMs: c.first, welcomeMs: c.welcomed, closed: c.closed, lead: c.lead, frames: c.frames, maxStateGapMs: c.maxGap,
|
|
92
92
|
tickHz: span ? (c.lastTick - c.firstTick) * 1000 / span : null, deliveredHz: span ? (c.frames - 1) * 1000 / span : null,
|
|
93
|
+
// Frames received for each tick the room advanced: 1 is every frame. It needs no clock, which matters where
|
|
94
|
+
// the caller's clock is a Worker's (it reads a timer's due time, and runs slow when the Worker runs late).
|
|
95
|
+
framesPerTick: c.lastTick > c.firstTick ? (c.frames - 1) / (c.lastTick - c.firstTick) : null,
|
|
93
96
|
downBytes: c.down, upBytes: c.up, inputs: c.inputs, reports: c.reports, late: c.late, undecodable: c.undecodable, ownMissing: c.ownMissing, errors: c.errors.slice(0, 8), fresh: c.at ? true : false };
|
|
94
97
|
},
|
|
95
98
|
};
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* A fixed piece of work, the same everywhere it runs: what a room's thread spends its time on, in miniature (parsing
|
|
3
|
+
* players' small messages, building and reading plain objects, arithmetic on floats, writing a block of text).
|
|
4
|
+
*
|
|
5
|
+
* A Worker cannot time a piece of its own work (its clock stands still while its code runs, and reads a timer's due
|
|
6
|
+
* time when the timer fires), so nothing measured on a laptop says by itself what a room costs on Cloudflare. This
|
|
7
|
+
* runs on both: the probe does it inside Cloudflare while `crowd-cloud.mjs --calibrate` holds the stopwatch on the
|
|
8
|
+
* request, and the same call is timed on the computer that asked. The ratio turns `test/crowd-room-cost.mjs`'s
|
|
9
|
+
* milliseconds into Cloudflare's.
|
|
10
|
+
*
|
|
11
|
+
* It returns a number that depends on every step, so no engine may skip the work, and the same number everywhere.
|
|
12
|
+
*/
|
|
13
|
+
export function referenceWork(rounds = 40) {
|
|
14
|
+
let check = 0, seed = 20261011;
|
|
15
|
+
const random = () => { seed = (Math.imul(seed, 1664525) + 1013904223) >>> 0; return seed / 4294967296; };
|
|
16
|
+
for (let round = 0; round < rounds; round += 1) {
|
|
17
|
+
// A tick's inputs: five hundred small messages parsed, held to numbers, queued by seat.
|
|
18
|
+
const queues = new Map();
|
|
19
|
+
for (let i = 0; i < 500; i += 1) {
|
|
20
|
+
const text = JSON.stringify({ t: 'in', e: 3914181188, k: 3000 + round, s: [[0, i % 2 ? 127 : -127, Math.round(random() * 70 - 35)]], r: i & 7, g: 2997 + round });
|
|
21
|
+
const m = JSON.parse(text), seat = i * 2;
|
|
22
|
+
const entry = { at: m.k >>> 0, values: { ax: Math.max(-127, Math.min(127, m.s[0][1] | 0)), ay: Math.max(-127, Math.min(127, m.s[0][2] | 0)) } };
|
|
23
|
+
let q = queues.get(seat); if (!q) { q = []; queues.set(seat, q); }
|
|
24
|
+
q.push(entry); check = (check + entry.values.ax + entry.values.ay + text.length) | 0;
|
|
25
|
+
}
|
|
26
|
+
// A tick of a thousand bodies: frozen vectors made and read, a little float arithmetic each.
|
|
27
|
+
const bodies = [];
|
|
28
|
+
for (let i = 0; i < 1000; i += 1) {
|
|
29
|
+
const q = queues.get(i), held = q ? q[q.length - 1].values : { ax: 0, ay: 0 };
|
|
30
|
+
const vel = Object.freeze({ x: Math.fround(held.ax / 127 * 6), y: Math.fround(held.ay / 127 * 6), z: 0 });
|
|
31
|
+
const pos = Object.freeze({ x: Math.fround(random() * 16 - 8 + vel.x * 0.05), y: Math.fround(random() * 120 - 60 + vel.y * 0.05), z: 0 });
|
|
32
|
+
const speed = Math.hypot(vel.x, vel.y);
|
|
33
|
+
bodies.push([`r${i}`, 0, round & 0xffff, [pos.x, pos.y], [vel.x, vel.y], [speed > 0 ? vel.x / speed : 1, speed > 0 ? vel.y / speed : 0], 0, [round, pos.x > 0], [], i, 0]);
|
|
34
|
+
check = (check + Math.round(pos.x * 100) + Math.round(pos.y * 100)) | 0;
|
|
35
|
+
}
|
|
36
|
+
// The world as it leaves: a block of numbers, and the room's checkpoint as text.
|
|
37
|
+
const block = new Float32Array(bodies.length * 6);
|
|
38
|
+
for (const [i, b] of bodies.entries()) block.set([b[3][0], b[3][1], b[4][0], b[4][1], b[5][0], b[5][1]], i * 6);
|
|
39
|
+
const bytes = new Uint8Array(block.buffer); let sum = 0;
|
|
40
|
+
for (let i = 0; i < bytes.length; i += 97) sum += bytes[i];
|
|
41
|
+
const text = JSON.stringify(bodies);
|
|
42
|
+
check = (check + sum + text.length + JSON.parse(text).length) | 0;
|
|
43
|
+
}
|
|
44
|
+
return check;
|
|
45
|
+
}
|
|
@@ -0,0 +1,207 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# A crowd trial of one game of a deployed studio, on your own Cloudflare account: each step from your computer and
|
|
3
|
+
# from inside Cloudflare, the in-Cloudflare steps three times each (one run does not tell a limit from luck), a
|
|
4
|
+
# step with slow players among healthy ones, the room's own figures a second beside every step, how fast Cloudflare
|
|
5
|
+
# ran a fixed piece of work against this computer, and one summary.md at the end.
|
|
6
|
+
#
|
|
7
|
+
# It deploys nothing and changes nothing in the studio. It writes under reports/<game>/trial/<stamp>/ in the
|
|
8
|
+
# studio's folder, creates temporary office keys for the diagnostics and revokes them.
|
|
9
|
+
#
|
|
10
|
+
# cd <your studio> # deployed, with this toolkit installed
|
|
11
|
+
# (cd node_modules/@homie-rocks/studio/lib/crowd-probe && KEY=$(openssl rand -hex 24) && npx wrangler deploy --var PROBE_KEY:$KEY)
|
|
12
|
+
# URL=https://<studio> WORKER=<its worker's name> PROBE_URL=https://homie-crowd-probe.<account>.workers.dev PROBE_KEY=$KEY \
|
|
13
|
+
# bash node_modules/@homie-rocks/studio/lib/crowd-probe/run-trial.sh
|
|
14
|
+
# npx wrangler delete --name homie-crowd-probe # when the trial is over
|
|
15
|
+
#
|
|
16
|
+
# Without PROBE_URL only the steps from this computer run. The players steer as the crowd-circuit fixture does:
|
|
17
|
+
# for another game, change the input lib/crowd-probe/client.mjs and lib/crowd-worker.mjs send.
|
|
18
|
+
#
|
|
19
|
+
# GAME=crowd-circuit the game's id
|
|
20
|
+
# PLAN=$'300 home\n600 cloud 3' a different plan: "players mode [runs]" a line (modes below)
|
|
21
|
+
# LABEL=evening a word added to the report folder's name
|
|
22
|
+
# SECONDS_PER_STEP=150 steady measurement per step
|
|
23
|
+
# EXPECTED_RUNTIME_SHA=<sha256> stop unless the installed and the deployed toolkit are this build
|
|
24
|
+
# CHROME_PATH=/path/to/chrome for the two real Chrome players in a step from this computer
|
|
25
|
+
#
|
|
26
|
+
# Read docs/crowd-transport-0462.md first for what each column can and cannot say. In short: nothing an object
|
|
27
|
+
# measures about itself by its own clock is to be trusted on Cloudflare; a Gate's round trip to the room, the share
|
|
28
|
+
# of worlds over 200 ms old and the players' own loop are the figures that show a room out of time.
|
|
29
|
+
set -euo pipefail
|
|
30
|
+
[ -d node_modules/@homie-rocks/studio ] || { echo 'Run this in a studio folder with @homie-rocks/studio installed.'; exit 1; }
|
|
31
|
+
: "${URL:?URL=https://<your studio>}"
|
|
32
|
+
: "${WORKER:?WORKER=<the name of the studio Worker, as in wrangler.jsonc>}"
|
|
33
|
+
GAME="${GAME:-crowd-circuit}"
|
|
34
|
+
export WRANGLER_SEND_METRICS=false
|
|
35
|
+
# The probe inflates every frame, as a browser does; give zlib's pool room.
|
|
36
|
+
export UV_THREADPOOL_SIZE=16
|
|
37
|
+
TOOLKIT="$PWD/node_modules/@homie-rocks/studio"
|
|
38
|
+
SECONDS_PER_STEP="${SECONDS_PER_STEP:-150}"
|
|
39
|
+
PROBE_URL="${PROBE_URL:-}"
|
|
40
|
+
PROBE_KEY="${PROBE_KEY:-}"
|
|
41
|
+
STAMP=$(date -u +%Y%m%dT%H%M%SZ)
|
|
42
|
+
OUT="$PWD/reports/$GAME/trial/$STAMP${LABEL:+-$LABEL}"
|
|
43
|
+
mkdir -p "$OUT"
|
|
44
|
+
|
|
45
|
+
# 1. The deployed Worker is this toolkit, byte for byte.
|
|
46
|
+
node --input-type=module - "$URL" "$TOOLKIT" "$OUT" <<'JS'
|
|
47
|
+
import { pathToFileURL } from 'node:url';
|
|
48
|
+
import { writeFile } from 'node:fs/promises';
|
|
49
|
+
const [url, toolkit, out] = process.argv.slice(2);
|
|
50
|
+
const { STUDIO_BUILD } = await import(pathToFileURL(`${toolkit}/worker/build-id.mjs`));
|
|
51
|
+
const expected = process.env.EXPECTED_RUNTIME_SHA;
|
|
52
|
+
if (expected && STUDIO_BUILD.sha256 !== expected) throw Error(`Installed toolkit's runtime is ${STUDIO_BUILD.sha256}, not this build's ${expected}: install that build first`);
|
|
53
|
+
const response = await fetch(`${url}/.well-known/homie-studio.json?verify=${Date.now()}`);
|
|
54
|
+
if (!response.ok) throw Error(`manifest HTTP ${response.status}`);
|
|
55
|
+
const manifest = await response.json();
|
|
56
|
+
await writeFile(`${out}/deployment.json`, JSON.stringify(manifest, null, 2));
|
|
57
|
+
if (manifest.toolkit?.sha256 !== STUDIO_BUILD.sha256 || manifest.toolkit?.version !== STUDIO_BUILD.version)
|
|
58
|
+
throw Error(`Wrong deployed toolkit: expected ${JSON.stringify(STUDIO_BUILD)}, got ${JSON.stringify(manifest.toolkit)}`);
|
|
59
|
+
console.log('Verified deployed runtime', STUDIO_BUILD);
|
|
60
|
+
JS
|
|
61
|
+
|
|
62
|
+
# What this connection carries: twelve parallel downloads from Cloudflare for eight seconds, to nowhere.
|
|
63
|
+
probe_link() {
|
|
64
|
+
local file="$1" tmp start end bytes pids=""
|
|
65
|
+
tmp=$(mktemp -d)
|
|
66
|
+
start=$(node -e 'console.log(Date.now())')
|
|
67
|
+
for i in 1 2 3 4 5 6 7 8 9 10 11 12; do
|
|
68
|
+
curl -s -o /dev/null -H 'Referer: https://speed.cloudflare.com/' -A 'Mozilla/5.0' --max-time 8 \
|
|
69
|
+
-w '%{size_download}\n' 'https://speed.cloudflare.com/__down?bytes=100000000' > "$tmp/$i" 2>/dev/null &
|
|
70
|
+
pids="$pids $!"
|
|
71
|
+
done
|
|
72
|
+
wait $pids
|
|
73
|
+
end=$(node -e 'console.log(Date.now())')
|
|
74
|
+
bytes=$(cat "$tmp"/* 2>/dev/null | awk '{s+=$1} END {print s+0}')
|
|
75
|
+
rm -rf "$tmp"
|
|
76
|
+
node -e '
|
|
77
|
+
const [file, bytes, start, end] = process.argv.slice(1);
|
|
78
|
+
const seconds = (end - start) / 1000, down = bytes > 0 && seconds > 0 ? bytes / seconds / 1e6 : null;
|
|
79
|
+
require("fs").writeFileSync(file, JSON.stringify({ at: new Date().toISOString(), streams: 12, seconds, bytes: Number(bytes), downMBps: down }, null, 2));
|
|
80
|
+
console.log("Link before this step:", down === null ? "not measured" : down.toFixed(2) + " MB/s (" + (down * 8).toFixed(0) + " Mbit/s)");
|
|
81
|
+
' "$file" "$bytes" "$start" "$end"
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
TAIL_PID=
|
|
85
|
+
DIAG_PID=
|
|
86
|
+
cleanup() {
|
|
87
|
+
if [ -n "$DIAG_PID" ]; then kill -TERM "$DIAG_PID" 2>/dev/null || true; wait "$DIAG_PID" 2>/dev/null || true; fi
|
|
88
|
+
if [ -n "$TAIL_PID" ]; then kill -TERM "$TAIL_PID" 2>/dev/null || true; wait "$TAIL_PID" 2>/dev/null || true; fi
|
|
89
|
+
# Whatever finished is summarised, even when a step failed.
|
|
90
|
+
node "$TOOLKIT/lib/crowd-summary.mjs" "$OUT" || true
|
|
91
|
+
printf 'All reports: %s\nSummary: %s\n' "$OUT" "$OUT/summary.md"
|
|
92
|
+
}
|
|
93
|
+
trap cleanup EXIT
|
|
94
|
+
trap 'exit 130' INT TERM
|
|
95
|
+
|
|
96
|
+
# The tail is sampled by Cloudflare under load and is not the record; diagnostics.jsonl is.
|
|
97
|
+
# Only failures are kept, so the sample is not spent on healthy requests.
|
|
98
|
+
./node_modules/.bin/wrangler tail "$WORKER" --format json --status error > "$OUT/worker-tail.jsonl" 2> "$OUT/worker-tail.stderr" &
|
|
99
|
+
TAIL_PID=$!
|
|
100
|
+
sleep 3
|
|
101
|
+
kill -0 "$TAIL_PID" 2>/dev/null || { printf 'Worker tail did not start (the run continues without it): %s\n' "$OUT/worker-tail.stderr"; TAIL_PID=; }
|
|
102
|
+
# One step: the diagnostics collector beside a driver. $1 the step's folder, $2 the room, the rest the driver's command.
|
|
103
|
+
run_step() {
|
|
104
|
+
local STEP="$1" ROOM="$2"; shift 2
|
|
105
|
+
mkdir -p "$STEP"
|
|
106
|
+
# The collector keeps a failed poll as a row and carries on; it is stopped when the step ends.
|
|
107
|
+
node "$TOOLKIT/lib/crowd-diagnostics.mjs" --root "$PWD" --url "$URL" \
|
|
108
|
+
--game "$GAME" --room "$ROOM" --seconds $((SECONDS_PER_STEP + 600)) --out "$STEP/diagnostics.jsonl" \
|
|
109
|
+
> "$STEP/diagnostics.log" 2>&1 &
|
|
110
|
+
DIAG_PID=$!
|
|
111
|
+
local READY=0 WAIT
|
|
112
|
+
for WAIT in $(seq 1 60); do
|
|
113
|
+
if [ -s "$STEP/diagnostics.jsonl" ]; then READY=1; break; fi
|
|
114
|
+
kill -0 "$DIAG_PID" 2>/dev/null || break
|
|
115
|
+
sleep 1
|
|
116
|
+
done
|
|
117
|
+
if [ "$READY" -ne 1 ]; then printf 'Diagnostics did not start for %s; the step runs without them:\n' "$ROOM"; cat "$STEP/diagnostics.log" || true; fi
|
|
118
|
+
local STATUS=0
|
|
119
|
+
"$@" > "$STEP.log" 2>&1 || STATUS=$?
|
|
120
|
+
printf '%s\n' "$STATUS" > "$STEP/driver-exit-code.txt"
|
|
121
|
+
kill -TERM "$DIAG_PID" 2>/dev/null || true
|
|
122
|
+
local DIAG_STATUS=0
|
|
123
|
+
wait "$DIAG_PID" 2>/dev/null || DIAG_STATUS=$?
|
|
124
|
+
printf '%s\n' "$DIAG_STATUS" > "$STEP/diagnostics-exit-code.txt"
|
|
125
|
+
DIAG_PID=
|
|
126
|
+
local ROWS FAILED
|
|
127
|
+
ROWS=$(wc -l < "$STEP/diagnostics.jsonl" 2>/dev/null | tr -d ' ' || echo 0)
|
|
128
|
+
FAILED=$(grep -c '"ok":false' "$STEP/diagnostics.jsonl" 2>/dev/null || true)
|
|
129
|
+
printf '%s finished (driver exit %s; diagnostics %s rows, %s failed polls): %s\n' "$ROOM" "$STATUS" "$ROWS" "${FAILED:-0}" "$STEP/report.json"
|
|
130
|
+
if [ -n "$TAIL_PID" ] && ! kill -0 "$TAIL_PID" 2>/dev/null; then
|
|
131
|
+
printf 'Worker tail stopped; inspect %s\n' "$OUT/worker-tail.stderr"; TAIL_PID=
|
|
132
|
+
fi
|
|
133
|
+
# A Gate gives its channel to the room back ten seconds after its last player leaves: the next step starts clean.
|
|
134
|
+
sleep 15
|
|
135
|
+
}
|
|
136
|
+
|
|
137
|
+
if [ -n "$PROBE_URL" ]; then
|
|
138
|
+
HELLO=$(curl -fsS -H "x-probe-key: $PROBE_KEY" "$PROBE_URL/hello" || true)
|
|
139
|
+
case "$HELLO" in
|
|
140
|
+
*'"probe":2'*) ;;
|
|
141
|
+
'') printf 'The probe at %s did not answer with that key; running from this computer only.\n' "$PROBE_URL"; PROBE_URL= ;;
|
|
142
|
+
*) printf 'The probe at %s is an older one (%s): deploy it again from the installed toolkit (lib/crowd-probe): it has the slow players and the calibration.\n' "$PROBE_URL" "$HELLO"; exit 1 ;;
|
|
143
|
+
esac
|
|
144
|
+
fi
|
|
145
|
+
|
|
146
|
+
# How fast Cloudflare runs a fixed piece of work against this computer: $1 the file. Eight probe requests.
|
|
147
|
+
calibrate() {
|
|
148
|
+
[ -n "$PROBE_URL" ] || return 0
|
|
149
|
+
node "$TOOLKIT/lib/crowd-cloud.mjs" --probe "$PROBE_URL" --key "$PROBE_KEY" --calibrate "$1" || true
|
|
150
|
+
}
|
|
151
|
+
uptime > "$OUT/load-average-start.txt" || true
|
|
152
|
+
calibrate "$OUT/calibration-start.json"
|
|
153
|
+
|
|
154
|
+
# The plan: one step a line, "players mode [repeat]". Modes:
|
|
155
|
+
# home this computer, compression on (what one home connection carries)
|
|
156
|
+
# home-slow the same with every tenth player's socket not read for four seconds in five (a real slow socket)
|
|
157
|
+
# home-plain this computer, compression off
|
|
158
|
+
# cloud inside Cloudflare, compression off
|
|
159
|
+
# cloud-deflate inside Cloudflare, compression on
|
|
160
|
+
# cloud-slow, cloud-deflate-slow the same with every tenth player slow (it takes six frames a second)
|
|
161
|
+
# The cloud steps run three times each: one run of each was not enough to tell a limit from luck.
|
|
162
|
+
# From home: sizes one connection carries with room to spare. A thousand players on one connection measure the connection.
|
|
163
|
+
PLAN="${PLAN:-300 home
|
|
164
|
+
300 home-slow
|
|
165
|
+
600 home
|
|
166
|
+
800 cloud 3
|
|
167
|
+
800 cloud-deflate 3
|
|
168
|
+
900 cloud 3
|
|
169
|
+
900 cloud-deflate 3
|
|
170
|
+
1000 cloud 3
|
|
171
|
+
1000 cloud-deflate 3
|
|
172
|
+
1000 cloud-slow
|
|
173
|
+
1000 cloud-deflate-slow}"
|
|
174
|
+
printf '%s\n' "$PLAN" | while read -r N MODE REPEAT; do
|
|
175
|
+
[ -n "$N" ] || continue
|
|
176
|
+
for RUN in $(seq 1 "${REPEAT:-1}"); do
|
|
177
|
+
SUFFIX=""; [ "$RUN" -gt 1 ] && SUFFIX="-r$RUN"
|
|
178
|
+
case "$MODE" in
|
|
179
|
+
home|home-plain|home-slow)
|
|
180
|
+
STEP="$OUT/n$N"; COMPRESS=on; EXTRA=()
|
|
181
|
+
if [ "$MODE" = home-plain ]; then STEP="$OUT/n$N-home-plain"; COMPRESS=off; fi
|
|
182
|
+
if [ "$MODE" = home-slow ]; then STEP="$OUT/n$N-home-slow"; EXTRA=(--slow-every 10); fi
|
|
183
|
+
STEP="$STEP$SUFFIX"
|
|
184
|
+
mkdir -p "$STEP"
|
|
185
|
+
uptime > "$STEP/load-average.txt" || true
|
|
186
|
+
probe_link "$STEP/link.json" || true
|
|
187
|
+
# A full room is 998 simulated and two Chrome players.
|
|
188
|
+
SIZE=(--n "$N")
|
|
189
|
+
if [ "$N" -ge 999 ]; then SIZE=(--total 1000); fi
|
|
190
|
+
ROOM="h$(printf '%s' "$MODE" | cut -c6-7)$N-$RUN-$STAMP"
|
|
191
|
+
run_step "$STEP" "$ROOM" node "$TOOLKIT/lib/crowd-load.mjs" --url "$URL" "${SIZE[@]}" \
|
|
192
|
+
--workers 4 --seconds "$SECONDS_PER_STEP" --ramp-ms 50 --rss-mb 3072 --compress "$COMPRESS" ${EXTRA[@]+"${EXTRA[@]}"} --room "$ROOM" --out "$STEP" < /dev/null
|
|
193
|
+
;;
|
|
194
|
+
cloud|cloud-deflate|cloud-slow|cloud-deflate-slow)
|
|
195
|
+
if [ -z "$PROBE_URL" ]; then continue; fi
|
|
196
|
+
STEP="$OUT/n$N-$MODE$SUFFIX"; COMPRESS=off; EXTRA=(); TAG=c
|
|
197
|
+
case "$MODE" in cloud-deflate*) COMPRESS=on; TAG=cd ;; esac
|
|
198
|
+
case "$MODE" in *-slow) EXTRA=(--slow-every 10); TAG="${TAG}s" ;; esac
|
|
199
|
+
ROOM="$TAG$N-$RUN-$STAMP"
|
|
200
|
+
run_step "$STEP" "$ROOM" node "$TOOLKIT/lib/crowd-cloud.mjs" --probe "$PROBE_URL" --key "$PROBE_KEY" \
|
|
201
|
+
--url "$URL" --n "$N" --seconds "$SECONDS_PER_STEP" --ramp-ms 50 --compress "$COMPRESS" --game "$GAME" ${EXTRA[@]+"${EXTRA[@]}"} --room "$ROOM" --out "$STEP" < /dev/null
|
|
202
|
+
;;
|
|
203
|
+
esac
|
|
204
|
+
done
|
|
205
|
+
done
|
|
206
|
+
calibrate "$OUT/calibration-end.json"
|
|
207
|
+
uptime > "$OUT/load-average-end.txt" || true
|