@homie-rocks/studio 0.18.2 → 0.19.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +31 -0
- package/bin/homie-studio.mjs +54 -4
- package/lib/build.mjs +20 -2
- package/lib/perf-profile.mjs +171 -0
- package/lib/perf-stats.mjs +249 -0
- package/lib/perf.mjs +680 -0
- package/lib/scaffold.mjs +8 -0
- package/lib/template-history.json +37 -0
- package/package.json +1 -1
- package/worker/version.mjs +1 -1
package/README.md
CHANGED
|
@@ -15,6 +15,7 @@ cd night-owls && npm install
|
|
|
15
15
|
npx homie-studio game new crown-thief --from gem-rush --name "Crown Thief"
|
|
16
16
|
npx homie-studio dev # the whole site locally
|
|
17
17
|
npx homie-studio check crown-thief --url http://127.0.0.1:8787
|
|
18
|
+
npx homie-studio perf crown-thief --url http://127.0.0.1:8787 # how fast it runs: a computer and an emulated phone, host and replica
|
|
18
19
|
npx homie-studio deploy --plan # what deploy will create, and what it costs; changes nothing
|
|
19
20
|
npx homie-studio deploy # the studio's own Cloudflare
|
|
20
21
|
npx homie-studio publish # the homie.rocks directory
|
|
@@ -409,6 +410,36 @@ feed every command behaves exactly as before. `lib/progress.mjs` has the whole f
|
|
|
409
410
|
names the Cloudflare account, often after its owner. A custom domain goes in studio.json
|
|
410
411
|
as `cloudflare.domain` and is what the directory claim, `publish`, `check` and `stats` use.
|
|
411
412
|
|
|
413
|
+
## How fast it runs: `homie-studio perf`
|
|
414
|
+
|
|
415
|
+
```sh
|
|
416
|
+
npx homie-studio perf crown-thief --url http://127.0.0.1:8787 [--device computer,phone] [--runs 3] [--profile]
|
|
417
|
+
npx homie-studio perf sizes crown-thief # every built file, raw and gzipped; with build --maps, the bundle's modules
|
|
418
|
+
npx homie-studio perf compare .perf/crown-thief/<before> .perf/crown-thief/<after> --goal phone.host.frame.p95
|
|
419
|
+
```
|
|
420
|
+
|
|
421
|
+
One run is two headless Chromes on this computer's GPU in a fresh room of their own (`?room=perf-…`): the host (the
|
|
422
|
+
rules, the bots, the snapshots) and a replica, both a computer (1280x800 at 2x) or both an emulated phone (390x844 at
|
|
423
|
+
3x, touch, Chrome's CPU throttle at 4x with `--cpu`, 4G; each run records the slow-down the throttle really gave, which
|
|
424
|
+
on a fast computer is less than its rate). They play the same seeded presses through a warm-up and a measured
|
|
425
|
+
window, and each run's JSON says, per browser: the time between animation frames (median, p95, p99, the share over 33
|
|
426
|
+
and 50 ms), the game's JavaScript per frame (every requestAnimationFrame callback timed), the main thread per frame
|
|
427
|
+
(Chrome's TaskDuration), time to the first frame, to a seat and to playable, the game's files on the wire, the heap
|
|
428
|
+
after a garbage collection, and netplay messages and kilobytes a second each way. `--profile` adds a CPU profile per
|
|
429
|
+
browser in a window of its own (a `.cpuprofile` for Chrome DevTools) and its hottest functions, read through the
|
|
430
|
+
game's source map when the build kept one (`homie-studio build --maps` keeps it in `.studio/maps/<id>/`, never in
|
|
431
|
+
`site/dist`).
|
|
432
|
+
|
|
433
|
+
The computer is shared, so every run waits for the 1-minute load to fall under 0.8 per core (`--max-load`) and
|
|
434
|
+
records the load before and after; a run that started busy says `loaded` and `compare` leaves it out. A software
|
|
435
|
+
renderer (SwiftShader) makes the run `blocked`: nothing on it is judged. `compare` calls a metric better only when a
|
|
436
|
+
rank test says it is unlikely to be chance (p < 0.05), the 95% bootstrap interval of the change stays below zero, and
|
|
437
|
+
it is at least 3% better; runs taken side by side with the same `--pair` label are judged as pairs (a signed-rank
|
|
438
|
+
test on their ratios), so a computer that drifted busier cancels. Guards (frame time, main thread per frame, time to
|
|
439
|
+
playable, heap, the host's upload) must not get worse. Results are files under `.perf/<id>/` (git-ignored); the
|
|
440
|
+
command prints paths and medians. The Homie plugin's `perf` skill runs the whole loop: baseline, one change at a time,
|
|
441
|
+
keep or revert, and a report in the studio's `perf/` folder.
|
|
442
|
+
|
|
412
443
|
## Checks on a computer without a GPU
|
|
413
444
|
|
|
414
445
|
`check`, `port check` and `look` run Chrome headless. On a Mac they use its GPU. On Linux (a Claude Code cloud
|
package/bin/homie-studio.mjs
CHANGED
|
@@ -16,10 +16,26 @@
|
|
|
16
16
|
* homie-studio game new <id> [--from gem-rush] [--name "<Game Name>"]
|
|
17
17
|
* homie-studio game remix <source.json url> --id <new id>
|
|
18
18
|
* homie-studio games
|
|
19
|
-
* homie-studio build [<id>]
|
|
19
|
+
* homie-studio build [<id>] [--maps] (--maps: also keep each bundle's source map and module sizes in .studio/maps/<id>/,
|
|
20
|
+
* never in site/dist: what `perf` reads a CPU profile through)
|
|
20
21
|
* homie-studio dev [--port 8787] [--remote-ai] (--stop: stop exactly this studio's dev server, nothing else;
|
|
21
22
|
* AI guides think scripted here unless --remote-ai: real Workers AI, billed)
|
|
22
23
|
* homie-studio check <id> [--url <site>] [--shots <dir>]
|
|
24
|
+
* homie-studio perf <id> [--url <site>] [--device computer,phone] [--runs 1] [--seconds 15] [--warm 3] [--profile]
|
|
25
|
+
* [--cpu 4] [--out <dir>] [--max-load 0.8] [--pair <label>]
|
|
26
|
+
* (how fast it runs: per device, two browsers in a fresh room, the host and a replica,
|
|
27
|
+
* playing: frame times (median, p95, p99, long frames), the game's JavaScript per frame,
|
|
28
|
+
* the main thread per frame, time to playable and what it downloaded, the heap, netplay
|
|
29
|
+
* messages a second; the computer's load with every run. --profile: a CPU profile of
|
|
30
|
+
* each browser and its hottest functions. Files under .perf/<id>/<time>/; the phone is
|
|
31
|
+
* emulated (a --cpu times slower CPU, 4G) on this computer's GPU. The plugin's perf
|
|
32
|
+
* skill runs the whole measure, change, compare, keep-or-revert loop)
|
|
33
|
+
* homie-studio perf sizes <id> (what a player downloads: every built file, raw and gzipped, the biggest first; with
|
|
34
|
+
* build --maps, which modules make up the bundle)
|
|
35
|
+
* homie-studio perf compare <before dir> <after dir> [--goal phone.host.frame.p95] [--min 3] [--guards a,b] [--also c,d]
|
|
36
|
+
* (better, worse or the same within the noise: medians, a 95% interval, a rank test,
|
|
37
|
+
* and the guards that must not get worse; runs taken side by side with the same
|
|
38
|
+
* --pair label on both sides are judged as pairs, so a computer's drift cancels)
|
|
23
39
|
* homie-studio chrome [install] [--fresh] (which Chrome the checks use; on Linux, `install` fetches Chrome for Testing;
|
|
24
40
|
* --fresh fetches it even when the machine has a Chrome)
|
|
25
41
|
* homie-studio look [<path>...] [--url <site>] [--shots <dir>] [--only computer,phone,sideways]
|
|
@@ -149,6 +165,7 @@ import { publish } from '../lib/directory.mjs';
|
|
|
149
165
|
import { look } from '../lib/look.mjs';
|
|
150
166
|
import { importPort, planPort } from '../lib/port.mjs';
|
|
151
167
|
import { portCheck } from '../lib/port-check.mjs';
|
|
168
|
+
import { DEFAULT_GOAL, perfCompare, perfRun, perfSizes } from '../lib/perf.mjs';
|
|
152
169
|
import { R2_COST, lineOf, mediaPlan, r2OverOf, recordUpload, resolveMedia, sizeOf, typeOf } from '../lib/media.mjs';
|
|
153
170
|
import { ensureMigrations, migrationWord, newStudio } from '../lib/scaffold.mjs';
|
|
154
171
|
import { lineDiff, upgradeApply, upgradePlan } from '../lib/upgrade.mjs';
|
|
@@ -170,7 +187,7 @@ import { formatHandoff, handoff } from '../lib/handoff.mjs';
|
|
|
170
187
|
|
|
171
188
|
const argv = process.argv.slice(2);
|
|
172
189
|
const flags = new Map();
|
|
173
|
-
const BOOL_FLAGS = ['revoke', 'json', 'yes', 'detach', 'no-install', 'plan', 'stop', 'share', 'apply', 'diff', 'template', 'ci', 'fresh', 'install', 'remove', 'replace', 'artifact', 'open', 'reopen', 'kids', 'remote-ai', 'dry-run', 'verify'];
|
|
190
|
+
const BOOL_FLAGS = ['revoke', 'json', 'yes', 'detach', 'no-install', 'plan', 'stop', 'share', 'apply', 'diff', 'template', 'ci', 'fresh', 'install', 'remove', 'replace', 'artifact', 'open', 'reopen', 'kids', 'remote-ai', 'dry-run', 'verify', 'maps', 'profile'];
|
|
174
191
|
const positional = [];
|
|
175
192
|
for (let i = 0; i < argv.length; i++) {
|
|
176
193
|
const a = argv[i];
|
|
@@ -451,6 +468,26 @@ function print(result) {
|
|
|
451
468
|
case 'chrome install':
|
|
452
469
|
lines.push(result.already || result.command === 'chrome' ? `Chrome: ${result.chrome}` : `Installed Chrome for Testing ${result.buildId}: ${result.chrome}`);
|
|
453
470
|
break;
|
|
471
|
+
case 'perf':
|
|
472
|
+
lines.push(`Measured ${result.game}: ${result.runs.length} run(s), in ${result.out}`, ...result.headline.map((l) => ` ${l}`),
|
|
473
|
+
...(result.blocked ? [` ${result.blocked}`] : []), ...(result.loaded ? [` ${result.loaded}`] : []),
|
|
474
|
+
`Each run: ${result.out}/<device>-<n>.json (and its screenshots); medians: ${result.summary}${result.sizes ? `; sizes: ${result.sizes}` : ''}`,
|
|
475
|
+
...(result.profiles?.length ? [`CPU profiles (Chrome DevTools opens them; the hottest functions are in each run's JSON): ${result.profiles.join(', ')}`] : []));
|
|
476
|
+
break;
|
|
477
|
+
case 'perf sizes': {
|
|
478
|
+
const kb = (b) => `${(b / 1024).toFixed(1)} KB`;
|
|
479
|
+
lines.push(`${result.game}: ${result.total.files} files, ${kb(result.total.bytes)} (${kb(result.total.gzip)} gzipped); JavaScript ${kb(result.js.bytes)} (${kb(result.js.gzip)} gzipped)`,
|
|
480
|
+
'Biggest:', ...result.biggest.slice(0, 8).map((f) => ` ${kb(f.bytes).padStart(10)} ${f.path}${f.gzip !== f.bytes ? ` (${kb(f.gzip)} gzipped)` : ''}`),
|
|
481
|
+
...(result.modules ? ['Bundle modules:', ...result.modules.top.slice(0, 8).map((m) => ` ${kb(m.bytes).padStart(10)} ${m.module}`)] : []),
|
|
482
|
+
...(result.apart ? [`Not loaded by the game: ${result.apart.files} file(s), ${kb(result.apart.bytes)} (${result.apart.note})`] : []));
|
|
483
|
+
break;
|
|
484
|
+
}
|
|
485
|
+
case 'perf compare':
|
|
486
|
+
lines.push(`${result.verdict.toUpperCase()}: ${result.why}`,
|
|
487
|
+
...result.guards.filter((g) => g.verdict !== 'same').map((g) => ` ${g.verdict} ${g.metric}: ${g.why}`),
|
|
488
|
+
...(result.left.before + result.left.after ? [` left out: ${result.left.before} run(s) before, ${result.left.after} after (blocked or started on a busy computer)`] : []),
|
|
489
|
+
`Written: ${result.file}`);
|
|
490
|
+
break;
|
|
454
491
|
case 'check':
|
|
455
492
|
lines.push(`PASS: two fresh browsers in room ${result.room} finished round ${result.round.n} (${result.round.humans} humans, ${result.round.bots} bots) in ${Math.round(result.totalMs / 1000)} s.`,
|
|
456
493
|
...result.seats.map((s) => ` ${s.browser}: seat ${s.seat} (${s.role}), seated in ${(s.seatedMs / 1000).toFixed(1)} s`),
|
|
@@ -490,6 +527,10 @@ async function main() {
|
|
|
490
527
|
if (line) process.stdout.write(`${line}\n`);
|
|
491
528
|
return { ok: true, command: 'help' };
|
|
492
529
|
}
|
|
530
|
+
if (cmd === 'perf' && sub === 'compare') {
|
|
531
|
+
if (!positional[2] || !positional[3]) return { ok: false, command: 'perf compare', why: 'usage: homie-studio perf compare <before dir> <after dir> [--goal <metric>] [--min 3]' };
|
|
532
|
+
return perfCompare(positional[2], positional[3], { goal: String(flags.get('goal') ?? DEFAULT_GOAL), min: flags.has('min') ? Number(flags.get('min')) / 100 : 0.03, guards: flags.get('guards') ? String(flags.get('guards')).split(',').map((x) => x.trim()).filter(Boolean) : null, also: flags.get('also') ? String(flags.get('also')).split(',').map((x) => x.trim()).filter(Boolean) : [] });
|
|
533
|
+
}
|
|
493
534
|
const root = requireStudio();
|
|
494
535
|
if (cmd === 'statusline') return installStatusLine(root, { remove: flags.has('remove'), replace: flags.has('replace'), project: flags.get('project') ?? null });
|
|
495
536
|
if (cmd === 'codex') return codexCommand(root, sub);
|
|
@@ -506,7 +547,16 @@ async function main() {
|
|
|
506
547
|
if (cmd === 'game' && sub === 'new') return newGame(root, positional[2], { from: flags.get('from') ?? 'gem-rush', name: flags.get('name') });
|
|
507
548
|
if (cmd === 'game' && sub === 'remix') return remixGame(root, positional[2], flags.get('id'), { name: flags.get('name') });
|
|
508
549
|
if (cmd === 'games') return { ok: true, command: 'games', games: listGames(root).map(({ dir, ...g }) => ({ ...g, dir: relative(root, dir) })) };
|
|
509
|
-
if (cmd === 'build') return tracked(root, 'build', () => build(root, { only: positional[1] ?? null, log }), 'build');
|
|
550
|
+
if (cmd === 'build') return tracked(root, 'build', () => build(root, { only: positional[1] ?? null, log, maps: flags.has('maps') }), 'build');
|
|
551
|
+
if (cmd === 'perf' && sub === 'sizes') return perfSizes(root, positional[2] ?? listGames(root)[0]?.id);
|
|
552
|
+
if (cmd === 'perf') {
|
|
553
|
+
const game = sub ?? listGames(root)[0]?.id;
|
|
554
|
+
const url = flags.get('url') ?? siteUrl(root);
|
|
555
|
+
if (!url) return { ok: false, command: 'perf', why: 'give --url (the local dev address or the live site)' };
|
|
556
|
+
const devices = String(flags.get('device') ?? 'computer,phone').split(',').map((d) => d.trim()).filter(Boolean);
|
|
557
|
+
const num = (k, d, lo, hi) => Math.max(lo, Math.min(hi, Number(flags.get(k) ?? d) || d));
|
|
558
|
+
return perfRun({ root, url, game, devices, runs: num('runs', 1, 1, 50), seconds: num('seconds', 15, 3, 300), warm: num('warm', 3, 0, 60), profile: flags.has('profile'), out: flags.get('out') ? resolve(String(flags.get('out'))) : null, maxLoad: num('max-load', 0.8, 0.05, 50), cpu: num('cpu', 4, 1, 20), pair: flags.get('pair') ? String(flags.get('pair')) : null, log });
|
|
559
|
+
}
|
|
510
560
|
if (cmd === 'status') {
|
|
511
561
|
const studio = readStudio(root);
|
|
512
562
|
return { ok: true, command: 'status', root, studio, site: siteUrl(root, studio), games: listGames(root).map((g) => g.id), cloudflareSignedIn: Boolean(wranglerBin(root) && whoami(root)) };
|
|
@@ -869,7 +919,7 @@ try {
|
|
|
869
919
|
if (result && result.command !== 'help') print(result);
|
|
870
920
|
if (result?.ok === false) process.exitCode = 1;
|
|
871
921
|
// Chrome's pipes can outlive browser.close(); a finished check must not hang its caller.
|
|
872
|
-
if (
|
|
922
|
+
if (['check', 'port check', 'look', 'perf'].includes(result?.command)) process.exit(process.exitCode ?? 0);
|
|
873
923
|
} catch (error) {
|
|
874
924
|
await flushProgress().catch(() => {});
|
|
875
925
|
print({ ok: false, why: error instanceof Error ? error.message : String(error) });
|
package/lib/build.mjs
CHANGED
|
@@ -30,6 +30,7 @@
|
|
|
30
30
|
* the game's own build (Vite, webpack…), then its output copied; base must be './'
|
|
31
31
|
*/
|
|
32
32
|
import { spawnSync } from 'node:child_process';
|
|
33
|
+
import { createHash } from 'node:crypto';
|
|
33
34
|
import { cpSync, existsSync, mkdirSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from 'node:fs';
|
|
34
35
|
import { createRequire } from 'node:module';
|
|
35
36
|
import { join, relative } from 'node:path';
|
|
@@ -205,7 +206,23 @@ export function seatsFor(g, net = netplayOf(g)) {
|
|
|
205
206
|
return { min: Math.min(max, minNamed ?? 1), max, asked };
|
|
206
207
|
}
|
|
207
208
|
|
|
208
|
-
|
|
209
|
+
/**
|
|
210
|
+
* `build --maps` (what `homie-studio perf` reads a CPU profile through): the bundle's source map and esbuild's metafile
|
|
211
|
+
* go to .studio/maps/<id>/ (git-ignored, this computer's own), never into site/dist, so a deploy never ships them. The
|
|
212
|
+
* bundle itself is byte for byte the one a plain build makes (an external map adds no comment to it); main.js.sha256
|
|
213
|
+
* says which bundle the map belongs to, so a map left over from an older build is never used.
|
|
214
|
+
*/
|
|
215
|
+
function keepMap(root, id, out, metafile) {
|
|
216
|
+
const dir = join(root, '.studio', 'maps', id);
|
|
217
|
+
rmSync(dir, { recursive: true, force: true });
|
|
218
|
+
mkdirSync(dir, { recursive: true });
|
|
219
|
+
const map = join(out, 'assets', 'main.js.map');
|
|
220
|
+
if (existsSync(map)) { cpSync(map, join(dir, 'main.js.map')); rmSync(map, { force: true }); }
|
|
221
|
+
writeFileSync(join(dir, 'meta.json'), JSON.stringify(metafile ?? {}));
|
|
222
|
+
writeFileSync(join(dir, 'main.js.sha256'), `${createHash('sha256').update(readFileSync(join(out, 'assets', 'main.js'))).digest('hex')}\n`);
|
|
223
|
+
}
|
|
224
|
+
|
|
225
|
+
export async function build(root, { only = null, log = () => {}, deploy = process.env.WORKERS_CI === '1', maps = false } = {}) {
|
|
209
226
|
const require = createRequire(join(root, 'package.json'));
|
|
210
227
|
let esbuild;
|
|
211
228
|
try { esbuild = require('esbuild'); } catch { esbuild = await import('esbuild'); }
|
|
@@ -228,7 +245,7 @@ export async function build(root, { only = null, log = () => {}, deploy = proces
|
|
|
228
245
|
const entry = join(g.dir, entryRel);
|
|
229
246
|
if (!existsSync(entry)) throw new Error(`games/${g.id}: entry ${entryRel} not found`);
|
|
230
247
|
const result = await esbuild.build({
|
|
231
|
-
entryPoints: [entry], bundle: true, format: 'esm', target: 'es2022', minify: true, sourcemap: false,
|
|
248
|
+
entryPoints: [entry], bundle: true, format: 'esm', target: 'es2022', minify: true, sourcemap: maps ? 'external' : false,
|
|
232
249
|
outfile: join(out, 'assets', 'main.js'), absWorkingDir: root, logLevel: 'silent', metafile: true,
|
|
233
250
|
loader: LOADERS, assetNames: '[name]-[hash]',
|
|
234
251
|
}).catch((error) => {
|
|
@@ -236,6 +253,7 @@ export async function build(root, { only = null, log = () => {}, deploy = proces
|
|
|
236
253
|
throw new Error(`games/${g.id} did not build: ${first ? `${first.text}${first.location ? ` (${first.location.file}:${first.location.line})` : ''}` : error.message}`);
|
|
237
254
|
});
|
|
238
255
|
warnings += result.warnings.length;
|
|
256
|
+
if (maps) keepMap(root, g.id, out, result.metafile);
|
|
239
257
|
};
|
|
240
258
|
if (mode === 'static') {
|
|
241
259
|
copyStatic(g.dir, out);
|
|
@@ -0,0 +1,171 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* WHERE A GAME'S TIME GOES: a Chrome CPU profile (`homie-studio perf --profile`, the .cpuprofile DevTools opens)
|
|
3
|
+
* summed per function, and the game's minified bundle read back through its source map, so the summary names
|
|
4
|
+
* `draw (src/main.ts:620)` rather than `Xe (main.js:1:48213)`.
|
|
5
|
+
*
|
|
6
|
+
* Self time is the time a function itself was on the stack's top (what to make cheaper); total time includes what it
|
|
7
|
+
* called (where a frame's time is spent overall). Idle, the garbage collector and Chrome's own work ("(program)") are
|
|
8
|
+
* kept apart, so a game that is mostly idle reads as idle, not as a list of tiny hot spots.
|
|
9
|
+
*/
|
|
10
|
+
|
|
11
|
+
const B64 = new Map([...'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'].map((c, i) => [c, i]));
|
|
12
|
+
|
|
13
|
+
/** A source map's `mappings` as one array of segments per generated line: [column, source, line, column, name?]. */
|
|
14
|
+
export function decodeMappings(mappings) {
|
|
15
|
+
const lines = [];
|
|
16
|
+
let src = 0;
|
|
17
|
+
let line = 0;
|
|
18
|
+
let col = 0;
|
|
19
|
+
let name = 0;
|
|
20
|
+
for (const text of String(mappings ?? '').split(';')) {
|
|
21
|
+
const segs = [];
|
|
22
|
+
let gen = 0;
|
|
23
|
+
for (const seg of text.split(',')) {
|
|
24
|
+
if (!seg) continue;
|
|
25
|
+
const v = [];
|
|
26
|
+
let value = 0;
|
|
27
|
+
let shift = 0;
|
|
28
|
+
for (const ch of seg) {
|
|
29
|
+
const d = B64.get(ch);
|
|
30
|
+
if (d === undefined) break;
|
|
31
|
+
value += (d & 31) << shift;
|
|
32
|
+
if (d & 32) { shift += 5; continue; }
|
|
33
|
+
v.push(value & 1 ? -(value >>> 1) : value >>> 1);
|
|
34
|
+
value = 0;
|
|
35
|
+
shift = 0;
|
|
36
|
+
}
|
|
37
|
+
gen += v[0] ?? 0;
|
|
38
|
+
if (v.length >= 4) {
|
|
39
|
+
src += v[1]; line += v[2]; col += v[3];
|
|
40
|
+
if (v.length >= 5) { name += v[4]; segs.push([gen, src, line, col, name]); } else segs.push([gen, src, line, col]);
|
|
41
|
+
}
|
|
42
|
+
}
|
|
43
|
+
lines.push(segs);
|
|
44
|
+
}
|
|
45
|
+
return lines;
|
|
46
|
+
}
|
|
47
|
+
|
|
48
|
+
/**
|
|
49
|
+
* A lookup for one source map: (0-based line, column) in the generated file → { source, line (1-based), column, name },
|
|
50
|
+
* the nearest mapping at or before that column on that line; null outside the map.
|
|
51
|
+
*/
|
|
52
|
+
export function sourceMapLookup(map) {
|
|
53
|
+
const lines = decodeMappings(map?.mappings);
|
|
54
|
+
const sources = (map?.sources ?? []).map((s) => String(s).replace(/^(\.\.\/)+/, '').replace(/^webpack:\/\/\//, ''));
|
|
55
|
+
const names = map?.names ?? [];
|
|
56
|
+
const content = map?.sourcesContent ?? [];
|
|
57
|
+
return (line, column) => {
|
|
58
|
+
const segs = lines[line];
|
|
59
|
+
if (!segs?.length) return null;
|
|
60
|
+
let lo = 0;
|
|
61
|
+
let hi = segs.length - 1;
|
|
62
|
+
let best = -1;
|
|
63
|
+
while (lo <= hi) { const mid = (lo + hi) >> 1; if (segs[mid][0] <= column) { best = mid; lo = mid + 1; } else hi = mid - 1; }
|
|
64
|
+
if (best < 0) return null;
|
|
65
|
+
// A function's own position often lands on its name token; prefer a named segment within a few columns.
|
|
66
|
+
let seg = segs[best];
|
|
67
|
+
for (let k = best; k >= 0 && column - segs[k][0] <= 12; k--) if (segs[k][4] !== undefined) { seg = segs[k]; break; }
|
|
68
|
+
const source = sources[seg[1]] ?? null;
|
|
69
|
+
const named = seg[4] !== undefined ? names[seg[4]] ?? null : null;
|
|
70
|
+
const text = typeof content[seg[1]] === 'string' ? content[seg[1]].split('\n')[seg[2]] ?? '' : '';
|
|
71
|
+
// What the line defines wins over a segment's name: at an arrow function V8 points at its first parameter, whose
|
|
72
|
+
// segment names the parameter, not the function.
|
|
73
|
+
return { source, line: seg[2] + 1, column: seg[3], name: nameFromLine(text, seg[3]) ?? named };
|
|
74
|
+
};
|
|
75
|
+
}
|
|
76
|
+
|
|
77
|
+
/**
|
|
78
|
+
* The function a line of source defines, when the line says: `function draw(`, `const step = (`, `draw(…) {` (a method),
|
|
79
|
+
* `step: (dt) =>`. The whole line first (a line usually defines one function), then from the mapped column on.
|
|
80
|
+
*/
|
|
81
|
+
function nameFromLine(text, column) {
|
|
82
|
+
const tries = [
|
|
83
|
+
/function\s*\*?\s*([A-Za-z_$][\w$]*)\s*[(<]/,
|
|
84
|
+
/(?:const|let|var)\s+([A-Za-z_$][\w$]*)\s*(?::[^=]+)?=\s*(?:async\s*)?(?:function\b|\([^)]*\)\s*(?::[^=]+)?=>|[A-Za-z_$][\w$]*\s*=>)/,
|
|
85
|
+
/^\s*(?:(?:public|private|protected|static|async|get|set)\s+)*([A-Za-z_$][\w$]*)\s*\([^)]*\)\s*(?::[^{]+)?\{/,
|
|
86
|
+
/([A-Za-z_$][\w$]*)\s*:\s*(?:async\s*)?(?:function\b|\([^)]*\)\s*(?::[^=]+)?=>)/,
|
|
87
|
+
];
|
|
88
|
+
for (const re of tries) { const m = re.exec(text); if (m && !['if', 'for', 'while', 'switch', 'catch', 'return'].includes(m[1])) return m[1]; }
|
|
89
|
+
void column;
|
|
90
|
+
return null;
|
|
91
|
+
}
|
|
92
|
+
|
|
93
|
+
const SPECIAL = new Set(['(root)', '(program)', '(idle)', '(garbage collector)']);
|
|
94
|
+
|
|
95
|
+
/**
|
|
96
|
+
* A CPU profile summed per function. `maps`: [{ match: (url) => boolean, lookup }] (sourceMapLookup), for scripts
|
|
97
|
+
* shipped minified. `game`: (url) => boolean, the game's own scripts (the rest is the play page and the browser).
|
|
98
|
+
* Returns milliseconds and shares of the busy (not idle) time, the hottest functions first.
|
|
99
|
+
*/
|
|
100
|
+
export function summarizeProfile(profile, { maps = [], game = () => true, top = 15 } = {}) {
|
|
101
|
+
const nodes = new Map((profile?.nodes ?? []).map((n) => [n.id, n]));
|
|
102
|
+
const parent = new Map();
|
|
103
|
+
for (const n of nodes.values()) for (const c of n.children ?? []) parent.set(c, n.id);
|
|
104
|
+
const samples = profile?.samples ?? [];
|
|
105
|
+
const deltas = profile?.timeDeltas ?? [];
|
|
106
|
+
// Each sample stands for the time until the next one (the last: the average step).
|
|
107
|
+
const span = (i) => (i + 1 < deltas.length ? deltas[i + 1] : (deltas.length ? deltas.reduce((s, x) => s + x, 0) / deltas.length : 0));
|
|
108
|
+
const keyOf = (cf) => `${cf.functionName || '(anonymous)'}|${cf.url}|${cf.lineNumber}|${cf.columnNumber}`;
|
|
109
|
+
const fns = new Map();
|
|
110
|
+
const fnOf = (n) => {
|
|
111
|
+
const cf = n.callFrame;
|
|
112
|
+
const k = keyOf(cf);
|
|
113
|
+
let f = fns.get(k);
|
|
114
|
+
if (!f) { f = { key: k, cf, self: 0, total: 0 }; fns.set(k, f); }
|
|
115
|
+
return f;
|
|
116
|
+
};
|
|
117
|
+
let all = 0;
|
|
118
|
+
let idle = 0;
|
|
119
|
+
let gc = 0;
|
|
120
|
+
let program = 0;
|
|
121
|
+
const scripts = new Map();
|
|
122
|
+
for (let i = 0; i < samples.length; i++) {
|
|
123
|
+
const n = nodes.get(samples[i]);
|
|
124
|
+
if (!n) continue;
|
|
125
|
+
const us = span(i);
|
|
126
|
+
all += us;
|
|
127
|
+
const name = n.callFrame.functionName;
|
|
128
|
+
if (name === '(idle)') { idle += us; continue; }
|
|
129
|
+
if (name === '(garbage collector)') gc += us;
|
|
130
|
+
if (name === '(program)') program += us;
|
|
131
|
+
fnOf(n).self += us;
|
|
132
|
+
const script = n.callFrame.url ? scriptName(n.callFrame.url) : SPECIAL.has(name) ? name : '(native)';
|
|
133
|
+
scripts.set(script, (scripts.get(script) ?? 0) + us);
|
|
134
|
+
// Total: once per function per sample, however deep the recursion.
|
|
135
|
+
const seen = new Set();
|
|
136
|
+
for (let id = n.id; id !== undefined; id = parent.get(id)) {
|
|
137
|
+
const p = nodes.get(id);
|
|
138
|
+
if (!p || p.callFrame.functionName === '(root)') break;
|
|
139
|
+
const f = fnOf(p);
|
|
140
|
+
if (seen.has(f)) continue;
|
|
141
|
+
seen.add(f);
|
|
142
|
+
f.total += us;
|
|
143
|
+
}
|
|
144
|
+
}
|
|
145
|
+
const busy = all - idle;
|
|
146
|
+
const ms = (us) => Math.round(us / 100) / 10;
|
|
147
|
+
const pct = (us) => (busy ? Math.round((us / busy) * 1000) / 10 : 0);
|
|
148
|
+
const place = (cf) => {
|
|
149
|
+
if (!cf.url) return null;
|
|
150
|
+
const map = maps.find((m) => m.match(cf.url));
|
|
151
|
+
const hit = map ? map.lookup(cf.lineNumber, cf.columnNumber) : null;
|
|
152
|
+
if (hit?.source) return { name: hit.name ?? null, where: `${hit.source}:${hit.line}`, mapped: true };
|
|
153
|
+
return { name: null, where: `${scriptName(cf.url)}:${cf.lineNumber + 1}:${cf.columnNumber + 1}`, mapped: false };
|
|
154
|
+
};
|
|
155
|
+
const rows = [...fns.values()].filter((f) => !SPECIAL.has(f.cf.functionName)).sort((a, b) => b.self - a.self).slice(0, top).map((f) => {
|
|
156
|
+
const at = place(f.cf);
|
|
157
|
+
const minified = f.cf.functionName || '(anonymous)';
|
|
158
|
+
// V8 names a function from where it was declared or assigned; one it calls anonymous (a callback) stays so, and
|
|
159
|
+
// `where` says which line it is on.
|
|
160
|
+
return { name: f.cf.functionName ? at?.name ?? minified : '(anonymous)', ...(at?.mapped && at.name && at.name !== minified ? { minified } : {}), where: at?.where ?? '(native)', game: Boolean(f.cf.url && game(f.cf.url)), selfMs: ms(f.self), selfPct: pct(f.self), totalPct: pct(f.total) };
|
|
161
|
+
});
|
|
162
|
+
const byScript = [...scripts.entries()].sort((a, b) => b[1] - a[1]).slice(0, 8).map(([script, us]) => ({ script, ms: ms(us), pct: pct(us) }));
|
|
163
|
+
let gameUs = 0;
|
|
164
|
+
for (const f of fns.values()) if (f.cf.url && game(f.cf.url)) gameUs += f.self;
|
|
165
|
+
return { ms: ms(all), busyMs: ms(busy), idlePct: all ? Math.round((idle / all) * 1000) / 10 : 0, gcPct: pct(gc), programPct: pct(program), gamePct: pct(gameUs), top: rows, scripts: byScript };
|
|
166
|
+
}
|
|
167
|
+
|
|
168
|
+
/** A script's address without the site: `/gem-rush/__game/assets/main.js`, `/_homie/site.js`. */
|
|
169
|
+
export function scriptName(url) {
|
|
170
|
+
try { const u = new URL(url); return u.pathname || url; } catch { return String(url).slice(0, 120); }
|
|
171
|
+
}
|
|
@@ -0,0 +1,249 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* WHETHER A CHANGE MADE A GAME FASTER, OR ONLY LOOKS LIKE IT (`homie-studio perf compare`).
|
|
3
|
+
*
|
|
4
|
+
* Two builds are measured in turns (before, after, after, before, ...), several runs each, and every run gives one
|
|
5
|
+
* number per metric (a run's 95th-percentile frame time, say). A browser's numbers move from run to run on their own:
|
|
6
|
+
* another app wakes up, a garbage collection lands in the window, the bots wander somewhere busier. So a difference
|
|
7
|
+
* between two medians means nothing by itself. A change counts as better only when all three hold:
|
|
8
|
+
*
|
|
9
|
+
* - it is unlikely to be chance: a permutation test on the runs' ranks (Mann-Whitney U, exact for the small run
|
|
10
|
+
* counts used here), one-sided, p < 0.05;
|
|
11
|
+
* - the 95% bootstrap interval of the change in medians stays on the better side of zero;
|
|
12
|
+
* - it is big enough to matter: at least `min` (3%) better at the median.
|
|
13
|
+
*
|
|
14
|
+
* A guard (another metric that must not get worse) is judged the same way, stricter (p < 0.01, at least 5% worse),
|
|
15
|
+
* because a handful of guards checked at 0.05 would flag a regression by chance on most changes. A metric that never
|
|
16
|
+
* varies (bytes of a build) has no noise to test: any change of at least 1% counts.
|
|
17
|
+
*
|
|
18
|
+
* PAIRS. A shared computer drifts: measured on an Apple M4 while other work ran, the main thread per frame rose from
|
|
19
|
+
* 0.88 to 1.25 ms over five runs of the SAME build as the load average went from 4 to 7.6, host and replica together.
|
|
20
|
+
* When the runs were taken side by side (`try` measures before, after, after, before, … and labels each neighbouring
|
|
21
|
+
* pair), the test is on the pairs instead: each after-run against the before-run next to it, as a ratio, so a drift
|
|
22
|
+
* that hits both cancels. Wilcoxon signed-rank, exact: with 6 pairs, p < 0.05 needs the after-run to win at least 5
|
|
23
|
+
* of the 6 pairs (and lose only the closest one). The change is the median ratio; its interval a bootstrap over pairs.
|
|
24
|
+
*
|
|
25
|
+
* Everything here is lower-is-better (milliseconds, bytes, messages); nothing reads a clock or the network.
|
|
26
|
+
*/
|
|
27
|
+
|
|
28
|
+
/** The q-quantile (0..1) of numbers, by linear interpolation between order statistics. null for none. */
|
|
29
|
+
export function quantile(values, q) {
|
|
30
|
+
const xs = values.filter(Number.isFinite).sort((a, b) => a - b);
|
|
31
|
+
if (!xs.length) return null;
|
|
32
|
+
const at = (xs.length - 1) * q;
|
|
33
|
+
const lo = Math.floor(at);
|
|
34
|
+
const hi = Math.ceil(at);
|
|
35
|
+
return xs[lo] + (xs[hi] - xs[lo]) * (at - lo);
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
export const median = (values) => quantile(values, 0.5);
|
|
39
|
+
|
|
40
|
+
/** A small summary of numbers: n, median, the middle half, min, max, and the spread as a share of the median. */
|
|
41
|
+
export function summarize(values) {
|
|
42
|
+
const xs = values.filter(Number.isFinite);
|
|
43
|
+
if (!xs.length) return { n: 0, median: null, q1: null, q3: null, min: null, max: null, spread: null };
|
|
44
|
+
const m = median(xs);
|
|
45
|
+
const q1 = quantile(xs, 0.25);
|
|
46
|
+
const q3 = quantile(xs, 0.75);
|
|
47
|
+
return { n: xs.length, median: round(m), q1: round(q1), q3: round(q3), min: round(Math.min(...xs)), max: round(Math.max(...xs)), spread: m ? round((q3 - q1) / Math.abs(m), 4) : null };
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
export const round = (x, digits = 3) => (Number.isFinite(x) ? Number(x.toFixed(digits)) : null);
|
|
51
|
+
|
|
52
|
+
/** Midranks of a joined list (ties share the average of their ranks), 1-based. */
|
|
53
|
+
function ranks(values) {
|
|
54
|
+
const order = values.map((v, i) => [v, i]).sort((a, b) => a[0] - b[0]);
|
|
55
|
+
const r = new Array(values.length);
|
|
56
|
+
for (let i = 0; i < order.length;) {
|
|
57
|
+
let j = i;
|
|
58
|
+
while (j + 1 < order.length && order[j + 1][0] === order[i][0]) j++;
|
|
59
|
+
const mid = (i + j) / 2 + 1;
|
|
60
|
+
for (let k = i; k <= j; k++) r[order[k][1]] = mid;
|
|
61
|
+
i = j + 1;
|
|
62
|
+
}
|
|
63
|
+
return r;
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
/**
|
|
67
|
+
* Mann-Whitney U as an exact permutation test on rank sums: every way of choosing which runs were "after" is equally
|
|
68
|
+
* likely when the change did nothing, so p is the share of those choices whose "after" ranks are as low (`less`) or as
|
|
69
|
+
* high (`greater`) as the ones measured. Exact up to 200,000 choices (10 + 10 runs is 184,756), the normal
|
|
70
|
+
* approximation with a tie correction past that.
|
|
71
|
+
*/
|
|
72
|
+
export function rankTest(before, after) {
|
|
73
|
+
const a = before.filter(Number.isFinite);
|
|
74
|
+
const b = after.filter(Number.isFinite);
|
|
75
|
+
const n1 = a.length;
|
|
76
|
+
const n2 = b.length;
|
|
77
|
+
if (!n1 || !n2) return { n1, n2, u: null, less: null, greater: null, twoSided: null, exact: false };
|
|
78
|
+
const r = ranks([...a, ...b]);
|
|
79
|
+
const observed = r.slice(n1).reduce((s, x) => s + x, 0);
|
|
80
|
+
const u = observed - (n2 * (n2 + 1)) / 2;
|
|
81
|
+
const total = binom(n1 + n2, n2);
|
|
82
|
+
const eps = 1e-9;
|
|
83
|
+
if (total <= 200_000) {
|
|
84
|
+
let le = 0;
|
|
85
|
+
let ge = 0;
|
|
86
|
+
let all = 0;
|
|
87
|
+
const pick = (start, left, sum) => {
|
|
88
|
+
if (left === 0) { all++; if (sum <= observed + eps) le++; if (sum >= observed - eps) ge++; return; }
|
|
89
|
+
for (let i = start; i <= r.length - left; i++) pick(i + 1, left - 1, sum + r[i]);
|
|
90
|
+
};
|
|
91
|
+
pick(0, n2, 0);
|
|
92
|
+
const less = le / all;
|
|
93
|
+
const greater = ge / all;
|
|
94
|
+
return { n1, n2, u, less: round(less, 5), greater: round(greater, 5), twoSided: round(Math.min(1, 2 * Math.min(less, greater)), 5), exact: true };
|
|
95
|
+
}
|
|
96
|
+
const n = n1 + n2;
|
|
97
|
+
const mean = (n2 * (n + 1)) / 2;
|
|
98
|
+
const counts = new Map();
|
|
99
|
+
for (const x of r) counts.set(x, (counts.get(x) ?? 0) + 1);
|
|
100
|
+
const ties = [...counts.values()].reduce((s, t) => s + (t ** 3 - t), 0);
|
|
101
|
+
const sd = Math.sqrt(((n1 * n2) / 12) * ((n + 1) - ties / (n * (n - 1))));
|
|
102
|
+
const z = sd ? (observed - mean) / sd : 0;
|
|
103
|
+
const less = normalCdf(z + 0.5 / (sd || 1));
|
|
104
|
+
const greater = 1 - normalCdf(z - 0.5 / (sd || 1));
|
|
105
|
+
return { n1, n2, u, less: round(less, 5), greater: round(greater, 5), twoSided: round(Math.min(1, 2 * Math.min(less, greater)), 5), exact: false };
|
|
106
|
+
}
|
|
107
|
+
|
|
108
|
+
function binom(n, k) {
|
|
109
|
+
let c = 1;
|
|
110
|
+
for (let i = 1; i <= k; i++) c = (c * (n - k + i)) / i;
|
|
111
|
+
return Math.round(c);
|
|
112
|
+
}
|
|
113
|
+
|
|
114
|
+
function normalCdf(z) {
|
|
115
|
+
// Abramowitz and Stegun 7.1.26, |error| < 1.5e-7.
|
|
116
|
+
const t = 1 / (1 + 0.3275911 * Math.abs(z) / Math.SQRT2);
|
|
117
|
+
const y = 1 - (((((1.061405429 * t - 1.453152027) * t) + 1.421413741) * t - 0.284496736) * t + 0.254829592) * t * Math.exp(-(z * z) / 2);
|
|
118
|
+
return z >= 0 ? (1 + y) / 2 : (1 - y) / 2;
|
|
119
|
+
}
|
|
120
|
+
|
|
121
|
+
/** A seeded random number generator (mulberry32): the same runs always give the same interval. */
|
|
122
|
+
export function seeded(seed = 1) {
|
|
123
|
+
let s = seed >>> 0;
|
|
124
|
+
return () => {
|
|
125
|
+
s = (s + 0x6d2b79f5) >>> 0;
|
|
126
|
+
let t = s;
|
|
127
|
+
t = Math.imul(t ^ (t >>> 15), t | 1);
|
|
128
|
+
t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
|
|
129
|
+
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
|
|
130
|
+
};
|
|
131
|
+
}
|
|
132
|
+
|
|
133
|
+
/**
|
|
134
|
+
* The 95% percentile-bootstrap interval of the relative change in medians, (after - before) / before: resample each
|
|
135
|
+
* side's runs with replacement 4,000 times. Seeded, so the same numbers give the same interval.
|
|
136
|
+
*/
|
|
137
|
+
export function bootstrapChange(before, after, { iterations = 4000, seed = 7 } = {}) {
|
|
138
|
+
const a = before.filter(Number.isFinite);
|
|
139
|
+
const b = after.filter(Number.isFinite);
|
|
140
|
+
if (!a.length || !b.length) return null;
|
|
141
|
+
const rnd = seeded(seed);
|
|
142
|
+
const draw = (xs) => { const out = new Array(xs.length); for (let i = 0; i < xs.length; i++) out[i] = xs[Math.floor(rnd() * xs.length)]; return out; };
|
|
143
|
+
const changes = [];
|
|
144
|
+
for (let i = 0; i < iterations; i++) {
|
|
145
|
+
const ma = median(draw(a));
|
|
146
|
+
const mb = median(draw(b));
|
|
147
|
+
if (ma) changes.push((mb - ma) / Math.abs(ma));
|
|
148
|
+
}
|
|
149
|
+
if (!changes.length) return null;
|
|
150
|
+
return [round(quantile(changes, 0.025), 4), round(quantile(changes, 0.975), 4)];
|
|
151
|
+
}
|
|
152
|
+
|
|
153
|
+
/**
|
|
154
|
+
* One metric, before against after (lower is better). Returns the medians, the change, p, the interval and a verdict:
|
|
155
|
+
* `better`, `worse` or `same` (within the noise), with `why` in words. `min` is the smallest change that counts as
|
|
156
|
+
* better (0.03 = 3%); `worseMin` and `worseAlpha` the stricter bar for calling it worse.
|
|
157
|
+
*/
|
|
158
|
+
export function judge(before, after, { min = 0.03, alpha = 0.05, worseMin = 0.05, worseAlpha = 0.01, unit = '' } = {}) {
|
|
159
|
+
const a = before.filter(Number.isFinite);
|
|
160
|
+
const b = after.filter(Number.isFinite);
|
|
161
|
+
const sa = summarize(a);
|
|
162
|
+
const sb = summarize(b);
|
|
163
|
+
const base = { before: sa, after: sb, change: null, p: null, pWorse: null, ci: null, verdict: 'unknown', why: null };
|
|
164
|
+
if (!a.length || !b.length) return { ...base, why: `not measured on both sides (${a.length} before, ${b.length} after)` };
|
|
165
|
+
const change = sa.median ? (sb.median - sa.median) / Math.abs(sa.median) : (sb.median === sa.median ? 0 : null);
|
|
166
|
+
const fixed = sa.min === sa.max && sb.min === sb.max;
|
|
167
|
+
const pct = (x) => `${x > 0 ? '+' : ''}${(x * 100).toFixed(1)}%`;
|
|
168
|
+
const say = (x) => `${round(x, 2)}${unit}`;
|
|
169
|
+
if (change === null) return { ...base, change: null, why: 'the before median is 0, so a relative change has no meaning' };
|
|
170
|
+
if (fixed) {
|
|
171
|
+
// Nothing to test: the same number every run (a build's bytes). Any real difference is the change itself.
|
|
172
|
+
const verdict = change <= -0.01 ? 'better' : change >= 0.01 ? 'worse' : 'same';
|
|
173
|
+
return { ...base, change: round(change, 4), verdict, exact: true, why: `${say(sa.median)} → ${say(sb.median)} (${pct(change)}, the same every run)` };
|
|
174
|
+
}
|
|
175
|
+
const t = rankTest(a, b);
|
|
176
|
+
const ci = bootstrapChange(a, b);
|
|
177
|
+
const res = { ...base, change: round(change, 4), p: t.less, pWorse: t.greater, ci, n: [t.n1, t.n2] };
|
|
178
|
+
const head = `${say(sa.median)} → ${say(sb.median)} (${pct(change)}; 95% interval ${ci ? `${pct(ci[0])} to ${pct(ci[1])}` : 'n/a'}; p = ${t.less} better, ${t.greater} worse; ${t.n1} + ${t.n2} runs)`;
|
|
179
|
+
if (t.less < alpha && ci && ci[1] < 0 && change <= -min) return { ...res, verdict: 'better', why: head };
|
|
180
|
+
if (t.greater < worseAlpha && ci && ci[0] > 0 && change >= worseMin) return { ...res, verdict: 'worse', why: head };
|
|
181
|
+
const reasons = [];
|
|
182
|
+
if (change > -min) reasons.push(`smaller than the ${Math.round(min * 100)}% that counts`);
|
|
183
|
+
if (t.less >= alpha) reasons.push(`p ${t.less} is not under ${alpha}`);
|
|
184
|
+
if (ci && ci[1] >= 0) reasons.push('the interval reaches zero');
|
|
185
|
+
return { ...res, verdict: 'same', why: `${head}: within the noise (${reasons.join('; ') || 'not clearly either way'})` };
|
|
186
|
+
}
|
|
187
|
+
|
|
188
|
+
/**
|
|
189
|
+
* Wilcoxon signed-rank test on paired differences (after - before, or their log ratios), exact for up to 20 pairs
|
|
190
|
+
* (every sign assignment), the normal approximation past that. Zeros are dropped. `less`: p that the after side is
|
|
191
|
+
* lower by chance; `greater`: higher.
|
|
192
|
+
*/
|
|
193
|
+
export function signedRankTest(diffs) {
|
|
194
|
+
const d = diffs.filter((x) => Number.isFinite(x) && x !== 0);
|
|
195
|
+
const n = d.length;
|
|
196
|
+
if (!n) return { n: 0, wins: 0, less: 1, greater: 1, exact: true };
|
|
197
|
+
const r = ranks(d.map(Math.abs));
|
|
198
|
+
const wPlus = d.reduce((s, x, i) => s + (x > 0 ? r[i] : 0), 0);
|
|
199
|
+
const wins = d.filter((x) => x < 0).length;
|
|
200
|
+
const eps = 1e-9;
|
|
201
|
+
if (n <= 20) {
|
|
202
|
+
let le = 0;
|
|
203
|
+
let ge = 0;
|
|
204
|
+
const total = 2 ** n;
|
|
205
|
+
for (let mask = 0; mask < total; mask++) {
|
|
206
|
+
let w = 0;
|
|
207
|
+
for (let i = 0; i < n; i++) if (mask & (1 << i)) w += r[i];
|
|
208
|
+
if (w <= wPlus + eps) le++;
|
|
209
|
+
if (w >= wPlus - eps) ge++;
|
|
210
|
+
}
|
|
211
|
+
return { n, wins, less: round(le / total, 5), greater: round(ge / total, 5), exact: true };
|
|
212
|
+
}
|
|
213
|
+
const mean = (n * (n + 1)) / 4;
|
|
214
|
+
const sd = Math.sqrt((n * (n + 1) * (2 * n + 1)) / 24);
|
|
215
|
+
return { n, wins, less: round(normalCdf((wPlus + 0.5 - mean) / sd), 5), greater: round(1 - normalCdf((wPlus - 0.5 - mean) / sd), 5), exact: false };
|
|
216
|
+
}
|
|
217
|
+
|
|
218
|
+
/**
|
|
219
|
+
* One metric from runs taken in pairs (before[k] beside after[k]); lower is better. The same verdicts and bars as
|
|
220
|
+
* judge(), on the pairs' ratios.
|
|
221
|
+
*/
|
|
222
|
+
export function judgePaired(before, after, { min = 0.03, alpha = 0.05, worseMin = 0.05, worseAlpha = 0.01, unit = '' } = {}) {
|
|
223
|
+
const pairs = before.map((a, i) => [a, after[i]]).filter(([a, b]) => Number.isFinite(a) && Number.isFinite(b) && a > 0 && b > 0);
|
|
224
|
+
const sa = summarize(pairs.map(([a]) => a));
|
|
225
|
+
const sb = summarize(pairs.map(([, b]) => b));
|
|
226
|
+
const base = { before: sa, after: sb, change: null, p: null, pWorse: null, ci: null, verdict: 'unknown', why: null, paired: true };
|
|
227
|
+
if (pairs.length < 3) return { ...base, why: `too few pairs measured on both sides (${pairs.length})` };
|
|
228
|
+
const logs = pairs.map(([a, b]) => Math.log(b / a));
|
|
229
|
+
const pct = (x) => `${x > 0 ? '+' : ''}${(x * 100).toFixed(1)}%`;
|
|
230
|
+
const say = (x) => `${round(x, 2)}${unit}`;
|
|
231
|
+
if (logs.every((x) => x === 0)) return { ...base, change: 0, p: 1, pWorse: 1, verdict: 'same', why: `${say(sa.median)} → ${say(sb.median)}: identical in every pair` };
|
|
232
|
+
const change = Math.exp(median(logs)) - 1;
|
|
233
|
+
const t = signedRankTest(logs);
|
|
234
|
+
const rnd = seeded(11);
|
|
235
|
+
const boots = [];
|
|
236
|
+
for (let i = 0; i < 4000; i++) { const draw = logs.map(() => logs[Math.floor(rnd() * logs.length)]); boots.push(Math.exp(median(draw)) - 1); }
|
|
237
|
+
const ci = [round(quantile(boots, 0.025), 4), round(quantile(boots, 0.975), 4)];
|
|
238
|
+
const res = { ...base, change: round(change, 4), p: t.less, pWorse: t.greater, ci, n: [pairs.length, pairs.length], wins: t.wins };
|
|
239
|
+
const head = `${say(sa.median)} → ${say(sb.median)} (${pct(change)} in the median pair; after was lower in ${t.wins} of ${pairs.length} pairs run side by side; 95% interval ${pct(ci[0])} to ${pct(ci[1])}; p = ${t.less} better, ${t.greater} worse)`;
|
|
240
|
+
if (t.less < alpha && ci[1] < 0 && change <= -min) return { ...res, verdict: 'better', why: head };
|
|
241
|
+
// Few pairs cannot reach p < 0.01 (6 pairs: 1/64 at best), so the guard's bar is then "worse in every pair".
|
|
242
|
+
const wa = Math.max(worseAlpha, 1.5 / 2 ** pairs.length);
|
|
243
|
+
if (t.greater < wa && ci[0] > 0 && change >= worseMin) return { ...res, verdict: 'worse', why: head };
|
|
244
|
+
const reasons = [];
|
|
245
|
+
if (change > -min) reasons.push(`smaller than the ${Math.round(min * 100)}% that counts`);
|
|
246
|
+
if (t.less >= alpha) reasons.push(`p ${t.less} is not under ${alpha}`);
|
|
247
|
+
if (ci[1] >= 0) reasons.push('the interval reaches zero');
|
|
248
|
+
return { ...res, verdict: 'same', why: `${head}: within the noise (${reasons.join('; ') || 'not clearly either way'})` };
|
|
249
|
+
}
|