personal-agent-benchmark 0.0.0-stage → 0.2.0

package/README.md

@@ -1,3 +1,129 @@
-# Temporary Holding Version
+# Personal Agent Benchmark
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+A benchmark for personal agents on real websites: shopping, posting, booking and more. Each task
+comes with a network-level recording of the sites it needs. During a run the agent's browser goes
+through a local replay proxy, so the agent sees the real URLs and real pages, while no request
+leaves the machine. Runs are repeatable, cannot trip a site's bot defenses, and can be repeated as
+often as needed.
+
+## How it works
+
+```text
+agent ─▶ browser ─▶ replay proxy (whistle + replay plugin) ─▶ recording of the task
+                                  │
+                                  └─ nothing is forwarded; misses are answered locally
+```
+
+- **Recording**: a person browses the task in a dedicated Chrome profile. Every HTTP exchange is
+  stored in a WARC file, with cookies and credentials removed.
+- **Matching**: each request is reduced to a key (method, host, path, the query parameters that
+  matter, the request body fields that matter). Site rules say which parameters matter, which
+  paths are the same page, which requests are searches, and which are trackers.
+- **Carts**: there is no cart arithmetic. Adding to the cart works for what was added while
+  recording; anything else misses. Cart and checkout pages answer with their last recording,
+  which is where the task left them.
+- **Clock**: pages see the time of the recording, so "deal ends in 2 hours" behaves as it did.
+
+## Repository layout
+
+```text
+tasks/<id>/        one task: task.yaml, recording/, rules/, grading.yaml
+sites/             site rules shared by all tasks (_default.yaml applies everywhere)
+src/               replay engine, whistle plugin, bundle format, CLI (pab)
+plugins/           whistle plugin package that loads the replay engine
+deploy/            replay and task images (built locally), a local Docker check
+docs/              task format, bundle format, site rules, deployment
+```
+
+## Quick start
+
+Needs Node 22+ and Google Chrome.
+
+```bash
+npm install
+npm run build
+node bin/pab.js list
+node bin/pab.js replay <task>
+```
+
+`replay` unpacks the task's recording into `.cache/`, starts the replay proxy and opens a Chrome
+window that uses it, at the first page opened while recording. Close the window to stop; hit rates are printed at the
+end.
+
+Connecting an agent:
+
+```bash
+node bin/pab.js replay <task> --cdp-port 9333   # the agent drives this Chrome over CDP
+node bin/pab.js replay <task> --no-browser      # proxy only; bring your own browser
+node bin/pab.js serve <task> --port 8899 --host 0.0.0.0   # in a container
+```
+
+A browser of your own needs `--proxy-server=http://127.0.0.1:<port> --ignore-certificate-errors
+--disable-quic`. The proxy also answers `http://rr.control/{health,stats,reset}`: `stats` counts
+hits and misses, `reset` starts a new run.
+
+Scoring a run: give the request log of the replay and the agent's final answer.
+
+```bash
+node bin/pab.js grade <task> --log .cache/logs/replay-<task>-<time>.jsonl --answer answer.md
+```
+
+It writes `result.json` and `report.html` (rendered from it) to `.cache/runs/<task>-<time>/`
+(`--out` to choose): the score, each graded item with its reason, and the pages the agent opened
+that came back 404 (see [docs/grading.md](docs/grading.md#run-report)).
+
+Other commands:
+
+```bash
+node bin/pab.js inspect <task> "https://www.amazon.com/s?k=airpods+pro"   # key and match of one request
+node bin/pab.js audit                          # check recordings are safe to publish
+node bin/pab.js stats .cache/logs/replay-*.jsonl
+```
+
+## As a package
+
+The replay engine, the recording format and the `pab` command are published to npm as
+`personal-agent-benchmark`; tasks are not part of the package.
+
+```bash
+npm install personal-agent-benchmark@0.2.0
+npx pab replay path/to/tasks/<id>
+```
+
+Modules are imported by path, for example `personal-agent-benchmark/scoring` or
+`personal-agent-benchmark/core/rules`; `exports` in `package.json` lists them. While the version
+is 0.x, a new minor version may change them. 0.1.x and 0.2.x read and write recordings in format 2
+([docs/bundle-format.md](docs/bundle-format.md)); a new format comes with a new minor version.
+
+## Tasks
+
+No task is published yet; `node bin/pab.js list` shows the tasks of a checkout.
+
+See [docs/tasks.md](docs/tasks.md) for the task format and how tasks are added.
+
+## Isolation
+
+- The proxy is a dedicated whistle instance with one rule that hands every request, HTTPS
+  included, to the replay plugin. The plugin only reads the recording.
+- Its DNS points at a dead address and proxy variables are removed, so even a request that slipped
+  past the plugin could not resolve a host.
+- The platform that runs the replay keeps it off the network and the agent's browser to the
+  proxy; [docs/deploy.md](docs/deploy.md) lists what it must ensure.
+
+## Recordings
+
+Recordings contain pages of third-party websites as they were served on the recording date, with
+cookies, credentials and personal data removed (`pab audit` checks this). They are published here
+for evaluating agents. To have content removed, open an issue.
+
+## Documentation
+
+- [docs/tasks.md](docs/tasks.md): task directory, task card, how a task is added
+- [docs/bundle-format.md](docs/bundle-format.md): what is inside `recording/`, and its format version
+- [docs/site-rules.md](docs/site-rules.md): request matching and site rules
+- [docs/grading.md](docs/grading.md): how answers to a task are scored
+- [docs/deploy.md](docs/deploy.md): replay and task images, what a platform running replays must ensure
+
+## License
+
+Not decided yet.

package/bin/pab.js

@@ -0,0 +1,5 @@
+#!/usr/bin/env node
+require('../dist/src/cli.js').main(process.argv.slice(2)).catch((err) => {
+  console.error(err && err.stack ? err.stack : err);
+  process.exit(1);
+});

package/dist/src/browser.d.ts

@@ -0,0 +1,29 @@
+import { type BrowserContext } from 'playwright-core';
+export interface BrowserOptions {
+    profileDir: string;
+    headless?: boolean;
+    /** Playwright channel; defaults to the locally installed Google Chrome. */
+    channel?: string;
+    executablePath?: string;
+    proxy?: {
+        server: string;
+        bypass?: string;
+    };
+    ignoreHTTPSErrors?: boolean;
+    locale?: string;
+    timezoneId?: string;
+    recordHarPath?: string;
+    /** Record a video of every page (tab) into this directory; saved when the page closes. */
+    recordVideo?: {
+        dir: string;
+        size?: {
+            width: number;
+            height: number;
+        };
+    };
+    /** Expose CDP on this port so an external agent can drive the same browser. */
+    cdpPort?: number;
+    args?: string[];
+}
+/** A dedicated, persistent Chrome profile; service workers blocked so every request hits the network. */
+export declare function launchBrowser(o: BrowserOptions): Promise<BrowserContext>;

package/dist/src/browser.js

@@ -0,0 +1,37 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.launchBrowser = launchBrowser;
+const playwright_core_1 = require("playwright-core");
+/** A dedicated, persistent Chrome profile; service workers blocked so every request hits the network. */
+async function launchBrowser(o) {
+    const args = [
+        '--disable-quic',
+        '--no-first-run',
+        '--no-default-browser-check',
+        // keep Chrome's own background traffic (sync, autofill, updates) out of recordings and replays
+        '--disable-background-networking',
+        '--disable-component-update',
+        '--disable-sync',
+        '--disable-domain-reliability',
+        '--disable-client-side-phishing-detection',
+        '--disable-features=AutofillServerCommunication,OptimizationHints,MediaRouter,Translate,InterestFeedContentSuggestions',
+        ...(o.args ?? []),
+    ];
+    if (o.cdpPort)
+        args.push(`--remote-debugging-port=${o.cdpPort}`);
+    return playwright_core_1.chromium.launchPersistentContext(o.profileDir, {
+        channel: o.executablePath ? undefined : o.channel ?? 'chrome',
+        executablePath: o.executablePath,
+        headless: o.headless ?? false,
+        proxy: o.proxy,
+        ignoreHTTPSErrors: o.ignoreHTTPSErrors,
+        locale: o.locale,
+        timezoneId: o.timezoneId,
+        viewport: { width: 1440, height: 900 },
+        serviceWorkers: 'block',
+        recordHar: o.recordHarPath ? { path: o.recordHarPath, content: 'embed', mode: 'full' } : undefined,
+        recordVideo: o.recordVideo,
+        args,
+    });
+}
+//# sourceMappingURL=browser.js.map

package/dist/src/browser.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"browser.js","sourceRoot":"","sources":["../../src/browser.ts"],"names":[],"mappings":";;;AAAA,qDAAgE;AAoBhE,yGAAyG;AAClG,KAAK,wBAAwB,CAAiB;IACnD,MAAM,IAAI,GAAG;QACX,gBAAgB;QAChB,gBAAgB;QAChB,4BAA4B;QAC5B,+FAA+F;QAC/F,iCAAiC;QACjC,4BAA4B;QAC5B,gBAAgB;QAChB,8BAA8B;QAC9B,0CAA0C;QAC1C,uHAAuH;QACvH,GAAG,CAAC,CAAC,CAAC,IAAI,IAAI,EAAE,CAAC;KAClB,CAAC;IACF,IAAI,CAAC,CAAC,OAAO;QAAE,IAAI,CAAC,IAAI,CAAC,2BAA2B,CAAC,CAAC,OAAO,EAAE,CAAC,CAAC;IACjE,OAAO,0BAAQ,CAAC,uBAAuB,CAAC,CAAC,CAAC,UAAU,EAAE;QACpD,OAAO,EAAE,CAAC,CAAC,cAAc,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,OAAO,IAAI,QAAQ;QAC7D,cAAc,EAAE,CAAC,CAAC,cAAc;QAChC,QAAQ,EAAE,CAAC,CAAC,QAAQ,IAAI,KAAK;QAC7B,KAAK,EAAE,CAAC,CAAC,KAAK;QACd,iBAAiB,EAAE,CAAC,CAAC,iBAAiB;QACtC,MAAM,EAAE,CAAC,CAAC,MAAM;QAChB,UAAU,EAAE,CAAC,CAAC,UAAU;QACxB,QAAQ,EAAE,EAAE,KAAK,EAAE,IAAI,EAAE,MAAM,EAAE,GAAG,EAAE;QACtC,cAAc,EAAE,OAAO;QACvB,SAAS,EAAE,CAAC,CAAC,aAAa,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC,aAAa,EAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,CAAC,CAAC,CAAC,SAAS;QAClG,WAAW,EAAE,CAAC,CAAC,WAAW;QAC1B,IAAI;KACL,CAAC,CAAC;AACL,CAAC"}

package/dist/src/cli.d.ts

@@ -0,0 +1,20 @@
+import { Bundle } from './core/bundle';
+import { RuleSet } from './core/rules';
+import { RecordStore } from './core/store';
+import type { ReplayStats } from './replay/engine';
+import { type ReplayTarget } from './tasks';
+export interface Args {
+    positional: string[];
+    flags: Record<string, string | boolean>;
+}
+export declare function parseArgs(argv: string[]): Args;
+export declare const str: (v: string | boolean | undefined) => string | undefined;
+export declare const num: (v: string | boolean | undefined, d: number) => number;
+export declare function main(argv: string[]): Promise<void>;
+/** Rules and recordings of a target, as the replay engine loads them. */
+export declare function loadTarget(target: ReplayTarget): {
+    bundle: Bundle;
+    rules: RuleSet;
+    store: RecordStore;
+};
+export declare function printStats(stats: ReplayStats): void;

package/dist/src/cli.js

@@ -0,0 +1,331 @@
+"use strict";
+var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    var desc = Object.getOwnPropertyDescriptor(m, k);
+    if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
+      desc = { enumerable: true, get: function() { return m[k]; } };
+    }
+    Object.defineProperty(o, k2, desc);
+}) : (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    o[k2] = m[k];
+}));
+var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) {
+    Object.defineProperty(o, "default", { enumerable: true, value: v });
+}) : function(o, v) {
+    o["default"] = v;
+});
+var __importStar = (this && this.__importStar) || (function () {
+    var ownKeys = function(o) {
+        ownKeys = Object.getOwnPropertyNames || function (o) {
+            var ar = [];
+            for (var k in o) if (Object.prototype.hasOwnProperty.call(o, k)) ar[ar.length] = k;
+            return ar;
+        };
+        return ownKeys(o);
+    };
+    return function (mod) {
+        if (mod && mod.__esModule) return mod;
+        var result = {};
+        if (mod != null) for (var k = ownKeys(mod), i = 0; i < k.length; i++) if (k[i] !== "default") __createBinding(result, mod, k[i]);
+        __setModuleDefault(result, mod);
+        return result;
+    };
+})();
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.num = exports.str = void 0;
+exports.parseArgs = parseArgs;
+exports.main = main;
+exports.loadTarget = loadTarget;
+exports.printStats = printStats;
+const fs = __importStar(require("node:fs"));
+const os = __importStar(require("node:os"));
+const path = __importStar(require("node:path"));
+const browser_1 = require("./browser");
+const audit_1 = require("./core/audit");
+const bundle_1 = require("./core/bundle");
+const lookup_1 = require("./core/lookup");
+const normalize_1 = require("./core/normalize");
+const rules_1 = require("./core/rules");
+const store_1 = require("./core/store");
+const util_1 = require("./core/util");
+const grading_1 = require("./grading");
+const judge_1 = require("./judge");
+const launcher_1 = require("./replay/launcher");
+const scoring_1 = require("./scoring");
+const tasks_1 = require("./tasks");
+const HELP = `pab - Personal Agent Benchmark: replay real-site tasks offline
+
+  pab list                                   tasks in this repository
+  pab replay <task> [--port 18899] [--url URL] [--cdp-port 9333] [--headless] [--no-browser] [--log FILE]
+                                             replay proxy plus a Chrome window that uses it
+  pab serve <task> [--port 8899] [--host 127.0.0.1] [--log FILE]
+                                             replay proxy only (containers, external browsers)
+  pab inspect <task> <url> [--method GET]    normalized key of a request and the recording it hits
+  pab audit [<task>...]                      check recordings are safe to publish (all tasks by default)
+  pab unpack <task>                          unpack the recording and print its directory
+  pab stats <replay-log.jsonl>               summarize a replay log
+  pab grade <task> --log <replay-log.jsonl> [--answer FILE] [--out DIR] [--judge api|cli] [--model M]
+                                             score a run and list the pages that returned 404;
+                                             writes result.json and report.html
+
+<task> is a task id, a task directory, or an unpacked bundle directory.
+--repo DIR (or PAB_REPO) points at another checkout of this repository.
+`;
+function parseArgs(argv) {
+    const positional = [];
+    const flags = {};
+    for (let i = 0; i < argv.length; i++) {
+        const a = argv[i];
+        if (a.startsWith('--')) {
+            const name = a.slice(2);
+            const next = argv[i + 1];
+            if (next !== undefined && !next.startsWith('--')) {
+                flags[name] = next;
+                i++;
+            }
+            else {
+                flags[name] = true;
+            }
+        }
+        else {
+            positional.push(a);
+        }
+    }
+    return { positional, flags };
+}
+const str = (v) => (typeof v === 'string' ? v : undefined);
+exports.str = str;
+const num = (v, d) => (typeof v === 'string' ? Number(v) : d);
+exports.num = num;
+const repoOf = (flags) => ((0, exports.str)(flags.repo) ? new tasks_1.Repo(path.resolve((0, exports.str)(flags.repo))) : tasks_1.Repo.default());
+async function main(argv) {
+    const [cmd, ...rest] = argv;
+    const { positional, flags } = parseArgs(rest);
+    switch (cmd) {
+        case 'list':
+            return cmdList(flags);
+        case 'replay':
+            return cmdReplay(positional, flags);
+        case 'serve':
+            return cmdServe(positional, flags);
+        case 'inspect':
+            return cmdInspect(positional, flags);
+        case 'audit':
+            return cmdAudit(positional, flags);
+        case 'unpack': {
+            const [arg] = positional;
+            if (!arg)
+                throw new Error('usage: pab unpack <task>');
+            console.log((await (0, tasks_1.resolveTarget)(arg, repoOf(flags))).bundleDir);
+            return;
+        }
+        case 'stats':
+            return cmdStats(positional);
+        case 'grade':
+            return cmdGrade(positional, flags);
+        default:
+            console.log(HELP);
+    }
+}
+function cmdList(flags) {
+    const repo = repoOf(flags);
+    for (const id of repo.taskIds()) {
+        const task = repo.task(id);
+        const card = task.card();
+        const recorded = task.recordingFiles().length ? '' : '  (no recording yet)';
+        console.log(`${id.padEnd(28)} ${card.title ?? ''}${recorded}`);
+    }
+}
+/** Rules and recordings of a target, as the replay engine loads them. */
+function loadTarget(target) {
+    const bundle = new bundle_1.Bundle(target.bundleDir);
+    const rules = rules_1.RuleSet.load([bundle.rulesDir, ...target.rulesDirs]);
+    const store = new store_1.RecordStore(bundle, rules);
+    return { bundle, rules, store };
+}
+async function cmdReplay(positional, flags) {
+    const [arg] = positional;
+    if (!arg)
+        throw new Error('usage: pab replay <task>');
+    const repo = repoOf(flags);
+    const target = await (0, tasks_1.resolveTarget)(arg, repo);
+    const bundle = new bundle_1.Bundle(target.bundleDir);
+    const manifest = bundle.manifest();
+    const port = (0, exports.num)(flags.port, 18899);
+    const logFile = (0, exports.str)(flags.log) ?? path.join(repo.cacheDir, 'logs', `replay-${target.name}-${Date.now()}.jsonl`);
+    const server = await (0, launcher_1.startReplayServer)({ bundleDir: target.bundleDir, rulesDirs: target.rulesDirs, port, logFile });
+    console.log(`replay proxy: ${server.proxy} (${target.name}, recorded ${manifest.captured_from})`);
+    console.log(`request log: ${logFile}`);
+    const stop = async () => {
+        const stats = await server.control('/stats').catch(() => undefined);
+        await server.stop();
+        if (stats)
+            printStats(stats);
+    };
+    if (flags['no-browser'] === true) {
+        console.log(`point a browser at it with: --proxy-server=${server.proxy} --ignore-certificate-errors --disable-quic`);
+        await new Promise((resolve) => process.once('SIGINT', () => resolve()));
+        await stop();
+        return;
+    }
+    const profile = fs.mkdtempSync(path.join(os.tmpdir(), 'pab-replay-'));
+    const cdpPort = flags['cdp-port'] ? (0, exports.num)(flags['cdp-port'], 9333) : undefined;
+    const context = await (0, browser_1.launchBrowser)({
+        profileDir: profile,
+        headless: flags.headless === true,
+        proxy: { server: server.proxy },
+        ignoreHTTPSErrors: true,
+        locale: manifest.locale,
+        timezoneId: manifest.timezone,
+        cdpPort,
+    });
+    await context.clock.install({ time: new Date(manifest.captured_from) });
+    if (cdpPort)
+        console.log(`CDP: http://127.0.0.1:${cdpPort} (an agent can drive this browser)`);
+    const page = context.pages()[0] ?? (await context.newPage());
+    // the first page opened while recording, unless told otherwise
+    const url = (0, exports.str)(flags.url) ?? bundle.pages().find((p) => p.note !== 'not-found-probe')?.url;
+    if (url)
+        await page.goto(url, { waitUntil: 'domcontentloaded' }).catch((e) => console.error(String(e)));
+    console.log('Close the browser window or press Ctrl-C to stop.');
+    await new Promise((resolve) => {
+        context.on('close', () => resolve());
+        process.once('SIGINT', () => resolve());
+    });
+    await context.close().catch(() => undefined);
+    fs.rmSync(profile, { recursive: true, force: true });
+    await stop();
+}
+async function cmdServe(positional, flags) {
+    const [arg] = positional;
+    if (!arg)
+        throw new Error('usage: pab serve <task>');
+    const target = await (0, tasks_1.resolveTarget)(arg, repoOf(flags));
+    const host = (0, exports.str)(flags.host) ?? '127.0.0.1';
+    const server = await (0, launcher_1.startReplayServer)({
+        bundleDir: target.bundleDir,
+        rulesDirs: target.rulesDirs,
+        port: (0, exports.num)(flags.port, 8899),
+        host,
+        logFile: (0, exports.str)(flags.log),
+    });
+    console.log(`replay proxy for ${target.name} listening on ${host}:${server.port}`);
+    await new Promise((resolve) => {
+        process.once('SIGINT', () => resolve());
+        process.once('SIGTERM', () => resolve());
+        server.child.once('exit', () => resolve());
+    });
+    await server.stop();
+}
+async function cmdInspect(positional, flags) {
+    const [arg, url] = positional;
+    if (!arg || !url)
+        throw new Error('usage: pab inspect <task> <url>');
+    const { rules, store } = loadTarget(await (0, tasks_1.resolveTarget)(arg, repoOf(flags)));
+    const norm = (0, normalize_1.normalizeRequest)({ method: (0, exports.str)(flags.method) ?? 'GET', url }, rules);
+    const found = (0, lookup_1.lookup)(store, norm);
+    console.log(JSON.stringify({ key: norm.key, kind: norm.kind, endpoint: norm.endpoint?.name, site: norm.site?.site, text: norm.text, match: found ? { source: found.source, entry: found.ix.entry.id, url: found.ix.entry.url, detail: found.detail } : null }, null, 2));
+}
+async function cmdAudit(positional, flags) {
+    const repo = repoOf(flags);
+    const args = positional.length ? positional : repo.taskIds().filter((id) => repo.task(id).recordingFiles().length);
+    let failed = 0;
+    for (const arg of args) {
+        const target = await (0, tasks_1.resolveTarget)(arg, repo);
+        const { bundle, rules } = loadTarget(target);
+        const findings = await (0, audit_1.auditBundle)(bundle, rules);
+        for (const part of target.task?.recordingFiles() ?? []) {
+            const mb = fs.statSync(part).size / 1024 / 1024;
+            if (mb > 50)
+                findings.push({ check: 'size', detail: `${path.basename(part)} is ${mb.toFixed(1)} MB (git hosts warn above 50 MB)` });
+        }
+        console.log(`${findings.length ? 'FAIL' : 'ok  '}  ${target.name}`);
+        for (const f of findings)
+            console.log(`      [${f.check}] ${f.detail}`);
+        if (findings.length)
+            failed++;
+    }
+    if (failed)
+        process.exitCode = 1;
+}
+function cmdStats(positional) {
+    const [file] = positional;
+    if (!file)
+        throw new Error('usage: pab stats <replay-log.jsonl>');
+    const rows = (0, util_1.readJsonl)(file);
+    const byClass = {};
+    const misses = {};
+    const bySource = {};
+    for (const r of rows) {
+        const c = (byClass[r.class] ??= { total: 0, hit: 0, miss: 0 });
+        c.total++;
+        bySource[r.source] = (bySource[r.source] ?? 0) + 1;
+        if (r.source === 'miss') {
+            c.miss++;
+            const m = (misses[r.url] ??= { count: 0, resourceClass: r.class, url: r.url });
+            m.count++;
+        }
+        else
+            c.hit++;
+    }
+    printStats({ requests: rows.length, byClass: byClass, bySource, misses: misses });
+}
+async function cmdGrade(positional, flags) {
+    const [arg] = positional;
+    const logFile = (0, exports.str)(flags.log);
+    if (!arg || !logFile)
+        throw new Error('usage: pab grade <task> --log <replay-log.jsonl> [--answer FILE] [--out DIR]');
+    const repo = repoOf(flags);
+    const target = await (0, tasks_1.resolveTarget)(arg, repo);
+    const { bundle, rules } = loadTarget(target);
+    const task = target.task;
+    const card = task?.card();
+    let grading;
+    if (task && fs.existsSync(task.gradingPath)) {
+        grading = (0, grading_1.parseGrading)(fs.readFileSync(task.gradingPath, 'utf8'));
+        const errors = (0, grading_1.validateGrading)(grading);
+        if (errors.length)
+            throw new Error(`${task.gradingPath}:\n  ${errors.join('\n  ')}`);
+    }
+    const answer = (0, exports.str)(flags.answer) ? fs.readFileSync((0, exports.str)(flags.answer), 'utf8') : undefined;
+    const result = await (0, scoring_1.scoreRun)({
+        task: { id: card?.id ?? target.name, title: card?.title, prompt: card?.prompt },
+        grading,
+        answer,
+        log: (0, util_1.readJsonl)(logFile),
+        logFile: path.resolve(logFile),
+        rules,
+        probes: Object.values(bundle.manifest().not_found ?? {}),
+        judge: grading && answer !== undefined ? (0, judge_1.pickJudge)((0, exports.str)(flags.judge), (0, exports.str)(flags.model)) : undefined,
+    });
+    const out = (0, exports.str)(flags.out) ?? path.join(repo.cacheDir, 'runs', `${target.name}-${new Date().toISOString().replace(/[:.]/g, '-')}`);
+    const files = (0, scoring_1.writeRun)(result, out);
+    if (result.score !== undefined)
+        console.log(`score ${(result.score * 100).toFixed(1)}%${result.grader ? ` (graded by ${result.grader.model})` : ''}`);
+    for (const n of result.notes)
+        console.log(n);
+    console.log(`pages that returned 404: ${result.pages.not_found.length}`);
+    for (const p of result.pages.not_found)
+        console.log(`  [${p.source === 'miss' ? 'not recorded' : 'recorded as 404'}] x${p.count} ${p.route.slice(0, 140)}`);
+    console.log(`result: ${files.json}`);
+    console.log(`report: ${files.html}`);
+    if (grading && answer !== undefined && result.score === undefined)
+        process.exitCode = 1;
+}
+function printStats(stats) {
+    console.log(`requests ${stats.requests}`);
+    for (const [cls, c] of Object.entries(stats.byClass)) {
+        if (!c.total)
+            continue;
+        console.log(`  ${cls.padEnd(9)} hit ${c.hit}/${c.total} (${((c.hit / c.total) * 100).toFixed(1)}%)`);
+    }
+    console.log(`  sources ${JSON.stringify(stats.bySource)}`);
+    const top = Object.values(stats.misses).sort((a, b) => b.count - a.count).slice(0, 15);
+    if (top.length) {
+        console.log('  misses (top 15):');
+        for (const m of top)
+            console.log(`    [${m.resourceClass}] x${m.count} ${m.url.slice(0, 140)}`);
+    }
+}
+//# sourceMappingURL=cli.js.map

package/dist/src/cli.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"cli.js","sourceRoot":"","sources":["../../src/cli.ts"],"names":[],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAAA,MAAY,EAAE,oCAAgB;AAC9B,MAAY,EAAE,oCAAgB;AAC9B,MAAY,IAAI,sCAAkB;AAClC,uCAA0C;AAC1C,wCAA2C;AAC3C,0CAAuC;AACvC,0CAAuC;AACvC,gDAAoD;AACpD,wCAAuC;AACvC,wCAA2C;AAC3C,sCAAwC;AACxC,uCAAwE;AACxE,mCAAoC;AAEpC,gDAAsD;AACtD,uCAA6D;AAC7D,mCAAiE;AAEjE,MAAM,IAAI,GAAG;;;;;;;;;;;;;;;;;CAiBZ,CAAC;AAOF,mBAA0B,IAAc;IACtC,MAAM,UAAU,GAAa,EAAE,CAAC;IAChC,MAAM,KAAK,GAAqC,EAAE,CAAC;IACnD,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QACrC,MAAM,CAAC,GAAG,IAAI,CAAC,CAAC,CAAC,CAAC;QAClB,IAAI,CAAC,CAAC,UAAU,CAAC,IAAI,CAAC,EAAE,CAAC;YACvB,MAAM,IAAI,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;YACxB,MAAM,IAAI,GAAG,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;YACzB,IAAI,IAAI,KAAK,SAAS,IAAI,CAAC,IAAI,CAAC,UAAU,CAAC,IAAI,CAAC,EAAE,CAAC;gBACjD,KAAK,CAAC,IAAI,CAAC,GAAG,IAAI,CAAC;gBACnB,CAAC,EAAE,CAAC;YACN,CAAC;iBAAM,CAAC;gBACN,KAAK,CAAC,IAAI,CAAC,GAAG,IAAI,CAAC;YACrB,CAAC;QACH,CAAC;aAAM,CAAC;YACN,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;QACrB,CAAC;IACH,CAAC;IACD,OAAO,EAAE,UAAU,EAAE,KAAK,EAAE,CAAC;AAC/B,CAAC;AAEM,MAAM,GAAG,GAAG,CAAC,CAA+B,EAAsB,EAAE,CAAC,CAAC,OAAO,CAAC,KAAK,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC;AAAvG,QAAA,GAAG,GAAH,GAAG,CAAoG;AAC7G,MAAM,GAAG,GAAG,CAAC,CAA+B,EAAE,CAAS,EAAU,EAAE,CAAC,CAAC,OAAO,CAAC,KAAK,QAAQ,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;AAAtG,QAAA,GAAG,GAAH,GAAG,CAAmG;AAEnH,MAAM,MAAM,GAAG,CAAC,KAAoB,EAAE,EAAE,CAAC,CAAC,IAAA,QAAA,GAAG,EAAC,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,YAAI,CAAC,IAAI,CAAC,OAAO,CAAC,IAAA,QAAA,GAAG,EAAC,KAAK,CAAC,IAAI,CAAE,CAAC,CAAC,CAAC,CAAC,CAAC,YAAI,CAAC,OAAO,EAAE,CAAC,CAAC;AAEhH,KAAK,eAAe,IAAc;IACvC,MAAM,CAAC,GAAG,EAAE,GAAG,IAAI,CAAC,GAAG,IAAI,CAAC;IAC5B,MAAM,EAAE,UAAU,EAAE,KAAK,EAAE,GAAG,SAAS,CAAC,IAAI,CAAC,CAAC;IAC9C,QAAQ,GAAG,EAAE,CAAC;QACZ,KAAK,MAAM;YACT,OAAO,OAAO,CAAC,KAAK,CAAC,CAAC;QACxB,KAAK,QAAQ;YACX,OAAO,SAAS,CAAC,UAAU,EAAE,KAAK,CAAC,CAAC;QACtC,KAAK,OAAO;YACV,OAAO,QAAQ,CAAC,UAAU,EAAE,KAAK,CAAC,CAAC;QACrC,KAAK,SAAS;YACZ,OAAO,UAAU,CAAC,UAAU,EAAE,KAAK,CAAC,CAAC;QACvC,KAAK,OAAO;YACV,OAAO,QAAQ,CAAC,UAAU,EAAE,KAAK,CAAC,CAAC;QACrC,KAAK,QAAQ,EAAE,CAAC;YACd,MAAM,CAAC,GAAG,CAAC,GAAG,UAAU,CAAC;YACzB,IAAI,CAAC,GAAG;gBAAE,MAAM,IAAI,KAAK,CAAC,0BAA0B,CAAC,CAAC;YACtD,OAAO,CAAC,GAAG,CAAC,CAAC,MAAM,IAAA,qBAAa,EAAC,GAAG,EAAE,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC;YACjE,OAAO;QACT,CAAC;QACD,KAAK,OAAO;YACV,OAAO,QAAQ,CAAC,UAAU,CAAC,CAAC;QAC9B,KAAK,OAAO;YACV,OAAO,QAAQ,CAAC,UAAU,EAAE,KAAK,CAAC,CAAC;QACrC;YACE,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;IACtB,CAAC;AACH,CAAC;AAED,SAAS,OAAO,CAAC,KAAoB;IACnC,MAAM,IAAI,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC;IAC3B,KAAK,MAAM,EAAE,IAAI,IAAI,CAAC,OAAO,EAAE,EAAE,CAAC;QAChC,MAAM,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;QAC3B,MAAM,IAAI,GAAG,IAAI,CAAC,IAAI,EAAE,CAAC;QACzB,MAAM,QAAQ,GAAG,IAAI,CAAC,cAAc,EAAE,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,sBAAsB,CAAC;QAC5E,OAAO,CAAC,GAAG,CAAC,GAAG,EAAE,CAAC,MAAM,CAAC,EAAE,CAAC,IAAI,IAAI,CAAC,KAAK,IAAI,EAAE,GAAG,QAAQ,EAAE,CAAC,CAAC;IACjE,CAAC;AACH,CAAC;AAED,yEAAyE;AACzE,oBAA2B,MAAoB;IAC7C,MAAM,MAAM,GAAG,IAAI,eAAM,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;IAC5C,MAAM,KAAK,GAAG,eAAO,CAAC,IAAI,CAAC,CAAC,MAAM,CAAC,QAAQ,EAAE,GAAG,MAAM,CAAC,SAAS,CAAC,CAAC,CAAC;IACnE,MAAM,KAAK,GAAG,IAAI,mBAAW,CAAC,MAAM,EAAE,KAAK,CAAC,CAAC;IAC7C,OAAO,EAAE,MAAM,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC;AAClC,CAAC;AAED,KAAK,UAAU,SAAS,CAAC,UAAoB,EAAE,KAAoB;IACjE,MAAM,CAAC,GAAG,CAAC,GAAG,UAAU,CAAC;IACzB,IAAI,CAAC,GAAG;QAAE,MAAM,IAAI,KAAK,CAAC,0BAA0B,CAAC,CAAC;IACtD,MAAM,IAAI,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC;IAC3B,MAAM,MAAM,GAAG,MAAM,IAAA,qBAAa,EAAC,GAAG,EAAE,IAAI,CAAC,CAAC;IAC9C,MAAM,MAAM,GAAG,IAAI,eAAM,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;IAC5C,MAAM,QAAQ,GAAG,MAAM,CAAC,QAAQ,EAAE,CAAC;IACnC,MAAM,IAAI,GAAG,IAAA,QAAA,GAAG,EAAC,KAAK,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC;IACpC,MAAM,OAAO,GAAG,IAAA,QAAA,GAAG,EAAC,KAAK,CAAC,GAAG,CAAC,IAAI,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,QAAQ,EAAE,MAAM,EAAE,UAAU,MAAM,CAAC,IAAI,IAAI,IAAI,CAAC,GAAG,EAAE,QAAQ,CAAC,CAAC;IAChH,MAAM,MAAM,GAAG,MAAM,IAAA,4BAAiB,EAAC,EAAE,SAAS,EAAE,MAAM,CAAC,SAAS,EAAE,SAAS,EAAE,MAAM,CAAC,SAAS,EAAE,IAAI,EAAE,OAAO,EAAE,CAAC,CAAC;IACpH,OAAO,CAAC,GAAG,CAAC,iBAAiB,MAAM,CAAC,KAAK,KAAK,MAAM,CAAC,IAAI,cAAc,QAAQ,CAAC,aAAa,GAAG,CAAC,CAAC;IAClG,OAAO,CAAC,GAAG,CAAC,gBAAgB,OAAO,EAAE,CAAC,CAAC;IACvC,MAAM,IAAI,GAAG,KAAK,IAAI,EAAE;QACtB,MAAM,KAAK,GAAG,MAAM,MAAM,CAAC,OAAO,CAAc,QAAQ,CAAC,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,SAAS,CAAC,CAAC;QACjF,MAAM,MAAM,CAAC,IAAI,EAAE,CAAC;QACpB,IAAI,KAAK;YAAE,UAAU,CAAC,KAAK,CAAC,CAAC;IAC/B,CAAC,CAAC;IACF,IAAI,KAAK,CAAC,YAAY,CAAC,KAAK,IAAI,EAAE,CAAC;QACjC,OAAO,CAAC,GAAG,CAAC,8CAA8C,MAAM,CAAC,KAAK,6CAA6C,CAAC,CAAC;QACrH,MAAM,IAAI,OAAO,CAAO,CAAC,OAAO,EAAE,EAAE,CAAC,OAAO,CAAC,IAAI,CAAC,QAAQ,EAAE,GAAG,EAAE,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC;QAC9E,MAAM,IAAI,EAAE,CAAC;QACb,OAAO;IACT,CAAC;IACD,MAAM,OAAO,GAAG,EAAE,CAAC,WAAW,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,MAAM,EAAE,EAAE,aAAa,CAAC,CAAC,CAAC;IACtE,MAAM,OAAO,GAAG,KAAK,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC,IAAA,QAAA,GAAG,EAAC,KAAK,CAAC,UAAU,CAAC,EAAE,IAAI,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC;IAC7E,MAAM,OAAO,GAAG,MAAM,IAAA,uBAAa,EAAC;QAClC,UAAU,EAAE,OAAO;QACnB,QAAQ,EAAE,KAAK,CAAC,QAAQ,KAAK,IAAI;QACjC,KAAK,EAAE,EAAE,MAAM,EAAE,MAAM,CAAC,KAAK,EAAE;QAC/B,iBAAiB,EAAE,IAAI;QACvB,MAAM,EAAE,QAAQ,CAAC,MAAM;QACvB,UAAU,EAAE,QAAQ,CAAC,QAAQ;QAC7B,OAAO;KACR,CAAC,CAAC;IACH,MAAM,OAAO,CAAC,KAAK,CAAC,OAAO,CAAC,EAAE,IAAI,EAAE,IAAI,IAAI,CAAC,QAAQ,CAAC,aAAa,CAAC,EAAE,CAAC,CAAC;IACxE,IAAI,OAAO;QAAE,OAAO,CAAC,GAAG,CAAC,yBAAyB,OAAO,oCAAoC,CAAC,CAAC;IAC/F,MAAM,IAAI,GAAG,OAAO,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,OAAO,CAAC,OAAO,EAAE,CAAC,CAAC;IAC7D,+DAA+D;IAC/D,MAAM,GAAG,GAAG,IAAA,QAAA,GAAG,EAAC,KAAK,CAAC,GAAG,CAAC,IAAI,MAAM,CAAC,KAAK,EAAE,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,iBAAiB,CAAC,EAAE,GAAG,CAAC;IAC5F,IAAI,GAAG;QAAE,MAAM,IAAI,CAAC,IAAI,CAAC,GAAG,EAAE,EAAE,SAAS,EAAE,kBAAkB,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,OAAO,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IACxG,OAAO,CAAC,GAAG,CAAC,mDAAmD,CAAC,CAAC;IACjE,MAAM,IAAI,OAAO,CAAO,CAAC,OAAO,EAAE,EAAE;QAClC,OAAO,CAAC,EAAE,CAAC,OAAO,EAAE,GAAG,EAAE,CAAC,OAAO,EAAE,CAAC,CAAC;QACrC,OAAO,CAAC,IAAI,CAAC,QAAQ,EAAE,GAAG,EAAE,CAAC,OAAO,EAAE,CAAC,CAAC;IAC1C,CAAC,CAAC,CAAC;IACH,MAAM,OAAO,CAAC,KAAK,EAAE,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,SAAS,CAAC,CAAC;IAC7C,EAAE,CAAC,MAAM,CAAC,OAAO,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,CAAC;IACrD,MAAM,IAAI,EAAE,CAAC;AACf,CAAC;AAED,KAAK,UAAU,QAAQ,CAAC,UAAoB,EAAE,KAAoB;IAChE,MAAM,CAAC,GAAG,CAAC,GAAG,UAAU,CAAC;IACzB,IAAI,CAAC,GAAG;QAAE,MAAM,IAAI,KAAK,CAAC,yBAAyB,CAAC,CAAC;IACrD,MAAM,MAAM,GAAG,MAAM,IAAA,qBAAa,EAAC,GAAG,EAAE,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC;IACvD,MAAM,IAAI,GAAG,IAAA,QAAA,GAAG,EAAC,KAAK,CAAC,IAAI,CAAC,IAAI,WAAW,CAAC;IAC5C,MAAM,MAAM,GAAG,MAAM,IAAA,4BAAiB,EAAC;QACrC,SAAS,EAAE,MAAM,CAAC,SAAS;QAC3B,SAAS,EAAE,MAAM,CAAC,SAAS;QAC3B,IAAI,EAAE,IAAA,QAAA,GAAG,EAAC,KAAK,CAAC,IAAI,EAAE,IAAI,CAAC;QAC3B,IAAI;QACJ,OAAO,EAAE,IAAA,QAAA,GAAG,EAAC,KAAK,CAAC,GAAG,CAAC;KACxB,CAAC,CAAC;IACH,OAAO,CAAC,GAAG,CAAC,oBAAoB,MAAM,CAAC,IAAI,iBAAiB,IAAI,IAAI,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC;IACnF,MAAM,IAAI,OAAO,CAAO,CAAC,OAAO,EAAE,EAAE;QAClC,OAAO,CAAC,IAAI,CAAC,QAAQ,EAAE,GAAG,EAAE,CAAC,OAAO,EAAE,CAAC,CAAC;QACxC,OAAO,CAAC,IAAI,CAAC,SAAS,EAAE,GAAG,EAAE,CAAC,OAAO,EAAE,CAAC,CAAC;QACzC,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,MAAM,EAAE,GAAG,EAAE,CAAC,OAAO,EAAE,CAAC,CAAC;IAC7C,CAAC,CAAC,CAAC;IACH,MAAM,MAAM,CAAC,IAAI,EAAE,CAAC;AACtB,CAAC;AAED,KAAK,UAAU,UAAU,CAAC,UAAoB,EAAE,KAAoB;IAClE,MAAM,CAAC,GAAG,EAAE,GAAG,CAAC,GAAG,UAAU,CAAC;IAC9B,IAAI,CAAC,GAAG,IAAI,CAAC,GAAG;QAAE,MAAM,IAAI,KAAK,CAAC,iCAAiC,CAAC,CAAC;IACrE,MAAM,EAAE,KAAK,EAAE,KAAK,EAAE,GAAG,UAAU,CAAC,MAAM,IAAA,qBAAa,EAAC,GAAG,EAAE,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;IAC7E,MAAM,IAAI,GAAG,IAAA,4BAAgB,EAAC,EAAE,MAAM,EAAE,IAAA,QAAA,GAAG,EAAC,KAAK,CAAC,MAAM,CAAC,IAAI,KAAK,EAAE,GAAG,EAAE,EAAE,KAAK,CAAC,CAAC;IAClF,MAAM,KAAK,GAAG,IAAA,eAAM,EAAC,KAAK,EAAE,IAAI,CAAC,CAAC;IAClC,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,EAAE,GAAG,EAAE,IAAI,CAAC,GAAG,EAAE,IAAI,EAAE,IAAI,CAAC,IAAI,EAAE,QAAQ,EAAE,IAAI,CAAC,QAAQ,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,CAAC,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,CAAC,IAAI,EAAE,KAAK,EAAE,KAAK,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,KAAK,CAAC,MAAM,EAAE,KAAK,EAAE,KAAK,CAAC,EAAE,CAAC,KAAK,CAAC,EAAE,EAAE,GAAG,EAAE,KAAK,CAAC,EAAE,CAAC,KAAK,CAAC,GAAG,EAAE,MAAM,EAAE,KAAK,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC;AAC3Q,CAAC;AAED,KAAK,UAAU,QAAQ,CAAC,UAAoB,EAAE,KAAoB;IAChE,MAAM,IAAI,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC;IAC3B,MAAM,IAAI,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC,CAAC,UAAU,CAAC,CAAC,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC,MAAM,CAAC,CAAC,EAAE,EAAE,EAAE,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,cAAc,EAAE,CAAC,MAAM,CAAC,CAAC;IACnH,IAAI,MAAM,GAAG,CAAC,CAAC;IACf,KAAK,MAAM,GAAG,IAAI,IAAI,EAAE,CAAC;QACvB,MAAM,MAAM,GAAG,MAAM,IAAA,qBAAa,EAAC,GAAG,EAAE,IAAI,CAAC,CAAC;QAC9C,MAAM,EAAE,MAAM,EAAE,KAAK,EAAE,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC;QAC7C,MAAM,QAAQ,GAAG,MAAM,IAAA,mBAAW,EAAC,MAAM,EAAE,KAAK,CAAC,CAAC;QAClD,KAAK,MAAM,IAAI,IAAI,MAAM,CAAC,IAAI,EAAE,cAAc,EAAE,IAAI,EAAE,EAAE,CAAC;YACvD,MAAM,EAAE,GAAG,EAAE,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,IAAI,GAAG,IAAI,GAAG,IAAI,CAAC;YAChD,IAAI,EAAE,GAAG,EAAE;gBAAE,QAAQ,CAAC,IAAI,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,MAAM,EAAE,GAAG,IAAI,CAAC,QAAQ,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC,OAAO,CAAC,CAAC,CAAC,kCAAkC,EAAE,CAAC,CAAC;QACtI,CAAC;QACD,OAAO,CAAC,GAAG,CAAC,GAAG,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,KAAK,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC;QACpE,KAAK,MAAM,CAAC,IAAI,QAAQ;YAAE,OAAO,CAAC,GAAG,CAAC,UAAU,CAAC,CAAC,KAAK,KAAK,CAAC,CAAC,MAAM,EAAE,CAAC,CAAC;QACxE,IAAI,QAAQ,CAAC,MAAM;YAAE,MAAM,EAAE,CAAC;IAChC,CAAC;IACD,IAAI,MAAM;QAAE,OAAO,CAAC,QAAQ,GAAG,CAAC,CAAC;AACnC,CAAC;AAED,SAAS,QAAQ,CAAC,UAAoB;IACpC,MAAM,CAAC,IAAI,CAAC,GAAG,UAAU,CAAC;IAC1B,IAAI,CAAC,IAAI;QAAE,MAAM,IAAI,KAAK,CAAC,qCAAqC,CAAC,CAAC;IAClE,MAAM,IAAI,GAAG,IAAA,gBAAS,EAAiD,IAAI,CAAC,CAAC;IAC7E,MAAM,OAAO,GAAiE,EAAE,CAAC;IACjF,MAAM,MAAM,GAA0E,EAAE,CAAC;IACzF,MAAM,QAAQ,GAA2B,EAAE,CAAC;IAC5C,KAAK,MAAM,CAAC,IAAI,IAAI,EAAE,CAAC;QACrB,MAAM,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,KAAK,CAAC,KAAK,EAAE,KAAK,EAAE,CAAC,EAAE,GAAG,EAAE,CAAC,EAAE,IAAI,EAAE,CAAC,EAAE,CAAC,CAAC;QAC/D,CAAC,CAAC,KAAK,EAAE,CAAC;QACV,QAAQ,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QACnD,IAAI,CAAC,CAAC,MAAM,KAAK,MAAM,EAAE,CAAC;YACxB,CAAC,CAAC,IAAI,EAAE,CAAC;YACT,MAAM,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,KAAK,EAAE,KAAK,EAAE,CAAC,EAAE,aAAa,EAAE,CAAC,CAAC,KAAK,EAAE,GAAG,EAAE,CAAC,CAAC,GAAG,EAAE,CAAC,CAAC;YAC/E,CAAC,CAAC,KAAK,EAAE,CAAC;QACZ,CAAC;;YAAM,CAAC,CAAC,GAAG,EAAE,CAAC;IACjB,CAAC;IACD,UAAU,CAAC,EAAE,QAAQ,EAAE,IAAI,CAAC,MAAM,EAAE,OAAO,EAAE,OAAiC,EAAE,QAAQ,EAAE,MAAM,EAAE,MAA+B,EAAE,CAAC,CAAC;AACvI,CAAC;AAED,KAAK,UAAU,QAAQ,CAAC,UAAoB,EAAE,KAAoB;IAChE,MAAM,CAAC,GAAG,CAAC,GAAG,UAAU,CAAC;IACzB,MAAM,OAAO,GAAG,IAAA,QAAA,GAAG,EAAC,KAAK,CAAC,GAAG,CAAC,CAAC;IAC/B,IAAI,CAAC,GAAG,IAAI,CAAC,OAAO;QAAE,MAAM,IAAI,KAAK,CAAC,8EAA8E,CAAC,CAAC;IACtH,MAAM,IAAI,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC;IAC3B,MAAM,MAAM,GAAG,MAAM,IAAA,qBAAa,EAAC,GAAG,EAAE,IAAI,CAAC,CAAC;IAC9C,MAAM,EAAE,MAAM,EAAE,KAAK,EAAE,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC;IAC7C,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,CAAC;IACzB,MAAM,IAAI,GAAG,IAAI,EAAE,IAAI,EAAE,CAAC;IAC1B,IAAI,OAA4B,CAAC;IACjC,IAAI,IAAI,IAAI,EAAE,CAAC,UAAU,CAAC,IAAI,CAAC,WAAW,CAAC,EAAE,CAAC;QAC5C,OAAO,GAAG,IAAA,sBAAY,EAAC,EAAE,CAAC,YAAY,CAAC,IAAI,CAAC,WAAW,EAAE,MAAM,CAAC,CAAC,CAAC;QAClE,MAAM,MAAM,GAAG,IAAA,yBAAe,EAAC,OAAO,CAAC,CAAC;QACxC,IAAI,MAAM,CAAC,MAAM;YAAE,MAAM,IAAI,KAAK,CAAC,GAAG,IAAI,CAAC,WAAW,QAAQ,MAAM,CAAC,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC;IACvF,CAAC;IACD,MAAM,MAAM,GAAG,IAAA,QAAA,GAAG,EAAC,KAAK,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,YAAY,CAAC,IAAA,QAAA,GAAG,EAAC,KAAK,CAAC,MAAM,CAAE,EAAE,MAAM,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC;IAC3F,MAAM,MAAM,GAAG,MAAM,IAAA,kBAAQ,EAAC;QAC5B,IAAI,EAAE,EAAE,EAAE,EAAE,IAAI,EAAE,EAAE,IAAI,MAAM,CAAC,IAAI,EAAE,KAAK,EAAE,IAAI,EAAE,KAAK,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE;QAC/E,OAAO;QACP,MAAM;QACN,GAAG,EAAE,IAAA,gBAAS,EAAU,OAAO,CAAC;QAChC,OAAO,EAAE,IAAI,CAAC,OAAO,CAAC,OAAO,CAAC;QAC9B,KAAK;QACL,MAAM,EAAE,MAAM,CAAC,MAAM,CAAC,MAAM,CAAC,QAAQ,EAAE,CAAC,SAAS,IAAI,EAAE,CAAC;QACxD,KAAK,EAAE,OAAO,IAAI,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,IAAA,iBAAS,EAAC,IAAA,QAAA,GAAG,EAAC,KAAK,CAAC,KAAK,CAAC,EAAE,IAAA,QAAA,GAAG,EAAC,KAAK,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,SAAS;KACnG,CAAC,CAAC;IACH,MAAM,GAAG,GAAG,IAAA,QAAA,GAAG,EAAC,KAAK,CAAC,GAAG,CAAC,IAAI,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,QAAQ,EAAE,MAAM,EAAE,GAAG,MAAM,CAAC,IAAI,IAAI,IAAI,IAAI,EAAE,CAAC,WAAW,EAAE,CAAC,OAAO,CAAC,OAAO,EAAE,GAAG,CAAC,EAAE,CAAC,CAAC;IACnI,MAAM,KAAK,GAAG,IAAA,kBAAQ,EAAC,MAAM,EAAE,GAAG,CAAC,CAAC;IACpC,IAAI,MAAM,CAAC,KAAK,KAAK,SAAS;QAAE,OAAO,CAAC,GAAG,CAAC,SAAS,CAAC,MAAM,CAAC,KAAK,GAAG,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,IAAI,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,eAAe,MAAM,CAAC,MAAM,CAAC,KAAK,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;IACtJ,KAAK,MAAM,CAAC,IAAI,MAAM,CAAC,KAAK;QAAE,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC;IAC7C,OAAO,CAAC,GAAG,CAAC,4BAA4B,MAAM,CAAC,KAAK,CAAC,SAAS,CAAC,MAAM,EAAE,CAAC,CAAC;IACzE,KAAK,MAAM,CAAC,IAAI,MAAM,CAAC,KAAK,CAAC,SAAS;QAAE,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC,MAAM,KAAK,MAAM,CAAC,CAAC,CAAC,cAAc,CAAC,CAAC,CAAC,iBAAiB,MAAM,CAAC,CAAC,KAAK,IAAI,CAAC,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,CAAC,CAAC;IAC5J,OAAO,CAAC,GAAG,CAAC,WAAW,KAAK,CAAC,IAAI,EAAE,CAAC,CAAC;IACrC,OAAO,CAAC,GAAG,CAAC,WAAW,KAAK,CAAC,IAAI,EAAE,CAAC,CAAC;IACrC,IAAI,OAAO,IAAI,MAAM,KAAK,SAAS,IAAI,MAAM,CAAC,KAAK,KAAK,SAAS;QAAE,OAAO,CAAC,QAAQ,GAAG,CAAC,CAAC;AAC1F,CAAC;AAED,oBAA2B,KAAkB;IAC3C,OAAO,CAAC,GAAG,CAAC,YAAY,KAAK,CAAC,QAAQ,EAAE,CAAC,CAAC;IAC1C,KAAK,MAAM,CAAC,GAAG,EAAE,CAAC,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,KAAK,CAAC,OAAO,CAAC,EAAE,CAAC;QACrD,IAAI,CAAC,CAAC,CAAC,KAAK;YAAE,SAAS;QACvB,OAAO,CAAC,GAAG,CAAC,KAAK,GAAG,CAAC,MAAM,CAAC,CAAC,CAAC,QAAQ,CAAC,CAAC,GAAG,IAAI,CAAC,CAAC,KAAK,KAAK,CAAC,CAAC,CAAC,CAAC,GAAG,GAAG,CAAC,CAAC,KAAK,CAAC,GAAG,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;IACvG,CAAC;IACD,OAAO,CAAC,GAAG,CAAC,aAAa,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC;IAC3D,MAAM,GAAG,GAAG,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;IACvF,IAAI,GAAG,CAAC,MAAM,EAAE,CAAC;QACf,OAAO,CAAC,GAAG,CAAC,oBAAoB,CAAC,CAAC;QAClC,KAAK,MAAM,CAAC,IAAI,GAAG;YAAE,OAAO,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC,aAAa,MAAM,CAAC,CAAC,KAAK,IAAI,CAAC,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,CAAC,CAAC;IAClG,CAAC;AACH,CAAC"}

package/dist/src/core/archive.d.ts

@@ -0,0 +1,49 @@
+/**
+ * Large files travel as gzip-compressed tar files: the WARC of a published recording (below) and
+ * the recording tool's hand-off archive. Git hosts reject large files (GitHub: 100 MB, with a
+ * warning from 50 MB), so archives above PART_BYTES are split into numbered parts:
+ * warc.tar.gz.000, warc.tar.gz.001, ... which are read back in order as one stream.
+ */
+export declare const PART_BYTES: number;
+/** Files of a bundle that can be replayed and published; everything else stays with the recorder. */
+export declare const PUBLIC_BUNDLE_FILES: string[];
+export declare const isBundleFile: (p: string) => boolean;
+/**
+ * Writes `files` (paths relative to `srcDir`, in this order) to `outFile`, a .tar.gz, replacing
+ * any earlier archive or parts there. Returns the files written: `outFile` or its parts.
+ */
+export declare function createArchive(srcDir: string, files: string[], outFile: string, partBytes?: number): Promise<string[]>;
+/** The archive at `file` as stored: the file itself, or its numbered parts in order. */
+export declare function archiveParts(file: string): string[];
+export declare function hashParts(parts: string[]): string;
+/**
+ * Extracts an archive (one file or its parts) into `destDir`. Only regular files and directories
+ * are extracted, and only paths accepted by `allow`; tar itself refuses absolute paths and "..".
+ */
+export declare function extractArchive(parts: string[], destDir: string, allow?: (p: string) => boolean): Promise<string[]>;
+/**
+ * Layout version of a recording published in a task. Replay reads only this version: a change to
+ * the layout or to the files in it bumps the number, and existing tasks are rewritten to match.
+ */
+export declare const RECORDING_FORMAT = 2;
+/**
+ * A bundle as published in a task (tasks/<id>/recording/):
+ *   manifest.json      the bundle's manifest plus "format"
+ *   pages.jsonl        pages/pages.jsonl
+ *   entries.jsonl.gz   indexes/entries.jsonl, gzip-compressed
+ *   warc.tar.gz        archive/data.warc (warc.tar.gz.000, .001, ... above PART_BYTES)
+ * The WARC sits apart from the files whose layout may change, so a new format rewrites only the
+ * small files and leaves the large archive as it was.
+ */
+export declare const RECORDING_FILES: {
+    readonly manifest: 'manifest.json';
+    readonly pages: 'pages.jsonl';
+    readonly entries: 'entries.jsonl.gz';
+    readonly warc: 'warc.tar.gz';
+};
+/** Writes the bundle at `bundleDir` to `outDir` in the published layout, replacing what is there. */
+export declare function publishRecording(bundleDir: string, outDir: string): Promise<string[]>;
+/** Files of the recording published in `dir`, in a fixed order; empty when there is none. */
+export declare function recordingFiles(dir: string): string[];
+/** Turns the recording published in `dir` back into a bundle at `destDir`; refuses other formats. */
+export declare function unpackRecording(dir: string, destDir: string): Promise<void>;