@parcha/agentrun-pi 0.1.0-beta.1 → 0.1.0-beta.2

package/README.md

@@ -2,24 +2,24 @@
 
 Describe a task in Pi. AgentRun turns it into a workflow you can inspect and runs its steps using your current Pi model. Requires Node 22.19+ and Pi 0.87.0.
 
-## Install from source
+## Install
 
-Build the extension and use the included Pi CLI. From a new terminal:
+If you do not have Pi, install the tested host version first:
 
 ```sh
-git clone --branch main --single-branch https://github.com/Parcha-ai/agentrun.git
-cd agentrun
-npm ci --ignore-scripts
-npm run build
-./node_modules/.bin/pi install ./packages/pi -l
-./node_modules/.bin/pi --offline
+npm install -g @earendil-works/pi-coding-agent@0.87.0
 ```
 
-`-l` writes project settings in `.pi/settings.json`. Omit it to install for all your Pi sessions. `--offline` skips startup downloads; it does not disable model calls.
+Then, in your project:
 
-On first launch, Pi asks whether you trust the project before loading its extension. Review the checkout before accepting.
+```sh
+pi install npm:@parcha/agentrun-pi@0.1.0-beta.2 -l
+pi --offline
+```
+
+`-l` writes project settings in `.pi/settings.json`. Omit it to install for all your Pi sessions. `--offline` skips startup downloads; it does not disable model calls. No AgentRun checkout or build is needed.
 
-Already built this checkout? Start at the `pi install` line. The included CLI is the tested Pi 0.87.0 version; a global Pi installation is unnecessary. If Pi 0.87.0 is already open, use `/reload`. Keep this checkout: Pi loads the extension and skill from it. Rebuild after source changes, then reload. `/reload` clears the current workflow definition; run the demo or describe your task again to create one.
+On first launch, Pi asks whether you trust the project before loading its extension. If Pi is already open, use `/reload`. Reloading clears the current workflow definition; run the demo or describe your task again to create one.
 
 For live tasks, configure model access in Pi first. AgentRun uses Pi's current model and authentication; it needs no separate model configuration. The scripted demo below works without provider access.
 
@@ -31,7 +31,7 @@ Try the bundled example inside Pi:
 
 The demo shows the workflow graph, progress, and a completed result with call counts. It uses scripted responses over fictional sources. It makes no model calls and does not test model quality. Run `/agentrun status` to check the loaded skill and Pi host version. The tested version is 0.87.0; status flags a different host without blocking it. Missing Pi or Jev access does not prevent the scripted demo. To repeat it, run `/agentrun run` or `/agentrun demo`. A scripted demo stays scripted on rerun, including its missing-evidence path. Use `/agentrun demo live` to switch to real Pi and Jev calls. Workflows authored by Pi use real adapters.
 
-For an offline Pi startup, use `./node_modules/.bin/pi --offline`. Without that flag, Pi may download optional command-line tools on first launch. This is separate from the scripted demo, which makes no network calls.
+For an offline Pi startup, use `pi --offline`. Without that flag, Pi may download optional command-line tools on first launch. This is separate from the scripted demo, which makes no network calls.
 
 ## Describe, inspect, run
 
@@ -64,7 +64,7 @@ One workflow runs at a time. Definitions and run state belong to the current Pi
 | `/agentrun demo empty` | Scripted insufficient-evidence path, without model calls. |
 | `/agentrun demo live` | Real Pi agent steps and Jev system one decisions over the same fictional sources. |
 
-Live mode uses your active Pi model and requires configured TypeSafe access for Jev. It still does not search the web or prove accuracy on real research. See the [Jev adapter configuration](../jev/README.md). Live calls can incur usage charges; scripted mode needs no provider access.
+Live mode uses your active Pi model and requires configured TypeSafe access for Jev. It still does not search the web or prove accuracy on real research. See the [Jev adapter configuration](https://github.com/Parcha-ai/agentrun/tree/main/packages/jev#readme). Live calls can incur usage charges; scripted mode needs no provider access.
 
 ## Tools and execution
 
@@ -196,6 +196,21 @@ With `rubricSections`, every supplied section must appear on every generated LLM
 
 The standalone CLI remains available as `node packages/pi/dist/cli.js --help` from source. It imports a trusted caller-owned `--config` module for `author` and `run`; importing that module executes its code. `validate` can evaluate code probes. Review executable workflows before using it.
 
+## Develop from source
+
+To change the extension itself, clone and build the repository:
+
+```sh
+git clone --branch main --single-branch https://github.com/Parcha-ai/agentrun.git
+cd agentrun
+npm ci --ignore-scripts
+npm run build
+./node_modules/.bin/pi install ./packages/pi -l
+./node_modules/.bin/pi --offline
+```
+
+Use a separate project for this source installation. Pi loads the extension and skill from the checkout, so keep it in place. After source changes, rebuild and run `/reload`.
+
 ## Dependencies and retained data
 
 Pi is optional for `@parcha/agentrun-dsl`. This package brings the Pi coding-agent SDK, its provider dependencies, and the Jev adapter; its dependency footprint is larger than the core. The release inventory records the locked versions.

package/dist/benchmark-contract.d.ts

@@ -0,0 +1,88 @@
+export type BenchmarkSplit = 'development' | 'heldout';
+export type BenchmarkFinding = {
+    topicId: string;
+    status: 'supported' | 'conflicting' | 'insufficient';
+    summary: string;
+    sourceIds: string[];
+};
+export type BenchmarkCase = {
+    id: string;
+    split: BenchmarkSplit;
+    question: string;
+    topics: {
+        id: string;
+        question: string;
+    }[];
+    sources: {
+        id: string;
+        text: string;
+    }[];
+    expected: Omit<BenchmarkFinding, 'summary'>[];
+};
+export declare const benchmarkHash: (value: unknown) => string;
+export declare const benchmarkInstructions = "Answer using only the fictional source corpus. Produce exactly one finding for each supplied topic. Status supported means concrete evidence establishes a finding, including a negative or qualified answer. Conflicting means incompatible accounts. Insufficient means no concrete information answers the question. Preserve conditions and contrary evidence; cite every material source by ID. Do not invent facts or citations. Submit the final structured report.";
+export declare const benchmarkReportSchema: {
+    type: string;
+    additionalProperties: boolean;
+    required: string[];
+    properties: {
+        findings: {
+            type: string;
+            minItems: number;
+            items: {
+                type: string;
+                additionalProperties: boolean;
+                required: string[];
+                properties: {
+                    topicId: {
+                        type: string;
+                        minLength: number;
+                    };
+                    status: {
+                        enum: string[];
+                    };
+                    summary: {
+                        type: string;
+                        minLength: number;
+                    };
+                    sourceIds: {
+                        type: string;
+                        uniqueItems: boolean;
+                        items: {
+                            type: string;
+                            minLength: number;
+                        };
+                    };
+                };
+            };
+        };
+    };
+};
+export declare const benchmarkEvidenceSchema: {
+    type: string;
+    additionalProperties: boolean;
+    required: string[];
+    properties: {
+        relevant: {
+            type: string;
+            description: string;
+            criteria: {
+                true: string;
+                false: string;
+            };
+        };
+    };
+};
+export declare function benchmarkCases(split: BenchmarkSplit): BenchmarkCase[];
+/** Frozen independent checks. Never return these results to an authoring agent. */
+export declare function evaluatePiBenchmark(caseId: string, output: unknown): {
+    passed: boolean;
+    issues: string[];
+};
+export declare function benchmarkContract(): {
+    version: number;
+    fixturesSha256: string;
+    verifierSha256: string;
+    outputSchemaSha256: string;
+    instructionsSha256: string;
+};

package/dist/benchmark-contract.js

@@ -0,0 +1,50 @@
+import { createHash } from 'node:crypto';
+import { readFileSync } from 'node:fs';
+import { Compile } from 'typebox/compile';
+const cases = [
+    { id: 'docs-editing', split: 'development', question: 'Can contributors edit our docs in the browser?', topics: [{ id: 'editing', question: 'Did contributors edit a page in the browser during our trial?' }], sources: [{ id: 'd1', text: 'All three contributors edited a page in the browser during the trial.' }, { id: 'd2', text: 'The documentation product has a new logo.' }], expected: [{ topicId: 'editing', status: 'supported', sourceIds: ['d1'] }] },
+    { id: 'migration', split: 'development', question: 'What changes with a repository-based wiki?', topics: [{ id: 'review', question: 'How are changes reviewed?' }, { id: 'editing', question: 'Could non-developers edit documentation?' }, { id: 'search', question: 'What did the search trial find?' }], sources: [{ id: 'm1', text: 'Documentation changes require an assigned reviewer to approve the pull request.' }, { id: 'm2', text: 'Non-developers edited in the browser but two needed help with merge conflicts.' }, { id: 'm3', text: 'Search found exact words but missed synonyms during our trial.' }, { id: 'm4', text: 'A revolutionary documentation experience.' }], expected: [{ topicId: 'review', status: 'supported', sourceIds: ['m1'] }, { topicId: 'editing', status: 'supported', sourceIds: ['m2'] }, { topicId: 'search', status: 'supported', sourceIds: ['m3'] }] },
+    { id: 'missing-conflicting', split: 'development', question: 'Can we rely on restricted editing and synonym search?', topics: [{ id: 'permissions', question: 'Who can edit restricted pages?' }, { id: 'search', question: 'Did synonym search work in this trial?' }], sources: [{ id: 'c1', text: 'Restricted-page permissions: details pending.' }, { id: 'c2', text: 'The trial report says all ten synonym searches found their target pages.' }, { id: 'c3', text: 'An audit of the same trial says none of those ten searches found their target pages.' }], expected: [{ topicId: 'permissions', status: 'insufficient', sourceIds: [] }, { topicId: 'search', status: 'conflicting', sourceIds: ['c2', 'c3'] }] },
+    { id: 'backup', split: 'heldout', question: 'Did the restore drill recover the archive?', topics: [{ id: 'restore', question: 'Was the archive recovered during the drill?' }], sources: [{ id: 'h1', text: 'The drill restored the archive and every file matched its recorded checksum.' }, { id: 'h2', text: 'Backups that give you peace of mind.' }], expected: [{ topicId: 'restore', status: 'supported', sourceIds: ['h1'] }] },
+    { id: 'queue-adoption', split: 'heldout', question: 'What should we know before adopting this task queue?', topics: [{ id: 'retries', question: 'What happens to jobs after worker failure?' }, { id: 'ordering', question: 'Is task order guaranteed?' }, { id: 'visibility', question: 'Can operators inspect stuck jobs?' }], sources: [{ id: 'q1', text: 'In the failure test, an unfinished job returned to the queue when its lease expired.' }, { id: 'q2', text: 'Tasks from different workers completed out of submission order in the trial.' }, { id: 'q3', text: 'Operators used the dashboard to inspect job age and the last worker error.' }, { id: 'q4', text: 'Supercharge your asynchronous future.' }], expected: [{ topicId: 'retries', status: 'supported', sourceIds: ['q1'] }, { topicId: 'ordering', status: 'supported', sourceIds: ['q2'] }, { topicId: 'visibility', status: 'supported', sourceIds: ['q3'] }] },
+    { id: 'cache-evidence', split: 'heldout', question: 'What do the cache trials tell us about deletion and encryption?', topics: [{ id: 'deletion', question: 'Did deleted entries disappear immediately in the trial?' }, { id: 'encryption', question: 'Which encryption protects cached entries?' }], sources: [{ id: 'k1', text: 'The trial log says every deleted entry disappeared immediately.' }, { id: 'k2', text: 'A second observer of that same trial recorded deleted entries remaining readable for a full minute.' }, { id: 'k3', text: 'Encryption: documentation forthcoming.' }], expected: [{ topicId: 'deletion', status: 'conflicting', sourceIds: ['k1', 'k2'] }, { topicId: 'encryption', status: 'insufficient', sourceIds: [] }] },
+];
+export const benchmarkHash = (value) => createHash('sha256').update(JSON.stringify(value)).digest('hex');
+export const benchmarkInstructions = 'Answer using only the fictional source corpus. Produce exactly one finding for each supplied topic. Status supported means concrete evidence establishes a finding, including a negative or qualified answer. Conflicting means incompatible accounts. Insufficient means no concrete information answers the question. Preserve conditions and contrary evidence; cite every material source by ID. Do not invent facts or citations. Submit the final structured report.';
+const str = { type: 'string', minLength: 1 };
+export const benchmarkReportSchema = { type: 'object', additionalProperties: false, required: ['findings'], properties: { findings: { type: 'array', minItems: 1, items: { type: 'object', additionalProperties: false, required: ['topicId', 'status', 'summary', 'sourceIds'], properties: { topicId: str, status: { enum: ['supported', 'conflicting', 'insufficient'] }, summary: str, sourceIds: { type: 'array', uniqueItems: true, items: str } } } } } };
+export const benchmarkEvidenceSchema = { type: 'object', additionalProperties: false, required: ['relevant'], properties: { relevant: { type: 'boolean', description: 'Does the passage contain a concrete fact, condition, limitation or contrary evidence bearing on the question?', criteria: { true: 'Concrete evidence relevant to the question, including qualified or contradictory answers.', false: 'Only topic mentions, marketing promises, missing details, or unrelated information.' } } } };
+const validator = Compile(benchmarkReportSchema);
+export function benchmarkCases(split) {
+    if (split !== 'development' && split !== 'heldout')
+        throw new Error('Choose development or heldout explicitly');
+    return structuredClone(cases.filter(row => row.split === split));
+}
+/** Frozen independent checks. Never return these results to an authoring agent. */
+export function evaluatePiBenchmark(caseId, output) {
+    const item = cases.find(row => row.id === caseId);
+    if (!item)
+        throw new Error('Unknown benchmark case');
+    if (!validator.Check(output))
+        return { passed: false, issues: ['schema'] };
+    const findings = output.findings;
+    const issues = [];
+    if (findings.length !== item.expected.length || new Set(findings.map(row => row.topicId)).size !== findings.length)
+        issues.push('coverage');
+    for (const expected of item.expected) {
+        const actual = findings.find(row => row.topicId === expected.topicId);
+        if (actual?.status !== expected.status)
+            issues.push(`${expected.topicId}:status`);
+        if (!actual || JSON.stringify([...actual.sourceIds].sort()) !== JSON.stringify([...expected.sourceIds].sort()))
+            issues.push(`${expected.topicId}:citations`);
+    }
+    return { passed: issues.length === 0, issues };
+}
+export function benchmarkContract() {
+    return {
+        version: 1, fixturesSha256: benchmarkHash(cases),
+        verifierSha256: createHash('sha256').update(readFileSync(new URL('./benchmark-contract.js', import.meta.url))).digest('hex'),
+        outputSchemaSha256: benchmarkHash(benchmarkReportSchema),
+        instructionsSha256: benchmarkHash(benchmarkInstructions),
+    };
+}

package/dist/benchmark-freeze.d.ts

@@ -0,0 +1,9 @@
+/** Frozen before any live sample. Changing these pins requires a new benchmark
+ * version and a fresh baseline; never update them to improve a skill's score. */
+export declare const frozenBenchmarkContract: {
+    readonly version: 1;
+    readonly fixturesSha256: "1a8f9ff022d5cdaf2455e327e59ae4593aeffcc8ae4c510a870b36ea0d9e7de2";
+    readonly verifierSha256: "f21d0c8b790186f571003907a5a6c7a6360423d77d1125c4398994cb1ae11f9a";
+    readonly outputSchemaSha256: "38a0d7ccd8590ad8a486a5b7b7edd36c0b8bd32e642ead5345a3f93d12f3e574";
+    readonly instructionsSha256: "d8d3fc6721ed99bb163ff3dbd93dffc15a1a92ab400f11f781bb25e9f8a61535";
+};

package/dist/benchmark-freeze.js

@@ -0,0 +1,9 @@
+/** Frozen before any live sample. Changing these pins requires a new benchmark
+ * version and a fresh baseline; never update them to improve a skill's score. */
+export const frozenBenchmarkContract = {
+    version: 1,
+    fixturesSha256: '1a8f9ff022d5cdaf2455e327e59ae4593aeffcc8ae4c510a870b36ea0d9e7de2',
+    verifierSha256: 'f21d0c8b790186f571003907a5a6c7a6360423d77d1125c4398994cb1ae11f9a',
+    outputSchemaSha256: '38a0d7ccd8590ad8a486a5b7b7edd36c0b8bd32e642ead5345a3f93d12f3e574',
+    instructionsSha256: 'd8d3fc6721ed99bb163ff3dbd93dffc15a1a92ab400f11f781bb25e9f8a61535',
+};

package/dist/benchmark.d.ts

@@ -0,0 +1,127 @@
+import { type WorkflowDeps } from '@parcha/agentrun-dsl';
+import type { PiHostContext } from './types.js';
+import { evaluatePiBenchmark, type BenchmarkSplit } from './benchmark-contract.js';
+export interface PiBenchmarkBudgets {
+    timeoutMs: number;
+    maxModelRequests: number;
+    maxToolCalls: number;
+    maxTurns: number;
+    maxSubmissions: number;
+}
+export interface PiSkillBenchmarkOptions {
+    host: PiHostContext;
+    runJudge: NonNullable<WorkflowDeps['runJudge']>;
+    /** Exact native skill plus its loaded references, supplied by the caller. No hidden authoring contract is added. */
+    skillText: string;
+    split: BenchmarkSplit;
+    mode: 'live' | 'scripted-transport';
+    signal?: AbortSignal;
+    budgets?: Partial<PiBenchmarkBudgets>;
+    onProgress?: (event: {
+        caseId: string;
+        arm: 'plain-pi' | 'skill-dsl';
+        stage: 'arm' | 'outer' | 'workflow';
+        event: 'start' | 'end';
+        status?: string;
+    }) => unknown;
+}
+type Metrics = {
+    pi: number;
+    jev: number;
+    tools: number;
+    submissions: number;
+    tokens: number;
+    usageSamples: number;
+    knownCostUsd: number;
+    costSamples: number;
+};
+type Arm = 'plain-pi' | 'skill-dsl';
+export interface PiBenchmarkArm {
+    arm: Arm;
+    status: 'passed' | 'rejected' | 'failed' | 'cancelled';
+    elapsedMs: number;
+    total: Metrics;
+    stages: {
+        outer: Metrics;
+        workflow: Metrics;
+    };
+    costComplete: boolean;
+    usageComplete: boolean;
+    candidates: {
+        digest?: string;
+        accepted: boolean;
+        workflow?: unknown;
+    }[];
+    workflowRuns: {
+        status: string;
+        digest: string;
+        output?: unknown;
+    }[];
+    output?: unknown;
+    evaluation?: ReturnType<typeof evaluatePiBenchmark>;
+    failure?: string;
+}
+/** End-to-end comparison: DSL authoring, repair, execution and delivery all count.
+ * Uses the selected host model, never a model selector or alternate provider. */
+export declare function runPiSkillBenchmark(options: PiSkillBenchmarkOptions): Promise<{
+    mode: "live" | "scripted-transport";
+    split: BenchmarkSplit;
+    status: "complete" | "cancelled";
+    passed: boolean;
+    contract: {
+        version: number;
+        fixturesSha256: string;
+        verifierSha256: string;
+        outputSchemaSha256: string;
+        instructionsSha256: string;
+    };
+    skillSha256: string;
+    budgets: {
+        timeoutMs: number;
+        maxModelRequests: number;
+        maxToolCalls: number;
+        maxTurns: number;
+        maxSubmissions: number;
+    };
+    runtime: {
+        model: string;
+        provider: string;
+        api: string;
+        thinking: import("./types.js").PiThinkingLevel;
+    };
+    runs: {
+        caseId: string;
+        inputSha256: string;
+        order: Arm[];
+        arms: PiBenchmarkArm[];
+    }[];
+    limitations: string[];
+}>;
+export type PiSkillBenchmarkReport = Awaited<ReturnType<typeof runPiSkillBenchmark>>;
+/** A development comparison receipt, not automatic promotion. Heldout is a final
+ * confirmation set, never the optimization target for this helper. */
+export declare function comparePiSkillBenchmarks(baseline: PiSkillBenchmarkReport, candidate: PiSkillBenchmarkReport): {
+    mode: "live" | "scripted-transport";
+    baselineSkillSha256: string;
+    candidateSkillSha256: string;
+    regressions: string[];
+    previous: {
+        passed: number;
+        requests: number;
+        tokens: number;
+        usageComplete: boolean;
+        knownCostUsd: number;
+        costComplete: boolean;
+    };
+    current: {
+        passed: number;
+        requests: number;
+        tokens: number;
+        usageComplete: boolean;
+        knownCostUsd: number;
+        costComplete: boolean;
+    };
+    improved: boolean;
+    eligibleForLiveHeldout: boolean;
+};
+export {};

package/dist/benchmark.js

@@ -0,0 +1,260 @@
+import { isDeepStrictEqual } from 'node:util';
+import { frozenBenchmarkContract } from './benchmark-freeze.js';
+import { Compile } from 'typebox/compile';
+import { formatWorkflowTree, runWorkflowSlice } from '@parcha/agentrun-dsl';
+import { createPiHostSessionFactory } from './host-session.js';
+import { createPiRunner, PiRunError } from './runner.js';
+import { WorkflowExtensionService, ExtensionServiceError } from './extension-service.js';
+import { benchmarkCases, benchmarkContract, benchmarkHash, benchmarkInstructions, benchmarkReportSchema, benchmarkEvidenceSchema, evaluatePiBenchmark } from './benchmark-contract.js';
+const emptyMetrics = () => ({ pi: 0, jev: 0, tools: 0, submissions: 0, tokens: 0, usageSamples: 0, knownCostUsd: 0, costSamples: 0 });
+const ceilings = { timeoutMs: 600_000, maxModelRequests: 24, maxToolCalls: 48, maxTurns: 24, maxSubmissions: 4 };
+class BenchmarkStop extends Error {
+    code;
+    constructor(code) {
+        super(code);
+        this.code = code;
+    }
+}
+const textResult = (value) => ({ content: [{ type: 'text', text: JSON.stringify(value) }], details: value });
+const selectionWorkflow = { v: 2, name: 'benchmark-evidence', schemas: { Evidence: benchmarkEvidenceSchema, Result: { type: 'object' } }, output: { schemaId: 'Result', path: 'evidence' }, root: { node: 'sift', label: 'screen', itemsPath: 'sources', state: { question: '{question}' }, out: 'Evidence', as: 'evidence', keep: { path: 'relevant', gte: 0.8 } } };
+/** End-to-end comparison: DSL authoring, repair, execution and delivery all count.
+ * Uses the selected host model, never a model selector or alternate provider. */
+export async function runPiSkillBenchmark(options) {
+    if (options.mode !== 'live' && options.mode !== 'scripted-transport')
+        throw new Error('Choose live or scripted-transport explicitly');
+    if (typeof options.skillText !== 'string' || !options.skillText.trim() || Buffer.byteLength(options.skillText) > 128 * 1024)
+        throw new Error('Provide the exact native skill bundle, at most 128 KiB');
+    if (typeof options.runJudge !== 'function')
+        throw new Error('Both arms require the same Jev capability');
+    const suite = benchmarkCases(options.split);
+    const contract = benchmarkContract();
+    if (!isDeepStrictEqual(contract, frozenBenchmarkContract))
+        throw new Error('Frozen benchmark contract changed; establish a new version and baseline');
+    const skillText = options.skillText;
+    const budgets = { ...ceilings, ...options.budgets };
+    for (const key of Object.keys(ceilings))
+        if (!Number.isSafeInteger(budgets[key]) || budgets[key] < 1 || budgets[key] > ceilings[key])
+            throw new Error(`Invalid benchmark budget: ${key}`);
+    const captured = { ...options.host };
+    createPiHostSessionFactory(captured);
+    const model = captured.model;
+    const runs = [];
+    for (const [index, item] of suite.entries()) {
+        if (options.signal?.aborted)
+            break;
+        const order = index % 2 ? ['skill-dsl', 'plain-pi'] : ['plain-pi', 'skill-dsl'];
+        const arms = [];
+        for (const arm of order) {
+            if (options.signal?.aborted)
+                break;
+            arms.push(await runArm(item, arm));
+        }
+        runs.push({ caseId: item.id, inputSha256: benchmarkHash({ question: item.question, topics: item.topics, sources: item.sources }), order: arms.map(row => row.arm), arms });
+    }
+    if (!isDeepStrictEqual(contract, benchmarkContract()))
+        throw new Error('Benchmark verifier changed during execution');
+    return {
+        mode: options.mode, split: options.split, status: options.signal?.aborted ? 'cancelled' : 'complete',
+        passed: runs.length === suite.length && runs.every(row => row.arms.length === 2 && row.arms.every(arm => arm.status === 'passed')),
+        contract, skillSha256: benchmarkHash(skillText), budgets,
+        runtime: { model: model.id, provider: model.provider, api: model.api, thinking: captured.thinkingLevel ?? 'low' }, runs,
+        limitations: ['Three fictional cases per split are not a statistical or real-world benchmark.', 'Gold checks cover statuses and citation sets; summary entailment remains unscored.', 'All outer-agent authoring, repair, final delivery, child execution and failed attempts count; initial package/skill development does not.', 'Request caps count adapter invocations, not provider-internal HTTP retries. There is no token or spending cap.', 'Known cost uses returned provider/SDK accounting only; missing samples are unknown, including failed requests.', 'Host caches are not reset; execution order alternates. Heldout cases must not be used to tune the skill.', 'The DSL arm uses the native bridge and service with a constrained benchmark tool surface; this is not a test of interactive skill discovery.'],
+    };
+    async function runArm(item, arm) {
+        const started = performance.now();
+        const notify = (stage, event, status) => {
+            try {
+                void Promise.resolve(options.onProgress?.({ caseId: item.id, arm, stage, event, ...(status ? { status } : {}) })).catch(() => { });
+            }
+            catch { /* UI hooks cannot change the measurement. */ }
+        };
+        notify('arm', 'start');
+        notify('outer', 'start');
+        const controller = new AbortController();
+        const signal = controller.signal;
+        const stop = (code) => { if (!signal.aborted)
+            controller.abort(new BenchmarkStop(code)); };
+        const externalAbort = () => stop('cancelled');
+        options.signal?.addEventListener('abort', externalAbort, { once: true });
+        if (options.signal?.aborted)
+            externalAbort();
+        const timer = setTimeout(() => stop('deadline'), budgets.timeoutMs);
+        const interrupted = new Promise((_resolve, reject) => {
+            signal.addEventListener('abort', () => reject(signal.reason), { once: true });
+            if (signal.aborted)
+                reject(signal.reason);
+        });
+        void interrupted.catch(() => { });
+        const bounded = (promise) => Promise.race([promise, interrupted]);
+        let stage = 'outer';
+        const stages = { outer: emptyMetrics(), workflow: emptyMetrics() };
+        const candidates = [], workflowRuns = [];
+        const total = () => Object.fromEntries(Object.keys(stages.outer).map(key => [key, stages.outer[key] + stages.workflow[key]]));
+        const admit = (kind) => {
+            signal.throwIfAborted();
+            const counts = total();
+            if (kind === 'tools' ? counts.tools >= budgets.maxToolCalls : counts.pi + counts.jev >= budgets.maxModelRequests)
+                stop(kind === 'tools' ? 'tool_limit' : 'model_limit');
+            signal.throwIfAborted();
+            stages[stage][kind]++;
+        };
+        const usage = (bucket, tokens, cost) => {
+            if (typeof tokens === 'number' && Number.isFinite(tokens) && tokens >= 0) {
+                bucket.tokens += tokens;
+                bucket.usageSamples++;
+            }
+            if (typeof cost === 'number' && Number.isFinite(cost) && cost >= 0) {
+                bucket.knownCostUsd += cost;
+                bucket.costSamples++;
+            }
+        };
+        const host = { ...captured, modelRegistry: {
+                getAll: () => captured.modelRegistry.getAll(),
+                streamSimple(selected, transcript, opts) {
+                    admit('pi');
+                    const bucket = stages[stage];
+                    const stream = captured.modelRegistry.streamSimple(selected, transcript, opts);
+                    void stream.result().then(result => {
+                        const value = result;
+                        usage(bucket, value.usage?.totalTokens, value.usage?.cost?.total);
+                    }).catch(() => { });
+                    return stream;
+                },
+            } };
+        const runJudge = async (request) => {
+            request.signal?.throwIfAborted();
+            admit('jev');
+            const bucket = stages[stage];
+            const result = await bounded(options.runJudge({ ...request, signal: request.signal ? AbortSignal.any([signal, request.signal]) : signal }));
+            usage(bucket, result.usage ? result.usage.input_tokens + result.usage.output_tokens : undefined, result.cost_usd);
+            return result;
+        };
+        const domainTools = [
+            { name: 'read_sources', label: 'Read sources', description: 'Return {sources:[{id,text}]} for this fictional task. Direct workflow calls return this same structured value. This is not web search.', parameters: { type: 'object', properties: {}, additionalProperties: false }, async execute() { return textResult({ sources: structuredClone(item.sources) }); } },
+            { name: 'screen_evidence', label: 'Screen evidence', description: 'Jev System One decisions select concrete evidence including limitations and contradictions at threshold 0.8.', parameters: { type: 'object', additionalProperties: false, required: ['topicId', 'sourceIds'], properties: { topicId: { type: 'string' }, sourceIds: { type: 'array', uniqueItems: true, items: { type: 'string' } } } },
+                async execute(_id, value, toolSignal) {
+                    const args = value;
+                    const topic = item.topics.find(row => row.id === args.topicId);
+                    if (!topic || args.sourceIds.some(id => !item.sources.some(source => source.id === id)))
+                        throw new Error('Unknown topic or source');
+                    const result = await bounded(runWorkflowSlice(selectionWorkflow, { question: topic.question, sources: item.sources.filter(source => args.sourceIds.includes(source.id)) }, { from: 'screen' }, { runJudge: request => runJudge({ ...request, signal: toolSignal }) }));
+                    if (result.status !== 'complete')
+                        throw new Error('Screening did not complete');
+                    return textResult(result.state.evidence);
+                } },
+        ];
+        const service = new WorkflowExtensionService({ allowedTools: domainTools.map(tool => tool.name), limits: { deadlineMs: budgets.timeoutMs } });
+        const input = { question: item.question, topics: structuredClone(item.topics) };
+        let lastOutput;
+        let workflowInput = input;
+        try {
+            const factory = createPiHostSessionFactory(host);
+            const makeRunner = (tools) => createPiRunner({ model, thinkingLevel: captured.thinkingLevel, cwd: captured.cwd, signal, tools,
+                maxTurns: budgets.maxTurns, maxSubmissions: budgets.maxSubmissions, timeoutMs: budgets.timeoutMs,
+                sessionFactory: opts => factory({ ...opts, customTools: opts.customTools.map(tool => ({ ...tool, async execute(...args) {
+                            args[2]?.throwIfAborted();
+                            admit('tools');
+                            if (tool.name === 'submit')
+                                stages[stage].submissions++;
+                            const validator = Compile(tool.parameters);
+                            if (!validator.Check(args[1]))
+                                throw new Error('Tool input does not match its schema');
+                            return tool.execute(...args);
+                        } })) }),
+            });
+            const child = makeRunner(domainTools);
+            const agentrun = { name: 'agentrun', label: 'AgentRun workflow', description: 'Describe available tools, inspect a workflow, then run the inspected workflow. Never run an inline candidate. No shell, code, file access or model selection.', parameters: { type: 'object', additionalProperties: false, required: ['action'], properties: { action: { enum: ['describe', 'inspect', 'run'] }, workflow: {}, input: { type: 'object' } } }, async execute(_id, value) {
+                    const args = value;
+                    if (args.action === 'inspect' && args.input !== undefined)
+                        workflowInput = args.input;
+                    if (args.action === 'describe')
+                        return textResult({ tools: domainTools.map(tool => ({ name: tool.name, description: tool.description, parameters: tool.parameters })), skill: true, pi: true, jev: true, running: false, workflow: !!service.inspect().current });
+                    if (args.action === 'inspect') {
+                        try {
+                            if (args.workflow === undefined) {
+                                const current = service.inspect().current;
+                                if (!current)
+                                    throw new Error('Inspect a workflow first');
+                                return textResult({ digest: current.digest, tree: formatWorkflowTree(current.inspection) });
+                            }
+                            const prepared = service.prepare(args.workflow);
+                            candidates.push({ digest: prepared.digest, accepted: true, workflow: prepared.workflow });
+                            return textResult({ digest: prepared.digest, tree: formatWorkflowTree(prepared.inspection) });
+                        }
+                        catch (error) {
+                            candidates.push({ accepted: false, workflow: structuredClone(args.workflow) });
+                            throw error;
+                        }
+                    }
+                    if (args.action !== 'run' || args.workflow !== undefined)
+                        throw new Error('Inspect the workflow first; run without a workflow argument');
+                    notify('outer', 'end');
+                    stage = 'workflow';
+                    notify('workflow', 'start');
+                    try {
+                        const report = await bounded(service.run(args.input ?? workflowInput, { signal, deps: {
+                                // A candidate cannot change the captured host thinking setting.
+                                runNode: request => child({ ...request, thinking: undefined }), runJudge,
+                                runEffect: async (request) => {
+                                    request.signal.throwIfAborted();
+                                    admit('tools');
+                                    const tool = domainTools.find(tool => tool.name === request.node.tool);
+                                    if (!tool || !Compile(tool.parameters).Check(request.input))
+                                        throw new Error('Unsupported tool input');
+                                    return (await bounded(tool.execute('benchmark-effect', request.input, request.signal, undefined, host))).details;
+                                },
+                            } }));
+                        workflowRuns.push({ status: report.status, digest: report.digest, ...(report.output === undefined ? {} : { output: report.output }) });
+                        lastOutput = report.status === 'complete' ? report.output : undefined;
+                        return textResult(report);
+                    }
+                    finally {
+                        notify('workflow', 'end');
+                        stage = 'outer';
+                        notify('outer', 'start');
+                    }
+                } };
+            const outer = makeRunner(arm === 'skill-dsl' ? [...domainTools, agentrun] : domainTools);
+            const output = await bounded(outer({ kind: 'agent', label: 'benchmark-outer',
+                system: [benchmarkInstructions, ...(arm === 'skill-dsl' ? [skillText, 'Use agentrun to inspect and execute your workflow. Deliver its completed output unchanged with submit. The benchmark forbids code, shell, files and changing models. Available task tools and their contracts are returned by agentrun describe.'] : [])],
+                user: JSON.stringify(input), schema: benchmarkReportSchema }));
+            signal.throwIfAborted();
+            if (arm === 'skill-dsl' && (lastOutput === undefined || !isDeepStrictEqual(lastOutput, output)))
+                return finish({ status: 'rejected', output, failure: 'workflow_delivery_missing_or_changed' });
+            const evaluation = evaluatePiBenchmark(item.id, output);
+            return finish({ status: evaluation.passed ? 'passed' : 'rejected', output, evaluation });
+        }
+        catch (error) {
+            return finish({ status: options.signal?.aborted ? 'cancelled' : 'failed', failure: signal.reason instanceof BenchmarkStop ? signal.reason.code : error instanceof PiRunError ? `pi_${error.reason}` : error instanceof ExtensionServiceError ? error.code : 'execution_failed' });
+        }
+        finally {
+            clearTimeout(timer);
+            options.signal?.removeEventListener('abort', externalAbort);
+            stop('arm_finished');
+            void service.dispose();
+        }
+        function finish(result) {
+            const counts = total();
+            notify('outer', 'end', result.status);
+            notify('arm', 'end', result.status);
+            return structuredClone({ arm, ...result, elapsedMs: Math.round(performance.now() - started), total: counts, stages, candidates, workflowRuns, costComplete: counts.costSamples === counts.pi + counts.jev, usageComplete: counts.usageSamples === counts.pi + counts.jev });
+        }
+    }
+}
+/** A development comparison receipt, not automatic promotion. Heldout is a final
+ * confirmation set, never the optimization target for this helper. */
+export function comparePiSkillBenchmarks(baseline, candidate) {
+    if (baseline.split !== 'development' || candidate.split !== 'development')
+        throw new Error('Hillclimb comparisons use development only');
+    if (baseline.status !== 'complete' || candidate.status !== 'complete' || baseline.mode !== candidate.mode || !isDeepStrictEqual(baseline.contract, candidate.contract) || !isDeepStrictEqual(baseline.budgets, candidate.budgets) || !isDeepStrictEqual(baseline.runtime, candidate.runtime))
+        throw new Error('Incomparable benchmark receipts');
+    const select = (report) => report.runs.map(run => ({ id: run.caseId, input: run.inputSha256, arm: run.arms.find(arm => arm.arm === 'skill-dsl') }));
+    const before = select(baseline), after = select(candidate);
+    if (before.length !== 3 || after.length !== 3 || !isDeepStrictEqual(before.map(row => [row.id, row.input]), after.map(row => [row.id, row.input])) || [...before, ...after].some(row => !row.arm))
+        throw new Error('Incomplete benchmark receipts');
+    const regressions = before.filter((row, i) => row.arm.status === 'passed' && after[i].arm.status !== 'passed').map(row => row.id);
+    const sums = (rows) => ({ passed: rows.filter(row => row.arm.status === 'passed').length, requests: rows.reduce((sum, row) => sum + row.arm.total.pi + row.arm.total.jev, 0), tokens: rows.reduce((sum, row) => sum + row.arm.total.tokens, 0), usageComplete: rows.every(row => row.arm.usageComplete), knownCostUsd: rows.reduce((sum, row) => sum + row.arm.total.knownCostUsd, 0), costComplete: rows.every(row => row.arm.costComplete) });
+    const previous = sums(before), current = sums(after);
+    const improved = !regressions.length && (current.passed > previous.passed || current.passed === previous.passed && current.passed === 3 && previous.usageComplete && current.usageComplete && current.tokens < previous.tokens);
+    return { mode: candidate.mode, baselineSkillSha256: baseline.skillSha256, candidateSkillSha256: candidate.skillSha256, regressions, previous, current, improved, eligibleForLiveHeldout: candidate.mode === 'live' && improved && current.passed === 3 };
+}

package/dist/comparison.d.ts

@@ -0,0 +1,69 @@
+import { type WorkflowDeps } from '@parcha/agentrun-dsl';
+import type { PiHostContext } from './types.js';
+export declare function evaluatePiComparison(caseId: string, output: unknown): {
+    passed: boolean;
+    issues: string[];
+};
+export interface PiComparisonBudgets {
+    timeoutMs: number;
+    maxModelRequests: number;
+    maxToolCalls: number;
+    maxTurns: number;
+    maxSubmissions: number;
+}
+export interface PiComparisonOptions {
+    host: PiHostContext;
+    runJudge: NonNullable<WorkflowDeps['runJudge']>;
+    signal?: AbortSignal;
+    mode: 'live' | 'scripted-transport';
+    budgets?: Partial<PiComparisonBudgets>;
+}
+export interface PiComparisonArm {
+    arm: 'plain-pi' | 'workflow';
+    status: 'passed' | 'rejected' | 'failed' | 'cancelled';
+    elapsedMs: number;
+    requests: {
+        pi: number;
+        jev: number;
+        total: number;
+    };
+    toolCalls: number;
+    submissions: number;
+    tokens: {
+        pi: number;
+        jev: number;
+        piUsageSamples: number;
+        jevUsageSamples: number;
+    };
+    output?: unknown;
+    evaluation?: ReturnType<typeof evaluatePiComparison>;
+    failure?: string;
+}
+export declare function runPiComparison(options: PiComparisonOptions): Promise<{
+    mode: "live" | "scripted-transport";
+    status: "complete" | "cancelled";
+    passed: boolean;
+    runtime: {
+        provider: string;
+        model: string;
+        api: string;
+        thinking: import("./types.js").PiThinkingLevel;
+    };
+    budgets: {
+        timeoutMs: number;
+        maxModelRequests: number;
+        maxToolCalls: number;
+        maxTurns: number;
+        maxSubmissions: number;
+    };
+    workflowSha256: string;
+    suiteSha256: string;
+    runs: {
+        caseId: string;
+        inputSha256: string;
+        expectedSha256: string;
+        executionOrder: string[];
+        arms: PiComparisonArm[];
+    }[];
+    limits: string[];
+}>;

package/dist/comparison.js

@@ -0,0 +1,229 @@
+import { createHash } from 'node:crypto';
+import { Compile } from 'typebox/compile';
+import { runWorkflow, runWorkflowSlice, workflowSha256 } from '@parcha/agentrun-dsl';
+import { createPiHostSessionFactory } from './host-session.js';
+import { createPiRunner, PiRunError } from './runner.js';
+import { demoData } from './demo-data.js';
+const cases = [
+    { id: 'single-fact', question: 'Can our contributors edit documentation in a browser?',
+        topics: [{ id: 'editing', question: 'Can contributors edit documentation in a browser?' }],
+        sources: [{ id: 'browser-trial', text: 'In our trial, all three contributors successfully edited a documentation page in the browser.' }, { id: 'advert', text: 'A beautiful new logo for documentation.' }],
+        expected: [{ topicId: 'editing', status: 'supported', sourceIds: ['browser-trial'] }] },
+    { id: 'parallel-research', question: 'What would change if our team moved its wiki into the code repository?',
+        topics: [{ id: 'reviews', question: demoData.subquestions[0] }, { id: 'editing', question: demoData.subquestions[1] }, { id: 'search', question: demoData.subquestions[2] }],
+        sources: structuredClone(demoData.sources),
+        expected: [{ topicId: 'reviews', status: 'supported', sourceIds: ['review-guide'] }, { topicId: 'editing', status: 'supported', sourceIds: ['editor-trial'] }, { topicId: 'search', status: 'supported', sourceIds: ['search-trial'] }] },
+    { id: 'gaps-and-conflicts', question: 'Does our repository trial support restricted editing and synonym search?',
+        topics: [{ id: 'permissions', question: 'Who can edit restricted documentation pages in our trial?' }, { id: 'search', question: 'Did synonym search work in our trial?' }],
+        sources: [{ id: 'permissions-heading', text: 'Restricted documentation permissions: details to be supplied.' }, { id: 'search-positive', text: 'The trial report says all ten synonym searches found the intended pages.' }, { id: 'search-negative', text: 'The audit of the same trial says none of those ten synonym searches found the intended pages.' }],
+        expected: [{ topicId: 'permissions', status: 'insufficient', sourceIds: [] }, { topicId: 'search', status: 'conflicting', sourceIds: ['search-positive', 'search-negative'] }] },
+];
+const str = { type: 'string', minLength: 1 };
+const findingSchema = { type: 'object', additionalProperties: false, required: ['topicId', 'status', 'summary', 'sourceIds'], properties: {
+        topicId: str, status: { enum: ['supported', 'conflicting', 'insufficient'] }, summary: str, sourceIds: { type: 'array', uniqueItems: true, items: str },
+    } };
+const reportSchema = { type: 'object', additionalProperties: false, required: ['answer', 'findings'], properties: {
+        answer: str, findings: { type: 'array', minItems: 1, items: findingSchema },
+    } };
+const sourcesSchema = { type: 'object', required: ['sources'], additionalProperties: false, properties: { sources: { type: 'array', items: { type: 'object', required: ['id', 'text'], additionalProperties: false, properties: { id: str, text: str } } } } };
+const evidenceSchema = demoData.workflow.root.steps[1].body.workflow.schemas.EvidenceDecision;
+const instructions = 'Use only the supplied fictional corpus. For each topic preserve all material sources, including contrary evidence, conditions and uncertainty. Status supported means the sources establish a finding, not necessarily a positive answer; conflicting means incompatible accounts; insufficient means no concrete information answers the topic. Cite source IDs. Do not turn qualified evidence into unconditional claims. You may read_sources or screen_evidence with Jev System One decisions. Submit the requested structured output.';
+const screen = { node: 'sift', label: 'screen-evidence', itemsPath: 'search.sources', state: { subquestion: '{item.question}' }, out: 'EvidenceDecision', as: 'evidence', keep: { path: 'answersQuestion', gte: 0.8 } };
+const workflow = {
+    v: 2, name: 'comparison-research', schemas: { Input: { type: 'object' }, Sources: sourcesSchema, Finding: findingSchema, Report: reportSchema, EvidenceDecision: evidenceSchema },
+    input: { schemaId: 'Input' }, output: { schemaId: 'Report', path: 'report' },
+    root: { node: 'chain', steps: [
+            { node: 'map', label: 'research-topics', itemsPath: 'topics', maxConcurrency: 3, resultPath: 'finding', as: 'findings', body: { node: 'chain', steps: [
+                        { node: 'call', label: 'read-sources', via: 'tool', tool: 'read_sources', args: {}, out: 'Sources', as: 'search', deadline_s: 15 }, screen,
+                        { node: 'agent', label: 'finding', instructions, state: { topic: '{item}', sources: '{evidence.items}' }, out: 'Finding', as: 'finding' },
+                    ] } },
+            { node: 'agent', label: 'report', instructions, state: { question: '{question}', topics: '{topics}', findings: '{findings}' }, out: 'Report', as: 'report' },
+        ] },
+};
+const selector = { ...workflow, name: 'comparison-screen', root: { node: 'chain', steps: [screen] } };
+const hash = (value) => createHash('sha256').update(JSON.stringify(value)).digest('hex');
+const validator = Compile(reportSchema);
+export function evaluatePiComparison(caseId, output) {
+    const item = cases.find(row => row.id === caseId);
+    if (!item)
+        throw new Error('Unknown comparison case');
+    if (!validator.Check(output))
+        return { passed: false, issues: ['output_schema'] };
+    const findings = output.findings;
+    const issues = [];
+    if (findings.length !== item.expected.length || new Set(findings.map(row => row.topicId)).size !== findings.length)
+        issues.push('topic_coverage');
+    for (const expected of item.expected) {
+        const actual = findings.find(row => row.topicId === expected.topicId);
+        if (!actual || actual.status !== expected.status)
+            issues.push(`${expected.topicId}:status`);
+        if (!actual || JSON.stringify([...actual.sourceIds].sort()) !== JSON.stringify([...expected.sourceIds].sort()))
+            issues.push(`${expected.topicId}:citations`);
+    }
+    return { passed: issues.length === 0, issues };
+}
+class ComparisonStop extends Error {
+    code;
+    constructor(code) {
+        super(code);
+        this.code = code;
+    }
+}
+const defaults = { timeoutMs: 120_000, maxModelRequests: 12, maxToolCalls: 24, maxTurns: 12, maxSubmissions: 3 };
+export async function runPiComparison(options) {
+    if (options.mode !== 'live' && options.mode !== 'scripted-transport')
+        throw new Error('Comparison requires an explicit execution mode');
+    if (typeof options.runJudge !== 'function')
+        throw new Error('Both arms require the same Jev capability');
+    const budgets = { ...defaults, ...options.budgets };
+    for (const key of Object.keys(defaults)) {
+        if (!Number.isSafeInteger(budgets[key]) || budgets[key] < 1 || budgets[key] > defaults[key])
+            throw new Error(`Comparison budget ${key} must be between 1 and ${defaults[key]}`);
+    }
+    const originalRegistry = options.host.modelRegistry;
+    const capturedHost = { ...options.host, modelRegistry: originalRegistry };
+    createPiHostSessionFactory(capturedHost);
+    const model = capturedHost.model;
+    const runs = [];
+    for (const [index, item] of cases.entries()) {
+        if (options.signal?.aborted)
+            break;
+        const order = index % 2 ? ['workflow', 'plain-pi'] : ['plain-pi', 'workflow'];
+        const arms = [];
+        for (const arm of order) {
+            if (options.signal?.aborted)
+                break;
+            arms.push(await runArm(item, arm));
+        }
+        runs.push({ caseId: item.id, inputSha256: hash({ question: item.question, topics: item.topics, sources: item.sources }), expectedSha256: hash(item.expected), executionOrder: arms.map(arm => arm.arm), arms });
+    }
+    return {
+        mode: options.mode, status: options.signal?.aborted ? 'cancelled' : 'complete',
+        passed: runs.length === cases.length && runs.every(run => run.arms.length === 2 && run.arms.every(arm => arm.status === 'passed')),
+        runtime: { provider: model.provider, model: model.id, api: model.api, thinking: capturedHost.thinkingLevel ?? 'low' },
+        budgets, workflowSha256: workflowSha256(workflow), suiteSha256: hash(cases), runs,
+        limits: ['Three fictional cases, one run per arm; not a statistical benchmark.', 'Workflow authoring and setup costs excluded; both arms receive identical prewritten topics.', 'Both arms can use the same corpus reader and Jev rubric; the plain loop chooses whether to call them.', 'Evaluation checks topic status and citation sets, not arbitrary prose entailment.', 'Token totals cover reported usage only; compare usage sample counts with request counts.', 'Host caches are not reset; execution order alternates.'],
+    };
+    async function runArm(item, arm) {
+        const started = performance.now();
+        const controller = new AbortController();
+        const signal = controller.signal;
+        const abort = (code) => { if (!signal.aborted)
+            controller.abort(new ComparisonStop(code)); };
+        const externalAbort = () => abort('cancelled');
+        options.signal?.addEventListener('abort', externalAbort, { once: true });
+        if (options.signal?.aborted)
+            externalAbort();
+        const timer = setTimeout(() => abort('deadline'), budgets.timeoutMs);
+        let rejectAbort;
+        const interrupted = new Promise((_resolve, reject) => { rejectAbort = reject; });
+        const onAbort = () => rejectAbort(signal.reason);
+        signal.addEventListener('abort', onAbort, { once: true });
+        if (signal.aborted)
+            onAbort();
+        void interrupted.catch(() => { });
+        const bounded = (work) => Promise.race([work, interrupted]);
+        const requests = { pi: 0, jev: 0, total: 0 };
+        const tokens = { pi: 0, jev: 0, piUsageSamples: 0, jevUsageSamples: 0 };
+        let toolCalls = 0, submissions = 0;
+        const admit = (kind) => {
+            signal.throwIfAborted();
+            if (kind === 'tool') {
+                if (toolCalls >= budgets.maxToolCalls)
+                    abort('tool_call_limit');
+                signal.throwIfAborted();
+                toolCalls++;
+            }
+            else {
+                if (requests.total >= budgets.maxModelRequests)
+                    abort('model_request_limit');
+                signal.throwIfAborted();
+                requests.total++;
+                requests[kind]++;
+            }
+        };
+        const host = { ...capturedHost, modelRegistry: {
+                getAll: () => originalRegistry.getAll(),
+                streamSimple(selected, transcript, streamOptions) {
+                    admit('pi');
+                    const stream = originalRegistry.streamSimple(selected, transcript, streamOptions);
+                    void stream.result().then(result => {
+                        const usage = result?.usage;
+                        if (Number.isFinite(usage?.totalTokens) && usage.totalTokens >= 0) {
+                            tokens.pi += usage.totalTokens;
+                            tokens.piUsageSamples++;
+                        }
+                    }).catch(() => { });
+                    return stream;
+                },
+            } };
+        const runJudge = async (request) => {
+            request.signal?.throwIfAborted();
+            admit('jev');
+            const result = await bounded(options.runJudge({ ...request, signal: request.signal ? AbortSignal.any([signal, request.signal]) : signal }));
+            if (result.usage && Number.isFinite(result.usage.input_tokens) && Number.isFinite(result.usage.output_tokens) && result.usage.input_tokens >= 0 && result.usage.output_tokens >= 0) {
+                tokens.jev += result.usage.input_tokens + result.usage.output_tokens;
+                tokens.jevUsageSamples++;
+            }
+            return result;
+        };
+        const read = () => ({ sources: structuredClone(item.sources) });
+        const tools = [
+            { name: 'read_sources', label: 'Read sources', description: 'Read the complete fictional corpus for this task.', parameters: { type: 'object', properties: {}, additionalProperties: false },
+                async execute() { const value = read(); return { content: [{ type: 'text', text: JSON.stringify(value) }], details: value }; } },
+            { name: 'screen_evidence', label: 'Screen evidence', description: 'Use Jev System One decisions to retain concrete evidence for a topic, including qualified and contrary evidence. Same rubric and 0.8 threshold as the workflow.',
+                parameters: { type: 'object', properties: { topicId: str, sourceIds: { type: 'array', uniqueItems: true, items: str } }, required: ['topicId', 'sourceIds'], additionalProperties: false },
+                async execute(_id, value, toolSignal) {
+                    const args = value;
+                    const topic = item.topics.find(row => row.id === args.topicId);
+                    if (!topic || args.sourceIds.some(id => !item.sources.some(source => source.id === id)))
+                        throw new Error('Unknown topic or source');
+                    const selected = item.sources.filter(source => args.sourceIds.includes(source.id));
+                    const result = await bounded(runWorkflowSlice(selector, {}, { from: 'screen-evidence', seed: { item: topic, search: { sources: selected } } }, { runJudge: request => runJudge({ ...request, signal: toolSignal }) }));
+                    if (result.status !== 'complete')
+                        throw new Error('Evidence screening did not complete');
+                    const evidence = result.state.evidence;
+                    return { content: [{ type: 'text', text: JSON.stringify(evidence) }], details: evidence };
+                } },
+        ];
+        try {
+            const factory = createPiHostSessionFactory(host);
+            const runNode = createPiRunner({ model, thinkingLevel: capturedHost.thinkingLevel, cwd: capturedHost.cwd, signal,
+                maxTurns: budgets.maxTurns, maxSubmissions: budgets.maxSubmissions, timeoutMs: budgets.timeoutMs, tools,
+                sessionFactory: opts => factory({ ...opts, customTools: opts.customTools.map(tool => ({ ...tool,
+                        async execute(...args) { args[2]?.throwIfAborted(); admit('tool'); if (tool.name === 'submit')
+                            submissions++; return tool.execute(...args); },
+                    })) }),
+            });
+            const input = { question: item.question, topics: structuredClone(item.topics) };
+            const output = await bounded((async () => {
+                if (arm === 'plain-pi')
+                    return runNode({ kind: 'agent', label: 'plain-pi', system: [instructions], user: JSON.stringify(input), schema: reportSchema });
+                const result = await runWorkflow(workflow, input, { runNode, runJudge, runEffect: async ({ node }) => {
+                        if (node.tool !== 'read_sources')
+                            throw new Error('Unknown comparison tool');
+                        admit('tool');
+                        return read();
+                    } });
+                if (result.status !== 'complete')
+                    throw new Error('Workflow did not complete');
+                return result.output;
+            })());
+            signal.throwIfAborted();
+            const evaluation = evaluatePiComparison(item.id, output);
+            return finish({ status: evaluation.passed ? 'passed' : 'rejected', output, evaluation });
+        }
+        catch (error) {
+            return finish({ status: options.signal?.aborted ? 'cancelled' : 'failed', failure: signal.reason instanceof ComparisonStop ? signal.reason.code : error instanceof PiRunError ? `pi_${error.reason}` : 'execution_failed' });
+        }
+        finally {
+            clearTimeout(timer);
+            options.signal?.removeEventListener('abort', externalAbort);
+            signal.removeEventListener('abort', onAbort);
+            abort('arm_finished');
+        }
+        function finish(result) {
+            return { arm, ...result, elapsedMs: Math.round(performance.now() - started), requests: { ...requests }, toolCalls, submissions, tokens: { ...tokens } };
+        }
+    }
+}

package/dist/extension-service.js

@@ -111,6 +111,7 @@ function localDiagnostic(error) {
             empty_selection: 'The selection list is empty and none is not allowed. Handle missing evidence explicitly.',
             missing_map_result: 'A map body did not produce its resultPath. Select a path in the completed body state; a body with as is collected automatically when resultPath is omitted.',
             parallel_write_conflict: 'Parallel branches changed the same top-level state key. Give independent branches distinct output keys and combine them after the parallel node.',
+            reserved_state_key: 'A node or host hook wrote an engine-owned "$"-prefixed state key such as $host. Choose another state key; host state is written only through decodeSubmission.',
         };
         return { code: error.code, stage: error.stage.slice(0, 200),
             // Parallel patch keys may be computed from source values, unlike authored paths.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@parcha/agentrun-pi",
-  "version": "0.1.0-beta.1",
+  "version": "0.1.0-beta.2",
   "description": "Pi extension for building, running and evaluating AgentRun workflows",
   "type": "module",
   "license": "Apache-2.0",
@@ -36,8 +36,8 @@
     "@earendil-works/pi-ai": "0.87.0"
   },
   "dependencies": {
-    "@parcha/agentrun-dsl": "0.1.0-beta.1",
-    "@parcha/agentrun-jev": "0.1.0-beta.1",
+    "@parcha/agentrun-dsl": "0.1.0-beta.2",
+    "@parcha/agentrun-jev": "0.1.0-beta.2",
     "@earendil-works/pi-agent-core": "0.87.0",
     "@earendil-works/pi-coding-agent": "0.87.0",
     "@earendil-works/pi-tui": "0.87.0",