carthagent 0.0.0-stage → 1.0.7

package/docs/webdriver.md

@@ -0,0 +1,83 @@
+# Real-browser checks
+
+`tools/webdriver-check.mjs` runs checks in a real browser. By default it uses
+the bundled `chrome-headless-shell` build — a lightweight headless-only
+browser downloaded on first use into `~/.carthagent/browser/` (pinned version,
+`CARTHAGENT_BROWSER_VERSION` to override, `CARTHAGENT_BROWSER` to supply your
+own binary; system Chrome/Chromium is used when present). The bundled browser
+is driven directly over the DevTools protocol, so no driver binary or npm
+dependency is required.
+
+```sh
+node tools/webdriver-check.mjs \
+  --root ./my-app --page index.html \
+  --assert-text '#status:ready' --click '#submit' \
+  --assert-text '#status:saved' --screenshot artifacts/browser.png
+```
+
+To drive an external W3C WebDriver service instead (e.g. geckodriver or
+chromedriver you manage), pass `--endpoint`:
+
+```sh
+geckodriver --host 127.0.0.1 --port 4444 &
+node tools/webdriver-check.mjs --endpoint http://127.0.0.1:4444 \
+  --root ./my-app --page index.html --assert-text '#status:ready'
+```
+
+Options are processed in order. `--assert` checks element existence;
+`--assert-text` checks visible text contains the supplied string; `--click`
+dispatches a click on the element after it exists. Screenshots use the same
+session after all steps. Exit 1 means failure, including unavailable browser
+or unsupported options. Requests and navigation have deadlines. The spawned
+browser is terminated on success and failure.
+
+For an already running Vite/Next.js or other application server, use
+`--url http://127.0.0.1:5173` instead of `--root`/`--page`. Start and stop that
+server separately. Only navigate to applications you trust: browser checks
+execute their scripts and clicks can change application data.
+
+Element existence and text assertions poll for up to `--wait-ms 5000` per step
+(default 5000; integer range 1–60000). Invalid selectors and driver errors fail
+immediately. Clicks wait for element existence but are executed exactly once;
+interaction errors are not retried. Page navigation has a separate 15-second
+timeout. This is a per-step bound, not a whole-run budget.
+
+Failures are JSON with `category` (configuration, launch, driver, capture,
+navigation, assertion, interaction, browser-errors, screenshot, or cleanup),
+`error`, and where applicable zero-based `stepIndex`, `selector`, and
+`expected`. Use this evidence to fix the relevant
+step rather than blindly repeating a mutating interaction.
+Use meaningful application-state assertions: element existence alone does not
+prove that a module ran. Console-error capture is automatic and mandatory —
+there is no `--console-clean` flag; it is rejected rather than silently
+ignored. Screenshot success is not visual
+review. The existing jsdom tool remains useful for classic-script DOM checks,
+but rejects unsupported module execution.
+
+Run the optional actual-browser regression from the repository root:
+
+```sh
+node tools/test-webdriver.mjs
+```
+
+It provisions the bundled browser (first run downloads ~100MB into
+`~/.carthagent/browser/`), exercises a module application and broken-module
+failure, and writes `artifacts/webdriver.png`.
+Default `npm test` tests CLI failures without requiring a browser.
+
+### Mandatory browser-error evidence
+
+Browser error capture subscribes before navigation (DevTools `Runtime` and
+`Log` domains; WebDriver endpoints use BiDi `log.entryAdded`), capturing
+startup console errors, uncaught exceptions and unhandled promise rejections.
+Unexpected error-level entries fail the check even when DOM assertions pass.
+Capture-channel failure fails closed; there is no silent DOM-only fallback. Diagnostics accompany assertion failures too and retain at most 50
+entries, 2,000 text characters and eight bounded stack frames per entry; dropped
+entries are counted. No broad ignore option is provided. Expected-error tests
+must currently use a separate explicit validator rather than suppressing logs.
+
+Coverage ends after the declared interactions/assertions and a BiDi round trip;
+this is not a promise to detect errors occurring arbitrarily later. Declare a
+readiness assertion for asynchronous work. Network failures are not classified
+as application errors unless the browser emits an error-level log. Warnings,
+visual correctness and all network traffic are not covered by this signal.

package/docs/workflow-modes.md

@@ -0,0 +1,26 @@
+# Workflow modes and evidence
+
+The ctg CLI defaults to `--workflow standard`. A valid required acceptance contract enables implementation without mandatory component/port/scenario review. Final verification remains strict: fresh check identities, new requirement executions, artifact/output existence, structured step completion and a snapshot-bound final review are required. Step `done` means implementation complete, not passing integrated checks.
+
+`--workflow strict` keeps tested-design approval before implementation. Direct extension hosts default to strict; restored runs retain their recorded workflowMode. Neither mode is an OS sandbox. Informational contracts cannot authorize implementation.
+
+`delivery_probe` exposes only Node/npm version argv with a ten-second bound; it is discovery, not check evidence. Unknown custom tools remain fail-closed.
+
+Partial revisions merge omitted design/workflow object fields; supplied arrays replace their respective arrays. Strict revisions still relock approval. Standard revisions continue implementation but invalidate affected evidence and final review. Validation observes the current plan and records findings without clearing approval; receipt identity still detects changed plans/source.
+
+Graph navigation is permitted during discovery, returns pagination/coverage and supports file/container filtering. Absolute workspace file paths normalize; directory filters return actionable errors. Name-based call matches remain candidates rather than runtime resolution.
+
+## Four-arm local pilot
+
+One natural-user trade-validation task, identical bundled engine/model/dependencies, external 900-second/120-tool ceilings, 47 frozen checks:
+
+| Arm | Elapsed | Recorded USD | External checks | Terminal outcome |
+|---|---:|---:|---:|---|
+| Plain Pi | 247.4 s | 0.033085 | 47/47 | Normal final answer |
+| Pi + graph | 535.7 s | 0.066948 | 47/47 | Provider failure; no final answer |
+| ctg standard | 509.5 s | 0.097460 | 47/47 | Verified handoff |
+| ctg strict | 900.3 s | 0.108461 | 12/47 | Timeout before product implementation |
+
+This is a single-task pilot, not a statistically significant cost/completion claim. Providers varied between requests; interrupted requests have unreconciled usage, so affected totals are lower bounds. Standard completed where strict did not, but plain Pi was faster and cheaper than standard. Graph benefit was not established. The evaluator's 12 strict passes are preserved pre-existing behavior/scope, not delivery completion. Candidate documented test commands are audited separately from external behavior.
+
+Local task, source freezes, events, usage receipts and reports reside in `/home/baissi/ctg_test/`; these paths describe this development machine, not distributed benchmark artifacts. Future evaluation should randomize repeated tasks, retain plain Pi, classify provider errors separately and report user task completion separately from internal handoff completion.

package/docs/workflow.d2

@@ -0,0 +1,17 @@
+direction: right
+inspect: "Inspect repository + installed APIs\nIdentify relevant knowledge bases/skills"
+plan: "Acceptance contract + plan\nScope, risks, concrete checks"
+pressure: "Pressure-test plan\nVerify checks detect failure"
+slice: "Runnable vertical slice"
+build: "Implement features + regression tests\nMark progress per step"
+verify: "Run checks against source fingerprint\nRecord exit status + output tail"
+review: "Adversarial review\nInteraction + visual + failure paths"
+repair: "Repair failures (bounded)\nFailing check + quoted output tail"
+adjust: "Adjust plan on wrong assumptions\nUpdate steps/checks to match reality"
+deliver: "Evidence + launch instructions\nExplicit limitations"
+inspect -> plan -> pressure -> slice -> build -> verify -> review -> deliver
+verify -> repair: failed / stale
+repair -> verify
+build -> adjust: wrong assumption
+adjust -> plan: revise contract
+review -> repair: defect

package/docs/workflow.svg

@@ -0,0 +1,149 @@
+<?xml version="1.0" encoding="utf-8"?>
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 3660 460" width="3660" height="460" preserveAspectRatio="xMinYMin meet">
+  <defs>
+    <style type="text/css"><![CDATA[
+      .node-text-bold { font-family: ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif; font-weight: 700; font-size: 15px; fill: #0A0F25; }
+      .node-text-sub { font-family: ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif; font-weight: 500; font-size: 13.5px; fill: #4B5563; }
+      .edge-label { font-family: ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif; font-style: italic; font-size: 14px; fill: #475569; }
+      .edge-label-success { font-family: ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif; font-weight: 600; font-size: 13.5px; fill: #047857; }
+      .edge-label-critique { font-family: ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif; font-weight: 600; font-size: 13.5px; fill: #6366F1; }
+      .arrow { stroke: #0D32B2; stroke-width: 2; fill: none; stroke-linecap: round; stroke-linejoin: round; }
+      .arrow-critique { stroke: #4F46E5; stroke-width: 2.2; fill: none; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 4,3; }
+      .arrow-success { stroke: #059669; stroke-width: 2.2; fill: none; stroke-linecap: round; stroke-linejoin: round; }
+      .arrow-repair { stroke: #D97706; stroke-width: 2; fill: none; stroke-linecap: round; stroke-linejoin: round; }
+    ]]></style>
+
+    <marker id="arrowhead" markerWidth="10" markerHeight="10" refX="8" refY="5" viewBox="0 0 10 10" orient="auto" markerUnits="userSpaceOnUse">
+      <polygon points="0,1.5 8,5 0,8.5" fill="#0D32B2" />
+    </marker>
+    <marker id="arrowhead-critique" markerWidth="10" markerHeight="10" refX="8" refY="5" viewBox="0 0 10 10" orient="auto" markerUnits="userSpaceOnUse">
+      <polygon points="0,1.5 8,5 0,8.5" fill="#4F46E5" />
+    </marker>
+    <marker id="arrowhead-success" markerWidth="10" markerHeight="10" refX="8" refY="5" viewBox="0 0 10 10" orient="auto" markerUnits="userSpaceOnUse">
+      <polygon points="0,1.5 8,5 0,8.5" fill="#059669" />
+    </marker>
+    <marker id="arrowhead-repair" markerWidth="10" markerHeight="10" refX="8" refY="5" viewBox="0 0 10 10" orient="auto" markerUnits="userSpaceOnUse">
+      <polygon points="0,1.5 8,5 0,8.5" fill="#D97706" />
+    </marker>
+  </defs>
+
+  <rect x="0" y="0" width="3660" height="460" fill="#FFFFFF" rx="8" />
+
+  <!-- 1. INSPECT NODE -->
+  <g id="node-inspect">
+    <rect x="20" y="175" width="270" height="70" rx="8" stroke="#0D32B2" stroke-width="2" fill="#F7F8FE" />
+    <text x="155" y="216" text-anchor="middle" class="node-text-bold">Inspect repository + installed APIs</text>
+  </g>
+
+  <!-- inspect -> contract -->
+  <path d="M 290 210 L 350 210" class="arrow" marker-end="url(#arrowhead)" />
+
+  <!-- 2. CONTRACT DRAFT NODE -->
+  <g id="node-contract">
+    <rect x="350" y="167" width="270" height="86" rx="8" stroke="#0D32B2" stroke-width="2" fill="#F7F8FE" />
+    <text x="485" y="205" text-anchor="middle" class="node-text-bold">Acceptance contract draft</text>
+    <text x="485" y="228" text-anchor="middle" class="node-text-sub">Scope, risks, concrete checks</text>
+  </g>
+
+  <!-- contract -> review_plan -->
+  <path d="M 620 210 L 685 210" class="arrow" marker-end="url(#arrowhead)" />
+
+  <!-- 3. REVIEW_PLAN (CRITIQUE LOOP) NODE -->
+  <g id="node-review_plan">
+    <rect x="690" y="156" width="340" height="106" rx="10" stroke="#4F46E5" stroke-width="2.5" fill="#EEF2FF" />
+    <rect x="702" y="166" width="94" height="20" rx="10" fill="#E0E7FF" />
+    <text x="749" y="180" text-anchor="middle" font-size="11" font-weight="700" fill="#4338CA">CRITIQUE LOOP</text>
+    <text x="860" y="205" text-anchor="middle" class="node-text-bold" fill="#1E1B4B">Plan &amp; architecture critique loop</text>
+    <text x="860" y="227" text-anchor="middle" class="node-text-sub" fill="#3730A3">Confidence &#x2265; 8/10 &amp; failure modes</text>
+    <text x="860" y="247" text-anchor="middle" class="node-text-sub" fill="#4338CA">Bounded iterations (&#x2265; 2 rounds)</text>
+  </g>
+
+  <!-- review_plan -> review_plan SELF LOOP (Low confidence / unhandled risks) -->
+  <path d="M 770 156 C 770 95, 950 95, 950 152" class="arrow-critique" marker-end="url(#arrowhead-critique)" />
+  <rect x="745" y="70" width="230" height="22" rx="4" fill="#FFFFFF" stroke="#E2E8F0" stroke-width="1" />
+  <text x="860" y="85" text-anchor="middle" class="edge-label-critique">Low confidence / unhandled risks</text>
+
+  <!-- review_plan -> plan (High confidence & approved) -->
+  <path d="M 1030 210 L 1315 210" class="arrow-success" marker-end="url(#arrowhead-success)" />
+  <rect x="1055" y="196" width="235" height="24" rx="4" fill="#ECFDF5" stroke="#A7F3D0" stroke-width="1" />
+  <text x="1172" y="213" text-anchor="middle" class="edge-label-success">High confidence (&#x2265; 8/10) &amp; approved</text>
+
+  <!-- 4. PLAN (LOCKED) NODE -->
+  <g id="node-plan">
+    <rect x="1320" y="167" width="250" height="86" rx="8" stroke="#0D32B2" stroke-width="2" fill="#F7F8FE" />
+    <text x="1445" y="205" text-anchor="middle" class="node-text-bold">D2 implementation plan (locked)</text>
+    <text x="1445" y="228" text-anchor="middle" class="node-text-sub">Frozen scope, bound checks</text>
+  </g>
+
+  <!-- plan -> slice -->
+  <path d="M 1570 210 L 1635 210" class="arrow" marker-end="url(#arrowhead)" />
+
+  <!-- 5. RUNNABLE VERTICAL SLICE NODE -->
+  <g id="node-slice">
+    <rect x="1640" y="167" width="280" height="86" rx="8" stroke="#0D32B2" stroke-width="2" fill="#F7F8FE" />
+    <text x="1780" y="205" text-anchor="middle" class="node-text-bold">Runnable vertical slice</text>
+    <text x="1780" y="228" text-anchor="middle" class="node-text-sub">Rendering verified first for games</text>
+  </g>
+
+  <!-- slice -> build -->
+  <path d="M 1920 210 L 1985 210" class="arrow" marker-end="url(#arrowhead)" />
+
+  <!-- 6. IMPLEMENT FEATURES NODE -->
+  <g id="node-build">
+    <rect x="1990" y="175" width="290" height="70" rx="8" stroke="#0D32B2" stroke-width="2" fill="#F7F8FE" />
+    <text x="2135" y="216" text-anchor="middle" class="node-text-bold">Implement features + regression tests</text>
+  </g>
+
+  <!-- build -> verify -->
+  <path d="M 2280 210 L 2345 210" class="arrow" marker-end="url(#arrowhead)" />
+
+  <!-- 7. VERIFY CHECKS NODE -->
+  <g id="node-verify">
+    <rect x="2350" y="167" width="310" height="86" rx="8" stroke="#0D32B2" stroke-width="2" fill="#F7F8FE" />
+    <text x="2505" y="205" text-anchor="middle" class="node-text-bold">Run checks against fingerprint</text>
+    <text x="2505" y="228" text-anchor="middle" class="node-text-sub">Record exit status + output tail</text>
+  </g>
+
+  <!-- 8. ADVERSARIAL REVIEW NODE -->
+  <g id="node-review">
+    <rect x="2770" y="30" width="290" height="84" rx="8" stroke="#0D32B2" stroke-width="2" fill="#F7F8FE" />
+    <text x="2915" y="68" text-anchor="middle" class="node-text-bold">Adversarial review</text>
+    <text x="2915" y="91" text-anchor="middle" class="node-text-sub">UX, edge cases, limitations</text>
+  </g>
+
+  <!-- verify -> review -->
+  <path d="M 2550 167 C 2630 100, 2700 72, 2765 72" class="arrow" marker-end="url(#arrowhead)" />
+
+  <!-- 9. REPAIR FAILURES NODE -->
+  <g id="node-repair">
+    <rect x="3230" y="167" width="290" height="86" rx="8" stroke="#D97706" stroke-width="2" fill="#FFFBEB" />
+    <text x="3375" y="205" text-anchor="middle" class="node-text-bold" fill="#92400E">Repair failures (bounded)</text>
+    <text x="3375" y="228" text-anchor="middle" class="node-text-sub" fill="#B45309">Failing check + quoted output tail</text>
+  </g>
+
+  <!-- 10. DELIVER NODE -->
+  <g id="node-deliver">
+    <rect x="3230" y="30" width="280" height="84" rx="8" stroke="#059669" stroke-width="2" fill="#ECFDF5" />
+    <text x="3370" y="68" text-anchor="middle" class="node-text-bold" fill="#065F46">Evidence + launch instructions</text>
+    <text x="3370" y="91" text-anchor="middle" class="node-text-sub" fill="#047857">Explicit limitations</text>
+  </g>
+
+  <!-- review -> deliver -->
+  <path d="M 3060 72 L 3225 72" class="arrow-success" marker-end="url(#arrowhead-success)" />
+
+  <!-- verify -> repair (failed / stale) -->
+  <path d="M 2660 210 L 3225 210" class="arrow-repair" marker-end="url(#arrowhead-repair)" />
+  <rect x="2860" y="198" width="110" height="22" rx="4" fill="#FEF3C7" stroke="#FDE68A" stroke-width="1" />
+  <text x="2915" y="214" text-anchor="middle" class="edge-label" fill="#B45309">failed / stale</text>
+
+  <!-- review -> repair (defect) -->
+  <path d="M 3060 92 C 3140 120, 3180 150, 3225 180" class="arrow-repair" marker-end="url(#arrowhead-repair)" />
+  <rect x="3130" y="118" width="60" height="20" rx="4" fill="#FEF3C7" stroke="#FDE68A" stroke-width="1" />
+  <text x="3160" y="132" text-anchor="middle" class="edge-label" fill="#B45309">defect</text>
+
+  <!-- repair -> verify (feedback loop) -->
+  <path d="M 3375 253 C 3375 350, 2505 350, 2505 258" class="arrow" marker-end="url(#arrowhead)" />
+  <rect x="2880" y="338" width="120" height="22" rx="4" fill="#FFFFFF" stroke="#CBD5E1" stroke-width="1" />
+  <text x="2940" y="353" text-anchor="middle" class="edge-label">repair &#x2192; verify</text>
+
+</svg>

package/docs/workspace-coordination.md

@@ -0,0 +1,9 @@
+# Workspace coordination
+
+CLI checks, extension delivery tools (TUI/headless), and dashboard mutations share an exclusive `.harness/workspace.lock` for the whole operation, including subprocess checks. Contention fails explicitly; operations are not silently replayed. Same-process extension calls retain FIFO ordering. Dashboard contention returns HTTP 409.
+
+An explicit new plan selects `.harness/active.json`. Status/dashboard/CLI selection uses this pointer rather than report modification time. Legacy workspaces without a pointer retain newest-report discovery until a coordinated operation selects a run. A missing or invalid selected report fails closed.
+
+Before each extension mutation, the latest same-run persisted state is reconciled. Superseded runs cannot write or trigger automatic repairs; explicitly creating a new plan can select fresh work once the workspace is idle. Dashboard mutations refresh state under the lock. Existing report version checks still reject non-cooperating stale writers. Failed operations are not automatically replayed, and new plans intentionally supersede idle runs rather than merging unrelated contracts.
+
+Locks are cooperative, local filesystem controls, not security isolation or a distributed lease. They do not serialize arbitrary shell edits, reviewers, or external tools. A crashed process can leave a lock: inspect its PID and the report, confirm no writer/check child is alive, then remove the lock manually. Never steal locks based solely on age. Manual state-file edits can bypass these controls. Separate worktrees have separate coordination state.

package/evaluate.mjs

@@ -0,0 +1,249 @@
+#!/usr/bin/env node
+/** Opt-in live A/B runs. Separate directories are NOT security sandboxes.
+ *  Task domains are pluggable scenario manifests under scenarios/<name>/ —
+ *  a game, a CLI task, or any future profile is one directory of config,
+ *  graders and seeds, not a special case in this file.
+ */
+import { parseArgs } from 'node:util';
+import { fileURLToPath } from 'node:url';
+import { basename, delimiter, dirname, join } from 'node:path';
+import { existsSync, mkdirSync, copyFileSync, cpSync, realpathSync, readdirSync, statSync, writeFileSync, readFileSync, appendFileSync } from 'node:fs';
+import { randomUUID } from 'node:crypto';
+import { atomicJson, runCommand } from './lib/delivery.mjs';
+import { loadScenario, expandPlaceholders } from './lib/scenarios.mjs';
+import { runNativeEvaluation, NATIVE_EVALUATION_RUNTIME } from './lib/opencode/evaluation.mjs';
+import { ensureService, createSession, stopService } from './lib/opencode/service.mjs';
+import { provisionRuntime } from './lib/opencode/runtime.mjs';
+import { OpenCode } from '@opencode/client';
+
+const { values: args } = parseArgs({ options: {
+  mode: { type: 'string', default: 'both' }, task: { type: 'string', default: 'cli' }, scenario: { type: 'string' },
+  model: { type: 'string', default: 'inception/mercury-2.5-preview' }, provider: { type: 'string', default: 'openrouter' },
+  thinking: { type: 'string', default: 'medium' },
+  timeout: { type: 'string', default: '600' }, 'max-turns': { type: 'string', default: '100' }, 'max-cost': { type: 'string', default: '3' },
+  python: { type: 'string' }, 'seed-from': { type: 'string' }, 'allow-live': { type: 'boolean', default: false },
+  'dry-run': { type: 'boolean', default: false },
+} });
+if (!args['allow-live'] && !args['dry-run']) throw Error('Live calls spend API credit and generated code runs with your permissions. Use --allow-live explicitly (or --dry-run to preview the assembled run without calling a provider).');
+if (!['baseline', 'custom', 'both'].includes(args.mode)) throw Error('Use --mode baseline|custom|both');
+for (const key of ['timeout', 'max-turns', 'max-cost']) if (!(Number(args[key]) > 0 && Number.isFinite(Number(args[key])))) throw Error(`Invalid ${key}`);
+if (Number(args['max-turns']) > 1000 || Number(args.timeout) > 3600) throw Error('Evaluation bounds may not exceed 1000 turns or 3600 seconds.');
+
+const root = dirname(fileURLToPath(import.meta.url)), project = dirname(root);
+// Layout: standalone checkouts keep evaluation fixtures beside this script;
+// when this package is nested inside a workspace, the venv and evidence
+// directories live in the parent project instead.
+const home = existsSync(join(root, 'scenarios')) ? root : project;
+const providerKey = process.env[`${args.provider.toUpperCase().replace(/[^A-Z0-9]/g, '_')}_API_KEY`];
+if (!args['dry-run'] && !providerKey) throw Error(`Set ${args.provider.toUpperCase().replace(/[^A-Z0-9]/g, '_')}_API_KEY explicitly; evaluation never discovers credentials from a home-directory agent config.`);
+const scenarioName = args.scenario || args.task;
+const scenario = loadScenario(root, scenarioName);
+const findExecutable = name => {
+  const names = process.platform === 'win32' ? [`${name}.exe`, name] : [name];
+  for (const dir of (process.env.PATH || '').split(delimiter)) {
+    for (const candidate of names.map(file => join(dir, file))) if (existsSync(candidate)) return candidate;
+  }
+  return null;
+};
+const venvPython = join(home, '.venv', process.platform === 'win32' ? 'Scripts/python.exe' : 'bin/python');
+const python = args.python || (existsSync(venvPython) ? venvPython : findExecutable('python3') || findExecutable('python') || venvPython);
+if (scenario.needsPython && !existsSync(python)) {
+  const msg = `Scenario ${scenarioName} needs Python — pass --python /absolute/path/to/python${existsSync(join(scenario.dir, 'requirements.txt')) ? ` after installing scenarios/${scenarioName}/requirements.txt` : ''}`;
+  if (args['dry-run']) console.error(`warning: ${msg}`); else throw Error(msg);
+}
+const seedFrom = args['seed-from'] ? realpathSync(args['seed-from']) : null;
+if (seedFrom && !statSync(seedFrom).isDirectory()) throw Error('--seed-from must name a candidate directory');
+const continuationEvidence = (() => {
+  if (!seedFrom) return '';
+  const summaryPath = join(dirname(seedFrom), 'summary.json');
+  if (!existsSync(summaryPath)) return '';
+  try {
+    const parsed = JSON.parse(readFileSync(summaryPath, 'utf8'));
+    const summary = Array.isArray(parsed) ? parsed.at(-1) : parsed;
+    const failed = (summary.finalChecks || [])
+      .filter(check => check.code !== 0)
+      .map(check => `${check.id}:\n${check.output}`)
+      .join('\n\n');
+    return failed.slice(-6000);
+  } catch {
+    return '';
+  }
+})();
+
+const vars = ({ cwd }) => ({ python, node: process.execPath, root, home, cwd, scenario: scenario.dir, ...(scenario.runtimes || {}) });
+const scenarioChecks = (list, cwd) => list.map(c => ({ ...c, argv: expandPlaceholders(c.argv, vars({ cwd })) }));
+
+const base = join(home, '.harness', 'evaluations', `${Date.now()}-${scenarioName}-${randomUUID().slice(0, 6)}`);
+if (!args['dry-run']) mkdirSync(base, { recursive: true });
+const runtimeInfo = [
+  scenario.needsPython ? `Python executable: ${JSON.stringify(python)}.` : null,
+  `Node: ${JSON.stringify(process.execPath)}.`,
+].filter(Boolean).join(' ');
+const environment = `Evaluation execution context:
+The current working directory is the isolated product workspace. Write product files with workspace-relative paths such as README.md or src/index.ts; do not prefix them with the current directory or recreate an absolute path as nested folders. ${runtimeInfo} Invoke these exact executable paths and quote them in shell commands. Absolute paths in this context identify external tools and are valid as-is; do not hunt for them.
+Set a bounded timeout on every shell command and never run filesystem-wide searches (find /, ls -R from the root, find of whole drives): one hung command can consume the entire time budget. Do not touch files outside this work directory or run nested engine agents. No network installs or deployment.
+You have at most ${args['max-turns']} productive model turns (recovered provider connection errors do not consume that budget) and ${args.timeout} seconds; reserve time for verification. If declared checks pass with budget unused, re-check every user requirement and finish missing behavior rather than stopping at the first passing slice. Generated logs/screenshots go in artifacts/.
+${expandPlaceholders(scenario.setup, vars({ cwd: '<the isolated product workspace>' }))}`;
+const environmentPath = join(base, 'execution-context.md');
+if (!args['dry-run']) writeFileSync(environmentPath, environment);
+const summaries = [];
+
+function copySeeds(cwd) {
+  if (seedFrom) {
+    const excluded = new Set(['.harness', 'artifacts', '__pycache__', '.pytest_cache', '.venv', 'node_modules', 'dist', 'build', 'coverage', '.turbo']);
+    for (const entry of readdirSync(seedFrom, { withFileTypes: true })) {
+      if (!excluded.has(entry.name)) {
+        cpSync(join(seedFrom, entry.name), join(cwd, entry.name), { recursive: true });
+      }
+    }
+    return;
+  }
+  for (const seed of scenario.seeds) {
+    const target = join(cwd, seed.to);
+    mkdirSync(dirname(target), { recursive: true });
+    copyFileSync(seed.from, target);
+  }
+}
+
+async function runScenarioChecks(mode, cwd, tier, checks) {
+  const results = [];
+  for (const check of scenarioChecks(checks, cwd)) {
+    const result = await runCommand(check.argv, {
+      cwd,
+      timeoutSeconds: check.timeoutSeconds,
+      logPath: join(base, `${mode}-${tier}-${check.id}.log`),
+    });
+    results.push({
+      id: check.id,
+      tier,
+      code: result.code,
+      timedOut: result.timedOut,
+      output: result.output,
+    });
+  }
+  return results;
+}
+
+for (const mode of args.mode === 'both' ? ['baseline', 'custom'] : [args.mode]) {
+  const cwd = join(base, mode);
+  if (!args['dry-run']) {
+    mkdirSync(cwd, { recursive: true });
+    copySeeds(cwd);
+  }
+  const systemPrompt = `${environment}\n\n${mode === 'custom' ? scenario.context : ''}`;
+  if (mode === 'custom') {
+    const manifest = join(base, 'required-validators.json');
+    if (!args['dry-run']) atomicJson(manifest, { version: 1, checks: scenarioChecks(scenario.developmentChecks, cwd) });
+    // The native plugin owns delivery configuration; the manifest remains an
+    // explicit piece of immutable evidence for that treatment.
+    // Domain context from the manifest, plus generic continuation guidance.
+    const context = [
+      scenario.context,
+      seedFrom && 'This workspace contains an earlier generated candidate. Every behavior not listed in the independent failures already passes. Preserve those behaviors, avoid subsystem rewrites, and make the smallest focused correction that addresses only the listed failures.',
+      continuationEvidence && `Independent failures from the candidate being continued:\n${continuationEvidence}`,
+    ].filter(Boolean).join('\n\n');
+    if (context) {
+      const contextPath = join(base, `${basename(scenarioName)}-context.md`);
+       if (!args['dry-run']) writeFileSync(contextPath, context);
+    }
+  }
+  const controller = new AbortController();
+  console.log(`Starting ${mode} ${scenarioName} with ${args.provider}/${args.model}; ${cwd}`);
+  if (args['dry-run']) {
+    console.log(JSON.stringify({
+      mode, cwd, runtime: NATIVE_EVALUATION_RUNTIME, nativeRequest: { directory: cwd, provider: args.provider, model: args.model, prompt: scenario.prompt, systemPrompt, timeoutSeconds: Number(args.timeout), maxTurns: Number(args['max-turns']), maxCost: Number(args['max-cost']) }, seeds: scenario.seeds.map(s => s.to),
+      ...(mode === 'custom' ? { requiredValidators: scenarioChecks(scenario.developmentChecks, cwd) } : {}),
+    }, null, 2));
+    continue;
+  }
+  const isolation = join(base, 'native-service');
+  let service;
+  try {
+    const providerID = args.provider === 'openrouter' ? 'openrouter' : args.provider;
+    const modelID = args.model;
+    await provisionRuntime({ root:isolation, profile:{
+      model:`${providerID}/${modelID}`,
+      default_agent:'build',
+      plugins:[join(root,'lib/opencode/evaluation-budget-plugin'), ...(mode === 'custom' ? [`${root}/plugins/carthagent`] : [])],
+      skills:mode === 'custom' ? [`${root}/skills`] : [],
+      instructions:[environmentPath],
+    } });
+    service = await ensureService({ root:isolation, env:{
+      OPENROUTER_API_KEY:providerKey,
+      [`${args.provider.toUpperCase().replace(/[^A-Z0-9]/g, '_')}_API_KEY`]:providerKey,
+      CTG_EVALUATION_MAX_TURNS:String(args['max-turns']),
+      CTG_EVALUATION_MAX_COST:String(args['max-cost']),
+    } });
+    process.env.CTG_OPENCODE_URL = service.url;
+    process.env.CTG_OPENCODE_AUTHORIZATION = service.headers.authorization;
+    const serviceClient = OpenCode.make({ baseUrl:service.url, headers:service.headers });
+    const probe = await createSession(service, cwd, { signal:AbortSignal.timeout(15000), body:{ model:{providerID,id:modelID}, title:'CTG evaluation plugin activation probe' } });
+    const pluginDeadline = Date.now() + 15000;
+    while (Date.now() < pluginDeadline) {
+      const inventory = await serviceClient.plugin.list({ location:{ directory:cwd } }, { signal:AbortSignal.timeout(5000) });
+      const guard = inventory.data.find(item => item.id === 'ctg.evaluation-budget');
+      if (guard?.state.status === 'active') break;
+      if (guard?.state.status === 'failed') throw Error(`Evaluation admission guard failed to activate: ${guard.state.error}`);
+      if (Date.now() + 100 >= pluginDeadline) throw Error(`Evaluation admission guard did not activate for probe session ${probe.id}`);
+      await new Promise(resolve => setTimeout(resolve, 100));
+    }
+    const result = await runNativeEvaluation({ directory: cwd, systemPrompt, prompt: scenario.prompt, model: args.model, provider: args.provider, timeoutSeconds: Number(args.timeout), maxTurns: Number(args['max-turns']), maxCost: Number(args['max-cost']), signal: controller.signal, onEvent: event => appendFileSync(join(base, `${mode}-events.jsonl`), `${JSON.stringify(event)}\n`) });
+  const summary = {
+    mode,
+    task: scenarioName,
+    oracleGuided: mode === 'custom',
+    provider: args.provider,
+    model: args.model,
+    cwd,
+    evidenceSchema: 2,
+    runtime: NATIVE_EVALUATION_RUNTIME,
+    runtimeVersion: result.opencodeVersion,
+    exitCode: result.exitCode,
+    timedOut: result.timedOut,
+    budgetReason: result.budgetReason,
+    outcome: result.outcome,
+    interruptionStatus: result.interruptionStatus,
+    sessionInfo: { costScope:'session', usageSource:'session.get', terminalAuthority:'session.outcome', outcome:result.outcome },
+    turns: result.turns,
+    productiveTurns: result.productiveTurns,
+    providerErrors: result.providerErrors,
+    endedOnProviderError: result.lastStop === 'error',
+    reportedCost: result.cost,
+    costEvidence: 'native-session-usage; fail-closed when absent',
+    reportedTokens: result.tokens,
+    lastStop: result.lastStop,
+    deliveryStatus: result.deliveryStatus,
+    errors: result.errors,
+    tools: result.tools,
+    sessionID: result.sessionID,
+    campaignComparability: 'not-comparable-to-pi-runtime',
+    durationMs: result.durationMs,
+  };
+  summary.developmentChecks = await runScenarioChecks(mode, cwd, 'development', scenario.developmentChecks);
+  summary.holdoutChecks = await runScenarioChecks(mode, cwd, 'holdout', scenario.holdoutChecks);
+  summary.finalChecks = [...summary.developmentChecks, ...summary.holdoutChecks];
+  summary.externalGrade = {
+    code: summary.holdoutChecks.some(check => check.code !== 0 || check.timedOut) ? 1 : 0,
+    output: summary.holdoutChecks.map(check => `# ${check.id}\n${check.output}`).join('\n'),
+  };
+  summaries.push(summary);
+  atomicJson(join(base, 'summary.json'), summaries);
+  console.log(JSON.stringify(summary, null, 2));
+  } finally {
+    try {
+      if (service) await stopService({ root:isolation });
+    } finally {
+      delete process.env.CTG_OPENCODE_URL;
+      delete process.env.CTG_OPENCODE_AUTHORIZATION;
+    }
+  }
+}
+console.log(`Evidence: ${join(base, 'summary.json')}`);
+if (summaries.some(s =>
+  s.exitCode !== 0 ||
+  (s.endedOnProviderError && s.deliveryStatus !== 'verified') ||
+  (s.mode === 'custom' && s.deliveryStatus !== 'verified') ||
+  s.timedOut ||
+  s.budgetReason ||
+  s.finalChecks?.some(check => check.code !== 0 || check.timedOut)
+)) process.exitCode = 1;

package/extensions/codegraph.ts

@@ -0,0 +1,36 @@
+import { truncateTail, type ExtensionAPI, type ExtensionContext } from '@earendil-works/pi-coding-agent';
+import { createCodeIndex } from '../lib/codegraph.mjs';
+import { boundedCodeNavResult } from '../lib/codegraph-query.mjs';
+
+type CodeIndex = Awaited<ReturnType<typeof createCodeIndex>>;
+export default function codegraph(pi: ExtensionAPI) {
+  pi.registerFlag('codegraph-max-files', { description: 'Positive maximum source files indexed by code_nav (default 8000)', type: 'string', default: '8000' });
+  const indexes = new Map<string, Promise<CodeIndex>>();
+  const indexFor = (ctx: ExtensionContext) => {
+    let pending = indexes.get(ctx.cwd);
+    if (!pending) {
+      const raw = pi.getFlag('codegraph-max-files') ?? '8000';
+      if (!/^\d+$/.test(String(raw)) || !Number.isSafeInteger(Number(raw)) || Number(raw) < 1) throw Error('codegraph-max-files must be a positive safe integer');
+      pending = createCodeIndex(ctx.cwd, { maxFiles: Number(raw) });
+      pending.catch(() => indexes.delete(ctx.cwd));
+      indexes.set(ctx.cwd, pending);
+    }
+    return pending;
+  };
+  pi.registerTool({
+    name: 'code_nav', label: 'Code navigation',
+    description: 'Read-only structural graph for JS/TS/Python/Rust. Use definition/symbols/outline to locate code, callers/importers before changing APIs, then targeted source reads. Calls are name-based candidates, not resolved bindings. Results include coverage, total and nextOffset; page or filter incomplete answers. path filters definitions/symbols or caller call-site files; container filters enclosing classes.',
+    parameters: { type: 'object', required: ['op'], additionalProperties: false, properties: {
+      op: { type: 'string', enum: ['definition', 'symbols', 'callers', 'callees', 'imports', 'importers', 'outline', 'status'] },
+      name: { type: 'string' }, path: { type: 'string' }, container: { type: 'string' },
+      limit: { type: 'integer', minimum: 1, maximum: 200, default: 50 }, offset: { type: 'integer', minimum: 0, default: 0 },
+    } },
+    async execute(_id, params: { op: string; name?: string; path?: string; container?: string; limit?: number; offset?: number }, _signal: unknown, _update: unknown, ctx: ExtensionContext) {
+      const index = await indexFor(ctx);
+      await index.refresh();
+      const serialized = boundedCodeNavResult(index, params);
+      const output = truncateTail(serialized, { maxBytes: 24000, maxLines: 400 });
+      return { content: [{ type: 'text' as const, text: output.content + (output.truncated ? '\n[Oversized record truncated; narrow query.]' : '') }], details: {} };
+    },
+  });
+}