testronaut 1.10.2 → 1.12.0

package/README.md

@@ -374,6 +374,204 @@ Notes:
 
 ---
 
+## 🔬 Experimental Jev shadow mode
+
+Jev shadow mode evaluates each agent turn without changing browser actions,
+mission status, or exit codes. Its typed recommendations, latency, model, and
+token usage are written to each step in the JSON report. A run-level
+`jevShadow` summary aggregates completed calls, errors, latency, and tokens.
+
+Enable it with a TypeSafe API key:
+
+```bash
+export TYPESAFE_API_KEY=YOUR_KEY
+TESTRONAUT_JEV_SHADOW=1 testronaut login.mission.js
+```
+
+Optional overrides:
+
+```bash
+export TESTRONAUT_JEV_MODEL=jev-latest
+export TESTRONAUT_JEV_TIMEOUT_MS=5000
+export TESTRONAUT_JEV_API_BASE=https://api.typesafe.ai
+```
+
+The shadow request contains a bounded representation of the mission, assistant
+response, redacted tool arguments, recent events, Ground Control state, and
+structured browser evidence derived from DOM snapshots already collected by
+Testronaut. Browser evidence includes the query-free URL, title, headings,
+control labels, element counts, and value-free live control flags (`hasValue`,
+`checked`, `disabled`, `readOnly`, and `visible`). It excludes raw DOM, body
+text, input values and value lengths, URL queries, and URL fragments. Because this data is sent to a
+third-party service, enable the experiment only for test data appropriate for that service.
+API errors and timeouts are recorded as shadow telemetry and do not stop the
+mission.
+
+### Experimental completion gate
+
+After validating shadow results, opt into early completion with:
+
+```bash
+TESTRONAUT_JEV_GATE=1 testronaut login.mission.js
+```
+
+The gate defaults to an 80% threshold and requires all three independent Jev
+signals—mission completion, complete progress, and finish-next probability—to
+meet it. It also requires structured browser evidence and a clean tool turn.
+Before returning success, Testronaut captures a final screenshot when
+screenshots are enabled and records completion telemetry. Jev errors, missing
+evidence, incomplete resource guards, and screenshot failures fail open to the
+ordinary agent loop.
+
+Use a different experimental threshold with, for example:
+
+```bash
+TESTRONAUT_JEV_GATE=1 TESTRONAUT_JEV_GATE_THRESHOLD=0.85 testronaut login.mission.js
+```
+
+JSON reports retain the full Jev evaluation telemetry. HTML reports show the
+models used and input/output token totals at run, mission, and turn levels
+without displaying internal strategy counts or estimated cost. The JSON cost
+estimate defaults to Jev's current `$0.042` per million input tokens. Override
+the reporting rate without changing API behavior:
+
+```bash
+TESTRONAUT_JEV_INPUT_USD_PER_MILLION=0.042 testronaut
+```
+
+If the gate is enabled without an API key, the report highlights every skipped
+evaluation. Ensure `TYPESAFE_API_KEY` or `TESTRONAUT_JEV_API_KEY` is exported in
+the same process that launches Testronaut.
+
+### Additional Jev savings experiments
+
+Enable the batched shadow strategy bundle without changing mission behavior:
+
+```bash
+TESTRONAUT_JEV_STRATEGIES=1 testronaut
+```
+
+The same Jev request then records:
+
+- relevant structured DOM regions for future context filtering;
+- a choice among controls already enumerated by Testronaut;
+- deterministic, fast-model, full-model, observe, or finish routing;
+- postcondition verification probability;
+- continue, retry, reobserve, escalate, or stop recovery policy;
+- mission stage and prerequisite readiness.
+
+Strategy results and aggregates appear in JSON reports. They remain advisory
+unless a separate experiment such as dynamic routing is explicitly enabled.
+
+### Stable Jev guardrails
+
+Run Jev completion and evidence guardrails without allowing any model switch:
+
+```bash
+export TESTRONAUT_JEV_API_KEY="your-key"
+testronaut --guardrails
+```
+
+This profile is fail-open: if Jev is unavailable, times out, or lacks sufficient
+evidence, the primary model continues normally. It does not request the larger
+experimental strategy question set and ignores fast-model routing configuration.
+The default completion probability is 0.8.
+
+```json
+{
+  "guardrails": {
+    "enabled": true,
+    "mode": "live",
+    "completionProbability": 0.8
+  }
+}
+```
+
+Guardrails are opt-in and use a user-supplied Jev key. The earlier
+`--optimize=tokens --optimization-mode=live` and nested optimization config
+remain supported as compatibility forms. Run `testronaut config --json` to
+check credential readiness without printing the key.
+
+Reports expose stable `jevGuardrails` telemetry separately from beta
+`modelRouting` telemetry. The legacy `jevShadow` field remains for compatibility.
+
+### Beta dynamic model routing
+
+Choose an optimization objective and whether it runs in shadow or live mode:
+
+```bash
+testronaut --optimize=cost --optimization-mode=live --beta-model-routing
+testronaut --optimize=tokens --optimization-mode=shadow
+testronaut --optimize=balanced --optimization-mode=live --beta-model-routing
+testronaut --optimize=off
+```
+
+The supported objectives are `off`, `cost`, `tokens`, `balanced`, and `speed`.
+`cost` requests cheaper family-local models; `tokens` uses stable guardrails
+without model switching; `balanced` combines guardrails and beta routing; and
+`speed` requests beta fast-model routing. The `cost`, `balanced`, and `speed`
+objectives cannot switch models unless beta routing is explicitly enabled.
+When an objective is supplied without a mode, it defaults to `shadow`.
+
+The equivalent `testronaut-config.json` configuration is:
+
+```json
+{
+  "optimization": {
+    "objective": "balanced",
+    "mode": "live",
+    "guardrails": {
+      "completionProbability": 0.8,
+      "minimumRoutingProbability": 0.4
+    },
+    "beta": {
+      "modelRouting": true
+    }
+  }
+}
+```
+
+CLI flags override configuration. Existing `TESTRONAUT_JEV_*` variables remain
+available as lower-level experimental overrides.
+
+Beta dynamic routing uses a high-confidence Jev `fast_model` recommendation from the
+completed turn to choose the model for the next turn. Start in shadow mode to
+record eligibility without changing models:
+
+```bash
+TESTRONAUT_JEV_ROUTING=shadow TESTRONAUT_JEV_FAST_MODEL=gpt-5.6-luna testronaut
+```
+
+After reviewing the recommendations, enable one-turn live routing:
+
+```bash
+TESTRONAUT_JEV_ROUTING=live TESTRONAUT_JEV_FAST_MODEL=gpt-5.6-luna testronaut
+```
+
+The family-local defaults are `gpt-5.6-luna` for OpenAI,
+`gemini-3.5-flash-lite` for Gemini, and `claude-haiku-4-5` for Anthropic. An
+explicit `TESTRONAUT_JEV_FAST_MODEL` must still belong to the active provider
+family. Routing uses Jev's selected `fast_model` probability with a default
+threshold of 40%; change it with `TESTRONAUT_JEV_ROUTING_THRESHOLD`.
+Authentication, completion, failed,
+uncertain recovery, and low-confidence turns stay on the primary model. If the
+routed request fails—or a routed action requires a retry—Testronaut retries on
+the primary model. Reports
+record the actual model and tokens for every turn and split aggregate tokens by
+model.
+
+Evaluate risk-based suite selection separately as a dry run:
+
+```bash
+TYPESAFE_API_KEY=YOUR_KEY testronaut select --base=origin/main
+```
+
+This reads the git diff, asks one bounded relevance question per discovered
+mission, and prints selected mission filenames without running or skipping any
+tests. Use repeated `--changed-file=path` arguments instead of a git diff when
+testing a hypothetical change. The default selection threshold is 70%; override
+it with `TESTRONAUT_JEV_SELECTION_THRESHOLD`.
+
 ## 📋 Reports
 
 Testronaut generates JSON, HTML, and run-specific screenshots under the configured `outputDir` (default):

package/TODO.md

@@ -20,6 +20,11 @@
 - [ ] Design an optional mission scope/side-effect policy that agents can enforce and reports can display.
 - [ ] Evaluate semantic DOM diffs/chunks after configurable limits, selector scoping, and table trimming have production data.
 
+## Optimization and Rate-Limit Follow-Ups
+
+- [ ] Run a history-informed model-routing benchmark before expanding cross-provider routing. Give Jev versioned, per-model/per-mission-class reliability, token, latency, and price observations from prior runs; require a configurable reliability floor; compare against the best fixed-model-plus-guardrails control with blocked repetitions; keep selection at the mission boundary; record the evidence and reason for every choice; and define cold-start, stale-data, low-confidence, and provider-failure fallbacks. Do not retry a failed mission on another provider in the benchmark because that would confound routing quality with retry recovery.
+- [ ] Add an opt-in CLI/config policy that reroutes an imminent throttled LLM request to a compatible model as an alternative to the default backoff behavior. Keep backoff as the default; use observed TPM/RPM headroom and projected request tokens; constrain failover to explicitly configured provider/model families; preserve tool/history compatibility; prevent routing loops; fall back safely when the alternate model is unavailable; and record the original model, routed model, trigger, token usage, latency, and outcome in reports.
+
 ## Automated MFA Follow-Ups
 
 - [ ] Add an integration smoke test against a local or staging API fixture for `get_mfa_code`.

package/bin/cli.js

@@ -48,7 +48,7 @@ import fetch from 'node-fetch';
 import crypto from 'crypto';
 import http from 'http';
 import { resolveProviderModel } from '../llm/modelResolver.js';
-import { exec as execCmd } from 'child_process';
+import { exec as execCmd, execFileSync } from 'child_process';
 import { promisify } from 'util';
 const exec = promisify(execCmd);
 import url from 'url';
@@ -57,6 +57,11 @@ import { discoverMissionFiles } from '../core/missionDiscovery.js';
 import { loadMissionModule } from '../core/missionLoader.js';
 import { loadConfig } from '../core/config.js';
 import { matchesTagFilter, normalizeTagMatch, normalizeTags } from '../core/tags.js';
+import { createTerminalUi } from './terminalUi.js';
+import { summarizeJevUsage } from '../core/jevUsage.js';
+import { evaluateJevShadow, resolveJevShadowConfig } from '../core/jevShadow.js';
+import { buildJevSuiteSelectionRequest, interpretJevSuiteSelection } from '../core/jevSuiteSelection.js';
+import { applyOptimizationPolicy, OPTIMIZATION_MODES, OPTIMIZATION_OBJECTIVES, resolveOptimizationPolicy } from '../core/optimization.js';
 
 // Keep PW browsers inside the project to avoid global cache skew
 process.env.PLAYWRIGHT_BROWSERS_PATH = process.env.PLAYWRIGHT_BROWSERS_PATH || '0';
@@ -78,6 +83,7 @@ const jsonOutput = args.includes('--json');
 const quietOutput = args.includes('--quiet') || jsonOutput;
 args = args.filter(arg => arg !== '--json' && arg !== '--quiet');
 const writeOutput = value => process.stdout.write(`${typeof value === 'string' ? value : JSON.stringify(value, null, 2)}\n`);
+const terminalUi = createTerminalUi({ quiet: quietOutput });
 if (quietOutput) {
   console.log = () => {};
   console.warn = () => {};
@@ -358,6 +364,7 @@ export const __test__ = {
   createVercelBypassHeader,
   parseProviderArgs,
   parseRunOptionsArgs,
+  parseOptimizationArgs,
   detectCliName,
   isDirectInvocation,
   resolveMissionPath,
@@ -387,6 +394,7 @@ function resolveReportFile(requested, reportDir, cwd = process.cwd()) {
 
 function buildEffectiveConfig(config = {}, cwd = process.cwd()) {
   const resolved = resolveProviderModel({ cwd });
+  const optimization = resolveOptimizationPolicy({ config });
   const source = key => process.env[key] ? 'environment' : undefined;
   return {
     configFile: path.resolve(cwd, 'testronaut-config.json'),
@@ -399,6 +407,21 @@ function buildEffectiveConfig(config = {}, cwd = process.cwd()) {
     addTags: normalizeTags([...(config.addTags || []), ...cliAddTags]),
     screenshots: screenshotsEnabled,
     authenticated: Boolean(config.sessionToken),
+    optimization,
+    guardrails: {
+      enabled: optimization.objective === 'tokens' && optimization.mode !== 'off',
+      mode: optimization.objective === 'tokens' ? optimization.mode : 'off',
+      completionProbability: optimization.completionProbability,
+      failOpen: true,
+      credentialConfigured: Boolean(resolveJevShadowConfig().apiKey),
+      credentialSource: process.env.TESTRONAUT_JEV_API_KEY
+        ? 'TESTRONAUT_JEV_API_KEY'
+        : process.env.TYPESAFE_API_KEY
+          ? 'TYPESAFE_API_KEY'
+          : process.env.JEV_API_KEY
+            ? 'JEV_API_KEY'
+            : null,
+    },
   };
 }
 
@@ -506,6 +529,56 @@ function parseTagArgs(argsList) {
 }
 __test__.parseTagArgs = parseTagArgs;
 
+function parseOptimizationArgs(argsList) {
+  const nextArgs = [...argsList];
+  const values = {};
+  let invalid = false;
+  const guardrailsIndex = nextArgs.findIndex(arg => arg === '--guardrails' || arg.startsWith('--guardrails='));
+  if (guardrailsIndex >= 0) {
+    const raw = nextArgs[guardrailsIndex];
+    const inline = raw.includes('=');
+    const following = nextArgs[guardrailsIndex + 1];
+    const consumesFollowing = !inline && ['live', 'shadow', 'off'].includes(String(following || '').toLowerCase());
+    const guardrailsMode = inline
+      ? raw.slice(raw.indexOf('=') + 1).trim().toLowerCase()
+      : consumesFollowing ? following.toLowerCase() : 'live';
+    if (!['live', 'shadow', 'off'].includes(guardrailsMode)) invalid = true;
+    else {
+      values.guardrailsMode = guardrailsMode;
+      values.objective = guardrailsMode === 'off' ? 'off' : 'tokens';
+      values.mode = guardrailsMode;
+    }
+    nextArgs.splice(guardrailsIndex, consumesFollowing ? 2 : 1);
+  }
+  const betaRoutingIndex = nextArgs.indexOf('--beta-model-routing');
+  if (betaRoutingIndex >= 0) {
+    values.betaModelRouting = true;
+    nextArgs.splice(betaRoutingIndex, 1);
+  }
+  const definitions = [
+    { names: ['--optimize'], key: 'objective' },
+    { names: ['--optimization-mode'], key: 'mode' },
+  ];
+  for (const definition of definitions) {
+    const index = nextArgs.findIndex(arg => definition.names.some(name => arg === name || arg.startsWith(`${name}=`)));
+    if (index < 0) continue;
+    const raw = nextArgs[index];
+    const inline = raw.includes('=');
+    const value = inline ? raw.slice(raw.indexOf('=') + 1) : nextArgs[index + 1];
+    if (!value || (!inline && value.startsWith('-'))) invalid = true;
+    else values[definition.key] = value.trim().toLowerCase();
+    nextArgs.splice(index, inline ? 1 : (value && !value.startsWith('-') ? 2 : 1));
+  }
+  const probe = resolveOptimizationPolicy({
+    env: {
+      ...(values.objective ? { TESTRONAUT_OPTIMIZE: values.objective } : {}),
+      ...(values.mode ? { TESTRONAUT_OPTIMIZATION_MODE: values.mode } : {}),
+    },
+  });
+  if (!probe.valid) invalid = true;
+  return { args: nextArgs, ...values, invalid };
+}
+
 // Look for --model=<id> or --model <id>
 let modelOverride;
 const modelFlagIndex = args.findIndex(a => a === '--model' || a.startsWith('--model='));
@@ -532,6 +605,16 @@ if (providerResult.invalid) {
   console.warn('⚠️ Invalid --provider value. Provide a non-empty provider id.');
 }
 args = providerResult.args;
+
+const optimizationArgs = parseOptimizationArgs(args);
+args = optimizationArgs.args;
+if (optimizationArgs.invalid) {
+  console.error(`❌ --optimize must be one of ${OPTIMIZATION_OBJECTIVES.join(', ')} and --optimization-mode must be one of ${OPTIMIZATION_MODES.join(', ')}.`);
+  process.exit(1);
+}
+if (optimizationArgs.objective) process.env.TESTRONAUT_OPTIMIZE = optimizationArgs.objective;
+if (optimizationArgs.mode) process.env.TESTRONAUT_OPTIMIZATION_MODE = optimizationArgs.mode;
+if (optimizationArgs.betaModelRouting) process.env.TESTRONAUT_BETA_MODEL_ROUTING = '1';
 const providerOverride = providerResult.provider;
 if (providerOverride) {
   process.env.TESTRONAUT_PROVIDER = providerOverride.trim();
@@ -754,6 +837,7 @@ Usage:
   ${cliName} <file>         Run a .mission.js or .mission.ts file by name or path
   ${cliName} login          Log in and store session token
   ${cliName} list           List discovered missions and tags without running them
+  ${cliName} select         Ask Jev which missions are relevant to the current git diff (dry-run only)
   ${cliName} config         Show effective configuration and value sources
   ${cliName} upload [file]  Upload the latest report or a selected report file/run ID
   ${cliName} serve        Serve & open the most recent HTML report (read-only)
@@ -765,6 +849,10 @@ Options:
   --debug[=<bool>]          Enable verbose debug logs (or set TESTRONAUT_DEBUG=1)
   --provider=<id>           Override LLM provider (openai, gemini, or anthropic)
   --model=<id>              Override the provider model for this run
+  --optimize=<objective>    Optimization objective: off, cost, tokens, balanced, or speed
+  --optimization-mode=<m>   Optimization control: off, shadow, or live
+  --guardrails[=<mode>]      Opt in to Jev guardrails: live (default), shadow, or off
+  --beta-model-routing      Enable beta model switching for cost, balanced, or speed
   --session=<mode>          Browser auth reuse: isolated (default) or shared-auth
   -o, --options key=value   Set run options, such as mfa=github-test-mfa or inbox=github-staging
   --dev                     Use the staging API base URL
@@ -778,6 +866,8 @@ Options:
   --no-upload-screenshots   Upload report JSON without its screenshots
   --json                    Emit machine-readable command/run output
   --quiet                   Suppress informational logs
+  --base=<git-ref>          Base ref for Jev suite selection (default: HEAD~1)
+  --changed-file=<path>     Explicit changed file for Jev suite selection; repeatable
   --retry_limit=<n>         Override agent turn retry limits (minimum 1, maximum 10)
   --tag=<tag>               Run missions matching a tag; repeat for multiple tags
   --tags=<tag,...>          Compact comma-list form (OR/any by default)
@@ -789,6 +879,7 @@ Examples:
   ${cliName}
   ${cliName} login
   ${cliName} list
+  ${cliName} select --base=origin/main
   ${cliName} upload
   ${cliName} serve
   ${cliName} --init
@@ -862,10 +953,91 @@ if (args.includes('serve') || args.includes('view')) {
 
 const { root: missionsRoot, files: discoveredMissions } = await discoverMissionFiles({ cwd: process.cwd() });
 const tagConfig = await loadConfig(process.cwd());
+const optimizationPolicy = resolveOptimizationPolicy({ config: tagConfig });
+if (!optimizationPolicy.valid) {
+  console.error('❌ Invalid optimization objective or mode in testronaut-config.json.');
+  process.exit(1);
+}
+applyOptimizationPolicy(optimizationPolicy, process.env, {
+  force: Boolean(optimizationArgs.objective || optimizationArgs.mode || optimizationArgs.betaModelRouting),
+});
+const jevPreflight = resolveJevShadowConfig();
+if (optimizationPolicy.objective === 'tokens' && optimizationPolicy.mode !== 'off' && !jevPreflight.apiKey) {
+  console.warn('⚠️ Jev guardrails are enabled but no user-supplied key was found. Set TESTRONAUT_JEV_API_KEY; this run will fail open to the primary model.');
+}
+if (optimizationPolicy.routingRequested && !optimizationPolicy.betaModelRouting) {
+  console.warn('⚠️ Model routing is beta and remains disabled. Add --beta-model-routing or optimization.beta.modelRouting=true to enable it.');
+}
 const outputDir = resolveReportDir(tagConfig);
 process.env.TESTRONAUT_OUTPUT_DIR = outputDir;
 process.env.TESTRONAUT_RUN_ID = runId;
 
+if (args[0] === 'select') {
+  const baseArg = args.find(arg => arg.startsWith('--base='));
+  const base = baseArg ? baseArg.slice('--base='.length) : 'HEAD~1';
+  const explicitChangedFiles = args
+    .filter(arg => arg.startsWith('--changed-file='))
+    .map(arg => arg.slice('--changed-file='.length))
+    .filter(Boolean);
+  let changedFiles = explicitChangedFiles;
+  if (!changedFiles.length) {
+    try {
+      const outputs = [
+        execFileSync('git', ['diff', '--name-only', `${base}...HEAD`], { cwd: process.cwd(), encoding: 'utf8' }),
+        execFileSync('git', ['diff', '--name-only'], { cwd: process.cwd(), encoding: 'utf8' }),
+        execFileSync('git', ['ls-files', '--others', '--exclude-standard'], { cwd: process.cwd(), encoding: 'utf8' }),
+      ];
+      changedFiles = [...new Set(outputs.flatMap(output => output.split(/\r?\n/).filter(Boolean)))].sort();
+    } catch (error) {
+      writeOutput({ mode: 'jev-suite-selection', status: 'error', error: `Could not resolve git diff from ${base}: ${error.message}` });
+      process.exitCode = 1;
+      return;
+    }
+  }
+
+  const missions = [];
+  for (const file of discoveredMissions) {
+    try {
+      const mission = await loadMissionModule(path.resolve(missionsRoot, file));
+      missions.push({ file, tags: normalizeTags(mission.tags) });
+    } catch (error) {
+      missions.push({ file, tags: [], loadError: error.message });
+    }
+  }
+  if (!missions.length) {
+    writeOutput({ mode: 'jev-suite-selection', status: 'error', error: 'No missions were discovered.' });
+    process.exitCode = 1;
+    return;
+  }
+  const request = buildJevSuiteSelectionRequest({ changedFiles, missions });
+  const config = { ...resolveJevShadowConfig(), enabled: true };
+  const evaluation = await evaluateJevShadow(request.state, { config, questions: request.questions });
+  const thresholdRaw = Number(process.env.TESTRONAUT_JEV_SELECTION_THRESHOLD);
+  const threshold = Number.isFinite(thresholdRaw) && thresholdRaw >= 0 && thresholdRaw <= 1 ? thresholdRaw : 0.7;
+  const selection = evaluation.status === 'ok'
+    ? interpretJevSuiteSelection(evaluation.answers, request.catalog, threshold)
+    : [];
+  writeOutput({
+    mode: 'jev-suite-selection',
+    dryRun: true,
+    base,
+    changedFiles,
+    threshold,
+    evaluation: {
+      status: evaluation.status,
+      model: evaluation.model,
+      usage: evaluation.usage,
+      latencyMs: evaluation.latencyMs,
+      reason: evaluation.reason,
+      error: evaluation.error,
+    },
+    missions: selection,
+    selected: selection.filter(mission => mission.selected).map(mission => mission.file),
+  });
+  if (evaluation.status !== 'ok') process.exitCode = 1;
+  return;
+}
+
 if (args.length === 1 && args[0] === 'config') {
   writeOutput(buildEffectiveConfig(tagConfig));
   return;
@@ -879,14 +1051,14 @@ if (args.length === 1 && args[0] === 'list') {
   const listedMissions = [];
   const listTags = cliTags ?? normalizeTags(tagConfig?.tags, { allowUntagged: true });
   const listTagMatch = normalizeTagMatch(cliTagMatch ?? tagConfig?.tagMatch);
-  if (!jsonOutput) console.log(`Missions in ${path.relative(process.cwd(), missionsRoot) || '.'}:`);
+  if (!jsonOutput) terminalUi.phase('preflight', `Missions in ${path.relative(process.cwd(), missionsRoot) || '.'}`);
   for (const file of discoveredMissions) {
     try {
       const mission = await loadMissionModule(path.resolve(missionsRoot, file));
       const tags = normalizeTags(mission.tags);
       const selected = matchesTagFilter(tags, listTags, listTagMatch);
       listedMissions.push({ file, tags, selected });
-      if (!jsonOutput) console.log(`  ${selected ? '✓' : '·'} ${file}${tags.length ? `  [${tags.join(', ')}]` : ''}`);
+      if (!jsonOutput) terminalUi.mission(selected, file, tags);
     } catch (error) {
       console.log(`  ${file}  [could not load: ${error.message}]`);
       process.exitCode = 1;
@@ -943,6 +1115,17 @@ if (dryRun) {
   return;
 }
 
+if (!jsonOutput) {
+  const previewModel = resolveProviderModel({ cwd: process.cwd() });
+  terminalUi.banner({
+    version: CLI_VERSION,
+    provider: previewModel.provider,
+    model: previewModel.model,
+    session: sessionMode,
+  });
+  terminalUi.phase('preflight', `${explicitFiles ? args.length : discoveredMissions.length} mission file(s) queued`);
+}
+
 const runFile = async (filePath) => {
   const modulePath = resolveMissionPath(filePath, { cwd: process.cwd(), missionsRoot });
   try {
@@ -960,6 +1143,7 @@ const runFile = async (filePath) => {
     }
 
     if (typeof missionsModule.executeMission === 'function') {
+      if (!jsonOutput) terminalUi.phase('launch', path.relative(process.cwd(), modulePath) || filePath);
       const result = await missionsModule.executeMission();
       if (result == null) {
         console.error(`❌ Mission did not return a result: ${filePath}`);
@@ -1018,19 +1202,26 @@ const flatMissions = allResults.flatMap(entry => {
         steps = mergeDuplicateTurns(fromJsonl);
       }
     }
-    return {
+    const mission = {
       ...m,
       steps,
       file: entry.file,
     };
+    const jevShadow = summarizeJevUsage(steps);
+    return jevShadow.evaluations ? { ...mission, jevShadow } : mission;
   });
 });
 
+const allMissionSteps = flatMissions
+  .flatMap(mission => Array.isArray(mission.steps) ? mission.steps : []);
+const jevShadowUsage = summarizeJevUsage(allMissionSteps);
+
 // Read provider/model from config (allow env override)
 const { provider: llmProvider, model: llmModel } = resolveProviderModel({ cwd: process.cwd() });
 
 
 const report = {
+  schemaVersion: 2,
   runId,
   cli: { version: CLI_VERSION },
   startTime: startTime.toISOString(),
@@ -1040,6 +1231,7 @@ const report = {
     model: llmModel,
   },
   session: { mode: sessionMode },
+  optimization: optimizationPolicy,
   summary: {
     totalMissions: flatMissions.length,
     passed: flatMissions.filter(m => m.status === 'passed').length,
@@ -1047,14 +1239,51 @@ const report = {
   },
   missions: flatMissions
 };
+if (process.env.TESTRONAUT_EXPERIMENT_ID) {
+  report.experiment = {
+    id: process.env.TESTRONAUT_EXPERIMENT_ID,
+    condition: process.env.TESTRONAUT_EXPERIMENT_CONDITION || null,
+    block: Number(process.env.TESTRONAUT_EXPERIMENT_BLOCK) || null,
+    unit: process.env.TESTRONAUT_EXPERIMENT_UNIT || null,
+    selectedBy: process.env.TESTRONAUT_EXPERIMENT_SELECTED_BY || null,
+    selectionScore: process.env.TESTRONAUT_EXPERIMENT_SELECTION_SCORE === ''
+      ? null
+      : Number(process.env.TESTRONAUT_EXPERIMENT_SELECTION_SCORE),
+  };
+}
+if (jevShadowUsage.evaluations) {
+  const gateMode = /^(1|true|yes|on)$/i.test(String(process.env.TESTRONAUT_JEV_GATE || ''));
+  report.jevShadow = {
+    mode: gateMode ? 'completion-gate' : 'shadow',
+    ...jevShadowUsage,
+  };
+  const { strategies, ...guardrailUsage } = jevShadowUsage;
+  report.jevGuardrails = {
+    mode: gateMode ? 'live' : 'shadow',
+    completionProbability: optimizationPolicy.completionProbability,
+    failOpen: true,
+    ...guardrailUsage,
+  };
+  if (optimizationPolicy.routingRequested) {
+    report.modelRouting = {
+      stability: 'beta',
+      optedIn: optimizationPolicy.betaModelRouting,
+      active: optimizationPolicy.routing,
+      mode: optimizationPolicy.routing ? optimizationPolicy.mode : 'off',
+      strategies,
+    };
+  }
+}
 report.tags = normalizeTags(flatMissions.flatMap(m =>
   m.submissionType === 'mission' ? (m.tags ?? []) : []
 ));
 
 fs.mkdirSync(outputDir, { recursive: true });
 fs.writeFileSync(path.join(outputDir, `${runId}.json`), JSON.stringify(report, null, 2));
-generateHtmlReport(report, path.join(outputDir, `${runId}.html`));
+const htmlReportPath = path.join(outputDir, `${runId}.html`);
+generateHtmlReport(report, htmlReportPath);
 if (jsonOutput) writeOutput(report);
+else terminalUi.summary({ ...report.summary, reportPath: path.relative(process.cwd(), htmlReportPath) });
 
   try {
     if (!process.env.TN_KEEP_TMP && fs.existsSync(TMP_DIR)) {

package/bin/terminalUi.js

@@ -0,0 +1,51 @@
+const ANSI = {
+  reset: '\u001b[0m',
+  bold: '\u001b[1m',
+  dim: '\u001b[2m',
+  orange: '\u001b[38;5;208m',
+  cyan: '\u001b[38;5;81m',
+  teal: '\u001b[38;5;43m',
+  red: '\u001b[38;5;203m',
+  cream: '\u001b[38;5;230m',
+};
+
+export function shouldUseColor(stream = process.stdout, env = process.env) {
+  return Boolean(stream?.isTTY) && !('NO_COLOR' in env) && env.TERM !== 'dumb';
+}
+
+export function createTerminalUi({
+  stream = process.stdout,
+  color = shouldUseColor(stream),
+  quiet = false,
+} = {}) {
+  const paint = (codes, value) => color ? `${codes}${value}${ANSI.reset}` : String(value);
+  const write = value => {
+    if (!quiet) stream.write(`${value}\n`);
+  };
+  const label = value => paint(`${ANSI.bold}${ANSI.orange}`, value.padEnd(10));
+
+  return {
+    banner({ version, provider, model, session }) {
+      write(paint(`${ANSI.bold}${ANSI.cream}`, '🧑‍🚀 TESTRONAUT  /  MISSION RUNNER'));
+      write(paint(ANSI.dim, `   v${version}  •  ${provider}/${model}  •  ${session}`));
+      write(paint(ANSI.dim, '   ─────────────────────────────────────────'));
+    },
+    phase(name, message) {
+      write(`${label(name.toUpperCase())}${paint(ANSI.cyan, message)}`);
+    },
+    mission(selected, file, tags = []) {
+      const marker = selected ? paint(ANSI.teal, '●') : paint(ANSI.dim, '○');
+      const tagText = tags.length ? paint(ANSI.dim, `  ${tags.map(tag => `#${tag}`).join(' ')}`) : '';
+      write(`   ${marker} ${file}${tagText}`);
+    },
+    summary({ totalMissions = 0, passed = 0, failed = 0, reportPath }) {
+      const status = failed === 0
+        ? paint(`${ANSI.bold}${ANSI.teal}`, 'MISSION COMPLETE')
+        : paint(`${ANSI.bold}${ANSI.red}`, 'MISSION NEEDS REVIEW');
+      write('');
+      write(`${label('COMPLETE')}${status}`);
+      write(`   ${paint(ANSI.teal, `${passed} passed`)}  ${paint(failed ? ANSI.red : ANSI.dim, `${failed} failed`)}  ${paint(ANSI.dim, `${totalMissions} total`)}`);
+      if (reportPath) write(paint(ANSI.dim, `   Report: ${reportPath}`));
+    },
+  };
+}

package/core/agent.js

@@ -259,6 +259,7 @@ Email invitation and magic links:
           steps,
           stepsArchive,
           missionName,
+          goal: userContent,
           retryLimit, 
           groundControl,
           resourceGuard: opts.resourceGuard,