evals 2.4.0 → 2.6.0

package/README.md

@@ -1,6 +1,6 @@
 # evals
 
-Go from zero to your first [Arize AX](https://arize.com/docs/ax) traces in one command. `evals` launches your coding agent, seeded with a guided onboarding prompt that walks you through signing up, instrumenting your app, and confirming traces land.
+Go from zero to your first [Arize AX](https://arize.com/docs/ax) traces in one command. `evals` launches your coding agent, seeded with a guided onboarding prompt that walks you through signing up, adding tracing, and confirming traces land — for your app, or for the coding agent itself.
 
 ## Quick start
 
@@ -12,9 +12,11 @@ You'll see a picker of the coding agents installed on your machine (Claude Code,
 
 No coding agent installed? The picker shows install links instead.
 
+Needs **Node.js 20 or newer** (the Ink UI does). On anything older, `npx evals` says so and points you at the shell launcher below rather than failing obscurely.
+
 ### Without Node
 
-If you don't have Node, use the shell launchers (they detect an agent and fetch the prompt from this published package via jsDelivr):
+If you don't have Node — or have one older than 20 — use the shell launchers (they detect an agent and fetch the prompt from this published package via jsDelivr):
 
 ```bash
 # macOS / Linux
@@ -26,13 +28,15 @@ bash <(curl -fsSL https://cdn.jsdelivr.net/npm/evals/start.sh)
 irm https://cdn.jsdelivr.net/npm/evals/start.ps1 | iex
 ```
 
-When Node **is** present, these hand off to `npx evals` for the richer UI.
+When **Node 20+** is present, these hand off to `npx evals` for the richer UI. On an older Node they say so and carry on in the shell, which needs no Node at all.
 
 ## What it does
 
 1. Detects an installed coding agent (never executes it — just a PATH lookup).
 2. Launches the agent in your current directory, seeded with the bundled onboarding prompt ([`onboarding-prompt.md`](./onboarding-prompt.md)).
-3. The agent walks you through: create/sign in to Arize AX → detect your stack → instrument it → verify your first traces.
+3. The agent walks you through: create/sign in to Arize AX → pick what to trace → instrument it → verify your first traces.
+
+You can trace an existing app (the one in the current folder, or another one you point it at by path), a starter app the agent creates, or **the coding agent itself** — so every session you run, in any project, shows up in Arize AX. Agent tracing is machine-wide and captures prompts and tool output by default, so the prompt asks for that explicitly and offers per-category opt-outs.
 
 The agent runs with its **normal permission model** — `evals` never passes skip-permissions, so you approve each step, and the prompt itself gates real changes on your confirmation.
 
@@ -43,26 +47,16 @@ The agent runs with its **normal permission model** — `evals` never passes ski
 | `ARIZE_AGENT=<id>` | `start.sh` / `start.ps1` | Skip the picker and use this agent (`claude`, `codex`, `cursor-agent`, `copilot`, `gemini`). |
 | `ARIZE_SKIP_NPX=1` | `start.sh` / `start.ps1` | Force the shell path even when Node/npx is available. |
 | `ARIZE_PROMPT_URL=<url>` | `start.sh` / `start.ps1` | Fetch the onboarding prompt from a custom URL (supports `file://`). |
+| `ARIZE_ONBOARDING_DIR=<dir>` | all three | Stage the prompt somewhere other than `~/.arize/onboarding`. |
 
 The shell launchers also accept `--agent <id>`.
 
-## Development
-
-```bash
-npm install
-npm test        # unit tests for the pure helpers (node --test)
-node cli.js     # run the interactive CLI locally
-```
-
-`npm test` covers the headless logic (URL tagging, agent detection, the agent catalog, and the picker). The interactive Ink flow and the live agent launch need a real terminal — verify those by running `node cli.js` in a terminal and picking an agent.
-
-### Layout
-
-- `cli.js` — the interactive `npx evals` entry point (Ink UI). Exports its pure helpers for testing; only launches the UI when run directly.
-- `onboarding-prompt.md` — the bundled onboarding prompt (a copy of the Arize docs landing-page prompt).
-- `start.sh` / `start.ps1` — the no-npm launchers, served to `curl`/`irm` via jsDelivr.
-- `test/` — unit tests.
+The prompt is staged in `~/.arize/onboarding` and handed to your agent by path, since it's too large to pass as a command-line argument. That directory is cleared at the start of each launch and holds nothing but the prompt and the offline tracing bundle. If your home directory isn't writable, the launcher says so and falls back to a temp directory — instrumenting an app still works, but tracing the coding agent itself needs a writable home.
 
 ## About Arize
 
 [Arize AX](https://arize.com/docs/ax) is the AI engineering platform for tracing, evaluating, and observing LLM and agent applications. Learn more at [arize.com](https://arize.com).
+
+## Contributing
+
+Development setup and the release process live in [CONTRIBUTING.md](https://github.com/Arize-ai/npm-evals/blob/main/CONTRIBUTING.md).

package/bin.js

@@ -0,0 +1,43 @@
+#!/usr/bin/env node
+//
+// Node version gate for `npx evals`.
+//
+// This is a separate file from cli.js on purpose. ESM `import` statements are
+// hoisted and evaluated before any of the importing module's own code runs, so a
+// check at the top of cli.js would execute *after* Ink's dependency graph loads —
+// too late. On Node 18 that graph dies while parsing string-width with
+// "SyntaxError: Invalid regular expression flags" (the regex `v` flag is Node 20+),
+// which tells the user nothing about Node versions. So: check here, where nothing
+// but Node builtins have loaded, then reach cli.js through a dynamic import.
+//
+// Keep this file dependency-free and conservative in syntax — it has to parse and
+// run on the very versions it exists to reject.
+//
+// MIN_MAJOR tracks ink's own `engines.node`, and package.json's `engines` field
+// must say the same thing; test/cli.test.js asserts they agree.
+
+const MIN_MAJOR = 20;
+
+const major = Number.parseInt(process.versions.node.split('.')[0], 10);
+
+if (!Number.isInteger(major) || major < MIN_MAJOR) {
+  const shell = process.platform === 'win32'
+    ? 'irm https://cdn.jsdelivr.net/npm/evals/start.ps1 | iex'
+    : 'bash <(curl -fsSL https://cdn.jsdelivr.net/npm/evals/start.sh)';
+
+  console.error(`evals needs Node.js ${MIN_MAJOR} or newer — this is ${process.version}.`);
+  console.error('');
+  console.error('Either upgrade Node (https://nodejs.org), or use the launcher that');
+  console.error('needs no Node at all — it runs the same onboarding flow:');
+  console.error('');
+  console.error(`  ${shell}`);
+  process.exit(1);
+}
+
+// cli.js only auto-runs when it is the entry point, so call main() explicitly.
+import('./cli.js')
+  .then((cli) => cli.main())
+  .catch((err) => {
+    console.error(`Could not start evals: ${err && err.message ? err.message : err}`);
+    process.exit(1);
+  });

package/cli.js

@@ -4,28 +4,198 @@ import React, { useState, useEffect } from 'react';
 import { render, Box, Text, useInput, useApp, Static } from 'ink';
 import Gradient from 'ink-gradient';
 import { exec, spawn } from 'child_process';
-import { existsSync, readFileSync, writeFileSync, mkdtempSync, realpathSync } from 'fs';
-import { join } from 'path';
-import { tmpdir } from 'os';
+import {
+  existsSync,
+  readFileSync,
+  writeFileSync,
+  mkdtempSync,
+  mkdirSync,
+  realpathSync,
+  readdirSync,
+  copyFileSync,
+  chmodSync,
+  renameSync,
+  rmSync,
+  rmdirSync,
+  unlinkSync,
+} from 'fs';
+import { join, isAbsolute, resolve } from 'path';
+import { tmpdir, homedir } from 'os';
 import { fileURLToPath } from 'url';
 
 const e = React.createElement;
 
-// The onboarding prompt is bundled with this package (onboarding-prompt.md, a
-// copy of the docs landing-page prompt). We read it, write it to a temp file,
-// and tell the agent to read that file — a ~27 KB prompt is too large to pass
-// reliably as a command-line argument.
-// TODO: sync this copy with the docs source (arize.com/docs) later.
+// The onboarding prompt is bundled with this package (onboarding-prompt.md).
+// This repo is its only home — there is no docs original to sync against, so
+// edit it here. We read it, write it to a temp file, and tell the agent to read
+// that file — a ~35 KB prompt is too large to pass reliably as a command-line
+// argument.
 const BUNDLED_PROMPT_PATH = fileURLToPath(new URL('./onboarding-prompt.md', import.meta.url));
 
-// Write the bundled prompt to a temp file and return a short seed instruction
-// that points the agent at it.
+// Wheels for the coding-agent tracing harness, built by scripts/build-harness-wheel.mjs.
+// Staging these next to the prompt lets Step 4A install with no network and no
+// remote code execution — which is what keeps it working under permission
+// classifiers that block piping a downloaded script into a shell.
+const VENDOR_DIR = fileURLToPath(new URL('./vendor', import.meta.url));
+const OFFLINE_DIR_NAME = 'arize-offline';
+
+// Files the offline install needs. Absent or incomplete vendor/ means we stage
+// nothing — the prompt keys off the directory existing, so a half-populated one
+// would send the agent down the offline path with no wheel to install.
+const OFFLINE_FILES = ['harness-install.sh', 'harness-install.bat', 'LICENSE-coding-harness-tracing'];
+
+const PROMPT_FILE_NAME = 'onboarding-prompt.md';
+
+// Step 4A writes the harness credentials here and deletes them itself. We name it
+// only so a stray one — left by a run that died between writing and deleting —
+// gets cleared at the next launch instead of sitting around holding a live key.
+const ENV_FILE_NAME = 'harness.env';
+
+// MANIFEST only lands in the shell launchers' bundles, never in ours. Naming it
+// anyway keeps one allowlist correct for every implementation, so a directory
+// staged by start.sh can be cleared by `npx evals` and vice versa.
+const BUNDLE_EXTRA_FILES = ['MANIFEST'];
+
+// Where the prompt and the offline bundle get staged: `~/.arize/onboarding`.
+// Deliberately stable rather than a fresh temp directory. The agent is asked to
+// read a file outside its workspace, and that is an approval a human can grant
+// with confidence for `~/.arize/onboarding/onboarding-prompt.md`, where a
+// `/var/folders/_s/3_t5nrxs…/T/arize-onboarding-c5uxb8/` path just looks alarming.
+//
+// This sits beside Step 4A's own `~/.arize/harness` install, which the launcher
+// must never touch — clearing our staging directory cannot uninstall a working
+// harness, and that boundary is why we only ever name files under `onboarding/`.
+export function resolveOnboardingDir({
+  home = homedir(),
+  override = process.env.ARIZE_ONBOARDING_DIR,
+} = {}) {
+  if (override) return resolve(override);
+  return join(home, '.arize', 'onboarding');
+}
+
+// Clear a staged bundle directory by naming every entry we could have put in it,
+// then rmdir. Bails out — deleting nothing further — the moment it meets an entry
+// it doesn't recognise. Wheel names carry a version we can't know ahead of time,
+// so `.whl` is matched by extension, non-recursively, inside this one directory.
+function clearBundleDir(dir) {
+  for (const entry of readdirSync(dir, { withFileTypes: true })) {
+    const known =
+      entry.name.endsWith('.whl') ||
+      OFFLINE_FILES.includes(entry.name) ||
+      BUNDLE_EXTRA_FILES.includes(entry.name);
+    if (entry.isDirectory() || !known) return false;
+    unlinkSync(join(dir, entry.name));
+  }
+  rmdirSync(dir);
+  return true;
+}
+
+// Empty the staging directory by deleting the exact files we create — never with
+// a recursive delete. Two things fall out of that: a bad path can at worst try to
+// unlink a handful of names that won't exist, and `rmdir` refuses a non-empty
+// directory, so anything unexpected in there stops us instead of being destroyed.
+// Unlinking a symlink removes the link and never the target.
+//
+// Returns true when the directory is gone or empty and safe to stage into.
+export function clearOnboardingDir(dir) {
+  if (!isAbsolute(dir)) return false;
+  if (!existsSync(dir)) return true;
+
+  try {
+    for (const entry of readdirSync(dir, { withFileTypes: true })) {
+      const path = join(dir, entry.name);
+
+      if (entry.isDirectory()) {
+        // arize-offline/ is ours; .staging-* is the leftover of a crashed run.
+        const ours = entry.name === OFFLINE_DIR_NAME || entry.name.startsWith('.staging-');
+        if (!ours || !clearBundleDir(path)) return false;
+        continue;
+      }
+
+      if (entry.name !== PROMPT_FILE_NAME && entry.name !== ENV_FILE_NAME) return false;
+      unlinkSync(path);
+    }
+    return true;
+  } catch {
+    return false;
+  }
+}
+
+// Clear and create the staging directory, degrading to a temp directory when the
+// home path can't be used: a read-only home, a directory owned by another user
+// after a past `sudo npx evals`, a full disk, an offline redirected profile on
+// Windows, or an unrecognised file we refuse to delete. We attempt the work and
+// degrade rather than probing for writability first — a probe lies on NFS and
+// ACL filesystems, races, and can't see ENOSPC at all.
+//
+// Mode 0700 because Step 4A writes credentials into this directory.
+export function prepareOnboardingDir(dir = resolveOnboardingDir()) {
+  if (clearOnboardingDir(dir)) {
+    try {
+      mkdirSync(dir, { recursive: true, mode: 0o700 });
+      // mkdir's mode is umask-filtered, so set it outright. Windows has no POSIX
+      // mode; there the file inherits the profile's own ACL, as it does today.
+      if (process.platform !== 'win32') chmodSync(dir, 0o700);
+      return { dir, fellBack: false };
+    } catch {
+      // Fall through to the temp directory.
+    }
+  }
+  return { dir: mkdtempSync(join(tmpdir(), 'arize-onboarding-')), fellBack: true };
+}
+
+// Copy the bundled wheels next to the prompt. Returns the staged directory, or
+// null when this package has no usable vendor/ (a git checkout that hasn't run
+// the build). The shell launchers stage the same bundle themselves, fetching it
+// from jsDelivr via vendor/MANIFEST.
+//
+// Never throws, and never leaves a partial directory behind. Both matter: the
+// prompt decides which install path to take purely on whether this directory
+// exists, so a half-copied one would send the agent offline with no wheel to
+// install — and this is an optional enhancement, so a broken vendor/ must not
+// take the launcher down with it. Either outcome degrades to the network path.
+export function stageOfflineHarness(dir, vendorDir = VENDOR_DIR) {
+  if (!existsSync(vendorDir)) return null;
+
+  let staging;
+  try {
+    const wheels = readdirSync(vendorDir).filter((f) => f.endsWith('.whl'));
+    const hasHarnessWheel = wheels.some((f) => f.startsWith('coding_harness_tracing-'));
+    if (!hasHarnessWheel || !OFFLINE_FILES.every((f) => existsSync(join(vendorDir, f)))) return null;
+
+    // Build under a temp name and rename into place, so the directory the prompt
+    // looks for only ever appears complete. Same parent, so the rename is atomic.
+    staging = mkdtempSync(join(dir, '.staging-'));
+    for (const name of [...wheels, ...OFFLINE_FILES]) {
+      copyFileSync(join(vendorDir, name), join(staging, name));
+    }
+    chmodSync(join(staging, 'harness-install.sh'), 0o755);
+
+    const target = join(dir, OFFLINE_DIR_NAME);
+    renameSync(staging, target);
+    return target;
+  } catch {
+    if (staging) rmSync(staging, { recursive: true, force: true });
+    return null;
+  }
+}
+
+// Stage the bundled prompt and return the seed instruction that points the agent
+// at it, plus whether we had to fall back off the home directory.
 function prepareSeedPrompt() {
   const promptText = readFileSync(BUNDLED_PROMPT_PATH, 'utf8');
-  const dir = mkdtempSync(join(tmpdir(), 'arize-onboarding-'));
-  const promptFile = join(dir, 'onboarding-prompt.md');
+  const { dir, fellBack } = prepareOnboardingDir();
+  const promptFile = join(dir, PROMPT_FILE_NAME);
   writeFileSync(promptFile, promptText, 'utf8');
-  return `Read the file ${promptFile} and follow it to set up Arize AX tracing in this project, walking me through each step and asking me questions as needed.`;
+  stageOfflineHarness(dir);
+  // Single quotes, not double: a home path is likelier to contain a space than a
+  // temp path was, but Windows spawns through the shell and embedded double
+  // quotes don't survive that reliably. cmd.exe leaves single quotes alone.
+  // No "in this project": the launcher may well be run from a home directory, and
+  // Step 4 is what scopes the work — an app here, an app at another path, a starter
+  // app, or the coding agent itself. Keep this wording in step with start.{sh,ps1}.
+  const seed = `Read the file '${promptFile}' and follow it to set up Arize AX tracing, walking me through each step and asking me questions as needed.`;
+  return { seed, dir, fellBack };
 }
 
 // Coding agents we can launch interactively, seeded with the prompt.
@@ -291,17 +461,26 @@ function App({ onDone }) {
 function launchAgent(agent) {
   const isWin = process.platform === 'win32';
 
-  let seed;
+  let prepared;
   try {
-    seed = prepareSeedPrompt();
+    prepared = prepareSeedPrompt();
   } catch (err) {
-    console.error(`Could not read the onboarding prompt: ${err.message}`);
+    console.error(`Could not prepare the onboarding prompt: ${err.message}`);
     process.exit(1);
   }
 
+  // Say so when we couldn't use the home directory. Otherwise the seed path
+  // silently reverts to the /var/folders/… form this change exists to avoid, and
+  // Step 4A is going to fail later anyway — it writes the harness and its
+  // credentials under ~/.arize regardless of where the prompt was staged.
+  if (prepared.fellBack) {
+    console.log(`\nCouldn't use ${resolveOnboardingDir()} — staged in ${prepared.dir} instead.`);
+    console.log('Instrumenting an app still works; tracing this coding agent (Step 4A) needs a writable home directory.');
+  }
+
   console.log(`\nLaunching ${agent.label}…\n`);
 
-  const child = spawn(agent.bin, agent.args(seed), {
+  const child = spawn(agent.bin, agent.args(prepared.seed), {
     stdio: 'inherit',
     shell: isWin // .cmd/.bat shims on Windows need the shell to resolve
   });