evals 2.4.0 → 2.6.0

package/onboarding-prompt.md

@@ -6,15 +6,14 @@ Work through the flow below in order. Installing the AX CLI and Arize skills, au
 
 ## Step 0: Welcome and confirm
 
-Greet the user and show the plan, then ask before doing anything:
+Greet the user and show the plan, then ask before doing anything. Send this block as written and nothing else: no added steps, no parentheticals on the ones here (not "browser OAuth", not a folder path), and no preamble about the network check or the approval gate. Both come later in the flow, and explaining them before the user has agreed to anything is noise.
 
 ```text
 Welcome to Arize AX. I'll get you set up with tracing. Here's what I'll do:
 
 1. Install the AX CLI and Arize skills
 2. Create a free Arize AX account or sign you in
-3. Instrument this project or a new app
-4. Capture traces from your app
+3. Add tracing to an existing app, a new starter app, or this coding agent itself
 
 Shall I proceed?
 ```
@@ -23,6 +22,26 @@ Do not proceed until the user approves.
 
 ## Prerequisites
 
+**Check network access first.** Every step below needs it: the CLI comes from PyPI, and auth and traces go to Arize. Agent sandboxes often block network by default, and it fails as a hang rather than an error:
+
+```bash
+for h in pypi.org registry.npmjs.org api.arize.com app.arize.com; do
+  code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 "https://$h")
+  [ "$code" != "000" ] && echo "$h ok" || echo "$h BLOCKED"
+done
+```
+
+`000` is a DNS or connect failure — your sandbox, not the user's connection.
+
+If anything is blocked, ask your harness to escalate: request network or elevated permissions for the commands that need them, rather than retrying silently or waiting. If escalation is refused or your harness doesn't offer it, stop — with no network this flow cannot complete, so don't retry and don't look for a workaround:
+
+```text
+My sandbox is blocking network access, so I can't install the CLI or reach
+Arize. Restart me with network access enabled and I'll pick this up.
+```
+
+Later in the flow, treat a connection or DNS error from any command the same way: it means network, not authentication, so stop and report rather than re-authenticating or recreating a profile.
+
 The AX CLI must be **arize-ax-cli `0.29.0` or newer** (Step 1 installs the latest) and needs **Python 3.11+** — a hard requirement. If Python is missing, stop and have the user install it from https://www.python.org/downloads/ and re-run. **Node.js 18+ with npx** is optional but installs the Arize skills; without it, don't stop — continue and use the docs paths in Step 6 (the Vercel AI SDK v7 starter needs Node.js 22+).
 
 ## Step 1: Install or update the AX CLI and Arize skills
@@ -35,7 +54,7 @@ pipx install arize-ax-cli   # already installed: pipx upgrade arize-ax-cli
 python3 -m pip install --upgrade arize-ax-cli
 ```
 
-If npx is available, install the Arize agent skills; otherwise skip and continue (Step 6 falls back to the docs paths):
+If npx is available, install the Arize agent skills; otherwise skip and continue (Step 6 falls back to the docs paths). If the install errors or doesn't finish promptly, skip it and continue rather than waiting — the same fallback applies:
 
 ```bash
 npx skills add Arize-ai/arize-skills --skill '*' --yes
@@ -72,7 +91,7 @@ If a `default` profile exists but the probe failed, it's signed out or expired,
   ax profiles create default --auth-method api-key --api-key "$ARIZE_API_KEY"
   ```
 
-  Reuse this key throughout — do **not** create a new one in Step 6.
+  Reuse this key throughout — do **not** create a new one in Step 6. Only if the probe passes, though: a stale or wrong-org `ARIZE_API_KEY` still looks present. If it fails, `ax profiles delete default`, take the OAuth path below, and let Step 6 create a key as normal.
 
 - **No key anywhere** — sign up or sign in with browser OAuth:
 
@@ -80,7 +99,7 @@ If a `default` profile exists but the probe failed, it's signed out or expired,
   ax profiles create default --auth-method oauth --utm-params "utm_source=npmevals&utm_medium=cli&utm_campaign=prompt-first-onboarding"
   ```
 
-The `--utm-params` flag tags a new sign-up with onboarding attribution. It's already included on the three browser OAuth commands that take it — the two `ax profiles create --auth-method oauth` calls and the `ax auth login` fallback. Pass those verbatim; don't drop it, change the values, or add the flag to the api-key `profiles create` or any other `ax` command.
+Pass `--utm-params` verbatim on the two browser OAuth commands that carry it (`profiles create --auth-method oauth` and the `ax auth login` fallback) and add it to no other `ax` command — it tags a new sign-up with onboarding attribution.
 
 Always pass the positional profile name `default`. Without it, the CLI prompts `profile name [default]:`, receives EOF from an agent-run command, and exits with `Goodbye!` without creating a profile.
 
@@ -88,39 +107,23 @@ The rest of this step applies only to the **browser OAuth** branch.
 
 Creating an OAuth profile **is** the browser login flow. Treat it as an interactive browser handoff: it opens a browser, starts a localhost callback server such as `127.0.0.1:<port>/callback`, and waits for the browser redirect. The command must stay alive until the redirect lands and the CLI exits on its own.
 
-While the OAuth command is waiting, do not close its stdin, send Ctrl-C, `pkill` the AX process, start a second auth command, or run an auth probe. Any of these aborts the in-progress browser flow. There is one deliberate exception — the new email/password signup case described below, where the callback never lands and the command must be restarted.
+While the OAuth command is waiting, do not close its stdin, send Ctrl-C, `pkill` the AX process, start a second auth command, or run an auth probe. Any of these aborts the in-progress browser flow. There is no exception: every sign-in path — existing account, Google/SSO, existing or brand-new email/password — completes through the same callback, so the command always gets there on its own.
 
 After launching the OAuth command, tell the user:
 
 ```text
 A browser window is opening for Arize AX sign-in.
 
-- Already have an account, or signing in with Google/SSO or an existing
-  email/password? Just finish in the browser and I'll continue automatically.
-- Creating a BRAND-NEW account with email and password? Arize emails you a
-  validation link. That link does NOT complete the CLI login, so this command
-  will hang. Click the link to finish creating your account, then tell me.
+Sign in — or create a new account — in the browser. I'll continue
+automatically once it completes.
 
-Did you just sign up for a new account with email and password?
+Creating a BRAND-NEW account with email and password? Arize emails you a
+validation link. Click it and finish in the browser; I'll wait.
 ```
 
-**If the user says yes (new email/password signup):** the localhost callback will never fire and the command will wait forever, so this is the one time you break the "keep it alive" rule. Once they confirm they've clicked the validation link and their account exists, terminate the waiting OAuth command (send Ctrl-C / kill that process), then re-run it:
-
-```bash
-ax profiles create default --auth-method oauth --utm-params "utm_source=npmevals&utm_medium=cli&utm_campaign=prompt-first-onboarding"
-```
-
-The second run is now a plain sign-in for the validated account. Its callback lands normally, so handle it with all the standard rules below — keep it alive and wait for it to exit on its own.
-
-**Otherwise (existing account, SSO, or existing email/password):** wait for the command to exit on its own. Treat exit code `0`, or CLI success output such as `Configuration saved to profile 'default'` or `Active profile set`, as the primary completion signal.
+Then wait for the command to exit on its own. Treat exit code `0`, or CLI success output such as `Configuration saved to profile 'default'` or `Active profile set`, as the primary completion signal. A new email/password sign-up can sit waiting for a while — the user has to find the validation email — so a long wait is not a hang; leave it alone.
 
-Only after the OAuth command completes, verify authentication with a non-secret probe:
-
-```bash
-ax spaces list --limit 1 --output json
-```
-
-If the probe succeeds, continue. If the default OAuth profile already existed and the probe returns an authentication error, the profile is signed out or expired — run the fallback:
+Only after the OAuth command completes, re-run the probe from the top of this step. If it succeeds, continue. If the default OAuth profile already existed and the probe returns an authentication error, the profile is signed out or expired — run the fallback:
 
 ```bash
 ax auth login --utm-params "utm_source=npmevals&utm_medium=cli&utm_campaign=prompt-first-onboarding"
@@ -144,9 +147,27 @@ If `ARIZE_SPACE_ID` is already set (environment or `.env` / `.env.local`), use i
 
 Capture the space's **ID** (not its display name) for `ARIZE_SPACE_ID`. The `arize-otel` tracing config requires the space ID; a name will silently fail to route traces.
 
-## Step 4: Inspect the folder and choose a path
+## Step 4: Choose what to trace
+
+Ask what the user wants to trace before inspecting anything — the third option has nothing to do with what is in the current folder:
+
+```text
+What would you like to trace?
+
+1. An existing app — in this folder or at another path
+2. A new starter app I create for you
+3. This coding agent itself — every session you run, in any project
+
+Which one?
+```
+
+If they pick **3**, go to [Step 4A](#step-4a-trace-this-coding-agent) and skip the folder inspection entirely. If they pick **2**, go straight to "Create a starter app" below — don't inspect anything. For **1**, continue.
+
+For options **1** and **2**, one directory ends up being **the app folder**: the app you instrument, which is not always the folder you were launched in. Establish it in this step and use it for every path in Steps 5 to 7.
+
+### Detecting an app
 
-Inspect the current folder to decide whether an app already exists. Do not change files during inspection. Look for:
+Inspect a candidate folder to decide whether an app already exists. Do not change files during inspection. Look for:
 
 - Python: `pyproject.toml`, `requirements.txt`, `setup.py`, `Pipfile`, imports.
 - TypeScript/JavaScript: `package.json`, lockfiles, `src`, `app`, `pages`, provider imports.
@@ -154,24 +175,48 @@ Inspect the current folder to decide whether an app already exists. Do not chang
 - Existing observability: `opentelemetry`, `TracerProvider`, `ARIZE_*`, `OTEL_*`, `OTLP_*`, Datadog, Honeycomb, Sentry, or other tracing.
 - Agent framework: identify it by its import/package — e.g. `langchain` / `langgraph`, `llama_index`, `crewai`, `autogen`, `semantic_kernel`, `pydantic_ai`, `google.adk`, `dspy`, `agent_framework`, and others. **Route on the framework, not the provider client it wraps** — an `openai` or `anthropic` import inside a framework app is not the thing to instrument; the framework almost certainly has its own integration (see Step 6).
 
-In a monorepo, check the git root to get oriented, but only instrument apps in or below the current working directory. If the project spans more than one language, instrument each one (route each through its own integration page in Step 6).
+In a monorepo, check the git root to get oriented, but only instrument apps in or below the folder you are scanning. If the project spans more than one language, instrument each one (route each through its own integration page in Step 6).
 
-Then branch on what you found:
+Run this detection on the current folder first, then branch on what you found.
 
 ### If an app exists in the current folder
 
-Do not offer the starter-app path. Summarize the detected stack and offer to instrument it:
+Summarize the detected stack and offer both routes — declining the local app must not be a dead end, because the user may well have run this from a folder above the app they care about:
 
 ```text
-I found a <language>/<framework> app in this folder. Want me to add Arize AX
-tracing to it?
+I found a <language>/<framework> app in this folder. I can trace that, or an
+app somewhere else.
+
+1. Trace the app in this folder
+2. Trace an app at a different path — tell me where
+
+Which one?
 ```
 
-If the user declines, stop here rather than falling back to a starter app.
+On **1**, the app folder is the current folder; continue to Step 5. On **2**, follow "If the user gives a path" below. Don't offer the starter-app path here — they already have an app.
 
 ### If no app exists in the current folder
 
-Go straight to the starter-app choice — do not ask whether to instrument the empty folder. Ask which folder to create it in, then offer these choices:
+Do not ask whether to instrument the empty folder, and do not assume a starter app is what they want. Ask for a path first:
+
+```text
+I don't see an app in this folder. Give me the path to the app you want to
+trace, and I'll instrument it — or say "starter" and I'll create a new app
+for you instead.
+```
+
+On a path, follow the next section. On "starter", go to "Create a starter app".
+
+### If the user gives a path
+
+- Expand `~` and resolve a relative path against the current folder, then confirm the resolved absolute path back to the user before you scan it. A typo caught here beats an API key written into the wrong repo.
+- Run the detection checklist on that folder. If it holds an app, that folder is the app folder from now on — summarize the stack you found there and continue to Step 5.
+- If the path doesn't exist, or exists but has no app in it, say what you actually found and ask for another path. Don't quietly fall through to a starter app.
+- If your own permission or sandbox layer won't let you read and write outside the folder you were launched in, say so plainly instead of half-instrumenting the app. Give the two ways out: the user grants access to that path, or they re-run `npx evals` from inside the app folder.
+
+### Create a starter app
+
+For option 2, or when the user asks for a starter app after the questions above. Ask which folder to create it in — that folder becomes the app folder — then offer these choices:
 
 - OpenAI — Python or TypeScript (with a tool call)
 - Anthropic — Python (with a tool call; official AX auto-instrumentation)
@@ -180,15 +225,157 @@ Go straight to the starter-app choice — do not ask whether to instrument the e
 
 If the user picks an unsupported pairing, explain the supported options and ask again.
 
+## Step 4A: Trace this coding agent
+
+Only for option 3. This traces the **agent harness**, not an app: every session the user runs, in every project on this machine. It edits files under their home directory, nothing in the current repo. When done, go to Step 8 — Steps 5 to 7 are app-only.
+
+Installer harness names: `claude`, `codex`, `cursor`, `copilot`, `gemini`, `kiro`, `opencode`, `omp` — note Cursor is `cursor`, not `cursor-agent`. Default to the agent you are running inside: state which you are and confirm, rather than asking.
+
+Per-agent setup, including the Claude Code and Cursor marketplace-plugin routes, is at `https://arize.com/docs/ax/integrations/platforms/<agent>/<agent>-tracing` (e.g. `.../claude-code/claude-code-tracing`) — the source of truth if anything below fails.
+
+### Check the home directory is writable first
+
+Everything below writes under `~/.arize` — the credentials file, then the harness itself. Confirm that works before asking for approval, so a read-only or restricted home fails here rather than halfway through an install the user already said yes to:
+
+```bash
+mkdir -p ~/.arize/onboarding && touch ~/.arize/onboarding/.probe && rm -f ~/.arize/onboarding/.probe
+```
+
+If that fails, say so and stop this path — agent tracing needs a writable home and there is no workaround from inside the session. Instrumenting an app (options 1 and 2) does not, so offer that instead. If the launcher already told the user it couldn't use `~/.arize/onboarding`, this is the same cause; don't re-diagnose it.
+
+### Get approval — this needs its own explicit yes
+
+Wider scope than an app install, so disclose it and wait:
+
+```text
+This traces <agent> itself. Three things first:
+
+- It applies to EVERY session you run, in every project on this machine.
+- Unless you say otherwise I'll turn on all three capture categories: your
+  prompts, what tools were asked to do (commands, file paths, URLs), and what
+  tools returned (file contents, command output).
+- It writes to ~/.arize/harness and <agent>'s settings file. Nothing in this
+  project changes.
+
+Want to skip any capture category — prompts, tool commands, or tool output?
+Shall I go ahead?
+```
+
+Do not proceed without a yes. Note which categories they accepted — you enable those explicitly below, and anything you leave out stays off.
+
+### Write the config file
+
+The installer reads credentials from a dotenv file, keeping the API key out of the command line, shell history, and this chat. Write it outside the project so it cannot be committed:
+
+```bash
+: > ~/.arize/onboarding/harness.env && chmod 600 ~/.arize/onboarding/harness.env
+```
+
+Add the Step 3 space ID as file contents, plus one `true` line per category the user **accepted**. Unattended installs capture nothing unless asked to, so a category you omit is off — omit the line for anything they declined:
+
+```dotenv
+ARIZE_SPACE_ID=<space-id-from-step-3>
+ARIZE_LOG_PROMPTS=true
+ARIZE_LOG_TOOL_DETAILS=true
+ARIZE_LOG_TOOL_CONTENT=true
+```
+
+Do **not** set `ARIZE_PROJECT_NAME` — each harness defaults to its own project (`claude-code`, `codex`, …), which keeps two traced agents apart, and you read the real name back after installing. Then create the key into the same file, written atomically and never printed:
+
+```bash
+ax api-keys create --name "Coding agent tracing" --env-file ~/.arize/onboarding/harness.env
+```
+
+**Skip that if `ARIZE_API_KEY` already existed in Step 2** — copy the existing value in without echoing it. At most one key, ever.
+
+### Install, then delete the file
+
+If a directory named `arize-offline/` sits beside this prompt file, install from it — no download, no remote script, so it is faster and far less likely to be refused:
+
+```bash
+# keep `< /dev/null`: an installer too old for --non-interactive then fails
+# fast instead of hanging on a prompt you cannot answer
+ARIZE_ENV_FILE=~/.arize/onboarding/harness.env \
+  bash <prompt-dir>/arize-offline/harness-install.sh \
+  <harness> --wheel-dir <prompt-dir>/arize-offline --non-interactive < /dev/null
+```
+
+Otherwise fetch it (the harness name must come first, before any flag):
+
+```bash
+ARIZE_ENV_FILE=~/.arize/onboarding/harness.env \
+  bash <(curl -fsSL https://raw.githubusercontent.com/Arize-ai/coding-harness-tracing/main/install.sh) \
+  <harness> --non-interactive < /dev/null
+```
+
+Delete the env file only once the install has actually run — a retry needs it, and so does the user if they end up running the command:
+
+```bash
+rm -f ~/.arize/onboarding/harness.env
+```
+
+On Windows, use `harness-install.bat` from the same directory — `cmd` cannot run the `.sh`:
+
+```powershell
+$env:ARIZE_ENV_FILE = "$HOME\.arize\onboarding\harness.env"
+& "<prompt-dir>\arize-offline\harness-install.bat" <harness> --wheel-dir "<prompt-dir>\arize-offline" --non-interactive
+```
+
+Without `arize-offline/`, use the install command from the agent's docs page with `$env:ARIZE_ENV_FILE` set first.
+
+An `EOFError` or hang means the installer predates `--non-interactive`: have the user run it without that flag and without `< /dev/null`, answering its prompts. On any other failure, report what it printed and stop — never retry with a different backend or type the key into a prompt.
+
+### If your permission layer refuses to run it
+
+Expected on the fetched command — auto-approval modes hold back piping a downloaded script into a shell. Retry **once** with the installer downloaded to a file first; if still refused, stop and do not hunt for a phrasing that slips through. Hand it over instead:
+
+```text
+My permission settings won't let me run the installer. It creates
+~/.arize/harness and adds hooks to <harness>'s settings file. Either run it
+yourself — prefix with ! so I see the output — or tell me to go ahead and
+I'll retry.
+
+<the command, one line>
+```
+
+A go-ahead authorizes that one command, nothing wider. **Never** frame this as bypassing a safety check, and never suggest an approval flag on the agent. **Keep the env file** until they confirm — they need those credentials.
+
+### Verify before asking the user for anything
+
+```bash
+~/.arize/harness/install.sh status --json
+```
+
+Exit `0` means configured **and** hooks wired up; `1` nothing configured; `2` configured but hooks missing (the `unregistered` list names which to re-install). Continue only on `0`. Take the project name for polling from `harnesses[].project_name` in the payload rather than guessing it. The payload holds no secrets.
+
+### Get the first traces
+
+Hooks load at session start, so **this session will never emit traces**. Do not restart or kill yourself — that would end the setup. Use a second session so you stay alive to poll:
+
+```text
+Tracing is installed. This session won't be traced — it started before the
+hooks existed. So:
+
+1. Open a new terminal and start <agent> there.
+2. Ask it something small, like "list the files in this folder".
+3. Tell me when you have, and I'll check for traces.
+```
+
+Then poll with the Step 7 command and counter, passing that project name and space ID, every ~15 seconds for up to ~3 minutes. On a non-zero count go to Step 8. On timeout say so plainly and give the likely causes: the new session started before the install finished, the agent was never asked to do anything, `ARIZE_TRACE_ENABLED` is `false` in the agent's settings, or the wrong space.
+
 ## Step 5: Present the plan and get approval
 
-Before creating any remote resource, writing files, or installing dependencies, present one consolidated plan and wait for approval. This is the gate the intro refers to — nothing so far has modified the app or created AX resources.
+Steps 5 to 7 are for the app paths only; if you took Step 4A, go straight to Step 8.
+
+Before creating any remote resource, writing files, or installing dependencies, present one consolidated plan and wait for approval.
 
 For an existing app, cover: detected language and framework, package manager, LLM provider or agent framework, any existing tracing to preserve, the env file that will be updated, the instrumentation packages and files that will change, whether a new AX user API key will be created or the existing `ARIZE_API_KEY` reused, and the project name that will be used.
 
 For a starter app, cover: the chosen provider and language, the target folder, the packages that will be installed, and the project name.
 
-Choose a default project name from the current folder or app name: lowercase it, replace spaces and unsupported punctuation with hyphens, and append `-arize-tracing` if it is too generic. Do not create an AX project explicitly — it is created on first trace ingestion.
+**State the app folder's absolute path in the plan whenever it isn't the folder you were launched in** — every file you touch lands there, and it's the one detail the user cannot verify from context.
+
+Choose a default project name from the app folder or app name: lowercase it, replace spaces and unsupported punctuation with hyphens, and append `-arize-tracing` if it is too generic. Do not create an AX project explicitly — it is created on first trace ingestion.
 
 ```text
 Here's my plan. Shall I proceed?
@@ -202,7 +389,7 @@ Only after approval, execute the plan in this order.
 
 ### Choose the env file
 
-Pick the env file to match the app and use it for every variable below:
+The env file lives in the **app folder** from Step 4. Pick its name to match the app and use that one file for every variable below:
 
 - Next.js, Vite, or browser-adjacent TypeScript apps: `.env.local`
 - Python apps, Node scripts, backend services, or unknown type: `.env`
@@ -210,7 +397,7 @@ Pick the env file to match the app and use it for every variable below:
 
 Do not read existing env file contents into chat. Preserve unrelated variables and never reveal their values.
 
-Make sure the env file is git-ignored before writing the API key to it — if the repo has a `.gitignore`, confirm it covers the file (add `.env` / `.env.local` if not); for a starter app you create, add one. The API key must never be committed to version control.
+Make sure the env file is git-ignored before writing the API key to it — check the `.gitignore` of the app folder's own repo. If it has one, confirm it covers the file (add `.env` / `.env.local` if not); for a starter app you create, add one. The API key must never be committed to version control.
 
 ### Write the non-secret variables
 
@@ -221,9 +408,11 @@ ARIZE_SPACE_ID=<space-id>
 ARIZE_PROJECT_NAME=<chosen-project-name>
 ```
 
+An exported variable beats this file — both `dotenv` implementations leave an existing environment variable alone by default. Check `env | grep ARIZE_`; if anything you just wrote is exported, tell the user to `unset` it in the shell they'll run the app from, or to load with `override=True`. Your Step 7 check reads the CLI profile, not the app's environment, so it would pass regardless.
+
 ### Create the API key
 
-**Skip this entirely if `ARIZE_API_KEY` was already present in Step 2** — reuse it and leave its env value untouched. Only create a key when you authenticated with browser OAuth and the app has no key yet.
+**Skip this entirely if `ARIZE_API_KEY` was already present and validated in Step 2** — reuse it and leave its env value untouched. Only create a key when you authenticated with browser OAuth and the app has no key yet.
 
 Create the key and write it into the env file in one step:
 
@@ -231,11 +420,11 @@ Create the key and write it into the env file in one step:
 ax api-keys create --name "Local Arize AX tracing" --env-file .env
 ```
 
-`--env-file` writes `ARIZE_API_KEY` atomically and **never prints it** — no temp file, no secret in your terminal or chat. Use `.env.local` if that's the app's convention; the file is created if missing, an existing `ARIZE_API_KEY` is replaced in place, and other variables are preserved.
+`--env-file` writes `ARIZE_API_KEY` atomically and **never prints it** — no temp file, no secret in your terminal or chat. Use `.env.local` if that's the app's convention, and always pass the app folder's path (`--env-file /path/to/app/.env`) — a bare `.env` would leave a stray key file in the wrong directory. The file is created if missing, an existing `ARIZE_API_KEY` is replaced in place, and other variables are preserved.
 
-**Create the key exactly once** — a second `ax api-keys create` just orphans a still-active key. Git-ignore the env file (see above) *before* running this so the key is never committed.
+**Create the key exactly once** — a second `ax api-keys create` just orphans a still-active key, and the env file must already be git-ignored.
 
-If key creation fails, have the user create one in the Arize AX UI and add it to the env file without exposing it in chat. Either way, create only one key.
+If key creation fails, have the user create one in the Arize AX UI and add it to the env file without exposing it in chat.
 
 Handle the LLM provider's own key (e.g. `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`) in the same env file — the app can't make a call or produce traces without it:
 
@@ -261,7 +450,7 @@ Common shortcuts:
 
 1. Use a framework-specific OpenInference instrumentor if one exists (`openinference-instrumentation-<name>` / `@arizeai/openinference-instrumentation-<name>`), wired up with `arize-otel` per the manual-instrumentation guide; install it unpinned.
 1. Instrument the underlying provider (OpenAI, Anthropic, Bedrock, …) with its instrumentor **only if the framework calls the provider SDK directly**. Many agent frameworks instead drive the model through their own client layer and emit their own OpenTelemetry spans — a provider instrumentor captures **no traces** for those. Never reach for the provider instrumentor as a blind fallback just because you recognize an `openai`/`anthropic` client.
-1. Otherwise instrument manually via `arize-otel` (see https://arize.com/docs/ax/instrument/manual-instrumentation), or stop and ask the user if you still can't determine a setup.
+1. Otherwise instrument manually by code — follow https://arize.com/docs/ax/instrument/manual-instrumentation#by-code — or stop and ask the user if you still can't determine a setup. `arize-otel` gives you a tracer, not LLM spans: **set the OpenInference span kind on every span you create** (`LLM` for a model call, `TOOL` for a tool, `AGENT` for the loop, `CHAIN` for a step), **and set the span status** — `ERROR` with the exception recorded on failure, `OK` otherwise. Without the kind, spans arrive as generic spans that AX cannot read as LLM calls: Step 7's span count comes back non-zero and looks like success while the trace shows no input, output, model, or token counts. Without the status, failed calls look like successful ones and errors never surface in AX.
 
 For existing apps:
 
@@ -281,7 +470,7 @@ For starter apps:
 
 ### Package guidance
 
-Install into the app's existing environment (its virtualenv if it has one) exactly the packages the detected framework's integration page lists — instrumentor names and peer dependencies differ per framework, so follow that page rather than copying from another stack or guessing versions.
+Install into the app's existing environment — the one in the app folder, using its virtualenv if it has one — exactly the packages the detected framework's integration page lists — instrumentor names and peer dependencies differ per framework, so follow that page rather than copying from another stack or guessing versions.
 
 ## Step 7: Run the app and poll for the first trace
 
@@ -293,11 +482,12 @@ If you created a starter app in this flow, offer to run it for the user:
 Your starter app is ready. Want me to run it for you?
 ```
 
-If they say yes, run it yourself with the run command, then poll. If they say no — or if you instrumented their existing app rather than creating a starter — tell them the exact run command and ask them to run it:
+If they say yes, run it yourself with the run command, then poll. If they say no — or if you instrumented their existing app rather than creating a starter — tell them the exact run command and ask them to run it. When the app folder isn't the folder they're sitting in, lead with the `cd` so the command works as pasted:
 
 ```text
 Run your app with:
 
+cd <app folder>
 <run command>
 
 It should make at least one LLM call. I'll poll Arize AX and let you know as
@@ -318,20 +508,22 @@ A non-zero count confirms traces are arriving. If you need to inspect a span to
 This uses the CLI profile from Step 2 (OAuth or api-key) — it works the same either way. If the export errors with an authentication failure, the profile isn't valid; re-run the Step 2 probe and re-authenticate, or fall back to having the user open the project in the Arize AX UI to confirm traces.
 
 - When spans come back, stop polling and continue to Step 8.
-- On timeout, do not fail silently. Tell the user no traces arrived yet, and give likely causes: app didn't make an LLM call, tracing initialized after the client was created, a short-lived script exited before flushing spans, or the wrong space/project/env file. Offer to re-check once they've run it again.
+- On timeout, do not fail silently. Tell the user no traces arrived yet, and give likely causes: app didn't make an LLM call, tracing initialized after the client was created, a short-lived script exited before flushing spans, the wrong space/project/env file, or **an exported `ARIZE_*` variable overriding the env file** (`env | grep ARIZE_`). Offer to re-check once they've run it again.
 
 Do not fabricate trace results. Only report traces the export command actually returned.
 
 ## Step 8: Report the first traces with a link
 
-Once spans arrive, report the span count and give the user a link into Arize AX for the project. Point them at the UI to explore the trace contents rather than printing span bodies into chat.
+Once spans arrive, report the span count and give the user a deep link to the project in Arize AX. Point them at the UI to explore the trace contents rather than printing span bodies into chat.
+
+Build that link with the **Arize link skill**: if you installed the skills in Step 1, load it now by reading `arize-link/SKILL.md` from your agent's skills directory (same paths as Step 6) and follow it. It owns the URL format and the `ax` commands that discover the organization and project IDs; you already have the project name and the space ID from Step 3.
 
 ```text
 Your first traces are in Arize AX. Open project `<ARIZE_PROJECT_NAME>` here:
-https://app.arize.com/
+<project link>
 ```
 
-If a space- or project-specific URL is known from the CLI or docs, prefer that deep link over the app root. Do not invent a URL structure you are not sure of; fall back to `https://app.arize.com/` plus instructions to select the project.
+If that skill file doesn't exist (e.g. npx was missing in Step 1), don't invent a URL structure: fall back to `https://app.arize.com/` plus instructions to select the project.
 
 ## Step 9: Point at docs
 
@@ -345,13 +537,25 @@ You're set up. To go further:
 - AX CLI: https://arize.com/docs/api-clients/cli/overview
 ```
 
+If you took Step 4A, add the controls that matter for agent tracing:
+
+```text
+- Your agent's tracing page: https://arize.com/docs/ax/integrations/platforms/<agent>/<agent>-tracing
+- Check it's still wired up: ~/.arize/harness/install.sh status
+- Pause it: set ARIZE_TRACE_ENABLED=false in <agent>'s settings
+- Remove it: ~/.arize/harness/install.sh uninstall <installer-name>
+```
+
 ## Critical rules
 
 - Get the user's approval (Step 5) before creating AX resources, editing files, or installing dependencies.
-- Authenticate the CLI before any other `ax` call — every command, including the final trace check, uses the active profile; env vars alone don't authenticate it. On the browser OAuth branch, keep the command alive until its localhost callback completes (never close stdin, Ctrl-C, `pkill`, or probe while it waits) — sole exception: a brand-new email/password signup, whose link never calls back, so ask the user, then kill and re-run as a sign-in.
+- Authenticate the CLI (Step 2) before any other `ax` call; env vars alone don't. Never interrupt a waiting OAuth command — no stdin close, Ctrl-C, `pkill`, or probe. A slow wait is not a hang.
 - Never print, log, or summarize secrets in chat — API keys, env-file contents/values, or span bodies (prompts, completions, tool args, user data) — and never read env files into chat. Only report traces a command actually returned.
-- Create at most one AX API key: skip if `ARIZE_API_KEY` already exists (reuse it), otherwise a **single** `ax api-keys create --env-file <file>`. Git-ignore that file before creating the key; never create a second key.
+- One AX API key, ever: reuse a **validated** `ARIZE_API_KEY`, else a **single** `ax api-keys create --env-file <file>` into an already-git-ignored file.
+- Ambient `ARIZE_*` variables are claims to verify, not facts to adopt: probe a found API key before reusing it, and remember that an exported variable overrides the env file you write.
+- Everything you write goes in the **app folder** (Step 4), which may not be the folder you were launched in. Name its absolute path in the plan; pass it explicitly to `--env-file` and the run command.
 - Write the space **ID** (not its name) to `ARIZE_SPACE_ID`, or traces won't route; never create an AX project explicitly (it's made on first ingestion).
-- Initialize tracing before LLM clients are created, and flush/shut down the tracer before short-lived scripts exit. Vercel AI SDK v7 also needs Node.js 22+, `@ai-sdk/otel` registered, and `experimental_telemetry: { isEnabled: true }` per call.
+- Initialize tracing before LLM clients exist; flush before short-lived scripts exit. Vercel AI SDK v7 also needs Node.js 22+, `@ai-sdk/otel` registered, and per-call `experimental_telemetry`.
+- Step 4A needs its own explicit yes, never folded into another approval — machine-wide, captures prompts and tool output. Run its installer with `--non-interactive` and `< /dev/null`, verify with `status --json`, and never restart the session you're in.
 
 Docs: https://arize.com/docs/llms.txt

package/package.json

@@ -1,13 +1,18 @@
 {
   "name": "evals",
-  "version": "2.4.0",
+  "version": "2.6.0",
   "description": "Arize AX onboarding — instrument your app with tracing via your coding agent",
   "type": "module",
   "main": "cli.js",
+  "engines": {
+    "node": ">=20"
+  },
   "scripts": {
     "postinstall": "node -e \"console.log('Arize evals installed — run: npx evals')\"",
     "start": "node cli.js",
-    "test": "node --test"
+    "test": "node --test",
+    "build:wheel": "node scripts/build-harness-wheel.mjs",
+    "prepack": "npm run build:wheel"
   },
   "dependencies": {
     "ink": "^6.0.0",
@@ -15,13 +20,15 @@
     "react": "^19.0.0"
   },
   "bin": {
-    "evals": "./cli.js"
+    "evals": "./bin.js"
   },
   "files": [
+    "bin.js",
     "cli.js",
     "onboarding-prompt.md",
     "start.sh",
-    "start.ps1"
+    "start.ps1",
+    "vendor/"
   ],
   "keywords": [
     "arize",