evals 2.2.8 → 2.3.0

package/onboarding-prompt.md

@@ -0,0 +1,355 @@
+# Get Started With Arize AX
+
+Guide the user from zero to their first traces in Arize AX. Use the AX CLI, the Arize agent skills, and the official docs.
+
+Work through the flow below in order. Installing the AX CLI and Arize skills, authenticating, and listing your spaces are all fine before approval — those are your own tooling. But do not create AX resources (like API keys), edit application files, or install the app's tracing dependencies before the user approves the plan in Step 5.
+
+## Step 0: Welcome and confirm
+
+Greet the user and show the plan, then ask before doing anything:
+
+```text
+Welcome to Arize AX. I'll get you set up with tracing. Here's what I'll do:
+
+1. Install the AX CLI and Arize skills
+2. Create a free Arize AX account or sign you in
+3. Instrument this project or a new app
+4. Capture traces from your app
+
+Shall I proceed?
+```
+
+Do not proceed until the user approves.
+
+## Prerequisites
+
+The AX CLI must be **arize-ax-cli `0.28.0` or newer** (Step 1 installs the latest) and needs **Python 3.11+** — a hard requirement. If Python is missing, stop and have the user install it from https://www.python.org/downloads/ and re-run. **Node.js 18+ with npx** is optional but installs the Arize skills; without it, don't stop — continue and use the docs paths in Step 6 (the Vercel AI SDK v7 starter needs Node.js 22+).
+
+## Step 1: Install or update the AX CLI and Arize skills
+
+Check the version with `ax --version`. If it's missing or older than the version above, install or upgrade `arize-ax-cli` — prefer `pipx`, otherwise pip:
+
+```bash
+pipx install arize-ax-cli   # already installed: pipx upgrade arize-ax-cli
+# without pipx:
+python3 -m pip install --upgrade arize-ax-cli
+```
+
+If npx is available, install the Arize agent skills; otherwise skip and continue (Step 6 falls back to the docs paths):
+
+```bash
+npx skills add Arize-ai/arize-skills --skill '*' --yes
+```
+
+This writes the skills to disk but does **not** load them into your current session — a mid-session install is not in your invocable skill list, so don't call it by name. When you need a skill (Step 6), load it by reading its `SKILL.md` directly from the install path (the command prints where it wrote them). Skill directories are named `arize-<name>`, e.g. `arize-instrumentation`.
+
+## Step 2: Authenticate the CLI
+
+The CLI authenticates through a profile named `default`, and every later `ax` command — including the trace check at the end — uses that profile. A bare `ARIZE_API_KEY` in the environment does **not** authenticate the CLI on its own; a profile must exist. Get one working before continuing.
+
+First check whether the CLI is already authenticated:
+
+```bash
+ax spaces list --limit 1 --output json
+```
+
+If this succeeds, a profile is already set up — skip the rest of this step.
+
+If it fails, first check whether a `default` profile already exists from a prior run:
+
+```bash
+ax profiles list
+```
+
+If a `default` profile exists but the probe failed, it's signed out or expired, not missing — for an OAuth profile, refresh it with `ax auth login` (see the OAuth handling below) and re-run the probe rather than recreating it. Only create a new profile when none exists (or an existing api-key profile has an invalid key). To create one, look for an existing key first: check for `ARIZE_API_KEY` in the environment **or** the project's `.env` / `.env.local`.
+
+- **A key is available** — the user already has an account and key, so skip the browser flow. Resolve the value (never print it): use `$ARIZE_API_KEY` if it's exported in your shell; if it's only in a dotenv file, load it from there for this one command. Then create an api-key profile and re-run the probe to confirm:
+
+  ```bash
+  # if the key is only in a dotenv file, load it first (nothing is printed);
+  # point at the file the app uses (.env or .env.local):
+  export ARIZE_API_KEY="$(grep -E '^ARIZE_API_KEY=' .env 2>/dev/null | tail -1 | cut -d= -f2-)"
+  ax profiles create default --auth-method api-key --api-key "$ARIZE_API_KEY"
+  ```
+
+  Reuse this key throughout — do **not** create a new one in Step 6.
+
+- **No key anywhere** — sign up or sign in with browser OAuth:
+
+  ```bash
+  ax profiles create default --auth-method oauth
+  ```
+
+Always pass the positional profile name `default`. Without it, the CLI prompts `profile name [default]:`, receives EOF from an agent-run command, and exits with `Goodbye!` without creating a profile.
+
+The rest of this step applies only to the **browser OAuth** branch.
+
+Creating an OAuth profile **is** the browser login flow. Treat it as an interactive browser handoff: it opens a browser, starts a localhost callback server such as `127.0.0.1:<port>/callback`, and waits for the browser redirect. The command must stay alive until the redirect lands and the CLI exits on its own.
+
+While the OAuth command is waiting, do not close its stdin, send Ctrl-C, `pkill` the AX process, start a second auth command, or run an auth probe. Any of these aborts the in-progress browser flow. There is one deliberate exception — the new email/password signup case described below, where the callback never lands and the command must be restarted.
+
+After launching the OAuth command, tell the user:
+
+```text
+A browser window is opening for Arize AX sign-in.
+
+- Already have an account, or signing in with Google/SSO or an existing
+  email/password? Just finish in the browser and I'll continue automatically.
+- Creating a BRAND-NEW account with email and password? Arize emails you a
+  validation link. That link does NOT complete the CLI login, so this command
+  will hang. Click the link to finish creating your account, then tell me.
+
+Did you just sign up for a new account with email and password?
+```
+
+**If the user says yes (new email/password signup):** the localhost callback will never fire and the command will wait forever, so this is the one time you break the "keep it alive" rule. Once they confirm they've clicked the validation link and their account exists, terminate the waiting OAuth command (send Ctrl-C / kill that process), then re-run it:
+
+```bash
+ax profiles create default --auth-method oauth
+```
+
+The second run is now a plain sign-in for the validated account. Its callback lands normally, so handle it with all the standard rules below — keep it alive and wait for it to exit on its own.
+
+**Otherwise (existing account, SSO, or existing email/password):** wait for the command to exit on its own. Treat exit code `0`, or CLI success output such as `Configuration saved to profile 'default'` or `Active profile set`, as the primary completion signal.
+
+Only after the OAuth command completes, verify authentication with a non-secret probe:
+
+```bash
+ax spaces list --limit 1 --output json
+```
+
+If the probe succeeds, continue. If the default OAuth profile already existed and the probe returns an authentication error, the profile is signed out or expired — run the fallback:
+
+```bash
+ax auth login
+```
+
+Handle `ax auth login` with the same rules: keep its callback server alive, wait for it to exit or print a success line, then rerun the probe. Do not run `ax auth login` immediately after creating an OAuth profile; the profile creation already performed login.
+
+Do not treat `ax profiles show` alone as proof that OAuth completed; it can show profile configuration even when the user still needs to authenticate. Use it only for troubleshooting profile configuration, and never with `--expand`.
+
+Run no other remote `ax` command (creating keys, listing all spaces, inspecting resources) until the probe succeeds.
+
+## Step 3: Select the AX space
+
+List the spaces the authenticated profile can access:
+
+```bash
+ax spaces list --output json
+```
+
+If `ARIZE_SPACE_ID` is already set (environment or `.env` / `.env.local`), use it **only if it appears in that list** — that confirms the active profile can reach it (the app's existing key and your CLI profile may point at different spaces). If it's set but not in the list, don't trust it; select from the list instead. Otherwise: if exactly one space is returned, use it; if multiple, ask the user which to use; if none, guide the user to create a space in the Arize AX UI (or with an organization ID if available), then re-list.
+
+Capture the space's **ID** (not its display name) for `ARIZE_SPACE_ID`. The `arize-otel` tracing config requires the space ID; a name will silently fail to route traces.
+
+## Step 4: Inspect the folder and choose a path
+
+Inspect the current folder to decide whether an app already exists. Do not change files during inspection. Look for:
+
+- Python: `pyproject.toml`, `requirements.txt`, `setup.py`, `Pipfile`, imports.
+- TypeScript/JavaScript: `package.json`, lockfiles, `src`, `app`, `pages`, provider imports.
+- Go: `go.mod`. Java: `pom.xml`, `build.gradle`, `build.gradle.kts`.
+- Existing observability: `opentelemetry`, `TracerProvider`, `ARIZE_*`, `OTEL_*`, `OTLP_*`, Datadog, Honeycomb, Sentry, or other tracing.
+- Agent framework: identify it by its import/package — e.g. `langchain` / `langgraph`, `llama_index`, `crewai`, `autogen`, `semantic_kernel`, `pydantic_ai`, `google.adk`, `dspy`, `agent_framework`, and others. **Route on the framework, not the provider client it wraps** — an `openai` or `anthropic` import inside a framework app is not the thing to instrument; the framework almost certainly has its own integration (see Step 6).
+
+In a monorepo, check the git root to get oriented, but only instrument apps in or below the current working directory. If the project spans more than one language, instrument each one (route each through its own integration page in Step 6).
+
+Then branch on what you found:
+
+### If an app exists in the current folder
+
+Do not offer the starter-app path. Summarize the detected stack and offer to instrument it:
+
+```text
+I found a <language>/<framework> app in this folder. Want me to add Arize AX
+tracing to it?
+```
+
+If the user declines, stop here rather than falling back to a starter app.
+
+### If no app exists in the current folder
+
+Go straight to the starter-app choice — do not ask whether to instrument the empty folder. Ask which folder to create it in, then offer these choices:
+
+- OpenAI — Python or TypeScript (with a tool call)
+- Anthropic — Python (with a tool call; official AX auto-instrumentation)
+- LangChain with OpenAI — Python or TypeScript (a tool-using agent)
+- Vercel AI SDK with OpenAI — TypeScript (a tool-using agent)
+
+If the user picks an unsupported pairing, explain the supported options and ask again.
+
+## Step 5: Present the plan and get approval
+
+Before creating any remote resource, writing files, or installing dependencies, present one consolidated plan and wait for approval. This is the gate the intro refers to — nothing so far has modified the app or created AX resources.
+
+For an existing app, cover: detected language and framework, package manager, LLM provider or agent framework, any existing tracing to preserve, the env file that will be updated, the instrumentation packages and files that will change, whether a new AX user API key will be created or the existing `ARIZE_API_KEY` reused, and the project name that will be used.
+
+For a starter app, cover: the chosen provider and language, the target folder, the packages that will be installed, and the project name.
+
+Choose a default project name from the current folder or app name: lowercase it, replace spaces and unsupported punctuation with hyphens, and append `-arize-tracing` if it is too generic. Do not create an AX project explicitly — it is created on first trace ingestion.
+
+```text
+Here's my plan. Shall I proceed?
+```
+
+Do not continue until the user approves.
+
+## Step 6: Create the API key, write the env file, and instrument
+
+Only after approval, execute the plan in this order.
+
+### Choose the env file
+
+Pick the env file to match the app and use it for every variable below:
+
+- Next.js, Vite, or browser-adjacent TypeScript apps: `.env.local`
+- Python apps, Node scripts, backend services, or unknown type: `.env`
+- If the project already has exactly one of `.env` or `.env.local`, follow the existing convention.
+
+Do not read existing env file contents into chat. Preserve unrelated variables and never reveal their values.
+
+Make sure the env file is git-ignored before writing the API key to it — if the repo has a `.gitignore`, confirm it covers the file (add `.env` / `.env.local` if not); for a starter app you create, add one. The API key must never be committed to version control.
+
+### Write the non-secret variables
+
+Add these two lines to the env file you chose — as file contents, not shell commands. Create the file if needed; if a line already exists, update it in place rather than duplicating. They're not secret. Skip `ARIZE_SPACE_ID` if it's already set to the value you're using.
+
+```dotenv
+ARIZE_SPACE_ID=<space-id>
+ARIZE_PROJECT_NAME=<chosen-project-name>
+```
+
+### Create the API key
+
+**Skip this entirely if `ARIZE_API_KEY` was already present in Step 2** — reuse it and leave its env value untouched. Only create a key when you authenticated with browser OAuth and the app has no key yet.
+
+Create the key and write it into the env file in one step:
+
+```bash
+ax api-keys create --name "Local Arize AX tracing" --env-file .env
+```
+
+`--env-file` writes `ARIZE_API_KEY` atomically and **never prints it** — no temp file, no secret in your terminal or chat. Use `.env.local` if that's the app's convention; the file is created if missing, an existing `ARIZE_API_KEY` is replaced in place, and other variables are preserved.
+
+**Create the key exactly once** — a second `ax api-keys create` just orphans a still-active key. Git-ignore the env file (see above) *before* running this so the key is never committed.
+
+If key creation fails, have the user create one in the Arize AX UI and add it to the env file without exposing it in chat. Either way, create only one key.
+
+Handle the LLM provider's own key (e.g. `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`) in the same env file — the app can't make a call or produce traces without it:
+
+- **Existing app:** confirm the key is already present. If it's missing, ask the user for it and add it without echoing the value.
+- **New starter app:** add a placeholder line for the provider key the app needs (e.g. `OPENAI_API_KEY=` with an empty value) to the env file, so the user knows exactly what to fill in. Tell them to set it before running; if they share it now, fill it in without echoing the value.
+
+### Add instrumentation
+
+**Prefer the Arize instrumentation skill.** If you installed the skills in Step 1, load it now by reading its `SKILL.md` directly (a mid-session install isn't in your invocable list, so don't call it by name): read `arize-instrumentation/SKILL.md` from your agent's skills directory — project-level (`.claude/skills/`, `.cursor/skills/`, `.codex/…`, `.windsurf/…`) or the matching `~/.<agent>/…` global path. Follow the skill as your source of truth and **skip the rest of this section**. Use the docs fallback below only if the skill file doesn't exist (e.g. npx was missing in Step 1).
+
+**Route by the framework you detected — search the index for *its* name and follow that page.** Look up the framework you identified in Step 4 (not the provider it wraps) in the index at https://arize.com/docs/ax/integrations (machine-readable: https://arize.com/docs/llms.txt). Most agent frameworks have a dedicated page even though they aren't in the shortcuts below, so actually search the index before concluding a framework is unsupported — it lists every supported provider and framework with the exact, verified setup. Don't force a stack onto a listed framework's setup, and don't wander into unrelated Arize docs once you're on the right page.
+
+Common shortcuts:
+
+- OpenAI: https://arize.com/docs/ax/integrations/llm-providers/openai/openai-tracing
+- Anthropic: https://arize.com/docs/ax/integrations/llm-providers/anthropic/anthropic-tracing
+- LangChain Python: https://arize.com/docs/ax/integrations/python-agent-frameworks/langchain/langchain-tracing
+- LangChain.js: https://arize.com/docs/ax/integrations/ts-js-agent-frameworks/langchain/langchain-js
+- Vercel AI SDK v7: https://arize.com/docs/ax/integrations/ts-js-agent-frameworks/vercel/vercel-ai-sdk-v7-tracing
+- Vercel AI SDK v6 and earlier: https://arize.com/docs/ax/integrations/ts-js-agent-frameworks/vercel/vercel-ai-sdk-tracing
+
+**Only if the framework genuinely has no page in the index**, fall back in this order — never hand-roll a raw OpenTelemetry `TracerProvider` + OTLP exporter when a helper exists:
+
+1. Use a framework-specific OpenInference instrumentor if one exists (`openinference-instrumentation-<name>` / `@arizeai/openinference-instrumentation-<name>`), wired up with `arize-otel` per the manual-instrumentation guide; install it unpinned.
+1. Instrument the underlying provider (OpenAI, Anthropic, Bedrock, …) with its instrumentor **only if the framework calls the provider SDK directly**. Many agent frameworks instead drive the model through their own client layer and emit their own OpenTelemetry spans — a provider instrumentor captures **no traces** for those. Never reach for the provider instrumentor as a blind fallback just because you recognize an `openai`/`anthropic` client.
+1. Otherwise instrument manually via `arize-otel` (see https://arize.com/docs/ax/instrument/manual-instrumentation), or stop and ask the user if you still can't determine a setup.
+
+For existing apps:
+
+- Use the app's existing package manager and style.
+- Initialize tracing before LLM clients or frameworks are imported or created.
+- Preserve existing OpenTelemetry providers; add Arize as an additional exporter when needed.
+- Do not change business logic except to enable tracing.
+- If the app uses tools/function calls and the integration does not capture tool spans, add manual tool spans only after explaining that in the plan.
+
+For starter apps:
+
+- Every starter should make a real LLM call that invokes at least one tool (e.g. a calculator or a canned data lookup), with the model deciding to call it and the result fed back for a final answer — so the first traces show a multi-step trajectory (the LLM call plus the tool call as its own span), not one flat span.
+  - Agent frameworks (LangChain, Vercel AI SDK): define the tool the framework's way; the instrumentor captures the tool and agent spans automatically.
+  - Plain providers (OpenAI, Anthropic): use the provider's native tool/function calling, and add tool spans with the OpenInference decorators — `@tracer.tool` on the tool function and `@tracer.agent` on the top-level loop. These decorators require wrapping the tracer in `OITracer` from `openinference-instrumentation` (the raw `arize.otel.register()` tracer doesn't expose them and `@tracer.tool` will raise `AttributeError`); see https://arize.com/docs/ax/instrument/manual-instrumentation.
+- Ensure short-lived scripts flush/shut down the tracer provider before exit, or spans won't export.
+- Give a clear run command and note which provider key env var is needed.
+
+### Package guidance
+
+Install into the app's existing environment (its virtualenv if it has one) exactly the packages the detected framework's integration page lists — instrumentor names and peer dependencies differ per framework, so follow that page rather than copying from another stack or guessing versions.
+
+## Step 7: Run the app and poll for the first trace
+
+First confirm the LLM provider's API key is set in the env file (from Step 6); a missing provider key is the most common reason the run produces no traces.
+
+If you created a starter app in this flow, offer to run it for the user:
+
+```text
+Your starter app is ready. Want me to run it for you?
+```
+
+If they say yes, run it yourself with the run command, then poll. If they say no — or if you instrumented their existing app rather than creating a starter — tell them the exact run command and ask them to run it:
+
+```text
+Run your app with:
+
+<run command>
+
+It should make at least one LLM call. I'll poll Arize AX and let you know as
+soon as your first traces arrive.
+```
+
+Once the app has run (whether you ran it or the user did), poll for spans. Pass the actual project name and space ID as literal arguments — do not use `$ARIZE_PROJECT_NAME`/`$ARIZE_SPACE_ID`, which live in the env file and are not exported to your shell. Query every ~15 seconds for up to ~3 minutes.
+
+Span bodies can contain prompts, completions, tool arguments, and user data, so don't dump them into chat. Pipe the export through a counter that surfaces only how many spans arrived:
+
+```bash
+ax spans export "<project-name>" --space "<space-id>" --limit 5 --stdout \
+  | python3 -c 'import json,sys; d=json.load(sys.stdin); s=d if isinstance(d,list) else (d.get("spans") or d.get("data") or []); print(f"{len(s)} span(s) found")'
+```
+
+A non-zero count confirms traces are arriving. If you need to inspect a span to debug, write the export to a file outside the repo and read only the fields you need — never paste raw span bodies into chat.
+
+This uses the CLI profile from Step 2 (OAuth or api-key) — it works the same either way. If the export errors with an authentication failure, the profile isn't valid; re-run the Step 2 probe and re-authenticate, or fall back to having the user open the project in the Arize AX UI to confirm traces.
+
+- When spans come back, stop polling and continue to Step 8.
+- On timeout, do not fail silently. Tell the user no traces arrived yet, and give likely causes: app didn't make an LLM call, tracing initialized after the client was created, a short-lived script exited before flushing spans, or the wrong space/project/env file. Offer to re-check once they've run it again.
+
+Do not fabricate trace results. Only report traces the export command actually returned.
+
+## Step 8: Report the first traces with a link
+
+Once spans arrive, report the span count and give the user a link into Arize AX for the project. Point them at the UI to explore the trace contents rather than printing span bodies into chat.
+
+```text
+Your first traces are in Arize AX. Open project `<ARIZE_PROJECT_NAME>` here:
+https://app.arize.com/
+```
+
+If a space- or project-specific URL is known from the CLI or docs, prefer that deep link over the app root. Do not invent a URL structure you are not sure of; fall back to `https://app.arize.com/` plus instructions to select the project.
+
+## Step 9: Point at docs
+
+Finish with links so the user can go deeper:
+
+```text
+You're set up. To go further:
+
+- Docs index: https://arize.com/docs/ax
+- Tracing integrations: https://arize.com/docs/ax/integrations
+- AX CLI: https://arize.com/docs/api-clients/cli/overview
+```
+
+## Critical rules
+
+- Get the user's approval (Step 5) before creating AX resources, editing files, or installing dependencies.
+- Authenticate the CLI before any other `ax` call — every command, including the final trace check, uses the active profile; env vars alone don't authenticate it. On the browser OAuth branch, keep the command alive until its localhost callback completes (never close stdin, Ctrl-C, `pkill`, or probe while it waits) — sole exception: a brand-new email/password signup, whose link never calls back, so ask the user, then kill and re-run as a sign-in.
+- Never print, log, or summarize secrets in chat — API keys, env-file contents/values, or span bodies (prompts, completions, tool args, user data) — and never read env files into chat. Only report traces a command actually returned.
+- Create at most one AX API key: skip if `ARIZE_API_KEY` already exists (reuse it), otherwise a **single** `ax api-keys create --env-file <file>`. Git-ignore that file before creating the key; never create a second key.
+- Write the space **ID** (not its name) to `ARIZE_SPACE_ID`, or traces won't route; never create an AX project explicitly (it's made on first ingestion).
+- Initialize tracing before LLM clients are created, and flush/shut down the tracer before short-lived scripts exit. Vercel AI SDK v7 also needs Node.js 22+, `@ai-sdk/otel` registered, and `experimental_telemetry: { isEnabled: true }` per call.
+
+Docs: https://arize.com/docs/llms.txt

package/package.json

@@ -1,27 +1,35 @@
-{
-  "name": "evals",
-  "version": "2.2.8",
-  "description": "Arize evals package",
-  "type": "module",
-  "main": "cli.js",
-  "scripts": {
-    "postinstall": "echo \"run npx evals to complete setup\" && node spawn-interactive.js",
-    "start": "node cli.js"
-  },
-  "dependencies": {
-    "@arizeai/phoenix-evals": "^0.6.5",
-    "@arizeai/phoenix-otel": "*",
-    "ink": "^6.0.0",
-    "ink-gradient": "^3.0.0",
-    "react": "^19.0.0"
-  },
-  "bin": {
-    "evals": "./cli.js"
-  },
-  "keywords": [
-    "arize",
-    "evals"
-  ],
-  "author": "",
-  "license": "ISC"
-}
+{
+  "name": "evals",
+  "version": "2.3.0",
+  "description": "Arize AX onboarding — instrument your app with tracing via your coding agent",
+  "type": "module",
+  "main": "cli.js",
+  "scripts": {
+    "postinstall": "node -e \"console.log('Arize evals installed — run: npx evals')\"",
+    "start": "node cli.js",
+    "test": "node --test"
+  },
+  "dependencies": {
+    "ink": "^6.0.0",
+    "ink-gradient": "^3.0.0",
+    "react": "^19.0.0"
+  },
+  "bin": {
+    "evals": "./cli.js"
+  },
+  "files": [
+    "cli.js",
+    "onboarding-prompt.md",
+    "start.sh",
+    "start.ps1"
+  ],
+  "keywords": [
+    "arize",
+    "evals",
+    "tracing",
+    "observability",
+    "llm"
+  ],
+  "author": "",
+  "license": "ISC"
+}

package/start.ps1

@@ -0,0 +1,128 @@
+#!/usr/bin/env pwsh
+#
+# Arize AX onboarding launcher (Windows / PowerShell).
+#
+# Lets you pick an installed coding agent and launches it seeded with the
+# onboarding prompt to walk you through signup -> instrument -> first trace.
+# This is the no-npm path; if you have Node, `npx evals` gives the same thing
+# with a nicer UI.
+#
+# Run it:
+#   irm https://cdn.jsdelivr.net/npm/evals/start.ps1 | iex
+#
+# Non-interactive (irm|iex can't take params, so use the env var):
+#   $env:ARIZE_AGENT='claude'; irm .../start.ps1 | iex
+# Or via the scriptblock form, which does accept -Agent:
+#   & ([scriptblock]::Create((irm .../start.ps1))) -Agent claude
+
+[CmdletBinding()]
+param(
+  [string]$Agent = $env:ARIZE_AGENT
+)
+
+$ErrorActionPreference = 'Stop'
+
+# The prompt ships inside the public `evals` npm package; jsDelivr serves package
+# files over HTTP, so it's fetchable even though the source repo is private.
+# Override with $env:ARIZE_PROMPT_URL if needed.
+$PromptUrl = if ($env:ARIZE_PROMPT_URL) { $env:ARIZE_PROMPT_URL } else { 'https://cdn.jsdelivr.net/npm/evals/onboarding-prompt.md' }
+
+# Prefer the richer, bundled-prompt `npx evals` experience when Node is present.
+# This script is the no-npm fallback; if npx exists, hand off to it.
+# Set $env:ARIZE_SKIP_NPX=1 to force this shell path even when npx is available.
+if (-not $env:ARIZE_SKIP_NPX -and (Get-Command npx -ErrorAction SilentlyContinue)) {
+  npx --yes evals
+  exit $LASTEXITCODE
+}
+
+# Supported agents: id -> label, install URL, and whether the REPL is seeded with `-i`.
+$Agents = [ordered]@{
+  'claude'       = @{ Label = 'Claude Code';    Install = 'https://docs.claude.com/en/docs/claude-code'; Flag = $null }
+  'codex'        = @{ Label = 'OpenAI Codex';   Install = 'https://developers.openai.com/codex/cli';      Flag = $null }
+  'cursor-agent' = @{ Label = 'Cursor';         Install = 'https://docs.cursor.com/en/cli/overview';      Flag = $null }
+  'copilot'      = @{ Label = 'GitHub Copilot'; Install = 'https://github.com/features/copilot/cli';       Flag = '-i' }
+  'gemini'       = @{ Label = 'Gemini CLI';     Install = 'https://github.com/google-gemini/gemini-cli';   Flag = '-i' }
+}
+
+function Test-Agent([string]$id) {
+  return [bool](Get-Command $id -ErrorAction SilentlyContinue)
+}
+
+# Resolve which agent to launch.
+function Resolve-Agent {
+  if ($Agent) {
+    if (Test-Agent $Agent) { return $Agent }
+    Write-Host "Requested agent '$Agent' is not on your PATH." -ForegroundColor Yellow
+    return $null
+  }
+
+  $detected = @($Agents.Keys | Where-Object { Test-Agent $_ })
+
+  if ($detected.Count -eq 0) {
+    Write-Host "No supported coding agent found on your PATH." -ForegroundColor Yellow
+    Write-Host "Install one of these, then re-run:"
+    foreach ($id in $Agents.Keys) {
+      "{0,-14} {1}" -f $Agents[$id].Label, $Agents[$id].Install | Write-Host
+    }
+    return $null
+  }
+
+  if ($detected.Count -eq 1) { return $detected[0] }
+
+  Write-Host "Choose your coding agent:"
+  for ($i = 0; $i -lt $detected.Count; $i++) {
+    "  {0}) {1}" -f ($i + 1), $Agents[$detected[$i]].Label | Write-Host
+  }
+  while ($true) {
+    $choice = Read-Host "Enter a number [1-$($detected.Count)]"
+    if ($choice -match '^\d+$' -and [int]$choice -ge 1 -and [int]$choice -le $detected.Count) {
+      return $detected[[int]$choice - 1]
+    }
+    Write-Host "Invalid choice."
+  }
+}
+
+# --- Welcome ---
+Write-Host ""
+Write-Host "Arize AX" -ForegroundColor Magenta -NoNewline
+Write-Host " - Evals & Observability for Agentic AI" -ForegroundColor DarkGray
+Write-Host ""
+Write-Host "Let's get you tracing. I'll launch your coding agent with a guided prompt that"
+Write-Host "walks you through signup, instrumenting your app, and seeing your first traces."
+Write-Host ""
+
+$chosen = Resolve-Agent
+if (-not $chosen) { exit 1 }
+
+# Download the prompt to a temp file (rename the .tmp to .md rather than
+# leaving both behind).
+$tmp = New-TemporaryFile
+$promptFile = [System.IO.Path]::ChangeExtension($tmp.FullName, 'md')
+Rename-Item -Path $tmp.FullName -NewName (Split-Path $promptFile -Leaf)
+try {
+  Write-Host "Fetching the onboarding prompt..."
+  try {
+    Invoke-RestMethod -Uri $PromptUrl -OutFile $promptFile
+  } catch {
+    Write-Host "Failed to download the onboarding prompt from $PromptUrl" -ForegroundColor Red
+    Write-Host "Check your connection and try again."
+    exit 1
+  }
+
+  $seed = "Read the file $promptFile and follow it to set up Arize AX tracing in this project, walking me through each step and asking me questions as needed."
+
+  Write-Host ("Launching {0}..." -f $Agents[$chosen].Label)
+  Write-Host ""
+
+  # Launch interactive + seeded. No skip-permissions — the agent's approval
+  # model and the prompt's own approval gate must stay intact. External commands
+  # attach to the console, so the agent gets a real terminal.
+  $flag = $Agents[$chosen].Flag
+  if ($flag) {
+    & $chosen $flag $seed
+  } else {
+    & $chosen $seed
+  }
+} finally {
+  Remove-Item $promptFile -ErrorAction SilentlyContinue
+}