evals 2.3.0 → 2.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +17 -4
- package/onboarding-prompt.md +6 -4
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -46,10 +46,23 @@ The agent runs with its **normal permission model** — `evals` never passes ski
|
|
|
46
46
|
|
|
47
47
|
The shell launchers also accept `--agent <id>`.
|
|
48
48
|
|
|
49
|
-
##
|
|
49
|
+
## Development
|
|
50
50
|
|
|
51
|
-
|
|
51
|
+
```bash
|
|
52
|
+
npm install
|
|
53
|
+
npm test # unit tests for the pure helpers (node --test)
|
|
54
|
+
node cli.js # run the interactive CLI locally
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
`npm test` covers the headless logic (URL tagging, agent detection, the agent catalog, and the picker). The interactive Ink flow and the live agent launch need a real terminal — verify those by running `node cli.js` in a terminal and picking an agent.
|
|
52
58
|
|
|
53
|
-
|
|
59
|
+
### Layout
|
|
54
60
|
|
|
55
|
-
|
|
61
|
+
- `cli.js` — the interactive `npx evals` entry point (Ink UI). Exports its pure helpers for testing; only launches the UI when run directly.
|
|
62
|
+
- `onboarding-prompt.md` — the bundled onboarding prompt (a copy of the Arize docs landing-page prompt).
|
|
63
|
+
- `start.sh` / `start.ps1` — the no-npm launchers, served to `curl`/`irm` via jsDelivr.
|
|
64
|
+
- `test/` — unit tests.
|
|
65
|
+
|
|
66
|
+
## About Arize
|
|
67
|
+
|
|
68
|
+
[Arize AX](https://arize.com/docs/ax) is the AI engineering platform for tracing, evaluating, and observing LLM and agent applications. Learn more at [arize.com](https://arize.com).
|
package/onboarding-prompt.md
CHANGED
|
@@ -23,7 +23,7 @@ Do not proceed until the user approves.
|
|
|
23
23
|
|
|
24
24
|
## Prerequisites
|
|
25
25
|
|
|
26
|
-
The AX CLI must be **arize-ax-cli `0.
|
|
26
|
+
The AX CLI must be **arize-ax-cli `0.29.0` or newer** (Step 1 installs the latest) and needs **Python 3.11+** — a hard requirement. If Python is missing, stop and have the user install it from https://www.python.org/downloads/ and re-run. **Node.js 18+ with npx** is optional but installs the Arize skills; without it, don't stop — continue and use the docs paths in Step 6 (the Vercel AI SDK v7 starter needs Node.js 22+).
|
|
27
27
|
|
|
28
28
|
## Step 1: Install or update the AX CLI and Arize skills
|
|
29
29
|
|
|
@@ -77,9 +77,11 @@ If a `default` profile exists but the probe failed, it's signed out or expired,
|
|
|
77
77
|
- **No key anywhere** — sign up or sign in with browser OAuth:
|
|
78
78
|
|
|
79
79
|
```bash
|
|
80
|
-
ax profiles create default --auth-method oauth
|
|
80
|
+
ax profiles create default --auth-method oauth --utm-params "utm_source=npmevals&utm_medium=cli&utm_campaign=prompt-first-onboarding"
|
|
81
81
|
```
|
|
82
82
|
|
|
83
|
+
The `--utm-params` flag tags a new sign-up with onboarding attribution. It's already included on the three browser OAuth commands that take it — the two `ax profiles create --auth-method oauth` calls and the `ax auth login` fallback. Pass those verbatim; don't drop it, change the values, or add the flag to the api-key `profiles create` or any other `ax` command.
|
|
84
|
+
|
|
83
85
|
Always pass the positional profile name `default`. Without it, the CLI prompts `profile name [default]:`, receives EOF from an agent-run command, and exits with `Goodbye!` without creating a profile.
|
|
84
86
|
|
|
85
87
|
The rest of this step applies only to the **browser OAuth** branch.
|
|
@@ -105,7 +107,7 @@ Did you just sign up for a new account with email and password?
|
|
|
105
107
|
**If the user says yes (new email/password signup):** the localhost callback will never fire and the command will wait forever, so this is the one time you break the "keep it alive" rule. Once they confirm they've clicked the validation link and their account exists, terminate the waiting OAuth command (send Ctrl-C / kill that process), then re-run it:
|
|
106
108
|
|
|
107
109
|
```bash
|
|
108
|
-
ax profiles create default --auth-method oauth
|
|
110
|
+
ax profiles create default --auth-method oauth --utm-params "utm_source=npmevals&utm_medium=cli&utm_campaign=prompt-first-onboarding"
|
|
109
111
|
```
|
|
110
112
|
|
|
111
113
|
The second run is now a plain sign-in for the validated account. Its callback lands normally, so handle it with all the standard rules below — keep it alive and wait for it to exit on its own.
|
|
@@ -121,7 +123,7 @@ ax spaces list --limit 1 --output json
|
|
|
121
123
|
If the probe succeeds, continue. If the default OAuth profile already existed and the probe returns an authentication error, the profile is signed out or expired — run the fallback:
|
|
122
124
|
|
|
123
125
|
```bash
|
|
124
|
-
ax auth login
|
|
126
|
+
ax auth login --utm-params "utm_source=npmevals&utm_medium=cli&utm_campaign=prompt-first-onboarding"
|
|
125
127
|
```
|
|
126
128
|
|
|
127
129
|
Handle `ax auth login` with the same rules: keep its callback server alive, wait for it to exit or print a success line, then rerun the probe. Do not run `ax auth login` immediately after creating an OAuth profile; the profile creation already performed login.
|