ax 0.10.0 → 0.10.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/.env.example CHANGED
@@ -1,18 +1,17 @@
1
1
  # ora platform API (agentic journeys / runs). Copy to `.env` and fill in.
2
- # The public commands (audit, webmcp-audit, deep-journey, skill) use the public
2
+ # The public commands (audit, webmcp-audit, deep-journey, skill, generate-files) use the public
3
3
  # ora.ai host and need none of these — `ax audit <url>` works with zero
4
4
  # configuration, and so does `ax webmcp-audit http://localhost:3000`.
5
5
 
6
6
  # Secret API key (looks like ora_sk_...). Required by `journey` and by
7
7
  # `audit --tunnel ora` (needs the tunnels:write + tunnels:connect scopes), which
8
- # exchange it for a short-lived bearer token. `webmcp-audit` sends it as a bearer
9
- # token to lift the anonymous ingest limits (10/min, 20 per 24h per IP); optional there.
8
+ # exchange it for a short-lived bearer token.
10
9
  ORA_API_KEY=
11
10
 
12
- # Platform API base. Defaults to staging if unset.
11
+ # Platform API base. Defaults to https://api.agentfront.sh (production) if unset.
13
12
  ORA_PLATFORM_URL=https://api.agentfront.sh
14
13
 
15
- # Public API base, used by audit, webmcp-audit, deep-journey and skill.
14
+ # Public API base, used by audit, webmcp-audit, deep-journey, skill and generate-files.
16
15
  # Defaults to https://ora.ai if unset. Point it at a local ora to test against
17
16
  # one (e.g. http://localhost:3000).
18
17
  # ORA_API_URL=https://ora.ai
@@ -34,5 +33,6 @@ ORA_PLATFORM_URL=https://api.agentfront.sh
34
33
 
35
34
  # ora-issued partner API key for `ax deep-journey` (same as --api-key):
36
35
  # unlocks free-text --task runs and the 1000 runs/24h keyed allowance
37
- # (no per-target cap, no burst guard). Issued manually by ora.
36
+ # (no per-target cap, no burst guard). Issued manually by ora. `ax webmcp-audit`
37
+ # also sends it (then ORA_SCAN_API_KEY) to lift its daily ingest limit.
38
38
  # ORA_PARTNER_API_KEY=
package/README.md CHANGED
@@ -18,13 +18,13 @@
18
18
  npx ax audit https://stripe.com
19
19
  ```
20
20
 
21
- **No account, no API key.** Everything below works out of the box — the one exception is [`journey`](#for-ora-partners), which drives ora's partner platform.
21
+ **No account, no API key.** Everything below works out of the box. The exceptions are [`journey`](#for-ora-partners) and `audit --tunnel ora`, which need an ora platform key, and `deep-journey --task`, which needs a partner key.
22
22
 
23
23
  <p align="center">
24
24
  <img src="https://raw.githubusercontent.com/ora/ax/main/.github/assets/demo.gif" width="880" alt="ax audit https://stripe.com — live progress, then the layered report and ora's ranked top fixes" />
25
25
  </p>
26
26
 
27
- Five commands:
27
+ Six commands:
28
28
 
29
29
  - **`audit <url>`** — run ora's hosted agent-readiness audit against a site: live progress, then a layered report of what passed, what's broken, and ora's ranked list of the highest-impact fixes (or `--json` for the raw contract payload). No account or API key needed. Gate CI with `--min-score`.
30
30
  - **`webmcp-audit <url>`** — audit a page's `document.modelContext` tools in a real browser. Works on any http(s) URL, localhost included, so you can check a surface before you publish it. Measures the page the way ora's capture worker does and has ora score it; nothing is stored, published, or ranked.
@@ -91,7 +91,7 @@ Exit codes are the contract:
91
91
  | `2` | usage error — bad flags, malformed URL, local target without a tunnel |
92
92
  | `3` | API unreachable, timeout, or rate limit exhausted |
93
93
 
94
- Budget notes: ora allows 30 scans + 6 `--force` scans per rolling 24h per IP (plus a 10/min burst limit). Results served from the freshness cache cost nothing, so a CI job that audits on every push stays well inside the budget — tune the window with `--max-age`, and reserve `--force` for verifying a fix you just deployed. High-volume callers can present an ora-issued scan API key (`--api-key` or `ORA_SCAN_API_KEY`), which exempts them from all of these limits; keys are issued manually by ora (no self-serve signup), and an unrecognized key silently falls back to the keyless limits rather than erroring. An auth-gated MCP target is not a failure: ora answers with an `MCP_AUTH_REQUIRED` error and no score; ax prints the notice (or the error object with `--json`), skips `--min-score`, and exits `0`.
94
+ Budget notes: ora allows 30 scans + 6 `--force` scans per rolling 24h per IP (plus a 10/min burst limit). Results served from the freshness cache cost nothing, so a CI job that audits on every push stays well inside the budget — tune the window with `--max-age`, and reserve `--force` for verifying a fix you just deployed. High-volume callers can present an ora-issued scan API key (`--api-key` or `ORA_SCAN_API_KEY`), which exempts them from these limits (a key scoped to `pap:read` keeps a daily cap and cannot `--force`); keys are issued manually by ora (no self-serve signup), and an unrecognized key silently falls back to the keyless limits rather than erroring. An auth-gated MCP target is not a failure: ora answers with an `MCP_AUTH_REQUIRED` error and no score; ax prints the notice (or the error object with `--json`), skips `--min-score`, and exits `0`.
95
95
 
96
96
  ### Auditing localhost (`--tunnel ora`, `--tunnel-cmd`)
97
97
 
@@ -144,7 +144,7 @@ or bundles a browser. Set `CHROME_PATH` if yours lives somewhere unusual.
144
144
  To drive a browser you started yourself, pass `--chrome-endpoint` (a `ws://` URL, an http origin,
145
145
  or `host:port`) and start it with `--remote-debugging-port`. The command never attaches to a
146
146
  browser you did not name — including anything already on port 9222 — because silently borrowing a
147
- Chrome with the WebMCP flags on would score your page higher here than it scores on ora.
147
+ Chrome with the WebMCP flags on would make your page read differently here than it does on ora.
148
148
 
149
149
  **Do not enable the WebMCP flags on a Chrome you point it at.** The audit asks whether a real
150
150
  browser gives an agent WebMCP on your page. A Chrome started with `--enable-features=WebMCPTesting`
@@ -158,15 +158,19 @@ present in order to call your tools.
158
158
  passes and the same probe code ora's worker runs, plus a viewport screenshot at ora's 1280×720. As
159
159
  ora does, it loads the page with images, media and webfonts refused — stylesheets still apply — so
160
160
  the screenshot the grader sees is the one ora would have taken. That capture is
161
- posted to ora, which runs the 16 checks, two model hops and the tool-selection simulation, and
162
- streams back a score. The CLI scores nothing itself (design decision #1) — grade, findings,
161
+ posted to ora, which runs its 19 checks, two model hops and the tool-selection simulation, and
162
+ streams back a score. The CLI never calls your tools, so the three checks that need a tool call
163
+ (`invoke-success-rate`, `visible-effect`, `hint-vs-observed`) read `Not applicable` on a local
164
+ audit. The CLI scores nothing itself (design decision #1) — grade, findings,
163
165
  category scores and the simulation are all rendered as ora sent them.
164
166
 
165
- **Availability is a gate, not a score.** Before anything is graded, ora decides whether an
166
- in-browser agent can use the page at all. A page that cannot comes back `not agent-ready` with the
167
- reasons why, and no score or grade — the reasons are the answer.
167
+ **Availability is reported, not scored.** ora says whether an in-browser agent can use the page
168
+ (`agent-ready`, `not agent-ready`, or unknown) and why, and the report leads with it. It never
169
+ changes the score: a `not agent-ready` page still gets its score and pillars. The one exception is
170
+ a page whose only tools are declarative forms: its pillars are scored, and its grade is withheld,
171
+ because ora finds those tools in the markup and cannot watch a browser register them.
168
172
 
169
- Past that gate the score is four pillars — `shared-experience` (30), `task-completion` (25),
173
+ The score is four pillars — `shared-experience` (30), `task-completion` (25),
170
174
  `tool-quality` (25), `trust` (20). It is **not** the 0–100 agent-readiness score from `ax audit`:
171
175
  different inputs, different rubric, never comparable.
172
176
 
@@ -190,8 +194,9 @@ the leaderboard, the badge, or your public scorecard — to appear there, publis
190
194
  on webmcp.ora.ai. Pass `--json` and pipe it to a file if you want to see exactly what was sent
191
195
  back; the capture itself is what the report's header describes.
192
196
 
193
- Anonymous use is rate-limited to 10/min and 20 per 24h per IP; an ora API key
194
- (`--api-key` / `ORA_API_KEY`) lifts that.
197
+ Anonymous use is rate-limited to 10/min and 20 per 24h per IP. An ora-issued partner or scan API
198
+ key (`--api-key`, or `ORA_PARTNER_API_KEY` / `ORA_SCAN_API_KEY`) lifts the daily limit; the
199
+ per-minute limit still applies.
195
200
 
196
201
  ### CI gate
197
202
 
@@ -201,8 +206,9 @@ Anonymous use is rate-limited to 10/min and 20 per 24h per IP; an ora API key
201
206
 
202
207
  Any CI image with Chrome installed works; nothing to launch first.
203
208
 
204
- Exit codes match `audit`: `0` ok · `1` below `--min-score` · `2` usage (including no debuggable
205
- Chrome) · `3` API. The gate reads the score, not the grade — a withheld grade would otherwise let
209
+ Exit codes follow `audit`: `0` ok · `1` below `--min-score` · `2` usage (no debuggable Chrome, the
210
+ page did not load, or a capture ora would refuse) · `3` API, a failed capture, or no score
211
+ returned. The gate reads the score, not the grade — a withheld grade would otherwise let
206
212
  everything through.
207
213
 
208
214
  ## deep-journey
@@ -230,18 +236,18 @@ Runs a real AI agent at a site through ora's **public** journey API — no key,
230
236
  For the 4-layer readiness audit, run: ax audit stripe.com
231
237
  ```
232
238
 
233
- Anonymous callers run curated intents (`pricing`, `signup`, `api-docs`, `integrate`, `support`, …) — list them with `GET https://ora.ai/api/journey/intents`, and the accepted agents with `GET https://ora.ai/api/journey/agents`. The public caps are 100 runs per rolling 24h per target and 20 runs per 24h per IP (plus a 20/min burst limit and a shared ceiling of 3,000 anonymous runs a day across all callers); a target at its cap answers with the most recent stored run (`rate_limited: true`) instead of an error, so repeat CI invocations are cheap. Verdict and step count come from ora's contract as-is — the CLI never re-judges a run.
239
+ Anonymous callers run curated intents (`pricing`, `signup`, `api-docs`, `integrate`, `support`, …) — list them with `GET https://ora.ai/api/journey/intents`, and the publicly offered agents with `GET https://ora.ai/api/journey/agents`. The public caps are 100 runs per rolling 24h per target and task, and 20 runs per 24h per IP (plus a 20/min burst limit and a shared ceiling of 3,000 anonymous runs a day across all callers); a target at its cap answers with the most recent stored run (`rate_limited: true`) instead of an error, so repeat CI invocations are cheap. Verdict and step count come from ora's contract as-is — the CLI never re-judges a run.
234
240
 
235
241
  **Keyed tier:** an ora-issued partner API key (`--api-key`, or `ORA_PARTNER_API_KEY` / `ORA_SCAN_API_KEY` in the environment) unlocks `--task` — a free-text task of 4–300 characters that replaces the curated intent — and moves the caller to an allowance of 1000 runs per rolling 24h per key with no per-target cap and no burst guard. Keys are issued manually by ora (no self-serve signup). A wrong or unrecognized key with a *curated* intent silently degrades to the anonymous tier; `--task` without a recognized key is an error.
236
242
 
237
- The CLI always follows the run's stream to the end, `--no-stream` included, because ora saves a finished run from inside the stream. If the stream drops or goes quiet (no frames for 120s), the CLI reopens it, which resumes the same run without re-running the agent. A stream `429` is waited out (up to 60s) before reopening, and a `404` or auth error ends the command at once. Only after five failed reopens does it fall back to polling the run detail, within a 15-minute budget shared with the stream.
243
+ The CLI always follows the run's stream to the end, `--no-stream` included, because ora saves a finished run from inside the stream. If the stream drops or goes quiet (no frames for 120s), the CLI reopens it, which resumes the same run without re-running the agent. A stream `429` is waited out (up to 60s) before reopening; any other HTTP error from the stream (a `404`, an auth error, a `5xx`) ends the command at once. Only after five failed reopens does it fall back to polling the run detail, within a 15-minute budget shared with the stream.
238
244
 
239
245
  | Flag | Effect |
240
246
  |---|---|
241
247
  | `--intent <id>` | Curated task id (server default when omitted) |
242
248
  | `--task <text>` | Free-text task — needs a partner API key; mutually exclusive with `--intent` |
243
249
  | `--api-key <k>` | ora partner API key; also read from `ORA_PARTNER_API_KEY` (then `ORA_SCAN_API_KEY`) |
244
- | `--agent <id>` | Agent from the public roster (default: ora's pick) |
250
+ | `--agent <id>` | Agent from the public roster (default: `cas-haiku`, Claude Code on the Haiku model ora's roster lists) |
245
251
  | `--no-stream` | Hide the live trajectory view and print only the final result |
246
252
  | `--json` | Print the terminal run detail (verdict, step_count, result) as JSON |
247
253
 
@@ -258,7 +264,7 @@ ax skill agent-ready-website --install # write .claude/skills/<name>/SKILL.md
258
264
  ax skill --json # the raw registry index
259
265
  ```
260
266
 
261
- Skills come from ora's public registry (`https://ora.ai/.well-known/agent-skills/`) and every byte is verified against the registry's sha256 digest before it is printed or installed. Skill content is never bundled into this package. `--dir <path>` overrides the install directory, and `--json` prints the registry index as JSON instead of the formatted list.
267
+ Skills come from ora's public registry (`https://ora.ai/.well-known/agent-skills/`) and every skill whose index entry carries a sha256 digest is verified against it, byte for byte, before it is printed or installed. Skill content is never bundled into this package. `--dir <path>` overrides the install directory, and `--json` prints the registry index as JSON instead of the formatted list.
262
268
 
263
269
  ## generate-files
264
270
 
@@ -268,7 +274,7 @@ ax generate-files example.com --out public # writes into public/ (your site's w
268
274
  ax generate-files example.com --json # ora's payload, files + notes, nothing written
269
275
  ```
270
276
 
271
- Drafts the files agents look for on a site, filled in from ora's last scan of the domain (no new crawl), so run `ax audit` first for a domain ora has never scanned:
277
+ Drafts the files agents look for on a site, filled in from ora's last scan of the domain (no new crawl). For a domain ora has never scanned you get only a website entry and a minimal `poppy.json`, so run `ax audit` first to fill in the rest:
272
278
 
273
279
  | File | Written to | What it is |
274
280
  |---|---|---|
@@ -277,9 +283,9 @@ Drafts the files agents look for on a site, filled in from ora's last scan of th
277
283
  | `llms.txt` | web root | An index of the docs, APIs and agent endpoints ora found; skipped when you already publish one |
278
284
  | `robots.txt` | web root | An edit of your current robots.txt: every line kept, a group added for AI search crawlers and user-requested agents your `User-agent: *` rules block (copying your other rules), and a `Sitemap:` line. Training crawlers and crawlers you block by name are never touched |
279
285
  | `api-catalog` | `.well-known/` | An [RFC 9727](https://www.rfc-editor.org/rfc/rfc9727) API catalog pointing at your OpenAPI description |
280
- | `server-card` | `<mcp path>/` | A draft MCP server card for each MCP server that has none. A card for a server on another host (`mcp.example.com`) is printed, not written |
286
+ | `server-card` | `<mcp path>/` | A draft MCP server card for each MCP server that has none. A card for a server on another host (`mcp.example.com`) is listed with its URL, not written; `--json` carries its body |
281
287
 
282
- A file ora cannot draft honestly (you already publish it, or ora never read the original) is left out and explained under "Not drafted". These are drafts: read each file and its notes before you publish.
288
+ A file ora cannot draft honestly (you already publish it, ora never read the original, or ora found nothing to put in it) is left out and explained under "Not drafted". These are drafts: read each file and its notes before you publish.
283
289
 
284
290
  Nothing is overwritten: if any target file exists - your `robots.txt` included - nothing is written and the command exits `2`. Pass `--force` to overwrite, or `--out <dir>` to write elsewhere and compare. Exit codes: `0` written, `2` usage (bad domain, files exist), `3` API. Same files as the web generator at [directory.ora.ai/generate](https://directory.ora.ai/generate) and `GET https://ora.ai/api/agent-files/{domain}?include=all`.
285
291
 
@@ -308,14 +314,14 @@ const { detail } = await deepJourney("stripe.com", {
308
314
  harness: agent.harness,
309
315
  model: agent.model,
310
316
  });
311
- console.log(detail.verdict, detail.step_count, detail.result?.insight.summary);
317
+ console.log(detail.verdict, detail.step_count, detail.result?.insight?.summary);
312
318
  ```
313
319
 
314
320
  `detail` is the contract's `JourneyRunDetail`; `fetchJourneyIntents` lists the curated intents. The same public caps apply — a capped target resolves with the most recent stored run (`cached: true` on the outcome), not an error.
315
321
 
316
322
  ## For ora partners
317
323
 
318
- Everything above runs without an account. The commands and flags in this section are for teams working with ora directly: they need an ora-issued API key, and there is no self-serve signup — keys are issued by ora (contact [hello@ora.ai](mailto:hello@ora.ai)). A partner key also lifts the anonymous rate limits on `audit`, `webmcp-audit`, and `deep-journey` via their `--api-key` flags.
324
+ Everything above runs without an account. The commands in this section are for teams working with ora directly, and there is no self-serve signup: keys are issued by ora (contact [hello@ora.ai](mailto:hello@ora.ai)). Two kinds of key exist. `journey` (and `audit --tunnel ora`) needs an ora platform key (`ora_sk_…`). A separate partner or scan key, passed with `--api-key`, lifts the anonymous rate limits on `audit` and `deep-journey`, and the daily limit on `webmcp-audit`.
319
325
 
320
326
  ### journey
321
327
 
@@ -386,14 +392,15 @@ A `.env` in the working directory is read on startup — copy `.env.example` and
386
392
 
387
393
  | Var | Used by | Default | Purpose |
388
394
  |---|---|---|---|
389
- | `ORA_API_URL` | audit, deep-journey, skill, generate-files | `https://ora.ai` | Public API base (no auth) |
395
+ | `ORA_API_URL` | audit, webmcp-audit, deep-journey, skill, generate-files | `https://ora.ai` | Public API base (no auth); also the ingest base for `webmcp-audit` |
390
396
  | `ORA_PLATFORM_URL` | journey, audit `--tunnel ora` | `https://api.agentfront.sh` | Authenticated platform API base |
391
397
  | `ORA_TUNNEL` | audit | — (optional) | `ora` opens ora's own tunnel for a local target (same as `--tunnel ora`) |
392
398
  | `ORA_TUNNEL_CMD` | audit | — (optional) | Tunnel command for a local target (same as `--tunnel-cmd`); wins over `ORA_TUNNEL` |
393
- | `ORA_API_URL` | webmcp-audit | `https://ora.ai` | Also the ingest base for `webmcp-audit` |
394
- | `ORA_API_KEY` | journey, audit `--tunnel ora`, webmcp-audit | — (required for journey and `--tunnel ora`) | Secret key (`ora_sk_…`); journey and `--tunnel ora` exchange it for a bearer token (the tunnel needs the `tunnels:write` + `tunnels:connect` scopes), webmcp-audit sends it to lift the ingest rate limits |
395
- | `ORA_SCAN_API_KEY` | audit, deep-journey | — (optional) | ora-issued scan API key; lifts every scan rate limit (issued manually by ora). deep-journey accepts it as a partner-key fallback |
396
- | `ORA_PARTNER_API_KEY` | deep-journey | — (optional) | ora-issued partner API key; unlocks `--task` and the 1000/24h keyed allowance |
399
+ | `ORA_API_KEY` | journey, audit `--tunnel ora` | — (required for both) | ora platform key (`ora_sk_…`); exchanged for a bearer token (the tunnel needs the `tunnels:write` + `tunnels:connect` scopes) |
400
+ | `ORA_SCAN_API_KEY` | audit, deep-journey, webmcp-audit | — (optional) | ora-issued scan API key; lifts every scan rate limit (issued manually by ora). deep-journey and webmcp-audit accept it as a partner-key fallback |
401
+ | `ORA_PARTNER_API_KEY` | deep-journey, webmcp-audit | — (optional) | ora-issued partner API key; unlocks `--task` and the 1000/24h keyed allowance on deep-journey, and lifts webmcp-audit's daily limit |
402
+ | `CHROME_PATH` | webmcp-audit | — (auto-detected) | The Chrome to launch, when yours is not in a standard location |
403
+ | `AX_WEBMCP_NO_SANDBOX` | webmcp-audit | — (unset) | `1` launches Chrome without its sandbox. Containers only: the audited page is untrusted and the sandbox contains it |
397
404
 
398
405
  ## Contract versioning
399
406