ax 0.6.0 → 0.7.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/.env.example CHANGED
@@ -1,14 +1,19 @@
1
1
  # ora platform API (agentic journeys / runs). Copy to `.env` and fill in.
2
- # The public commands (audit, deep-journey, skill) use the public ora.ai host
3
- # and need none of these — `ax audit <url>` works with zero configuration.
2
+ # The public commands (audit, webmcp-audit, deep-journey, skill) use the public
3
+ # ora.ai host and need none of these — `ax audit <url>` works with zero
4
+ # configuration, and so does `ax webmcp-audit http://localhost:3000`.
4
5
 
5
- # Secret API key (looks like ora_sk_...). Exchanged for a short-lived bearer token.
6
+ # Secret API key (looks like ora_sk_...). Required by `journey`, which exchanges
7
+ # it for a short-lived bearer token. `webmcp-audit` sends it as a bearer token to
8
+ # lift the anonymous ingest limits (10/min, 20 per 24h per IP); optional there.
6
9
  ORA_API_KEY=
7
10
 
8
11
  # Platform API base. Defaults to staging if unset.
9
12
  ORA_PLATFORM_URL=https://api.agentfront.sh
10
13
 
11
- # Public scan API base. Defaults to https://ora.ai if unset.
14
+ # Public API base, used by audit, webmcp-audit, deep-journey and skill.
15
+ # Defaults to https://ora.ai if unset. Point it at a local ora to test against
16
+ # one (e.g. http://localhost:3000).
12
17
  # ORA_API_URL=https://ora.ai
13
18
 
14
19
  # Tunnel command for auditing a local target (same as --tunnel-cmd):
package/README.md CHANGED
@@ -6,9 +6,10 @@ Score any site's agent readiness — and watch real AI agents navigate it. Power
6
6
  npx ax audit https://stripe.com
7
7
  ```
8
8
 
9
- Four commands:
9
+ Five commands:
10
10
 
11
11
  - **`audit <url>`** — run ora's hosted agent-readiness audit against a site: live progress, then a layered report of what passed, what's broken, and ora's ranked list of the highest-impact fixes (or `--json` for the raw contract payload). No account or API key needed. Gate CI with `--min-score`.
12
+ - **`webmcp-audit <url>`** — audit a page's `document.modelContext` tools in a real browser. Works on any http(s) URL, localhost included, so you can check a surface before you publish it. Measures the page the way ora's capture worker does and has ora score it; nothing is stored, published, or ranked.
12
13
  - **`deep-journey <url>`** — run a real AI agent at a site on one of ora's curated tasks through the public journey API: no key, no workspace. A partner API key unlocks free-text tasks (`--task`) and a larger allowance.
13
14
  - **`journey "<intent>"`** — send a real AI agent (claude-code, codex, …) at a site and watch its navigation live as a boxed node-graph, then get the scored insight, tokens, and cost. Requires an `ORA_API_KEY`.
14
15
  - **`skill [name]`** — list, print, or install ora's agent skills, digest-verified from the public registry.
@@ -85,6 +86,94 @@ ax audit localhost:3000 --tunnel-cmd 'ngrok http 3000 --log stdout'
85
86
  - Off-site checks (registry listings, brand search) usually fail for a throwaway tunnel hostname — the report says so. Use tunnel audits to iterate on your on-site surface, not to compare scores.
86
87
  - Free tiers of some tunnel vendors serve an interstitial warning page to browser-like requests, which can distort what the scanner sees — prefer a vendor/plan that serves your origin directly.
87
88
 
89
+ ## webmcp-audit
90
+
91
+ ```
92
+ ax webmcp-audit <url> [--chrome-endpoint e] [--min-score n] [--api-key k] [--json] [--show-passing]
93
+ ```
94
+
95
+ ```bash
96
+ ax webmcp-audit http://localhost:3000
97
+ ```
98
+
99
+ The dev-loop counterpart to the audit at [webmcp.ora.ai](https://webmcp.ora.ai): check the
100
+ `document.modelContext` tools your page registers *before* you ship it, and get the same answer
101
+ you will get once it is public.
102
+
103
+ Any `http(s)` URL works — a dev server, a staging host, or a site you do not own. A page with no
104
+ WebMCP surface is not an error: it comes back `absent` with the checks that cannot apply marked
105
+ `Not applicable`, which is a perfectly good way to see what the audit looks like before you have
106
+ written any tools.
107
+
108
+ **It uses a Chrome you already have.** WebMCP tools only exist inside a live page, so the audit
109
+ opens yours in a real browser. It finds your installed Chrome, starts a headless one on a throwaway
110
+ profile, and stops it when the audit finishes — nothing to set up, and this package never downloads
111
+ or bundles a browser. Set `CHROME_PATH` if yours lives somewhere unusual.
112
+
113
+ To drive a browser you started yourself, pass `--chrome-endpoint` (a `ws://` URL, an http origin,
114
+ or `host:port`) and start it with `--remote-debugging-port`. The command never attaches to a
115
+ browser you did not name — including anything already on port 9222 — because silently borrowing a
116
+ Chrome with the WebMCP flags on would score your page higher here than it scores on ora.
117
+
118
+ **Do not enable the WebMCP flags on a Chrome you point it at.** The audit asks whether a real
119
+ browser gives an agent WebMCP on your page. A Chrome started with `--enable-features=WebMCPTesting`
120
+ answers yes for every site, so a page can come back `active` here and `testing-only` on ora. The
121
+ report tells you when the browser answered for the page; where that actually moved a check, ora
122
+ annotates that check itself. Keep a flag-enabled Chrome for
123
+ [`@ora-ai/webmcp-verify`](https://www.npmjs.com/package/@ora-ai/webmcp-verify), which needs the API
124
+ present in order to call your tools.
125
+
126
+ **What it measures, and where.** The capture is local: three passes over your entry page, the same
127
+ passes and the same probe code ora's worker runs, plus a viewport screenshot at ora's 1280×720. As
128
+ ora does, it loads the page with images, media and webfonts refused — stylesheets still apply — so
129
+ the screenshot the grader sees is the one ora would have taken. That capture is
130
+ posted to ora, which runs the 16 checks, two model hops and the tool-selection simulation, and
131
+ streams back a score. The CLI scores nothing itself (design decision #1) — grade, findings,
132
+ category scores and the simulation are all rendered as ora sent them.
133
+
134
+ **Availability is a gate, not a score.** Before anything is graded, ora decides whether an
135
+ in-browser agent can use the page at all. A page that cannot comes back `not agent-ready` with the
136
+ reasons why, and no score or grade — the reasons are the answer.
137
+
138
+ Past that gate the score is four pillars — `shared-experience` (30), `task-completion` (25),
139
+ `tool-quality` (25), `trust` (20). It is **not** the 0–100 agent-readiness score from `ax audit`:
140
+ different inputs, different rubric, never comparable.
141
+
142
+ A pillar can read 0 because it earned nothing or because nothing in it could be measured; the
143
+ report marks the second case rather than leaving you to guess.
144
+
145
+ **Two check statuses mean "no score" and are not the same thing.** `Not applicable` means the check
146
+ had nothing to measure and your page is not charged for it. `Not measured` means it applied and
147
+ could not be measured — that counts against evidence coverage, and below the coverage gate ora
148
+ withholds the letter grade while still reporting the score. A withheld grade beside a real score is
149
+ normal, not an error.
150
+
151
+ **What leaves your machine.** Scoring happens on ora's side, so the capture is uploaded: the page
152
+ URL, every tool your page registers (names, descriptions, JSON Schemas, annotations), the WebMCP
153
+ evidence found in your page's own scripts, and **a 1280×720 screenshot of the page**. Point it at a
154
+ logged-in or seeded dev environment and that screenshot shows whatever was on screen. Your source
155
+ code is never read and never sent.
156
+
157
+ **Nothing is stored.** A local audit is scored and returned and then forgotten. It never reaches
158
+ the leaderboard, the badge, or your public scorecard — to appear there, publish and run the audit
159
+ on webmcp.ora.ai. Pass `--json` and pipe it to a file if you want to see exactly what was sent
160
+ back; the capture itself is what the report's header describes.
161
+
162
+ Anonymous use is rate-limited to 10/min and 20 per 24h per IP; an ora API key
163
+ (`--api-key` / `ORA_API_KEY`) lifts that.
164
+
165
+ ### CI gate
166
+
167
+ ```yaml
168
+ - run: npx ax webmcp-audit http://localhost:3000 --min-score 70
169
+ ```
170
+
171
+ Any CI image with Chrome installed works; nothing to launch first.
172
+
173
+ Exit codes match `audit`: `0` ok · `1` below `--min-score` · `2` usage (including no debuggable
174
+ Chrome) · `3` API. The gate reads the score, not the grade — a withheld grade would otherwise let
175
+ everything through.
176
+
88
177
  ## deep-journey
89
178
 
90
179
  ```
@@ -242,7 +331,8 @@ A `.env` in the working directory is read on startup — copy `.env.example` and
242
331
  | `ORA_API_URL` | audit, deep-journey, skill | `https://ora.ai` | Public API base (no auth) |
243
332
  | `ORA_PLATFORM_URL` | journey | `https://api.agentfront.sh` | Authenticated platform API base |
244
333
  | `ORA_TUNNEL_CMD` | audit | — (optional) | Tunnel command for a local target (same as `--tunnel-cmd`) |
245
- | `ORA_API_KEY` | journey | — (required) | Secret key (`ora_sk_…`), exchanged for a short-lived bearer token |
334
+ | `ORA_API_URL` | webmcp-audit | `https://ora.ai` | Also the ingest base for `webmcp-audit` |
335
+ | `ORA_API_KEY` | journey, webmcp-audit | — (required for journey) | Secret key (`ora_sk_…`); journey exchanges it for a bearer token, webmcp-audit sends it to lift the ingest rate limits |
246
336
  | `ORA_SCAN_API_KEY` | audit, deep-journey | — (optional) | ora-issued scan API key; lifts every scan rate limit (issued manually by ora). deep-journey accepts it as a partner-key fallback |
247
337
  | `ORA_PARTNER_API_KEY` | deep-journey | — (optional) | ora-issued partner API key; unlocks `--task` and the 1000/24h keyed allowance |
248
338
 
@@ -267,17 +357,39 @@ pnpm lint
267
357
  pnpm contract:gen # regenerate src/contract/ from the served OpenAPI spec
268
358
  pnpm build # tsup → dist/main.cjs (bin) + dist/index.* (library)
269
359
  pnpm smoke # run the built binary the way a user would (needs build)
360
+ pnpm smoke:webmcp # webmcp-audit end to end: real Chrome + mock ingest (needs build)
270
361
  pnpm verify:pack # check the tarball npm would publish (needs build)
271
362
  node dist/main.cjs audit https://example.com
272
363
  ```
273
364
 
274
- Manual testing without burning the live scan rate limit:
365
+ `webmcp-audit` is the one command whose core cannot be unit-tested — it needs a browser. Two
366
+ commands cover it, and CI runs both on every PR:
367
+
368
+ ```sh
369
+ AX_WEBMCP_LIVE=1 pnpm test:capture # the three capture passes against a real Chrome
370
+ pnpm build && pnpm smoke:webmcp # the whole chain: browser → capture → ingest → report
371
+ ```
372
+
373
+ Both launch their own headless Chrome and stop it again. In a container where Chrome's sandbox
374
+ cannot start, set `AX_WEBMCP_NO_SANDBOX=1` — never on a development machine, since the page being
375
+ audited is untrusted and the sandbox is what contains it.
376
+
377
+ Manual testing without burning the live rate limits:
275
378
 
276
379
  ```sh
277
380
  node scripts/mock-scan-server.mjs &
278
381
  ORA_API_URL=http://localhost:8799 node dist/main.cjs audit https://example.com
279
382
  ```
280
383
 
384
+ ```sh
385
+ node scripts/mock-webmcp-ingest.mjs &
386
+ ORA_API_URL=http://localhost:8798 node dist/main.cjs webmcp-audit http://localhost:3000
387
+ ```
388
+
389
+ The webmcp mock replays the checked-in real ingest fixture, and takes a failure mode as its second
390
+ argument — `shim`, `rate`, `scoring` or `stream` — so every error branch is reachable without the
391
+ live endpoint.
392
+
281
393
  To exercise the published-package experience locally: `npm pack`, then `npx ./ora-ai-ax-<version>.tgz audit https://example.com`, or `pnpm link --global` and use `ax` directly.
282
394
 
283
395
  ## Releasing
@@ -1,4 +1,4 @@
1
- import type { JourneyRun, JourneyRunDetail } from "../contract";
1
+ import type { JourneyRun, JourneyRunDetail, JourneyTrajectoryStep } from "../contract";
2
2
  /** Any failure to obtain a run from ora: network, HTTP, rate limit, stream error. */
3
3
  export declare class DeepJourneyApiError extends Error {
4
4
  }
@@ -45,6 +45,12 @@ export interface DeepJourneyOptions {
45
45
  model: string;
46
46
  /** Receives one-line progress updates while the run streams/polls. */
47
47
  progress?: (line: string) => void;
48
+ /**
49
+ * Receives the cumulative trajectory on every `trajectory` frame - the raw
50
+ * material the caller redraws the live attribution graph from. Streaming
51
+ * only; polling has no per-step frames to hand back.
52
+ */
53
+ onTrajectory?: (steps: JourneyTrajectoryStep[]) => void;
48
54
  /** Base URL override; otherwise $ORA_API_URL, otherwise https://ora.ai. */
49
55
  baseUrl?: string;
50
56
  /** Abort when the stream is silent for this long (default 120s). */
package/dist/index.cjs CHANGED
@@ -435,6 +435,7 @@ async function followJourneyStream(base, streamPath, options) {
435
435
  const steps = payload.steps ?? [];
436
436
  const active = steps.filter((s) => s.type === "tool_call").at(-1);
437
437
  options.progress?.(`step ${steps.length} \xB7 ${activityLabel(active ?? steps.at(-1))}`);
438
+ options.onTrajectory?.(steps);
438
439
  } else if (decoded.event === "processing") {
439
440
  options.progress?.(
440
441
  typeof payload.message === "string" ? payload.message : "generating insights\u2026"
package/dist/index.js CHANGED
@@ -399,6 +399,7 @@ async function followJourneyStream(base, streamPath, options) {
399
399
  const steps = payload.steps ?? [];
400
400
  const active = steps.filter((s) => s.type === "tool_call").at(-1);
401
401
  options.progress?.(`step ${steps.length} \xB7 ${activityLabel(active ?? steps.at(-1))}`);
402
+ options.onTrajectory?.(steps);
402
403
  } else if (decoded.event === "processing") {
403
404
  options.progress?.(
404
405
  typeof payload.message === "string" ? payload.message : "generating insights\u2026"