ax 0.6.0 → 0.7.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.env.example +9 -4
- package/README.md +115 -3
- package/dist/api/deep-journey.d.ts +7 -1
- package/dist/index.cjs +1 -0
- package/dist/index.js +1 -0
- package/dist/main.cjs +593 -36
- package/package.json +6 -2
package/.env.example
CHANGED
|
@@ -1,14 +1,19 @@
|
|
|
1
1
|
# ora platform API (agentic journeys / runs). Copy to `.env` and fill in.
|
|
2
|
-
# The public commands (audit, deep-journey, skill) use the public
|
|
3
|
-
# and need none of these — `ax audit <url>` works with zero
|
|
2
|
+
# The public commands (audit, webmcp-audit, deep-journey, skill) use the public
|
|
3
|
+
# ora.ai host and need none of these — `ax audit <url>` works with zero
|
|
4
|
+
# configuration, and so does `ax webmcp-audit http://localhost:3000`.
|
|
4
5
|
|
|
5
|
-
# Secret API key (looks like ora_sk_...).
|
|
6
|
+
# Secret API key (looks like ora_sk_...). Required by `journey`, which exchanges
|
|
7
|
+
# it for a short-lived bearer token. `webmcp-audit` sends it as a bearer token to
|
|
8
|
+
# lift the anonymous ingest limits (10/min, 20 per 24h per IP); optional there.
|
|
6
9
|
ORA_API_KEY=
|
|
7
10
|
|
|
8
11
|
# Platform API base. Defaults to staging if unset.
|
|
9
12
|
ORA_PLATFORM_URL=https://api.agentfront.sh
|
|
10
13
|
|
|
11
|
-
# Public
|
|
14
|
+
# Public API base, used by audit, webmcp-audit, deep-journey and skill.
|
|
15
|
+
# Defaults to https://ora.ai if unset. Point it at a local ora to test against
|
|
16
|
+
# one (e.g. http://localhost:3000).
|
|
12
17
|
# ORA_API_URL=https://ora.ai
|
|
13
18
|
|
|
14
19
|
# Tunnel command for auditing a local target (same as --tunnel-cmd):
|
package/README.md
CHANGED
|
@@ -6,9 +6,10 @@ Score any site's agent readiness — and watch real AI agents navigate it. Power
|
|
|
6
6
|
npx ax audit https://stripe.com
|
|
7
7
|
```
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
Five commands:
|
|
10
10
|
|
|
11
11
|
- **`audit <url>`** — run ora's hosted agent-readiness audit against a site: live progress, then a layered report of what passed, what's broken, and ora's ranked list of the highest-impact fixes (or `--json` for the raw contract payload). No account or API key needed. Gate CI with `--min-score`.
|
|
12
|
+
- **`webmcp-audit <url>`** — audit a page's `document.modelContext` tools in a real browser. Works on any http(s) URL, localhost included, so you can check a surface before you publish it. Measures the page the way ora's capture worker does and has ora score it; nothing is stored, published, or ranked.
|
|
12
13
|
- **`deep-journey <url>`** — run a real AI agent at a site on one of ora's curated tasks through the public journey API: no key, no workspace. A partner API key unlocks free-text tasks (`--task`) and a larger allowance.
|
|
13
14
|
- **`journey "<intent>"`** — send a real AI agent (claude-code, codex, …) at a site and watch its navigation live as a boxed node-graph, then get the scored insight, tokens, and cost. Requires an `ORA_API_KEY`.
|
|
14
15
|
- **`skill [name]`** — list, print, or install ora's agent skills, digest-verified from the public registry.
|
|
@@ -85,6 +86,94 @@ ax audit localhost:3000 --tunnel-cmd 'ngrok http 3000 --log stdout'
|
|
|
85
86
|
- Off-site checks (registry listings, brand search) usually fail for a throwaway tunnel hostname — the report says so. Use tunnel audits to iterate on your on-site surface, not to compare scores.
|
|
86
87
|
- Free tiers of some tunnel vendors serve an interstitial warning page to browser-like requests, which can distort what the scanner sees — prefer a vendor/plan that serves your origin directly.
|
|
87
88
|
|
|
89
|
+
## webmcp-audit
|
|
90
|
+
|
|
91
|
+
```
|
|
92
|
+
ax webmcp-audit <url> [--chrome-endpoint e] [--min-score n] [--api-key k] [--json] [--show-passing]
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
```bash
|
|
96
|
+
ax webmcp-audit http://localhost:3000
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
The dev-loop counterpart to the audit at [webmcp.ora.ai](https://webmcp.ora.ai): check the
|
|
100
|
+
`document.modelContext` tools your page registers *before* you ship it, and get the same answer
|
|
101
|
+
you will get once it is public.
|
|
102
|
+
|
|
103
|
+
Any `http(s)` URL works — a dev server, a staging host, or a site you do not own. A page with no
|
|
104
|
+
WebMCP surface is not an error: it comes back `absent` with the checks that cannot apply marked
|
|
105
|
+
`Not applicable`, which is a perfectly good way to see what the audit looks like before you have
|
|
106
|
+
written any tools.
|
|
107
|
+
|
|
108
|
+
**It uses a Chrome you already have.** WebMCP tools only exist inside a live page, so the audit
|
|
109
|
+
opens yours in a real browser. It finds your installed Chrome, starts a headless one on a throwaway
|
|
110
|
+
profile, and stops it when the audit finishes — nothing to set up, and this package never downloads
|
|
111
|
+
or bundles a browser. Set `CHROME_PATH` if yours lives somewhere unusual.
|
|
112
|
+
|
|
113
|
+
To drive a browser you started yourself, pass `--chrome-endpoint` (a `ws://` URL, an http origin,
|
|
114
|
+
or `host:port`) and start it with `--remote-debugging-port`. The command never attaches to a
|
|
115
|
+
browser you did not name — including anything already on port 9222 — because silently borrowing a
|
|
116
|
+
Chrome with the WebMCP flags on would score your page higher here than it scores on ora.
|
|
117
|
+
|
|
118
|
+
**Do not enable the WebMCP flags on a Chrome you point it at.** The audit asks whether a real
|
|
119
|
+
browser gives an agent WebMCP on your page. A Chrome started with `--enable-features=WebMCPTesting`
|
|
120
|
+
answers yes for every site, so a page can come back `active` here and `testing-only` on ora. The
|
|
121
|
+
report tells you when the browser answered for the page; where that actually moved a check, ora
|
|
122
|
+
annotates that check itself. Keep a flag-enabled Chrome for
|
|
123
|
+
[`@ora-ai/webmcp-verify`](https://www.npmjs.com/package/@ora-ai/webmcp-verify), which needs the API
|
|
124
|
+
present in order to call your tools.
|
|
125
|
+
|
|
126
|
+
**What it measures, and where.** The capture is local: three passes over your entry page, the same
|
|
127
|
+
passes and the same probe code ora's worker runs, plus a viewport screenshot at ora's 1280×720. As
|
|
128
|
+
ora does, it loads the page with images, media and webfonts refused — stylesheets still apply — so
|
|
129
|
+
the screenshot the grader sees is the one ora would have taken. That capture is
|
|
130
|
+
posted to ora, which runs the 16 checks, two model hops and the tool-selection simulation, and
|
|
131
|
+
streams back a score. The CLI scores nothing itself (design decision #1) — grade, findings,
|
|
132
|
+
category scores and the simulation are all rendered as ora sent them.
|
|
133
|
+
|
|
134
|
+
**Availability is a gate, not a score.** Before anything is graded, ora decides whether an
|
|
135
|
+
in-browser agent can use the page at all. A page that cannot comes back `not agent-ready` with the
|
|
136
|
+
reasons why, and no score or grade — the reasons are the answer.
|
|
137
|
+
|
|
138
|
+
Past that gate the score is four pillars — `shared-experience` (30), `task-completion` (25),
|
|
139
|
+
`tool-quality` (25), `trust` (20). It is **not** the 0–100 agent-readiness score from `ax audit`:
|
|
140
|
+
different inputs, different rubric, never comparable.
|
|
141
|
+
|
|
142
|
+
A pillar can read 0 because it earned nothing or because nothing in it could be measured; the
|
|
143
|
+
report marks the second case rather than leaving you to guess.
|
|
144
|
+
|
|
145
|
+
**Two check statuses mean "no score" and are not the same thing.** `Not applicable` means the check
|
|
146
|
+
had nothing to measure and your page is not charged for it. `Not measured` means it applied and
|
|
147
|
+
could not be measured — that counts against evidence coverage, and below the coverage gate ora
|
|
148
|
+
withholds the letter grade while still reporting the score. A withheld grade beside a real score is
|
|
149
|
+
normal, not an error.
|
|
150
|
+
|
|
151
|
+
**What leaves your machine.** Scoring happens on ora's side, so the capture is uploaded: the page
|
|
152
|
+
URL, every tool your page registers (names, descriptions, JSON Schemas, annotations), the WebMCP
|
|
153
|
+
evidence found in your page's own scripts, and **a 1280×720 screenshot of the page**. Point it at a
|
|
154
|
+
logged-in or seeded dev environment and that screenshot shows whatever was on screen. Your source
|
|
155
|
+
code is never read and never sent.
|
|
156
|
+
|
|
157
|
+
**Nothing is stored.** A local audit is scored and returned and then forgotten. It never reaches
|
|
158
|
+
the leaderboard, the badge, or your public scorecard — to appear there, publish and run the audit
|
|
159
|
+
on webmcp.ora.ai. Pass `--json` and pipe it to a file if you want to see exactly what was sent
|
|
160
|
+
back; the capture itself is what the report's header describes.
|
|
161
|
+
|
|
162
|
+
Anonymous use is rate-limited to 10/min and 20 per 24h per IP; an ora API key
|
|
163
|
+
(`--api-key` / `ORA_API_KEY`) lifts that.
|
|
164
|
+
|
|
165
|
+
### CI gate
|
|
166
|
+
|
|
167
|
+
```yaml
|
|
168
|
+
- run: npx ax webmcp-audit http://localhost:3000 --min-score 70
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
Any CI image with Chrome installed works; nothing to launch first.
|
|
172
|
+
|
|
173
|
+
Exit codes match `audit`: `0` ok · `1` below `--min-score` · `2` usage (including no debuggable
|
|
174
|
+
Chrome) · `3` API. The gate reads the score, not the grade — a withheld grade would otherwise let
|
|
175
|
+
everything through.
|
|
176
|
+
|
|
88
177
|
## deep-journey
|
|
89
178
|
|
|
90
179
|
```
|
|
@@ -242,7 +331,8 @@ A `.env` in the working directory is read on startup — copy `.env.example` and
|
|
|
242
331
|
| `ORA_API_URL` | audit, deep-journey, skill | `https://ora.ai` | Public API base (no auth) |
|
|
243
332
|
| `ORA_PLATFORM_URL` | journey | `https://api.agentfront.sh` | Authenticated platform API base |
|
|
244
333
|
| `ORA_TUNNEL_CMD` | audit | — (optional) | Tunnel command for a local target (same as `--tunnel-cmd`) |
|
|
245
|
-
| `
|
|
334
|
+
| `ORA_API_URL` | webmcp-audit | `https://ora.ai` | Also the ingest base for `webmcp-audit` |
|
|
335
|
+
| `ORA_API_KEY` | journey, webmcp-audit | — (required for journey) | Secret key (`ora_sk_…`); journey exchanges it for a bearer token, webmcp-audit sends it to lift the ingest rate limits |
|
|
246
336
|
| `ORA_SCAN_API_KEY` | audit, deep-journey | — (optional) | ora-issued scan API key; lifts every scan rate limit (issued manually by ora). deep-journey accepts it as a partner-key fallback |
|
|
247
337
|
| `ORA_PARTNER_API_KEY` | deep-journey | — (optional) | ora-issued partner API key; unlocks `--task` and the 1000/24h keyed allowance |
|
|
248
338
|
|
|
@@ -267,17 +357,39 @@ pnpm lint
|
|
|
267
357
|
pnpm contract:gen # regenerate src/contract/ from the served OpenAPI spec
|
|
268
358
|
pnpm build # tsup → dist/main.cjs (bin) + dist/index.* (library)
|
|
269
359
|
pnpm smoke # run the built binary the way a user would (needs build)
|
|
360
|
+
pnpm smoke:webmcp # webmcp-audit end to end: real Chrome + mock ingest (needs build)
|
|
270
361
|
pnpm verify:pack # check the tarball npm would publish (needs build)
|
|
271
362
|
node dist/main.cjs audit https://example.com
|
|
272
363
|
```
|
|
273
364
|
|
|
274
|
-
|
|
365
|
+
`webmcp-audit` is the one command whose core cannot be unit-tested — it needs a browser. Two
|
|
366
|
+
commands cover it, and CI runs both on every PR:
|
|
367
|
+
|
|
368
|
+
```sh
|
|
369
|
+
AX_WEBMCP_LIVE=1 pnpm test:capture # the three capture passes against a real Chrome
|
|
370
|
+
pnpm build && pnpm smoke:webmcp # the whole chain: browser → capture → ingest → report
|
|
371
|
+
```
|
|
372
|
+
|
|
373
|
+
Both launch their own headless Chrome and stop it again. In a container where Chrome's sandbox
|
|
374
|
+
cannot start, set `AX_WEBMCP_NO_SANDBOX=1` — never on a development machine, since the page being
|
|
375
|
+
audited is untrusted and the sandbox is what contains it.
|
|
376
|
+
|
|
377
|
+
Manual testing without burning the live rate limits:
|
|
275
378
|
|
|
276
379
|
```sh
|
|
277
380
|
node scripts/mock-scan-server.mjs &
|
|
278
381
|
ORA_API_URL=http://localhost:8799 node dist/main.cjs audit https://example.com
|
|
279
382
|
```
|
|
280
383
|
|
|
384
|
+
```sh
|
|
385
|
+
node scripts/mock-webmcp-ingest.mjs &
|
|
386
|
+
ORA_API_URL=http://localhost:8798 node dist/main.cjs webmcp-audit http://localhost:3000
|
|
387
|
+
```
|
|
388
|
+
|
|
389
|
+
The webmcp mock replays the checked-in real ingest fixture, and takes a failure mode as its second
|
|
390
|
+
argument — `shim`, `rate`, `scoring` or `stream` — so every error branch is reachable without the
|
|
391
|
+
live endpoint.
|
|
392
|
+
|
|
281
393
|
To exercise the published-package experience locally: `npm pack`, then `npx ./ora-ai-ax-<version>.tgz audit https://example.com`, or `pnpm link --global` and use `ax` directly.
|
|
282
394
|
|
|
283
395
|
## Releasing
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import type { JourneyRun, JourneyRunDetail } from "../contract";
|
|
1
|
+
import type { JourneyRun, JourneyRunDetail, JourneyTrajectoryStep } from "../contract";
|
|
2
2
|
/** Any failure to obtain a run from ora: network, HTTP, rate limit, stream error. */
|
|
3
3
|
export declare class DeepJourneyApiError extends Error {
|
|
4
4
|
}
|
|
@@ -45,6 +45,12 @@ export interface DeepJourneyOptions {
|
|
|
45
45
|
model: string;
|
|
46
46
|
/** Receives one-line progress updates while the run streams/polls. */
|
|
47
47
|
progress?: (line: string) => void;
|
|
48
|
+
/**
|
|
49
|
+
* Receives the cumulative trajectory on every `trajectory` frame - the raw
|
|
50
|
+
* material the caller redraws the live attribution graph from. Streaming
|
|
51
|
+
* only; polling has no per-step frames to hand back.
|
|
52
|
+
*/
|
|
53
|
+
onTrajectory?: (steps: JourneyTrajectoryStep[]) => void;
|
|
48
54
|
/** Base URL override; otherwise $ORA_API_URL, otherwise https://ora.ai. */
|
|
49
55
|
baseUrl?: string;
|
|
50
56
|
/** Abort when the stream is silent for this long (default 120s). */
|
package/dist/index.cjs
CHANGED
|
@@ -435,6 +435,7 @@ async function followJourneyStream(base, streamPath, options) {
|
|
|
435
435
|
const steps = payload.steps ?? [];
|
|
436
436
|
const active = steps.filter((s) => s.type === "tool_call").at(-1);
|
|
437
437
|
options.progress?.(`step ${steps.length} \xB7 ${activityLabel(active ?? steps.at(-1))}`);
|
|
438
|
+
options.onTrajectory?.(steps);
|
|
438
439
|
} else if (decoded.event === "processing") {
|
|
439
440
|
options.progress?.(
|
|
440
441
|
typeof payload.message === "string" ? payload.message : "generating insights\u2026"
|
package/dist/index.js
CHANGED
|
@@ -399,6 +399,7 @@ async function followJourneyStream(base, streamPath, options) {
|
|
|
399
399
|
const steps = payload.steps ?? [];
|
|
400
400
|
const active = steps.filter((s) => s.type === "tool_call").at(-1);
|
|
401
401
|
options.progress?.(`step ${steps.length} \xB7 ${activityLabel(active ?? steps.at(-1))}`);
|
|
402
|
+
options.onTrajectory?.(steps);
|
|
402
403
|
} else if (decoded.event === "processing") {
|
|
403
404
|
options.progress?.(
|
|
404
405
|
typeof payload.message === "string" ? payload.message : "generating insights\u2026"
|