@ora-ai/cli 0.25.0 → 0.27.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +39 -1
- package/README.md +569 -462
- package/dist/command-tree.d.ts +30 -0
- package/dist/command-tree.d.ts.map +1 -0
- package/dist/command-tree.js +144 -0
- package/dist/command-tree.js.map +1 -0
- package/dist/core/error-envelope.d.ts +13 -4
- package/dist/core/error-envelope.d.ts.map +1 -1
- package/dist/core/error-envelope.js +19 -4
- package/dist/core/error-envelope.js.map +1 -1
- package/dist/core/error-stories.d.ts +36 -0
- package/dist/core/error-stories.d.ts.map +1 -0
- package/dist/core/error-stories.js +155 -0
- package/dist/core/error-stories.js.map +1 -0
- package/dist/core/http.d.ts +14 -3
- package/dist/core/http.d.ts.map +1 -1
- package/dist/core/http.js +24 -6
- package/dist/core/http.js.map +1 -1
- package/dist/core/limit-story.d.ts +2 -0
- package/dist/core/limit-story.d.ts.map +1 -1
- package/dist/core/limit-story.js +9 -4
- package/dist/core/limit-story.js.map +1 -1
- package/dist/core/operation-command.d.ts +10 -0
- package/dist/core/operation-command.d.ts.map +1 -0
- package/dist/core/operation-command.js +15 -0
- package/dist/core/operation-command.js.map +1 -0
- package/dist/core/operation-meta.d.ts +62 -0
- package/dist/core/operation-meta.d.ts.map +1 -0
- package/dist/core/operation-meta.js +15 -0
- package/dist/core/operation-meta.js.map +1 -0
- package/dist/core/output.d.ts +7 -5
- package/dist/core/output.d.ts.map +1 -1
- package/dist/core/output.js +21 -80
- package/dist/core/output.js.map +1 -1
- package/dist/core/problem.d.ts +34 -0
- package/dist/core/problem.d.ts.map +1 -0
- package/dist/core/problem.js +44 -0
- package/dist/core/problem.js.map +1 -0
- package/dist/core/retry-after.d.ts +4 -3
- package/dist/core/retry-after.d.ts.map +1 -1
- package/dist/core/retry-after.js +4 -3
- package/dist/core/retry-after.js.map +1 -1
- package/dist/core/run-operation.d.ts +9 -4
- package/dist/core/run-operation.d.ts.map +1 -1
- package/dist/core/run-operation.js +17 -4
- package/dist/core/run-operation.js.map +1 -1
- package/dist/generated/operations.d.ts +37 -1
- package/dist/generated/operations.d.ts.map +1 -1
- package/dist/generated/operations.js +27562 -181
- package/dist/generated/operations.js.map +1 -1
- package/dist/generated/types.d.ts +1995 -135
- package/dist/generated/types.d.ts.map +1 -1
- package/dist/generated/types.js +1 -1
- package/dist/interactive/derive-prompts.d.ts +13 -5
- package/dist/interactive/derive-prompts.d.ts.map +1 -1
- package/dist/interactive/derive-prompts.js +25 -1
- package/dist/interactive/derive-prompts.js.map +1 -1
- package/dist/interactive/session.d.ts +2 -2
- package/dist/interactive/session.d.ts.map +1 -1
- package/dist/interactive/session.js +31 -2
- package/dist/interactive/session.js.map +1 -1
- package/dist/mcp/defined-tools/defined-call.d.ts +18 -0
- package/dist/mcp/defined-tools/defined-call.d.ts.map +1 -0
- package/dist/mcp/defined-tools/defined-call.js +114 -0
- package/dist/mcp/defined-tools/defined-call.js.map +1 -0
- package/dist/mcp/defined-tools/defined-tool.d.ts +22 -0
- package/dist/mcp/defined-tools/defined-tool.d.ts.map +1 -0
- package/dist/mcp/defined-tools/defined-tool.js +35 -0
- package/dist/mcp/defined-tools/defined-tool.js.map +1 -0
- package/dist/mcp/defined-tools/json-conform.d.ts +28 -0
- package/dist/mcp/defined-tools/json-conform.d.ts.map +1 -0
- package/dist/mcp/defined-tools/json-conform.js +144 -0
- package/dist/mcp/defined-tools/json-conform.js.map +1 -0
- package/dist/mcp/defined-tools/problem-mapping.d.ts +39 -0
- package/dist/mcp/defined-tools/problem-mapping.d.ts.map +1 -0
- package/dist/mcp/defined-tools/problem-mapping.js +179 -0
- package/dist/mcp/defined-tools/problem-mapping.js.map +1 -0
- package/dist/mcp/defined-tools/request-split.d.ts +19 -0
- package/dist/mcp/defined-tools/request-split.d.ts.map +1 -0
- package/dist/mcp/defined-tools/request-split.js +25 -0
- package/dist/mcp/defined-tools/request-split.js.map +1 -0
- package/dist/mcp/defined-tools/tool-problem.d.ts +40 -0
- package/dist/mcp/defined-tools/tool-problem.d.ts.map +1 -0
- package/dist/mcp/defined-tools/tool-problem.js +42 -0
- package/dist/mcp/defined-tools/tool-problem.js.map +1 -0
- package/dist/mcp/defined-tools/tool-results.d.ts +12 -0
- package/dist/mcp/defined-tools/tool-results.d.ts.map +1 -0
- package/dist/mcp/defined-tools/tool-results.js +21 -0
- package/dist/mcp/defined-tools/tool-results.js.map +1 -0
- package/dist/mcp/local-tools/credential-minting.d.ts +24 -0
- package/dist/mcp/local-tools/credential-minting.d.ts.map +1 -0
- package/dist/mcp/local-tools/credential-minting.js +49 -0
- package/dist/mcp/local-tools/credential-minting.js.map +1 -0
- package/dist/mcp/local-tools/env-file.d.ts +30 -0
- package/dist/mcp/local-tools/env-file.d.ts.map +1 -0
- package/dist/mcp/local-tools/env-file.js +71 -0
- package/dist/mcp/local-tools/env-file.js.map +1 -0
- package/dist/mcp/local-tools/local-tools.d.ts +16 -0
- package/dist/mcp/local-tools/local-tools.d.ts.map +1 -0
- package/dist/mcp/local-tools/local-tools.js +16 -0
- package/dist/mcp/local-tools/local-tools.js.map +1 -0
- package/dist/mcp/local-tools/save-transmitter-credential.d.ts +21 -0
- package/dist/mcp/local-tools/save-transmitter-credential.d.ts.map +1 -0
- package/dist/mcp/local-tools/save-transmitter-credential.js +119 -0
- package/dist/mcp/local-tools/save-transmitter-credential.js.map +1 -0
- package/dist/mcp/mcp-call.d.ts +2 -2
- package/dist/mcp/mcp-call.d.ts.map +1 -1
- package/dist/mcp/mcp-call.js +8 -2
- package/dist/mcp/mcp-call.js.map +1 -1
- package/dist/mcp/mcp-command.d.ts +6 -2
- package/dist/mcp/mcp-command.d.ts.map +1 -1
- package/dist/mcp/mcp-command.js +36 -12
- package/dist/mcp/mcp-command.js.map +1 -1
- package/dist/mcp/mcp-install.d.ts +2 -0
- package/dist/mcp/mcp-install.d.ts.map +1 -1
- package/dist/mcp/mcp-install.js +7 -1
- package/dist/mcp/mcp-install.js.map +1 -1
- package/dist/mcp/mcp-protocol.d.ts +2 -0
- package/dist/mcp/mcp-protocol.d.ts.map +1 -1
- package/dist/mcp/mcp-protocol.js.map +1 -1
- package/dist/mcp/mcp-tools.d.ts +28 -25
- package/dist/mcp/mcp-tools.d.ts.map +1 -1
- package/dist/mcp/mcp-tools.js +39 -16
- package/dist/mcp/mcp-tools.js.map +1 -1
- package/dist/mcp/tool-binding.d.ts +35 -0
- package/dist/mcp/tool-binding.d.ts.map +1 -0
- package/dist/mcp/tool-binding.js +7 -0
- package/dist/mcp/tool-binding.js.map +1 -0
- package/dist/mcp/tool-bindings.d.ts +20 -0
- package/dist/mcp/tool-bindings.d.ts.map +1 -0
- package/dist/mcp/tool-bindings.js +31 -0
- package/dist/mcp/tool-bindings.js.map +1 -0
- package/dist/mcp/tool-filter.d.ts +26 -0
- package/dist/mcp/tool-filter.d.ts.map +1 -0
- package/dist/mcp/tool-filter.js +23 -0
- package/dist/mcp/tool-filter.js.map +1 -0
- package/dist/program.d.ts +9 -9
- package/dist/program.d.ts.map +1 -1
- package/dist/program.js +12 -83
- package/dist/program.js.map +1 -1
- package/dist/watch.d.ts +3 -3
- package/dist/watch.d.ts.map +1 -1
- package/dist/watch.js +10 -2
- package/dist/watch.js.map +1 -1
- package/package.json +6 -7
package/README.md
CHANGED
|
@@ -1,498 +1,605 @@
|
|
|
1
|
-
# ora
|
|
1
|
+
# ora CLI
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
3
|
+
The command line for [ora](https://ora.ai). Connect your website to ora Front, measure how AI
|
|
4
|
+
agents treat your business, and run studies and agent experiments, from a terminal, a script, CI
|
|
5
|
+
or your coding agent.
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
npm install -g @ora-ai/cli
|
|
10
|
-
ora --help
|
|
7
|
+
Every command is generated from ora's public API, so the CLI always matches it. The API's OpenAPI
|
|
8
|
+
document is at <https://api.ora.ai/openapi.json>.
|
|
11
9
|
|
|
12
|
-
(
|
|
13
|
-
|
|
10
|
+
**Contents:** [Install](#install) · [Sign in](#sign-in) ·
|
|
11
|
+
[Connect a website to ora Front](#connect-a-website-to-ora-front) ·
|
|
12
|
+
[Use it from an AI agent](#use-it-from-an-ai-agent) · [Commands](#commands) ·
|
|
13
|
+
[Scripting and CI](#scripting-and-ci) · [Errors](#errors) ·
|
|
14
|
+
[Files and network](#files-and-network) · [Support](#support)
|
|
14
15
|
|
|
15
|
-
|
|
16
|
-
version is out (never with `--json`, so its stderr stays JSON). Set
|
|
17
|
-
`ORA_NO_UPDATE_CHECK=1` to turn it off.
|
|
16
|
+
## Install
|
|
18
17
|
|
|
19
|
-
|
|
18
|
+
Requires Node.js 20 or newer.
|
|
20
19
|
|
|
21
|
-
|
|
22
|
-
|
|
20
|
+
```bash
|
|
21
|
+
npm install -g @ora-ai/cli
|
|
22
|
+
ora --help
|
|
23
|
+
```
|
|
23
24
|
|
|
24
|
-
|
|
25
|
+
You can also run it without installing, with `npx @ora-ai/cli <command>`. Always use the scoped
|
|
26
|
+
name: bare `npx ora` fetches an unrelated package.
|
|
25
27
|
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
prints `--help` instead — nothing changes for existing scripts.
|
|
28
|
+
Run `ora` on its own in a terminal to browse every command: it prompts for the arguments and shows
|
|
29
|
+
the command it will run. Piped or in CI, bare `ora` prints `--help` instead.
|
|
29
30
|
|
|
30
|
-
##
|
|
31
|
+
## Sign in
|
|
31
32
|
|
|
32
|
-
|
|
33
|
+
```bash
|
|
34
|
+
ora login # opens your browser; a new person signs up on the same page
|
|
35
|
+
ora whoami
|
|
36
|
+
```
|
|
33
37
|
|
|
34
|
-
|
|
35
|
-
|
|
38
|
+
Approve the sign-in in your browser and the CLI stays signed in on this machine, refreshing on its
|
|
39
|
+
own, until `ora logout`, which also revokes it. Approve only a sign-in you started yourself: a link
|
|
40
|
+
and code someone sends you would sign _their_ terminal in as you. If a code expires before you
|
|
41
|
+
approve it, the CLI shows a fresh one.
|
|
36
42
|
|
|
37
|
-
`ora login
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
terminal in as you. If a code expires before you approve it, the CLI shows a
|
|
42
|
-
fresh one; nothing to restart. Over SSH, in CI, or with `--no-browser` it only
|
|
43
|
-
prints the link. `ora login --read-only` asks for read permissions only, and
|
|
44
|
-
`ora login --json` prints JSON lines (the code to show, then who signed in) for
|
|
45
|
-
an agent driving the CLI.
|
|
43
|
+
- `ora login --read-only` asks for read permissions only.
|
|
44
|
+
- `ora login --no-browser` only prints the link. Over SSH and in CI that is the default.
|
|
45
|
+
- `ora login --json` prints JSON lines for an agent driving the CLI: first the link and code to
|
|
46
|
+
show, then who signed in.
|
|
46
47
|
|
|
47
|
-
The
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
`ora autopilot …` does not take a browser sign-in yet and needs an API key.
|
|
48
|
+
The approval page lists the permissions the CLI asks for, and you choose which to grant. A command
|
|
49
|
+
that needs one you did not grant says which; run `ora login` again to allow it. `ora autopilot …`
|
|
50
|
+
does not take a browser sign-in yet and needs an API key.
|
|
51
51
|
|
|
52
|
-
API keys
|
|
53
|
-
keys), then:
|
|
52
|
+
### API keys
|
|
54
53
|
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
ORA_API_KEY=ora_sk_... ora whoami
|
|
54
|
+
API keys are for CI and scripts. Create one in [ora](https://app.ora.ai) under Settings → API
|
|
55
|
+
keys, then:
|
|
58
56
|
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
says so and points here).
|
|
57
|
+
```bash
|
|
58
|
+
ora login --api-key ora_sk_... # or `ora login --api-key` alone to paste it, masked
|
|
59
|
+
echo "$KEY" | ora login # a key piped on stdin
|
|
60
|
+
ORA_API_KEY=ora_sk_... ora whoami
|
|
61
|
+
```
|
|
65
62
|
|
|
66
|
-
|
|
63
|
+
`ORA_API_KEY` wins over anything stored, so in CI just export it, preferably a short-lived key. A
|
|
64
|
+
stored key is exchanged for a short-lived token on every call: the key itself never travels beyond
|
|
65
|
+
that exchange. The CLI keeps one credential at a time: signing in through the browser removes a
|
|
67
66
|
stored key, and `ora login --api-key` revokes and removes a browser sign-in.
|
|
68
67
|
|
|
69
68
|
### Keyless sign-in from GitHub Actions
|
|
70
69
|
|
|
71
|
-
A job can sign in with no ora secret stored in the repository.
|
|
72
|
-
(Settings → API keys → CI sign-in)
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
70
|
+
A job can sign in with no ora secret stored in the repository. In ora, trust the repository on an
|
|
71
|
+
API key (Settings → API keys → CI sign-in). Then hand the job's identity token to the CLI, with the
|
|
72
|
+
key's ID:
|
|
73
|
+
|
|
74
|
+
```yaml
|
|
75
|
+
permissions:
|
|
76
|
+
id-token: write
|
|
77
|
+
steps:
|
|
78
|
+
- uses: actions/github-script@v7
|
|
79
|
+
id: identity
|
|
80
|
+
with:
|
|
81
|
+
script: |
|
|
82
|
+
core.setSecret(process.env.ACTIONS_ID_TOKEN_REQUEST_TOKEN);
|
|
83
|
+
core.setOutput("url", process.env.ACTIONS_ID_TOKEN_REQUEST_URL);
|
|
84
|
+
core.setOutput("token", process.env.ACTIONS_ID_TOKEN_REQUEST_TOKEN);
|
|
85
|
+
- run: npx --yes @ora-ai/cli whoami
|
|
86
|
+
env:
|
|
87
|
+
ORA_API_KEY_ID: 1b0f6a2c-… # the key's ID; not a secret
|
|
88
|
+
ACTIONS_ID_TOKEN_REQUEST_URL: ${{ steps.identity.outputs.url }}
|
|
89
|
+
ACTIONS_ID_TOKEN_REQUEST_TOKEN: ${{ steps.identity.outputs.token }}
|
|
90
|
+
```
|
|
80
91
|
|
|
81
92
|
The job signs in **as that key**: its scopes, its expiry, its revocation. `ora whoami` prints a
|
|
82
93
|
`via:` line naming the repository and ref that signed in.
|
|
83
94
|
|
|
84
|
-
Running the CLI from your own step instead: set `ORA_API_KEY_ID`, and pass
|
|
85
|
-
`ACTIONS_ID_TOKEN_REQUEST_URL` and `ACTIONS_ID_TOKEN_REQUEST_TOKEN` to the step (read them in an
|
|
86
|
-
`actions/github-script` step; a shell step may not see them). The identity token is requested for
|
|
87
|
-
the API origin the CLI is pointed at; `ORA_CI_AUDIENCE` overrides that.
|
|
88
|
-
|
|
89
95
|
Which credential is used, first match wins: `ORA_API_KEY`, then `ORA_API_KEY_ID` in a job that can
|
|
90
|
-
request its identity token, then the
|
|
91
|
-
job has no identity token, the CLI stops and says so. It never falls back to a stored key.
|
|
96
|
+
request its identity token, then the credential `ora login` stored. If `ORA_API_KEY_ID` is set and
|
|
97
|
+
the job has no identity token, the CLI stops and says so. It never falls back to a stored key.
|
|
98
|
+
|
|
99
|
+
| Refusal | Meaning |
|
|
100
|
+
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
|
|
101
|
+
| `CI_NOT_TRUSTED` | No trust on the key admits this job: check the repository, the ref (`refs/pull/*/merge` for pull requests) and the environment |
|
|
102
|
+
| `CI_TOKEN_INVALID` | The identity token did not verify, or was requested for another audience |
|
|
103
|
+
| `CI_IDENTITY_UNAVAILABLE` | GitHub gave the job no identity token: add `permissions: id-token: write` |
|
|
104
|
+
|
|
105
|
+
## Connect a website to ora Front
|
|
106
|
+
|
|
107
|
+
ora Front makes your website work for the AI agents that visit it. Connecting a site means
|
|
108
|
+
installing a small transmitter where the site runs (Cloudflare, Vercel, Next.js or Express), or
|
|
109
|
+
letting ora's managed CDN serve it. Do it yourself, or give this section to your coding agent.
|
|
110
|
+
|
|
111
|
+
1. **Sign in** with `ora login`, using an email at the website's domain (dana@acme.com for
|
|
112
|
+
acme.com).
|
|
113
|
+
2. **Create the site.** If it already exists you get the existing one. Note its `id`. Its first build
|
|
114
|
+
starts at once (`--watch` follows it). Once it is published, ora also sets up tools in the background,
|
|
115
|
+
as the app does: request forms, and tools it finds on your site. `ora front actions list <site-id>`
|
|
116
|
+
lists them.
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
echo '{"url":"acme.com"}' | ora front sites create --file - --watch
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
3. **See where it is hosted:** `ora front hosting detect --url acme.com`
|
|
123
|
+
4. **Save how it connects:** `cloudflare`, `vercel`, `nextjs` or `express`, and the hostname visitors
|
|
124
|
+
land on (`acme.com` or `www.acme.com`). (`managed` also takes the site's host and DNS: see
|
|
125
|
+
`ora front connection update --help`.)
|
|
126
|
+
|
|
127
|
+
```bash
|
|
128
|
+
echo '{"method":"nextjs","hostname":"acme.com"}' | ora front connection update <site-id> --file -
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
5. **Read the install steps:** `ora front connection install get <site-id>` lists the files to
|
|
132
|
+
add, the variables to set and the commands to run for your stack.
|
|
133
|
+
6. **Create the transmitter's key, install and deploy.** `ORA_INGEST_KEY` is the secret the
|
|
134
|
+
transmitter sends, and the install steps never show it. `ora front connection credentials create
|
|
135
|
+
<site-id>` creates one and shows it once (with `--json`, it is the answer's `secret`): set it where
|
|
136
|
+
the site runs. It is an organization API key that can only report traffic and serve, the same key
|
|
137
|
+
the app's Connect step creates: it works for any of your sites, and the connection lists it as this
|
|
138
|
+
site's key. To rotate it, create a new one, update the transmitter, then revoke the old one with
|
|
139
|
+
`ora front connection credentials delete <site-id> <credential-id>`, which stops it on every site.
|
|
140
|
+
Creating one takes an organization owner or admin, signed in with the Secrets permission (allow it
|
|
141
|
+
on the approval page).
|
|
142
|
+
7. **Check it:** `ora front connection checks create <site-id>`, then
|
|
143
|
+
`ora front sites stream <site-id>` to follow the site's status live.
|
|
144
|
+
8. **Switch Agent Serving on**, the app's last setup step (the site's next steps list it as
|
|
145
|
+
`serve_agents` once the website is connected). For a new site this also turns on Ask and MCP, at
|
|
146
|
+
the agent address and on your website, plus intents and Markdown for agents. MCP waits if your
|
|
147
|
+
site has its own MCP server.
|
|
148
|
+
|
|
149
|
+
```bash
|
|
150
|
+
echo '{"mode":"autopilot"}' | ora front serving update <site-id> --file -
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
If you are a coding agent doing this for someone:
|
|
154
|
+
|
|
155
|
+
- Sign-in waits for your user to approve it. If no browser opens, run `ora login --json` in the
|
|
156
|
+
background and show your user the link from its first line.
|
|
157
|
+
- Never ask your user to paste a key into the chat, and never print the transmitter's key yourself.
|
|
158
|
+
Through `ora mcp serve`, `front_save_transmitter_credential` creates it and writes it into the
|
|
159
|
+
site's env file without showing it to you. Otherwise your user runs
|
|
160
|
+
`ora front connection credentials create` in their own terminal and sets it on the host.
|
|
161
|
+
- Add the transmitter to an existing middleware or proxy file. Don't replace the file.
|
|
162
|
+
- Your website's Ask and MCP answer only while Agent Serving is on: switch it on (step 8) before
|
|
163
|
+
testing `/ask` or `/mcp` on the website.
|
|
164
|
+
- Ask before you deploy, and use a preview deployment if the host has one.
|
|
165
|
+
- Finish by telling your user what you changed and what the checks said.
|
|
166
|
+
|
|
167
|
+
| Refusal | What to do |
|
|
168
|
+
| ---------------------------------------- | --------------------------------------------------------- |
|
|
169
|
+
| `WORK_EMAIL_REQUIRED`, `DOMAIN_MISMATCH` | Sign in with an email at the website's domain |
|
|
170
|
+
| `ROLE_REQUIRED` | Only an organization owner or admin can connect a website |
|
|
171
|
+
| `INSUFFICIENT_SCOPE` creating the key | Run `ora login` again and allow Secrets |
|
|
172
|
+
|
|
173
|
+
## Use it from an AI agent
|
|
174
|
+
|
|
175
|
+
Claude and ChatGPT can use ora without the CLI: add it as an app at `https://app.ora.ai/mcp`.
|
|
176
|
+
Coding agents can run the CLI directly, and two commands help them further.
|
|
177
|
+
|
|
178
|
+
### MCP: the CLI's commands as tools
|
|
179
|
+
|
|
180
|
+
```bash
|
|
181
|
+
ora mcp install --product study # this project, for Claude Code (.mcp.json)
|
|
182
|
+
ora mcp install --agent cursor --product study # Cursor (.cursor/mcp.json)
|
|
183
|
+
ora mcp install --command front # ora Front's tools, the same as the ora app's
|
|
184
|
+
ora mcp config --product study --read-only # print the entry, for any other agent
|
|
185
|
+
ora mcp serve --product study # what the agent runs: an MCP server on stdio
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
`ora mcp serve` is the CLI served over the Model Context Protocol. Every tool is a command:
|
|
189
|
+
`study_get` runs `ora study get <id>`, takes the same parameters (a JSON body goes in `body`) and
|
|
190
|
+
answers with the API's JSON.
|
|
191
|
+
|
|
192
|
+
ora Front's commands are served as ora Front's own tools instead, exactly as `https://app.ora.ai/mcp`
|
|
193
|
+
serves them to Claude and ChatGPT: the same names (`front_list_sites`, `front_connect_website`, …),
|
|
194
|
+
descriptions, schemas and hints. Each answers its result as structured content, and a failure
|
|
195
|
+
answers a problem with its code, whether anything changed and whether retrying can help. One Front
|
|
196
|
+
tool is only here, because only a server on your machine can write your files:
|
|
197
|
+
`front_save_transmitter_credential` creates the transmitter's key and writes it into an env file of
|
|
198
|
+
the project without showing it to the agent.
|
|
199
|
+
|
|
200
|
+
- `--product <key>` (repeatable) serves one product's commands: `studio`, `study`, `experiment` or
|
|
201
|
+
`platform`. Without it the server offers every command, which is a long tool list for an agent.
|
|
202
|
+
- `--command <name>` (repeatable) serves only the commands under it: `front`, or `"front sites"`.
|
|
203
|
+
`--command front` serves exactly ora Front's tools, plus `front_save_transmitter_credential`.
|
|
204
|
+
- `--read-only` serves only the commands that read. A read sent as a POST (a study's run query)
|
|
205
|
+
stays. The flags combine: a tool is served when every flag given keeps it.
|
|
206
|
+
- **Your credential still decides what any call may do.** The server uses the one the CLI already
|
|
207
|
+
has (`ora login`, or `ORA_API_KEY` in the agent's environment) and never widens it.
|
|
208
|
+
- `install` writes the command and its flags into the agent's configuration, never a key. It merges
|
|
209
|
+
into an existing file and leaves the other servers in it alone.
|
|
210
|
+
- A refusal reaches the agent as a tool error with the API's status, code and words, so it can fix
|
|
211
|
+
the call. A write is never retried.
|
|
212
|
+
- Commands that stream are not offered: a tool call answers once. An answer over 200,000
|
|
213
|
+
characters is refused with its size, not cut off: ask a narrower command.
|
|
214
|
+
|
|
215
|
+
### Skills: teach your coding agent to use ora Studio
|
|
216
|
+
|
|
217
|
+
```bash
|
|
218
|
+
ora skills list # the skills your account can install
|
|
219
|
+
ora skills add ora-studio # this project: .claude/skills (Claude Code) and .agents/skills (Codex and others)
|
|
220
|
+
ora skills add ora-studio --global # every project, in your home folder
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
`ora-studio` is the playbook for measuring how agents treat your business: set the objective,
|
|
224
|
+
create the intent, define the win, measure, read the results, improve and schedule, with a recipe
|
|
225
|
+
per objective (be recommended, don't get steered away, book a demo, MCP discovery, MCP usage). It
|
|
226
|
+
comes from your ora account, so it needs a sign-in or API key with the `intents:read` scope.
|
|
227
|
+
|
|
228
|
+
## Commands
|
|
229
|
+
|
|
230
|
+
Every command's `--help` ends with a ready-to-run example, including a request body where one is
|
|
231
|
+
needed:
|
|
232
|
+
|
|
233
|
+
```bash
|
|
234
|
+
ora --help # every command group
|
|
235
|
+
ora experiment runs --help # everything you can do with runs
|
|
236
|
+
ora experiment runs create --help # flags and an example run.json
|
|
237
|
+
```
|
|
238
|
+
|
|
239
|
+
Commands are named after the API's operations, each part in kebab-case
|
|
240
|
+
(`ora study benchmark-definitions list`). A query parameter the API requires is a required flag. A
|
|
241
|
+
request body comes from a JSON file (`--file <path>`, or `--file -` for stdin). An operation that
|
|
242
|
+
takes an idempotency key takes `--idempotency-key <key>`: send the same key again to retry, and the
|
|
243
|
+
change never runs twice.
|
|
244
|
+
|
|
245
|
+
### Intents and goals: measure how agents treat your business (Studio)
|
|
246
|
+
|
|
247
|
+
```bash
|
|
248
|
+
ora agents # the harness × model pairs you can measure with
|
|
249
|
+
ora intents list
|
|
250
|
+
cat > intent.json <<'EOF'
|
|
251
|
+
{ "title": "Best managed Postgres for a B2B SaaS app", "layer": "discovery" }
|
|
252
|
+
EOF
|
|
253
|
+
ora intents create --file intent.json
|
|
254
|
+
ora intents measurements-status # are the daily measurements running?
|
|
255
|
+
ora intents measurements-pause-all # pause them
|
|
256
|
+
ora intents measurements-resume-all # resume them
|
|
257
|
+
echo '{}' | ora intents measure <intent-id> --file - --watch # measure now and follow it until it finishes
|
|
258
|
+
ora intents measure-status <intent-id> <measurement-id> --watch # each prompt × agent run, live
|
|
259
|
+
ora intents stream <intent-id> # the raw live events
|
|
260
|
+
cat > schedule.json <<'EOF'
|
|
261
|
+
{ "cadence": "daily" }
|
|
262
|
+
EOF
|
|
263
|
+
ora intents measurement-policy <intent-id> --file schedule.json # daily | weekly | monthly | manual
|
|
264
|
+
ora intents get <intent-id> # its measurement policy and MCP servers: read before you replace its agents
|
|
265
|
+
ora mcp servers list # the organization's MCP servers and their ids (scope mcp:read)
|
|
266
|
+
```
|
|
267
|
+
|
|
268
|
+
Goals are your own definition of a win, scored on every run:
|
|
269
|
+
|
|
270
|
+
```bash
|
|
271
|
+
cat > goal.json <<'EOF'
|
|
272
|
+
{ "kind": "answer_check", "title": "Acme DB is the first pick", "criterionGroup": "choose",
|
|
273
|
+
"statement": "The answer recommends Acme DB as its first option for hosting the database." }
|
|
274
|
+
EOF
|
|
275
|
+
ora goals create <intent-id> --file goal.json
|
|
276
|
+
ora goals list <intent-id>
|
|
277
|
+
ora goals report <intent-id> --window 30d # answer score per question type, weakest checks, prompt × agent grid
|
|
278
|
+
ora intents insights <intent-id> # what to fix: issues, pages that failed, MCP and API tool calls, what each agent said
|
|
279
|
+
ora intents answers <intent-id> # every answer: each brand's status, quote, reasons, drawbacks, who was recommended instead
|
|
280
|
+
ora intents analytics <intent-id> --window 30d # layers, positioning per question type, agents, brands, cited sources
|
|
281
|
+
ora intents history <intent-id> # each measurement, newest first, with its answer score
|
|
282
|
+
ora goals delete <goal-id> # deactivate it (kept, no longer scored)
|
|
283
|
+
```
|
|
284
|
+
|
|
285
|
+
### Runs: launch and follow agent runs (Experiments)
|
|
286
|
+
|
|
287
|
+
```bash
|
|
288
|
+
ora experiment runs list --page-size 20
|
|
289
|
+
|
|
290
|
+
cat > run.json <<'EOF'
|
|
291
|
+
{ "intent": "check pricing for acme.com", "harness": "eve", "model": "claude-sonnet-5" }
|
|
292
|
+
EOF
|
|
293
|
+
ora experiment runs create --file run.json # returns {runId, streamUrl, status}
|
|
294
|
+
ora experiment runs create --file run.json --watch # the same, then streams live events until done
|
|
295
|
+
ora experiment runs stream <run-id> # attach to a run already in flight
|
|
296
|
+
ora experiment runs feed # the organization's run status changes, live
|
|
297
|
+
```
|
|
298
|
+
|
|
299
|
+
After a run completes, read what it produced:
|
|
300
|
+
|
|
301
|
+
```bash
|
|
302
|
+
ora experiment runs insight <run-id> # the analyzed outcome
|
|
303
|
+
ora experiment runs trajectory <run-id> # the step-by-step trace
|
|
304
|
+
ora experiment runs raw-trace <run-id> # the full raw event log
|
|
305
|
+
```
|
|
306
|
+
|
|
307
|
+
### Sessions: long-lived agent environments
|
|
308
|
+
|
|
309
|
+
```bash
|
|
310
|
+
ora sessions catalog # the harnesses and models you can launch
|
|
311
|
+
ora sessions list
|
|
312
|
+
ora sessions hibernate <id>
|
|
313
|
+
ora sessions wake <id>
|
|
314
|
+
```
|
|
315
|
+
|
|
316
|
+
### Studies: compare agent setups
|
|
317
|
+
|
|
318
|
+
A study runs one task set over 1 to 6 named **groups**, each a configuration of the agent (add-ons,
|
|
319
|
+
skills, tools). One group ranks its agents (a leaderboard); two or more groups are compared.
|
|
320
|
+
|
|
321
|
+
```bash
|
|
322
|
+
cat > study.json <<'EOF'
|
|
323
|
+
{
|
|
324
|
+
"name": "A real browser against the agent as published",
|
|
325
|
+
"subjectName": "Playwright browser automation",
|
|
326
|
+
"promise": "An agent with a real browser reports content a page renders only after a click.",
|
|
327
|
+
"comparisonMode": "multi_arm",
|
|
328
|
+
"arms": [
|
|
329
|
+
{ "label": "As published", "role": "reference", "config": {} },
|
|
330
|
+
{ "label": "With a real browser", "config": { "addons": { "webdriver": {} } } }
|
|
331
|
+
],
|
|
332
|
+
"matrix": [{ "harness": "claude-code", "provider": "anthropic", "model": "claude-haiku-4-5-20251001" }],
|
|
333
|
+
"tasks": [{
|
|
334
|
+
"prompt": "Open https://the-internet.herokuapp.com/dynamic_loading/2, click Start, and report the text that appears.",
|
|
335
|
+
"rubric": ["Reports the text exactly as \"Hello World!\".", "Does not report a loading indicator as the answer."]
|
|
336
|
+
}]
|
|
337
|
+
}
|
|
338
|
+
EOF
|
|
339
|
+
ora study catalog # valid harness × model cells
|
|
340
|
+
ora study addons list --harness claude-code --surface run # add-on ids and their config
|
|
341
|
+
ora study preflight --file study.json # what it would be: runs, comparisons; creates nothing
|
|
342
|
+
ora study create --file study.json # an authored task set lands `ready`
|
|
343
|
+
ora study launch <id> --watch
|
|
344
|
+
ora study get <id> # status, runs, results
|
|
345
|
+
ora study preview <id> # the page as it would publish
|
|
346
|
+
ora study publish <id> --file publish.json # {"slug": "my-study", "visibility": "public", "skillNames": "withheld"}
|
|
347
|
+
ora study update-details <id> --file details.json # rename or re-describe it, any time
|
|
348
|
+
```
|
|
349
|
+
|
|
350
|
+
**Before you spend,** prove a design on its smallest real run first:
|
|
351
|
+
|
|
352
|
+
```bash
|
|
353
|
+
ora study rehearse --file study.json # one real run per group; the same body as create
|
|
354
|
+
ora study rehearsal-report <id> # what it proved, group by group
|
|
355
|
+
ora study ci rehearse --file study.json --mode setup # wait for the answer; setup and mock start no agent
|
|
356
|
+
```
|
|
357
|
+
|
|
358
|
+
**A study as a merge check.** A gate answers pass, fail, or "not enough runs to decide", against a
|
|
359
|
+
floor or against an earlier study of the same series:
|
|
360
|
+
|
|
361
|
+
```bash
|
|
362
|
+
ora study gate <id> --file gate.json # {"maxDrop": {"pp": 10}, "baseline": {"series": {"branch": "main"}}}
|
|
363
|
+
ora study ci run --file study.json --gate gate.json --series cli/pr-gate
|
|
364
|
+
# create, launch, wait, answer: exit 0 pass, 1 fail, 2 undecided
|
|
365
|
+
ora study series # the organization's series
|
|
366
|
+
ora study series-trend --series cli/pr-gate # one series over time, split where it stopped being comparable
|
|
367
|
+
ora study carry-plan <id> # what a series study would carry instead of run, and why the rest runs
|
|
368
|
+
ora study set-carry <id> --file carry.json # turn carrying on or off before launch: {"carry": null} runs every job
|
|
369
|
+
ora study ci run … --no-carry # this one study runs every job, whatever its body file says
|
|
370
|
+
```
|
|
371
|
+
|
|
372
|
+
A study that declares `"carry": {"from": "series"}` takes the runs of its unchanged jobs from the
|
|
373
|
+
series' previous study instead of running them again. Whether it does is yours to change until the
|
|
374
|
+
study launches.
|
|
375
|
+
|
|
376
|
+
**Reading the runs:**
|
|
377
|
+
|
|
378
|
+
```bash
|
|
379
|
+
ora study query <id> --file query.json # which runs, split how, measured on what
|
|
380
|
+
ora study query-vocabulary <id> # what its runs can be split by and measured on
|
|
381
|
+
ora study save-view <id> --file view.json # keep a question; also `views`, `run-view`, `delete-view`
|
|
382
|
+
ora study behaviors <id> # the study's own questions about each run, and how they held up
|
|
383
|
+
ora study run-screens <id> <runId> # whether a run is valid evidence, with the quotes it stands on
|
|
384
|
+
ora study run-spans <id> <runId> # the run's stored spans
|
|
385
|
+
ora study run-delivered <id> <runId> # the files and images the run delivered
|
|
386
|
+
```
|
|
387
|
+
|
|
388
|
+
**After it finished:**
|
|
389
|
+
|
|
390
|
+
```bash
|
|
391
|
+
ora study revise-grading <id> --file revision.json # change a finished study's checks, with a reason; re-grades
|
|
392
|
+
ora study grading-revisions <id> # the record of every revision
|
|
393
|
+
ora study retest <id> --file retest.json # run the benchmark's own checker again over the files runs saved
|
|
394
|
+
ora study retests <id> # what every re-check found
|
|
395
|
+
ora study rerun-infra-failures <id> # re-run, in place, the jobs ora's own infrastructure failed
|
|
396
|
+
```
|
|
397
|
+
|
|
398
|
+
**Scopes:** `study.studies:read` and `study.studies:write`, plus `study.addons:read` for the add-on
|
|
399
|
+
catalog.
|
|
400
|
+
|
|
401
|
+
### Image builds: build and run custom harness images
|
|
402
|
+
|
|
403
|
+
```bash
|
|
404
|
+
ora experiment images builds create --help # every source kind, with an example body
|
|
405
|
+
```
|
|
406
|
+
|
|
407
|
+
Build a custom harness image from a context you already host (`archive`) or from a git repository
|
|
408
|
+
(`git`):
|
|
409
|
+
|
|
410
|
+
```bash
|
|
411
|
+
cat > build.json <<'EOF'
|
|
412
|
+
{ "name": "my-harness", "source": { "kind": "archive", "url": "https://example.com/ctx.tar.gz", "sha256": "…" } }
|
|
413
|
+
EOF
|
|
414
|
+
ora experiment images builds create --file build.json # -> {id, phase}
|
|
415
|
+
ora experiment images builds get <id> # phase, image, digest once succeeded
|
|
416
|
+
ora experiment images builds stream <id> # follow it live
|
|
417
|
+
ora experiment images builds list --phase succeeded
|
|
418
|
+
ora experiment images builds cancel <id> # in-flight builds only
|
|
419
|
+
|
|
420
|
+
cat > build-git.json <<'EOF'
|
|
421
|
+
{ "name": "my-harness", "source": { "kind": "git", "url": "https://github.com/you/repo", "ref": "main" } }
|
|
422
|
+
EOF
|
|
423
|
+
ora experiment images builds create --file build-git.json
|
|
424
|
+
```
|
|
92
425
|
|
|
93
|
-
|
|
94
|
-
| --- | --- |
|
|
95
|
-
| `CI_NOT_TRUSTED` | No trust of the key admits this job: check the repository, the ref (`refs/pull/*/merge` for pull requests) and the environment |
|
|
96
|
-
| `CI_TOKEN_INVALID` | The identity token did not verify, or was requested for another audience |
|
|
97
|
-
| `CI_IDENTITY_UNAVAILABLE` | GitHub issued the job no identity token: add `permissions: id-token: write` |
|
|
426
|
+
Then launch a run from a build that reached `succeeded`:
|
|
98
427
|
|
|
99
|
-
|
|
428
|
+
```bash
|
|
429
|
+
cat > run.json <<'EOF'
|
|
430
|
+
{ "buildId": "<id>", "mode": "harness", "task": "say hello" }
|
|
431
|
+
EOF
|
|
432
|
+
ora experiment images runs create --file run.json # -> {runId}; follow it like any other run
|
|
433
|
+
```
|
|
434
|
+
|
|
435
|
+
Builds from an uploaded file (the `upload` and `inline` sources) are not available from the CLI
|
|
436
|
+
yet: use `archive` or `git`.
|
|
437
|
+
|
|
438
|
+
**Scopes:** `images:read` (get, list, stream) and `images:write` (create, cancel, runs create).
|
|
439
|
+
|
|
440
|
+
### Autopilot: standing teams of agents
|
|
441
|
+
|
|
442
|
+
A standing team of agents, each member with its own harness and model, that you send missions and
|
|
443
|
+
messages to, then watch live. Request bodies are JSON files: `--file <path>`, or `--file -` for
|
|
444
|
+
stdin.
|
|
445
|
+
|
|
446
|
+
```bash
|
|
447
|
+
cat > team.json <<'EOF'
|
|
448
|
+
{
|
|
449
|
+
"name": "billing-crew",
|
|
450
|
+
"members": [
|
|
451
|
+
{ "handle": "nova", "harness": "claude-code" },
|
|
452
|
+
{ "handle": "juno", "harness": "codex" }
|
|
453
|
+
],
|
|
454
|
+
"squads": [{ "name": "reviewers", "members": ["juno"] }]
|
|
455
|
+
}
|
|
456
|
+
EOF
|
|
457
|
+
ora autopilot strategies list # the strategies a team can be configured with
|
|
458
|
+
ora autopilot teams create --file team.json
|
|
459
|
+
ora autopilot teams list
|
|
460
|
+
|
|
461
|
+
cat > task.json <<'EOF'
|
|
462
|
+
{ "target": { "kind": "squad", "id": "reviewers" }, "instructions": "Review PR 812 against the v2 schema." }
|
|
463
|
+
EOF
|
|
464
|
+
ora autopilot missions create billing-crew --file task.json --watch # start it, then stream its events
|
|
465
|
+
printf '%s' '{"to":["nova"],"text":"Use the v2 schema.","interrupt":true}' \
|
|
466
|
+
| ora autopilot messages send billing-crew --file -
|
|
467
|
+
ora autopilot missions pause billing-crew <mission-id>
|
|
468
|
+
ora autopilot missions stop billing-crew <mission-id> --mode immediate
|
|
469
|
+
ora autopilot teams stop billing-crew --mode graceful
|
|
470
|
+
```
|
|
471
|
+
|
|
472
|
+
Every streamed line starts with its `#seq`. A stream does not reconnect on its own: to pick up
|
|
473
|
+
where one left off, pass the last seq you saw:
|
|
474
|
+
|
|
475
|
+
```bash
|
|
476
|
+
ora autopilot teams stream billing-crew --after 1840 # replays the gap, then live
|
|
477
|
+
ora autopilot events list billing-crew --mission-id <mission-id> --after-seq 1840 --limit 500 --json
|
|
478
|
+
```
|
|
479
|
+
|
|
480
|
+
**Scopes:** `autopilot:read` (lists, gets, streams), `autopilot:write` (team setup, missions,
|
|
481
|
+
messages, pause, resume, stop) and `autopilot:export:write` for `ora autopilot exports …`: an
|
|
482
|
+
export is bulk data, so a read-only key cannot run it. Every command's `--help` names the scope it
|
|
483
|
+
needs. Grant the key the `experiment` product too, and create it as a person: Autopilot acts for
|
|
484
|
+
the key's creator, so a key whose creator has left the organization is refused. A team file that
|
|
485
|
+
binds a repository credential (`resources.repos[].credential.secretRef`) also needs `secrets:bind`
|
|
486
|
+
and a key role other than Viewer: it attaches an organization secret by reference, and no command
|
|
487
|
+
returns its value, but the workload it is bound to can read it. The same rule covers every other
|
|
488
|
+
binding: an add-on's secret on `ora experiment runs create`, a session or a study, and an MCP
|
|
489
|
+
server's `secretId`.
|
|
490
|
+
|
|
491
|
+
An export is a manifest plus a signed `artifactUrl` that lives 15 minutes. Fetch the dataset with
|
|
492
|
+
any HTTP client:
|
|
493
|
+
|
|
494
|
+
```bash
|
|
495
|
+
curl -o dataset.tar "$(ora autopilot exports get billing-crew <export-id> --json | jq -r .artifactUrl)"
|
|
496
|
+
```
|
|
497
|
+
|
|
498
|
+
### Tunnels: let ora reach a site on your machine
|
|
499
|
+
|
|
500
|
+
Like `ngrok http`: ora's scans and runs reach a site running on your machine. One command prints a
|
|
501
|
+
working address and forwards until Ctrl-C:
|
|
502
|
+
|
|
503
|
+
```console
|
|
504
|
+
$ ora tunnel 3000
|
|
505
|
+
|
|
506
|
+
🚇 ora tunnel — dev-3000
|
|
507
|
+
forwards localhost:3000 → 924649e0…
|
|
508
|
+
url https://924649e0-….t.ora.ai/?ora_token=ora_tt_…
|
|
509
|
+
only holders of this link reach it — paste it as a journey targetDomain
|
|
510
|
+
● connected — Ctrl-C to stop
|
|
511
|
+
200 GET / 8ms
|
|
512
|
+
```
|
|
513
|
+
|
|
514
|
+
The port is chosen on your machine when you connect: ora can only ask your connected CLI to fetch,
|
|
515
|
+
and never reaches into your machine. Running `ora tunnel 3000` again reuses the same tunnel and
|
|
516
|
+
address.
|
|
517
|
+
|
|
518
|
+
| `--access` | Who reaches `https://<id>.t.ora.ai` | Use it for |
|
|
519
|
+
| --------------------- | ------------------------------------------------------------------------------ | ----------------------------------------- |
|
|
520
|
+
| `protected` (default) | Holders of the printed link (`?ora_token=…` or an `X-Ora-Tunnel-Token` header) | Runs, sharing with a teammate |
|
|
521
|
+
| `public` | Anyone, with no credential | Letting real AI agents reach the dev site |
|
|
522
|
+
| `private` | Nobody but ora's own scanners | Scans only |
|
|
523
|
+
|
|
524
|
+
Anything else answers 404, so a private tunnel looks the same as one that does not exist. A revoked
|
|
525
|
+
tunnel answers 410, and a tunnel whose CLI is not connected answers 502 `TUNNEL_NOT_CONNECTED`.
|
|
526
|
+
|
|
527
|
+
To point a run at your tunnel, create it with `--access public` and use its hostname as the
|
|
528
|
+
journey's `targetDomain` (for example `924649e0-….t.ora.ai`). To score a local site's agent
|
|
529
|
+
readiness, [`ax`](https://www.npmjs.com/package/ax) does it end to end with just `ORA_API_KEY`:
|
|
530
|
+
`npx ax audit localhost:3000 --tunnel ora`.
|
|
531
|
+
|
|
532
|
+
```bash
|
|
533
|
+
ora tunnels list # every tunnel, with its status
|
|
534
|
+
ora tunnels get <id>
|
|
535
|
+
ora tunnels revoke <id> # end the live connection, keep the tunnel
|
|
536
|
+
ora tunnels delete <id> # revoke it and remove it entirely
|
|
537
|
+
```
|
|
538
|
+
|
|
539
|
+
A quick local site to try it with:
|
|
540
|
+
|
|
541
|
+
```bash
|
|
542
|
+
node -e 'require("http").createServer((q,r)=>{r.writeHead(200,{"content-type":"text/html"});r.end("<h1>it works</h1>")}).listen(3000)'
|
|
543
|
+
ora tunnel 3000 # then open the printed url
|
|
544
|
+
```
|
|
545
|
+
|
|
546
|
+
## Scripting and CI
|
|
547
|
+
|
|
548
|
+
`--json` prints the raw API response; with `--watch` it becomes NDJSON (the create response first,
|
|
549
|
+
then one line per event). Exit codes: 0 ok, 1 API or sign-in error, 2 usage error.
|
|
550
|
+
|
|
551
|
+
With `--json`, a failure is JSON too: the API's error body, as sent, on one stderr line
|
|
552
|
+
(`{error, code, details}`). A failure the API did not answer in JSON (not signed in, an unreadable
|
|
553
|
+
`--file`, a refused key exchange, a sign-in that did not complete) prints `{error, code}`, plus the
|
|
554
|
+
HTTP `status` (and `details.retryAfter`, in seconds) when the API answered:
|
|
555
|
+
|
|
556
|
+
```bash
|
|
557
|
+
ora autopilot members add billing-crew --file member.json --json 2> err.json \
|
|
558
|
+
|| jq -r '.code, .details.issues[0].refusal.wave' err.json
|
|
559
|
+
|
|
560
|
+
id=$(ora experiment runs create --file run.json --json | jq -r .runId)
|
|
561
|
+
ora experiment runs stream "$id" --json | jq -c 'select(.event == "step")'
|
|
562
|
+
|
|
563
|
+
ora experiment runs list --json | jq '.results[] | {id, harness, status}'
|
|
564
|
+
```
|
|
565
|
+
|
|
566
|
+
In CI, export `ORA_API_KEY` instead of signing in, or sign in with no stored secret
|
|
567
|
+
([keyless sign-in](#keyless-sign-in-from-github-actions)).
|
|
100
568
|
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
self-hosted ora, set the API base **once at login** — it's remembered in
|
|
104
|
-
`~/.config/ora/config.json`:
|
|
569
|
+
Studio views follow your terminal's light or dark background, read from `COLORFGBG`. Set
|
|
570
|
+
`ORA_THEME=light` or `ORA_THEME=dark` when your terminal does not report it.
|
|
105
571
|
|
|
106
|
-
|
|
107
|
-
ora login --api-url http://api.ora.localhost:8081 # local k3d dev
|
|
572
|
+
## Errors
|
|
108
573
|
|
|
109
|
-
|
|
110
|
-
`
|
|
111
|
-
|
|
574
|
+
- **402**: a plan limit was hit. The message names the limit and where to upgrade.
|
|
575
|
+
- **403 `INSUFFICIENT_SCOPE`**: the credential lacks a permission, and the message names it. With a
|
|
576
|
+
browser sign-in, run `ora login` again and allow it. With an API key, create a key with that scope
|
|
577
|
+
and run `ora login --api-key` again.
|
|
578
|
+
- **429**: too many requests. The message says how long to wait.
|
|
579
|
+
- **A problem** (RFC 9457, `application/problem+json`): the CLI prints its code and detail, each
|
|
580
|
+
field error, the next steps with the command for each, whether to retry (and after how long),
|
|
581
|
+
whether a change may already have been applied (retry with the same `--idempotency-key`), and the
|
|
582
|
+
request id to quote if you contact us.
|
|
112
583
|
|
|
113
|
-
|
|
114
|
-
`config.json` > the spec's default. So one-off calls can override without
|
|
115
|
-
re-logging-in:
|
|
584
|
+
## Files and network
|
|
116
585
|
|
|
117
|
-
|
|
118
|
-
ORA_API_URL=https://api.staging.agentfront.sh ora whoami
|
|
586
|
+
The CLI keeps its state in `~/.config/ora/`, readable only by you:
|
|
119
587
|
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
ora tunnel 3000
|
|
126
|
-
|
|
127
|
-
🚇 ora tunnel — dev-3000
|
|
128
|
-
forwards localhost:3000 → 924649e0…
|
|
129
|
-
url https://924649e0-….t.agentfront.sh/?ora_token=ora_tt_…
|
|
130
|
-
only holders of this link reach it — paste it as a journey targetDomain
|
|
131
|
-
● connected — Ctrl-C to stop
|
|
132
|
-
200 GET / 8ms
|
|
133
|
-
|
|
134
|
-
The port is chosen here, at connect time — the server can only ask your
|
|
135
|
-
connected CLI to fetch; it never reaches into your machine. Re-running
|
|
136
|
-
`ora tunnel 3000` reuses the same tunnel (and URL): the per-tunnel secret is
|
|
137
|
-
remembered in `~/.config/ora/tunnel-secrets.json` (0600), like ngrok's
|
|
138
|
-
authtoken.
|
|
139
|
-
|
|
140
|
-
### Access modes (`--access`)
|
|
141
|
-
|
|
142
|
-
ora tunnel 3000 # protected (default): the printed
|
|
143
|
-
# link works; strangers get 404
|
|
144
|
-
ora tunnel 3000 --access public # open URL — answer engines can
|
|
145
|
-
# browse/index the dev site
|
|
146
|
-
ora tunnel 3000 --access private # no URL; only ora's own scanners
|
|
147
|
-
|
|
148
|
-
| mode | who reaches `https://<id>.t.<domain>` | use it for |
|
|
149
|
-
|---|---|---|
|
|
150
|
-
| `protected` | holders of the printed link (`?ora_token=…` or an `X-Ora-Tunnel-Token` header) | journeys, sharing with a teammate |
|
|
151
|
-
| `public` | anyone — no credential | letting real answer engines measure the dev site |
|
|
152
|
-
| `private` | nobody — 404; only ora's in-cluster scanners via `tunnel://<id>/` | scan-only workflows |
|
|
153
|
-
|
|
154
|
-
Everything else answers 404 identically (a private tunnel is
|
|
155
|
-
indistinguishable from a nonexistent one), revoked answers 410, and a tunnel
|
|
156
|
-
whose CLI isn't currently connected answers 502 `TUNNEL_NOT_CONNECTED`.
|
|
157
|
-
|
|
158
|
-
### Point a run at your tunnel
|
|
159
|
-
|
|
160
|
-
- **Journey**: create the tunnel `--access public`, then use the hostname as
|
|
161
|
-
the journey's `targetDomain` — e.g. `924649e0-….t.agentfront.sh`. The
|
|
162
|
-
agent browses it like any site.
|
|
163
|
-
- **Audit** (the agent-readiness audit at ora.ai): create the tunnel
|
|
164
|
-
`--access public` and let `ax` run it end to end —
|
|
165
|
-
`ax audit localhost:<port> --tunnel-cmd 'ora tunnel <port> --access public'`
|
|
166
|
-
(npm `ax`; `ax audit localhost:<port> --tunnel ora` does the same with just
|
|
167
|
-
`ORA_API_KEY`). `tunnel://<id>/` targets are for ora's in-cluster scanners
|
|
168
|
-
only, not a public API.
|
|
169
|
-
|
|
170
|
-
### Manage tunnels
|
|
171
|
-
|
|
172
|
-
ora tunnels list # status-glyph table (● connected ✖ revoked …)
|
|
173
|
-
ora tunnels get <id>
|
|
174
|
-
ora tunnels revoke <id> # kill the live connection, keep the row
|
|
175
|
-
ora tunnels delete <id> # revoke + remove the row entirely
|
|
176
|
-
|
|
177
|
-
Quick local target to test with:
|
|
178
|
-
|
|
179
|
-
node -e 'require("http").createServer((q,r)=>{r.writeHead(200,{"content-type":"text/html"});r.end("<h1>it works</h1>")}).listen(3000)'
|
|
180
|
-
ora tunnel 3000 # then open the printed url
|
|
181
|
-
|
|
182
|
-
## Use
|
|
183
|
-
|
|
184
|
-
Every command's `--help` ends with a ready-to-run example (including a request
|
|
185
|
-
body where one is needed) — the fastest way to learn a command is to ask it:
|
|
186
|
-
|
|
187
|
-
ora --help # all command groups
|
|
188
|
-
ora experiment runs --help # everything you can do with runs
|
|
189
|
-
ora experiment runs create --help # flags + an example run.json
|
|
190
|
-
|
|
191
|
-
### Runs — launch and follow agent executions
|
|
192
|
-
|
|
193
|
-
ora experiment runs list --page-size 20
|
|
194
|
-
|
|
195
|
-
cat > run.json <<'EOF'
|
|
196
|
-
{ "intent": "check pricing for acme.com", "harness": "eve", "model": "claude-sonnet-5" }
|
|
197
|
-
EOF
|
|
198
|
-
ora experiment runs create --file run.json # returns {runId, streamUrl, status}
|
|
199
|
-
ora experiment runs create --file run.json --watch # same, then streams live events until done
|
|
200
|
-
ora experiment runs stream <run-id> # attach to a run already in flight
|
|
201
|
-
ora experiment runs feed # org-wide live feed of run status changes
|
|
202
|
-
|
|
203
|
-
After a run completes, read its artifacts:
|
|
204
|
-
|
|
205
|
-
ora experiment runs insight <run-id> # the analyzed outcome
|
|
206
|
-
ora experiment runs trajectory <run-id> # step-by-step trace
|
|
207
|
-
ora experiment runs raw-trace <run-id> # the full raw event log
|
|
208
|
-
|
|
209
|
-
### Intents — AEO measurement (studio)
|
|
210
|
-
|
|
211
|
-
ora agents # the harness × model pairs you can measure with
|
|
212
|
-
ora intents list
|
|
213
|
-
cat > intent.json <<'EOF'
|
|
214
|
-
{ "title": "Best AEO platform for B2B SaaS", "layer": "discovery" }
|
|
215
|
-
EOF
|
|
216
|
-
ora intents create --file intent.json
|
|
217
|
-
ora intents measurements-status # are the daily measurements running?
|
|
218
|
-
ora intents measurements-pause-all # stop the scans
|
|
219
|
-
ora intents measurements-resume-all # restart them
|
|
220
|
-
echo '{}' | ora intents measure <intent-id> --file - --watch # measure now and follow it live until it finishes
|
|
221
|
-
ora intents measure-status <intent-id> <measurement-id> --watch # each prompt × agent run, live; done: true once every run is analyzed
|
|
222
|
-
ora intents stream <intent-id> # the raw live events (a `measurement` event when one finishes)
|
|
223
|
-
cat > schedule.json <<'EOF'
|
|
224
|
-
{ "cadence": "daily" }
|
|
225
|
-
EOF
|
|
226
|
-
ora intents measurement-policy <intent-id> --file schedule.json # daily | weekly | monthly | manual
|
|
227
|
-
ora intents get <intent-id> # measurementPolicy (agents with their MCP servers) + mcpServers: read before you replace the agents list
|
|
228
|
-
ora mcp servers list # the org's MCP servers and their ids (needs the mcp:read scope)
|
|
229
|
-
|
|
230
|
-
### Goals — your own definition of a win, scored on every run (studio)
|
|
231
|
-
|
|
232
|
-
cat > goal.json <<'EOF'
|
|
233
|
-
{ "kind": "answer_check", "title": "Acme DB is the first pick", "criterionGroup": "choose",
|
|
234
|
-
"statement": "The answer recommends Acme DB as its first option for hosting the database." }
|
|
235
|
-
EOF
|
|
236
|
-
ora goals create <intent-id> --file goal.json
|
|
237
|
-
ora goals list <intent-id>
|
|
238
|
-
ora goals report <intent-id> --window 30d # answer score per question type, weakest checks, prompt × agent grid, Studio link
|
|
239
|
-
ora intents insights <intent-id> # what to fix: issues, pages that failed, MCP / API tool calls, what each agent said
|
|
240
|
-
ora intents answers <intent-id> # every answer of a measurement: each brand's status, quote, reasons, drawbacks, who was recommended instead
|
|
241
|
-
ora intents analytics <intent-id> --window 30d # layers, positioning per question type, agents, brands, cited sources
|
|
242
|
-
ora intents history <intent-id> # each measurement, newest first, with its answer score
|
|
243
|
-
ora goals delete <goal-id> # deactivate (kept, no longer scored)
|
|
244
|
-
|
|
245
|
-
### Skills — teach your coding agent to use Studio
|
|
246
|
-
|
|
247
|
-
ora skills list # the skills your Studio account can install
|
|
248
|
-
ora skills add ora-studio # this project: .claude/skills (Claude Code) and .agents/skills (Codex and others)
|
|
249
|
-
ora skills add ora-studio --global # every project, in your home folder
|
|
250
|
-
|
|
251
|
-
`ora-studio` is the Studio playbook: define the objective, create the intent, define the win, measure,
|
|
252
|
-
read the results, improve, schedule, with a recipe per objective (be recommended, don't get steered
|
|
253
|
-
away, book a demo, MCP discovery, MCP usage). It comes from your Studio account, so it needs the same
|
|
254
|
-
sign-in or API key as the other commands (scope `intents:read`).
|
|
255
|
-
|
|
256
|
-
### MCP — give your coding agent the CLI's commands as tools
|
|
257
|
-
|
|
258
|
-
ora mcp install --product study # this project, for Claude Code (.mcp.json)
|
|
259
|
-
ora mcp install --agent cursor --product study # Cursor (.cursor/mcp.json)
|
|
260
|
-
ora mcp config --product study --read-only # print the entry, for any other agent
|
|
261
|
-
ora mcp serve --product study # what the agent runs: an MCP server on stdio
|
|
262
|
-
|
|
263
|
-
`ora mcp serve` is the CLI served over the Model Context Protocol. Every tool is a command: `study_get`
|
|
264
|
-
runs `ora study get <id>`, takes the same parameters (a JSON body goes in `body`) and answers the API's
|
|
265
|
-
JSON. There is no second list: a command added to the CLI is a tool the next time the server starts.
|
|
266
|
-
|
|
267
|
-
- `--product <key>` (repeatable) serves one product's commands: `study`, `studio`, `experiment` or
|
|
268
|
-
`platform`. Without it the server offers every command, which is a long tool list for an agent.
|
|
269
|
-
- `--read-only` serves only the commands that read. A read sent as a POST (a study's run query) stays.
|
|
270
|
-
- **The API key still decides what any call may do.** The server never widens it. Give the agent a key
|
|
271
|
-
with the scopes you want it to have.
|
|
272
|
-
- A refusal reaches the agent as a tool error with the API's own status, code and words, so it can fix
|
|
273
|
-
the call. A write is never retried.
|
|
274
|
-
- The server uses the key the CLI already has (`ora login`, or `ORA_API_KEY` in the agent's
|
|
275
|
-
environment). `install` writes the command and its flags into the agent's configuration, and never a key.
|
|
276
|
-
It merges into an existing file and leaves every other server in it alone.
|
|
277
|
-
- Commands that stream (`… stream`) are not offered: a tool call answers once.
|
|
278
|
-
- An answer over 200,000 characters is refused with its size, not cut in half: ask a narrower command.
|
|
279
|
-
|
|
280
|
-
### Sessions — long-lived agent environments
|
|
281
|
-
|
|
282
|
-
ora sessions catalog # harnesses + models you can launch
|
|
283
|
-
ora sessions list
|
|
284
|
-
ora sessions hibernate <id>
|
|
285
|
-
ora sessions wake <id>
|
|
286
|
-
|
|
287
|
-
### Studies — compare agent setups (Groups)
|
|
288
|
-
|
|
289
|
-
A study runs one task set over 1–6 named **groups**, each a configuration of
|
|
290
|
-
the agent (add-ons, skills, tools). One group ranks its agents (a
|
|
291
|
-
leaderboard); two or more groups are compared.
|
|
292
|
-
|
|
293
|
-
cat > study.json <<'EOF'
|
|
294
|
-
{
|
|
295
|
-
"name": "A real browser against the agent as published",
|
|
296
|
-
"subjectName": "Playwright browser automation",
|
|
297
|
-
"promise": "An agent with a real browser reports content a page renders only after a click.",
|
|
298
|
-
"comparisonMode": "multi_arm",
|
|
299
|
-
"arms": [
|
|
300
|
-
{ "label": "As published", "role": "reference", "config": {} },
|
|
301
|
-
{ "label": "With a real browser", "config": { "addons": { "webdriver": {} } } }
|
|
302
|
-
],
|
|
303
|
-
"matrix": [{ "harness": "claude-code", "provider": "anthropic", "model": "claude-haiku-4-5-20251001" }],
|
|
304
|
-
"tasks": [{
|
|
305
|
-
"prompt": "Open https://the-internet.herokuapp.com/dynamic_loading/2, click Start, and report the text that appears.",
|
|
306
|
-
"rubric": ["Reports the text exactly as \"Hello World!\".", "Does not report a loading indicator as the answer."]
|
|
307
|
-
}]
|
|
308
|
-
}
|
|
309
|
-
EOF
|
|
310
|
-
ora study catalog # valid harness × model cells
|
|
311
|
-
ora study addons list --harness claude-code --surface run # add-on ids + their config
|
|
312
|
-
ora study preflight --file study.json # what it would be: runs, comparisons — creates nothing
|
|
313
|
-
ora study create --file study.json # an authored task set lands `ready`
|
|
314
|
-
ora study launch <id> --watch
|
|
315
|
-
ora study get <id> # status, runs, results
|
|
316
|
-
ora study preview <id> # the page as it would publish
|
|
317
|
-
ora study publish <id> --file publish.json # {"slug": "my-study", "visibility": "public", "skillNames": "withheld"}
|
|
318
|
-
ora study update-details <id> --file details.json # rename or re-describe it, any time
|
|
319
|
-
|
|
320
|
-
**Before you spend.** Prove a design on its smallest real run first:
|
|
321
|
-
|
|
322
|
-
ora study rehearse --file study.json # one real run per group; same body as create
|
|
323
|
-
ora study rehearsal-report <id> # what it proved, group by group
|
|
324
|
-
ora study ci rehearse --file study.json --mode setup # wait for the answer; setup / mock start no agent
|
|
325
|
-
|
|
326
|
-
**A study as a merge check.** A gate answers pass, fail, or "not enough runs
|
|
327
|
-
to decide", against a floor or against an earlier study of the same series:
|
|
328
|
-
|
|
329
|
-
ora study gate <id> --file gate.json # {"maxDrop": {"pp": 10}, "baseline": {"series": {"branch": "main"}}}
|
|
330
|
-
ora study ci run --file study.json --gate gate.json --series cli/pr-gate
|
|
331
|
-
# create, launch, wait, answer: exit 0 pass, 1 fail, 2 undecided
|
|
332
|
-
ora study series # the organization's series
|
|
333
|
-
ora study series-trend --series cli/pr-gate # one series over time, split where it stopped being comparable
|
|
334
|
-
ora study carry-plan <id> # what a series study would carry instead of run, and why the rest runs
|
|
335
|
-
ora study set-carry <id> --file carry.json # turn carrying on or off before launch: {"carry": null} runs every job
|
|
336
|
-
ora study ci run … --no-carry # this one study runs every job, whatever its body file says
|
|
337
|
-
|
|
338
|
-
A study that declares `"carry": {"from": "series"}` takes the runs of its
|
|
339
|
-
unchanged jobs from the series' previous study instead of running them again.
|
|
340
|
-
Whether it does is yours to change until the study launches.
|
|
341
|
-
|
|
342
|
-
**Reading the runs.**
|
|
343
|
-
|
|
344
|
-
ora study query <id> --file query.json # which runs, split how, measured on what
|
|
345
|
-
ora study query-vocabulary <id> # what its runs can be split by and measured on
|
|
346
|
-
ora study save-view <id> --file view.json # keep a question; `views`, `run-view`, `delete-view`
|
|
347
|
-
ora study behaviors <id> # the study's own questions about each run, and how they held up
|
|
348
|
-
ora study run-screens <id> <runId> # whether a run is valid evidence, with the quotes it stands on
|
|
349
|
-
ora study run-spans <id> <runId> # the run's stored spans
|
|
350
|
-
ora study run-delivered <id> <runId> # the files and images the run delivered
|
|
351
|
-
|
|
352
|
-
**After it finished.**
|
|
353
|
-
|
|
354
|
-
ora study revise-grading <id> --file revision.json # change a finished study's checks, with a reason; re-grades
|
|
355
|
-
ora study grading-revisions <id> # the record of every revision
|
|
356
|
-
ora study retest <id> --file retest.json # run the benchmark's own checker again over the files runs saved
|
|
357
|
-
ora study retests <id> # what every re-check found
|
|
358
|
-
ora study rerun-infra-failures <id> # re-run, in place, the jobs ora's own infrastructure failed
|
|
359
|
-
|
|
360
|
-
**Scopes**: `study.studies:read` / `study.studies:write`, plus `study.addons:read`
|
|
361
|
-
for the add-on catalog.
|
|
362
|
-
|
|
363
|
-
### Image builds — build and run custom harness images
|
|
364
|
-
|
|
365
|
-
ora experiment images builds create --help # every source kind + an example body
|
|
366
|
-
|
|
367
|
-
Build a custom harness image from a context you already host somewhere
|
|
368
|
-
(`archive`) or a git repo (`git`) — both work end to end from the CLI alone,
|
|
369
|
-
api-key auth included:
|
|
370
|
-
|
|
371
|
-
cat > build.json <<'EOF'
|
|
372
|
-
{ "name": "my-harness", "source": { "kind": "archive", "url": "https://example.com/ctx.tar.gz", "sha256": "…" } }
|
|
373
|
-
EOF
|
|
374
|
-
ora experiment images builds create --file build.json # -> {id, phase}
|
|
375
|
-
ora experiment images builds get <id> # phase, image, digest once succeeded
|
|
376
|
-
ora experiment images builds stream <id> # SSE — follow it live instead of polling
|
|
377
|
-
ora experiment images builds list --phase succeeded
|
|
378
|
-
ora experiment images builds cancel <id> # in-flight builds only
|
|
379
|
-
|
|
380
|
-
cat > build-git.json <<'EOF'
|
|
381
|
-
{ "name": "my-harness", "source": { "kind": "git", "url": "https://github.com/you/repo", "ref": "main" } }
|
|
382
|
-
EOF
|
|
383
|
-
ora experiment images builds create --file build-git.json
|
|
588
|
+
| File | What it holds |
|
|
589
|
+
| --------------------- | ------------------------------------------------------------ |
|
|
590
|
+
| `session.json` | Your browser sign-in, until `ora logout` |
|
|
591
|
+
| `config.json` | A stored API key and your settings |
|
|
592
|
+
| `tunnel-secrets.json` | Each tunnel's secret, so `ora tunnel 3000` keeps its address |
|
|
384
593
|
|
|
385
|
-
|
|
594
|
+
It talks to ora (`api.ora.ai`, `account.ora.ai` for sign-in, `tunnel.ora.ai` for tunnels). Once a
|
|
595
|
+
day it also asks the npm registry whether a newer version is out and, if so, prints one line on
|
|
596
|
+
stderr (never with `--json`). Set `ORA_NO_UPDATE_CHECK=1` to turn that off.
|
|
597
|
+
|
|
598
|
+
## Support
|
|
599
|
+
|
|
600
|
+
Questions, problems or ideas: [hello@ora.ai](mailto:hello@ora.ai). When a command fails, include
|
|
601
|
+
the request id it printed.
|
|
602
|
+
|
|
603
|
+
## License
|
|
386
604
|
|
|
387
|
-
|
|
388
|
-
{ "buildId": "<id>", "mode": "harness", "task": "say hello" }
|
|
389
|
-
EOF
|
|
390
|
-
ora experiment images runs create --file run.json # -> {runId} — follow it like any other run
|
|
391
|
-
|
|
392
|
-
**`upload`/`inline` sources are NOT reachable from this CLI.** Both need a
|
|
393
|
-
files-service `fileId` (`upload` directly; `inline` materializes one for
|
|
394
|
-
you), and files-service is closed to api-key callers ENTIRELY — every one of
|
|
395
|
-
its routes 403s `NOT_PUBLIC_API` for an `api_key`-typed token, `inline`
|
|
396
|
-
additionally requires an attributable human (`created_by` is a real
|
|
397
|
-
`users.id`, which an api-key principal's synthetic grant id can't satisfy).
|
|
398
|
-
Only a real user SESSION bearer (`POST /api/auth/login`, not `ora login`'s
|
|
399
|
-
api key) can call `POST /files/upload/direct` to mint a `fileId` first — a
|
|
400
|
-
script driving both steps needs that separate login, this CLI's own
|
|
401
|
-
`ora login` cannot supply it. `archive`/`git` have no such requirement:
|
|
402
|
-
that's why they're the two kinds shown above.
|
|
403
|
-
|
|
404
|
-
**Scopes**: these commands need an api key minted with `images:read`
|
|
405
|
-
(get/list/stream) and/or `images:write` (create/cancel/runs create) — see
|
|
406
|
-
"Errors that explain themselves" below if a command answers 403
|
|
407
|
-
`INSUFFICIENT_SCOPE`.
|
|
408
|
-
|
|
409
|
-
### Autopilot — teams
|
|
410
|
-
|
|
411
|
-
A standing team of agents (each member its own harness and model) that you
|
|
412
|
-
send missions and messages to, then watch live. Request bodies are JSON
|
|
413
|
-
files: `--file <path>`, or `--file -` for stdin (YAML is not accepted yet).
|
|
414
|
-
|
|
415
|
-
cat > team.json <<'EOF'
|
|
416
|
-
{
|
|
417
|
-
"name": "billing-crew",
|
|
418
|
-
"members": [
|
|
419
|
-
{ "handle": "nova", "harness": "claude-code" },
|
|
420
|
-
{ "handle": "juno", "harness": "codex" }
|
|
421
|
-
],
|
|
422
|
-
"squads": [{ "name": "reviewers", "members": ["juno"] }]
|
|
423
|
-
}
|
|
424
|
-
EOF
|
|
425
|
-
ora autopilot strategies list # the strategy catalog a team can be configured with
|
|
426
|
-
ora autopilot teams create --file team.json
|
|
427
|
-
ora autopilot teams list
|
|
428
|
-
|
|
429
|
-
cat > task.json <<'EOF'
|
|
430
|
-
{ "target": { "kind": "squad", "id": "reviewers" }, "instructions": "Review PR 812 against the v2 schema." }
|
|
431
|
-
EOF
|
|
432
|
-
ora autopilot missions create billing-crew --file task.json --watch # start it, then stream its events
|
|
433
|
-
printf '%s' '{"to":["nova"],"text":"Use the v2 schema.","interrupt":true}' \
|
|
434
|
-
| ora autopilot messages send billing-crew --file -
|
|
435
|
-
ora autopilot missions pause billing-crew <mission-id>
|
|
436
|
-
ora autopilot missions stop billing-crew <mission-id> --mode immediate
|
|
437
|
-
ora autopilot teams stop billing-crew --mode graceful
|
|
438
|
-
|
|
439
|
-
Every streamed line starts with its `#seq`. A stream does not reconnect on
|
|
440
|
-
its own: to pick up where one left off, pass the last seq you saw:
|
|
441
|
-
|
|
442
|
-
ora autopilot teams stream billing-crew --after 1840 # replays the gap, then live
|
|
443
|
-
ora autopilot events list billing-crew --mission-id <mission-id> --after-seq 1840 --limit 500 --json
|
|
444
|
-
|
|
445
|
-
**Scopes**: `autopilot:read` (lists, gets, streams), `autopilot:write`
|
|
446
|
-
(team setup, missions, messages, pause/resume/stop), and
|
|
447
|
-
`autopilot:export:write` for `ora autopilot exports …` — a dataset export is
|
|
448
|
-
bulk data egress, so a read-only key cannot run it. Every command's `--help`
|
|
449
|
-
names the scope it needs. Grant the key the `experiment` product too (the
|
|
450
|
-
Autopilot scopes belong to it), and mint it as a person: Autopilot acts for
|
|
451
|
-
the key's creator, so a key whose creator has left the organization is
|
|
452
|
-
refused. A team file that binds a repo credential
|
|
453
|
-
(`resources.repos[].credential.secretRef`) also needs `secrets:bind` on the
|
|
454
|
-
key, and a key role other than Viewer: it attaches an org secret by reference
|
|
455
|
-
and no route returns its value — but the workload it is bound to can read it
|
|
456
|
-
(keys minted before `secrets:bind` existed hold `secrets:reveal`, which still
|
|
457
|
-
works). The same rule covers every other binding: an add-on's secret on
|
|
458
|
-
`ora experiment runs create`, a session or a study, and an MCP server's
|
|
459
|
-
`secretId`. A person needs any organization role except Viewer to bind.
|
|
460
|
-
|
|
461
|
-
An export is a manifest plus a signed `artifactUrl` that lives 15 minutes;
|
|
462
|
-
fetch the dataset with any HTTP client:
|
|
463
|
-
|
|
464
|
-
curl -o dataset.tar "$(ora autopilot exports get billing-crew <export-id> --json | jq -r .artifactUrl)"
|
|
465
|
-
|
|
466
|
-
### Scripting
|
|
467
|
-
|
|
468
|
-
`--json` prints the raw API response; with `--watch` it becomes NDJSON (the
|
|
469
|
-
create response first, then one line per event). Exit codes: 0 ok, 1 API/auth
|
|
470
|
-
error, 2 usage error.
|
|
471
|
-
|
|
472
|
-
With `--json`, a failure is JSON too: the API's error body, as sent, on one
|
|
473
|
-
stderr line — `{error, code, details}`, the fields a client generated from the
|
|
474
|
-
public API reads. An Autopilot refusal names its wave and blocker there. A
|
|
475
|
-
failure the API did not answer in JSON (not signed in, an unreadable `--file`,
|
|
476
|
-
a refused key exchange, a sign-in that did not complete) prints `{error, code}`:
|
|
477
|
-
|
|
478
|
-
ora autopilot members add billing-crew --file member.json --json 2> err.json \
|
|
479
|
-
|| jq -r '.code, .details.issues[0].refusal.wave' err.json
|
|
480
|
-
|
|
481
|
-
id=$(ora experiment runs create --file run.json --json | jq -r .runId)
|
|
482
|
-
ora experiment runs stream "$id" --json | jq -c 'select(.event == "step")'
|
|
483
|
-
|
|
484
|
-
ora experiment runs list --json | jq '.results[] | {id, harness, status}'
|
|
485
|
-
|
|
486
|
-
In CI, export `ORA_API_KEY` instead of logging in, or sign in with no stored secret
|
|
487
|
-
([keyless sign-in](#keyless-sign-in-from-github-actions)). Point at another
|
|
488
|
-
environment with `ORA_API_URL` or `--api-url`.
|
|
489
|
-
|
|
490
|
-
Studio views are drawn in Ora cream on a dark terminal and in Ora ink on a light one, read from
|
|
491
|
-
the terminal's `COLORFGBG`. Set `ORA_THEME=light` or `ORA_THEME=dark` when your terminal does not report it.
|
|
492
|
-
|
|
493
|
-
## Errors that explain themselves
|
|
494
|
-
|
|
495
|
-
- **402** — a plan limit was hit; the message names the dimension and where to upgrade.
|
|
496
|
-
- **403 INSUFFICIENT_SCOPE** — the credential lacks a scope; the message names
|
|
497
|
-
it. With a browser sign-in, run `ora login` again and allow that permission.
|
|
498
|
-
With an api key, mint a key with that scope and `ora login --api-key` again.
|
|
605
|
+
MIT
|