@ora-ai/cli 0.25.0 → 0.27.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (145) hide show
  1. package/CHANGELOG.md +39 -1
  2. package/README.md +569 -462
  3. package/dist/command-tree.d.ts +30 -0
  4. package/dist/command-tree.d.ts.map +1 -0
  5. package/dist/command-tree.js +144 -0
  6. package/dist/command-tree.js.map +1 -0
  7. package/dist/core/error-envelope.d.ts +13 -4
  8. package/dist/core/error-envelope.d.ts.map +1 -1
  9. package/dist/core/error-envelope.js +19 -4
  10. package/dist/core/error-envelope.js.map +1 -1
  11. package/dist/core/error-stories.d.ts +36 -0
  12. package/dist/core/error-stories.d.ts.map +1 -0
  13. package/dist/core/error-stories.js +155 -0
  14. package/dist/core/error-stories.js.map +1 -0
  15. package/dist/core/http.d.ts +14 -3
  16. package/dist/core/http.d.ts.map +1 -1
  17. package/dist/core/http.js +24 -6
  18. package/dist/core/http.js.map +1 -1
  19. package/dist/core/limit-story.d.ts +2 -0
  20. package/dist/core/limit-story.d.ts.map +1 -1
  21. package/dist/core/limit-story.js +9 -4
  22. package/dist/core/limit-story.js.map +1 -1
  23. package/dist/core/operation-command.d.ts +10 -0
  24. package/dist/core/operation-command.d.ts.map +1 -0
  25. package/dist/core/operation-command.js +15 -0
  26. package/dist/core/operation-command.js.map +1 -0
  27. package/dist/core/operation-meta.d.ts +62 -0
  28. package/dist/core/operation-meta.d.ts.map +1 -0
  29. package/dist/core/operation-meta.js +15 -0
  30. package/dist/core/operation-meta.js.map +1 -0
  31. package/dist/core/output.d.ts +7 -5
  32. package/dist/core/output.d.ts.map +1 -1
  33. package/dist/core/output.js +21 -80
  34. package/dist/core/output.js.map +1 -1
  35. package/dist/core/problem.d.ts +34 -0
  36. package/dist/core/problem.d.ts.map +1 -0
  37. package/dist/core/problem.js +44 -0
  38. package/dist/core/problem.js.map +1 -0
  39. package/dist/core/retry-after.d.ts +4 -3
  40. package/dist/core/retry-after.d.ts.map +1 -1
  41. package/dist/core/retry-after.js +4 -3
  42. package/dist/core/retry-after.js.map +1 -1
  43. package/dist/core/run-operation.d.ts +9 -4
  44. package/dist/core/run-operation.d.ts.map +1 -1
  45. package/dist/core/run-operation.js +17 -4
  46. package/dist/core/run-operation.js.map +1 -1
  47. package/dist/generated/operations.d.ts +37 -1
  48. package/dist/generated/operations.d.ts.map +1 -1
  49. package/dist/generated/operations.js +27562 -181
  50. package/dist/generated/operations.js.map +1 -1
  51. package/dist/generated/types.d.ts +1995 -135
  52. package/dist/generated/types.d.ts.map +1 -1
  53. package/dist/generated/types.js +1 -1
  54. package/dist/interactive/derive-prompts.d.ts +13 -5
  55. package/dist/interactive/derive-prompts.d.ts.map +1 -1
  56. package/dist/interactive/derive-prompts.js +25 -1
  57. package/dist/interactive/derive-prompts.js.map +1 -1
  58. package/dist/interactive/session.d.ts +2 -2
  59. package/dist/interactive/session.d.ts.map +1 -1
  60. package/dist/interactive/session.js +31 -2
  61. package/dist/interactive/session.js.map +1 -1
  62. package/dist/mcp/defined-tools/defined-call.d.ts +18 -0
  63. package/dist/mcp/defined-tools/defined-call.d.ts.map +1 -0
  64. package/dist/mcp/defined-tools/defined-call.js +114 -0
  65. package/dist/mcp/defined-tools/defined-call.js.map +1 -0
  66. package/dist/mcp/defined-tools/defined-tool.d.ts +22 -0
  67. package/dist/mcp/defined-tools/defined-tool.d.ts.map +1 -0
  68. package/dist/mcp/defined-tools/defined-tool.js +35 -0
  69. package/dist/mcp/defined-tools/defined-tool.js.map +1 -0
  70. package/dist/mcp/defined-tools/json-conform.d.ts +28 -0
  71. package/dist/mcp/defined-tools/json-conform.d.ts.map +1 -0
  72. package/dist/mcp/defined-tools/json-conform.js +144 -0
  73. package/dist/mcp/defined-tools/json-conform.js.map +1 -0
  74. package/dist/mcp/defined-tools/problem-mapping.d.ts +39 -0
  75. package/dist/mcp/defined-tools/problem-mapping.d.ts.map +1 -0
  76. package/dist/mcp/defined-tools/problem-mapping.js +179 -0
  77. package/dist/mcp/defined-tools/problem-mapping.js.map +1 -0
  78. package/dist/mcp/defined-tools/request-split.d.ts +19 -0
  79. package/dist/mcp/defined-tools/request-split.d.ts.map +1 -0
  80. package/dist/mcp/defined-tools/request-split.js +25 -0
  81. package/dist/mcp/defined-tools/request-split.js.map +1 -0
  82. package/dist/mcp/defined-tools/tool-problem.d.ts +40 -0
  83. package/dist/mcp/defined-tools/tool-problem.d.ts.map +1 -0
  84. package/dist/mcp/defined-tools/tool-problem.js +42 -0
  85. package/dist/mcp/defined-tools/tool-problem.js.map +1 -0
  86. package/dist/mcp/defined-tools/tool-results.d.ts +12 -0
  87. package/dist/mcp/defined-tools/tool-results.d.ts.map +1 -0
  88. package/dist/mcp/defined-tools/tool-results.js +21 -0
  89. package/dist/mcp/defined-tools/tool-results.js.map +1 -0
  90. package/dist/mcp/local-tools/credential-minting.d.ts +24 -0
  91. package/dist/mcp/local-tools/credential-minting.d.ts.map +1 -0
  92. package/dist/mcp/local-tools/credential-minting.js +49 -0
  93. package/dist/mcp/local-tools/credential-minting.js.map +1 -0
  94. package/dist/mcp/local-tools/env-file.d.ts +30 -0
  95. package/dist/mcp/local-tools/env-file.d.ts.map +1 -0
  96. package/dist/mcp/local-tools/env-file.js +71 -0
  97. package/dist/mcp/local-tools/env-file.js.map +1 -0
  98. package/dist/mcp/local-tools/local-tools.d.ts +16 -0
  99. package/dist/mcp/local-tools/local-tools.d.ts.map +1 -0
  100. package/dist/mcp/local-tools/local-tools.js +16 -0
  101. package/dist/mcp/local-tools/local-tools.js.map +1 -0
  102. package/dist/mcp/local-tools/save-transmitter-credential.d.ts +21 -0
  103. package/dist/mcp/local-tools/save-transmitter-credential.d.ts.map +1 -0
  104. package/dist/mcp/local-tools/save-transmitter-credential.js +119 -0
  105. package/dist/mcp/local-tools/save-transmitter-credential.js.map +1 -0
  106. package/dist/mcp/mcp-call.d.ts +2 -2
  107. package/dist/mcp/mcp-call.d.ts.map +1 -1
  108. package/dist/mcp/mcp-call.js +8 -2
  109. package/dist/mcp/mcp-call.js.map +1 -1
  110. package/dist/mcp/mcp-command.d.ts +6 -2
  111. package/dist/mcp/mcp-command.d.ts.map +1 -1
  112. package/dist/mcp/mcp-command.js +36 -12
  113. package/dist/mcp/mcp-command.js.map +1 -1
  114. package/dist/mcp/mcp-install.d.ts +2 -0
  115. package/dist/mcp/mcp-install.d.ts.map +1 -1
  116. package/dist/mcp/mcp-install.js +7 -1
  117. package/dist/mcp/mcp-install.js.map +1 -1
  118. package/dist/mcp/mcp-protocol.d.ts +2 -0
  119. package/dist/mcp/mcp-protocol.d.ts.map +1 -1
  120. package/dist/mcp/mcp-protocol.js.map +1 -1
  121. package/dist/mcp/mcp-tools.d.ts +28 -25
  122. package/dist/mcp/mcp-tools.d.ts.map +1 -1
  123. package/dist/mcp/mcp-tools.js +39 -16
  124. package/dist/mcp/mcp-tools.js.map +1 -1
  125. package/dist/mcp/tool-binding.d.ts +35 -0
  126. package/dist/mcp/tool-binding.d.ts.map +1 -0
  127. package/dist/mcp/tool-binding.js +7 -0
  128. package/dist/mcp/tool-binding.js.map +1 -0
  129. package/dist/mcp/tool-bindings.d.ts +20 -0
  130. package/dist/mcp/tool-bindings.d.ts.map +1 -0
  131. package/dist/mcp/tool-bindings.js +31 -0
  132. package/dist/mcp/tool-bindings.js.map +1 -0
  133. package/dist/mcp/tool-filter.d.ts +26 -0
  134. package/dist/mcp/tool-filter.d.ts.map +1 -0
  135. package/dist/mcp/tool-filter.js +23 -0
  136. package/dist/mcp/tool-filter.js.map +1 -0
  137. package/dist/program.d.ts +9 -9
  138. package/dist/program.d.ts.map +1 -1
  139. package/dist/program.js +12 -83
  140. package/dist/program.js.map +1 -1
  141. package/dist/watch.d.ts +3 -3
  142. package/dist/watch.d.ts.map +1 -1
  143. package/dist/watch.js +10 -2
  144. package/dist/watch.js.map +1 -1
  145. package/package.json +6 -7
package/README.md CHANGED
@@ -1,498 +1,605 @@
1
- # ora — the ora CLI
1
+ # ora CLI
2
2
 
3
- Drive the [ora](https://agentfront.sh) public API from your terminal. Every
4
- command is generated from ora's public OpenAPI document, so the CLI always
5
- matches the API: `https://api.agentfront.sh/openapi.json`.
3
+ The command line for [ora](https://ora.ai). Connect your website to ora Front, measure how AI
4
+ agents treat your business, and run studies and agent experiments, from a terminal, a script, CI
5
+ or your coding agent.
6
6
 
7
- ## Install
8
-
9
- npm install -g @ora-ai/cli
10
- ora --help
7
+ Every command is generated from ora's public API, so the CLI always matches it. The API's OpenAPI
8
+ document is at <https://api.ora.ai/openapi.json>.
11
9
 
12
- (`npx @ora-ai/cli` also works. Note: bare `npx ora` fetches an unrelated npm
13
- spinner library — always use the scoped package name with npx.)
10
+ **Contents:** [Install](#install) · [Sign in](#sign-in) ·
11
+ [Connect a website to ora Front](#connect-a-website-to-ora-front) ·
12
+ [Use it from an AI agent](#use-it-from-an-ai-agent) · [Commands](#commands) ·
13
+ [Scripting and CI](#scripting-and-ci) · [Errors](#errors) ·
14
+ [Files and network](#files-and-network) · [Support](#support)
14
15
 
15
- Once a day the CLI checks npm and prints one line on stderr when a newer
16
- version is out (never with `--json`, so its stderr stays JSON). Set
17
- `ORA_NO_UPDATE_CHECK=1` to turn it off.
16
+ ## Install
18
17
 
19
- ## Interactive
18
+ Requires Node.js 20 or newer.
20
19
 
21
- Just run `ora` — browse every command, get prompted for arguments, and see
22
- the equivalent scriptable command before it runs.
20
+ ```bash
21
+ npm install -g @ora-ai/cli
22
+ ora --help
23
+ ```
23
24
 
24
- ora
25
+ You can also run it without installing, with `npx @ora-ai/cli <command>`. Always use the scoped
26
+ name: bare `npx ora` fetches an unrelated package.
25
27
 
26
- This only opens in a real terminal (both stdin and stdout attached to a tty).
27
- Piped or scripted (`ora | less`, `ora > out.txt`, inside CI), bare `ora`
28
- prints `--help` instead — nothing changes for existing scripts.
28
+ Run `ora` on its own in a terminal to browse every command: it prompts for the arguments and shows
29
+ the command it will run. Piped or in CI, bare `ora` prints `--help` instead.
29
30
 
30
- ## Authenticate
31
+ ## Sign in
31
32
 
32
- Sign in with your browser (a new person signs up on the same page):
33
+ ```bash
34
+ ora login # opens your browser; a new person signs up on the same page
35
+ ora whoami
36
+ ```
33
37
 
34
- ora login # prints a link and a code, opens your browser
35
- ora whoami
38
+ Approve the sign-in in your browser and the CLI stays signed in on this machine, refreshing on its
39
+ own, until `ora logout`, which also revokes it. Approve only a sign-in you started yourself: a link
40
+ and code someone sends you would sign _their_ terminal in as you. If a code expires before you
41
+ approve it, the CLI shows a fresh one.
36
42
 
37
- `ora login` is device sign-in (RFC 8628): approve the CLI in the browser and it
38
- stays signed in on this machine — `~/.config/ora/session.json` (0600), refreshed
39
- on its own — until `ora logout`, which also revokes it. Approve only a sign-in
40
- you started yourself: a link and code someone sends you would sign *their*
41
- terminal in as you. If a code expires before you approve it, the CLI shows a
42
- fresh one; nothing to restart. Over SSH, in CI, or with `--no-browser` it only
43
- prints the link. `ora login --read-only` asks for read permissions only, and
44
- `ora login --json` prints JSON lines (the code to show, then who signed in) for
45
- an agent driving the CLI.
43
+ - `ora login --read-only` asks for read permissions only.
44
+ - `ora login --no-browser` only prints the link. Over SSH and in CI that is the default.
45
+ - `ora login --json` prints JSON lines for an agent driving the CLI: first the link and code to
46
+ show, then who signed in.
46
47
 
47
- The sign-in asks for the permissions this CLI's commands use; the approval page
48
- shows them and you decide which to grant. A command that needs one you did not
49
- grant says which, and `ora login` again lets you allow it. One exception:
50
- `ora autopilot …` does not take a browser sign-in yet and needs an API key.
48
+ The approval page lists the permissions the CLI asks for, and you choose which to grant. A command
49
+ that needs one you did not grant says which; run `ora login` again to allow it. `ora autopilot …`
50
+ does not take a browser sign-in yet and needs an API key.
51
51
 
52
- API keys are for CI and scripts. Mint one in the ora dashboard (Settings → API
53
- keys), then:
52
+ ### API keys
54
53
 
55
- ora login --api-key ora_sk_... # or `ora login --api-key` to paste it (masked)
56
- echo "$KEY" | ora login # a key piped on stdin
57
- ORA_API_KEY=ora_sk_... ora whoami
54
+ API keys are for CI and scripts. Create one in [ora](https://app.ora.ai) under Settings → API
55
+ keys, then:
58
56
 
59
- In CI, skip login and export `ORA_API_KEY` (prefer short-expiry keys); it wins
60
- over anything stored. A stored key lives in `~/.config/ora/config.json` (0600)
61
- and is exchanged for a short-lived token on every call — the key itself never
62
- travels beyond the token exchange. Keys work exactly as they always have, also
63
- against a server that does not offer browser sign-in yet (there, `ora login`
64
- says so and points here).
57
+ ```bash
58
+ ora login --api-key ora_sk_... # or `ora login --api-key` alone to paste it, masked
59
+ echo "$KEY" | ora login # a key piped on stdin
60
+ ORA_API_KEY=ora_sk_... ora whoami
61
+ ```
65
62
 
66
- One credential is stored at a time: signing in through the browser removes a
63
+ `ORA_API_KEY` wins over anything stored, so in CI just export it, preferably a short-lived key. A
64
+ stored key is exchanged for a short-lived token on every call: the key itself never travels beyond
65
+ that exchange. The CLI keeps one credential at a time: signing in through the browser removes a
67
66
  stored key, and `ora login --api-key` revokes and removes a browser sign-in.
68
67
 
69
68
  ### Keyless sign-in from GitHub Actions
70
69
 
71
- A job can sign in with no ora secret stored in the repository. Trust the repository on an api key
72
- (Settings → API keys → CI sign-in), then give the job an identity token and the key's ID:
73
-
74
- permissions:
75
- id-token: write
76
- steps:
77
- - uses: ./.github/actions/ora-study-gate # hands the CLI the job's identity token
78
- with:
79
- api-key-id: 1b0f6a2c-… # the key's ID. Not a secret
70
+ A job can sign in with no ora secret stored in the repository. In ora, trust the repository on an
71
+ API key (Settings → API keys → CI sign-in). Then hand the job's identity token to the CLI, with the
72
+ key's ID:
73
+
74
+ ```yaml
75
+ permissions:
76
+ id-token: write
77
+ steps:
78
+ - uses: actions/github-script@v7
79
+ id: identity
80
+ with:
81
+ script: |
82
+ core.setSecret(process.env.ACTIONS_ID_TOKEN_REQUEST_TOKEN);
83
+ core.setOutput("url", process.env.ACTIONS_ID_TOKEN_REQUEST_URL);
84
+ core.setOutput("token", process.env.ACTIONS_ID_TOKEN_REQUEST_TOKEN);
85
+ - run: npx --yes @ora-ai/cli whoami
86
+ env:
87
+ ORA_API_KEY_ID: 1b0f6a2c-… # the key's ID; not a secret
88
+ ACTIONS_ID_TOKEN_REQUEST_URL: ${{ steps.identity.outputs.url }}
89
+ ACTIONS_ID_TOKEN_REQUEST_TOKEN: ${{ steps.identity.outputs.token }}
90
+ ```
80
91
 
81
92
  The job signs in **as that key**: its scopes, its expiry, its revocation. `ora whoami` prints a
82
93
  `via:` line naming the repository and ref that signed in.
83
94
 
84
- Running the CLI from your own step instead: set `ORA_API_KEY_ID`, and pass
85
- `ACTIONS_ID_TOKEN_REQUEST_URL` and `ACTIONS_ID_TOKEN_REQUEST_TOKEN` to the step (read them in an
86
- `actions/github-script` step; a shell step may not see them). The identity token is requested for
87
- the API origin the CLI is pointed at; `ORA_CI_AUDIENCE` overrides that.
88
-
89
95
  Which credential is used, first match wins: `ORA_API_KEY`, then `ORA_API_KEY_ID` in a job that can
90
- request its identity token, then the key `ora login` stored. If `ORA_API_KEY_ID` is set and the
91
- job has no identity token, the CLI stops and says so. It never falls back to a stored key.
96
+ request its identity token, then the credential `ora login` stored. If `ORA_API_KEY_ID` is set and
97
+ the job has no identity token, the CLI stops and says so. It never falls back to a stored key.
98
+
99
+ | Refusal | Meaning |
100
+ | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
101
+ | `CI_NOT_TRUSTED` | No trust on the key admits this job: check the repository, the ref (`refs/pull/*/merge` for pull requests) and the environment |
102
+ | `CI_TOKEN_INVALID` | The identity token did not verify, or was requested for another audience |
103
+ | `CI_IDENTITY_UNAVAILABLE` | GitHub gave the job no identity token: add `permissions: id-token: write` |
104
+
105
+ ## Connect a website to ora Front
106
+
107
+ ora Front makes your website work for the AI agents that visit it. Connecting a site means
108
+ installing a small transmitter where the site runs (Cloudflare, Vercel, Next.js or Express), or
109
+ letting ora's managed CDN serve it. Do it yourself, or give this section to your coding agent.
110
+
111
+ 1. **Sign in** with `ora login`, using an email at the website's domain (dana@acme.com for
112
+ acme.com).
113
+ 2. **Create the site.** If it already exists you get the existing one. Note its `id`. Its first build
114
+ starts at once (`--watch` follows it). Once it is published, ora also sets up tools in the background,
115
+ as the app does: request forms, and tools it finds on your site. `ora front actions list <site-id>`
116
+ lists them.
117
+
118
+ ```bash
119
+ echo '{"url":"acme.com"}' | ora front sites create --file - --watch
120
+ ```
121
+
122
+ 3. **See where it is hosted:** `ora front hosting detect --url acme.com`
123
+ 4. **Save how it connects:** `cloudflare`, `vercel`, `nextjs` or `express`, and the hostname visitors
124
+ land on (`acme.com` or `www.acme.com`). (`managed` also takes the site's host and DNS: see
125
+ `ora front connection update --help`.)
126
+
127
+ ```bash
128
+ echo '{"method":"nextjs","hostname":"acme.com"}' | ora front connection update <site-id> --file -
129
+ ```
130
+
131
+ 5. **Read the install steps:** `ora front connection install get <site-id>` lists the files to
132
+ add, the variables to set and the commands to run for your stack.
133
+ 6. **Create the transmitter's key, install and deploy.** `ORA_INGEST_KEY` is the secret the
134
+ transmitter sends, and the install steps never show it. `ora front connection credentials create
135
+ <site-id>` creates one and shows it once (with `--json`, it is the answer's `secret`): set it where
136
+ the site runs. It is an organization API key that can only report traffic and serve, the same key
137
+ the app's Connect step creates: it works for any of your sites, and the connection lists it as this
138
+ site's key. To rotate it, create a new one, update the transmitter, then revoke the old one with
139
+ `ora front connection credentials delete <site-id> <credential-id>`, which stops it on every site.
140
+ Creating one takes an organization owner or admin, signed in with the Secrets permission (allow it
141
+ on the approval page).
142
+ 7. **Check it:** `ora front connection checks create <site-id>`, then
143
+ `ora front sites stream <site-id>` to follow the site's status live.
144
+ 8. **Switch Agent Serving on**, the app's last setup step (the site's next steps list it as
145
+ `serve_agents` once the website is connected). For a new site this also turns on Ask and MCP, at
146
+ the agent address and on your website, plus intents and Markdown for agents. MCP waits if your
147
+ site has its own MCP server.
148
+
149
+ ```bash
150
+ echo '{"mode":"autopilot"}' | ora front serving update <site-id> --file -
151
+ ```
152
+
153
+ If you are a coding agent doing this for someone:
154
+
155
+ - Sign-in waits for your user to approve it. If no browser opens, run `ora login --json` in the
156
+ background and show your user the link from its first line.
157
+ - Never ask your user to paste a key into the chat, and never print the transmitter's key yourself.
158
+ Through `ora mcp serve`, `front_save_transmitter_credential` creates it and writes it into the
159
+ site's env file without showing it to you. Otherwise your user runs
160
+ `ora front connection credentials create` in their own terminal and sets it on the host.
161
+ - Add the transmitter to an existing middleware or proxy file. Don't replace the file.
162
+ - Your website's Ask and MCP answer only while Agent Serving is on: switch it on (step 8) before
163
+ testing `/ask` or `/mcp` on the website.
164
+ - Ask before you deploy, and use a preview deployment if the host has one.
165
+ - Finish by telling your user what you changed and what the checks said.
166
+
167
+ | Refusal | What to do |
168
+ | ---------------------------------------- | --------------------------------------------------------- |
169
+ | `WORK_EMAIL_REQUIRED`, `DOMAIN_MISMATCH` | Sign in with an email at the website's domain |
170
+ | `ROLE_REQUIRED` | Only an organization owner or admin can connect a website |
171
+ | `INSUFFICIENT_SCOPE` creating the key | Run `ora login` again and allow Secrets |
172
+
173
+ ## Use it from an AI agent
174
+
175
+ Claude and ChatGPT can use ora without the CLI: add it as an app at `https://app.ora.ai/mcp`.
176
+ Coding agents can run the CLI directly, and two commands help them further.
177
+
178
+ ### MCP: the CLI's commands as tools
179
+
180
+ ```bash
181
+ ora mcp install --product study # this project, for Claude Code (.mcp.json)
182
+ ora mcp install --agent cursor --product study # Cursor (.cursor/mcp.json)
183
+ ora mcp install --command front # ora Front's tools, the same as the ora app's
184
+ ora mcp config --product study --read-only # print the entry, for any other agent
185
+ ora mcp serve --product study # what the agent runs: an MCP server on stdio
186
+ ```
187
+
188
+ `ora mcp serve` is the CLI served over the Model Context Protocol. Every tool is a command:
189
+ `study_get` runs `ora study get <id>`, takes the same parameters (a JSON body goes in `body`) and
190
+ answers with the API's JSON.
191
+
192
+ ora Front's commands are served as ora Front's own tools instead, exactly as `https://app.ora.ai/mcp`
193
+ serves them to Claude and ChatGPT: the same names (`front_list_sites`, `front_connect_website`, …),
194
+ descriptions, schemas and hints. Each answers its result as structured content, and a failure
195
+ answers a problem with its code, whether anything changed and whether retrying can help. One Front
196
+ tool is only here, because only a server on your machine can write your files:
197
+ `front_save_transmitter_credential` creates the transmitter's key and writes it into an env file of
198
+ the project without showing it to the agent.
199
+
200
+ - `--product <key>` (repeatable) serves one product's commands: `studio`, `study`, `experiment` or
201
+ `platform`. Without it the server offers every command, which is a long tool list for an agent.
202
+ - `--command <name>` (repeatable) serves only the commands under it: `front`, or `"front sites"`.
203
+ `--command front` serves exactly ora Front's tools, plus `front_save_transmitter_credential`.
204
+ - `--read-only` serves only the commands that read. A read sent as a POST (a study's run query)
205
+ stays. The flags combine: a tool is served when every flag given keeps it.
206
+ - **Your credential still decides what any call may do.** The server uses the one the CLI already
207
+ has (`ora login`, or `ORA_API_KEY` in the agent's environment) and never widens it.
208
+ - `install` writes the command and its flags into the agent's configuration, never a key. It merges
209
+ into an existing file and leaves the other servers in it alone.
210
+ - A refusal reaches the agent as a tool error with the API's status, code and words, so it can fix
211
+ the call. A write is never retried.
212
+ - Commands that stream are not offered: a tool call answers once. An answer over 200,000
213
+ characters is refused with its size, not cut off: ask a narrower command.
214
+
215
+ ### Skills: teach your coding agent to use ora Studio
216
+
217
+ ```bash
218
+ ora skills list # the skills your account can install
219
+ ora skills add ora-studio # this project: .claude/skills (Claude Code) and .agents/skills (Codex and others)
220
+ ora skills add ora-studio --global # every project, in your home folder
221
+ ```
222
+
223
+ `ora-studio` is the playbook for measuring how agents treat your business: set the objective,
224
+ create the intent, define the win, measure, read the results, improve and schedule, with a recipe
225
+ per objective (be recommended, don't get steered away, book a demo, MCP discovery, MCP usage). It
226
+ comes from your ora account, so it needs a sign-in or API key with the `intents:read` scope.
227
+
228
+ ## Commands
229
+
230
+ Every command's `--help` ends with a ready-to-run example, including a request body where one is
231
+ needed:
232
+
233
+ ```bash
234
+ ora --help # every command group
235
+ ora experiment runs --help # everything you can do with runs
236
+ ora experiment runs create --help # flags and an example run.json
237
+ ```
238
+
239
+ Commands are named after the API's operations, each part in kebab-case
240
+ (`ora study benchmark-definitions list`). A query parameter the API requires is a required flag. A
241
+ request body comes from a JSON file (`--file <path>`, or `--file -` for stdin). An operation that
242
+ takes an idempotency key takes `--idempotency-key <key>`: send the same key again to retry, and the
243
+ change never runs twice.
244
+
245
+ ### Intents and goals: measure how agents treat your business (Studio)
246
+
247
+ ```bash
248
+ ora agents # the harness × model pairs you can measure with
249
+ ora intents list
250
+ cat > intent.json <<'EOF'
251
+ { "title": "Best managed Postgres for a B2B SaaS app", "layer": "discovery" }
252
+ EOF
253
+ ora intents create --file intent.json
254
+ ora intents measurements-status # are the daily measurements running?
255
+ ora intents measurements-pause-all # pause them
256
+ ora intents measurements-resume-all # resume them
257
+ echo '{}' | ora intents measure <intent-id> --file - --watch # measure now and follow it until it finishes
258
+ ora intents measure-status <intent-id> <measurement-id> --watch # each prompt × agent run, live
259
+ ora intents stream <intent-id> # the raw live events
260
+ cat > schedule.json <<'EOF'
261
+ { "cadence": "daily" }
262
+ EOF
263
+ ora intents measurement-policy <intent-id> --file schedule.json # daily | weekly | monthly | manual
264
+ ora intents get <intent-id> # its measurement policy and MCP servers: read before you replace its agents
265
+ ora mcp servers list # the organization's MCP servers and their ids (scope mcp:read)
266
+ ```
267
+
268
+ Goals are your own definition of a win, scored on every run:
269
+
270
+ ```bash
271
+ cat > goal.json <<'EOF'
272
+ { "kind": "answer_check", "title": "Acme DB is the first pick", "criterionGroup": "choose",
273
+ "statement": "The answer recommends Acme DB as its first option for hosting the database." }
274
+ EOF
275
+ ora goals create <intent-id> --file goal.json
276
+ ora goals list <intent-id>
277
+ ora goals report <intent-id> --window 30d # answer score per question type, weakest checks, prompt × agent grid
278
+ ora intents insights <intent-id> # what to fix: issues, pages that failed, MCP and API tool calls, what each agent said
279
+ ora intents answers <intent-id> # every answer: each brand's status, quote, reasons, drawbacks, who was recommended instead
280
+ ora intents analytics <intent-id> --window 30d # layers, positioning per question type, agents, brands, cited sources
281
+ ora intents history <intent-id> # each measurement, newest first, with its answer score
282
+ ora goals delete <goal-id> # deactivate it (kept, no longer scored)
283
+ ```
284
+
285
+ ### Runs: launch and follow agent runs (Experiments)
286
+
287
+ ```bash
288
+ ora experiment runs list --page-size 20
289
+
290
+ cat > run.json <<'EOF'
291
+ { "intent": "check pricing for acme.com", "harness": "eve", "model": "claude-sonnet-5" }
292
+ EOF
293
+ ora experiment runs create --file run.json # returns {runId, streamUrl, status}
294
+ ora experiment runs create --file run.json --watch # the same, then streams live events until done
295
+ ora experiment runs stream <run-id> # attach to a run already in flight
296
+ ora experiment runs feed # the organization's run status changes, live
297
+ ```
298
+
299
+ After a run completes, read what it produced:
300
+
301
+ ```bash
302
+ ora experiment runs insight <run-id> # the analyzed outcome
303
+ ora experiment runs trajectory <run-id> # the step-by-step trace
304
+ ora experiment runs raw-trace <run-id> # the full raw event log
305
+ ```
306
+
307
+ ### Sessions: long-lived agent environments
308
+
309
+ ```bash
310
+ ora sessions catalog # the harnesses and models you can launch
311
+ ora sessions list
312
+ ora sessions hibernate <id>
313
+ ora sessions wake <id>
314
+ ```
315
+
316
+ ### Studies: compare agent setups
317
+
318
+ A study runs one task set over 1 to 6 named **groups**, each a configuration of the agent (add-ons,
319
+ skills, tools). One group ranks its agents (a leaderboard); two or more groups are compared.
320
+
321
+ ```bash
322
+ cat > study.json <<'EOF'
323
+ {
324
+ "name": "A real browser against the agent as published",
325
+ "subjectName": "Playwright browser automation",
326
+ "promise": "An agent with a real browser reports content a page renders only after a click.",
327
+ "comparisonMode": "multi_arm",
328
+ "arms": [
329
+ { "label": "As published", "role": "reference", "config": {} },
330
+ { "label": "With a real browser", "config": { "addons": { "webdriver": {} } } }
331
+ ],
332
+ "matrix": [{ "harness": "claude-code", "provider": "anthropic", "model": "claude-haiku-4-5-20251001" }],
333
+ "tasks": [{
334
+ "prompt": "Open https://the-internet.herokuapp.com/dynamic_loading/2, click Start, and report the text that appears.",
335
+ "rubric": ["Reports the text exactly as \"Hello World!\".", "Does not report a loading indicator as the answer."]
336
+ }]
337
+ }
338
+ EOF
339
+ ora study catalog # valid harness × model cells
340
+ ora study addons list --harness claude-code --surface run # add-on ids and their config
341
+ ora study preflight --file study.json # what it would be: runs, comparisons; creates nothing
342
+ ora study create --file study.json # an authored task set lands `ready`
343
+ ora study launch <id> --watch
344
+ ora study get <id> # status, runs, results
345
+ ora study preview <id> # the page as it would publish
346
+ ora study publish <id> --file publish.json # {"slug": "my-study", "visibility": "public", "skillNames": "withheld"}
347
+ ora study update-details <id> --file details.json # rename or re-describe it, any time
348
+ ```
349
+
350
+ **Before you spend,** prove a design on its smallest real run first:
351
+
352
+ ```bash
353
+ ora study rehearse --file study.json # one real run per group; the same body as create
354
+ ora study rehearsal-report <id> # what it proved, group by group
355
+ ora study ci rehearse --file study.json --mode setup # wait for the answer; setup and mock start no agent
356
+ ```
357
+
358
+ **A study as a merge check.** A gate answers pass, fail, or "not enough runs to decide", against a
359
+ floor or against an earlier study of the same series:
360
+
361
+ ```bash
362
+ ora study gate <id> --file gate.json # {"maxDrop": {"pp": 10}, "baseline": {"series": {"branch": "main"}}}
363
+ ora study ci run --file study.json --gate gate.json --series cli/pr-gate
364
+ # create, launch, wait, answer: exit 0 pass, 1 fail, 2 undecided
365
+ ora study series # the organization's series
366
+ ora study series-trend --series cli/pr-gate # one series over time, split where it stopped being comparable
367
+ ora study carry-plan <id> # what a series study would carry instead of run, and why the rest runs
368
+ ora study set-carry <id> --file carry.json # turn carrying on or off before launch: {"carry": null} runs every job
369
+ ora study ci run … --no-carry # this one study runs every job, whatever its body file says
370
+ ```
371
+
372
+ A study that declares `"carry": {"from": "series"}` takes the runs of its unchanged jobs from the
373
+ series' previous study instead of running them again. Whether it does is yours to change until the
374
+ study launches.
375
+
376
+ **Reading the runs:**
377
+
378
+ ```bash
379
+ ora study query <id> --file query.json # which runs, split how, measured on what
380
+ ora study query-vocabulary <id> # what its runs can be split by and measured on
381
+ ora study save-view <id> --file view.json # keep a question; also `views`, `run-view`, `delete-view`
382
+ ora study behaviors <id> # the study's own questions about each run, and how they held up
383
+ ora study run-screens <id> <runId> # whether a run is valid evidence, with the quotes it stands on
384
+ ora study run-spans <id> <runId> # the run's stored spans
385
+ ora study run-delivered <id> <runId> # the files and images the run delivered
386
+ ```
387
+
388
+ **After it finished:**
389
+
390
+ ```bash
391
+ ora study revise-grading <id> --file revision.json # change a finished study's checks, with a reason; re-grades
392
+ ora study grading-revisions <id> # the record of every revision
393
+ ora study retest <id> --file retest.json # run the benchmark's own checker again over the files runs saved
394
+ ora study retests <id> # what every re-check found
395
+ ora study rerun-infra-failures <id> # re-run, in place, the jobs ora's own infrastructure failed
396
+ ```
397
+
398
+ **Scopes:** `study.studies:read` and `study.studies:write`, plus `study.addons:read` for the add-on
399
+ catalog.
400
+
401
+ ### Image builds: build and run custom harness images
402
+
403
+ ```bash
404
+ ora experiment images builds create --help # every source kind, with an example body
405
+ ```
406
+
407
+ Build a custom harness image from a context you already host (`archive`) or from a git repository
408
+ (`git`):
409
+
410
+ ```bash
411
+ cat > build.json <<'EOF'
412
+ { "name": "my-harness", "source": { "kind": "archive", "url": "https://example.com/ctx.tar.gz", "sha256": "…" } }
413
+ EOF
414
+ ora experiment images builds create --file build.json # -> {id, phase}
415
+ ora experiment images builds get <id> # phase, image, digest once succeeded
416
+ ora experiment images builds stream <id> # follow it live
417
+ ora experiment images builds list --phase succeeded
418
+ ora experiment images builds cancel <id> # in-flight builds only
419
+
420
+ cat > build-git.json <<'EOF'
421
+ { "name": "my-harness", "source": { "kind": "git", "url": "https://github.com/you/repo", "ref": "main" } }
422
+ EOF
423
+ ora experiment images builds create --file build-git.json
424
+ ```
92
425
 
93
- | Refusal | Meaning |
94
- | --- | --- |
95
- | `CI_NOT_TRUSTED` | No trust of the key admits this job: check the repository, the ref (`refs/pull/*/merge` for pull requests) and the environment |
96
- | `CI_TOKEN_INVALID` | The identity token did not verify, or was requested for another audience |
97
- | `CI_IDENTITY_UNAVAILABLE` | GitHub issued the job no identity token: add `permissions: id-token: write` |
426
+ Then launch a run from a build that reached `succeeded`:
98
427
 
99
- ### Point the CLI at an environment (`--api-url`)
428
+ ```bash
429
+ cat > run.json <<'EOF'
430
+ { "buildId": "<id>", "mode": "harness", "task": "say hello" }
431
+ EOF
432
+ ora experiment images runs create --file run.json # -> {runId}; follow it like any other run
433
+ ```
434
+
435
+ Builds from an uploaded file (the `upload` and `inline` sources) are not available from the CLI
436
+ yet: use `archive` or `git`.
437
+
438
+ **Scopes:** `images:read` (get, list, stream) and `images:write` (create, cancel, runs create).
439
+
440
+ ### Autopilot: standing teams of agents
441
+
442
+ A standing team of agents, each member with its own harness and model, that you send missions and
443
+ messages to, then watch live. Request bodies are JSON files: `--file <path>`, or `--file -` for
444
+ stdin.
445
+
446
+ ```bash
447
+ cat > team.json <<'EOF'
448
+ {
449
+ "name": "billing-crew",
450
+ "members": [
451
+ { "handle": "nova", "harness": "claude-code" },
452
+ { "handle": "juno", "harness": "codex" }
453
+ ],
454
+ "squads": [{ "name": "reviewers", "members": ["juno"] }]
455
+ }
456
+ EOF
457
+ ora autopilot strategies list # the strategies a team can be configured with
458
+ ora autopilot teams create --file team.json
459
+ ora autopilot teams list
460
+
461
+ cat > task.json <<'EOF'
462
+ { "target": { "kind": "squad", "id": "reviewers" }, "instructions": "Review PR 812 against the v2 schema." }
463
+ EOF
464
+ ora autopilot missions create billing-crew --file task.json --watch # start it, then stream its events
465
+ printf '%s' '{"to":["nova"],"text":"Use the v2 schema.","interrupt":true}' \
466
+ | ora autopilot messages send billing-crew --file -
467
+ ora autopilot missions pause billing-crew <mission-id>
468
+ ora autopilot missions stop billing-crew <mission-id> --mode immediate
469
+ ora autopilot teams stop billing-crew --mode graceful
470
+ ```
471
+
472
+ Every streamed line starts with its `#seq`. A stream does not reconnect on its own: to pick up
473
+ where one left off, pass the last seq you saw:
474
+
475
+ ```bash
476
+ ora autopilot teams stream billing-crew --after 1840 # replays the gap, then live
477
+ ora autopilot events list billing-crew --mission-id <mission-id> --after-seq 1840 --limit 500 --json
478
+ ```
479
+
480
+ **Scopes:** `autopilot:read` (lists, gets, streams), `autopilot:write` (team setup, missions,
481
+ messages, pause, resume, stop) and `autopilot:export:write` for `ora autopilot exports …`: an
482
+ export is bulk data, so a read-only key cannot run it. Every command's `--help` names the scope it
483
+ needs. Grant the key the `experiment` product too, and create it as a person: Autopilot acts for
484
+ the key's creator, so a key whose creator has left the organization is refused. A team file that
485
+ binds a repository credential (`resources.repos[].credential.secretRef`) also needs `secrets:bind`
486
+ and a key role other than Viewer: it attaches an organization secret by reference, and no command
487
+ returns its value, but the workload it is bound to can read it. The same rule covers every other
488
+ binding: an add-on's secret on `ora experiment runs create`, a session or a study, and an MCP
489
+ server's `secretId`.
490
+
491
+ An export is a manifest plus a signed `artifactUrl` that lives 15 minutes. Fetch the dataset with
492
+ any HTTP client:
493
+
494
+ ```bash
495
+ curl -o dataset.tar "$(ora autopilot exports get billing-crew <export-id> --json | jq -r .artifactUrl)"
496
+ ```
497
+
498
+ ### Tunnels: let ora reach a site on your machine
499
+
500
+ Like `ngrok http`: ora's scans and runs reach a site running on your machine. One command prints a
501
+ working address and forwards until Ctrl-C:
502
+
503
+ ```console
504
+ $ ora tunnel 3000
505
+
506
+ 🚇 ora tunnel — dev-3000
507
+ forwards localhost:3000 → 924649e0…
508
+ url https://924649e0-….t.ora.ai/?ora_token=ora_tt_…
509
+ only holders of this link reach it — paste it as a journey targetDomain
510
+ ● connected — Ctrl-C to stop
511
+ 200 GET / 8ms
512
+ ```
513
+
514
+ The port is chosen on your machine when you connect: ora can only ask your connected CLI to fetch,
515
+ and never reaches into your machine. Running `ora tunnel 3000` again reuses the same tunnel and
516
+ address.
517
+
518
+ | `--access` | Who reaches `https://<id>.t.ora.ai` | Use it for |
519
+ | --------------------- | ------------------------------------------------------------------------------ | ----------------------------------------- |
520
+ | `protected` (default) | Holders of the printed link (`?ora_token=…` or an `X-Ora-Tunnel-Token` header) | Runs, sharing with a teammate |
521
+ | `public` | Anyone, with no credential | Letting real AI agents reach the dev site |
522
+ | `private` | Nobody but ora's own scanners | Scans only |
523
+
524
+ Anything else answers 404, so a private tunnel looks the same as one that does not exist. A revoked
525
+ tunnel answers 410, and a tunnel whose CLI is not connected answers 502 `TUNNEL_NOT_CONNECTED`.
526
+
527
+ To point a run at your tunnel, create it with `--access public` and use its hostname as the
528
+ journey's `targetDomain` (for example `924649e0-….t.ora.ai`). To score a local site's agent
529
+ readiness, [`ax`](https://www.npmjs.com/package/ax) does it end to end with just `ORA_API_KEY`:
530
+ `npx ax audit localhost:3000 --tunnel ora`.
531
+
532
+ ```bash
533
+ ora tunnels list # every tunnel, with its status
534
+ ora tunnels get <id>
535
+ ora tunnels revoke <id> # end the live connection, keep the tunnel
536
+ ora tunnels delete <id> # revoke it and remove it entirely
537
+ ```
538
+
539
+ A quick local site to try it with:
540
+
541
+ ```bash
542
+ node -e 'require("http").createServer((q,r)=>{r.writeHead(200,{"content-type":"text/html"});r.end("<h1>it works</h1>")}).listen(3000)'
543
+ ora tunnel 3000 # then open the printed url
544
+ ```
545
+
546
+ ## Scripting and CI
547
+
548
+ `--json` prints the raw API response; with `--watch` it becomes NDJSON (the create response first,
549
+ then one line per event). Exit codes: 0 ok, 1 API or sign-in error, 2 usage error.
550
+
551
+ With `--json`, a failure is JSON too: the API's error body, as sent, on one stderr line
552
+ (`{error, code, details}`). A failure the API did not answer in JSON (not signed in, an unreadable
553
+ `--file`, a refused key exchange, a sign-in that did not complete) prints `{error, code}`, plus the
554
+ HTTP `status` (and `details.retryAfter`, in seconds) when the API answered:
555
+
556
+ ```bash
557
+ ora autopilot members add billing-crew --file member.json --json 2> err.json \
558
+ || jq -r '.code, .details.issues[0].refusal.wave' err.json
559
+
560
+ id=$(ora experiment runs create --file run.json --json | jq -r .runId)
561
+ ora experiment runs stream "$id" --json | jq -c 'select(.event == "step")'
562
+
563
+ ora experiment runs list --json | jq '.results[] | {id, harness, status}'
564
+ ```
565
+
566
+ In CI, export `ORA_API_KEY` instead of signing in, or sign in with no stored secret
567
+ ([keyless sign-in](#keyless-sign-in-from-github-actions)).
100
568
 
101
- By default the CLI talks to the URL embedded in its spec
102
- (`https://api.agentfront.sh`). To target staging, a dev cluster, or any
103
- self-hosted ora, set the API base **once at login** — it's remembered in
104
- `~/.config/ora/config.json`:
569
+ Studio views follow your terminal's light or dark background, read from `COLORFGBG`. Set
570
+ `ORA_THEME=light` or `ORA_THEME=dark` when your terminal does not report it.
105
571
 
106
- ora login --api-url https://api.staging.agentfront.sh # staging
107
- ora login --api-url http://api.ora.localhost:8081 # local k3d dev
572
+ ## Errors
108
573
 
109
- Browser sign-in finds its sign-in server from that URL (`api.<domain>` →
110
- `account.<domain>`); when yours lives elsewhere, name it with `--auth-url`
111
- (or `ORA_AUTH_URL`).
574
+ - **402**: a plan limit was hit. The message names the limit and where to upgrade.
575
+ - **403 `INSUFFICIENT_SCOPE`**: the credential lacks a permission, and the message names it. With a
576
+ browser sign-in, run `ora login` again and allow it. With an API key, create a key with that scope
577
+ and run `ora login --api-key` again.
578
+ - **429**: too many requests. The message says how long to wait.
579
+ - **A problem** (RFC 9457, `application/problem+json`): the CLI prints its code and detail, each
580
+ field error, the next steps with the command for each, whether to retry (and after how long),
581
+ whether a change may already have been applied (retry with the same `--idempotency-key`), and the
582
+ request id to quote if you contact us.
112
583
 
113
- Precedence for every command: `--api-url` flag > `ORA_API_URL` env >
114
- `config.json` > the spec's default. So one-off calls can override without
115
- re-logging-in:
584
+ ## Files and network
116
585
 
117
- ora tunnels list --api-url https://api.staging.agentfront.sh
118
- ORA_API_URL=https://api.staging.agentfront.sh ora whoami
586
+ The CLI keeps its state in `~/.config/ora/`, readable only by you:
119
587
 
120
- ## Tunnels — expose your localhost to ora (like `ngrok http`)
121
-
122
- Let ora's scans and journeys reach a dev site running on your machine.
123
- One command, prints a working URL, forwards until Ctrl-C:
124
-
125
- ora tunnel 3000
126
-
127
- 🚇 ora tunnel — dev-3000
128
- forwards localhost:3000 → 924649e0…
129
- url https://924649e0-….t.agentfront.sh/?ora_token=ora_tt_…
130
- only holders of this link reach it — paste it as a journey targetDomain
131
- ● connected — Ctrl-C to stop
132
- 200 GET / 8ms
133
-
134
- The port is chosen here, at connect time — the server can only ask your
135
- connected CLI to fetch; it never reaches into your machine. Re-running
136
- `ora tunnel 3000` reuses the same tunnel (and URL): the per-tunnel secret is
137
- remembered in `~/.config/ora/tunnel-secrets.json` (0600), like ngrok's
138
- authtoken.
139
-
140
- ### Access modes (`--access`)
141
-
142
- ora tunnel 3000 # protected (default): the printed
143
- # link works; strangers get 404
144
- ora tunnel 3000 --access public # open URL — answer engines can
145
- # browse/index the dev site
146
- ora tunnel 3000 --access private # no URL; only ora's own scanners
147
-
148
- | mode | who reaches `https://<id>.t.<domain>` | use it for |
149
- |---|---|---|
150
- | `protected` | holders of the printed link (`?ora_token=…` or an `X-Ora-Tunnel-Token` header) | journeys, sharing with a teammate |
151
- | `public` | anyone — no credential | letting real answer engines measure the dev site |
152
- | `private` | nobody — 404; only ora's in-cluster scanners via `tunnel://<id>/` | scan-only workflows |
153
-
154
- Everything else answers 404 identically (a private tunnel is
155
- indistinguishable from a nonexistent one), revoked answers 410, and a tunnel
156
- whose CLI isn't currently connected answers 502 `TUNNEL_NOT_CONNECTED`.
157
-
158
- ### Point a run at your tunnel
159
-
160
- - **Journey**: create the tunnel `--access public`, then use the hostname as
161
- the journey's `targetDomain` — e.g. `924649e0-….t.agentfront.sh`. The
162
- agent browses it like any site.
163
- - **Audit** (the agent-readiness audit at ora.ai): create the tunnel
164
- `--access public` and let `ax` run it end to end —
165
- `ax audit localhost:<port> --tunnel-cmd 'ora tunnel <port> --access public'`
166
- (npm `ax`; `ax audit localhost:<port> --tunnel ora` does the same with just
167
- `ORA_API_KEY`). `tunnel://<id>/` targets are for ora's in-cluster scanners
168
- only, not a public API.
169
-
170
- ### Manage tunnels
171
-
172
- ora tunnels list # status-glyph table (● connected ✖ revoked …)
173
- ora tunnels get <id>
174
- ora tunnels revoke <id> # kill the live connection, keep the row
175
- ora tunnels delete <id> # revoke + remove the row entirely
176
-
177
- Quick local target to test with:
178
-
179
- node -e 'require("http").createServer((q,r)=>{r.writeHead(200,{"content-type":"text/html"});r.end("<h1>it works</h1>")}).listen(3000)'
180
- ora tunnel 3000 # then open the printed url
181
-
182
- ## Use
183
-
184
- Every command's `--help` ends with a ready-to-run example (including a request
185
- body where one is needed) — the fastest way to learn a command is to ask it:
186
-
187
- ora --help # all command groups
188
- ora experiment runs --help # everything you can do with runs
189
- ora experiment runs create --help # flags + an example run.json
190
-
191
- ### Runs — launch and follow agent executions
192
-
193
- ora experiment runs list --page-size 20
194
-
195
- cat > run.json <<'EOF'
196
- { "intent": "check pricing for acme.com", "harness": "eve", "model": "claude-sonnet-5" }
197
- EOF
198
- ora experiment runs create --file run.json # returns {runId, streamUrl, status}
199
- ora experiment runs create --file run.json --watch # same, then streams live events until done
200
- ora experiment runs stream <run-id> # attach to a run already in flight
201
- ora experiment runs feed # org-wide live feed of run status changes
202
-
203
- After a run completes, read its artifacts:
204
-
205
- ora experiment runs insight <run-id> # the analyzed outcome
206
- ora experiment runs trajectory <run-id> # step-by-step trace
207
- ora experiment runs raw-trace <run-id> # the full raw event log
208
-
209
- ### Intents — AEO measurement (studio)
210
-
211
- ora agents # the harness × model pairs you can measure with
212
- ora intents list
213
- cat > intent.json <<'EOF'
214
- { "title": "Best AEO platform for B2B SaaS", "layer": "discovery" }
215
- EOF
216
- ora intents create --file intent.json
217
- ora intents measurements-status # are the daily measurements running?
218
- ora intents measurements-pause-all # stop the scans
219
- ora intents measurements-resume-all # restart them
220
- echo '{}' | ora intents measure <intent-id> --file - --watch # measure now and follow it live until it finishes
221
- ora intents measure-status <intent-id> <measurement-id> --watch # each prompt × agent run, live; done: true once every run is analyzed
222
- ora intents stream <intent-id> # the raw live events (a `measurement` event when one finishes)
223
- cat > schedule.json <<'EOF'
224
- { "cadence": "daily" }
225
- EOF
226
- ora intents measurement-policy <intent-id> --file schedule.json # daily | weekly | monthly | manual
227
- ora intents get <intent-id> # measurementPolicy (agents with their MCP servers) + mcpServers: read before you replace the agents list
228
- ora mcp servers list # the org's MCP servers and their ids (needs the mcp:read scope)
229
-
230
- ### Goals — your own definition of a win, scored on every run (studio)
231
-
232
- cat > goal.json <<'EOF'
233
- { "kind": "answer_check", "title": "Acme DB is the first pick", "criterionGroup": "choose",
234
- "statement": "The answer recommends Acme DB as its first option for hosting the database." }
235
- EOF
236
- ora goals create <intent-id> --file goal.json
237
- ora goals list <intent-id>
238
- ora goals report <intent-id> --window 30d # answer score per question type, weakest checks, prompt × agent grid, Studio link
239
- ora intents insights <intent-id> # what to fix: issues, pages that failed, MCP / API tool calls, what each agent said
240
- ora intents answers <intent-id> # every answer of a measurement: each brand's status, quote, reasons, drawbacks, who was recommended instead
241
- ora intents analytics <intent-id> --window 30d # layers, positioning per question type, agents, brands, cited sources
242
- ora intents history <intent-id> # each measurement, newest first, with its answer score
243
- ora goals delete <goal-id> # deactivate (kept, no longer scored)
244
-
245
- ### Skills — teach your coding agent to use Studio
246
-
247
- ora skills list # the skills your Studio account can install
248
- ora skills add ora-studio # this project: .claude/skills (Claude Code) and .agents/skills (Codex and others)
249
- ora skills add ora-studio --global # every project, in your home folder
250
-
251
- `ora-studio` is the Studio playbook: define the objective, create the intent, define the win, measure,
252
- read the results, improve, schedule, with a recipe per objective (be recommended, don't get steered
253
- away, book a demo, MCP discovery, MCP usage). It comes from your Studio account, so it needs the same
254
- sign-in or API key as the other commands (scope `intents:read`).
255
-
256
- ### MCP — give your coding agent the CLI's commands as tools
257
-
258
- ora mcp install --product study # this project, for Claude Code (.mcp.json)
259
- ora mcp install --agent cursor --product study # Cursor (.cursor/mcp.json)
260
- ora mcp config --product study --read-only # print the entry, for any other agent
261
- ora mcp serve --product study # what the agent runs: an MCP server on stdio
262
-
263
- `ora mcp serve` is the CLI served over the Model Context Protocol. Every tool is a command: `study_get`
264
- runs `ora study get <id>`, takes the same parameters (a JSON body goes in `body`) and answers the API's
265
- JSON. There is no second list: a command added to the CLI is a tool the next time the server starts.
266
-
267
- - `--product <key>` (repeatable) serves one product's commands: `study`, `studio`, `experiment` or
268
- `platform`. Without it the server offers every command, which is a long tool list for an agent.
269
- - `--read-only` serves only the commands that read. A read sent as a POST (a study's run query) stays.
270
- - **The API key still decides what any call may do.** The server never widens it. Give the agent a key
271
- with the scopes you want it to have.
272
- - A refusal reaches the agent as a tool error with the API's own status, code and words, so it can fix
273
- the call. A write is never retried.
274
- - The server uses the key the CLI already has (`ora login`, or `ORA_API_KEY` in the agent's
275
- environment). `install` writes the command and its flags into the agent's configuration, and never a key.
276
- It merges into an existing file and leaves every other server in it alone.
277
- - Commands that stream (`… stream`) are not offered: a tool call answers once.
278
- - An answer over 200,000 characters is refused with its size, not cut in half: ask a narrower command.
279
-
280
- ### Sessions — long-lived agent environments
281
-
282
- ora sessions catalog # harnesses + models you can launch
283
- ora sessions list
284
- ora sessions hibernate <id>
285
- ora sessions wake <id>
286
-
287
- ### Studies — compare agent setups (Groups)
288
-
289
- A study runs one task set over 1–6 named **groups**, each a configuration of
290
- the agent (add-ons, skills, tools). One group ranks its agents (a
291
- leaderboard); two or more groups are compared.
292
-
293
- cat > study.json <<'EOF'
294
- {
295
- "name": "A real browser against the agent as published",
296
- "subjectName": "Playwright browser automation",
297
- "promise": "An agent with a real browser reports content a page renders only after a click.",
298
- "comparisonMode": "multi_arm",
299
- "arms": [
300
- { "label": "As published", "role": "reference", "config": {} },
301
- { "label": "With a real browser", "config": { "addons": { "webdriver": {} } } }
302
- ],
303
- "matrix": [{ "harness": "claude-code", "provider": "anthropic", "model": "claude-haiku-4-5-20251001" }],
304
- "tasks": [{
305
- "prompt": "Open https://the-internet.herokuapp.com/dynamic_loading/2, click Start, and report the text that appears.",
306
- "rubric": ["Reports the text exactly as \"Hello World!\".", "Does not report a loading indicator as the answer."]
307
- }]
308
- }
309
- EOF
310
- ora study catalog # valid harness × model cells
311
- ora study addons list --harness claude-code --surface run # add-on ids + their config
312
- ora study preflight --file study.json # what it would be: runs, comparisons — creates nothing
313
- ora study create --file study.json # an authored task set lands `ready`
314
- ora study launch <id> --watch
315
- ora study get <id> # status, runs, results
316
- ora study preview <id> # the page as it would publish
317
- ora study publish <id> --file publish.json # {"slug": "my-study", "visibility": "public", "skillNames": "withheld"}
318
- ora study update-details <id> --file details.json # rename or re-describe it, any time
319
-
320
- **Before you spend.** Prove a design on its smallest real run first:
321
-
322
- ora study rehearse --file study.json # one real run per group; same body as create
323
- ora study rehearsal-report <id> # what it proved, group by group
324
- ora study ci rehearse --file study.json --mode setup # wait for the answer; setup / mock start no agent
325
-
326
- **A study as a merge check.** A gate answers pass, fail, or "not enough runs
327
- to decide", against a floor or against an earlier study of the same series:
328
-
329
- ora study gate <id> --file gate.json # {"maxDrop": {"pp": 10}, "baseline": {"series": {"branch": "main"}}}
330
- ora study ci run --file study.json --gate gate.json --series cli/pr-gate
331
- # create, launch, wait, answer: exit 0 pass, 1 fail, 2 undecided
332
- ora study series # the organization's series
333
- ora study series-trend --series cli/pr-gate # one series over time, split where it stopped being comparable
334
- ora study carry-plan <id> # what a series study would carry instead of run, and why the rest runs
335
- ora study set-carry <id> --file carry.json # turn carrying on or off before launch: {"carry": null} runs every job
336
- ora study ci run … --no-carry # this one study runs every job, whatever its body file says
337
-
338
- A study that declares `"carry": {"from": "series"}` takes the runs of its
339
- unchanged jobs from the series' previous study instead of running them again.
340
- Whether it does is yours to change until the study launches.
341
-
342
- **Reading the runs.**
343
-
344
- ora study query <id> --file query.json # which runs, split how, measured on what
345
- ora study query-vocabulary <id> # what its runs can be split by and measured on
346
- ora study save-view <id> --file view.json # keep a question; `views`, `run-view`, `delete-view`
347
- ora study behaviors <id> # the study's own questions about each run, and how they held up
348
- ora study run-screens <id> <runId> # whether a run is valid evidence, with the quotes it stands on
349
- ora study run-spans <id> <runId> # the run's stored spans
350
- ora study run-delivered <id> <runId> # the files and images the run delivered
351
-
352
- **After it finished.**
353
-
354
- ora study revise-grading <id> --file revision.json # change a finished study's checks, with a reason; re-grades
355
- ora study grading-revisions <id> # the record of every revision
356
- ora study retest <id> --file retest.json # run the benchmark's own checker again over the files runs saved
357
- ora study retests <id> # what every re-check found
358
- ora study rerun-infra-failures <id> # re-run, in place, the jobs ora's own infrastructure failed
359
-
360
- **Scopes**: `study.studies:read` / `study.studies:write`, plus `study.addons:read`
361
- for the add-on catalog.
362
-
363
- ### Image builds — build and run custom harness images
364
-
365
- ora experiment images builds create --help # every source kind + an example body
366
-
367
- Build a custom harness image from a context you already host somewhere
368
- (`archive`) or a git repo (`git`) — both work end to end from the CLI alone,
369
- api-key auth included:
370
-
371
- cat > build.json <<'EOF'
372
- { "name": "my-harness", "source": { "kind": "archive", "url": "https://example.com/ctx.tar.gz", "sha256": "…" } }
373
- EOF
374
- ora experiment images builds create --file build.json # -> {id, phase}
375
- ora experiment images builds get <id> # phase, image, digest once succeeded
376
- ora experiment images builds stream <id> # SSE — follow it live instead of polling
377
- ora experiment images builds list --phase succeeded
378
- ora experiment images builds cancel <id> # in-flight builds only
379
-
380
- cat > build-git.json <<'EOF'
381
- { "name": "my-harness", "source": { "kind": "git", "url": "https://github.com/you/repo", "ref": "main" } }
382
- EOF
383
- ora experiment images builds create --file build-git.json
588
+ | File | What it holds |
589
+ | --------------------- | ------------------------------------------------------------ |
590
+ | `session.json` | Your browser sign-in, until `ora logout` |
591
+ | `config.json` | A stored API key and your settings |
592
+ | `tunnel-secrets.json` | Each tunnel's secret, so `ora tunnel 3000` keeps its address |
384
593
 
385
- Then launch a run from a build that reached `succeeded`:
594
+ It talks to ora (`api.ora.ai`, `account.ora.ai` for sign-in, `tunnel.ora.ai` for tunnels). Once a
595
+ day it also asks the npm registry whether a newer version is out and, if so, prints one line on
596
+ stderr (never with `--json`). Set `ORA_NO_UPDATE_CHECK=1` to turn that off.
597
+
598
+ ## Support
599
+
600
+ Questions, problems or ideas: [hello@ora.ai](mailto:hello@ora.ai). When a command fails, include
601
+ the request id it printed.
602
+
603
+ ## License
386
604
 
387
- cat > run.json <<'EOF'
388
- { "buildId": "<id>", "mode": "harness", "task": "say hello" }
389
- EOF
390
- ora experiment images runs create --file run.json # -> {runId} — follow it like any other run
391
-
392
- **`upload`/`inline` sources are NOT reachable from this CLI.** Both need a
393
- files-service `fileId` (`upload` directly; `inline` materializes one for
394
- you), and files-service is closed to api-key callers ENTIRELY — every one of
395
- its routes 403s `NOT_PUBLIC_API` for an `api_key`-typed token, `inline`
396
- additionally requires an attributable human (`created_by` is a real
397
- `users.id`, which an api-key principal's synthetic grant id can't satisfy).
398
- Only a real user SESSION bearer (`POST /api/auth/login`, not `ora login`'s
399
- api key) can call `POST /files/upload/direct` to mint a `fileId` first — a
400
- script driving both steps needs that separate login, this CLI's own
401
- `ora login` cannot supply it. `archive`/`git` have no such requirement:
402
- that's why they're the two kinds shown above.
403
-
404
- **Scopes**: these commands need an api key minted with `images:read`
405
- (get/list/stream) and/or `images:write` (create/cancel/runs create) — see
406
- "Errors that explain themselves" below if a command answers 403
407
- `INSUFFICIENT_SCOPE`.
408
-
409
- ### Autopilot — teams
410
-
411
- A standing team of agents (each member its own harness and model) that you
412
- send missions and messages to, then watch live. Request bodies are JSON
413
- files: `--file <path>`, or `--file -` for stdin (YAML is not accepted yet).
414
-
415
- cat > team.json <<'EOF'
416
- {
417
- "name": "billing-crew",
418
- "members": [
419
- { "handle": "nova", "harness": "claude-code" },
420
- { "handle": "juno", "harness": "codex" }
421
- ],
422
- "squads": [{ "name": "reviewers", "members": ["juno"] }]
423
- }
424
- EOF
425
- ora autopilot strategies list # the strategy catalog a team can be configured with
426
- ora autopilot teams create --file team.json
427
- ora autopilot teams list
428
-
429
- cat > task.json <<'EOF'
430
- { "target": { "kind": "squad", "id": "reviewers" }, "instructions": "Review PR 812 against the v2 schema." }
431
- EOF
432
- ora autopilot missions create billing-crew --file task.json --watch # start it, then stream its events
433
- printf '%s' '{"to":["nova"],"text":"Use the v2 schema.","interrupt":true}' \
434
- | ora autopilot messages send billing-crew --file -
435
- ora autopilot missions pause billing-crew <mission-id>
436
- ora autopilot missions stop billing-crew <mission-id> --mode immediate
437
- ora autopilot teams stop billing-crew --mode graceful
438
-
439
- Every streamed line starts with its `#seq`. A stream does not reconnect on
440
- its own: to pick up where one left off, pass the last seq you saw:
441
-
442
- ora autopilot teams stream billing-crew --after 1840 # replays the gap, then live
443
- ora autopilot events list billing-crew --mission-id <mission-id> --after-seq 1840 --limit 500 --json
444
-
445
- **Scopes**: `autopilot:read` (lists, gets, streams), `autopilot:write`
446
- (team setup, missions, messages, pause/resume/stop), and
447
- `autopilot:export:write` for `ora autopilot exports …` — a dataset export is
448
- bulk data egress, so a read-only key cannot run it. Every command's `--help`
449
- names the scope it needs. Grant the key the `experiment` product too (the
450
- Autopilot scopes belong to it), and mint it as a person: Autopilot acts for
451
- the key's creator, so a key whose creator has left the organization is
452
- refused. A team file that binds a repo credential
453
- (`resources.repos[].credential.secretRef`) also needs `secrets:bind` on the
454
- key, and a key role other than Viewer: it attaches an org secret by reference
455
- and no route returns its value — but the workload it is bound to can read it
456
- (keys minted before `secrets:bind` existed hold `secrets:reveal`, which still
457
- works). The same rule covers every other binding: an add-on's secret on
458
- `ora experiment runs create`, a session or a study, and an MCP server's
459
- `secretId`. A person needs any organization role except Viewer to bind.
460
-
461
- An export is a manifest plus a signed `artifactUrl` that lives 15 minutes;
462
- fetch the dataset with any HTTP client:
463
-
464
- curl -o dataset.tar "$(ora autopilot exports get billing-crew <export-id> --json | jq -r .artifactUrl)"
465
-
466
- ### Scripting
467
-
468
- `--json` prints the raw API response; with `--watch` it becomes NDJSON (the
469
- create response first, then one line per event). Exit codes: 0 ok, 1 API/auth
470
- error, 2 usage error.
471
-
472
- With `--json`, a failure is JSON too: the API's error body, as sent, on one
473
- stderr line — `{error, code, details}`, the fields a client generated from the
474
- public API reads. An Autopilot refusal names its wave and blocker there. A
475
- failure the API did not answer in JSON (not signed in, an unreadable `--file`,
476
- a refused key exchange, a sign-in that did not complete) prints `{error, code}`:
477
-
478
- ora autopilot members add billing-crew --file member.json --json 2> err.json \
479
- || jq -r '.code, .details.issues[0].refusal.wave' err.json
480
-
481
- id=$(ora experiment runs create --file run.json --json | jq -r .runId)
482
- ora experiment runs stream "$id" --json | jq -c 'select(.event == "step")'
483
-
484
- ora experiment runs list --json | jq '.results[] | {id, harness, status}'
485
-
486
- In CI, export `ORA_API_KEY` instead of logging in, or sign in with no stored secret
487
- ([keyless sign-in](#keyless-sign-in-from-github-actions)). Point at another
488
- environment with `ORA_API_URL` or `--api-url`.
489
-
490
- Studio views are drawn in Ora cream on a dark terminal and in Ora ink on a light one, read from
491
- the terminal's `COLORFGBG`. Set `ORA_THEME=light` or `ORA_THEME=dark` when your terminal does not report it.
492
-
493
- ## Errors that explain themselves
494
-
495
- - **402** — a plan limit was hit; the message names the dimension and where to upgrade.
496
- - **403 INSUFFICIENT_SCOPE** — the credential lacks a scope; the message names
497
- it. With a browser sign-in, run `ora login` again and allow that permission.
498
- With an api key, mint a key with that scope and `ora login --api-key` again.
605
+ MIT