oui-spec 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +680 -0
- package/dist/chunk-2ON5IEXB.cjs +167 -0
- package/dist/chunk-2ON5IEXB.cjs.map +1 -0
- package/dist/chunk-3DCIL4FV.cjs +14 -0
- package/dist/chunk-3DCIL4FV.cjs.map +1 -0
- package/dist/chunk-7EXSMPGR.js +207 -0
- package/dist/chunk-7EXSMPGR.js.map +1 -0
- package/dist/chunk-CVUUSPF4.js +12 -0
- package/dist/chunk-CVUUSPF4.js.map +1 -0
- package/dist/chunk-JO5LGOB6.js +88 -0
- package/dist/chunk-JO5LGOB6.js.map +1 -0
- package/dist/chunk-LEXIU2MH.cjs +90 -0
- package/dist/chunk-LEXIU2MH.cjs.map +1 -0
- package/dist/chunk-UIGRLZOX.js +164 -0
- package/dist/chunk-UIGRLZOX.js.map +1 -0
- package/dist/chunk-WZXUKJ7E.cjs +210 -0
- package/dist/chunk-WZXUKJ7E.cjs.map +1 -0
- package/dist/core/index.cjs +17 -0
- package/dist/core/index.cjs.map +1 -0
- package/dist/core/index.d.cts +120 -0
- package/dist/core/index.d.ts +120 -0
- package/dist/core/index.js +4 -0
- package/dist/core/index.js.map +1 -0
- package/dist/index.cjs +35 -0
- package/dist/index.cjs.map +1 -0
- package/dist/index.d.cts +4 -0
- package/dist/index.d.ts +4 -0
- package/dist/index.js +6 -0
- package/dist/index.js.map +1 -0
- package/dist/react/index.cjs +16 -0
- package/dist/react/index.cjs.map +1 -0
- package/dist/react/index.d.cts +62 -0
- package/dist/react/index.d.ts +62 -0
- package/dist/react/index.js +3 -0
- package/dist/react/index.js.map +1 -0
- package/dist/spec/index.cjs +12 -0
- package/dist/spec/index.cjs.map +1 -0
- package/dist/spec/index.d.cts +249 -0
- package/dist/spec/index.d.ts +249 -0
- package/dist/spec/index.js +3 -0
- package/dist/spec/index.js.map +1 -0
- package/dist/transport/index.cjs +16 -0
- package/dist/transport/index.cjs.map +1 -0
- package/dist/transport/index.d.cts +91 -0
- package/dist/transport/index.d.ts +91 -0
- package/dist/transport/index.js +3 -0
- package/dist/transport/index.js.map +1 -0
- package/package.json +109 -0
- package/spec/OUI-SPEC-v0.1.md +1323 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Closure Studio
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,680 @@
|
|
|
1
|
+
# OUI - The OpenUI Specification
|
|
2
|
+
|
|
3
|
+
**The machine-readable contract for agent-controllable user interfaces.**
|
|
4
|
+
|
|
5
|
+
> _"Oui"_ — French for "yes." As in: yes, an AI agent can control this UI.
|
|
6
|
+
|
|
7
|
+
<!-- Badges -->
|
|
8
|
+
|
|
9
|
+
[](https://www.npmjs.com/package/oui-spec)
|
|
10
|
+
[](LICENSE)
|
|
11
|
+
[](https://www.typescriptlang.org/)
|
|
12
|
+
[](https://github.com/wesreid/oui/actions/workflows/ci.yml)
|
|
13
|
+
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
## The Problem
|
|
17
|
+
|
|
18
|
+
AI agents that control UIs today rely on fragile, expensive hacks:
|
|
19
|
+
|
|
20
|
+
- **Computer vision** — screenshot the screen, feed it to a vision model, infer what's clickable, guess coordinates, click, screenshot again to see if it worked. Slow, expensive, unreliable.
|
|
21
|
+
- **DOM scraping** — find elements by CSS selector, simulate clicks and keystrokes. Breaks on every layout change. Requires intimate knowledge of the app's internal structure.
|
|
22
|
+
- **MCP tool servers** — powerful for backend operations, but have no concept of _UI state_. The agent can call APIs but can't observe what the user is seeing or coordinate with the frontend.
|
|
23
|
+
|
|
24
|
+
All of these share a fundamental flaw: **the agent is guessing what the application can do by examining its implementation details.** It's like reading a restaurant menu by peering into the kitchen.
|
|
25
|
+
|
|
26
|
+
## The Solution
|
|
27
|
+
|
|
28
|
+
OUI inverts the model. **The application declares a typed semantic surface** — what it can do, what state is observable, and what parameters each operation accepts. The agent reads this manifest and operates through typed function calls. No pixels. No DOM. No guessing.
|
|
29
|
+
|
|
30
|
+
```
|
|
31
|
+
OpenAPI + HTTP Server = any client can consume the API
|
|
32
|
+
OUI + UI App = any agent can control the UI
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
```mermaid
|
|
36
|
+
flowchart TD
|
|
37
|
+
Agent[Agent Runtime / LLM]
|
|
38
|
+
Manifest[OUI Manifest]
|
|
39
|
+
Dispatch[Dispatch Channel]
|
|
40
|
+
Observe[Observation Channel]
|
|
41
|
+
Surface[UI Surface]
|
|
42
|
+
App[Application State]
|
|
43
|
+
|
|
44
|
+
Agent -->|reads| Manifest
|
|
45
|
+
Agent -->|fire-and-forget action| Dispatch
|
|
46
|
+
Dispatch -->|WebSocket| Surface
|
|
47
|
+
Surface -->|executes handler| App
|
|
48
|
+
App -->|state changes| Surface
|
|
49
|
+
Surface -->|pushes observations| Observe
|
|
50
|
+
Observe -->|WebSocket| Agent
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
### Fully Async. Fully Stateless. No Waiting.
|
|
54
|
+
|
|
55
|
+
OUI uses **two one-way channels** — not request/response. This is the critical architectural decision:
|
|
56
|
+
|
|
57
|
+
| Channel | Direction | Purpose |
|
|
58
|
+
| --------------- | --------------- | ------------------------------------ |
|
|
59
|
+
| **Dispatch** | Server → Client | Fire action instructions at the UI |
|
|
60
|
+
| **Observation** | Client → Server | Push state updates back to the agent |
|
|
61
|
+
|
|
62
|
+
The agent runtime (e.g., a Lambda function) dispatches an action and **terminates immediately**. It does not wait for a response. The browser executes the action, starts any async work, and pushes observation updates over the observation channel as results arrive. The agent picks up those observations on its next invocation.
|
|
63
|
+
|
|
64
|
+
This means:
|
|
65
|
+
|
|
66
|
+
- **No Lambda waiting** — no 30-second timeouts holding a connection open
|
|
67
|
+
- **No request/response correlation** — no correlation IDs, no pending promises
|
|
68
|
+
- **No WebSocket held open by the server** — the server writes and disconnects
|
|
69
|
+
- **Long-running operations work naturally** — the UI polls or subscribes and pushes status updates over time
|
|
70
|
+
|
|
71
|
+
---
|
|
72
|
+
|
|
73
|
+
## Quick Start
|
|
74
|
+
|
|
75
|
+
The simplest possible OUI surface — a counter the agent can increment:
|
|
76
|
+
|
|
77
|
+
```typescript
|
|
78
|
+
// counter.surface.ts
|
|
79
|
+
import { defineSurface } from "oui-spec/core";
|
|
80
|
+
|
|
81
|
+
export const counterSurface = defineSurface({
|
|
82
|
+
id: "counter",
|
|
83
|
+
name: "Counter",
|
|
84
|
+
description: "A simple counter that can be incremented or reset",
|
|
85
|
+
actions: [
|
|
86
|
+
{
|
|
87
|
+
id: "increment",
|
|
88
|
+
description: "Increment the counter by a given amount",
|
|
89
|
+
input: {
|
|
90
|
+
type: "object",
|
|
91
|
+
properties: {
|
|
92
|
+
amount: { type: "number", description: "Amount to add (default 1)" },
|
|
93
|
+
},
|
|
94
|
+
},
|
|
95
|
+
handler: async (params, ctx) => {
|
|
96
|
+
const amount = (params.amount as number) ?? 1;
|
|
97
|
+
ctx.setCount(ctx.count + amount);
|
|
98
|
+
return { success: true, data: { newValue: ctx.count + amount } };
|
|
99
|
+
},
|
|
100
|
+
},
|
|
101
|
+
{
|
|
102
|
+
id: "reset",
|
|
103
|
+
description: "Reset the counter to zero",
|
|
104
|
+
input: { type: "object", properties: {} },
|
|
105
|
+
handler: async (_params, ctx) => {
|
|
106
|
+
ctx.setCount(0);
|
|
107
|
+
return { success: true, data: { newValue: 0 } };
|
|
108
|
+
},
|
|
109
|
+
},
|
|
110
|
+
],
|
|
111
|
+
observations: [
|
|
112
|
+
{
|
|
113
|
+
id: "current_value",
|
|
114
|
+
description: "The current counter value",
|
|
115
|
+
schema: { type: "object", properties: { value: { type: "number" } } },
|
|
116
|
+
},
|
|
117
|
+
],
|
|
118
|
+
});
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
```tsx
|
|
122
|
+
// CounterPage.tsx
|
|
123
|
+
import { useSurface, useObservation } from "oui-spec/react";
|
|
124
|
+
import { counterSurface } from "./counter.surface";
|
|
125
|
+
|
|
126
|
+
function CounterPage({ transport }) {
|
|
127
|
+
const [count, setCount] = useState(0);
|
|
128
|
+
|
|
129
|
+
const { handleActionRequest } = useSurface({
|
|
130
|
+
surface: counterSurface,
|
|
131
|
+
context: { count, setCount },
|
|
132
|
+
send: transport.send,
|
|
133
|
+
});
|
|
134
|
+
|
|
135
|
+
// Push observation whenever count changes
|
|
136
|
+
useObservation(
|
|
137
|
+
counterSurface.id,
|
|
138
|
+
"current_value",
|
|
139
|
+
{ value: count },
|
|
140
|
+
transport.send,
|
|
141
|
+
);
|
|
142
|
+
|
|
143
|
+
return <div>Count: {count}</div>;
|
|
144
|
+
}
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
That's it. The agent now sees a `counter` surface with two tools (`increment`, `reset`) and one live observation (`current_value`).
|
|
148
|
+
|
|
149
|
+
---
|
|
150
|
+
|
|
151
|
+
## Architecture
|
|
152
|
+
|
|
153
|
+
### Two One-Way Channels
|
|
154
|
+
|
|
155
|
+
```mermaid
|
|
156
|
+
sequenceDiagram
|
|
157
|
+
participant Lambda as Agent Lambda
|
|
158
|
+
participant WS as WebSocket Server
|
|
159
|
+
participant Browser as Browser / UI
|
|
160
|
+
|
|
161
|
+
Note over Lambda: Agent decides to act
|
|
162
|
+
Lambda->>WS: dispatch(surfaceId, actionId, params)
|
|
163
|
+
Note over Lambda: Lambda terminates immediately
|
|
164
|
+
|
|
165
|
+
WS->>Browser: oui:dispatch event
|
|
166
|
+
Note over Browser: Handler executes
|
|
167
|
+
Note over Browser: Async work begins
|
|
168
|
+
|
|
169
|
+
loop Polling / Subscribe
|
|
170
|
+
Browser->>Browser: Check job status
|
|
171
|
+
Browser->>WS: oui:observation (interim update)
|
|
172
|
+
end
|
|
173
|
+
|
|
174
|
+
Browser->>WS: oui:observation (final result)
|
|
175
|
+
Note over Lambda: Next invocation reads observations
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
The dispatch channel carries action instructions **downward** (server → client). The observation channel carries state updates **upward** (client → server). They never mix. There is no concept of "waiting for a response."
|
|
179
|
+
|
|
180
|
+
### Why Not Request/Response?
|
|
181
|
+
|
|
182
|
+
In a typical agent architecture:
|
|
183
|
+
|
|
184
|
+
1. The agent runs in a Lambda (or similar serverless function)
|
|
185
|
+
2. The Lambda has a 30-second timeout
|
|
186
|
+
3. The UI operation might take 2 minutes (file upload, ML inference, user confirmation)
|
|
187
|
+
|
|
188
|
+
Request/response forces the Lambda to hold a connection open until the operation completes. That either times out or requires expensive long-running compute.
|
|
189
|
+
|
|
190
|
+
OUI's two-channel model means the Lambda does O(1) work — write a message to the dispatch channel and exit. The browser handles the rest on its own timeline and pushes results as observations. The agent reads those observations whenever it runs next.
|
|
191
|
+
|
|
192
|
+
---
|
|
193
|
+
|
|
194
|
+
## Polling & Async Operations
|
|
195
|
+
|
|
196
|
+
Many UI operations are not instant. A DataViz chart generation might take 30 seconds. A file export might need to wait for server-side rendering. OUI handles this with **declarative polling configuration**:
|
|
197
|
+
|
|
198
|
+
```typescript
|
|
199
|
+
{
|
|
200
|
+
id: 'generate_chart',
|
|
201
|
+
description: 'Generate a visualization from the configured dataset',
|
|
202
|
+
async: true,
|
|
203
|
+
estimatedDuration: '10-30s',
|
|
204
|
+
input: {
|
|
205
|
+
type: 'object',
|
|
206
|
+
properties: {
|
|
207
|
+
chartType: { type: 'string', enum: ['bar', 'line', 'scatter', 'pie'] },
|
|
208
|
+
dataset: { type: 'string', description: 'Dataset ID to visualize' },
|
|
209
|
+
},
|
|
210
|
+
required: ['chartType', 'dataset'],
|
|
211
|
+
},
|
|
212
|
+
polling: {
|
|
213
|
+
intervalMs: 2000,
|
|
214
|
+
maxAttempts: 30,
|
|
215
|
+
resolve: async (dispatchResult, ctx) => {
|
|
216
|
+
const job = await ctx.api.getJobStatus(dispatchResult.jobId);
|
|
217
|
+
if (job.status === 'complete') {
|
|
218
|
+
return { done: true, data: { status: 'complete', chartUrl: job.outputUrl } };
|
|
219
|
+
}
|
|
220
|
+
return { done: false, data: { status: job.status, progress: job.progress } };
|
|
221
|
+
},
|
|
222
|
+
},
|
|
223
|
+
handler: async (params, ctx) => {
|
|
224
|
+
const job = await ctx.api.startChartGeneration(params);
|
|
225
|
+
return { success: true, data: { jobId: job.id } };
|
|
226
|
+
},
|
|
227
|
+
}
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
**What happens at runtime:**
|
|
231
|
+
|
|
232
|
+
1. Agent dispatches `generate_chart` with params
|
|
233
|
+
2. Handler fires, calls the API, returns `{ jobId: 'xyz' }` immediately
|
|
234
|
+
3. OUI runtime starts polling every 2 seconds using the `resolve` function
|
|
235
|
+
4. Each poll pushes an observation: `{ status: 'processing', progress: 45, interim: true }`
|
|
236
|
+
5. When `resolve` returns `{ done: true }`, polling stops and a final observation is pushed
|
|
237
|
+
6. The agent sees the final `{ status: 'complete', chartUrl: '...' }` observation on its next invocation
|
|
238
|
+
|
|
239
|
+
### Subscribe Mode (Alternative to Polling)
|
|
240
|
+
|
|
241
|
+
If you have realtime infrastructure (WebSocket events, SSE, etc.), you can use subscribe mode instead of polling:
|
|
242
|
+
|
|
243
|
+
```typescript
|
|
244
|
+
polling: {
|
|
245
|
+
intervalMs: 0, // Not used in subscribe mode
|
|
246
|
+
subscribe: {
|
|
247
|
+
event: 'job:complete',
|
|
248
|
+
filter: { jobId: '$dispatchResult.jobId' },
|
|
249
|
+
},
|
|
250
|
+
},
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
The `$dispatchResult.` prefix tells the runtime to resolve the filter value from the handler's return data. The transport layer listens for the named event and pushes the observation when it arrives.
|
|
254
|
+
|
|
255
|
+
---
|
|
256
|
+
|
|
257
|
+
## Real-World Example: DataViz Surface
|
|
258
|
+
|
|
259
|
+
A complete DataViz wizard surface — the kind of thing you'd build for an AI-assisted analytics tool:
|
|
260
|
+
|
|
261
|
+
```typescript
|
|
262
|
+
// dataviz.surface.ts
|
|
263
|
+
import { defineSurface } from "oui-spec/core";
|
|
264
|
+
import type { DataVizContext } from "./types";
|
|
265
|
+
|
|
266
|
+
export const datavizSurface = defineSurface<DataVizContext>({
|
|
267
|
+
id: "dataviz-wizard",
|
|
268
|
+
name: "Data Visualization Wizard",
|
|
269
|
+
description:
|
|
270
|
+
"AI-controllable data visualization builder. Supports dataset selection, chart configuration, filtering, and export.",
|
|
271
|
+
version: "1.0.0",
|
|
272
|
+
|
|
273
|
+
activation: {
|
|
274
|
+
routes: ["/studio/dataviz", "/studio/dataviz/*"],
|
|
275
|
+
condition: "User has an active project with at least one dataset",
|
|
276
|
+
},
|
|
277
|
+
|
|
278
|
+
actions: [
|
|
279
|
+
{
|
|
280
|
+
id: "select_dataset",
|
|
281
|
+
description:
|
|
282
|
+
"Select a dataset to visualize. Must be called before configuring a chart.",
|
|
283
|
+
input: {
|
|
284
|
+
type: "object",
|
|
285
|
+
properties: {
|
|
286
|
+
datasetId: {
|
|
287
|
+
type: "string",
|
|
288
|
+
description: "The dataset ID from available_datasets observation",
|
|
289
|
+
},
|
|
290
|
+
},
|
|
291
|
+
required: ["datasetId"],
|
|
292
|
+
},
|
|
293
|
+
handler: async (params, ctx) => {
|
|
294
|
+
const dataset = await ctx.api.loadDataset(params.datasetId as string);
|
|
295
|
+
ctx.updateState({ selectedDataset: dataset, step: "configure" });
|
|
296
|
+
return {
|
|
297
|
+
success: true,
|
|
298
|
+
data: { columns: dataset.columns, rowCount: dataset.rowCount },
|
|
299
|
+
};
|
|
300
|
+
},
|
|
301
|
+
},
|
|
302
|
+
{
|
|
303
|
+
id: "configure_chart",
|
|
304
|
+
description: "Configure the chart type, axes, and visual options",
|
|
305
|
+
input: {
|
|
306
|
+
type: "object",
|
|
307
|
+
properties: {
|
|
308
|
+
chartType: {
|
|
309
|
+
type: "string",
|
|
310
|
+
enum: ["bar", "line", "scatter", "pie", "heatmap", "treemap"],
|
|
311
|
+
},
|
|
312
|
+
xAxis: { type: "string", description: "Column name for x-axis" },
|
|
313
|
+
yAxis: { type: "string", description: "Column name for y-axis" },
|
|
314
|
+
colorBy: {
|
|
315
|
+
type: "string",
|
|
316
|
+
description: "Column to use for color encoding (optional)",
|
|
317
|
+
},
|
|
318
|
+
aggregation: {
|
|
319
|
+
type: "string",
|
|
320
|
+
enum: ["sum", "avg", "count", "min", "max"],
|
|
321
|
+
description: "Aggregation function for y-axis",
|
|
322
|
+
},
|
|
323
|
+
},
|
|
324
|
+
required: ["chartType", "xAxis", "yAxis"],
|
|
325
|
+
},
|
|
326
|
+
preconditions: "A dataset must be selected first (select_dataset)",
|
|
327
|
+
handler: async (params, ctx) => {
|
|
328
|
+
ctx.updateState({ chartConfig: params, step: "preview" });
|
|
329
|
+
return {
|
|
330
|
+
success: true,
|
|
331
|
+
data: { configured: true, chartType: params.chartType },
|
|
332
|
+
};
|
|
333
|
+
},
|
|
334
|
+
},
|
|
335
|
+
{
|
|
336
|
+
id: "apply_filter",
|
|
337
|
+
description: "Add a filter to narrow the visualized data",
|
|
338
|
+
input: {
|
|
339
|
+
type: "object",
|
|
340
|
+
properties: {
|
|
341
|
+
column: { type: "string" },
|
|
342
|
+
operator: {
|
|
343
|
+
type: "string",
|
|
344
|
+
enum: ["eq", "neq", "gt", "gte", "lt", "lte", "in", "contains"],
|
|
345
|
+
},
|
|
346
|
+
value: { description: "Filter value — type depends on the column" },
|
|
347
|
+
},
|
|
348
|
+
required: ["column", "operator", "value"],
|
|
349
|
+
},
|
|
350
|
+
handler: async (params, ctx) => {
|
|
351
|
+
const filters = [...ctx.state.filters, params];
|
|
352
|
+
ctx.updateState({ filters });
|
|
353
|
+
return { success: true, data: { filterCount: filters.length } };
|
|
354
|
+
},
|
|
355
|
+
},
|
|
356
|
+
{
|
|
357
|
+
id: "render_chart",
|
|
358
|
+
description:
|
|
359
|
+
"Render the configured chart. This is async — the chart is generated server-side.",
|
|
360
|
+
async: true,
|
|
361
|
+
estimatedDuration: "5-20s",
|
|
362
|
+
input: {
|
|
363
|
+
type: "object",
|
|
364
|
+
properties: {
|
|
365
|
+
quality: {
|
|
366
|
+
type: "string",
|
|
367
|
+
enum: ["draft", "production"],
|
|
368
|
+
description: "Render quality",
|
|
369
|
+
},
|
|
370
|
+
},
|
|
371
|
+
},
|
|
372
|
+
polling: {
|
|
373
|
+
intervalMs: 2000,
|
|
374
|
+
maxDurationMs: 60000,
|
|
375
|
+
resolve: async (dispatchResult, ctx) => {
|
|
376
|
+
const result = dispatchResult as { jobId: string };
|
|
377
|
+
const status = await ctx.api.getRenderStatus(result.jobId);
|
|
378
|
+
if (status.state === "complete") {
|
|
379
|
+
return {
|
|
380
|
+
done: true,
|
|
381
|
+
data: {
|
|
382
|
+
status: "complete",
|
|
383
|
+
imageUrl: status.imageUrl,
|
|
384
|
+
renderTimeMs: status.durationMs,
|
|
385
|
+
},
|
|
386
|
+
};
|
|
387
|
+
}
|
|
388
|
+
if (status.state === "failed") {
|
|
389
|
+
return {
|
|
390
|
+
done: true,
|
|
391
|
+
data: { status: "failed", error: status.error },
|
|
392
|
+
};
|
|
393
|
+
}
|
|
394
|
+
return {
|
|
395
|
+
done: false,
|
|
396
|
+
data: { status: "rendering", progress: status.progress },
|
|
397
|
+
};
|
|
398
|
+
},
|
|
399
|
+
},
|
|
400
|
+
handler: async (params, ctx) => {
|
|
401
|
+
const job = await ctx.api.startRender({
|
|
402
|
+
dataset: ctx.state.selectedDataset.id,
|
|
403
|
+
config: ctx.state.chartConfig,
|
|
404
|
+
filters: ctx.state.filters,
|
|
405
|
+
quality: (params.quality as string) ?? "draft",
|
|
406
|
+
});
|
|
407
|
+
return { success: true, data: { jobId: job.id } };
|
|
408
|
+
},
|
|
409
|
+
},
|
|
410
|
+
{
|
|
411
|
+
id: "export_chart",
|
|
412
|
+
description: "Export the rendered chart as PNG, SVG, or PDF",
|
|
413
|
+
input: {
|
|
414
|
+
type: "object",
|
|
415
|
+
properties: {
|
|
416
|
+
format: { type: "string", enum: ["png", "svg", "pdf"] },
|
|
417
|
+
width: {
|
|
418
|
+
type: "number",
|
|
419
|
+
description: "Export width in pixels (default 1200)",
|
|
420
|
+
},
|
|
421
|
+
height: {
|
|
422
|
+
type: "number",
|
|
423
|
+
description: "Export height in pixels (default 800)",
|
|
424
|
+
},
|
|
425
|
+
},
|
|
426
|
+
required: ["format"],
|
|
427
|
+
},
|
|
428
|
+
preconditions: "A chart must be rendered first (render_chart)",
|
|
429
|
+
handler: async (params, ctx) => {
|
|
430
|
+
const url = await ctx.api.exportChart(
|
|
431
|
+
ctx.state.renderedChart.id,
|
|
432
|
+
params,
|
|
433
|
+
);
|
|
434
|
+
return {
|
|
435
|
+
success: true,
|
|
436
|
+
data: { downloadUrl: url, format: params.format },
|
|
437
|
+
};
|
|
438
|
+
},
|
|
439
|
+
},
|
|
440
|
+
],
|
|
441
|
+
|
|
442
|
+
observations: [
|
|
443
|
+
{
|
|
444
|
+
id: "available_datasets",
|
|
445
|
+
description:
|
|
446
|
+
"List of datasets available for visualization in the current project",
|
|
447
|
+
schema: {
|
|
448
|
+
type: "array",
|
|
449
|
+
items: {
|
|
450
|
+
type: "object",
|
|
451
|
+
properties: {
|
|
452
|
+
id: { type: "string" },
|
|
453
|
+
name: { type: "string" },
|
|
454
|
+
rowCount: { type: "number" },
|
|
455
|
+
columns: { type: "array", items: { type: "string" } },
|
|
456
|
+
},
|
|
457
|
+
},
|
|
458
|
+
},
|
|
459
|
+
updateFrequency: "on-change",
|
|
460
|
+
},
|
|
461
|
+
{
|
|
462
|
+
id: "wizard_state",
|
|
463
|
+
description:
|
|
464
|
+
"Current state of the DataViz wizard — which step the user is on and what is configured",
|
|
465
|
+
schema: {
|
|
466
|
+
type: "object",
|
|
467
|
+
properties: {
|
|
468
|
+
step: {
|
|
469
|
+
type: "string",
|
|
470
|
+
enum: ["select", "configure", "preview", "export"],
|
|
471
|
+
},
|
|
472
|
+
selectedDatasetId: { type: "string" },
|
|
473
|
+
chartConfig: { type: "object" },
|
|
474
|
+
filters: { type: "array" },
|
|
475
|
+
renderedChartUrl: { type: "string" },
|
|
476
|
+
},
|
|
477
|
+
},
|
|
478
|
+
updateFrequency: "on-change",
|
|
479
|
+
},
|
|
480
|
+
],
|
|
481
|
+
});
|
|
482
|
+
```
|
|
483
|
+
|
|
484
|
+
### Using the Surface in React
|
|
485
|
+
|
|
486
|
+
```tsx
|
|
487
|
+
// DataVizPage.tsx
|
|
488
|
+
import { useSurface, useObservation } from "oui-spec/react";
|
|
489
|
+
import { createWebSocketTransport } from "oui-spec/transport";
|
|
490
|
+
import { datavizSurface } from "./dataviz.surface";
|
|
491
|
+
|
|
492
|
+
export function DataVizPage() {
|
|
493
|
+
const { state, updateState, api } = useDataVizWizard();
|
|
494
|
+
const transport = useTransport(); // your app's transport setup
|
|
495
|
+
|
|
496
|
+
const { handleActionRequest } = useSurface({
|
|
497
|
+
surface: datavizSurface,
|
|
498
|
+
context: { state, updateState, api },
|
|
499
|
+
send: transport.pushObservation,
|
|
500
|
+
active: true,
|
|
501
|
+
});
|
|
502
|
+
|
|
503
|
+
// Push observations whenever relevant state changes
|
|
504
|
+
useObservation(
|
|
505
|
+
datavizSurface.id,
|
|
506
|
+
"wizard_state",
|
|
507
|
+
{
|
|
508
|
+
step: state.step,
|
|
509
|
+
selectedDatasetId: state.selectedDataset?.id,
|
|
510
|
+
chartConfig: state.chartConfig,
|
|
511
|
+
filters: state.filters,
|
|
512
|
+
renderedChartUrl: state.renderedChart?.imageUrl,
|
|
513
|
+
},
|
|
514
|
+
transport.pushObservation,
|
|
515
|
+
);
|
|
516
|
+
|
|
517
|
+
useObservation(
|
|
518
|
+
datavizSurface.id,
|
|
519
|
+
"available_datasets",
|
|
520
|
+
state.datasets,
|
|
521
|
+
transport.pushObservation,
|
|
522
|
+
);
|
|
523
|
+
|
|
524
|
+
// Wire transport to surface
|
|
525
|
+
useEffect(() => {
|
|
526
|
+
return transport.onAction((surfaceId, actionId, params) => {
|
|
527
|
+
handleActionRequest({
|
|
528
|
+
surfaceId,
|
|
529
|
+
actionId,
|
|
530
|
+
params,
|
|
531
|
+
requestId: "",
|
|
532
|
+
timestamp: Date.now(),
|
|
533
|
+
});
|
|
534
|
+
});
|
|
535
|
+
}, [handleActionRequest, transport]);
|
|
536
|
+
|
|
537
|
+
return <DataVizWizardUI state={state} />;
|
|
538
|
+
}
|
|
539
|
+
```
|
|
540
|
+
|
|
541
|
+
### What the Agent Sees
|
|
542
|
+
|
|
543
|
+
The agent runtime receives the manifest and exposes these tools to the LLM:
|
|
544
|
+
|
|
545
|
+
```
|
|
546
|
+
Tools available:
|
|
547
|
+
- select_dataset(datasetId: string) — Select a dataset to visualize
|
|
548
|
+
- configure_chart(chartType, xAxis, yAxis, ...) — Configure chart type and axes
|
|
549
|
+
- apply_filter(column, operator, value) — Add a data filter
|
|
550
|
+
- render_chart(quality?) — Render the chart (async, 5-20s)
|
|
551
|
+
- export_chart(format, width?, height?) — Export as PNG/SVG/PDF
|
|
552
|
+
|
|
553
|
+
Observations:
|
|
554
|
+
- available_datasets — [{id, name, rowCount, columns}]
|
|
555
|
+
- wizard_state — {step, selectedDatasetId, chartConfig, ...}
|
|
556
|
+
```
|
|
557
|
+
|
|
558
|
+
The agent can orchestrate the full wizard: select data → configure chart → filter → render → export. All through typed, validated function calls. No screenshots. No DOM selectors.
|
|
559
|
+
|
|
560
|
+
---
|
|
561
|
+
|
|
562
|
+
## Subpath Imports
|
|
563
|
+
|
|
564
|
+
The `oui-spec` package exposes subpath exports for granular imports:
|
|
565
|
+
|
|
566
|
+
```typescript
|
|
567
|
+
import { defineSurface } from "oui-spec/core";
|
|
568
|
+
import { useSurface } from "oui-spec/react";
|
|
569
|
+
import { createWebSocketTransport } from "oui-spec/transport";
|
|
570
|
+
import type { OUISurface, OUIAction } from "oui-spec/spec";
|
|
571
|
+
|
|
572
|
+
// Or import everything from the root
|
|
573
|
+
import { defineSurface, useSurface, createWebSocketTransport } from "oui-spec";
|
|
574
|
+
```
|
|
575
|
+
|
|
576
|
+
| Subpath | Description |
|
|
577
|
+
| -------------------- | ---------------------------------------------------------------------------- |
|
|
578
|
+
| `oui-spec/spec` | TypeScript types + JSON Schema for the OUI specification. Zero runtime deps. |
|
|
579
|
+
| `oui-spec/core` | `defineSurface()`, manifest extraction, action execution, polling config. |
|
|
580
|
+
| `oui-spec/react` | `useSurface()` hook, `useObservation()` helper. React bindings. |
|
|
581
|
+
| `oui-spec/transport` | Two-channel transport layer. WebSocket (Socket.IO), direct (in-memory). |
|
|
582
|
+
|
|
583
|
+
---
|
|
584
|
+
|
|
585
|
+
## How OUI Relates to OpenAPI
|
|
586
|
+
|
|
587
|
+
OUI and OpenAPI are **complementary**, not competing:
|
|
588
|
+
|
|
589
|
+
| | OpenAPI | OUI |
|
|
590
|
+
| -------------------- | --------------------------- | ------------------------------ |
|
|
591
|
+
| **Describes** | HTTP endpoints | UI capabilities |
|
|
592
|
+
| **Operates on** | Network requests | Application state |
|
|
593
|
+
| **Agent uses it to** | Call backend APIs | Control frontend UIs |
|
|
594
|
+
| **State awareness** | None (stateless HTTP) | Observations push live state |
|
|
595
|
+
| **Async model** | Webhooks / polling (manual) | Built-in polling runtime |
|
|
596
|
+
| **Schema format** | JSON Schema | JSON Schema |
|
|
597
|
+
| **Transport** | HTTP | WebSocket, postMessage, direct |
|
|
598
|
+
|
|
599
|
+
An application can expose **both**: an OpenAPI spec for its REST API and an OUI manifest for its UI. An agent that needs to "call the API" uses OpenAPI. An agent that needs to "use the app" uses OUI.
|
|
600
|
+
|
|
601
|
+
They even share JSON Schema for parameter validation — a deliberate design choice so the ecosystem tooling (validators, code generators, documentation) works with both.
|
|
602
|
+
|
|
603
|
+
---
|
|
604
|
+
|
|
605
|
+
## Comparison Table
|
|
606
|
+
|
|
607
|
+
| | Vision / Screenshot | DOM Scraping | MCP Tools | **OUI** |
|
|
608
|
+
| ---------------------- | ------------------------------ | ------------------------- | ------------------------- | --------------------------------------- |
|
|
609
|
+
| **Discovery** | Infer from pixels | Parse HTML structure | Read tool manifests | Read surface manifest |
|
|
610
|
+
| **Actions** | Click (x, y) | `querySelector().click()` | `call_tool(name, params)` | `dispatch(surfaceId, actionId, params)` |
|
|
611
|
+
| **Feedback** | Screenshot & compare | Check DOM changed | Tool returns response | Observation pushed async |
|
|
612
|
+
| **UI awareness** | Full (but expensive) | Structural only | None | Declared observations |
|
|
613
|
+
| **Reliability** | Breaks on style/layout changes | Breaks on DOM changes | Stable (typed) | Stable (typed) |
|
|
614
|
+
| **Cost per action** | ~$0.01-0.05 (vision model) | Near zero | Near zero | Near zero |
|
|
615
|
+
| **Latency** | 2-10s (screenshot + inference) | <100ms | <100ms | <100ms |
|
|
616
|
+
| **Async operations** | Poll screenshots | Poll DOM | Not built-in | Declarative polling runtime |
|
|
617
|
+
| **Consent model** | Everything visible | Everything in DOM | Server declares tools | App declares surface |
|
|
618
|
+
| **Framework coupling** | None | Tight (HTML-specific) | None | None (spec is universal) |
|
|
619
|
+
| **Complex workflows** | Dozens of screenshots | Dozens of selectors | Multiple tool calls | Multiple dispatches + observations |
|
|
620
|
+
|
|
621
|
+
---
|
|
622
|
+
|
|
623
|
+
## Principles
|
|
624
|
+
|
|
625
|
+
1. **App-declared, not inferred.** The application explicitly states what it supports. Agents don't guess from pixels or DOM.
|
|
626
|
+
2. **Semantic, not structural.** Actions are named operations (`render_chart`), not DOM paths (`click #btn-render`).
|
|
627
|
+
3. **Typed end-to-end.** Inputs and outputs use JSON Schema. The agent knows exactly what to send and expect.
|
|
628
|
+
4. **Fully async.** Dispatch is fire-and-forget. Results arrive as observations. No blocking, no timeouts.
|
|
629
|
+
5. **Two channels, one direction each.** Dispatch goes down. Observations go up. Never mixed.
|
|
630
|
+
6. **Framework-agnostic.** The spec works with React, Vue, Svelte, vanilla JS, native apps — anything that can handle a function call and emit an event.
|
|
631
|
+
7. **Observable.** The agent reads relevant state without screenshots. The app pushes state changes when they happen.
|
|
632
|
+
8. **Consent-based.** The app controls what's exposed. Users and developers decide what agents can do.
|
|
633
|
+
9. **Composable.** Multiple surfaces coexist. A complex app exposes many surfaces (one per feature/page).
|
|
634
|
+
10. **Complementary.** OUI doesn't replace OpenAPI, MCP, or any other standard. It fills the "UI control" gap they leave open.
|
|
635
|
+
|
|
636
|
+
---
|
|
637
|
+
|
|
638
|
+
## Status
|
|
639
|
+
|
|
640
|
+
🚧 **Active development.** The specification (v0.1) and core packages are functional and in use within the Closure AI Studio platform. The public API is stabilizing but may change before 1.0.
|
|
641
|
+
|
|
642
|
+
### Roadmap
|
|
643
|
+
|
|
644
|
+
- [x] Specification v0.1
|
|
645
|
+
- [x] `oui-spec/spec` — Types package
|
|
646
|
+
- [x] `oui-spec/core` — `defineSurface()` + manifest extraction
|
|
647
|
+
- [x] `oui-spec/react` — `useSurface()` hook + observation helpers
|
|
648
|
+
- [x] `oui-spec/transport` — WebSocket + Direct transports
|
|
649
|
+
- [ ] `oui-spec/devtools` — Surface inspector / debugger
|
|
650
|
+
- [ ] `oui-spec/vue` — Vue bindings
|
|
651
|
+
- [ ] `oui-spec/validator` — Runtime schema validation
|
|
652
|
+
- [ ] Specification v1.0
|
|
653
|
+
|
|
654
|
+
---
|
|
655
|
+
|
|
656
|
+
## Contributing
|
|
657
|
+
|
|
658
|
+
Contributions welcome. Please read the spec at [`spec/OUI-SPEC-v0.1.md`](./spec/OUI-SPEC-v0.1.md) before proposing changes to ensure alignment with the design principles.
|
|
659
|
+
|
|
660
|
+
```bash
|
|
661
|
+
# Clone and install
|
|
662
|
+
git clone https://github.com/wesreid/oui.git
|
|
663
|
+
cd oui
|
|
664
|
+
npm install
|
|
665
|
+
|
|
666
|
+
# Build
|
|
667
|
+
npm run build
|
|
668
|
+
|
|
669
|
+
# Run tests
|
|
670
|
+
npm test
|
|
671
|
+
|
|
672
|
+
# Typecheck
|
|
673
|
+
npm run typecheck
|
|
674
|
+
```
|
|
675
|
+
|
|
676
|
+
---
|
|
677
|
+
|
|
678
|
+
## License
|
|
679
|
+
|
|
680
|
+
MIT
|