oui-spec 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (50) hide show
  1. package/LICENSE +21 -0
  2. package/README.md +680 -0
  3. package/dist/chunk-2ON5IEXB.cjs +167 -0
  4. package/dist/chunk-2ON5IEXB.cjs.map +1 -0
  5. package/dist/chunk-3DCIL4FV.cjs +14 -0
  6. package/dist/chunk-3DCIL4FV.cjs.map +1 -0
  7. package/dist/chunk-7EXSMPGR.js +207 -0
  8. package/dist/chunk-7EXSMPGR.js.map +1 -0
  9. package/dist/chunk-CVUUSPF4.js +12 -0
  10. package/dist/chunk-CVUUSPF4.js.map +1 -0
  11. package/dist/chunk-JO5LGOB6.js +88 -0
  12. package/dist/chunk-JO5LGOB6.js.map +1 -0
  13. package/dist/chunk-LEXIU2MH.cjs +90 -0
  14. package/dist/chunk-LEXIU2MH.cjs.map +1 -0
  15. package/dist/chunk-UIGRLZOX.js +164 -0
  16. package/dist/chunk-UIGRLZOX.js.map +1 -0
  17. package/dist/chunk-WZXUKJ7E.cjs +210 -0
  18. package/dist/chunk-WZXUKJ7E.cjs.map +1 -0
  19. package/dist/core/index.cjs +17 -0
  20. package/dist/core/index.cjs.map +1 -0
  21. package/dist/core/index.d.cts +120 -0
  22. package/dist/core/index.d.ts +120 -0
  23. package/dist/core/index.js +4 -0
  24. package/dist/core/index.js.map +1 -0
  25. package/dist/index.cjs +35 -0
  26. package/dist/index.cjs.map +1 -0
  27. package/dist/index.d.cts +4 -0
  28. package/dist/index.d.ts +4 -0
  29. package/dist/index.js +6 -0
  30. package/dist/index.js.map +1 -0
  31. package/dist/react/index.cjs +16 -0
  32. package/dist/react/index.cjs.map +1 -0
  33. package/dist/react/index.d.cts +62 -0
  34. package/dist/react/index.d.ts +62 -0
  35. package/dist/react/index.js +3 -0
  36. package/dist/react/index.js.map +1 -0
  37. package/dist/spec/index.cjs +12 -0
  38. package/dist/spec/index.cjs.map +1 -0
  39. package/dist/spec/index.d.cts +249 -0
  40. package/dist/spec/index.d.ts +249 -0
  41. package/dist/spec/index.js +3 -0
  42. package/dist/spec/index.js.map +1 -0
  43. package/dist/transport/index.cjs +16 -0
  44. package/dist/transport/index.cjs.map +1 -0
  45. package/dist/transport/index.d.cts +91 -0
  46. package/dist/transport/index.d.ts +91 -0
  47. package/dist/transport/index.js +3 -0
  48. package/dist/transport/index.js.map +1 -0
  49. package/package.json +109 -0
  50. package/spec/OUI-SPEC-v0.1.md +1323 -0
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Closure Studio
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md ADDED
@@ -0,0 +1,680 @@
1
+ # OUI - The OpenUI Specification
2
+
3
+ **The machine-readable contract for agent-controllable user interfaces.**
4
+
5
+ > _"Oui"_ — French for "yes." As in: yes, an AI agent can control this UI.
6
+
7
+ <!-- Badges -->
8
+
9
+ [![npm](https://img.shields.io/npm/v/oui-spec)](https://www.npmjs.com/package/oui-spec)
10
+ [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
11
+ [![TypeScript](https://img.shields.io/badge/TypeScript-5.x-blue)](https://www.typescriptlang.org/)
12
+ [![CI](https://github.com/wesreid/oui/actions/workflows/ci.yml/badge.svg)](https://github.com/wesreid/oui/actions/workflows/ci.yml)
13
+
14
+ ---
15
+
16
+ ## The Problem
17
+
18
+ AI agents that control UIs today rely on fragile, expensive hacks:
19
+
20
+ - **Computer vision** — screenshot the screen, feed it to a vision model, infer what's clickable, guess coordinates, click, screenshot again to see if it worked. Slow, expensive, unreliable.
21
+ - **DOM scraping** — find elements by CSS selector, simulate clicks and keystrokes. Breaks on every layout change. Requires intimate knowledge of the app's internal structure.
22
+ - **MCP tool servers** — powerful for backend operations, but have no concept of _UI state_. The agent can call APIs but can't observe what the user is seeing or coordinate with the frontend.
23
+
24
+ All of these share a fundamental flaw: **the agent is guessing what the application can do by examining its implementation details.** It's like reading a restaurant menu by peering into the kitchen.
25
+
26
+ ## The Solution
27
+
28
+ OUI inverts the model. **The application declares a typed semantic surface** — what it can do, what state is observable, and what parameters each operation accepts. The agent reads this manifest and operates through typed function calls. No pixels. No DOM. No guessing.
29
+
30
+ ```
31
+ OpenAPI + HTTP Server = any client can consume the API
32
+ OUI + UI App = any agent can control the UI
33
+ ```
34
+
35
+ ```mermaid
36
+ flowchart TD
37
+ Agent[Agent Runtime / LLM]
38
+ Manifest[OUI Manifest]
39
+ Dispatch[Dispatch Channel]
40
+ Observe[Observation Channel]
41
+ Surface[UI Surface]
42
+ App[Application State]
43
+
44
+ Agent -->|reads| Manifest
45
+ Agent -->|fire-and-forget action| Dispatch
46
+ Dispatch -->|WebSocket| Surface
47
+ Surface -->|executes handler| App
48
+ App -->|state changes| Surface
49
+ Surface -->|pushes observations| Observe
50
+ Observe -->|WebSocket| Agent
51
+ ```
52
+
53
+ ### Fully Async. Fully Stateless. No Waiting.
54
+
55
+ OUI uses **two one-way channels** — not request/response. This is the critical architectural decision:
56
+
57
+ | Channel | Direction | Purpose |
58
+ | --------------- | --------------- | ------------------------------------ |
59
+ | **Dispatch** | Server → Client | Fire action instructions at the UI |
60
+ | **Observation** | Client → Server | Push state updates back to the agent |
61
+
62
+ The agent runtime (e.g., a Lambda function) dispatches an action and **terminates immediately**. It does not wait for a response. The browser executes the action, starts any async work, and pushes observation updates over the observation channel as results arrive. The agent picks up those observations on its next invocation.
63
+
64
+ This means:
65
+
66
+ - **No Lambda waiting** — no 30-second timeouts holding a connection open
67
+ - **No request/response correlation** — no correlation IDs, no pending promises
68
+ - **No WebSocket held open by the server** — the server writes and disconnects
69
+ - **Long-running operations work naturally** — the UI polls or subscribes and pushes status updates over time
70
+
71
+ ---
72
+
73
+ ## Quick Start
74
+
75
+ The simplest possible OUI surface — a counter the agent can increment:
76
+
77
+ ```typescript
78
+ // counter.surface.ts
79
+ import { defineSurface } from "oui-spec/core";
80
+
81
+ export const counterSurface = defineSurface({
82
+ id: "counter",
83
+ name: "Counter",
84
+ description: "A simple counter that can be incremented or reset",
85
+ actions: [
86
+ {
87
+ id: "increment",
88
+ description: "Increment the counter by a given amount",
89
+ input: {
90
+ type: "object",
91
+ properties: {
92
+ amount: { type: "number", description: "Amount to add (default 1)" },
93
+ },
94
+ },
95
+ handler: async (params, ctx) => {
96
+ const amount = (params.amount as number) ?? 1;
97
+ ctx.setCount(ctx.count + amount);
98
+ return { success: true, data: { newValue: ctx.count + amount } };
99
+ },
100
+ },
101
+ {
102
+ id: "reset",
103
+ description: "Reset the counter to zero",
104
+ input: { type: "object", properties: {} },
105
+ handler: async (_params, ctx) => {
106
+ ctx.setCount(0);
107
+ return { success: true, data: { newValue: 0 } };
108
+ },
109
+ },
110
+ ],
111
+ observations: [
112
+ {
113
+ id: "current_value",
114
+ description: "The current counter value",
115
+ schema: { type: "object", properties: { value: { type: "number" } } },
116
+ },
117
+ ],
118
+ });
119
+ ```
120
+
121
+ ```tsx
122
+ // CounterPage.tsx
123
+ import { useSurface, useObservation } from "oui-spec/react";
124
+ import { counterSurface } from "./counter.surface";
125
+
126
+ function CounterPage({ transport }) {
127
+ const [count, setCount] = useState(0);
128
+
129
+ const { handleActionRequest } = useSurface({
130
+ surface: counterSurface,
131
+ context: { count, setCount },
132
+ send: transport.send,
133
+ });
134
+
135
+ // Push observation whenever count changes
136
+ useObservation(
137
+ counterSurface.id,
138
+ "current_value",
139
+ { value: count },
140
+ transport.send,
141
+ );
142
+
143
+ return <div>Count: {count}</div>;
144
+ }
145
+ ```
146
+
147
+ That's it. The agent now sees a `counter` surface with two tools (`increment`, `reset`) and one live observation (`current_value`).
148
+
149
+ ---
150
+
151
+ ## Architecture
152
+
153
+ ### Two One-Way Channels
154
+
155
+ ```mermaid
156
+ sequenceDiagram
157
+ participant Lambda as Agent Lambda
158
+ participant WS as WebSocket Server
159
+ participant Browser as Browser / UI
160
+
161
+ Note over Lambda: Agent decides to act
162
+ Lambda->>WS: dispatch(surfaceId, actionId, params)
163
+ Note over Lambda: Lambda terminates immediately
164
+
165
+ WS->>Browser: oui:dispatch event
166
+ Note over Browser: Handler executes
167
+ Note over Browser: Async work begins
168
+
169
+ loop Polling / Subscribe
170
+ Browser->>Browser: Check job status
171
+ Browser->>WS: oui:observation (interim update)
172
+ end
173
+
174
+ Browser->>WS: oui:observation (final result)
175
+ Note over Lambda: Next invocation reads observations
176
+ ```
177
+
178
+ The dispatch channel carries action instructions **downward** (server → client). The observation channel carries state updates **upward** (client → server). They never mix. There is no concept of "waiting for a response."
179
+
180
+ ### Why Not Request/Response?
181
+
182
+ In a typical agent architecture:
183
+
184
+ 1. The agent runs in a Lambda (or similar serverless function)
185
+ 2. The Lambda has a 30-second timeout
186
+ 3. The UI operation might take 2 minutes (file upload, ML inference, user confirmation)
187
+
188
+ Request/response forces the Lambda to hold a connection open until the operation completes. That either times out or requires expensive long-running compute.
189
+
190
+ OUI's two-channel model means the Lambda does O(1) work — write a message to the dispatch channel and exit. The browser handles the rest on its own timeline and pushes results as observations. The agent reads those observations whenever it runs next.
191
+
192
+ ---
193
+
194
+ ## Polling & Async Operations
195
+
196
+ Many UI operations are not instant. A DataViz chart generation might take 30 seconds. A file export might need to wait for server-side rendering. OUI handles this with **declarative polling configuration**:
197
+
198
+ ```typescript
199
+ {
200
+ id: 'generate_chart',
201
+ description: 'Generate a visualization from the configured dataset',
202
+ async: true,
203
+ estimatedDuration: '10-30s',
204
+ input: {
205
+ type: 'object',
206
+ properties: {
207
+ chartType: { type: 'string', enum: ['bar', 'line', 'scatter', 'pie'] },
208
+ dataset: { type: 'string', description: 'Dataset ID to visualize' },
209
+ },
210
+ required: ['chartType', 'dataset'],
211
+ },
212
+ polling: {
213
+ intervalMs: 2000,
214
+ maxAttempts: 30,
215
+ resolve: async (dispatchResult, ctx) => {
216
+ const job = await ctx.api.getJobStatus(dispatchResult.jobId);
217
+ if (job.status === 'complete') {
218
+ return { done: true, data: { status: 'complete', chartUrl: job.outputUrl } };
219
+ }
220
+ return { done: false, data: { status: job.status, progress: job.progress } };
221
+ },
222
+ },
223
+ handler: async (params, ctx) => {
224
+ const job = await ctx.api.startChartGeneration(params);
225
+ return { success: true, data: { jobId: job.id } };
226
+ },
227
+ }
228
+ ```
229
+
230
+ **What happens at runtime:**
231
+
232
+ 1. Agent dispatches `generate_chart` with params
233
+ 2. Handler fires, calls the API, returns `{ jobId: 'xyz' }` immediately
234
+ 3. OUI runtime starts polling every 2 seconds using the `resolve` function
235
+ 4. Each poll pushes an observation: `{ status: 'processing', progress: 45, interim: true }`
236
+ 5. When `resolve` returns `{ done: true }`, polling stops and a final observation is pushed
237
+ 6. The agent sees the final `{ status: 'complete', chartUrl: '...' }` observation on its next invocation
238
+
239
+ ### Subscribe Mode (Alternative to Polling)
240
+
241
+ If you have realtime infrastructure (WebSocket events, SSE, etc.), you can use subscribe mode instead of polling:
242
+
243
+ ```typescript
244
+ polling: {
245
+ intervalMs: 0, // Not used in subscribe mode
246
+ subscribe: {
247
+ event: 'job:complete',
248
+ filter: { jobId: '$dispatchResult.jobId' },
249
+ },
250
+ },
251
+ ```
252
+
253
+ The `$dispatchResult.` prefix tells the runtime to resolve the filter value from the handler's return data. The transport layer listens for the named event and pushes the observation when it arrives.
254
+
255
+ ---
256
+
257
+ ## Real-World Example: DataViz Surface
258
+
259
+ A complete DataViz wizard surface — the kind of thing you'd build for an AI-assisted analytics tool:
260
+
261
+ ```typescript
262
+ // dataviz.surface.ts
263
+ import { defineSurface } from "oui-spec/core";
264
+ import type { DataVizContext } from "./types";
265
+
266
+ export const datavizSurface = defineSurface<DataVizContext>({
267
+ id: "dataviz-wizard",
268
+ name: "Data Visualization Wizard",
269
+ description:
270
+ "AI-controllable data visualization builder. Supports dataset selection, chart configuration, filtering, and export.",
271
+ version: "1.0.0",
272
+
273
+ activation: {
274
+ routes: ["/studio/dataviz", "/studio/dataviz/*"],
275
+ condition: "User has an active project with at least one dataset",
276
+ },
277
+
278
+ actions: [
279
+ {
280
+ id: "select_dataset",
281
+ description:
282
+ "Select a dataset to visualize. Must be called before configuring a chart.",
283
+ input: {
284
+ type: "object",
285
+ properties: {
286
+ datasetId: {
287
+ type: "string",
288
+ description: "The dataset ID from available_datasets observation",
289
+ },
290
+ },
291
+ required: ["datasetId"],
292
+ },
293
+ handler: async (params, ctx) => {
294
+ const dataset = await ctx.api.loadDataset(params.datasetId as string);
295
+ ctx.updateState({ selectedDataset: dataset, step: "configure" });
296
+ return {
297
+ success: true,
298
+ data: { columns: dataset.columns, rowCount: dataset.rowCount },
299
+ };
300
+ },
301
+ },
302
+ {
303
+ id: "configure_chart",
304
+ description: "Configure the chart type, axes, and visual options",
305
+ input: {
306
+ type: "object",
307
+ properties: {
308
+ chartType: {
309
+ type: "string",
310
+ enum: ["bar", "line", "scatter", "pie", "heatmap", "treemap"],
311
+ },
312
+ xAxis: { type: "string", description: "Column name for x-axis" },
313
+ yAxis: { type: "string", description: "Column name for y-axis" },
314
+ colorBy: {
315
+ type: "string",
316
+ description: "Column to use for color encoding (optional)",
317
+ },
318
+ aggregation: {
319
+ type: "string",
320
+ enum: ["sum", "avg", "count", "min", "max"],
321
+ description: "Aggregation function for y-axis",
322
+ },
323
+ },
324
+ required: ["chartType", "xAxis", "yAxis"],
325
+ },
326
+ preconditions: "A dataset must be selected first (select_dataset)",
327
+ handler: async (params, ctx) => {
328
+ ctx.updateState({ chartConfig: params, step: "preview" });
329
+ return {
330
+ success: true,
331
+ data: { configured: true, chartType: params.chartType },
332
+ };
333
+ },
334
+ },
335
+ {
336
+ id: "apply_filter",
337
+ description: "Add a filter to narrow the visualized data",
338
+ input: {
339
+ type: "object",
340
+ properties: {
341
+ column: { type: "string" },
342
+ operator: {
343
+ type: "string",
344
+ enum: ["eq", "neq", "gt", "gte", "lt", "lte", "in", "contains"],
345
+ },
346
+ value: { description: "Filter value — type depends on the column" },
347
+ },
348
+ required: ["column", "operator", "value"],
349
+ },
350
+ handler: async (params, ctx) => {
351
+ const filters = [...ctx.state.filters, params];
352
+ ctx.updateState({ filters });
353
+ return { success: true, data: { filterCount: filters.length } };
354
+ },
355
+ },
356
+ {
357
+ id: "render_chart",
358
+ description:
359
+ "Render the configured chart. This is async — the chart is generated server-side.",
360
+ async: true,
361
+ estimatedDuration: "5-20s",
362
+ input: {
363
+ type: "object",
364
+ properties: {
365
+ quality: {
366
+ type: "string",
367
+ enum: ["draft", "production"],
368
+ description: "Render quality",
369
+ },
370
+ },
371
+ },
372
+ polling: {
373
+ intervalMs: 2000,
374
+ maxDurationMs: 60000,
375
+ resolve: async (dispatchResult, ctx) => {
376
+ const result = dispatchResult as { jobId: string };
377
+ const status = await ctx.api.getRenderStatus(result.jobId);
378
+ if (status.state === "complete") {
379
+ return {
380
+ done: true,
381
+ data: {
382
+ status: "complete",
383
+ imageUrl: status.imageUrl,
384
+ renderTimeMs: status.durationMs,
385
+ },
386
+ };
387
+ }
388
+ if (status.state === "failed") {
389
+ return {
390
+ done: true,
391
+ data: { status: "failed", error: status.error },
392
+ };
393
+ }
394
+ return {
395
+ done: false,
396
+ data: { status: "rendering", progress: status.progress },
397
+ };
398
+ },
399
+ },
400
+ handler: async (params, ctx) => {
401
+ const job = await ctx.api.startRender({
402
+ dataset: ctx.state.selectedDataset.id,
403
+ config: ctx.state.chartConfig,
404
+ filters: ctx.state.filters,
405
+ quality: (params.quality as string) ?? "draft",
406
+ });
407
+ return { success: true, data: { jobId: job.id } };
408
+ },
409
+ },
410
+ {
411
+ id: "export_chart",
412
+ description: "Export the rendered chart as PNG, SVG, or PDF",
413
+ input: {
414
+ type: "object",
415
+ properties: {
416
+ format: { type: "string", enum: ["png", "svg", "pdf"] },
417
+ width: {
418
+ type: "number",
419
+ description: "Export width in pixels (default 1200)",
420
+ },
421
+ height: {
422
+ type: "number",
423
+ description: "Export height in pixels (default 800)",
424
+ },
425
+ },
426
+ required: ["format"],
427
+ },
428
+ preconditions: "A chart must be rendered first (render_chart)",
429
+ handler: async (params, ctx) => {
430
+ const url = await ctx.api.exportChart(
431
+ ctx.state.renderedChart.id,
432
+ params,
433
+ );
434
+ return {
435
+ success: true,
436
+ data: { downloadUrl: url, format: params.format },
437
+ };
438
+ },
439
+ },
440
+ ],
441
+
442
+ observations: [
443
+ {
444
+ id: "available_datasets",
445
+ description:
446
+ "List of datasets available for visualization in the current project",
447
+ schema: {
448
+ type: "array",
449
+ items: {
450
+ type: "object",
451
+ properties: {
452
+ id: { type: "string" },
453
+ name: { type: "string" },
454
+ rowCount: { type: "number" },
455
+ columns: { type: "array", items: { type: "string" } },
456
+ },
457
+ },
458
+ },
459
+ updateFrequency: "on-change",
460
+ },
461
+ {
462
+ id: "wizard_state",
463
+ description:
464
+ "Current state of the DataViz wizard — which step the user is on and what is configured",
465
+ schema: {
466
+ type: "object",
467
+ properties: {
468
+ step: {
469
+ type: "string",
470
+ enum: ["select", "configure", "preview", "export"],
471
+ },
472
+ selectedDatasetId: { type: "string" },
473
+ chartConfig: { type: "object" },
474
+ filters: { type: "array" },
475
+ renderedChartUrl: { type: "string" },
476
+ },
477
+ },
478
+ updateFrequency: "on-change",
479
+ },
480
+ ],
481
+ });
482
+ ```
483
+
484
+ ### Using the Surface in React
485
+
486
+ ```tsx
487
+ // DataVizPage.tsx
488
+ import { useSurface, useObservation } from "oui-spec/react";
489
+ import { createWebSocketTransport } from "oui-spec/transport";
490
+ import { datavizSurface } from "./dataviz.surface";
491
+
492
+ export function DataVizPage() {
493
+ const { state, updateState, api } = useDataVizWizard();
494
+ const transport = useTransport(); // your app's transport setup
495
+
496
+ const { handleActionRequest } = useSurface({
497
+ surface: datavizSurface,
498
+ context: { state, updateState, api },
499
+ send: transport.pushObservation,
500
+ active: true,
501
+ });
502
+
503
+ // Push observations whenever relevant state changes
504
+ useObservation(
505
+ datavizSurface.id,
506
+ "wizard_state",
507
+ {
508
+ step: state.step,
509
+ selectedDatasetId: state.selectedDataset?.id,
510
+ chartConfig: state.chartConfig,
511
+ filters: state.filters,
512
+ renderedChartUrl: state.renderedChart?.imageUrl,
513
+ },
514
+ transport.pushObservation,
515
+ );
516
+
517
+ useObservation(
518
+ datavizSurface.id,
519
+ "available_datasets",
520
+ state.datasets,
521
+ transport.pushObservation,
522
+ );
523
+
524
+ // Wire transport to surface
525
+ useEffect(() => {
526
+ return transport.onAction((surfaceId, actionId, params) => {
527
+ handleActionRequest({
528
+ surfaceId,
529
+ actionId,
530
+ params,
531
+ requestId: "",
532
+ timestamp: Date.now(),
533
+ });
534
+ });
535
+ }, [handleActionRequest, transport]);
536
+
537
+ return <DataVizWizardUI state={state} />;
538
+ }
539
+ ```
540
+
541
+ ### What the Agent Sees
542
+
543
+ The agent runtime receives the manifest and exposes these tools to the LLM:
544
+
545
+ ```
546
+ Tools available:
547
+ - select_dataset(datasetId: string) — Select a dataset to visualize
548
+ - configure_chart(chartType, xAxis, yAxis, ...) — Configure chart type and axes
549
+ - apply_filter(column, operator, value) — Add a data filter
550
+ - render_chart(quality?) — Render the chart (async, 5-20s)
551
+ - export_chart(format, width?, height?) — Export as PNG/SVG/PDF
552
+
553
+ Observations:
554
+ - available_datasets — [{id, name, rowCount, columns}]
555
+ - wizard_state — {step, selectedDatasetId, chartConfig, ...}
556
+ ```
557
+
558
+ The agent can orchestrate the full wizard: select data → configure chart → filter → render → export. All through typed, validated function calls. No screenshots. No DOM selectors.
559
+
560
+ ---
561
+
562
+ ## Subpath Imports
563
+
564
+ The `oui-spec` package exposes subpath exports for granular imports:
565
+
566
+ ```typescript
567
+ import { defineSurface } from "oui-spec/core";
568
+ import { useSurface } from "oui-spec/react";
569
+ import { createWebSocketTransport } from "oui-spec/transport";
570
+ import type { OUISurface, OUIAction } from "oui-spec/spec";
571
+
572
+ // Or import everything from the root
573
+ import { defineSurface, useSurface, createWebSocketTransport } from "oui-spec";
574
+ ```
575
+
576
+ | Subpath | Description |
577
+ | -------------------- | ---------------------------------------------------------------------------- |
578
+ | `oui-spec/spec` | TypeScript types + JSON Schema for the OUI specification. Zero runtime deps. |
579
+ | `oui-spec/core` | `defineSurface()`, manifest extraction, action execution, polling config. |
580
+ | `oui-spec/react` | `useSurface()` hook, `useObservation()` helper. React bindings. |
581
+ | `oui-spec/transport` | Two-channel transport layer. WebSocket (Socket.IO), direct (in-memory). |
582
+
583
+ ---
584
+
585
+ ## How OUI Relates to OpenAPI
586
+
587
+ OUI and OpenAPI are **complementary**, not competing:
588
+
589
+ | | OpenAPI | OUI |
590
+ | -------------------- | --------------------------- | ------------------------------ |
591
+ | **Describes** | HTTP endpoints | UI capabilities |
592
+ | **Operates on** | Network requests | Application state |
593
+ | **Agent uses it to** | Call backend APIs | Control frontend UIs |
594
+ | **State awareness** | None (stateless HTTP) | Observations push live state |
595
+ | **Async model** | Webhooks / polling (manual) | Built-in polling runtime |
596
+ | **Schema format** | JSON Schema | JSON Schema |
597
+ | **Transport** | HTTP | WebSocket, postMessage, direct |
598
+
599
+ An application can expose **both**: an OpenAPI spec for its REST API and an OUI manifest for its UI. An agent that needs to "call the API" uses OpenAPI. An agent that needs to "use the app" uses OUI.
600
+
601
+ They even share JSON Schema for parameter validation — a deliberate design choice so the ecosystem tooling (validators, code generators, documentation) works with both.
602
+
603
+ ---
604
+
605
+ ## Comparison Table
606
+
607
+ | | Vision / Screenshot | DOM Scraping | MCP Tools | **OUI** |
608
+ | ---------------------- | ------------------------------ | ------------------------- | ------------------------- | --------------------------------------- |
609
+ | **Discovery** | Infer from pixels | Parse HTML structure | Read tool manifests | Read surface manifest |
610
+ | **Actions** | Click (x, y) | `querySelector().click()` | `call_tool(name, params)` | `dispatch(surfaceId, actionId, params)` |
611
+ | **Feedback** | Screenshot & compare | Check DOM changed | Tool returns response | Observation pushed async |
612
+ | **UI awareness** | Full (but expensive) | Structural only | None | Declared observations |
613
+ | **Reliability** | Breaks on style/layout changes | Breaks on DOM changes | Stable (typed) | Stable (typed) |
614
+ | **Cost per action** | ~$0.01-0.05 (vision model) | Near zero | Near zero | Near zero |
615
+ | **Latency** | 2-10s (screenshot + inference) | <100ms | <100ms | <100ms |
616
+ | **Async operations** | Poll screenshots | Poll DOM | Not built-in | Declarative polling runtime |
617
+ | **Consent model** | Everything visible | Everything in DOM | Server declares tools | App declares surface |
618
+ | **Framework coupling** | None | Tight (HTML-specific) | None | None (spec is universal) |
619
+ | **Complex workflows** | Dozens of screenshots | Dozens of selectors | Multiple tool calls | Multiple dispatches + observations |
620
+
621
+ ---
622
+
623
+ ## Principles
624
+
625
+ 1. **App-declared, not inferred.** The application explicitly states what it supports. Agents don't guess from pixels or DOM.
626
+ 2. **Semantic, not structural.** Actions are named operations (`render_chart`), not DOM paths (`click #btn-render`).
627
+ 3. **Typed end-to-end.** Inputs and outputs use JSON Schema. The agent knows exactly what to send and expect.
628
+ 4. **Fully async.** Dispatch is fire-and-forget. Results arrive as observations. No blocking, no timeouts.
629
+ 5. **Two channels, one direction each.** Dispatch goes down. Observations go up. Never mixed.
630
+ 6. **Framework-agnostic.** The spec works with React, Vue, Svelte, vanilla JS, native apps — anything that can handle a function call and emit an event.
631
+ 7. **Observable.** The agent reads relevant state without screenshots. The app pushes state changes when they happen.
632
+ 8. **Consent-based.** The app controls what's exposed. Users and developers decide what agents can do.
633
+ 9. **Composable.** Multiple surfaces coexist. A complex app exposes many surfaces (one per feature/page).
634
+ 10. **Complementary.** OUI doesn't replace OpenAPI, MCP, or any other standard. It fills the "UI control" gap they leave open.
635
+
636
+ ---
637
+
638
+ ## Status
639
+
640
+ 🚧 **Active development.** The specification (v0.1) and core packages are functional and in use within the Closure AI Studio platform. The public API is stabilizing but may change before 1.0.
641
+
642
+ ### Roadmap
643
+
644
+ - [x] Specification v0.1
645
+ - [x] `oui-spec/spec` — Types package
646
+ - [x] `oui-spec/core` — `defineSurface()` + manifest extraction
647
+ - [x] `oui-spec/react` — `useSurface()` hook + observation helpers
648
+ - [x] `oui-spec/transport` — WebSocket + Direct transports
649
+ - [ ] `oui-spec/devtools` — Surface inspector / debugger
650
+ - [ ] `oui-spec/vue` — Vue bindings
651
+ - [ ] `oui-spec/validator` — Runtime schema validation
652
+ - [ ] Specification v1.0
653
+
654
+ ---
655
+
656
+ ## Contributing
657
+
658
+ Contributions welcome. Please read the spec at [`spec/OUI-SPEC-v0.1.md`](./spec/OUI-SPEC-v0.1.md) before proposing changes to ensure alignment with the design principles.
659
+
660
+ ```bash
661
+ # Clone and install
662
+ git clone https://github.com/wesreid/oui.git
663
+ cd oui
664
+ npm install
665
+
666
+ # Build
667
+ npm run build
668
+
669
+ # Run tests
670
+ npm test
671
+
672
+ # Typecheck
673
+ npm run typecheck
674
+ ```
675
+
676
+ ---
677
+
678
+ ## License
679
+
680
+ MIT