deepseek-harness-intelligent-ui 0.2.3 → 0.2.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +35 -1
- package/README.md +145 -35
- package/README.zh-CN.md +144 -34
- package/THIRD_PARTY_NOTICES.md +8 -0
- package/client/client.js +17 -17
- package/package.json +13 -6
- package/renderer/renderer.js +500 -0
- package/src/catalog/patch.js +283 -0
- package/src/guidance.js +6 -1
- package/src/plugin.js +6 -2
- package/src/portable/artifacts.js +87 -0
- package/src/tools/patch.js +139 -0
- package/src/tools/render.js +4 -2
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,39 @@ All notable changes to this project are documented here. The format follows
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [0.2.4]
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- `ui_patch`: change a UI shown earlier in the conversation by component id (`setProps`, `replaceNode`, `insertChild`, `removeNode`) instead of re-sending the whole tree ([#14](https://github.com/dreamfarer-space/deepseek-harness-intelligent-ui/issues/14)). `ui_render` results now report the UI's `uiId` and version. Patches are atomic, revalidated as a whole tree, version-checked (`baseVersion`; stale patches are refused with the current version), and limited to UIs of the same conversation. The Host rebuilds versions by replaying the session history, so it keeps no state.
|
|
14
|
+
- Untouched components keep their interaction state across versions (table sort, tab, form input and submitted lock, Plot sliders). Streaming, refused, failed or stopped patches never replace the previous version; superseded copies collapse to a pointer; a reload restores the latest version.
|
|
15
|
+
- Browser E2E: a cross-turn patch scenario against real DSH Web (refusal and retry, in-place bar → line, state kept, reload).
|
|
16
|
+
- Real-model quality baseline ([#13](https://github.com/dreamfarer-space/deepseek-harness-intelligent-ui/issues/13)): `npm run eval:real` runs three separately reported layers: the scripted-model browser E2E, a real-model **API** stage, and a new real-model **browser** stage in real DSH Web + Chrome. It exits non-zero on any failure.
|
|
17
|
+
- API stage: the model sees the plugin's real three tools (now including `ui_patch`) and system guidance. The tools' own `execute()` produces the tool results, so `ui_patch` edits are scored on the UI the user ends up with. Each case records tool choice, first-attempt schema validity versus recovery, and per-criterion checks of the rendered components and data. New cases: two successive chart edits (D3); D2 accepts an in-place patch.
|
|
18
|
+
- Browser stage: plain text, multi-series chart (legend, Q1–Q4, hover), sortable table, form submit → follow-up using the submitted values, Plot sliders, Three.js (non-blank WebGL canvas, drag rotates), stop → regenerate, two in-place edits, Fast 3G with cache off, CDN unreachable, and phone width. Every case runs in a new conversation and records console errors.
|
|
19
|
+
- Timings are recorded per stage: API time per model call, and in the browser first visible preview, complete generation and browser-side execution, each with p50/p95. `--repeat N` gives per-case success rates.
|
|
20
|
+
- The report (`docs/real-model-eval-report.json` + `.md`) records date, model (requested, API-reported and fingerprint), prompt hash, assertion version, environment and failure reasons. It is redacted, and the run refuses to write a report that still contains the key.
|
|
21
|
+
- A manually triggered `Real-model eval` workflow uses the `DEEPSEEK_API_KEY` secret.
|
|
22
|
+
- `scripts/eval/edit-demo.mjs` (`npm run eval:demo`) turns the browser-stage recording into a designed demo video, narrated by the DeepSeek whale girl in 26 poses from [Sutera-Diffusus/dsh-whale-musume](https://github.com/Sutera-Diffusus/dsh-whale-musume) (MIT). She switches poses with crossfades to fit each beat (hello, thinking at her laptop, an umbrella for the slow network, tears for a failure, confetti for a success), with motion on top. Each chapter is marked ✓/✗ and the narration follows the real outcome. Every frame is rendered deterministically.
|
|
23
|
+
- The recorder marks non-content spans (new conversations, screenshots, viewport switches) to cut, and holds each case's result on screen for a few seconds.
|
|
24
|
+
- Model waits are sped up; the recording's real-time stopwatch and a speed-factor sticker keep that visible.
|
|
25
|
+
- `scripts/eval/gallery.mjs` puts the README screenshots on one uniform 1200×750 canvas.
|
|
26
|
+
- Portable renderer for remote clients ([#22](https://github.com/dreamfarer-space/deepseek-harness-intelligent-ui/issues/22), experimental):
|
|
27
|
+
- `renderer/renderer.js` is a prebuilt, host-neutral WebView bundle (`DshIntelligentUiRenderer.mount(element, host)`) that renders accepted `ui_render`, `ui_patch` and `ui_render_widget` artifacts without DSH Web. It ships in the npm package and is checked in CI and before release.
|
|
28
|
+
- `deepseek-harness-intelligent-ui/portable` exports `projectNativeUiEvents`, which turns native Session events into artifacts.
|
|
29
|
+
- Widget errors inside the sandbox are reported to the remote client.
|
|
30
|
+
- See [docs/portable-renderer.md](https://github.com/dreamfarer-space/deepseek-harness-intelligent-ui/blob/main/docs/portable-renderer.md).
|
|
31
|
+
|
|
32
|
+
### Fixed
|
|
33
|
+
|
|
34
|
+
- Browser E2E: the dark-theme check polls until the host card and the widget frame have both switched, instead of measuring after a fixed 1.2 s. The nightly `dsh latest · macos-latest` job had measured too early.
|
|
35
|
+
|
|
36
|
+
### Changed
|
|
37
|
+
|
|
38
|
+
- The guidance prefers `ui_patch` for changes to an existing `ui_render` UI; widgets are still re-rendered in full.
|
|
39
|
+
- `Form` fields added by a later version start at their defaults; a `Table` sort on a removed column is ignored.
|
|
40
|
+
- The 2026-10-10 real-model report (15/15, including the B3 false positive) moved to `docs/history/` as a labelled historical snapshot. A test now pins that the historical B3 answer fails.
|
|
41
|
+
|
|
9
42
|
## [0.2.3]
|
|
10
43
|
|
|
11
44
|
### Changed
|
|
@@ -64,7 +97,8 @@ All notable changes to this project are documented here. The format follows
|
|
|
64
97
|
- The declarative component catalog (`ui_render`) and sandboxed streaming HTML widgets (`ui_render_widget`).
|
|
65
98
|
- The `/ui` command and the guidance system-prompt section.
|
|
66
99
|
|
|
67
|
-
[Unreleased]: https://github.com/dreamfarer-space/deepseek-harness-intelligent-ui/compare/v0.2.
|
|
100
|
+
[Unreleased]: https://github.com/dreamfarer-space/deepseek-harness-intelligent-ui/compare/v0.2.4...HEAD
|
|
101
|
+
[0.2.4]: https://github.com/dreamfarer-space/deepseek-harness-intelligent-ui/compare/v0.2.3...v0.2.4
|
|
68
102
|
[0.2.3]: https://github.com/dreamfarer-space/deepseek-harness-intelligent-ui/compare/v0.2.2...v0.2.3
|
|
69
103
|
[0.2.2]: https://github.com/dreamfarer-space/deepseek-harness-intelligent-ui/compare/v0.2.1...v0.2.2
|
|
70
104
|
[0.2.1]: https://github.com/dreamfarer-space/deepseek-harness-intelligent-ui/releases/tag/v0.2.1
|
package/README.md
CHANGED
|
@@ -11,7 +11,7 @@
|
|
|
11
11
|
[](https://github.com/dreamfarer-space/deepseek-harness-intelligent-ui/blob/main/docs/compatibility.md)
|
|
12
12
|
[](LICENSE)
|
|
13
13
|
|
|
14
|
-
[Demo](#-demo) · [Quick start](#-quick-start) · [How it decides](#-how-the-agent-decides) · [Widgets](#-sandboxed-widgets-ui_render_widget) · [Security](#-security-model) · [Configuration](#%EF%B8%8F-configuration) · [Develop](#-develop-and-verify)
|
|
14
|
+
[Demo](#-demo) · [Quality](#-real-model-quality-baseline) · [Quick start](#-quick-start) · [How it decides](#-how-the-agent-decides) · [Widgets](#-sandboxed-widgets-ui_render_widget) · [Security](#-security-model) · [Configuration](#%EF%B8%8F-configuration) · [Develop](#-develop-and-verify)
|
|
15
15
|
|
|
16
16
|
</div>
|
|
17
17
|
|
|
@@ -66,37 +66,60 @@ Open tabs, sort order, form values, sliders and `UI.state` survive turn folding,
|
|
|
66
66
|
|
|
67
67
|
## 🎬 Demo
|
|
68
68
|
|
|
69
|
-
|
|
69
|
+
### The real DeepSeek model, end to end
|
|
70
70
|
|
|
71
|
-
|
|
71
|
+
https://github.com/user-attachments/assets/e4eb0adc-659e-4c80-b473-449aa42ca00f
|
|
72
|
+
|
|
73
|
+
<details>
|
|
74
|
+
<summary><b>🎞️ GIF highlights</b> (for viewers that cannot play the video, e.g. npm)</summary>
|
|
75
|
+
<br>
|
|
76
|
+
|
|
77
|
+
<p align="center">
|
|
78
|
+
<img src="https://raw.githubusercontent.com/dreamfarer-space/deepseek-harness-intelligent-ui/main/docs/assets/real-model/demo.gif" alt="Real deepseek-flash model in DSH Web, narrated by the whale girl: a form round trip, a damped-oscillation Plot with sliders, and two in-place chart edits via ui_patch" width="100%">
|
|
79
|
+
</p>
|
|
80
|
+
|
|
81
|
+
</details>
|
|
82
|
+
|
|
83
|
+
<img align="right" width="120" src="https://raw.githubusercontent.com/dreamfarer-space/deepseek-harness-intelligent-ui/main/scripts/eval/demo/assets/whale/greet.webp" alt="The DeepSeek whale girl who narrates the demo">
|
|
84
|
+
|
|
85
|
+
🎥 **Full video (H.264, 2 min 27 s, 9.5 MB):** [docs/assets/real-model/demo.mp4](https://github.com/dreamfarer-space/deepseek-harness-intelligent-ui/blob/main/docs/assets/real-model/demo.mp4). It covers all 11 browser cases, and every chapter ends on its real result: text, chart, table, form round trip, Plot, Three.js with "Ask to fix", stop → regenerate, two in-place edits, Fast 3G, CDN offline and phone width. The DeepSeek whale girl (鲸鱼娘) narrates; her art is from [dsh-whale-musume](https://github.com/Sutera-Diffusus/dsh-whale-musume) (MIT, see the [credits](#credits)).
|
|
72
86
|
|
|
73
87
|
> [!NOTE]
|
|
74
|
-
>
|
|
88
|
+
> Recorded on 2026-10-16 by `npm run eval:real -- --stage browser --record` and edited with `npm run eval:demo`. Every card was generated live by the real **`deepseek-flash`** model in real **DSH Web 0.2.0-rc.1** with this plugin installed. Nothing is scripted, and **failures are left in**: each chapter card is marked ✓ or ✗, and the narration says what went wrong. While the model is working the video is sped up; the stopwatch in the corner shows the real elapsed time. The results are in the [real-model quality baseline](#-real-model-quality-baseline).
|
|
89
|
+
|
|
90
|
+
| Multi-series chart: **✗** (call cut off mid-stream) | Two in-place edits with `ui_patch`: ✓ |
|
|
91
|
+
|:---:|:---:|
|
|
92
|
+
|  |  |
|
|
93
|
+
| *The chart is right, but the card says "generation interrupted": more components were still pending* | *"Make it a line chart", then "add 小程序 30/45/60/80": one card, version 3, exact values* |
|
|
94
|
+
| **Form submit → the agent uses the values: ✓** | **Damped oscillation `Plot` with live sliders: ✓** |
|
|
95
|
+
|  |  |
|
|
96
|
+
| *The submitted 全国 / 4 / 乐观 drive a quarterly forecast* | *y = A·e^(−γt)·cos(ωt) with envelopes; the sliders never call the model* |
|
|
97
|
+
| **Three.js: ✗, then repaired with "Ask to fix"** | **Stopped mid-stream → regenerated: ✓** |
|
|
98
|
+
|  |  |
|
|
99
|
+
| *First attempt: `view is not defined`. After "Ask to fix": sphere centred at (0,0,R) with axes* | *The interrupted card keeps a static preview; Regenerate produces the full scene* |
|
|
100
|
+
| **Fast 3G, cache off: ✓** | **CDN unreachable: ✓ (degrades as designed)** |
|
|
101
|
+
|  |  |
|
|
102
|
+
| *three.js loads slowly but still renders* | *The failure is shown on the card; the chat keeps working* |
|
|
103
|
+
| **Phone width (390 px): ✓** | **Sortable table: ✗ (no table within 5 min)** |
|
|
104
|
+
|  |  |
|
|
105
|
+
| *No overflow; KPI text and charts fit* | *The agent spent the whole budget researching car specs on the web* |
|
|
75
106
|
|
|
76
107
|
<details>
|
|
77
|
-
<summary><b
|
|
108
|
+
<summary><b>🧪 Scripted-model demo</b> (the pipeline E2E: dark mode, reload, sandbox isolation)</summary>
|
|
78
109
|
<br>
|
|
79
110
|
|
|
80
|
-
|
|
81
|
-
|:---:|:---:|
|
|
82
|
-
|  |  |
|
|
83
|
-
| *Stat, Grid, Tabs, stacked bar chart with hover tooltip, sortable table, timeline, form* | *`x²+y²+z²=2Rz` via `UI.three()`: the slider re-shapes the surface locally* |
|
|
111
|
+
|
|
84
112
|
|
|
85
|
-
|
|
86
|
-
|:---:|:---:|
|
|
87
|
-
|  |  |
|
|
88
|
-
| *KaTeX formula, `Plot` with parameter sliders (no model call), Mermaid flowchart* | *Submitted values reach the agent, which answers with a forecast chart* |
|
|
113
|
+
🎥 [docs/assets/demo.mp4](https://github.com/dreamfarer-space/deepseek-harness-intelligent-ui/blob/v0.2.1/docs/assets/demo.mp4). This recording is from `npm run e2e:all -- --record`. The model behind it is the **scripted mock** in `scripts/e2e/`, so the answers are canned. It shows the plugin pipeline (streaming, rendering, round trips, persistence, isolation), not how a real model behaves.
|
|
89
114
|
|
|
90
|
-
|
|
|
115
|
+
| Dashboard streams in, then settles | Three.js widget: R slider, rotate, zoom |
|
|
91
116
|
|:---:|:---:|
|
|
92
|
-
|  |  |
|
|
93
118
|
|
|
94
119
|
| Dark theme | Phone width (dark) | After reload |
|
|
95
120
|
|:---:|:---:|:---:|
|
|
96
121
|
|  |  |  |
|
|
97
122
|
|
|
98
|
-
To get an inline player on GitHub, drag `docs/assets/demo.mp4` into a GitHub comment or release editor and paste the resulting `https://github.com/user-attachments/assets/…` URL on its own line here. GitHub only renders inline video for uploaded attachments, not for repository files.
|
|
99
|
-
|
|
100
123
|
</details>
|
|
101
124
|
|
|
102
125
|
## 🚀 Quick start
|
|
@@ -120,7 +143,7 @@ npx deepseek-harness-intelligent-ui doctor # diagnose only
|
|
|
120
143
|
|---|---|
|
|
121
144
|
| Upgrade | `dsh plugin --profile web add deepseek-harness-intelligent-ui@latest` |
|
|
122
145
|
| Try a prerelease (only when the `next` dist-tag exists) | `dsh plugin --profile web add deepseek-harness-intelligent-ui@next` |
|
|
123
|
-
| Pin a version | `dsh plugin --profile web add deepseek-harness-intelligent-ui@0.2.
|
|
146
|
+
| Pin a version | `dsh plugin --profile web add deepseek-harness-intelligent-ui@0.2.4` |
|
|
124
147
|
| Uninstall | `dsh plugin --profile web remove deepseek-harness-intelligent-ui` (or `npx deepseek-harness-intelligent-ui uninstall`) |
|
|
125
148
|
| From a checkout | `dsh plugin --profile web add file:/path/to/deepseek-harness-intelligent-ui` |
|
|
126
149
|
|
|
@@ -163,6 +186,7 @@ The plugin injects a system-prompt section (`src/guidance.js`) that makes the mo
|
|
|
163
186
|
| Tool | Use it for | Rendering |
|
|
164
187
|
|---|---|---|
|
|
165
188
|
| `ui_render` | Structured answers | Native React components from a fixed catalog, rendered progressively as the JSON streams |
|
|
189
|
+
| `ui_patch` | Changing a UI shown earlier | Component-level operations on that UI; untouched components keep their state ([details](#-updating-a-ui-in-place-ui_patch)) |
|
|
166
190
|
| `ui_render_widget` | Custom visuals, 3D, simulations, mini-apps | HTML/CSS/JS in an opaque-origin sandboxed iframe: CSS first, HTML morphed in live, JS last, all in **one** iframe |
|
|
167
191
|
|
|
168
192
|
## 🧩 Component catalog (`ui_render`)
|
|
@@ -180,6 +204,18 @@ A UI travels as a **flat node list with id references**, because DSH's tool-sche
|
|
|
180
204
|
|
|
181
205
|
`Math` and `Mermaid` render inside the same sandbox as widgets, loading KaTeX or Mermaid from the CDN allowlist. No third-party code runs in the DSH page and the host bundle stays small (~127 KB). If the CDN is unreachable, the source text is shown instead.
|
|
182
206
|
|
|
207
|
+
## 🔁 Updating a UI in place (`ui_patch`)
|
|
208
|
+
|
|
209
|
+
When the user asks to change a UI shown earlier ("make that bar chart a line chart; leave the rest"), the agent patches it by component id instead of re-sending the whole tree.
|
|
210
|
+
|
|
211
|
+
- **Identity:** every `ui_render` result tells the model the UI's `uiId` (its call id) and `version` 1. Each accepted patch reports the new version and an `id:Type` outline of the components.
|
|
212
|
+
- **Operations:** `setProps` (merge props, `unset` to remove), `replaceNode` (same id, new type/props/children), `insertChild` (new components under a container), and `removeNode`. They apply in order and **all-or-nothing**. The whole next version then passes the same checks as a fresh `ui_render` (types, props, references, cycles, depth, sizes, Tabs labels).
|
|
213
|
+
- **Concurrency:** `baseVersion` is required. A patch based on an older version is refused as stale and reports the current version, so the model re-derives the patch and retries. Patches run exclusively (never in parallel), so versions form one ordered chain.
|
|
214
|
+
- **Scope:** the Host stores nothing. It rebuilds the current version by replaying the conversation's own history (accepted renders, then accepted patches), so a patch can only address UIs of the same conversation. Unknown ids, other conversations' UIs and UIs dropped by context compaction are refused, and the model falls back to `ui_render`.
|
|
215
|
+
- **State:** components keep their ids across versions, and interaction state is keyed by `uiId` + component id. Every component a patch does not replace keeps its table sort, selected tab, form input, submitted lock and Plot slider positions. A `replaceNode`d component starts fresh; to reset any component, give it a new id. Form fields added by a patch start at their defaults.
|
|
216
|
+
- **Display:** while a patch streams or runs, its card is a one-line "Updating the UI…" status and the previous version stays live, so a half-applied patch is never drawn. On success the next version renders beneath the new reply and the older copy collapses to "Updated to version N". A refused, failed or stopped patch leaves the previous version untouched. The full snapshot travels in the durable result meta, so a reload shows the latest version.
|
|
217
|
+
- **Widgets** (`ui_render_widget`) are not patched: the agent re-renders them, and they keep their `UI.state` when it does.
|
|
218
|
+
|
|
183
219
|
## 🪟 Sandboxed widgets (`ui_render_widget`)
|
|
184
220
|
|
|
185
221
|
<img src="https://raw.githubusercontent.com/dreamfarer-space/deepseek-harness-intelligent-ui/main/docs/assets/readme/widget-streaming.svg" alt="Widget fields stream in order: placeholder, then a live preview with morphed HTML, then the same iframe is finalized and the code runs" width="100%">
|
|
@@ -206,7 +242,7 @@ Fields follow the OpenIntelligentUI streaming contract, **in this order**: `titl
|
|
|
206
242
|
|
|
207
243
|
- While a turn runs, the UI renders inline inside the tool call. One component instance spans preparing → running → result, so interaction state carries over.
|
|
208
244
|
- When the turn ends, DSH folds tool calls into the "process" section. The plugin re-presents each UI **beneath the reply** (`conversation.chat.turnTail`), and the inline copy shrinks to a pointer.
|
|
209
|
-
- **Interaction state persists per tool call
|
|
245
|
+
- **Interaction state persists per UI** (per tool call; patched versions share their `ui_render` call's state): the open tab, table sort, form values, the submitted lock, Plot sliders, widget controls and `UI.state`, and frame heights. It lives in memory and in `localStorage` (bounded), so moving beneath the reply or reloading the page restores it. The UIs themselves are rebuilt from the durable session log.
|
|
210
246
|
- **Stopped generation:**
|
|
211
247
|
- If the call already reached DSH, the view keeps the finished part (a static widget preview or the completed components) with a notice and a **Regenerate** button.
|
|
212
248
|
- If the user stops while the arguments are still streaming, DSH drops the call entirely. The plugin stashes what was streamed and shows it beneath the stopped turn the same way.
|
|
@@ -235,7 +271,7 @@ The bundle patch inserts the plugin row automatically. To change settings, add t
|
|
|
235
271
|
|
|
236
272
|
| Option | Default | Meaning |
|
|
237
273
|
|---|---|---|
|
|
238
|
-
| `enableCatalog` | `true` | Register `ui_render` |
|
|
274
|
+
| `enableCatalog` | `true` | Register `ui_render` and `ui_patch` |
|
|
239
275
|
| `enableWidget` | `true` | Register `ui_render_widget` |
|
|
240
276
|
| `injectGuidance` | `true` | Add the "when to use UI" system-prompt section |
|
|
241
277
|
| `cdnAllowlist` | esm.sh, jsdelivr, unpkg | Origins widgets (and Math/Mermaid) may load resources from |
|
|
@@ -252,11 +288,12 @@ npm install
|
|
|
252
288
|
npm run build # client-src/ → client/client.js (DSH __ModuleLoader__ factory; React and Cordis external)
|
|
253
289
|
npm test # vitest (catalog, expressions, tools, plugin mount, streaming parser, frame/CSP/bridge, state, theme, views, interruption, turn tail, DSH compatibility gate, installer CLI)
|
|
254
290
|
npm run typecheck
|
|
255
|
-
npm run e2e:all # build → install into a temp DSH home → mock model + real DSH Web →
|
|
291
|
+
npm run e2e:all # build → install into a temp DSH home → mock model + real DSH Web → 30 browser checks
|
|
256
292
|
npm run e2e:all -- --record # also re-records docs/assets/demo.mp4 / demo.gif (needs ffmpeg)
|
|
257
293
|
npm run e2e:all -- --spec ./deepseek-harness-intelligent-ui-<v>.tgz # E2E the packed tarball instead of the checkout
|
|
258
294
|
npm run smoke -- --spec ./deepseek-harness-intelligent-ui-<v>.tgz # install smoke test (no browser): version gate, bundle row, DSH Web boot
|
|
259
295
|
npm run docs:art # regenerate the animated README illustrations in docs/assets/readme/
|
|
296
|
+
npm run eval:real # real-model quality baseline (needs DEEPSEEK_API_KEY): see below
|
|
260
297
|
```
|
|
261
298
|
|
|
262
299
|
The E2E runs on Windows, Linux and macOS. It finds Chrome per platform (set `CHROME_PATH` to override) and uses `DSH_BIN` to test another `dsh` binary. CI runs everything on all three OSes against every supported DSH line ([compatibility](https://github.com/dreamfarer-space/deepseek-harness-intelligent-ui/blob/main/docs/compatibility.md)).
|
|
@@ -274,6 +311,7 @@ Report: [`docs/e2e-report.json`](https://github.com/dreamfarer-space/deepseek-ha
|
|
|
274
311
|
- The dashboard streams progressively.
|
|
275
312
|
- Chart hover, table sort and tabs are local.
|
|
276
313
|
- A form round-trip returns a new UI.
|
|
314
|
+
- A second-turn `ui_patch` turns the dashboard chart from bar to line: an invalid first attempt is refused, the old copy collapses, and sort, tab and form state are kept, including after a reload.
|
|
277
315
|
- The preview iframe is finalized in place.
|
|
278
316
|
- The Three.js scene renders, and the R slider, drag-rotate and wheel-zoom work.
|
|
279
317
|
- The `Bridge.sendPrompt` round trip works.
|
|
@@ -306,32 +344,102 @@ See [release security and OIDC setup](https://github.com/dreamfarer-space/deepse
|
|
|
306
344
|
|
|
307
345
|
</details>
|
|
308
346
|
|
|
309
|
-
|
|
310
|
-
<summary><b>🤖 Real-model evaluation</b></summary>
|
|
311
|
-
<br>
|
|
347
|
+
## 📊 Real-model quality baseline
|
|
312
348
|
|
|
313
|
-
|
|
349
|
+
A tool call whose arguments parse is not the same as a UI the user can use. `npm run eval:real` measures three layers **separately**, and a pass in one is not evidence for another:
|
|
314
350
|
|
|
315
|
-
|
|
351
|
+
| Layer | What it proves | Latest · 2026-10-16 · `deepseek-flash` |
|
|
352
|
+
| --- | --- | :---: |
|
|
353
|
+
| **Mock E2E** | Rendering, streaming, interaction, persistence and isolation, with **scripted** model answers | **30 / 30** |
|
|
354
|
+
| **Real model · API** | The model picks the right tool, emits valid arguments on the **first** try, and the rendered UI contains the requested components and data. The model sees the plugin's real tools and guidance; the tools' own `execute()` runs, including `ui_patch` | **30 / 32** (94%) |
|
|
355
|
+
| **Real model · browser** | What the model generated actually works in real DSH Web + Chrome: it renders, responds to interaction, survives stop/resume and bad networks, and logs no console errors | **9 / 11** automated<br>**8 / 11** after manual review |
|
|
316
356
|
|
|
317
|
-
|
|
357
|
+
| Timing (real model) | p50 | p95 |
|
|
358
|
+
| --- | ---: | ---: |
|
|
359
|
+
| First visible UI preview (browser) | 7.6 s | 70.2 s |
|
|
360
|
+
| UI generation complete (browser) | 11.8 s | 73.4 s |
|
|
361
|
+
| Browser-side execution after the stream (widget code, three.js from the CDN) | 26 ms | 2.9 s |
|
|
362
|
+
| One model API call | 2.4 s | 75.7 s |
|
|
318
363
|
|
|
319
|
-
|
|
364
|
+
The long tail comes from large Three.js widgets: the model writes 10–20 KB of code before the preview starts.
|
|
365
|
+
|
|
366
|
+
**What the run found.** These are real model and agent behaviours, not harness noise; each one is a failed or flagged case in the [full report](https://github.com/dreamfarer-space/deepseek-harness-intelligent-ui/blob/main/docs/real-model-eval.md):
|
|
367
|
+
|
|
368
|
+
- **A tool call cut off mid-stream (R-chart):** the chart rendered correctly, but the turn ended while more components were still streaming. The plugin correctly downgraded the card to "generation interrupted, showing the completed part" with a Regenerate button. The automated content checks passed, but the manual review counts it as a failure, and a new check now fails any case that leaves an interrupted card.
|
|
369
|
+
- **Agent research instead of a UI (R-table):** DSH's agent spent the whole 5-minute budget researching car specs on the web (20 steps, about 1.3M tokens) and never rendered the table.
|
|
370
|
+
- **Three.js scope slips:** two recordings, two variants of the same bug. `view is not defined` in this run and `THREE is not defined` in the previous one: the widget code used a variable outside its scope. The card flags "1 个脚本错误", and the user's **"让模型修复"** button produced a working scene both times.
|
|
371
|
+
- **Invalid first attempts (API):** a missing closing brace in nested `props` (D2, both turns) and duplicate component ids (B1). The plugin refuses these and the model fixes them on retry, but the user briefly sees a refused card, so these count as failures.
|
|
372
|
+
- **Agent behaviour around the UI:** DSH's agent may first search the workspace, ask a clarifying question, or send sub-agents to research facts. The browser stage answers clarifying questions like a user would (picking the recommended option) and waits for the UI itself, not just for the turn to go idle.
|
|
373
|
+
|
|
374
|
+
<details>
|
|
375
|
+
<summary><b>▶️ Run it</b></summary>
|
|
376
|
+
<br>
|
|
377
|
+
|
|
378
|
+
Put the key in the environment. No credentials file is read, and the run refuses to write a report that still contains the key.
|
|
320
379
|
|
|
321
380
|
```sh
|
|
322
|
-
export DEEPSEEK_API_KEY='<your-key>'
|
|
323
|
-
|
|
381
|
+
export DEEPSEEK_API_KEY='<your-key>' # PowerShell: $env:DEEPSEEK_API_KEY = '<your-key>'
|
|
382
|
+
npm run eval:real # mock E2E + API + browser; exits 1 on any failure
|
|
383
|
+
npm run eval:real -- --repeat 3 # 3 runs per API case → per-case success rates, p50/p95
|
|
384
|
+
npm run eval:real -- --record && npm run eval:demo # also record and cut the demo video
|
|
385
|
+
npm run eval:gallery # README screenshots on one uniform 1200×750 canvas
|
|
324
386
|
```
|
|
325
387
|
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
388
|
+
| Option | Effect |
|
|
389
|
+
| --- | --- |
|
|
390
|
+
| `--stage mock,api,browser` | Run only some layers (`all` by default). `A`/`B`/`C`/`D` selects API categories, as before. |
|
|
391
|
+
| `--only B3,R-chart` | Run only these cases. |
|
|
392
|
+
| `--keep-other-layers` | Keep the layers (or, with `--only`, the cases) this run did not execute. Each keeps its own date. |
|
|
393
|
+
| `--append` | With `--only`: add the rerun as extra samples. A failure is never replaced by a lucky retry. |
|
|
394
|
+
| `--rescore` | Re-judge the last API run from its saved transcripts with the current assertions, with no model calls. |
|
|
395
|
+
| `DEEPSEEK_EVAL_MODEL` | Model id (default `deepseek-flash`). |
|
|
330
396
|
|
|
331
|
-
|
|
397
|
+
The **Real-model eval** GitHub workflow runs the same command on demand with the `DEEPSEEK_API_KEY` secret and uploads the report and screenshots. A full run costs about ¥1.5 with `deepseek-flash`.
|
|
332
398
|
|
|
333
399
|
</details>
|
|
334
400
|
|
|
401
|
+
<details>
|
|
402
|
+
<summary><b>🔬 What each case checks</b></summary>
|
|
403
|
+
<br>
|
|
404
|
+
|
|
405
|
+
**API stage** (`scripts/eval/api-stage.mjs`, 16 cases, assertions v3). Every criterion is its own check, so the report says *which* one failed:
|
|
406
|
+
|
|
407
|
+
- **Plain text (A1–A4):** no UI tool, a substantive answer.
|
|
408
|
+
- **Charts/tables (B1–B3, B5):**
|
|
409
|
+
- KPI tiles named by meaning (revenue, active users incl. 月活/MAU, conversion, order value);
|
|
410
|
+
- a sortable table with all five requested columns;
|
|
411
|
+
- a **bar chart with Q1–Q4 and both an online and an offline series** (the 2026-10-10 B3 answer, a form without a chart, is kept as a test fixture and must fail);
|
|
412
|
+
- a chronology with 4+ milestones (a `Timeline`, or a Mermaid diagram with 4+ ordered years).
|
|
413
|
+
- **Form (B4, D1):** range, checkbox and select fields. D1 submits the form exactly as the browser would and requires the follow-up to use the submitted region, quarter count and scenario.
|
|
414
|
+
- **Math Plot (C2):** three sliders, *every slider used in the expression*, and an `exp(…)·sin/cos(…)` shape.
|
|
415
|
+
- **Three.js and widgets (C1, C3, C4):** `UI.three()`/`import("three")`, animation, lights, start/pause and parameter controls.
|
|
416
|
+
- **Multi-turn edits:** D2 adds a column with every per-model value correct and every old value kept. D3 makes two successive edits (bar → line, then a third series with exact values, original series unchanged). `ui_patch` in place is recorded.
|
|
417
|
+
|
|
418
|
+
Each case records tool choice, first-attempt schema validity versus recovery, API time per call, the model id the API reports and its `system_fingerprint`, and the prompt hash.
|
|
419
|
+
|
|
420
|
+
**Browser stage** (`scripts/eval/browser-stage.mjs`, 11 cases, each in a new conversation):
|
|
421
|
+
|
|
422
|
+
- **Plain text:** no card is created.
|
|
423
|
+
- **Chart:** legend, Q1–Q4, 8+ bars, local hover tooltip.
|
|
424
|
+
- **Table:** clicking a header re-sorts rows with no model call.
|
|
425
|
+
- **Form:** filled and submitted with the real mouse and keyboard; the follow-up must mention the submitted values; the form locks.
|
|
426
|
+
- **Plot:** dragging a slider redraws the curve.
|
|
427
|
+
- **Three.js:** the canvas is **drawn, not blank** (pixel statistics), dragging changes the image, and there are no widget script errors. On failure, the "Ask to fix" path is tried and reported.
|
|
428
|
+
- **Stop/resume:** stop mid-stream → static preview, notice and Regenerate → a complete scene.
|
|
429
|
+
- **Two in-place edits:** one live card, version 3, line chart, third series.
|
|
430
|
+
- **Weak network:** Fast 3G with the cache off.
|
|
431
|
+
- **No network:** the CDN is blocked; the failure must be shown on the card, the DSH page must have no uncaught errors, and the chat must keep working.
|
|
432
|
+
- **Phone width (390 px):** no overflow; the KPI grid uses at most 2 columns.
|
|
433
|
+
|
|
434
|
+
Every case also records console errors, with the host page separated from widget frames. DSH run failures such as quota errors are reported as *infrastructure* errors, outside the model's success rate.
|
|
435
|
+
|
|
436
|
+
Automated checks cannot judge visual quality, so the report also carries a **sampled manual review** of every screenshot.
|
|
437
|
+
|
|
438
|
+
</details>
|
|
439
|
+
|
|
440
|
+
> [!IMPORTANT]
|
|
441
|
+
> The [2026-10-10 report](https://github.com/dreamfarer-space/deepseek-harness-intelligent-ui/blob/main/docs/history/2026-10-10-real-model-eval-report.json) (15/15) is a **historical snapshot**, not a pass rate. It came from an API-only evaluator that passed B3 although no chart was rendered, and it did not keep the arguments needed to rescore it. The current numbers are in [`docs/real-model-eval.md`](https://github.com/dreamfarer-space/deepseek-harness-intelligent-ui/blob/main/docs/real-model-eval.md) (human-readable) and [`docs/real-model-eval-report.json`](https://github.com/dreamfarer-space/deepseek-harness-intelligent-ui/blob/main/docs/real-model-eval-report.json).
|
|
442
|
+
|
|
335
443
|
## Limitations
|
|
336
444
|
|
|
337
445
|
- Widgets, Math and Mermaid need the CDN allowlist to be reachable. Offline, Math/Mermaid fall back to source text and `UI.three` reports a script error.
|
|
@@ -341,3 +449,5 @@ Each invocation writes only that run's cases to `docs/real-model-eval-report.jso
|
|
|
341
449
|
## Credits
|
|
342
450
|
|
|
343
451
|
Concepts, the streaming widget contract, and parts of the prompt guidance come from [CopilotKit/OpenIntelligentUI](https://github.com/CopilotKit/OpenIntelligentUI) (MIT). See [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md). Licensed under MIT.
|
|
452
|
+
|
|
453
|
+
**Whale girl in the demo video.** The DeepSeek whale girl (鲸鱼娘) who narrates the demo, in 26 poses, is by [Sutera-Diffusus/dsh-whale-musume](https://github.com/Sutera-Diffusus/dsh-whale-musume) (MIT © Sutera-Diffusus; illustrations by [@SuteraWu](https://github.com/SuteraWu)). The unmodified files and their licence are in [`scripts/eval/demo/assets/whale/`](https://github.com/dreamfarer-space/deepseek-harness-intelligent-ui/tree/main/scripts/eval/demo/assets/whale). They are not part of the npm package.
|