@gui-chat-plugin/sequence 0.2.0 → 0.3.1

package/CHANGELOG.md

@@ -1,5 +1,25 @@
 # Changelog
 
+## 0.3.1
+
+- **"HTML slides" go to presentSlide**: the prompt says that when the user asks for HTML slides,
+  presentSlide is the tool, one HTML slide per call, not one HTML page. OpenAI Realtime, asked to
+  "explain this article in HTML slides", wrote a single presentHtml page with its own slide
+  navigation (found in MulmoChat). With this and the hosts' presentHtml prompt, 6 of 6 runs in
+  MulmoChat and 3 of 3 in MulmoGlass made presentSlide HTML slides.
+
+## 0.3.0
+
+- **HTML slides**: `presentSlide` takes `html` instead of `imagePrompt` for a slide designed in HTML
+  (words, numbers, lists, comparisons, diagrams), styled with Tailwind CSS v4 and animated, so it
+  builds up while the model explains it. The model picks per slide; a slideshow can mix them. The
+  View shows it at 1280x720, scaled to fit, in a sandboxed iframe (`sandbox="allow-scripts"`, no
+  network: `connect-src 'none'`, images from `data:`/`blob:` only; the slide's own scripts, event
+  handlers, `<meta>` refreshes and links don't run, since they could navigate the frame to a URL
+  carrying the slide). Nothing is drawn, so it needs
+  no image backend. `slideHtmlDocument(html)` makes the page, for a host that shows one elsewhere.
+  Records and results carry `html`; `imagePrompt` is `""` and `imagePath` absent for these slides.
+
 ## 0.2.0
 
 - **State per conversation**: the slideshow in progress, the steps and panels waiting for the

package/README.md

@@ -3,11 +3,11 @@
 Tools that show things one step at a time, as pictures, for
 [GUI Chat Protocol](https://github.com/receptron/gui-chat-protocol) hosts:
 
-| Tool               | Shows                                                                                                                                     |
-| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
-| `presentSlide`     | One slide of a spoken slideshow (mode `"presentation"`), or one step of a step-by-step guide the user follows along with (mode `"steps"`) |
-| `defineStoryboard` | A story's cast: a reference sheet for each recurring character                                                                            |
-| `presentPanel`     | One panel of the story, with its characters drawn as in their sheets; an interactive story offers choices                                 |
+| Tool               | Shows                                                                                                                                                                              |
+| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `presentSlide`     | One slide of a spoken slideshow (mode `"presentation"`), or one step of a step-by-step guide the user follows along with (mode `"steps"`): a generated picture, or a slide in HTML |
+| `defineStoryboard` | A story's cast: a reference sheet for each recurring character                                                                                                                     |
+| `presentPanel`     | One panel of the story, with its characters drawn as in their sheets; an interactive story offers choices                                                                          |
 
 From MulmoChat and MulmoGlass, tested there by voice on OpenAI Realtime, Gemini Live and Grok, and
 in text chat.
@@ -60,11 +60,34 @@ overlap (a host that runs `execute()` on a server gets them as concurrent reques
 claimed before anything is awaited, so an identical one waits for it and a later step is held while
 it is drawn.
 
+## HTML slides
+
+A slide is a generated picture (`imagePrompt`) or HTML (`html`), the model's choice per slide: a
+picture for a scene or an object, HTML for words, numbers, lists, comparisons and diagrams, which
+picture models draw badly. The model writes the inside of `<body>` for a 1280x720 canvas, with
+Tailwind CSS v4 classes and these entrance animations: `animate-fade-in`, `animate-fade-up`,
+`animate-fade-down`, `animate-slide-in-left`, `animate-slide-in-right`, `animate-zoom-in`,
+`animate-pop` and `animate-draw` (an SVG path drawing itself), staggered with
+`[animation-delay:…]`. The slide's own scripts don't run: its animations are CSS.
+
+`slideHtmlDocument(html)` (core entry) makes the page: Tailwind's browser build from jsDelivr (one
+version, checked by its hash), the animations, and a Content Security Policy that lets it send
+nothing out: `connect-src 'none'`; images, media and fonts only from `data:`/`blob:` URLs and
+Google Fonts; and no script but the page's own and Tailwind's, so nothing can navigate the frame to
+a URL carrying the page (a sandbox allows that, and CSP can't forbid it). The slide's `<meta>`,
+`<base>` and `<link>` tags are made plain text (a refresh would navigate), and a link doesn't
+navigate when clicked. The View shows it in an iframe with `sandbox="allow-scripts"` and no
+`allow-same-origin`, so the page can't reach the host's pages, storage or cookies, scaled to fit. Tailwind is loaded from the network: offline, the slide
+appears unstyled.
+
+A host that sends `currentResult` by its saved picture's path only (as MulmoChat does) can't tell
+that an HTML step is on the screen; a guide step asked for again is then shown again.
+
 ## Records
 
 Saved through `files.artifacts`, with their pictures' paths:
 
-- `slideshows/<id>.json`: `{ id, title, mode, totalSlides, slides: { "1": { title, imagePrompt, imagePath } } }`
+- `slideshows/<id>.json`: `{ id, title, mode, totalSlides, slides: { "1": { title, imagePrompt, imagePath } } }`; an HTML slide has `html`, an `imagePrompt` of `""` and no `imagePath`
 - `storyboards/<id>.json`: `{ id, title, style, totalPanels, interactive, characters: [{ name, description, imagePath }], panels: { "1": { caption, characters, imagePrompt, imagePath, choices } } }`
 
 IDs are 12 hex digits. The results name them, so a later tool call can refer to a slideshow or a

package/dist/core/definitions.d.ts

@@ -13,12 +13,17 @@ export interface SlideArgs {
     slide: number;
     totalSlides: number;
     title: string;
+    /** The picture's prompt; "" for an HTML slide. */
     imagePrompt: string;
+    /** An HTML slide's body; absent for a picture. */
+    html?: string;
     mode: SlideMode;
 }
-export declare const SLIDE_ARGS_ERROR = "presentSlide needs slide and totalSlides (whole numbers, slide from 1 to totalSlides) and an imagePrompt";
-/** The slide arguments, or null when the model sent something else. */
-export declare function parseSlideArgs(args: Record<string, unknown>): SlideArgs | null;
+export declare const SLIDE_ARGS_ERROR = "presentSlide needs slide and totalSlides (whole numbers, slide from 1 to totalSlides), and either an imagePrompt or html";
+export declare const SLIDE_HTML_TOO_LONG = "presentSlide's html is too long: keep a slide under 60000 characters";
+/** The slide arguments, or what is wrong with them (said to the model). A
+ *  slide with html is an HTML slide, whatever imagePrompt says. */
+export declare function parseSlideArgs(args: Record<string, unknown>): SlideArgs | string;
 /** What the model does once a slide is on the screen. */
 export declare const slideShownInstructions: ({ slide, totalSlides, title, mode, }: SlideArgs) => string;
 /** A slideshow as saved in artifacts/slideshows/<id>.json. */
@@ -30,14 +35,19 @@ export interface Slideshow {
     totalSlides: number;
     slides: Record<string, {
         title: string;
+        /** "" for an HTML slide. */
         imagePrompt: string;
         imagePath?: string;
+        /** An HTML slide's body (slideHtmlDocument makes it a page). */
+        html?: string;
     }>;
 }
-/** A shown slide's result data. */
+/** A shown slide's result data: a picture, or an HTML slide's body. */
 export interface SlideData {
-    imageData: string;
+    imageData?: string;
     imagePath?: string;
+    html?: string;
+    /** The picture's prompt; "" for an HTML slide. */
     prompt: string;
     slideshowId: string;
     slide: number;

package/dist/core/host.d.ts

@@ -11,9 +11,15 @@ export declare function imageOf(result: ToolResult | null | undefined): {
     imageData?: string;
     imagePath?: string;
 };
+/** An HTML slide's body, when the result is one. */
+export declare function htmlOf(result: ToolResult | null | undefined): string;
+/** Whether a result shows something: a picture or an HTML slide. */
+export declare const shownContent: (result: ToolResult) => boolean;
 /** Whether two results show the same picture: by its saved path when both
  *  have one (a host may send the current result without its image data),
- *  else by the picture itself. */
+ *  else by the picture itself, or the same HTML slide. A host that sends the
+ *  current result by its path only can't tell an HTML slide is on the
+ *  screen: one asked for again is shown again. */
 export declare function samePicture(a: ToolResult | null | undefined, b: ToolResult | null | undefined): boolean;
 /** When the user spoke after `time` (ms since the epoch). A host that doesn't
  *  say when the user spoke holds nothing. */

package/dist/core/index.d.ts

@@ -4,5 +4,6 @@ export declare const presentSlidePluginCore: SequencePluginCore;
 export declare const defineStoryboardPluginCore: SequencePluginCore;
 export declare const presentPanelPluginCore: SequencePluginCore;
 export * from "./definitions";
+export { MAX_SLIDE_HTML, SLIDE_ANIMATIONS, SLIDE_HEIGHT, SLIDE_WIDTH, slideHtmlDocument, } from "./slideHtml";
 export { SLIDESHOWS_DIR, STORYBOARDS_DIR, loadRecord } from "./records";
 export type { SequenceContext } from "./host";

package/dist/core/slideHtml.d.ts

@@ -0,0 +1,23 @@
+export declare const SLIDE_WIDTH = 1280;
+export declare const SLIDE_HEIGHT = 720;
+/** The most HTML a slide may have, in characters. */
+export declare const MAX_SLIDE_HTML = 60000;
+/** The animations every slide has, as `animate-<name>` classes. They keep
+ *  their first frame while delayed (fill mode both), so an element with a
+ *  delay stays hidden until its turn. */
+export declare const SLIDE_ANIMATIONS: readonly ["fade-in", "fade-up", "fade-down", "slide-in-left", "slide-in-right", "zoom-in", "pop", "draw"];
+/** PAGE_SCRIPT's hash, which the policy allows (checked by a test). */
+export declare const PAGE_SCRIPT_HASH = "sha256-MZyL5rvWmIVxl3WUuDU5BqzPJGR2/fV/YPh01MNyUoM=";
+/** The slide's HTML, made safe to show:
+ *  - the tags that act on the page from outside the body are plain text:
+ *    <meta> (a refresh navigates the frame, a policy of its own), <base> and
+ *    <link> (prefetches). No tag can be rebuilt by the replacement, which only
+ *    adds "&lt;".
+ *  - no declarative shadow root (<template shadowrootmode>, the only way to
+ *    make one without a script): a click inside one reaches the page's link
+ *    guard retargeted to its host, so a link there would navigate. Every
+ *    "shadowroot" is renamed, which leaves no attribute that makes one.
+ *    Everything stays in the light DOM, where the guard sees it. */
+export declare const neutralizeSlideHtml: (html: string) => string;
+/** The page an HTML slide is shown as. */
+export declare function slideHtmlDocument(html: string): string;