@gui-chat-plugin/sequence 0.6.3 → 0.7.1

package/CHANGELOG.md

@@ -1,5 +1,28 @@
 # Changelog
 
+## 0.7.1
+
+- **A web page's address shows the page's preview picture.** In MulmoChat, the model had only
+  pages' addresses from its searches and browsing (neither returns image files'), and gave one as
+  `imageUrl`. A page's `og:image` (else `twitter:image`) is now fetched in its place, under the
+  same checks; one that is a page in turn isn't followed. Tried on Car and Driver's and Wired's
+  articles about the Tesla Roadster: each slide showed the article's photo.
+- **An `imageUrl` that isn't used is said so.** The same model put the address on an HTML slide
+  without `{{image}}`; it was left out without a word. The result now says the picture isn't
+  shown and why, and the tool's description says the slide must have `{{image}}`.
+
+## 0.7.0
+
+- **A slide's picture can come from the web (`imageUrl`).** The slides' policy lets an image come
+  only from a `data:` URL, so a picture the model found in a search couldn't be shown. `imageUrl`
+  is an image file's address: `execute()` fetches it, shows it as a `data:` URL (the policy is
+  unchanged) and saves it under `artifacts/images/` like a generated picture. It works on a picture
+  slide and at an HTML or Markdown slide's `{{image}}`. Only `https` addresses of named sites are
+  fetched, redirects are checked the same way, and only a PNG, JPEG, GIF or WebP picture of at
+  most 5 MB is shown. One that can't be fetched is drawn from `imagePrompt` (or the alt text)
+  instead, and the model is told; with nothing to draw, the slide isn't shown and the model is
+  told why. The result and the record keep the address in `imageUrl`.
+
 ## 0.6.3
 
 - **`& % #` inside `\text{}` are escaped before KaTeX reads the math.** In MulmoChat, a model wrote

package/README.md

@@ -25,14 +25,14 @@ The core entry (`@gui-chat-plugin/sequence`) has the same tools without Views, f
 
 ### What `execute()` needs from its context
 
-|                                      | Used for                                                                                       | Without it                                                                          |
-| ------------------------------------ | ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
-| `app.generateImage(prompt)`          | every picture                                                                                  | "image generation isn't available"                                                  |
-| `app.editImages(prompt, imagePaths)` | a panel drawn from its characters' sheets, a guide step drawn from the step before             | drawn from the prompt alone                                                         |
-| `files.artifacts` (`FileOps`)        | the records, and checking that a reference picture is still there                              | records are kept in memory only (the results say they weren't saved); no references |
-| `userSpokeAt`                        | holding a guide's next step, or a story's next panel after a choice, until the user has spoken | nothing is held                                                                     |
-| `conversationId`                     | keeping each conversation's slideshows, holds and repeats apart                                | one conversation                                                                    |
-| `currentResult`                      | telling "show step 2 again" (it isn't on the screen) from a repeated call (it is)              | a step asked for again is shown again                                               |
+|                                      | Used for                                                                                         | Without it                                                                                                                         |
+| ------------------------------------ | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- |
+| `app.generateImage(prompt)`          | every picture                                                                                    | "image generation isn't available"                                                                                                 |
+| `app.editImages(prompt, imagePaths)` | a panel drawn from its characters' sheets, a guide step drawn from the step before               | drawn from the prompt alone                                                                                                        |
+| `files.artifacts` (`FileOps`)        | the records, saving a picture from the web, and checking that a reference picture is still there | records are kept in memory only (the results say they weren't saved); a picture from the web is shown but not saved; no references |
+| `userSpokeAt`                        | holding a guide's next step, or a story's next panel after a choice, until the user has spoken   | nothing is held                                                                                                                    |
+| `conversationId`                     | keeping each conversation's slideshows, holds and repeats apart                                  | one conversation                                                                                                                   |
+| `currentResult`                      | telling "show step 2 again" (it isn't on the screen) from a repeated call (it is)                | a step asked for again is shown again                                                                                              |
 
 `generateImage` and `editImages` are gui-chat-protocol's `context.app` conventions: they save each
 picture as `artifacts/images/…` and return its path in `data.imagePath`. `currentResult` is compared
@@ -67,8 +67,8 @@ it is drawn.
 
 ## Slides that aren't pictures
 
-A slide is a generated picture (`imagePrompt`), a chart (`chart`), Markdown (`markdown`) or HTML
-(`html`), the model's choice per slide: a picture for a scene or an object; a chart for numbers to
+A slide is a generated picture (`imagePrompt`), a picture from the web (`imageUrl`, below), a chart
+(`chart`), Markdown (`markdown`) or HTML (`html`), the model's choice per slide: a picture for a scene or an object; a chart for numbers to
 compare, a trend or proportions; Markdown for equations (TeX math), definitions, short lists and
 small tables; HTML for designed layouts, comparisons, timelines and diagrams. Picture models draw
 words, numbers and math badly. A slide sent with more than one is one kind: `html` first, then
@@ -89,6 +89,35 @@ Without `{{image}}`, an `imagePrompt` sent with `html` or `markdown` is left out
 without `imagePrompt` is drawn from the picture's alt text, or the slide's title when it has none. When the picture can't be drawn, the slide is shown without it
 and the model is told.
 
+### A picture from the web
+
+`imageUrl` is the address of an image file the model found (in a search, or from the user), or of
+a web page about the thing (an article from a search), whose preview picture (`og:image`, else
+`twitter:image`) is shown: models have pages' addresses far more often than image files'. It is
+shown instead of a generated picture: on a picture slide, or at an HTML or Markdown slide's `{{image}}`
+in place of the one `imagePrompt` would draw. `execute()` fetches it and returns it as a `data:`
+URL (`imageData`), so the slides' policy (`img-src data: blob:`) is unchanged and a slide still
+can't send anything out in an image URL. It is saved as the hosts save their own pictures,
+`artifacts/images/<YYYY>/<MM>/<id>.<ext>` (`imagePath`), so a movie of the slideshow, or the next
+step of a guide drawn from it, can use it; the result and the record keep the address in
+`imageUrl`.
+
+Since MulmoChat fetches it on its server, from an address the model chose: only `https`, no IP
+addresses and no local names (`localhost`, `*.local`, a name without a dot…), every redirect (at
+most 3) and a page's preview picture checked the same way (a preview that is a page in turn isn't
+followed), 10 seconds and 5 MB at most, and only bytes that are a PNG, JPEG, GIF
+or WebP picture, whatever the server says its type is. A public name that resolves to a private
+address isn't caught: that needs a DNS lookup, which a page can't make; what comes back is shown
+only if it is a picture. MulmoChat's server runs on the user's machine, so it reaches what their
+browser reaches; a host whose server runs elsewhere should fetch on a network that has nothing to
+reach. MulmoGlass fetches it in the page, where a site that doesn't allow CORS
+can't be read.
+
+An `imageUrl` on a slide with no place for a picture (an HTML slide without `{{image}}`, a chart)
+isn't fetched, and the model is told. A picture that can't be fetched is drawn instead, from `imagePrompt` (an HTML or Markdown slide's
+from its alt text or title, as without `imageUrl`), and the model is told; a picture slide with no
+`imagePrompt` isn't shown, and the model is told why, so it can give another address.
+
 ### Markdown slides
 
 `markdown` is GitHub-flavored Markdown with TeX math between `$…$` (inline) or `$$…$$` (a
@@ -150,7 +179,7 @@ that an HTML step is on the screen; a guide step asked for again is then shown a
 
 Saved through `files.artifacts`, with their pictures' paths:
 
-- `slideshows/<id>.json`: `{ id, title, mode, totalSlides, slides: { "1": { title, imagePrompt, imagePath } } }`; a slide that isn't a picture has `html` (an HTML slide), `html` and `markdown` (a Markdown slide) or `chart` (a chart slide), and an `imagePrompt` of `""` and no `imagePath` unless its `html` places a picture at `{{image}}`
+- `slideshows/<id>.json`: `{ id, title, mode, totalSlides, slides: { "1": { title, imagePrompt, imagePath, imageUrl } } }` (`imageUrl` for a picture from the web); a slide that isn't a picture has `html` (an HTML slide), `html` and `markdown` (a Markdown slide) or `chart` (a chart slide), and an `imagePrompt` of `""` and no `imagePath` unless its `html` places a picture at `{{image}}`
 - `storyboards/<id>.json`: `{ id, title, style, totalPanels, interactive, characters: [{ name, description, imagePath }], panels: { "1": { caption, characters, imagePrompt, imagePath, choices } } }`
 
 IDs are 12 hex digits. The results name them, so a later tool call can refer to a slideshow or a

package/dist/core/definitions.d.ts

@@ -17,6 +17,12 @@ export interface SlideArgs {
     /** The picture's prompt: a picture slide's, or the picture an HTML or
      *  Markdown slide places at {{image}}; "" for a slide without one. */
     imagePrompt: string;
+    /** A picture from the web, shown instead of a drawn one: a picture
+     *  slide's, or the one an HTML or Markdown slide places at {{image}}. */
+    imageUrl?: string;
+    /** An imageUrl sent for a slide with no place for a picture (an HTML slide
+     *  without {{image}}, a chart): not shown, and the model is told. */
+    unplacedImageUrl?: string;
     /** An HTML slide's body. */
     html?: string;
     /** A Markdown slide's text, with TeX math. */
@@ -25,7 +31,7 @@ export interface SlideArgs {
     chart?: SlideChart;
     mode: SlideMode;
 }
-export declare const SLIDE_ARGS_ERROR = "presentSlide needs slide and totalSlides (whole numbers, slide from 1 to totalSlides), and one of imagePrompt, html, markdown or chart";
+export declare const SLIDE_ARGS_ERROR = "presentSlide needs slide and totalSlides (whole numbers, slide from 1 to totalSlides), and one of imagePrompt, imageUrl, html, markdown or chart";
 export declare const SLIDE_HTML_TOO_LONG = "presentSlide's slide is too long: keep its html, markdown or chart under 60000 characters";
 export declare const SLIDE_CHART_INVALID: string;
 /** The slide arguments, or what is wrong with them (said to the model). A
@@ -48,6 +54,8 @@ export interface Slideshow {
         /** A picture slide's picture, or the one an HTML or Markdown slide
          *  places at {{image}} (withSlideImage). */
         imagePath?: string;
+        /** Where that picture came from, when it is from the web. */
+        imageUrl?: string;
         /** An HTML slide's body (slideHtmlDocument makes it a page); for a
          *  Markdown slide, the HTML made of it. Keeps {{image}}. */
         html?: string;
@@ -62,6 +70,8 @@ export interface Slideshow {
 export interface SlideData {
     imageData?: string;
     imagePath?: string;
+    /** Where the picture came from, when it is from the web. */
+    imageUrl?: string;
     html?: string;
     markdown?: string;
     chart?: SlideChart;

package/dist/core/webImage.d.ts

@@ -0,0 +1,26 @@
+import type { FileOps } from "gui-chat-protocol";
+export declare const WEB_IMAGE_TIMEOUT_MS = 10000;
+export declare const MAX_WEB_IMAGE_BYTES: number;
+/** Why a URL isn't one to fetch, or null when it is. */
+export declare function webImageUrlProblem(url: URL): string | null;
+/** The picture's type and file extension from its first bytes: the server's
+ *  Content-Type isn't trusted. */
+export declare function webImageType(bytes: Uint8Array): {
+    mime: string;
+    extension: string;
+} | null;
+/** The address of a page's preview picture (og:image, twitter:image), from
+ *  its <meta> tags, or null. Read as text: no DOM where execute() runs on a
+ *  server. */
+export declare function pagePreviewImage(html: string, pageUrl: string): string | null;
+/** A picture from the web, saved under artifacts/images/ when the host has
+ *  files, or why it couldn't be fetched. A page's address gives its preview
+ *  picture, whose address is `source`; a page's preview that is a page
+ *  isn't followed. */
+export declare function fetchWebImage(address: string, artifacts: FileOps | undefined, fromPage?: boolean): Promise<{
+    imageData: string;
+    imagePath?: string;
+    source?: string;
+} | {
+    error: string;
+}>;