@sable-ai/sdk-core 0.1.4 → 0.1.5

package/README.md

@@ -1,88 +1,332 @@
-# @sable-ai/sdk-core
+# Adding a Sable agent to your website
 
-Browser runtime for Sable: joins a LiveKit room and lets users talk to an
-agent worker with optional vision + browser-bridge tools. Ships as a
-Stripe.js-style split bundle:
+> **Beta.** The platform UI, the SDK API, and the error codes described below
+> may still change before general availability. If you're integrating now,
+> pin the SDK to an exact version.
 
-| File | Size (gzipped) | When it loads |
-| --- | --- | --- |
-| `sable.js` (loader) | **~530 B** | Page load |
-| `sable-core.mjs` (full SDK, livekit inlined) | ~150 KB | First `Sable.start()` call |
+Sable is a voice + vision agent that lives inside your users' browsers. Drop
+a single `<script>` tag on the pages where you want it, call `Sable.start()`,
+and an agent you configured on the Sable platform can talk to your user and
+see what they see — no iframes, no overlays you don't control, no changes to
+your backend.
+
+This guide walks you through the end-to-end integration as a web engineer at
+a company that already has a Sable account.
+
+---
+
+## 1. Create an agent on the Sable platform
+
+1. Sign in to <https://platform.withsable.com>.
+2. **Agents → New agent**. Give it a name and pick a template (voice-only,
+   voice + vision, etc).
+3. Write the agent's system prompt and configure its tools. This is the same
+   agent config you'd use in any Sable channel — the SDK just exposes it in
+   your user's browser instead of a Sable-hosted call page.
+4. Publish the agent. You'll land on the agent detail page.
+
+## 2. Add your domain to the allowlist
+
+Still on the agent detail page:
+
+1. Open the **Web SDK** tab.
+2. Under **Allowed domains**, add every origin where you intend to load the
+   SDK. Exact hostnames only — `example.com`, `www.example.com`,
+   `app.example.com`. Wildcards are supported as a leading `*.`
+   (e.g. `*.example.com` covers every subdomain).
+3. Save.
+
+Why this matters: the SDK is loaded inside the user's browser and talks
+directly to the Sable API from the page. Sable rejects requests from any
+origin that isn't on this list, so adding your domain here is the gate
+that lets the script tag on your site actually connect. **If you forget
+this step, you'll see a CORS error in the devtools console and nothing
+else — the call never starts.**
+
+Allowed domains are scoped to the agent. An agent for your marketing site
+and an agent for your product dashboard can have different allowlists.
+
+## 3. Copy your public key
+
+On the same **Web SDK** tab, under **Public key**, you'll see a value like:
+
+```
+pk_live_8f3a9c2d4e5b6a7c
+```
 
-Pages that never open a session pay only the loader cost.
+This is the key you'll paste into your page. It's a **publishable** key —
+safe to ship in client-side code, visible in devtools, not a secret. It
+identifies the agent and is validated by Sable against the allowed-domains
+list above, so even if someone copies it onto their own site it won't work
+unless their origin is on your allowlist.
 
-## Install
+If you ever need to rotate it (e.g. the key shows up somewhere you don't
+want it), click **Rotate key** and update the snippet on your site. The old
+key is immediately invalidated.
 
-### Script tag (recommended for no-build sites + extensions)
+## 4. Load the SDK
+
+You have two options. Both expose the same API; pick whichever fits your
+stack.
+
+### Option A: Script tag (works everywhere)
+
+Add this to every page you want the agent to be available on:
 
 ```html
-<script src="https://sdk.withsable.com/v0.1.4/sable.js"></script>
+<script src="https://sdk.withsable.com/v1/sable.js" async></script>
 ```
 
-The loader installs `window.Sable` synchronously. On the first
-`Sable.start()` call it lazy-imports `sable-core.mjs` from the same CDN
-path and forwards the call.
+That's a **~530 B gzipped loader stub** — no dependencies, no
+stylesheet, no livekit, no vision runtime. The loader installs a single
+global (`window.Sable`) and does nothing else until you call `start()`:
+no network requests, no microphone prompt, no DOM mutations. Idle cost
+to your site is effectively zero.
+
+On the first `Sable.start()` call, the loader dynamic-imports the full
+SDK (`sable-core.mjs`, ~150 KB gzipped with livekit-client inlined) from
+the **same CDN path** it was served from — so loading
+`https://sdk.withsable.com/v0.1.5/sable.js` always pulls
+`https://sdk.withsable.com/v0.1.5/sable-core.mjs`, and version cohesion
+is automatic. Pages that never start a session pay only the loader cost;
+pages that do start a session pay the core download exactly once, cached
+aggressively for the lifetime of the version pin.
 
-Available CDN pins:
+The script is served from Sable's CDN (`sdk.withsable.com`, Cloudflare
+Pages), cached at the edge, and versioned. Three path conventions:
 
-| Path | Cache | Use case |
+| URL | Cache | Use case |
 | --- | --- | --- |
-| `https://sdk.withsable.com/v0.1.4/sable.js` | 1 year, immutable | **Recommended for production** — exact version pin |
-| `https://sdk.withsable.com/v1/sable.js` | 1 hour | Latest 0.x (accepts patch releases automatically) |
+| `https://sdk.withsable.com/v0.1.5/sable.js` | 1 year, immutable | **Recommended for production** — exact version pin |
+| `https://sdk.withsable.com/v1/sable.js` | 1 hour | Latest `0.x` — accepts patch releases automatically |
 | `https://sdk.withsable.com/latest/sable.js` | 5 minutes | Demos and smoke tests only |
 
-### npm (ESM)
+`v1` is a stable major line — the API won't change without a major
+bump. Pin to an exact version (`v0.1.5`) if you want bit-for-bit
+reproducibility.
+
+### Option B: npm package (React, Vue, Svelte, Next, etc.)
 
 ```bash
-bun add @sable-ai/sdk-core
+npm install @sable-ai/sdk-core
 ```
 
 ```js
 import Sable from "@sable-ai/sdk-core";
-// Importing also installs window.Sable (first-write-wins), so mixed
-// script-tag + npm usage on the same page stays coherent.
+
+await Sable.start({ publicKey: "pk_live_..." });
 ```
 
-The npm build keeps `livekit-client` as an external peer so your bundler
-can dedupe it.
+The npm package is a standalone ESM bundle (livekit-client declared as a
+regular dependency so your bundler can dedupe it) — it does **not**
+fetch from the CDN. Importing the module also installs `window.Sable`
+with first-write-wins semantics, so mixing the script tag and the npm
+package on the same page is safe: whichever loads first installs the
+global and the other becomes a no-op. You get TypeScript types and
+bundler-integrated imports. Use this path if you want typed autocomplete
+and a bundler-controlled dependency graph.
+
+## 5. Start a session from your app code
 
-## Usage
+When you want the agent to actually connect — on a button click, on page
+load, when the user opens a help menu, whatever — call:
 
 ```js
 await window.Sable.start({
-  agentPublicId: "agt_...",
-  apiUrl: "https://sable-api-gateway-9dfmhij9.wl.gateway.dev", // optional, default
-  nickelRegion: "us-east1", // optional
+  publicKey: "pk_live_8f3a9c2d4e5b6a7c",
+
+  // Optional — what the agent can see.
+  vision: {
+    enabled: true,
+    // How frames are produced. Defaults to the built-in wireframe renderer.
+    // Discriminated on `type`:
+    //   { type: "wireframe", features: { includeImages?: boolean } }
+    //   { type: "fn", captureFn: () => HTMLCanvasElement | ImageBitmap }
+    frameSource: {
+      type: "wireframe",
+      rate: 2, // frames per second; default 2
+      features: {
+        includeImages: true, // include rendered images, not just layout boxes
+      },
+    },
+  },
+
+  // Optional — implementations for methods the agent can RPC into your page.
+  // The SDK defines a small set of "UI stub" methods (e.g. showMessage,
+  // highlightElement). You can override any of them here, and add your own
+  // for agent tools specific to your app. Anything you pass becomes callable
+  // by the agent.
+  runtime: {
+    showMessage: (text) => myToast.show(text),
+    highlightElement: (selector) => { /* ... */ },
+    openDocument: (docId) => router.push(`/docs/${docId}`),
+  },
+
+  // Optional — arbitrary context forwarded to the agent at session start.
+  // Surfaces verbatim in the agent's initial prompt.
+  context: {
+    userId: currentUser.id,
+    userName: currentUser.name,
+    currentPage: "dashboard",
+  },
 });
+```
 
-// ... user talks to the agent ...
+`Sable.start()` returns a promise that resolves when the mic is live and
+the agent has greeted the user. It rejects if the public key is invalid,
+the origin isn't on the allowlist, or the user denies microphone access.
 
+To end the session:
+
+```js
 await window.Sable.stop();
 ```
 
-`Sable.on(event, handler)` subscriptions registered before `start()`
-resolves are buffered by the loader and flushed onto the real session
-once `sable-core.mjs` finishes loading.
+You can call `start()` and `stop()` as many times as you like during a page
+lifetime. Only one session can be active at a time.
+
+## 6. React to session events (optional)
+
+If you want to, say, show your own UI when the agent is talking, subscribe
+to events:
+
+```js
+window.Sable.on("session:started", () => { ... });
+window.Sable.on("session:ended", (reason) => { ... });
+window.Sable.on("agent:speaking", (speaking) => { ... });
+window.Sable.on("user:speaking", (speaking) => { ... });
+window.Sable.on("error", (err) => { ... });
+```
+
+All events are fire-and-forget; the SDK does not care whether you subscribe.
+
+---
+
+## Reference: `Sable.start()` options
+
+| Option | Type | Default | Description |
+| --- | --- | --- | --- |
+| `publicKey` | `string` | **required** | The `pk_live_*` key from the platform. |
+| `vision.enabled` | `boolean` | `false` | Whether to publish a video track of the page to the agent. |
+| `vision.frameSource` | `FrameSource` | `{ type: "wireframe" }` | Where frames come from. Either the built-in wireframe renderer or a custom function (see below). |
+| `vision.frameSource.rate` | `number` | `2` | Capture rate in frames per second. Applies to both `wireframe` and `fn`. Higher = more responsive agent vision, more bandwidth. |
+| `vision.frameSource.features.includeImages` | `boolean` | `true` | (Wireframe only.) Include rendered images, not just layout boxes. Set to `false` to ship only layout boxes (lower bandwidth). |
+| `vision.frameSource.captureFn` | `() => HTMLCanvasElement \| ImageBitmap` | — | (Required when `type: "fn"`.) Called at `rate` Hz. Useful for feeding the agent a custom canvas (e.g. a 3D scene, a video element, a WebGL surface). |
+| `runtime` | `Record<string, Function>` | `{}` | Implementations for UI-stub methods the agent can call, plus any additional methods you want to expose as agent tools. |
+| `context` | `Record<string, unknown>` | `{}` | Forwarded verbatim to the agent at session start. Appears in the agent's initial prompt. |
+
+## Reference: errors
+
+| Code | Cause | Fix |
+| --- | --- | --- |
+| `SABLE_INVALID_KEY` | Public key doesn't exist or was rotated. | Copy the current key from the platform. |
+| `SABLE_ORIGIN_NOT_ALLOWED` | Page's origin isn't on the agent's allowlist. | Add the exact origin in the platform's Web SDK → Allowed domains. |
+| `SABLE_MIC_DENIED` | User denied microphone permission. | Prompt them again, or show a message explaining why voice is needed. |
+| `SABLE_RATE_LIMITED` | Too many sessions started from this origin in a short window. | Back off and retry. |
+| `SABLE_NETWORK` | Couldn't reach the Sable API. | Usually transient — retry with backoff. |
+
+---
+
+## FAQ
+
+**Do I need a backend integration?**
+No. The SDK talks directly to the Sable API from the page. There's no
+webhook to install, no server-to-server auth, no token exchange you have to
+implement. The public key + allowed-domains check is the entire trust model.
+
+**Is the public key a secret?**
+No — it's designed to be shipped in client-side code. The security boundary
+is the allowed-domains list, not the key itself. Someone who copies your
+key to their own site can't use it unless their origin is also on your
+allowlist.
 
-## Architecture
+**Script tag or npm package — which should I use?**
+Either. Use the script tag for static HTML or if you want to avoid
+bundler configuration — you get the Stripe.js-style split bundle
+(~530 B on page load, core lazy-loaded from the CDN on first
+`Sable.start()`). Use the npm package if you want TypeScript types and a
+bundler-integrated, self-contained dependency graph — the package is a
+standalone ESM build that does not fetch from the CDN. Mixing on the
+same page is safe: both install `window.Sable` with first-write-wins
+semantics, so whichever runs first wins and the other becomes a no-op.
 
-The loader is a tiny IIFE proxy (`src/loader.ts`) that:
+**Why a global (`window.Sable`) instead of an import-only API?**
+To keep the drop-in story honest. A single `<script>` tag works in every
+web framework — React, Vue, Svelte, Next, Rails, plain HTML — without
+bundler setup. The npm package also installs the same global on import,
+so mixed usage stays coherent.
 
-1. Captures `document.currentScript.src` synchronously at top-level
-   (the only time `currentScript` is defined).
-2. Installs `window.Sable` with stub `start/stop/on` methods.
-3. On first `start()`, resolves `new URL("./sable-core.mjs", scriptSrc)`
-   and dynamically imports it — so `/v0.1.4/sable.js` always pulls
-   `/v0.1.4/sable-core.mjs`. Version cohesion is automatic.
-4. Memoises the import and flushes any queued `on()` subscriptions.
+**Does the SDK render anything by default?**
+The SDK itself is headless — it handles voice, vision, and agent
+communication but renders nothing on its own. A default UI (mic button,
+avatar, agent-driven overlays) can be enabled via a separate package, or
+you can build your own using the event API and your own component library.
 
-See `docs/superpowers/specs/` and `infra/cdn/README.md` for more.
+**Does the agent see my users' data?**
+Only what's on the page at the moment the session is active, and only if
+`vision.enabled` is `true`. The wireframe is generated client-side and
+streamed as a video track; Sable never scrapes, indexes, or stores page
+content server-side. Microphone audio is end-to-end within the voice
+session.
 
-## Local development
+**What happens if the user navigates away?**
+The session ends. The SDK tears down the connection, stops the microphone,
+and unmounts anything it mounted. A fresh `Sable.start()` on the next page
+creates a new session.
+
+**How do I test this locally?**
+Add `http://localhost:3000` (or whatever port you use) to the allowed
+domains list alongside your production origin. The SDK treats localhost
+the same as any other origin — there's no special dev bypass.
+
+**Can I self-host the script?**
+Not recommended. The CDN copy is versioned, cached globally, and updated
+automatically. If you really need to — e.g. an air-gapped customer —
+you need to host **both** files side-by-side under the same path:
+`sable.js` (the loader) and `sable-core.mjs` (the full SDK). The loader
+resolves the core via `new URL("./sable-core.mjs", document.currentScript.src)`,
+so they must live in the same directory. Contact support for details.
+
+**Does the SDK work with strict CSP (`script-src` nonces, etc.)?**
+Yes. Apply your nonce to the `<script>` tag as usual. The loader
+dynamic-imports `sable-core.mjs` from the **same origin** it was served
+from (`sdk.withsable.com` by default), so your CSP needs to allow that
+origin in `script-src` — typically `script-src 'self' https://sdk.withsable.com`
+if you load from the public CDN. No third-party origins are contacted
+at runtime beyond `sdk.withsable.com` (for the core bundle) and
+`sable-api-gateway-9dfmhij9.wl.gateway.dev` (for the session — plus
+the LiveKit WebSocket origin returned by the session endpoint). If
+you self-host `sable.js` + `sable-core.mjs` under your own origin,
+`'self'` covers the script side; the API + LiveKit origins still
+need to be in `connect-src`.
+
+---
+
+## Development
+
+> This section is for contributors to `@sable-ai/sdk-core` itself. If
+> you're integrating Sable into your site, stop here.
+
+The package source lives at
+[`sable-inc/js-sdk`](https://github.com/sable-inc/js-sdk) under
+`packages/sdk-core`. Build:
 
 ```bash
 bun install
 bun run --filter @sable-ai/sdk-core build
+```
+
+This produces three artifacts in `dist/`:
+
+| Artifact | Entry | Consumer |
+| --- | --- | --- |
+| `dist/sable.iife.js` | `src/loader.ts` (minified IIFE) | CDN script tag |
+| `dist/sable-core.mjs` | `src/index.ts` (minified ESM, livekit inlined) | CDN lazy-import from the loader |
+| `dist/esm/index.js` + `dist/types/` | `src/index.ts` (ESM, livekit external) | `npm install @sable-ai/sdk-core` |
+
+### Local smoke test
+
+```bash
 python3 -m http.server 5173 --directory packages/sdk-core
 ```
 
@@ -92,14 +336,14 @@ ID, click Start. Grant mic permission when prompted.
 **Only `http://localhost:*` origins work against the hosted API** —
 `sable-api`'s CORS policy does not yet allow arbitrary origins.
 
-## Releasing
+### Releasing
 
-Tagging `sdk-core-v<x.y.z>` triggers `.github/workflows/release-sdk-core.yml`
-which:
+Tagging `sdk-core-v<x.y.z>` triggers
+`.github/workflows/release-sdk-core.yml`, which:
 
-1. Runs typecheck + tests + build.
+1. Runs typecheck + tests + `bun run build`.
 2. Publishes to npm with OIDC trusted-publisher provenance.
-3. Stages `dist/sable.iife.js` + `dist/sable-core.mjs` under
-   `/v<version>/`, `/v1/`, and `/latest/` via `infra/cdn/stage.sh`.
+3. Stages `dist/sable.iife.js` + `dist/sable-core.mjs` under `/v<version>/`,
+   `/v1/`, and `/latest/` via `infra/cdn/stage.sh`.
 4. Deploys the staged directory to Cloudflare Pages
    (`sable-sdk-cdn` project, custom domain `sdk.withsable.com`).

package/dist/esm/index.js

@@ -2001,7 +2001,7 @@ async function publishCanvasAsVideoTrack(room, lib, canvas, fps) {
 var DEFAULT_FRAME_SOURCE = {
   type: "wireframe",
   rate: 2,
-  features: { includeImages: false }
+  features: { includeImages: true }
 };
 async function startVision(args) {
   const source = args.options.frameSource ?? DEFAULT_FRAME_SOURCE;

package/dist/sable-core.mjs

@@ -1483,4 +1483,4 @@ console.log('  await new Wireframe().toDataURL()     — base64 for LLM');
         });
         ${is};
         return Wireframe;
-      })()`);return _n}var vn=null;function ru(){if(!vn)vn=(0,eval)(`(${ss})`);return vn}async function cs(){let t=ru()();return{screenshot_jpeg_b64:(await new(Dl())(document.body,{}).toDataURL()).replace(/^data:[^,]+,/,""),elements:t,viewport:{width:window.innerWidth,height:window.innerHeight},url:window.location.href}}async function hs(){let t=()=>new Promise((e)=>requestAnimationFrame(()=>requestAnimationFrame(()=>e())));await t(),await new Promise((e)=>{let r=performance.now(),o=performance.now(),s=new MutationObserver(()=>{o=performance.now()});s.observe(document.documentElement,{subtree:!0,childList:!0,attributes:!0,characterData:!0});let c=()=>{let i=performance.now();if(i-o>=30){s.disconnect(),e();return}if(i-r>=30){s.disconnect(),e();return}requestAnimationFrame(c)};requestAnimationFrame(c)}),await t()}function Qe(t,e){return async(l)=>{let n={};try{n=l.payload?JSON.parse(l.payload):{}}catch(r){console.warn(`[Sable] ${t}: bad JSON payload`,r)}try{let r=await e(n);return JSON.stringify(r??{})}catch(r){let o=r instanceof Error?r.message:String(r);return console.warn(`[Sable] ${t}: handler error`,o),JSON.stringify({error:o})}}}function fs(t){t.registerRpcMethod("browser.execute_action",Qe("browser.execute_action",async(e)=>{let l=e.action;if(!l||typeof l!=="object")throw Error("execute_action: missing action");return await os(l),{}})),t.registerRpcMethod("browser.get_dom_state",Qe("browser.get_dom_state",async()=>cs())),t.registerRpcMethod("browser.get_url",Qe("browser.get_url",async()=>({url:window.location.href}))),t.registerRpcMethod("browser.get_viewport",Qe("browser.get_viewport",async()=>({width:window.innerWidth,height:window.innerHeight}))),t.registerRpcMethod("browser.verify_selector",Qe("browser.verify_selector",async(e)=>{let l=typeof e.selector==="string"?e.selector:"",n=!1;try{n=!!document.querySelector(l)}catch{n=!1}return{matches:n}})),t.registerRpcMethod("browser.settle",Qe("browser.settle",async()=>{return await hs(),{}})),console.log("[Sable] browser bridge RPCs registered")}var ou=2;function su(t){let e=typeof t==="number"&&t>0?t:ou;return Math.max(1,Math.round(1000/e))}function iu(t){let e=Math.max(1,window.innerWidth),l=Math.max(1,window.innerHeight);if(t.width!==e||t.height!==l)t.width=e,t.height=l}function us(t,e){let l=e.getContext("2d",{alpha:!1});if(!l)return console.warn("[Sable] frame source: 2d context unavailable"),()=>{};let n=su(t.rate),r=!1,o,s=!1,c=async()=>{if(r)return;if(!s){s=!0;try{if(iu(e),t.type==="wireframe"){let i=t.features?.includeImages===!0,f=new(Dl())(document.body,{images:i}),{canvas:u}=await f.capture();l.fillStyle="#ffffff",l.fillRect(0,0,e.width,e.height),l.drawImage(u,0,0,e.width,e.height)}else if(t.type==="fn"){let i=t.captureFn();l.fillStyle="#ffffff",l.fillRect(0,0,e.width,e.height),l.drawImage(i,0,0,e.width,e.height)}}catch(i){console.warn("[Sable] frame source tick failed",i)}finally{s=!1}}if(!r)o=setTimeout(c,n)};return c(),()=>{if(r=!0,o!==void 0)clearTimeout(o)}}async function ds(t,e,l,n){let o=l.captureStream(n).getVideoTracks();if(o.length===0)throw Error("canvas.captureStream produced no video tracks");let s=o[0],c=new e.LocalVideoTrack(s,void 0,!0),i=await t.localParticipant.publishTrack(c,{source:e.Track.Source.ScreenShare,name:"browser"});return console.log("[Sable] vision track published",{trackSid:i.trackSid,fps:n}),async()=>{try{await t.localParticipant.unpublishTrack(c,!0)}catch(h){console.warn("[Sable] vision unpublishTrack failed",h)}try{s.stop()}catch{}console.log("[Sable] vision track stopped")}}var cu={type:"wireframe",rate:2,features:{includeImages:!1}};async function as(t){let e=t.options.frameSource??cu,l=typeof e.rate==="number"&&e.rate>0?e.rate:2,n=document.createElement("canvas");n.width=Math.max(1,window.innerWidth),n.height=Math.max(1,window.innerHeight);let r=us(e,n),o;try{o=await ds(t.room,t.lib,n,l)}catch(s){throw r(),s}return{canvas:n,stop:async()=>{r(),await o()}}}var tr="0.1.4";function er(){try{let t=window.localStorage?.getItem("sable:debug:panel");if(!t)return{};let e=JSON.parse(t);return typeof e==="object"&&e!==null?e:{}}catch{return{}}}function bs(t){try{window.localStorage?.setItem("sable:debug:panel",JSON.stringify(t))}catch{}}function xs(t){if(t)return!0;try{if(new URL(window.location.href).searchParams.get("sable-debug")==="1")return!0}catch{}try{if(window.localStorage?.getItem("sable:debug")==="1")return!0}catch{}return!1}function gs(t){let e=er(),l=document.createElement("div");if(l.setAttribute("data-sable-debug","vision"),Object.assign(l.style,{position:"fixed",width:"240px",zIndex:"2147483647",background:"#111",color:"#ddd",border:"1px solid #444",borderRadius:"8px",font:"11px/1.3 system-ui, sans-serif",boxShadow:"0 8px 24px rgba(0,0,0,.4)",pointerEvents:"none",userSelect:"none",overflow:"hidden"}),typeof e.left==="number"&&typeof e.top==="number")l.style.left=`${e.left}px`,l.style.top=`${e.top}px`;else l.style.right="12px",l.style.top="12px";let n=document.createElement("div");Object.assign(n.style,{display:"flex",alignItems:"center",justifyContent:"space-between",gap:"6px",padding:"6px 8px",background:"#1a1a1a",borderBottom:"1px solid #333",cursor:"move",pointerEvents:"auto"});let r=document.createElement("div");r.textContent="sable: agent vision",Object.assign(r.style,{opacity:"0.75",fontWeight:"600",flex:"1",pointerEvents:"none"}),n.appendChild(r);let o=document.createElement("button");o.setAttribute("aria-label","Minimize vision panel"),Object.assign(o.style,{background:"transparent",color:"#ddd",border:"1px solid #444",borderRadius:"4px",width:"20px",height:"20px",cursor:"pointer",fontSize:"12px",lineHeight:"1",padding:"0",pointerEvents:"auto"}),n.appendChild(o),l.appendChild(n);let s=document.createElement("div");Object.assign(s.style,{padding:"6px",background:"#111",pointerEvents:"none"}),t.style.width="100%",t.style.height="auto",t.style.display="block",t.style.background="#fff",t.style.borderRadius="4px",t.style.pointerEvents="none",s.appendChild(t),l.appendChild(s);let c=!!e.minimized,i=()=>{s.style.display=c?"none":"block",o.textContent=c?"▢":"–",o.setAttribute("aria-label",c?"Restore vision panel":"Minimize vision panel")};i(),o.addEventListener("click",(x)=>{x.stopPropagation(),c=!c,i(),bs({...er(),minimized:c})});let h=null,f=(x)=>{if(x.target instanceof HTMLElement&&x.target.closest("button"))return;let g=l.getBoundingClientRect();h={offsetX:x.clientX-g.left,offsetY:x.clientY-g.top},l.style.left=`${g.left}px`,l.style.top=`${g.top}px`,l.style.right="auto",l.style.bottom="auto",n.setPointerCapture(x.pointerId),x.preventDefault()},u=(x)=>{if(!h)return;let{innerWidth:g,innerHeight:w}=window,p=l.offsetWidth,y=x.clientX-h.offsetX,$=x.clientY-h.offsetY;y=Math.min(Math.max(y,-p+48),g-48),$=Math.min(Math.max($,0),w-24),l.style.left=`${y}px`,l.style.top=`${$}px`},d=(x)=>{if(!h)return;h=null;try{n.releasePointerCapture(x.pointerId)}catch{}let g=parseFloat(l.style.left)||0,w=parseFloat(l.style.top)||0;bs({...er(),left:g,top:w})};return n.addEventListener("pointerdown",f),n.addEventListener("pointermove",u),n.addEventListener("pointerup",d),n.addEventListener("pointercancel",d),document.body.appendChild(l),console.log("[Sable] debug vision panel mounted",{minimized:c,restoredPosition:typeof e.left==="number"&&typeof e.top==="number"}),()=>{try{n.removeEventListener("pointerdown",f),n.removeEventListener("pointermove",u),n.removeEventListener("pointerup",d),n.removeEventListener("pointercancel",d),l.remove()}catch{}}}var Nx=5,Tf=500;function Mx(t){let e=t.remoteParticipants?Array.from(t.remoteParticipants.values()):[];return e.find((n)=>typeof n.identity==="string"&&n.identity.startsWith("agent"))?.identity??e[0]?.identity??null}async function Hx(t){for(let e=1;e<=Nx;e++){let l=Mx(t);if(!l){console.warn("[Sable] sendUiReady: no agent participant yet",{attempt:e}),await new Promise((n)=>setTimeout(n,Tf));continue}try{await t.localParticipant.performRpc({destinationIdentity:l,method:"uiReady",payload:JSON.stringify({timestamp:Date.now()})}),console.log("[Sable] uiReady sent",{identity:l,attempt:e});return}catch(n){console.warn("[Sable] uiReady RPC failed",{attempt:e,err:n}),await new Promise((r)=>setTimeout(r,Tf))}}console.error("[Sable] uiReady: exhausted retries — agent will not greet")}class _o{version=tr;emitter=new kn;activeRoom=null;visionHandle=null;unmountDebugPanel=null;on(t,e){return this.emitter.on(t,e)}async start(t){if(this.activeRoom)throw Error("Sable already started; call stop() first");let e=t.publicKey??t.agentPublicId;if(!e)throw Error("Sable.start: `publicKey` is required");let l=t.apiUrl??ts;console.log("[Sable] fetching connection details",{apiUrl:l});let n=await es({apiUrl:l,publicKey:e});console.log("[Sable] connection details received",{roomName:n.roomName,participantName:n.participantName});let r=await Promise.resolve().then(() => (qf(),jf)),{Room:o,RoomEvent:s,LocalVideoTrack:c,Track:i}=r,h=new o,f={LocalVideoTrack:c,Track:i};if(h.registerRpcMethod("agentReady",async()=>{return console.log("[Sable] RPC agentReady received"),Hx(h),JSON.stringify({success:!0})}),fs(h),rs(h,t.runtime),this.wireRoomEvents(h,s),await h.connect(n.serverUrl,n.participantToken),await h.localParticipant.setMicrophoneEnabled(!0),this.activeRoom=h,t.vision?.enabled)try{if(this.visionHandle=await as({room:h,lib:f,options:t.vision}),xs(t.debug))this.unmountDebugPanel=gs(this.visionHandle.canvas)}catch(u){console.warn("[Sable] failed to start vision",u)}console.log("[Sable] session live",{roomName:n.roomName,participantName:n.participantName}),this.emitter.emit("session:started",{roomName:n.roomName,participantName:n.participantName}),setTimeout(()=>{if(this.activeRoom!==h)return;let u=h,x=(u.remoteParticipants?Array.from(u.remoteParticipants.values()):[]).map((w)=>({identity:w.identity,tracks:w.trackPublications?Array.from(w.trackPublications.values()).map((p)=>({kind:p.kind,subscribed:p.isSubscribed})):[]}));if(!x.some((w)=>w.tracks.some((p)=>p.kind==="audio")))console.warn("[Sable] no remote audio track after 10s — agent worker probably failed to publish. Remote participants:",x)},1e4)}async stop(){let t=this.activeRoom;if(!t)return;if(this.activeRoom=null,this.unmountDebugPanel){try{this.unmountDebugPanel()}catch(e){console.warn("[Sable] debug panel unmount failed",e)}this.unmountDebugPanel=null}if(this.visionHandle){try{await this.visionHandle.stop()}catch(e){console.warn("[Sable] vision stop failed",e)}this.visionHandle=null}try{await t.localParticipant.setMicrophoneEnabled(!1)}catch(e){console.warn("[Sable] setMicrophoneEnabled(false) failed",e)}await t.disconnect(),console.log("[Sable] session ended"),this.emitter.emit("session:ended",{})}wireRoomEvents(t,e){t.on(e.ConnectionStateChanged,(l)=>{console.log("[Sable] ConnectionStateChanged",l)}),t.on(e.Disconnected,(l)=>{if(console.log("[Sable] Disconnected",l),this.activeRoom===t)this.stop().catch((n)=>console.warn("[Sable] stop on disconnect failed",n))}),t.on(e.ParticipantConnected,(l)=>{let n=l;console.log("[Sable] ParticipantConnected",{identity:n.identity,sid:n.sid})}),t.on(e.ParticipantDisconnected,(l)=>{console.warn("[Sable] ParticipantDisconnected",{identity:l.identity})}),t.on(e.TrackSubscribed,(l,n,r)=>{let o=l,s=r;if(console.log("[Sable] TrackSubscribed",{kind:o.kind,participant:s.identity}),o.kind==="audio"&&typeof o.attach==="function"){let c=o.attach();c.setAttribute("data-sable","1"),c.setAttribute("playsinline",""),c.autoplay=!0,document.body.appendChild(c),console.log("[Sable] attached remote audio element")}}),t.on(e.TrackUnsubscribed,(l)=>{let n=l;if(typeof n.detach==="function")for(let r of n.detach())r.remove()}),t.on(e.ActiveSpeakersChanged,(l)=>{let n=l??[],r=n.some((s)=>typeof s.identity==="string"&&s.identity.startsWith("agent")),o=n.some((s)=>typeof s.identity==="string"&&!s.identity.startsWith("agent"));this.emitter.emit("agent:speaking",r),this.emitter.emit("user:speaking",o)})}}var Sn=new _o;function Ff(){if(typeof window>"u")return;if(window.Sable)return;window.Sable=Sn,console.log("[Sable] SDK loaded",Sn.version)}Ff();var Ag=Sn;export{Ag as default,tr as VERSION};
+      })()`);return _n}var vn=null;function ru(){if(!vn)vn=(0,eval)(`(${ss})`);return vn}async function cs(){let t=ru()();return{screenshot_jpeg_b64:(await new(Dl())(document.body,{}).toDataURL()).replace(/^data:[^,]+,/,""),elements:t,viewport:{width:window.innerWidth,height:window.innerHeight},url:window.location.href}}async function hs(){let t=()=>new Promise((e)=>requestAnimationFrame(()=>requestAnimationFrame(()=>e())));await t(),await new Promise((e)=>{let r=performance.now(),o=performance.now(),s=new MutationObserver(()=>{o=performance.now()});s.observe(document.documentElement,{subtree:!0,childList:!0,attributes:!0,characterData:!0});let c=()=>{let i=performance.now();if(i-o>=30){s.disconnect(),e();return}if(i-r>=30){s.disconnect(),e();return}requestAnimationFrame(c)};requestAnimationFrame(c)}),await t()}function Qe(t,e){return async(l)=>{let n={};try{n=l.payload?JSON.parse(l.payload):{}}catch(r){console.warn(`[Sable] ${t}: bad JSON payload`,r)}try{let r=await e(n);return JSON.stringify(r??{})}catch(r){let o=r instanceof Error?r.message:String(r);return console.warn(`[Sable] ${t}: handler error`,o),JSON.stringify({error:o})}}}function fs(t){t.registerRpcMethod("browser.execute_action",Qe("browser.execute_action",async(e)=>{let l=e.action;if(!l||typeof l!=="object")throw Error("execute_action: missing action");return await os(l),{}})),t.registerRpcMethod("browser.get_dom_state",Qe("browser.get_dom_state",async()=>cs())),t.registerRpcMethod("browser.get_url",Qe("browser.get_url",async()=>({url:window.location.href}))),t.registerRpcMethod("browser.get_viewport",Qe("browser.get_viewport",async()=>({width:window.innerWidth,height:window.innerHeight}))),t.registerRpcMethod("browser.verify_selector",Qe("browser.verify_selector",async(e)=>{let l=typeof e.selector==="string"?e.selector:"",n=!1;try{n=!!document.querySelector(l)}catch{n=!1}return{matches:n}})),t.registerRpcMethod("browser.settle",Qe("browser.settle",async()=>{return await hs(),{}})),console.log("[Sable] browser bridge RPCs registered")}var ou=2;function su(t){let e=typeof t==="number"&&t>0?t:ou;return Math.max(1,Math.round(1000/e))}function iu(t){let e=Math.max(1,window.innerWidth),l=Math.max(1,window.innerHeight);if(t.width!==e||t.height!==l)t.width=e,t.height=l}function us(t,e){let l=e.getContext("2d",{alpha:!1});if(!l)return console.warn("[Sable] frame source: 2d context unavailable"),()=>{};let n=su(t.rate),r=!1,o,s=!1,c=async()=>{if(r)return;if(!s){s=!0;try{if(iu(e),t.type==="wireframe"){let i=t.features?.includeImages===!0,f=new(Dl())(document.body,{images:i}),{canvas:u}=await f.capture();l.fillStyle="#ffffff",l.fillRect(0,0,e.width,e.height),l.drawImage(u,0,0,e.width,e.height)}else if(t.type==="fn"){let i=t.captureFn();l.fillStyle="#ffffff",l.fillRect(0,0,e.width,e.height),l.drawImage(i,0,0,e.width,e.height)}}catch(i){console.warn("[Sable] frame source tick failed",i)}finally{s=!1}}if(!r)o=setTimeout(c,n)};return c(),()=>{if(r=!0,o!==void 0)clearTimeout(o)}}async function ds(t,e,l,n){let o=l.captureStream(n).getVideoTracks();if(o.length===0)throw Error("canvas.captureStream produced no video tracks");let s=o[0],c=new e.LocalVideoTrack(s,void 0,!0),i=await t.localParticipant.publishTrack(c,{source:e.Track.Source.ScreenShare,name:"browser"});return console.log("[Sable] vision track published",{trackSid:i.trackSid,fps:n}),async()=>{try{await t.localParticipant.unpublishTrack(c,!0)}catch(h){console.warn("[Sable] vision unpublishTrack failed",h)}try{s.stop()}catch{}console.log("[Sable] vision track stopped")}}var cu={type:"wireframe",rate:2,features:{includeImages:!0}};async function as(t){let e=t.options.frameSource??cu,l=typeof e.rate==="number"&&e.rate>0?e.rate:2,n=document.createElement("canvas");n.width=Math.max(1,window.innerWidth),n.height=Math.max(1,window.innerHeight);let r=us(e,n),o;try{o=await ds(t.room,t.lib,n,l)}catch(s){throw r(),s}return{canvas:n,stop:async()=>{r(),await o()}}}var tr="0.1.4";function er(){try{let t=window.localStorage?.getItem("sable:debug:panel");if(!t)return{};let e=JSON.parse(t);return typeof e==="object"&&e!==null?e:{}}catch{return{}}}function bs(t){try{window.localStorage?.setItem("sable:debug:panel",JSON.stringify(t))}catch{}}function xs(t){if(t)return!0;try{if(new URL(window.location.href).searchParams.get("sable-debug")==="1")return!0}catch{}try{if(window.localStorage?.getItem("sable:debug")==="1")return!0}catch{}return!1}function gs(t){let e=er(),l=document.createElement("div");if(l.setAttribute("data-sable-debug","vision"),Object.assign(l.style,{position:"fixed",width:"240px",zIndex:"2147483647",background:"#111",color:"#ddd",border:"1px solid #444",borderRadius:"8px",font:"11px/1.3 system-ui, sans-serif",boxShadow:"0 8px 24px rgba(0,0,0,.4)",pointerEvents:"none",userSelect:"none",overflow:"hidden"}),typeof e.left==="number"&&typeof e.top==="number")l.style.left=`${e.left}px`,l.style.top=`${e.top}px`;else l.style.right="12px",l.style.top="12px";let n=document.createElement("div");Object.assign(n.style,{display:"flex",alignItems:"center",justifyContent:"space-between",gap:"6px",padding:"6px 8px",background:"#1a1a1a",borderBottom:"1px solid #333",cursor:"move",pointerEvents:"auto"});let r=document.createElement("div");r.textContent="sable: agent vision",Object.assign(r.style,{opacity:"0.75",fontWeight:"600",flex:"1",pointerEvents:"none"}),n.appendChild(r);let o=document.createElement("button");o.setAttribute("aria-label","Minimize vision panel"),Object.assign(o.style,{background:"transparent",color:"#ddd",border:"1px solid #444",borderRadius:"4px",width:"20px",height:"20px",cursor:"pointer",fontSize:"12px",lineHeight:"1",padding:"0",pointerEvents:"auto"}),n.appendChild(o),l.appendChild(n);let s=document.createElement("div");Object.assign(s.style,{padding:"6px",background:"#111",pointerEvents:"none"}),t.style.width="100%",t.style.height="auto",t.style.display="block",t.style.background="#fff",t.style.borderRadius="4px",t.style.pointerEvents="none",s.appendChild(t),l.appendChild(s);let c=!!e.minimized,i=()=>{s.style.display=c?"none":"block",o.textContent=c?"▢":"–",o.setAttribute("aria-label",c?"Restore vision panel":"Minimize vision panel")};i(),o.addEventListener("click",(x)=>{x.stopPropagation(),c=!c,i(),bs({...er(),minimized:c})});let h=null,f=(x)=>{if(x.target instanceof HTMLElement&&x.target.closest("button"))return;let g=l.getBoundingClientRect();h={offsetX:x.clientX-g.left,offsetY:x.clientY-g.top},l.style.left=`${g.left}px`,l.style.top=`${g.top}px`,l.style.right="auto",l.style.bottom="auto",n.setPointerCapture(x.pointerId),x.preventDefault()},u=(x)=>{if(!h)return;let{innerWidth:g,innerHeight:w}=window,p=l.offsetWidth,y=x.clientX-h.offsetX,$=x.clientY-h.offsetY;y=Math.min(Math.max(y,-p+48),g-48),$=Math.min(Math.max($,0),w-24),l.style.left=`${y}px`,l.style.top=`${$}px`},d=(x)=>{if(!h)return;h=null;try{n.releasePointerCapture(x.pointerId)}catch{}let g=parseFloat(l.style.left)||0,w=parseFloat(l.style.top)||0;bs({...er(),left:g,top:w})};return n.addEventListener("pointerdown",f),n.addEventListener("pointermove",u),n.addEventListener("pointerup",d),n.addEventListener("pointercancel",d),document.body.appendChild(l),console.log("[Sable] debug vision panel mounted",{minimized:c,restoredPosition:typeof e.left==="number"&&typeof e.top==="number"}),()=>{try{n.removeEventListener("pointerdown",f),n.removeEventListener("pointermove",u),n.removeEventListener("pointerup",d),n.removeEventListener("pointercancel",d),l.remove()}catch{}}}var Nx=5,Tf=500;function Mx(t){let e=t.remoteParticipants?Array.from(t.remoteParticipants.values()):[];return e.find((n)=>typeof n.identity==="string"&&n.identity.startsWith("agent"))?.identity??e[0]?.identity??null}async function Hx(t){for(let e=1;e<=Nx;e++){let l=Mx(t);if(!l){console.warn("[Sable] sendUiReady: no agent participant yet",{attempt:e}),await new Promise((n)=>setTimeout(n,Tf));continue}try{await t.localParticipant.performRpc({destinationIdentity:l,method:"uiReady",payload:JSON.stringify({timestamp:Date.now()})}),console.log("[Sable] uiReady sent",{identity:l,attempt:e});return}catch(n){console.warn("[Sable] uiReady RPC failed",{attempt:e,err:n}),await new Promise((r)=>setTimeout(r,Tf))}}console.error("[Sable] uiReady: exhausted retries — agent will not greet")}class _o{version=tr;emitter=new kn;activeRoom=null;visionHandle=null;unmountDebugPanel=null;on(t,e){return this.emitter.on(t,e)}async start(t){if(this.activeRoom)throw Error("Sable already started; call stop() first");let e=t.publicKey??t.agentPublicId;if(!e)throw Error("Sable.start: `publicKey` is required");let l=t.apiUrl??ts;console.log("[Sable] fetching connection details",{apiUrl:l});let n=await es({apiUrl:l,publicKey:e});console.log("[Sable] connection details received",{roomName:n.roomName,participantName:n.participantName});let r=await Promise.resolve().then(() => (qf(),jf)),{Room:o,RoomEvent:s,LocalVideoTrack:c,Track:i}=r,h=new o,f={LocalVideoTrack:c,Track:i};if(h.registerRpcMethod("agentReady",async()=>{return console.log("[Sable] RPC agentReady received"),Hx(h),JSON.stringify({success:!0})}),fs(h),rs(h,t.runtime),this.wireRoomEvents(h,s),await h.connect(n.serverUrl,n.participantToken),await h.localParticipant.setMicrophoneEnabled(!0),this.activeRoom=h,t.vision?.enabled)try{if(this.visionHandle=await as({room:h,lib:f,options:t.vision}),xs(t.debug))this.unmountDebugPanel=gs(this.visionHandle.canvas)}catch(u){console.warn("[Sable] failed to start vision",u)}console.log("[Sable] session live",{roomName:n.roomName,participantName:n.participantName}),this.emitter.emit("session:started",{roomName:n.roomName,participantName:n.participantName}),setTimeout(()=>{if(this.activeRoom!==h)return;let u=h,x=(u.remoteParticipants?Array.from(u.remoteParticipants.values()):[]).map((w)=>({identity:w.identity,tracks:w.trackPublications?Array.from(w.trackPublications.values()).map((p)=>({kind:p.kind,subscribed:p.isSubscribed})):[]}));if(!x.some((w)=>w.tracks.some((p)=>p.kind==="audio")))console.warn("[Sable] no remote audio track after 10s — agent worker probably failed to publish. Remote participants:",x)},1e4)}async stop(){let t=this.activeRoom;if(!t)return;if(this.activeRoom=null,this.unmountDebugPanel){try{this.unmountDebugPanel()}catch(e){console.warn("[Sable] debug panel unmount failed",e)}this.unmountDebugPanel=null}if(this.visionHandle){try{await this.visionHandle.stop()}catch(e){console.warn("[Sable] vision stop failed",e)}this.visionHandle=null}try{await t.localParticipant.setMicrophoneEnabled(!1)}catch(e){console.warn("[Sable] setMicrophoneEnabled(false) failed",e)}await t.disconnect(),console.log("[Sable] session ended"),this.emitter.emit("session:ended",{})}wireRoomEvents(t,e){t.on(e.ConnectionStateChanged,(l)=>{console.log("[Sable] ConnectionStateChanged",l)}),t.on(e.Disconnected,(l)=>{if(console.log("[Sable] Disconnected",l),this.activeRoom===t)this.stop().catch((n)=>console.warn("[Sable] stop on disconnect failed",n))}),t.on(e.ParticipantConnected,(l)=>{let n=l;console.log("[Sable] ParticipantConnected",{identity:n.identity,sid:n.sid})}),t.on(e.ParticipantDisconnected,(l)=>{console.warn("[Sable] ParticipantDisconnected",{identity:l.identity})}),t.on(e.TrackSubscribed,(l,n,r)=>{let o=l,s=r;if(console.log("[Sable] TrackSubscribed",{kind:o.kind,participant:s.identity}),o.kind==="audio"&&typeof o.attach==="function"){let c=o.attach();c.setAttribute("data-sable","1"),c.setAttribute("playsinline",""),c.autoplay=!0,document.body.appendChild(c),console.log("[Sable] attached remote audio element")}}),t.on(e.TrackUnsubscribed,(l)=>{let n=l;if(typeof n.detach==="function")for(let r of n.detach())r.remove()}),t.on(e.ActiveSpeakersChanged,(l)=>{let n=l??[],r=n.some((s)=>typeof s.identity==="string"&&s.identity.startsWith("agent")),o=n.some((s)=>typeof s.identity==="string"&&!s.identity.startsWith("agent"));this.emitter.emit("agent:speaking",r),this.emitter.emit("user:speaking",o)})}}var Sn=new _o;function Ff(){if(typeof window>"u")return;if(window.Sable)return;window.Sable=Sn,console.log("[Sable] SDK loaded",Sn.version)}Ff();var Ag=Sn;export{Ag as default,tr as VERSION};

package/dist/types/types/index.d.ts

@@ -12,7 +12,7 @@ export interface WireframeFrameSource {
     features?: {
         /**
          * Include rendered images in the wireframe (instead of placeholder boxes).
-         * Slightly higher CPU + bandwidth. Default: false.
+         * Slightly higher CPU + bandwidth. Default: true.
          */
         includeImages?: boolean;
     };
@@ -39,7 +39,7 @@ export interface VisionOptions {
     enabled?: boolean;
     /**
      * Where video frames come from. Defaults to the built-in wireframe renderer
-     * at 2 fps with images disabled.
+     * at 2 fps with images enabled.
      */
     frameSource?: FrameSource;
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@sable-ai/sdk-core",
-  "version": "0.1.4",
+  "version": "0.1.5",
   "description": "Sable SDK core — headless runtime (voice + vision + agent RPC) that runs in the user's browser.",
   "license": "MIT",
   "type": "module",

package/src/types/index.ts

@@ -19,7 +19,7 @@ export interface WireframeFrameSource {
   features?: {
     /**
      * Include rendered images in the wireframe (instead of placeholder boxes).
-     * Slightly higher CPU + bandwidth. Default: false.
+     * Slightly higher CPU + bandwidth. Default: true.
      */
     includeImages?: boolean;
   };
@@ -51,7 +51,7 @@ export interface VisionOptions {
   enabled?: boolean;
   /**
    * Where video frames come from. Defaults to the built-in wireframe renderer
-   * at 2 fps with images disabled.
+   * at 2 fps with images enabled.
    */
   frameSource?: FrameSource;
 }

package/src/vision/index.ts

@@ -26,7 +26,7 @@ export type { LiveKitPublishLib, PublishCapableRoom } from "./publisher";
 const DEFAULT_FRAME_SOURCE: FrameSource = {
   type: "wireframe",
   rate: 2,
-  features: { includeImages: false },
+  features: { includeImages: true },
 };
 
 export interface StartVisionArgs {