@elizaos/plugin-xr 2.0.3-beta.5

package/AGENTS.md

@@ -0,0 +1,151 @@
+# @elizaos/plugin-xr
+
+WebXR audio/video streaming for elizaOS — Quest 3 and XReal glasses.
+
+## Purpose / role
+
+Adds a WebXR streaming surface to an Eliza agent: a WebSocket server accepts
+connections from XR headsets (Quest 3, XReal, or a browser simulator), pipes
+microphone audio through the runtime's TRANSCRIPTION model, and routes the
+transcript through the standard message pipeline. Voice responses are generated
+via the TEXT_TO_SPEECH model and sent back as binary audio frames. The plugin
+is opt-in — register `xrPlugin` in your character's plugins array.
+
+## Plugin surface
+
+**Service**
+- `XRSessionService` (`xr-session`) — WebSocket server on port 31338 (default).
+  Manages per-connection lifecycle, delegates audio to `AudioPipeline` and
+  camera frames to `VisionPipeline`, routes transcripts into the agent message
+  pipeline, and sends TTS audio back to the headset.
+
+**Actions**
+- `XR_QUERY_VISION` — describes what the user's XR camera currently sees
+  (calls `VisionPipeline.describeFrame` → `ModelType.IMAGE_DESCRIPTION`).
+  Only validates when a recent camera frame exists.
+- `XR_OPEN_VIEW` — opens a named view panel on the headset (sends
+  `view_open` control message).
+- `XR_CLOSE_VIEW` — closes a named view (or all views if no id given).
+- `XR_SWITCH_VIEW` — brings a view to the foreground without closing others.
+- `XR_LIST_VIEWS` — enumerates views with `viewType: "xr"` from all loaded
+  plugins and optionally sends the catalog to the device.
+- `XR_RESIZE_VIEW` — resizes/repositions the active panel (`scale`,
+  `distance`, `fullscreen` options).
+
+**Provider**
+- `XR_SESSION` (`xr-context.ts`) — injects connected device list and camera
+  state into the agent context block when at least one headset is connected.
+
+**Routes** (all under `/api/xr/`)
+- `GET /xr/status` — JSON list of connected sessions and camera-frame state.
+- `GET /xr/connect` — HTML page with QR code for pairing a headset.
+- `GET /xr/views` — JSON list of all `viewType: "xr"` views registered by
+  loaded plugins, plus active connections.
+- `GET /xr/view-host/:id` — self-contained HTML shell that dynamically imports
+  a plugin view bundle and renders it with an XR-optimised chrome.
+- `GET /xr/simulator.js` — serves the built WebXR emulator bundle (only
+  available after `bun run build:all`).
+
+## Layout
+
+```
+plugins/plugin-xr/
+  src/
+    index.ts                  Plugin entry — exports xrPlugin and public types
+    protocol.ts               Wire types: XRClientControl, XRServerControl,
+                              XRBinaryHeader, XRTTSAudioHeader; encode/decode helpers
+    actions/
+      xr-query-vision.ts      XR_QUERY_VISION action
+      xr-view-actions.ts      XR_OPEN/CLOSE/SWITCH/LIST/RESIZE_VIEW actions
+    providers/
+      xr-context.ts           XR_SESSION provider
+    services/
+      xr-session-service.ts   XRSessionService (WebSocket server, main orchestrator)
+      audio-pipeline.ts       AudioPipeline — buffers audio chunks, flushes to
+                              TRANSCRIPTION model after 2 s or 1.5 s silence
+      vision-pipeline.ts      VisionPipeline — stores latest camera frame (max age
+                              10 s), calls IMAGE_DESCRIPTION model on demand
+    routes/
+      xr-status.ts            GET /xr/status
+      xr-connect.ts           GET /xr/connect
+      xr-views.ts             GET /xr/views
+      xr-view-host.ts         GET /xr/view-host/:id
+      xr-simulator-route.ts   GET /xr/simulator.js
+    __tests__/
+      audio-pipeline.test.ts
+      protocol.test.ts
+      vision-pipeline.test.ts
+      xr-bundle-coverage.test.ts
+      xr-feature-parity.test.ts
+      xr-functional-parity.test.ts
+      xr-view-host.test.ts
+      xr-view-host-http.test.ts
+  simulator/                  Browser-side WebXR emulator (Vite build)
+```
+
+## Commands
+
+```bash
+bun run --cwd plugins/plugin-xr typecheck
+bun run --cwd plugins/plugin-xr lint
+bun run --cwd plugins/plugin-xr test
+bun run --cwd plugins/plugin-xr build
+bun run --cwd plugins/plugin-xr build:all    # also builds simulator/
+bun run --cwd plugins/plugin-xr simulator:build
+bun run --cwd plugins/plugin-xr simulator:watch
+bun run --cwd plugins/plugin-xr clean
+```
+
+## Config / env vars
+
+| Var | Default | Required | Purpose |
+|-----|---------|----------|---------|
+| `XR_WS_PORT` | `31338` | no | WebSocket server port |
+| `XR_AGENT_URL` | `http://localhost:<agent-port>` | no | Public base URL sent to the headset for view bundles |
+| `XR_APP_URL` | derived from `VITE_PORT` | no | URL shown on the `/xr/connect` pairing page |
+
+The plugin sets `config: { XR_WS_PORT: 31338 }` in the Plugin object so the
+runtime exposes it via `runtime.getSetting("XR_WS_PORT")`.
+
+The agent must have a TRANSCRIPTION model and TEXT_TO_SPEECH model configured
+(e.g., via `@elizaos/plugin-openai` or a local inference plugin) for audio
+streaming to work. IMAGE_DESCRIPTION is required for `XR_QUERY_VISION`.
+
+## How to extend
+
+**Add an action** — create `src/actions/<name>.ts`, implement `Action` from
+`@elizaos/core`, get `XRSessionService` via
+`runtime.getService<XRSessionService>(XR_SERVICE_TYPE)`, then add the import
+and the object to the `actions` array in `src/index.ts`.
+
+**Add a provider** — create `src/providers/<name>.ts`, implement `Provider`,
+add to `providers` array in `src/index.ts`.
+
+**Add a route** — create `src/routes/<name>.ts`, implement `Route`, add to
+`routes` array in `src/index.ts`.
+
+**Expose a view in XR** — in any other plugin, add a `views` array entry with
+`viewType: "xr"`. `XR_LIST_VIEWS` and `GET /xr/views` collect views with this
+field across all loaded plugins at runtime.
+
+## Conventions / gotchas
+
+- **WebXR requires HTTPS on device.** The `/xr/connect` page warns when the
+  URL is plain HTTP. Use a local tunnel (e.g., `cloudflared`) and set
+  `XR_APP_URL` to the HTTPS tunnel URL.
+- **Binary frame framing** is defined in `src/protocol.ts`: 4-byte big-endian
+  header length, then UTF-8 JSON header, then raw payload. Use
+  `encodeBinaryFrame` / `decodeBinaryFrame` from that module — do not
+  reimplement the framing.
+- **Audio buffering** — `AudioPipeline` accumulates chunks and flushes after
+  2 000 ms of audio or 1 500 ms of silence. Chunks shorter than 512 bytes are
+  dropped. `pcm-f32` encoding (ScriptProcessorNode fallback) is wrapped in a
+  WAV header before being passed to TRANSCRIPTION.
+- **Simulator bundle** — `simulator/` is a separate Vite project. Run
+  `bun run build:all` (or `simulator:build`) before the `/xr/simulator.js`
+  route will serve anything. The route returns 404 until the bundle exists.
+- **`XR_AGENT_URL`** must be a reachable URL from inside the XR headset
+  browser when loading view bundles. `localhost` will only work when testing
+  on the same machine via the browser simulator.
+- See repo root `AGENTS.md` for repo-wide architecture rules, logger
+  conventions, ESM requirements, and naming standards.

package/CLAUDE.md

@@ -0,0 +1,151 @@
+# @elizaos/plugin-xr
+
+WebXR audio/video streaming for elizaOS — Quest 3 and XReal glasses.
+
+## Purpose / role
+
+Adds a WebXR streaming surface to an Eliza agent: a WebSocket server accepts
+connections from XR headsets (Quest 3, XReal, or a browser simulator), pipes
+microphone audio through the runtime's TRANSCRIPTION model, and routes the
+transcript through the standard message pipeline. Voice responses are generated
+via the TEXT_TO_SPEECH model and sent back as binary audio frames. The plugin
+is opt-in — register `xrPlugin` in your character's plugins array.
+
+## Plugin surface
+
+**Service**
+- `XRSessionService` (`xr-session`) — WebSocket server on port 31338 (default).
+  Manages per-connection lifecycle, delegates audio to `AudioPipeline` and
+  camera frames to `VisionPipeline`, routes transcripts into the agent message
+  pipeline, and sends TTS audio back to the headset.
+
+**Actions**
+- `XR_QUERY_VISION` — describes what the user's XR camera currently sees
+  (calls `VisionPipeline.describeFrame` → `ModelType.IMAGE_DESCRIPTION`).
+  Only validates when a recent camera frame exists.
+- `XR_OPEN_VIEW` — opens a named view panel on the headset (sends
+  `view_open` control message).
+- `XR_CLOSE_VIEW` — closes a named view (or all views if no id given).
+- `XR_SWITCH_VIEW` — brings a view to the foreground without closing others.
+- `XR_LIST_VIEWS` — enumerates views with `viewType: "xr"` from all loaded
+  plugins and optionally sends the catalog to the device.
+- `XR_RESIZE_VIEW` — resizes/repositions the active panel (`scale`,
+  `distance`, `fullscreen` options).
+
+**Provider**
+- `XR_SESSION` (`xr-context.ts`) — injects connected device list and camera
+  state into the agent context block when at least one headset is connected.
+
+**Routes** (all under `/api/xr/`)
+- `GET /xr/status` — JSON list of connected sessions and camera-frame state.
+- `GET /xr/connect` — HTML page with QR code for pairing a headset.
+- `GET /xr/views` — JSON list of all `viewType: "xr"` views registered by
+  loaded plugins, plus active connections.
+- `GET /xr/view-host/:id` — self-contained HTML shell that dynamically imports
+  a plugin view bundle and renders it with an XR-optimised chrome.
+- `GET /xr/simulator.js` — serves the built WebXR emulator bundle (only
+  available after `bun run build:all`).
+
+## Layout
+
+```
+plugins/plugin-xr/
+  src/
+    index.ts                  Plugin entry — exports xrPlugin and public types
+    protocol.ts               Wire types: XRClientControl, XRServerControl,
+                              XRBinaryHeader, XRTTSAudioHeader; encode/decode helpers
+    actions/
+      xr-query-vision.ts      XR_QUERY_VISION action
+      xr-view-actions.ts      XR_OPEN/CLOSE/SWITCH/LIST/RESIZE_VIEW actions
+    providers/
+      xr-context.ts           XR_SESSION provider
+    services/
+      xr-session-service.ts   XRSessionService (WebSocket server, main orchestrator)
+      audio-pipeline.ts       AudioPipeline — buffers audio chunks, flushes to
+                              TRANSCRIPTION model after 2 s or 1.5 s silence
+      vision-pipeline.ts      VisionPipeline — stores latest camera frame (max age
+                              10 s), calls IMAGE_DESCRIPTION model on demand
+    routes/
+      xr-status.ts            GET /xr/status
+      xr-connect.ts           GET /xr/connect
+      xr-views.ts             GET /xr/views
+      xr-view-host.ts         GET /xr/view-host/:id
+      xr-simulator-route.ts   GET /xr/simulator.js
+    __tests__/
+      audio-pipeline.test.ts
+      protocol.test.ts
+      vision-pipeline.test.ts
+      xr-bundle-coverage.test.ts
+      xr-feature-parity.test.ts
+      xr-functional-parity.test.ts
+      xr-view-host.test.ts
+      xr-view-host-http.test.ts
+  simulator/                  Browser-side WebXR emulator (Vite build)
+```
+
+## Commands
+
+```bash
+bun run --cwd plugins/plugin-xr typecheck
+bun run --cwd plugins/plugin-xr lint
+bun run --cwd plugins/plugin-xr test
+bun run --cwd plugins/plugin-xr build
+bun run --cwd plugins/plugin-xr build:all    # also builds simulator/
+bun run --cwd plugins/plugin-xr simulator:build
+bun run --cwd plugins/plugin-xr simulator:watch
+bun run --cwd plugins/plugin-xr clean
+```
+
+## Config / env vars
+
+| Var | Default | Required | Purpose |
+|-----|---------|----------|---------|
+| `XR_WS_PORT` | `31338` | no | WebSocket server port |
+| `XR_AGENT_URL` | `http://localhost:<agent-port>` | no | Public base URL sent to the headset for view bundles |
+| `XR_APP_URL` | derived from `VITE_PORT` | no | URL shown on the `/xr/connect` pairing page |
+
+The plugin sets `config: { XR_WS_PORT: 31338 }` in the Plugin object so the
+runtime exposes it via `runtime.getSetting("XR_WS_PORT")`.
+
+The agent must have a TRANSCRIPTION model and TEXT_TO_SPEECH model configured
+(e.g., via `@elizaos/plugin-openai` or a local inference plugin) for audio
+streaming to work. IMAGE_DESCRIPTION is required for `XR_QUERY_VISION`.
+
+## How to extend
+
+**Add an action** — create `src/actions/<name>.ts`, implement `Action` from
+`@elizaos/core`, get `XRSessionService` via
+`runtime.getService<XRSessionService>(XR_SERVICE_TYPE)`, then add the import
+and the object to the `actions` array in `src/index.ts`.
+
+**Add a provider** — create `src/providers/<name>.ts`, implement `Provider`,
+add to `providers` array in `src/index.ts`.
+
+**Add a route** — create `src/routes/<name>.ts`, implement `Route`, add to
+`routes` array in `src/index.ts`.
+
+**Expose a view in XR** — in any other plugin, add a `views` array entry with
+`viewType: "xr"`. `XR_LIST_VIEWS` and `GET /xr/views` collect views with this
+field across all loaded plugins at runtime.
+
+## Conventions / gotchas
+
+- **WebXR requires HTTPS on device.** The `/xr/connect` page warns when the
+  URL is plain HTTP. Use a local tunnel (e.g., `cloudflared`) and set
+  `XR_APP_URL` to the HTTPS tunnel URL.
+- **Binary frame framing** is defined in `src/protocol.ts`: 4-byte big-endian
+  header length, then UTF-8 JSON header, then raw payload. Use
+  `encodeBinaryFrame` / `decodeBinaryFrame` from that module — do not
+  reimplement the framing.
+- **Audio buffering** — `AudioPipeline` accumulates chunks and flushes after
+  2 000 ms of audio or 1 500 ms of silence. Chunks shorter than 512 bytes are
+  dropped. `pcm-f32` encoding (ScriptProcessorNode fallback) is wrapped in a
+  WAV header before being passed to TRANSCRIPTION.
+- **Simulator bundle** — `simulator/` is a separate Vite project. Run
+  `bun run build:all` (or `simulator:build`) before the `/xr/simulator.js`
+  route will serve anything. The route returns 404 until the bundle exists.
+- **`XR_AGENT_URL`** must be a reachable URL from inside the XR headset
+  browser when loading view bundles. `localhost` will only work when testing
+  on the same machine via the browser simulator.
+- See repo root `AGENTS.md` for repo-wide architecture rules, logger
+  conventions, ESM requirements, and naming standards.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Shaw Walters and elizaOS Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,106 @@
+# @elizaos/plugin-xr
+
+WebXR audio/video streaming for elizaOS — connects Quest 3, XReal glasses, or
+any WebXR-capable browser to an Eliza agent.
+
+## What it does
+
+- Opens a WebSocket server (default port 31338) that XR headsets connect to.
+- Streams microphone audio from the headset, transcribes it via the runtime's
+  TRANSCRIPTION model, and routes the transcript through the agent's normal
+  message pipeline.
+- Sends voice responses back to the headset as TTS audio (TEXT_TO_SPEECH model).
+- Captures camera frames from the headset and can describe them using the
+  IMAGE_DESCRIPTION model (`XR_QUERY_VISION` action).
+- Manages floating view panels on the headset — open, close, switch, resize,
+  and list views that other plugins have declared with `viewType: "xr"`.
+
+## Capabilities added to an Eliza agent
+
+| Action | What it does |
+|--------|-------------|
+| `XR_QUERY_VISION` | Describes what the user's XR camera currently sees |
+| `XR_OPEN_VIEW` | Opens a named floating panel on the headset |
+| `XR_CLOSE_VIEW` | Closes a named panel (or all panels) |
+| `XR_SWITCH_VIEW` | Brings a panel to the foreground |
+| `XR_LIST_VIEWS` | Lists all XR-capable views and sends a launcher catalog to the device |
+| `XR_RESIZE_VIEW` | Resizes or repositions the active panel |
+
+The `XR_SESSION` context provider automatically injects connected-device status
+and camera state into every conversation turn when a headset is connected.
+
+## HTTP routes
+
+| Route | Purpose |
+|-------|---------|
+| `GET /api/xr/status` | JSON status of connected sessions |
+| `GET /api/xr/connect` | HTML pairing page with QR code for the headset browser |
+| `GET /api/xr/views` | JSON list of all registered XR views |
+| `GET /api/xr/view-host/:id` | Self-contained XR shell page for a specific view |
+| `GET /api/xr/simulator.js` | WebXR emulator bundle for testing (after build) |
+
+## Enabling the plugin
+
+Add `@elizaos/plugin-xr` to your character's plugin list:
+
+```json
+{
+  "name": "my-agent",
+  "plugins": ["@elizaos/plugin-xr"]
+}
+```
+
+The plugin also requires a character configuration that loads:
+- A TRANSCRIPTION model provider (for speech-to-text)
+- A TEXT_TO_SPEECH model provider (for voice responses)
+- An IMAGE_DESCRIPTION model provider (for `XR_QUERY_VISION`)
+
+## Configuration
+
+| Environment variable | Default | Purpose |
+|---------------------|---------|---------|
+| `XR_WS_PORT` | `31338` | WebSocket server port |
+| `XR_AGENT_URL` | `http://localhost:<agent-port>` | Public base URL used by the headset to load view bundles — must be reachable from the device |
+| `XR_APP_URL` | derived from `VITE_PORT` | URL shown on the `/api/xr/connect` pairing page |
+
+## Connecting a headset
+
+1. Start the agent with this plugin enabled.
+2. Open `http://localhost:31337/api/xr/connect` in a browser (adjust port to
+   your agent's API port).
+3. Scan the QR code with your Quest 3 or XReal browser, or navigate to the
+   shown URL.
+4. Allow microphone and camera access when prompted.
+5. The agent will start receiving audio immediately.
+
+> **Note:** WebXR APIs require HTTPS on physical devices. For local
+> development, run a tunnel (e.g., `cloudflared tunnel`) and set `XR_APP_URL`
+> to the HTTPS tunnel URL before starting the agent.
+
+## Building the simulator
+
+The `simulator/` sub-project provides a browser-side WebXR emulator for
+Playwright tests.
+
+```bash
+bun run --cwd plugins/plugin-xr build:all
+# or just the simulator:
+bun run --cwd plugins/plugin-xr simulator:build
+```
+
+After building, `GET /api/xr/simulator.js` serves the emulator bundle.
+
+## Wire protocol
+
+The plugin uses a custom binary framing on top of WebSocket:
+
+- **Text frames** — JSON control messages (`XRClientControl` / `XRServerControl`).
+- **Binary frames** — 4-byte big-endian header length, UTF-8 JSON header
+  (`XRAudioHeader` or `XRFrameHeader`), then raw payload (audio or JPEG/WebP).
+
+The wire types (`XRClientControl`, `XRServerControl`, `XRBinaryHeader`,
+`XRTTSAudioHeader`, `XRPanelConfig`, and the rest of `protocol.ts`) are
+re-exported as types from the package root via `export type *`. The framing
+helpers `encodeBinaryFrame` / `decodeBinaryFrame` are runtime functions that
+live in `src/protocol.ts` and are used internally by the WebSocket server;
+they are not part of the package's public runtime exports.

package/package.json

@@ -0,0 +1,57 @@
+{
+  "name": "@elizaos/plugin-xr",
+  "version": "2.0.3-beta.5",
+  "type": "module",
+  "description": "WebXR audio/video streaming for elizaOS — Quest 3 and XReal glasses",
+  "main": "./dist/index.js",
+  "exports": {
+    "./package.json": "./package.json",
+    ".": {
+      "types": "./dist/index.d.ts",
+      "eliza-source": {
+        "types": "./src/index.ts",
+        "import": "./src/index.ts",
+        "default": "./src/index.ts"
+      },
+      "import": "./dist/index.js",
+      "default": "./dist/index.js"
+    }
+  },
+  "scripts": {
+    "typecheck": "tsgo --noEmit -p tsconfig.json",
+    "lint": "bunx @biomejs/biome check src",
+    "test": "bunx vitest run --config ./vitest.config.ts",
+    "build": "bun run build:js && bun run build:types",
+    "build:all": "bun run build && bun run simulator:build",
+    "clean": "node ../../packages/scripts/rm-path-recursive.mjs dist simulator/dist",
+    "build:js": "tsup --config ../tsup.plugin-packages.shared.ts",
+    "build:types": "tsc --noCheck -p tsconfig.build.json",
+    "simulator:build": "cd simulator && bun install && bun run build",
+    "simulator:watch": "cd simulator && bun run build:watch"
+  },
+  "dependencies": {
+    "@elizaos/agent": "2.0.3-beta.5",
+    "@elizaos/core": "2.0.3-beta.5",
+    "ws": "^8.18.0",
+    "zod": "^4.4.3"
+  },
+  "devDependencies": {
+    "@biomejs/biome": "^2.4.14",
+    "@types/node": "^25.0.3",
+    "@types/ws": "^8.5.10",
+    "bun-types": "^1.3.14",
+    "tsup": "^8.5.1",
+    "typescript": "^6.0.3",
+    "vitest": "^4.0.17"
+  },
+  "elizaos": {
+    "plugin": {
+      "displayName": "XR Streaming",
+      "category": "media"
+    }
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "gitHead": "ff6157011c9459670021cc28a6797592a78b8817"
+}