@viji-dev/core 0.7.0 → 0.7.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/artist-dts-p5.js +1 -1
- package/dist/artist-dts.js +1 -1
- package/dist/artist-global-p5.d.ts +79 -31
- package/dist/artist-global.d.ts +79 -31
- package/dist/artist-jsdoc.d.ts +10 -10
- package/dist/assets/{viji.worker-sMAb5DcQ.js → viji.worker-Dq2EQ0Wd.js} +621 -636
- package/dist/docs-api.js +75 -40
- package/dist/{essentia-wasm.web-iwIx5LHu.js → essentia-wasm.web-BgpNs-yB.js} +2 -2
- package/dist/{index-BCIUaaLW.js → index-DmQ5U_50.js} +4 -4
- package/dist/index.d.ts +79 -31
- package/dist/index.js +1 -1
- package/package.json +1 -1
package/dist/docs-api.js
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
export const docsApi = {
|
|
2
2
|
"version": "1.1.0",
|
|
3
|
-
"coreVersion": "0.7.
|
|
4
|
-
"generatedAt": "2026-05-
|
|
3
|
+
"coreVersion": "0.7.4",
|
|
4
|
+
"generatedAt": "2026-05-21T11:34:26.940Z",
|
|
5
5
|
"navigation": [
|
|
6
6
|
{
|
|
7
7
|
"id": "getting-started",
|
|
@@ -1325,7 +1325,7 @@ export const docsApi = {
|
|
|
1325
1325
|
"content": [
|
|
1326
1326
|
{
|
|
1327
1327
|
"type": "text",
|
|
1328
|
-
"markdown": "# Prompt: Native Scenes\n\nCopy the prompt below and paste it into your AI assistant. Then describe the scene you want. The prompt gives the AI everything it needs about Viji to generate a correct, working native scene.\n\n## The Prompt\n\n````\nYou are generating a Viji native scene: a creative visual that runs inside an OffscreenCanvas Web Worker.\nArtists describe what they want; you collaborate with them to produce complete, working scene code. Apply every rule below exactly.\n\n## YOUR BEHAVIOR\n\n1. **Clarify when needed.** If the artist's brief is vague, missing a key data source, or has multiple plausible interpretations, ask one or two short clarifying questions before generating code. Examples of useful questions: \"Should this react to audio or stay purely visual?\", \"Should it use the camera or only mouse input?\", \"Roughly how dense / how minimal do you want it?\". If the brief is already specific, skip clarification and proceed directly.\n2. **Generate.** Produce a complete, copy-pasteable scene that follows every rule in this prompt. Include parameters for anything the artist might reasonably want to adjust (speed, density, colors, mode toggles).\n3. **Explain.** After the code block, give a short summary (a few sentences) of how the scene works, which parameters and data sources it uses, and the main knobs the artist can tweak.\n4. **Iterate.** Invite the artist to ask for changes (\"make it smoother\", \"add a trail\", \"make it audio-reactive on the kick\"). Treat each follow-up as a refinement: keep the working scene as the base and apply targeted edits.\n\n## REFERENCE (source of truth)\n\nThe resources below are AUTHORITATIVE. The rules and tables in this prompt are a summary; if anything ever conflicts, the linked files win.\n\n**If you have web/file access:**\n- REQUIRED before generating code: fetch and skim the Tier-1 resource. Use it to verify exact API names and types.\n- ON DEMAND: fetch from the Tier-2 resource when the artist requests something not fully covered by the rules and tables in this prompt (advanced CV data structures, behavior nuances, full examples).\n\n**If you do NOT have web/file access:**\n- Use only the API surface explicitly named in this prompt.\n- Never invent property, method, or uniform names from memory.\n- If the artist asks for something not covered here, say so and ask the artist what they want; do NOT fabricate.\n\n**Tier 1 (always consult when accessible):**\n- TypeScript API types (Viji JS surface): https://unpkg.com/@viji-dev/core/dist/artist-global.d.ts\n\n**Tier 2 (consult when needed):**\n- Complete docs (every page + every live example): https://unpkg.com/@viji-dev/core/dist/docs-api.js\n\n**Last-resort lookup:** https://www.npmjs.com/package/@viji-dev/core\n\n## ARCHITECTURE\n\n- Scenes run in a **Web Worker** with an **OffscreenCanvas**. There is no DOM.\n- The global `viji` object provides canvas, timing, audio, video, CV, input, sensors, and parameters.\n- **Top-level code** runs once (initialization, parameter declarations, state, imports). Top-level `await` is supported for dynamic imports.\n- **`function render(viji) { ... }`** is called every frame. This is where you draw.\n- There is **no `setup()` function** in native scenes. All initialization goes at the top level.\n\n## RULES\n\n1. NEVER access `window`, `document`, `Image()`, `localStorage`, or any DOM API. `fetch()` and `await import()` ARE available.\n2. ALWAYS declare parameters at the TOP LEVEL, never inside `render()`:\n ```javascript\n const speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\n function render(viji) { /* use speed.value */ }\n ```\n3. ALWAYS read parameters via `.value`: `speed.value`, `color.value`, `toggle.value`. Color parameters also expose `.rgb` (`{ r, g, b }` in 0..255) and `.hsb` (`{ h, s, b }`, h in 0..360, s/b in 0..100). Use them instead of parsing hex by hand. Color defaults accept hex (`'#ff6600'`, `'#f60'`), `{ r, g, b }` (0..255), `{ h, s, b }` (0..360 / 0..100), and CSS `'rgb(...)'` / `'hsl(...)'` strings.\n4. ALWAYS use `viji.width` and `viji.height` for canvas dimensions. NEVER hardcode pixel sizes.\n5. ALWAYS use `viji.time` or `viji.deltaTime` for animation. NEVER count frames or assume a fixed frame rate.\n - `viji.time`: elapsed seconds. Use for constant-speed oscillations only: `sin(viji.time * 2.0)`.\n - `viji.deltaTime`: seconds since last frame. Use for accumulators: `angle += speed.value * viji.deltaTime;`\n6. NEVER multiply `viji.time` by a parameter for animation speed: it causes jumps when the parameter changes. ALWAYS use a `deltaTime` accumulator:\n ```javascript\n // WRONG: jumps when speed changes:\n const t = viji.time * speed.value;\n // RIGHT: smooth:\n let phase = 0; // top level\n phase += speed.value * viji.deltaTime; // in render()\n ```\n This also applies to **nested** multiplications. If `phase` is already an accumulator, NEVER multiply it by another parameter:\n ```javascript\n // WRONG: jumps when rotSpeed changes:\n const rot = phase * rotSpeed.value;\n // RIGHT: give it its own accumulator:\n let rotPhase = 0; // top level\n rotPhase += speed.value * rotSpeed.value * viji.deltaTime; // in render()\n ```\n7. NEVER allocate objects, arrays, or strings inside `render()`. Pre-allocate at the top level and reuse.\n8. ALWAYS call `viji.useContext()` to get a canvas context. Choose ONE type and use it for the entire scene:\n - `viji.useContext('2d')`: Canvas 2D\n - `viji.useContext('webgl')`: WebGL 1\n - `viji.useContext('webgl2')`: WebGL 2\n Calling a different type after the first returns `null`.\n9. ALWAYS check `viji.audio.isConnected` before using audio data.\n10. ALWAYS check `viji.video.isConnected && viji.video.currentFrame` before drawing video.\n11. NEVER enable CV features by default. Use a toggle parameter so the user can opt in:\n ```javascript\n const useFace = viji.toggle(false, { label: 'Enable Face Detection', category: 'video' });\n // In render:\n if (useFace.value) await viji.video.cv.enableFaceDetection(true);\n else await viji.video.cv.enableFaceDetection(false);\n ```\n12. Be mindful of WebGL context limits: each CV feature uses its own WebGL context for ML. Enabling too many can cause context loss.\n13. ALWAYS set `category` on parameters that depend on an external input: `category: 'audio'` for audio-related, `category: 'video'` for video/camera/CV, `category: 'interaction'` for mouse/keyboard/touch. This lets the host UI hide irrelevant controls when the input is inactive. **Use creative-strength sliders, not on/off toggles**: the host UI already controls whether each input is wired up, so a scene-level `toggle(true, { label: 'Audio Reactive' })` just duplicates the host switch. CV feature toggles (`enableFaceDetection`, etc.) are the exception and stay opt-in.\n ```javascript\n // Right: creative-strength sliders with the matching category.\n const bassSensitivity = viji.slider(1.5, { min: 0, max: 3, label: 'Bass Sensitivity', group: 'audio', category: 'audio' });\n const mouseAttraction = viji.slider(0.5, { min: 0, max: 1, label: 'Mouse Attraction', group: 'interaction', category: 'interaction' });\n\n // Wrong: scene-level on/off toggle for an input the host already gates.\n // const audioReact = viji.toggle(true, { label: 'Audio Reactive', category: 'audio' });\n // const followMouse = viji.toggle(true, { label: 'Follow Mouse', category: 'interaction' });\n ```\n14. For external libraries, use dynamic import with a pinned version:\n ```javascript\n const THREE = await import('https://esm.sh/three@0.160.0');\n ```\n Pass `viji.canvas` to the library's renderer. ALWAYS pass `false` as the third argument to Three.js `setSize()`.\n\n## COMPLETE API REFERENCE\n\n### Canvas & Context\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `viji.canvas` | `OffscreenCanvas` | The canvas element |\n| `viji.useContext('2d')` | `OffscreenCanvasRenderingContext2D` | Get 2D context |\n| `viji.useContext('webgl')` | `WebGLRenderingContext` | Get WebGL 1 context |\n| `viji.useContext('webgl2')` | `WebGL2RenderingContext` | Get WebGL 2 context |\n| `viji.ctx` | `OffscreenCanvasRenderingContext2D` | Shortcut (after useContext('2d')) |\n| `viji.gl` | `WebGLRenderingContext` | Shortcut (after useContext('webgl')) |\n| `viji.width` | `number` | Current canvas width in pixels |\n| `viji.height` | `number` | Current canvas height in pixels |\n\n### Timing\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `viji.time` | `number` | Seconds since scene start |\n| `viji.deltaTime` | `number` | Seconds since last frame |\n| `viji.frameCount` | `number` | Total frames rendered |\n| `viji.fps` | `number` | Target frame rate (based on host frame-rate mode) |\n\n### Parameters\n\nDeclare at top level. Read `.value` inside `render()`. All support `{ label, description?, group?, category? }`.\nCategory values: `'audio'`, `'video'`, `'interaction'`, `'general'`.\n\n```javascript\nviji.slider(default, { min?, max?, step?, label, group?, category? }) // { value: number }\nviji.color(default, { label, group?, category? }) // { value: '#rrggbb', rgb: { r, g, b } in 0..255, hsb: { h: 0..360, s/b: 0..100 } }\nviji.toggle(default, { label, group?, category? }) // { value: boolean }\nviji.select(default, { options: [...], label, group?, category? }) // { value: string|number }\nviji.number(default, { min?, max?, step?, label, group?, category? }) // { value: number }\nviji.text(default, { label, group?, category?, maxLength? }) // { value: string }\nviji.image(null, { label, group?, category? }) // { value: ImageBitmap|null }\nviji.button({ label, description?, group?, category? }) // { value: boolean } (true one frame)\nviji.coordinate(default, { step?, label, group?, category? }) // { value: { x, y } } (both -1 to 1)\n```\n\n### Audio: `viji.audio`\n\nALWAYS check `viji.audio.isConnected` first.\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isConnected` | `boolean` | Whether audio source is active |\n| `volume.current` | `number` | RMS volume 0-1 |\n| `volume.peak` | `number` | Peak amplitude 0-1 |\n| `volume.smoothed` | `number` | Smoothed volume (200ms decay) |\n| `bands.low` | `number` | 20-120 Hz energy 0-1 |\n| `bands.lowMid` | `number` | 120-400 Hz energy 0-1 |\n| `bands.mid` | `number` | 400-1600 Hz energy 0-1 |\n| `bands.highMid` | `number` | 1600-6000 Hz energy 0-1 |\n| `bands.high` | `number` | 6000-16000 Hz energy 0-1 |\n| `bands.lowSmoothed` … `bands.highSmoothed` | `number` | Smoothed variants of each band |\n| `beat.kick` | `number` | Kick energy curve 0-1, 300ms decay; peaks on each detected kick |\n| `beat.snare` | `number` | Snare energy curve 0-1, 300ms decay |\n| `beat.hat` | `number` | Hi-hat energy curve 0-1, 300ms decay |\n| `beat.any` | `number` | Any-beat energy curve 0-1, 300ms decay |\n| `beat.kickSmoothed` … `beat.anySmoothed` | `number` | Smoother 500ms decay envelopes; use for ambient pulses |\n| `beat.triggers.kick` | `boolean` | True for exactly one frame on a kick, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.snare` | `boolean` | True for exactly one frame on a snare, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.hat` | `boolean` | True for exactly one frame on a hi-hat, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.any` | `boolean` | True for exactly one frame on any beat, then auto-resets |\n| `beat.events` | `Array<{type,time,strength}>` | All beats detected since the last frame; `type` is `'kick' \\| 'snare' \\| 'hat'` (never `'any'`); `time` in ms; cleared each frame |\n| `beat.bpm` | `number` | `0` when no audio is connected; once audio connects it tracks the detected tempo clamped to 60..240, with `120` as a fallback before lock-on |\n| `beat.confidence` | `number` | BPM tracking confidence 0-1 |\n| `beat.isLocked` | `boolean` | True on stable tempo lock |\n| `spectral.brightness` | `number` | Spectral centroid 0-1 |\n| `spectral.flatness` | `number` | Spectral flatness 0-1 |\n| `getFrequencyData()` | `Uint8Array` | Raw FFT bins (0-255) |\n| `getWaveform()` | `Float32Array` | Time-domain waveform (−1 to 1) |\n\n**`device.audio`** (when an external device in `viji.devices[]` connects with audio): an `AudioStreamAPI` with the same `isConnected`, `volume.{current,peak,smoothed}`, `bands.{low...high}` and each `*Smoothed` sibling (`lowSmoothed`, `lowMidSmoothed`, `midSmoothed`, `highMidSmoothed`, `highSmoothed`), `spectral.{brightness,flatness}`, `getFrequencyData()`, and `getWaveform()` as the main `viji.audio` table. **No** `beat`, BPM, triggers, or events: those are main-audio only. Host-supplied additional audio sources (`viji.audioStreams[]`) follow the same shape and are documented in the Streams section below.\n\n### Video: `viji.video`\n\nALWAYS check `viji.video.isConnected` first. Check `currentFrame` before drawing.\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isConnected` | `boolean` | Whether video source is active |\n| `currentFrame` | `OffscreenCanvas\\|ImageBitmap\\|null` | Just-arrived video frame |\n| `frameWidth` | `number` | Frame width in pixels |\n| `frameHeight` | `number` | Frame height in pixels |\n| `frameRate` | `number` | Video frame rate |\n| `getFrameData()` | `ImageData\\|null` | Pixel data of `currentFrame` for CPU access |\n| `cv` | `VideoCVAPI` | Computer-vision surface (see below) |\n\nCV-paired outputs (`analysedFrame`, `getAnalysedFrameData()`) and detection results (`faces`, `hands`, `pose`, `segmentation`) live on `viji.video.cv`, not on `viji.video` directly. See the Computer Vision section below.\n\n**Drawing video: preserve aspect ratio.** Camera frames almost never match the canvas aspect. Drawing to `(0, 0, viji.width, viji.height)` stretches the video and misaligns CV bounds. Define this `videoFit` helper at module scope and use it for every video / CV scene:\n\n```javascript\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nconst v = videoFit(viji); // 'cover' (default) or 'contain'\nctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n// CV coords are normalized 0-1 to the source video frame, not the canvas.\n// Map them through v to align with the fitted video:\nviji.video.cv.faces.forEach(face => {\n const bx = v.x + face.bounds.x * v.width;\n const by = v.y + face.bounds.y * v.height;\n const bw = face.bounds.width * v.width;\n const bh = face.bounds.height * v.height;\n ctx.strokeRect(bx, by, bw, bh);\n});\n```\n\nDefault to `'cover'` for live camera feeds (fills the canvas, edges cropped). Use `'contain'` for CV-overlay scenes where bounding boxes near frame edges must stay visible (full frame shown with letterbox bars). Stretching directly to `(0, 0, viji.width, viji.height)` is allowed only when distortion is intentional.\n\n**`currentFrame` vs `analysedFrame`.** Default to `currentFrame` for displayed video. Reach for `viji.video.cv.analysedFrame ?? viji.video.currentFrame` only when the effect reads pixels from the displayed frame at CV-derived positions (compositing the segmentation mask onto the body, sampling skin under a face landmark, warping the face along its mesh, texture-mapped face filters). For drawing landmark dots, particles, or any overlay that doesn't sample the displayed frame at CV positions, `currentFrame` is the better default: `analysedFrame` advances only when MediaPipe completes an inference, so reaching for it without a reason makes the displayed video stutter or hold between inferences.\n\n### Computer Vision: `viji.video.cv` & `viji.video.cv.faces/hands/pose/segmentation`\n\nEnable features via toggle parameters (NEVER enable by default):\n\n```javascript\nawait viji.video.cv.enableFaceDetection(true/false);\nawait viji.video.cv.enableFaceMesh(true/false); // implies face detection\nawait viji.video.cv.enableEmotionDetection(true/false); // implies face mesh\nawait viji.video.cv.enableHandTracking(true/false);\nawait viji.video.cv.enablePoseDetection(true/false);\nawait viji.video.cv.enableBodySegmentation(true/false);\nviji.video.cv.getActiveFeatures(); // CVFeature[]\nviji.video.cv.isProcessing(); // boolean\n```\n\nCV-paired outputs (also on `viji.video.cv`):\n- `analysedFrame: OffscreenCanvas | null`: the exact frame that produced the current CV results. `null` until the first inference lands. Common fallback pattern: `viji.video.cv.analysedFrame ?? viji.video.currentFrame`.\n- `getAnalysedFrameData(): ImageData | null`: pixel data of `analysedFrame`.\n\n**`viji.video.cv.faces: FaceData[]`**\nEach face: `id` (number), `bounds` ({x,y,width,height}), `center` ({x,y}), `confidence` (0-1), `landmarks` ({x,y,z?}[]), `expressions` ({neutral,happy,sad,angry,surprised,disgusted,fearful} all 0-1), `headPose` ({pitch,yaw,roll}), `blendshapes` (52 ARKit coefficients: browDownLeft, browDownRight, browInnerUp, browOuterUpLeft, browOuterUpRight, cheekPuff, cheekSquintLeft, cheekSquintRight, eyeBlinkLeft, eyeBlinkRight, eyeLookDownLeft, eyeLookDownRight, eyeLookInLeft, eyeLookInRight, eyeLookOutLeft, eyeLookOutRight, eyeLookUpLeft, eyeLookUpRight, eyeSquintLeft, eyeSquintRight, eyeWideLeft, eyeWideRight, jawForward, jawLeft, jawOpen, jawRight, mouthClose, mouthDimpleLeft, mouthDimpleRight, mouthFrownLeft, mouthFrownRight, mouthFunnel, mouthLeft, mouthLowerDownLeft, mouthLowerDownRight, mouthPressLeft, mouthPressRight, mouthPucker, mouthRight, mouthRollLower, mouthRollUpper, mouthShrugLower, mouthShrugUpper, mouthSmileLeft, mouthSmileRight, mouthStretchLeft, mouthStretchRight, mouthUpperUpLeft, mouthUpperUpRight, noseSneerLeft, noseSneerRight, tongueOut: all 0-1).\n\n**`viji.video.cv.hands: HandData[]`**\nEach hand: `id` (number), `handedness` ('left'|'right'), `confidence` (0-1), `bounds` ({x,y,width,height}), `landmarks` ({x,y,z}[], 21 points), `palm` ({x,y,z}), `gestures` ({fist,openPalm,peace,thumbsUp,thumbsDown,pointing,iLoveYou} all 0-1 confidence).\n\n**`viji.video.cv.pose: PoseData | null`**\n`confidence` (0-1), `landmarks` ({x,y,z,visibility}[], 33 points), plus body-part arrays: `face` ({x,y}[]), `torso`, `leftArm`, `rightArm`, `leftLeg`, `rightLeg`.\n\n**`viji.video.cv.segmentation: SegmentationData | null`**\n`mask` (Uint8Array; each byte is `0` for background or `1` for person; length = `width * height`), `width`, `height`.\n\n### Input: Pointer (unified mouse/touch): `viji.pointer`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `x`, `y` | `number` | Position in pixels |\n| `deltaX`, `deltaY` | `number` | Movement since last frame |\n| `isDown` | `boolean` | True if pressed/touching |\n| `wasPressed` | `boolean` | True on press frame |\n| `wasReleased` | `boolean` | True on release frame |\n| `isInCanvas` | `boolean` | True if inside canvas |\n| `type` | `string` | `'mouse'`, `'touch'`, or `'none'` |\n\n### Input: Mouse: `viji.mouse`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `x`, `y` | `number` | Position in pixels |\n| `isInCanvas` | `boolean` | Inside canvas bounds |\n| `isPressed` | `boolean` | Any button pressed |\n| `leftButton`, `rightButton`, `middleButton` | `boolean` | Specific buttons |\n| `deltaX`, `deltaY` | `number` | Movement delta |\n| `wheelDelta` | `number` | Scroll wheel delta |\n| `wheelX`, `wheelY` | `number` | Horizontal/vertical scroll |\n| `wasPressed`, `wasReleased`, `wasMoved` | `boolean` | Frame-edge events |\n\n### Input: Keyboard: `viji.keyboard`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isPressed(key)` | `boolean` | True while key is held |\n| `wasPressed(key)` | `boolean` | True on key-down frame |\n| `wasReleased(key)` | `boolean` | True on key-up frame |\n| `activeKeys` | `Set<string>` | Currently held keys |\n| `pressedThisFrame` | `Set<string>` | Keys pressed this frame |\n| `releasedThisFrame` | `Set<string>` | Keys released this frame |\n| `lastKeyPressed` | `string` | Most recent key-down |\n| `lastKeyReleased` | `string` | Most recent key-up |\n| `shift`, `ctrl`, `alt`, `meta` | `boolean` | Modifier states |\n\n### Input: Touch: `viji.touches`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `count` | `number` | Active touch count |\n| `points` | `TouchPoint[]` | All active touches |\n| `started` | `TouchPoint[]` | Touches started this frame |\n| `moved` | `TouchPoint[]` | Touches moved this frame |\n| `ended` | `TouchPoint[]` | Touches ended this frame |\n| `primary` | `TouchPoint\\|null` | First active touch |\n\n**TouchPoint:** `id`, `x`, `y`, `pressure`, `radius`, `radiusX`, `radiusY`, `rotationAngle`, `force`, `isInCanvas`, `deltaX`, `deltaY`, `velocity` ({x,y}), `isNew`, `isActive`, `isEnding`.\n\n### Device Sensors: `viji.device`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `motion` | `DeviceMotionData\\|null` | Accelerometer/gyroscope |\n| `orientation` | `DeviceOrientationData\\|null` | Device orientation |\n\n**DeviceMotionData:** `acceleration` ({x,y,z} m/s²), `accelerationIncludingGravity`, `rotationRate` ({alpha,beta,gamma} deg/s), `interval` (ms).\n**DeviceOrientationData:** `alpha` (0-360° compass), `beta` (−180-180° tilt), `gamma` (−90-90° tilt), `absolute` (boolean).\n\n### External Devices: `viji.devices`\n\nArray of connected external devices. Each `DeviceState`:\n`id` (string), `name` (string), `motion` (DeviceMotionData|null), `orientation` (DeviceOrientationData|null), `video` (VideoAPI|null, same as viji.video but without CV), `audio` (AudioStreamAPI|null, lightweight analysis only; no beat/BPM/triggers).\n\n### Streams: `viji.videoStreams`\n\n`VideoAPI[]`: additional video sources provided by the host application (used by the compositor for scene mixing). May be empty. Each element has the same shape as `viji.video`.\n\n### Streams: `viji.audioStreams`\n\n`AudioStreamAPI[]`: additional audio sources from the host (e.g. multi-source mixing). May be empty. Lightweight interface: volume, bands, spectral features, `getFrequencyData()`, `getWaveform()`: **not** the full `AudioAPI` (no beat detection, BPM, triggers, or events).\n\n### External Libraries\n\n```javascript\nconst THREE = await import('https://esm.sh/three@0.160.0');\nconst renderer = new THREE.WebGLRenderer({ canvas: viji.canvas, antialias: true });\nrenderer.setSize(viji.width, viji.height, false); // false = no CSS styles\n```\n\nALWAYS pin library versions. ALWAYS pass `viji.canvas` to the renderer. Handle resize in `render()`.\n\n## BEST PRACTICES\n\n1. NEVER use `viji.time * speed.value`: use a `deltaTime` accumulator instead (see rule 6). Same for nested: never multiply an accumulator by another parameter.\n2. Guard audio/video with `isConnected` checks.\n3. Pre-allocate all objects/arrays at top level: never inside `render()`.\n4. For CV, use toggle parameters: never enable by default.\n5. ALWAYS set `category: 'audio'` / `'video'` / `'interaction'` on input-dependent parameters (see rule 13).\n6. For WebGL scenes with Three.js, handle resize by comparing `viji.width/height` with previous values.\n\n## TEMPLATE\n\n```javascript\nconst bgColor = viji.color('#1a1a2e', { label: 'Background' });\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nconst count = viji.slider(12, { min: 3, max: 30, step: 1, label: 'Count' });\n\nlet angle = 0;\n\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n angle += speed.value * viji.deltaTime;\n\n ctx.fillStyle = bgColor.value;\n ctx.fillRect(0, 0, viji.width, viji.height);\n\n const cx = viji.width / 2;\n const cy = viji.height / 2;\n const radius = Math.min(viji.width, viji.height) * 0.3;\n const dotSize = Math.min(viji.width, viji.height) * 0.02;\n const n = Math.floor(count.value);\n\n for (let i = 0; i < n; i++) {\n const a = angle + (i / n) * Math.PI * 2;\n const x = cx + Math.cos(a) * radius;\n const y = cy + Math.sin(a) * radius;\n const hue = (i / n) * 360;\n ctx.beginPath();\n ctx.arc(x, y, dotSize, 0, Math.PI * 2);\n ctx.fillStyle = `hsl(${hue}, 80%, 60%)`;\n ctx.fill();\n }\n}\n```\n\nNow help the artist build a Viji native scene based on their description below.\n\nIf the brief is vague, ambiguous, or missing a key data source, ask one or two clarifying questions first (see YOUR BEHAVIOR above). Otherwise, proceed.\n\nWhen you generate the scene:\n- Follow every rule in this prompt.\n- Use `viji.deltaTime` for animation. Use parameters for anything the user might want to adjust. Check `isConnected` before using audio or video.\n- Output the scene code in a single fenced code block.\n- After the code block, write a short explanation (a few sentences) of how the scene works and what the artist can tweak.\n- Invite the artist to ask for changes.\n````\n\n## Usage\n\n1. Copy the entire prompt block above.\n2. Paste it into your AI assistant (ChatGPT, Claude, etc.).\n3. After the prompt, describe the scene you want: be as specific as you like.\n4. The AI will return a complete Viji native scene.\n\n> [!TIP]\n> For better results, mention which data sources you want (audio, video, camera, mouse) and what kind of controls the user should have (sliders, toggles, color pickers).\n\n## Related\n\n- [Create Your First Scene](/ai-prompts/create-first-scene): guided prompt for beginners\n- [Prompting Tips](/ai-prompts/prompting-tips): how to get better results from AI\n- [Native Quick Start](/native/quickstart): your first Viji native scene\n- [Native API Reference](/native/api-reference): full API reference\n- [Best Practices](/getting-started/best-practices): essential patterns for reliable scenes\n- [Common Mistakes](/getting-started/common-mistakes): pitfalls to avoid"
|
|
1328
|
+
"markdown": "# Prompt: Native Scenes\n\nCopy the prompt below and paste it into your AI assistant. Then describe the scene you want. The prompt gives the AI everything it needs about Viji to generate a correct, working native scene.\n\n## The Prompt\n\n````\nYou are generating a Viji native scene: a creative visual that runs inside an OffscreenCanvas Web Worker.\nArtists describe what they want; you collaborate with them to produce complete, working scene code. Apply every rule below exactly.\n\n## YOUR BEHAVIOR\n\n1. **Clarify when needed.** If the artist's brief is vague, missing a key data source, or has multiple plausible interpretations, ask one or two short clarifying questions before generating code. Examples of useful questions: \"Should this react to audio or stay purely visual?\", \"Should it use the camera or only mouse input?\", \"Roughly how dense / how minimal do you want it?\". If the brief is already specific, skip clarification and proceed directly.\n2. **Generate.** Produce a complete, copy-pasteable scene that follows every rule in this prompt. Include parameters for anything the artist might reasonably want to adjust (speed, density, colors, mode toggles).\n3. **Explain.** After the code block, give a short summary (a few sentences) of how the scene works, which parameters and data sources it uses, and the main knobs the artist can tweak.\n4. **Iterate.** Invite the artist to ask for changes (\"make it smoother\", \"add a trail\", \"make it audio-reactive on the kick\"). Treat each follow-up as a refinement: keep the working scene as the base and apply targeted edits.\n\n## REFERENCE (source of truth)\n\nThe resources below are AUTHORITATIVE. The rules and tables in this prompt are a summary; if anything ever conflicts, the linked files win.\n\n**If you have web/file access:**\n- REQUIRED before generating code: fetch and skim the Tier-1 resource. Use it to verify exact API names and types.\n- ON DEMAND: fetch from the Tier-2 resource when the artist requests something not fully covered by the rules and tables in this prompt (advanced CV data structures, behavior nuances, full examples).\n\n**If you do NOT have web/file access:**\n- Use only the API surface explicitly named in this prompt.\n- Never invent property, method, or uniform names from memory.\n- If the artist asks for something not covered here, say so and ask the artist what they want; do NOT fabricate.\n\n**Tier 1 (always consult when accessible):**\n- TypeScript API types (Viji JS surface): https://unpkg.com/@viji-dev/core/dist/artist-global.d.ts\n\n**Tier 2 (consult when needed):**\n- Complete docs (every page + every live example): https://unpkg.com/@viji-dev/core/dist/docs-api.js\n\n**Last-resort lookup:** https://www.npmjs.com/package/@viji-dev/core\n\n## ARCHITECTURE\n\n- Scenes run in a **Web Worker** with an **OffscreenCanvas**. There is no DOM.\n- The global `viji` object provides canvas, timing, audio, video, CV, input, sensors, and parameters.\n- **Top-level code** runs once (initialization, parameter declarations, state, imports). Top-level `await` is supported for dynamic imports.\n- **`function render(viji) { ... }`** is called every frame. This is where you draw.\n- There is **no `setup()` function** in native scenes. All initialization goes at the top level.\n\n## RULES\n\n1. NEVER access `window`, `document`, `Image()`, `localStorage`, or any DOM API. `fetch()` and `await import()` ARE available.\n2. ALWAYS declare parameters at the TOP LEVEL, never inside `render()`:\n ```javascript\n const speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\n function render(viji) { /* use speed.value */ }\n ```\n3. ALWAYS read parameters via `.value`: `speed.value`, `color.value`, `toggle.value`. Color parameters also expose `.rgb` (`{ r, g, b }` in 0..255) and `.hsb` (`{ h, s, b }`, h in 0..360, s/b in 0..100). Use them instead of parsing hex by hand. Color defaults accept hex (`'#ff6600'`, `'#f60'`), `{ r, g, b }` (0..255), `{ h, s, b }` (0..360 / 0..100), and CSS `'rgb(...)'` / `'hsl(...)'` strings.\n4. ALWAYS use `viji.width` and `viji.height` for canvas dimensions. NEVER hardcode pixel sizes.\n5. ALWAYS use `viji.time` or `viji.deltaTime` for animation. NEVER count frames or assume a fixed frame rate.\n - `viji.time`: elapsed seconds. Use for constant-speed oscillations only: `sin(viji.time * 2.0)`.\n - `viji.deltaTime`: seconds since last frame. Use for accumulators: `angle += speed.value * viji.deltaTime;`\n6. NEVER multiply `viji.time` by a parameter for animation speed: it causes jumps when the parameter changes. ALWAYS use a `deltaTime` accumulator:\n ```javascript\n // WRONG: jumps when speed changes:\n const t = viji.time * speed.value;\n // RIGHT: smooth:\n let phase = 0; // top level\n phase += speed.value * viji.deltaTime; // in render()\n ```\n This also applies to **nested** multiplications. If `phase` is already an accumulator, NEVER multiply it by another parameter:\n ```javascript\n // WRONG: jumps when rotSpeed changes:\n const rot = phase * rotSpeed.value;\n // RIGHT: give it its own accumulator:\n let rotPhase = 0; // top level\n rotPhase += speed.value * rotSpeed.value * viji.deltaTime; // in render()\n ```\n7. NEVER allocate objects, arrays, or strings inside `render()`. Pre-allocate at the top level and reuse.\n8. ALWAYS call `viji.useContext()` to get a canvas context. Choose ONE type and use it for the entire scene:\n - `viji.useContext('2d')`: Canvas 2D\n - `viji.useContext('webgl')`: WebGL 1\n - `viji.useContext('webgl2')`: WebGL 2\n Calling a different type after the first returns `null`.\n9. ALWAYS check `viji.audio.isConnected` before using audio data.\n10. ALWAYS check `viji.video.isConnected && viji.video.currentFrame` before drawing video.\n11. NEVER enable CV features by default. Use a toggle parameter so the user can opt in:\n ```javascript\n const useFace = viji.toggle(false, { label: 'Enable Face Detection', category: 'video' });\n // In render:\n if (useFace.value) await viji.video.cv.enableFaceDetection(true);\n else await viji.video.cv.enableFaceDetection(false);\n ```\n12. Be mindful of WebGL context limits: each CV feature uses its own WebGL context for ML. Enabling too many can cause context loss.\n13. ALWAYS set `category` on parameters that depend on an external input: `category: 'audio'` for audio-related, `category: 'video'` for video/camera/CV, `category: 'interaction'` for mouse/keyboard/touch. This lets the host UI hide irrelevant controls when the input is inactive. **Use creative-strength sliders, not on/off toggles**: the host UI already controls whether each input is wired up, so a scene-level `toggle(true, { label: 'Audio Reactive' })` just duplicates the host switch. CV feature toggles (`enableFaceDetection`, etc.) are the exception and stay opt-in.\n ```javascript\n // Right: creative-strength sliders with the matching category.\n const bassSensitivity = viji.slider(1.5, { min: 0, max: 3, label: 'Bass Sensitivity', group: 'audio', category: 'audio' });\n const mouseAttraction = viji.slider(0.5, { min: 0, max: 1, label: 'Mouse Attraction', group: 'interaction', category: 'interaction' });\n\n // Wrong: scene-level on/off toggle for an input the host already gates.\n // const audioReact = viji.toggle(true, { label: 'Audio Reactive', category: 'audio' });\n // const followMouse = viji.toggle(true, { label: 'Follow Mouse', category: 'interaction' });\n ```\n14. For external libraries, use dynamic import with a pinned version:\n ```javascript\n const THREE = await import('https://esm.sh/three@0.160.0');\n ```\n Pass `viji.canvas` to the library's renderer. ALWAYS pass `false` as the third argument to Three.js `setSize()`.\n\n## COMPLETE API REFERENCE\n\n### Canvas & Context\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `viji.canvas` | `OffscreenCanvas` | The canvas element |\n| `viji.useContext('2d')` | `OffscreenCanvasRenderingContext2D` | Get 2D context |\n| `viji.useContext('webgl')` | `WebGLRenderingContext` | Get WebGL 1 context |\n| `viji.useContext('webgl2')` | `WebGL2RenderingContext` | Get WebGL 2 context |\n| `viji.ctx` | `OffscreenCanvasRenderingContext2D` | Shortcut (after useContext('2d')) |\n| `viji.gl` | `WebGLRenderingContext` | Shortcut (after useContext('webgl')) |\n| `viji.width` | `number` | Current canvas width in pixels |\n| `viji.height` | `number` | Current canvas height in pixels |\n\n### Timing\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `viji.time` | `number` | Seconds since scene start |\n| `viji.deltaTime` | `number` | Seconds since last frame |\n| `viji.frameCount` | `number` | Total frames rendered |\n| `viji.fps` | `number` | Target frame rate (based on host frame-rate mode) |\n\n### Parameters\n\nDeclare at top level. Read `.value` inside `render()`. All support `{ label, description?, group?, category? }`.\nCategory values: `'audio'`, `'video'`, `'interaction'`, `'general'`.\n\n```javascript\nviji.slider(default, { min?, max?, step?, label, group?, category? }) // { value: number }\nviji.color(default, { label, group?, category? }) // { value: '#rrggbb', rgb: { r, g, b } in 0..255, hsb: { h: 0..360, s/b: 0..100 } }\nviji.toggle(default, { label, group?, category? }) // { value: boolean }\nviji.select(default, { options: [...], label, group?, category? }) // { value: string|number }\nviji.number(default, { min?, max?, step?, label, group?, category? }) // { value: number }\nviji.text(default, { label, group?, category?, maxLength? }) // { value: string }\nviji.image(null, { label, group?, category? }) // { value: ImageBitmap|null }\nviji.button({ label, description?, group?, category? }) // { value: boolean } (true one frame)\nviji.coordinate(default, { step?, label, group?, category? }) // { value: { x, y } } (both -1 to 1)\n```\n\n### Audio: `viji.audio`\n\nALWAYS check `viji.audio.isConnected` first.\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isConnected` | `boolean` | Whether audio source is active |\n| `volume.current` | `number` | RMS volume 0-1 |\n| `volume.peak` | `number` | Peak amplitude 0-1 |\n| `volume.smoothed` | `number` | Smoothed volume (200ms decay) |\n| `bands.low` | `number` | 20-120 Hz energy 0-1 |\n| `bands.lowMid` | `number` | 120-400 Hz energy 0-1 |\n| `bands.mid` | `number` | 400-1600 Hz energy 0-1 |\n| `bands.highMid` | `number` | 1600-6000 Hz energy 0-1 |\n| `bands.high` | `number` | 6000-16000 Hz energy 0-1 |\n| `bands.lowSmoothed` … `bands.highSmoothed` | `number` | Smoothed variants of each band |\n| `beat.kick` | `number` | Kick energy curve 0-1, 300ms decay; peaks on each detected kick |\n| `beat.snare` | `number` | Snare energy curve 0-1, 300ms decay |\n| `beat.hat` | `number` | Hi-hat energy curve 0-1, 300ms decay |\n| `beat.any` | `number` | Any-beat energy curve 0-1, 300ms decay |\n| `beat.kickSmoothed` … `beat.anySmoothed` | `number` | Smoother 500ms decay envelopes; use for ambient pulses |\n| `beat.triggers.kick` | `boolean` | True for exactly one frame on a kick, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.snare` | `boolean` | True for exactly one frame on a snare, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.hat` | `boolean` | True for exactly one frame on a hi-hat, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.any` | `boolean` | True for exactly one frame on any beat, then auto-resets |\n| `beat.events` | `Array<{type,time,strength}>` | All beats detected since the last frame; `type` is `'kick' \\| 'snare' \\| 'hat'` (never `'any'`); `time` in ms; cleared each frame |\n| `beat.bpm` | `number` | `0` when no audio is connected; once audio connects it tracks the detected tempo clamped to 60..240, with `120` as a fallback before lock-on |\n| `beat.confidence` | `number` | BPM tracking confidence 0-1 |\n| `beat.isLocked` | `boolean` | True on stable tempo lock |\n| `spectral.brightness` | `number` | Spectral centroid 0-1 |\n| `spectral.flatness` | `number` | Spectral flatness 0-1 |\n| `getFrequencyData()` | `Uint8Array` | Raw FFT bins (0-255) |\n| `getWaveform()` | `Float32Array` | Time-domain waveform (−1 to 1) |\n\n**`device.audio`** (when an external device in `viji.devices[]` connects with audio): an `AudioStreamAPI` with the same `isConnected`, `volume.{current,peak,smoothed}`, `bands.{low...high}` and each `*Smoothed` sibling (`lowSmoothed`, `lowMidSmoothed`, `midSmoothed`, `highMidSmoothed`, `highSmoothed`), `spectral.{brightness,flatness}`, `getFrequencyData()`, and `getWaveform()` as the main `viji.audio` table. **No** `beat`, BPM, triggers, or events: those are main-audio only. Host-supplied additional audio sources (`viji.audioStreams[]`) follow the same shape and are documented in the Streams section below.\n\n### Video: `viji.video`\n\nALWAYS check `viji.video.isConnected` first. Check `currentFrame` before drawing.\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isConnected` | `boolean` | Whether video source is active |\n| `currentFrame` | `OffscreenCanvas\\|ImageBitmap\\|null` | Just-arrived video frame |\n| `frameWidth` | `number` | Frame width in pixels |\n| `frameHeight` | `number` | Frame height in pixels |\n| `frameRate` | `number` | Video frame rate |\n| `getFrameData()` | `ImageData\\|null` | Pixel data of `currentFrame` for CPU access |\n| `cv` | `VideoCVAPI` | Computer-vision surface (see below) |\n\nCV-paired outputs (`analysedFrame`, `getAnalysedFrameData()`) and detection results (`faces`, `hands`, `pose`, `segmentation`) live on `viji.video.cv`, not on `viji.video` directly. See the Computer Vision section below.\n\n**Drawing video: preserve aspect ratio.** Camera frames almost never match the canvas aspect. Drawing to `(0, 0, viji.width, viji.height)` stretches the video and misaligns CV bounds. Define this `videoFit` helper at module scope and use it for every video / CV scene:\n\n```javascript\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nconst v = videoFit(viji); // 'cover' (default) or 'contain'\nctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n// CV coords are normalized 0-1 to the source video frame, not the canvas.\n// Map them through v to align with the fitted video:\nviji.video.cv.faces.forEach(face => {\n const bx = v.x + face.bounds.x * v.width;\n const by = v.y + face.bounds.y * v.height;\n const bw = face.bounds.width * v.width;\n const bh = face.bounds.height * v.height;\n ctx.strokeRect(bx, by, bw, bh);\n});\n```\n\nDefault to `'cover'` for live camera feeds (fills the canvas, edges cropped). Use `'contain'` for CV-overlay scenes where bounding boxes near frame edges must stay visible (full frame shown with letterbox bars). Stretching directly to `(0, 0, viji.width, viji.height)` is allowed only when distortion is intentional.\n\n**`currentFrame` vs `analysedFrame`.** Default to `currentFrame` for displayed video. Reach for `viji.video.cv.analysedFrame ?? viji.video.currentFrame` only when the effect reads pixels from the displayed frame at CV-derived positions (compositing the segmentation mask onto the body, sampling skin under a face landmark, warping the face along its mesh, texture-mapped face filters). For drawing landmark dots, particles, or any overlay that doesn't sample the displayed frame at CV positions, `currentFrame` is the better default: `analysedFrame` advances only when MediaPipe completes an inference, so reaching for it without a reason makes the displayed video stutter or hold between inferences.\n\n### Computer Vision: `viji.video.cv` & `viji.video.cv.faces/hands/pose/segmentation`\n\nEach CV feature is independent and populates only its own subset of fields. Each active feature uses its own WebGL context for MediaPipe — enable only what you need (4+ on low-end hardware risks context-limit failures).\n\n```javascript\n// Verbs return Promise<void>. await is optional. Safe to call from module\n// scope (always-on CV) or per-frame inside render() gated by a viji.toggle(...)\n// (opt-in CV). Idempotent + reference-counted.\nawait viji.video.cv.enableFaceDetection(true/false); // populates face.bounds, face.center, face.confidence\nawait viji.video.cv.enableFaceMesh(true/false); // populates face.landmarks (468 pts) + face.headPose\nawait viji.video.cv.enableEmotionDetection(true/false); // populates face.expressions + face.blendshapes; also loads landmarker (face.landmarks + face.headPose populated)\nawait viji.video.cv.enableHandTracking(true/false); // populates viji.video.cv.hands[]\nawait viji.video.cv.enablePoseDetection(true/false); // populates viji.video.cv.pose\nawait viji.video.cv.enableBodySegmentation(true/false); // populates viji.video.cv.segmentation\nviji.video.cv.getActiveFeatures(); // CVFeature[]\nviji.video.cv.isProcessing(); // boolean\n```\n\nCV-paired outputs (also on `viji.video.cv`):\n- `analysedFrame: OffscreenCanvas | null`: the exact frame that produced the current CV results. `null` until the first inference lands. Common fallback pattern: `viji.video.cv.analysedFrame ?? viji.video.currentFrame`.\n- `getAnalysedFrameData(): ImageData | null`: pixel data of `analysedFrame`.\n\n**`viji.video.cv.faces: FaceData[]`**\n`id` always present. Other fields populated only by their source model and read `null` (or `[]` for `landmarks`) otherwise — always null-check.\n- `bounds: {x,y,width,height} | null` (normalized 0..1) — populated by face detection\n- `center: {x,y} | null` (normalized 0..1) — populated by face detection\n- `confidence: number | null` (0..1) — populated by face detection\n- `landmarks: {x,y,z?}[]` (empty `[]` unless face mesh enabled; 468 pts when populated)\n- `headPose: {pitch, yaw, roll} | null` (degrees) — populated by face mesh\n- `expressions: {neutral, happy, sad, angry, surprised, disgusted, fearful} | null` (0..1 each) — populated by emotion detection\n- `blendshapes | null` (52 ARKit coefficients 0..1) — populated by emotion detection. Names: browDownLeft/Right, browInnerUp, browOuterUpLeft/Right, cheekPuff, cheekSquintLeft/Right, eyeBlinkLeft/Right, eyeLookDownLeft/Right, eyeLookInLeft/Right, eyeLookOutLeft/Right, eyeLookUpLeft/Right, eyeSquintLeft/Right, eyeWideLeft/Right, jawForward/Left/Open/Right, mouthClose/DimpleLeft/Right/FrownLeft/Right/Funnel/Left/LowerDownLeft/Right/PressLeft/Right/Pucker/Right/RollLower/Upper/ShrugLower/Upper/SmileLeft/Right/StretchLeft/Right/UpperUpLeft/Right, noseSneerLeft/Right, tongueOut.\n\nIf you need `bounds` while only running face mesh, either also enable face detection or compute it from `face.landmarks` min/max.\n\n**`viji.video.cv.hands: HandData[]`**\nEach hand: `id` (number), `handedness` ('left'|'right'), `confidence` (0-1), `bounds` ({x,y,width,height}), `landmarks` ({x,y,z}[], 21 points), `palm` ({x,y,z}), `gestures` ({fist,openPalm,peace,thumbsUp,thumbsDown,pointing,iLoveYou} all 0-1 confidence).\n\n**`viji.video.cv.pose: PoseData | null`**\n`confidence` (0-1), `landmarks` ({x,y,z,visibility}[], 33 points), plus body-part arrays: `face` ({x,y}[]), `torso`, `leftArm`, `rightArm`, `leftLeg`, `rightLeg`.\n\n**`viji.video.cv.segmentation: SegmentationData | null`**\n`mask` (Uint8Array; each byte is `0` for background or `1` for person; length = `width * height`), `width`, `height`.\n\n### Input: Pointer (unified mouse/touch): `viji.pointer`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `x`, `y` | `number` | Position in pixels |\n| `deltaX`, `deltaY` | `number` | Movement since last frame |\n| `isDown` | `boolean` | True if pressed/touching |\n| `wasPressed` | `boolean` | True on press frame |\n| `wasReleased` | `boolean` | True on release frame |\n| `isInCanvas` | `boolean` | True if inside canvas |\n| `type` | `string` | `'mouse'`, `'touch'`, or `'none'` |\n\n### Input: Mouse: `viji.mouse`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `x`, `y` | `number` | Position in pixels |\n| `isInCanvas` | `boolean` | Inside canvas bounds |\n| `isPressed` | `boolean` | Any button pressed |\n| `leftButton`, `rightButton`, `middleButton` | `boolean` | Specific buttons |\n| `deltaX`, `deltaY` | `number` | Movement delta |\n| `wheelDelta` | `number` | Scroll wheel delta |\n| `wheelX`, `wheelY` | `number` | Horizontal/vertical scroll |\n| `wasPressed`, `wasReleased`, `wasMoved` | `boolean` | Frame-edge events |\n\n### Input: Keyboard: `viji.keyboard`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isPressed(key)` | `boolean` | True while key is held |\n| `wasPressed(key)` | `boolean` | True on key-down frame |\n| `wasReleased(key)` | `boolean` | True on key-up frame |\n| `activeKeys` | `Set<string>` | Currently held keys |\n| `pressedThisFrame` | `Set<string>` | Keys pressed this frame |\n| `releasedThisFrame` | `Set<string>` | Keys released this frame |\n| `lastKeyPressed` | `string` | Most recent key-down |\n| `lastKeyReleased` | `string` | Most recent key-up |\n| `shift`, `ctrl`, `alt`, `meta` | `boolean` | Modifier states |\n\n### Input: Touch: `viji.touches`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `count` | `number` | Active touch count |\n| `points` | `TouchPoint[]` | All active touches |\n| `started` | `TouchPoint[]` | Touches started this frame |\n| `moved` | `TouchPoint[]` | Touches moved this frame |\n| `ended` | `TouchPoint[]` | Touches ended this frame |\n| `primary` | `TouchPoint\\|null` | First active touch |\n\n**TouchPoint:** `id`, `x`, `y`, `pressure`, `radius`, `radiusX`, `radiusY`, `rotationAngle`, `force`, `isInCanvas`, `deltaX`, `deltaY`, `velocity` ({x,y}), `isNew`, `isActive`, `isEnding`.\n\n### Device Sensors: `viji.device`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `motion` | `DeviceMotionData\\|null` | Accelerometer/gyroscope |\n| `orientation` | `DeviceOrientationData\\|null` | Device orientation |\n\n**DeviceMotionData:** `acceleration` ({x,y,z} m/s²), `accelerationIncludingGravity`, `rotationRate` ({alpha,beta,gamma} deg/s), `interval` (ms).\n**DeviceOrientationData:** `alpha` (0-360° compass), `beta` (−180-180° tilt), `gamma` (−90-90° tilt), `absolute` (boolean).\n\n### External Devices: `viji.devices`\n\nArray of connected external devices. Each `DeviceState`:\n`id` (string), `name` (string), `motion` (DeviceMotionData|null), `orientation` (DeviceOrientationData|null), `video` (VideoAPI|null, same as viji.video but without CV), `audio` (AudioStreamAPI|null, lightweight analysis only; no beat/BPM/triggers).\n\n### Streams: `viji.videoStreams`\n\n`VideoAPI[]`: additional video sources provided by the host application (used by the compositor for scene mixing). May be empty. Each element has the same shape as `viji.video`.\n\n### Streams: `viji.audioStreams`\n\n`AudioStreamAPI[]`: additional audio sources from the host (e.g. multi-source mixing). May be empty. Lightweight interface: volume, bands, spectral features, `getFrequencyData()`, `getWaveform()`: **not** the full `AudioAPI` (no beat detection, BPM, triggers, or events).\n\n### External Libraries\n\n```javascript\nconst THREE = await import('https://esm.sh/three@0.160.0');\nconst renderer = new THREE.WebGLRenderer({ canvas: viji.canvas, antialias: true });\nrenderer.setSize(viji.width, viji.height, false); // false = no CSS styles\n```\n\nALWAYS pin library versions. ALWAYS pass `viji.canvas` to the renderer. Handle resize in `render()`.\n\n## BEST PRACTICES\n\n1. NEVER use `viji.time * speed.value`: use a `deltaTime` accumulator instead (see rule 6). Same for nested: never multiply an accumulator by another parameter.\n2. Guard audio/video with `isConnected` checks.\n3. Pre-allocate all objects/arrays at top level: never inside `render()`.\n4. For CV, use toggle parameters: never enable by default.\n5. ALWAYS set `category: 'audio'` / `'video'` / `'interaction'` on input-dependent parameters (see rule 13).\n6. For WebGL scenes with Three.js, handle resize by comparing `viji.width/height` with previous values.\n\n## TEMPLATE\n\n```javascript\nconst bgColor = viji.color('#1a1a2e', { label: 'Background' });\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nconst count = viji.slider(12, { min: 3, max: 30, step: 1, label: 'Count' });\n\nlet angle = 0;\n\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n angle += speed.value * viji.deltaTime;\n\n ctx.fillStyle = bgColor.value;\n ctx.fillRect(0, 0, viji.width, viji.height);\n\n const cx = viji.width / 2;\n const cy = viji.height / 2;\n const radius = Math.min(viji.width, viji.height) * 0.3;\n const dotSize = Math.min(viji.width, viji.height) * 0.02;\n const n = Math.floor(count.value);\n\n for (let i = 0; i < n; i++) {\n const a = angle + (i / n) * Math.PI * 2;\n const x = cx + Math.cos(a) * radius;\n const y = cy + Math.sin(a) * radius;\n const hue = (i / n) * 360;\n ctx.beginPath();\n ctx.arc(x, y, dotSize, 0, Math.PI * 2);\n ctx.fillStyle = `hsl(${hue}, 80%, 60%)`;\n ctx.fill();\n }\n}\n```\n\nNow help the artist build a Viji native scene based on their description below.\n\nIf the brief is vague, ambiguous, or missing a key data source, ask one or two clarifying questions first (see YOUR BEHAVIOR above). Otherwise, proceed.\n\nWhen you generate the scene:\n- Follow every rule in this prompt.\n- Use `viji.deltaTime` for animation. Use parameters for anything the user might want to adjust. Check `isConnected` before using audio or video.\n- Output the scene code in a single fenced code block.\n- After the code block, write a short explanation (a few sentences) of how the scene works and what the artist can tweak.\n- Invite the artist to ask for changes.\n````\n\n## Usage\n\n1. Copy the entire prompt block above.\n2. Paste it into your AI assistant (ChatGPT, Claude, etc.).\n3. After the prompt, describe the scene you want: be as specific as you like.\n4. The AI will return a complete Viji native scene.\n\n> [!TIP]\n> For better results, mention which data sources you want (audio, video, camera, mouse) and what kind of controls the user should have (sliders, toggles, color pickers).\n\n## Related\n\n- [Create Your First Scene](/ai-prompts/create-first-scene): guided prompt for beginners\n- [Prompting Tips](/ai-prompts/prompting-tips): how to get better results from AI\n- [Native Quick Start](/native/quickstart): your first Viji native scene\n- [Native API Reference](/native/api-reference): full API reference\n- [Best Practices](/getting-started/best-practices): essential patterns for reliable scenes\n- [Common Mistakes](/getting-started/common-mistakes): pitfalls to avoid"
|
|
1329
1329
|
}
|
|
1330
1330
|
]
|
|
1331
1331
|
},
|
|
@@ -1353,7 +1353,7 @@ export const docsApi = {
|
|
|
1353
1353
|
"content": [
|
|
1354
1354
|
{
|
|
1355
1355
|
"type": "text",
|
|
1356
|
-
"markdown": "# Prompt: P5 Scenes\n\nCopy the prompt below and paste it into your AI assistant. Then describe the scene you want. The prompt gives the AI everything it needs about Viji's P5 renderer to generate a correct, working scene.\n\n## The Prompt\n\n````\nYou are generating a Viji P5.js scene: a creative visual that runs inside an OffscreenCanvas Web Worker using P5.js v1.9.4.\nArtists describe what they want; you collaborate with them to produce complete, working scene code. Apply every rule below exactly.\n\n## YOUR BEHAVIOR\n\n1. **Clarify when needed.** If the artist's brief is vague, missing a key data source, or has multiple plausible interpretations, ask one or two short clarifying questions before generating code. Examples: \"Should this react to audio or stay purely visual?\", \"2D canvas or WEBGL / 3D mode?\", \"Should it use the camera or only mouse input?\". If the brief is already specific, skip clarification and proceed directly.\n2. **Generate.** Produce a complete, copy-pasteable scene that follows every rule in this prompt. Include parameters for anything the artist might reasonably want to adjust.\n3. **Explain.** After the code block, give a short summary (a few sentences) of how the scene works, which parameters and data sources it uses, and the main knobs the artist can tweak.\n4. **Iterate.** Invite the artist to ask for changes. Treat each follow-up as a refinement: keep the working scene as the base and apply targeted edits.\n\n## REFERENCE (source of truth)\n\nThe resources below are AUTHORITATIVE. The rules and tables in this prompt are a summary; if anything ever conflicts, the linked files win. Viji pins **p5.js v1.9.4**: when in doubt about a P5 call, the p5.js v1.x reference is the truth.\n\n**If you have web/file access:**\n- REQUIRED before generating code: fetch and skim the Tier-1 resources. Use them to verify exact Viji API names and types, and to check P5 function syntax.\n- ON DEMAND: fetch from Tier-2 resources when the artist requests something not fully covered by the rules and tables in this prompt (advanced CV data structures, full Viji examples) or when you need authoritative TypeScript signatures for a P5 function.\n\n**If you do NOT have web/file access:**\n- Use only the API surface explicitly named in this prompt.\n- Never invent property, method, or P5 function names from memory.\n- If the artist asks for something not covered here, say so and ask the artist what they want; do NOT fabricate.\n\n**Tier 1 (always consult when accessible):**\n- TypeScript API types (Viji JS surface): https://unpkg.com/@viji-dev/core/dist/artist-global.d.ts\n- P5.js v1.x reference (HTML, authoritative for P5 syntax): https://p5js.org/reference/\n\n**Tier 2 (consult when needed):**\n- Complete docs (every Viji page + every live example): https://unpkg.com/@viji-dev/core/dist/docs-api.js\n- Bundled Viji + P5.js v1.9.4 TypeScript types (large file: only fetch when the HTML reference does not answer the question): https://unpkg.com/@viji-dev/core/dist/artist-global-p5.d.ts\n\n**Last-resort lookup:** https://www.npmjs.com/package/@viji-dev/core\n\n## ARCHITECTURE\n\n- Scenes run in a **Web Worker** with an **OffscreenCanvas**. There is no DOM.\n- Viji automatically loads **P5.js v1.9.4** when you use `// @renderer p5` or `// @renderer p5 webgl`.\n- The global `viji` object provides canvas, timing, audio, video, CV, input, sensors, and parameters.\n- **Top-level code** runs once (initialization, parameter declarations, state).\n- **`function render(viji, p5) { ... }`** is called every frame. This is where you draw.\n- Optional **`function setup(viji, p5) { ... }`** runs once for configuration (e.g., `p5.colorMode()`).\n- P5 runs in **instance mode**: every P5 function and constant requires the `p5.` prefix.\n\n## RULES\n\n1. ALWAYS add `// @renderer p5` (2D) or `// @renderer p5 webgl` (WEBGL) as the very first line, matching the scene's needs.\n2. ALWAYS use `render(viji, p5)`: not `draw()`. ALWAYS use `setup(viji, p5)`: not `setup()`.\n3. ALWAYS prefix every P5 function and constant with `p5.`:\n - `background(0)` → `p5.background(0)`\n - `fill(255)` → `p5.fill(255)`\n - `PI` → `p5.PI`, `TWO_PI` → `p5.TWO_PI`, `HSB` → `p5.HSB`\n - `createVector(1, 0)` → `p5.createVector(1, 0)`\n - `map(v, 0, 1, 0, 255)` → `p5.map(v, 0, 1, 0, 255)`\n - `noise(x)` → `p5.noise(x)`, `random()` → `p5.random()`\n This applies to ALL P5 functions and constants without exception.\n4. NEVER call `createCanvas()`. The canvas is created and managed by Viji.\n5. NEVER use `preload()`. Use `viji.image(null, { label: 'Name' })` for images, or `fetch()` in `setup()`.\n6. NEVER use P5 event callbacks: `mousePressed()`, `mouseDragged()`, `mouseReleased()`, `keyPressed()`, `keyReleased()`, `keyTyped()`, `touchStarted()`, `touchMoved()`, `touchEnded()`. Check state in `render()`:\n - `mouseIsPressed` → `viji.pointer.isDown` or `viji.mouse.isPressed`\n - `mouseX` / `mouseY` → `viji.pointer.x` / `viji.pointer.y` or `viji.mouse.x` / `viji.mouse.y`\n - `keyIsPressed` → `viji.keyboard.isPressed('keyName')`\n - For press-edge detection: `viji.pointer.wasPressed` / `viji.pointer.wasReleased`.\n7. NEVER use `loadImage()`, `loadFont()`, `loadJSON()`, `loadModel()`, `loadShader()`. Use `viji.image()` or `fetch()`.\n8. NEVER use `p5.frameRate()`, `p5.save()`, `p5.saveCanvas()`, `p5.saveFrames()`.\n9. NEVER use `createCapture()`, `createVideo()`. Use `viji.video.*` instead.\n10. NEVER use `p5.dom` or `p5.sound` libraries. Use Viji parameters for UI and `viji.audio.*` for audio.\n11. NEVER access `window`, `document`, `Image()`, or `localStorage`. `fetch()` IS available.\n12. ALWAYS declare parameters at the TOP LEVEL, never inside `render()` or `setup()`.\n13. ALWAYS read parameters via `.value`: `size.value`, `color.value`, `toggle.value`. Color parameters also expose `.rgb` (`{ r, g, b }` in 0..255; matches `colorMode(RGB, 255)`) and `.hsb` (`{ h, s, b }`, h in 0..360, s/b in 0..100; matches `colorMode(HSB, 360, 100, 100)`); prefer those over parsing hex. Color defaults accept hex (`'#ff6600'`, `'#f60'`), `{ r, g, b }` (0..255), `{ h, s, b }` (0..360 / 0..100), and CSS `'rgb(...)'` / `'hsl(...)'` strings.\n14. ALWAYS use `viji.width` and `viji.height` for canvas dimensions. NEVER hardcode pixel sizes.\n15. ALWAYS use `viji.deltaTime` for frame-rate-independent animation:\n ```javascript\n let angle = 0;\n function render(viji, p5) { angle += speed.value * viji.deltaTime; }\n ```\n16. NEVER multiply `viji.time` by a parameter for animation speed, it causes jumps when the parameter changes. ALWAYS use a `deltaTime` accumulator (rule 15). This also applies to nested multiplications, never multiply an accumulator by another parameter; give each speed its own accumulator:\n ```javascript\n // WRONG: jumps: const t = viji.time * speed.value;\n // WRONG: nested: const rot = phase * rotSpeed.value;\n // RIGHT:\n let phase = 0, rotPhase = 0; // top level\n phase += speed.value * viji.deltaTime;\n rotPhase += speed.value * rotSpeed.value * viji.deltaTime;\n ```\n17. NEVER allocate objects, arrays, or strings inside `render()`. Pre-allocate at the top level and reuse.\n18. For image parameters displayed with P5, use `.p5` (not `.value`) with `p5.image()`:\n ```javascript\n const photo = viji.image(null, { label: 'Photo' });\n function render(viji, p5) {\n if (photo.value) p5.image(photo.p5, 0, 0, viji.width, viji.height);\n }\n ```\n19. For video frames: in **2D** (`// @renderer p5`) you may use `p5.image(viji.video.currentFrame, ...)` or `p5.drawingContext.drawImage(...)`. In **WEBGL** (`// @renderer p5 webgl`), use `p5.image(viji.video.currentFrame, ...)` only: `p5.drawingContext` is WebGL, not Canvas 2D. **Always preserve the source aspect ratio with the `videoFit` helper.** Drawing to `(0, 0, viji.width, viji.height)` stretches the video and misaligns CV bounds.\n ```javascript\n function videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n }\n if (viji.video.isConnected && viji.video.currentFrame) {\n const v = videoFit(viji); // 'cover' (default) or 'contain'\n p5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n }\n // CV coords (face.bounds, hand.landmarks, etc.) are normalized 0-1 to the\n // SOURCE video frame, not the canvas. Map through v: x = v.x + pt.x * v.width.\n ```\n Default to `'cover'` for live cameras. Use `'contain'` for CV-overlay scenes where bounding boxes near frame edges must stay visible. Stretching with `(0, 0, viji.width, viji.height)` is allowed only when distortion is intentional.\n20. `p5.createGraphics()` works (creates OffscreenCanvas internally). Use for off-screen buffers.\n21. Fonts: `p5.textFont()` only with CSS generic names (`monospace`, `serif`, `sans-serif`). `loadFont()` is NOT available.\n22. `p5.tint()` and `p5.blendMode()` work normally.\n23. **Canvas mode:** Use `// @renderer p5` for a **2D** main canvas. For **WEBGL / 3D**, the first line MUST be `// @renderer p5 webgl`. NEVER call `createCanvas()` or `createCanvas(..., p5.WEBGL)`: Viji creates the canvas in the correct mode.\n24. In **WEBGL** scenes, `p5.drawingContext` is a WebGL context: never use Canvas 2D-only APIs on it. Use P5 3D drawing, `p5.image()` / textures for images and video.\n25. `p5.createGraphics(w, h)` is **2D only**. `createGraphics(w, h, p5.WEBGL)` is NOT supported.\n26. `p5.pixelDensity()` defaults to 1 in the worker. `p5.loadPixels()` and `p5.pixels[]` work (2D scenes; WEBGL pixel readback follows P5.js rules).\n27. ALWAYS check `viji.audio.isConnected` before using audio data.\n28. ALWAYS check `viji.video.isConnected && viji.video.currentFrame` before drawing video.\n29. NEVER enable CV features by default: use toggle parameters for user opt-in.\n30. ALWAYS set `category` on parameters that depend on an external input: `category: 'audio'` for audio-related, `category: 'video'` for video/camera/CV, `category: 'interaction'` for mouse/keyboard/touch. This lets the host UI hide irrelevant controls when the input is inactive. **Use creative-strength sliders, not on/off toggles**: the host UI already controls whether each input is wired up, so a scene-level `toggle(true, { label: 'Audio Reactive' })` just duplicates the host switch. CV feature toggles (`enableFaceDetection`, etc.) are the exception and stay opt-in.\n ```javascript\n // Right: creative-strength sliders with the matching category.\n const bassSensitivity = viji.slider(1.5, { min: 0, max: 3, label: 'Bass Sensitivity', group: 'audio', category: 'audio' });\n const mouseAttraction = viji.slider(0.5, { min: 0, max: 1, label: 'Mouse Attraction', group: 'interaction', category: 'interaction' });\n\n // Wrong: scene-level on/off toggle for an input the host already gates.\n // const audioReact = viji.toggle(true, { label: 'Audio Reactive', category: 'audio' });\n // const followMouse = viji.toggle(true, { label: 'Follow Mouse', category: 'interaction' });\n ```\n31. `viji.useContext()` is NOT available in P5 scenes: the canvas is managed by P5.\n\n## COMPLETE API REFERENCE\n\nAll `viji.*` members are identical to the native renderer (same object, same types).\n\n### Canvas & Timing\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `viji.canvas` | `OffscreenCanvas` | The canvas element (managed by P5) |\n| `viji.width` | `number` | Current canvas width in pixels |\n| `viji.height` | `number` | Current canvas height in pixels |\n| `viji.time` | `number` | Seconds since scene start |\n| `viji.deltaTime` | `number` | Seconds since last frame |\n| `viji.frameCount` | `number` | Total frames rendered |\n| `viji.fps` | `number` | Target frame rate (based on host frame-rate mode) |\n\nNote: `viji.useContext()` is NOT available in P5. The canvas context is managed by P5 internally.\n\n### Parameters\n\nDeclare at top level. Read `.value` inside `render()`. All support `{ label, description?, group?, category? }`.\nCategory values: `'audio'`, `'video'`, `'interaction'`, `'general'`.\n\n```javascript\nviji.slider(default, { min?, max?, step?, label, group?, category? }) // { value: number }\nviji.color(default, { label, group?, category? }) // { value: '#rrggbb', rgb: { r, g, b } in 0..255, hsb: { h: 0..360, s/b: 0..100 } }\nviji.toggle(default, { label, group?, category? }) // { value: boolean }\nviji.select(default, { options: [...], label, group?, category? }) // { value: string|number }\nviji.number(default, { min?, max?, step?, label, group?, category? }) // { value: number }\nviji.text(default, { label, group?, category?, maxLength? }) // { value: string }\nviji.image(null, { label, group?, category? }) // { value: ImageBitmap|null, p5: P5Image }\nviji.button({ label, description?, group?, category? }) // { value: boolean } (true one frame)\nviji.coordinate(default, { step?, label, group?, category? }) // { value: { x, y } } (both -1 to 1)\n```\n\n### Audio: `viji.audio`\n\nALWAYS check `viji.audio.isConnected` first.\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isConnected` | `boolean` | Whether audio source is active |\n| `volume.current` | `number` | RMS volume 0-1 |\n| `volume.peak` | `number` | Peak amplitude 0-1 |\n| `volume.smoothed` | `number` | Smoothed volume (200ms decay) |\n| `bands.low` | `number` | 20-120 Hz energy 0-1 |\n| `bands.lowMid` | `number` | 120-400 Hz energy 0-1 |\n| `bands.mid` | `number` | 400-1600 Hz energy 0-1 |\n| `bands.highMid` | `number` | 1600-6000 Hz energy 0-1 |\n| `bands.high` | `number` | 6000-16000 Hz energy 0-1 |\n| `bands.lowSmoothed` … `bands.highSmoothed` | `number` | Smoothed variants of each band |\n| `beat.kick` | `number` | Kick energy curve 0-1, 300ms decay; peaks on each detected kick |\n| `beat.snare` | `number` | Snare energy curve 0-1, 300ms decay |\n| `beat.hat` | `number` | Hi-hat energy curve 0-1, 300ms decay |\n| `beat.any` | `number` | Any-beat energy curve 0-1, 300ms decay |\n| `beat.kickSmoothed` … `beat.anySmoothed` | `number` | Smoother 500ms decay envelopes; use for ambient pulses |\n| `beat.triggers.kick` | `boolean` | True for exactly one frame on a kick, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.snare` | `boolean` | True for exactly one frame on a snare, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.hat` | `boolean` | True for exactly one frame on a hi-hat, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.any` | `boolean` | True for exactly one frame on any beat, then auto-resets |\n| `beat.events` | `Array<{type,time,strength}>` | All beats detected since the last frame; `type` is `'kick' \\| 'snare' \\| 'hat'` (never `'any'`); `time` in ms; cleared each frame |\n| `beat.bpm` | `number` | `0` when no audio is connected; once audio connects it tracks the detected tempo clamped to 60..240, with `120` as a fallback before lock-on |\n| `beat.confidence` | `number` | BPM tracking confidence 0-1 |\n| `beat.isLocked` | `boolean` | True on stable tempo lock |\n| `spectral.brightness` | `number` | Spectral centroid 0-1 |\n| `spectral.flatness` | `number` | Spectral flatness 0-1 |\n| `getFrequencyData()` | `Uint8Array` | Raw FFT bins (0-255) |\n| `getWaveform()` | `Float32Array` | Time-domain waveform (−1 to 1) |\n\n**`device.audio`** (when an external device in `viji.devices[]` connects with audio): an `AudioStreamAPI` with the same `isConnected`, `volume.{current,peak,smoothed}`, `bands.{low...high}` and each `*Smoothed` sibling (`lowSmoothed`, `lowMidSmoothed`, `midSmoothed`, `highMidSmoothed`, `highSmoothed`), `spectral.{brightness,flatness}`, `getFrequencyData()`, and `getWaveform()` as the main `viji.audio` table. **No** `beat`, BPM, triggers, or events: those are main-audio only. Host-supplied additional audio sources (`viji.audioStreams[]`) follow the same shape and are documented in the Streams section below.\n\n### Video: `viji.video`\n\nALWAYS check `viji.video.isConnected` first. Check `currentFrame` before drawing.\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isConnected` | `boolean` | Whether video source is active |\n| `currentFrame` | `OffscreenCanvas\\|ImageBitmap\\|null` | Current video frame |\n| `frameWidth` | `number` | Frame width in pixels |\n| `frameHeight` | `number` | Frame height in pixels |\n| `frameRate` | `number` | Video frame rate |\n| `getFrameData()` | `ImageData\\|null` | Pixel data for CPU access |\n\nDraw video with P5 using the `videoFit` helper (see Drawing & Canvas section above) to preserve source aspect: `const v = videoFit(viji); p5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);`. Never `(0, 0, viji.width, viji.height)` unless distortion is intentional.\n\n### Computer Vision: `viji.video.cv` & `viji.video.cv.faces/hands/pose/segmentation`\n\nEnable features via toggle parameters (NEVER enable by default):\n\n```javascript\nawait viji.video.cv.enableFaceDetection(true/false);\nawait viji.video.cv.enableFaceMesh(true/false); // implies face detection\nawait viji.video.cv.enableEmotionDetection(true/false); // implies face mesh\nawait viji.video.cv.enableHandTracking(true/false);\nawait viji.video.cv.enablePoseDetection(true/false);\nawait viji.video.cv.enableBodySegmentation(true/false);\nviji.video.cv.getActiveFeatures(); // CVFeature[]\nviji.video.cv.isProcessing(); // boolean\n```\n\nCV-paired outputs (also on `viji.video.cv`, not on `viji.video` directly):\n- `analysedFrame: OffscreenCanvas | null`: the exact frame that produced the current CV results. `null` until the first inference lands. Common fallback pattern: `viji.video.cv.analysedFrame ?? viji.video.currentFrame`.\n- `getAnalysedFrameData(): ImageData | null`: pixel data of `analysedFrame`.\n\n**`viji.video.cv.faces: FaceData[]`**\nEach face: `id` (number), `bounds` ({x,y,width,height}), `center` ({x,y}), `confidence` (0-1), `landmarks` ({x,y,z?}[]), `expressions` ({neutral,happy,sad,angry,surprised,disgusted,fearful} all 0-1), `headPose` ({pitch,yaw,roll}), `blendshapes` (52 ARKit coefficients: browDownLeft, browDownRight, browInnerUp, browOuterUpLeft, browOuterUpRight, cheekPuff, cheekSquintLeft, cheekSquintRight, eyeBlinkLeft, eyeBlinkRight, eyeLookDownLeft, eyeLookDownRight, eyeLookInLeft, eyeLookInRight, eyeLookOutLeft, eyeLookOutRight, eyeLookUpLeft, eyeLookUpRight, eyeSquintLeft, eyeSquintRight, eyeWideLeft, eyeWideRight, jawForward, jawLeft, jawOpen, jawRight, mouthClose, mouthDimpleLeft, mouthDimpleRight, mouthFrownLeft, mouthFrownRight, mouthFunnel, mouthLeft, mouthLowerDownLeft, mouthLowerDownRight, mouthPressLeft, mouthPressRight, mouthPucker, mouthRight, mouthRollLower, mouthRollUpper, mouthShrugLower, mouthShrugUpper, mouthSmileLeft, mouthSmileRight, mouthStretchLeft, mouthStretchRight, mouthUpperUpLeft, mouthUpperUpRight, noseSneerLeft, noseSneerRight, tongueOut: all 0-1).\n\n**`viji.video.cv.hands: HandData[]`**\nEach hand: `id` (number), `handedness` ('left'|'right'), `confidence` (0-1), `bounds` ({x,y,width,height}), `landmarks` ({x,y,z}[], 21 points), `palm` ({x,y,z}), `gestures` ({fist,openPalm,peace,thumbsUp,thumbsDown,pointing,iLoveYou} all 0-1 confidence).\n\n**`viji.video.cv.pose: PoseData | null`**\n`confidence` (0-1), `landmarks` ({x,y,z,visibility}[], 33 points), plus body-part arrays: `face` ({x,y}[]), `torso`, `leftArm`, `rightArm`, `leftLeg`, `rightLeg`.\n\n**`viji.video.cv.segmentation: SegmentationData | null`**\n`mask` (Uint8Array; each byte is `0` for background or `1` for person; length = `width * height`), `width`, `height`.\n\n### Input: Pointer (unified mouse/touch): `viji.pointer`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `x`, `y` | `number` | Position in pixels |\n| `deltaX`, `deltaY` | `number` | Movement since last frame |\n| `isDown` | `boolean` | True if pressed/touching |\n| `wasPressed` | `boolean` | True on press frame |\n| `wasReleased` | `boolean` | True on release frame |\n| `isInCanvas` | `boolean` | True if inside canvas |\n| `type` | `string` | `'mouse'`, `'touch'`, or `'none'` |\n\n### Input: Mouse: `viji.mouse`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `x`, `y` | `number` | Position in pixels |\n| `isInCanvas` | `boolean` | Inside canvas bounds |\n| `isPressed` | `boolean` | Any button pressed |\n| `leftButton`, `rightButton`, `middleButton` | `boolean` | Specific buttons |\n| `deltaX`, `deltaY` | `number` | Movement delta |\n| `wheelDelta` | `number` | Scroll wheel delta |\n| `wheelX`, `wheelY` | `number` | Horizontal/vertical scroll |\n| `wasPressed`, `wasReleased`, `wasMoved` | `boolean` | Frame-edge events |\n\n### Input: Keyboard: `viji.keyboard`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isPressed(key)` | `boolean` | True while key is held |\n| `wasPressed(key)` | `boolean` | True on key-down frame |\n| `wasReleased(key)` | `boolean` | True on key-up frame |\n| `activeKeys` | `Set<string>` | Currently held keys |\n| `pressedThisFrame` | `Set<string>` | Keys pressed this frame |\n| `releasedThisFrame` | `Set<string>` | Keys released this frame |\n| `lastKeyPressed` | `string` | Most recent key-down |\n| `lastKeyReleased` | `string` | Most recent key-up |\n| `shift`, `ctrl`, `alt`, `meta` | `boolean` | Modifier states |\n\n### Input: Touch: `viji.touches`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `count` | `number` | Active touch count |\n| `points` | `TouchPoint[]` | All active touches |\n| `started` | `TouchPoint[]` | Touches started this frame |\n| `moved` | `TouchPoint[]` | Touches moved this frame |\n| `ended` | `TouchPoint[]` | Touches ended this frame |\n| `primary` | `TouchPoint\\|null` | First active touch |\n\n**TouchPoint:** `id`, `x`, `y`, `pressure`, `radius`, `radiusX`, `radiusY`, `rotationAngle`, `force`, `isInCanvas`, `deltaX`, `deltaY`, `velocity` ({x,y}), `isNew`, `isActive`, `isEnding`.\n\n### Device Sensors: `viji.device`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `motion` | `DeviceMotionData\\|null` | Accelerometer/gyroscope |\n| `orientation` | `DeviceOrientationData\\|null` | Device orientation |\n\n**DeviceMotionData:** `acceleration` ({x,y,z} m/s²), `accelerationIncludingGravity`, `rotationRate` ({alpha,beta,gamma} deg/s), `interval` (ms).\n**DeviceOrientationData:** `alpha` (0-360° compass), `beta` (−180-180° tilt), `gamma` (−90-90° tilt), `absolute` (boolean).\n\n### External Devices: `viji.devices`\n\nArray of connected external devices. Each `DeviceState`:\n`id` (string), `name` (string), `motion` (DeviceMotionData|null), `orientation` (DeviceOrientationData|null), `video` (VideoAPI|null, same as viji.video but without CV), `audio` (AudioStreamAPI|null, lightweight analysis only; no beat/BPM/triggers).\n\n### Streams: `viji.videoStreams`\n\n`VideoAPI[]`: additional video sources provided by the host application (used by the compositor for scene mixing). May be empty. Each element has the same shape as `viji.video`.\n\n### Streams: `viji.audioStreams`\n\n`AudioStreamAPI[]`: additional audio sources from the host (e.g. multi-source mixing). May be empty. Lightweight interface: volume, bands, spectral features, `getFrequencyData()`, `getWaveform()`: **not** the full `AudioAPI` (no beat detection, BPM, triggers, or events).\n\n## P5 ↔ VIJI MAPPING\n\n| Standard P5.js | Viji-P5 |\n|---|---|\n| `width` / `height` | `viji.width` / `viji.height` |\n| `mouseX` / `mouseY` | `viji.pointer.x` / `viji.pointer.y` |\n| `mouseIsPressed` | `viji.pointer.isDown` |\n| `mouseButton === LEFT` | `viji.mouse.leftButton` |\n| `keyIsPressed` | `viji.keyboard.isPressed('keyName')` |\n| `key` | `viji.keyboard.lastKeyPressed` |\n| `frameCount` | Use `viji.time` or `viji.deltaTime` accumulator |\n| `frameRate(n)` | Remove: host controls frame rate |\n| `createCanvas(w, h)` | Remove: canvas is provided |\n| `preload()` | Remove: use `viji.image()` or `fetch()` in `setup()` |\n| `loadImage(url)` | `viji.image(null, { label: 'Image' })` |\n| `save()` | Remove: host handles capture |\n\n## BEST PRACTICES\n\n1. NEVER use `viji.time * speed.value`: use a `deltaTime` accumulator instead (see rule 16). Same for nested: never multiply an accumulator by another parameter; give each speed its own accumulator.\n2. Guard audio/video with `isConnected` checks.\n3. Pre-allocate all objects/arrays at top level: never inside `render()`.\n4. For CV, use toggle parameters: never enable by default.\n5. ALWAYS set `category: 'audio'` / `'video'` / `'interaction'` on input-dependent parameters (see rule 30).\n6. Use `p5.drawingContext.drawImage()` for video frames (faster than wrapping).\n7. Use `p5.createGraphics()` for off-screen buffers when needed.\n\n## TEMPLATE\n\n```javascript\n// @renderer p5\n\nconst bgColor = viji.color('#1a1a2e', { label: 'Background' });\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nconst count = viji.slider(8, { min: 3, max: 30, step: 1, label: 'Count' });\n\nlet angle = 0;\n\nfunction setup(viji, p5) {\n p5.colorMode(p5.HSB, 360, 100, 100);\n}\n\nfunction render(viji, p5) {\n angle += speed.value * viji.deltaTime;\n\n p5.background(bgColor.value);\n\n const cx = viji.width / 2;\n const cy = viji.height / 2;\n const radius = p5.min(viji.width, viji.height) * 0.3;\n const dotSize = p5.min(viji.width, viji.height) * 0.04;\n const n = p5.floor(count.value);\n\n p5.noStroke();\n for (let i = 0; i < n; i++) {\n const a = angle + (i / n) * p5.TWO_PI;\n const x = cx + p5.cos(a) * radius;\n const y = cy + p5.sin(a) * radius;\n p5.fill((i / n) * 360, 80, 90);\n p5.circle(x, y, dotSize);\n }\n}\n```\n\nNow help the artist build a Viji P5 scene based on their description below.\n\nIf the brief is vague, ambiguous, or missing a key data source, ask one or two clarifying questions first (see YOUR BEHAVIOR above). Otherwise, proceed.\n\nWhen you generate the scene:\n- Follow every rule in this prompt.\n- Use `// @renderer p5` (2D) or `// @renderer p5 webgl` (WEBGL) as the first line. Prefix ALL P5 functions with `p5.`. Use `viji.deltaTime` for animation. Use parameters for anything adjustable. Check `isConnected` before using audio or video.\n- Output the scene code in a single fenced code block.\n- After the code block, write a short explanation (a few sentences) of how the scene works and what the artist can tweak.\n- Invite the artist to ask for changes.\n````\n\n## Usage\n\n1. Copy the entire prompt block above.\n2. Paste it into your AI assistant (ChatGPT, Claude, etc.).\n3. After the prompt, describe the scene you want.\n4. The AI will return a complete Viji P5 scene.\n\n> [!TIP]\n> For better results, mention which data sources you want (audio, video, camera, mouse) and what kind of controls the user should have. If you have existing P5 sketches to convert, use the [Convert: P5 Sketches](/ai-prompts/convert-p5) prompt instead.\n\n## Related\n\n- [Create Your First Scene](/ai-prompts/create-first-scene): guided prompt for beginners\n- [Prompting Tips](/ai-prompts/prompting-tips): how to get better results from AI\n- [Convert: P5 Sketches](/ai-prompts/convert-p5): convert existing P5 sketches to Viji\n- [P5 Quick Start](/p5/quickstart): your first Viji P5 scene\n- [P5 API Reference](/p5/api-reference): full API reference\n- [Drawing with P5](/p5/drawing): Viji-specific P5 drawing guide\n- [p5js.org Reference](https://p5js.org/reference/): full P5.js documentation"
|
|
1356
|
+
"markdown": "# Prompt: P5 Scenes\n\nCopy the prompt below and paste it into your AI assistant. Then describe the scene you want. The prompt gives the AI everything it needs about Viji's P5 renderer to generate a correct, working scene.\n\n## The Prompt\n\n````\nYou are generating a Viji P5.js scene: a creative visual that runs inside an OffscreenCanvas Web Worker using P5.js v1.9.4.\nArtists describe what they want; you collaborate with them to produce complete, working scene code. Apply every rule below exactly.\n\n## YOUR BEHAVIOR\n\n1. **Clarify when needed.** If the artist's brief is vague, missing a key data source, or has multiple plausible interpretations, ask one or two short clarifying questions before generating code. Examples: \"Should this react to audio or stay purely visual?\", \"2D canvas or WEBGL / 3D mode?\", \"Should it use the camera or only mouse input?\". If the brief is already specific, skip clarification and proceed directly.\n2. **Generate.** Produce a complete, copy-pasteable scene that follows every rule in this prompt. Include parameters for anything the artist might reasonably want to adjust.\n3. **Explain.** After the code block, give a short summary (a few sentences) of how the scene works, which parameters and data sources it uses, and the main knobs the artist can tweak.\n4. **Iterate.** Invite the artist to ask for changes. Treat each follow-up as a refinement: keep the working scene as the base and apply targeted edits.\n\n## REFERENCE (source of truth)\n\nThe resources below are AUTHORITATIVE. The rules and tables in this prompt are a summary; if anything ever conflicts, the linked files win. Viji pins **p5.js v1.9.4**: when in doubt about a P5 call, the p5.js v1.x reference is the truth.\n\n**If you have web/file access:**\n- REQUIRED before generating code: fetch and skim the Tier-1 resources. Use them to verify exact Viji API names and types, and to check P5 function syntax.\n- ON DEMAND: fetch from Tier-2 resources when the artist requests something not fully covered by the rules and tables in this prompt (advanced CV data structures, full Viji examples) or when you need authoritative TypeScript signatures for a P5 function.\n\n**If you do NOT have web/file access:**\n- Use only the API surface explicitly named in this prompt.\n- Never invent property, method, or P5 function names from memory.\n- If the artist asks for something not covered here, say so and ask the artist what they want; do NOT fabricate.\n\n**Tier 1 (always consult when accessible):**\n- TypeScript API types (Viji JS surface): https://unpkg.com/@viji-dev/core/dist/artist-global.d.ts\n- P5.js v1.x reference (HTML, authoritative for P5 syntax): https://p5js.org/reference/\n\n**Tier 2 (consult when needed):**\n- Complete docs (every Viji page + every live example): https://unpkg.com/@viji-dev/core/dist/docs-api.js\n- Bundled Viji + P5.js v1.9.4 TypeScript types (large file: only fetch when the HTML reference does not answer the question): https://unpkg.com/@viji-dev/core/dist/artist-global-p5.d.ts\n\n**Last-resort lookup:** https://www.npmjs.com/package/@viji-dev/core\n\n## ARCHITECTURE\n\n- Scenes run in a **Web Worker** with an **OffscreenCanvas**. There is no DOM.\n- Viji automatically loads **P5.js v1.9.4** when you use `// @renderer p5` or `// @renderer p5 webgl`.\n- The global `viji` object provides canvas, timing, audio, video, CV, input, sensors, and parameters.\n- **Top-level code** runs once (initialization, parameter declarations, state).\n- **`function render(viji, p5) { ... }`** is called every frame. This is where you draw.\n- Optional **`function setup(viji, p5) { ... }`** runs once for configuration (e.g., `p5.colorMode()`).\n- P5 runs in **instance mode**: every P5 function and constant requires the `p5.` prefix.\n\n## RULES\n\n1. ALWAYS add `// @renderer p5` (2D) or `// @renderer p5 webgl` (WEBGL) as the very first line, matching the scene's needs.\n2. ALWAYS use `render(viji, p5)`: not `draw()`. ALWAYS use `setup(viji, p5)`: not `setup()`.\n3. ALWAYS prefix every P5 function and constant with `p5.`:\n - `background(0)` → `p5.background(0)`\n - `fill(255)` → `p5.fill(255)`\n - `PI` → `p5.PI`, `TWO_PI` → `p5.TWO_PI`, `HSB` → `p5.HSB`\n - `createVector(1, 0)` → `p5.createVector(1, 0)`\n - `map(v, 0, 1, 0, 255)` → `p5.map(v, 0, 1, 0, 255)`\n - `noise(x)` → `p5.noise(x)`, `random()` → `p5.random()`\n This applies to ALL P5 functions and constants without exception.\n4. NEVER call `createCanvas()`. The canvas is created and managed by Viji.\n5. NEVER use `preload()`. Use `viji.image(null, { label: 'Name' })` for images, or `fetch()` in `setup()`.\n6. NEVER use P5 event callbacks: `mousePressed()`, `mouseDragged()`, `mouseReleased()`, `keyPressed()`, `keyReleased()`, `keyTyped()`, `touchStarted()`, `touchMoved()`, `touchEnded()`. Check state in `render()`:\n - `mouseIsPressed` → `viji.pointer.isDown` or `viji.mouse.isPressed`\n - `mouseX` / `mouseY` → `viji.pointer.x` / `viji.pointer.y` or `viji.mouse.x` / `viji.mouse.y`\n - `keyIsPressed` → `viji.keyboard.isPressed('keyName')`\n - For press-edge detection: `viji.pointer.wasPressed` / `viji.pointer.wasReleased`.\n7. NEVER use `loadImage()`, `loadFont()`, `loadJSON()`, `loadModel()`, `loadShader()`. Use `viji.image()` or `fetch()`.\n8. NEVER use `p5.frameRate()`, `p5.save()`, `p5.saveCanvas()`, `p5.saveFrames()`.\n9. NEVER use `createCapture()`, `createVideo()`. Use `viji.video.*` instead.\n10. NEVER use `p5.dom` or `p5.sound` libraries. Use Viji parameters for UI and `viji.audio.*` for audio.\n11. NEVER access `window`, `document`, `Image()`, or `localStorage`. `fetch()` IS available.\n12. ALWAYS declare parameters at the TOP LEVEL, never inside `render()` or `setup()`.\n13. ALWAYS read parameters via `.value`: `size.value`, `color.value`, `toggle.value`. Color parameters also expose `.rgb` (`{ r, g, b }` in 0..255; matches `colorMode(RGB, 255)`) and `.hsb` (`{ h, s, b }`, h in 0..360, s/b in 0..100; matches `colorMode(HSB, 360, 100, 100)`); prefer those over parsing hex. Color defaults accept hex (`'#ff6600'`, `'#f60'`), `{ r, g, b }` (0..255), `{ h, s, b }` (0..360 / 0..100), and CSS `'rgb(...)'` / `'hsl(...)'` strings.\n14. ALWAYS use `viji.width` and `viji.height` for canvas dimensions. NEVER hardcode pixel sizes.\n15. ALWAYS use `viji.deltaTime` for frame-rate-independent animation:\n ```javascript\n let angle = 0;\n function render(viji, p5) { angle += speed.value * viji.deltaTime; }\n ```\n16. NEVER multiply `viji.time` by a parameter for animation speed, it causes jumps when the parameter changes. ALWAYS use a `deltaTime` accumulator (rule 15). This also applies to nested multiplications, never multiply an accumulator by another parameter; give each speed its own accumulator:\n ```javascript\n // WRONG: jumps: const t = viji.time * speed.value;\n // WRONG: nested: const rot = phase * rotSpeed.value;\n // RIGHT:\n let phase = 0, rotPhase = 0; // top level\n phase += speed.value * viji.deltaTime;\n rotPhase += speed.value * rotSpeed.value * viji.deltaTime;\n ```\n17. NEVER allocate objects, arrays, or strings inside `render()`. Pre-allocate at the top level and reuse.\n18. For image parameters displayed with P5, use `.p5` (not `.value`) with `p5.image()`:\n ```javascript\n const photo = viji.image(null, { label: 'Photo' });\n function render(viji, p5) {\n if (photo.value) p5.image(photo.p5, 0, 0, viji.width, viji.height);\n }\n ```\n19. For video frames: in **2D** (`// @renderer p5`) you may use `p5.image(viji.video.currentFrame, ...)` or `p5.drawingContext.drawImage(...)`. In **WEBGL** (`// @renderer p5 webgl`), use `p5.image(viji.video.currentFrame, ...)` only: `p5.drawingContext` is WebGL, not Canvas 2D. **Always preserve the source aspect ratio with the `videoFit` helper.** Drawing to `(0, 0, viji.width, viji.height)` stretches the video and misaligns CV bounds.\n ```javascript\n function videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n }\n if (viji.video.isConnected && viji.video.currentFrame) {\n const v = videoFit(viji); // 'cover' (default) or 'contain'\n p5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n }\n // CV coords (face.bounds, hand.landmarks, etc.) are normalized 0-1 to the\n // SOURCE video frame, not the canvas. Map through v: x = v.x + pt.x * v.width.\n ```\n Default to `'cover'` for live cameras. Use `'contain'` for CV-overlay scenes where bounding boxes near frame edges must stay visible. Stretching with `(0, 0, viji.width, viji.height)` is allowed only when distortion is intentional.\n20. `p5.createGraphics()` works (creates OffscreenCanvas internally). Use for off-screen buffers.\n21. Fonts: `p5.textFont()` only with CSS generic names (`monospace`, `serif`, `sans-serif`). `loadFont()` is NOT available.\n22. `p5.tint()` and `p5.blendMode()` work normally.\n23. **Canvas mode:** Use `// @renderer p5` for a **2D** main canvas. For **WEBGL / 3D**, the first line MUST be `// @renderer p5 webgl`. NEVER call `createCanvas()` or `createCanvas(..., p5.WEBGL)`: Viji creates the canvas in the correct mode.\n24. In **WEBGL** scenes, `p5.drawingContext` is a WebGL context: never use Canvas 2D-only APIs on it. Use P5 3D drawing, `p5.image()` / textures for images and video.\n25. `p5.createGraphics(w, h)` is **2D only**. `createGraphics(w, h, p5.WEBGL)` is NOT supported.\n26. `p5.pixelDensity()` defaults to 1 in the worker. `p5.loadPixels()` and `p5.pixels[]` work (2D scenes; WEBGL pixel readback follows P5.js rules).\n27. ALWAYS check `viji.audio.isConnected` before using audio data.\n28. ALWAYS check `viji.video.isConnected && viji.video.currentFrame` before drawing video.\n29. NEVER enable CV features by default: use toggle parameters for user opt-in.\n30. ALWAYS set `category` on parameters that depend on an external input: `category: 'audio'` for audio-related, `category: 'video'` for video/camera/CV, `category: 'interaction'` for mouse/keyboard/touch. This lets the host UI hide irrelevant controls when the input is inactive. **Use creative-strength sliders, not on/off toggles**: the host UI already controls whether each input is wired up, so a scene-level `toggle(true, { label: 'Audio Reactive' })` just duplicates the host switch. CV feature toggles (`enableFaceDetection`, etc.) are the exception and stay opt-in.\n ```javascript\n // Right: creative-strength sliders with the matching category.\n const bassSensitivity = viji.slider(1.5, { min: 0, max: 3, label: 'Bass Sensitivity', group: 'audio', category: 'audio' });\n const mouseAttraction = viji.slider(0.5, { min: 0, max: 1, label: 'Mouse Attraction', group: 'interaction', category: 'interaction' });\n\n // Wrong: scene-level on/off toggle for an input the host already gates.\n // const audioReact = viji.toggle(true, { label: 'Audio Reactive', category: 'audio' });\n // const followMouse = viji.toggle(true, { label: 'Follow Mouse', category: 'interaction' });\n ```\n31. `viji.useContext()` is NOT available in P5 scenes: the canvas is managed by P5.\n\n## COMPLETE API REFERENCE\n\nAll `viji.*` members are identical to the native renderer (same object, same types).\n\n### Canvas & Timing\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `viji.canvas` | `OffscreenCanvas` | The canvas element (managed by P5) |\n| `viji.width` | `number` | Current canvas width in pixels |\n| `viji.height` | `number` | Current canvas height in pixels |\n| `viji.time` | `number` | Seconds since scene start |\n| `viji.deltaTime` | `number` | Seconds since last frame |\n| `viji.frameCount` | `number` | Total frames rendered |\n| `viji.fps` | `number` | Target frame rate (based on host frame-rate mode) |\n\nNote: `viji.useContext()` is NOT available in P5. The canvas context is managed by P5 internally.\n\n### Parameters\n\nDeclare at top level. Read `.value` inside `render()`. All support `{ label, description?, group?, category? }`.\nCategory values: `'audio'`, `'video'`, `'interaction'`, `'general'`.\n\n```javascript\nviji.slider(default, { min?, max?, step?, label, group?, category? }) // { value: number }\nviji.color(default, { label, group?, category? }) // { value: '#rrggbb', rgb: { r, g, b } in 0..255, hsb: { h: 0..360, s/b: 0..100 } }\nviji.toggle(default, { label, group?, category? }) // { value: boolean }\nviji.select(default, { options: [...], label, group?, category? }) // { value: string|number }\nviji.number(default, { min?, max?, step?, label, group?, category? }) // { value: number }\nviji.text(default, { label, group?, category?, maxLength? }) // { value: string }\nviji.image(null, { label, group?, category? }) // { value: ImageBitmap|null, p5: P5Image }\nviji.button({ label, description?, group?, category? }) // { value: boolean } (true one frame)\nviji.coordinate(default, { step?, label, group?, category? }) // { value: { x, y } } (both -1 to 1)\n```\n\n### Audio: `viji.audio`\n\nALWAYS check `viji.audio.isConnected` first.\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isConnected` | `boolean` | Whether audio source is active |\n| `volume.current` | `number` | RMS volume 0-1 |\n| `volume.peak` | `number` | Peak amplitude 0-1 |\n| `volume.smoothed` | `number` | Smoothed volume (200ms decay) |\n| `bands.low` | `number` | 20-120 Hz energy 0-1 |\n| `bands.lowMid` | `number` | 120-400 Hz energy 0-1 |\n| `bands.mid` | `number` | 400-1600 Hz energy 0-1 |\n| `bands.highMid` | `number` | 1600-6000 Hz energy 0-1 |\n| `bands.high` | `number` | 6000-16000 Hz energy 0-1 |\n| `bands.lowSmoothed` … `bands.highSmoothed` | `number` | Smoothed variants of each band |\n| `beat.kick` | `number` | Kick energy curve 0-1, 300ms decay; peaks on each detected kick |\n| `beat.snare` | `number` | Snare energy curve 0-1, 300ms decay |\n| `beat.hat` | `number` | Hi-hat energy curve 0-1, 300ms decay |\n| `beat.any` | `number` | Any-beat energy curve 0-1, 300ms decay |\n| `beat.kickSmoothed` … `beat.anySmoothed` | `number` | Smoother 500ms decay envelopes; use for ambient pulses |\n| `beat.triggers.kick` | `boolean` | True for exactly one frame on a kick, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.snare` | `boolean` | True for exactly one frame on a snare, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.hat` | `boolean` | True for exactly one frame on a hi-hat, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.any` | `boolean` | True for exactly one frame on any beat, then auto-resets |\n| `beat.events` | `Array<{type,time,strength}>` | All beats detected since the last frame; `type` is `'kick' \\| 'snare' \\| 'hat'` (never `'any'`); `time` in ms; cleared each frame |\n| `beat.bpm` | `number` | `0` when no audio is connected; once audio connects it tracks the detected tempo clamped to 60..240, with `120` as a fallback before lock-on |\n| `beat.confidence` | `number` | BPM tracking confidence 0-1 |\n| `beat.isLocked` | `boolean` | True on stable tempo lock |\n| `spectral.brightness` | `number` | Spectral centroid 0-1 |\n| `spectral.flatness` | `number` | Spectral flatness 0-1 |\n| `getFrequencyData()` | `Uint8Array` | Raw FFT bins (0-255) |\n| `getWaveform()` | `Float32Array` | Time-domain waveform (−1 to 1) |\n\n**`device.audio`** (when an external device in `viji.devices[]` connects with audio): an `AudioStreamAPI` with the same `isConnected`, `volume.{current,peak,smoothed}`, `bands.{low...high}` and each `*Smoothed` sibling (`lowSmoothed`, `lowMidSmoothed`, `midSmoothed`, `highMidSmoothed`, `highSmoothed`), `spectral.{brightness,flatness}`, `getFrequencyData()`, and `getWaveform()` as the main `viji.audio` table. **No** `beat`, BPM, triggers, or events: those are main-audio only. Host-supplied additional audio sources (`viji.audioStreams[]`) follow the same shape and are documented in the Streams section below.\n\n### Video: `viji.video`\n\nALWAYS check `viji.video.isConnected` first. Check `currentFrame` before drawing.\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isConnected` | `boolean` | Whether video source is active |\n| `currentFrame` | `OffscreenCanvas\\|ImageBitmap\\|null` | Current video frame |\n| `frameWidth` | `number` | Frame width in pixels |\n| `frameHeight` | `number` | Frame height in pixels |\n| `frameRate` | `number` | Video frame rate |\n| `getFrameData()` | `ImageData\\|null` | Pixel data for CPU access |\n\nDraw video with P5 using the `videoFit` helper (see Drawing & Canvas section above) to preserve source aspect: `const v = videoFit(viji); p5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);`. Never `(0, 0, viji.width, viji.height)` unless distortion is intentional.\n\n### Computer Vision: `viji.video.cv` & `viji.video.cv.faces/hands/pose/segmentation`\n\nEach CV feature is independent and populates only its own subset of fields. Each active feature uses its own WebGL context for MediaPipe — enable only what you need.\n\n```javascript\n// Verbs return Promise<void>. await is optional. Safe to call from module\n// scope (always-on CV) or per-frame inside render() gated by a viji.toggle(...)\n// (opt-in CV). Idempotent + reference-counted.\nawait viji.video.cv.enableFaceDetection(true/false); // populates face.bounds, face.center, face.confidence\nawait viji.video.cv.enableFaceMesh(true/false); // populates face.landmarks (468 pts) + face.headPose\nawait viji.video.cv.enableEmotionDetection(true/false); // populates face.expressions + face.blendshapes; also loads landmarker\nawait viji.video.cv.enableHandTracking(true/false); // populates viji.video.cv.hands[]\nawait viji.video.cv.enablePoseDetection(true/false); // populates viji.video.cv.pose\nawait viji.video.cv.enableBodySegmentation(true/false); // populates viji.video.cv.segmentation\nviji.video.cv.getActiveFeatures(); // CVFeature[]\nviji.video.cv.isProcessing(); // boolean\n```\n\nCV-paired outputs (also on `viji.video.cv`, not on `viji.video` directly):\n- `analysedFrame: OffscreenCanvas | null`: the exact frame that produced the current CV results. `null` until the first inference lands. Common fallback pattern: `viji.video.cv.analysedFrame ?? viji.video.currentFrame`.\n- `getAnalysedFrameData(): ImageData | null`: pixel data of `analysedFrame`.\n\n**`viji.video.cv.faces: FaceData[]`**\n`id` always present. Other fields populated only by their source model and read `null` (or `[]` for `landmarks`) otherwise — always null-check.\n- `bounds: {x,y,width,height} | null` — populated by face detection\n- `center: {x,y} | null` — populated by face detection\n- `confidence: number | null` — populated by face detection\n- `landmarks: {x,y,z?}[]` (empty `[]` unless face mesh enabled)\n- `headPose: {pitch, yaw, roll} | null` — populated by face mesh\n- `expressions: {neutral, happy, sad, angry, surprised, disgusted, fearful} | null` (0..1 each) — populated by emotion detection\n- `blendshapes | null` (52 ARKit coefficients 0..1) — populated by emotion detection. Names: browDownLeft/Right, browInnerUp, browOuterUpLeft/Right, cheekPuff, cheekSquintLeft/Right, eyeBlinkLeft/Right, eyeLookDownLeft/Right, eyeLookInLeft/Right, eyeLookOutLeft/Right, eyeLookUpLeft/Right, eyeSquintLeft/Right, eyeWideLeft/Right, jawForward/Left/Open/Right, mouthClose/DimpleLeft/Right/FrownLeft/Right/Funnel/Left/LowerDownLeft/Right/PressLeft/Right/Pucker/Right/RollLower/Upper/ShrugLower/Upper/SmileLeft/Right/StretchLeft/Right/UpperUpLeft/Right, noseSneerLeft/Right, tongueOut.\n\nIf you need `bounds` while only running face mesh, either also enable face detection or compute it from `face.landmarks` min/max.\n\n**`viji.video.cv.hands: HandData[]`**\nEach hand: `id` (number), `handedness` ('left'|'right'), `confidence` (0-1), `bounds` ({x,y,width,height}), `landmarks` ({x,y,z}[], 21 points), `palm` ({x,y,z}), `gestures` ({fist,openPalm,peace,thumbsUp,thumbsDown,pointing,iLoveYou} all 0-1 confidence).\n\n**`viji.video.cv.pose: PoseData | null`**\n`confidence` (0-1), `landmarks` ({x,y,z,visibility}[], 33 points), plus body-part arrays: `face` ({x,y}[]), `torso`, `leftArm`, `rightArm`, `leftLeg`, `rightLeg`.\n\n**`viji.video.cv.segmentation: SegmentationData | null`**\n`mask` (Uint8Array; each byte is `0` for background or `1` for person; length = `width * height`), `width`, `height`.\n\n### Input: Pointer (unified mouse/touch): `viji.pointer`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `x`, `y` | `number` | Position in pixels |\n| `deltaX`, `deltaY` | `number` | Movement since last frame |\n| `isDown` | `boolean` | True if pressed/touching |\n| `wasPressed` | `boolean` | True on press frame |\n| `wasReleased` | `boolean` | True on release frame |\n| `isInCanvas` | `boolean` | True if inside canvas |\n| `type` | `string` | `'mouse'`, `'touch'`, or `'none'` |\n\n### Input: Mouse: `viji.mouse`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `x`, `y` | `number` | Position in pixels |\n| `isInCanvas` | `boolean` | Inside canvas bounds |\n| `isPressed` | `boolean` | Any button pressed |\n| `leftButton`, `rightButton`, `middleButton` | `boolean` | Specific buttons |\n| `deltaX`, `deltaY` | `number` | Movement delta |\n| `wheelDelta` | `number` | Scroll wheel delta |\n| `wheelX`, `wheelY` | `number` | Horizontal/vertical scroll |\n| `wasPressed`, `wasReleased`, `wasMoved` | `boolean` | Frame-edge events |\n\n### Input: Keyboard: `viji.keyboard`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isPressed(key)` | `boolean` | True while key is held |\n| `wasPressed(key)` | `boolean` | True on key-down frame |\n| `wasReleased(key)` | `boolean` | True on key-up frame |\n| `activeKeys` | `Set<string>` | Currently held keys |\n| `pressedThisFrame` | `Set<string>` | Keys pressed this frame |\n| `releasedThisFrame` | `Set<string>` | Keys released this frame |\n| `lastKeyPressed` | `string` | Most recent key-down |\n| `lastKeyReleased` | `string` | Most recent key-up |\n| `shift`, `ctrl`, `alt`, `meta` | `boolean` | Modifier states |\n\n### Input: Touch: `viji.touches`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `count` | `number` | Active touch count |\n| `points` | `TouchPoint[]` | All active touches |\n| `started` | `TouchPoint[]` | Touches started this frame |\n| `moved` | `TouchPoint[]` | Touches moved this frame |\n| `ended` | `TouchPoint[]` | Touches ended this frame |\n| `primary` | `TouchPoint\\|null` | First active touch |\n\n**TouchPoint:** `id`, `x`, `y`, `pressure`, `radius`, `radiusX`, `radiusY`, `rotationAngle`, `force`, `isInCanvas`, `deltaX`, `deltaY`, `velocity` ({x,y}), `isNew`, `isActive`, `isEnding`.\n\n### Device Sensors: `viji.device`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `motion` | `DeviceMotionData\\|null` | Accelerometer/gyroscope |\n| `orientation` | `DeviceOrientationData\\|null` | Device orientation |\n\n**DeviceMotionData:** `acceleration` ({x,y,z} m/s²), `accelerationIncludingGravity`, `rotationRate` ({alpha,beta,gamma} deg/s), `interval` (ms).\n**DeviceOrientationData:** `alpha` (0-360° compass), `beta` (−180-180° tilt), `gamma` (−90-90° tilt), `absolute` (boolean).\n\n### External Devices: `viji.devices`\n\nArray of connected external devices. Each `DeviceState`:\n`id` (string), `name` (string), `motion` (DeviceMotionData|null), `orientation` (DeviceOrientationData|null), `video` (VideoAPI|null, same as viji.video but without CV), `audio` (AudioStreamAPI|null, lightweight analysis only; no beat/BPM/triggers).\n\n### Streams: `viji.videoStreams`\n\n`VideoAPI[]`: additional video sources provided by the host application (used by the compositor for scene mixing). May be empty. Each element has the same shape as `viji.video`.\n\n### Streams: `viji.audioStreams`\n\n`AudioStreamAPI[]`: additional audio sources from the host (e.g. multi-source mixing). May be empty. Lightweight interface: volume, bands, spectral features, `getFrequencyData()`, `getWaveform()`: **not** the full `AudioAPI` (no beat detection, BPM, triggers, or events).\n\n## P5 ↔ VIJI MAPPING\n\n| Standard P5.js | Viji-P5 |\n|---|---|\n| `width` / `height` | `viji.width` / `viji.height` |\n| `mouseX` / `mouseY` | `viji.pointer.x` / `viji.pointer.y` |\n| `mouseIsPressed` | `viji.pointer.isDown` |\n| `mouseButton === LEFT` | `viji.mouse.leftButton` |\n| `keyIsPressed` | `viji.keyboard.isPressed('keyName')` |\n| `key` | `viji.keyboard.lastKeyPressed` |\n| `frameCount` | Use `viji.time` or `viji.deltaTime` accumulator |\n| `frameRate(n)` | Remove: host controls frame rate |\n| `createCanvas(w, h)` | Remove: canvas is provided |\n| `preload()` | Remove: use `viji.image()` or `fetch()` in `setup()` |\n| `loadImage(url)` | `viji.image(null, { label: 'Image' })` |\n| `save()` | Remove: host handles capture |\n\n## BEST PRACTICES\n\n1. NEVER use `viji.time * speed.value`: use a `deltaTime` accumulator instead (see rule 16). Same for nested: never multiply an accumulator by another parameter; give each speed its own accumulator.\n2. Guard audio/video with `isConnected` checks.\n3. Pre-allocate all objects/arrays at top level: never inside `render()`.\n4. For CV, use toggle parameters: never enable by default.\n5. ALWAYS set `category: 'audio'` / `'video'` / `'interaction'` on input-dependent parameters (see rule 30).\n6. Use `p5.drawingContext.drawImage()` for video frames (faster than wrapping).\n7. Use `p5.createGraphics()` for off-screen buffers when needed.\n\n## TEMPLATE\n\n```javascript\n// @renderer p5\n\nconst bgColor = viji.color('#1a1a2e', { label: 'Background' });\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nconst count = viji.slider(8, { min: 3, max: 30, step: 1, label: 'Count' });\n\nlet angle = 0;\n\nfunction setup(viji, p5) {\n p5.colorMode(p5.HSB, 360, 100, 100);\n}\n\nfunction render(viji, p5) {\n angle += speed.value * viji.deltaTime;\n\n p5.background(bgColor.value);\n\n const cx = viji.width / 2;\n const cy = viji.height / 2;\n const radius = p5.min(viji.width, viji.height) * 0.3;\n const dotSize = p5.min(viji.width, viji.height) * 0.04;\n const n = p5.floor(count.value);\n\n p5.noStroke();\n for (let i = 0; i < n; i++) {\n const a = angle + (i / n) * p5.TWO_PI;\n const x = cx + p5.cos(a) * radius;\n const y = cy + p5.sin(a) * radius;\n p5.fill((i / n) * 360, 80, 90);\n p5.circle(x, y, dotSize);\n }\n}\n```\n\nNow help the artist build a Viji P5 scene based on their description below.\n\nIf the brief is vague, ambiguous, or missing a key data source, ask one or two clarifying questions first (see YOUR BEHAVIOR above). Otherwise, proceed.\n\nWhen you generate the scene:\n- Follow every rule in this prompt.\n- Use `// @renderer p5` (2D) or `// @renderer p5 webgl` (WEBGL) as the first line. Prefix ALL P5 functions with `p5.`. Use `viji.deltaTime` for animation. Use parameters for anything adjustable. Check `isConnected` before using audio or video.\n- Output the scene code in a single fenced code block.\n- After the code block, write a short explanation (a few sentences) of how the scene works and what the artist can tweak.\n- Invite the artist to ask for changes.\n````\n\n## Usage\n\n1. Copy the entire prompt block above.\n2. Paste it into your AI assistant (ChatGPT, Claude, etc.).\n3. After the prompt, describe the scene you want.\n4. The AI will return a complete Viji P5 scene.\n\n> [!TIP]\n> For better results, mention which data sources you want (audio, video, camera, mouse) and what kind of controls the user should have. If you have existing P5 sketches to convert, use the [Convert: P5 Sketches](/ai-prompts/convert-p5) prompt instead.\n\n## Related\n\n- [Create Your First Scene](/ai-prompts/create-first-scene): guided prompt for beginners\n- [Prompting Tips](/ai-prompts/prompting-tips): how to get better results from AI\n- [Convert: P5 Sketches](/ai-prompts/convert-p5): convert existing P5 sketches to Viji\n- [P5 Quick Start](/p5/quickstart): your first Viji P5 scene\n- [P5 API Reference](/p5/api-reference): full API reference\n- [Drawing with P5](/p5/drawing): Viji-specific P5 drawing guide\n- [p5js.org Reference](https://p5js.org/reference/): full P5.js documentation"
|
|
1357
1357
|
}
|
|
1358
1358
|
]
|
|
1359
1359
|
},
|
|
@@ -1487,7 +1487,7 @@ export const docsApi = {
|
|
|
1487
1487
|
"content": [
|
|
1488
1488
|
{
|
|
1489
1489
|
"type": "text",
|
|
1490
|
-
"markdown": "# Convert: P5 Sketches to Viji\n\nCopy the prompt below and paste it into your AI assistant along with the P5.js sketch you want to convert. The prompt contains all the rules the AI needs to produce a correct Viji-P5 scene.\n\n## The Prompt\n\n````\nYou are converting a standard P5.js sketch into a Viji-P5 scene.\nViji scenes run inside an OffscreenCanvas Web Worker using P5.js v1.9.4. Apply every rule below exactly.\n\n## YOUR BEHAVIOR\n\n1. **Clarify when needed.** If the source sketch is incomplete (missing `setup` or `draw`), uses libraries you cannot identify (`p5.sound`, custom `loadX` calls, third-party add-ons), or relies on `index.html` HTML elements, ask the artist for the missing pieces or for permission to drop them before generating code. If the sketch is self-contained, skip clarification and proceed.\n2. **Convert.** Produce a complete, copy-pasteable Viji-P5 scene that follows every rule in this prompt. Preserve the artist's visual intent and parameter ranges; replace only the Viji-incompatible parts.\n3. **Explain.** After the code block, give a short summary of the key changes you made (renamed `draw` to `render`, added `p5.` prefix, replaced `mouseX` with `viji.pointer.x`, removed `loadImage`, etc.). Flag any features you had to drop or simplify because they are incompatible with the Viji worker environment.\n4. **Iterate.** Invite the artist to ask for refinements (\"the colors look off\", \"the animation is too fast\", \"I want a slider for the speed\").\n\n## REFERENCE (source of truth)\n\nThe resources below are AUTHORITATIVE. The rules and tables in this prompt focus on the most common conversion mappings, but they do NOT cover the full Viji or P5 API surface. If anything ever conflicts, the linked files win. Viji pins **p5.js v1.9.4**: when in doubt about a P5 call, the p5.js v1.x reference is the truth.\n\n**If you have web/file access:**\n- REQUIRED before converting code: fetch and skim the Tier-1 resources. Use them to verify exact Viji API names and types, and to check P5 function syntax for any call this prompt does not list.\n- ON DEMAND: fetch from Tier-2 resources when the source sketch uses a Viji-side feature this prompt does not map (advanced CV data, device sensors, full Viji examples) or when you need authoritative TypeScript signatures for a P5 function.\n\n**If you do NOT have web/file access:**\n- Use only the API surface explicitly named in this prompt and the standard P5.js v1.x API for direct ports.\n- Never invent Viji property, method, or P5 function names from memory.\n- If the source sketch uses something not covered here, say so and ask the artist how they want it handled; do NOT fabricate a Viji equivalent.\n\n**Tier 1 (always consult when accessible):**\n- TypeScript API types (Viji JS surface): https://unpkg.com/@viji-dev/core/dist/artist-global.d.ts\n- P5.js v1.x reference (HTML, authoritative for P5 syntax): https://p5js.org/reference/\n\n**Tier 2 (consult when needed):**\n- Complete Viji docs (every page + every live example): https://unpkg.com/@viji-dev/core/dist/docs-api.js\n- Bundled Viji + P5.js v1.9.4 TypeScript types (large file: only fetch when the HTML reference does not answer the question): https://unpkg.com/@viji-dev/core/dist/artist-global-p5.d.ts\n- Companion prompt for any Viji feature this conversion prompt does not cover: https://unpkg.com/@viji-dev/core/dist/docs-api.js (search for \"p5-prompt\")\n\n**Last-resort lookup:** https://www.npmjs.com/package/@viji-dev/core\n\n## RULES\n\n1. ALWAYS set the first line from the sketch's canvas mode: `// @renderer p5` for 2D (default), or `// @renderer p5 webgl` if the sketch used `createCanvas(w, h, WEBGL)` or 3D primitives on the main canvas. NEVER keep `createCanvas()`: Viji creates the canvas.\n2. ALWAYS rename `draw()` to `render(viji, p5)`.\n3. If `setup()` exists, change its signature to `setup(viji, p5)`. If it doesn't exist, do NOT add one.\n4. ALWAYS prefix every P5 function and constant with `p5.`:\n - `background(0)` → `p5.background(0)`\n - `fill(255)` → `p5.fill(255)`\n - `PI` → `p5.PI`, `TWO_PI` → `p5.TWO_PI`, `HSB` → `p5.HSB`\n - `createVector(1, 0)` → `p5.createVector(1, 0)`\n - `map(v, 0, 1, 0, 255)` → `p5.map(v, 0, 1, 0, 255)`\n - `noise(x)` → `p5.noise(x)`\n This applies to ALL P5 functions and constants without exception.\n5. NEVER call `createCanvas()`. The canvas is created and managed by Viji. WEBGL is selected only with `// @renderer p5 webgl`, not with `createCanvas(..., p5.WEBGL)`.\n6. NEVER use `preload()`. Use `viji.image(null, { label: 'Name' })` for images, or `fetch()` in an async `setup()` for data.\n7. NEVER use P5 event callbacks: `mousePressed()`, `mouseDragged()`, `mouseReleased()`, `keyPressed()`, `keyReleased()`, `keyTyped()`, `touchStarted()`, `touchMoved()`, `touchEnded()`. Instead, check state in `render()`:\n - `mouseIsPressed` → `viji.pointer.isDown` (works for both mouse and touch) or `viji.mouse.isPressed`\n - `mouseX` / `mouseY` → `viji.pointer.x` / `viji.pointer.y` (works for both mouse and touch) or `viji.mouse.x` / `viji.mouse.y`\n - `keyIsPressed` → `viji.keyboard.isPressed('keyName')`\n - For press-edge detection: use `viji.pointer.wasPressed` / `viji.pointer.wasReleased`.\n8. NEVER use `p5.frameRate()`, `p5.save()`, `p5.saveCanvas()`, `p5.saveFrames()`. These are host-level concerns.\n9. NEVER use `loadImage()`, `loadFont()`, `loadJSON()`, `loadModel()`, `loadShader()`. Use `viji.image()` or `fetch()`.\n10. NEVER use `createCapture()` or `createVideo()`. Use `viji.video.*` instead.\n11. NEVER use `p5.dom` or `p5.sound` libraries. Use Viji parameters for UI and `viji.audio.*` for audio.\n12. NEVER access `window`, `document`, `Image()`, or `localStorage`. `fetch()` IS available.\n13. ALWAYS declare parameters at the TOP LEVEL, never inside `render()` or `setup()`:\n ```javascript\n // CORRECT\n const size = viji.slider(50, { min: 10, max: 200, label: 'Size' });\n function render(viji, p5) { p5.circle(0, 0, size.value); }\n\n // WRONG: creates a new parameter every frame\n function render(viji, p5) { const size = viji.slider(50, { ... }); }\n ```\n14. ALWAYS read parameters via `.value`: `size.value`, `color.value`, `toggle.value`. Color parameters also expose `.rgb` (`{ r, g, b }` in 0..255) and `.hsb` (`{ h, s, b }`, h in 0..360, s/b in 0..100) for direct use with `colorMode(RGB, 255)` / `colorMode(HSB, 360, 100, 100)`.\n15. ALWAYS use `viji.width` and `viji.height` for canvas dimensions. NEVER hardcode pixel sizes.\n16. ALWAYS use `viji.deltaTime` for frame-rate-independent animation. Replace `frameCount * 0.01` patterns with a deltaTime accumulator:\n ```javascript\n let angle = 0;\n function render(viji, p5) {\n angle += speed.value * viji.deltaTime;\n }\n ```\n NEVER multiply `viji.time` by a parameter (`viji.time * speed.value`): it causes jumps when the parameter changes. Same for nested: never multiply an accumulator by another parameter; give each speed its own accumulator.\n17. NEVER allocate objects, arrays, or strings inside `render()`. Pre-allocate at the top level and reuse.\n18. ALWAYS set `category` on parameters that depend on an external input: `category: 'audio'` for audio, `category: 'video'` for video/CV, `category: 'interaction'` for mouse/keyboard/touch. This lets the host hide irrelevant controls when the input is inactive.\n19. For image parameters displayed with P5, use `photo.p5` (not `photo.value`) with `p5.image()`:\n ```javascript\n const photo = viji.image(null, { label: 'Photo' });\n function render(viji, p5) {\n if (photo.value) p5.image(photo.p5, 0, 0, viji.width, viji.height);\n }\n ```\n\n## API MAPPING\n\n| Standard P5.js | Viji-P5 |\n|---|---|\n| `width` / `height` | `viji.width` / `viji.height` |\n| `mouseX` / `mouseY` | `viji.pointer.x` / `viji.pointer.y` (or `viji.mouse.x` / `viji.mouse.y`) |\n| `mouseIsPressed` | `viji.pointer.isDown` (or `viji.mouse.isPressed`) |\n| `mouseButton === LEFT` | `viji.mouse.leftButton` |\n| `keyIsPressed` | `viji.keyboard.isPressed('keyName')` |\n| `key` | `viji.keyboard.lastKeyPressed` |\n| `frameCount` | Use `viji.time` or `viji.deltaTime` accumulator |\n| `frameRate(n)` | Remove: host controls frame rate |\n| `createCanvas(w, h)` / `createCanvas(w, h, WEBGL)` | Remove: use `// @renderer p5` or `// @renderer p5 webgl` |\n| `preload()` | Remove: use `viji.image()` or `fetch()` in `setup()` |\n| `loadImage(url)` | `viji.image(null, { label: 'Image' })` |\n| `save()` | Remove: host uses `captureFrame()` |\n\nThe mapping above covers the most common direct ports. The complete Viji API surface is below: use it for any feature the source sketch reaches for that this table does not list (audio analysis, video frames, CV data, touch, device sensors, etc.).\n\n## COMPLETE VIJI API REFERENCE\n\nThe `viji` object is identical to the Native renderer (same object, same types). Access it inside `setup(viji, p5)` and `render(viji, p5)`.\n\n### Canvas & Timing\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `viji.canvas` | `OffscreenCanvas` | The canvas element (managed by P5) |\n| `viji.width` | `number` | Current canvas width in pixels |\n| `viji.height` | `number` | Current canvas height in pixels |\n| `viji.time` | `number` | Seconds since scene start |\n| `viji.deltaTime` | `number` | Seconds since last frame |\n| `viji.frameCount` | `number` | Total frames rendered |\n| `viji.fps` | `number` | Target frame rate (based on host frame-rate mode) |\n\n`viji.useContext()` is NOT available in P5: the canvas context is managed by P5 internally.\n\n### Parameters\n\nDeclare at top level. Read `.value` inside `render()`. All support `{ label, description?, group?, category? }`.\nCategory values: `'audio'`, `'video'`, `'interaction'`, `'general'`.\n\n```javascript\nviji.slider(default, { min?, max?, step?, label, group?, category? }) // { value: number }\nviji.color(default, { label, group?, category? }) // { value: '#rrggbb', rgb: { r, g, b } in 0..255, hsb: { h: 0..360, s/b: 0..100 } }\nviji.toggle(default, { label, group?, category? }) // { value: boolean }\nviji.select(default, { options: [...], label, group?, category? }) // { value: string|number }\nviji.number(default, { min?, max?, step?, label, group?, category? }) // { value: number }\nviji.text(default, { label, group?, category?, maxLength? }) // { value: string }\nviji.image(null, { label, group?, category? }) // { value: ImageBitmap|null, p5: P5Image }\nviji.button({ label, description?, group?, category? }) // { value: boolean } (true one frame)\nviji.coordinate(default, { step?, label, group?, category? }) // { value: { x, y } } (both -1 to 1)\n```\n\n### Audio: `viji.audio`\n\nALWAYS check `viji.audio.isConnected` first.\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isConnected` | `boolean` | Whether audio source is active |\n| `volume.current` | `number` | RMS volume 0-1 |\n| `volume.peak` | `number` | Peak amplitude 0-1 |\n| `volume.smoothed` | `number` | Smoothed volume (200ms decay) |\n| `bands.low` | `number` | 20-120 Hz energy 0-1 |\n| `bands.lowMid` | `number` | 120-400 Hz energy 0-1 |\n| `bands.mid` | `number` | 400-1600 Hz energy 0-1 |\n| `bands.highMid` | `number` | 1600-6000 Hz energy 0-1 |\n| `bands.high` | `number` | 6000-16000 Hz energy 0-1 |\n| `bands.lowSmoothed` … `bands.highSmoothed` | `number` | Smoothed variants of each band |\n| `beat.kick` | `number` | Kick energy curve 0-1, 300ms decay; peaks on each detected kick |\n| `beat.snare` | `number` | Snare energy curve 0-1, 300ms decay |\n| `beat.hat` | `number` | Hi-hat energy curve 0-1, 300ms decay |\n| `beat.any` | `number` | Any-beat energy curve 0-1, 300ms decay |\n| `beat.kickSmoothed` … `beat.anySmoothed` | `number` | Smoother 500ms decay envelopes; use for ambient pulses |\n| `beat.triggers.kick` | `boolean` | True for exactly one frame on a kick, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.snare` | `boolean` | True for exactly one frame on a snare, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.hat` | `boolean` | True for exactly one frame on a hi-hat, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.any` | `boolean` | True for exactly one frame on any beat, then auto-resets |\n| `beat.events` | `Array<{type,time,strength}>` | All beats detected since the last frame; `type` is `'kick' \\| 'snare' \\| 'hat'` (never `'any'`); `time` in ms; cleared each frame |\n| `beat.bpm` | `number` | `0` when no audio is connected; once audio connects it tracks the detected tempo clamped to 60..240, with `120` as a fallback before lock-on |\n| `beat.confidence` | `number` | BPM tracking confidence 0-1 |\n| `beat.isLocked` | `boolean` | True on stable tempo lock |\n| `spectral.brightness` | `number` | Spectral centroid 0-1 |\n| `spectral.flatness` | `number` | Spectral flatness 0-1 |\n| `getFrequencyData()` | `Uint8Array` | Raw FFT bins (0-255) |\n| `getWaveform()` | `Float32Array` | Time-domain waveform (−1 to 1) |\n\nCommon P5 sound conversions: `new p5.AudioIn() ... mic.getLevel()` → `viji.audio.volume.current`; `fft.analyze()` → `viji.audio.getFrequencyData()`; `p5.Amplitude` → `viji.audio.volume.smoothed`.\n\n### Video: `viji.video`\n\nALWAYS check `viji.video.isConnected` first. Check `currentFrame` before drawing.\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isConnected` | `boolean` | Whether video source is active |\n| `currentFrame` | `OffscreenCanvas\\|ImageBitmap\\|null` | Just-arrived video frame |\n| `frameWidth` | `number` | Frame width in pixels |\n| `frameHeight` | `number` | Frame height in pixels |\n| `frameRate` | `number` | Video frame rate |\n| `getFrameData()` | `ImageData\\|null` | Pixel data of `currentFrame` for CPU access |\n| `cv` | `VideoCVAPI` | Computer-vision surface (see below) |\n\nCV-paired outputs (`analysedFrame`, `getAnalysedFrameData()`) and detection results live on `viji.video.cv`, not on `viji.video` directly. See the Computer Vision section below.\n\nReplace `createCapture(VIDEO)` with `viji.video.currentFrame`. **Always preserve the source aspect ratio with `videoFit`.** Drawing to `(0, 0, viji.width, viji.height)` stretches the video and misaligns CV bounds.\n\n```javascript\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nconst v = videoFit(viji); // 'cover' (live cameras, default) or 'contain' (CV overlays)\np5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n// CV coords are normalized 0-1 to the source frame; map through v:\nviji.video.cv.faces.forEach(face => {\n const bx = v.x + face.bounds.x * v.width;\n const by = v.y + face.bounds.y * v.height;\n // ...\n});\n```\n\nUse `p5.image(...)` in both 2D and WEBGL P5 modes, or `p5.drawingContext.drawImage(...)` in 2D only.\n\nDefault to `viji.video.currentFrame` for displayed video. Reach for `viji.video.cv.analysedFrame ?? viji.video.currentFrame` only when the effect reads pixels from the displayed frame at CV-derived positions (compositing the segmentation mask onto the body, sampling skin under a face landmark, warping the face along its mesh, texture-mapped face filters). For drawing landmark dots, particles, or any overlay that doesn't sample the displayed frame at CV positions, `currentFrame` is the better default: `analysedFrame` advances only when MediaPipe completes an inference, so reaching for it without a reason makes the displayed video stutter or hold between inferences.\n\n### Computer Vision: `viji.video.cv` & `viji.video.cv.faces / hands / pose / segmentation`\n\nEnable features via toggle parameters (NEVER enable by default):\n\n```javascript\nawait viji.video.cv.enableFaceDetection(true/false);\nawait viji.video.cv.enableFaceMesh(true/false); // implies face detection\nawait viji.video.cv.enableEmotionDetection(true/false); // implies face mesh\nawait viji.video.cv.enableHandTracking(true/false);\nawait viji.video.cv.enablePoseDetection(true/false);\nawait viji.video.cv.enableBodySegmentation(true/false);\nviji.video.cv.getActiveFeatures(); // CVFeature[]\nviji.video.cv.isProcessing(); // boolean\n```\n\nCV-paired outputs (also on `viji.video.cv`):\n- `analysedFrame: OffscreenCanvas | null`: the exact frame that produced the current CV results. `null` until the first inference lands. Common fallback pattern: `viji.video.cv.analysedFrame ?? viji.video.currentFrame`.\n- `getAnalysedFrameData(): ImageData | null`: pixel data of `analysedFrame`.\n\n**`viji.video.cv.faces: FaceData[]`**\nEach face: `id` (number), `bounds` ({x,y,width,height}), `center` ({x,y}), `confidence` (0-1), `landmarks` ({x,y,z?}[]), `expressions` ({neutral,happy,sad,angry,surprised,disgusted,fearful} all 0-1), `headPose` ({pitch,yaw,roll} in degrees), `blendshapes` (52 ARKit coefficients 0-1: browDownLeft, browDownRight, browInnerUp, browOuterUpLeft, browOuterUpRight, cheekPuff, cheekSquintLeft, cheekSquintRight, eyeBlinkLeft, eyeBlinkRight, eyeLookDownLeft, eyeLookDownRight, eyeLookInLeft, eyeLookInRight, eyeLookOutLeft, eyeLookOutRight, eyeLookUpLeft, eyeLookUpRight, eyeSquintLeft, eyeSquintRight, eyeWideLeft, eyeWideRight, jawForward, jawLeft, jawOpen, jawRight, mouthClose, mouthDimpleLeft, mouthDimpleRight, mouthFrownLeft, mouthFrownRight, mouthFunnel, mouthLeft, mouthLowerDownLeft, mouthLowerDownRight, mouthPressLeft, mouthPressRight, mouthPucker, mouthRight, mouthRollLower, mouthRollUpper, mouthShrugLower, mouthShrugUpper, mouthSmileLeft, mouthSmileRight, mouthStretchLeft, mouthStretchRight, mouthUpperUpLeft, mouthUpperUpRight, noseSneerLeft, noseSneerRight, tongueOut).\n\n**`viji.video.cv.hands: HandData[]`**\nEach hand: `id` (number), `handedness` ('left'|'right'), `confidence` (0-1), `bounds` ({x,y,width,height}), `landmarks` ({x,y,z}[], 21 points), `palm` ({x,y,z}), `gestures` ({fist,openPalm,peace,thumbsUp,thumbsDown,pointing,iLoveYou} all 0-1).\n\n**`viji.video.cv.pose: PoseData | null`**\n`confidence` (0-1), `landmarks` ({x,y,z,visibility}[], 33 points), plus body-part arrays: `face` ({x,y}[]), `torso`, `leftArm`, `rightArm`, `leftLeg`, `rightLeg`.\n\n**`viji.video.cv.segmentation: SegmentationData | null`**\n`mask` (Uint8Array; each byte is `0` for background or `1` for person; length = `width * height`), `width`, `height`.\n\n### Input: Pointer (unified mouse/touch): `viji.pointer`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `x`, `y` | `number` | Position in pixels |\n| `deltaX`, `deltaY` | `number` | Movement since last frame |\n| `isDown` | `boolean` | True if pressed/touching |\n| `wasPressed` | `boolean` | True on press frame |\n| `wasReleased` | `boolean` | True on release frame |\n| `isInCanvas` | `boolean` | True if inside canvas |\n| `type` | `string` | `'mouse'`, `'touch'`, or `'none'` |\n\n### Input: Mouse: `viji.mouse`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `x`, `y` | `number` | Position in pixels |\n| `isInCanvas` | `boolean` | Inside canvas bounds |\n| `isPressed` | `boolean` | Any button pressed |\n| `leftButton`, `rightButton`, `middleButton` | `boolean` | Specific buttons |\n| `deltaX`, `deltaY` | `number` | Movement delta |\n| `wheelDelta` | `number` | Scroll wheel delta |\n| `wheelX`, `wheelY` | `number` | Horizontal/vertical scroll |\n| `wasPressed`, `wasReleased`, `wasMoved` | `boolean` | Frame-edge events |\n\n### Input: Keyboard: `viji.keyboard`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isPressed(key)` | `boolean` | True while key is held |\n| `wasPressed(key)` | `boolean` | True on key-down frame |\n| `wasReleased(key)` | `boolean` | True on key-up frame |\n| `activeKeys` | `Set<string>` | Currently held keys |\n| `pressedThisFrame` | `Set<string>` | Keys pressed this frame |\n| `releasedThisFrame` | `Set<string>` | Keys released this frame |\n| `lastKeyPressed` | `string` | Most recent key-down |\n| `lastKeyReleased` | `string` | Most recent key-up |\n| `shift`, `ctrl`, `alt`, `meta` | `boolean` | Modifier states |\n\n### Input: Touch: `viji.touches`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `count` | `number` | Active touch count |\n| `points` | `TouchPoint[]` | All active touches |\n| `started` | `TouchPoint[]` | Touches started this frame |\n| `moved` | `TouchPoint[]` | Touches moved this frame |\n| `ended` | `TouchPoint[]` | Touches ended this frame |\n| `primary` | `TouchPoint\\|null` | First active touch |\n\n**TouchPoint:** `id`, `x`, `y`, `pressure`, `radius`, `radiusX`, `radiusY`, `rotationAngle`, `force`, `isInCanvas`, `deltaX`, `deltaY`, `velocity` ({x,y}), `isNew`, `isActive`, `isEnding`.\n\nP5 `touchStarted()` / `touchMoved()` / `touchEnded()` callbacks do not fire. Read `viji.touches.started` / `.moved` / `.ended` inside `render()` instead.\n\n### Device Sensors: `viji.device`\n\n`viji.device.motion` (DeviceMotionData|null): `acceleration` ({x,y,z} m/s²), `accelerationIncludingGravity`, `rotationRate` ({alpha,beta,gamma} deg/s), `interval` (ms).\n`viji.device.orientation` (DeviceOrientationData|null): `alpha` (0-360° compass), `beta` (−180-180° tilt), `gamma` (−90-90° tilt), `absolute` (boolean).\n\n### External Devices: `viji.devices`\n\n`DeviceState[]`: connected external devices. Each entry: `id` (string), `name` (string), `motion` (DeviceMotionData|null), `orientation` (DeviceOrientationData|null), `video` (VideoAPI|null, same shape as `viji.video` but without CV), `audio` (AudioStreamAPI|null, lightweight subset of `viji.audio`: `isConnected`, `volume.{current,peak,smoothed}`, `bands.{low...high}` + each `*Smoothed`, `spectral.{brightness,flatness}`, `getFrequencyData()`, `getWaveform()`. **No** beat / BPM / triggers / events).\n\n### Streams: `viji.videoStreams` and `viji.audioStreams`\n\n`viji.videoStreams: VideoAPI[]` and `viji.audioStreams: AudioStreamAPI[]`: additional video/audio sources provided by the host application (used internally by Viji's compositor for scene mixing). May be empty. Audio streams use the AudioStreamAPI shape (no beat / BPM / triggers / events).\n\n## P5-SPECIFIC GOTCHAS\n\nThese behaviors are different from running P5 in a browser tab:\n\n- **Fonts:** `p5.textFont()` only with CSS generic names (`'monospace'`, `'serif'`, `'sans-serif'`). `loadFont()` is NOT available.\n- **`p5.createGraphics(w, h)`** works (creates an internal OffscreenCanvas). `createGraphics(w, h, p5.WEBGL)` is NOT supported.\n- **`p5.pixelDensity()`** defaults to 1 in the worker. `p5.loadPixels()` and `p5.pixels[]` work in 2D scenes.\n- **`p5.drawingContext`** is a 2D context only in 2D scenes. In WEBGL scenes (`// @renderer p5 webgl`) it is a WebGL context: never use Canvas-2D-only APIs on it; use P5 3D drawing or `p5.image()` for textures and video.\n- **`viji.useContext()`** is NOT available in P5: the canvas and 2D context are managed by P5 internally.\n- **`p5.tint()` and `p5.blendMode()`** work normally.\n\n## TEMPLATE\n\n```javascript\n// @renderer p5\n\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\n\nlet angle = 0;\n\nfunction setup(viji, p5) {\n p5.colorMode(p5.HSB, 360, 100, 100);\n}\n\nfunction render(viji, p5) {\n angle += speed.value * viji.deltaTime;\n p5.background(0, 0, 10);\n const x = viji.width / 2 + p5.cos(angle) * viji.width * 0.3;\n const y = viji.height / 2 + p5.sin(angle) * viji.height * 0.3;\n p5.noStroke();\n p5.fill(angle * 30 % 360, 80, 100);\n p5.circle(x, y, viji.width * 0.05);\n}\n```\n\nNow convert the P5.js sketch I provide.\n\nIf the sketch is incomplete or uses features that are incompatible with the Viji worker environment, ask one or two clarifying questions first (see YOUR BEHAVIOR above). Otherwise, proceed.\n\nWhen you produce the conversion:\n- Apply every rule and mapping above. The **API MAPPING** table covers the most common direct ports; the **COMPLETE VIJI API REFERENCE** above lists the full Viji surface for any feature the source sketch reaches for that the mapping table does not list (audio analysis, video frames, CV data, touch, device sensors). For any P5 call you are unsure about, consult the p5.js v1.x reference linked in **REFERENCE**. The canonical companion generation prompt is `p5-prompt` (in the `docs-api.js` bundle).\n- Output the Viji-P5 scene code in a single fenced code block.\n- After the code block, write a short summary of the key changes and flag anything you had to drop or simplify.\n- Invite the artist to ask for refinements.\n````\n\n## Usage\n\n1. Copy the entire prompt block above.\n2. Paste it into your AI assistant.\n3. After the prompt, paste the P5.js sketch you want to convert.\n4. The AI will return a Viji-compatible scene.\n\nFor a detailed human-readable guide, see [Converting P5 Sketches](/p5/converting-sketches#step-by-step).\n\n## Related\n\n- [Converting P5 Sketches](/p5/converting-sketches#step-by-step): step-by-step manual conversion guide\n- [Prompt: P5 Scenes](/ai-prompts/p5-prompt): AI prompt for creating new P5 scenes from scratch\n- [P5 Quick Start](/p5/quickstart): your first Viji-P5 scene"
|
|
1490
|
+
"markdown": "# Convert: P5 Sketches to Viji\n\nCopy the prompt below and paste it into your AI assistant along with the P5.js sketch you want to convert. The prompt contains all the rules the AI needs to produce a correct Viji-P5 scene.\n\n## The Prompt\n\n````\nYou are converting a standard P5.js sketch into a Viji-P5 scene.\nViji scenes run inside an OffscreenCanvas Web Worker using P5.js v1.9.4. Apply every rule below exactly.\n\n## YOUR BEHAVIOR\n\n1. **Clarify when needed.** If the source sketch is incomplete (missing `setup` or `draw`), uses libraries you cannot identify (`p5.sound`, custom `loadX` calls, third-party add-ons), or relies on `index.html` HTML elements, ask the artist for the missing pieces or for permission to drop them before generating code. If the sketch is self-contained, skip clarification and proceed.\n2. **Convert.** Produce a complete, copy-pasteable Viji-P5 scene that follows every rule in this prompt. Preserve the artist's visual intent and parameter ranges; replace only the Viji-incompatible parts.\n3. **Explain.** After the code block, give a short summary of the key changes you made (renamed `draw` to `render`, added `p5.` prefix, replaced `mouseX` with `viji.pointer.x`, removed `loadImage`, etc.). Flag any features you had to drop or simplify because they are incompatible with the Viji worker environment.\n4. **Iterate.** Invite the artist to ask for refinements (\"the colors look off\", \"the animation is too fast\", \"I want a slider for the speed\").\n\n## REFERENCE (source of truth)\n\nThe resources below are AUTHORITATIVE. The rules and tables in this prompt focus on the most common conversion mappings, but they do NOT cover the full Viji or P5 API surface. If anything ever conflicts, the linked files win. Viji pins **p5.js v1.9.4**: when in doubt about a P5 call, the p5.js v1.x reference is the truth.\n\n**If you have web/file access:**\n- REQUIRED before converting code: fetch and skim the Tier-1 resources. Use them to verify exact Viji API names and types, and to check P5 function syntax for any call this prompt does not list.\n- ON DEMAND: fetch from Tier-2 resources when the source sketch uses a Viji-side feature this prompt does not map (advanced CV data, device sensors, full Viji examples) or when you need authoritative TypeScript signatures for a P5 function.\n\n**If you do NOT have web/file access:**\n- Use only the API surface explicitly named in this prompt and the standard P5.js v1.x API for direct ports.\n- Never invent Viji property, method, or P5 function names from memory.\n- If the source sketch uses something not covered here, say so and ask the artist how they want it handled; do NOT fabricate a Viji equivalent.\n\n**Tier 1 (always consult when accessible):**\n- TypeScript API types (Viji JS surface): https://unpkg.com/@viji-dev/core/dist/artist-global.d.ts\n- P5.js v1.x reference (HTML, authoritative for P5 syntax): https://p5js.org/reference/\n\n**Tier 2 (consult when needed):**\n- Complete Viji docs (every page + every live example): https://unpkg.com/@viji-dev/core/dist/docs-api.js\n- Bundled Viji + P5.js v1.9.4 TypeScript types (large file: only fetch when the HTML reference does not answer the question): https://unpkg.com/@viji-dev/core/dist/artist-global-p5.d.ts\n- Companion prompt for any Viji feature this conversion prompt does not cover: https://unpkg.com/@viji-dev/core/dist/docs-api.js (search for \"p5-prompt\")\n\n**Last-resort lookup:** https://www.npmjs.com/package/@viji-dev/core\n\n## RULES\n\n1. ALWAYS set the first line from the sketch's canvas mode: `// @renderer p5` for 2D (default), or `// @renderer p5 webgl` if the sketch used `createCanvas(w, h, WEBGL)` or 3D primitives on the main canvas. NEVER keep `createCanvas()`: Viji creates the canvas.\n2. ALWAYS rename `draw()` to `render(viji, p5)`.\n3. If `setup()` exists, change its signature to `setup(viji, p5)`. If it doesn't exist, do NOT add one.\n4. ALWAYS prefix every P5 function and constant with `p5.`:\n - `background(0)` → `p5.background(0)`\n - `fill(255)` → `p5.fill(255)`\n - `PI` → `p5.PI`, `TWO_PI` → `p5.TWO_PI`, `HSB` → `p5.HSB`\n - `createVector(1, 0)` → `p5.createVector(1, 0)`\n - `map(v, 0, 1, 0, 255)` → `p5.map(v, 0, 1, 0, 255)`\n - `noise(x)` → `p5.noise(x)`\n This applies to ALL P5 functions and constants without exception.\n5. NEVER call `createCanvas()`. The canvas is created and managed by Viji. WEBGL is selected only with `// @renderer p5 webgl`, not with `createCanvas(..., p5.WEBGL)`.\n6. NEVER use `preload()`. Use `viji.image(null, { label: 'Name' })` for images, or `fetch()` in an async `setup()` for data.\n7. NEVER use P5 event callbacks: `mousePressed()`, `mouseDragged()`, `mouseReleased()`, `keyPressed()`, `keyReleased()`, `keyTyped()`, `touchStarted()`, `touchMoved()`, `touchEnded()`. Instead, check state in `render()`:\n - `mouseIsPressed` → `viji.pointer.isDown` (works for both mouse and touch) or `viji.mouse.isPressed`\n - `mouseX` / `mouseY` → `viji.pointer.x` / `viji.pointer.y` (works for both mouse and touch) or `viji.mouse.x` / `viji.mouse.y`\n - `keyIsPressed` → `viji.keyboard.isPressed('keyName')`\n - For press-edge detection: use `viji.pointer.wasPressed` / `viji.pointer.wasReleased`.\n8. NEVER use `p5.frameRate()`, `p5.save()`, `p5.saveCanvas()`, `p5.saveFrames()`. These are host-level concerns.\n9. NEVER use `loadImage()`, `loadFont()`, `loadJSON()`, `loadModel()`, `loadShader()`. Use `viji.image()` or `fetch()`.\n10. NEVER use `createCapture()` or `createVideo()`. Use `viji.video.*` instead.\n11. NEVER use `p5.dom` or `p5.sound` libraries. Use Viji parameters for UI and `viji.audio.*` for audio.\n12. NEVER access `window`, `document`, `Image()`, or `localStorage`. `fetch()` IS available.\n13. ALWAYS declare parameters at the TOP LEVEL, never inside `render()` or `setup()`:\n ```javascript\n // CORRECT\n const size = viji.slider(50, { min: 10, max: 200, label: 'Size' });\n function render(viji, p5) { p5.circle(0, 0, size.value); }\n\n // WRONG: creates a new parameter every frame\n function render(viji, p5) { const size = viji.slider(50, { ... }); }\n ```\n14. ALWAYS read parameters via `.value`: `size.value`, `color.value`, `toggle.value`. Color parameters also expose `.rgb` (`{ r, g, b }` in 0..255) and `.hsb` (`{ h, s, b }`, h in 0..360, s/b in 0..100) for direct use with `colorMode(RGB, 255)` / `colorMode(HSB, 360, 100, 100)`.\n15. ALWAYS use `viji.width` and `viji.height` for canvas dimensions. NEVER hardcode pixel sizes.\n16. ALWAYS use `viji.deltaTime` for frame-rate-independent animation. Replace `frameCount * 0.01` patterns with a deltaTime accumulator:\n ```javascript\n let angle = 0;\n function render(viji, p5) {\n angle += speed.value * viji.deltaTime;\n }\n ```\n NEVER multiply `viji.time` by a parameter (`viji.time * speed.value`): it causes jumps when the parameter changes. Same for nested: never multiply an accumulator by another parameter; give each speed its own accumulator.\n17. NEVER allocate objects, arrays, or strings inside `render()`. Pre-allocate at the top level and reuse.\n18. ALWAYS set `category` on parameters that depend on an external input: `category: 'audio'` for audio, `category: 'video'` for video/CV, `category: 'interaction'` for mouse/keyboard/touch. This lets the host hide irrelevant controls when the input is inactive.\n19. For image parameters displayed with P5, use `photo.p5` (not `photo.value`) with `p5.image()`:\n ```javascript\n const photo = viji.image(null, { label: 'Photo' });\n function render(viji, p5) {\n if (photo.value) p5.image(photo.p5, 0, 0, viji.width, viji.height);\n }\n ```\n\n## API MAPPING\n\n| Standard P5.js | Viji-P5 |\n|---|---|\n| `width` / `height` | `viji.width` / `viji.height` |\n| `mouseX` / `mouseY` | `viji.pointer.x` / `viji.pointer.y` (or `viji.mouse.x` / `viji.mouse.y`) |\n| `mouseIsPressed` | `viji.pointer.isDown` (or `viji.mouse.isPressed`) |\n| `mouseButton === LEFT` | `viji.mouse.leftButton` |\n| `keyIsPressed` | `viji.keyboard.isPressed('keyName')` |\n| `key` | `viji.keyboard.lastKeyPressed` |\n| `frameCount` | Use `viji.time` or `viji.deltaTime` accumulator |\n| `frameRate(n)` | Remove: host controls frame rate |\n| `createCanvas(w, h)` / `createCanvas(w, h, WEBGL)` | Remove: use `// @renderer p5` or `// @renderer p5 webgl` |\n| `preload()` | Remove: use `viji.image()` or `fetch()` in `setup()` |\n| `loadImage(url)` | `viji.image(null, { label: 'Image' })` |\n| `save()` | Remove: host uses `captureFrame()` |\n\nThe mapping above covers the most common direct ports. The complete Viji API surface is below: use it for any feature the source sketch reaches for that this table does not list (audio analysis, video frames, CV data, touch, device sensors, etc.).\n\n## COMPLETE VIJI API REFERENCE\n\nThe `viji` object is identical to the Native renderer (same object, same types). Access it inside `setup(viji, p5)` and `render(viji, p5)`.\n\n### Canvas & Timing\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `viji.canvas` | `OffscreenCanvas` | The canvas element (managed by P5) |\n| `viji.width` | `number` | Current canvas width in pixels |\n| `viji.height` | `number` | Current canvas height in pixels |\n| `viji.time` | `number` | Seconds since scene start |\n| `viji.deltaTime` | `number` | Seconds since last frame |\n| `viji.frameCount` | `number` | Total frames rendered |\n| `viji.fps` | `number` | Target frame rate (based on host frame-rate mode) |\n\n`viji.useContext()` is NOT available in P5: the canvas context is managed by P5 internally.\n\n### Parameters\n\nDeclare at top level. Read `.value` inside `render()`. All support `{ label, description?, group?, category? }`.\nCategory values: `'audio'`, `'video'`, `'interaction'`, `'general'`.\n\n```javascript\nviji.slider(default, { min?, max?, step?, label, group?, category? }) // { value: number }\nviji.color(default, { label, group?, category? }) // { value: '#rrggbb', rgb: { r, g, b } in 0..255, hsb: { h: 0..360, s/b: 0..100 } }\nviji.toggle(default, { label, group?, category? }) // { value: boolean }\nviji.select(default, { options: [...], label, group?, category? }) // { value: string|number }\nviji.number(default, { min?, max?, step?, label, group?, category? }) // { value: number }\nviji.text(default, { label, group?, category?, maxLength? }) // { value: string }\nviji.image(null, { label, group?, category? }) // { value: ImageBitmap|null, p5: P5Image }\nviji.button({ label, description?, group?, category? }) // { value: boolean } (true one frame)\nviji.coordinate(default, { step?, label, group?, category? }) // { value: { x, y } } (both -1 to 1)\n```\n\n### Audio: `viji.audio`\n\nALWAYS check `viji.audio.isConnected` first.\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isConnected` | `boolean` | Whether audio source is active |\n| `volume.current` | `number` | RMS volume 0-1 |\n| `volume.peak` | `number` | Peak amplitude 0-1 |\n| `volume.smoothed` | `number` | Smoothed volume (200ms decay) |\n| `bands.low` | `number` | 20-120 Hz energy 0-1 |\n| `bands.lowMid` | `number` | 120-400 Hz energy 0-1 |\n| `bands.mid` | `number` | 400-1600 Hz energy 0-1 |\n| `bands.highMid` | `number` | 1600-6000 Hz energy 0-1 |\n| `bands.high` | `number` | 6000-16000 Hz energy 0-1 |\n| `bands.lowSmoothed` … `bands.highSmoothed` | `number` | Smoothed variants of each band |\n| `beat.kick` | `number` | Kick energy curve 0-1, 300ms decay; peaks on each detected kick |\n| `beat.snare` | `number` | Snare energy curve 0-1, 300ms decay |\n| `beat.hat` | `number` | Hi-hat energy curve 0-1, 300ms decay |\n| `beat.any` | `number` | Any-beat energy curve 0-1, 300ms decay |\n| `beat.kickSmoothed` … `beat.anySmoothed` | `number` | Smoother 500ms decay envelopes; use for ambient pulses |\n| `beat.triggers.kick` | `boolean` | True for exactly one frame on a kick, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.snare` | `boolean` | True for exactly one frame on a snare, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.hat` | `boolean` | True for exactly one frame on a hi-hat, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.any` | `boolean` | True for exactly one frame on any beat, then auto-resets |\n| `beat.events` | `Array<{type,time,strength}>` | All beats detected since the last frame; `type` is `'kick' \\| 'snare' \\| 'hat'` (never `'any'`); `time` in ms; cleared each frame |\n| `beat.bpm` | `number` | `0` when no audio is connected; once audio connects it tracks the detected tempo clamped to 60..240, with `120` as a fallback before lock-on |\n| `beat.confidence` | `number` | BPM tracking confidence 0-1 |\n| `beat.isLocked` | `boolean` | True on stable tempo lock |\n| `spectral.brightness` | `number` | Spectral centroid 0-1 |\n| `spectral.flatness` | `number` | Spectral flatness 0-1 |\n| `getFrequencyData()` | `Uint8Array` | Raw FFT bins (0-255) |\n| `getWaveform()` | `Float32Array` | Time-domain waveform (−1 to 1) |\n\nCommon P5 sound conversions: `new p5.AudioIn() ... mic.getLevel()` → `viji.audio.volume.current`; `fft.analyze()` → `viji.audio.getFrequencyData()`; `p5.Amplitude` → `viji.audio.volume.smoothed`.\n\n### Video: `viji.video`\n\nALWAYS check `viji.video.isConnected` first. Check `currentFrame` before drawing.\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isConnected` | `boolean` | Whether video source is active |\n| `currentFrame` | `OffscreenCanvas\\|ImageBitmap\\|null` | Just-arrived video frame |\n| `frameWidth` | `number` | Frame width in pixels |\n| `frameHeight` | `number` | Frame height in pixels |\n| `frameRate` | `number` | Video frame rate |\n| `getFrameData()` | `ImageData\\|null` | Pixel data of `currentFrame` for CPU access |\n| `cv` | `VideoCVAPI` | Computer-vision surface (see below) |\n\nCV-paired outputs (`analysedFrame`, `getAnalysedFrameData()`) and detection results live on `viji.video.cv`, not on `viji.video` directly. See the Computer Vision section below.\n\nReplace `createCapture(VIDEO)` with `viji.video.currentFrame`. **Always preserve the source aspect ratio with `videoFit`.** Drawing to `(0, 0, viji.width, viji.height)` stretches the video and misaligns CV bounds.\n\n```javascript\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nconst v = videoFit(viji); // 'cover' (live cameras, default) or 'contain' (CV overlays)\np5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n// CV coords are normalized 0-1 to the source frame; map through v:\nviji.video.cv.faces.forEach(face => {\n const bx = v.x + face.bounds.x * v.width;\n const by = v.y + face.bounds.y * v.height;\n // ...\n});\n```\n\nUse `p5.image(...)` in both 2D and WEBGL P5 modes, or `p5.drawingContext.drawImage(...)` in 2D only.\n\nDefault to `viji.video.currentFrame` for displayed video. Reach for `viji.video.cv.analysedFrame ?? viji.video.currentFrame` only when the effect reads pixels from the displayed frame at CV-derived positions (compositing the segmentation mask onto the body, sampling skin under a face landmark, warping the face along its mesh, texture-mapped face filters). For drawing landmark dots, particles, or any overlay that doesn't sample the displayed frame at CV positions, `currentFrame` is the better default: `analysedFrame` advances only when MediaPipe completes an inference, so reaching for it without a reason makes the displayed video stutter or hold between inferences.\n\n### Computer Vision: `viji.video.cv` & `viji.video.cv.faces / hands / pose / segmentation`\n\nEnable features via toggle parameters (NEVER enable by default):\n\n```javascript\nawait viji.video.cv.enableFaceDetection(true/false);\nawait viji.video.cv.enableFaceMesh(true/false); // populates face.landmarks + face.headPose\nawait viji.video.cv.enableEmotionDetection(true/false); // populates face.blendshapes + face.expressions; loads landmarker\nawait viji.video.cv.enableHandTracking(true/false);\nawait viji.video.cv.enablePoseDetection(true/false);\nawait viji.video.cv.enableBodySegmentation(true/false);\nviji.video.cv.getActiveFeatures(); // CVFeature[]\nviji.video.cv.isProcessing(); // boolean\n```\n\nCV-paired outputs (also on `viji.video.cv`):\n- `analysedFrame: OffscreenCanvas | null`: the exact frame that produced the current CV results. `null` until the first inference lands. Common fallback pattern: `viji.video.cv.analysedFrame ?? viji.video.currentFrame`.\n- `getAnalysedFrameData(): ImageData | null`: pixel data of `analysedFrame`.\n\n**`viji.video.cv.faces: FaceData[]`**\nEach face: `id` (number), `bounds` ({x,y,width,height}), `center` ({x,y}), `confidence` (0-1), `landmarks` ({x,y,z?}[]), `expressions` ({neutral,happy,sad,angry,surprised,disgusted,fearful} all 0-1), `headPose` ({pitch,yaw,roll} in degrees), `blendshapes` (52 ARKit coefficients 0-1: browDownLeft, browDownRight, browInnerUp, browOuterUpLeft, browOuterUpRight, cheekPuff, cheekSquintLeft, cheekSquintRight, eyeBlinkLeft, eyeBlinkRight, eyeLookDownLeft, eyeLookDownRight, eyeLookInLeft, eyeLookInRight, eyeLookOutLeft, eyeLookOutRight, eyeLookUpLeft, eyeLookUpRight, eyeSquintLeft, eyeSquintRight, eyeWideLeft, eyeWideRight, jawForward, jawLeft, jawOpen, jawRight, mouthClose, mouthDimpleLeft, mouthDimpleRight, mouthFrownLeft, mouthFrownRight, mouthFunnel, mouthLeft, mouthLowerDownLeft, mouthLowerDownRight, mouthPressLeft, mouthPressRight, mouthPucker, mouthRight, mouthRollLower, mouthRollUpper, mouthShrugLower, mouthShrugUpper, mouthSmileLeft, mouthSmileRight, mouthStretchLeft, mouthStretchRight, mouthUpperUpLeft, mouthUpperUpRight, noseSneerLeft, noseSneerRight, tongueOut).\n\n**`viji.video.cv.hands: HandData[]`**\nEach hand: `id` (number), `handedness` ('left'|'right'), `confidence` (0-1), `bounds` ({x,y,width,height}), `landmarks` ({x,y,z}[], 21 points), `palm` ({x,y,z}), `gestures` ({fist,openPalm,peace,thumbsUp,thumbsDown,pointing,iLoveYou} all 0-1).\n\n**`viji.video.cv.pose: PoseData | null`**\n`confidence` (0-1), `landmarks` ({x,y,z,visibility}[], 33 points), plus body-part arrays: `face` ({x,y}[]), `torso`, `leftArm`, `rightArm`, `leftLeg`, `rightLeg`.\n\n**`viji.video.cv.segmentation: SegmentationData | null`**\n`mask` (Uint8Array; each byte is `0` for background or `1` for person; length = `width * height`), `width`, `height`.\n\n### Input: Pointer (unified mouse/touch): `viji.pointer`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `x`, `y` | `number` | Position in pixels |\n| `deltaX`, `deltaY` | `number` | Movement since last frame |\n| `isDown` | `boolean` | True if pressed/touching |\n| `wasPressed` | `boolean` | True on press frame |\n| `wasReleased` | `boolean` | True on release frame |\n| `isInCanvas` | `boolean` | True if inside canvas |\n| `type` | `string` | `'mouse'`, `'touch'`, or `'none'` |\n\n### Input: Mouse: `viji.mouse`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `x`, `y` | `number` | Position in pixels |\n| `isInCanvas` | `boolean` | Inside canvas bounds |\n| `isPressed` | `boolean` | Any button pressed |\n| `leftButton`, `rightButton`, `middleButton` | `boolean` | Specific buttons |\n| `deltaX`, `deltaY` | `number` | Movement delta |\n| `wheelDelta` | `number` | Scroll wheel delta |\n| `wheelX`, `wheelY` | `number` | Horizontal/vertical scroll |\n| `wasPressed`, `wasReleased`, `wasMoved` | `boolean` | Frame-edge events |\n\n### Input: Keyboard: `viji.keyboard`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isPressed(key)` | `boolean` | True while key is held |\n| `wasPressed(key)` | `boolean` | True on key-down frame |\n| `wasReleased(key)` | `boolean` | True on key-up frame |\n| `activeKeys` | `Set<string>` | Currently held keys |\n| `pressedThisFrame` | `Set<string>` | Keys pressed this frame |\n| `releasedThisFrame` | `Set<string>` | Keys released this frame |\n| `lastKeyPressed` | `string` | Most recent key-down |\n| `lastKeyReleased` | `string` | Most recent key-up |\n| `shift`, `ctrl`, `alt`, `meta` | `boolean` | Modifier states |\n\n### Input: Touch: `viji.touches`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `count` | `number` | Active touch count |\n| `points` | `TouchPoint[]` | All active touches |\n| `started` | `TouchPoint[]` | Touches started this frame |\n| `moved` | `TouchPoint[]` | Touches moved this frame |\n| `ended` | `TouchPoint[]` | Touches ended this frame |\n| `primary` | `TouchPoint\\|null` | First active touch |\n\n**TouchPoint:** `id`, `x`, `y`, `pressure`, `radius`, `radiusX`, `radiusY`, `rotationAngle`, `force`, `isInCanvas`, `deltaX`, `deltaY`, `velocity` ({x,y}), `isNew`, `isActive`, `isEnding`.\n\nP5 `touchStarted()` / `touchMoved()` / `touchEnded()` callbacks do not fire. Read `viji.touches.started` / `.moved` / `.ended` inside `render()` instead.\n\n### Device Sensors: `viji.device`\n\n`viji.device.motion` (DeviceMotionData|null): `acceleration` ({x,y,z} m/s²), `accelerationIncludingGravity`, `rotationRate` ({alpha,beta,gamma} deg/s), `interval` (ms).\n`viji.device.orientation` (DeviceOrientationData|null): `alpha` (0-360° compass), `beta` (−180-180° tilt), `gamma` (−90-90° tilt), `absolute` (boolean).\n\n### External Devices: `viji.devices`\n\n`DeviceState[]`: connected external devices. Each entry: `id` (string), `name` (string), `motion` (DeviceMotionData|null), `orientation` (DeviceOrientationData|null), `video` (VideoAPI|null, same shape as `viji.video` but without CV), `audio` (AudioStreamAPI|null, lightweight subset of `viji.audio`: `isConnected`, `volume.{current,peak,smoothed}`, `bands.{low...high}` + each `*Smoothed`, `spectral.{brightness,flatness}`, `getFrequencyData()`, `getWaveform()`. **No** beat / BPM / triggers / events).\n\n### Streams: `viji.videoStreams` and `viji.audioStreams`\n\n`viji.videoStreams: VideoAPI[]` and `viji.audioStreams: AudioStreamAPI[]`: additional video/audio sources provided by the host application (used internally by Viji's compositor for scene mixing). May be empty. Audio streams use the AudioStreamAPI shape (no beat / BPM / triggers / events).\n\n## P5-SPECIFIC GOTCHAS\n\nThese behaviors are different from running P5 in a browser tab:\n\n- **Fonts:** `p5.textFont()` only with CSS generic names (`'monospace'`, `'serif'`, `'sans-serif'`). `loadFont()` is NOT available.\n- **`p5.createGraphics(w, h)`** works (creates an internal OffscreenCanvas). `createGraphics(w, h, p5.WEBGL)` is NOT supported.\n- **`p5.pixelDensity()`** defaults to 1 in the worker. `p5.loadPixels()` and `p5.pixels[]` work in 2D scenes.\n- **`p5.drawingContext`** is a 2D context only in 2D scenes. In WEBGL scenes (`// @renderer p5 webgl`) it is a WebGL context: never use Canvas-2D-only APIs on it; use P5 3D drawing or `p5.image()` for textures and video.\n- **`viji.useContext()`** is NOT available in P5: the canvas and 2D context are managed by P5 internally.\n- **`p5.tint()` and `p5.blendMode()`** work normally.\n\n## TEMPLATE\n\n```javascript\n// @renderer p5\n\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\n\nlet angle = 0;\n\nfunction setup(viji, p5) {\n p5.colorMode(p5.HSB, 360, 100, 100);\n}\n\nfunction render(viji, p5) {\n angle += speed.value * viji.deltaTime;\n p5.background(0, 0, 10);\n const x = viji.width / 2 + p5.cos(angle) * viji.width * 0.3;\n const y = viji.height / 2 + p5.sin(angle) * viji.height * 0.3;\n p5.noStroke();\n p5.fill(angle * 30 % 360, 80, 100);\n p5.circle(x, y, viji.width * 0.05);\n}\n```\n\nNow convert the P5.js sketch I provide.\n\nIf the sketch is incomplete or uses features that are incompatible with the Viji worker environment, ask one or two clarifying questions first (see YOUR BEHAVIOR above). Otherwise, proceed.\n\nWhen you produce the conversion:\n- Apply every rule and mapping above. The **API MAPPING** table covers the most common direct ports; the **COMPLETE VIJI API REFERENCE** above lists the full Viji surface for any feature the source sketch reaches for that the mapping table does not list (audio analysis, video frames, CV data, touch, device sensors). For any P5 call you are unsure about, consult the p5.js v1.x reference linked in **REFERENCE**. The canonical companion generation prompt is `p5-prompt` (in the `docs-api.js` bundle).\n- Output the Viji-P5 scene code in a single fenced code block.\n- After the code block, write a short summary of the key changes and flag anything you had to drop or simplify.\n- Invite the artist to ask for refinements.\n````\n\n## Usage\n\n1. Copy the entire prompt block above.\n2. Paste it into your AI assistant.\n3. After the prompt, paste the P5.js sketch you want to convert.\n4. The AI will return a Viji-compatible scene.\n\nFor a detailed human-readable guide, see [Converting P5 Sketches](/p5/converting-sketches#step-by-step).\n\n## Related\n\n- [Converting P5 Sketches](/p5/converting-sketches#step-by-step): step-by-step manual conversion guide\n- [Prompt: P5 Scenes](/ai-prompts/p5-prompt): AI prompt for creating new P5 scenes from scratch\n- [P5 Quick Start](/p5/quickstart): your first Viji-P5 scene"
|
|
1491
1491
|
}
|
|
1492
1492
|
]
|
|
1493
1493
|
},
|
|
@@ -1543,7 +1543,7 @@ export const docsApi = {
|
|
|
1543
1543
|
"content": [
|
|
1544
1544
|
{
|
|
1545
1545
|
"type": "text",
|
|
1546
|
-
"markdown": "# Convert: Three.js to Viji\n\nCopy the prompt below and paste it into your AI assistant along with the Three.js code you want to convert. The prompt contains all the rules the AI needs to produce a correct Viji native scene with Three.js.\n\n## The Prompt\n\n````\nYou are converting a standalone Three.js application into a Viji native scene.\nViji scenes run inside an OffscreenCanvas Web Worker. Apply every rule below exactly.\n\n## YOUR BEHAVIOR\n\n1. **Clarify when needed.** If the source code is incomplete (missing init or render loop), uses a framework on top of Three.js (React Three Fiber, Drei, Theatre.js), or relies on DOM elements outside the canvas, ask the artist for the missing pieces or for permission to drop the framework wrapper before generating code. If the code is plain Three.js and self-contained, skip clarification and proceed.\n2. **Convert.** Produce a complete, copy-pasteable Viji native scene that follows every rule in this prompt. Preserve the artist's visual intent, scene graph, and material setup; replace only the Viji-incompatible parts (window/document access, `requestAnimationFrame`, `THREE.Clock`, DOM event listeners).\n3. **Explain.** After the code block, give a short summary of the key changes you made (e.g., \"wrapped scene init in top-level code, moved per-frame logic into `render(viji)`, replaced `clock.getDelta()` with `viji.deltaTime`, replaced mouse listeners with `viji.pointer`\"). Flag any features you had to drop or simplify (e.g., `OrbitControls`, postprocessing that depends on DOM events).\n4. **Iterate.** Invite the artist to ask for refinements (\"add a slider for camera distance\", \"make the cube react to audio\", \"swap the material for a wireframe\").\n\n## REFERENCE (source of truth)\n\nThe resources below are AUTHORITATIVE. The rules and tables in this prompt focus on the most common Three.js → Viji mappings, but they do NOT cover the full Viji API surface. If anything ever conflicts, the linked files win.\n\n**If you have web/file access:**\n- REQUIRED before converting code: fetch and skim the Tier-1 resource. Use it to verify exact Viji API names, parameter types, and any Viji feature the source code may reach for (audio, video, CV, sensors, parameters).\n- ON DEMAND: fetch from the Tier-2 resource when the source uses a Viji-side feature this prompt does not map (advanced CV data structures, device sensors, full Viji examples) or when you need authoritative TypeScript signatures.\n\n**If you do NOT have web/file access:**\n- Use only the API surface explicitly named in this prompt and the standard Three.js API for direct ports.\n- Never invent Viji property or method names from memory.\n- If the source uses something not covered here, say so and ask the artist how they want it handled; do NOT fabricate a Viji equivalent.\n\n**Tier 1 (always consult when accessible):**\n- TypeScript API types (Viji JS surface, all renderers): https://unpkg.com/@viji-dev/core/dist/artist-global.d.ts\n\n**Tier 2 (consult when needed):**\n- Complete Viji docs (every page + every live example): https://unpkg.com/@viji-dev/core/dist/docs-api.js\n- Companion prompt for any Viji feature this conversion prompt does not cover (search for \"native-prompt\" in the docs above).\n\n**Last-resort lookup:** https://www.npmjs.com/package/@viji-dev/core\n\n## RULES\n\n1. ALWAYS import Three.js dynamically at the top level using `await import()`:\n ```javascript\n const THREE = await import('https://esm.sh/three@0.160.0');\n ```\n NEVER use `<script>` tags, `require()`, or static `import` statements.\n ALWAYS pin the version number in the URL.\n\n2. ALWAYS use `viji.canvas` as the renderer's canvas:\n ```javascript\n const renderer = new THREE.WebGLRenderer({ canvas: viji.canvas, antialias: true });\n renderer.setSize(viji.width, viji.height, false);\n ```\n ALWAYS pass `false` as the third argument to `setSize()`: this prevents Three.js from setting CSS styles, which would fail in the worker.\n\n3. NEVER use `requestAnimationFrame()`. Viji controls the render loop. Write all per-frame logic inside `function render(viji) { ... }` and call `renderer.render(scene, camera)` at the end.\n\n4. ALWAYS handle resize by checking `viji.width` / `viji.height` in `render()`:\n ```javascript\n let prevWidth = viji.width;\n let prevHeight = viji.height;\n\n function render(viji) {\n if (viji.width !== prevWidth || viji.height !== prevHeight) {\n renderer.setSize(viji.width, viji.height, false);\n camera.aspect = viji.width / viji.height;\n camera.updateProjectionMatrix();\n prevWidth = viji.width;\n prevHeight = viji.height;\n }\n renderer.render(scene, camera);\n }\n ```\n\n5. NEVER access `window`, `document`, `Image()`, or `localStorage`. `fetch()` IS available.\n\n6. NEVER use `window.innerWidth` / `window.innerHeight`. Use `viji.width` / `viji.height`.\n\n7. Replace hardcoded values with Viji parameters declared at the top level:\n ```javascript\n const speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\n const color = viji.color('#049ef4', { label: 'Color' });\n ```\n NEVER declare parameters inside `render()`.\n ALWAYS read via `.value`: `speed.value`, `color.value`. Color parameters also expose `.rgb` (`{ r, g, b }` in 0..255) and `.hsb` (`{ h, s, b }`, h in 0..360, s/b in 0..100). Three.js color objects can take `.rgb` directly, e.g. `material.color.setRGB(c.rgb.r/255, c.rgb.g/255, c.rgb.b/255)`, or just keep using `material.color.set(c.value)`.\n\n8. ALWAYS use `viji.deltaTime` for animation timing:\n ```javascript\n cube.rotation.y += speed.value * viji.deltaTime;\n ```\n NEVER use `clock.getDelta()` or `Date.now()`. Remove any `THREE.Clock` usage.\n NEVER multiply `viji.time` by a parameter (`viji.time * speed.value`): it causes animation jumps when the parameter changes. Same for nested multiplications: never multiply an accumulator by another parameter; give each speed its own accumulator.\n\n9. Replace mouse/keyboard event listeners with Viji APIs:\n - `event.clientX` → `viji.pointer.x` (works for both mouse and touch) or `viji.mouse.x`\n - `event.clientY` → `viji.pointer.y` or `viji.mouse.y`\n - Mouse buttons → `viji.mouse.leftButton`, `viji.mouse.rightButton`\n - Key presses → `viji.keyboard.isPressed('keyName')`\n\n10. `OrbitControls` and other controls that depend on DOM events will NOT work in the worker.\n For camera interaction, read `viji.pointer` (handles both mouse and touch) and update the camera manually.\n\n11. For Three.js addons, import from the examples directory:\n ```javascript\n const { GLTFLoader } = await import('https://esm.sh/three@0.160.0/examples/jsm/loaders/GLTFLoader.js');\n const { EffectComposer } = await import('https://esm.sh/three@0.160.0/examples/jsm/postprocessing/EffectComposer.js');\n ```\n ALWAYS use the same Three.js version for addons as for the main library.\n\n12. For textures from file inputs, use Viji's image parameters:\n ```javascript\n const photo = viji.image(null, { label: 'Texture' });\n // In render():\n if (photo.value && !texture) {\n texture = new THREE.CanvasTexture(photo.value);\n material.map = texture;\n material.needsUpdate = true;\n }\n ```\n\n13. For video textures, use `viji.video`:\n ```javascript\n if (viji.video.isConnected && viji.video.currentFrame) {\n if (!videoTexture) {\n videoTexture = new THREE.CanvasTexture(viji.video.currentFrame);\n material.map = videoTexture;\n }\n videoTexture.needsUpdate = true;\n }\n ```\n\n14. NEVER allocate new objects inside `render()`. Pre-create vectors, colors, and materials at the top level.\n\n15. ALWAYS set `category` on parameters that depend on an external input: `category: 'audio'` for audio, `category: 'video'` for video/CV, `category: 'interaction'` for mouse/keyboard/touch. This lets the host hide irrelevant controls when the input is inactive.\n\n16. Remove any `window.addEventListener('resize', ...)`: resize is handled in `render()` (see rule 4).\n\n17. Remove any CSS, HTML, or DOM manipulation code. Viji scenes produce only canvas output.\n\n## COMPLETE VIJI API REFERENCE\n\nThe `viji` object is identical across all renderers (same object, same types). The mapping table above covers the most common Three.js → Viji ports; the reference below is the full surface for any Viji-side feature the source code reaches for (audio analysis, video frames, CV data, touch, device sensors).\n\n### Canvas & Context\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `viji.canvas` | `OffscreenCanvas` | The canvas element (pass to Three.js renderer) |\n| `viji.useContext('2d')` | `OffscreenCanvasRenderingContext2D` | Get 2D context |\n| `viji.useContext('webgl')` | `WebGLRenderingContext` | Get WebGL 1 context |\n| `viji.useContext('webgl2')` | `WebGL2RenderingContext` | Get WebGL 2 context |\n| `viji.ctx` | `OffscreenCanvasRenderingContext2D` | Shortcut (after useContext('2d')) |\n| `viji.gl` | `WebGLRenderingContext` | Shortcut (after useContext('webgl')) |\n| `viji.width` | `number` | Current canvas width in pixels |\n| `viji.height` | `number` | Current canvas height in pixels |\n\nWhen using Three.js, do NOT call `viji.useContext()`: pass `viji.canvas` to the `THREE.WebGLRenderer({ canvas: viji.canvas })` and Three.js manages its own GL context.\n\n### Timing\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `viji.time` | `number` | Seconds since scene start |\n| `viji.deltaTime` | `number` | Seconds since last frame |\n| `viji.frameCount` | `number` | Total frames rendered |\n| `viji.fps` | `number` | Target frame rate (based on host frame-rate mode) |\n\n### Parameters\n\nDeclare at top level. Read `.value` inside `render()`. All support `{ label, description?, group?, category? }`.\nCategory values: `'audio'`, `'video'`, `'interaction'`, `'general'`.\n\n```javascript\nviji.slider(default, { min?, max?, step?, label, group?, category? }) // { value: number }\nviji.color(default, { label, group?, category? }) // { value: '#rrggbb', rgb: { r, g, b } in 0..255, hsb: { h: 0..360, s/b: 0..100 } }\nviji.toggle(default, { label, group?, category? }) // { value: boolean }\nviji.select(default, { options: [...], label, group?, category? }) // { value: string|number }\nviji.number(default, { min?, max?, step?, label, group?, category? }) // { value: number }\nviji.text(default, { label, group?, category?, maxLength? }) // { value: string }\nviji.image(null, { label, group?, category? }) // { value: ImageBitmap|null }\nviji.button({ label, description?, group?, category? }) // { value: boolean } (true one frame)\nviji.coordinate(default, { step?, label, group?, category? }) // { value: { x, y } } (both -1 to 1)\n```\n\n### Audio: `viji.audio`\n\nALWAYS check `viji.audio.isConnected` first.\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isConnected` | `boolean` | Whether audio source is active |\n| `volume.current` | `number` | RMS volume 0-1 |\n| `volume.peak` | `number` | Peak amplitude 0-1 |\n| `volume.smoothed` | `number` | Smoothed volume (200ms decay) |\n| `bands.low` | `number` | 20-120 Hz energy 0-1 |\n| `bands.lowMid` | `number` | 120-400 Hz energy 0-1 |\n| `bands.mid` | `number` | 400-1600 Hz energy 0-1 |\n| `bands.highMid` | `number` | 1600-6000 Hz energy 0-1 |\n| `bands.high` | `number` | 6000-16000 Hz energy 0-1 |\n| `bands.lowSmoothed` … `bands.highSmoothed` | `number` | Smoothed variants of each band |\n| `beat.kick` | `number` | Kick energy curve 0-1, 300ms decay; peaks on each detected kick |\n| `beat.snare` | `number` | Snare energy curve 0-1, 300ms decay |\n| `beat.hat` | `number` | Hi-hat energy curve 0-1, 300ms decay |\n| `beat.any` | `number` | Any-beat energy curve 0-1, 300ms decay |\n| `beat.kickSmoothed` … `beat.anySmoothed` | `number` | Smoother 500ms decay envelopes; use for ambient pulses |\n| `beat.triggers.kick` | `boolean` | True for exactly one frame on a kick, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.snare` | `boolean` | True for exactly one frame on a snare, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.hat` | `boolean` | True for exactly one frame on a hi-hat, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.any` | `boolean` | True for exactly one frame on any beat, then auto-resets |\n| `beat.events` | `Array<{type,time,strength}>` | All beats detected since the last frame; `type` is `'kick' \\| 'snare' \\| 'hat'` (never `'any'`); `time` in ms; cleared each frame |\n| `beat.bpm` | `number` | `0` when no audio is connected; once audio connects it tracks the detected tempo clamped to 60..240, with `120` as a fallback before lock-on |\n| `beat.confidence` | `number` | BPM tracking confidence 0-1 |\n| `beat.isLocked` | `boolean` | True on stable tempo lock |\n| `spectral.brightness` | `number` | Spectral centroid 0-1 |\n| `spectral.flatness` | `number` | Spectral flatness 0-1 |\n| `getFrequencyData()` | `Uint8Array` | Raw FFT bins (0-255) |\n| `getWaveform()` | `Float32Array` | Time-domain waveform (−1 to 1) |\n\n### Video: `viji.video`\n\nALWAYS check `viji.video.isConnected` first. Check `currentFrame` before drawing.\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isConnected` | `boolean` | Whether video source is active |\n| `currentFrame` | `OffscreenCanvas\\|ImageBitmap\\|null` | Current video frame |\n| `frameWidth` | `number` | Frame width in pixels |\n| `frameHeight` | `number` | Frame height in pixels |\n| `frameRate` | `number` | Video frame rate |\n| `getFrameData()` | `ImageData\\|null` | Pixel data for CPU access |\n| `cv` | `VideoCVAPI` | Computer-vision surface (see below) |\n\nCV-paired outputs (`analysedFrame`, `getAnalysedFrameData()`) and detection results live on `viji.video.cv`, not on `viji.video` directly. For Three.js video textures, see rule 13.\n\n### Computer Vision: `viji.video.cv` & `viji.video.cv.faces / hands / pose / segmentation`\n\nEnable features via toggle parameters (NEVER enable by default):\n\n```javascript\nawait viji.video.cv.enableFaceDetection(true/false);\nawait viji.video.cv.enableFaceMesh(true/false); // implies face detection\nawait viji.video.cv.enableEmotionDetection(true/false); // implies face mesh\nawait viji.video.cv.enableHandTracking(true/false);\nawait viji.video.cv.enablePoseDetection(true/false);\nawait viji.video.cv.enableBodySegmentation(true/false);\nviji.video.cv.getActiveFeatures(); // CVFeature[]\nviji.video.cv.isProcessing(); // boolean\n```\n\nCV-paired outputs (also on `viji.video.cv`):\n- `analysedFrame: OffscreenCanvas | null`: the exact frame that produced the current CV results. `null` until the first inference lands. Common fallback pattern: `viji.video.cv.analysedFrame ?? viji.video.currentFrame`.\n- `getAnalysedFrameData(): ImageData | null`: pixel data of `analysedFrame`.\n\n**`viji.video.cv.faces: FaceData[]`**\nEach face: `id` (number), `bounds` ({x,y,width,height}), `center` ({x,y}), `confidence` (0-1), `landmarks` ({x,y,z?}[]), `expressions` ({neutral,happy,sad,angry,surprised,disgusted,fearful} all 0-1), `headPose` ({pitch,yaw,roll} in degrees), `blendshapes` (52 ARKit coefficients 0-1: browDownLeft, browDownRight, browInnerUp, browOuterUpLeft, browOuterUpRight, cheekPuff, cheekSquintLeft, cheekSquintRight, eyeBlinkLeft, eyeBlinkRight, eyeLookDownLeft, eyeLookDownRight, eyeLookInLeft, eyeLookInRight, eyeLookOutLeft, eyeLookOutRight, eyeLookUpLeft, eyeLookUpRight, eyeSquintLeft, eyeSquintRight, eyeWideLeft, eyeWideRight, jawForward, jawLeft, jawOpen, jawRight, mouthClose, mouthDimpleLeft, mouthDimpleRight, mouthFrownLeft, mouthFrownRight, mouthFunnel, mouthLeft, mouthLowerDownLeft, mouthLowerDownRight, mouthPressLeft, mouthPressRight, mouthPucker, mouthRight, mouthRollLower, mouthRollUpper, mouthShrugLower, mouthShrugUpper, mouthSmileLeft, mouthSmileRight, mouthStretchLeft, mouthStretchRight, mouthUpperUpLeft, mouthUpperUpRight, noseSneerLeft, noseSneerRight, tongueOut).\n\n**`viji.video.cv.hands: HandData[]`**\nEach hand: `id` (number), `handedness` ('left'|'right'), `confidence` (0-1), `bounds` ({x,y,width,height}), `landmarks` ({x,y,z}[], 21 points), `palm` ({x,y,z}), `gestures` ({fist,openPalm,peace,thumbsUp,thumbsDown,pointing,iLoveYou} all 0-1).\n\n**`viji.video.cv.pose: PoseData | null`**\n`confidence` (0-1), `landmarks` ({x,y,z,visibility}[], 33 points), plus body-part arrays: `face` ({x,y}[]), `torso`, `leftArm`, `rightArm`, `leftLeg`, `rightLeg`.\n\n**`viji.video.cv.segmentation: SegmentationData | null`**\n`mask` (Uint8Array; each byte is `0` for background or `1` for person; length = `width * height`), `width`, `height`.\n\n### Input: Pointer (unified mouse/touch): `viji.pointer`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `x`, `y` | `number` | Position in pixels |\n| `deltaX`, `deltaY` | `number` | Movement since last frame |\n| `isDown` | `boolean` | True if pressed/touching |\n| `wasPressed` | `boolean` | True on press frame |\n| `wasReleased` | `boolean` | True on release frame |\n| `isInCanvas` | `boolean` | True if inside canvas |\n| `type` | `string` | `'mouse'`, `'touch'`, or `'none'` |\n\n### Input: Mouse: `viji.mouse`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `x`, `y` | `number` | Position in pixels |\n| `isInCanvas` | `boolean` | Inside canvas bounds |\n| `isPressed` | `boolean` | Any button pressed |\n| `leftButton`, `rightButton`, `middleButton` | `boolean` | Specific buttons |\n| `deltaX`, `deltaY` | `number` | Movement delta |\n| `wheelDelta` | `number` | Scroll wheel delta |\n| `wheelX`, `wheelY` | `number` | Horizontal/vertical scroll |\n| `wasPressed`, `wasReleased`, `wasMoved` | `boolean` | Frame-edge events |\n\n### Input: Keyboard: `viji.keyboard`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isPressed(key)` | `boolean` | True while key is held |\n| `wasPressed(key)` | `boolean` | True on key-down frame |\n| `wasReleased(key)` | `boolean` | True on key-up frame |\n| `activeKeys` | `Set<string>` | Currently held keys |\n| `pressedThisFrame` | `Set<string>` | Keys pressed this frame |\n| `releasedThisFrame` | `Set<string>` | Keys released this frame |\n| `lastKeyPressed` | `string` | Most recent key-down |\n| `lastKeyReleased` | `string` | Most recent key-up |\n| `shift`, `ctrl`, `alt`, `meta` | `boolean` | Modifier states |\n\n### Input: Touch: `viji.touches`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `count` | `number` | Active touch count |\n| `points` | `TouchPoint[]` | All active touches |\n| `started` | `TouchPoint[]` | Touches started this frame |\n| `moved` | `TouchPoint[]` | Touches moved this frame |\n| `ended` | `TouchPoint[]` | Touches ended this frame |\n| `primary` | `TouchPoint\\|null` | First active touch |\n\n**TouchPoint:** `id`, `x`, `y`, `pressure`, `radius`, `radiusX`, `radiusY`, `rotationAngle`, `force`, `isInCanvas`, `deltaX`, `deltaY`, `velocity` ({x,y}), `isNew`, `isActive`, `isEnding`.\n\n### Device Sensors: `viji.device`\n\n`viji.device.motion` (DeviceMotionData|null): `acceleration` ({x,y,z} m/s²), `accelerationIncludingGravity`, `rotationRate` ({alpha,beta,gamma} deg/s), `interval` (ms).\n`viji.device.orientation` (DeviceOrientationData|null): `alpha` (0-360° compass), `beta` (−180-180° tilt), `gamma` (−90-90° tilt), `absolute` (boolean).\n\n### External Devices: `viji.devices`\n\n`DeviceState[]`: connected external devices. Each entry: `id` (string), `name` (string), `motion` (DeviceMotionData|null), `orientation` (DeviceOrientationData|null), `video` (VideoAPI|null, same shape as `viji.video` but without CV), `audio` (AudioStreamAPI|null, lightweight subset of `viji.audio`: `isConnected`, `volume.{current,peak,smoothed}`, `bands.{low...high}` + each `*Smoothed`, `spectral.{brightness,flatness}`, `getFrequencyData()`, `getWaveform()`. **No** beat / BPM / triggers / events).\n\n### Streams: `viji.videoStreams` and `viji.audioStreams`\n\n`viji.videoStreams: VideoAPI[]` and `viji.audioStreams: AudioStreamAPI[]`: additional video/audio sources provided by the host application (used internally by Viji's compositor for scene mixing). May be empty. Audio streams use the AudioStreamAPI shape (no beat / BPM / triggers / events).\n\n## TEMPLATE\n\n```javascript\nconst THREE = await import('https://esm.sh/three@0.160.0');\n\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nconst color = viji.color('#049ef4', { label: 'Color' });\n\nconst scene = new THREE.Scene();\nconst camera = new THREE.PerspectiveCamera(60, viji.width / viji.height, 0.1, 100);\ncamera.position.set(0, 1, 3);\ncamera.lookAt(0, 0, 0);\n\nconst renderer = new THREE.WebGLRenderer({ canvas: viji.canvas, antialias: true });\nrenderer.setSize(viji.width, viji.height, false);\n\nconst geometry = new THREE.BoxGeometry();\nconst material = new THREE.MeshStandardMaterial({ color: color.value });\nconst mesh = new THREE.Mesh(geometry, material);\nscene.add(mesh);\n\nscene.add(new THREE.DirectionalLight(0xffffff, 1.5));\nscene.add(new THREE.AmbientLight(0x404040));\n\nlet prevWidth = viji.width;\nlet prevHeight = viji.height;\n\nfunction render(viji) {\n mesh.rotation.y += speed.value * viji.deltaTime;\n material.color.set(color.value);\n\n if (viji.width !== prevWidth || viji.height !== prevHeight) {\n renderer.setSize(viji.width, viji.height, false);\n camera.aspect = viji.width / viji.height;\n camera.updateProjectionMatrix();\n prevWidth = viji.width;\n prevHeight = viji.height;\n }\n\n renderer.render(scene, camera);\n}\n```\n\nNow convert the Three.js code I provide.\n\nIf the source is incomplete or uses a framework that needs to be unwrapped, ask one or two clarifying questions first (see YOUR BEHAVIOR above). Otherwise, proceed.\n\nWhen you produce the conversion:\n- Apply every rule above. The conversion-specific rules (1-17) cover the Three.js → Viji mappings most artists need; the **COMPLETE VIJI API REFERENCE** above lists the full Viji surface for any Viji-side feature the source code reaches for (audio, video, CV, touch, device sensors, parameters beyond the basics). For details not in this prompt, consult the **REFERENCE** links: the canonical companion generation prompt is `native-prompt` (in the `docs-api.js` bundle).\n- Output the Viji scene code in a single fenced code block.\n- After the code block, write a short summary of the key changes and flag anything you had to drop or simplify (controls, postprocessing, framework wrappers).\n- Invite the artist to ask for refinements.\n````\n\n## Usage\n\n1. Copy the entire prompt block above.\n2. Paste it into your AI assistant.\n3. After the prompt, paste the Three.js code you want to convert.\n4. The AI will return a Viji-compatible native scene.\n\n> [!NOTE]\n> This prompt handles standard Three.js scenes. If the original code uses a framework (React Three Fiber, Drei, etc.), you may need to manually extract the Three.js scene setup first.\n\n## Related\n\n- [External Libraries](/native/external-libraries): detailed guide for using Three.js and other libraries in Viji\n- [Prompt: Native Scenes](/ai-prompts/native-prompt): AI prompt for creating new native scenes from scratch\n- [Native Quick Start](/native/quickstart): your first Viji native scene"
|
|
1546
|
+
"markdown": "# Convert: Three.js to Viji\n\nCopy the prompt below and paste it into your AI assistant along with the Three.js code you want to convert. The prompt contains all the rules the AI needs to produce a correct Viji native scene with Three.js.\n\n## The Prompt\n\n````\nYou are converting a standalone Three.js application into a Viji native scene.\nViji scenes run inside an OffscreenCanvas Web Worker. Apply every rule below exactly.\n\n## YOUR BEHAVIOR\n\n1. **Clarify when needed.** If the source code is incomplete (missing init or render loop), uses a framework on top of Three.js (React Three Fiber, Drei, Theatre.js), or relies on DOM elements outside the canvas, ask the artist for the missing pieces or for permission to drop the framework wrapper before generating code. If the code is plain Three.js and self-contained, skip clarification and proceed.\n2. **Convert.** Produce a complete, copy-pasteable Viji native scene that follows every rule in this prompt. Preserve the artist's visual intent, scene graph, and material setup; replace only the Viji-incompatible parts (window/document access, `requestAnimationFrame`, `THREE.Clock`, DOM event listeners).\n3. **Explain.** After the code block, give a short summary of the key changes you made (e.g., \"wrapped scene init in top-level code, moved per-frame logic into `render(viji)`, replaced `clock.getDelta()` with `viji.deltaTime`, replaced mouse listeners with `viji.pointer`\"). Flag any features you had to drop or simplify (e.g., `OrbitControls`, postprocessing that depends on DOM events).\n4. **Iterate.** Invite the artist to ask for refinements (\"add a slider for camera distance\", \"make the cube react to audio\", \"swap the material for a wireframe\").\n\n## REFERENCE (source of truth)\n\nThe resources below are AUTHORITATIVE. The rules and tables in this prompt focus on the most common Three.js → Viji mappings, but they do NOT cover the full Viji API surface. If anything ever conflicts, the linked files win.\n\n**If you have web/file access:**\n- REQUIRED before converting code: fetch and skim the Tier-1 resource. Use it to verify exact Viji API names, parameter types, and any Viji feature the source code may reach for (audio, video, CV, sensors, parameters).\n- ON DEMAND: fetch from the Tier-2 resource when the source uses a Viji-side feature this prompt does not map (advanced CV data structures, device sensors, full Viji examples) or when you need authoritative TypeScript signatures.\n\n**If you do NOT have web/file access:**\n- Use only the API surface explicitly named in this prompt and the standard Three.js API for direct ports.\n- Never invent Viji property or method names from memory.\n- If the source uses something not covered here, say so and ask the artist how they want it handled; do NOT fabricate a Viji equivalent.\n\n**Tier 1 (always consult when accessible):**\n- TypeScript API types (Viji JS surface, all renderers): https://unpkg.com/@viji-dev/core/dist/artist-global.d.ts\n\n**Tier 2 (consult when needed):**\n- Complete Viji docs (every page + every live example): https://unpkg.com/@viji-dev/core/dist/docs-api.js\n- Companion prompt for any Viji feature this conversion prompt does not cover (search for \"native-prompt\" in the docs above).\n\n**Last-resort lookup:** https://www.npmjs.com/package/@viji-dev/core\n\n## RULES\n\n1. ALWAYS import Three.js dynamically at the top level using `await import()`:\n ```javascript\n const THREE = await import('https://esm.sh/three@0.160.0');\n ```\n NEVER use `<script>` tags, `require()`, or static `import` statements.\n ALWAYS pin the version number in the URL.\n\n2. ALWAYS use `viji.canvas` as the renderer's canvas:\n ```javascript\n const renderer = new THREE.WebGLRenderer({ canvas: viji.canvas, antialias: true });\n renderer.setSize(viji.width, viji.height, false);\n ```\n ALWAYS pass `false` as the third argument to `setSize()`: this prevents Three.js from setting CSS styles, which would fail in the worker.\n\n3. NEVER use `requestAnimationFrame()`. Viji controls the render loop. Write all per-frame logic inside `function render(viji) { ... }` and call `renderer.render(scene, camera)` at the end.\n\n4. ALWAYS handle resize by checking `viji.width` / `viji.height` in `render()`:\n ```javascript\n let prevWidth = viji.width;\n let prevHeight = viji.height;\n\n function render(viji) {\n if (viji.width !== prevWidth || viji.height !== prevHeight) {\n renderer.setSize(viji.width, viji.height, false);\n camera.aspect = viji.width / viji.height;\n camera.updateProjectionMatrix();\n prevWidth = viji.width;\n prevHeight = viji.height;\n }\n renderer.render(scene, camera);\n }\n ```\n\n5. NEVER access `window`, `document`, `Image()`, or `localStorage`. `fetch()` IS available.\n\n6. NEVER use `window.innerWidth` / `window.innerHeight`. Use `viji.width` / `viji.height`.\n\n7. Replace hardcoded values with Viji parameters declared at the top level:\n ```javascript\n const speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\n const color = viji.color('#049ef4', { label: 'Color' });\n ```\n NEVER declare parameters inside `render()`.\n ALWAYS read via `.value`: `speed.value`, `color.value`. Color parameters also expose `.rgb` (`{ r, g, b }` in 0..255) and `.hsb` (`{ h, s, b }`, h in 0..360, s/b in 0..100). Three.js color objects can take `.rgb` directly, e.g. `material.color.setRGB(c.rgb.r/255, c.rgb.g/255, c.rgb.b/255)`, or just keep using `material.color.set(c.value)`.\n\n8. ALWAYS use `viji.deltaTime` for animation timing:\n ```javascript\n cube.rotation.y += speed.value * viji.deltaTime;\n ```\n NEVER use `clock.getDelta()` or `Date.now()`. Remove any `THREE.Clock` usage.\n NEVER multiply `viji.time` by a parameter (`viji.time * speed.value`): it causes animation jumps when the parameter changes. Same for nested multiplications: never multiply an accumulator by another parameter; give each speed its own accumulator.\n\n9. Replace mouse/keyboard event listeners with Viji APIs:\n - `event.clientX` → `viji.pointer.x` (works for both mouse and touch) or `viji.mouse.x`\n - `event.clientY` → `viji.pointer.y` or `viji.mouse.y`\n - Mouse buttons → `viji.mouse.leftButton`, `viji.mouse.rightButton`\n - Key presses → `viji.keyboard.isPressed('keyName')`\n\n10. `OrbitControls` and other controls that depend on DOM events will NOT work in the worker.\n For camera interaction, read `viji.pointer` (handles both mouse and touch) and update the camera manually.\n\n11. For Three.js addons, import from the examples directory:\n ```javascript\n const { GLTFLoader } = await import('https://esm.sh/three@0.160.0/examples/jsm/loaders/GLTFLoader.js');\n const { EffectComposer } = await import('https://esm.sh/three@0.160.0/examples/jsm/postprocessing/EffectComposer.js');\n ```\n ALWAYS use the same Three.js version for addons as for the main library.\n\n12. For textures from file inputs, use Viji's image parameters:\n ```javascript\n const photo = viji.image(null, { label: 'Texture' });\n // In render():\n if (photo.value && !texture) {\n texture = new THREE.CanvasTexture(photo.value);\n material.map = texture;\n material.needsUpdate = true;\n }\n ```\n\n13. For video textures, use `viji.video`:\n ```javascript\n if (viji.video.isConnected && viji.video.currentFrame) {\n if (!videoTexture) {\n videoTexture = new THREE.CanvasTexture(viji.video.currentFrame);\n material.map = videoTexture;\n }\n videoTexture.needsUpdate = true;\n }\n ```\n\n14. NEVER allocate new objects inside `render()`. Pre-create vectors, colors, and materials at the top level.\n\n15. ALWAYS set `category` on parameters that depend on an external input: `category: 'audio'` for audio, `category: 'video'` for video/CV, `category: 'interaction'` for mouse/keyboard/touch. This lets the host hide irrelevant controls when the input is inactive.\n\n16. Remove any `window.addEventListener('resize', ...)`: resize is handled in `render()` (see rule 4).\n\n17. Remove any CSS, HTML, or DOM manipulation code. Viji scenes produce only canvas output.\n\n## COMPLETE VIJI API REFERENCE\n\nThe `viji` object is identical across all renderers (same object, same types). The mapping table above covers the most common Three.js → Viji ports; the reference below is the full surface for any Viji-side feature the source code reaches for (audio analysis, video frames, CV data, touch, device sensors).\n\n### Canvas & Context\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `viji.canvas` | `OffscreenCanvas` | The canvas element (pass to Three.js renderer) |\n| `viji.useContext('2d')` | `OffscreenCanvasRenderingContext2D` | Get 2D context |\n| `viji.useContext('webgl')` | `WebGLRenderingContext` | Get WebGL 1 context |\n| `viji.useContext('webgl2')` | `WebGL2RenderingContext` | Get WebGL 2 context |\n| `viji.ctx` | `OffscreenCanvasRenderingContext2D` | Shortcut (after useContext('2d')) |\n| `viji.gl` | `WebGLRenderingContext` | Shortcut (after useContext('webgl')) |\n| `viji.width` | `number` | Current canvas width in pixels |\n| `viji.height` | `number` | Current canvas height in pixels |\n\nWhen using Three.js, do NOT call `viji.useContext()`: pass `viji.canvas` to the `THREE.WebGLRenderer({ canvas: viji.canvas })` and Three.js manages its own GL context.\n\n### Timing\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `viji.time` | `number` | Seconds since scene start |\n| `viji.deltaTime` | `number` | Seconds since last frame |\n| `viji.frameCount` | `number` | Total frames rendered |\n| `viji.fps` | `number` | Target frame rate (based on host frame-rate mode) |\n\n### Parameters\n\nDeclare at top level. Read `.value` inside `render()`. All support `{ label, description?, group?, category? }`.\nCategory values: `'audio'`, `'video'`, `'interaction'`, `'general'`.\n\n```javascript\nviji.slider(default, { min?, max?, step?, label, group?, category? }) // { value: number }\nviji.color(default, { label, group?, category? }) // { value: '#rrggbb', rgb: { r, g, b } in 0..255, hsb: { h: 0..360, s/b: 0..100 } }\nviji.toggle(default, { label, group?, category? }) // { value: boolean }\nviji.select(default, { options: [...], label, group?, category? }) // { value: string|number }\nviji.number(default, { min?, max?, step?, label, group?, category? }) // { value: number }\nviji.text(default, { label, group?, category?, maxLength? }) // { value: string }\nviji.image(null, { label, group?, category? }) // { value: ImageBitmap|null }\nviji.button({ label, description?, group?, category? }) // { value: boolean } (true one frame)\nviji.coordinate(default, { step?, label, group?, category? }) // { value: { x, y } } (both -1 to 1)\n```\n\n### Audio: `viji.audio`\n\nALWAYS check `viji.audio.isConnected` first.\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isConnected` | `boolean` | Whether audio source is active |\n| `volume.current` | `number` | RMS volume 0-1 |\n| `volume.peak` | `number` | Peak amplitude 0-1 |\n| `volume.smoothed` | `number` | Smoothed volume (200ms decay) |\n| `bands.low` | `number` | 20-120 Hz energy 0-1 |\n| `bands.lowMid` | `number` | 120-400 Hz energy 0-1 |\n| `bands.mid` | `number` | 400-1600 Hz energy 0-1 |\n| `bands.highMid` | `number` | 1600-6000 Hz energy 0-1 |\n| `bands.high` | `number` | 6000-16000 Hz energy 0-1 |\n| `bands.lowSmoothed` … `bands.highSmoothed` | `number` | Smoothed variants of each band |\n| `beat.kick` | `number` | Kick energy curve 0-1, 300ms decay; peaks on each detected kick |\n| `beat.snare` | `number` | Snare energy curve 0-1, 300ms decay |\n| `beat.hat` | `number` | Hi-hat energy curve 0-1, 300ms decay |\n| `beat.any` | `number` | Any-beat energy curve 0-1, 300ms decay |\n| `beat.kickSmoothed` … `beat.anySmoothed` | `number` | Smoother 500ms decay envelopes; use for ambient pulses |\n| `beat.triggers.kick` | `boolean` | True for exactly one frame on a kick, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.snare` | `boolean` | True for exactly one frame on a snare, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.hat` | `boolean` | True for exactly one frame on a hi-hat, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.any` | `boolean` | True for exactly one frame on any beat, then auto-resets |\n| `beat.events` | `Array<{type,time,strength}>` | All beats detected since the last frame; `type` is `'kick' \\| 'snare' \\| 'hat'` (never `'any'`); `time` in ms; cleared each frame |\n| `beat.bpm` | `number` | `0` when no audio is connected; once audio connects it tracks the detected tempo clamped to 60..240, with `120` as a fallback before lock-on |\n| `beat.confidence` | `number` | BPM tracking confidence 0-1 |\n| `beat.isLocked` | `boolean` | True on stable tempo lock |\n| `spectral.brightness` | `number` | Spectral centroid 0-1 |\n| `spectral.flatness` | `number` | Spectral flatness 0-1 |\n| `getFrequencyData()` | `Uint8Array` | Raw FFT bins (0-255) |\n| `getWaveform()` | `Float32Array` | Time-domain waveform (−1 to 1) |\n\n### Video: `viji.video`\n\nALWAYS check `viji.video.isConnected` first. Check `currentFrame` before drawing.\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isConnected` | `boolean` | Whether video source is active |\n| `currentFrame` | `OffscreenCanvas\\|ImageBitmap\\|null` | Current video frame |\n| `frameWidth` | `number` | Frame width in pixels |\n| `frameHeight` | `number` | Frame height in pixels |\n| `frameRate` | `number` | Video frame rate |\n| `getFrameData()` | `ImageData\\|null` | Pixel data for CPU access |\n| `cv` | `VideoCVAPI` | Computer-vision surface (see below) |\n\nCV-paired outputs (`analysedFrame`, `getAnalysedFrameData()`) and detection results live on `viji.video.cv`, not on `viji.video` directly. For Three.js video textures, see rule 13.\n\n### Computer Vision: `viji.video.cv` & `viji.video.cv.faces / hands / pose / segmentation`\n\nEnable features via toggle parameters (NEVER enable by default):\n\n```javascript\nawait viji.video.cv.enableFaceDetection(true/false);\nawait viji.video.cv.enableFaceMesh(true/false); // populates face.landmarks + face.headPose\nawait viji.video.cv.enableEmotionDetection(true/false); // populates face.blendshapes + face.expressions; loads landmarker\nawait viji.video.cv.enableHandTracking(true/false);\nawait viji.video.cv.enablePoseDetection(true/false);\nawait viji.video.cv.enableBodySegmentation(true/false);\nviji.video.cv.getActiveFeatures(); // CVFeature[]\nviji.video.cv.isProcessing(); // boolean\n```\n\nCV-paired outputs (also on `viji.video.cv`):\n- `analysedFrame: OffscreenCanvas | null`: the exact frame that produced the current CV results. `null` until the first inference lands. Common fallback pattern: `viji.video.cv.analysedFrame ?? viji.video.currentFrame`.\n- `getAnalysedFrameData(): ImageData | null`: pixel data of `analysedFrame`.\n\n**`viji.video.cv.faces: FaceData[]`**\nEach face: `id` (number), `bounds` ({x,y,width,height}), `center` ({x,y}), `confidence` (0-1), `landmarks` ({x,y,z?}[]), `expressions` ({neutral,happy,sad,angry,surprised,disgusted,fearful} all 0-1), `headPose` ({pitch,yaw,roll} in degrees), `blendshapes` (52 ARKit coefficients 0-1: browDownLeft, browDownRight, browInnerUp, browOuterUpLeft, browOuterUpRight, cheekPuff, cheekSquintLeft, cheekSquintRight, eyeBlinkLeft, eyeBlinkRight, eyeLookDownLeft, eyeLookDownRight, eyeLookInLeft, eyeLookInRight, eyeLookOutLeft, eyeLookOutRight, eyeLookUpLeft, eyeLookUpRight, eyeSquintLeft, eyeSquintRight, eyeWideLeft, eyeWideRight, jawForward, jawLeft, jawOpen, jawRight, mouthClose, mouthDimpleLeft, mouthDimpleRight, mouthFrownLeft, mouthFrownRight, mouthFunnel, mouthLeft, mouthLowerDownLeft, mouthLowerDownRight, mouthPressLeft, mouthPressRight, mouthPucker, mouthRight, mouthRollLower, mouthRollUpper, mouthShrugLower, mouthShrugUpper, mouthSmileLeft, mouthSmileRight, mouthStretchLeft, mouthStretchRight, mouthUpperUpLeft, mouthUpperUpRight, noseSneerLeft, noseSneerRight, tongueOut).\n\n**`viji.video.cv.hands: HandData[]`**\nEach hand: `id` (number), `handedness` ('left'|'right'), `confidence` (0-1), `bounds` ({x,y,width,height}), `landmarks` ({x,y,z}[], 21 points), `palm` ({x,y,z}), `gestures` ({fist,openPalm,peace,thumbsUp,thumbsDown,pointing,iLoveYou} all 0-1).\n\n**`viji.video.cv.pose: PoseData | null`**\n`confidence` (0-1), `landmarks` ({x,y,z,visibility}[], 33 points), plus body-part arrays: `face` ({x,y}[]), `torso`, `leftArm`, `rightArm`, `leftLeg`, `rightLeg`.\n\n**`viji.video.cv.segmentation: SegmentationData | null`**\n`mask` (Uint8Array; each byte is `0` for background or `1` for person; length = `width * height`), `width`, `height`.\n\n### Input: Pointer (unified mouse/touch): `viji.pointer`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `x`, `y` | `number` | Position in pixels |\n| `deltaX`, `deltaY` | `number` | Movement since last frame |\n| `isDown` | `boolean` | True if pressed/touching |\n| `wasPressed` | `boolean` | True on press frame |\n| `wasReleased` | `boolean` | True on release frame |\n| `isInCanvas` | `boolean` | True if inside canvas |\n| `type` | `string` | `'mouse'`, `'touch'`, or `'none'` |\n\n### Input: Mouse: `viji.mouse`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `x`, `y` | `number` | Position in pixels |\n| `isInCanvas` | `boolean` | Inside canvas bounds |\n| `isPressed` | `boolean` | Any button pressed |\n| `leftButton`, `rightButton`, `middleButton` | `boolean` | Specific buttons |\n| `deltaX`, `deltaY` | `number` | Movement delta |\n| `wheelDelta` | `number` | Scroll wheel delta |\n| `wheelX`, `wheelY` | `number` | Horizontal/vertical scroll |\n| `wasPressed`, `wasReleased`, `wasMoved` | `boolean` | Frame-edge events |\n\n### Input: Keyboard: `viji.keyboard`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isPressed(key)` | `boolean` | True while key is held |\n| `wasPressed(key)` | `boolean` | True on key-down frame |\n| `wasReleased(key)` | `boolean` | True on key-up frame |\n| `activeKeys` | `Set<string>` | Currently held keys |\n| `pressedThisFrame` | `Set<string>` | Keys pressed this frame |\n| `releasedThisFrame` | `Set<string>` | Keys released this frame |\n| `lastKeyPressed` | `string` | Most recent key-down |\n| `lastKeyReleased` | `string` | Most recent key-up |\n| `shift`, `ctrl`, `alt`, `meta` | `boolean` | Modifier states |\n\n### Input: Touch: `viji.touches`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `count` | `number` | Active touch count |\n| `points` | `TouchPoint[]` | All active touches |\n| `started` | `TouchPoint[]` | Touches started this frame |\n| `moved` | `TouchPoint[]` | Touches moved this frame |\n| `ended` | `TouchPoint[]` | Touches ended this frame |\n| `primary` | `TouchPoint\\|null` | First active touch |\n\n**TouchPoint:** `id`, `x`, `y`, `pressure`, `radius`, `radiusX`, `radiusY`, `rotationAngle`, `force`, `isInCanvas`, `deltaX`, `deltaY`, `velocity` ({x,y}), `isNew`, `isActive`, `isEnding`.\n\n### Device Sensors: `viji.device`\n\n`viji.device.motion` (DeviceMotionData|null): `acceleration` ({x,y,z} m/s²), `accelerationIncludingGravity`, `rotationRate` ({alpha,beta,gamma} deg/s), `interval` (ms).\n`viji.device.orientation` (DeviceOrientationData|null): `alpha` (0-360° compass), `beta` (−180-180° tilt), `gamma` (−90-90° tilt), `absolute` (boolean).\n\n### External Devices: `viji.devices`\n\n`DeviceState[]`: connected external devices. Each entry: `id` (string), `name` (string), `motion` (DeviceMotionData|null), `orientation` (DeviceOrientationData|null), `video` (VideoAPI|null, same shape as `viji.video` but without CV), `audio` (AudioStreamAPI|null, lightweight subset of `viji.audio`: `isConnected`, `volume.{current,peak,smoothed}`, `bands.{low...high}` + each `*Smoothed`, `spectral.{brightness,flatness}`, `getFrequencyData()`, `getWaveform()`. **No** beat / BPM / triggers / events).\n\n### Streams: `viji.videoStreams` and `viji.audioStreams`\n\n`viji.videoStreams: VideoAPI[]` and `viji.audioStreams: AudioStreamAPI[]`: additional video/audio sources provided by the host application (used internally by Viji's compositor for scene mixing). May be empty. Audio streams use the AudioStreamAPI shape (no beat / BPM / triggers / events).\n\n## TEMPLATE\n\n```javascript\nconst THREE = await import('https://esm.sh/three@0.160.0');\n\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nconst color = viji.color('#049ef4', { label: 'Color' });\n\nconst scene = new THREE.Scene();\nconst camera = new THREE.PerspectiveCamera(60, viji.width / viji.height, 0.1, 100);\ncamera.position.set(0, 1, 3);\ncamera.lookAt(0, 0, 0);\n\nconst renderer = new THREE.WebGLRenderer({ canvas: viji.canvas, antialias: true });\nrenderer.setSize(viji.width, viji.height, false);\n\nconst geometry = new THREE.BoxGeometry();\nconst material = new THREE.MeshStandardMaterial({ color: color.value });\nconst mesh = new THREE.Mesh(geometry, material);\nscene.add(mesh);\n\nscene.add(new THREE.DirectionalLight(0xffffff, 1.5));\nscene.add(new THREE.AmbientLight(0x404040));\n\nlet prevWidth = viji.width;\nlet prevHeight = viji.height;\n\nfunction render(viji) {\n mesh.rotation.y += speed.value * viji.deltaTime;\n material.color.set(color.value);\n\n if (viji.width !== prevWidth || viji.height !== prevHeight) {\n renderer.setSize(viji.width, viji.height, false);\n camera.aspect = viji.width / viji.height;\n camera.updateProjectionMatrix();\n prevWidth = viji.width;\n prevHeight = viji.height;\n }\n\n renderer.render(scene, camera);\n}\n```\n\nNow convert the Three.js code I provide.\n\nIf the source is incomplete or uses a framework that needs to be unwrapped, ask one or two clarifying questions first (see YOUR BEHAVIOR above). Otherwise, proceed.\n\nWhen you produce the conversion:\n- Apply every rule above. The conversion-specific rules (1-17) cover the Three.js → Viji mappings most artists need; the **COMPLETE VIJI API REFERENCE** above lists the full Viji surface for any Viji-side feature the source code reaches for (audio, video, CV, touch, device sensors, parameters beyond the basics). For details not in this prompt, consult the **REFERENCE** links: the canonical companion generation prompt is `native-prompt` (in the `docs-api.js` bundle).\n- Output the Viji scene code in a single fenced code block.\n- After the code block, write a short summary of the key changes and flag anything you had to drop or simplify (controls, postprocessing, framework wrappers).\n- Invite the artist to ask for refinements.\n````\n\n## Usage\n\n1. Copy the entire prompt block above.\n2. Paste it into your AI assistant.\n3. After the prompt, paste the Three.js code you want to convert.\n4. The AI will return a Viji-compatible native scene.\n\n> [!NOTE]\n> This prompt handles standard Three.js scenes. If the original code uses a framework (React Three Fiber, Drei, etc.), you may need to manually extract the Three.js scene setup first.\n\n## Related\n\n- [External Libraries](/native/external-libraries): detailed guide for using Three.js and other libraries in Viji\n- [Prompt: Native Scenes](/ai-prompts/native-prompt): AI prompt for creating new native scenes from scratch\n- [Native Quick Start](/native/quickstart): your first Viji native scene"
|
|
1547
1547
|
}
|
|
1548
1548
|
]
|
|
1549
1549
|
},
|
|
@@ -2580,7 +2580,7 @@ export const docsApi = {
|
|
|
2580
2580
|
},
|
|
2581
2581
|
"native-audio-overview": {
|
|
2582
2582
|
"id": "native-audio-overview",
|
|
2583
|
-
"title": "
|
|
2583
|
+
"title": "native-audio-overview",
|
|
2584
2584
|
"description": "Real-time audio analysis API: volume, frequency bands, beat detection, spectral features, and raw FFT/waveform data.",
|
|
2585
2585
|
"headings": [
|
|
2586
2586
|
{
|
|
@@ -2607,12 +2607,12 @@ export const docsApi = {
|
|
|
2607
2607
|
"content": [
|
|
2608
2608
|
{
|
|
2609
2609
|
"type": "text",
|
|
2610
|
-
"markdown": "# Audio\n\nViji provides real-time audio analysis when the host application connects an audio stream. All analysis runs on the host side and results are delivered to the scene through `viji.audio`.\n\n## API Overview\n\n| Sub-object | Description | Page |\n|------------|-------------|------|\n| [`isConnected`](connection/) | Whether an audio stream is active | [Connection & Lifecycle](connection/) |\n| [`volume`](volume/) | RMS level, peak amplitude, smoothed volume | [Volume](volume/) |\n| [`bands`](bands/) | Five frequency bands (instant and smoothed) | [Frequency Bands](bands/) |\n| [`beat`](beat/) | Beat energy curves, triggers, events, BPM | [Beat Detection](beat/) |\n| [`spectral`](spectral/) | Brightness and flatness features | [Spectral Analysis](spectral/) |\n| [`getFrequencyData()`](frequency-data/) | Raw FFT spectrum as `Uint8Array` | [Frequency Data](frequency-data/) |\n| [`getWaveform()`](waveform/) | Raw time-domain samples as `Float32Array` | [Waveform](waveform/) |\n\n## Basic Usage\n\n```javascript\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n const w = viji.width;\n const h = viji.height;\n\n ctx.fillStyle = '#111';\n ctx.fillRect(0, 0, w, h);\n\n if (!viji.audio.isConnected) {\n ctx.fillStyle = '#555';\n ctx.font = `${Math.min(w, h) * 0.04}px sans-serif`;\n ctx.textAlign = 'center';\n ctx.fillText('Waiting for audio...', w / 2, h / 2);\n return;\n }\n\n const vol = viji.audio.volume.smoothed;\n const r = Math.min(w, h) * 0.1 + vol * Math.min(w, h) * 0.3;\n\n ctx.beginPath();\n ctx.arc(w / 2, h / 2, r, 0, Math.PI * 2);\n ctx.fillStyle = `hsl(${200 + vol * 160}, 80%, 60%)`;\n ctx.fill();\n}\n```\n\n> [!NOTE]\n> Always check [`viji.audio.isConnected`](connection/) before reading audio values. When no audio stream is connected, all numeric values are `0` (`bpm` included). The `120` value sometimes seen in BPM contexts is an internal tracker fallback after audio connects but before tempo lock-on; before any audio connects, `bpm` reads `0`."
|
|
2610
|
+
"markdown": "# Audio\r\n\r\nViji provides real-time audio analysis when the host application connects an audio stream. All analysis runs on the host side and results are delivered to the scene through `viji.audio`.\r\n\r\n## API Overview\r\n\r\n| Sub-object | Description | Page |\r\n|------------|-------------|------|\r\n| [`isConnected`](connection/) | Whether an audio stream is active | [Connection & Lifecycle](connection/) |\r\n| [`volume`](volume/) | RMS level, peak amplitude, smoothed volume | [Volume](volume/) |\r\n| [`bands`](bands/) | Five frequency bands (instant and smoothed) | [Frequency Bands](bands/) |\r\n| [`beat`](beat/) | Beat energy curves, triggers, events, BPM | [Beat Detection](beat/) |\r\n| [`spectral`](spectral/) | Brightness and flatness features | [Spectral Analysis](spectral/) |\r\n| [`getFrequencyData()`](frequency-data/) | Raw FFT spectrum as `Uint8Array` | [Frequency Data](frequency-data/) |\r\n| [`getWaveform()`](waveform/) | Raw time-domain samples as `Float32Array` | [Waveform](waveform/) |\r\n\r\n## Basic Usage\r\n\r\n```javascript\r\nfunction render(viji) {\r\n const ctx = viji.useContext('2d');\r\n const w = viji.width;\r\n const h = viji.height;\r\n\r\n ctx.fillStyle = '#111';\r\n ctx.fillRect(0, 0, w, h);\r\n\r\n if (!viji.audio.isConnected) {\r\n ctx.fillStyle = '#555';\r\n ctx.font = `${Math.min(w, h) * 0.04}px sans-serif`;\r\n ctx.textAlign = 'center';\r\n ctx.fillText('Waiting for audio...', w / 2, h / 2);\r\n return;\r\n }\r\n\r\n const vol = viji.audio.volume.smoothed;\r\n const r = Math.min(w, h) * 0.1 + vol * Math.min(w, h) * 0.3;\r\n\r\n ctx.beginPath();\r\n ctx.arc(w / 2, h / 2, r, 0, Math.PI * 2);\r\n ctx.fillStyle = `hsl(${200 + vol * 160}, 80%, 60%)`;\r\n ctx.fill();\r\n}\r\n```\r\n\r\n> [!NOTE]\r\n> Always check [`viji.audio.isConnected`](connection/) before reading audio values. When no audio stream is connected, all numeric values are `0` (`bpm` included). The `120` value sometimes seen in BPM contexts is an internal tracker fallback after audio connects but before tempo lock-on; before any audio connects, `bpm` reads `0`."
|
|
2611
2611
|
},
|
|
2612
2612
|
{
|
|
2613
2613
|
"type": "live-example",
|
|
2614
2614
|
"title": "Audio-Reactive Circle",
|
|
2615
|
-
"sceneCode": "function render(viji) {\n const ctx = viji.useContext('2d');\n const w = viji.width;\n const h = viji.height;\n\n ctx.fillStyle = '#111';\n ctx.fillRect(0, 0, w, h);\n\n if (!viji.audio.isConnected) {\n ctx.fillStyle = '#555';\n ctx.font = `${Math.min(w, h) * 0.04}px sans-serif`;\n ctx.textAlign = 'center';\n ctx.fillText('Waiting for audio...', w / 2, h / 2);\n return;\n }\n\n const vol = viji.audio.volume.smoothed;\n const r = Math.min(w, h) * 0.1 + vol * Math.min(w, h) * 0.3;\n\n ctx.beginPath();\n ctx.arc(w / 2, h / 2, r, 0, Math.PI * 2);\n ctx.fillStyle = `hsl(${200 + vol * 160}, 80%, 60%)`;\n ctx.fill();\n}\n",
|
|
2615
|
+
"sceneCode": "function render(viji) {\r\n const ctx = viji.useContext('2d');\r\n const w = viji.width;\r\n const h = viji.height;\r\n\r\n ctx.fillStyle = '#111';\r\n ctx.fillRect(0, 0, w, h);\r\n\r\n if (!viji.audio.isConnected) {\r\n ctx.fillStyle = '#555';\r\n ctx.font = `${Math.min(w, h) * 0.04}px sans-serif`;\r\n ctx.textAlign = 'center';\r\n ctx.fillText('Waiting for audio...', w / 2, h / 2);\r\n return;\r\n }\r\n\r\n const vol = viji.audio.volume.smoothed;\r\n const r = Math.min(w, h) * 0.1 + vol * Math.min(w, h) * 0.3;\r\n\r\n ctx.beginPath();\r\n ctx.arc(w / 2, h / 2, r, 0, Math.PI * 2);\r\n ctx.fillStyle = `hsl(${200 + vol * 160}, 80%, 60%)`;\r\n ctx.fill();\r\n}\r\n",
|
|
2616
2616
|
"sceneFile": "audio-overview.scene.js",
|
|
2617
2617
|
"capabilities": {
|
|
2618
2618
|
"audio": true
|
|
@@ -2620,13 +2620,13 @@ export const docsApi = {
|
|
|
2620
2620
|
},
|
|
2621
2621
|
{
|
|
2622
2622
|
"type": "text",
|
|
2623
|
-
"markdown": "## Beyond the Main Stream\n\nWhen an external device (phone, tablet, second machine) connects with an audio source, it surfaces as `device.audio` on the matching entry in `viji.devices[]`. Each `device.audio` is an `AudioStreamAPI`: a lightweight subset of the full `AudioAPI` on this page. It includes `volume` (current/peak/smoothed), `bands` (low/lowMid/mid/highMid/high plus each `*Smoothed` sibling), `spectral` (brightness/flatness), `getFrequencyData()`, and `getWaveform()`, but **not** `beat` energies, triggers, events, or BPM. Those remain on `viji.audio` only.\n\nSee [Device Audio](/native/external-devices/audio) for the full `device.audio` reference.\n\nThe host may also expose additional standalone audio sources via `viji.audioStreams[]` (same `AudioStreamAPI` shape). This array is mostly used internally by Viji's compositor for mixing scenes; standalone scenes rarely read it directly. See the [Native API Reference](/native/api-reference) for details.\n\n## Related\n\n- [Connection & Lifecycle](connection/)\n- [Volume](volume/)\n- [Frequency Bands](bands/)\n- [Beat Detection](beat/)\n- [Spectral Analysis](spectral/)\n- [Frequency Data](frequency-data/)\n- [Waveform](waveform/)\n- [P5 Audio](/p5/audio)\n- [Shader Audio Uniforms](/shader/audio)"
|
|
2623
|
+
"markdown": "## Beyond the Main Stream\r\n\r\nWhen an external device (phone, tablet, second machine) connects with an audio source, it surfaces as `device.audio` on the matching entry in `viji.devices[]`. Each `device.audio` is an `AudioStreamAPI`: a lightweight subset of the full `AudioAPI` on this page. It includes `volume` (current/peak/smoothed), `bands` (low/lowMid/mid/highMid/high plus each `*Smoothed` sibling), `spectral` (brightness/flatness), `getFrequencyData()`, and `getWaveform()`, but **not** `beat` energies, triggers, events, or BPM. Those remain on `viji.audio` only.\r\n\r\nSee [Device Audio](/native/external-devices/audio) for the full `device.audio` reference.\r\n\r\nThe host may also expose additional standalone audio sources via `viji.audioStreams[]` (same `AudioStreamAPI` shape). This array is mostly used internally by Viji's compositor for mixing scenes; standalone scenes rarely read it directly. See the [Native API Reference](/native/api-reference) for details.\r\n\r\n## Related\r\n\r\n- [Connection & Lifecycle](connection/)\r\n- [Volume](volume/)\r\n- [Frequency Bands](bands/)\r\n- [Beat Detection](beat/)\r\n- [Spectral Analysis](spectral/)\r\n- [Frequency Data](frequency-data/)\r\n- [Waveform](waveform/)\r\n- [P5 Audio](/p5/audio)\r\n- [Shader Audio Uniforms](/shader/audio)"
|
|
2624
2624
|
}
|
|
2625
2625
|
]
|
|
2626
2626
|
},
|
|
2627
2627
|
"native-audio-connection": {
|
|
2628
2628
|
"id": "native-audio-connection",
|
|
2629
|
-
"title": "
|
|
2629
|
+
"title": "native-audio-connection",
|
|
2630
2630
|
"description": "Audio connection state, guard patterns, and default values when no audio stream is active.",
|
|
2631
2631
|
"headings": [
|
|
2632
2632
|
{
|
|
@@ -2658,12 +2658,12 @@ export const docsApi = {
|
|
|
2658
2658
|
"content": [
|
|
2659
2659
|
{
|
|
2660
2660
|
"type": "text",
|
|
2661
|
-
"markdown": "# Connection & Lifecycle\n\nThe `viji.audio.isConnected` property indicates whether the host application has provided an active audio stream. All other audio properties depend on this: when disconnected, they hold default values.\n\n## Property Reference\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `viji.audio.isConnected` | `boolean` | `true` when an audio stream is active and analysis results are flowing |\n\n## Guard Pattern\n\nAlways check `isConnected` before using audio data. This prevents your scene from reacting to default values as if they were real audio input.\n\n```javascript\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n const w = viji.width;\n const h = viji.height;\n\n ctx.fillStyle = '#111';\n ctx.fillRect(0, 0, w, h);\n\n if (!viji.audio.isConnected) {\n ctx.fillStyle = '#444';\n ctx.font = `${Math.min(w, h) * 0.04}px sans-serif`;\n ctx.textAlign = 'center';\n ctx.fillText('No audio connected', w / 2, h / 2);\n return;\n }\n\n // Safe to use audio data here\n const vol = viji.audio.volume.current;\n ctx.fillStyle = `rgba(100, 200, 255, ${vol})`;\n ctx.fillRect(0, 0, w * vol, h);\n}\n```\n\n## Connection Lifecycle\n\n1. **Disconnected (default)**: `isConnected` is `false`. All audio values are at their defaults.\n2. **Connected**: The host provides a `MediaStream`. `isConnected` becomes `true` and audio analysis values begin updating every frame.\n3. **Disconnected again**: The stream is removed. `isConnected` returns to `false` and all values reset to defaults.\n\n## Default Values\n\nWhen `isConnected` is `false`, all audio properties hold these values:\n\n| Property | Default |\n|----------|---------|\n| `volume.current`, `volume.peak`, `volume.smoothed` | `0` |\n| All `bands.*` (instant and smoothed) | `0` |\n| `beat.kick`, `.snare`, `.hat`, `.any` (and smoothed) | `0` |\n| `beat.triggers.kick`, `.snare`, `.hat`, `.any` | `false` |\n| `beat.events` | `[]` (empty array) |\n| `beat.bpm` | `0` |\n| `beat.confidence` | `0` |\n| `beat.isLocked` | `false` |\n| `spectral.brightness`, `spectral.flatness` | `0` |\n| `getFrequencyData()` | Empty `Uint8Array` (length 0) |\n| `getWaveform()` | Empty `Float32Array` (length 0) |\n\n> [!NOTE]\n> `beat.bpm` is `0` when no audio is connected (the \"no signal\" sentinel). Once audio connects, it tracks the detected tempo clamped to 60-240, with `120` as a fallback before the tracker locks on. If your scene needs a fallback default for BPM-based calculations before audio connects, write `viji.audio.beat.bpm || 120` explicitly."
|
|
2661
|
+
"markdown": "# Connection & Lifecycle\r\n\r\nThe `viji.audio.isConnected` property indicates whether the host application has provided an active audio stream. All other audio properties depend on this: when disconnected, they hold default values.\r\n\r\n## Property Reference\r\n\r\n| Property | Type | Description |\r\n|----------|------|-------------|\r\n| `viji.audio.isConnected` | `boolean` | `true` when an audio stream is active and analysis results are flowing |\r\n\r\n## Guard Pattern\r\n\r\nAlways check `isConnected` before using audio data. This prevents your scene from reacting to default values as if they were real audio input.\r\n\r\n```javascript\r\nfunction render(viji) {\r\n const ctx = viji.useContext('2d');\r\n const w = viji.width;\r\n const h = viji.height;\r\n\r\n ctx.fillStyle = '#111';\r\n ctx.fillRect(0, 0, w, h);\r\n\r\n if (!viji.audio.isConnected) {\r\n ctx.fillStyle = '#444';\r\n ctx.font = `${Math.min(w, h) * 0.04}px sans-serif`;\r\n ctx.textAlign = 'center';\r\n ctx.fillText('No audio connected', w / 2, h / 2);\r\n return;\r\n }\r\n\r\n // Safe to use audio data here\r\n const vol = viji.audio.volume.current;\r\n ctx.fillStyle = `rgba(100, 200, 255, ${vol})`;\r\n ctx.fillRect(0, 0, w * vol, h);\r\n}\r\n```\r\n\r\n## Connection Lifecycle\r\n\r\n1. **Disconnected (default)**: `isConnected` is `false`. All audio values are at their defaults.\r\n2. **Connected**: The host provides a `MediaStream`. `isConnected` becomes `true` and audio analysis values begin updating every frame.\r\n3. **Disconnected again**: The stream is removed. `isConnected` returns to `false` and all values reset to defaults.\r\n\r\n## Default Values\r\n\r\nWhen `isConnected` is `false`, all audio properties hold these values:\r\n\r\n| Property | Default |\r\n|----------|---------|\r\n| `volume.current`, `volume.peak`, `volume.smoothed` | `0` |\r\n| All `bands.*` (instant and smoothed) | `0` |\r\n| `beat.kick`, `.snare`, `.hat`, `.any` (and smoothed) | `0` |\r\n| `beat.triggers.kick`, `.snare`, `.hat`, `.any` | `false` |\r\n| `beat.events` | `[]` (empty array) |\r\n| `beat.bpm` | `0` |\r\n| `beat.confidence` | `0` |\r\n| `beat.isLocked` | `false` |\r\n| `spectral.brightness`, `spectral.flatness` | `0` |\r\n| `getFrequencyData()` | Empty `Uint8Array` (length 0) |\r\n| `getWaveform()` | Empty `Float32Array` (length 0) |\r\n\r\n> [!NOTE]\r\n> `beat.bpm` is `0` when no audio is connected (the \"no signal\" sentinel). Once audio connects, it tracks the detected tempo clamped to 60-240, with `120` as a fallback before the tracker locks on. If your scene needs a fallback default for BPM-based calculations before audio connects, write `viji.audio.beat.bpm || 120` explicitly."
|
|
2662
2662
|
},
|
|
2663
2663
|
{
|
|
2664
2664
|
"type": "live-example",
|
|
2665
2665
|
"title": "Connection State",
|
|
2666
|
-
"sceneCode": "function render(viji) {\n const ctx = viji.useContext('2d');\n const w = viji.width;\n const h = viji.height;\n\n ctx.fillStyle = '#111';\n ctx.fillRect(0, 0, w, h);\n\n const fontSize = Math.min(w, h) * 0.035;\n ctx.font = `${fontSize}px sans-serif`;\n ctx.textAlign = 'center';\n\n if (!viji.audio.isConnected) {\n const pulse = 0.4 + Math.sin(viji.time * 2) * 0.15;\n ctx.fillStyle = `rgba(255, 255, 255, ${pulse})`;\n ctx.fillText('Waiting for audio stream...', w / 2, h / 2 - fontSize);\n ctx.fillStyle = '#444';\n ctx.fillText('Connect a microphone or audio source', w / 2, h / 2 + fontSize);\n return;\n }\n\n const vol = viji.audio.volume.smoothed;\n const barW = w * 0.6;\n const barH = Math.min(w, h) * 0.06;\n const barX = (w - barW) / 2;\n const barY = h / 2 - barH / 2;\n\n ctx.fillStyle = '#222';\n ctx.fillRect(barX, barY, barW, barH);\n ctx.fillStyle = '#4CAF50';\n ctx.fillRect(barX, barY, barW * vol, barH);\n\n ctx.fillStyle = '#aaa';\n ctx.fillText('Audio connected: volume: ' + vol.toFixed(2), w / 2, barY - fontSize);\n}\n",
|
|
2666
|
+
"sceneCode": "function render(viji) {\r\n const ctx = viji.useContext('2d');\r\n const w = viji.width;\r\n const h = viji.height;\r\n\r\n ctx.fillStyle = '#111';\r\n ctx.fillRect(0, 0, w, h);\r\n\r\n const fontSize = Math.min(w, h) * 0.035;\r\n ctx.font = `${fontSize}px sans-serif`;\r\n ctx.textAlign = 'center';\r\n\r\n if (!viji.audio.isConnected) {\r\n const pulse = 0.4 + Math.sin(viji.time * 2) * 0.15;\r\n ctx.fillStyle = `rgba(255, 255, 255, ${pulse})`;\r\n ctx.fillText('Waiting for audio stream...', w / 2, h / 2 - fontSize);\r\n ctx.fillStyle = '#444';\r\n ctx.fillText('Connect a microphone or audio source', w / 2, h / 2 + fontSize);\r\n return;\r\n }\r\n\r\n const vol = viji.audio.volume.smoothed;\r\n const barW = w * 0.6;\r\n const barH = Math.min(w, h) * 0.06;\r\n const barX = (w - barW) / 2;\r\n const barY = h / 2 - barH / 2;\r\n\r\n ctx.fillStyle = '#222';\r\n ctx.fillRect(barX, barY, barW, barH);\r\n ctx.fillStyle = '#4CAF50';\r\n ctx.fillRect(barX, barY, barW * vol, barH);\r\n\r\n ctx.fillStyle = '#aaa';\r\n ctx.fillText('Audio connected: volume: ' + vol.toFixed(2), w / 2, barY - fontSize);\r\n}\r\n",
|
|
2667
2667
|
"sceneFile": "connection-demo.scene.js",
|
|
2668
2668
|
"capabilities": {
|
|
2669
2669
|
"audio": true
|
|
@@ -2671,7 +2671,7 @@ export const docsApi = {
|
|
|
2671
2671
|
},
|
|
2672
2672
|
{
|
|
2673
2673
|
"type": "text",
|
|
2674
|
-
"markdown": "## Related\n\n- [Audio Overview](../)\n- [Volume](../volume/)\n- [Beat Detection](../beat/)\n- [P5 Connection & Lifecycle](/p5/audio/connection)"
|
|
2674
|
+
"markdown": "## Related\r\n\r\n- [Audio Overview](../)\r\n- [Volume](../volume/)\r\n- [Beat Detection](../beat/)\r\n- [P5 Connection & Lifecycle](/p5/audio/connection)"
|
|
2675
2675
|
}
|
|
2676
2676
|
]
|
|
2677
2677
|
},
|
|
@@ -3052,6 +3052,11 @@ export const docsApi = {
|
|
|
3052
3052
|
"level": 2,
|
|
3053
3053
|
"text": "CV Control API"
|
|
3054
3054
|
},
|
|
3055
|
+
{
|
|
3056
|
+
"id": "where-to-call-enable",
|
|
3057
|
+
"level": 3,
|
|
3058
|
+
"text": "Where to call enable*"
|
|
3059
|
+
},
|
|
3055
3060
|
{
|
|
3056
3061
|
"id": "guard-pattern",
|
|
3057
3062
|
"level": 2,
|
|
@@ -3076,7 +3081,7 @@ export const docsApi = {
|
|
|
3076
3081
|
"content": [
|
|
3077
3082
|
{
|
|
3078
3083
|
"type": "text",
|
|
3079
|
-
"markdown": "# Connection & Lifecycle\n\nThe `viji.video.isConnected` property indicates whether the host application has provided an active video stream. All other video and CV properties depend on this: when disconnected, they hold default values.\n\n## Property Reference\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `viji.video.isConnected` | `boolean` | `true` when a video stream is active |\n| `viji.video.currentFrame` | `OffscreenCanvas \\| null` | Current video frame: drawable with `ctx.drawImage()` |\n| `viji.video.frameWidth` | `number` | Video frame width in pixels |\n| `viji.video.frameHeight` | `number` | Video frame height in pixels |\n| `viji.video.frameRate` | `number` | Video frame rate (Hz) |\n| `viji.video.getFrameData()` | `ImageData \\| null` | Raw pixel data for per-pixel analysis |\n\n> [!TIP]\n> Use [`viji.video.currentFrame`](../basics/) for drawing video to canvas (fast, GPU-friendly). Use `viji.video.getFrameData()` only when you need per-pixel access: it is much slower as it reads back pixel data.\n\n## CV Control API\n\nThe `viji.video.cv` object provides methods to enable and disable individual CV features. All methods accept a boolean parameter.\n\n| Method |
|
|
3084
|
+
"markdown": "# Connection & Lifecycle\n\nThe `viji.video.isConnected` property indicates whether the host application has provided an active video stream. All other video and CV properties depend on this: when disconnected, they hold default values.\n\n## Property Reference\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `viji.video.isConnected` | `boolean` | `true` when a video stream is active |\n| `viji.video.currentFrame` | `OffscreenCanvas \\| null` | Current video frame: drawable with `ctx.drawImage()` |\n| `viji.video.frameWidth` | `number` | Video frame width in pixels |\n| `viji.video.frameHeight` | `number` | Video frame height in pixels |\n| `viji.video.frameRate` | `number` | Video frame rate (Hz) |\n| `viji.video.getFrameData()` | `ImageData \\| null` | Raw pixel data for per-pixel analysis |\n\n> [!TIP]\n> Use [`viji.video.currentFrame`](../basics/) for drawing video to canvas (fast, GPU-friendly). Use `viji.video.getFrameData()` only when you need per-pixel access: it is much slower as it reads back pixel data.\n\n## CV Control API\n\nThe `viji.video.cv` object provides methods to enable and disable individual CV features. All methods accept a boolean parameter and return a `Promise<void>` that resolves once the underlying pipeline has finished switching.\n\n| Method | Populates these `face` / `viji.video.cv.*` fields |\n|--------|---------|\n| `enableFaceDetection(enabled)` | `face.bounds`, `face.center`, `face.confidence` |\n| `enableFaceMesh(enabled)` | `face.landmarks` (468 pts), `face.headPose` |\n| `enableEmotionDetection(enabled)` | `face.expressions` (7), `face.blendshapes` (52) — internally also loads the face landmarker, so `face.landmarks` and `face.headPose` are populated |\n| `enableHandTracking(enabled)` | `viji.video.cv.hands[]` (up to 2, landmarks + palm + bounds + gestures) |\n| `enablePoseDetection(enabled)` | `viji.video.cv.pose` (33 BlazePose landmarks + body-part groups) |\n| `enableBodySegmentation(enabled)` | `viji.video.cv.segmentation` (per-pixel mask) |\n\nEach feature is independent: enabling one does NOT populate fields owned by another. If your scene needs both face bounds AND head pose, enable face detection AND face mesh. The verbs are idempotent and reference-counted, so calling them many times is cheap.\n\n### Where to call enable*\n\n`viji.video.cv.enableX(true)` is safe to call from either:\n\n- **Module scope** — for scenes where the CV feature is always required (a face filter that does not work without face mesh).\n- **Per-frame inside `render()`** — the canonical pattern for opt-in CV gated by a `viji.toggle(...)` parameter. The host UI controls activation; per-frame calls are cheap.\n\n`await` is optional. Without it, the MediaPipe model loads asynchronously while the render loop runs; the corresponding data fields populate as soon as the first inference returns. With `await` you can sequence post-load code.\n\n> [!WARNING]\n> Each active CV feature consumes its own WebGL context for MediaPipe inference. Browsers typically allow 8-16 active WebGL contexts. Enabling many CV features simultaneously on lower-end hardware can cause context eviction and break the scene's own rendering. Enable only the features you need; prefer the toggle-gated per-frame pattern for opt-in CV.\n\n| Method | Returns | Description |\n|--------|---------|-------------|\n| `getActiveFeatures()` | `CVFeature[]` | Array of currently active feature strings |\n| `isProcessing()` | `boolean` | `true` if CV worker is actively processing frames |\n\n## Guard Pattern\n\nAlways check `isConnected` and `currentFrame` before using video data. This prevents your scene from drawing a null frame or reacting to default CV values as if they were real input.\n\nVideo drawing follows the [aspect-correct pattern](../basics/#drawing-video).\n\n```javascript\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n const w = viji.width;\n const h = viji.height;\n\n ctx.fillStyle = '#111';\n ctx.fillRect(0, 0, w, h);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) {\n ctx.fillStyle = '#444';\n ctx.font = `${Math.min(w, h) * 0.04}px sans-serif`;\n ctx.textAlign = 'center';\n ctx.fillText('No video connected', w / 2, h / 2);\n return;\n }\n\n const v = videoFit(viji);\n ctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n}\n```\n\n## Connection Lifecycle\n\n1. **Disconnected (default)**: `isConnected` is `false`. All video values are at their defaults.\n2. **Connected**: The host provides a video stream. `isConnected` becomes `true` and `currentFrame` begins updating every frame.\n3. **Disconnected again**: The stream is removed. `isConnected` returns to `false` and all values reset to defaults.\n\n## Default Values\n\nWhen `isConnected` is `false`, all video properties hold these values:\n\n| Property | Default |\n|----------|---------|\n| `currentFrame` | `null` |\n| `frameWidth` | `0` |\n| `frameHeight` | `0` |\n| `frameRate` | `0` |\n| `getFrameData()` | `null` |\n| `faces` | `[]` (empty array) |\n| `hands` | `[]` (empty array) |\n| `pose` | `null` |\n| `segmentation` | `null` |\n\nWhen CV features are not enabled, the corresponding fields on each face / hand entry are also `null` rather than fake zeros: `face.bounds` / `face.center` / `face.confidence` are `null` unless face detection is on, `face.headPose` is `null` unless face mesh is on, `face.expressions` / `face.blendshapes` are `null` unless emotion detection is on. Always null-check before reading.\n\nWhen the user leaves the camera frame, CV data also resets: `faces` and `hands` become empty arrays, `pose` and `segmentation` become `null`. This prevents stale data from persisting."
|
|
3080
3085
|
},
|
|
3081
3086
|
{
|
|
3082
3087
|
"type": "live-example",
|
|
@@ -3193,7 +3198,7 @@ export const docsApi = {
|
|
|
3193
3198
|
"content": [
|
|
3194
3199
|
{
|
|
3195
3200
|
"type": "text",
|
|
3196
|
-
"markdown": "# Face Detection\n\nFace detection provides the position, size, and confidence of faces in the video stream. Enable it with [`viji.video.cv.enableFaceDetection(true)`](../connection/).\n\n## Property Reference\n\nResults appear in `viji.video.cv.faces`: an array of `FaceData` objects. When no faces are detected or the feature is disabled, the array is empty.\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `face.id` | `number` | Index-based face identifier (0, 1, 2, ...) |\n| `face.bounds` | `{ x, y, width, height }` | Bounding box, normalized 0-1 |\n| `face.center` | `{ x, y }` | Bounding box center, normalized 0-1 |\n| `face.confidence` | `number` | Detection confidence (0-1) |\n| `face.landmarks` | `{ x, y, z? }[]` | Empty `[]
|
|
3201
|
+
"markdown": "# Face Detection\n\nFace detection provides the position, size, and confidence of faces in the video stream. Enable it with [`viji.video.cv.enableFaceDetection(true)`](../connection/).\n\nFace detection populates `face.bounds`, `face.center`, and `face.confidence` on each detected face. It does NOT populate `face.landmarks` / `face.headPose` (use [Face Mesh](../face-mesh/)) or `face.expressions` / `face.blendshapes` (use [Emotion Detection](../emotion-detection/)). The three CV features are independent — enable each one you need.\n\n`viji.video.cv.enableFaceDetection(true)` is safe to call from module scope (always-on CV) or per-frame inside `render()` gated by a toggle parameter (opt-in CV). The verb is idempotent and reference-counted; per-frame calls are cheap. `await` is optional.\n\n## Property Reference\n\nResults appear in `viji.video.cv.faces`: an array of `FaceData` objects. When no faces are detected or the feature is disabled, the array is empty.\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `face.id` | `number` | Index-based face identifier (0, 1, 2, ...) |\n| `face.bounds` | `{ x, y, width, height } \\| null` | Bounding box, normalized 0-1. `null` when face detection is disabled. |\n| `face.center` | `{ x, y } \\| null` | Bounding box center, normalized 0-1. `null` when face detection is disabled. |\n| `face.confidence` | `number \\| null` | Detection confidence (0-1). `null` when face detection is disabled. |\n| `face.landmarks` | `{ x, y, z? }[]` | Empty `[]` unless [Face Mesh](../face-mesh/) is enabled. |\n| `face.headPose` | `{ pitch, yaw, roll } \\| null` | `null` unless [Face Mesh](../face-mesh/) is enabled. |\n| `face.expressions` | `object \\| null` | `null` unless [Emotion Detection](../emotion-detection/) is enabled. |\n| `face.blendshapes` | `FaceBlendshapes \\| null` | `null` unless [Emotion Detection](../emotion-detection/) is enabled. |\n\n### Coordinate System\n\nAll coordinates are normalized 0-1. To draw on the canvas, multiply by canvas dimensions:\n\n```javascript\nconst screenX = face.bounds.x * viji.width;\nconst screenY = face.bounds.y * viji.height;\n```\n\n## Usage\n\nVideo drawing follows the [aspect-correct pattern](../basics/#drawing-video). The `videoFit` helper returns the rectangle the video occupies inside the canvas; multiply CV bounds by `v.width` / `v.height` and offset by `v.x` / `v.y` so they stay aligned. This demo uses `'contain'` so face boxes near the frame edges are not clipped against the canvas.\n\n```javascript\nconst useFace = viji.toggle(false, { label: 'Face Detection', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n\n ctx.fillStyle = '#111';\n ctx.fillRect(0, 0, viji.width, viji.height);\n\n if (useFace.value) viji.video.cv.enableFaceDetection(true);\n else viji.video.cv.enableFaceDetection(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) return;\n\n const v = videoFit(viji, 'contain');\n ctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n viji.video.cv.faces.forEach(face => {\n const bx = v.x + face.bounds.x * v.width;\n const by = v.y + face.bounds.y * v.height;\n const bw = face.bounds.width * v.width;\n const bh = face.bounds.height * v.height;\n\n ctx.strokeStyle = '#4ecdc4';\n ctx.lineWidth = 2;\n ctx.strokeRect(bx, by, bw, bh);\n\n ctx.fillStyle = '#4ecdc4';\n ctx.beginPath();\n ctx.arc(v.x + face.center.x * v.width, v.y + face.center.y * v.height, 4, 0, Math.PI * 2);\n ctx.fill();\n\n ctx.font = `${Math.min(viji.width, viji.height) * 0.025}px sans-serif`;\n ctx.fillText('Face #' + face.id + ' (' + (face.confidence * 100).toFixed(0) + '%)', bx, by - 6);\n });\n}\n```\n\n**Cost: Low.** Face detection is the lightest CV feature, providing only bounding boxes and basic metadata.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\n> [!TIP]\n> **Best practice:** Don't enable CV features by default. Instead, expose a toggle parameter so users can activate them on capable devices:\n> ```javascript\n> const useFace = viji.toggle(false, { label: 'Enable Face Detection', category: 'video' });\n> if (useFace.value) {\n> await viji.video.cv.enableFaceDetection(true);\n> }\n> ```\n\nWhen face detection is disabled or no faces are visible, `viji.video.cv.faces` becomes an empty array `[]`."
|
|
3197
3202
|
},
|
|
3198
3203
|
{
|
|
3199
3204
|
"type": "live-example",
|
|
@@ -3235,11 +3240,21 @@ export const docsApi = {
|
|
|
3235
3240
|
"level": 2,
|
|
3236
3241
|
"text": "Usage"
|
|
3237
3242
|
},
|
|
3243
|
+
{
|
|
3244
|
+
"id": "computing-a-face-center-from-landmarks",
|
|
3245
|
+
"level": 2,
|
|
3246
|
+
"text": "Computing a Face Center From Landmarks"
|
|
3247
|
+
},
|
|
3238
3248
|
{
|
|
3239
3249
|
"id": "pairing-frame-and-landmarks",
|
|
3240
3250
|
"level": 2,
|
|
3241
3251
|
"text": "Pairing Frame and Landmarks"
|
|
3242
3252
|
},
|
|
3253
|
+
{
|
|
3254
|
+
"id": "enabling-face-mesh",
|
|
3255
|
+
"level": 2,
|
|
3256
|
+
"text": "Enabling face mesh"
|
|
3257
|
+
},
|
|
3243
3258
|
{
|
|
3244
3259
|
"id": "related",
|
|
3245
3260
|
"level": 2,
|
|
@@ -3249,12 +3264,12 @@ export const docsApi = {
|
|
|
3249
3264
|
"content": [
|
|
3250
3265
|
{
|
|
3251
3266
|
"type": "text",
|
|
3252
|
-
"markdown": "# Face Mesh\n\nFace mesh provides 468 detailed facial landmark points and head pose estimation (pitch, yaw, roll). Enable it with [`viji.video.cv.enableFaceMesh(true)`](../connection/)
|
|
3267
|
+
"markdown": "# Face Mesh\n\nFace mesh provides 468 detailed facial landmark points and head pose estimation (pitch, yaw, roll). Enable it with [`viji.video.cv.enableFaceMesh(true)`](../connection/).\n\nFace mesh is independent of [Face Detection](../face-detection/). Each feature populates its own subset of fields on `FaceData`:\n\n- Face mesh populates `face.landmarks` and `face.headPose`.\n- Face detection populates `face.bounds`, `face.center`, `face.confidence`.\n\nIf you also need `face.bounds` or `face.center` (e.g., for positioning a hat or glasses around the face), enable face detection alongside mesh. Alternatively, compute a bounding box from `face.landmarks` min/max yourself — the data is the same, with no extra MediaPipe model loaded.\n\n## Property Reference\n\nFace mesh data appears on each `FaceData` object in `viji.video.cv.faces`:\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `face.landmarks` | `{ x, y, z? }[]` | 468 facial landmark points, normalized 0-1. Empty `[]` when face mesh is disabled. |\n| `face.headPose` | `{ pitch, yaw, roll } \\| null` | Head rotation in degrees, or `null` when face mesh is disabled. |\n\n### Landmarks\n\nThe 468-point mesh covers the entire face surface including eyes, eyebrows, nose, mouth, jawline, and forehead. Each point has `x` and `y` in normalized 0-1 coordinates, with an optional `z` for depth.\n\nWhen face mesh is disabled, `landmarks` is an empty array `[]`.\n\n### Head Pose\n\nHead pose is computed from the face mesh landmark geometry. Values are in degrees:\n\n| Axis | Range | Description |\n|------|-------|-------------|\n| `pitch` | -90 to 90 | Looking down (negative) or up (positive) |\n| `yaw` | -90 to 90 | Looking left (negative) or right (positive) |\n| `roll` | -180 to 180 | Tilting head left (negative) or right (positive) |\n\nWhen face mesh is disabled, `face.headPose` reads as `null`. Always null-check (`if (face.headPose) { ... }`) before reading `pitch`/`yaw`/`roll`.\n\n## Usage\n\n```javascript\nconst useMesh = viji.toggle(false, { label: 'Face Mesh', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n const w = viji.width;\n const h = viji.height;\n\n ctx.fillStyle = '#111';\n ctx.fillRect(0, 0, w, h);\n\n if (useMesh.value) viji.video.cv.enableFaceMesh(true);\n else viji.video.cv.enableFaceMesh(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) return;\n\n // 'contain' so face landmarks near the frame edges stay visible.\n const v = videoFit(viji, 'contain');\n ctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n viji.video.cv.faces.forEach(face => {\n if (face.landmarks.length === 0) return;\n\n ctx.fillStyle = 'rgba(69, 183, 209, 0.6)';\n face.landmarks.forEach(pt => {\n ctx.beginPath();\n ctx.arc(v.x + pt.x * v.width, v.y + pt.y * v.height, 1, 0, Math.PI * 2);\n ctx.fill();\n });\n\n const fontSize = Math.min(w, h) * 0.025;\n ctx.fillStyle = '#fff';\n ctx.font = `${fontSize}px sans-serif`;\n ctx.textAlign = 'left';\n const hp = face.headPose;\n ctx.fillText(\n 'Pitch: ' + hp.pitch.toFixed(1) + ' Yaw: ' + hp.yaw.toFixed(1) + ' Roll: ' + hp.roll.toFixed(1),\n w * 0.03, h - fontSize\n );\n });\n}\n```\n\n**Cost: Medium-High.** Face mesh processes 468 landmarks per face and computes head pose. More demanding than basic face detection.\n\n## Computing a Face Center From Landmarks\n\n`face.center` and `face.bounds` are populated by [Face Detection](../face-detection/), not by face mesh. When only face mesh is enabled, both read as `null`. For mesh-driven visualizations that need a face-anchored position (e.g. drawing a head-pose arrow at the face), compute the center from landmark min/max instead of enabling face detection just for that:\n\n```javascript\nfunction meshCenterFromLandmarks(landmarks) {\n if (!landmarks || landmarks.length === 0) return null;\n let minX = 1, minY = 1, maxX = 0, maxY = 0;\n for (const p of landmarks) {\n if (p.x < minX) minX = p.x;\n if (p.y < minY) minY = p.y;\n if (p.x > maxX) maxX = p.x;\n if (p.y > maxY) maxY = p.y;\n }\n return { x: (minX + maxX) / 2, y: (minY + maxY) / 2 };\n}\n\nviji.video.cv.faces.forEach(face => {\n const center = meshCenterFromLandmarks(face.landmarks);\n if (!center) return;\n const cx = v.x + center.x * v.width;\n const cy = v.y + center.y * v.height;\n // ...anchor your visualization at (cx, cy)\n});\n```\n\nThis saves a CV feature's worth of WebGL-context and inference cost compared to enabling face detection just for the center value.\n\n## Pairing Frame and Landmarks\n\nFace-mesh effects split into two camps with different defaults.\n\n**Effects that read pixels from the displayed frame at landmark positions** (texturing the face mesh with the underlying skin, warping the face along its mesh, masks that follow the face contour, makeup-style overlays that blend with skin tone): use [`viji.video.cv.analysedFrame`](../basics/), the exact frame the landmarks were computed from. The displayed image will stutter or hold briefly between inferences, but the effect aligns pixel-for-pixel with the face geometry.\n\n```javascript\n// Texture-mapped face filter: analysedFrame is required\nconst v = videoFit(viji, 'contain'); // see basics for the helper\nconst frame = viji.video.cv.analysedFrame ?? viji.video.currentFrame;\nctx.drawImage(frame, v.x, v.y, v.width, v.height);\nviji.video.cv.faces.forEach(face => { /* warp/texture along the mesh */ });\n```\n\n**Effects that draw on top of the face without reading its pixels** (landmark dots, bounding boxes, lines between landmarks, particles attached to landmark positions, debug overlays, parameter modulation): use `viji.video.currentFrame`. Landmarks lag the displayed face by ~1 frame under fast motion, but the camera feels live. Reaching for `analysedFrame` here would freeze the displayed video between inferences in exchange for alignment you don't need.\n\n```javascript\n// Live camera with landmark dots: currentFrame is the right default\nconst v = videoFit(viji, 'contain');\nctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\nviji.video.cv.faces.forEach(face => { /* dots, lines, particles at v.x + pt.x * v.width, etc. */ });\n```\n\n`analysedFrame` is `null` until MediaPipe completes its first inference after face mesh is enabled. The `?? currentFrame` fallback covers that startup window; expect a brief visual hitch when the first inference lands and the source switches.\n\nSee [Best Practices](/getting-started/best-practices/#currentframe-vs-analysedframe-pick-the-right-trade-off) for the cross-renderer trade-off discussion.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\n## Enabling face mesh\n\n`viji.video.cv.enableFaceMesh(true)` is safe to call from anywhere: module scope (for scenes where face mesh is always on) or per-frame inside `render()` gated by a toggle (the canonical pattern for opt-in CV). The verb is idempotent and reference-counted, so per-frame calls are cheap. `await` is optional — without it, the model loads asynchronously while the render loop runs; landmarks populate when ready.\n\n**Always-on CV** (mesh is mandatory for the scene):\n\n```javascript\n// Module scope — runs once when the scene loads.\nawait viji.video.cv.enableFaceMesh(true);\n\nfunction render(viji) { /* read face.landmarks every frame */ }\n```\n\n**Opt-in CV** (toggle-gated, recommended for general-purpose scenes):\n\n```javascript\nconst useMesh = viji.toggle(false, { label: 'Enable Face Mesh', category: 'video' });\n\nfunction render(viji) {\n // Per-frame, gated by toggle. Cheap (idempotent + reference-counted).\n if (useMesh.value) viji.video.cv.enableFaceMesh(true);\n else viji.video.cv.enableFaceMesh(false);\n // ...\n}\n```\n\nWhen face mesh is disabled, `face.landmarks` becomes `[]` and `face.headPose` reads as `null`."
|
|
3253
3268
|
},
|
|
3254
3269
|
{
|
|
3255
3270
|
"type": "live-example",
|
|
3256
3271
|
"title": "Face Mesh",
|
|
3257
|
-
"sceneCode": "const useMesh = viji.toggle(false, { label: 'Face Mesh', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction cvHint(ctx, viji, label) {\n const s = Math.min(viji.width, viji.height);\n const t = (Math.sin(viji.time * 3) + 1) / 2;\n const bandH = s * 0.18;\n const bandY = (viji.height - bandH) / 2;\n ctx.fillStyle = `rgba(255, 200, 0, ${0.85 + t * 0.15})`;\n ctx.fillRect(0, bandY, viji.width, bandH);\n ctx.fillStyle = '#111';\n ctx.font = `bold ${s * 0.045}px sans-serif`;\n ctx.textAlign = 'center';\n ctx.textBaseline = 'middle';\n ctx.fillText(`Enable \"${label}\" in parameters`, viji.width / 2, viji.height / 2);\n ctx.textBaseline = 'alphabetic';\n}\n\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n const w = viji.width;\n const h = viji.height;\n\n ctx.fillStyle = '#111';\n ctx.fillRect(0, 0, w, h);\n\n if (useMesh.value) viji.video.cv.enableFaceMesh(true);\n else viji.video.cv.enableFaceMesh(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) {\n ctx.fillStyle = '#555';\n ctx.font = `${Math.min(w, h) * 0.04}px sans-serif`;\n ctx.textAlign = 'center';\n ctx.fillText('Waiting for video...', w / 2, h / 2);\n return;\n }\n\n const v = videoFit(viji, 'contain');\n ctx.globalAlpha = 0.4;\n ctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n ctx.globalAlpha = 1.0;\n\n if (!useMesh.value) {\n cvHint(ctx, viji, 'Face Mesh');\n return;\n }\n\n viji.video.cv.faces.forEach(face => {\n if (face.landmarks.length === 0) return;\n\n ctx.fillStyle = 'rgba(69, 183, 209, 0.7)';\n face.landmarks.forEach(pt => {\n ctx.beginPath();\n ctx.arc(v.x + pt.x * v.width, v.y + pt.y * v.height, 1, 0, Math.PI * 2);\n ctx.fill();\n });\n\n const fontSize = Math.min(w, h) * 0.025;\n ctx.fillStyle = '#fff';\n ctx.font = `${fontSize}px sans-serif`;\n ctx.textAlign = 'left';\n
|
|
3272
|
+
"sceneCode": "const useMesh = viji.toggle(false, { label: 'Face Mesh', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\n// Compute face center from face-mesh landmark min/max. face.center is null\n// when only face mesh is enabled (it is populated by face detection), so\n// mesh-driven visualizations derive their anchor point from landmarks.\nfunction meshCenterFromLandmarks(landmarks) {\n if (!landmarks || landmarks.length === 0) return null;\n let minX = 1, minY = 1, maxX = 0, maxY = 0;\n for (const p of landmarks) {\n if (p.x < minX) minX = p.x;\n if (p.y < minY) minY = p.y;\n if (p.x > maxX) maxX = p.x;\n if (p.y > maxY) maxY = p.y;\n }\n return { x: (minX + maxX) / 2, y: (minY + maxY) / 2 };\n}\n\nfunction cvHint(ctx, viji, label) {\n const s = Math.min(viji.width, viji.height);\n const t = (Math.sin(viji.time * 3) + 1) / 2;\n const bandH = s * 0.18;\n const bandY = (viji.height - bandH) / 2;\n ctx.fillStyle = `rgba(255, 200, 0, ${0.85 + t * 0.15})`;\n ctx.fillRect(0, bandY, viji.width, bandH);\n ctx.fillStyle = '#111';\n ctx.font = `bold ${s * 0.045}px sans-serif`;\n ctx.textAlign = 'center';\n ctx.textBaseline = 'middle';\n ctx.fillText(`Enable \"${label}\" in parameters`, viji.width / 2, viji.height / 2);\n ctx.textBaseline = 'alphabetic';\n}\n\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n const w = viji.width;\n const h = viji.height;\n\n ctx.fillStyle = '#111';\n ctx.fillRect(0, 0, w, h);\n\n if (useMesh.value) viji.video.cv.enableFaceMesh(true);\n else viji.video.cv.enableFaceMesh(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) {\n ctx.fillStyle = '#555';\n ctx.font = `${Math.min(w, h) * 0.04}px sans-serif`;\n ctx.textAlign = 'center';\n ctx.fillText('Waiting for video...', w / 2, h / 2);\n return;\n }\n\n const v = videoFit(viji, 'contain');\n ctx.globalAlpha = 0.4;\n ctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n ctx.globalAlpha = 1.0;\n\n if (!useMesh.value) {\n cvHint(ctx, viji, 'Face Mesh');\n return;\n }\n\n viji.video.cv.faces.forEach(face => {\n if (face.landmarks.length === 0) return;\n const hp = face.headPose;\n if (!hp) return;\n\n // Landmark dots\n ctx.fillStyle = 'rgba(69, 183, 209, 0.7)';\n face.landmarks.forEach(pt => {\n ctx.beginPath();\n ctx.arc(v.x + pt.x * v.width, v.y + pt.y * v.height, 1, 0, Math.PI * 2);\n ctx.fill();\n });\n\n // Face-anchored head-pose arrow + roll horizon. Yaw drives X, pitch\n // drives Y (negated for Canvas2D Y-down), roll rotates the arrow vector\n // so a head tilt visibly rolls the direction. Center is computed from\n // landmarks because face.center is null in mesh-only mode.\n const center = meshCenterFromLandmarks(face.landmarks);\n if (center) {\n const fcx = v.x + center.x * v.width;\n const fcy = v.y + center.y * v.height;\n const yawRad = hp.yaw * Math.PI / 180;\n const pitchRad = hp.pitch * Math.PI / 180;\n const rollRad = hp.roll * Math.PI / 180;\n const armLen = Math.min(v.width, v.height) * 0.12;\n const horizonLen = Math.min(v.width, v.height) * 0.07;\n\n const dx = Math.sin(yawRad) * armLen;\n const dy = -Math.sin(pitchRad) * armLen;\n const cr = Math.cos(rollRad), sr = Math.sin(rollRad);\n const rx = dx * cr - dy * sr;\n const ry = dx * sr + dy * cr;\n\n ctx.strokeStyle = '#00fff0';\n ctx.lineWidth = 2;\n ctx.beginPath();\n ctx.moveTo(fcx, fcy);\n ctx.lineTo(fcx + rx, fcy + ry);\n ctx.stroke();\n ctx.fillStyle = '#00fff0';\n ctx.beginPath();\n ctx.arc(fcx + rx, fcy + ry, 3, 0, Math.PI * 2);\n ctx.fill();\n\n // Roll-only horizon, so head tilt is legible even when looking forward.\n const hx = Math.cos(rollRad) * horizonLen;\n const hy = Math.sin(rollRad) * horizonLen;\n ctx.strokeStyle = 'rgba(0, 200, 200, 0.85)';\n ctx.lineWidth = 1.5;\n ctx.beginPath();\n ctx.moveTo(fcx - hx, fcy - hy);\n ctx.lineTo(fcx + hx, fcy + hy);\n ctx.stroke();\n }\n\n // Numeric overlay\n const fontSize = Math.min(w, h) * 0.025;\n ctx.fillStyle = '#fff';\n ctx.font = `${fontSize}px sans-serif`;\n ctx.textAlign = 'left';\n ctx.fillText(\n face.landmarks.length + ' landmarks | Pitch: ' + hp.pitch.toFixed(1) +\n ' Yaw: ' + hp.yaw.toFixed(1) + ' Roll: ' + hp.roll.toFixed(1),\n w * 0.03, h - fontSize * 0.8\n );\n });\n}\n",
|
|
3258
3273
|
"sceneFile": "face-mesh-demo.scene.js",
|
|
3259
3274
|
"capabilities": {
|
|
3260
3275
|
"video": true
|
|
@@ -3300,7 +3315,7 @@ export const docsApi = {
|
|
|
3300
3315
|
"content": [
|
|
3301
3316
|
{
|
|
3302
3317
|
"type": "text",
|
|
3303
|
-
"markdown": "# Emotion Detection\n\nEmotion detection provides 7 expression scores and 52 ARKit-compatible blendshape coefficients for each detected face. Enable it with [`viji.video.cv.enableEmotionDetection(true)`](../connection/).\n\n## Property Reference\n\nEmotion data appears on each `FaceData` object in `viji.video.cv.faces`:\n\n### Expressions (7 emotions)\n\nEach value is a confidence score from 0 to 1.\n\n| Property | Type | Range | Description |\n|----------|------|-------|-------------|\n| `face.expressions.neutral` | `number` | 0-1 | Neutral expression |\n| `face.expressions.happy` | `number` | 0-1 | Happy / smiling |\n| `face.expressions.sad` | `number` | 0-1 | Sad |\n| `face.expressions.angry` | `number` | 0-1 | Angry |\n| `face.expressions.surprised` | `number` | 0-1 | Surprised |\n| `face.expressions.disgusted` | `number` | 0-1 | Disgusted |\n| `face.expressions.fearful` | `number` | 0-1 | Fearful |\n\n### Blendshapes (52 ARKit coefficients)\n\nThe `face.blendshapes` object contains 52 ARKit-compatible coefficients derived from MediaPipe FaceLandmarker, each ranging from 0 to 1. These provide fine-grained facial muscle control data:\n\n**Brow:** `browDownLeft`, `browDownRight`, `browInnerUp`, `browOuterUpLeft`, `browOuterUpRight`\n\n**Cheek:** `cheekPuff`, `cheekSquintLeft`, `cheekSquintRight`\n\n**Eye:** `eyeBlinkLeft`, `eyeBlinkRight`, `eyeLookDownLeft`, `eyeLookDownRight`, `eyeLookInLeft`, `eyeLookInRight`, `eyeLookOutLeft`, `eyeLookOutRight`, `eyeLookUpLeft`, `eyeLookUpRight`, `eyeSquintLeft`, `eyeSquintRight`, `eyeWideLeft`, `eyeWideRight`\n\n**Jaw:** `jawForward`, `jawLeft`, `jawOpen`, `jawRight`\n\n**Mouth:** `mouthClose`, `mouthDimpleLeft`, `mouthDimpleRight`, `mouthFrownLeft`, `mouthFrownRight`, `mouthFunnel`, `mouthLeft`, `mouthLowerDownLeft`, `mouthLowerDownRight`, `mouthPressLeft`, `mouthPressRight`, `mouthPucker`, `mouthRight`, `mouthRollLower`, `mouthRollUpper`, `mouthShrugLower`, `mouthShrugUpper`, `mouthSmileLeft`, `mouthSmileRight`, `mouthStretchLeft`, `mouthStretchRight`, `mouthUpperUpLeft`, `mouthUpperUpRight`\n\n**Nose & Tongue:** `noseSneerLeft`, `noseSneerRight`, `tongueOut`\n\n## Usage\n\nVideo drawing follows the [aspect-correct pattern](../basics/#drawing-video). This demo uses `'contain'` so the face stays visible at the frame edges.\n\n```javascript\nconst useEmotion = viji.toggle(false, { label: 'Emotion Detection', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n const w = viji.width;\n const h = viji.height;\n\n ctx.fillStyle = '#111';\n ctx.fillRect(0, 0, w, h);\n\n if (useEmotion.value) viji.video.cv.enableEmotionDetection(true);\n else viji.video.cv.enableEmotionDetection(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) return;\n\n const v = videoFit(viji, 'contain');\n ctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n const face = viji.video.cv.faces[0];\n if (!face) return;\n\n const expr = face.expressions;\n const labels = ['neutral', 'happy', 'sad', 'angry', 'surprised', 'disgusted', 'fearful'];\n const values = [expr.neutral, expr.happy, expr.sad, expr.angry, expr.surprised, expr.disgusted, expr.fearful];\n\n const barH = h * 0.04;\n const barW = w * 0.3;\n const x = w * 0.65;\n let y = h * 0.15;\n const fontSize = barH * 0.7;\n\n ctx.font = `${fontSize}px sans-serif`;\n labels.forEach((label, i) => {\n ctx.fillStyle = '#aaa';\n ctx.textAlign = 'right';\n ctx.fillText(label, x - 8, y + barH * 0.75);\n\n ctx.fillStyle = '#333';\n ctx.fillRect(x, y, barW, barH);\n\n ctx.fillStyle = `hsl(${i * 50}, 70%, 55%)`;\n ctx.fillRect(x, y, barW * values[i], barH);\n\n y += barH * 1.8;\n });\n}\n```\n\n**Cost: High.** Emotion detection computes 7 expression scores and 52 blendshape coefficients, requiring significant processing per face.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\n> [!TIP]\n> **Best practice:** Don't enable CV features by default. Instead, expose a toggle parameter so users can activate them on capable devices:\n> ```javascript\n> const useEmotion = viji.toggle(false, { label: 'Enable Emotion Detection', category: 'video' });\n> if (useEmotion.value) {\n> await viji.video.cv.enableEmotionDetection(true);\n> }\n> ```\n\nWhen emotion detection is disabled, all `expressions` values are `0` and all `blendshapes` coefficients are `0`."
|
|
3318
|
+
"markdown": "# Emotion Detection\n\nEmotion detection provides 7 expression scores and 52 ARKit-compatible blendshape coefficients for each detected face. Enable it with [`viji.video.cv.enableEmotionDetection(true)`](../connection/).\n\nEmotion detection populates `face.expressions` and `face.blendshapes`. Both are `null` until emotion detection is enabled — always null-check before reading individual scores. Emotion detection internally loads the face landmarker model, so `face.landmarks` and `face.headPose` are also populated alongside.\n\n`face.bounds`, `face.center`, and `face.confidence` are produced by [Face Detection](../face-detection/), not by emotion detection — enable both if you need a bounding box for positioning overlays.\n\n`viji.video.cv.enableEmotionDetection(true)` is safe to call from module scope (always-on CV) or per-frame inside `render()` gated by a toggle parameter (opt-in CV). The verb is idempotent and reference-counted; per-frame calls are cheap. `await` is optional.\n\n## Property Reference\n\nEmotion data appears on each `FaceData` object in `viji.video.cv.faces`:\n\n### Expressions (7 emotions)\n\nEach value is a confidence score from 0 to 1.\n\n| Property | Type | Range | Description |\n|----------|------|-------|-------------|\n| `face.expressions.neutral` | `number` | 0-1 | Neutral expression |\n| `face.expressions.happy` | `number` | 0-1 | Happy / smiling |\n| `face.expressions.sad` | `number` | 0-1 | Sad |\n| `face.expressions.angry` | `number` | 0-1 | Angry |\n| `face.expressions.surprised` | `number` | 0-1 | Surprised |\n| `face.expressions.disgusted` | `number` | 0-1 | Disgusted |\n| `face.expressions.fearful` | `number` | 0-1 | Fearful |\n\n### Blendshapes (52 ARKit coefficients)\n\nThe `face.blendshapes` object contains 52 ARKit-compatible coefficients derived from MediaPipe FaceLandmarker, each ranging from 0 to 1. These provide fine-grained facial muscle control data:\n\n**Brow:** `browDownLeft`, `browDownRight`, `browInnerUp`, `browOuterUpLeft`, `browOuterUpRight`\n\n**Cheek:** `cheekPuff`, `cheekSquintLeft`, `cheekSquintRight`\n\n**Eye:** `eyeBlinkLeft`, `eyeBlinkRight`, `eyeLookDownLeft`, `eyeLookDownRight`, `eyeLookInLeft`, `eyeLookInRight`, `eyeLookOutLeft`, `eyeLookOutRight`, `eyeLookUpLeft`, `eyeLookUpRight`, `eyeSquintLeft`, `eyeSquintRight`, `eyeWideLeft`, `eyeWideRight`\n\n**Jaw:** `jawForward`, `jawLeft`, `jawOpen`, `jawRight`\n\n**Mouth:** `mouthClose`, `mouthDimpleLeft`, `mouthDimpleRight`, `mouthFrownLeft`, `mouthFrownRight`, `mouthFunnel`, `mouthLeft`, `mouthLowerDownLeft`, `mouthLowerDownRight`, `mouthPressLeft`, `mouthPressRight`, `mouthPucker`, `mouthRight`, `mouthRollLower`, `mouthRollUpper`, `mouthShrugLower`, `mouthShrugUpper`, `mouthSmileLeft`, `mouthSmileRight`, `mouthStretchLeft`, `mouthStretchRight`, `mouthUpperUpLeft`, `mouthUpperUpRight`\n\n**Nose & Tongue:** `noseSneerLeft`, `noseSneerRight`, `tongueOut`\n\n## Usage\n\nVideo drawing follows the [aspect-correct pattern](../basics/#drawing-video). This demo uses `'contain'` so the face stays visible at the frame edges.\n\n```javascript\nconst useEmotion = viji.toggle(false, { label: 'Emotion Detection', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n const w = viji.width;\n const h = viji.height;\n\n ctx.fillStyle = '#111';\n ctx.fillRect(0, 0, w, h);\n\n if (useEmotion.value) viji.video.cv.enableEmotionDetection(true);\n else viji.video.cv.enableEmotionDetection(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) return;\n\n const v = videoFit(viji, 'contain');\n ctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n const face = viji.video.cv.faces[0];\n if (!face) return;\n\n const expr = face.expressions;\n const labels = ['neutral', 'happy', 'sad', 'angry', 'surprised', 'disgusted', 'fearful'];\n const values = [expr.neutral, expr.happy, expr.sad, expr.angry, expr.surprised, expr.disgusted, expr.fearful];\n\n const barH = h * 0.04;\n const barW = w * 0.3;\n const x = w * 0.65;\n let y = h * 0.15;\n const fontSize = barH * 0.7;\n\n ctx.font = `${fontSize}px sans-serif`;\n labels.forEach((label, i) => {\n ctx.fillStyle = '#aaa';\n ctx.textAlign = 'right';\n ctx.fillText(label, x - 8, y + barH * 0.75);\n\n ctx.fillStyle = '#333';\n ctx.fillRect(x, y, barW, barH);\n\n ctx.fillStyle = `hsl(${i * 50}, 70%, 55%)`;\n ctx.fillRect(x, y, barW * values[i], barH);\n\n y += barH * 1.8;\n });\n}\n```\n\n**Cost: High.** Emotion detection computes 7 expression scores and 52 blendshape coefficients, requiring significant processing per face.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\n> [!TIP]\n> **Best practice:** Don't enable CV features by default. Instead, expose a toggle parameter so users can activate them on capable devices:\n> ```javascript\n> const useEmotion = viji.toggle(false, { label: 'Enable Emotion Detection', category: 'video' });\n> if (useEmotion.value) {\n> await viji.video.cv.enableEmotionDetection(true);\n> }\n> ```\n\nWhen emotion detection is disabled, all `expressions` values are `0` and all `blendshapes` coefficients are `0`."
|
|
3304
3319
|
},
|
|
3305
3320
|
{
|
|
3306
3321
|
"type": "live-example",
|
|
@@ -3346,7 +3361,7 @@ export const docsApi = {
|
|
|
3346
3361
|
"content": [
|
|
3347
3362
|
{
|
|
3348
3363
|
"type": "text",
|
|
3349
|
-
"markdown": "# Hand Tracking\n\nHand tracking provides 21-point landmarks, palm position, bounding boxes, and ML-based gesture recognition for up to two hands. Enable it with [`viji.video.cv.enableHandTracking(true)`](../connection/).\n\n## Property Reference\n\nResults appear in `viji.video.cv.hands`: an array of up to 2 `HandData` objects. When no hands are detected or the feature is disabled, the array is empty.\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `hand.id` | `number` | Index-based hand identifier (0, 1) |\n| `hand.handedness` | `'left' \\| 'right'` | Which hand (always lowercase) |\n| `hand.confidence` | `number` | Detection confidence (0-1) |\n| `hand.bounds` | `{ x, y, width, height }` | Bounding box, normalized 0-1 |\n| `hand.landmarks` | `{ x, y, z }[]` | 21 MediaPipe hand landmarks, normalized 0-1 |\n| `hand.palm` | `{ x, y, z }` | Palm center: `landmarks[9]` (middle finger MCP) |\n| `hand.gestures` | object | 7 ML gesture confidence scores (0-1 each) |\n\n### Gestures\n\nGestures are classified by MediaPipe's GestureRecognizer ML model, not computed from geometric heuristics. Each value is a confidence score from 0 to 1.\n\n| Property | Gesture |\n|----------|---------|\n| `hand.gestures.fist` | Closed fist |\n| `hand.gestures.openPalm` | Open hand |\n| `hand.gestures.peace` | Victory / peace sign |\n| `hand.gestures.thumbsUp` | Thumbs up |\n| `hand.gestures.thumbsDown` | Thumbs down |\n| `hand.gestures.pointing` | Pointing up |\n| `hand.gestures.iLoveYou` | ASL I-love-you sign |\n\n## Usage\n\nVideo drawing follows the [aspect-correct pattern](../basics/#drawing-video). This demo uses `'contain'` so hands near the frame edges stay visible.\n\n```javascript\nconst useHands = viji.toggle(false, { label: 'Hand Tracking', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n const w = viji.width;\n const h = viji.height;\n\n ctx.fillStyle = '#111';\n ctx.fillRect(0, 0, w, h);\n\n if (useHands.value) viji.video.cv.enableHandTracking(true);\n else viji.video.cv.enableHandTracking(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) return;\n\n const v = videoFit(viji, 'contain');\n ctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n viji.video.cv.hands.forEach(hand => {\n const color = hand.handedness === 'left' ? '#ff9ff3' : '#54a0ff';\n ctx.fillStyle = color;\n\n hand.landmarks.forEach(pt => {\n ctx.beginPath();\n ctx.arc(v.x + pt.x * v.width, v.y + pt.y * v.height, 3, 0, Math.PI * 2);\n ctx.fill();\n });\n\n ctx.beginPath();\n ctx.arc(v.x + hand.palm.x * v.width, v.y + hand.palm.y * v.height, 8, 0, Math.PI * 2);\n ctx.strokeStyle = color;\n ctx.lineWidth = 2;\n ctx.stroke();\n\n const g = hand.gestures;\n const gestures = [\n ['fist', g.fist], ['open', g.openPalm], ['peace', g.peace],\n ['thumbsUp', g.thumbsUp], ['pointing', g.pointing]\n ];\n const top = gestures.reduce((a, b) => b[1] > a[1] ? b : a);\n if (top[1] > 0.5) {\n ctx.fillStyle = '#fff';\n ctx.font = `${Math.min(w, h) * 0.03}px sans-serif`;\n ctx.textAlign = 'center';\n ctx.fillText(top[0], v.x + hand.palm.x * v.width, v.y + hand.bounds.y * v.height - 8);\n }\n });\n}\n```\n\n**Cost: Medium.** Hand tracking processes up to 2 hands with 21 landmarks each, plus ML gesture classification.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\n> [!TIP]\n> **Best practice:** Don't enable CV features by default. Instead, expose a toggle parameter so users can activate them on capable devices:\n> ```javascript\n> const useHands = viji.toggle(false, { label: 'Enable Hand Tracking', category: 'video' });\n> if (useHands.value) {\n> await viji.video.cv.enableHandTracking(true);\n> }\n> ```\n\nWhen hand tracking is disabled or no hands are visible, `viji.video.cv.hands` becomes an empty array `[]`."
|
|
3364
|
+
"markdown": "# Hand Tracking\n\nHand tracking provides 21-point landmarks, palm position, bounding boxes, and ML-based gesture recognition for up to two hands. Enable it with [`viji.video.cv.enableHandTracking(true)`](../connection/).\n\n`viji.video.cv.enableHandTracking(true)` is safe to call from module scope (always-on CV) or per-frame inside `render()` gated by a toggle parameter (opt-in CV). The verb is idempotent and reference-counted; per-frame calls are cheap. `await` is optional.\n\n## Property Reference\n\nResults appear in `viji.video.cv.hands`: an array of up to 2 `HandData` objects. When no hands are detected or the feature is disabled, the array is empty.\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `hand.id` | `number` | Index-based hand identifier (0, 1) |\n| `hand.handedness` | `'left' \\| 'right'` | Which hand (always lowercase) |\n| `hand.confidence` | `number` | Detection confidence (0-1) |\n| `hand.bounds` | `{ x, y, width, height }` | Bounding box, normalized 0-1 |\n| `hand.landmarks` | `{ x, y, z }[]` | 21 MediaPipe hand landmarks, normalized 0-1 |\n| `hand.palm` | `{ x, y, z }` | Palm center: `landmarks[9]` (middle finger MCP) |\n| `hand.gestures` | object | 7 ML gesture confidence scores (0-1 each) |\n\n### Gestures\n\nGestures are classified by MediaPipe's GestureRecognizer ML model, not computed from geometric heuristics. Each value is a confidence score from 0 to 1.\n\n| Property | Gesture |\n|----------|---------|\n| `hand.gestures.fist` | Closed fist |\n| `hand.gestures.openPalm` | Open hand |\n| `hand.gestures.peace` | Victory / peace sign |\n| `hand.gestures.thumbsUp` | Thumbs up |\n| `hand.gestures.thumbsDown` | Thumbs down |\n| `hand.gestures.pointing` | Pointing up |\n| `hand.gestures.iLoveYou` | ASL I-love-you sign |\n\n## Usage\n\nVideo drawing follows the [aspect-correct pattern](../basics/#drawing-video). This demo uses `'contain'` so hands near the frame edges stay visible.\n\n```javascript\nconst useHands = viji.toggle(false, { label: 'Hand Tracking', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n const w = viji.width;\n const h = viji.height;\n\n ctx.fillStyle = '#111';\n ctx.fillRect(0, 0, w, h);\n\n if (useHands.value) viji.video.cv.enableHandTracking(true);\n else viji.video.cv.enableHandTracking(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) return;\n\n const v = videoFit(viji, 'contain');\n ctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n viji.video.cv.hands.forEach(hand => {\n const color = hand.handedness === 'left' ? '#ff9ff3' : '#54a0ff';\n ctx.fillStyle = color;\n\n hand.landmarks.forEach(pt => {\n ctx.beginPath();\n ctx.arc(v.x + pt.x * v.width, v.y + pt.y * v.height, 3, 0, Math.PI * 2);\n ctx.fill();\n });\n\n ctx.beginPath();\n ctx.arc(v.x + hand.palm.x * v.width, v.y + hand.palm.y * v.height, 8, 0, Math.PI * 2);\n ctx.strokeStyle = color;\n ctx.lineWidth = 2;\n ctx.stroke();\n\n const g = hand.gestures;\n const gestures = [\n ['fist', g.fist], ['open', g.openPalm], ['peace', g.peace],\n ['thumbsUp', g.thumbsUp], ['pointing', g.pointing]\n ];\n const top = gestures.reduce((a, b) => b[1] > a[1] ? b : a);\n if (top[1] > 0.5) {\n ctx.fillStyle = '#fff';\n ctx.font = `${Math.min(w, h) * 0.03}px sans-serif`;\n ctx.textAlign = 'center';\n ctx.fillText(top[0], v.x + hand.palm.x * v.width, v.y + hand.bounds.y * v.height - 8);\n }\n });\n}\n```\n\n**Cost: Medium.** Hand tracking processes up to 2 hands with 21 landmarks each, plus ML gesture classification.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\n> [!TIP]\n> **Best practice:** Don't enable CV features by default. Instead, expose a toggle parameter so users can activate them on capable devices:\n> ```javascript\n> const useHands = viji.toggle(false, { label: 'Enable Hand Tracking', category: 'video' });\n> if (useHands.value) {\n> await viji.video.cv.enableHandTracking(true);\n> }\n> ```\n\nWhen hand tracking is disabled or no hands are visible, `viji.video.cv.hands` becomes an empty array `[]`."
|
|
3350
3365
|
},
|
|
3351
3366
|
{
|
|
3352
3367
|
"type": "live-example",
|
|
@@ -3392,7 +3407,7 @@ export const docsApi = {
|
|
|
3392
3407
|
"content": [
|
|
3393
3408
|
{
|
|
3394
3409
|
"type": "text",
|
|
3395
|
-
"markdown": "# Pose Detection\n\nPose detection provides 33 body landmarks using MediaPipe's BlazePose model, with named groups for easy access to face, torso, arms, and legs. Enable it with [`viji.video.cv.enablePoseDetection(true)`](../connection/).\n\n## Property Reference\n\nResults appear in `viji.video.cv.pose`: a single `PoseData` object, or `null` when no pose is detected or the feature is disabled.\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `pose.confidence` | `number` | Average landmark visibility (0-1) |\n| `pose.landmarks` | `{ x, y, z, visibility }[]` | 33 BlazePose points, normalized 0-1 |\n| `pose.face` | `{ x, y }[]` | Face region landmarks (indices 0-10) |\n| `pose.torso` | `{ x, y }[]` | Torso landmarks (indices 11, 12, 23, 24) |\n| `pose.leftArm` | `{ x, y }[]` | Left arm (indices 11, 13, 15) |\n| `pose.rightArm` | `{ x, y }[]` | Right arm (indices 12, 14, 16) |\n| `pose.leftLeg` | `{ x, y }[]` | Left leg (indices 23, 25, 27, 29, 31) |\n| `pose.rightLeg` | `{ x, y }[]` | Right leg (indices 24, 26, 28, 30, 32) |\n\n### Key Landmark Indices\n\n| Index | Landmark | Index | Landmark |\n|-------|----------|-------|----------|\n| 0 | Nose | 15 | Left wrist |\n| 11 | Left shoulder | 16 | Right wrist |\n| 12 | Right shoulder | 23 | Left hip |\n| 13 | Left elbow | 24 | Right hip |\n| 14 | Right elbow | 25 | Left knee |\n| 27 | Left ankle | 26 | Right knee |\n| 28 | Right ankle | | |\n\nEach landmark has a `visibility` score (0-1) indicating how confident the model is that this point is visible.\n\n## Usage\n\nVideo drawing follows the [aspect-correct pattern](../basics/#drawing-video). This demo uses `'contain'` so the full body stays visible at the frame edges.\n\n```javascript\nconst usePose = viji.toggle(false, { label: 'Pose Detection', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n const w = viji.width;\n const h = viji.height;\n\n ctx.fillStyle = '#111';\n ctx.fillRect(0, 0, w, h);\n\n if (usePose.value) viji.video.cv.enablePoseDetection(true);\n else viji.video.cv.enablePoseDetection(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) return;\n\n const v = videoFit(viji, 'contain');\n ctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n const pose = viji.video.cv.pose;\n if (!pose) return;\n\n ctx.fillStyle = '#ff6b6b';\n pose.landmarks.forEach(pt => {\n if (pt.visibility > 0.5) {\n ctx.beginPath();\n ctx.arc(v.x + pt.x * v.width, v.y + pt.y * v.height, 4, 0, Math.PI * 2);\n ctx.fill();\n }\n });\n\n ctx.strokeStyle = '#ff6b6b';\n ctx.lineWidth = 2;\n const drawGroup = (group) => {\n if (group.length < 2) return;\n ctx.beginPath();\n ctx.moveTo(v.x + group[0].x * v.width, v.y + group[0].y * v.height);\n for (let i = 1; i < group.length; i++) {\n ctx.lineTo(v.x + group[i].x * v.width, v.y + group[i].y * v.height);\n }\n ctx.stroke();\n };\n\n drawGroup(pose.leftArm);\n drawGroup(pose.rightArm);\n drawGroup(pose.leftLeg);\n drawGroup(pose.rightLeg);\n drawGroup(pose.torso);\n}\n```\n\n**Cost: Medium.** Pose detection processes 33 body landmarks with visibility scores.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\n> [!TIP]\n> **Best practice:** Don't enable CV features by default. Instead, expose a toggle parameter so users can activate them on capable devices:\n> ```javascript\n> const usePose = viji.toggle(false, { label: 'Enable Pose Detection', category: 'video' });\n> if (usePose.value) {\n> await viji.video.cv.enablePoseDetection(true);\n> }\n> ```\n\nWhen pose detection is disabled or no body is visible, `viji.video.cv.pose` becomes `null`."
|
|
3410
|
+
"markdown": "# Pose Detection\n\nPose detection provides 33 body landmarks using MediaPipe's BlazePose model, with named groups for easy access to face, torso, arms, and legs. Enable it with [`viji.video.cv.enablePoseDetection(true)`](../connection/).\n\n`viji.video.cv.enablePoseDetection(true)` is safe to call from module scope (always-on CV) or per-frame inside `render()` gated by a toggle parameter (opt-in CV). The verb is idempotent and reference-counted; per-frame calls are cheap. `await` is optional.\n\n## Property Reference\n\nResults appear in `viji.video.cv.pose`: a single `PoseData` object, or `null` when no pose is detected or the feature is disabled.\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `pose.confidence` | `number` | Average landmark visibility (0-1) |\n| `pose.landmarks` | `{ x, y, z, visibility }[]` | 33 BlazePose points, normalized 0-1 |\n| `pose.face` | `{ x, y }[]` | Face region landmarks (indices 0-10) |\n| `pose.torso` | `{ x, y }[]` | Torso landmarks (indices 11, 12, 23, 24) |\n| `pose.leftArm` | `{ x, y }[]` | Left arm (indices 11, 13, 15) |\n| `pose.rightArm` | `{ x, y }[]` | Right arm (indices 12, 14, 16) |\n| `pose.leftLeg` | `{ x, y }[]` | Left leg (indices 23, 25, 27, 29, 31) |\n| `pose.rightLeg` | `{ x, y }[]` | Right leg (indices 24, 26, 28, 30, 32) |\n\n### Key Landmark Indices\n\n| Index | Landmark | Index | Landmark |\n|-------|----------|-------|----------|\n| 0 | Nose | 15 | Left wrist |\n| 11 | Left shoulder | 16 | Right wrist |\n| 12 | Right shoulder | 23 | Left hip |\n| 13 | Left elbow | 24 | Right hip |\n| 14 | Right elbow | 25 | Left knee |\n| 27 | Left ankle | 26 | Right knee |\n| 28 | Right ankle | | |\n\nEach landmark has a `visibility` score (0-1) indicating how confident the model is that this point is visible.\n\n## Usage\n\nVideo drawing follows the [aspect-correct pattern](../basics/#drawing-video). This demo uses `'contain'` so the full body stays visible at the frame edges.\n\n```javascript\nconst usePose = viji.toggle(false, { label: 'Pose Detection', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n const w = viji.width;\n const h = viji.height;\n\n ctx.fillStyle = '#111';\n ctx.fillRect(0, 0, w, h);\n\n if (usePose.value) viji.video.cv.enablePoseDetection(true);\n else viji.video.cv.enablePoseDetection(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) return;\n\n const v = videoFit(viji, 'contain');\n ctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n const pose = viji.video.cv.pose;\n if (!pose) return;\n\n ctx.fillStyle = '#ff6b6b';\n pose.landmarks.forEach(pt => {\n if (pt.visibility > 0.5) {\n ctx.beginPath();\n ctx.arc(v.x + pt.x * v.width, v.y + pt.y * v.height, 4, 0, Math.PI * 2);\n ctx.fill();\n }\n });\n\n ctx.strokeStyle = '#ff6b6b';\n ctx.lineWidth = 2;\n const drawGroup = (group) => {\n if (group.length < 2) return;\n ctx.beginPath();\n ctx.moveTo(v.x + group[0].x * v.width, v.y + group[0].y * v.height);\n for (let i = 1; i < group.length; i++) {\n ctx.lineTo(v.x + group[i].x * v.width, v.y + group[i].y * v.height);\n }\n ctx.stroke();\n };\n\n drawGroup(pose.leftArm);\n drawGroup(pose.rightArm);\n drawGroup(pose.leftLeg);\n drawGroup(pose.rightLeg);\n drawGroup(pose.torso);\n}\n```\n\n**Cost: Medium.** Pose detection processes 33 body landmarks with visibility scores.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\n> [!TIP]\n> **Best practice:** Don't enable CV features by default. Instead, expose a toggle parameter so users can activate them on capable devices:\n> ```javascript\n> const usePose = viji.toggle(false, { label: 'Enable Pose Detection', category: 'video' });\n> if (usePose.value) {\n> await viji.video.cv.enablePoseDetection(true);\n> }\n> ```\n\nWhen pose detection is disabled or no body is visible, `viji.video.cv.pose` becomes `null`."
|
|
3396
3411
|
},
|
|
3397
3412
|
{
|
|
3398
3413
|
"type": "live-example",
|
|
@@ -3443,7 +3458,7 @@ export const docsApi = {
|
|
|
3443
3458
|
"content": [
|
|
3444
3459
|
{
|
|
3445
3460
|
"type": "text",
|
|
3446
|
-
"markdown": "# Body Segmentation\n\nBody segmentation provides a per-pixel mask that separates the person from the background. Enable it with [`viji.video.cv.enableBodySegmentation(true)`](../connection/).\n\n## Property Reference\n\nResults appear in `viji.video.cv.segmentation`: a `SegmentationData` object, or `null` when no mask is available or the feature is disabled.\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `segmentation.mask` | `Uint8Array` | Per-pixel mask: 0 = background, 1 = person |\n| `segmentation.width` | `number` | Mask width in pixels |\n| `segmentation.height` | `number` | Mask height in pixels |\n\n### Mask Format\n\nThe mask is a flat `Uint8Array` with `width * height` elements. Each element is `0` (background) or `1` (person). The mask dimensions may differ from the video frame dimensions: they reflect the ML model's output resolution.\n\nTo count person pixels or calculate presence ratio, iterate the mask array manually:\n\n```javascript\nlet personPixels = 0;\nfor (let i = 0; i < segmentation.mask.length; i++) {\n if (segmentation.mask[i] > 0) personPixels++;\n}\nconst personRatio = personPixels / segmentation.mask.length;\n```\n\n## Usage\n\n```javascript\nconst useSeg = viji.toggle(false, { label: 'Body Segmentation', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n const w = viji.width;\n const h = viji.height;\n\n ctx.fillStyle = '#111';\n ctx.fillRect(0, 0, w, h);\n\n if (useSeg.value) viji.video.cv.enableBodySegmentation(true);\n else viji.video.cv.enableBodySegmentation(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) return;\n\n // 'contain' so the body silhouette stays visible at frame edges.\n const v = videoFit(viji, 'contain');\n ctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n const seg = viji.video.cv.segmentation;\n if (!seg) return;\n\n let personPixels = 0;\n for (let i = 0; i < seg.mask.length; i++) {\n if (seg.mask[i] > 0) personPixels++;\n }\n const personRatio = personPixels / seg.mask.length;\n\n if (personRatio > 0.05) {\n ctx.shadowBlur = 30 * personRatio;\n ctx.shadowColor = `hsl(${170 + personRatio * 60}, 80%, 60%)`;\n ctx.strokeStyle = `hsla(${170 + personRatio * 60}, 80%, 60%, 0.6)`;\n ctx.lineWidth = 4;\n ctx.strokeRect(v.x, v.y, v.width, v.height);\n ctx.shadowBlur = 0;\n }\n\n ctx.fillStyle = '#fff';\n ctx.font = `${Math.min(w, h) * 0.03}px sans-serif`;\n ctx.textAlign = 'left';\n ctx.fillText(\n 'Person: ' + (personRatio * 100).toFixed(0) + '% (' + seg.width + 'x' + seg.height + ' mask)',\n w * 0.03, h - Math.min(w, h) * 0.03\n );\n}\n```\n\n**Cost: High.** Body segmentation produces a per-pixel mask with a large tensor output, making it one of the most expensive CV features.\n\n## Pairing Frame and Mask\n\nBody segmentation splits into two camps with different defaults.\n\n**Effects that composite the mask onto the displayed body** (background replacement, silhouette outlining, recolor-the-person effects, anything where the mask edge must follow the actual body contour): use [`viji.video.cv.analysedFrame`](../basics/), the frame the mask was computed from. If you composite against `currentFrame` instead, the silhouette edge drifts off the moving body and the cutout shows visible bleed/holes during motion. The cost is that the displayed image stutters or holds briefly between inferences.\n\n```javascript\n// Background replacement: analysedFrame is required\nconst v = videoFit(viji, 'contain'); // see basics for the helper\nconst frame = viji.video.cv.analysedFrame ?? viji.video.currentFrame;\nctx.drawImage(frame, v.x, v.y, v.width, v.height);\n// composite the mask against the same frame, using v to position mask pixels...\n```\n\n**Effects that use the mask as a generative input without compositing it onto the displayed body** (driving particle systems from the silhouette, feeding the mask into a fluid/light/distortion effect, scenes that don't display the camera at all): `viji.video.currentFrame` (or skipping the displayed video altogether) is usually fine. The exact mask edge does not need to match a specific displayed frame, and reaching for `analysedFrame` would freeze the displayed video between inferences for no visual benefit.\n\n`analysedFrame` is `null` until the first segmentation result lands. The `?? currentFrame` fallback covers that startup window; expect a brief visual hitch when the first inference lands and the source switches.\n\nSee [Best Practices](/getting-started/best-practices/#currentframe-vs-analysedframe-pick-the-right-trade-off) for the cross-renderer trade-off discussion.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\n> [!TIP]\n> **Best practice:** Don't enable CV features by default. Instead, expose a toggle parameter so users can activate them on capable devices:\n> ```javascript\n> const useSeg = viji.toggle(false, { label: 'Enable Body Segmentation', category: 'video' });\n> if (useSeg.value) {\n> await viji.video.cv.enableBodySegmentation(true);\n> }\n> ```\n\nWhen body segmentation is disabled or no body is visible, `viji.video.cv.segmentation` becomes `null`."
|
|
3461
|
+
"markdown": "# Body Segmentation\n\nBody segmentation provides a per-pixel mask that separates the person from the background. Enable it with [`viji.video.cv.enableBodySegmentation(true)`](../connection/).\n\n`viji.video.cv.enableBodySegmentation(true)` is safe to call from module scope (always-on CV) or per-frame inside `render()` gated by a toggle parameter (opt-in CV). The verb is idempotent and reference-counted; per-frame calls are cheap. `await` is optional.\n\n## Property Reference\n\nResults appear in `viji.video.cv.segmentation`: a `SegmentationData` object, or `null` when no mask is available or the feature is disabled.\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `segmentation.mask` | `Uint8Array` | Per-pixel mask: 0 = background, 1 = person |\n| `segmentation.width` | `number` | Mask width in pixels |\n| `segmentation.height` | `number` | Mask height in pixels |\n\n### Mask Format\n\nThe mask is a flat `Uint8Array` with `width * height` elements. Each element is `0` (background) or `1` (person). The mask dimensions may differ from the video frame dimensions: they reflect the ML model's output resolution.\n\nTo count person pixels or calculate presence ratio, iterate the mask array manually:\n\n```javascript\nlet personPixels = 0;\nfor (let i = 0; i < segmentation.mask.length; i++) {\n if (segmentation.mask[i] > 0) personPixels++;\n}\nconst personRatio = personPixels / segmentation.mask.length;\n```\n\n## Usage\n\n```javascript\nconst useSeg = viji.toggle(false, { label: 'Body Segmentation', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n const w = viji.width;\n const h = viji.height;\n\n ctx.fillStyle = '#111';\n ctx.fillRect(0, 0, w, h);\n\n if (useSeg.value) viji.video.cv.enableBodySegmentation(true);\n else viji.video.cv.enableBodySegmentation(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) return;\n\n // 'contain' so the body silhouette stays visible at frame edges.\n const v = videoFit(viji, 'contain');\n ctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n const seg = viji.video.cv.segmentation;\n if (!seg) return;\n\n let personPixels = 0;\n for (let i = 0; i < seg.mask.length; i++) {\n if (seg.mask[i] > 0) personPixels++;\n }\n const personRatio = personPixels / seg.mask.length;\n\n if (personRatio > 0.05) {\n ctx.shadowBlur = 30 * personRatio;\n ctx.shadowColor = `hsl(${170 + personRatio * 60}, 80%, 60%)`;\n ctx.strokeStyle = `hsla(${170 + personRatio * 60}, 80%, 60%, 0.6)`;\n ctx.lineWidth = 4;\n ctx.strokeRect(v.x, v.y, v.width, v.height);\n ctx.shadowBlur = 0;\n }\n\n ctx.fillStyle = '#fff';\n ctx.font = `${Math.min(w, h) * 0.03}px sans-serif`;\n ctx.textAlign = 'left';\n ctx.fillText(\n 'Person: ' + (personRatio * 100).toFixed(0) + '% (' + seg.width + 'x' + seg.height + ' mask)',\n w * 0.03, h - Math.min(w, h) * 0.03\n );\n}\n```\n\n**Cost: High.** Body segmentation produces a per-pixel mask with a large tensor output, making it one of the most expensive CV features.\n\n## Pairing Frame and Mask\n\nBody segmentation splits into two camps with different defaults.\n\n**Effects that composite the mask onto the displayed body** (background replacement, silhouette outlining, recolor-the-person effects, anything where the mask edge must follow the actual body contour): use [`viji.video.cv.analysedFrame`](../basics/), the frame the mask was computed from. If you composite against `currentFrame` instead, the silhouette edge drifts off the moving body and the cutout shows visible bleed/holes during motion. The cost is that the displayed image stutters or holds briefly between inferences.\n\n```javascript\n// Background replacement: analysedFrame is required\nconst v = videoFit(viji, 'contain'); // see basics for the helper\nconst frame = viji.video.cv.analysedFrame ?? viji.video.currentFrame;\nctx.drawImage(frame, v.x, v.y, v.width, v.height);\n// composite the mask against the same frame, using v to position mask pixels...\n```\n\n**Effects that use the mask as a generative input without compositing it onto the displayed body** (driving particle systems from the silhouette, feeding the mask into a fluid/light/distortion effect, scenes that don't display the camera at all): `viji.video.currentFrame` (or skipping the displayed video altogether) is usually fine. The exact mask edge does not need to match a specific displayed frame, and reaching for `analysedFrame` would freeze the displayed video between inferences for no visual benefit.\n\n`analysedFrame` is `null` until the first segmentation result lands. The `?? currentFrame` fallback covers that startup window; expect a brief visual hitch when the first inference lands and the source switches.\n\nSee [Best Practices](/getting-started/best-practices/#currentframe-vs-analysedframe-pick-the-right-trade-off) for the cross-renderer trade-off discussion.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\n> [!TIP]\n> **Best practice:** Don't enable CV features by default. Instead, expose a toggle parameter so users can activate them on capable devices:\n> ```javascript\n> const useSeg = viji.toggle(false, { label: 'Enable Body Segmentation', category: 'video' });\n> if (useSeg.value) {\n> await viji.video.cv.enableBodySegmentation(true);\n> }\n> ```\n\nWhen body segmentation is disabled or no body is visible, `viji.video.cv.segmentation` becomes `null`."
|
|
3447
3462
|
},
|
|
3448
3463
|
{
|
|
3449
3464
|
"type": "live-example",
|
|
@@ -4150,7 +4165,7 @@ export const docsApi = {
|
|
|
4150
4165
|
},
|
|
4151
4166
|
"native-ext-audio": {
|
|
4152
4167
|
"id": "native-ext-audio",
|
|
4153
|
-
"title": "
|
|
4168
|
+
"title": "native-ext-audio",
|
|
4154
4169
|
"description": "Lightweight audio analysis from externally connected devices: volume, bands, spectral features, and raw FFT/waveform via AudioStreamAPI.",
|
|
4155
4170
|
"headings": [
|
|
4156
4171
|
{
|
|
@@ -4167,7 +4182,7 @@ export const docsApi = {
|
|
|
4167
4182
|
"content": [
|
|
4168
4183
|
{
|
|
4169
4184
|
"type": "text",
|
|
4170
|
-
"markdown": "# Device Audio\n\nEach entry in `viji.devices` may expose **`device.audio`**: an [`AudioStreamAPI`](../../audio/) (or `null` when the host has not attached an audio source for that device).\n\n## Behavior\n\n- Check **`device.audio?.isConnected`** before reading values.\n- **Lightweight subset only:** `isConnected`, `volume` (`current` / `peak` / `smoothed`), `bands` (`low` / `lowMid` / `mid` / `highMid` / `high` plus each `*Smoothed` sibling), `spectral` (`brightness` / `flatness`), `getFrequencyData()`, `getWaveform()`.\n- **Not available:** beat energy, triggers, BPM, or beat events (those exist only on the main [`viji.audio`](../../audio/) stream).\n\n## Related\n\n- [External Devices: Overview](../)\n- [Native Audio](../../audio/)\n- [Shader: Audio stream uniforms](/shader/api-reference#audio-streams)"
|
|
4185
|
+
"markdown": "# Device Audio\r\n\r\nEach entry in `viji.devices` may expose **`device.audio`**: an [`AudioStreamAPI`](../../audio/) (or `null` when the host has not attached an audio source for that device).\r\n\r\n## Behavior\r\n\r\n- Check **`device.audio?.isConnected`** before reading values.\r\n- **Lightweight subset only:** `isConnected`, `volume` (`current` / `peak` / `smoothed`), `bands` (`low` / `lowMid` / `mid` / `highMid` / `high` plus each `*Smoothed` sibling), `spectral` (`brightness` / `flatness`), `getFrequencyData()`, `getWaveform()`.\r\n- **Not available:** beat energy, triggers, BPM, or beat events (those exist only on the main [`viji.audio`](../../audio/) stream).\r\n\r\n## Related\r\n\r\n- [External Devices: Overview](../)\r\n- [Native Audio](../../audio/)\r\n- [Shader: Audio stream uniforms](/shader/api-reference#audio-streams)"
|
|
4171
4186
|
}
|
|
4172
4187
|
]
|
|
4173
4188
|
},
|
|
@@ -5802,6 +5817,11 @@ export const docsApi = {
|
|
|
5802
5817
|
"level": 2,
|
|
5803
5818
|
"text": "CV Control API"
|
|
5804
5819
|
},
|
|
5820
|
+
{
|
|
5821
|
+
"id": "where-to-call-enable",
|
|
5822
|
+
"level": 3,
|
|
5823
|
+
"text": "Where to call enable*"
|
|
5824
|
+
},
|
|
5805
5825
|
{
|
|
5806
5826
|
"id": "guard-pattern",
|
|
5807
5827
|
"level": 2,
|
|
@@ -5821,7 +5841,7 @@ export const docsApi = {
|
|
|
5821
5841
|
"content": [
|
|
5822
5842
|
{
|
|
5823
5843
|
"type": "text",
|
|
5824
|
-
"markdown": "# Connection & Lifecycle\n\nThe `viji.video.isConnected` property indicates whether the host application has provided an active video stream. All other video and CV properties depend on this: when disconnected, they hold default values. The video API is identical to the [Native renderer](../../../native/video/connection/).\n\n## Property Reference\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `viji.video.isConnected` | `boolean` | `true` when a video stream is active |\n| `viji.video.currentFrame` | `OffscreenCanvas \\| null` | Current video frame: drawable with `p5.image()` |\n| `viji.video.frameWidth` | `number` | Video frame width in pixels |\n| `viji.video.frameHeight` | `number` | Video frame height in pixels |\n| `viji.video.frameRate` | `number` | Video frame rate (Hz) |\n| `viji.video.getFrameData()` | `ImageData \\| null` | Raw pixel data for per-pixel analysis |\n\n> [!TIP]\n> Use [`viji.video.currentFrame`](../basics/) for drawing video with `p5.image()` (fast, GPU-friendly). Use `viji.video.getFrameData()` only when you need per-pixel access: it is much slower as it reads back pixel data.\n\n## CV Control API\n\nThe `viji.video.cv` object provides methods to enable and disable individual CV features. All methods accept a boolean parameter.\n\n| Method |
|
|
5844
|
+
"markdown": "# Connection & Lifecycle\n\nThe `viji.video.isConnected` property indicates whether the host application has provided an active video stream. All other video and CV properties depend on this: when disconnected, they hold default values. The video API is identical to the [Native renderer](../../../native/video/connection/).\n\n## Property Reference\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `viji.video.isConnected` | `boolean` | `true` when a video stream is active |\n| `viji.video.currentFrame` | `OffscreenCanvas \\| null` | Current video frame: drawable with `p5.image()` |\n| `viji.video.frameWidth` | `number` | Video frame width in pixels |\n| `viji.video.frameHeight` | `number` | Video frame height in pixels |\n| `viji.video.frameRate` | `number` | Video frame rate (Hz) |\n| `viji.video.getFrameData()` | `ImageData \\| null` | Raw pixel data for per-pixel analysis |\n\n> [!TIP]\n> Use [`viji.video.currentFrame`](../basics/) for drawing video with `p5.image()` (fast, GPU-friendly). Use `viji.video.getFrameData()` only when you need per-pixel access: it is much slower as it reads back pixel data.\n\n## CV Control API\n\nThe `viji.video.cv` object provides methods to enable and disable individual CV features. All methods accept a boolean parameter and return a `Promise<void>` that resolves once the underlying pipeline has finished switching.\n\n| Method | Populates these `face` / `viji.video.cv.*` fields |\n|--------|---------|\n| `enableFaceDetection(enabled)` | `face.bounds`, `face.center`, `face.confidence` |\n| `enableFaceMesh(enabled)` | `face.landmarks` (468 pts), `face.headPose` |\n| `enableEmotionDetection(enabled)` | `face.expressions` (7), `face.blendshapes` (52) — internally also loads the face landmarker, so `face.landmarks` and `face.headPose` are populated |\n| `enableHandTracking(enabled)` | `viji.video.cv.hands[]` (up to 2, landmarks + palm + bounds + gestures) |\n| `enablePoseDetection(enabled)` | `viji.video.cv.pose` (33 BlazePose landmarks + body-part groups) |\n| `enableBodySegmentation(enabled)` | `viji.video.cv.segmentation` (per-pixel mask) |\n\nEach feature is independent: enabling one does NOT populate fields owned by another. If your scene needs both face bounds AND head pose, enable face detection AND face mesh. The verbs are idempotent and reference-counted, so calling them many times is cheap.\n\n### Where to call enable*\n\n`viji.video.cv.enableX(true)` is safe to call from either:\n\n- **Module scope** — for scenes where the CV feature is always required (a face filter that does not work without face mesh).\n- **Per-frame inside `render()`** — the canonical pattern for opt-in CV gated by a `viji.toggle(...)` parameter. The host UI controls activation; per-frame calls are cheap.\n\n`await` is optional. Without it, the MediaPipe model loads asynchronously while the render loop runs; the corresponding data fields populate as soon as the first inference returns. With `await` you can sequence post-load code.\n\n> [!WARNING]\n> Each active CV feature consumes its own WebGL context for MediaPipe inference. Browsers typically allow 8-16 active WebGL contexts. Enabling many CV features simultaneously on lower-end hardware can cause context eviction and break the scene's own rendering. Enable only the features you need; prefer the toggle-gated per-frame pattern for opt-in CV.\n\n| Method | Returns | Description |\n|--------|---------|-------------|\n| `getActiveFeatures()` | `CVFeature[]` | Array of currently active feature strings |\n| `isProcessing()` | `boolean` | `true` if CV worker is actively processing frames |\n\n## Guard Pattern\n\nAlways check `isConnected` and `currentFrame` before using video data. Video drawing follows the [aspect-correct pattern](../basics/#drawing-video).\n\n```javascript\n// @renderer p5\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji, p5) {\n p5.background(17);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) {\n p5.fill(80);\n p5.textAlign(p5.CENTER, p5.CENTER);\n p5.textSize(Math.min(viji.width, viji.height) * 0.04);\n p5.text('No video connected', viji.width / 2, viji.height / 2);\n return;\n }\n\n const v = videoFit(viji);\n p5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n}\n```\n\n## Default Values\n\nWhen `isConnected` is `false`, all video properties hold these values:\n\n| Property | Default |\n|----------|---------|\n| `currentFrame` | `null` |\n| `frameWidth` | `0` |\n| `frameHeight` | `0` |\n| `frameRate` | `0` |\n| `getFrameData()` | `null` |\n| `faces` | `[]` (empty array) |\n| `hands` | `[]` (empty array) |\n| `pose` | `null` |\n| `segmentation` | `null` |\n\nWhen the user leaves the camera frame, CV data also resets: `faces` and `hands` become empty arrays, `pose` and `segmentation` become `null`.\n\nWhen CV features are not enabled, the corresponding fields on each face / hand entry are also `null` rather than fake zeros: `face.bounds` / `face.center` / `face.confidence` are `null` unless face detection is on, `face.headPose` is `null` unless face mesh is on, `face.expressions` / `face.blendshapes` are `null` unless emotion detection is on. Always null-check before reading."
|
|
5825
5845
|
},
|
|
5826
5846
|
{
|
|
5827
5847
|
"type": "live-example",
|
|
@@ -5928,7 +5948,7 @@ export const docsApi = {
|
|
|
5928
5948
|
"content": [
|
|
5929
5949
|
{
|
|
5930
5950
|
"type": "text",
|
|
5931
|
-
"markdown": "# Face Detection\n\nFace detection provides the position, size, and confidence of faces in the video stream. Enable it with [`viji.video.cv.enableFaceDetection(true)`](../connection/). The data API is identical to the [Native renderer](../../../native/video/face-detection/).\n\n## Property Reference\n\nResults appear in `viji.video.cv.faces`: an array of `FaceData` objects. When no faces are detected or the feature is disabled, the array is empty.\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `face.id` | `number` | Index-based face identifier (0, 1, 2, ...) |\n| `face.bounds` | `{ x, y, width, height }` | Bounding box, normalized 0-1 |\n| `face.center` | `{ x, y }` | Bounding box center, normalized 0-1 |\n| `face.confidence` | `number` | Detection confidence (0-1) |\n| `face.landmarks` | `{ x, y, z? }[]` | Empty `[]
|
|
5951
|
+
"markdown": "# Face Detection\n\nFace detection provides the position, size, and confidence of faces in the video stream. Enable it with [`viji.video.cv.enableFaceDetection(true)`](../connection/). The data API is identical to the [Native renderer](../../../native/video/face-detection/).\n\nFace detection populates `face.bounds`, `face.center`, and `face.confidence`. It does NOT populate `face.landmarks` / `face.headPose` (use [Face Mesh](../face-mesh/)) or `face.expressions` / `face.blendshapes` (use [Emotion Detection](../emotion-detection/)). The three CV features are independent — enable each one you need.\n\n`viji.video.cv.enableFaceDetection(true)` is safe to call from module scope (always-on CV) or per-frame inside `render()` gated by a toggle parameter (opt-in CV). The verb is idempotent and reference-counted; per-frame calls are cheap. `await` is optional.\n\n## Property Reference\n\nResults appear in `viji.video.cv.faces`: an array of `FaceData` objects. When no faces are detected or the feature is disabled, the array is empty.\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `face.id` | `number` | Index-based face identifier (0, 1, 2, ...) |\n| `face.bounds` | `{ x, y, width, height } \\| null` | Bounding box, normalized 0-1. `null` when face detection is disabled. |\n| `face.center` | `{ x, y } \\| null` | Bounding box center, normalized 0-1. `null` when face detection is disabled. |\n| `face.confidence` | `number \\| null` | Detection confidence (0-1). `null` when face detection is disabled. |\n| `face.landmarks` | `{ x, y, z? }[]` | Empty `[]` unless [Face Mesh](../face-mesh/) is enabled. |\n| `face.headPose` | `{ pitch, yaw, roll } \\| null` | `null` unless [Face Mesh](../face-mesh/) is enabled. |\n| `face.expressions` | `object \\| null` | `null` unless [Emotion Detection](../emotion-detection/) is enabled. |\n| `face.blendshapes` | `FaceBlendshapes \\| null` | `null` unless [Emotion Detection](../emotion-detection/) is enabled. |\n\n## Usage\n\nVideo drawing follows the [aspect-correct pattern](../basics/#drawing-video). The `videoFit` helper returns the rectangle the video occupies inside the canvas; multiply CV bounds by `v.width` / `v.height` and offset by `v.x` / `v.y` so they stay aligned. This demo uses `'contain'` so face boxes near the frame edges are not clipped against the canvas.\n\n```javascript\n// @renderer p5\n\nconst useFace = viji.toggle(false, { label: 'Face Detection', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji, p5) {\n p5.background(17);\n\n if (useFace.value) viji.video.cv.enableFaceDetection(true);\n else viji.video.cv.enableFaceDetection(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) return;\n\n const v = videoFit(viji, 'contain');\n p5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n viji.video.cv.faces.forEach(face => {\n const bx = v.x + face.bounds.x * v.width;\n const by = v.y + face.bounds.y * v.height;\n const bw = face.bounds.width * v.width;\n const bh = face.bounds.height * v.height;\n\n p5.noFill();\n p5.stroke(78, 205, 196);\n p5.strokeWeight(2);\n p5.rect(bx, by, bw, bh);\n\n p5.noStroke();\n p5.fill(78, 205, 196);\n p5.circle(v.x + face.center.x * v.width, v.y + face.center.y * v.height, 8);\n\n p5.textSize(Math.min(viji.width, viji.height) * 0.025);\n p5.textAlign(p5.LEFT, p5.BOTTOM);\n p5.text('Face #' + face.id + ' (' + (face.confidence * 100).toFixed(0) + '%)', bx, by - 4);\n });\n}\n```\n\n**Cost: Low.** Face detection is the lightest CV feature.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\n> [!TIP]\n> **Best practice:** Don't enable CV features by default. Instead, expose a toggle parameter so users can activate them on capable devices:\n> ```javascript\n> const useFace = viji.toggle(false, { label: 'Enable Face Detection', category: 'video' });\n> if (useFace.value) {\n> await viji.video.cv.enableFaceDetection(true);\n> }\n> ```\n\nWhen face detection is disabled or no faces are visible, `viji.video.cv.faces` becomes an empty array `[]`."
|
|
5932
5952
|
},
|
|
5933
5953
|
{
|
|
5934
5954
|
"type": "live-example",
|
|
@@ -5955,11 +5975,21 @@ export const docsApi = {
|
|
|
5955
5975
|
"level": 2,
|
|
5956
5976
|
"text": "Property Reference"
|
|
5957
5977
|
},
|
|
5978
|
+
{
|
|
5979
|
+
"id": "enabling-face-mesh",
|
|
5980
|
+
"level": 2,
|
|
5981
|
+
"text": "Enabling face mesh"
|
|
5982
|
+
},
|
|
5958
5983
|
{
|
|
5959
5984
|
"id": "usage",
|
|
5960
5985
|
"level": 2,
|
|
5961
5986
|
"text": "Usage"
|
|
5962
5987
|
},
|
|
5988
|
+
{
|
|
5989
|
+
"id": "computing-a-face-center-from-landmarks",
|
|
5990
|
+
"level": 2,
|
|
5991
|
+
"text": "Computing a Face Center From Landmarks"
|
|
5992
|
+
},
|
|
5963
5993
|
{
|
|
5964
5994
|
"id": "pairing-frame-and-landmarks",
|
|
5965
5995
|
"level": 2,
|
|
@@ -5974,12 +6004,12 @@ export const docsApi = {
|
|
|
5974
6004
|
"content": [
|
|
5975
6005
|
{
|
|
5976
6006
|
"type": "text",
|
|
5977
|
-
"markdown": "# Face Mesh\n\nFace mesh provides 468 detailed facial landmark points and head pose estimation (pitch, yaw, roll). Enable it with [`viji.video.cv.enableFaceMesh(true)`](../connection/). The data API is identical to the [Native renderer](../../../native/video/face-mesh/).\n\n## Property Reference\n\nFace mesh data appears on each `FaceData` object in `viji.video.cv.faces`:\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `face.landmarks` | `{ x, y, z? }[]` | 468 facial landmark points, normalized 0-1 |\n| `face.headPose
|
|
6007
|
+
"markdown": "# Face Mesh\n\nFace mesh provides 468 detailed facial landmark points and head pose estimation (pitch, yaw, roll). Enable it with [`viji.video.cv.enableFaceMesh(true)`](../connection/). The data API is identical to the [Native renderer](../../../native/video/face-mesh/).\n\nFace mesh is independent of [Face Detection](../face-detection/). Each feature populates its own subset of fields on `FaceData`:\n\n- Face mesh populates `face.landmarks` and `face.headPose`.\n- Face detection populates `face.bounds`, `face.center`, `face.confidence`.\n\nIf you also need `face.bounds` or `face.center`, enable face detection alongside mesh, or compute a bounding box from `face.landmarks` min/max yourself.\n\n## Property Reference\n\nFace mesh data appears on each `FaceData` object in `viji.video.cv.faces`:\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `face.landmarks` | `{ x, y, z? }[]` | 468 facial landmark points, normalized 0-1. Empty `[]` when face mesh is disabled. |\n| `face.headPose` | `{ pitch, yaw, roll } \\| null` | Head rotation in degrees, or `null` when face mesh is disabled. |\n\nWhen face mesh is disabled, `landmarks` is `[]` and `headPose` is `null` (always null-check before reading `pitch`/`yaw`/`roll`).\n\n## Enabling face mesh\n\n`viji.video.cv.enableFaceMesh(true)` is safe to call from module scope (always-on CV) or per-frame inside `render()` gated by a toggle parameter (opt-in CV). The verb is idempotent and reference-counted; per-frame calls are cheap. `await` is optional.\n\n## Usage\n\n```javascript\n// @renderer p5\n\nconst useMesh = viji.toggle(false, { label: 'Face Mesh', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji, p5) {\n p5.background(17);\n\n if (useMesh.value) viji.video.cv.enableFaceMesh(true);\n else viji.video.cv.enableFaceMesh(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) return;\n\n // 'contain' so face landmarks near the edge stay visible.\n const v = videoFit(viji, 'contain');\n p5.tint(255, 100);\n p5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n p5.noTint();\n\n viji.video.cv.faces.forEach(face => {\n if (face.landmarks.length === 0) return;\n\n p5.noStroke();\n p5.fill(69, 183, 209, 180);\n face.landmarks.forEach(pt => {\n p5.circle(v.x + pt.x * v.width, v.y + pt.y * v.height, 2);\n });\n\n const hp = face.headPose;\n p5.fill(255);\n p5.textSize(Math.min(viji.width, viji.height) * 0.025);\n p5.textAlign(p5.LEFT, p5.BOTTOM);\n p5.text(\n 'Pitch: ' + hp.pitch.toFixed(1) + ' Yaw: ' + hp.yaw.toFixed(1) + ' Roll: ' + hp.roll.toFixed(1),\n viji.width * 0.03, viji.height - 10\n );\n });\n}\n```\n\n**Cost: Medium-High.** Face mesh processes 468 landmarks per face and computes head pose.\n\n## Computing a Face Center From Landmarks\n\n`face.center` and `face.bounds` are populated by [Face Detection](../face-detection/), not by face mesh. When only face mesh is enabled, both read as `null`. For mesh-driven visualizations that need a face-anchored position, compute the center from landmark min/max instead of enabling face detection just for that:\n\n```javascript\nfunction meshCenterFromLandmarks(landmarks) {\n if (!landmarks || landmarks.length === 0) return null;\n let minX = 1, minY = 1, maxX = 0, maxY = 0;\n for (const p of landmarks) {\n if (p.x < minX) minX = p.x;\n if (p.y < minY) minY = p.y;\n if (p.x > maxX) maxX = p.x;\n if (p.y > maxY) maxY = p.y;\n }\n return { x: (minX + maxX) / 2, y: (minY + maxY) / 2 };\n}\n\nviji.video.cv.faces.forEach(face => {\n const center = meshCenterFromLandmarks(face.landmarks);\n if (!center) return;\n const cx = v.x + center.x * v.width;\n const cy = v.y + center.y * v.height;\n // ...anchor your visualization at (cx, cy)\n});\n```\n\nThis saves a CV feature's worth of WebGL-context and inference cost compared to enabling face detection just for the center value.\n\n## Pairing Frame and Landmarks\n\nFace-mesh effects split into two camps with different defaults.\n\n**Effects that read pixels from the displayed frame at landmark positions** (texturing the face mesh with the underlying skin, warping the face along its mesh, masks that follow the face contour, makeup that blends with skin tone): use [`viji.video.cv.analysedFrame`](../basics/), the frame the landmarks were computed from. The displayed image stutters or holds briefly between inferences, but the effect aligns pixel-for-pixel with the face geometry.\n\n```javascript\n// Texture-mapped face filter: analysedFrame is required\nconst frame = viji.video.cv.analysedFrame ?? viji.video.currentFrame;\np5.image(frame, 0, 0, viji.width, viji.height);\nviji.video.cv.faces.forEach(face => { /* warp/texture along the mesh */ });\n```\n\n**Effects that draw on top of the face without reading its pixels** (landmark dots, lines, particles, debug overlays, parameter modulation): use `viji.video.currentFrame`. Landmarks lag the displayed face by ~1 frame under fast motion, but the camera feels live. Reaching for `analysedFrame` here would freeze the displayed video between inferences for no visual benefit.\n\n`analysedFrame` is `null` until the first inference completes after enabling face mesh. The `?? currentFrame` fallback covers that startup window; expect a brief visual hitch when the first inference lands.\n\nSee [Best Practices](/getting-started/best-practices/#currentframe-vs-analysedframe-pick-the-right-trade-off) for the cross-renderer trade-off discussion.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\n> [!TIP]\n> **Best practice:** Don't enable CV features by default. Instead, expose a toggle parameter so users can activate them on capable devices:\n> ```javascript\n> const useMesh = viji.toggle(false, { label: 'Enable Face Mesh', category: 'video' });\n> if (useMesh.value) {\n> await viji.video.cv.enableFaceMesh(true);\n> }\n> ```\n\nWhen face mesh is disabled, `face.landmarks` becomes `[]` and `face.headPose` reads as `null` (always null-check before reading `pitch`/`yaw`/`roll`)."
|
|
5978
6008
|
},
|
|
5979
6009
|
{
|
|
5980
6010
|
"type": "live-example",
|
|
5981
6011
|
"title": "Face Mesh",
|
|
5982
|
-
"sceneCode": "// @renderer p5\n\nconst useMesh = viji.toggle(false, { label: 'Face Mesh', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction cvHint(viji, p5, label) {\n const s = Math.min(viji.width, viji.height);\n const t = (p5.sin(viji.time * 3) + 1) / 2;\n const bandH = s * 0.18;\n const bandY = (viji.height - bandH) / 2;\n p5.noStroke();\n p5.fill(255, 200, 0, (0.85 + t * 0.15) * 255);\n p5.rect(0, bandY, viji.width, bandH);\n p5.fill(17);\n p5.textAlign(p5.CENTER, p5.CENTER);\n p5.textSize(s * 0.045);\n p5.textStyle(p5.BOLD);\n p5.text(`Enable \"${label}\" in parameters`, viji.width / 2, viji.height / 2);\n p5.textStyle(p5.NORMAL);\n}\n\nfunction render(viji, p5) {\n p5.background(17);\n\n if (useMesh.value) viji.video.cv.enableFaceMesh(true);\n else viji.video.cv.enableFaceMesh(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) {\n p5.fill(100);\n p5.textAlign(p5.CENTER, p5.CENTER);\n p5.textSize(Math.min(viji.width, viji.height) * 0.04);\n p5.text('Waiting for video...', viji.width / 2, viji.height / 2);\n return;\n }\n\n const v = videoFit(viji, 'contain');\n p5.tint(255, 100);\n p5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n p5.noTint();\n\n if (!useMesh.value) {\n cvHint(viji, p5, 'Face Mesh');\n return;\n }\n\n viji.video.cv.faces.forEach(face => {\n if (face.landmarks.length === 0) return;\n\n p5.noStroke();\n p5.fill(69, 183, 209, 180);\n face.landmarks.forEach(pt => {\n p5.circle(v.x + pt.x * v.width, v.y + pt.y * v.height, 2);\n });\n\n const
|
|
6012
|
+
"sceneCode": "// @renderer p5\n\nconst useMesh = viji.toggle(false, { label: 'Face Mesh', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\n// Compute face center from face-mesh landmark min/max. face.center is null\n// when only face mesh is enabled (it is populated by face detection), so\n// mesh-driven visualizations derive their anchor point from landmarks.\nfunction meshCenterFromLandmarks(landmarks) {\n if (!landmarks || landmarks.length === 0) return null;\n let minX = 1, minY = 1, maxX = 0, maxY = 0;\n for (const p of landmarks) {\n if (p.x < minX) minX = p.x;\n if (p.y < minY) minY = p.y;\n if (p.x > maxX) maxX = p.x;\n if (p.y > maxY) maxY = p.y;\n }\n return { x: (minX + maxX) / 2, y: (minY + maxY) / 2 };\n}\n\nfunction cvHint(viji, p5, label) {\n const s = Math.min(viji.width, viji.height);\n const t = (p5.sin(viji.time * 3) + 1) / 2;\n const bandH = s * 0.18;\n const bandY = (viji.height - bandH) / 2;\n p5.noStroke();\n p5.fill(255, 200, 0, (0.85 + t * 0.15) * 255);\n p5.rect(0, bandY, viji.width, bandH);\n p5.fill(17);\n p5.textAlign(p5.CENTER, p5.CENTER);\n p5.textSize(s * 0.045);\n p5.textStyle(p5.BOLD);\n p5.text(`Enable \"${label}\" in parameters`, viji.width / 2, viji.height / 2);\n p5.textStyle(p5.NORMAL);\n}\n\nfunction render(viji, p5) {\n p5.background(17);\n\n if (useMesh.value) viji.video.cv.enableFaceMesh(true);\n else viji.video.cv.enableFaceMesh(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) {\n p5.fill(100);\n p5.textAlign(p5.CENTER, p5.CENTER);\n p5.textSize(Math.min(viji.width, viji.height) * 0.04);\n p5.text('Waiting for video...', viji.width / 2, viji.height / 2);\n return;\n }\n\n const v = videoFit(viji, 'contain');\n p5.tint(255, 100);\n p5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n p5.noTint();\n\n if (!useMesh.value) {\n cvHint(viji, p5, 'Face Mesh');\n return;\n }\n\n viji.video.cv.faces.forEach(face => {\n if (face.landmarks.length === 0) return;\n const hp = face.headPose;\n if (!hp) return;\n\n // Landmark dots\n p5.noStroke();\n p5.fill(69, 183, 209, 180);\n face.landmarks.forEach(pt => {\n p5.circle(v.x + pt.x * v.width, v.y + pt.y * v.height, 2);\n });\n\n // Face-anchored head-pose arrow + roll horizon. Yaw drives X, pitch\n // drives Y (negated because p5 default canvas is Y-down), roll rotates\n // the arrow vector. Center is computed from landmarks because\n // face.center is null in mesh-only mode.\n const center = meshCenterFromLandmarks(face.landmarks);\n if (center) {\n const fcx = v.x + center.x * v.width;\n const fcy = v.y + center.y * v.height;\n const yawRad = hp.yaw * Math.PI / 180;\n const pitchRad = hp.pitch * Math.PI / 180;\n const rollRad = hp.roll * Math.PI / 180;\n const armLen = Math.min(v.width, v.height) * 0.12;\n const horizonLen = Math.min(v.width, v.height) * 0.07;\n\n const dx = Math.sin(yawRad) * armLen;\n const dy = -Math.sin(pitchRad) * armLen;\n const cr = Math.cos(rollRad), sr = Math.sin(rollRad);\n const rx = dx * cr - dy * sr;\n const ry = dx * sr + dy * cr;\n\n p5.stroke(0, 255, 240);\n p5.strokeWeight(2);\n p5.line(fcx, fcy, fcx + rx, fcy + ry);\n p5.noStroke();\n p5.fill(0, 255, 240);\n p5.circle(fcx + rx, fcy + ry, 6);\n\n // Roll-only horizon, so head tilt is legible even when looking forward.\n const hx = Math.cos(rollRad) * horizonLen;\n const hy = Math.sin(rollRad) * horizonLen;\n p5.stroke(0, 200, 200, 220);\n p5.strokeWeight(1.5);\n p5.line(fcx - hx, fcy - hy, fcx + hx, fcy + hy);\n p5.noStroke();\n }\n\n // Numeric overlay\n p5.fill(255);\n p5.textSize(Math.min(viji.width, viji.height) * 0.025);\n p5.textAlign(p5.LEFT, p5.BOTTOM);\n p5.text(\n face.landmarks.length + ' landmarks | Pitch: ' + hp.pitch.toFixed(1) +\n ' Yaw: ' + hp.yaw.toFixed(1) + ' Roll: ' + hp.roll.toFixed(1),\n viji.width * 0.03, viji.height - 10\n );\n });\n}\n",
|
|
5983
6013
|
"sceneFile": "face-mesh-demo.scene.js",
|
|
5984
6014
|
"capabilities": {
|
|
5985
6015
|
"video": true
|
|
@@ -6025,7 +6055,7 @@ export const docsApi = {
|
|
|
6025
6055
|
"content": [
|
|
6026
6056
|
{
|
|
6027
6057
|
"type": "text",
|
|
6028
|
-
"markdown": "# Emotion Detection\n\nEmotion detection provides 7 expression scores and 52 ARKit-compatible blendshape coefficients for each detected face. Enable it with [`viji.video.cv.enableEmotionDetection(true)`](../connection/). The data API is identical to the [Native renderer](../../../native/video/emotion-detection/).\n\n## Property Reference\n\n### Expressions (7 emotions)\n\nEach value is a confidence score from 0 to 1.\n\n| Property | Type | Range | Description |\n|----------|------|-------|-------------|\n| `face.expressions.neutral` | `number` | 0-1 | Neutral expression |\n| `face.expressions.happy` | `number` | 0-1 | Happy / smiling |\n| `face.expressions.sad` | `number` | 0-1 | Sad |\n| `face.expressions.angry` | `number` | 0-1 | Angry |\n| `face.expressions.surprised` | `number` | 0-1 | Surprised |\n| `face.expressions.disgusted` | `number` | 0-1 | Disgusted |\n| `face.expressions.fearful` | `number` | 0-1 | Fearful |\n\n### Blendshapes (52 ARKit coefficients)\n\nThe `face.blendshapes` object contains 52 ARKit-compatible coefficients (0-1 each). See [Native Emotion Detection](../../../native/video/emotion-detection/) for the full list.\n\n## Usage\n\n```javascript\n// @renderer p5\n\nconst useEmotion = viji.toggle(false, { label: 'Emotion Detection', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji, p5) {\n p5.background(17);\n\n if (useEmotion.value) viji.video.cv.enableEmotionDetection(true);\n else viji.video.cv.enableEmotionDetection(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) return;\n\n // 'contain' so the face stays visible at the frame edges.\n const v = videoFit(viji, 'contain');\n p5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n const face = viji.video.cv.faces[0];\n if (!face) return;\n\n const expr = face.expressions;\n const labels = ['neutral', 'happy', 'sad', 'angry', 'surprised', 'disgusted', 'fearful'];\n const values = [expr.neutral, expr.happy, expr.sad, expr.angry, expr.surprised, expr.disgusted, expr.fearful];\n const colors = [[136,136,136], [76,175,80], [33,150,243], [244,67,54], [255,152,0], [156,39,176], [96,125,139]];\n\n const barH = viji.height * 0.04;\n const barW = viji.width * 0.3;\n const x = viji.width * 0.65;\n let y = viji.height * 0.12;\n const fontSize = barH * 0.7;\n\n p5.textSize(fontSize);\n labels.forEach((label, i) => {\n p5.fill(170);\n p5.noStroke();\n p5.textAlign(p5.RIGHT, p5.CENTER);\n p5.text(label, x - 8, y + barH / 2);\n\n p5.fill(34);\n p5.rect(x, y, barW, barH);\n p5.fill(colors[i][0], colors[i][1], colors[i][2]);\n p5.rect(x, y, barW * values[i], barH);\n\n p5.fill(220);\n p5.textAlign(p5.LEFT, p5.CENTER);\n p5.text((values[i] * 100).toFixed(0) + '%', x + barW + 6, y + barH / 2);\n\n y += barH * 1.8;\n });\n}\n```\n\n**Cost: High.** Emotion detection computes 7 expressions and 52 blendshape coefficients per face.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\n> [!TIP]\n> **Best practice:** Don't enable CV features by default. Instead, expose a toggle parameter so users can activate them on capable devices:\n> ```javascript\n> const useEmotion = viji.toggle(false, { label: 'Enable Emotion Detection', category: 'video' });\n> if (useEmotion.value) {\n> await viji.video.cv.enableEmotionDetection(true);\n> }\n> ```\n\nWhen emotion detection is disabled, all `expressions` values are `0` and all `blendshapes` coefficients are `0`."
|
|
6058
|
+
"markdown": "# Emotion Detection\n\nEmotion detection provides 7 expression scores and 52 ARKit-compatible blendshape coefficients for each detected face. Enable it with [`viji.video.cv.enableEmotionDetection(true)`](../connection/). The data API is identical to the [Native renderer](../../../native/video/emotion-detection/).\n\n`face.expressions` and `face.blendshapes` are `null` until emotion detection is enabled — always null-check before reading individual scores. Emotion detection internally loads the face landmarker model, so `face.landmarks` and `face.headPose` are also populated. `face.bounds`, `face.center`, and `face.confidence` are produced by [Face Detection](../face-detection/), not by emotion detection — enable both if you need a bounding box.\n\n`viji.video.cv.enableEmotionDetection(true)` is safe to call from module scope (always-on CV) or per-frame inside `render()` gated by a toggle parameter (opt-in CV). The verb is idempotent and reference-counted; per-frame calls are cheap. `await` is optional.\n\n## Property Reference\n\n### Expressions (7 emotions)\n\nEach value is a confidence score from 0 to 1.\n\n| Property | Type | Range | Description |\n|----------|------|-------|-------------|\n| `face.expressions.neutral` | `number` | 0-1 | Neutral expression |\n| `face.expressions.happy` | `number` | 0-1 | Happy / smiling |\n| `face.expressions.sad` | `number` | 0-1 | Sad |\n| `face.expressions.angry` | `number` | 0-1 | Angry |\n| `face.expressions.surprised` | `number` | 0-1 | Surprised |\n| `face.expressions.disgusted` | `number` | 0-1 | Disgusted |\n| `face.expressions.fearful` | `number` | 0-1 | Fearful |\n\n### Blendshapes (52 ARKit coefficients)\n\nThe `face.blendshapes` object contains 52 ARKit-compatible coefficients (0-1 each). See [Native Emotion Detection](../../../native/video/emotion-detection/) for the full list.\n\n## Usage\n\n```javascript\n// @renderer p5\n\nconst useEmotion = viji.toggle(false, { label: 'Emotion Detection', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji, p5) {\n p5.background(17);\n\n if (useEmotion.value) viji.video.cv.enableEmotionDetection(true);\n else viji.video.cv.enableEmotionDetection(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) return;\n\n // 'contain' so the face stays visible at the frame edges.\n const v = videoFit(viji, 'contain');\n p5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n const face = viji.video.cv.faces[0];\n if (!face) return;\n\n const expr = face.expressions;\n const labels = ['neutral', 'happy', 'sad', 'angry', 'surprised', 'disgusted', 'fearful'];\n const values = [expr.neutral, expr.happy, expr.sad, expr.angry, expr.surprised, expr.disgusted, expr.fearful];\n const colors = [[136,136,136], [76,175,80], [33,150,243], [244,67,54], [255,152,0], [156,39,176], [96,125,139]];\n\n const barH = viji.height * 0.04;\n const barW = viji.width * 0.3;\n const x = viji.width * 0.65;\n let y = viji.height * 0.12;\n const fontSize = barH * 0.7;\n\n p5.textSize(fontSize);\n labels.forEach((label, i) => {\n p5.fill(170);\n p5.noStroke();\n p5.textAlign(p5.RIGHT, p5.CENTER);\n p5.text(label, x - 8, y + barH / 2);\n\n p5.fill(34);\n p5.rect(x, y, barW, barH);\n p5.fill(colors[i][0], colors[i][1], colors[i][2]);\n p5.rect(x, y, barW * values[i], barH);\n\n p5.fill(220);\n p5.textAlign(p5.LEFT, p5.CENTER);\n p5.text((values[i] * 100).toFixed(0) + '%', x + barW + 6, y + barH / 2);\n\n y += barH * 1.8;\n });\n}\n```\n\n**Cost: High.** Emotion detection computes 7 expressions and 52 blendshape coefficients per face.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\n> [!TIP]\n> **Best practice:** Don't enable CV features by default. Instead, expose a toggle parameter so users can activate them on capable devices:\n> ```javascript\n> const useEmotion = viji.toggle(false, { label: 'Enable Emotion Detection', category: 'video' });\n> if (useEmotion.value) {\n> await viji.video.cv.enableEmotionDetection(true);\n> }\n> ```\n\nWhen emotion detection is disabled, all `expressions` values are `0` and all `blendshapes` coefficients are `0`."
|
|
6029
6059
|
},
|
|
6030
6060
|
{
|
|
6031
6061
|
"type": "live-example",
|
|
@@ -6071,7 +6101,7 @@ export const docsApi = {
|
|
|
6071
6101
|
"content": [
|
|
6072
6102
|
{
|
|
6073
6103
|
"type": "text",
|
|
6074
|
-
"markdown": "# Hand Tracking\n\nHand tracking provides 21-point landmarks, palm position, bounding boxes, and ML-based gesture recognition for up to two hands. Enable it with [`viji.video.cv.enableHandTracking(true)`](../connection/). The data API is identical to the [Native renderer](../../../native/video/hand-tracking/).\n\n## Property Reference\n\nResults appear in `viji.video.cv.hands`: an array of up to 2 `HandData` objects.\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `hand.id` | `number` | Index-based hand identifier (0, 1) |\n| `hand.handedness` | `'left' \\| 'right'` | Which hand (always lowercase) |\n| `hand.confidence` | `number` | Detection confidence (0-1) |\n| `hand.bounds` | `{ x, y, width, height }` | Bounding box, normalized 0-1 |\n| `hand.landmarks` | `{ x, y, z }[]` | 21 MediaPipe hand landmarks, normalized 0-1 |\n| `hand.palm` | `{ x, y, z }` | Palm center: `landmarks[9]` (middle finger MCP) |\n| `hand.gestures` | object | 7 ML gesture confidence scores (0-1 each) |\n\n### Gestures\n\n| Property | Gesture |\n|----------|---------|\n| `hand.gestures.fist` | Closed fist |\n| `hand.gestures.openPalm` | Open hand |\n| `hand.gestures.peace` | Victory / peace sign |\n| `hand.gestures.thumbsUp` | Thumbs up |\n| `hand.gestures.thumbsDown` | Thumbs down |\n| `hand.gestures.pointing` | Pointing up |\n| `hand.gestures.iLoveYou` | ASL I-love-you sign |\n\n## Usage\n\nVideo drawing follows the [aspect-correct pattern](../basics/#drawing-video). This demo uses `'contain'` so hands near the frame edges stay visible.\n\n```javascript\n// @renderer p5\n\nconst useHands = viji.toggle(false, { label: 'Hand Tracking', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji, p5) {\n p5.background(17);\n\n if (useHands.value) viji.video.cv.enableHandTracking(true);\n else viji.video.cv.enableHandTracking(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) return;\n\n const v = videoFit(viji, 'contain');\n p5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n viji.video.cv.hands.forEach(hand => {\n const col = hand.handedness === 'left' ? [255, 159, 243] : [84, 160, 255];\n p5.noStroke();\n p5.fill(col[0], col[1], col[2]);\n\n hand.landmarks.forEach(pt => {\n p5.circle(v.x + pt.x * v.width, v.y + pt.y * v.height, 6);\n });\n\n p5.noFill();\n p5.stroke(col[0], col[1], col[2]);\n p5.strokeWeight(2);\n p5.circle(v.x + hand.palm.x * v.width, v.y + hand.palm.y * v.height, 16);\n });\n}\n```\n\n**Cost: Medium.** Hand tracking processes up to 2 hands with 21 landmarks each, plus ML gesture classification.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\n> [!TIP]\n> **Best practice:** Don't enable CV features by default. Instead, expose a toggle parameter so users can activate them on capable devices:\n> ```javascript\n> const useHands = viji.toggle(false, { label: 'Enable Hand Tracking', category: 'video' });\n> if (useHands.value) {\n> await viji.video.cv.enableHandTracking(true);\n> }\n> ```\n\nWhen hand tracking is disabled or no hands are visible, `viji.video.cv.hands` becomes an empty array `[]`."
|
|
6104
|
+
"markdown": "# Hand Tracking\n\nHand tracking provides 21-point landmarks, palm position, bounding boxes, and ML-based gesture recognition for up to two hands. Enable it with [`viji.video.cv.enableHandTracking(true)`](../connection/). The data API is identical to the [Native renderer](../../../native/video/hand-tracking/).\n\n`viji.video.cv.enableHandTracking(true)` is safe to call from module scope (always-on CV) or per-frame inside `render()` gated by a toggle parameter (opt-in CV). The verb is idempotent and reference-counted; per-frame calls are cheap. `await` is optional.\n\n## Property Reference\n\nResults appear in `viji.video.cv.hands`: an array of up to 2 `HandData` objects.\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `hand.id` | `number` | Index-based hand identifier (0, 1) |\n| `hand.handedness` | `'left' \\| 'right'` | Which hand (always lowercase) |\n| `hand.confidence` | `number` | Detection confidence (0-1) |\n| `hand.bounds` | `{ x, y, width, height }` | Bounding box, normalized 0-1 |\n| `hand.landmarks` | `{ x, y, z }[]` | 21 MediaPipe hand landmarks, normalized 0-1 |\n| `hand.palm` | `{ x, y, z }` | Palm center: `landmarks[9]` (middle finger MCP) |\n| `hand.gestures` | object | 7 ML gesture confidence scores (0-1 each) |\n\n### Gestures\n\n| Property | Gesture |\n|----------|---------|\n| `hand.gestures.fist` | Closed fist |\n| `hand.gestures.openPalm` | Open hand |\n| `hand.gestures.peace` | Victory / peace sign |\n| `hand.gestures.thumbsUp` | Thumbs up |\n| `hand.gestures.thumbsDown` | Thumbs down |\n| `hand.gestures.pointing` | Pointing up |\n| `hand.gestures.iLoveYou` | ASL I-love-you sign |\n\n## Usage\n\nVideo drawing follows the [aspect-correct pattern](../basics/#drawing-video). This demo uses `'contain'` so hands near the frame edges stay visible.\n\n```javascript\n// @renderer p5\n\nconst useHands = viji.toggle(false, { label: 'Hand Tracking', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji, p5) {\n p5.background(17);\n\n if (useHands.value) viji.video.cv.enableHandTracking(true);\n else viji.video.cv.enableHandTracking(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) return;\n\n const v = videoFit(viji, 'contain');\n p5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n viji.video.cv.hands.forEach(hand => {\n const col = hand.handedness === 'left' ? [255, 159, 243] : [84, 160, 255];\n p5.noStroke();\n p5.fill(col[0], col[1], col[2]);\n\n hand.landmarks.forEach(pt => {\n p5.circle(v.x + pt.x * v.width, v.y + pt.y * v.height, 6);\n });\n\n p5.noFill();\n p5.stroke(col[0], col[1], col[2]);\n p5.strokeWeight(2);\n p5.circle(v.x + hand.palm.x * v.width, v.y + hand.palm.y * v.height, 16);\n });\n}\n```\n\n**Cost: Medium.** Hand tracking processes up to 2 hands with 21 landmarks each, plus ML gesture classification.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\n> [!TIP]\n> **Best practice:** Don't enable CV features by default. Instead, expose a toggle parameter so users can activate them on capable devices:\n> ```javascript\n> const useHands = viji.toggle(false, { label: 'Enable Hand Tracking', category: 'video' });\n> if (useHands.value) {\n> await viji.video.cv.enableHandTracking(true);\n> }\n> ```\n\nWhen hand tracking is disabled or no hands are visible, `viji.video.cv.hands` becomes an empty array `[]`."
|
|
6075
6105
|
},
|
|
6076
6106
|
{
|
|
6077
6107
|
"type": "live-example",
|
|
@@ -6112,7 +6142,7 @@ export const docsApi = {
|
|
|
6112
6142
|
"content": [
|
|
6113
6143
|
{
|
|
6114
6144
|
"type": "text",
|
|
6115
|
-
"markdown": "# Pose Detection\n\nPose detection provides 33 body landmarks using MediaPipe's BlazePose model, with named groups for easy access to face, torso, arms, and legs. Enable it with [`viji.video.cv.enablePoseDetection(true)`](../connection/). The data API is identical to the [Native renderer](../../../native/video/pose-detection/).\n\n## Property Reference\n\nResults appear in `viji.video.cv.pose`: a single `PoseData` object, or `null` when no pose is detected.\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `pose.confidence` | `number` | Average landmark visibility (0-1) |\n| `pose.landmarks` | `{ x, y, z, visibility }[]` | 33 BlazePose points, normalized 0-1 |\n| `pose.face` | `{ x, y }[]` | Face region landmarks (indices 0-10) |\n| `pose.torso` | `{ x, y }[]` | Torso landmarks (indices 11, 12, 23, 24) |\n| `pose.leftArm` | `{ x, y }[]` | Left arm (indices 11, 13, 15) |\n| `pose.rightArm` | `{ x, y }[]` | Right arm (indices 12, 14, 16) |\n| `pose.leftLeg` | `{ x, y }[]` | Left leg (indices 23, 25, 27, 29, 31) |\n| `pose.rightLeg` | `{ x, y }[]` | Right leg (indices 24, 26, 28, 30, 32) |\n\n## Usage\n\n```javascript\n// @renderer p5\n\nconst usePose = viji.toggle(false, { label: 'Pose Detection', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji, p5) {\n p5.background(17);\n\n if (usePose.value) viji.video.cv.enablePoseDetection(true);\n else viji.video.cv.enablePoseDetection(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) return;\n\n // 'contain' so the full body stays visible at frame edges.\n const v = videoFit(viji, 'contain');\n p5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n const pose = viji.video.cv.pose;\n if (!pose) return;\n\n p5.noStroke();\n p5.fill(255, 107, 107);\n pose.landmarks.forEach(pt => {\n if (pt.visibility > 0.5) {\n p5.circle(v.x + pt.x * v.width, v.y + pt.y * v.height, 8);\n }\n });\n\n p5.strokeWeight(2);\n p5.noFill();\n const drawGroup = (group, col) => {\n if (group.length < 2) return;\n p5.stroke(col[0], col[1], col[2]);\n p5.beginShape();\n group.forEach(pt => p5.vertex(v.x + pt.x * v.width, v.y + pt.y * v.height));\n p5.endShape();\n };\n\n drawGroup(pose.leftArm, [255, 159, 243]);\n drawGroup(pose.rightArm, [84, 160, 255]);\n drawGroup(pose.leftLeg, [255, 159, 243]);\n drawGroup(pose.rightLeg, [84, 160, 255]);\n drawGroup(pose.torso, [254, 202, 87]);\n}\n```\n\n**Cost: Medium.** Pose detection processes 33 body landmarks with visibility scores.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\n> [!TIP]\n> **Best practice:** Don't enable CV features by default. Instead, expose a toggle parameter so users can activate them on capable devices:\n> ```javascript\n> const usePose = viji.toggle(false, { label: 'Enable Pose Detection', category: 'video' });\n> if (usePose.value) {\n> await viji.video.cv.enablePoseDetection(true);\n> }\n> ```\n\nWhen pose detection is disabled or no body is visible, `viji.video.cv.pose` becomes `null`."
|
|
6145
|
+
"markdown": "# Pose Detection\n\nPose detection provides 33 body landmarks using MediaPipe's BlazePose model, with named groups for easy access to face, torso, arms, and legs. Enable it with [`viji.video.cv.enablePoseDetection(true)`](../connection/). The data API is identical to the [Native renderer](../../../native/video/pose-detection/).\n\n`viji.video.cv.enablePoseDetection(true)` is safe to call from module scope (always-on CV) or per-frame inside `render()` gated by a toggle parameter (opt-in CV). The verb is idempotent and reference-counted; per-frame calls are cheap. `await` is optional.\n\n## Property Reference\n\nResults appear in `viji.video.cv.pose`: a single `PoseData` object, or `null` when no pose is detected.\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `pose.confidence` | `number` | Average landmark visibility (0-1) |\n| `pose.landmarks` | `{ x, y, z, visibility }[]` | 33 BlazePose points, normalized 0-1 |\n| `pose.face` | `{ x, y }[]` | Face region landmarks (indices 0-10) |\n| `pose.torso` | `{ x, y }[]` | Torso landmarks (indices 11, 12, 23, 24) |\n| `pose.leftArm` | `{ x, y }[]` | Left arm (indices 11, 13, 15) |\n| `pose.rightArm` | `{ x, y }[]` | Right arm (indices 12, 14, 16) |\n| `pose.leftLeg` | `{ x, y }[]` | Left leg (indices 23, 25, 27, 29, 31) |\n| `pose.rightLeg` | `{ x, y }[]` | Right leg (indices 24, 26, 28, 30, 32) |\n\n## Usage\n\n```javascript\n// @renderer p5\n\nconst usePose = viji.toggle(false, { label: 'Pose Detection', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji, p5) {\n p5.background(17);\n\n if (usePose.value) viji.video.cv.enablePoseDetection(true);\n else viji.video.cv.enablePoseDetection(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) return;\n\n // 'contain' so the full body stays visible at frame edges.\n const v = videoFit(viji, 'contain');\n p5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n const pose = viji.video.cv.pose;\n if (!pose) return;\n\n p5.noStroke();\n p5.fill(255, 107, 107);\n pose.landmarks.forEach(pt => {\n if (pt.visibility > 0.5) {\n p5.circle(v.x + pt.x * v.width, v.y + pt.y * v.height, 8);\n }\n });\n\n p5.strokeWeight(2);\n p5.noFill();\n const drawGroup = (group, col) => {\n if (group.length < 2) return;\n p5.stroke(col[0], col[1], col[2]);\n p5.beginShape();\n group.forEach(pt => p5.vertex(v.x + pt.x * v.width, v.y + pt.y * v.height));\n p5.endShape();\n };\n\n drawGroup(pose.leftArm, [255, 159, 243]);\n drawGroup(pose.rightArm, [84, 160, 255]);\n drawGroup(pose.leftLeg, [255, 159, 243]);\n drawGroup(pose.rightLeg, [84, 160, 255]);\n drawGroup(pose.torso, [254, 202, 87]);\n}\n```\n\n**Cost: Medium.** Pose detection processes 33 body landmarks with visibility scores.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\n> [!TIP]\n> **Best practice:** Don't enable CV features by default. Instead, expose a toggle parameter so users can activate them on capable devices:\n> ```javascript\n> const usePose = viji.toggle(false, { label: 'Enable Pose Detection', category: 'video' });\n> if (usePose.value) {\n> await viji.video.cv.enablePoseDetection(true);\n> }\n> ```\n\nWhen pose detection is disabled or no body is visible, `viji.video.cv.pose` becomes `null`."
|
|
6116
6146
|
},
|
|
6117
6147
|
{
|
|
6118
6148
|
"type": "live-example",
|
|
@@ -6158,7 +6188,7 @@ export const docsApi = {
|
|
|
6158
6188
|
"content": [
|
|
6159
6189
|
{
|
|
6160
6190
|
"type": "text",
|
|
6161
|
-
"markdown": "# Body Segmentation\n\nBody segmentation provides a per-pixel mask that separates the person from the background. Enable it with [`viji.video.cv.enableBodySegmentation(true)`](../connection/). The data API is identical to the [Native renderer](../../../native/video/body-segmentation/).\n\n## Property Reference\n\nResults appear in `viji.video.cv.segmentation`: a `SegmentationData` object, or `null` when no mask is available.\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `segmentation.mask` | `Uint8Array` | Per-pixel mask: 0 = background, 1 = person |\n| `segmentation.width` | `number` | Mask width in pixels |\n| `segmentation.height` | `number` | Mask height in pixels |\n\nThe mask dimensions may differ from the video frame dimensions: they reflect the ML model's output resolution.\n\n## Usage\n\n```javascript\n// @renderer p5\n\nconst useSeg = viji.toggle(false, { label: 'Body Segmentation', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji, p5) {\n p5.background(17);\n\n if (useSeg.value) viji.video.cv.enableBodySegmentation(true);\n else viji.video.cv.enableBodySegmentation(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) return;\n\n // 'contain' so the body silhouette stays visible at frame edges.\n const v = videoFit(viji, 'contain');\n p5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n const seg = viji.video.cv.segmentation;\n if (!seg) return;\n\n let personPixels = 0;\n for (let i = 0; i < seg.mask.length; i++) {\n if (seg.mask[i] > 0) personPixels++;\n }\n const personRatio = personPixels / seg.mask.length;\n\n if (personRatio > 0.05) {\n const hue = 170 + personRatio * 60;\n p5.noFill();\n p5.stroke(hue % 360, 80, 60);\n p5.strokeWeight(4);\n p5.rect(v.x, v.y, v.width, v.height);\n }\n\n p5.noStroke();\n p5.fill(255);\n p5.textSize(Math.min(viji.width, viji.height) * 0.03);\n p5.textAlign(p5.LEFT, p5.BOTTOM);\n p5.text(\n 'Person: ' + (personRatio * 100).toFixed(0) + '% (' + seg.width + 'x' + seg.height + ' mask)',\n viji.width * 0.03, viji.height - 10\n );\n}\n```\n\n**Cost: High.** Body segmentation produces a per-pixel mask with a large tensor output.\n\n## Pairing Frame and Mask\n\nBody segmentation splits into two camps with different defaults.\n\n**Effects that composite the mask onto the displayed body** (background replacement, silhouette outlining, recolor-the-person effects, anything where the mask edge must follow the actual body contour): use [`viji.video.cv.analysedFrame`](../basics/), the frame the mask was computed from. Compositing against `currentFrame` makes the silhouette edge drift off the moving body and the cutout shows visible bleed during motion. The cost of `analysedFrame` is that the displayed image stutters or holds briefly between inferences.\n\n```javascript\n// Background replacement: analysedFrame is required\nconst v = videoFit(viji, 'contain'); // see basics for the helper\nconst frame = viji.video.cv.analysedFrame ?? viji.video.currentFrame;\np5.image(frame, v.x, v.y, v.width, v.height);\n// composite mask against the same frame, using v for mask positioning...\n```\n\n**Effects that use the mask as a generative input without compositing it onto the displayed body** (driving particle systems from the silhouette, feeding the mask into a fluid/light/distortion effect, scenes that don't display the camera at all): `viji.video.currentFrame` (or no displayed video) is fine. The exact mask edge does not need to match a specific displayed frame.\n\n`analysedFrame` is `null` before the first CV result. The `?? currentFrame` fallback covers the startup window; expect a brief visual hitch when the first inference lands.\n\nSee [Best Practices](/getting-started/best-practices/#currentframe-vs-analysedframe-pick-the-right-trade-off) for the cross-renderer trade-off discussion.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\n> [!TIP]\n> **Best practice:** Don't enable CV features by default. Instead, expose a toggle parameter so users can activate them on capable devices:\n> ```javascript\n> const useSeg = viji.toggle(false, { label: 'Enable Body Segmentation', category: 'video' });\n> if (useSeg.value) {\n> await viji.video.cv.enableBodySegmentation(true);\n> }\n> ```\n\nWhen body segmentation is disabled or no body is visible, `viji.video.cv.segmentation` becomes `null`."
|
|
6191
|
+
"markdown": "# Body Segmentation\n\nBody segmentation provides a per-pixel mask that separates the person from the background. Enable it with [`viji.video.cv.enableBodySegmentation(true)`](../connection/). The data API is identical to the [Native renderer](../../../native/video/body-segmentation/).\n\n`viji.video.cv.enableBodySegmentation(true)` is safe to call from module scope (always-on CV) or per-frame inside `render()` gated by a toggle parameter (opt-in CV). The verb is idempotent and reference-counted; per-frame calls are cheap. `await` is optional.\n\n## Property Reference\n\nResults appear in `viji.video.cv.segmentation`: a `SegmentationData` object, or `null` when no mask is available.\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `segmentation.mask` | `Uint8Array` | Per-pixel mask: 0 = background, 1 = person |\n| `segmentation.width` | `number` | Mask width in pixels |\n| `segmentation.height` | `number` | Mask height in pixels |\n\nThe mask dimensions may differ from the video frame dimensions: they reflect the ML model's output resolution.\n\n## Usage\n\n```javascript\n// @renderer p5\n\nconst useSeg = viji.toggle(false, { label: 'Body Segmentation', category: 'video' });\n\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nfunction render(viji, p5) {\n p5.background(17);\n\n if (useSeg.value) viji.video.cv.enableBodySegmentation(true);\n else viji.video.cv.enableBodySegmentation(false);\n\n if (!viji.video.isConnected || !viji.video.currentFrame) return;\n\n // 'contain' so the body silhouette stays visible at frame edges.\n const v = videoFit(viji, 'contain');\n p5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n const seg = viji.video.cv.segmentation;\n if (!seg) return;\n\n let personPixels = 0;\n for (let i = 0; i < seg.mask.length; i++) {\n if (seg.mask[i] > 0) personPixels++;\n }\n const personRatio = personPixels / seg.mask.length;\n\n if (personRatio > 0.05) {\n const hue = 170 + personRatio * 60;\n p5.noFill();\n p5.stroke(hue % 360, 80, 60);\n p5.strokeWeight(4);\n p5.rect(v.x, v.y, v.width, v.height);\n }\n\n p5.noStroke();\n p5.fill(255);\n p5.textSize(Math.min(viji.width, viji.height) * 0.03);\n p5.textAlign(p5.LEFT, p5.BOTTOM);\n p5.text(\n 'Person: ' + (personRatio * 100).toFixed(0) + '% (' + seg.width + 'x' + seg.height + ' mask)',\n viji.width * 0.03, viji.height - 10\n );\n}\n```\n\n**Cost: High.** Body segmentation produces a per-pixel mask with a large tensor output.\n\n## Pairing Frame and Mask\n\nBody segmentation splits into two camps with different defaults.\n\n**Effects that composite the mask onto the displayed body** (background replacement, silhouette outlining, recolor-the-person effects, anything where the mask edge must follow the actual body contour): use [`viji.video.cv.analysedFrame`](../basics/), the frame the mask was computed from. Compositing against `currentFrame` makes the silhouette edge drift off the moving body and the cutout shows visible bleed during motion. The cost of `analysedFrame` is that the displayed image stutters or holds briefly between inferences.\n\n```javascript\n// Background replacement: analysedFrame is required\nconst v = videoFit(viji, 'contain'); // see basics for the helper\nconst frame = viji.video.cv.analysedFrame ?? viji.video.currentFrame;\np5.image(frame, v.x, v.y, v.width, v.height);\n// composite mask against the same frame, using v for mask positioning...\n```\n\n**Effects that use the mask as a generative input without compositing it onto the displayed body** (driving particle systems from the silhouette, feeding the mask into a fluid/light/distortion effect, scenes that don't display the camera at all): `viji.video.currentFrame` (or no displayed video) is fine. The exact mask edge does not need to match a specific displayed frame.\n\n`analysedFrame` is `null` before the first CV result. The `?? currentFrame` fallback covers the startup window; expect a brief visual hitch when the first inference lands.\n\nSee [Best Practices](/getting-started/best-practices/#currentframe-vs-analysedframe-pick-the-right-trade-off) for the cross-renderer trade-off discussion.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\n> [!TIP]\n> **Best practice:** Don't enable CV features by default. Instead, expose a toggle parameter so users can activate them on capable devices:\n> ```javascript\n> const useSeg = viji.toggle(false, { label: 'Enable Body Segmentation', category: 'video' });\n> if (useSeg.value) {\n> await viji.video.cv.enableBodySegmentation(true);\n> }\n> ```\n\nWhen body segmentation is disabled or no body is visible, `viji.video.cv.segmentation` becomes `null`."
|
|
6162
6192
|
},
|
|
6163
6193
|
{
|
|
6164
6194
|
"type": "live-example",
|
|
@@ -6855,7 +6885,7 @@ export const docsApi = {
|
|
|
6855
6885
|
},
|
|
6856
6886
|
"p5-ext-audio": {
|
|
6857
6887
|
"id": "p5-ext-audio",
|
|
6858
|
-
"title": "
|
|
6888
|
+
"title": "p5-ext-audio",
|
|
6859
6889
|
"description": "Lightweight audio analysis from externally connected devices in P5 scenes: volume, bands, spectral features, and raw FFT/waveform via AudioStreamAPI.",
|
|
6860
6890
|
"headings": [
|
|
6861
6891
|
{
|
|
@@ -6872,7 +6902,7 @@ export const docsApi = {
|
|
|
6872
6902
|
"content": [
|
|
6873
6903
|
{
|
|
6874
6904
|
"type": "text",
|
|
6875
|
-
"markdown": "# Device Audio\n\nEach entry in `viji.devices` may expose **`device.audio`**: an [`AudioStreamAPI`](../../audio/) (or `null` when the host has not attached an audio source for that device).\n\n## Behavior\n\n- Check **`device.audio?.isConnected`** before reading values.\n- **Lightweight subset only:** `isConnected`, `volume` (`current` / `peak` / `smoothed`), `bands` (`low` / `lowMid` / `mid` / `highMid` / `high` plus each `*Smoothed` sibling), `spectral` (`brightness` / `flatness`), `getFrequencyData()`, `getWaveform()`.\n- **Not available:** beat energy, triggers, BPM, or beat events (those exist only on the main [`viji.audio`](../../audio/) stream).\n\n## Related\n\n- [External Devices: Overview](../)\n- [P5 Audio](../../audio/)\n- [Shader: Audio stream uniforms](/shader/api-reference#audio-streams)"
|
|
6905
|
+
"markdown": "# Device Audio\r\n\r\nEach entry in `viji.devices` may expose **`device.audio`**: an [`AudioStreamAPI`](../../audio/) (or `null` when the host has not attached an audio source for that device).\r\n\r\n## Behavior\r\n\r\n- Check **`device.audio?.isConnected`** before reading values.\r\n- **Lightweight subset only:** `isConnected`, `volume` (`current` / `peak` / `smoothed`), `bands` (`low` / `lowMid` / `mid` / `highMid` / `high` plus each `*Smoothed` sibling), `spectral` (`brightness` / `flatness`), `getFrequencyData()`, `getWaveform()`.\r\n- **Not available:** beat energy, triggers, BPM, or beat events (those exist only on the main [`viji.audio`](../../audio/) stream).\r\n\r\n## Related\r\n\r\n- [External Devices: Overview](../)\r\n- [P5 Audio](../../audio/)\r\n- [Shader: Audio stream uniforms](/shader/api-reference#audio-streams)"
|
|
6876
6906
|
}
|
|
6877
6907
|
]
|
|
6878
6908
|
},
|
|
@@ -8617,6 +8647,11 @@ export const docsApi = {
|
|
|
8617
8647
|
"level": 2,
|
|
8618
8648
|
"text": "Usage"
|
|
8619
8649
|
},
|
|
8650
|
+
{
|
|
8651
|
+
"id": "anchoring-visualizations-to-the-face",
|
|
8652
|
+
"level": 2,
|
|
8653
|
+
"text": "Anchoring Visualizations to the Face"
|
|
8654
|
+
},
|
|
8620
8655
|
{
|
|
8621
8656
|
"id": "pairing-texture-and-face-uniforms",
|
|
8622
8657
|
"level": 2,
|
|
@@ -8631,7 +8666,7 @@ export const docsApi = {
|
|
|
8631
8666
|
"content": [
|
|
8632
8667
|
{
|
|
8633
8668
|
"type": "text",
|
|
8634
|
-
"markdown": "# Face Mesh Uniforms\n\nFace mesh provides head pose estimation as a single `vec3` uniform. Activate the pipeline with the `// @viji-cv:faceMesh` directive at the top of your shader, either bare (always on) or toggleable (`// @viji-cv:faceMesh label:\"Face Mesh\" default:false`).\n\n> [!NOTE]\n> Individual face mesh landmark positions (468 points) are not exposed as shader uniforms: only the computed head pose is available. For full landmark access, use the [Native](../../../native/video/face-mesh/) or [P5](../../../p5/video/face-mesh/) renderer.\n\n## Uniform Reference\n\n| Uniform | Type | Description |\n|---------|------|-------------|\n| `u_face0HeadPose` | `vec3` | Head rotation `(pitch, yaw, roll)` in **degrees** |\n| `u_videoAnalysed` | `sampler2D` | The exact frame MediaPipe ran on. Pair with `u_face*` for pixel-precise effects |\n| `u_videoAnalysedAvailable` | `bool` | True after the first CV result lands |\n\n### Head Pose Values\n\n| Component | Range | Description |\n|-----------|-------|-------------|\n| `u_face0HeadPose.x` (pitch) | -90 to 90 | Looking down (negative) or up (positive) |\n| `u_face0HeadPose.y` (yaw) | -90 to 90 | Looking left (negative) or right (positive) |\n| `u_face0HeadPose.z` (roll) | -180 to 180 | Tilting head right (negative) or left (positive). Sign is chosen so `rotate(roll)` in Y-up GLSL space rotates the same visual direction as the head; this is opposite the `face.headPose.roll` value exposed to native and p5 scenes, where rotation primitives are Y-down. |\n\n## Usage\n\n```glsl\n// @renderer shader\n\nvoid main() {\n vec2 uv = gl_FragCoord.xy / u_resolution;\n vec4 video = texture2D(u_video, uv);\n\n float yaw = u_face0HeadPose.y / 90.0;\n float pitch = u_face0HeadPose.x / 90.0;\n\n vec2 offset = vec2(yaw, pitch) * 0.05;\n vec4 shifted = texture2D(u_video, uv + offset);\n\n vec3 col = mix(video.rgb, shifted.rgb, 0.5);\n gl_FragColor = vec4(col, 1.0);\n}\n```\n\n**Cost: Medium-High.** Face mesh processes 468 landmarks per face and computes head pose from the geometry.\n\n## Pairing Texture and Face Uniforms\n\nFace-mesh shaders split into two camps with different defaults.\n\n**Shaders that read pixels from the displayed frame at face-derived positions** (texture-mapped facepaint that samples skin under the mesh, warps that displace pixels along the mesh, masks that need a clean edge along the face contour): sample `u_videoAnalysed`. It is the exact frame the `u_face*` uniforms correspond to, so the effect aligns pixel-for-pixel. The displayed image stutters or holds briefly between inferences as the cost.\n\n```glsl\n// Texture-mapped face effect: u_videoAnalysed is required\nvec4 source = u_videoAnalysedAvailable\n ? texture2D(u_videoAnalysed, uv)\n : texture2D(u_video, uv);\n```\n\n**Shaders that draw on top of the face without reading its pixels** (head-pose-driven color shifts, geometry that uses `u_face0HeadPose` as a parameter source, generative effects that consume face uniforms without sampling at face positions): sample `u_video`. The face uniforms lag the displayed face by ~1 frame under fast motion, but the camera stays smooth. Reaching for `u_videoAnalysed` here would freeze the displayed video between inferences for no visual benefit.\n\n`u_videoAnalysedAvailable` becomes `true` after the first CV inference completes; before that, `u_videoAnalysed` samples a 1×1 black fallback (safe but not what you want as the visible source; use the gate above). Both textures share `u_videoResolution`.\n\nSee [Best Practices](/getting-started/best-practices/#currentframe-vs-analysedframe-pick-the-right-trade-off) for the cross-renderer trade-off discussion.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\nWhen face mesh is disabled, `u_face0HeadPose` is `vec3(0.0)`."
|
|
8669
|
+
"markdown": "# Face Mesh Uniforms\n\nFace mesh provides head pose estimation as a single `vec3` uniform. Activate the pipeline with the `// @viji-cv:faceMesh` directive at the top of your shader, either bare (always on) or toggleable (`// @viji-cv:faceMesh label:\"Face Mesh\" default:false`).\n\nFace mesh is independent of [Face Detection](../face-detection/). Each feature populates its own subset of face uniforms:\n\n- `// @viji-cv:faceMesh` populates `u_face0HeadPose` (and `u_videoAnalysed` / `u_videoAnalysedAvailable`).\n- `// @viji-cv:faceDetection` populates `u_faceCount`, `u_face0Bounds`, `u_face0Center`, `u_face0Confidence`.\n\nIf you need both head pose and a bounding box, declare both `@viji-cv:` directives.\n\n> [!NOTE]\n> Individual face mesh landmark positions (468 points) are not exposed as shader uniforms: only the computed head pose is available. For full landmark access, use the [Native](../../../native/video/face-mesh/) or [P5](../../../p5/video/face-mesh/) renderer.\n\n## Uniform Reference\n\n| Uniform | Type | Description |\n|---------|------|-------------|\n| `u_face0HeadPose` | `vec3` | Head rotation `(pitch, yaw, roll)` in **degrees** |\n| `u_videoAnalysed` | `sampler2D` | The exact frame MediaPipe ran on. Pair with `u_face*` for pixel-precise effects |\n| `u_videoAnalysedAvailable` | `bool` | True after the first CV result lands |\n\n### Head Pose Values\n\n| Component | Range | Description |\n|-----------|-------|-------------|\n| `u_face0HeadPose.x` (pitch) | -90 to 90 | Looking down (negative) or up (positive) |\n| `u_face0HeadPose.y` (yaw) | -90 to 90 | Looking left (negative) or right (positive) |\n| `u_face0HeadPose.z` (roll) | -180 to 180 | Tilting head right (negative) or left (positive). Sign is chosen so `rotate(roll)` in Y-up GLSL space rotates the same visual direction as the head; this is opposite the `face.headPose.roll` value exposed to native and p5 scenes, where rotation primitives are Y-down. |\n\n## Usage\n\n```glsl\n// @renderer shader\n\nvoid main() {\n vec2 uv = gl_FragCoord.xy / u_resolution;\n vec4 video = texture2D(u_video, uv);\n\n float yaw = u_face0HeadPose.y / 90.0;\n float pitch = u_face0HeadPose.x / 90.0;\n\n vec2 offset = vec2(yaw, pitch) * 0.05;\n vec4 shifted = texture2D(u_video, uv + offset);\n\n vec3 col = mix(video.rgb, shifted.rgb, 0.5);\n gl_FragColor = vec4(col, 1.0);\n}\n```\n\n**Cost: Medium-High.** Face mesh processes 468 landmarks per face and computes head pose from the geometry.\n\n## Anchoring Visualizations to the Face\n\nIndividual face mesh landmarks are not exposed as shader uniforms (468 × 2 floats would blow the uniform budget), so a shader cannot compute a face center from mesh data alone. If your effect anchors at the face position (head-pose arrows, face-centered glows, geometry placed on the face), also declare `// @viji-cv:faceDetection` so `u_face0Center` and `u_face0Bounds` are populated; gate the effect on `u_faceCount > 0`:\n\n```glsl\n// @viji-cv:faceDetection\n// @viji-cv:faceMesh\n\nvoid main() {\n // ...\n if (u_faceCount > 0 && length(u_face0HeadPose) > 0.0) {\n vec2 fc = u_face0Center;\n // Anchored head-pose visualization at fc using u_face0HeadPose components\n }\n}\n```\n\nThe native and p5 renderers can compute a mesh-derived center directly from landmarks (see the [Native](../../../native/video/face-mesh/#computing-a-face-center-from-landmarks) and [P5](../../../p5/video/face-mesh/#computing-a-face-center-from-landmarks) pages); shader scenes use face detection for the anchor.\n\n## Pairing Texture and Face Uniforms\n\nFace-mesh shaders split into two camps with different defaults.\n\n**Shaders that read pixels from the displayed frame at face-derived positions** (texture-mapped facepaint that samples skin under the mesh, warps that displace pixels along the mesh, masks that need a clean edge along the face contour): sample `u_videoAnalysed`. It is the exact frame the `u_face*` uniforms correspond to, so the effect aligns pixel-for-pixel. The displayed image stutters or holds briefly between inferences as the cost.\n\n```glsl\n// Texture-mapped face effect: u_videoAnalysed is required\nvec4 source = u_videoAnalysedAvailable\n ? texture2D(u_videoAnalysed, uv)\n : texture2D(u_video, uv);\n```\n\n**Shaders that draw on top of the face without reading its pixels** (head-pose-driven color shifts, geometry that uses `u_face0HeadPose` as a parameter source, generative effects that consume face uniforms without sampling at face positions): sample `u_video`. The face uniforms lag the displayed face by ~1 frame under fast motion, but the camera stays smooth. Reaching for `u_videoAnalysed` here would freeze the displayed video between inferences for no visual benefit.\n\n`u_videoAnalysedAvailable` becomes `true` after the first CV inference completes; before that, `u_videoAnalysed` samples a 1×1 black fallback (safe but not what you want as the visible source; use the gate above). Both textures share `u_videoResolution`.\n\nSee [Best Practices](/getting-started/best-practices/#currentframe-vs-analysedframe-pick-the-right-trade-off) for the cross-renderer trade-off discussion.\n\n> [!WARNING]\n> **WebGL Context Limits:** Each CV feature requires its own WebGL context for ML inference. Browsers typically allow 8-16 active WebGL contexts. Enabling too many CV features simultaneously can cause context eviction, potentially breaking the scene's own rendering. Use only the CV features you need.\n\nWhen face mesh is disabled, `u_face0HeadPose` is `vec3(0.0)`."
|
|
8635
8670
|
},
|
|
8636
8671
|
{
|
|
8637
8672
|
"type": "live-example",
|
|
@@ -9803,15 +9838,15 @@ export const docsApi = {
|
|
|
9803
9838
|
},
|
|
9804
9839
|
"ai": {
|
|
9805
9840
|
"systemPrompts": {
|
|
9806
|
-
"base": "# Viji API: Base Reference\n\nThis document is the renderer-agnostic foundation for generating Viji scenes. It is loaded into every AI turn alongside the renderer-specific reference for the renderer in use.\n\n## Architecture\n\nViji scenes run inside a **Web Worker** on an **OffscreenCanvas**. There is no DOM.\n\n- Top-level code runs once when the scene loads (parameter declarations, state, imports).\n- A `render` function (or, for shaders, `void main()`) runs every frame.\n- `fetch()` and `await import()` are available. `window`, `document`, `Image()`, and `localStorage` are not.\n- The global `viji` object exposes everything: canvas, timing, audio, video, computer vision, input, sensors, and parameters. Each renderer also has its own surface (P5's `p5.*` API; the shader's auto-injected uniforms).\n\n## Renderers\n\nThree renderers. Pick by the visual goal, not by perceived difficulty.\n\n| | Native | P5 | Shader |\n|---|---|---|---|\n| Language | JavaScript (Canvas 2D / WebGL) | JavaScript + P5.js 1.9.4 | GLSL fragment shader |\n| Best for | Full control, Three.js, generative art, pixel-perfect Canvas 2D | Creative coding with shapes, colors, transforms; ports of existing P5 sketches | GPU effects, patterns, raymarching, SDF, post-processing |\n| 3D | Yes (WebGL, Three.js via `await import()`) | Yes (`// @renderer p5 webgl`) | Yes (raymarching, SDF) |\n| External libraries | Yes (any ESM via `await import()`) | P5.js built-in | None (GPU only) |\n\n### Decision criteria\n\n- **Per-pixel GPU effects** (raymarching, fractals, distance fields, fullscreen post-processing, patterns that read every pixel): use **Shader**.\n- **3D with Three.js, custom WebGL, pixel-perfect Canvas 2D, or any external ESM library**: use **Native**.\n- **Classic P5 shapes and transforms** (`ellipse`, `line`, `bezier`, color modes, `push`/`pop`), or a port of an existing P5 sketch: use **P5**.\n- **When ambiguous, default to Native**. Audio reactivity, camera/CV, parameters, and input work in every renderer. Pick on visual style, not data sources.\n\n## Scene shape per renderer\n\nEach example below is the minimum viable entry point. Full surface lives in the matching renderer reference.\n\n**Native**:\n```javascript\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nlet angle = 0;\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n angle += speed.value * viji.deltaTime;\n ctx.clearRect(0, 0, viji.width, viji.height);\n // draw with ctx\n}\n```\n\n**P5**:\n```javascript\n// @renderer p5\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nlet angle = 0;\nfunction setup(viji, p5) { p5.colorMode(p5.HSB, 360, 100, 100); }\nfunction render(viji, p5) {\n angle += speed.value * viji.deltaTime;\n p5.background(0);\n // draw with p5.* prefixed calls\n}\n```\n\n**Shader (GLSL)**:\n```glsl\n// @renderer shader\n// @viji-slider:speed label:\"Speed\" default:1.0 min:0.1 max:5.0\n// @viji-accumulator:phase rate:speed\nvoid main() {\n vec2 uv = gl_FragCoord.xy / u_resolution;\n gl_FragColor = vec4(uv, sin(phase) * 0.5 + 0.5, 1.0);\n}\n```\n\n## Cross-cutting rules (apply to every renderer)\n\n1. **Never** touch the DOM. No `window`, `document`, `Image()`, `localStorage`. `fetch()` and `await import()` are available.\n2. **Always** declare parameters at the **top level**, never inside `render()` or `main()`. Parameters declared inside a render call are recreated every frame and break the host UI.\n3. **Always** use `viji.width` / `viji.height` (or `u_resolution` in shaders) for canvas dimensions. Never hardcode pixel sizes.\n4. **Always** use `viji.deltaTime` (or `u_deltaTime` / `@viji-accumulator` in shaders) for animation. Never count frames or assume a fixed frame rate.\n5. **Never** multiply `viji.time` (or `u_time`) by a parameter to drive animation speed. It causes visible jumps when the slider changes. Use a `deltaTime` accumulator at the top level instead. In shaders, use `@viji-accumulator`. This also applies to nested multiplications: never multiply an accumulator by another parameter. Each independent speed gets its own accumulator.\n6. **Never** allocate objects, arrays, or strings inside `render()`. Pre-allocate at the top level and reuse.\n7. **Always** check `viji.audio.isConnected` (or `u_audioVolume > 0.0`) before reading audio data. Check `viji.video.isConnected && viji.video.currentFrame` (or `u_videoConnected`) before reading video.\n8. **
|
|
9807
|
-
"native": "# Viji API: Native Renderer Reference\n\nNative scenes run JavaScript on an `OffscreenCanvas` in a Web Worker, with full access to Canvas 2D, WebGL, and WebGL2 contexts. External libraries load via `await import()`. Top-level code runs once; a `render(viji)` function runs every frame.\n\n## Architecture\n\n- The global `viji` object exposes canvas, timing, audio, video, CV, input, sensors, parameters.\n- Call `viji.useContext('2d' | 'webgl' | 'webgl2')` once at the top of `render()` (or top-level) to obtain a rendering context. After the first call, `viji.ctx` / `viji.gl` shortcuts are populated.\n- **Pick one context type for the entire scene.** Calling `useContext` with a different type after the first call returns `null`.\n- There is **no `setup()` function** in Native. All initialization goes at the top level (which supports `await` for dynamic imports).\n- The DOM is unavailable. `fetch()` and `await import()` are available.\n\n## Rules\n\n1. **Never** access `window`, `document`, `Image()`, `localStorage`, or any DOM API. `fetch()` and `await import()` are available.\n2. **Always** declare parameters at the top level, never inside `render()`.\n3. **Always** read parameters via `.value`. Color parameters also expose `.rgb` (`{ r, g, b }` in 0..255) and `.hsb` (`{ h, s, b }` with `h` in 0..360, `s` / `b` in 0..100). Color defaults accept hex (`'#ff6600'`, `'#f60'`), `{ r, g, b }` (0..255), `{ h, s, b }` (0..360 / 0..100), or CSS `'rgb(...)'` / `'hsl(...)'` strings.\n4. **Always** use `viji.width` and `viji.height` for canvas dimensions. Never hardcode pixel sizes.\n5. **Always** use `viji.deltaTime` for animation. Use `viji.time` only for constant-speed oscillations (`Math.sin(viji.time * 2.0)`). For anything driven by a parameter, use a `deltaTime` accumulator.\n6. **Never** multiply `viji.time` by a parameter; it causes visible jumps when the parameter changes. Use a `deltaTime` accumulator at the top level. This also applies to nested multiplications: each independent speed gets its own accumulator.\n\n ```javascript\n // WRONG\n const t = viji.time * speed.value;\n const rot = phase * rotSpeed.value;\n\n // RIGHT\n let phase = 0, rotPhase = 0;\n phase += speed.value * viji.deltaTime;\n rotPhase += speed.value * rotSpeed.value * viji.deltaTime;\n ```\n7. **Never** allocate objects, arrays, or strings inside `render()`. Pre-allocate at the top level and reuse.\n8. **Always** call `viji.useContext()` to get a context. Pick one type and use it for the entire scene.\n9. **Always** check `viji.audio.isConnected` before reading audio data. Check `viji.video.isConnected && viji.video.currentFrame` before drawing video.\n10. **Never** enable CV features by default. Use a toggle parameter so the user opts in.\n11. **Always** set `category` on parameters that depend on an external input: `'audio'`, `'video'`, `'interaction'`. Use creative-strength sliders (sensitivity, intensity), not on/off toggles, for inputs the host already gates. CV feature toggles (`enableFaceDetection` etc.) are the exception and stay opt-in.\n\n## API reference\n\n### Canvas and context\n\n| Member | Type | Description |\n|---|---|---|\n| `viji.canvas` | `OffscreenCanvas` | The canvas |\n| `viji.useContext('2d')` | `OffscreenCanvasRenderingContext2D` | Acquire 2D context |\n| `viji.useContext('webgl')` | `WebGLRenderingContext` | Acquire WebGL 1 |\n| `viji.useContext('webgl2')` | `WebGL2RenderingContext` | Acquire WebGL 2 |\n| `viji.ctx` | `OffscreenCanvasRenderingContext2D` | Shortcut after `useContext('2d')` |\n| `viji.gl` | `WebGLRenderingContext \\| WebGL2RenderingContext` | Shortcut after WebGL context acquired |\n| `viji.width`, `viji.height` | `number` | Current canvas dimensions in pixels |\n\n### Timing\n\n| Member | Type | Description |\n|---|---|---|\n| `viji.time` | `number` | Seconds since scene start |\n| `viji.deltaTime` | `number` | Seconds since last frame |\n| `viji.frameCount` | `number` | Total frames rendered |\n| `viji.fps` | `number` | Target frame rate (host-controlled) |\n\n### Parameters\n\nDeclare at top level. All accept `{ label, description?, group?, category? }`. Category values: `'audio'`, `'video'`, `'interaction'`, `'general'` (default).\n\n```javascript\nviji.slider(default, { min?, max?, step?, label, group?, category? }) // .value: number\nviji.color(default, { label, group?, category? }) // .value: '#rrggbb', .rgb: {r,g,b} 0..255, .hsb: {h:0..360, s/b:0..100}\nviji.toggle(default, { label, group?, category? }) // .value: boolean\nviji.select(default, { options, label, group?, category? }) // .value: string | number\nviji.number(default, { min?, max?, step?, label, group?, category? }) // .value: number\nviji.text(default, { label, group?, category?, maxLength? }) // .value: string\nviji.image(null, { label, group?, category? }) // .value: ImageBitmap | null\nviji.button({ label, description?, group?, category? }) // .value: boolean (true for 1 frame)\nviji.coordinate(default, { step?, label, group?, category? }) // .value: { x, y }, range -1..1\n```\n\n### Audio: `viji.audio`\n\nCheck `isConnected` first.\n\n| Member | Type | Notes |\n|---|---|---|\n| `isConnected` | `boolean` | Audio source active |\n| `volume.current`, `volume.peak`, `volume.smoothed` | `number` 0..1 | `smoothed` = 200ms decay envelope of `current` |\n| `bands.low`, `lowMid`, `mid`, `highMid`, `high` | `number` 0..1 | Instant band energies (20-120, 120-400, 400-1600, 1600-6000, 6000-16000 Hz) |\n| `bands.lowSmoothed`, `lowMidSmoothed`, `midSmoothed`, `highMidSmoothed`, `highSmoothed` | `number` 0..1 | 150ms decay envelope siblings (not nested) |\n| `beat.kick`, `snare`, `hat`, `any` | `number` 0..1 | Beat energy curves with a 300ms decay; peak on each detected beat then fall off |\n| `beat.kickSmoothed`, `snareSmoothed`, `hatSmoothed`, `anySmoothed` | `number` 0..1 | Smoother 500ms decay envelopes; use for ambient pulses |\n| `beat.triggers.kick`, `snare`, `hat`, `any` | `boolean` | True for exactly one frame on the matching beat, then auto-resets. OR-accumulated between frames so no beat is lost |\n| `beat.events` | `Array<{type, time, strength}>` | All beats detected since the last frame; `type` is `'kick' \\| 'snare' \\| 'hat'` (never `'any'`); `time` in milliseconds; cleared each frame |\n| `beat.bpm`, `beat.confidence`, `beat.isLocked` | `number`, `number`, `boolean` | `bpm` is `0` when no audio is connected; once audio connects it tracks the detected tempo (clamped to 60..240) with `120` as a fallback before lock-on. `confidence` in 0..1. `isLocked` is `true` on stable tempo lock |\n| `spectral.brightness`, `spectral.flatness` | `number` 0..1 | Spectral centroid; spectral flatness |\n| `getFrequencyData()` | `Uint8Array` | Raw FFT bins 0..255 |\n| `getWaveform()` | `Float32Array` | Time-domain waveform -1..1 |\n\nExternal-device audio (`viji.devices[i].audio`) and host-supplied audio streams (`viji.audioStreams[i]`) use the `AudioStreamAPI` shape: `isConnected`, `volume.{current,peak,smoothed}`, `bands.{low...high}` plus each `*Smoothed` sibling, `spectral.{brightness,flatness}`, `getFrequencyData()`, `getWaveform()`. No `beat`, BPM, triggers, or events on streams: those exist only on the main `viji.audio`.\n\n### Video: `viji.video`\n\nCheck `isConnected && currentFrame` before drawing.\n\n| Member | Type | Notes |\n|---|---|---|\n| `isConnected` | `boolean` | Video source active |\n| `currentFrame` | `OffscreenCanvas \\| ImageBitmap \\| null` | Most recent frame |\n| `frameWidth`, `frameHeight` | `number` | Source frame size |\n| `frameRate` | `number` | Source frame rate |\n| `getFrameData()` | `ImageData \\| null` | Pixel data of `currentFrame` |\n| `cv` | `VideoCVAPI` | Computer-vision surface: see below |\n\nCV data outputs (`analysedFrame`, `getAnalysedFrameData()`, `faces`, `hands`, `pose`, `segmentation`) and CV verbs (`enableFaceDetection`, etc.) all live on `viji.video.cv`, not on `viji.video` directly.\n\n**Aspect ratio.** Camera frames almost never match the canvas aspect. Drawing to `(0, 0, viji.width, viji.height)` stretches video and misaligns CV bounds. Use this helper at module scope:\n\n```javascript\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nconst v = videoFit(viji); // 'cover' or 'contain'\nctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n// CV coords are normalized 0..1 to the source frame. Map through v to align with the fitted video:\nviji.video.cv.faces.forEach(face => {\n const bx = v.x + face.bounds.x * v.width;\n const by = v.y + face.bounds.y * v.height;\n const bw = face.bounds.width * v.width;\n const bh = face.bounds.height * v.height;\n ctx.strokeRect(bx, by, bw, bh);\n});\n```\n\nDefault to `'cover'` for live cameras. Use `'contain'` for CV-overlay scenes where bounding boxes near frame edges must stay visible. `viji.video.currentFrame` is the right default for displayed video; reach for `viji.video.cv.analysedFrame` only when sampling pixels at CV-derived positions (segmentation compositing, face-mesh texturing, landmark-anchored color sampling). The common fallback pattern is `viji.video.cv.analysedFrame ?? viji.video.currentFrame` during the brief window before the first CV inference lands.\n\n### Computer Vision: `viji.video.cv`\n\nEnable features explicitly. Never enable by default.\n\n```javascript\nawait viji.video.cv.enableFaceDetection(true | false);\nawait viji.video.cv.enableFaceMesh(true | false); // implies face detection\nawait viji.video.cv.enableEmotionDetection(true | false); // implies face mesh\nawait viji.video.cv.enableHandTracking(true | false);\nawait viji.video.cv.enablePoseDetection(true | false);\nawait viji.video.cv.enableBodySegmentation(true | false);\nviji.video.cv.getActiveFeatures(); // CVFeature[]\nviji.video.cv.isProcessing(); // boolean\n```\n\nData outputs on `viji.video.cv`:\n- `analysedFrame: OffscreenCanvas | null`: the exact frame paired with the current CV results. `null` until the first inference lands after a feature is enabled. Use the pattern `viji.video.cv.analysedFrame ?? viji.video.currentFrame` to fall back during the startup window.\n- `getAnalysedFrameData(): ImageData | null`: pixel data of `analysedFrame`, cached and re-extracted only on new CV results.\n\n**`viji.video.cv.faces: FaceData[]`**: `id`, `bounds: {x,y,width,height}` (normalized 0..1), `center: {x,y}`, `confidence`, `landmarks: {x,y,z?}[]`, `expressions: { neutral, happy, sad, angry, surprised, disgusted, fearful }` (0..1 each), `headPose: { pitch, yaw, roll }`, `blendshapes` (52 ARKit coefficients: `browDownLeft`/`Right`, `browInnerUp`, `browOuterUpLeft`/`Right`, `cheekPuff`, `cheekSquintLeft`/`Right`, `eyeBlinkLeft`/`Right`, `eyeLookDown`/`In`/`Out`/`UpLeft`/`Right`, `eyeSquintLeft`/`Right`, `eyeWideLeft`/`Right`, `jawForward`/`Left`/`Open`/`Right`, `mouthClose`/`DimpleLeft`/`Right`/`FrownLeft`/`Right`/`Funnel`/`Left`/`LowerDownLeft`/`Right`/`PressLeft`/`Right`/`Pucker`/`Right`/`RollLower`/`Upper`/`ShrugLower`/`Upper`/`SmileLeft`/`Right`/`StretchLeft`/`Right`/`UpperUpLeft`/`Right`, `noseSneerLeft`/`Right`, `tongueOut`: all 0..1).\n\n**`viji.video.cv.hands: HandData[]`**: `id`, `handedness: 'left' | 'right'`, `confidence`, `bounds`, `landmarks: {x,y,z}[]` (21 points), `palm: {x,y,z}`, `gestures: { fist, openPalm, peace, thumbsUp, thumbsDown, pointing, iLoveYou }` (0..1 confidence each).\n\n**`viji.video.cv.pose: PoseData | null`**: `confidence`, `landmarks: {x,y,z,visibility}[]` (33 points), plus body-part arrays `face`, `torso`, `leftArm`, `rightArm`, `leftLeg`, `rightLeg`.\n\n**`viji.video.cv.segmentation: SegmentationData | null`**: `mask: Uint8Array` (each byte is `0` for background or `1` for person; length = `width * height`), `width`, `height`. Note: the shader `u_segmentationMask` sampler is sampled as a normalized `float` (0.0 = background, 1.0 = person); the JS `mask` byte values are 0/1, not 0/255.\n\n### Input\n\n**Pointer** (unified mouse and touch): `viji.pointer`\n| Member | Type | Notes |\n|---|---|---|\n| `x`, `y` | `number` | Position in pixels |\n| `deltaX`, `deltaY` | `number` | Per-frame movement |\n| `isDown` | `boolean` | Pressed or touching |\n| `wasPressed`, `wasReleased` | `boolean` | One-frame edge triggers |\n| `isInCanvas` | `boolean` | Inside canvas bounds |\n| `type` | `'mouse' \\| 'touch' \\| 'none'` | Active input type |\n\n**Mouse**: `viji.mouse`: `x`, `y`, `isInCanvas`, `isPressed`, `leftButton`, `rightButton`, `middleButton`, `deltaX`, `deltaY`, `wheelDelta`, `wheelX`, `wheelY`, `wasPressed`, `wasReleased`, `wasMoved`.\n\n**Keyboard**: `viji.keyboard`: `isPressed(key)`, `wasPressed(key)`, `wasReleased(key)`, `activeKeys: Set<string>`, `pressedThisFrame`, `releasedThisFrame`, `lastKeyPressed`, `lastKeyReleased`, `shift`, `ctrl`, `alt`, `meta`.\n\n**Touch**: `viji.touches`: `count`, `points: TouchPoint[]`, `started`, `moved`, `ended` (each a `TouchPoint[]`), `primary: TouchPoint | null`. Each `TouchPoint`: `id`, `x`, `y`, `pressure`, `radius`, `radiusX`, `radiusY`, `rotationAngle`, `force`, `isInCanvas`, `deltaX`, `deltaY`, `velocity: {x,y}`, `isNew`, `isActive`, `isEnding`.\n\n### Sensors and external devices\n\n**`viji.device`**: host device motion and orientation:\n- `motion: DeviceMotionData | null`: `acceleration: {x,y,z}` (m/s²), `accelerationIncludingGravity`, `rotationRate: {alpha,beta,gamma}` (deg/s), `interval` (ms).\n- `orientation: DeviceOrientationData | null`: `alpha` (0-360° compass), `beta` (-180..180° front-back tilt), `gamma` (-90..90° left-right tilt), `absolute`.\n\n**`viji.devices`**: `DeviceState[]`, externally connected devices. Each: `id`, `name`, `motion`, `orientation`, `video` (`VideoAPI | null`, same shape as `viji.video` but without CV), `audio` (`AudioStreamAPI | null`, lightweight; no beat or BPM).\n\n### Host-supplied streams\n\n- `viji.videoStreams: VideoAPI[]`: extra video sources from the host compositor. Same shape as `viji.video`. May be empty.\n- `viji.audioStreams: AudioStreamAPI[]`: extra audio sources from the host. Lightweight shape (no beat or BPM). May be empty.\n\n### External libraries\n\n```javascript\nconst THREE = await import('https://esm.sh/three@0.160.0');\nconst renderer = new THREE.WebGLRenderer({ canvas: viji.canvas, antialias: true });\nrenderer.setSize(viji.width, viji.height, false); // always pass false to skip CSS styling\n```\n\nPin library versions in the import URL. Always pass `viji.canvas` to the renderer. Always pass `false` as the third argument to Three.js `setSize`. Handle resize by comparing `viji.width` / `viji.height` against previous values inside `render()`.\n\n## Template\n\n```javascript\nconst bgColor = viji.color('#1a1a2e', { label: 'Background' });\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nconst count = viji.slider(12, { min: 3, max: 30, step: 1, label: 'Count' });\n\nlet angle = 0;\n\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n angle += speed.value * viji.deltaTime;\n\n ctx.fillStyle = bgColor.value;\n ctx.fillRect(0, 0, viji.width, viji.height);\n\n const cx = viji.width / 2;\n const cy = viji.height / 2;\n const radius = Math.min(viji.width, viji.height) * 0.3;\n const dotSize = Math.min(viji.width, viji.height) * 0.02;\n const n = Math.floor(count.value);\n\n for (let i = 0; i < n; i++) {\n const a = angle + (i / n) * Math.PI * 2;\n const x = cx + Math.cos(a) * radius;\n const y = cy + Math.sin(a) * radius;\n const hue = (i / n) * 360;\n ctx.beginPath();\n ctx.arc(x, y, dotSize, 0, Math.PI * 2);\n ctx.fillStyle = `hsl(${hue}, 80%, 60%)`;\n ctx.fill();\n }\n}\n```\n",
|
|
9808
|
-
"p5": "# Viji API: P5 Renderer Reference\n\nP5 scenes use P5.js 1.9.4 inside a Web Worker on an `OffscreenCanvas`. P5 runs in **instance mode**: every P5 function and constant requires the `p5.` prefix. The first line of the scene must be `// @renderer p5` (2D) or `// @renderer p5 webgl` (3D / WEBGL).\n\n## Architecture\n\n- The global `viji` object exposes canvas, timing, audio, video, CV, input, sensors, parameters: identical to Native.\n- The canvas and its rendering context are managed by P5. **Do not call `createCanvas()`.** Viji creates the canvas in the mode declared by the `// @renderer` directive.\n- `viji.useContext()` is **not available** in P5 mode (the context belongs to P5).\n- Top-level code runs once. `function setup(viji, p5)` runs once for configuration. `function render(viji, p5)` runs every frame.\n- The `p5` instance passed to `setup` and `render` is the P5 instance. Use `p5.background`, `p5.fill`, `p5.circle`, `p5.PI`, etc. Unprefixed names will not resolve.\n\n## Rules\n\n1. **Always** add `// @renderer p5` (2D) or `// @renderer p5 webgl` (WEBGL) as the first line.\n2. **Always** use `render(viji, p5)` (not `draw()`) and `setup(viji, p5)` (not `setup()`).\n3. **Always** prefix every P5 function and constant with `p5.`:\n - `background(0)` -> `p5.background(0)`\n - `fill(255)` -> `p5.fill(255)`\n - `PI` -> `p5.PI`, `TWO_PI` -> `p5.TWO_PI`, `HSB` -> `p5.HSB`\n - `createVector(1, 0)` -> `p5.createVector(1, 0)`\n - `map(v, 0, 1, 0, 255)` -> `p5.map(v, 0, 1, 0, 255)`\n - `noise(x)` -> `p5.noise(x)`, `random()` -> `p5.random()`\n4. **Never** call `createCanvas()`. The canvas is created and managed by Viji.\n5. **Never** use `preload()`. Use `viji.image(null, { label })` for images, or `fetch()` inside `setup()`.\n6. **Never** use P5 event callbacks (`mousePressed`, `mouseDragged`, `mouseReleased`, `keyPressed`, `keyReleased`, `keyTyped`, `touchStarted`, `touchMoved`, `touchEnded`). Read state in `render()` via `viji.pointer`, `viji.mouse`, `viji.keyboard`, `viji.touches`. Use `wasPressed` / `wasReleased` for one-frame edges.\n7. **Never** use `loadImage()`, `loadFont()`, `loadJSON()`, `loadModel()`, `loadShader()`. Use `viji.image()` or `fetch()`.\n8. **Never** use `p5.frameRate()`, `p5.save()`, `p5.saveCanvas()`, `p5.saveFrames()`. The host controls frame rate and capture.\n9. **Never** use `createCapture()` or `createVideo()`. Use `viji.video.*`.\n10. **Never** use `p5.dom` or `p5.sound`. Use Viji parameters and `viji.audio.*`.\n11. **Never** access `window`, `document`, `Image()`, `localStorage`. `fetch()` is available.\n12. **Always** declare parameters at the top level, never inside `render()` or `setup()`.\n13. **Always** read parameters via `.value`. Color parameters also expose `.rgb` (matches `colorMode(RGB, 255)`) and `.hsb` (matches `colorMode(HSB, 360, 100, 100)`). Color defaults accept hex, `{ r, g, b }` (0..255), `{ h, s, b }` (0..360 / 0..100), or CSS color strings.\n14. **Always** use `viji.width` and `viji.height` for canvas dimensions. Never hardcode pixel sizes.\n15. **Always** use `viji.deltaTime` for animation:\n ```javascript\n let angle = 0;\n function render(viji, p5) { angle += speed.value * viji.deltaTime; }\n ```\n16. **Never** multiply `viji.time` by a parameter; it causes visible jumps when the parameter changes. Use a `deltaTime` accumulator. This also applies to nested multiplications; each independent speed gets its own accumulator.\n17. **Never** allocate objects, arrays, or strings inside `render()`. Pre-allocate at the top level and reuse.\n18. **Image parameters with P5:** use `.p5` (a `P5Image`) instead of `.value` when passing to `p5.image()`:\n ```javascript\n const photo = viji.image(null, { label: 'Photo' });\n function render(viji, p5) {\n if (photo.value) p5.image(photo.p5, 0, 0, viji.width, viji.height);\n }\n ```\n19. **Video drawing.** In 2D, use `p5.image(viji.video.currentFrame, ...)` or `p5.drawingContext.drawImage(...)`. In WEBGL, use `p5.image(viji.video.currentFrame, ...)` only (`p5.drawingContext` is WebGL, not Canvas 2D). **Always preserve aspect ratio** via the `videoFit` helper below.\n20. `p5.createGraphics(w, h)` is **2D only** (`createGraphics(w, h, p5.WEBGL)` is not supported). It creates an OffscreenCanvas internally.\n21. Fonts: `p5.textFont()` only with CSS generic names (`monospace`, `serif`, `sans-serif`). `loadFont()` is not available.\n22. `p5.tint()` and `p5.blendMode()` work normally.\n23. **Canvas mode is declared by the `// @renderer` directive.** Use `// @renderer p5` for 2D, `// @renderer p5 webgl` for WEBGL. Never call `createCanvas()` or `createCanvas(..., p5.WEBGL)`.\n24. In WEBGL scenes, `p5.drawingContext` is a WebGL context. Use P5 3D drawing primitives or `p5.image()` / textures for images and video.\n25. `p5.pixelDensity()` defaults to 1 in the worker. `p5.loadPixels()` and `p5.pixels[]` work in 2D scenes; WEBGL pixel readback follows standard P5.js rules.\n26. **Always** check `viji.audio.isConnected` before reading audio data.\n27. **Always** check `viji.video.isConnected && viji.video.currentFrame` before drawing video.\n28. **Never** enable CV features by default; use toggle parameters.\n29. **Always** set `category` on input-dependent parameters (`'audio'`, `'video'`, `'interaction'`). Use creative-strength sliders, not on/off toggles, for inputs the host already gates. CV feature toggles stay opt-in.\n\n## Video aspect helper\n\n```javascript\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nif (viji.video.isConnected && viji.video.currentFrame) {\n const v = videoFit(viji, 'cover');\n p5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n}\n\n// CV bounds and landmarks are normalized 0..1 to the source frame.\n// Map them through v to align with the drawn video.\n```\n\n## P5 to Viji mapping\n\n| Standard P5.js | Viji P5 equivalent |\n|---|---|\n| `width` / `height` | `viji.width` / `viji.height` |\n| `mouseX` / `mouseY` | `viji.pointer.x` / `viji.pointer.y` (or `viji.mouse.x` / `viji.mouse.y`) |\n| `mouseIsPressed` | `viji.pointer.isDown` |\n| `mouseButton === LEFT` | `viji.mouse.leftButton` |\n| `keyIsPressed` | `viji.keyboard.activeKeys.size > 0` |\n| `keyIsDown(code)` | `viji.keyboard.isPressed('keyName')` |\n| `key` | `viji.keyboard.lastKeyPressed` |\n| `frameCount` | `viji.frameCount` (or a `viji.deltaTime` accumulator) |\n| `frameRate(n)` | Remove. Host controls frame rate. |\n| `createCanvas(w, h)` | Remove. Canvas is provided. |\n| `preload()` | Remove. Use `viji.image()` or `fetch()` in `setup()`. |\n| `loadImage(url)` | `viji.image(null, { label: 'Image' })`; pass `.p5` to `p5.image()`. |\n| `save()` / `saveCanvas()` | Remove. Host handles capture. |\n| `mousePressed()` / `mouseReleased()` callbacks | Check `viji.pointer.wasPressed` / `wasReleased` inside `render()`. |\n| `keyPressed()` / `keyReleased()` callbacks | Check `viji.keyboard.wasPressed(key)` / `wasReleased(key)` inside `render()`. |\n\n## API reference\n\nAll `viji.*` members are identical to Native (same object, same types).\n\n### Canvas and timing\n\n| Member | Type | Notes |\n|---|---|---|\n| `viji.canvas` | `OffscreenCanvas` | Managed by P5 |\n| `viji.width`, `viji.height` | `number` | Canvas dimensions |\n| `viji.time` | `number` | Seconds since scene start |\n| `viji.deltaTime` | `number` | Seconds since last frame |\n| `viji.frameCount` | `number` | Total frames rendered |\n| `viji.fps` | `number` | Target frame rate |\n\n### Parameters\n\n```javascript\nviji.slider(default, { min?, max?, step?, label, group?, category? }) // .value: number\nviji.color(default, { label, group?, category? }) // .value, .rgb, .hsb\nviji.toggle(default, { label, group?, category? }) // .value: boolean\nviji.select(default, { options, label, group?, category? }) // .value: string | number\nviji.number(default, { min?, max?, step?, label, group?, category? }) // .value: number\nviji.text(default, { label, group?, category?, maxLength? }) // .value: string\nviji.image(null, { label, group?, category? }) // .value: ImageBitmap | null, .p5: P5Image\nviji.button({ label, description?, group?, category? }) // .value: boolean (1 frame)\nviji.coordinate(default, { step?, label, group?, category? }) // .value: { x, y }, range -1..1\n```\n\n### Audio: `viji.audio`\n\nCheck `isConnected` first. Members:\n\n`isConnected` (boolean); `volume.{current, peak, smoothed}` (0..1; `smoothed` 200ms decay); `bands.{low, lowMid, mid, highMid, high}` (0..1; ranges 20-120, 120-400, 400-1600, 1600-6000, 6000-16000 Hz) and the sibling `*Smoothed` envelopes (150ms decay); `beat.{kick, snare, hat, any}` (0..1, 300ms decay curves; peak on each detected beat); `beat.{kickSmoothed, snareSmoothed, hatSmoothed, anySmoothed}` (500ms decay envelopes); `beat.triggers.{kick, snare, hat, any}` (boolean, true for exactly one frame, OR-accumulated between frames); `beat.events: Array<{type, time, strength}>` (`type` is `'kick' | 'snare' | 'hat'`; `time` ms; cleared each frame); `beat.bpm` (`0` when no audio is connected; once audio connects it tracks the detected tempo clamped to 60..240, with `120` as a fallback before lock-on), `beat.confidence` (0..1), `beat.isLocked` (boolean); `spectral.{brightness, flatness}` (0..1); `getFrequencyData(): Uint8Array` (1024 FFT bins, 0..255); `getWaveform(): Float32Array` (2048 samples, -1..1).\n\nExternal-device audio (`viji.devices[i].audio`) and host streams (`viji.audioStreams[i]`) follow the `AudioStreamAPI` shape: `isConnected`, `volume`, `bands` (instant + smoothed siblings), `spectral`, `getFrequencyData`, `getWaveform`. No beat, BPM, triggers, or events on streams.\n\n### Video: `viji.video`\n\nCheck `isConnected && currentFrame` before drawing. Members:\n\n`isConnected` (boolean); `currentFrame` (`OffscreenCanvas | ImageBitmap | null`); `frameWidth`, `frameHeight`, `frameRate`; `getFrameData(): ImageData | null`; `cv: VideoCVAPI` (CV outputs and verbs live here, not on `viji.video` directly).\n\n### Computer Vision: `viji.video.cv`\n\nEnable explicitly:\n\n```javascript\nawait viji.video.cv.enableFaceDetection(true | false);\nawait viji.video.cv.enableFaceMesh(true | false);\nawait viji.video.cv.enableEmotionDetection(true | false);\nawait viji.video.cv.enableHandTracking(true | false);\nawait viji.video.cv.enablePoseDetection(true | false);\nawait viji.video.cv.enableBodySegmentation(true | false);\nviji.video.cv.getActiveFeatures(); // CVFeature[]\nviji.video.cv.isProcessing(); // boolean\n```\n\nData outputs on `viji.video.cv`:\n- `analysedFrame: OffscreenCanvas | null`: the frame paired with the current CV results. `null` until the first inference lands. Use `viji.video.cv.analysedFrame ?? viji.video.currentFrame` to fall back during startup.\n- `getAnalysedFrameData(): ImageData | null`: pixel data of `analysedFrame`.\n\n**`faces: FaceData[]`**: `id`, `bounds`, `center`, `confidence`, `landmarks`, `expressions` (`neutral`, `happy`, `sad`, `angry`, `surprised`, `disgusted`, `fearful` 0..1), `headPose` (`pitch`, `yaw`, `roll`), `blendshapes` (52 ARKit coefficients 0..1).\n\n**`hands: HandData[]`**: `id`, `handedness: 'left' | 'right'`, `confidence`, `bounds`, `landmarks` (21 points), `palm`, `gestures` (`fist`, `openPalm`, `peace`, `thumbsUp`, `thumbsDown`, `pointing`, `iLoveYou` 0..1).\n\n**`pose: PoseData | null`**: `confidence`, `landmarks` (33 points), body-part arrays `face`, `torso`, `leftArm`, `rightArm`, `leftLeg`, `rightLeg`.\n\n**`segmentation: SegmentationData | null`**: `mask: Uint8Array` (each byte is `0` for background or `1` for person; length = `width * height`), `width`, `height`.\n\n### Input\n\n- `viji.pointer`: `x`, `y`, `deltaX`, `deltaY`, `isDown`, `wasPressed`, `wasReleased`, `isInCanvas`, `type` (`'mouse' | 'touch' | 'none'`).\n- `viji.mouse`: `x`, `y`, `isInCanvas`, `isPressed`, `leftButton`, `rightButton`, `middleButton`, `deltaX`, `deltaY`, `wheelDelta`, `wheelX`, `wheelY`, `wasPressed`, `wasReleased`, `wasMoved`.\n- `viji.keyboard`: `isPressed(key)`, `wasPressed(key)`, `wasReleased(key)`, `activeKeys`, `pressedThisFrame`, `releasedThisFrame`, `lastKeyPressed`, `lastKeyReleased`, `shift`, `ctrl`, `alt`, `meta`.\n- `viji.touches`: `count`, `points`, `started`, `moved`, `ended`, `primary`. Each `TouchPoint`: `id`, `x`, `y`, `pressure`, `radius`, `radiusX`, `radiusY`, `rotationAngle`, `force`, `isInCanvas`, `deltaX`, `deltaY`, `velocity`, `isNew`, `isActive`, `isEnding`.\n\n### Sensors and external devices\n\n`viji.device.motion`: `acceleration {x,y,z}` (m/s²), `accelerationIncludingGravity`, `rotationRate {alpha,beta,gamma}` (deg/s), `interval` (ms).\n`viji.device.orientation`: `alpha` (0-360°), `beta` (-180..180°), `gamma` (-90..90°), `absolute`.\n\n`viji.devices: DeviceState[]`: each entry: `id`, `name`, `motion`, `orientation`, `video` (`VideoAPI | null`, no CV), `audio` (`AudioStreamAPI | null`, lightweight).\n\n### Host streams\n\n`viji.videoStreams: VideoAPI[]`: extra video sources. May be empty.\n`viji.audioStreams: AudioStreamAPI[]`: extra audio sources. May be empty. Lightweight shape (no beat / BPM).\n\n## Template\n\n```javascript\n// @renderer p5\n\nconst bgColor = viji.color('#1a1a2e', { label: 'Background' });\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nconst count = viji.slider(8, { min: 3, max: 30, step: 1, label: 'Count' });\n\nlet angle = 0;\n\nfunction setup(viji, p5) {\n p5.colorMode(p5.HSB, 360, 100, 100);\n}\n\nfunction render(viji, p5) {\n angle += speed.value * viji.deltaTime;\n p5.background(bgColor.value);\n\n const cx = viji.width / 2;\n const cy = viji.height / 2;\n const radius = p5.min(viji.width, viji.height) * 0.3;\n const dotSize = p5.min(viji.width, viji.height) * 0.04;\n const n = p5.floor(count.value);\n\n p5.noStroke();\n for (let i = 0; i < n; i++) {\n const a = angle + (i / n) * p5.TWO_PI;\n const x = cx + p5.cos(a) * radius;\n const y = cy + p5.sin(a) * radius;\n p5.fill((i / n) * 360, 80, 90);\n p5.circle(x, y, dotSize);\n }\n}\n```\n",
|
|
9841
|
+
"base": "# Viji API: Base Reference\n\nThis document is the renderer-agnostic foundation for generating Viji scenes. It is loaded into every AI turn alongside the renderer-specific reference for the renderer in use.\n\n## Architecture\n\nViji scenes run inside a **Web Worker** on an **OffscreenCanvas**. There is no DOM.\n\n- Top-level code runs once when the scene loads (parameter declarations, state, imports).\n- A `render` function (or, for shaders, `void main()`) runs every frame.\n- `fetch()` and `await import()` are available. `window`, `document`, `Image()`, and `localStorage` are not.\n- The global `viji` object exposes everything: canvas, timing, audio, video, computer vision, input, sensors, and parameters. Each renderer also has its own surface (P5's `p5.*` API; the shader's auto-injected uniforms).\n\n## Renderers\n\nThree renderers. Pick by the visual goal, not by perceived difficulty.\n\n| | Native | P5 | Shader |\n|---|---|---|---|\n| Language | JavaScript (Canvas 2D / WebGL) | JavaScript + P5.js 1.9.4 | GLSL fragment shader |\n| Best for | Full control, Three.js, generative art, pixel-perfect Canvas 2D | Creative coding with shapes, colors, transforms; ports of existing P5 sketches | GPU effects, patterns, raymarching, SDF, post-processing |\n| 3D | Yes (WebGL, Three.js via `await import()`) | Yes (`// @renderer p5 webgl`) | Yes (raymarching, SDF) |\n| External libraries | Yes (any ESM via `await import()`) | P5.js built-in | None (GPU only) |\n\n### Decision criteria\n\n- **Per-pixel GPU effects** (raymarching, fractals, distance fields, fullscreen post-processing, patterns that read every pixel): use **Shader**.\n- **3D with Three.js, custom WebGL, pixel-perfect Canvas 2D, or any external ESM library**: use **Native**.\n- **Classic P5 shapes and transforms** (`ellipse`, `line`, `bezier`, color modes, `push`/`pop`), or a port of an existing P5 sketch: use **P5**.\n- **When ambiguous, default to Native**. Audio reactivity, camera/CV, parameters, and input work in every renderer. Pick on visual style, not data sources.\n\n## Scene shape per renderer\n\nEach example below is the minimum viable entry point. Full surface lives in the matching renderer reference.\n\n**Native**:\n```javascript\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nlet angle = 0;\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n angle += speed.value * viji.deltaTime;\n ctx.clearRect(0, 0, viji.width, viji.height);\n // draw with ctx\n}\n```\n\n**P5**:\n```javascript\n// @renderer p5\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nlet angle = 0;\nfunction setup(viji, p5) { p5.colorMode(p5.HSB, 360, 100, 100); }\nfunction render(viji, p5) {\n angle += speed.value * viji.deltaTime;\n p5.background(0);\n // draw with p5.* prefixed calls\n}\n```\n\n**Shader (GLSL)**:\n```glsl\n// @renderer shader\n// @viji-slider:speed label:\"Speed\" default:1.0 min:0.1 max:5.0\n// @viji-accumulator:phase rate:speed\nvoid main() {\n vec2 uv = gl_FragCoord.xy / u_resolution;\n gl_FragColor = vec4(uv, sin(phase) * 0.5 + 0.5, 1.0);\n}\n```\n\n## Cross-cutting rules (apply to every renderer)\n\n1. **Never** touch the DOM. No `window`, `document`, `Image()`, `localStorage`. `fetch()` and `await import()` are available.\n2. **Always** declare parameters at the **top level**, never inside `render()` or `main()`. Parameters declared inside a render call are recreated every frame and break the host UI.\n3. **Always** use `viji.width` / `viji.height` (or `u_resolution` in shaders) for canvas dimensions. Never hardcode pixel sizes.\n4. **Always** use `viji.deltaTime` (or `u_deltaTime` / `@viji-accumulator` in shaders) for animation. Never count frames or assume a fixed frame rate.\n5. **Never** multiply `viji.time` (or `u_time`) by a parameter to drive animation speed. It causes visible jumps when the slider changes. Use a `deltaTime` accumulator at the top level instead. In shaders, use `@viji-accumulator`. This also applies to nested multiplications: never multiply an accumulator by another parameter. Each independent speed gets its own accumulator.\n6. **Never** allocate objects, arrays, or strings inside `render()`. Pre-allocate at the top level and reuse.\n7. **Always** check `viji.audio.isConnected` (or `u_audioVolume > 0.0`) before reading audio data. Check `viji.video.isConnected && viji.video.currentFrame` (or `u_videoConnected`) before reading video.\n8. **CV features are independent**: each `enableX()` populates only its own subset of fields. Face detection populates `face.bounds` / `center` / `confidence`; face mesh populates `face.landmarks` / `headPose`; emotion detection populates `face.expressions` / `blendshapes`. Fields whose source model is not enabled read as `null` (or `[]` for `landmarks`) — always null-check before reading. Enable from module scope for always-on CV scenes, or per-frame inside `render()` gated by a toggle parameter for opt-in CV (canonical pattern). Each active CV feature uses its own WebGL context; 4+ on low-end hardware can hit context limits, so enable only what you need.\n9. **Always** set `category` on parameters that depend on an external input: `category: 'audio'` for audio-driven controls, `category: 'video'` for video / CV-driven controls, `category: 'interaction'` for mouse / keyboard / touch controls. Omit `category` (defaults to `'general'`) for parameters that work without external input. Use creative-strength sliders, not on/off toggles, for inputs the host already gates.\n10. **In P5**: prefix every P5 function and constant with `p5.` (instance mode). Never call `createCanvas()`; Viji creates the canvas.\n11. **In shaders**: never redeclare `precision`, never redeclare built-in uniforms (`u_time`, `u_resolution`, etc.), never redeclare parameter uniforms. Never use the `u_` prefix for your own parameter names; it is reserved for Viji's built-ins.\n\n## Parameter declaration shape (cross-renderer)\n\nNative and P5 use JavaScript calls at the top level:\n\n```javascript\nviji.slider(default, { min, max, step, label, group, category }) // .value: number\nviji.color(default, { label }) // .value: '#rrggbb', .rgb, .hsb\nviji.toggle(default, { label }) // .value: boolean\nviji.select(default, { options, label }) // .value: string | number\nviji.number(default, { min, max, step, label }) // .value: number\nviji.text(default, { label, maxLength }) // .value: string\nviji.image(null, { label }) // .value: ImageBitmap | null\nviji.button({ label }) // .value: boolean (true for 1 frame)\nviji.coordinate({ x: 0, y: 0 }, { step, label, group, category }) // .value: { x, y }, range -1..1\n```\n\nShaders use GLSL comment directives that compile to typed uniforms:\n\n```glsl\n// @viji-slider:name label:\"Label\" default:1.0 min:0.0 max:5.0 -> uniform float name;\n// @viji-color:name label:\"Color\" default:#ff6600 -> uniform vec3 name;\n// @viji-toggle:name label:\"Toggle\" default:false -> uniform bool name;\n// @viji-select:name label:\"Mode\" default:0 options:[\"A\",\"B\"] -> uniform int name;\n// @viji-number:name label:\"Count\" default:10.0 min:1.0 max:100.0 -> uniform float name;\n// @viji-image:name label:\"Texture\" -> uniform sampler2D name;\n// @viji-button:name label:\"Reset\" -> uniform bool name;\n// @viji-coordinate:name label:\"Pos\" default:[0.0,0.0] -> uniform vec2 name; // range -1..1\n// @viji-accumulator:name rate:speed -> uniform float name;\n```\n\nDirectives use `//` line comments only. Block comments (`/* */`) are not parsed.\n\n## Where to look next\n\nFor the full API surface of the renderer in use, consult the matching reference: Native, P5, or Shader. Each renderer reference includes the complete data shapes for audio, video, CV, input, sensors, external devices, and renderer-specific patterns (e.g. `viji.useContext` for Native, `videoFit` for P5, the complete uniform table for Shader).\n",
|
|
9842
|
+
"native": "# Viji API: Native Renderer Reference\n\nNative scenes run JavaScript on an `OffscreenCanvas` in a Web Worker, with full access to Canvas 2D, WebGL, and WebGL2 contexts. External libraries load via `await import()`. Top-level code runs once; a `render(viji)` function runs every frame.\n\n## Architecture\n\n- The global `viji` object exposes canvas, timing, audio, video, CV, input, sensors, parameters.\n- Call `viji.useContext('2d' | 'webgl' | 'webgl2')` once at the top of `render()` (or top-level) to obtain a rendering context. After the first call, `viji.ctx` / `viji.gl` shortcuts are populated.\n- **Pick one context type for the entire scene.** Calling `useContext` with a different type after the first call returns `null`.\n- There is **no `setup()` function** in Native. All initialization goes at the top level (which supports `await` for dynamic imports).\n- The DOM is unavailable. `fetch()` and `await import()` are available.\n\n## Rules\n\n1. **Never** access `window`, `document`, `Image()`, `localStorage`, or any DOM API. `fetch()` and `await import()` are available.\n2. **Always** declare parameters at the top level, never inside `render()`.\n3. **Always** read parameters via `.value`. Color parameters also expose `.rgb` (`{ r, g, b }` in 0..255) and `.hsb` (`{ h, s, b }` with `h` in 0..360, `s` / `b` in 0..100). Color defaults accept hex (`'#ff6600'`, `'#f60'`), `{ r, g, b }` (0..255), `{ h, s, b }` (0..360 / 0..100), or CSS `'rgb(...)'` / `'hsl(...)'` strings.\n4. **Always** use `viji.width` and `viji.height` for canvas dimensions. Never hardcode pixel sizes.\n5. **Always** use `viji.deltaTime` for animation. Use `viji.time` only for constant-speed oscillations (`Math.sin(viji.time * 2.0)`). For anything driven by a parameter, use a `deltaTime` accumulator.\n6. **Never** multiply `viji.time` by a parameter; it causes visible jumps when the parameter changes. Use a `deltaTime` accumulator at the top level. This also applies to nested multiplications: each independent speed gets its own accumulator.\n\n ```javascript\n // WRONG\n const t = viji.time * speed.value;\n const rot = phase * rotSpeed.value;\n\n // RIGHT\n let phase = 0, rotPhase = 0;\n phase += speed.value * viji.deltaTime;\n rotPhase += speed.value * rotSpeed.value * viji.deltaTime;\n ```\n7. **Never** allocate objects, arrays, or strings inside `render()`. Pre-allocate at the top level and reuse.\n8. **Always** call `viji.useContext()` to get a context. Pick one type and use it for the entire scene.\n9. **Always** check `viji.audio.isConnected` before reading audio data. Check `viji.video.isConnected && viji.video.currentFrame` before drawing video.\n10. **Never** enable CV features by default. Use a toggle parameter so the user opts in.\n11. **Always** set `category` on parameters that depend on an external input: `'audio'`, `'video'`, `'interaction'`. Use creative-strength sliders (sensitivity, intensity), not on/off toggles, for inputs the host already gates. CV feature toggles (`enableFaceDetection` etc.) are the exception and stay opt-in.\n\n## API reference\n\n### Canvas and context\n\n| Member | Type | Description |\n|---|---|---|\n| `viji.canvas` | `OffscreenCanvas` | The canvas |\n| `viji.useContext('2d')` | `OffscreenCanvasRenderingContext2D` | Acquire 2D context |\n| `viji.useContext('webgl')` | `WebGLRenderingContext` | Acquire WebGL 1 |\n| `viji.useContext('webgl2')` | `WebGL2RenderingContext` | Acquire WebGL 2 |\n| `viji.ctx` | `OffscreenCanvasRenderingContext2D` | Shortcut after `useContext('2d')` |\n| `viji.gl` | `WebGLRenderingContext \\| WebGL2RenderingContext` | Shortcut after WebGL context acquired |\n| `viji.width`, `viji.height` | `number` | Current canvas dimensions in pixels |\n\n### Timing\n\n| Member | Type | Description |\n|---|---|---|\n| `viji.time` | `number` | Seconds since scene start |\n| `viji.deltaTime` | `number` | Seconds since last frame |\n| `viji.frameCount` | `number` | Total frames rendered |\n| `viji.fps` | `number` | Target frame rate (host-controlled) |\n\n### Parameters\n\nDeclare at top level. All accept `{ label, description?, group?, category? }`. Category values: `'audio'`, `'video'`, `'interaction'`, `'general'` (default).\n\n```javascript\nviji.slider(default, { min?, max?, step?, label, group?, category? }) // .value: number\nviji.color(default, { label, group?, category? }) // .value: '#rrggbb', .rgb: {r,g,b} 0..255, .hsb: {h:0..360, s/b:0..100}\nviji.toggle(default, { label, group?, category? }) // .value: boolean\nviji.select(default, { options, label, group?, category? }) // .value: string | number\nviji.number(default, { min?, max?, step?, label, group?, category? }) // .value: number\nviji.text(default, { label, group?, category?, maxLength? }) // .value: string\nviji.image(null, { label, group?, category? }) // .value: ImageBitmap | null\nviji.button({ label, description?, group?, category? }) // .value: boolean (true for 1 frame)\nviji.coordinate(default, { step?, label, group?, category? }) // .value: { x, y }, range -1..1\n```\n\n### Audio: `viji.audio`\n\nCheck `isConnected` first.\n\n| Member | Type | Notes |\n|---|---|---|\n| `isConnected` | `boolean` | Audio source active |\n| `volume.current`, `volume.peak`, `volume.smoothed` | `number` 0..1 | `smoothed` = 200ms decay envelope of `current` |\n| `bands.low`, `lowMid`, `mid`, `highMid`, `high` | `number` 0..1 | Instant band energies (20-120, 120-400, 400-1600, 1600-6000, 6000-16000 Hz) |\n| `bands.lowSmoothed`, `lowMidSmoothed`, `midSmoothed`, `highMidSmoothed`, `highSmoothed` | `number` 0..1 | 150ms decay envelope siblings (not nested) |\n| `beat.kick`, `snare`, `hat`, `any` | `number` 0..1 | Beat energy curves with a 300ms decay; peak on each detected beat then fall off |\n| `beat.kickSmoothed`, `snareSmoothed`, `hatSmoothed`, `anySmoothed` | `number` 0..1 | Smoother 500ms decay envelopes; use for ambient pulses |\n| `beat.triggers.kick`, `snare`, `hat`, `any` | `boolean` | True for exactly one frame on the matching beat, then auto-resets. OR-accumulated between frames so no beat is lost |\n| `beat.events` | `Array<{type, time, strength}>` | All beats detected since the last frame; `type` is `'kick' \\| 'snare' \\| 'hat'` (never `'any'`); `time` in milliseconds; cleared each frame |\n| `beat.bpm`, `beat.confidence`, `beat.isLocked` | `number`, `number`, `boolean` | `bpm` is `0` when no audio is connected; once audio connects it tracks the detected tempo (clamped to 60..240) with `120` as a fallback before lock-on. `confidence` in 0..1. `isLocked` is `true` on stable tempo lock |\n| `spectral.brightness`, `spectral.flatness` | `number` 0..1 | Spectral centroid; spectral flatness |\n| `getFrequencyData()` | `Uint8Array` | Raw FFT bins 0..255 |\n| `getWaveform()` | `Float32Array` | Time-domain waveform -1..1 |\n\nExternal-device audio (`viji.devices[i].audio`) and host-supplied audio streams (`viji.audioStreams[i]`) use the `AudioStreamAPI` shape: `isConnected`, `volume.{current,peak,smoothed}`, `bands.{low...high}` plus each `*Smoothed` sibling, `spectral.{brightness,flatness}`, `getFrequencyData()`, `getWaveform()`. No `beat`, BPM, triggers, or events on streams: those exist only on the main `viji.audio`.\n\n### Video: `viji.video`\n\nCheck `isConnected && currentFrame` before drawing.\n\n| Member | Type | Notes |\n|---|---|---|\n| `isConnected` | `boolean` | Video source active |\n| `currentFrame` | `OffscreenCanvas \\| ImageBitmap \\| null` | Most recent frame |\n| `frameWidth`, `frameHeight` | `number` | Source frame size |\n| `frameRate` | `number` | Source frame rate |\n| `getFrameData()` | `ImageData \\| null` | Pixel data of `currentFrame` |\n| `cv` | `VideoCVAPI` | Computer-vision surface: see below |\n\nCV data outputs (`analysedFrame`, `getAnalysedFrameData()`, `faces`, `hands`, `pose`, `segmentation`) and CV verbs (`enableFaceDetection`, etc.) all live on `viji.video.cv`, not on `viji.video` directly.\n\n**Aspect ratio.** Camera frames almost never match the canvas aspect. Drawing to `(0, 0, viji.width, viji.height)` stretches video and misaligns CV bounds. Use this helper at module scope:\n\n```javascript\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nconst v = videoFit(viji); // 'cover' or 'contain'\nctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n// CV coords are normalized 0..1 to the source frame. Map through v to align with the fitted video:\nviji.video.cv.faces.forEach(face => {\n const bx = v.x + face.bounds.x * v.width;\n const by = v.y + face.bounds.y * v.height;\n const bw = face.bounds.width * v.width;\n const bh = face.bounds.height * v.height;\n ctx.strokeRect(bx, by, bw, bh);\n});\n```\n\nDefault to `'cover'` for live cameras. Use `'contain'` for CV-overlay scenes where bounding boxes near frame edges must stay visible. `viji.video.currentFrame` is the right default for displayed video; reach for `viji.video.cv.analysedFrame` only when sampling pixels at CV-derived positions (segmentation compositing, face-mesh texturing, landmark-anchored color sampling). The common fallback pattern is `viji.video.cv.analysedFrame ?? viji.video.currentFrame` during the brief window before the first CV inference lands.\n\n### Computer Vision: `viji.video.cv`\n\nEach CV feature is independent and populates only its own subset of fields. Enable only what you need (each active feature consumes a separate WebGL context for MediaPipe; 4+ on low-end hardware risks context-limit failures).\n\n```javascript\n// Verbs return Promise<void>. await is optional — without it, the model loads\n// asynchronously while render runs and data fields populate when ready.\n// Safe to call from module scope (always-on CV) or per-frame inside render()\n// gated by a viji.toggle(...) (opt-in CV). Idempotent + reference-counted.\nawait viji.video.cv.enableFaceDetection(true | false); // populates face.bounds, face.center, face.confidence\nawait viji.video.cv.enableFaceMesh(true | false); // populates face.landmarks (468 pts) + face.headPose\nawait viji.video.cv.enableEmotionDetection(true | false); // populates face.expressions + face.blendshapes; also loads landmarker (face.landmarks + face.headPose populated)\nawait viji.video.cv.enableHandTracking(true | false); // populates viji.video.cv.hands[]\nawait viji.video.cv.enablePoseDetection(true | false); // populates viji.video.cv.pose\nawait viji.video.cv.enableBodySegmentation(true | false); // populates viji.video.cv.segmentation\nviji.video.cv.getActiveFeatures(); // CVFeature[]\nviji.video.cv.isProcessing(); // boolean\n```\n\nData outputs on `viji.video.cv`:\n- `analysedFrame: OffscreenCanvas | null`: the exact frame paired with the current CV results. `null` until the first inference lands after a feature is enabled. Use the pattern `viji.video.cv.analysedFrame ?? viji.video.currentFrame` to fall back during the startup window.\n- `getAnalysedFrameData(): ImageData | null`: pixel data of `analysedFrame`, cached and re-extracted only on new CV results.\n\n**`viji.video.cv.faces: FaceData[]`**: `id` always present. Other fields are populated only by their source model and read `null` (or empty array for `landmarks`) otherwise — always null-check before reading.\n- `bounds: {x,y,width,height} | null` (normalized 0..1) — populated by face detection\n- `center: {x,y} | null` (normalized 0..1) — populated by face detection\n- `confidence: number | null` (0..1) — populated by face detection\n- `landmarks: {x,y,z?}[]` (empty `[]` unless face mesh enabled; 468 pts when populated)\n- `headPose: { pitch, yaw, roll } | null` (degrees) — populated by face mesh\n- `expressions: { neutral, happy, sad, angry, surprised, disgusted, fearful } | null` (0..1 each) — populated by emotion detection\n- `blendshapes | null` — populated by emotion detection; 52 ARKit coefficients (`browDownLeft`/`Right`, `browInnerUp`, `browOuterUpLeft`/`Right`, `cheekPuff`, `cheekSquintLeft`/`Right`, `eyeBlinkLeft`/`Right`, `eyeLookDown`/`In`/`Out`/`UpLeft`/`Right`, `eyeSquintLeft`/`Right`, `eyeWideLeft`/`Right`, `jawForward`/`Left`/`Open`/`Right`, `mouthClose`/`DimpleLeft`/`Right`/`FrownLeft`/`Right`/`Funnel`/`Left`/`LowerDownLeft`/`Right`/`PressLeft`/`Right`/`Pucker`/`Right`/`RollLower`/`Upper`/`ShrugLower`/`Upper`/`SmileLeft`/`Right`/`StretchLeft`/`Right`/`UpperUpLeft`/`Right`, `noseSneerLeft`/`Right`, `tongueOut`: all 0..1)\n\nIf you need bounds while only running face mesh, either also enable face detection or compute the bounding box from `face.landmarks` min/max yourself.\n\n**`viji.video.cv.hands: HandData[]`**: `id`, `handedness: 'left' | 'right'`, `confidence`, `bounds`, `landmarks: {x,y,z}[]` (21 points), `palm: {x,y,z}`, `gestures: { fist, openPalm, peace, thumbsUp, thumbsDown, pointing, iLoveYou }` (0..1 confidence each).\n\n**`viji.video.cv.pose: PoseData | null`**: `confidence`, `landmarks: {x,y,z,visibility}[]` (33 points), plus body-part arrays `face`, `torso`, `leftArm`, `rightArm`, `leftLeg`, `rightLeg`.\n\n**`viji.video.cv.segmentation: SegmentationData | null`**: `mask: Uint8Array` (each byte is `0` for background or `1` for person; length = `width * height`), `width`, `height`. Note: the shader `u_segmentationMask` sampler is sampled as a normalized `float` (0.0 = background, 1.0 = person); the JS `mask` byte values are 0/1, not 0/255.\n\n### Input\n\n**Pointer** (unified mouse and touch): `viji.pointer`\n| Member | Type | Notes |\n|---|---|---|\n| `x`, `y` | `number` | Position in pixels |\n| `deltaX`, `deltaY` | `number` | Per-frame movement |\n| `isDown` | `boolean` | Pressed or touching |\n| `wasPressed`, `wasReleased` | `boolean` | One-frame edge triggers |\n| `isInCanvas` | `boolean` | Inside canvas bounds |\n| `type` | `'mouse' \\| 'touch' \\| 'none'` | Active input type |\n\n**Mouse**: `viji.mouse`: `x`, `y`, `isInCanvas`, `isPressed`, `leftButton`, `rightButton`, `middleButton`, `deltaX`, `deltaY`, `wheelDelta`, `wheelX`, `wheelY`, `wasPressed`, `wasReleased`, `wasMoved`.\n\n**Keyboard**: `viji.keyboard`: `isPressed(key)`, `wasPressed(key)`, `wasReleased(key)`, `activeKeys: Set<string>`, `pressedThisFrame`, `releasedThisFrame`, `lastKeyPressed`, `lastKeyReleased`, `shift`, `ctrl`, `alt`, `meta`.\n\n**Touch**: `viji.touches`: `count`, `points: TouchPoint[]`, `started`, `moved`, `ended` (each a `TouchPoint[]`), `primary: TouchPoint | null`. Each `TouchPoint`: `id`, `x`, `y`, `pressure`, `radius`, `radiusX`, `radiusY`, `rotationAngle`, `force`, `isInCanvas`, `deltaX`, `deltaY`, `velocity: {x,y}`, `isNew`, `isActive`, `isEnding`.\n\n### Sensors and external devices\n\n**`viji.device`**: host device motion and orientation:\n- `motion: DeviceMotionData | null`: `acceleration: {x,y,z}` (m/s²), `accelerationIncludingGravity`, `rotationRate: {alpha,beta,gamma}` (deg/s), `interval` (ms).\n- `orientation: DeviceOrientationData | null`: `alpha` (0-360° compass), `beta` (-180..180° front-back tilt), `gamma` (-90..90° left-right tilt), `absolute`.\n\n**`viji.devices`**: `DeviceState[]`, externally connected devices. Each: `id`, `name`, `motion`, `orientation`, `video` (`VideoAPI | null`, same shape as `viji.video` but without CV), `audio` (`AudioStreamAPI | null`, lightweight; no beat or BPM).\n\n### Host-supplied streams\n\n- `viji.videoStreams: VideoAPI[]`: extra video sources from the host compositor. Same shape as `viji.video`. May be empty.\n- `viji.audioStreams: AudioStreamAPI[]`: extra audio sources from the host. Lightweight shape (no beat or BPM). May be empty.\n\n### External libraries\n\n```javascript\nconst THREE = await import('https://esm.sh/three@0.160.0');\nconst renderer = new THREE.WebGLRenderer({ canvas: viji.canvas, antialias: true });\nrenderer.setSize(viji.width, viji.height, false); // always pass false to skip CSS styling\n```\n\nPin library versions in the import URL. Always pass `viji.canvas` to the renderer. Always pass `false` as the third argument to Three.js `setSize`. Handle resize by comparing `viji.width` / `viji.height` against previous values inside `render()`.\n\n## Template\n\n```javascript\nconst bgColor = viji.color('#1a1a2e', { label: 'Background' });\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nconst count = viji.slider(12, { min: 3, max: 30, step: 1, label: 'Count' });\n\nlet angle = 0;\n\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n angle += speed.value * viji.deltaTime;\n\n ctx.fillStyle = bgColor.value;\n ctx.fillRect(0, 0, viji.width, viji.height);\n\n const cx = viji.width / 2;\n const cy = viji.height / 2;\n const radius = Math.min(viji.width, viji.height) * 0.3;\n const dotSize = Math.min(viji.width, viji.height) * 0.02;\n const n = Math.floor(count.value);\n\n for (let i = 0; i < n; i++) {\n const a = angle + (i / n) * Math.PI * 2;\n const x = cx + Math.cos(a) * radius;\n const y = cy + Math.sin(a) * radius;\n const hue = (i / n) * 360;\n ctx.beginPath();\n ctx.arc(x, y, dotSize, 0, Math.PI * 2);\n ctx.fillStyle = `hsl(${hue}, 80%, 60%)`;\n ctx.fill();\n }\n}\n```\n",
|
|
9843
|
+
"p5": "# Viji API: P5 Renderer Reference\n\nP5 scenes use P5.js 1.9.4 inside a Web Worker on an `OffscreenCanvas`. P5 runs in **instance mode**: every P5 function and constant requires the `p5.` prefix. The first line of the scene must be `// @renderer p5` (2D) or `// @renderer p5 webgl` (3D / WEBGL).\n\n## Architecture\n\n- The global `viji` object exposes canvas, timing, audio, video, CV, input, sensors, parameters: identical to Native.\n- The canvas and its rendering context are managed by P5. **Do not call `createCanvas()`.** Viji creates the canvas in the mode declared by the `// @renderer` directive.\n- `viji.useContext()` is **not available** in P5 mode (the context belongs to P5).\n- Top-level code runs once. `function setup(viji, p5)` runs once for configuration. `function render(viji, p5)` runs every frame.\n- The `p5` instance passed to `setup` and `render` is the P5 instance. Use `p5.background`, `p5.fill`, `p5.circle`, `p5.PI`, etc. Unprefixed names will not resolve.\n\n## Rules\n\n1. **Always** add `// @renderer p5` (2D) or `// @renderer p5 webgl` (WEBGL) as the first line.\n2. **Always** use `render(viji, p5)` (not `draw()`) and `setup(viji, p5)` (not `setup()`).\n3. **Always** prefix every P5 function and constant with `p5.`:\n - `background(0)` -> `p5.background(0)`\n - `fill(255)` -> `p5.fill(255)`\n - `PI` -> `p5.PI`, `TWO_PI` -> `p5.TWO_PI`, `HSB` -> `p5.HSB`\n - `createVector(1, 0)` -> `p5.createVector(1, 0)`\n - `map(v, 0, 1, 0, 255)` -> `p5.map(v, 0, 1, 0, 255)`\n - `noise(x)` -> `p5.noise(x)`, `random()` -> `p5.random()`\n4. **Never** call `createCanvas()`. The canvas is created and managed by Viji.\n5. **Never** use `preload()`. Use `viji.image(null, { label })` for images, or `fetch()` inside `setup()`.\n6. **Never** use P5 event callbacks (`mousePressed`, `mouseDragged`, `mouseReleased`, `keyPressed`, `keyReleased`, `keyTyped`, `touchStarted`, `touchMoved`, `touchEnded`). Read state in `render()` via `viji.pointer`, `viji.mouse`, `viji.keyboard`, `viji.touches`. Use `wasPressed` / `wasReleased` for one-frame edges.\n7. **Never** use `loadImage()`, `loadFont()`, `loadJSON()`, `loadModel()`, `loadShader()`. Use `viji.image()` or `fetch()`.\n8. **Never** use `p5.frameRate()`, `p5.save()`, `p5.saveCanvas()`, `p5.saveFrames()`. The host controls frame rate and capture.\n9. **Never** use `createCapture()` or `createVideo()`. Use `viji.video.*`.\n10. **Never** use `p5.dom` or `p5.sound`. Use Viji parameters and `viji.audio.*`.\n11. **Never** access `window`, `document`, `Image()`, `localStorage`. `fetch()` is available.\n12. **Always** declare parameters at the top level, never inside `render()` or `setup()`.\n13. **Always** read parameters via `.value`. Color parameters also expose `.rgb` (matches `colorMode(RGB, 255)`) and `.hsb` (matches `colorMode(HSB, 360, 100, 100)`). Color defaults accept hex, `{ r, g, b }` (0..255), `{ h, s, b }` (0..360 / 0..100), or CSS color strings.\n14. **Always** use `viji.width` and `viji.height` for canvas dimensions. Never hardcode pixel sizes.\n15. **Always** use `viji.deltaTime` for animation:\n ```javascript\n let angle = 0;\n function render(viji, p5) { angle += speed.value * viji.deltaTime; }\n ```\n16. **Never** multiply `viji.time` by a parameter; it causes visible jumps when the parameter changes. Use a `deltaTime` accumulator. This also applies to nested multiplications; each independent speed gets its own accumulator.\n17. **Never** allocate objects, arrays, or strings inside `render()`. Pre-allocate at the top level and reuse.\n18. **Image parameters with P5:** use `.p5` (a `P5Image`) instead of `.value` when passing to `p5.image()`:\n ```javascript\n const photo = viji.image(null, { label: 'Photo' });\n function render(viji, p5) {\n if (photo.value) p5.image(photo.p5, 0, 0, viji.width, viji.height);\n }\n ```\n19. **Video drawing.** In 2D, use `p5.image(viji.video.currentFrame, ...)` or `p5.drawingContext.drawImage(...)`. In WEBGL, use `p5.image(viji.video.currentFrame, ...)` only (`p5.drawingContext` is WebGL, not Canvas 2D). **Always preserve aspect ratio** via the `videoFit` helper below.\n20. `p5.createGraphics(w, h)` is **2D only** (`createGraphics(w, h, p5.WEBGL)` is not supported). It creates an OffscreenCanvas internally.\n21. Fonts: `p5.textFont()` only with CSS generic names (`monospace`, `serif`, `sans-serif`). `loadFont()` is not available.\n22. `p5.tint()` and `p5.blendMode()` work normally.\n23. **Canvas mode is declared by the `// @renderer` directive.** Use `// @renderer p5` for 2D, `// @renderer p5 webgl` for WEBGL. Never call `createCanvas()` or `createCanvas(..., p5.WEBGL)`.\n24. In WEBGL scenes, `p5.drawingContext` is a WebGL context. Use P5 3D drawing primitives or `p5.image()` / textures for images and video.\n25. `p5.pixelDensity()` defaults to 1 in the worker. `p5.loadPixels()` and `p5.pixels[]` work in 2D scenes; WEBGL pixel readback follows standard P5.js rules.\n26. **Always** check `viji.audio.isConnected` before reading audio data.\n27. **Always** check `viji.video.isConnected && viji.video.currentFrame` before drawing video.\n28. **Never** enable CV features by default; use toggle parameters.\n29. **Always** set `category` on input-dependent parameters (`'audio'`, `'video'`, `'interaction'`). Use creative-strength sliders, not on/off toggles, for inputs the host already gates. CV feature toggles stay opt-in.\n\n## Video aspect helper\n\n```javascript\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nif (viji.video.isConnected && viji.video.currentFrame) {\n const v = videoFit(viji, 'cover');\n p5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n}\n\n// CV bounds and landmarks are normalized 0..1 to the source frame.\n// Map them through v to align with the drawn video.\n```\n\n## P5 to Viji mapping\n\n| Standard P5.js | Viji P5 equivalent |\n|---|---|\n| `width` / `height` | `viji.width` / `viji.height` |\n| `mouseX` / `mouseY` | `viji.pointer.x` / `viji.pointer.y` (or `viji.mouse.x` / `viji.mouse.y`) |\n| `mouseIsPressed` | `viji.pointer.isDown` |\n| `mouseButton === LEFT` | `viji.mouse.leftButton` |\n| `keyIsPressed` | `viji.keyboard.activeKeys.size > 0` |\n| `keyIsDown(code)` | `viji.keyboard.isPressed('keyName')` |\n| `key` | `viji.keyboard.lastKeyPressed` |\n| `frameCount` | `viji.frameCount` (or a `viji.deltaTime` accumulator) |\n| `frameRate(n)` | Remove. Host controls frame rate. |\n| `createCanvas(w, h)` | Remove. Canvas is provided. |\n| `preload()` | Remove. Use `viji.image()` or `fetch()` in `setup()`. |\n| `loadImage(url)` | `viji.image(null, { label: 'Image' })`; pass `.p5` to `p5.image()`. |\n| `save()` / `saveCanvas()` | Remove. Host handles capture. |\n| `mousePressed()` / `mouseReleased()` callbacks | Check `viji.pointer.wasPressed` / `wasReleased` inside `render()`. |\n| `keyPressed()` / `keyReleased()` callbacks | Check `viji.keyboard.wasPressed(key)` / `wasReleased(key)` inside `render()`. |\n\n## API reference\n\nAll `viji.*` members are identical to Native (same object, same types).\n\n### Canvas and timing\n\n| Member | Type | Notes |\n|---|---|---|\n| `viji.canvas` | `OffscreenCanvas` | Managed by P5 |\n| `viji.width`, `viji.height` | `number` | Canvas dimensions |\n| `viji.time` | `number` | Seconds since scene start |\n| `viji.deltaTime` | `number` | Seconds since last frame |\n| `viji.frameCount` | `number` | Total frames rendered |\n| `viji.fps` | `number` | Target frame rate |\n\n### Parameters\n\n```javascript\nviji.slider(default, { min?, max?, step?, label, group?, category? }) // .value: number\nviji.color(default, { label, group?, category? }) // .value, .rgb, .hsb\nviji.toggle(default, { label, group?, category? }) // .value: boolean\nviji.select(default, { options, label, group?, category? }) // .value: string | number\nviji.number(default, { min?, max?, step?, label, group?, category? }) // .value: number\nviji.text(default, { label, group?, category?, maxLength? }) // .value: string\nviji.image(null, { label, group?, category? }) // .value: ImageBitmap | null, .p5: P5Image\nviji.button({ label, description?, group?, category? }) // .value: boolean (1 frame)\nviji.coordinate(default, { step?, label, group?, category? }) // .value: { x, y }, range -1..1\n```\n\n### Audio: `viji.audio`\n\nCheck `isConnected` first. Members:\n\n`isConnected` (boolean); `volume.{current, peak, smoothed}` (0..1; `smoothed` 200ms decay); `bands.{low, lowMid, mid, highMid, high}` (0..1; ranges 20-120, 120-400, 400-1600, 1600-6000, 6000-16000 Hz) and the sibling `*Smoothed` envelopes (150ms decay); `beat.{kick, snare, hat, any}` (0..1, 300ms decay curves; peak on each detected beat); `beat.{kickSmoothed, snareSmoothed, hatSmoothed, anySmoothed}` (500ms decay envelopes); `beat.triggers.{kick, snare, hat, any}` (boolean, true for exactly one frame, OR-accumulated between frames); `beat.events: Array<{type, time, strength}>` (`type` is `'kick' | 'snare' | 'hat'`; `time` ms; cleared each frame); `beat.bpm` (`0` when no audio is connected; once audio connects it tracks the detected tempo clamped to 60..240, with `120` as a fallback before lock-on), `beat.confidence` (0..1), `beat.isLocked` (boolean); `spectral.{brightness, flatness}` (0..1); `getFrequencyData(): Uint8Array` (1024 FFT bins, 0..255); `getWaveform(): Float32Array` (2048 samples, -1..1).\n\nExternal-device audio (`viji.devices[i].audio`) and host streams (`viji.audioStreams[i]`) follow the `AudioStreamAPI` shape: `isConnected`, `volume`, `bands` (instant + smoothed siblings), `spectral`, `getFrequencyData`, `getWaveform`. No beat, BPM, triggers, or events on streams.\n\n### Video: `viji.video`\n\nCheck `isConnected && currentFrame` before drawing. Members:\n\n`isConnected` (boolean); `currentFrame` (`OffscreenCanvas | ImageBitmap | null`); `frameWidth`, `frameHeight`, `frameRate`; `getFrameData(): ImageData | null`; `cv: VideoCVAPI` (CV outputs and verbs live here, not on `viji.video` directly).\n\n### Computer Vision: `viji.video.cv`\n\nEach CV feature is independent and populates only its own subset of fields. Enable only what you need (each active feature uses a separate WebGL context for MediaPipe; 4+ on low-end hardware risks context-limit failures).\n\n```javascript\n// Verbs return Promise<void>. await is optional. Safe to call from module\n// scope (always-on CV) or per-frame inside render() gated by a viji.toggle(...)\n// (opt-in CV). Idempotent + reference-counted.\nawait viji.video.cv.enableFaceDetection(true | false); // face.bounds, face.center, face.confidence\nawait viji.video.cv.enableFaceMesh(true | false); // face.landmarks (468 pts) + face.headPose\nawait viji.video.cv.enableEmotionDetection(true | false); // face.expressions + face.blendshapes; also loads landmarker\nawait viji.video.cv.enableHandTracking(true | false); // viji.video.cv.hands[]\nawait viji.video.cv.enablePoseDetection(true | false); // viji.video.cv.pose\nawait viji.video.cv.enableBodySegmentation(true | false); // viji.video.cv.segmentation\nviji.video.cv.getActiveFeatures(); // CVFeature[]\nviji.video.cv.isProcessing(); // boolean\n```\n\nData outputs on `viji.video.cv`:\n- `analysedFrame: OffscreenCanvas | null`: the frame paired with the current CV results. `null` until the first inference lands. Use `viji.video.cv.analysedFrame ?? viji.video.currentFrame` to fall back during startup.\n- `getAnalysedFrameData(): ImageData | null`: pixel data of `analysedFrame`.\n\n**`faces: FaceData[]`**: `id` always present. Other fields populated only by their source model; otherwise `null` (`landmarks` is `[]` when empty). Always null-check.\n- `bounds: {x,y,width,height} | null` — populated by face detection\n- `center: {x,y} | null` — populated by face detection\n- `confidence: number | null` — populated by face detection\n- `landmarks: {x,y,z?}[]` (empty `[]` unless face mesh enabled)\n- `headPose: {pitch, yaw, roll} | null` — populated by face mesh\n- `expressions: {neutral, happy, sad, angry, surprised, disgusted, fearful} | null` (0..1 each) — populated by emotion detection\n- `blendshapes | null` (52 ARKit coefficients 0..1) — populated by emotion detection\n\nIf you need `bounds` while only running face mesh, either also enable face detection or compute it from `face.landmarks` min/max.\n\n**`hands: HandData[]`**: `id`, `handedness: 'left' | 'right'`, `confidence`, `bounds`, `landmarks` (21 points), `palm`, `gestures` (`fist`, `openPalm`, `peace`, `thumbsUp`, `thumbsDown`, `pointing`, `iLoveYou` 0..1).\n\n**`pose: PoseData | null`**: `confidence`, `landmarks` (33 points), body-part arrays `face`, `torso`, `leftArm`, `rightArm`, `leftLeg`, `rightLeg`.\n\n**`segmentation: SegmentationData | null`**: `mask: Uint8Array` (each byte is `0` for background or `1` for person; length = `width * height`), `width`, `height`.\n\n### Input\n\n- `viji.pointer`: `x`, `y`, `deltaX`, `deltaY`, `isDown`, `wasPressed`, `wasReleased`, `isInCanvas`, `type` (`'mouse' | 'touch' | 'none'`).\n- `viji.mouse`: `x`, `y`, `isInCanvas`, `isPressed`, `leftButton`, `rightButton`, `middleButton`, `deltaX`, `deltaY`, `wheelDelta`, `wheelX`, `wheelY`, `wasPressed`, `wasReleased`, `wasMoved`.\n- `viji.keyboard`: `isPressed(key)`, `wasPressed(key)`, `wasReleased(key)`, `activeKeys`, `pressedThisFrame`, `releasedThisFrame`, `lastKeyPressed`, `lastKeyReleased`, `shift`, `ctrl`, `alt`, `meta`.\n- `viji.touches`: `count`, `points`, `started`, `moved`, `ended`, `primary`. Each `TouchPoint`: `id`, `x`, `y`, `pressure`, `radius`, `radiusX`, `radiusY`, `rotationAngle`, `force`, `isInCanvas`, `deltaX`, `deltaY`, `velocity`, `isNew`, `isActive`, `isEnding`.\n\n### Sensors and external devices\n\n`viji.device.motion`: `acceleration {x,y,z}` (m/s²), `accelerationIncludingGravity`, `rotationRate {alpha,beta,gamma}` (deg/s), `interval` (ms).\n`viji.device.orientation`: `alpha` (0-360°), `beta` (-180..180°), `gamma` (-90..90°), `absolute`.\n\n`viji.devices: DeviceState[]`: each entry: `id`, `name`, `motion`, `orientation`, `video` (`VideoAPI | null`, no CV), `audio` (`AudioStreamAPI | null`, lightweight).\n\n### Host streams\n\n`viji.videoStreams: VideoAPI[]`: extra video sources. May be empty.\n`viji.audioStreams: AudioStreamAPI[]`: extra audio sources. May be empty. Lightweight shape (no beat / BPM).\n\n## Template\n\n```javascript\n// @renderer p5\n\nconst bgColor = viji.color('#1a1a2e', { label: 'Background' });\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nconst count = viji.slider(8, { min: 3, max: 30, step: 1, label: 'Count' });\n\nlet angle = 0;\n\nfunction setup(viji, p5) {\n p5.colorMode(p5.HSB, 360, 100, 100);\n}\n\nfunction render(viji, p5) {\n angle += speed.value * viji.deltaTime;\n p5.background(bgColor.value);\n\n const cx = viji.width / 2;\n const cy = viji.height / 2;\n const radius = p5.min(viji.width, viji.height) * 0.3;\n const dotSize = p5.min(viji.width, viji.height) * 0.04;\n const n = p5.floor(count.value);\n\n p5.noStroke();\n for (let i = 0; i < n; i++) {\n const a = angle + (i / n) * p5.TWO_PI;\n const x = cx + p5.cos(a) * radius;\n const y = cy + p5.sin(a) * radius;\n p5.fill((i / n) * 360, 80, 90);\n p5.circle(x, y, dotSize);\n }\n}\n```\n",
|
|
9809
9844
|
"shader": "# Viji API: Shader Renderer Reference\n\nShader scenes are GLSL fragment shaders running on a fullscreen quad. Write helper functions and `void main()`. Precision and all uniforms (built-in plus those generated from `@viji-*` directives) are auto-injected by Viji. The first line must be `// @renderer shader` (or `#version 300 es` followed by `// @renderer shader` for GLSL ES 3.00).\n\n## Architecture\n\n- The shader is **GLSL ES 1.00** by default. Opt into ES 3.00 by making `#version 300 es` the literal first line.\n- Viji auto-injects `precision mediump float;` and every uniform declaration. Never declare them yourself.\n- ES 3.00 requires `out vec4 fragColor;` before `main` and `fragColor = ...` instead of `gl_FragColor`. ES 3.00 uses `texture()` instead of `texture2D()`.\n- If the shader references `fwidth`, Viji auto-injects `#extension GL_OES_standard_derivatives : enable`.\n- The `backbuffer` sampler is auto-enabled when referenced. It carries the previous frame.\n\n## Rules\n\n1. **Always** add `// @renderer shader` as the first line (or as line 2 after `#version 300 es`).\n2. **Never** declare `precision mediump float;` or `precision highp float;`. Viji auto-injects precision.\n3. **Never** redeclare built-in uniforms (`u_time`, `u_resolution`, `u_mouse`, audio / video / CV uniforms, etc.). They are auto-injected.\n4. **Never** redeclare parameter uniforms. They are auto-generated from `@viji-*` directives.\n5. **Never** use the `u_` prefix for your own parameter names. It is reserved for Viji's built-ins. Use descriptive names (`speed`, `colorMix`, `intensity`).\n6. `@viji-*` parameter directives work with `//` comments only. Block comments (`/* */`) are not parsed.\n7. **Always** use `@viji-accumulator` instead of `u_time * speed` for parameter-driven animation. Multiplying `u_time` by a parameter causes visible jumps when sliders change.\n\n ```glsl\n // @viji-slider:speed label:\"Speed\" default:1.0 min:0.1 max:5.0\n // @viji-accumulator:phase rate:speed\n float wave = sin(phase);\n ```\n\n This also applies to **nested** multiplications. Each independent speed needs its own accumulator:\n\n ```glsl\n // @viji-accumulator:phase rate:speed\n // @viji-accumulator:rotPhase rate:rotSpeed\n ```\n8. For `backbuffer` (previous frame): reference it directly in code. Viji auto-detects and enables it. Sample with `texture2D(backbuffer, uv)` (ES 1.00) or `texture(backbuffer, uv)` (ES 3.00).\n9. Remove any `#ifdef GL_ES` / `precision` blocks; Viji handles them.\n10. **Always** set `category` on input-dependent directives: `category:audio`, `category:video`, `category:interaction`. Use creative-strength sliders, not on/off toggles, for inputs the host already gates.\n11. **CV features must be activated via `// @viji-cv:<featureToken>` directives.** Without the matching `@viji-cv:` activation, CV uniforms (`u_faceCount`, `u_face0*`, `u_handCount`, `u_poseDetected`, `u_segmentationMask`, etc.) read zero. Use the bare form (`// @viji-cv:faceDetection`) for always-on detection, or the toggleable form (`// @viji-cv:faceDetection label:\"Face Detection\" default:false`) which synthesizes both a host-side toggle parameter and a `bool <featureToken>` shader uniform mirroring its state. The six tokens: `faceDetection`, `faceMesh`, `emotionDetection`, `handTracking`, `poseDetection`, `bodySegmentation`.\n\n## Parameter directives\n\n```glsl\n// @viji-slider:name label:\"Label\" default:1.0 min:0.0 max:5.0 step:0.1\n// -> uniform float name;\n\n// @viji-color:name label:\"Color\" default:#ff6600\n// -> uniform vec3 name; (RGB 0..1)\n// default: accepts #rrggbb, #rgb, vec3(r,g,b) (0..1), rgb(r,g,b) (0..255),\n// hsl(h, s%, l%) (h: 0..360), hsb(h, s, b) (h: 0..360, s/b: 0..100)\n\n// @viji-toggle:name label:\"Toggle\" default:false\n// -> uniform bool name;\n\n// @viji-select:name label:\"Mode\" default:0 options:[\"Solid\",\"Gradient\",\"Noise\"]\n// -> uniform int name; (0-based)\n\n// @viji-number:name label:\"Count\" default:10.0 min:1.0 max:100.0 step:1.0\n// -> uniform float name;\n\n// @viji-image:name label:\"Texture\"\n// -> uniform sampler2D name;\n\n// @viji-button:name label:\"Reset\"\n// -> uniform bool name; (true for one frame on press)\n\n// @viji-coordinate:name label:\"Origin\" default:[0.0,0.0]\n// -> uniform vec2 name; (each component -1..1)\n\n// @viji-accumulator:name rate:speed\n// -> uniform float name; (CPU-side: += rate * deltaTime each frame)\n```\n\nAll directives support `group:\"GroupName\"` and `category:\"audio|video|interaction|general\"`.\n\n## Uniform reference\n\nAll uniforms below are always available. Do not declare them.\n\n### Core\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_resolution` | `vec2` | Canvas width and height in pixels |\n| `u_time` | `float` | Seconds since scene start |\n| `u_deltaTime` | `float` | Seconds since last frame |\n| `u_frame` | `int` | Current frame number |\n| `u_fps` | `float` | Target frame rate (host-controlled) |\n\n### Mouse\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_mouse` | `vec2` | Mouse position in pixels (WebGL coords, bottom-left origin) |\n| `u_mouseInCanvas` | `bool` | Mouse inside canvas |\n| `u_mousePressed` | `bool` | Any button pressed |\n| `u_mouseLeft`, `u_mouseRight`, `u_mouseMiddle` | `bool` | Specific buttons |\n| `u_mouseDelta` | `vec2` | Per-frame movement |\n| `u_mouseWheel` | `float` | Scroll wheel delta |\n| `u_mouseWasPressed`, `u_mouseWasReleased` | `bool` | One-frame edges |\n\n### Pointer (unified mouse / touch)\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_pointer` | `vec2` | Primary input position in pixels (WebGL coords) |\n| `u_pointerDelta` | `vec2` | Primary input movement delta |\n| `u_pointerDown` | `bool` | Primary input active |\n| `u_pointerWasPressed`, `u_pointerWasReleased` | `bool` | One-frame edges |\n| `u_pointerInCanvas` | `bool` | Inside canvas |\n\n### Keyboard\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_keySpace`, `u_keyShift`, `u_keyCtrl`, `u_keyAlt` | `bool` | Modifier and space |\n| `u_keyW`, `u_keyA`, `u_keyS`, `u_keyD` | `bool` | WASD |\n| `u_keyUp`, `u_keyDown`, `u_keyLeft`, `u_keyRight` | `bool` | Arrows |\n| `u_keyboard` | `sampler2D` | Full keyboard state (256x3 LUMINANCE). Row 0 = held, row 1 = pressed-this-frame, row 2 = toggle. Access: `texelFetch(u_keyboard, ivec2(keyCode, row), 0).r` |\n\n### Touch\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_touchCount` | `int` | Active touches (0-5) |\n| `u_touch0`-`u_touch4` | `vec2` | Touch point positions in pixels |\n\n### Audio (scalars)\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_audioVolume`, `u_audioPeak` | `float` | RMS / peak 0..1 |\n| `u_audioVolumeSmoothed` | `float` | 200ms decay envelope of `u_audioVolume` |\n| `u_audioLow`, `u_audioLowMid`, `u_audioMid`, `u_audioHighMid`, `u_audioHigh` | `float` | Band energies 0..1 (20-120, 120-400, 400-1600, 1600-6000, 6000-16000 Hz) |\n| `u_audioLowSmoothed` - `u_audioHighSmoothed` | `float` | 150ms decay envelopes of the bands |\n| `u_audioKick`, `u_audioSnare`, `u_audioHat`, `u_audioAny` | `float` | Beat energy curves with 300ms decay; peak on each detected beat then fall off |\n| `u_audioKickSmoothed` - `u_audioAnySmoothed` | `float` | Smoother 500ms decay envelopes |\n| `u_audioKickTrigger`, `u_audioSnareTrigger`, `u_audioHatTrigger`, `u_audioAnyTrigger` | `bool` | True for exactly one frame on the matching beat, then auto-resets; OR-accumulated between frames |\n| `u_audioBPM`, `u_audioConfidence`, `u_audioIsLocked` | `float`, `float`, `bool` | `u_audioBPM` is `0.0` when no audio is connected; once audio connects it tracks the detected tempo (clamped to 60..240) with `120.0` as a fallback before lock-on. `u_audioConfidence` 0..1. `u_audioIsLocked` true on stable tempo lock |\n| `u_audioBrightness`, `u_audioFlatness` | `float` | Spectral features 0..1 |\n\n### Audio (textures, main source only)\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_audioFFT` | `sampler2D` | FFT spectrum (1024 bins, 0..255) |\n| `u_audioWaveform` | `sampler2D` | Time-domain waveform (-1..1) |\n\nAdditional audio streams and device audio expose scalar uniforms only (no FFT / waveform textures): see \"Streams\" below.\n\n### Video\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_video` | `sampler2D` | Most recent video frame |\n| `u_videoAnalysed` | `sampler2D` | Frame paired with current CV uniforms |\n| `u_videoAnalysedAvailable` | `bool` | True after first CV result lands |\n| `u_videoResolution` | `vec2` | Source frame size in pixels |\n| `u_videoFrameRate` | `float` | Source frame rate |\n| `u_videoConnected` | `bool` | Video source active |\n\n**Aspect handling.** Camera frames almost never match the canvas aspect. Sampling `texture2D(u_video, uv)` with canvas UV stretches the video and misaligns CV uniforms. Use these helpers at the top of every video / CV shader:\n\n```glsl\n// mode: 1 = cover (fills canvas, video edges cropped)\n// 0 = contain (fits video, canvas letterboxed)\nvec2 vijiVideoUV(vec2 uv, int mode) {\n vec2 canvas = u_resolution;\n vec2 video = u_videoResolution;\n if (video.x == 0.0 || video.y == 0.0) return uv;\n float canvasAspect = canvas.x / canvas.y;\n float videoAspect = video.x / video.y;\n vec2 scale = vec2(1.0);\n if (mode == 1) {\n scale = canvasAspect > videoAspect\n ? vec2(1.0, canvasAspect / videoAspect)\n : vec2(videoAspect / canvasAspect, 1.0);\n } else {\n scale = canvasAspect > videoAspect\n ? vec2(videoAspect / canvasAspect, 1.0)\n : vec2(1.0, canvasAspect / videoAspect);\n }\n return (uv - 0.5) / scale + 0.5;\n}\n\nbool vijiInVideo(vec2 uv) {\n return all(greaterThanEqual(uv, vec2(0.0))) && all(lessThanEqual(uv, vec2(1.0)));\n}\n```\n\nDefault to mode `1` (cover) for live cameras. Use `0` (contain) for CV-overlay shaders where features near edges must stay visible. CV uniforms (`u_face0Center`, etc.) share the same bottom-up convention as `videoUV`; compare them directly without flipping.\n\n**`u_video` vs `u_videoAnalysed`.** Default to `u_video` for displayed video. Reach for `u_videoAnalysed` only when sampling pixels at CV-derived positions (segmentation compositing onto the body, sampling skin under face landmarks, face-mesh texturing). For shaders that consume CV uniforms without sampling at CV positions, stay on `u_video` (`u_videoAnalysed` advances only on inference completion and will stutter the displayed video). When `u_videoAnalysed` is the right choice, gate on `u_videoAnalysedAvailable` for the brief startup window:\n\n```glsl\nvec4 source = u_videoAnalysedAvailable\n ? texture2D(u_videoAnalysed, uv)\n : texture2D(u_video, uv);\n```\n\n### CV: face detection\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_faceCount` | `int` | Detected faces (0-1) |\n| `u_face0Bounds` | `vec4` | `(x, y, width, height)` normalized 0..1 |\n| `u_face0Center` | `vec2` | Center normalized 0..1 |\n| `u_face0HeadPose` | `vec3` | `(pitch, yaw, roll)` in degrees |\n| `u_face0Confidence` | `float` | 0..1 |\n| `u_face0Neutral` - `u_face0Fearful` | `float` | 7 expression scores 0..1 |\n\n**52 blendshape uniforms** (all `float`, 0..1, ARKit names prefixed with `u_face0`):\n`BrowDownLeft`, `BrowDownRight`, `BrowInnerUp`, `BrowOuterUpLeft`, `BrowOuterUpRight`, `CheekPuff`, `CheekSquintLeft`, `CheekSquintRight`, `EyeBlinkLeft`, `EyeBlinkRight`, `EyeLookDownLeft`, `EyeLookDownRight`, `EyeLookInLeft`, `EyeLookInRight`, `EyeLookOutLeft`, `EyeLookOutRight`, `EyeLookUpLeft`, `EyeLookUpRight`, `EyeSquintLeft`, `EyeSquintRight`, `EyeWideLeft`, `EyeWideRight`, `JawForward`, `JawLeft`, `JawOpen`, `JawRight`, `MouthClose`, `MouthDimpleLeft`, `MouthDimpleRight`, `MouthFrownLeft`, `MouthFrownRight`, `MouthFunnel`, `MouthLeft`, `MouthLowerDownLeft`, `MouthLowerDownRight`, `MouthPressLeft`, `MouthPressRight`, `MouthPucker`, `MouthRight`, `MouthRollLower`, `MouthRollUpper`, `MouthShrugLower`, `MouthShrugUpper`, `MouthSmileLeft`, `MouthSmileRight`, `MouthStretchLeft`, `MouthStretchRight`, `MouthUpperUpLeft`, `MouthUpperUpRight`, `NoseSneerLeft`, `NoseSneerRight`, `TongueOut`.\n\n### CV: hands\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_handCount` | `int` | Detected hands (0-2) |\n| `u_leftHandPalm`, `u_rightHandPalm` | `vec3` | Palm position `(x, y, z)` |\n| `u_leftHandConfidence`, `u_rightHandConfidence` | `float` | 0..1 |\n| `u_leftHandBounds`, `u_rightHandBounds` | `vec4` | Bounds normalized 0..1 |\n| `u_leftHandFist` - `u_leftHandILoveYou` | `float` | 7 left-hand gesture scores |\n| `u_rightHandFist` - `u_rightHandILoveYou` | `float` | 7 right-hand gesture scores |\n\n### CV: pose\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_poseDetected` | `bool` | Pose detected |\n| `u_poseConfidence` | `float` | 0..1 |\n| `u_nosePosition` | `vec2` | Nose landmark, normalized 0..1 |\n| `u_leftShoulderPosition` … `u_rightAnklePosition` | `vec2` | Joint landmarks (shoulders, elbows, wrists, hips, knees, ankles) |\n\n### CV: body segmentation\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_segmentationMask` | `sampler2D` | 0 = background, 1 = person |\n| `u_segmentationRes` | `vec2` | Mask resolution in pixels |\n\n### Device sensors\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_deviceAcceleration` | `vec3` | Acceleration without gravity (m/s²) |\n| `u_deviceAccelerationGravity` | `vec3` | Acceleration including gravity (m/s²) |\n| `u_deviceRotationRate` | `vec3` | Rotation rate (deg/s) |\n| `u_deviceOrientation` | `vec3` | `(alpha, beta, gamma)` in degrees |\n| `u_deviceOrientationAbsolute` | `bool` | Using magnetometer |\n\n### External devices\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_deviceCount` | `int` | Device video sources (0-8) |\n| `u_externalDeviceCount` | `int` | External devices (0-8) |\n| `u_device0` - `u_device7` | `sampler2D` | Device camera textures |\n| `u_device0Resolution` - `u_device7Resolution` | `vec2` | Per-device resolution |\n| `u_device0Connected` - `u_device7Connected` | `bool` | Per-device connection |\n| `u_device0Acceleration` - `u_device7Acceleration` | `vec3` | Per-device acceleration (no gravity) |\n| `u_device0AccelerationGravity` - `u_device7AccelerationGravity` | `vec3` | Per-device acceleration (with gravity) |\n| `u_device0RotationRate` - `u_device7RotationRate` | `vec3` | Per-device rotation rate |\n| `u_device0Orientation` - `u_device7Orientation` | `vec3` | Per-device orientation |\n\n### Streams (compositor)\n\nVideo streams (host-provided):\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_videoStreamCount` | `int` | Active streams (0-8) |\n| `u_videoStream0` - `u_videoStream7` | `sampler2D` | Stream textures |\n| `u_videoStream0Resolution` - `u_videoStream7Resolution` | `vec2` | Per-stream resolution |\n| `u_videoStream0Connected` - `u_videoStream7Connected` | `bool` | Per-stream connection |\n\nAudio streams (additional sources, including device audio):\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_audioStreamCount` | `int` | Active streams (0-8) |\n| `u_audioStream0Connected` - `u_audioStream7Connected` | `bool` | Per-slot active |\n| `u_audioStream{i}Volume` | `float` | Volume 0..1 |\n| `u_audioStream{i}Low` - `u_audioStream{i}High` | `float` | Band energies 0..1 |\n| `u_audioStream{i}Brightness`, `u_audioStream{i}Flatness` | `float` | Spectral features 0..1 |\n\n`i` ranges 0..7. **Scalars only**: no `u_audioFFT` / `u_audioWaveform` per stream. Beat / BPM / trigger uniforms are main audio only.\n\n### Backbuffer\n\n| Uniform | Type | Description |\n|---|---|---|\n| `backbuffer` | `sampler2D` | Previous frame (auto-enabled when referenced) |\n\nNo `u_` prefix. RGBA 8-bit, `LINEAR` filtering, `CLAMP_TO_EDGE` wrapping. First frame samples as black. Clears on canvas resize.\n\n## Template\n\n```glsl\n// @renderer shader\n// @viji-slider:speed label:\"Speed\" default:1.0 min:0.1 max:5.0\n// @viji-color:baseColor label:\"Color\" default:#ff6600\n// @viji-accumulator:phase rate:speed\n\nvoid main() {\n vec2 uv = gl_FragCoord.xy / u_resolution;\n\n float wave = sin(uv.x * 10.0 + phase) * 0.5 + 0.5;\n float pulse = 1.0 + u_audioLow * 0.5;\n vec3 color = baseColor * wave * pulse;\n\n gl_FragColor = vec4(color, 1.0);\n}\n```\n"
|
|
9810
9845
|
},
|
|
9811
9846
|
"conversionGuides": {
|
|
9812
|
-
"p5": "# Converting P5.js Sketches to Viji P5\n\nThis guide is loaded by the Viji-Backend AI when the user wants to convert an existing P5.js sketch into a Viji scene. It is self-contained: it includes both the source-to-target mapping and the Viji P5 target reference needed to produce correct converted output, without requiring the Viji P5 system prompt to also be loaded.\n\n## Source platform: standard P5.js sketches\n\nStandard P5.js sketches run in a browser tab. They typically:\n- Use `setup()` to call `createCanvas(w, h)` and configure the sketch (`colorMode`, `frameRate`, etc.).\n- Use `draw()` as the per-frame loop (`background`, `fill`, shape primitives, etc.).\n- Use global `mouseX`, `mouseY`, `mouseIsPressed`, `keyIsPressed`, `frameCount`, `width`, `height`.\n- Define event callbacks at the top level: `mousePressed`, `keyPressed`, `touchStarted`, etc.\n- Use `preload()` to fetch assets via `loadImage`, `loadFont`, `loadJSON`.\n- Possibly use `createCapture(VIDEO)`, `p5.AudioIn()`, `p5.sound`, `p5.dom` add-ons.\n\nP5.js v1.9.4 is what Viji pins. When in doubt about a P5 function, the p5.js v1.x reference is authoritative.\n\n## Target platform: Viji P5\n\nViji P5 scenes run inside a Web Worker on an `OffscreenCanvas` and use P5.js v1.9.4 in **instance mode**. The DOM is unavailable. Every P5 function and constant requires the `p5.` prefix. The canvas is created and managed by Viji; never call `createCanvas()`.\n\n## Conversion rules\n\n1. **Always** set the first line based on the source canvas mode: `// @renderer p5` for 2D (default), or `// @renderer p5 webgl` if the sketch used `createCanvas(w, h, WEBGL)` or any 3D primitives. Never keep `createCanvas()`.\n2. Rename `draw()` to `render(viji, p5)`.\n3. If `setup()` exists, change its signature to `setup(viji, p5)`. If it does not exist, do not add one.\n4. **Always** prefix every P5 function and constant with `p5.`. No exceptions: `background` -> `p5.background`, `fill` -> `p5.fill`, `PI` -> `p5.PI`, `TWO_PI` -> `p5.TWO_PI`, `HSB` -> `p5.HSB`, `createVector(...)` -> `p5.createVector(...)`, `map(...)` -> `p5.map(...)`, `noise(x)` -> `p5.noise(x)`, `random()` -> `p5.random()`.\n5. **Never** keep `createCanvas()`. The canvas is created by Viji. WEBGL is selected only via `// @renderer p5 webgl`, never via `createCanvas(..., p5.WEBGL)`.\n6. **Never** keep `preload()`. Use `viji.image(null, { label: 'Name' })` for images, or `fetch()` inside an async `setup()` for other data.\n7. **Never** keep P5 event callbacks (`mousePressed`, `mouseDragged`, `mouseReleased`, `keyPressed`, `keyReleased`, `keyTyped`, `touchStarted`, `touchMoved`, `touchEnded`). Read state in `render()` via `viji.pointer`, `viji.mouse`, `viji.keyboard`, `viji.touches`. Use `wasPressed` / `wasReleased` for one-frame edge detection.\n8. **Never** keep `p5.frameRate()`, `p5.save()`, `p5.saveCanvas()`, or `p5.saveFrames()`. The host controls frame rate and capture.\n9. **Never** keep `loadImage()`, `loadFont()`, `loadJSON()`, `loadModel()`, `loadShader()`. Use `viji.image()` or `fetch()`.\n10. **Never** keep `createCapture()` or `createVideo()`. Use `viji.video.*`.\n11. **Never** keep `p5.dom` or `p5.sound` add-ons. Replace UI with Viji parameters and audio with `viji.audio.*`.\n12. **Never** keep `window`, `document`, `Image()`, or `localStorage`. `fetch()` is available.\n13. Lift constants and tunable values to top-level parameters using `viji.slider`, `viji.color`, `viji.toggle`, etc. Read via `.value` inside `render()`.\n14. Replace `frameCount`-based timing with a `viji.deltaTime` accumulator. Never multiply `viji.time` by a parameter; it causes visible jumps when the parameter changes. Same rule for nested multiplications: each independent speed needs its own accumulator.\n15. **Always** check `viji.audio.isConnected` before reading audio data. Check `viji.video.isConnected && viji.video.currentFrame` before drawing video.\n16. **Never** enable CV features by default; use toggle parameters.\n17. **Always** set `category` on input-dependent parameters: `'audio'`, `'video'`, `'interaction'`.\n\n## Direct mapping table\n\n| Standard P5.js | Viji P5 equivalent |\n|---|---|\n| `width` / `height` | `viji.width` / `viji.height` |\n| `mouseX` / `mouseY` | `viji.pointer.x` / `viji.pointer.y` (or `viji.mouse.x` / `viji.mouse.y`) |\n| `mouseIsPressed` | `viji.pointer.isDown` (or `viji.mouse.isPressed`) |\n| `mouseButton === LEFT` | `viji.mouse.leftButton` |\n| `keyIsPressed` | `viji.keyboard.activeKeys.size > 0` |\n| `keyIsDown(code)` | `viji.keyboard.isPressed('keyName')` |\n| `key` | `viji.keyboard.lastKeyPressed` |\n| `frameCount` | `viji.frameCount` (or a `viji.deltaTime` accumulator) |\n| `frameRate(n)` | Remove. Host controls frame rate. |\n| `createCanvas(w, h)` | Remove. Canvas is provided. Use `// @renderer p5`. |\n| `createCanvas(w, h, WEBGL)` | Remove. Use `// @renderer p5 webgl` as the first line. |\n| `preload()` | Remove. Use `viji.image()` or `fetch()` in `setup()`. |\n| `loadImage(url)` | `viji.image(null, { label: 'Image' })`; pass `.p5` to `p5.image()`. |\n| `loadFont(url)` | Not available. Use CSS generic font names with `p5.textFont('monospace' \\| 'serif' \\| 'sans-serif')`. |\n| `loadJSON(url)` | `await fetch(url).then(r => r.json())` inside async `setup()`. |\n| `save()` / `saveCanvas()` | Remove. Host handles capture. |\n| `createCapture(VIDEO)` | `viji.video.currentFrame` (plus `viji.video.isConnected` check). |\n| `new p5.AudioIn(); mic.getLevel()` | `viji.audio.volume.current` (after `viji.audio.isConnected` check). |\n| `new p5.FFT(); fft.analyze()` | `viji.audio.getFrequencyData()` (`Uint8Array` of 1024 bins). |\n| `mousePressed()` callback | `if (viji.pointer.wasPressed) { ... }` inside `render()`. |\n| `keyPressed()` callback | `if (viji.keyboard.wasPressed(key)) { ... }` inside `render()`. |\n| `touchStarted()` callback | `viji.touches.started.forEach(t => ...)` inside `render()`. |\n\n## Target API reference (Viji P5)\n\nThe `viji` object is identical to Native. Access it inside `setup(viji, p5)` and `render(viji, p5)`. `viji.useContext()` is **not available** in P5; the context is managed by P5.\n\n**Canvas and timing**: `viji.canvas` (`OffscreenCanvas`), `viji.width`, `viji.height`, `viji.time`, `viji.deltaTime`, `viji.frameCount`, `viji.fps`.\n\n**Parameters** (top-level only, read via `.value`):\n```javascript\nviji.slider(default, { min?, max?, step?, label, group?, category? }) // .value: number\nviji.color(default, { label, group?, category? }) // .value: '#rrggbb', .rgb, .hsb\nviji.toggle(default, { label, group?, category? }) // .value: boolean\nviji.select(default, { options, label, group?, category? }) // .value: string | number\nviji.number(default, { min?, max?, step?, label, group?, category? }) // .value: number\nviji.text(default, { label, group?, category?, maxLength? }) // .value: string\nviji.image(null, { label, group?, category? }) // .value: ImageBitmap|null, .p5: P5Image\nviji.button({ label, description?, group?, category? }) // .value: boolean (1 frame)\nviji.coordinate(default, { step?, label, group?, category? }) // .value: { x, y }, range -1..1\n```\nFor images displayed with P5, pass `param.p5` (a `P5Image`) to `p5.image()`, not `param.value`.\n\n**Audio: `viji.audio`**: check `isConnected` first. Members: `volume.{current, peak, smoothed}` (0..1; `smoothed` 200ms decay); `bands.{low, lowMid, mid, highMid, high}` (instant 0..1) and the sibling `*Smoothed` envelopes (150ms decay); `beat.{kick, snare, hat, any}` (300ms decay curves; peak on detected beats); `beat.{kickSmoothed, snareSmoothed, hatSmoothed, anySmoothed}` (500ms decay envelopes); `beat.triggers.{kick, snare, hat, any}` (boolean, true for one frame, OR-accumulated); `beat.events: Array<{type, time, strength}>` (`type` is `'kick' | 'snare' | 'hat'`; cleared each frame); `beat.bpm` (`0` when no audio is connected; once audio connects it tracks the detected tempo clamped to 60..240, with `120` as a fallback before lock-on), `beat.confidence` (0..1), `beat.isLocked` (boolean); `spectral.{brightness, flatness}` (0..1); `getFrequencyData(): Uint8Array` (1024 FFT bins 0..255); `getWaveform(): Float32Array` (2048 samples -1..1).\n\n**Video: `viji.video`**: check `isConnected && currentFrame` first. Members: `currentFrame` (`OffscreenCanvas | ImageBitmap | null`), `frameWidth`, `frameHeight`, `frameRate`, `getFrameData()`, `cv` (VideoCVAPI). CV outputs (`analysedFrame`, `getAnalysedFrameData`, `faces`, `hands`, `pose`, `segmentation`) and CV verbs live on `viji.video.cv`, not on `viji.video` directly.\n\n**Aspect handling** (use for every video / CV conversion):\n```javascript\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\nconst v = videoFit(viji, 'cover');\np5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n// CV coords are normalized 0..1 to the source frame; map through v to align.\n```\nDefault `viji.video.currentFrame` for displayed video; reach for `viji.video.cv.analysedFrame` only when sampling pixels at CV-derived positions. Common fallback pattern: `viji.video.cv.analysedFrame ?? viji.video.currentFrame`.\n\n**CV: `viji.video.cv`**: enable explicitly (never by default):\n```javascript\nawait viji.video.cv.enableFaceDetection(true);\nawait viji.video.cv.enableFaceMesh(true); // implies face detection\nawait viji.video.cv.enableEmotionDetection(true); // implies face mesh\nawait viji.video.cv.enableHandTracking(true);\nawait viji.video.cv.enablePoseDetection(true);\nawait viji.video.cv.enableBodySegmentation(true);\n```\nAlso on `viji.video.cv`: `analysedFrame: OffscreenCanvas | null` (frame paired with current CV results), `getAnalysedFrameData(): ImageData | null`, `getActiveFeatures(): CVFeature[]`, `isProcessing(): boolean`.\n\nData shapes on `viji.video.cv`: `faces: FaceData[]` (`{id, bounds, center, confidence, landmarks, expressions, headPose, blendshapes}`; `blendshapes` are 52 ARKit coefficients 0..1), `hands: HandData[]` (`{id, handedness, confidence, bounds, landmarks (21 pts), palm, gestures: {fist, openPalm, peace, thumbsUp, thumbsDown, pointing, iLoveYou}}`), `pose: PoseData | null` (`{confidence, landmarks (33 pts), face, torso, leftArm, rightArm, leftLeg, rightLeg}`), `segmentation: SegmentationData | null` (`{mask: Uint8Array (byte values 0 or 1), width, height}`).\n\n**Input**:\n- `viji.pointer`: `x`, `y`, `deltaX`, `deltaY`, `isDown`, `wasPressed`, `wasReleased`, `isInCanvas`, `type`.\n- `viji.mouse`: `x`, `y`, `isInCanvas`, `isPressed`, `leftButton`, `rightButton`, `middleButton`, `deltaX`, `deltaY`, `wheelDelta`, `wasPressed`, `wasReleased`, `wasMoved`.\n- `viji.keyboard`: `isPressed(key)`, `wasPressed(key)`, `wasReleased(key)`, `activeKeys`, `lastKeyPressed`, `lastKeyReleased`, `shift`, `ctrl`, `alt`, `meta`.\n- `viji.touches`: `count`, `points`, `started`, `moved`, `ended`, `primary`: each `TouchPoint` has `id`, `x`, `y`, `pressure`, `radius`, `velocity`, etc.\n\n**Sensors / streams**: `viji.device.{motion, orientation}`, `viji.devices[]`, `viji.videoStreams[]`, `viji.audioStreams[]` (lightweight `AudioStreamAPI`: no beat / BPM).\n\n## P5 gotchas in the worker\n\n- **Fonts**: `p5.textFont()` only with CSS generic names (`'monospace'`, `'serif'`, `'sans-serif'`). `loadFont()` is unavailable.\n- **`p5.createGraphics(w, h)`** works (creates an internal OffscreenCanvas). `createGraphics(w, h, p5.WEBGL)` is not supported.\n- **`p5.pixelDensity()`** defaults to 1 in the worker. `p5.loadPixels()` and `p5.pixels[]` work in 2D scenes.\n- **`p5.drawingContext`** is a 2D context only in 2D scenes. In WEBGL scenes, it is a WebGL context; do not call Canvas-2D APIs on it.\n- **`viji.useContext()`** is not available in P5; the context belongs to P5.\n- **`p5.tint()` and `p5.blendMode()`** work normally.\n\n## Worked example\n\nSource P5 sketch:\n```javascript\nlet angle = 0;\nfunction setup() {\n createCanvas(800, 600);\n colorMode(HSB, 360, 100, 100);\n}\nfunction draw() {\n angle += 0.02;\n background(0, 0, 10);\n const x = width / 2 + cos(angle) * width * 0.3;\n const y = height / 2 + sin(angle) * height * 0.3;\n noStroke();\n fill((frameCount * 0.5) % 360, 80, 100);\n circle(x, y, width * 0.05);\n}\nfunction mousePressed() {\n angle = 0;\n}\n```\n\nConverted Viji P5 scene:\n```javascript\n// @renderer p5\n\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nconst reactivity = viji.slider(0, { min: 0, max: 1, label: 'Audio Reactivity', category: 'audio' });\n\nlet angle = 0;\nlet huePhase = 0;\n\nfunction setup(viji, p5) {\n p5.colorMode(p5.HSB, 360, 100, 100);\n}\n\nfunction render(viji, p5) {\n if (viji.pointer.wasPressed) angle = 0;\n\n const audioBoost = viji.audio.isConnected ? viji.audio.volume.smoothed * reactivity.value : 0;\n angle += (speed.value + audioBoost * 4) * viji.deltaTime;\n huePhase += 30 * viji.deltaTime;\n\n p5.background(0, 0, 10);\n const x = viji.width / 2 + p5.cos(angle) * viji.width * 0.3;\n const y = viji.height / 2 + p5.sin(angle) * viji.height * 0.3;\n p5.noStroke();\n p5.fill(huePhase % 360, 80, 100);\n p5.circle(x, y, viji.width * 0.05);\n}\n```\n\nKey changes: removed `createCanvas`, added `// @renderer p5`; renamed `draw` to `render(viji, p5)`; gave `setup` the `(viji, p5)` signature; added `p5.` prefix everywhere; replaced `width`/`height` with `viji.width`/`viji.height`; replaced the implicit `0.02` frame-rate-coupled animation with a `viji.deltaTime` accumulator driven by a `speed` slider; replaced `(frameCount * 0.5) % 360` hue with a separate `huePhase` accumulator (so changing speed never jumps the color); replaced the `mousePressed()` callback with a `viji.pointer.wasPressed` check inside `render`; added an opt-in audio-reactive `reactivity` slider with `category: 'audio'`.\n",
|
|
9847
|
+
"p5": "# Converting P5.js Sketches to Viji P5\n\nThis guide is loaded by the Viji-Backend AI when the user wants to convert an existing P5.js sketch into a Viji scene. It is self-contained: it includes both the source-to-target mapping and the Viji P5 target reference needed to produce correct converted output, without requiring the Viji P5 system prompt to also be loaded.\n\n## Source platform: standard P5.js sketches\n\nStandard P5.js sketches run in a browser tab. They typically:\n- Use `setup()` to call `createCanvas(w, h)` and configure the sketch (`colorMode`, `frameRate`, etc.).\n- Use `draw()` as the per-frame loop (`background`, `fill`, shape primitives, etc.).\n- Use global `mouseX`, `mouseY`, `mouseIsPressed`, `keyIsPressed`, `frameCount`, `width`, `height`.\n- Define event callbacks at the top level: `mousePressed`, `keyPressed`, `touchStarted`, etc.\n- Use `preload()` to fetch assets via `loadImage`, `loadFont`, `loadJSON`.\n- Possibly use `createCapture(VIDEO)`, `p5.AudioIn()`, `p5.sound`, `p5.dom` add-ons.\n\nP5.js v1.9.4 is what Viji pins. When in doubt about a P5 function, the p5.js v1.x reference is authoritative.\n\n## Target platform: Viji P5\n\nViji P5 scenes run inside a Web Worker on an `OffscreenCanvas` and use P5.js v1.9.4 in **instance mode**. The DOM is unavailable. Every P5 function and constant requires the `p5.` prefix. The canvas is created and managed by Viji; never call `createCanvas()`.\n\n## Conversion rules\n\n1. **Always** set the first line based on the source canvas mode: `// @renderer p5` for 2D (default), or `// @renderer p5 webgl` if the sketch used `createCanvas(w, h, WEBGL)` or any 3D primitives. Never keep `createCanvas()`.\n2. Rename `draw()` to `render(viji, p5)`.\n3. If `setup()` exists, change its signature to `setup(viji, p5)`. If it does not exist, do not add one.\n4. **Always** prefix every P5 function and constant with `p5.`. No exceptions: `background` -> `p5.background`, `fill` -> `p5.fill`, `PI` -> `p5.PI`, `TWO_PI` -> `p5.TWO_PI`, `HSB` -> `p5.HSB`, `createVector(...)` -> `p5.createVector(...)`, `map(...)` -> `p5.map(...)`, `noise(x)` -> `p5.noise(x)`, `random()` -> `p5.random()`.\n5. **Never** keep `createCanvas()`. The canvas is created by Viji. WEBGL is selected only via `// @renderer p5 webgl`, never via `createCanvas(..., p5.WEBGL)`.\n6. **Never** keep `preload()`. Use `viji.image(null, { label: 'Name' })` for images, or `fetch()` inside an async `setup()` for other data.\n7. **Never** keep P5 event callbacks (`mousePressed`, `mouseDragged`, `mouseReleased`, `keyPressed`, `keyReleased`, `keyTyped`, `touchStarted`, `touchMoved`, `touchEnded`). Read state in `render()` via `viji.pointer`, `viji.mouse`, `viji.keyboard`, `viji.touches`. Use `wasPressed` / `wasReleased` for one-frame edge detection.\n8. **Never** keep `p5.frameRate()`, `p5.save()`, `p5.saveCanvas()`, or `p5.saveFrames()`. The host controls frame rate and capture.\n9. **Never** keep `loadImage()`, `loadFont()`, `loadJSON()`, `loadModel()`, `loadShader()`. Use `viji.image()` or `fetch()`.\n10. **Never** keep `createCapture()` or `createVideo()`. Use `viji.video.*`.\n11. **Never** keep `p5.dom` or `p5.sound` add-ons. Replace UI with Viji parameters and audio with `viji.audio.*`.\n12. **Never** keep `window`, `document`, `Image()`, or `localStorage`. `fetch()` is available.\n13. Lift constants and tunable values to top-level parameters using `viji.slider`, `viji.color`, `viji.toggle`, etc. Read via `.value` inside `render()`.\n14. Replace `frameCount`-based timing with a `viji.deltaTime` accumulator. Never multiply `viji.time` by a parameter; it causes visible jumps when the parameter changes. Same rule for nested multiplications: each independent speed needs its own accumulator.\n15. **Always** check `viji.audio.isConnected` before reading audio data. Check `viji.video.isConnected && viji.video.currentFrame` before drawing video.\n16. **Never** enable CV features by default; use toggle parameters.\n17. **Always** set `category` on input-dependent parameters: `'audio'`, `'video'`, `'interaction'`.\n\n## Direct mapping table\n\n| Standard P5.js | Viji P5 equivalent |\n|---|---|\n| `width` / `height` | `viji.width` / `viji.height` |\n| `mouseX` / `mouseY` | `viji.pointer.x` / `viji.pointer.y` (or `viji.mouse.x` / `viji.mouse.y`) |\n| `mouseIsPressed` | `viji.pointer.isDown` (or `viji.mouse.isPressed`) |\n| `mouseButton === LEFT` | `viji.mouse.leftButton` |\n| `keyIsPressed` | `viji.keyboard.activeKeys.size > 0` |\n| `keyIsDown(code)` | `viji.keyboard.isPressed('keyName')` |\n| `key` | `viji.keyboard.lastKeyPressed` |\n| `frameCount` | `viji.frameCount` (or a `viji.deltaTime` accumulator) |\n| `frameRate(n)` | Remove. Host controls frame rate. |\n| `createCanvas(w, h)` | Remove. Canvas is provided. Use `// @renderer p5`. |\n| `createCanvas(w, h, WEBGL)` | Remove. Use `// @renderer p5 webgl` as the first line. |\n| `preload()` | Remove. Use `viji.image()` or `fetch()` in `setup()`. |\n| `loadImage(url)` | `viji.image(null, { label: 'Image' })`; pass `.p5` to `p5.image()`. |\n| `loadFont(url)` | Not available. Use CSS generic font names with `p5.textFont('monospace' \\| 'serif' \\| 'sans-serif')`. |\n| `loadJSON(url)` | `await fetch(url).then(r => r.json())` inside async `setup()`. |\n| `save()` / `saveCanvas()` | Remove. Host handles capture. |\n| `createCapture(VIDEO)` | `viji.video.currentFrame` (plus `viji.video.isConnected` check). |\n| `new p5.AudioIn(); mic.getLevel()` | `viji.audio.volume.current` (after `viji.audio.isConnected` check). |\n| `new p5.FFT(); fft.analyze()` | `viji.audio.getFrequencyData()` (`Uint8Array` of 1024 bins). |\n| `mousePressed()` callback | `if (viji.pointer.wasPressed) { ... }` inside `render()`. |\n| `keyPressed()` callback | `if (viji.keyboard.wasPressed(key)) { ... }` inside `render()`. |\n| `touchStarted()` callback | `viji.touches.started.forEach(t => ...)` inside `render()`. |\n\n## Target API reference (Viji P5)\n\nThe `viji` object is identical to Native. Access it inside `setup(viji, p5)` and `render(viji, p5)`. `viji.useContext()` is **not available** in P5; the context is managed by P5.\n\n**Canvas and timing**: `viji.canvas` (`OffscreenCanvas`), `viji.width`, `viji.height`, `viji.time`, `viji.deltaTime`, `viji.frameCount`, `viji.fps`.\n\n**Parameters** (top-level only, read via `.value`):\n```javascript\nviji.slider(default, { min?, max?, step?, label, group?, category? }) // .value: number\nviji.color(default, { label, group?, category? }) // .value: '#rrggbb', .rgb, .hsb\nviji.toggle(default, { label, group?, category? }) // .value: boolean\nviji.select(default, { options, label, group?, category? }) // .value: string | number\nviji.number(default, { min?, max?, step?, label, group?, category? }) // .value: number\nviji.text(default, { label, group?, category?, maxLength? }) // .value: string\nviji.image(null, { label, group?, category? }) // .value: ImageBitmap|null, .p5: P5Image\nviji.button({ label, description?, group?, category? }) // .value: boolean (1 frame)\nviji.coordinate(default, { step?, label, group?, category? }) // .value: { x, y }, range -1..1\n```\nFor images displayed with P5, pass `param.p5` (a `P5Image`) to `p5.image()`, not `param.value`.\n\n**Audio: `viji.audio`**: check `isConnected` first. Members: `volume.{current, peak, smoothed}` (0..1; `smoothed` 200ms decay); `bands.{low, lowMid, mid, highMid, high}` (instant 0..1) and the sibling `*Smoothed` envelopes (150ms decay); `beat.{kick, snare, hat, any}` (300ms decay curves; peak on detected beats); `beat.{kickSmoothed, snareSmoothed, hatSmoothed, anySmoothed}` (500ms decay envelopes); `beat.triggers.{kick, snare, hat, any}` (boolean, true for one frame, OR-accumulated); `beat.events: Array<{type, time, strength}>` (`type` is `'kick' | 'snare' | 'hat'`; cleared each frame); `beat.bpm` (`0` when no audio is connected; once audio connects it tracks the detected tempo clamped to 60..240, with `120` as a fallback before lock-on), `beat.confidence` (0..1), `beat.isLocked` (boolean); `spectral.{brightness, flatness}` (0..1); `getFrequencyData(): Uint8Array` (1024 FFT bins 0..255); `getWaveform(): Float32Array` (2048 samples -1..1).\n\n**Video: `viji.video`**: check `isConnected && currentFrame` first. Members: `currentFrame` (`OffscreenCanvas | ImageBitmap | null`), `frameWidth`, `frameHeight`, `frameRate`, `getFrameData()`, `cv` (VideoCVAPI). CV outputs (`analysedFrame`, `getAnalysedFrameData`, `faces`, `hands`, `pose`, `segmentation`) and CV verbs live on `viji.video.cv`, not on `viji.video` directly.\n\n**Aspect handling** (use for every video / CV conversion):\n```javascript\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\nconst v = videoFit(viji, 'cover');\np5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n// CV coords are normalized 0..1 to the source frame; map through v to align.\n```\nDefault `viji.video.currentFrame` for displayed video; reach for `viji.video.cv.analysedFrame` only when sampling pixels at CV-derived positions. Common fallback pattern: `viji.video.cv.analysedFrame ?? viji.video.currentFrame`.\n\n**CV: `viji.video.cv`**: enable explicitly (never by default):\n```javascript\nawait viji.video.cv.enableFaceDetection(true);\nawait viji.video.cv.enableFaceMesh(true); // populates face.landmarks + face.headPose\nawait viji.video.cv.enableEmotionDetection(true); // populates face.blendshapes + face.expressions; loads landmarker\nawait viji.video.cv.enableHandTracking(true);\nawait viji.video.cv.enablePoseDetection(true);\nawait viji.video.cv.enableBodySegmentation(true);\n```\nAlso on `viji.video.cv`: `analysedFrame: OffscreenCanvas | null` (frame paired with current CV results), `getAnalysedFrameData(): ImageData | null`, `getActiveFeatures(): CVFeature[]`, `isProcessing(): boolean`.\n\nData shapes on `viji.video.cv`: `faces: FaceData[]` (`{id, bounds, center, confidence, landmarks, expressions, headPose, blendshapes}`; `blendshapes` are 52 ARKit coefficients 0..1), `hands: HandData[]` (`{id, handedness, confidence, bounds, landmarks (21 pts), palm, gestures: {fist, openPalm, peace, thumbsUp, thumbsDown, pointing, iLoveYou}}`), `pose: PoseData | null` (`{confidence, landmarks (33 pts), face, torso, leftArm, rightArm, leftLeg, rightLeg}`), `segmentation: SegmentationData | null` (`{mask: Uint8Array (byte values 0 or 1), width, height}`).\n\n**Input**:\n- `viji.pointer`: `x`, `y`, `deltaX`, `deltaY`, `isDown`, `wasPressed`, `wasReleased`, `isInCanvas`, `type`.\n- `viji.mouse`: `x`, `y`, `isInCanvas`, `isPressed`, `leftButton`, `rightButton`, `middleButton`, `deltaX`, `deltaY`, `wheelDelta`, `wasPressed`, `wasReleased`, `wasMoved`.\n- `viji.keyboard`: `isPressed(key)`, `wasPressed(key)`, `wasReleased(key)`, `activeKeys`, `lastKeyPressed`, `lastKeyReleased`, `shift`, `ctrl`, `alt`, `meta`.\n- `viji.touches`: `count`, `points`, `started`, `moved`, `ended`, `primary`: each `TouchPoint` has `id`, `x`, `y`, `pressure`, `radius`, `velocity`, etc.\n\n**Sensors / streams**: `viji.device.{motion, orientation}`, `viji.devices[]`, `viji.videoStreams[]`, `viji.audioStreams[]` (lightweight `AudioStreamAPI`: no beat / BPM).\n\n## P5 gotchas in the worker\n\n- **Fonts**: `p5.textFont()` only with CSS generic names (`'monospace'`, `'serif'`, `'sans-serif'`). `loadFont()` is unavailable.\n- **`p5.createGraphics(w, h)`** works (creates an internal OffscreenCanvas). `createGraphics(w, h, p5.WEBGL)` is not supported.\n- **`p5.pixelDensity()`** defaults to 1 in the worker. `p5.loadPixels()` and `p5.pixels[]` work in 2D scenes.\n- **`p5.drawingContext`** is a 2D context only in 2D scenes. In WEBGL scenes, it is a WebGL context; do not call Canvas-2D APIs on it.\n- **`viji.useContext()`** is not available in P5; the context belongs to P5.\n- **`p5.tint()` and `p5.blendMode()`** work normally.\n\n## Worked example\n\nSource P5 sketch:\n```javascript\nlet angle = 0;\nfunction setup() {\n createCanvas(800, 600);\n colorMode(HSB, 360, 100, 100);\n}\nfunction draw() {\n angle += 0.02;\n background(0, 0, 10);\n const x = width / 2 + cos(angle) * width * 0.3;\n const y = height / 2 + sin(angle) * height * 0.3;\n noStroke();\n fill((frameCount * 0.5) % 360, 80, 100);\n circle(x, y, width * 0.05);\n}\nfunction mousePressed() {\n angle = 0;\n}\n```\n\nConverted Viji P5 scene:\n```javascript\n// @renderer p5\n\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nconst reactivity = viji.slider(0, { min: 0, max: 1, label: 'Audio Reactivity', category: 'audio' });\n\nlet angle = 0;\nlet huePhase = 0;\n\nfunction setup(viji, p5) {\n p5.colorMode(p5.HSB, 360, 100, 100);\n}\n\nfunction render(viji, p5) {\n if (viji.pointer.wasPressed) angle = 0;\n\n const audioBoost = viji.audio.isConnected ? viji.audio.volume.smoothed * reactivity.value : 0;\n angle += (speed.value + audioBoost * 4) * viji.deltaTime;\n huePhase += 30 * viji.deltaTime;\n\n p5.background(0, 0, 10);\n const x = viji.width / 2 + p5.cos(angle) * viji.width * 0.3;\n const y = viji.height / 2 + p5.sin(angle) * viji.height * 0.3;\n p5.noStroke();\n p5.fill(huePhase % 360, 80, 100);\n p5.circle(x, y, viji.width * 0.05);\n}\n```\n\nKey changes: removed `createCanvas`, added `// @renderer p5`; renamed `draw` to `render(viji, p5)`; gave `setup` the `(viji, p5)` signature; added `p5.` prefix everywhere; replaced `width`/`height` with `viji.width`/`viji.height`; replaced the implicit `0.02` frame-rate-coupled animation with a `viji.deltaTime` accumulator driven by a `speed` slider; replaced `(frameCount * 0.5) % 360` hue with a separate `huePhase` accumulator (so changing speed never jumps the color); replaced the `mousePressed()` callback with a `viji.pointer.wasPressed` check inside `render`; added an opt-in audio-reactive `reactivity` slider with `category: 'audio'`.\n",
|
|
9813
9848
|
"shadertoy": "# Converting Shadertoy Shaders to Viji Shader Scenes\n\nThis guide is loaded by the Viji-Backend AI when the user wants to convert an existing Shadertoy shader into a Viji shader scene. It is self-contained: it includes both the source-to-target mapping and the Viji shader target reference needed to produce correct converted output, without requiring the Viji shader system prompt to also be loaded.\n\n## Source platform: Shadertoy\n\nShadertoy shaders are GLSL fragment shaders authored against a fixed API:\n- Entry point: `void mainImage(out vec4 fragColor, in vec2 fragCoord)`.\n- Built-in uniforms: `iResolution` (`vec3`: `.xy` size, `.z` aspect), `iTime`, `iTimeDelta`, `iFrame`, `iMouse` (`vec4`: `.xy` current position, `.zw` click origin), `iDate` (year, month, day, seconds), `iChannel0`-`iChannel3` (textures whose meaning depends on the channel binding: Image, FFT, Sound, Music, Microphone, Keyboard, Cubemap).\n- Side metadata: `iChannelResolution[i]`, `iChannelTime[i]`, `iSampleRate`.\n- Multi-buffer pipelines (Buffer A, B, C, D), Cubemap buffer, Sound output buffer, VR mode (`mainVR`). None of these have Viji equivalents.\n\n## Target platform: Viji shader\n\nGLSL fragment shader on a fullscreen quad. Standard `void main()` entry. Precision and all uniforms (built-ins and `@viji-*` directive uniforms) are auto-injected by Viji. **Never declare them.** Artists declare parameters via `// @viji-*` line-comment directives, which compile to typed uniforms.\n\nGLSL ES 1.00 by default. Opt into ES 3.00 by making `#version 300 es` the literal first line.\n\n## Conversion rules\n\n1. **Always** add `// @renderer shader` as the very first line (or as line 2 after `#version 300 es`).\n2. **Never** declare `precision mediump float;` or `precision highp float;`. Viji auto-injects precision.\n3. **Never** redeclare built-in uniforms. They are auto-injected.\n4. **Never** redeclare parameter uniforms. They are auto-generated from `@viji-*` directives.\n5. Convert the `mainImage` signature to standard `void main()`:\n - Replace `fragCoord` with `gl_FragCoord.xy`.\n - Replace `fragColor` (the `out` parameter) with `gl_FragColor` (ES 1.00) or the declared `out vec4 fragColor` (ES 3.00).\n - Remove the `mainImage` wrapper entirely.\n\n6. Replace Shadertoy uniforms with Viji equivalents:\n\n | Shadertoy | Viji | Notes |\n |---|---|---|\n | `iResolution.xy` | `u_resolution` | Viji's `u_resolution` is `vec2`. |\n | `iResolution.z` | `u_resolution.x / u_resolution.y` | Aspect ratio. |\n | `iResolution.x` / `.y` | `u_resolution.x` / `.y` | Direct match. |\n | `iTime` | `u_time` | Elapsed seconds. |\n | `iTimeDelta` | `u_deltaTime` | Seconds since last frame. |\n | `iFrame` | `u_frame` (`int`) | Frame counter. |\n | `iMouse.xy` | `u_mouse` | Current mouse position in pixels. |\n | `iMouse.z` | `u_mouseLeft ? u_mouse.x : 0.0` | Viji does not track click origin. |\n | `iMouse.w` | `u_mouseLeft ? u_mouse.y : 0.0` | Viji does not track click origin. |\n | `iChannelN` (static Image) | `@viji-image:channelN label:\"…\"` | See rule 7. |\n | `iChannelN` (FFT) | `u_audioFFT` | Auto-injected `sampler2D`, 1024 bins 0..255. |\n | `iChannelN` (Sound / Music / Microphone) | `u_audioFFT` and/or `u_audioWaveform` | See two-row note in rule 7. |\n | `iChannelN` (Keyboard) | `u_keyboard` | See rule 8. |\n | `iChannelN` (Cubemap) | Not supported | Only 2D textures available. |\n | `iChannelResolution[i]` | Not available | Track manually if needed. |\n | `iChannelTime[i]` | Not available | Per-channel time is not tracked. |\n | `iDate` | Not available | Use `u_time`. |\n | `iSampleRate` | Not available | Not applicable. |\n\n7. **Static-image `iChannel` slots** become `@viji-image` parameters:\n ```glsl\n // @viji-image:channel0 label:\"Texture 1\"\n ```\n Then replace `texture(iChannel0, uv)` with `texture2D(channel0, uv)` (ES 1.00) or `texture(channel0, uv)` (ES 3.00).\n\n **Audio iChannel slots** (Sound / Music / Microphone) sample the auto-injected audio textures directly:\n ```glsl\n float spectrum = texture2D(u_audioFFT, vec2(uv.x, 0.0)).r;\n float waveform = texture2D(u_audioWaveform, vec2(uv.x, 0.0)).r;\n ```\n\n **Two-row audio channel**: Shadertoy's Sound / Music / Microphone iChannels encode FFT in row 0 (`y` near 0.0) and waveform in row 1 (`y` near 0.5+) of a single texture. When converting:\n - `texture(iChannelN, vec2(uv.x, 0.0)).r` (row 0) -> `texture2D(u_audioFFT, vec2(uv.x, 0.0)).r`.\n - `texture(iChannelN, vec2(uv.x, 0.5)).r` or `vec2(uv.x, 0.75)` (row 1) -> `texture2D(u_audioWaveform, vec2(uv.x, 0.0)).r`.\n - When unsure: prefer `u_audioFFT` for spectrum-style visuals, `u_audioWaveform` for oscilloscope-style visuals.\n\n8. **Keyboard `iChannel` slots** become `u_keyboard`:\n ```glsl\n // Shadertoy: texelFetch(iChannel0, ivec2(KEY, 0), 0).x\n // Viji: texelFetch(u_keyboard, ivec2(KEY, 0), 0).x\n ```\n `u_keyboard` is a built-in `sampler2D` (256x3). Row 0 = held, row 1 = pressed-this-frame, row 2 = toggle. Do not declare it.\n\n9. `iResolution` used as `vec3`: replace with `vec3(u_resolution, u_resolution.x / u_resolution.y)`, or refactor to use the `vec2` form directly.\n\n10. **Parameter-driven animation must use `@viji-accumulator`, never `u_time * speed`.** Multiplying `u_time` by a parameter causes visible jumps when the slider changes.\n\n ```glsl\n // WRONG (source pattern)\n float t = iTime * speed;\n\n // RIGHT (converted)\n // @viji-slider:speed label:\"Speed\" default:1.0 min:0.1 max:5.0\n // @viji-accumulator:phase rate:speed\n float t = phase;\n ```\n\n Same rule for nested multiplications. Each independent speed gets its own accumulator.\n\n11. Adding artist-controllable parameters uses `@viji-*` directives:\n ```glsl\n // @viji-slider:name label:\"Label\" default:1.0 min:0.0 max:5.0\n // @viji-color:name label:\"Label\" default:#ff6600\n // @viji-toggle:name label:\"Label\" default:true\n // @viji-select:name label:\"Mode\" default:0 options:[\"A\",\"B\",\"C\"]\n // @viji-image:name label:\"Texture\"\n // @viji-button:name label:\"Reset\"\n // @viji-coordinate:name label:\"Origin\" default:[0.0,0.0]\n // @viji-accumulator:name rate:speed\n ```\n Never use the `u_` prefix for parameter names; it is reserved for built-in uniforms. Directives work with `//` line comments only.\n\n12. **`#version 300 es`** stays as the literal first line if present. Then `// @renderer shader` on line 2. Replace `gl_FragColor = ...` with a declared `out vec4 fragColor;` (before `main`) and `fragColor = ...`. Replace `texture2D()` with `texture()`.\n\n13. Remove any `#ifdef GL_ES` / `precision` blocks. Viji handles them.\n\n14. **Always** set `category` on input-dependent directives: `category:audio` for audio controls, `category:video` for video controls, `category:interaction` for mouse / touch controls.\n\n15. **Single Buffer feedback** becomes Viji's `backbuffer`:\n ```glsl\n vec4 prev = texture2D(backbuffer, uv);\n ```\n `backbuffer` is auto-detected and enabled when referenced. RGBA 8-bit, `LINEAR`, `CLAMP_TO_EDGE`, no `u_` prefix. First frame samples as black.\n\n## Unsupported features (warn the artist)\n\n- **Multi-buffer pipelines** (Buffer A -> B -> C -> D): only single `backbuffer` is available.\n- **Cubemap buffer** (`samplerCube`): not supported.\n- **3D textures** (`sampler3D`): not supported.\n- **`iChannelTime[i]`**, **`iChannelResolution[i]`**, **`iSampleRate`**, **`iDate`**: not available.\n- **Sound output buffer**: not supported.\n- **`mainVR()`**: not supported.\n- **Texture wrap/filter modes**: Viji uses fixed `CLAMP_TO_EDGE` + `LINEAR`. Use `fract(uv)` for repeat.\n\n## Target uniform reference (Viji shader)\n\nAll uniforms below are auto-injected. Never declare them.\n\n**Core**: `u_resolution` (`vec2`), `u_time` (`float`), `u_deltaTime` (`float`), `u_frame` (`int`), `u_fps` (`float`).\n\n**Mouse**: `u_mouse` (`vec2`, pixels, bottom-left origin), `u_mouseInCanvas`, `u_mousePressed`, `u_mouseLeft`, `u_mouseRight`, `u_mouseMiddle` (`bool`), `u_mouseDelta` (`vec2`), `u_mouseWheel` (`float`), `u_mouseWasPressed`, `u_mouseWasReleased` (`bool`).\n\n**Pointer (unified mouse / touch)**: `u_pointer` (`vec2`), `u_pointerDelta` (`vec2`), `u_pointerDown`, `u_pointerWasPressed`, `u_pointerWasReleased`, `u_pointerInCanvas` (`bool`).\n\n**Keyboard**: `u_keySpace`, `u_keyShift`, `u_keyCtrl`, `u_keyAlt`, `u_keyW`/`A`/`S`/`D`, `u_keyUp`/`Down`/`Left`/`Right` (`bool`). `u_keyboard` (`sampler2D`, 256x3): row 0 held, row 1 pressed-this-frame, row 2 toggle.\n\n**Touch**: `u_touchCount` (`int`), `u_touch0`-`u_touch4` (`vec2`).\n\n**Audio scalars**: `u_audioVolume`, `u_audioPeak`, `u_audioVolumeSmoothed` (200ms decay), `u_audioLow`/`LowMid`/`Mid`/`HighMid`/`High` (band energies 20-120 / 120-400 / 400-1600 / 1600-6000 / 6000-16000 Hz), `u_audioLowSmoothed`-`u_audioHighSmoothed` (150ms decay), `u_audioKick`/`Snare`/`Hat`/`Any` (300ms decay curves), `u_audioKickSmoothed`-`u_audioAnySmoothed` (500ms decay), `u_audioKickTrigger`/`SnareTrigger`/`HatTrigger`/`AnyTrigger` (`bool`, true for one frame, OR-accumulated), `u_audioBPM` (`0.0` when no audio is connected; once audio connects it tracks the detected tempo clamped to 60..240, with `120.0` as a fallback before lock-on), `u_audioConfidence`, `u_audioIsLocked` (`bool`), `u_audioBrightness`, `u_audioFlatness`.\n\n**Audio textures (main source only)**: `u_audioFFT` (`sampler2D`, 1024 FFT bins 0..255), `u_audioWaveform` (`sampler2D`, time-domain -1..1).\n\n**Video**: `u_video` (`sampler2D`), `u_videoAnalysed` (`sampler2D`, paired with CV uniforms), `u_videoAnalysedAvailable` (`bool`), `u_videoResolution` (`vec2`), `u_videoFrameRate` (`float`), `u_videoConnected` (`bool`). Use the `vijiVideoUV` helper below for aspect handling.\n\n**CV face**: `u_faceCount` (`int`), `u_face0Bounds` (`vec4`, normalized 0..1), `u_face0Center` (`vec2`, normalized), `u_face0HeadPose` (`vec3` degrees), `u_face0Confidence` (`float`), `u_face0Neutral`-`u_face0Fearful` (7 expressions 0..1), 52 blendshape uniforms prefixed `u_face0` (e.g. `u_face0JawOpen`, `u_face0EyeBlinkLeft`, `u_face0MouthSmileRight`). Activated via `// @viji-cv:faceDetection` / `// @viji-cv:faceMesh` / `// @viji-cv:emotionDetection`.\n\n**CV hands**: `u_handCount` (`int`), `u_leftHandPalm`/`u_rightHandPalm` (`vec3`), `u_leftHandConfidence`/`u_rightHandConfidence`, `u_leftHandBounds`/`u_rightHandBounds` (`vec4`), 7 gesture uniforms per hand (`Fist`, `OpenPalm`, `Peace`, `ThumbsUp`, `ThumbsDown`, `Pointing`, `ILoveYou`). Activated via `// @viji-cv:handTracking`.\n\n**CV pose**: `u_poseDetected` (`bool`), `u_poseConfidence`, joint landmarks (`u_nosePosition`, `u_leftShoulderPosition`-`u_rightAnklePosition`, all `vec2` normalized 0..1). Activated via `// @viji-cv:poseDetection`.\n\n**CV segmentation**: `u_segmentationMask` (`sampler2D`, 0 = background, 1 = person), `u_segmentationRes` (`vec2`). Activated via `// @viji-cv:bodySegmentation`.\n\n**Device sensors**: `u_deviceAcceleration` (`vec3`, m/s² without gravity), `u_deviceAccelerationGravity`, `u_deviceRotationRate` (deg/s), `u_deviceOrientation` (`vec3`, degrees), `u_deviceOrientationAbsolute` (`bool`).\n\n**External devices**: `u_deviceCount` (`int`, 0..8), `u_device0`-`u_device7` (`sampler2D` camera textures), `u_device0Resolution`-`u_device7Resolution` (`vec2`), `u_device0Connected`-`u_device7Connected` (`bool`), plus per-device acceleration / rotation / orientation `vec3` uniforms.\n\n**Compositor streams**: `u_videoStreamCount`, `u_videoStream0`-`u_videoStream7` (`sampler2D`), with `*Resolution` and `*Connected` per slot. `u_audioStreamCount`, `u_audioStream0Connected`-`u_audioStream7Connected` (`bool`), `u_audioStream{i}Volume`, `u_audioStream{i}Low`-`u_audioStream{i}High`, `u_audioStream{i}Brightness`, `u_audioStream{i}Flatness` (`float` 0..1). Scalars only on streams: no per-stream FFT / waveform. Beat / BPM / trigger uniforms are main audio only.\n\n**Backbuffer**: `backbuffer` (`sampler2D`, no `u_` prefix). Auto-enabled when referenced.\n\n## Video aspect helper\n\nCamera frames almost never match the canvas aspect. Sampling `texture2D(u_video, uv)` with canvas UV stretches the video and misaligns CV uniforms. Use these at the top of any video / CV shader:\n\n```glsl\n// mode: 1 = cover (fills canvas, video edges cropped)\n// 0 = contain (fits video, canvas letterboxed)\nvec2 vijiVideoUV(vec2 uv, int mode) {\n vec2 canvas = u_resolution;\n vec2 video = u_videoResolution;\n if (video.x == 0.0 || video.y == 0.0) return uv;\n float canvasAspect = canvas.x / canvas.y;\n float videoAspect = video.x / video.y;\n vec2 scale = vec2(1.0);\n if (mode == 1) {\n scale = canvasAspect > videoAspect\n ? vec2(1.0, canvasAspect / videoAspect)\n : vec2(videoAspect / canvasAspect, 1.0);\n } else {\n scale = canvasAspect > videoAspect\n ? vec2(videoAspect / canvasAspect, 1.0)\n : vec2(1.0, canvasAspect / videoAspect);\n }\n return (uv - 0.5) / scale + 0.5;\n}\n\nbool vijiInVideo(vec2 uv) {\n return all(greaterThanEqual(uv, vec2(0.0))) && all(lessThanEqual(uv, vec2(1.0)));\n}\n```\n\n## Worked example\n\nSource Shadertoy shader:\n```glsl\nvoid mainImage(out vec4 fragColor, in vec2 fragCoord) {\n vec2 uv = fragCoord / iResolution.xy;\n float t = iTime * 0.5;\n float wave = sin(uv.x * 10.0 + t) * 0.5 + 0.5;\n vec3 col = vec3(uv, wave);\n fragColor = vec4(col, 1.0);\n}\n```\n\nConverted Viji shader:\n```glsl\n// @renderer shader\n// @viji-slider:speed label:\"Speed\" default:0.5 min:0.1 max:3.0\n// @viji-color:tint label:\"Tint\" default:#ffffff\n// @viji-accumulator:phase rate:speed\n\nvoid main() {\n vec2 uv = gl_FragCoord.xy / u_resolution;\n float wave = sin(uv.x * 10.0 + phase) * 0.5 + 0.5;\n vec3 col = vec3(uv, wave) * tint;\n gl_FragColor = vec4(col, 1.0);\n}\n```\n\nKey changes: removed the `mainImage(out vec4 fragColor, in vec2 fragCoord)` wrapper and used `void main()` with `gl_FragCoord.xy`; replaced `iResolution.xy` with `u_resolution`; replaced `iTime * 0.5` with a `@viji-accumulator` keyed to a `speed` slider (so changing speed never jumps the animation); added a `@viji-color` for the tint so the artist can adjust it; preserved the visual intent (UV gradient + horizontal wave).\n",
|
|
9814
|
-
"threejs": "# Converting Three.js Applications to Viji Native\n\nThis guide is loaded by the Viji-Backend AI when the user wants to convert a standalone Three.js application into a Viji scene. The target renderer is Viji **Native** (Three.js loads as an ESM dynamic import). This guide is self-contained: it includes both the source-to-target mapping and the Viji Native target reference needed to produce correct converted output, without requiring the Viji Native system prompt to also be loaded.\n\n## Source platform: standalone Three.js\n\nStandalone Three.js apps typically:\n- Create their own `<canvas>` element or accept one from the DOM.\n- Construct a `THREE.Scene`, `THREE.PerspectiveCamera`, and `THREE.WebGLRenderer`.\n- Use `requestAnimationFrame()` as the per-frame loop.\n- Use `THREE.Clock` / `clock.getDelta()` for timing.\n- Attach `window.addEventListener('resize', ...)` and use `window.innerWidth` / `window.innerHeight`.\n- Listen to DOM mouse / keyboard events, or use `OrbitControls`.\n- Load assets via `THREE.TextureLoader`, `GLTFLoader`, etc.\n\n## Target platform: Viji Native + Three.js\n\nViji Native scenes run on an `OffscreenCanvas` inside a Web Worker. The DOM is unavailable. Three.js loads as an ESM dynamic import. The canvas is created by Viji and passed to the Three.js renderer. Viji controls the render loop; the artist's `render(viji)` function runs each frame.\n\n## Conversion rules\n\n1. **Always** import Three.js dynamically at the top level with a pinned version:\n ```javascript\n const THREE = await import('https://esm.sh/three@0.160.0');\n ```\n Never use `<script>` tags, `require()`, or static `import` statements.\n\n2. **Always** use `viji.canvas` as the renderer's canvas:\n ```javascript\n const renderer = new THREE.WebGLRenderer({ canvas: viji.canvas, antialias: true });\n renderer.setSize(viji.width, viji.height, false);\n ```\n **Always** pass `false` as the third argument to `setSize()`. The Worker has no DOM, so any attempt to update CSS styles (Three.js's default) will throw.\n\n3. **Never** use `requestAnimationFrame()`. Viji controls the render loop. Put all per-frame logic inside `function render(viji) { ... }` and call `renderer.render(scene, camera)` at the end.\n\n4. **Always** handle resize by checking `viji.width` / `viji.height` inside `render()`:\n ```javascript\n let prevWidth = viji.width;\n let prevHeight = viji.height;\n\n function render(viji) {\n if (viji.width !== prevWidth || viji.height !== prevHeight) {\n renderer.setSize(viji.width, viji.height, false);\n camera.aspect = viji.width / viji.height;\n camera.updateProjectionMatrix();\n prevWidth = viji.width;\n prevHeight = viji.height;\n }\n // ...\n renderer.render(scene, camera);\n }\n ```\n Remove any `window.addEventListener('resize', ...)`. Resize is handled here.\n\n5. **Never** access `window`, `document`, `Image()`, or `localStorage`. `fetch()` and `await import()` are available.\n\n6. **Never** use `window.innerWidth` / `window.innerHeight`. Use `viji.width` / `viji.height`.\n\n7. **Lift hardcoded values to Viji parameters** at the top level:\n ```javascript\n const speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\n const meshColor = viji.color('#049ef4', { label: 'Color' });\n ```\n Read via `.value` inside `render()`. Three.js colors accept the hex string: `material.color.set(meshColor.value)`. For more control, color parameters also expose `.rgb` (0..255) and `.hsb`.\n\n8. **Always** use `viji.deltaTime` for animation timing:\n ```javascript\n cube.rotation.y += speed.value * viji.deltaTime;\n ```\n Remove `THREE.Clock` and `clock.getDelta()`. Never use `Date.now()` or `performance.now()` directly.\n\n **Never** multiply `viji.time` by a parameter; it causes visible jumps when the parameter changes. Same rule for nested multiplications: each independent speed needs its own accumulator at the top level.\n\n9. **Replace mouse / keyboard event listeners with Viji input APIs**:\n - `event.clientX` -> `viji.pointer.x` (works for both mouse and touch), or `viji.mouse.x`.\n - `event.clientY` -> `viji.pointer.y`, or `viji.mouse.y`.\n - Mouse buttons -> `viji.mouse.leftButton`, `viji.mouse.rightButton`, `viji.mouse.middleButton`.\n - Key presses -> `viji.keyboard.isPressed('keyName')`, `viji.keyboard.wasPressed('keyName')` for one-frame edges.\n\n10. **`OrbitControls` and other DOM-event-based controls do not work** in the Worker. For camera interaction, read `viji.pointer` and `viji.mouse.wheelDelta` directly and update the camera manually. (You can typically build a working orbit with: pointer delta when down -> yaw / pitch; wheel delta -> radius.)\n\n11. **Three.js addons** import from the examples directory with the same pinned version:\n ```javascript\n const { GLTFLoader } = await import('https://esm.sh/three@0.160.0/examples/jsm/loaders/GLTFLoader.js');\n const { EffectComposer } = await import('https://esm.sh/three@0.160.0/examples/jsm/postprocessing/EffectComposer.js');\n ```\n Always pin the same version for addons as for the main library.\n\n12. **File textures** become `viji.image` parameters:\n ```javascript\n const photo = viji.image(null, { label: 'Texture' });\n let texture = null;\n\n function render(viji) {\n if (photo.value && !texture) {\n texture = new THREE.CanvasTexture(photo.value);\n material.map = texture;\n material.needsUpdate = true;\n }\n // ...\n }\n ```\n\n13. **Video textures** use `viji.video`:\n ```javascript\n let videoTexture = null;\n\n function render(viji) {\n if (viji.video.isConnected && viji.video.currentFrame) {\n if (!videoTexture) {\n videoTexture = new THREE.CanvasTexture(viji.video.currentFrame);\n material.map = videoTexture;\n }\n videoTexture.needsUpdate = true;\n }\n // ...\n }\n ```\n\n14. **Never** allocate new objects (vectors, colors, materials, geometries) inside `render()`. Pre-create at the top level and mutate in place. Three.js reuses `Vector3` and `Color` mutably; this is the idiomatic pattern.\n\n15. **Always** set `category` on parameters that depend on an external input: `category: 'audio'`, `category: 'video'`, `category: 'interaction'`.\n\n16. **Remove** any CSS, HTML, or DOM manipulation code. Viji scenes produce canvas output only.\n\n17. If the source uses a framework on top of Three.js (React Three Fiber, Drei, Theatre.js): flag this to the artist. These cannot be converted directly; they would need to be rewritten as plain Three.js first.\n\n## Target API reference (Viji Native + Three.js patterns)\n\n**Canvas and context**: `viji.canvas` (`OffscreenCanvas`: pass to `new THREE.WebGLRenderer({ canvas: viji.canvas })`), `viji.width`, `viji.height`. `viji.useContext()` is available for non-Three.js Native scenes, but with Three.js you let it manage the GL context.\n\n**Timing**: `viji.time` (seconds since scene start), `viji.deltaTime` (seconds since last frame), `viji.frameCount`, `viji.fps`.\n\n**Parameters** (top-level only, read via `.value`):\n```javascript\nviji.slider(default, { min?, max?, step?, label, group?, category? }) // .value: number\nviji.color(default, { label, group?, category? }) // .value, .rgb, .hsb\nviji.toggle(default, { label, group?, category? }) // .value: boolean\nviji.select(default, { options, label, group?, category? }) // .value: string | number\nviji.number(default, { min?, max?, step?, label, group?, category? }) // .value: number\nviji.text(default, { label, group?, category?, maxLength? }) // .value: string\nviji.image(null, { label, group?, category? }) // .value: ImageBitmap | null\nviji.button({ label, description?, group?, category? }) // .value: boolean (1 frame)\nviji.coordinate(default, { step?, label, group?, category? }) // .value: { x, y }, range -1..1\n```\n\n**Audio: `viji.audio`**: check `isConnected` first. Members: `volume.{current, peak, smoothed}` (0..1; `smoothed` 200ms decay); `bands.{low, lowMid, mid, highMid, high}` (instant 0..1) + each `*Smoothed` sibling (150ms decay); `beat.{kick, snare, hat, any}` (300ms decay curves) + each `*Smoothed` (500ms); `beat.triggers.{kick, snare, hat, any}` (boolean, true for one frame, OR-accumulated); `beat.events: Array<{type, time, strength}>` (`type` is `'kick' | 'snare' | 'hat'`); `beat.bpm` (`0` when no audio is connected; once audio connects it tracks the detected tempo clamped to 60..240, with `120` as a fallback before lock-on), `beat.confidence` (0..1), `beat.isLocked`; `spectral.{brightness, flatness}`; `getFrequencyData(): Uint8Array` (1024 bins 0..255); `getWaveform(): Float32Array` (2048 samples -1..1). Apply audio reactivity to Three.js objects via `material.uniforms` (for `ShaderMaterial`), `material.emissiveIntensity`, mesh `scale`, `rotation`, position, etc.\n\n**Video: `viji.video`**: check `isConnected && currentFrame` first. `currentFrame` (`OffscreenCanvas | ImageBitmap`), `frameWidth`, `frameHeight`, `frameRate`, `getFrameData()`, `cv: VideoCVAPI`. CV outputs (`analysedFrame`, `getAnalysedFrameData`, `faces`, `hands`, `pose`, `segmentation`) and CV verbs all live on `viji.video.cv`, not on `viji.video` directly. Wrap `viji.video.currentFrame` in `THREE.CanvasTexture(viji.video.currentFrame)` and set `texture.needsUpdate = true` each frame the frame changes.\n\n**Computer Vision: `viji.video.cv`**: enable explicitly (never by default). All CV state lives on `viji.video.cv`:\n```javascript\nawait viji.video.cv.enableFaceDetection(true);\nawait viji.video.cv.enableFaceMesh(true); // implies face detection\nawait viji.video.cv.enableEmotionDetection(true); // implies face mesh\nawait viji.video.cv.enableHandTracking(true);\nawait viji.video.cv.enablePoseDetection(true);\nawait viji.video.cv.enableBodySegmentation(true);\n```\nAlso on `viji.video.cv`: `analysedFrame: OffscreenCanvas | null` (paired with the current CV results; `null` until first inference), `getAnalysedFrameData(): ImageData | null`, `getActiveFeatures()`, `isProcessing()`. Common fallback pattern when texturing from a CV-paired frame: `viji.video.cv.analysedFrame ?? viji.video.currentFrame`.\n\nData shapes on `viji.video.cv`: `faces: FaceData[]` (`{id, bounds, center, confidence, landmarks, expressions, headPose, blendshapes (52 ARKit coefficients)}`), `hands: HandData[]` (`{id, handedness, confidence, bounds, landmarks (21 pts), palm, gestures}`), `pose: PoseData | null` (`{confidence, landmarks (33 pts), face, torso, leftArm, rightArm, leftLeg, rightLeg}`), `segmentation: SegmentationData | null` (`{mask: Uint8Array (byte values 0 or 1), width, height}`).\n\n**Input**:\n- `viji.pointer`: `x`, `y`, `deltaX`, `deltaY`, `isDown`, `wasPressed`, `wasReleased`, `isInCanvas`, `type` (`'mouse' | 'touch' | 'none'`). Unified across mouse and touch: preferred for OrbitControls replacements.\n- `viji.mouse`: `x`, `y`, `isInCanvas`, `isPressed`, `leftButton`, `rightButton`, `middleButton`, `deltaX`, `deltaY`, `wheelDelta`, `wheelX`, `wheelY`, `wasPressed`, `wasReleased`, `wasMoved`.\n- `viji.keyboard`: `isPressed(key)`, `wasPressed(key)`, `wasReleased(key)`, `activeKeys`, `lastKeyPressed`, `lastKeyReleased`, `shift`, `ctrl`, `alt`, `meta`.\n- `viji.touches`: `count`, `points`, `started`, `moved`, `ended`, `primary`.\n\n**Sensors / streams**: `viji.device.{motion, orientation}` (for tilt-driven cameras), `viji.devices[]`, `viji.videoStreams[]`, `viji.audioStreams[]`.\n\n## Three.js setup pattern\n\nThe canonical Viji-Native + Three.js scaffold:\n\n```javascript\nconst THREE = await import('https://esm.sh/three@0.160.0');\n\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nconst meshColor = viji.color('#049ef4', { label: 'Color' });\n\nconst renderer = new THREE.WebGLRenderer({ canvas: viji.canvas, antialias: true });\nrenderer.setSize(viji.width, viji.height, false);\nrenderer.setPixelRatio(1);\n\nconst scene = new THREE.Scene();\nconst camera = new THREE.PerspectiveCamera(50, viji.width / viji.height, 0.1, 100);\ncamera.position.set(0, 0, 5);\nscene.add(new THREE.AmbientLight(0xffffff, 0.4));\nconst dir = new THREE.DirectionalLight(0xffffff, 0.9);\ndir.position.set(2, 3, 4);\nscene.add(dir);\n\nconst geometry = new THREE.BoxGeometry(1.5, 1.5, 1.5);\nconst material = new THREE.MeshStandardMaterial({ color: meshColor.value, roughness: 0.5 });\nconst cube = new THREE.Mesh(geometry, material);\nscene.add(cube);\n\nlet prevWidth = viji.width;\nlet prevHeight = viji.height;\n\nfunction render(viji) {\n if (viji.width !== prevWidth || viji.height !== prevHeight) {\n renderer.setSize(viji.width, viji.height, false);\n camera.aspect = viji.width / viji.height;\n camera.updateProjectionMatrix();\n prevWidth = viji.width;\n prevHeight = viji.height;\n }\n\n material.color.set(meshColor.value);\n cube.rotation.x += speed.value * viji.deltaTime;\n cube.rotation.y += speed.value * viji.deltaTime * 0.7;\n\n renderer.render(scene, camera);\n}\n```\n\n## Worked example\n\nSource Three.js app:\n```javascript\nimport * as THREE from 'three';\nimport { OrbitControls } from 'three/examples/jsm/controls/OrbitControls.js';\n\nconst renderer = new THREE.WebGLRenderer({ antialias: true });\nrenderer.setSize(window.innerWidth, window.innerHeight);\ndocument.body.appendChild(renderer.domElement);\n\nconst scene = new THREE.Scene();\nconst camera = new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 0.1, 1000);\ncamera.position.z = 5;\n\nconst controls = new OrbitControls(camera, renderer.domElement);\n\nconst cube = new THREE.Mesh(\n new THREE.BoxGeometry(),\n new THREE.MeshNormalMaterial()\n);\nscene.add(cube);\n\nconst clock = new THREE.Clock();\nfunction animate() {\n requestAnimationFrame(animate);\n const dt = clock.getDelta();\n cube.rotation.x += dt;\n cube.rotation.y += dt;\n controls.update();\n renderer.render(scene, camera);\n}\nanimate();\n\nwindow.addEventListener('resize', () => {\n renderer.setSize(window.innerWidth, window.innerHeight);\n camera.aspect = window.innerWidth / window.innerHeight;\n camera.updateProjectionMatrix();\n});\n```\n\nConverted Viji Native scene:\n```javascript\nconst THREE = await import('https://esm.sh/three@0.160.0');\n\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Rotation Speed' });\nconst orbitSensitivity = viji.slider(0.005, { min: 0.001, max: 0.02, step: 0.001, label: 'Orbit Sensitivity', category: 'interaction' });\n\nconst renderer = new THREE.WebGLRenderer({ canvas: viji.canvas, antialias: true });\nrenderer.setSize(viji.width, viji.height, false);\n\nconst scene = new THREE.Scene();\nconst camera = new THREE.PerspectiveCamera(75, viji.width / viji.height, 0.1, 1000);\n\nlet cameraYaw = 0;\nlet cameraPitch = 0;\nlet cameraRadius = 5;\n\nconst cube = new THREE.Mesh(\n new THREE.BoxGeometry(),\n new THREE.MeshNormalMaterial()\n);\nscene.add(cube);\n\nlet prevWidth = viji.width;\nlet prevHeight = viji.height;\n\nfunction render(viji) {\n if (viji.width !== prevWidth || viji.height !== prevHeight) {\n renderer.setSize(viji.width, viji.height, false);\n camera.aspect = viji.width / viji.height;\n camera.updateProjectionMatrix();\n prevWidth = viji.width;\n prevHeight = viji.height;\n }\n\n // Manual orbit replacement (OrbitControls cannot run in the worker).\n if (viji.pointer.isDown) {\n cameraYaw -= viji.pointer.deltaX * orbitSensitivity.value;\n cameraPitch = Math.max(-Math.PI / 2 + 0.01, Math.min(Math.PI / 2 - 0.01,\n cameraPitch - viji.pointer.deltaY * orbitSensitivity.value));\n }\n cameraRadius = Math.max(1.5, Math.min(20, cameraRadius - viji.mouse.wheelDelta * 0.002));\n camera.position.set(\n cameraRadius * Math.cos(cameraPitch) * Math.sin(cameraYaw),\n cameraRadius * Math.sin(cameraPitch),\n cameraRadius * Math.cos(cameraPitch) * Math.cos(cameraYaw)\n );\n camera.lookAt(0, 0, 0);\n\n cube.rotation.x += speed.value * viji.deltaTime;\n cube.rotation.y += speed.value * viji.deltaTime;\n\n renderer.render(scene, camera);\n}\n```\n\nKey changes: replaced static `import` with dynamic `await import('https://esm.sh/three@0.160.0')` pinned to a version; passed `viji.canvas` to the renderer and added the mandatory `false` third argument to `setSize`; removed `document.body.appendChild`; removed `requestAnimationFrame` (Viji owns the loop) and moved per-frame logic into `render(viji)`; replaced `THREE.Clock` / `clock.getDelta()` with `viji.deltaTime`; replaced the `OrbitControls` instance with a manual orbit driven by `viji.pointer.deltaX/Y` and `viji.mouse.wheelDelta` (with an `orbitSensitivity` slider tagged `category: 'interaction'`); moved resize logic from a `window.addEventListener('resize', ...)` callback into a `prevWidth`/`prevHeight` check inside `render`; lifted the implicit rotation speed to a `speed` slider so the artist can adjust it.\n"
|
|
9849
|
+
"threejs": "# Converting Three.js Applications to Viji Native\n\nThis guide is loaded by the Viji-Backend AI when the user wants to convert a standalone Three.js application into a Viji scene. The target renderer is Viji **Native** (Three.js loads as an ESM dynamic import). This guide is self-contained: it includes both the source-to-target mapping and the Viji Native target reference needed to produce correct converted output, without requiring the Viji Native system prompt to also be loaded.\n\n## Source platform: standalone Three.js\n\nStandalone Three.js apps typically:\n- Create their own `<canvas>` element or accept one from the DOM.\n- Construct a `THREE.Scene`, `THREE.PerspectiveCamera`, and `THREE.WebGLRenderer`.\n- Use `requestAnimationFrame()` as the per-frame loop.\n- Use `THREE.Clock` / `clock.getDelta()` for timing.\n- Attach `window.addEventListener('resize', ...)` and use `window.innerWidth` / `window.innerHeight`.\n- Listen to DOM mouse / keyboard events, or use `OrbitControls`.\n- Load assets via `THREE.TextureLoader`, `GLTFLoader`, etc.\n\n## Target platform: Viji Native + Three.js\n\nViji Native scenes run on an `OffscreenCanvas` inside a Web Worker. The DOM is unavailable. Three.js loads as an ESM dynamic import. The canvas is created by Viji and passed to the Three.js renderer. Viji controls the render loop; the artist's `render(viji)` function runs each frame.\n\n## Conversion rules\n\n1. **Always** import Three.js dynamically at the top level with a pinned version:\n ```javascript\n const THREE = await import('https://esm.sh/three@0.160.0');\n ```\n Never use `<script>` tags, `require()`, or static `import` statements.\n\n2. **Always** use `viji.canvas` as the renderer's canvas:\n ```javascript\n const renderer = new THREE.WebGLRenderer({ canvas: viji.canvas, antialias: true });\n renderer.setSize(viji.width, viji.height, false);\n ```\n **Always** pass `false` as the third argument to `setSize()`. The Worker has no DOM, so any attempt to update CSS styles (Three.js's default) will throw.\n\n3. **Never** use `requestAnimationFrame()`. Viji controls the render loop. Put all per-frame logic inside `function render(viji) { ... }` and call `renderer.render(scene, camera)` at the end.\n\n4. **Always** handle resize by checking `viji.width` / `viji.height` inside `render()`:\n ```javascript\n let prevWidth = viji.width;\n let prevHeight = viji.height;\n\n function render(viji) {\n if (viji.width !== prevWidth || viji.height !== prevHeight) {\n renderer.setSize(viji.width, viji.height, false);\n camera.aspect = viji.width / viji.height;\n camera.updateProjectionMatrix();\n prevWidth = viji.width;\n prevHeight = viji.height;\n }\n // ...\n renderer.render(scene, camera);\n }\n ```\n Remove any `window.addEventListener('resize', ...)`. Resize is handled here.\n\n5. **Never** access `window`, `document`, `Image()`, or `localStorage`. `fetch()` and `await import()` are available.\n\n6. **Never** use `window.innerWidth` / `window.innerHeight`. Use `viji.width` / `viji.height`.\n\n7. **Lift hardcoded values to Viji parameters** at the top level:\n ```javascript\n const speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\n const meshColor = viji.color('#049ef4', { label: 'Color' });\n ```\n Read via `.value` inside `render()`. Three.js colors accept the hex string: `material.color.set(meshColor.value)`. For more control, color parameters also expose `.rgb` (0..255) and `.hsb`.\n\n8. **Always** use `viji.deltaTime` for animation timing:\n ```javascript\n cube.rotation.y += speed.value * viji.deltaTime;\n ```\n Remove `THREE.Clock` and `clock.getDelta()`. Never use `Date.now()` or `performance.now()` directly.\n\n **Never** multiply `viji.time` by a parameter; it causes visible jumps when the parameter changes. Same rule for nested multiplications: each independent speed needs its own accumulator at the top level.\n\n9. **Replace mouse / keyboard event listeners with Viji input APIs**:\n - `event.clientX` -> `viji.pointer.x` (works for both mouse and touch), or `viji.mouse.x`.\n - `event.clientY` -> `viji.pointer.y`, or `viji.mouse.y`.\n - Mouse buttons -> `viji.mouse.leftButton`, `viji.mouse.rightButton`, `viji.mouse.middleButton`.\n - Key presses -> `viji.keyboard.isPressed('keyName')`, `viji.keyboard.wasPressed('keyName')` for one-frame edges.\n\n10. **`OrbitControls` and other DOM-event-based controls do not work** in the Worker. For camera interaction, read `viji.pointer` and `viji.mouse.wheelDelta` directly and update the camera manually. (You can typically build a working orbit with: pointer delta when down -> yaw / pitch; wheel delta -> radius.)\n\n11. **Three.js addons** import from the examples directory with the same pinned version:\n ```javascript\n const { GLTFLoader } = await import('https://esm.sh/three@0.160.0/examples/jsm/loaders/GLTFLoader.js');\n const { EffectComposer } = await import('https://esm.sh/three@0.160.0/examples/jsm/postprocessing/EffectComposer.js');\n ```\n Always pin the same version for addons as for the main library.\n\n12. **File textures** become `viji.image` parameters:\n ```javascript\n const photo = viji.image(null, { label: 'Texture' });\n let texture = null;\n\n function render(viji) {\n if (photo.value && !texture) {\n texture = new THREE.CanvasTexture(photo.value);\n material.map = texture;\n material.needsUpdate = true;\n }\n // ...\n }\n ```\n\n13. **Video textures** use `viji.video`:\n ```javascript\n let videoTexture = null;\n\n function render(viji) {\n if (viji.video.isConnected && viji.video.currentFrame) {\n if (!videoTexture) {\n videoTexture = new THREE.CanvasTexture(viji.video.currentFrame);\n material.map = videoTexture;\n }\n videoTexture.needsUpdate = true;\n }\n // ...\n }\n ```\n\n14. **Never** allocate new objects (vectors, colors, materials, geometries) inside `render()`. Pre-create at the top level and mutate in place. Three.js reuses `Vector3` and `Color` mutably; this is the idiomatic pattern.\n\n15. **Always** set `category` on parameters that depend on an external input: `category: 'audio'`, `category: 'video'`, `category: 'interaction'`.\n\n16. **Remove** any CSS, HTML, or DOM manipulation code. Viji scenes produce canvas output only.\n\n17. If the source uses a framework on top of Three.js (React Three Fiber, Drei, Theatre.js): flag this to the artist. These cannot be converted directly; they would need to be rewritten as plain Three.js first.\n\n## Target API reference (Viji Native + Three.js patterns)\n\n**Canvas and context**: `viji.canvas` (`OffscreenCanvas`: pass to `new THREE.WebGLRenderer({ canvas: viji.canvas })`), `viji.width`, `viji.height`. `viji.useContext()` is available for non-Three.js Native scenes, but with Three.js you let it manage the GL context.\n\n**Timing**: `viji.time` (seconds since scene start), `viji.deltaTime` (seconds since last frame), `viji.frameCount`, `viji.fps`.\n\n**Parameters** (top-level only, read via `.value`):\n```javascript\nviji.slider(default, { min?, max?, step?, label, group?, category? }) // .value: number\nviji.color(default, { label, group?, category? }) // .value, .rgb, .hsb\nviji.toggle(default, { label, group?, category? }) // .value: boolean\nviji.select(default, { options, label, group?, category? }) // .value: string | number\nviji.number(default, { min?, max?, step?, label, group?, category? }) // .value: number\nviji.text(default, { label, group?, category?, maxLength? }) // .value: string\nviji.image(null, { label, group?, category? }) // .value: ImageBitmap | null\nviji.button({ label, description?, group?, category? }) // .value: boolean (1 frame)\nviji.coordinate(default, { step?, label, group?, category? }) // .value: { x, y }, range -1..1\n```\n\n**Audio: `viji.audio`**: check `isConnected` first. Members: `volume.{current, peak, smoothed}` (0..1; `smoothed` 200ms decay); `bands.{low, lowMid, mid, highMid, high}` (instant 0..1) + each `*Smoothed` sibling (150ms decay); `beat.{kick, snare, hat, any}` (300ms decay curves) + each `*Smoothed` (500ms); `beat.triggers.{kick, snare, hat, any}` (boolean, true for one frame, OR-accumulated); `beat.events: Array<{type, time, strength}>` (`type` is `'kick' | 'snare' | 'hat'`); `beat.bpm` (`0` when no audio is connected; once audio connects it tracks the detected tempo clamped to 60..240, with `120` as a fallback before lock-on), `beat.confidence` (0..1), `beat.isLocked`; `spectral.{brightness, flatness}`; `getFrequencyData(): Uint8Array` (1024 bins 0..255); `getWaveform(): Float32Array` (2048 samples -1..1). Apply audio reactivity to Three.js objects via `material.uniforms` (for `ShaderMaterial`), `material.emissiveIntensity`, mesh `scale`, `rotation`, position, etc.\n\n**Video: `viji.video`**: check `isConnected && currentFrame` first. `currentFrame` (`OffscreenCanvas | ImageBitmap`), `frameWidth`, `frameHeight`, `frameRate`, `getFrameData()`, `cv: VideoCVAPI`. CV outputs (`analysedFrame`, `getAnalysedFrameData`, `faces`, `hands`, `pose`, `segmentation`) and CV verbs all live on `viji.video.cv`, not on `viji.video` directly. Wrap `viji.video.currentFrame` in `THREE.CanvasTexture(viji.video.currentFrame)` and set `texture.needsUpdate = true` each frame the frame changes.\n\n**Computer Vision: `viji.video.cv`**: enable explicitly (never by default). All CV state lives on `viji.video.cv`:\n```javascript\nawait viji.video.cv.enableFaceDetection(true);\nawait viji.video.cv.enableFaceMesh(true); // populates face.landmarks + face.headPose\nawait viji.video.cv.enableEmotionDetection(true); // populates face.blendshapes + face.expressions; loads landmarker\nawait viji.video.cv.enableHandTracking(true);\nawait viji.video.cv.enablePoseDetection(true);\nawait viji.video.cv.enableBodySegmentation(true);\n```\nAlso on `viji.video.cv`: `analysedFrame: OffscreenCanvas | null` (paired with the current CV results; `null` until first inference), `getAnalysedFrameData(): ImageData | null`, `getActiveFeatures()`, `isProcessing()`. Common fallback pattern when texturing from a CV-paired frame: `viji.video.cv.analysedFrame ?? viji.video.currentFrame`.\n\nData shapes on `viji.video.cv`: `faces: FaceData[]` (`{id, bounds, center, confidence, landmarks, expressions, headPose, blendshapes (52 ARKit coefficients)}`), `hands: HandData[]` (`{id, handedness, confidence, bounds, landmarks (21 pts), palm, gestures}`), `pose: PoseData | null` (`{confidence, landmarks (33 pts), face, torso, leftArm, rightArm, leftLeg, rightLeg}`), `segmentation: SegmentationData | null` (`{mask: Uint8Array (byte values 0 or 1), width, height}`).\n\n**Input**:\n- `viji.pointer`: `x`, `y`, `deltaX`, `deltaY`, `isDown`, `wasPressed`, `wasReleased`, `isInCanvas`, `type` (`'mouse' | 'touch' | 'none'`). Unified across mouse and touch: preferred for OrbitControls replacements.\n- `viji.mouse`: `x`, `y`, `isInCanvas`, `isPressed`, `leftButton`, `rightButton`, `middleButton`, `deltaX`, `deltaY`, `wheelDelta`, `wheelX`, `wheelY`, `wasPressed`, `wasReleased`, `wasMoved`.\n- `viji.keyboard`: `isPressed(key)`, `wasPressed(key)`, `wasReleased(key)`, `activeKeys`, `lastKeyPressed`, `lastKeyReleased`, `shift`, `ctrl`, `alt`, `meta`.\n- `viji.touches`: `count`, `points`, `started`, `moved`, `ended`, `primary`.\n\n**Sensors / streams**: `viji.device.{motion, orientation}` (for tilt-driven cameras), `viji.devices[]`, `viji.videoStreams[]`, `viji.audioStreams[]`.\n\n## Three.js setup pattern\n\nThe canonical Viji-Native + Three.js scaffold:\n\n```javascript\nconst THREE = await import('https://esm.sh/three@0.160.0');\n\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nconst meshColor = viji.color('#049ef4', { label: 'Color' });\n\nconst renderer = new THREE.WebGLRenderer({ canvas: viji.canvas, antialias: true });\nrenderer.setSize(viji.width, viji.height, false);\nrenderer.setPixelRatio(1);\n\nconst scene = new THREE.Scene();\nconst camera = new THREE.PerspectiveCamera(50, viji.width / viji.height, 0.1, 100);\ncamera.position.set(0, 0, 5);\nscene.add(new THREE.AmbientLight(0xffffff, 0.4));\nconst dir = new THREE.DirectionalLight(0xffffff, 0.9);\ndir.position.set(2, 3, 4);\nscene.add(dir);\n\nconst geometry = new THREE.BoxGeometry(1.5, 1.5, 1.5);\nconst material = new THREE.MeshStandardMaterial({ color: meshColor.value, roughness: 0.5 });\nconst cube = new THREE.Mesh(geometry, material);\nscene.add(cube);\n\nlet prevWidth = viji.width;\nlet prevHeight = viji.height;\n\nfunction render(viji) {\n if (viji.width !== prevWidth || viji.height !== prevHeight) {\n renderer.setSize(viji.width, viji.height, false);\n camera.aspect = viji.width / viji.height;\n camera.updateProjectionMatrix();\n prevWidth = viji.width;\n prevHeight = viji.height;\n }\n\n material.color.set(meshColor.value);\n cube.rotation.x += speed.value * viji.deltaTime;\n cube.rotation.y += speed.value * viji.deltaTime * 0.7;\n\n renderer.render(scene, camera);\n}\n```\n\n## Worked example\n\nSource Three.js app:\n```javascript\nimport * as THREE from 'three';\nimport { OrbitControls } from 'three/examples/jsm/controls/OrbitControls.js';\n\nconst renderer = new THREE.WebGLRenderer({ antialias: true });\nrenderer.setSize(window.innerWidth, window.innerHeight);\ndocument.body.appendChild(renderer.domElement);\n\nconst scene = new THREE.Scene();\nconst camera = new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 0.1, 1000);\ncamera.position.z = 5;\n\nconst controls = new OrbitControls(camera, renderer.domElement);\n\nconst cube = new THREE.Mesh(\n new THREE.BoxGeometry(),\n new THREE.MeshNormalMaterial()\n);\nscene.add(cube);\n\nconst clock = new THREE.Clock();\nfunction animate() {\n requestAnimationFrame(animate);\n const dt = clock.getDelta();\n cube.rotation.x += dt;\n cube.rotation.y += dt;\n controls.update();\n renderer.render(scene, camera);\n}\nanimate();\n\nwindow.addEventListener('resize', () => {\n renderer.setSize(window.innerWidth, window.innerHeight);\n camera.aspect = window.innerWidth / window.innerHeight;\n camera.updateProjectionMatrix();\n});\n```\n\nConverted Viji Native scene:\n```javascript\nconst THREE = await import('https://esm.sh/three@0.160.0');\n\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Rotation Speed' });\nconst orbitSensitivity = viji.slider(0.005, { min: 0.001, max: 0.02, step: 0.001, label: 'Orbit Sensitivity', category: 'interaction' });\n\nconst renderer = new THREE.WebGLRenderer({ canvas: viji.canvas, antialias: true });\nrenderer.setSize(viji.width, viji.height, false);\n\nconst scene = new THREE.Scene();\nconst camera = new THREE.PerspectiveCamera(75, viji.width / viji.height, 0.1, 1000);\n\nlet cameraYaw = 0;\nlet cameraPitch = 0;\nlet cameraRadius = 5;\n\nconst cube = new THREE.Mesh(\n new THREE.BoxGeometry(),\n new THREE.MeshNormalMaterial()\n);\nscene.add(cube);\n\nlet prevWidth = viji.width;\nlet prevHeight = viji.height;\n\nfunction render(viji) {\n if (viji.width !== prevWidth || viji.height !== prevHeight) {\n renderer.setSize(viji.width, viji.height, false);\n camera.aspect = viji.width / viji.height;\n camera.updateProjectionMatrix();\n prevWidth = viji.width;\n prevHeight = viji.height;\n }\n\n // Manual orbit replacement (OrbitControls cannot run in the worker).\n if (viji.pointer.isDown) {\n cameraYaw -= viji.pointer.deltaX * orbitSensitivity.value;\n cameraPitch = Math.max(-Math.PI / 2 + 0.01, Math.min(Math.PI / 2 - 0.01,\n cameraPitch - viji.pointer.deltaY * orbitSensitivity.value));\n }\n cameraRadius = Math.max(1.5, Math.min(20, cameraRadius - viji.mouse.wheelDelta * 0.002));\n camera.position.set(\n cameraRadius * Math.cos(cameraPitch) * Math.sin(cameraYaw),\n cameraRadius * Math.sin(cameraPitch),\n cameraRadius * Math.cos(cameraPitch) * Math.cos(cameraYaw)\n );\n camera.lookAt(0, 0, 0);\n\n cube.rotation.x += speed.value * viji.deltaTime;\n cube.rotation.y += speed.value * viji.deltaTime;\n\n renderer.render(scene, camera);\n}\n```\n\nKey changes: replaced static `import` with dynamic `await import('https://esm.sh/three@0.160.0')` pinned to a version; passed `viji.canvas` to the renderer and added the mandatory `false` third argument to `setSize`; removed `document.body.appendChild`; removed `requestAnimationFrame` (Viji owns the loop) and moved per-frame logic into `render(viji)`; replaced `THREE.Clock` / `clock.getDelta()` with `viji.deltaTime`; replaced the `OrbitControls` instance with a manual orbit driven by `viji.pointer.deltaX/Y` and `viji.mouse.wheelDelta` (with an `orbitSensitivity` slider tagged `category: 'interaction'`); moved resize logic from a `window.addEventListener('resize', ...)` callback into a `prevWidth`/`prevHeight` check inside `render`; lifted the implicit rotation speed to a `speed` slider so the artist can adjust it.\n"
|
|
9815
9850
|
}
|
|
9816
9851
|
}
|
|
9817
9852
|
};
|