@viji-dev/core 0.13.0 → 0.15.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/artist-dts-p5.js +1 -1
- package/dist/artist-dts.js +1 -1
- package/dist/artist-global-p5.d.ts +3447 -1
- package/dist/artist-global.d.ts +3447 -1
- package/dist/artist-jsdoc.d.ts +2 -0
- package/dist/assets/viji.worker-A8uAea-6.js +608 -0
- package/dist/docs-api.js +44 -19
- package/dist/{essentia-wasm.web-DGWIGRe8.js → essentia-wasm.web-BE_39Nld.js} +1 -1
- package/dist/{index-CkmT_70R.js → index-CuQT5y06.js} +916 -519
- package/dist/index.d.ts +74 -3
- package/dist/index.js +1 -1
- package/dist/{scene-analysis-DNN8jTaH.js → scene-analysis-CEQRlNUL.js} +30 -25
- package/dist/scene-analysis.d.ts +67 -23
- package/dist/scene-analysis.js +2 -2
- package/package.json +117 -116
- package/dist/assets/viji.worker-Clz2HjaH.js +0 -587
package/dist/docs-api.js
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
export const docsApi = {
|
|
2
2
|
"version": "1.1.0",
|
|
3
|
-
"coreVersion": "0.
|
|
4
|
-
"generatedAt": "2026-08-
|
|
3
|
+
"coreVersion": "0.15.0",
|
|
4
|
+
"generatedAt": "2026-08-25T15:25:24.042Z",
|
|
5
5
|
"navigation": [
|
|
6
6
|
{
|
|
7
7
|
"id": "getting-started",
|
|
@@ -890,7 +890,7 @@ export const docsApi = {
|
|
|
890
890
|
},
|
|
891
891
|
"renderers-overview": {
|
|
892
892
|
"id": "renderers-overview",
|
|
893
|
-
"title": "
|
|
893
|
+
"title": "renderers-overview",
|
|
894
894
|
"description": "How to choose and activate each of Viji's three rendering modes, Native, P5.js, and Shader, and what makes each one unique.",
|
|
895
895
|
"headings": [
|
|
896
896
|
{
|
|
@@ -927,7 +927,7 @@ export const docsApi = {
|
|
|
927
927
|
"content": [
|
|
928
928
|
{
|
|
929
929
|
"type": "text",
|
|
930
|
-
"markdown": "# Renderers Overview\n\nViji supports three rendering modes. Each produces visuals on the same canvas and shares the same Artist API for parameters, audio, video, interaction, and sensors. The difference is the language and paradigm you use to draw.\n\n## Choosing a Renderer\n\nYou pick a renderer with a **comment directive** at the top of your scene code:\n\n```javascript\n// @renderer p5 → P5.js renderer (2D default canvas)\n// @renderer p5 webgl → P5.js with WEBGL main canvas (3D / shaders)\n// @renderer shader → Shader renderer\n// (no directive) → Native renderer (default)\n```\n\nIf no `@renderer` directive is present, the scene runs in **Native** mode. You can also write `// @renderer native` for clarity, but it's not required.\n\n> [!IMPORTANT]\n> P5 and shader scenes must declare their renderer type as the first comment. For example:\n> ```\n> // @renderer p5\n> ```\n> For P5's **WEBGL** main canvas, use `// @renderer p5 webgl` as the first line instead of `// @renderer p5`.\n> Shader scenes:\n> ```\n> // @renderer shader\n> ```\n> Without a matching directive, the scene defaults to the native renderer.\n\n---\n\n## Native Renderer\n\nThe native renderer gives you direct access to the canvas. Write standard JavaScript (or TypeScript) and pick any rendering approach you like, whether Canvas 2D, WebGL, or an external library like Three.js.\n\n**Entry point:** `render(viji)`"
|
|
930
|
+
"markdown": "# Renderers Overview\r\n\r\nViji supports three rendering modes. Each produces visuals on the same canvas and shares the same Artist API for parameters, audio, video, interaction, and sensors. The difference is the language and paradigm you use to draw.\r\n\r\n## Choosing a Renderer\r\n\r\nYou pick a renderer with a **comment directive** at the top of your scene code:\r\n\r\n```javascript\r\n// @renderer p5 → P5.js renderer (2D default canvas)\r\n// @renderer p5 webgl → P5.js with WEBGL main canvas (3D / shaders)\r\n// @renderer shader → Shader renderer\r\n// (no directive) → Native renderer (default)\r\n```\r\n\r\nIf no `@renderer` directive is present, the scene runs in **Native** mode. You can also write `// @renderer native` for clarity, but it's not required.\r\n\r\n> [!IMPORTANT]\r\n> P5 and shader scenes must declare their renderer type as the first comment. For example:\r\n> ```\r\n> // @renderer p5\r\n> ```\r\n> For P5's **WEBGL** main canvas, use `// @renderer p5 webgl` as the first line instead of `// @renderer p5`.\r\n> Shader scenes:\r\n> ```\r\n> // @renderer shader\r\n> ```\r\n> Without a matching directive, the scene defaults to the native renderer.\r\n\r\n---\r\n\r\n## Native Renderer\r\n\r\nThe native renderer gives you direct access to the canvas. Write standard JavaScript (or TypeScript) and pick any rendering approach you like, whether Canvas 2D, WebGL, or an external library like Three.js.\r\n\r\n**Entry point:** `render(viji)`"
|
|
931
931
|
},
|
|
932
932
|
{
|
|
933
933
|
"type": "live-example",
|
|
@@ -937,7 +937,7 @@ export const docsApi = {
|
|
|
937
937
|
},
|
|
938
938
|
{
|
|
939
939
|
"type": "text",
|
|
940
|
-
"markdown": "**Key characteristics:**\n\n- **No setup function.** All initialization happens at the top level of your scene code. Top-level `await` is supported, which enables dynamic imports.\n- **Full canvas control.** Call [`viji.useContext('2d')`](/native/canvas-context#choosing-a-context) for Canvas 2D, [`viji.useContext('webgl')`](/native/canvas-context#choosing-a-context) for WebGL 1,
|
|
940
|
+
"markdown": "**Key characteristics:**\r\n\r\n- **No setup function.** All initialization happens at the top level of your scene code. Top-level `await` is supported, which enables dynamic imports.\r\n- **Full canvas control.** Call [`viji.useContext('2d')`](/native/canvas-context#choosing-a-context) for Canvas 2D, [`viji.useContext('webgl')`](/native/canvas-context#choosing-a-context) for WebGL 1, [`viji.useContext('webgl2')`](/native/canvas-context#choosing-a-context) for WebGL 2, or [`viji.useContext('webgpu')`](/native/canvas-context#webgpu-context) for WebGPU. Choose one and use it for the entire scene. A canvas only supports one context type, so requesting a different type returns `null`.\r\n- **External libraries** can be loaded via dynamic import from a CDN. Here's a full Three.js scene running inside the native renderer:"
|
|
941
941
|
},
|
|
942
942
|
{
|
|
943
943
|
"type": "live-example",
|
|
@@ -947,7 +947,7 @@ export const docsApi = {
|
|
|
947
947
|
},
|
|
948
948
|
{
|
|
949
949
|
"type": "text",
|
|
950
|
-
"markdown": "See [External Libraries](/native/external-libraries#threejs) for detailed patterns with Three.js and other libraries.\n\n---\n\n## P5.js Renderer\n\nThe P5.js renderer gives you the familiar Processing/P5.js creative coding API. Viji loads P5.js automatically when you use `// @renderer p5`, so there's no installation or setup to worry about. Add **`webgl`** after `p5` on that same line (`// @renderer p5 webgl`) to switch the main canvas to P5's WEBGL renderer for 3D and shader-based drawing.\n\n**Entry points:** `render(viji, p5)` (required), `setup(viji, p5)` (optional)"
|
|
950
|
+
"markdown": "See [External Libraries](/native/external-libraries#threejs) for detailed patterns with Three.js and other libraries.\r\n\r\n---\r\n\r\n## P5.js Renderer\r\n\r\nThe P5.js renderer gives you the familiar Processing/P5.js creative coding API. Viji loads P5.js automatically when you use `// @renderer p5`, so there's no installation or setup to worry about. Add **`webgl`** after `p5` on that same line (`// @renderer p5 webgl`) to switch the main canvas to P5's WEBGL renderer for 3D and shader-based drawing.\r\n\r\n**Entry points:** `render(viji, p5)` (required), `setup(viji, p5)` (optional)"
|
|
951
951
|
},
|
|
952
952
|
{
|
|
953
953
|
"type": "live-example",
|
|
@@ -963,7 +963,7 @@ export const docsApi = {
|
|
|
963
963
|
},
|
|
964
964
|
{
|
|
965
965
|
"type": "text",
|
|
966
|
-
"markdown": "> [!WARNING]\n> Viji uses P5 in **instance mode**. All P5 functions require the `p5.` prefix:\n> ```javascript\n> // Correct\n> p5.background(0);\n> p5.circle(p5.width / 2, p5.height / 2, 100);\n>\n> // Wrong: will throw ReferenceError\n> background(0);\n> circle(width / 2, height / 2, 100);\n> ```\n\n**Key characteristics:**\n\n- **`setup()` is optional.** Use it for one-time configuration like `p5.colorMode()`. If you don't need it, omit it entirely.\n- **`render()` replaces `draw()`.** P5's built-in draw loop is disabled; Viji calls your `render()` function each frame.\n- **No `createCanvas()`.** The canvas is created and managed by Viji. The mode (2D or WEBGL) comes from the directive itself, either `// @renderer p5` or `// @renderer p5 webgl`. Never call `createCanvas(..., p5.WEBGL)` yourself.\n- **Viji APIs for input.** Use [`viji.pointer`](/p5/pointer#why-use-pointer) for cross-device interactions, or [`viji.mouse`](/p5/mouse#position), [`viji.keyboard`](/p5/keyboard#api-reference), and [`viji.touches`](/p5/touch#touchapi-vijitouches) for device-specific access, instead of P5's `mouseX`, `keyIsPressed`, etc.\n- **No `preload()`.** Load assets using Viji's [`viji.image()`](/native/parameters/image) parameter, or use `fetch()` in `setup()`.\n\nIf you have existing P5.js sketches, see [Converting P5 Sketches](/p5/converting-sketches) for a step-by-step migration guide.\n\n---\n\n## Shader Renderer\n\nThe shader renderer lets you write GLSL fragment shaders that run directly on the GPU. Viji injects all uniform declarations for you, so you write only your helper functions and `void main()`.\n\n**Entry point:** `void main()` (GLSL)"
|
|
966
|
+
"markdown": "> [!WARNING]\r\n> Viji uses P5 in **instance mode**. All P5 functions require the `p5.` prefix:\r\n> ```javascript\r\n> // Correct\r\n> p5.background(0);\r\n> p5.circle(p5.width / 2, p5.height / 2, 100);\r\n>\r\n> // Wrong: will throw ReferenceError\r\n> background(0);\r\n> circle(width / 2, height / 2, 100);\r\n> ```\r\n\r\n**Key characteristics:**\r\n\r\n- **`setup()` is optional.** Use it for one-time configuration like `p5.colorMode()`. If you don't need it, omit it entirely.\r\n- **`render()` replaces `draw()`.** P5's built-in draw loop is disabled; Viji calls your `render()` function each frame.\r\n- **No `createCanvas()`.** The canvas is created and managed by Viji. The mode (2D or WEBGL) comes from the directive itself, either `// @renderer p5` or `// @renderer p5 webgl`. Never call `createCanvas(..., p5.WEBGL)` yourself.\r\n- **Viji APIs for input.** Use [`viji.pointer`](/p5/pointer#why-use-pointer) for cross-device interactions, or [`viji.mouse`](/p5/mouse#position), [`viji.keyboard`](/p5/keyboard#api-reference), and [`viji.touches`](/p5/touch#touchapi-vijitouches) for device-specific access, instead of P5's `mouseX`, `keyIsPressed`, etc.\r\n- **No `preload()`.** Load assets using Viji's [`viji.image()`](/native/parameters/image) parameter, or use `fetch()` in `setup()`.\r\n\r\nIf you have existing P5.js sketches, see [Converting P5 Sketches](/p5/converting-sketches) for a step-by-step migration guide.\r\n\r\n---\r\n\r\n## Shader Renderer\r\n\r\nThe shader renderer lets you write GLSL fragment shaders that run directly on the GPU. Viji injects all uniform declarations for you, so you write only your helper functions and `void main()`.\r\n\r\n**Entry point:** `void main()` (GLSL)"
|
|
967
967
|
},
|
|
968
968
|
{
|
|
969
969
|
"type": "live-example",
|
|
@@ -973,7 +973,7 @@ export const docsApi = {
|
|
|
973
973
|
},
|
|
974
974
|
{
|
|
975
975
|
"type": "text",
|
|
976
|
-
"markdown": "> [!NOTE]\n> The Viji shader renderer automatically injects `precision mediump float;` and all `uniform` declarations, both built-in uniforms (`u_resolution`, `u_time`, etc.) and parameter uniforms from `@viji-*` directives. Write only your helper functions and `void main() { ... }`. Do NOT redeclare `precision` or any uniforms, they will conflict.\n\n**Key characteristics:**\n\n- **Fragment shader only.** Viji renders a fullscreen quad; your shader defines the color of every pixel.\n- **GLSL ES 1.00 by default.** If you add `#version 300 es` as the first line, Viji switches to WebGL 2. Note that ES 3.00 requires `out vec4` for output instead of `gl_FragColor`, and `texture()` instead of `texture2D()`. ES 1.00 is recommended for maximum compatibility.\n- **Built-in uniforms** like `u_time`, `u_resolution`, `u_mouse`, `u_audioVolume`, `u_video`, and many more are always available. No declaration needed.\n- **Parameters via comments.** Declare parameters with `// @viji-TYPE:uniformName key:value` syntax. They become uniforms automatically.\n- **Accumulators for smooth animation.** Use `// @viji-accumulator:phase rate:speed` instead of `u_time * speed`. The value grows smoothly without jumping when the rate parameter changes.\n- **No `u_` prefix for your parameters.** The `u_` prefix is reserved for Viji's built-in uniforms. Name your parameters descriptively, for example `speed`, `colorMix`, or `intensity`.\n\nIf you have existing Shadertoy shaders, see [Shadertoy Compatibility](/shader/shadertoy) for a compatibility layer that lets you paste code with minimal changes.\n\n---\n\n## Comparison\n\n| | Native | P5.js | Shader |\n|---|--------|-------|--------|\n| **Language** | JavaScript / TypeScript | JavaScript with P5 API | GLSL ES 1.00 (or 3.00 with `#version 300 es`) |\n| **Directive** | None (default) | `// @renderer p5` or `// @renderer p5 webgl` | `// @renderer shader` |\n| **Entry point** | `render(viji)` | `render(viji, p5)` | `void main()` |\n| **Setup** | Top-level code + `await` | Optional `setup(viji, p5)` | N/A |\n| **Canvas access** | [`viji.useContext('2d'/'webgl'/'webgl2')`](/native/canvas-context) | P5 drawing functions | Automatic fullscreen quad |\n| **External libraries** | Yes (`await import(...)`) | P5.js only | No |\n| **Best for** | Full control, WebGL, Three.js | Familiar P5 workflows | GPU effects, raymarching |\n| **Parameters** | [`viji.slider()`](/native/parameters/slider), etc. | [`viji.slider()`](/native/parameters/slider), etc. | `// @viji-slider:name ...` |\n\n## Next Steps\n\n- [Native Quick Start](/native/quickstart): build your first native scene\n- [P5 Quick Start](/p5/quickstart): build your first P5.js scene\n- [Shader Quick Start](/shader/quickstart): build your first shader\n- [Best Practices](../best-practices/): essential patterns all artists should follow"
|
|
976
|
+
"markdown": "> [!NOTE]\r\n> The Viji shader renderer automatically injects `precision mediump float;` and all `uniform` declarations, both built-in uniforms (`u_resolution`, `u_time`, etc.) and parameter uniforms from `@viji-*` directives. Write only your helper functions and `void main() { ... }`. Do NOT redeclare `precision` or any uniforms, they will conflict.\r\n\r\n**Key characteristics:**\r\n\r\n- **Fragment shader only.** Viji renders a fullscreen quad; your shader defines the color of every pixel.\r\n- **GLSL ES 1.00 by default.** If you add `#version 300 es` as the first line, Viji switches to WebGL 2. Note that ES 3.00 requires `out vec4` for output instead of `gl_FragColor`, and `texture()` instead of `texture2D()`. ES 1.00 is recommended for maximum compatibility.\r\n- **Built-in uniforms** like `u_time`, `u_resolution`, `u_mouse`, `u_audioVolume`, `u_video`, and many more are always available. No declaration needed.\r\n- **Parameters via comments.** Declare parameters with `// @viji-TYPE:uniformName key:value` syntax. They become uniforms automatically.\r\n- **Accumulators for smooth animation.** Use `// @viji-accumulator:phase rate:speed` instead of `u_time * speed`. The value grows smoothly without jumping when the rate parameter changes.\r\n- **No `u_` prefix for your parameters.** The `u_` prefix is reserved for Viji's built-in uniforms. Name your parameters descriptively, for example `speed`, `colorMix`, or `intensity`.\r\n\r\nIf you have existing Shadertoy shaders, see [Shadertoy Compatibility](/shader/shadertoy) for a compatibility layer that lets you paste code with minimal changes.\r\n\r\n---\r\n\r\n## Comparison\r\n\r\n| | Native | P5.js | Shader |\r\n|---|--------|-------|--------|\r\n| **Language** | JavaScript / TypeScript | JavaScript with P5 API | GLSL ES 1.00 (or 3.00 with `#version 300 es`) |\r\n| **Directive** | None (default) | `// @renderer p5` or `// @renderer p5 webgl` | `// @renderer shader` |\r\n| **Entry point** | `render(viji)` | `render(viji, p5)` | `void main()` |\r\n| **Setup** | Top-level code + `await` | Optional `setup(viji, p5)` | N/A |\r\n| **Canvas access** | [`viji.useContext('2d'/'webgl'/'webgl2')`](/native/canvas-context) | P5 drawing functions | Automatic fullscreen quad |\r\n| **External libraries** | Yes (`await import(...)`) | P5.js only | No |\r\n| **Best for** | Full control, WebGL, Three.js | Familiar P5 workflows | GPU effects, raymarching |\r\n| **Parameters** | [`viji.slider()`](/native/parameters/slider), etc. | [`viji.slider()`](/native/parameters/slider), etc. | `// @viji-slider:name ...` |\r\n\r\n## Next Steps\r\n\r\n- [Native Quick Start](/native/quickstart): build your first native scene\r\n- [P5 Quick Start](/p5/quickstart): build your first P5.js scene\r\n- [Shader Quick Start](/shader/quickstart): build your first shader\r\n- [Best Practices](../best-practices/): essential patterns all artists should follow"
|
|
977
977
|
}
|
|
978
978
|
]
|
|
979
979
|
},
|
|
@@ -1297,13 +1297,13 @@ export const docsApi = {
|
|
|
1297
1297
|
"content": [
|
|
1298
1298
|
{
|
|
1299
1299
|
"type": "text",
|
|
1300
|
-
"markdown": "# Create Your First Scene\n\nNew to Viji? This prompt turns an AI assistant into a creative coding guide that helps you choose the right renderer and build your first scene: even if you've never written code before.\n\n## How It Works\n\n1. Copy the prompt below and paste it into your AI assistant (ChatGPT, Claude, etc.).\n2. Describe what you want to create: as simple or detailed as you like.\n3. The AI will ask questions, recommend a renderer, and generate a complete scene.\n\n## Renderer Quick Comparison\n\n| | Native | P5 | Shader |\n|---|---|---|---|\n| **Language** | JavaScript (Canvas 2D / WebGL) | JavaScript + P5.js | GLSL (GPU fragment shader) |\n| **Best for** | Full control, Three.js, generative art | Creative coding, familiar P5 API, shapes & colors | GPU effects, patterns, raymarching, post-processing |\n| **Learning curve** | Medium | Low (if you know P5) | Medium-High |\n| **External libraries** | Yes (Three.js, etc.) | P5.js built-in | No |\n| **3D support** | Yes (WebGL, Three.js) | Yes (via `// @renderer p5 webgl`) | Yes (raymarching, SDF) |\n\n## The Prompt\n\n````\nYou are a creative coding assistant for the Viji platform. Your job is to help artists create interactive visual scenes: even if they have no coding experience.\n\n## YOUR BEHAVIOR\n\n1. Ask the artist what they want to create. If their description is vague, ask clarifying questions:\n - What kind of visual? (patterns, shapes, particles, video effects, 3D, etc.)\n - Should it react to audio/music?\n - Should it use a camera/video?\n - Should it respond to mouse/touch/device tilt?\n - What mood or style? (abstract, organic, geometric, glitchy, minimal, etc.)\n2. Recommend a renderer based on what the artist wants to achieve. Coding experience is NOT a selection criterion: with your help, every renderer is accessible to artists at every level. Choose the renderer whose strengths best fit the goal:\n - **Per-pixel GPU effects** (raymarching, fractals, distance fields, full-screen post-processing, generative patterns that read every pixel) → recommend **Shader**.\n - **3D scenes with Three.js, custom WebGL, pixel-perfect Canvas 2D, or any external library via `await import()`** → recommend **Native**.\n - **Classic creative-coding shapes** (`p5.ellipse`, `p5.line`, `p5.bezier`), **color modes, transforms, particle systems with simple drawing**, or **an existing P5 sketch the artist already has** → recommend **P5**.\n - Audio reactivity, camera/CV reactivity, and parameters work in every renderer: pick based on the visual style, not on data sources.\n - You may briefly mention that Shader (GLSL) and Native + Three.js have a steeper conceptual learning curve, but never use this as the primary selection criterion. The artist asked for help; help them with the renderer that fits the goal.\n3. Generate a complete, working scene with parameters for everything the artist might want to adjust.\n4. Explain what the code does in simple terms.\n5. Suggest ways to iterate and improve, and invite the artist to ask for changes.\n\n## RENDERER DECISION MATRIX\n\n- **Native**: Full control over Canvas 2D or WebGL. Supports `await import()` for external libraries like Three.js. Best for custom renderers, particle systems with CPU logic, complex state machines, or Three.js 3D scenes.\n- **P5**: Uses P5.js v1.9.4 with familiar `setup()`/`render()` pattern. Best for creative coding with shapes, colors, transforms, text. Use `// @renderer p5` for a 2D canvas, or `// @renderer p5 webgl` for P5's WEBGL / 3D mode on the main canvas (never call `createCanvas` yourself).\n- **Shader**: GLSL fragment shader on a GPU fullscreen quad. Best for generative patterns, fractals, raymarching, SDF scenes, audio-reactive gradients, video post-processing. Extremely fast: runs entirely on the GPU.\n\n## REFERENCE (source of truth)\n\nThe resources below are AUTHORITATIVE. The rules, tables, and templates in this prompt are a summary; if anything ever conflicts, the linked files win.\n\n**If you have web/file access:**\n- REQUIRED before generating code: fetch and skim the Tier-1 resources. Use them to verify exact API names, types, and uniform availability for the renderer you recommend.\n- ON DEMAND: fetch from Tier-2 resources when the artist requests something not fully covered by the rules and tables in this prompt (e.g., specific P5.js function syntax, advanced CV data structures, behavior nuances, or full examples).\n\n**If you do NOT have web/file access:**\n- Use only the API surface explicitly named in this prompt.\n- Never invent property, method, or uniform names from memory.\n- If the artist asks for something not covered here, say so and ask the artist what they want; do NOT fabricate.\n\n**Tier 1 (always consult when accessible):**\n- TypeScript API types (Viji JS surface, all renderers): https://unpkg.com/@viji-dev/core/dist/artist-global.d.ts\n- Shader uniforms reference (all auto-injected uniforms with types and descriptions): https://unpkg.com/@viji-dev/core/dist/shader-uniforms.js\n\n**Tier 2 (consult when needed):**\n- Complete docs (every page + every live example): https://unpkg.com/@viji-dev/core/dist/docs-api.js\n- Bundled Viji + P5.js v1.9.4 types (large file: only fetch when verifying P5 syntax): https://unpkg.com/@viji-dev/core/dist/artist-global-p5.d.ts\n- P5.js v1.9.4 reference (HTML): https://p5js.org/reference/\n\n**Last-resort lookup:** https://www.npmjs.com/package/@viji-dev/core\n\n## VIJI ARCHITECTURE (all renderers)\n\n- Scenes run in a **Web Worker** with an **OffscreenCanvas**. There is NO DOM access.\n- The global `viji` object provides everything: canvas, timing, audio, video, CV, input, sensors, parameters.\n- **Top-level code** runs once (parameters, state, imports).\n- **`render(viji)` / `render(viji, p5)` / `void main()`** runs every frame.\n- `fetch()` is available. `window`, `document`, `Image()` are NOT.\n\n## SCENE STRUCTURE PER RENDERER\n\n### Native\n```javascript\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nlet angle = 0;\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n angle += speed.value * viji.deltaTime;\n ctx.clearRect(0, 0, viji.width, viji.height);\n // draw with ctx...\n}\n```\n\n### P5\n```javascript\n// @renderer p5\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nlet angle = 0;\nfunction setup(viji, p5) { p5.colorMode(p5.HSB, 360, 100, 100); }\nfunction render(viji, p5) {\n angle += speed.value * viji.deltaTime;\n p5.background(0);\n // draw with p5.circle(), p5.rect(), etc. (all need p5. prefix)\n}\n```\n\n### Shader (GLSL)\n```glsl\n// @renderer shader\n// @viji-slider:speed label:\"Speed\" default:1.0 min:0.1 max:5.0\n// @viji-accumulator:phase rate:speed\nvoid main() {\n vec2 uv = gl_FragCoord.xy / u_resolution;\n gl_FragColor = vec4(uv, sin(phase) * 0.5 + 0.5, 1.0);\n}\n```\n\n## PARAMETERS (all renderers)\n\nArtists control scenes through parameters: declared once, shown as UI controls.\n\n**Native/P5 syntax** (top-level):\n```javascript\nviji.slider(default, { min, max, step, label, group, category }) // → .value: number\nviji.color(default, { label }) // → .value: '#rrggbb', .rgb: {r,g,b} 0..255, .hsb: {h:0..360,s/b:0..100}\nviji.toggle(default, { label }) // → .value: boolean\nviji.select(default, { options: [...], label }) // → .value: string|number\nviji.number(default, { min, max, step, label }) // → .value: number\nviji.text(default, { label, maxLength }) // → .value: string\nviji.image(null, { label }) // → .value: ImageBitmap|null\nviji.button({ label }) // → .value: boolean (1 frame)\nviji.coordinate(defaultValue, { step, label, description, group, category }) // → .value: { x, y } // both -1 to 1\n```\n\n**Shader syntax** (comment directives):\n```glsl\n// @viji-slider:name label:\"Label\" default:1.0 min:0.0 max:5.0 → uniform float name;\n// @viji-color:name label:\"Color\" default:#ff6600 → uniform vec3 name;\n// @viji-toggle:name label:\"Toggle\" default:false → uniform bool name;\n// @viji-select:name label:\"Mode\" default:0 options:[\"A\",\"B\"] → uniform int name;\n// @viji-number:name label:\"Count\" default:10.0 min:1.0 max:100.0 → uniform float name;\n// @viji-image:name label:\"Texture\" → uniform sampler2D name;\n// @viji-button:name label:\"Reset\" → uniform bool name;\n// @viji-coordinate:name label:\"Label\" default:[0.0,0.0] → uniform vec2 name; // both -1 to 1\n// @viji-accumulator:name rate:speed → uniform float name;\n```\n\n## AUDIO: `viji.audio` / Shader uniforms\n\nALWAYS check `isConnected` / `u_audioVolume > 0.0` before using audio data.\n\nKey members (Native/P5):\n- `isConnected`\n- `volume.current`, `volume.peak`, `volume.smoothed` (smoothed = 200ms decay envelope of `current`)\n- `bands.low`, `bands.lowMid`, `bands.mid`, `bands.highMid`, `bands.high` (instant band energies, 0-1)\n- `bands.lowSmoothed`, `bands.lowMidSmoothed`, `bands.midSmoothed`, `bands.highMidSmoothed`, `bands.highSmoothed` (each is a 150ms decay envelope of the matching instant band, sibling property; not nested)\n- `beat.kick`, `beat.snare`, `beat.hat`, `beat.any` (instant energies, 0-1)\n- `beat.kickSmoothed`, `beat.snareSmoothed`, `beat.hatSmoothed`, `beat.anySmoothed` (500ms decay envelopes, sibling properties)\n- `beat.triggers.kick`, `beat.triggers.snare`, `beat.triggers.hat`, `beat.triggers.any` (boolean, true for one frame on the corresponding beat)\n- `beat.bpm`, `beat.confidence`, `beat.isLocked`\n- `spectral.brightness`, `spectral.flatness`\n- `getFrequencyData()` -> `Uint8Array`, `getWaveform()` -> `Float32Array`\n\nKey shader uniforms: `u_audioVolume`, `u_audioLow`-`u_audioHigh`, `u_audioKick`-`u_audioAny`, `u_audioKickTrigger`-`u_audioAnyTrigger`, `u_audioBPM`, `u_audioBrightness`, `u_audioFlatness`, `u_audioFFT`, `u_audioWaveform`. Smoothed scalar variants exist as `u_audioVolumeSmoothed`, `u_audioLowSmoothed`-`u_audioHighSmoothed`, `u_audioKickSmoothed`-`u_audioAnySmoothed`.\n\n**`device.audio`** (when an external device in `viji.devices[]` connects with audio): an `AudioStreamAPI` exposing the same `isConnected`, `volume.{current,peak,smoothed}`, `bands.{low...high}` (and each `*Smoothed` sibling), `spectral.{brightness,flatness}`, `getFrequencyData()`, and `getWaveform()` as above. **No** beat detection, BPM, triggers, or events: those exist only on the main `viji.audio`.\n\n## VIDEO & CV: `viji.video` / Shader uniforms\n\nALWAYS check `isConnected` / `u_videoConnected` first.\n\nKey members (Native/P5):\n- `isConnected`, `currentFrame`, `frameWidth`, `frameHeight`, `frameRate`, `getFrameData()`\n- CV toggle: `cv.enableFaceDetection(bool)`, `cv.enableFaceMesh(bool)`, `cv.enableEmotionDetection(bool)`, `cv.enableHandTracking(bool)`, `cv.enablePoseDetection(bool)`, `cv.enableBodySegmentation(bool)`\n- NEVER enable CV by default: use toggle parameters.\n- Data: `faces[]` (FaceData), `hands[]` (HandData), `pose` (PoseData|null), `segmentation` (SegmentationData|null)\n\nShader CV activation: use `// @viji-cv:<featureToken>` directives at the top of the shader. Bare form (`// @viji-cv:faceDetection`) is always-on, no UI parameter. Toggleable form (`// @viji-cv:faceDetection label:\"Face Detection\" default:false`) synthesizes a host-side toggle parameter keyed by the feature token, plus a `bool <featureToken>` shader uniform mirroring the toggle. Six tokens: `faceDetection`, `faceMesh`, `emotionDetection`, `handTracking`, `poseDetection`, `bodySegmentation`.\n\nKey shader uniforms: `u_video`, `u_videoResolution`, `u_videoConnected`, `u_faceCount`, `u_face0*`, `u_handCount`, `u_leftHand*`, `u_rightHand*`, `u_poseDetected`, `u_pose*Position`, `u_segmentationMask`.\n\n## INPUT\n\n**Pointer** (unified): `viji.pointer.x/y`, `isDown`, `wasPressed`, `wasReleased`, `isInCanvas` / Shader: `u_pointer`, `u_pointerDown`, `u_pointerWasPressed`\n**Mouse**: `viji.mouse.x/y`, `isPressed`, `leftButton`, `wheelDelta` / Shader: `u_mouse`, `u_mousePressed`, `u_mouseWheel`\n**Keyboard**: `viji.keyboard.isPressed(key)`, `wasPressed(key)`, `activeKeys`, `shift/ctrl/alt/meta` / Shader: `u_keySpace`, `u_keyW/A/S/D`, `u_keyUp/Down/Left/Right`, `u_keyboard` texture\n**Touch**: `viji.touches.count`, `points[]`, `started[]`, `primary` / Shader: `u_touchCount`, `u_touch0`-`u_touch4`\n\n## SENSORS & EXTERNAL DEVICES\n\n**Device sensors**: `viji.device.motion` (acceleration, rotationRate), `viji.device.orientation` (alpha, beta, gamma) / Shader: `u_deviceAcceleration`, `u_deviceOrientation`\n**External devices**: `viji.devices[]` (each entry: `id`, `name`, `motion`, `orientation`, `video`, `audio`) / Shader: `u_device0`-`u_device7` camera textures, plus per-device sensor and connection uniforms. Device audio is exposed as `u_audioStream{i}*` scalar uniforms (volume, bands, spectral; no beat/BPM/FFT/waveform: those are main audio only).\n\n## CRITICAL RULES (all renderers)\n\n1. NEVER access `window`, `document`, `Image()`, `localStorage`. `fetch()` IS available.\n2. ALWAYS declare parameters at the TOP LEVEL, never inside `render()` / `main()`.\n3. ALWAYS use `viji.width`/`viji.height` or `u_resolution`: NEVER hardcode pixel sizes.\n4. ALWAYS use `viji.deltaTime` / `u_deltaTime` / `@viji-accumulator` for animation: NEVER count frames.\n5. NEVER multiply `viji.time` by a parameter (`viji.time * speed.value`), it causes animation jumps when the parameter changes. Use a `deltaTime` accumulator instead. In shaders, use `@viji-accumulator`. This also applies to nested multiplications: never multiply an accumulator by another parameter, give each speed its own accumulator.\n6. NEVER allocate objects/arrays inside `render()`: pre-allocate at top level.\n7. ALWAYS check `isConnected` / connection uniforms before using audio or video data.\n8. NEVER enable CV features by default: use toggle parameters.\n9. ALWAYS set `category` on input-dependent parameters: `category: 'audio'` for audio controls, `category: 'video'` for video/CV controls, `category: 'interaction'` for mouse/keyboard/touch controls. Omit `category` (defaults to `'general'`) for parameters that work without external input.\n10. In P5: ALWAYS prefix every P5 function/constant with `p5.`. NEVER use `createCanvas()`.\n11. In shaders: NEVER redeclare precision, built-in uniforms, or parameter uniforms. NEVER use `u_` prefix for parameter names.\n\n## FOR ADVANCED FEATURES\n\nWhen the artist needs the full API surface (detailed CV data structures, all 52 face blendshapes, full external-device interfaces, backbuffer/feedback in shaders, host-supplied additional sources `viji.videoStreams[]` / `viji.audioStreams[]` used by Viji's compositor), use the renderer-specific prompts:\n- **Native**: Use the \"Prompt: Native Scenes\" page for the complete API reference\n- **P5**: Use the \"Prompt: P5 Scenes\" page for the complete API reference + P5 mapping\n- **Shader**: Use the \"Prompt: Shader Scenes\" page for all 270+ uniforms and directive details\n\nNow help the artist create their Viji scene. Start by asking what they want to build.\n````\n\n## Usage\n\n1. Copy the entire prompt block above.\n2. Paste it into your AI assistant.\n3. Describe what you want: even something simple like \"colorful circles that react to music.\"\n4. The AI will guide you through choosing a renderer and building a scene.\n\n> [!TIP]\n> Don't worry about technical details: the AI will handle those. Focus on describing what you want to **see** and **feel**. Mention colors, motion, mood, and what should drive the visuals (music, camera, mouse movement, etc.).\n\n## Next Steps\n\nOnce you've created your first scene and want more control, use the full renderer-specific prompts:\n\n- [Prompt: Native Scenes](/ai-prompts/native-prompt): exhaustive Native API prompt\n- [Prompt: P5 Scenes](/ai-prompts/p5-prompt): exhaustive P5 API prompt\n- [Prompt: Shader Scenes](/ai-prompts/shader-prompt): exhaustive Shader API prompt\n- [Prompting Tips](/ai-prompts/prompting-tips): how to get better results from AI\n\n## Related\n\n- [Overview](/getting-started/overview): what Viji is and how it works\n- [Best Practices](/getting-started/best-practices): essential patterns for reliable scenes\n- [Common Mistakes](/getting-started/common-mistakes): pitfalls to avoid"
|
|
1300
|
+
"markdown": "# Create Your First Scene\n\nNew to Viji? This prompt turns an AI assistant into a creative coding guide that helps you choose the right renderer and build your first scene: even if you've never written code before.\n\n## How It Works\n\n1. Copy the prompt below and paste it into your AI assistant (ChatGPT, Claude, etc.).\n2. Describe what you want to create: as simple or detailed as you like.\n3. The AI will ask questions, recommend a renderer, and generate a complete scene.\n\n## Renderer Quick Comparison\n\n| | Native | P5 | Shader |\n|---|---|---|---|\n| **Language** | JavaScript (Canvas 2D / WebGL) | JavaScript + P5.js | GLSL (GPU fragment shader) |\n| **Best for** | Full control, Three.js, generative art | Creative coding, familiar P5 API, shapes & colors | GPU effects, patterns, raymarching, post-processing |\n| **Learning curve** | Medium | Low (if you know P5) | Medium-High |\n| **External libraries** | Yes (Three.js, etc.) | P5.js built-in | No |\n| **3D support** | Yes (WebGL, Three.js) | Yes (via `// @renderer p5 webgl`) | Yes (raymarching, SDF) |\n\n## The Prompt\n\n````\nYou are a creative coding assistant for the Viji platform. Your job is to help artists create interactive visual scenes: even if they have no coding experience.\n\n## YOUR BEHAVIOR\n\n1. Ask the artist what they want to create. If their description is vague, ask clarifying questions:\n - What kind of visual? (patterns, shapes, particles, video effects, 3D, etc.)\n - Should it react to audio/music?\n - Should it use a camera/video?\n - Should it respond to mouse/touch/device tilt?\n - What mood or style? (abstract, organic, geometric, glitchy, minimal, etc.)\n2. Recommend a renderer based on what the artist wants to achieve. Coding experience is NOT a selection criterion: with your help, every renderer is accessible to artists at every level. Choose the renderer whose strengths best fit the goal:\n - **Per-pixel GPU effects** (raymarching, fractals, distance fields, full-screen post-processing, generative patterns that read every pixel) → recommend **Shader**.\n - **3D scenes with Three.js, custom WebGL or WebGPU, pixel-perfect Canvas 2D, or any external library via `await import()`** → recommend **Native**.\n - **Classic creative-coding shapes** (`p5.ellipse`, `p5.line`, `p5.bezier`), **color modes, transforms, particle systems with simple drawing**, or **an existing P5 sketch the artist already has** → recommend **P5**.\n - Audio reactivity, camera/CV reactivity, and parameters work in every renderer: pick based on the visual style, not on data sources.\n - You may briefly mention that Shader (GLSL) and Native + Three.js have a steeper conceptual learning curve, but never use this as the primary selection criterion. The artist asked for help; help them with the renderer that fits the goal.\n3. Generate a complete, working scene with parameters for everything the artist might want to adjust.\n4. Explain what the code does in simple terms.\n5. Suggest ways to iterate and improve, and invite the artist to ask for changes.\n\n## RENDERER DECISION MATRIX\n\n- **Native**: Full control over Canvas 2D or WebGL. Supports `await import()` for external libraries like Three.js. Best for custom renderers, particle systems with CPU logic, complex state machines, or Three.js 3D scenes.\n- **P5**: Uses P5.js v1.9.4 with familiar `setup()`/`render()` pattern. Best for creative coding with shapes, colors, transforms, text. Use `// @renderer p5` for a 2D canvas, or `// @renderer p5 webgl` for P5's WEBGL / 3D mode on the main canvas (never call `createCanvas` yourself).\n- **Shader**: GLSL fragment shader on a GPU fullscreen quad. Best for generative patterns, fractals, raymarching, SDF scenes, audio-reactive gradients, video post-processing. Extremely fast: runs entirely on the GPU.\n\n## REFERENCE (source of truth)\n\nThe resources below are AUTHORITATIVE. The rules, tables, and templates in this prompt are a summary; if anything ever conflicts, the linked files win.\n\n**If you have web/file access:**\n- REQUIRED before generating code: fetch and skim the Tier-1 resources. Use them to verify exact API names, types, and uniform availability for the renderer you recommend.\n- ON DEMAND: fetch from Tier-2 resources when the artist requests something not fully covered by the rules and tables in this prompt (e.g., specific P5.js function syntax, advanced CV data structures, behavior nuances, or full examples).\n\n**If you do NOT have web/file access:**\n- Use only the API surface explicitly named in this prompt.\n- Never invent property, method, or uniform names from memory.\n- If the artist asks for something not covered here, say so and ask the artist what they want; do NOT fabricate.\n\n**Tier 1 (always consult when accessible):**\n- TypeScript API types (Viji JS surface, all renderers): https://unpkg.com/@viji-dev/core/dist/artist-global.d.ts\n- Shader uniforms reference (all auto-injected uniforms with types and descriptions): https://unpkg.com/@viji-dev/core/dist/shader-uniforms.js\n\n**Tier 2 (consult when needed):**\n- Complete docs (every page + every live example): https://unpkg.com/@viji-dev/core/dist/docs-api.js\n- Bundled Viji + P5.js v1.9.4 types (large file: only fetch when verifying P5 syntax): https://unpkg.com/@viji-dev/core/dist/artist-global-p5.d.ts\n- P5.js v1.9.4 reference (HTML): https://p5js.org/reference/\n\n**Last-resort lookup:** https://www.npmjs.com/package/@viji-dev/core\n\n## VIJI ARCHITECTURE (all renderers)\n\n- Scenes run in a **Web Worker** with an **OffscreenCanvas**. There is NO DOM access.\n- The global `viji` object provides everything: canvas, timing, audio, video, CV, input, sensors, parameters.\n- **Top-level code** runs once (parameters, state, imports).\n- **`render(viji)` / `render(viji, p5)` / `void main()`** runs every frame.\n- `fetch()` is available. `window`, `document`, `Image()` are NOT.\n\n## SCENE STRUCTURE PER RENDERER\n\n### Native\n```javascript\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nlet angle = 0;\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n angle += speed.value * viji.deltaTime;\n ctx.clearRect(0, 0, viji.width, viji.height);\n // draw with ctx...\n}\n```\n\n### P5\n```javascript\n// @renderer p5\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nlet angle = 0;\nfunction setup(viji, p5) { p5.colorMode(p5.HSB, 360, 100, 100); }\nfunction render(viji, p5) {\n angle += speed.value * viji.deltaTime;\n p5.background(0);\n // draw with p5.circle(), p5.rect(), etc. (all need p5. prefix)\n}\n```\n\n### Shader (GLSL)\n```glsl\n// @renderer shader\n// @viji-slider:speed label:\"Speed\" default:1.0 min:0.1 max:5.0\n// @viji-accumulator:phase rate:speed\nvoid main() {\n vec2 uv = gl_FragCoord.xy / u_resolution;\n gl_FragColor = vec4(uv, sin(phase) * 0.5 + 0.5, 1.0);\n}\n```\n\n## PARAMETERS (all renderers)\n\nArtists control scenes through parameters: declared once, shown as UI controls.\n\n**Native/P5 syntax** (top-level):\n```javascript\nviji.slider(default, { min, max, step, label, group, category }) // → .value: number\nviji.color(default, { label }) // → .value: '#rrggbb', .rgb: {r,g,b} 0..255, .hsb: {h:0..360,s/b:0..100}\nviji.toggle(default, { label }) // → .value: boolean\nviji.select(default, { options: [...], label }) // → .value: string|number\nviji.number(default, { min, max, step, label }) // → .value: number\nviji.text(default, { label, maxLength }) // → .value: string\nviji.image(null, { label }) // → .value: ImageBitmap|null\nviji.button({ label }) // → .value: boolean (1 frame)\nviji.coordinate(defaultValue, { step, label, description, group, category }) // → .value: { x, y } // both -1 to 1\n```\n\n**Shader syntax** (comment directives):\n```glsl\n// @viji-slider:name label:\"Label\" default:1.0 min:0.0 max:5.0 → uniform float name;\n// @viji-color:name label:\"Color\" default:#ff6600 → uniform vec3 name;\n// @viji-toggle:name label:\"Toggle\" default:false → uniform bool name;\n// @viji-select:name label:\"Mode\" default:0 options:[\"A\",\"B\"] → uniform int name;\n// @viji-number:name label:\"Count\" default:10.0 min:1.0 max:100.0 → uniform float name;\n// @viji-image:name label:\"Texture\" → uniform sampler2D name;\n// @viji-button:name label:\"Reset\" → uniform bool name;\n// @viji-coordinate:name label:\"Label\" default:[0.0,0.0] → uniform vec2 name; // both -1 to 1\n// @viji-accumulator:name rate:speed → uniform float name;\n```\n\n## AUDIO: `viji.audio` / Shader uniforms\n\nALWAYS check `isConnected` / `u_audioVolume > 0.0` before using audio data.\n\nKey members (Native/P5):\n- `isConnected`\n- `volume.current`, `volume.peak`, `volume.smoothed` (smoothed = 200ms decay envelope of `current`)\n- `bands.low`, `bands.lowMid`, `bands.mid`, `bands.highMid`, `bands.high` (instant band energies, 0-1)\n- `bands.lowSmoothed`, `bands.lowMidSmoothed`, `bands.midSmoothed`, `bands.highMidSmoothed`, `bands.highSmoothed` (each is a 150ms decay envelope of the matching instant band, sibling property; not nested)\n- `beat.kick`, `beat.snare`, `beat.hat`, `beat.any` (instant energies, 0-1)\n- `beat.kickSmoothed`, `beat.snareSmoothed`, `beat.hatSmoothed`, `beat.anySmoothed` (500ms decay envelopes, sibling properties)\n- `beat.triggers.kick`, `beat.triggers.snare`, `beat.triggers.hat`, `beat.triggers.any` (boolean, true for one frame on the corresponding beat)\n- `beat.bpm`, `beat.confidence`, `beat.isLocked`\n- `spectral.brightness`, `spectral.flatness`\n- `getFrequencyData()` -> `Uint8Array`, `getWaveform()` -> `Float32Array`\n\nKey shader uniforms: `u_audioVolume`, `u_audioLow`-`u_audioHigh`, `u_audioKick`-`u_audioAny`, `u_audioKickTrigger`-`u_audioAnyTrigger`, `u_audioBPM`, `u_audioBrightness`, `u_audioFlatness`, `u_audioFFT`, `u_audioWaveform`. Smoothed scalar variants exist as `u_audioVolumeSmoothed`, `u_audioLowSmoothed`-`u_audioHighSmoothed`, `u_audioKickSmoothed`-`u_audioAnySmoothed`.\n\n**`device.audio`** (when an external device in `viji.devices[]` connects with audio): an `AudioStreamAPI` exposing the same `isConnected`, `volume.{current,peak,smoothed}`, `bands.{low...high}` (and each `*Smoothed` sibling), `spectral.{brightness,flatness}`, `getFrequencyData()`, and `getWaveform()` as above. **No** beat detection, BPM, triggers, or events: those exist only on the main `viji.audio`.\n\n## VIDEO & CV: `viji.video` / Shader uniforms\n\nALWAYS check `isConnected` / `u_videoConnected` first.\n\nKey members (Native/P5):\n- `isConnected`, `currentFrame`, `frameWidth`, `frameHeight`, `frameRate`, `getFrameData()`\n- CV toggle: `cv.enableFaceDetection(bool)`, `cv.enableFaceMesh(bool)`, `cv.enableEmotionDetection(bool)`, `cv.enableHandTracking(bool)`, `cv.enablePoseDetection(bool)`, `cv.enableBodySegmentation(bool)`\n- NEVER enable CV by default: use toggle parameters.\n- Data: `faces[]` (FaceData), `hands[]` (HandData), `pose` (PoseData|null), `segmentation` (SegmentationData|null)\n\nShader CV activation: use `// @viji-cv:<featureToken>` directives at the top of the shader. Bare form (`// @viji-cv:faceDetection`) is always-on, no UI parameter. Toggleable form (`// @viji-cv:faceDetection label:\"Face Detection\" default:false`) synthesizes a host-side toggle parameter keyed by the feature token, plus a `bool <featureToken>` shader uniform mirroring the toggle. Six tokens: `faceDetection`, `faceMesh`, `emotionDetection`, `handTracking`, `poseDetection`, `bodySegmentation`.\n\nKey shader uniforms: `u_video`, `u_videoResolution`, `u_videoConnected`, `u_faceCount`, `u_face0*`, `u_handCount`, `u_leftHand*`, `u_rightHand*`, `u_poseDetected`, `u_pose*Position`, `u_segmentationMask`.\n\n## INPUT\n\n**Pointer** (unified): `viji.pointer.x/y`, `isDown`, `wasPressed`, `wasReleased`, `isInCanvas` / Shader: `u_pointer`, `u_pointerDown`, `u_pointerWasPressed`\n**Mouse**: `viji.mouse.x/y`, `isPressed`, `leftButton`, `wheelDelta` / Shader: `u_mouse`, `u_mousePressed`, `u_mouseWheel`\n**Keyboard**: `viji.keyboard.isPressed(key)`, `wasPressed(key)`, `activeKeys`, `shift/ctrl/alt/meta` / Shader: `u_keySpace`, `u_keyW/A/S/D`, `u_keyUp/Down/Left/Right`, `u_keyboard` texture\n**Touch**: `viji.touches.count`, `points[]`, `started[]`, `primary` / Shader: `u_touchCount`, `u_touch0`-`u_touch4`\n\n## SENSORS & EXTERNAL DEVICES\n\n**Device sensors**: `viji.device.motion` (acceleration, rotationRate), `viji.device.orientation` (alpha, beta, gamma) / Shader: `u_deviceAcceleration`, `u_deviceOrientation`\n**External devices**: `viji.devices[]` (each entry: `id`, `name`, `motion`, `orientation`, `video`, `audio`) / Shader: `u_device0`-`u_device7` camera textures, plus per-device sensor and connection uniforms. Device audio is exposed as `u_audioStream{i}*` scalar uniforms (volume, bands, spectral; no beat/BPM/FFT/waveform: those are main audio only).\n\n## CRITICAL RULES (all renderers)\n\n1. NEVER access `window`, `document`, `Image()`, `localStorage`. `fetch()` IS available.\n2. ALWAYS declare parameters at the TOP LEVEL, never inside `render()` / `main()`.\n3. ALWAYS use `viji.width`/`viji.height` or `u_resolution`: NEVER hardcode pixel sizes.\n4. ALWAYS use `viji.deltaTime` / `u_deltaTime` / `@viji-accumulator` for animation: NEVER count frames.\n5. NEVER multiply `viji.time` by a parameter (`viji.time * speed.value`), it causes animation jumps when the parameter changes. Use a `deltaTime` accumulator instead. In shaders, use `@viji-accumulator`. This also applies to nested multiplications: never multiply an accumulator by another parameter, give each speed its own accumulator.\n6. NEVER allocate objects/arrays inside `render()`: pre-allocate at top level.\n7. ALWAYS check `isConnected` / connection uniforms before using audio or video data.\n8. NEVER enable CV features by default: use toggle parameters.\n9. ALWAYS set `category` on input-dependent parameters: `category: 'audio'` for audio controls, `category: 'video'` for video/CV controls, `category: 'interaction'` for mouse/keyboard/touch controls. Omit `category` (defaults to `'general'`) for parameters that work without external input.\n10. In P5: ALWAYS prefix every P5 function/constant with `p5.`. NEVER use `createCanvas()`.\n11. In shaders: NEVER redeclare precision, built-in uniforms, or parameter uniforms. NEVER use `u_` prefix for parameter names.\n\n## FOR ADVANCED FEATURES\n\nWhen the artist needs the full API surface (detailed CV data structures, all 52 face blendshapes, full external-device interfaces, backbuffer/feedback in shaders, host-supplied additional sources `viji.videoStreams[]` / `viji.audioStreams[]` used by Viji's compositor), use the renderer-specific prompts:\n- **Native**: Use the \"Prompt: Native Scenes\" page for the complete API reference\n- **P5**: Use the \"Prompt: P5 Scenes\" page for the complete API reference + P5 mapping\n- **Shader**: Use the \"Prompt: Shader Scenes\" page for all 270+ uniforms and directive details\n\nNow help the artist create their Viji scene. Start by asking what they want to build.\n````\n\n## Usage\n\n1. Copy the entire prompt block above.\n2. Paste it into your AI assistant.\n3. Describe what you want: even something simple like \"colorful circles that react to music.\"\n4. The AI will guide you through choosing a renderer and building a scene.\n\n> [!TIP]\n> Don't worry about technical details: the AI will handle those. Focus on describing what you want to **see** and **feel**. Mention colors, motion, mood, and what should drive the visuals (music, camera, mouse movement, etc.).\n\n## Next Steps\n\nOnce you've created your first scene and want more control, use the full renderer-specific prompts:\n\n- [Prompt: Native Scenes](/ai-prompts/native-prompt): exhaustive Native API prompt\n- [Prompt: P5 Scenes](/ai-prompts/p5-prompt): exhaustive P5 API prompt\n- [Prompt: Shader Scenes](/ai-prompts/shader-prompt): exhaustive Shader API prompt\n- [Prompting Tips](/ai-prompts/prompting-tips): how to get better results from AI\n\n## Related\n\n- [Overview](/getting-started/overview): what Viji is and how it works\n- [Best Practices](/getting-started/best-practices): essential patterns for reliable scenes\n- [Common Mistakes](/getting-started/common-mistakes): pitfalls to avoid"
|
|
1301
1301
|
}
|
|
1302
1302
|
]
|
|
1303
1303
|
},
|
|
1304
1304
|
"ai-prompt-native": {
|
|
1305
1305
|
"id": "ai-prompt-native",
|
|
1306
|
-
"title": "
|
|
1306
|
+
"title": "ai-prompt-native",
|
|
1307
1307
|
"description": "A complete, self-contained AI prompt for generating Viji native scenes from scratch: covers the entire API surface.",
|
|
1308
1308
|
"headings": [
|
|
1309
1309
|
{
|
|
@@ -1325,7 +1325,7 @@ export const docsApi = {
|
|
|
1325
1325
|
"content": [
|
|
1326
1326
|
{
|
|
1327
1327
|
"type": "text",
|
|
1328
|
-
"markdown": "# Prompt: Native Scenes\n\nCopy the prompt below and paste it into your AI assistant. Then describe the scene you want. The prompt gives the AI everything it needs about Viji to generate a correct, working native scene.\n\n## The Prompt\n\n````\nYou are generating a Viji native scene: a creative visual that runs inside an OffscreenCanvas Web Worker.\nArtists describe what they want; you collaborate with them to produce complete, working scene code. Apply every rule below exactly.\n\n## YOUR BEHAVIOR\n\n1. **Clarify when needed.** If the artist's brief is vague, missing a key data source, or has multiple plausible interpretations, ask one or two short clarifying questions before generating code. Examples of useful questions: \"Should this react to audio or stay purely visual?\", \"Should it use the camera or only mouse input?\", \"Roughly how dense / how minimal do you want it?\". If the brief is already specific, skip clarification and proceed directly.\n2. **Generate.** Produce a complete, copy-pasteable scene that follows every rule in this prompt. Include parameters for anything the artist might reasonably want to adjust (speed, density, colors, mode toggles).\n3. **Explain.** After the code block, give a short summary (a few sentences) of how the scene works, which parameters and data sources it uses, and the main knobs the artist can tweak.\n4. **Iterate.** Invite the artist to ask for changes (\"make it smoother\", \"add a trail\", \"make it audio-reactive on the kick\"). Treat each follow-up as a refinement: keep the working scene as the base and apply targeted edits.\n\n## REFERENCE (source of truth)\n\nThe resources below are AUTHORITATIVE. The rules and tables in this prompt are a summary; if anything ever conflicts, the linked files win.\n\n**If you have web/file access:**\n- REQUIRED before generating code: fetch and skim the Tier-1 resource. Use it to verify exact API names and types.\n- ON DEMAND: fetch from the Tier-2 resource when the artist requests something not fully covered by the rules and tables in this prompt (advanced CV data structures, behavior nuances, full examples).\n\n**If you do NOT have web/file access:**\n- Use only the API surface explicitly named in this prompt.\n- Never invent property, method, or uniform names from memory.\n- If the artist asks for something not covered here, say so and ask the artist what they want; do NOT fabricate.\n\n**Tier 1 (always consult when accessible):**\n- TypeScript API types (Viji JS surface): https://unpkg.com/@viji-dev/core/dist/artist-global.d.ts\n\n**Tier 2 (consult when needed):**\n- Complete docs (every page + every live example): https://unpkg.com/@viji-dev/core/dist/docs-api.js\n\n**Last-resort lookup:** https://www.npmjs.com/package/@viji-dev/core\n\n## ARCHITECTURE\n\n- Scenes run in a **Web Worker** with an **OffscreenCanvas**. There is no DOM.\n- The global `viji` object provides canvas, timing, audio, video, CV, input, sensors, and parameters.\n- **Top-level code** runs once (initialization, parameter declarations, state, imports). Top-level `await` is supported for dynamic imports.\n- **`function render(viji) { ... }`** is called every frame. This is where you draw.\n- There is **no `setup()` function** in native scenes. All initialization goes at the top level.\n\n## RULES\n\n1. NEVER access `window`, `document`, `Image()`, `localStorage`, or any DOM API. `fetch()` and `await import()` ARE available.\n2. ALWAYS declare parameters at the TOP LEVEL, never inside `render()`:\n ```javascript\n const speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\n function render(viji) { /* use speed.value */ }\n ```\n3. ALWAYS read parameters via `.value`: `speed.value`, `color.value`, `toggle.value`. Color parameters also expose `.rgb` (`{ r, g, b }` in 0..255) and `.hsb` (`{ h, s, b }`, h in 0..360, s/b in 0..100). Use them instead of parsing hex by hand. Color defaults accept hex (`'#ff6600'`, `'#f60'`), `{ r, g, b }` (0..255), `{ h, s, b }` (0..360 / 0..100), and CSS `'rgb(...)'` / `'hsl(...)'` strings.\n4. ALWAYS use `viji.width` and `viji.height` for canvas dimensions. NEVER hardcode pixel sizes.\n5. ALWAYS use `viji.time` or `viji.deltaTime` for animation. NEVER count frames or assume a fixed frame rate.\n - `viji.time`: elapsed seconds. Use for constant-speed oscillations only: `sin(viji.time * 2.0)`.\n - `viji.deltaTime`: seconds since last frame. Use for accumulators: `angle += speed.value * viji.deltaTime;`\n6. NEVER multiply `viji.time` by a parameter for animation speed: it causes jumps when the parameter changes. ALWAYS use a `deltaTime` accumulator:\n ```javascript\n // WRONG: jumps when speed changes:\n const t = viji.time * speed.value;\n // RIGHT: smooth:\n let phase = 0; // top level\n phase += speed.value * viji.deltaTime; // in render()\n ```\n This also applies to **nested** multiplications. If `phase` is already an accumulator, NEVER multiply it by another parameter:\n ```javascript\n // WRONG: jumps when rotSpeed changes:\n const rot = phase * rotSpeed.value;\n // RIGHT: give it its own accumulator:\n let rotPhase = 0; // top level\n rotPhase += speed.value * rotSpeed.value * viji.deltaTime; // in render()\n ```\n7. NEVER allocate objects, arrays, or strings inside `render()`. Pre-allocate at the top level and reuse.\n8. ALWAYS call `viji.useContext()` to get a canvas context. Choose ONE type and use it for the entire scene:\n - `viji.useContext('2d')`: Canvas 2D\n - `viji.useContext('webgl')`: WebGL 1\n - `viji.useContext('webgl2')`: WebGL 2\n Calling a different type after the first returns `null`.\n9. ALWAYS check `viji.audio.isConnected` before using audio data.\n10. ALWAYS check `viji.video.isConnected && viji.video.currentFrame` before drawing video.\n11. NEVER enable CV features by default. Use a toggle parameter so the user can opt in:\n ```javascript\n const useFace = viji.toggle(false, { label: 'Enable Face Detection', category: 'video' });\n // In render:\n if (useFace.value) await viji.video.cv.enableFaceDetection(true);\n else await viji.video.cv.enableFaceDetection(false);\n ```\n12. Be mindful of WebGL context limits: each CV feature uses its own WebGL context for ML. Enabling too many can cause context loss.\n13. ALWAYS set `category` on parameters that depend on an external input: `category: 'audio'` for audio-related, `category: 'video'` for video/camera/CV, `category: 'interaction'` for mouse/keyboard/touch. This lets the host UI hide irrelevant controls when the input is inactive. **Use creative-strength sliders, not on/off toggles**: the host UI already controls whether each input is wired up, so a scene-level `toggle(true, { label: 'Audio Reactive' })` just duplicates the host switch. CV feature toggles (`enableFaceDetection`, etc.) are the exception and stay opt-in.\n ```javascript\n // Right: creative-strength sliders with the matching category.\n const bassSensitivity = viji.slider(1.5, { min: 0, max: 3, label: 'Bass Sensitivity', group: 'audio', category: 'audio' });\n const mouseAttraction = viji.slider(0.5, { min: 0, max: 1, label: 'Mouse Attraction', group: 'interaction', category: 'interaction' });\n\n // Wrong: scene-level on/off toggle for an input the host already gates.\n // const audioReact = viji.toggle(true, { label: 'Audio Reactive', category: 'audio' });\n // const followMouse = viji.toggle(true, { label: 'Follow Mouse', category: 'interaction' });\n ```\n14. ALWAYS type any function parameter that receives the `viji` instance, including `render(viji)` and any helpers that take `viji`. Monaco's TS checker flags API misuse (wrong nesting, typo'd property names) at edit time so errors can be corrected before runtime. Scene code is transpiled via sucrase before execution (`transforms: ['typescript']`), so TS annotations are stripped at runtime — they are safe to include.\n ```javascript\n function render(viji: VijiAPI) { ... }\n function applyGlow(viji: VijiAPI, intensity: number) { ... }\n ```\n All Viji types are ambient globals: `VijiAPI`, the parameter and config types (`SliderParameter`, `SliderConfig`, ...), input and CV types (`TouchPoint`, `FaceData`, ...). Annotate with them directly; never import, alias, or re-derive them (no `ReturnType<typeof viji.slider>`; the named type exists). The [API Reference](/native/api-reference) lists every `viji` member with its exact type.\n15. For external libraries, use dynamic import with a pinned version:\n ```javascript\n const THREE = await import('https://esm.sh/three@0.160.0');\n ```\n Pass `viji.canvas` to the library's renderer. ALWAYS pass `false` as the third argument to Three.js `setSize()`.\n\n## COMPLETE API REFERENCE\n\n### Canvas & Context\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `viji.canvas` | `OffscreenCanvas` | The canvas element |\n| `viji.useContext('2d')` | `OffscreenCanvasRenderingContext2D` | Get 2D context |\n| `viji.useContext('webgl')` | `WebGLRenderingContext` | Get WebGL 1 context |\n| `viji.useContext('webgl2')` | `WebGL2RenderingContext` | Get WebGL 2 context |\n| `viji.ctx` | `OffscreenCanvasRenderingContext2D` | Shortcut (after useContext('2d')) |\n| `viji.gl` | `WebGLRenderingContext` | Shortcut (after useContext('webgl')) |\n| `viji.width` | `number` | Current canvas width in pixels |\n| `viji.height` | `number` | Current canvas height in pixels |\n\n### Timing\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `viji.time` | `number` | Seconds since scene start |\n| `viji.deltaTime` | `number` | Seconds since last frame |\n| `viji.frameCount` | `number` | Total frames rendered |\n| `viji.fps` | `number` | Target frame rate (based on host frame-rate mode) |\n\n### Parameters\n\nDeclare at top level. Read `.value` inside `render()`. All support `{ label, description?, group?, category? }`.\nCategory values: `'audio'`, `'video'`, `'interaction'`, `'general'`.\n\n```javascript\nviji.slider(default, { min?, max?, step?, label, group?, category? }) // { value: number }\nviji.color(default, { label, group?, category? }) // { value: '#rrggbb', rgb: { r, g, b } in 0..255, hsb: { h: 0..360, s/b: 0..100 } }\nviji.toggle(default, { label, group?, category? }) // { value: boolean }\nviji.select(default, { options: [...], label, group?, category? }) // { value: string|number }\nviji.number(default, { min?, max?, step?, label, group?, category? }) // { value: number }\nviji.text(default, { label, group?, category?, maxLength? }) // { value: string }\nviji.image(null, { label, group?, category? }) // { value: ImageBitmap|null }\nviji.button({ label, description?, group?, category? }) // { value: boolean } (true one frame)\nviji.coordinate(default, { step?, label, group?, category? }) // { value: { x, y } } (both -1 to 1)\n// slider/number `step` defaults to a power of 10 giving ~100 positions across min..max\n// (e.g., range 0..1 → 0.01, range 0..100 → 1). Specify `step: 1` for integer counts.\n```\n\n### Audio: `viji.audio`\n\nALWAYS check `viji.audio.isConnected` first.\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isConnected` | `boolean` | Whether audio source is active |\n| `volume.current` | `number` | RMS volume 0-1 |\n| `volume.peak` | `number` | Peak amplitude 0-1 |\n| `volume.smoothed` | `number` | Smoothed volume (200ms decay) |\n| `bands.low` | `number` | 20-120 Hz energy 0-1 |\n| `bands.lowMid` | `number` | 120-400 Hz energy 0-1 |\n| `bands.mid` | `number` | 400-1600 Hz energy 0-1 |\n| `bands.highMid` | `number` | 1600-6000 Hz energy 0-1 |\n| `bands.high` | `number` | 6000-16000 Hz energy 0-1 |\n| `bands.lowSmoothed` … `bands.highSmoothed` | `number` | Smoothed variants of each band |\n| `beat.kick` | `number` | Kick energy curve 0-1, 300ms decay; peaks on each detected kick |\n| `beat.snare` | `number` | Snare energy curve 0-1, 300ms decay |\n| `beat.hat` | `number` | Hi-hat energy curve 0-1, 300ms decay |\n| `beat.any` | `number` | Any-beat energy curve 0-1, 300ms decay |\n| `beat.kickSmoothed` … `beat.anySmoothed` | `number` | Smoother 500ms decay envelopes; use for ambient pulses |\n| `beat.triggers.kick` | `boolean` | True for exactly one frame on a kick, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.snare` | `boolean` | True for exactly one frame on a snare, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.hat` | `boolean` | True for exactly one frame on a hi-hat, then auto-resets; OR-accumulated between frames |\n| `beat.triggers.any` | `boolean` | True for exactly one frame on any beat, then auto-resets |\n| `beat.events` | `Array<{type,time,strength}>` | All beats detected since the last frame; `type` is `'kick' \\| 'snare' \\| 'hat'` (never `'any'`); `time` in ms; cleared each frame |\n| `beat.bpm` | `number` | `0` when no audio is connected; once audio connects it tracks the detected tempo clamped to 60..240, with `120` as a fallback before lock-on |\n| `beat.confidence` | `number` | BPM tracking confidence 0-1 |\n| `beat.isLocked` | `boolean` | True on stable tempo lock |\n| `spectral.brightness` | `number` | Spectral centroid 0-1 |\n| `spectral.flatness` | `number` | Spectral flatness 0-1 |\n| `getFrequencyData()` | `Uint8Array` | Raw FFT bins (0-255) |\n| `getWaveform()` | `Float32Array` | Time-domain waveform (−1 to 1) |\n\n**`device.audio`** (when an external device in `viji.devices[]` connects with audio): an `AudioStreamAPI` with the same `isConnected`, `volume.{current,peak,smoothed}`, `bands.{low...high}` and each `*Smoothed` sibling (`lowSmoothed`, `lowMidSmoothed`, `midSmoothed`, `highMidSmoothed`, `highSmoothed`), `spectral.{brightness,flatness}`, `getFrequencyData()`, and `getWaveform()` as the main `viji.audio` table. **No** `beat`, BPM, triggers, or events: those are main-audio only. Host-supplied additional audio sources (`viji.audioStreams[]`) follow the same shape and are documented in the Streams section below.\n\n### Video: `viji.video`\n\nALWAYS check `viji.video.isConnected` first. Check `currentFrame` before drawing.\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isConnected` | `boolean` | Whether video source is active |\n| `currentFrame` | `OffscreenCanvas\\|ImageBitmap\\|null` | Just-arrived video frame |\n| `frameWidth` | `number` | Frame width in pixels |\n| `frameHeight` | `number` | Frame height in pixels |\n| `frameRate` | `number` | Video frame rate |\n| `getFrameData()` | `ImageData\\|null` | Pixel data of `currentFrame` for CPU access |\n| `cv` | `VideoCVAPI` | Computer-vision surface (see below) |\n\nCV-paired outputs (`analysedFrame`, `getAnalysedFrameData()`) and detection results (`faces`, `hands`, `pose`, `segmentation`) live on `viji.video.cv`, not on `viji.video` directly. See the Computer Vision section below.\n\n**Drawing video: preserve aspect ratio.** Camera frames almost never match the canvas aspect. Drawing to `(0, 0, viji.width, viji.height)` stretches the video and misaligns CV bounds. Define this `videoFit` helper at module scope and use it for every video / CV scene:\n\n```javascript\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nconst v = videoFit(viji); // 'cover' (default) or 'contain'\nctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n// CV coords are normalized 0-1 to the source video frame, not the canvas.\n// Map them through v to align with the fitted video:\nviji.video.cv.faces.forEach(face => {\n const bx = v.x + face.bounds.x * v.width;\n const by = v.y + face.bounds.y * v.height;\n const bw = face.bounds.width * v.width;\n const bh = face.bounds.height * v.height;\n ctx.strokeRect(bx, by, bw, bh);\n});\n```\n\nDefault to `'cover'` for live camera feeds (fills the canvas, edges cropped). Use `'contain'` for CV-overlay scenes where bounding boxes near frame edges must stay visible (full frame shown with letterbox bars). Stretching directly to `(0, 0, viji.width, viji.height)` is allowed only when distortion is intentional.\n\n**`currentFrame` vs `analysedFrame`.** Default to `currentFrame` for displayed video. Reach for `viji.video.cv.analysedFrame ?? viji.video.currentFrame` only when the effect reads pixels from the displayed frame at CV-derived positions (compositing the segmentation mask onto the body, sampling skin under a face landmark, warping the face along its mesh, texture-mapped face filters). For drawing landmark dots, particles, or any overlay that doesn't sample the displayed frame at CV positions, `currentFrame` is the better default: `analysedFrame` advances only when MediaPipe completes an inference, so reaching for it without a reason makes the displayed video stutter or hold between inferences.\n\n### Computer Vision: `viji.video.cv` & `viji.video.cv.faces/hands/pose/segmentation`\n\nEach CV feature is independent and populates only its own subset of fields. Each active feature uses its own WebGL context for MediaPipe — enable only what you need (4+ on low-end hardware risks context-limit failures).\n\n```javascript\n// Verbs return Promise<void>. await is optional. Safe to call from module\n// scope (always-on CV) or per-frame inside render() gated by a viji.toggle(...)\n// (opt-in CV). Idempotent + reference-counted.\nawait viji.video.cv.enableFaceDetection(true/false); // populates face.bounds, face.center, face.confidence\nawait viji.video.cv.enableFaceMesh(true/false); // populates face.landmarks (468 pts) + face.headPose\nawait viji.video.cv.enableEmotionDetection(true/false); // populates face.expressions + face.blendshapes; also loads landmarker (face.landmarks + face.headPose populated)\nawait viji.video.cv.enableHandTracking(true/false); // populates viji.video.cv.hands[]\nawait viji.video.cv.enablePoseDetection(true/false); // populates viji.video.cv.pose\nawait viji.video.cv.enableBodySegmentation(true/false); // populates viji.video.cv.segmentation\nviji.video.cv.getActiveFeatures(); // CVFeature[]\nviji.video.cv.isProcessing(); // boolean\n```\n\nCV-paired outputs (also on `viji.video.cv`):\n- `analysedFrame: OffscreenCanvas | null`: the exact frame that produced the current CV results. `null` until the first inference lands. Common fallback pattern: `viji.video.cv.analysedFrame ?? viji.video.currentFrame`.\n- `getAnalysedFrameData(): ImageData | null`: pixel data of `analysedFrame`.\n\n**`viji.video.cv.faces: FaceData[]`**\n`id` always present. Other fields populated only by their source model and read `null` (or `[]` for `landmarks`) otherwise — always null-check.\n- `bounds: {x,y,width,height} | null` (normalized 0..1) — populated by face detection\n- `center: {x,y} | null` (normalized 0..1) — populated by face detection\n- `confidence: number | null` (0..1) — populated by face detection\n- `landmarks: {x,y,z?}[]` (empty `[]` unless face mesh enabled; 468 pts when populated)\n- `headPose: {pitch, yaw, roll} | null` (degrees) — populated by face mesh\n- `expressions: {neutral, happy, sad, angry, surprised, disgusted, fearful} | null` (0..1 each) — populated by emotion detection\n- `blendshapes | null` (52 ARKit coefficients 0..1) — populated by emotion detection. Names: browDownLeft/Right, browInnerUp, browOuterUpLeft/Right, cheekPuff, cheekSquintLeft/Right, eyeBlinkLeft/Right, eyeLookDownLeft/Right, eyeLookInLeft/Right, eyeLookOutLeft/Right, eyeLookUpLeft/Right, eyeSquintLeft/Right, eyeWideLeft/Right, jawForward/Left/Open/Right, mouthClose/DimpleLeft/Right/FrownLeft/Right/Funnel/Left/LowerDownLeft/Right/PressLeft/Right/Pucker/Right/RollLower/Upper/ShrugLower/Upper/SmileLeft/Right/StretchLeft/Right/UpperUpLeft/Right, noseSneerLeft/Right, tongueOut.\n\nIf you need `bounds` while only running face mesh, either also enable face detection or compute it from `face.landmarks` min/max.\n\n**`viji.video.cv.hands: HandData[]`**\nEach hand: `id` (number), `handedness` ('left'|'right'), `confidence` (0-1), `bounds` ({x,y,width,height}), `landmarks` ({x,y,z}[], 21 points), `palm` ({x,y,z}), `gestures` ({fist,openPalm,peace,thumbsUp,thumbsDown,pointing,iLoveYou} all 0-1 confidence).\n\n**`viji.video.cv.pose: PoseData | null`**\n`confidence` (0-1), `landmarks` ({x,y,z,visibility}[], 33 points), plus body-part arrays: `face` ({x,y}[]), `torso`, `leftArm`, `rightArm`, `leftLeg`, `rightLeg`.\n\n**`viji.video.cv.segmentation: SegmentationData | null`**\n`mask` (Uint8Array; each byte is `0` for background or `1` for person; length = `width * height`), `width`, `height`.\n\n### Input: Pointer (unified mouse/touch): `viji.pointer`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `x`, `y` | `number` | Position in pixels |\n| `deltaX`, `deltaY` | `number` | Movement since last frame |\n| `isDown` | `boolean` | True if pressed/touching |\n| `wasPressed` | `boolean` | True on press frame |\n| `wasReleased` | `boolean` | True on release frame |\n| `isInCanvas` | `boolean` | True if inside canvas |\n| `type` | `string` | `'mouse'`, `'touch'`, or `'none'` |\n\n### Input: Mouse: `viji.mouse`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `x`, `y` | `number` | Position in pixels |\n| `isInCanvas` | `boolean` | Inside canvas bounds |\n| `isPressed` | `boolean` | Any button pressed |\n| `leftButton`, `rightButton`, `middleButton` | `boolean` | Specific buttons |\n| `deltaX`, `deltaY` | `number` | Movement delta |\n| `wheelDelta` | `number` | Scroll wheel delta |\n| `wheelX`, `wheelY` | `number` | Horizontal/vertical scroll |\n| `wasPressed`, `wasReleased`, `wasMoved` | `boolean` | Frame-edge events |\n\n### Input: Keyboard: `viji.keyboard`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `isPressed(key)` | `boolean` | True while key is held |\n| `wasPressed(key)` | `boolean` | True on key-down frame |\n| `wasReleased(key)` | `boolean` | True on key-up frame |\n| `activeKeys` | `Set<string>` | Currently held keys |\n| `pressedThisFrame` | `Set<string>` | Keys pressed this frame |\n| `releasedThisFrame` | `Set<string>` | Keys released this frame |\n| `lastKeyPressed` | `string` | Most recent key-down |\n| `lastKeyReleased` | `string` | Most recent key-up |\n| `shift`, `ctrl`, `alt`, `meta` | `boolean` | Modifier states |\n\n### Input: Touch: `viji.touches`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `count` | `number` | Active touch count |\n| `points` | `TouchPoint[]` | All active touches |\n| `started` | `TouchPoint[]` | Touches started this frame |\n| `moved` | `TouchPoint[]` | Touches moved this frame |\n| `ended` | `TouchPoint[]` | Touches ended this frame |\n| `primary` | `TouchPoint\\|null` | First active touch |\n\n**TouchPoint:** `id`, `x`, `y`, `pressure`, `radius`, `radiusX`, `radiusY`, `rotationAngle`, `force`, `isInCanvas`, `deltaX`, `deltaY`, `velocity` ({x,y}), `isNew`, `isActive`, `isEnding`.\n\n### Device Sensors: `viji.device`\n\n| Member | Type | Description |\n|--------|------|-------------|\n| `motion` | `DeviceMotionData\\|null` | Accelerometer/gyroscope |\n| `orientation` | `DeviceOrientationData\\|null` | Device orientation |\n\n**DeviceMotionData:** `acceleration` ({x,y,z} m/s²), `accelerationIncludingGravity`, `rotationRate` ({alpha,beta,gamma} deg/s), `interval` (ms).\n**DeviceOrientationData:** `alpha` (0-360° compass), `beta` (−180-180° tilt), `gamma` (−90-90° tilt), `absolute` (boolean).\n\n### External Devices: `viji.devices`\n\nArray of connected external devices. Each `DeviceState`:\n`id` (string), `name` (string), `motion` (DeviceMotionData|null), `orientation` (DeviceOrientationData|null), `video` (VideoAPI|null, same as viji.video but without CV), `audio` (AudioStreamAPI|null, lightweight analysis only; no beat/BPM/triggers).\n\n### Streams: `viji.videoStreams`\n\n`VideoAPI[]`: additional video sources provided by the host application (used by the compositor for scene mixing). May be empty. Each element has the same shape as `viji.video`.\n\n### Streams: `viji.audioStreams`\n\n`AudioStreamAPI[]`: additional audio sources from the host (e.g. multi-source mixing). May be empty. Lightweight interface: volume, bands, spectral features, `getFrequencyData()`, `getWaveform()`: **not** the full `AudioAPI` (no beat detection, BPM, triggers, or events).\n\n### External Libraries\n\n```javascript\nconst THREE = await import('https://esm.sh/three@0.160.0');\nconst renderer = new THREE.WebGLRenderer({ canvas: viji.canvas, antialias: true });\nrenderer.setSize(viji.width, viji.height, false); // false = no CSS styles\n```\n\nALWAYS pin library versions. ALWAYS pass `viji.canvas` to the renderer. Handle resize in `render()`.\n\n## BEST PRACTICES\n\n1. NEVER use `viji.time * speed.value`: use a `deltaTime` accumulator instead (see rule 6). Same for nested: never multiply an accumulator by another parameter.\n2. Guard audio/video with `isConnected` checks.\n3. Pre-allocate all objects/arrays at top level: never inside `render()`.\n4. For CV, use toggle parameters: never enable by default.\n5. ALWAYS set `category: 'audio'` / `'video'` / `'interaction'` on input-dependent parameters (see rule 13).\n6. For WebGL scenes with Three.js, handle resize by comparing `viji.width/height` with previous values.\n\n## TEMPLATE\n\n```javascript\nconst bgColor = viji.color('#1a1a2e', { label: 'Background' });\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nconst count = viji.slider(12, { min: 3, max: 30, step: 1, label: 'Count' });\n\nlet angle = 0;\n\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n angle += speed.value * viji.deltaTime;\n\n ctx.fillStyle = bgColor.value;\n ctx.fillRect(0, 0, viji.width, viji.height);\n\n const cx = viji.width / 2;\n const cy = viji.height / 2;\n const radius = Math.min(viji.width, viji.height) * 0.3;\n const dotSize = Math.min(viji.width, viji.height) * 0.02;\n const n = Math.floor(count.value);\n\n for (let i = 0; i < n; i++) {\n const a = angle + (i / n) * Math.PI * 2;\n const x = cx + Math.cos(a) * radius;\n const y = cy + Math.sin(a) * radius;\n const hue = (i / n) * 360;\n ctx.beginPath();\n ctx.arc(x, y, dotSize, 0, Math.PI * 2);\n ctx.fillStyle = `hsl(${hue}, 80%, 60%)`;\n ctx.fill();\n }\n}\n```\n\nNow help the artist build a Viji native scene based on their description below.\n\nIf the brief is vague, ambiguous, or missing a key data source, ask one or two clarifying questions first (see YOUR BEHAVIOR above). Otherwise, proceed.\n\nWhen you generate the scene:\n- Follow every rule in this prompt.\n- Use `viji.deltaTime` for animation. Use parameters for anything the user might want to adjust. Check `isConnected` before using audio or video.\n- Output the scene code in a single fenced code block.\n- After the code block, write a short explanation (a few sentences) of how the scene works and what the artist can tweak.\n- Invite the artist to ask for changes.\n````\n\n## Usage\n\n1. Copy the entire prompt block above.\n2. Paste it into your AI assistant (ChatGPT, Claude, etc.).\n3. After the prompt, describe the scene you want: be as specific as you like.\n4. The AI will return a complete Viji native scene.\n\n> [!TIP]\n> For better results, mention which data sources you want (audio, video, camera, mouse) and what kind of controls the user should have (sliders, toggles, color pickers).\n\n## Related\n\n- [Create Your First Scene](/ai-prompts/create-first-scene): guided prompt for beginners\n- [Prompting Tips](/ai-prompts/prompting-tips): how to get better results from AI\n- [Native Quick Start](/native/quickstart): your first Viji native scene\n- [Native API Reference](/native/api-reference): full API reference\n- [Best Practices](/getting-started/best-practices): essential patterns for reliable scenes\n- [Common Mistakes](/getting-started/common-mistakes): pitfalls to avoid"
|
|
1328
|
+
"markdown": "# Prompt: Native Scenes\r\n\r\nCopy the prompt below and paste it into your AI assistant. Then describe the scene you want. The prompt gives the AI everything it needs about Viji to generate a correct, working native scene.\r\n\r\n## The Prompt\r\n\r\n````\r\nYou are generating a Viji native scene: a creative visual that runs inside an OffscreenCanvas Web Worker.\r\nArtists describe what they want; you collaborate with them to produce complete, working scene code. Apply every rule below exactly.\r\n\r\n## YOUR BEHAVIOR\r\n\r\n1. **Clarify when needed.** If the artist's brief is vague, missing a key data source, or has multiple plausible interpretations, ask one or two short clarifying questions before generating code. Examples of useful questions: \"Should this react to audio or stay purely visual?\", \"Should it use the camera or only mouse input?\", \"Roughly how dense / how minimal do you want it?\". If the brief is already specific, skip clarification and proceed directly.\r\n2. **Generate.** Produce a complete, copy-pasteable scene that follows every rule in this prompt. Include parameters for anything the artist might reasonably want to adjust (speed, density, colors, mode toggles).\r\n3. **Explain.** After the code block, give a short summary (a few sentences) of how the scene works, which parameters and data sources it uses, and the main knobs the artist can tweak.\r\n4. **Iterate.** Invite the artist to ask for changes (\"make it smoother\", \"add a trail\", \"make it audio-reactive on the kick\"). Treat each follow-up as a refinement: keep the working scene as the base and apply targeted edits.\r\n\r\n## REFERENCE (source of truth)\r\n\r\nThe resources below are AUTHORITATIVE. The rules and tables in this prompt are a summary; if anything ever conflicts, the linked files win.\r\n\r\n**If you have web/file access:**\r\n- REQUIRED before generating code: fetch and skim the Tier-1 resource. Use it to verify exact API names and types.\r\n- ON DEMAND: fetch from the Tier-2 resource when the artist requests something not fully covered by the rules and tables in this prompt (advanced CV data structures, behavior nuances, full examples).\r\n\r\n**If you do NOT have web/file access:**\r\n- Use only the API surface explicitly named in this prompt.\r\n- Never invent property, method, or uniform names from memory.\r\n- If the artist asks for something not covered here, say so and ask the artist what they want; do NOT fabricate.\r\n\r\n**Tier 1 (always consult when accessible):**\r\n- TypeScript API types (Viji JS surface): https://unpkg.com/@viji-dev/core/dist/artist-global.d.ts\r\n\r\n**Tier 2 (consult when needed):**\r\n- Complete docs (every page + every live example): https://unpkg.com/@viji-dev/core/dist/docs-api.js\r\n\r\n**Last-resort lookup:** https://www.npmjs.com/package/@viji-dev/core\r\n\r\n## ARCHITECTURE\r\n\r\n- Scenes run in a **Web Worker** with an **OffscreenCanvas**. There is no DOM.\r\n- The global `viji` object provides canvas, timing, audio, video, CV, input, sensors, and parameters.\r\n- **Top-level code** runs once (initialization, parameter declarations, state, imports). Top-level `await` is supported for dynamic imports.\r\n- **`function render(viji) { ... }`** is called every frame. This is where you draw.\r\n- There is **no `setup()` function** in native scenes. All initialization goes at the top level.\r\n\r\n## RULES\r\n\r\n1. NEVER access `window`, `document`, `Image()`, `localStorage`, or any DOM API. `fetch()` and `await import()` ARE available.\r\n2. ALWAYS declare parameters at the TOP LEVEL, never inside `render()`:\r\n ```javascript\r\n const speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\r\n function render(viji) { /* use speed.value */ }\r\n ```\r\n3. ALWAYS read parameters via `.value`: `speed.value`, `color.value`, `toggle.value`. Color parameters also expose `.rgb` (`{ r, g, b }` in 0..255) and `.hsb` (`{ h, s, b }`, h in 0..360, s/b in 0..100). Use them instead of parsing hex by hand. Color defaults accept hex (`'#ff6600'`, `'#f60'`), `{ r, g, b }` (0..255), `{ h, s, b }` (0..360 / 0..100), and CSS `'rgb(...)'` / `'hsl(...)'` strings.\r\n4. ALWAYS use `viji.width` and `viji.height` for canvas dimensions. NEVER hardcode pixel sizes.\r\n5. ALWAYS use `viji.time` or `viji.deltaTime` for animation. NEVER count frames or assume a fixed frame rate.\r\n - `viji.time`: elapsed seconds. Use for constant-speed oscillations only: `sin(viji.time * 2.0)`.\r\n - `viji.deltaTime`: seconds since last frame. Use for accumulators: `angle += speed.value * viji.deltaTime;`\r\n6. NEVER multiply `viji.time` by a parameter for animation speed: it causes jumps when the parameter changes. ALWAYS use a `deltaTime` accumulator:\r\n ```javascript\r\n // WRONG: jumps when speed changes:\r\n const t = viji.time * speed.value;\r\n // RIGHT: smooth:\r\n let phase = 0; // top level\r\n phase += speed.value * viji.deltaTime; // in render()\r\n ```\r\n This also applies to **nested** multiplications. If `phase` is already an accumulator, NEVER multiply it by another parameter:\r\n ```javascript\r\n // WRONG: jumps when rotSpeed changes:\r\n const rot = phase * rotSpeed.value;\r\n // RIGHT: give it its own accumulator:\r\n let rotPhase = 0; // top level\r\n rotPhase += speed.value * rotSpeed.value * viji.deltaTime; // in render()\r\n ```\r\n7. NEVER allocate objects, arrays, or strings inside `render()`. Pre-allocate at the top level and reuse.\r\n8. ALWAYS call `viji.useContext()` to get a canvas context. Choose ONE type and use it for the entire scene:\r\n - `viji.useContext('2d')`: Canvas 2D\r\n - `viji.useContext('webgl')`: WebGL 1\r\n - `viji.useContext('webgl2')`: WebGL 2\r\n - `viji.useContext('webgpu')`: WebGPU\r\n Calling a different type after the first returns `null`.\r\n9. ALWAYS check `viji.audio.isConnected` before using audio data.\r\n10. ALWAYS check `viji.video.isConnected && viji.video.currentFrame` before drawing video.\r\n11. NEVER enable CV features by default. Use a toggle parameter so the user can opt in:\r\n ```javascript\r\n const useFace = viji.toggle(false, { label: 'Enable Face Detection', category: 'video' });\r\n // In render:\r\n if (useFace.value) await viji.video.cv.enableFaceDetection(true);\r\n else await viji.video.cv.enableFaceDetection(false);\r\n ```\r\n12. Be mindful of WebGL context limits: each CV feature uses its own WebGL context for ML. Enabling too many can cause context loss.\r\n13. ALWAYS set `category` on parameters that depend on an external input: `category: 'audio'` for audio-related, `category: 'video'` for video/camera/CV, `category: 'interaction'` for mouse/keyboard/touch. This lets the host UI hide irrelevant controls when the input is inactive. **Use creative-strength sliders, not on/off toggles**: the host UI already controls whether each input is wired up, so a scene-level `toggle(true, { label: 'Audio Reactive' })` just duplicates the host switch. CV feature toggles (`enableFaceDetection`, etc.) are the exception and stay opt-in.\r\n ```javascript\r\n // Right: creative-strength sliders with the matching category.\r\n const bassSensitivity = viji.slider(1.5, { min: 0, max: 3, label: 'Bass Sensitivity', group: 'audio', category: 'audio' });\r\n const mouseAttraction = viji.slider(0.5, { min: 0, max: 1, label: 'Mouse Attraction', group: 'interaction', category: 'interaction' });\r\n\r\n // Wrong: scene-level on/off toggle for an input the host already gates.\r\n // const audioReact = viji.toggle(true, { label: 'Audio Reactive', category: 'audio' });\r\n // const followMouse = viji.toggle(true, { label: 'Follow Mouse', category: 'interaction' });\r\n ```\r\n14. ALWAYS type any function parameter that receives the `viji` instance, including `render(viji)` and any helpers that take `viji`. Monaco's TS checker flags API misuse (wrong nesting, typo'd property names) at edit time so errors can be corrected before runtime. Scene code is transpiled via sucrase before execution (`transforms: ['typescript']`), so TS annotations are stripped at runtime — they are safe to include.\r\n ```javascript\r\n function render(viji: VijiAPI) { ... }\r\n function applyGlow(viji: VijiAPI, intensity: number) { ... }\r\n ```\r\n All Viji types are ambient globals: `VijiAPI`, the parameter and config types (`SliderParameter`, `SliderConfig`, ...), input and CV types (`TouchPoint`, `FaceData`, ...). Annotate with them directly; never import, alias, or re-derive them (no `ReturnType<typeof viji.slider>`; the named type exists). The [API Reference](/native/api-reference) lists every `viji` member with its exact type.\r\n15. For external libraries, use dynamic import with a pinned version:\r\n ```javascript\r\n const THREE = await import('https://esm.sh/three@0.160.0');\r\n ```\r\n Pass `viji.canvas` to the library's renderer. ALWAYS pass `false` as the third argument to Three.js `setSize()`.\r\n\r\n## COMPLETE API REFERENCE\r\n\r\n### Canvas & Context\r\n\r\n| Member | Type | Description |\r\n|--------|------|-------------|\r\n| `viji.canvas` | `OffscreenCanvas` | The canvas element |\r\n| `viji.useContext('2d')` | `OffscreenCanvasRenderingContext2D` | Get 2D context |\r\n| `viji.useContext('webgl')` | `WebGLRenderingContext` | Get WebGL 1 context |\r\n| `viji.useContext('webgl2')` | `WebGL2RenderingContext` | Get WebGL 2 context |\r\n| `viji.useContext('webgpu')` | `GPUCanvasContext \\| null` | Get WebGPU context (null when unsupported) |\r\n| `viji.ctx` | `OffscreenCanvasRenderingContext2D` | Shortcut (after useContext('2d')) |\r\n| `viji.gl` | `WebGLRenderingContext` | Shortcut (after useContext('webgl')) |\r\n| `viji.gpu` | `GPUCanvasContext` | Shortcut (after useContext('webgpu')) |\r\n| `viji.width` | `number` | Current canvas width in pixels |\r\n| `viji.height` | `number` | Current canvas height in pixels |\r\n\r\n### Timing\r\n\r\n| Member | Type | Description |\r\n|--------|------|-------------|\r\n| `viji.time` | `number` | Seconds since scene start |\r\n| `viji.deltaTime` | `number` | Seconds since last frame |\r\n| `viji.frameCount` | `number` | Total frames rendered |\r\n| `viji.fps` | `number` | Target frame rate (based on host frame-rate mode) |\r\n\r\n### Parameters\r\n\r\nDeclare at top level. Read `.value` inside `render()`. All support `{ label, description?, group?, category? }`.\r\nCategory values: `'audio'`, `'video'`, `'interaction'`, `'general'`.\r\n\r\n```javascript\r\nviji.slider(default, { min?, max?, step?, label, group?, category? }) // { value: number }\r\nviji.color(default, { label, group?, category? }) // { value: '#rrggbb', rgb: { r, g, b } in 0..255, hsb: { h: 0..360, s/b: 0..100 } }\r\nviji.toggle(default, { label, group?, category? }) // { value: boolean }\r\nviji.select(default, { options: [...], label, group?, category? }) // { value: string|number }\r\nviji.number(default, { min?, max?, step?, label, group?, category? }) // { value: number }\r\nviji.text(default, { label, group?, category?, maxLength? }) // { value: string }\r\nviji.image(null, { label, group?, category? }) // { value: ImageBitmap|null }\r\nviji.button({ label, description?, group?, category? }) // { value: boolean } (true one frame)\r\nviji.coordinate(default, { step?, label, group?, category? }) // { value: { x, y } } (both -1 to 1)\r\n// slider/number `step` defaults to a power of 10 giving ~100 positions across min..max\r\n// (e.g., range 0..1 → 0.01, range 0..100 → 1). Specify `step: 1` for integer counts.\r\n```\r\n\r\n### Audio: `viji.audio`\r\n\r\nALWAYS check `viji.audio.isConnected` first.\r\n\r\n| Member | Type | Description |\r\n|--------|------|-------------|\r\n| `isConnected` | `boolean` | Whether audio source is active |\r\n| `volume.current` | `number` | RMS volume 0-1 |\r\n| `volume.peak` | `number` | Peak amplitude 0-1 |\r\n| `volume.smoothed` | `number` | Smoothed volume (200ms decay) |\r\n| `bands.low` | `number` | 20-120 Hz energy 0-1 |\r\n| `bands.lowMid` | `number` | 120-400 Hz energy 0-1 |\r\n| `bands.mid` | `number` | 400-1600 Hz energy 0-1 |\r\n| `bands.highMid` | `number` | 1600-6000 Hz energy 0-1 |\r\n| `bands.high` | `number` | 6000-16000 Hz energy 0-1 |\r\n| `bands.lowSmoothed` … `bands.highSmoothed` | `number` | Smoothed variants of each band |\r\n| `beat.kick` | `number` | Kick energy curve 0-1, 300ms decay; peaks on each detected kick |\r\n| `beat.snare` | `number` | Snare energy curve 0-1, 300ms decay |\r\n| `beat.hat` | `number` | Hi-hat energy curve 0-1, 300ms decay |\r\n| `beat.any` | `number` | Any-beat energy curve 0-1, 300ms decay |\r\n| `beat.kickSmoothed` … `beat.anySmoothed` | `number` | Smoother 500ms decay envelopes; use for ambient pulses |\r\n| `beat.triggers.kick` | `boolean` | True for exactly one frame on a kick, then auto-resets; OR-accumulated between frames |\r\n| `beat.triggers.snare` | `boolean` | True for exactly one frame on a snare, then auto-resets; OR-accumulated between frames |\r\n| `beat.triggers.hat` | `boolean` | True for exactly one frame on a hi-hat, then auto-resets; OR-accumulated between frames |\r\n| `beat.triggers.any` | `boolean` | True for exactly one frame on any beat, then auto-resets |\r\n| `beat.events` | `Array<{type,time,strength}>` | All beats detected since the last frame; `type` is `'kick' \\| 'snare' \\| 'hat'` (never `'any'`); `time` in ms; cleared each frame |\r\n| `beat.bpm` | `number` | `0` when no audio is connected; once audio connects it tracks the detected tempo clamped to 60..240, with `120` as a fallback before lock-on |\r\n| `beat.confidence` | `number` | BPM tracking confidence 0-1 |\r\n| `beat.isLocked` | `boolean` | True on stable tempo lock |\r\n| `spectral.brightness` | `number` | Spectral centroid 0-1 |\r\n| `spectral.flatness` | `number` | Spectral flatness 0-1 |\r\n| `getFrequencyData()` | `Uint8Array` | Raw FFT bins (0-255) |\r\n| `getWaveform()` | `Float32Array` | Time-domain waveform (−1 to 1) |\r\n\r\n**`device.audio`** (when an external device in `viji.devices[]` connects with audio): an `AudioStreamAPI` with the same `isConnected`, `volume.{current,peak,smoothed}`, `bands.{low...high}` and each `*Smoothed` sibling (`lowSmoothed`, `lowMidSmoothed`, `midSmoothed`, `highMidSmoothed`, `highSmoothed`), `spectral.{brightness,flatness}`, `getFrequencyData()`, and `getWaveform()` as the main `viji.audio` table. **No** `beat`, BPM, triggers, or events: those are main-audio only. Host-supplied additional audio sources (`viji.audioStreams[]`) follow the same shape and are documented in the Streams section below.\r\n\r\n### Video: `viji.video`\r\n\r\nALWAYS check `viji.video.isConnected` first. Check `currentFrame` before drawing.\r\n\r\n| Member | Type | Description |\r\n|--------|------|-------------|\r\n| `isConnected` | `boolean` | Whether video source is active |\r\n| `currentFrame` | `OffscreenCanvas\\|ImageBitmap\\|null` | Just-arrived video frame |\r\n| `frameWidth` | `number` | Frame width in pixels |\r\n| `frameHeight` | `number` | Frame height in pixels |\r\n| `frameRate` | `number` | Video frame rate |\r\n| `getFrameData()` | `ImageData\\|null` | Pixel data of `currentFrame` for CPU access |\r\n| `cv` | `VideoCVAPI` | Computer-vision surface (see below) |\r\n\r\nCV-paired outputs (`analysedFrame`, `getAnalysedFrameData()`) and detection results (`faces`, `hands`, `pose`, `segmentation`) live on `viji.video.cv`, not on `viji.video` directly. See the Computer Vision section below.\r\n\r\n**Drawing video: preserve aspect ratio.** Camera frames almost never match the canvas aspect. Drawing to `(0, 0, viji.width, viji.height)` stretches the video and misaligns CV bounds. Define this `videoFit` helper at module scope and use it for every video / CV scene:\r\n\r\n```javascript\r\nfunction videoFit(viji, mode = 'cover') {\r\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\r\n const w = viji.width, h = viji.height;\r\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\r\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\r\n const dw = vw * scale, dh = vh * scale;\r\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\r\n}\r\n\r\nconst v = videoFit(viji); // 'cover' (default) or 'contain'\r\nctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\r\n\r\n// CV coords are normalized 0-1 to the source video frame, not the canvas.\r\n// Map them through v to align with the fitted video:\r\nviji.video.cv.faces.forEach(face => {\r\n const bx = v.x + face.bounds.x * v.width;\r\n const by = v.y + face.bounds.y * v.height;\r\n const bw = face.bounds.width * v.width;\r\n const bh = face.bounds.height * v.height;\r\n ctx.strokeRect(bx, by, bw, bh);\r\n});\r\n```\r\n\r\nDefault to `'cover'` for live camera feeds (fills the canvas, edges cropped). Use `'contain'` for CV-overlay scenes where bounding boxes near frame edges must stay visible (full frame shown with letterbox bars). Stretching directly to `(0, 0, viji.width, viji.height)` is allowed only when distortion is intentional.\r\n\r\n**`currentFrame` vs `analysedFrame`.** Default to `currentFrame` for displayed video. Reach for `viji.video.cv.analysedFrame ?? viji.video.currentFrame` only when the effect reads pixels from the displayed frame at CV-derived positions (compositing the segmentation mask onto the body, sampling skin under a face landmark, warping the face along its mesh, texture-mapped face filters). For drawing landmark dots, particles, or any overlay that doesn't sample the displayed frame at CV positions, `currentFrame` is the better default: `analysedFrame` advances only when MediaPipe completes an inference, so reaching for it without a reason makes the displayed video stutter or hold between inferences.\r\n\r\n### Computer Vision: `viji.video.cv` & `viji.video.cv.faces/hands/pose/segmentation`\r\n\r\nEach CV feature is independent and populates only its own subset of fields. Each active feature uses its own WebGL context for MediaPipe — enable only what you need (4+ on low-end hardware risks context-limit failures).\r\n\r\n```javascript\r\n// Verbs return Promise<void>. await is optional. Safe to call from module\r\n// scope (always-on CV) or per-frame inside render() gated by a viji.toggle(...)\r\n// (opt-in CV). Idempotent + reference-counted.\r\nawait viji.video.cv.enableFaceDetection(true/false); // populates face.bounds, face.center, face.confidence\r\nawait viji.video.cv.enableFaceMesh(true/false); // populates face.landmarks (468 pts) + face.headPose\r\nawait viji.video.cv.enableEmotionDetection(true/false); // populates face.expressions + face.blendshapes; also loads landmarker (face.landmarks + face.headPose populated)\r\nawait viji.video.cv.enableHandTracking(true/false); // populates viji.video.cv.hands[]\r\nawait viji.video.cv.enablePoseDetection(true/false); // populates viji.video.cv.pose\r\nawait viji.video.cv.enableBodySegmentation(true/false); // populates viji.video.cv.segmentation\r\nviji.video.cv.getActiveFeatures(); // CVFeature[]\r\nviji.video.cv.isProcessing(); // boolean\r\n```\r\n\r\nCV-paired outputs (also on `viji.video.cv`):\r\n- `analysedFrame: OffscreenCanvas | null`: the exact frame that produced the current CV results. `null` until the first inference lands. Common fallback pattern: `viji.video.cv.analysedFrame ?? viji.video.currentFrame`.\r\n- `getAnalysedFrameData(): ImageData | null`: pixel data of `analysedFrame`.\r\n\r\n**`viji.video.cv.faces: FaceData[]`**\r\n`id` always present. Other fields populated only by their source model and read `null` (or `[]` for `landmarks`) otherwise — always null-check.\r\n- `bounds: {x,y,width,height} | null` (normalized 0..1) — populated by face detection\r\n- `center: {x,y} | null` (normalized 0..1) — populated by face detection\r\n- `confidence: number | null` (0..1) — populated by face detection\r\n- `landmarks: {x,y,z?}[]` (empty `[]` unless face mesh enabled; 468 pts when populated)\r\n- `headPose: {pitch, yaw, roll} | null` (degrees) — populated by face mesh\r\n- `expressions: {neutral, happy, sad, angry, surprised, disgusted, fearful} | null` (0..1 each) — populated by emotion detection\r\n- `blendshapes | null` (52 ARKit coefficients 0..1) — populated by emotion detection. Names: browDownLeft/Right, browInnerUp, browOuterUpLeft/Right, cheekPuff, cheekSquintLeft/Right, eyeBlinkLeft/Right, eyeLookDownLeft/Right, eyeLookInLeft/Right, eyeLookOutLeft/Right, eyeLookUpLeft/Right, eyeSquintLeft/Right, eyeWideLeft/Right, jawForward/Left/Open/Right, mouthClose/DimpleLeft/Right/FrownLeft/Right/Funnel/Left/LowerDownLeft/Right/PressLeft/Right/Pucker/Right/RollLower/Upper/ShrugLower/Upper/SmileLeft/Right/StretchLeft/Right/UpperUpLeft/Right, noseSneerLeft/Right, tongueOut.\r\n\r\nIf you need `bounds` while only running face mesh, either also enable face detection or compute it from `face.landmarks` min/max.\r\n\r\n**`viji.video.cv.hands: HandData[]`**\r\nEach hand: `id` (number), `handedness` ('left'|'right'), `confidence` (0-1), `bounds` ({x,y,width,height}), `landmarks` ({x,y,z}[], 21 points), `palm` ({x,y,z}), `gestures` ({fist,openPalm,peace,thumbsUp,thumbsDown,pointing,iLoveYou} all 0-1 confidence).\r\n\r\n**`viji.video.cv.pose: PoseData | null`**\r\n`confidence` (0-1), `landmarks` ({x,y,z,visibility}[], 33 points), plus body-part arrays: `face` ({x,y}[]), `torso`, `leftArm`, `rightArm`, `leftLeg`, `rightLeg`.\r\n\r\n**`viji.video.cv.segmentation: SegmentationData | null`**\r\n`mask` (Uint8Array; each byte is `0` for background or `1` for person; length = `width * height`), `width`, `height`.\r\n\r\n### Input: Pointer (unified mouse/touch): `viji.pointer`\r\n\r\n| Member | Type | Description |\r\n|--------|------|-------------|\r\n| `x`, `y` | `number` | Position in pixels |\r\n| `deltaX`, `deltaY` | `number` | Movement since last frame |\r\n| `isDown` | `boolean` | True if pressed/touching |\r\n| `wasPressed` | `boolean` | True on press frame |\r\n| `wasReleased` | `boolean` | True on release frame |\r\n| `isInCanvas` | `boolean` | True if inside canvas |\r\n| `type` | `string` | `'mouse'`, `'touch'`, or `'none'` |\r\n\r\n### Input: Mouse: `viji.mouse`\r\n\r\n| Member | Type | Description |\r\n|--------|------|-------------|\r\n| `x`, `y` | `number` | Position in pixels |\r\n| `isInCanvas` | `boolean` | Inside canvas bounds |\r\n| `isPressed` | `boolean` | Any button pressed |\r\n| `leftButton`, `rightButton`, `middleButton` | `boolean` | Specific buttons |\r\n| `deltaX`, `deltaY` | `number` | Movement delta |\r\n| `wheelDelta` | `number` | Scroll wheel delta |\r\n| `wheelX`, `wheelY` | `number` | Horizontal/vertical scroll |\r\n| `wasPressed`, `wasReleased`, `wasMoved` | `boolean` | Frame-edge events |\r\n\r\n### Input: Keyboard: `viji.keyboard`\r\n\r\n| Member | Type | Description |\r\n|--------|------|-------------|\r\n| `isPressed(key)` | `boolean` | True while key is held |\r\n| `wasPressed(key)` | `boolean` | True on key-down frame |\r\n| `wasReleased(key)` | `boolean` | True on key-up frame |\r\n| `activeKeys` | `Set<string>` | Currently held keys |\r\n| `pressedThisFrame` | `Set<string>` | Keys pressed this frame |\r\n| `releasedThisFrame` | `Set<string>` | Keys released this frame |\r\n| `lastKeyPressed` | `string` | Most recent key-down |\r\n| `lastKeyReleased` | `string` | Most recent key-up |\r\n| `shift`, `ctrl`, `alt`, `meta` | `boolean` | Modifier states |\r\n\r\n### Input: Touch: `viji.touches`\r\n\r\n| Member | Type | Description |\r\n|--------|------|-------------|\r\n| `count` | `number` | Active touch count |\r\n| `points` | `TouchPoint[]` | All active touches |\r\n| `started` | `TouchPoint[]` | Touches started this frame |\r\n| `moved` | `TouchPoint[]` | Touches moved this frame |\r\n| `ended` | `TouchPoint[]` | Touches ended this frame |\r\n| `primary` | `TouchPoint\\|null` | First active touch |\r\n\r\n**TouchPoint:** `id`, `x`, `y`, `pressure`, `radius`, `radiusX`, `radiusY`, `rotationAngle`, `force`, `isInCanvas`, `deltaX`, `deltaY`, `velocity` ({x,y}), `isNew`, `isActive`, `isEnding`.\r\n\r\n### Device Sensors: `viji.device`\r\n\r\n| Member | Type | Description |\r\n|--------|------|-------------|\r\n| `motion` | `DeviceMotionData\\|null` | Accelerometer/gyroscope |\r\n| `orientation` | `DeviceOrientationData\\|null` | Device orientation |\r\n\r\n**DeviceMotionData:** `acceleration` ({x,y,z} m/s²), `accelerationIncludingGravity`, `rotationRate` ({alpha,beta,gamma} deg/s), `interval` (ms).\r\n**DeviceOrientationData:** `alpha` (0-360° compass), `beta` (−180-180° tilt), `gamma` (−90-90° tilt), `absolute` (boolean).\r\n\r\n### External Devices: `viji.devices`\r\n\r\nArray of connected external devices. Each `DeviceState`:\r\n`id` (string), `name` (string), `motion` (DeviceMotionData|null), `orientation` (DeviceOrientationData|null), `video` (VideoAPI|null, same as viji.video but without CV), `audio` (AudioStreamAPI|null, lightweight analysis only; no beat/BPM/triggers).\r\n\r\n### Streams: `viji.videoStreams`\r\n\r\n`VideoAPI[]`: additional video sources provided by the host application (used by the compositor for scene mixing). May be empty. Each element has the same shape as `viji.video`.\r\n\r\n### Streams: `viji.audioStreams`\r\n\r\n`AudioStreamAPI[]`: additional audio sources from the host (e.g. multi-source mixing). May be empty. Lightweight interface: volume, bands, spectral features, `getFrequencyData()`, `getWaveform()`: **not** the full `AudioAPI` (no beat detection, BPM, triggers, or events).\r\n\r\n### External Libraries\r\n\r\n```javascript\r\nconst THREE = await import('https://esm.sh/three@0.160.0');\r\nconst renderer = new THREE.WebGLRenderer({ canvas: viji.canvas, antialias: true });\r\nrenderer.setSize(viji.width, viji.height, false); // false = no CSS styles\r\n```\r\n\r\nALWAYS pin library versions. ALWAYS pass `viji.canvas` to the renderer. Handle resize in `render()`.\r\n\r\n## BEST PRACTICES\r\n\r\n1. NEVER use `viji.time * speed.value`: use a `deltaTime` accumulator instead (see rule 6). Same for nested: never multiply an accumulator by another parameter.\r\n2. Guard audio/video with `isConnected` checks.\r\n3. Pre-allocate all objects/arrays at top level: never inside `render()`.\r\n4. For CV, use toggle parameters: never enable by default.\r\n5. ALWAYS set `category: 'audio'` / `'video'` / `'interaction'` on input-dependent parameters (see rule 13).\r\n6. For WebGL scenes with Three.js, handle resize by comparing `viji.width/height` with previous values.\r\n\r\n## TEMPLATE\r\n\r\n```javascript\r\nconst bgColor = viji.color('#1a1a2e', { label: 'Background' });\r\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\r\nconst count = viji.slider(12, { min: 3, max: 30, step: 1, label: 'Count' });\r\n\r\nlet angle = 0;\r\n\r\nfunction render(viji) {\r\n const ctx = viji.useContext('2d');\r\n angle += speed.value * viji.deltaTime;\r\n\r\n ctx.fillStyle = bgColor.value;\r\n ctx.fillRect(0, 0, viji.width, viji.height);\r\n\r\n const cx = viji.width / 2;\r\n const cy = viji.height / 2;\r\n const radius = Math.min(viji.width, viji.height) * 0.3;\r\n const dotSize = Math.min(viji.width, viji.height) * 0.02;\r\n const n = Math.floor(count.value);\r\n\r\n for (let i = 0; i < n; i++) {\r\n const a = angle + (i / n) * Math.PI * 2;\r\n const x = cx + Math.cos(a) * radius;\r\n const y = cy + Math.sin(a) * radius;\r\n const hue = (i / n) * 360;\r\n ctx.beginPath();\r\n ctx.arc(x, y, dotSize, 0, Math.PI * 2);\r\n ctx.fillStyle = `hsl(${hue}, 80%, 60%)`;\r\n ctx.fill();\r\n }\r\n}\r\n```\r\n\r\nNow help the artist build a Viji native scene based on their description below.\r\n\r\nIf the brief is vague, ambiguous, or missing a key data source, ask one or two clarifying questions first (see YOUR BEHAVIOR above). Otherwise, proceed.\r\n\r\nWhen you generate the scene:\r\n- Follow every rule in this prompt.\r\n- Use `viji.deltaTime` for animation. Use parameters for anything the user might want to adjust. Check `isConnected` before using audio or video.\r\n- Output the scene code in a single fenced code block.\r\n- After the code block, write a short explanation (a few sentences) of how the scene works and what the artist can tweak.\r\n- Invite the artist to ask for changes.\r\n````\r\n\r\n## Usage\r\n\r\n1. Copy the entire prompt block above.\r\n2. Paste it into your AI assistant (ChatGPT, Claude, etc.).\r\n3. After the prompt, describe the scene you want: be as specific as you like.\r\n4. The AI will return a complete Viji native scene.\r\n\r\n> [!TIP]\r\n> For better results, mention which data sources you want (audio, video, camera, mouse) and what kind of controls the user should have (sliders, toggles, color pickers).\r\n\r\n## Related\r\n\r\n- [Create Your First Scene](/ai-prompts/create-first-scene): guided prompt for beginners\r\n- [Prompting Tips](/ai-prompts/prompting-tips): how to get better results from AI\r\n- [Native Quick Start](/native/quickstart): your first Viji native scene\r\n- [Native API Reference](/native/api-reference): full API reference\r\n- [Best Practices](/getting-started/best-practices): essential patterns for reliable scenes\r\n- [Common Mistakes](/getting-started/common-mistakes): pitfalls to avoid"
|
|
1329
1329
|
}
|
|
1330
1330
|
]
|
|
1331
1331
|
},
|
|
@@ -1705,7 +1705,7 @@ export const docsApi = {
|
|
|
1705
1705
|
},
|
|
1706
1706
|
"native-canvas": {
|
|
1707
1707
|
"id": "native-canvas",
|
|
1708
|
-
"title": "
|
|
1708
|
+
"title": "native-canvas",
|
|
1709
1709
|
"description": "Access the canvas, choose a rendering context, and work with resolution-agnostic dimensions.",
|
|
1710
1710
|
"headings": [
|
|
1711
1711
|
{
|
|
@@ -1738,6 +1738,16 @@ export const docsApi = {
|
|
|
1738
1738
|
"level": 3,
|
|
1739
1739
|
"text": "Shorthand: viji.gl"
|
|
1740
1740
|
},
|
|
1741
|
+
{
|
|
1742
|
+
"id": "webgpu-context",
|
|
1743
|
+
"level": 2,
|
|
1744
|
+
"text": "WebGPU Context"
|
|
1745
|
+
},
|
|
1746
|
+
{
|
|
1747
|
+
"id": "shorthand-vijigpu",
|
|
1748
|
+
"level": 3,
|
|
1749
|
+
"text": "Shorthand: viji.gpu"
|
|
1750
|
+
},
|
|
1741
1751
|
{
|
|
1742
1752
|
"id": "resolution-agnostic-drawing",
|
|
1743
1753
|
"level": 2,
|
|
@@ -1777,7 +1787,7 @@ export const docsApi = {
|
|
|
1777
1787
|
"content": [
|
|
1778
1788
|
{
|
|
1779
1789
|
"type": "text",
|
|
1780
|
-
"markdown": "# Canvas & Context\n\nEvery native scene gets a canvas and a set of properties for resolution-agnostic drawing. This page covers `viji.canvas`, `viji.useContext()`, `viji.width`, `viji.height`, `viji.ctx`, and `viji.gl`.\n\n## The Canvas\n\n`viji.canvas` is an [`OffscreenCanvas`](https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCanvas): not a DOM `<canvas>` element. It lives inside a Web Worker and has no associated DOM node. You never create or resize it yourself; the host application manages its lifecycle and dimensions.\n\n| Property | Type | Description |\n|----------|------|-------------|\n| `viji.canvas` | `OffscreenCanvas` | The raw canvas. Rarely needed: use `viji.useContext()` instead. |\n| `viji.width` | `number` | Current canvas width in pixels |\n| `viji.height` | `number` | Current canvas height in pixels |\n\n`viji.width` and `viji.height` always reflect the actual pixel dimensions of the canvas and update automatically when the host resizes it. Use them for all positioning and sizing.\n\n## Choosing a Context\n\nCall `viji.useContext()` to get a rendering context:\n\n```javascript\nconst ctx
|
|
1790
|
+
"markdown": "# Canvas & Context\r\n\r\nEvery native scene gets a canvas and a set of properties for resolution-agnostic drawing. This page covers `viji.canvas`, `viji.useContext()`, `viji.width`, `viji.height`, `viji.ctx`, and `viji.gl`.\r\n\r\n## The Canvas\r\n\r\n`viji.canvas` is an [`OffscreenCanvas`](https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCanvas): not a DOM `<canvas>` element. It lives inside a Web Worker and has no associated DOM node. You never create or resize it yourself; the host application manages its lifecycle and dimensions.\r\n\r\n| Property | Type | Description |\r\n|----------|------|-------------|\r\n| `viji.canvas` | `OffscreenCanvas` | The raw canvas. Rarely needed: use `viji.useContext()` instead. |\r\n| `viji.width` | `number` | Current canvas width in pixels |\r\n| `viji.height` | `number` | Current canvas height in pixels |\r\n\r\n`viji.width` and `viji.height` always reflect the actual pixel dimensions of the canvas and update automatically when the host resizes it. Use them for all positioning and sizing.\r\n\r\n## Choosing a Context\r\n\r\nCall `viji.useContext()` to get a rendering context:\r\n\r\n```javascript\r\nconst ctx = viji.useContext('2d'); // OffscreenCanvasRenderingContext2D\r\nconst gl = viji.useContext('webgl'); // WebGLRenderingContext\r\nconst gl2 = viji.useContext('webgl2'); // WebGL2RenderingContext\r\nconst gctx = viji.useContext('webgpu'); // GPUCanvasContext | null\r\n```\r\n\r\n| Argument | Return Type | Also Sets |\r\n|----------|------------|-----------|\r\n| `'2d'` | `OffscreenCanvasRenderingContext2D` | `viji.ctx` |\r\n| `'webgl'` | `WebGLRenderingContext` | `viji.gl` |\r\n| `'webgl2'` | `WebGL2RenderingContext` | `viji.gl` |\r\n| `'webgpu'` | `GPUCanvasContext \\| null` | `viji.gpu` |\r\n\r\nThe context is created once and cached: calling `useContext()` again with the same argument returns the same instance. Call it at the top of `render()` or once at the top level and store the result.\r\n\r\n> [!WARNING]\r\n> A canvas only supports one context type. If you call `useContext('2d')` and later call `useContext('webgl')` (or vice versa), the second call returns `null`. Choose one context type and use it for the entire scene.\r\n\r\n## 2D Context\r\n\r\nThe most common choice. You get a standard [`CanvasRenderingContext2D`](https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D) with full access to paths, fills, strokes, transforms, gradients, and compositing."
|
|
1781
1791
|
},
|
|
1782
1792
|
{
|
|
1783
1793
|
"type": "live-example",
|
|
@@ -1787,7 +1797,7 @@ export const docsApi = {
|
|
|
1787
1797
|
},
|
|
1788
1798
|
{
|
|
1789
1799
|
"type": "text",
|
|
1790
|
-
"markdown": "### Shorthand: `viji.ctx`\n\nAfter calling `viji.useContext('2d')`, the same context is also available as `viji.ctx`:\n\n```javascript\nfunction render(viji) {\n viji.useContext('2d');\n viji.ctx.fillStyle = '#000';\n viji.ctx.fillRect(0, 0, viji.width, viji.height);\n}\n```\n\nBoth `viji.useContext('2d')` and `viji.ctx` reference the same object.\n\n## WebGL Context\n\nFor GPU-accelerated rendering, request a WebGL context. The returned object is a standard [`WebGLRenderingContext`](https://developer.mozilla.org/en-US/docs/Web/API/WebGLRenderingContext) or [`WebGL2RenderingContext`](https://developer.mozilla.org/en-US/docs/Web/API/WebGL2RenderingContext): you manage shaders, buffers, and draw calls yourself."
|
|
1800
|
+
"markdown": "### Shorthand: `viji.ctx`\r\n\r\nAfter calling `viji.useContext('2d')`, the same context is also available as `viji.ctx`:\r\n\r\n```javascript\r\nfunction render(viji) {\r\n viji.useContext('2d');\r\n viji.ctx.fillStyle = '#000';\r\n viji.ctx.fillRect(0, 0, viji.width, viji.height);\r\n}\r\n```\r\n\r\nBoth `viji.useContext('2d')` and `viji.ctx` reference the same object.\r\n\r\n## WebGL Context\r\n\r\nFor GPU-accelerated rendering, request a WebGL context. The returned object is a standard [`WebGLRenderingContext`](https://developer.mozilla.org/en-US/docs/Web/API/WebGLRenderingContext) or [`WebGL2RenderingContext`](https://developer.mozilla.org/en-US/docs/Web/API/WebGL2RenderingContext): you manage shaders, buffers, and draw calls yourself."
|
|
1791
1801
|
},
|
|
1792
1802
|
{
|
|
1793
1803
|
"type": "live-example",
|
|
@@ -1797,7 +1807,17 @@ export const docsApi = {
|
|
|
1797
1807
|
},
|
|
1798
1808
|
{
|
|
1799
1809
|
"type": "text",
|
|
1800
|
-
"markdown": "### Shorthand: `viji.gl`\n\nAfter calling `viji.useContext('webgl')` or `viji.useContext('webgl2')`, the context is also available as `viji.gl`. Its runtime type matches whichever variant you requested.\n\n> [!TIP]\n> If you want full GPU control but don't want to manage WebGL boilerplate, consider the [Shader Renderer](/shader/quickstart): it handles the fullscreen quad, uniform injection, and compilation for you.\n\n##
|
|
1810
|
+
"markdown": "### Shorthand: `viji.gl`\r\n\r\nAfter calling `viji.useContext('webgl')` or `viji.useContext('webgl2')`, the context is also available as `viji.gl`. Its runtime type matches whichever variant you requested.\r\n\r\n> [!TIP]\r\n> If you want full GPU control but don't want to manage WebGL boilerplate, consider the [Shader Renderer](/shader/quickstart): it handles the fullscreen quad, uniform injection, and compilation for you.\r\n\r\n## WebGPU Context\r\n\r\nThe modern GPU API, for compute shaders and custom render pipelines. `useContext('webgpu')` returns a raw [`GPUCanvasContext`](https://developer.mozilla.org/en-US/docs/Web/API/GPUCanvasContext). Unlike the other context types, it can also return `null` at runtime on a browser without WebGPU support, so always guard and provide a fallback. Requesting the adapter and device, and configuring the context, stay in your scene code (top-level `await` works):\r\n\r\n```javascript\r\nconst gctx = viji.useContext('webgpu');\r\nif (!navigator.gpu || !gctx) {\r\n // fall back: render a message with a 2D context instead\r\n} else {\r\n const adapter = await navigator.gpu.requestAdapter();\r\n const device = await adapter.requestDevice();\r\n const format = navigator.gpu.getPreferredCanvasFormat();\r\n gctx.configure({ device, format, alphaMode: 'opaque' });\r\n}\r\n```"
|
|
1811
|
+
},
|
|
1812
|
+
{
|
|
1813
|
+
"type": "live-example",
|
|
1814
|
+
"title": "WebGPU: Hello Triangle",
|
|
1815
|
+
"sceneCode": "// @renderer native\nconst speed = viji.slider(1, { min: 0, max: 3, step: 0.05, label: 'Spin Speed' });\n\nlet draw;\nconst gctx = viji.useContext('webgpu');\nif (!navigator.gpu || !gctx) {\n const ctx = viji.useContext('2d');\n draw = () => {\n ctx.fillStyle = '#101018';\n ctx.fillRect(0, 0, viji.width, viji.height);\n ctx.fillStyle = '#cfd2ff';\n ctx.font = '15px system-ui';\n ctx.textAlign = 'center';\n ctx.fillText('WebGPU is not available in this browser', viji.width / 2, viji.height / 2);\n };\n} else {\n const adapter = await navigator.gpu.requestAdapter();\n const device = await adapter.requestDevice();\n const format = navigator.gpu.getPreferredCanvasFormat();\n gctx.configure({ device, format, alphaMode: 'opaque' });\n\n const wgsl =\n 'struct U { t: f32, aspect: f32 };\\n' +\n '@group(0) @binding(0) var<uniform> u: U;\\n' +\n 'struct VOut { @builtin(position) pos: vec4<f32>, @location(0) col: vec3<f32> };\\n' +\n '@vertex fn vs(@builtin(vertex_index) i: u32) -> VOut {\\n' +\n ' var p = array<vec2<f32>, 3>(vec2<f32>(0.0, 0.62), vec2<f32>(-0.54, -0.31), vec2<f32>(0.54, -0.31));\\n' +\n ' var c = array<vec3<f32>, 3>(vec3<f32>(1.0, 0.3, 0.5), vec3<f32>(0.3, 0.9, 1.0), vec3<f32>(0.7, 0.4, 1.0));\\n' +\n ' let r = mat2x2<f32>(cos(u.t), -sin(u.t), sin(u.t), cos(u.t));\\n' +\n ' var q = r * p[i];\\n' +\n ' q.x = q.x / u.aspect;\\n' +\n ' var out: VOut;\\n' +\n ' out.pos = vec4<f32>(q, 0.0, 1.0);\\n' +\n ' out.col = c[i];\\n' +\n ' return out;\\n' +\n '}\\n' +\n '@fragment fn fs(in: VOut) -> @location(0) vec4<f32> {\\n' +\n ' return vec4<f32>(in.col, 1.0);\\n' +\n '}\\n';\n\n const module = device.createShaderModule({ code: wgsl });\n const pipeline = device.createRenderPipeline({\n layout: 'auto',\n vertex: { module, entryPoint: 'vs' },\n fragment: { module, entryPoint: 'fs', targets: [{ format }] },\n primitive: { topology: 'triangle-list' },\n });\n const uniforms = device.createBuffer({ size: 8, usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST });\n const bindGroup = device.createBindGroup({\n layout: pipeline.getBindGroupLayout(0),\n entries: [{ binding: 0, resource: { buffer: uniforms } }],\n });\n\n let angle = 0;\n draw = () => {\n angle += viji.deltaTime * speed.value;\n device.queue.writeBuffer(uniforms, 0, new Float32Array([angle, viji.width / viji.height]));\n const encoder = device.createCommandEncoder();\n const pass = encoder.beginRenderPass({\n colorAttachments: [{\n view: gctx.getCurrentTexture().createView(),\n loadOp: 'clear',\n storeOp: 'store',\n clearValue: { r: 0.05, g: 0.04, b: 0.1, a: 1 },\n }],\n });\n pass.setPipeline(pipeline);\n pass.setBindGroup(0, bindGroup);\n pass.draw(3);\n pass.end();\n device.queue.submit([encoder.finish()]);\n };\n}\n\nfunction render() {\n draw();\n}\n",
|
|
1816
|
+
"sceneFile": "canvas-context-webgpu.scene.js"
|
|
1817
|
+
},
|
|
1818
|
+
{
|
|
1819
|
+
"type": "text",
|
|
1820
|
+
"markdown": "### Shorthand: `viji.gpu`\r\n\r\nAfter a successful `viji.useContext('webgpu')` (or a direct `viji.canvas.getContext('webgpu')`), the same context is also available as `viji.gpu`.\r\n\r\nGet a fresh swapchain texture with `getCurrentTexture()` every frame inside `render()`; never cache it across frames. A failed WebGPU request doesn't bind the canvas, so the fallback branch can still acquire `'2d'`. A GPU device can be lost (driver reset, tab backgrounding on some platforms): listen on `device.lost`, then re-request the adapter and device and `configure()` again. The engine tolerates reconfiguration at any time. For running ONNX models on the WebGPU execution provider, see [External Libraries](/native/external-libraries).\r\n\r\n## Resolution-Agnostic Drawing\r\n\r\nThe canvas can be any size, from a small preview to fullscreen 4K, and the host can resize it at any time. Your scene must adapt automatically.\r\n\r\n> [!NOTE]\r\n> Always use `viji.width` and `viji.height` for positioning and sizing, and `viji.deltaTime` for frame-rate-independent animation. Never hardcode pixel values or assume a specific frame rate.\r\n\r\n### Dimension Properties\r\n\r\n| Property | Type | Description |\r\n|----------|------|-------------|\r\n| `viji.width` | `number` | Current canvas width in pixels |\r\n| `viji.height` | `number` | Current canvas height in pixels |\r\n\r\nBoth values update automatically when the host resizes the canvas. Read them every frame inside `render()`: don't cache them at the top level.\r\n\r\n### Centering and Proportional Sizing\r\n\r\nExpress all positions and sizes relative to the canvas dimensions:\r\n\r\n```javascript\r\n// Good: scales with canvas\r\nconst cx = viji.width / 2;\r\nconst cy = viji.height / 2;\r\nconst r = Math.min(viji.width, viji.height) * 0.3;\r\n\r\n// Bad: breaks at different resolutions\r\nconst cx = 400;\r\nconst cy = 300;\r\nconst r = 150;\r\n```\r\n\r\nUse `Math.min(viji.width, viji.height)` when you need uniform scaling: it ensures elements stay proportional regardless of aspect ratio. Use `viji.width` or `viji.height` individually when you want an element to stretch with one axis (e.g., a full-width bar).\r\n\r\n### Proportional Margins and Spacing\r\n\r\n```javascript\r\nconst margin = Math.min(viji.width, viji.height) * 0.05;\r\nconst usableW = viji.width - margin * 2;\r\nconst usableH = viji.height - margin * 2;\r\n```\r\n\r\n### Comparison Across Renderers\r\n\r\n| Concept | Native | P5 | Shader |\r\n|---------|--------|-----|--------|\r\n| Canvas width | `viji.width` | `viji.width` / `p5.width` | `u_resolution.x` |\r\n| Canvas height | `viji.height` | `viji.height` / `p5.height` | `u_resolution.y` |\r\n| Aspect ratio | `viji.width / viji.height` | `viji.width / viji.height` | `u_resolution.x / u_resolution.y` |\r\n| Uniform scale | `Math.min(viji.width, viji.height)` | `Math.min(viji.width, viji.height)` | `min(u_resolution.x, u_resolution.y)` |\r\n\r\n## Environment Constraints\r\n\r\n> [!WARNING]\r\n> Scenes run in a Web Worker. There is no `window`, `document`, `Image()`, `localStorage`, or any DOM API. All inputs (audio, video, images) are provided through the Viji API. One exception: `fetch()` is available, so you can load external data (JSON, etc.) from CDNs.\r\n\r\n## Next Steps\r\n\r\n- [Timing](/native/timing): [`viji.time`](/native/timing), [`viji.deltaTime`](/native/timing), frame counting\r\n- [Parameters](/native/parameters): sliders, colors, toggles\r\n- [P5 Canvas & Resolution](/p5/canvas-resolution): how P5 handles canvas and sizing\r\n- [Shader Resolution](/shader/resolution): `u_resolution` and coordinate normalization\r\n- [API Reference](/native/api-reference): full list of everything available"
|
|
1801
1821
|
}
|
|
1802
1822
|
]
|
|
1803
1823
|
},
|
|
@@ -1881,7 +1901,7 @@ export const docsApi = {
|
|
|
1881
1901
|
},
|
|
1882
1902
|
"native-external-libs": {
|
|
1883
1903
|
"id": "native-external-libs",
|
|
1884
|
-
"title": "
|
|
1904
|
+
"title": "native-external-libs",
|
|
1885
1905
|
"description": "How to use Three.js, math libraries, and other ES modules in native Viji scenes.",
|
|
1886
1906
|
"headings": [
|
|
1887
1907
|
{
|
|
@@ -1919,6 +1939,11 @@ export const docsApi = {
|
|
|
1919
1939
|
"level": 2,
|
|
1920
1940
|
"text": "WebGL Context"
|
|
1921
1941
|
},
|
|
1942
|
+
{
|
|
1943
|
+
"id": "machine-learning-libraries",
|
|
1944
|
+
"level": 2,
|
|
1945
|
+
"text": "Machine Learning Libraries"
|
|
1946
|
+
},
|
|
1922
1947
|
{
|
|
1923
1948
|
"id": "tips",
|
|
1924
1949
|
"level": 2,
|
|
@@ -1933,7 +1958,7 @@ export const docsApi = {
|
|
|
1933
1958
|
"content": [
|
|
1934
1959
|
{
|
|
1935
1960
|
"type": "text",
|
|
1936
|
-
"markdown": "# External Libraries\n\nThe native renderer supports **top-level `await`**, which means you can dynamically import any ES module from a CDN directly in your scene code. No bundler, no install step: just `await import()`.\n\n> [!TIP]\n> Have an existing Three.js project? See [Convert: Three.js](/ai-prompts/convert-threejs) for a ready-to-paste AI prompt that adapts your code to Viji automatically.\n\n## Importing a Library\n\n```javascript\nconst THREE = await import('https://esm.sh/three@0.160.0');\n```\n\nThis works because Viji scene code runs in a module-like context where top-level `await` is available. The import happens once during scene initialization, before `render()` is ever called.\n\n> [!TIP]\n> [esm.sh](https://esm.sh) is a recommended CDN for ES modules. It serves any npm package as a standard ES module and handles dependency resolution automatically.\n\n## Three.js\n\nThree.js is the most common external library used with Viji. It works naturally in native mode because it accepts an external canvas, has no automatic render loop, and uses standard WebGL."
|
|
1961
|
+
"markdown": "# External Libraries\r\n\r\nThe native renderer supports **top-level `await`**, which means you can dynamically import any ES module from a CDN directly in your scene code. No bundler, no install step: just `await import()`.\r\n\r\n> [!TIP]\r\n> Have an existing Three.js project? See [Convert: Three.js](/ai-prompts/convert-threejs) for a ready-to-paste AI prompt that adapts your code to Viji automatically.\r\n\r\n## Importing a Library\r\n\r\n```javascript\r\nconst THREE = await import('https://esm.sh/three@0.160.0');\r\n```\r\n\r\nThis works because Viji scene code runs in a module-like context where top-level `await` is available. The import happens once during scene initialization, before `render()` is ever called.\r\n\r\n> [!TIP]\r\n> [esm.sh](https://esm.sh) is a recommended CDN for ES modules. It serves any npm package as a standard ES module and handles dependency resolution automatically.\r\n\r\n## Three.js\r\n\r\nThree.js is the most common external library used with Viji. It works naturally in native mode because it accepts an external canvas, has no automatic render loop, and uses standard WebGL."
|
|
1937
1962
|
},
|
|
1938
1963
|
{
|
|
1939
1964
|
"type": "live-example",
|
|
@@ -1943,7 +1968,7 @@ export const docsApi = {
|
|
|
1943
1968
|
},
|
|
1944
1969
|
{
|
|
1945
1970
|
"type": "text",
|
|
1946
|
-
"markdown": "### Key Integration Points\n\n**Canvas:** Pass `viji.canvas` to the Three.js renderer:\n\n```javascript\nconst renderer = new THREE.WebGLRenderer({\n canvas: viji.canvas,\n antialias: true\n});\n```\n\n**Resize:** Three.js doesn't auto-resize. Check for dimension changes in `render()`:\n\n```javascript\nlet prevWidth = viji.width;\nlet prevHeight = viji.height;\n\nfunction render(viji) {\n if (viji.width !== prevWidth || viji.height !== prevHeight) {\n renderer.setSize(viji.width, viji.height, false);\n camera.aspect = viji.width / viji.height;\n camera.updateProjectionMatrix();\n prevWidth = viji.width;\n prevHeight = viji.height;\n }\n\n renderer.render(scene, camera);\n}\n```\n\n> [!IMPORTANT]\n> Pass `false` as the third argument to `renderer.setSize()`: this prevents Three.js from setting CSS styles on the canvas, which would fail in the worker environment.\n\n**No `requestAnimationFrame`:** Viji controls the render loop. Write your animation logic inside `render(viji)` and call `renderer.render(scene, camera)` at the end.\n\n**Parameters:** All Viji parameter types work as expected:\n\n```javascript\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nconst color = viji.color('#049ef4', { label: 'Color' });\n\nfunction render(viji) {\n cube.rotation.y += speed.value * viji.deltaTime;\n material.color.set(color.value);\n renderer.render(scene, camera);\n}\n```\n\n### Three.js Addons\n\nImport addons from the Three.js examples directory:\n\n```javascript\nconst THREE = await import('https://esm.sh/three@0.160.0');\nconst { OrbitControls } = await import('https://esm.sh/three@0.160.0/examples/jsm/controls/OrbitControls.js');\nconst { GLTFLoader } = await import('https://esm.sh/three@0.160.0/examples/jsm/loaders/GLTFLoader.js');\nconst { EffectComposer } = await import('https://esm.sh/three@0.160.0/examples/jsm/postprocessing/EffectComposer.js');\n```\n\n> [!NOTE]\n> `OrbitControls` requires DOM events and will not work as expected in the worker environment. For camera interaction, use [`viji.pointer`](/native/pointer) to drive the camera manually: it handles both mouse and touch input.\n\n### Video and Image Textures\n\nThree.js can consume Viji's video frames and image parameters directly:\n\n```javascript\nconst photo = viji.image(null, { label: 'Texture' });\nlet texture = null;\n\nfunction render(viji) {\n if (photo.value && !texture) {\n texture = new THREE.CanvasTexture(photo.value);\n material.map = texture;\n material.needsUpdate = true;\n }\n\n if (viji.video.isConnected && viji.video.currentFrame) {\n if (!videoTexture) {\n videoTexture = new THREE.CanvasTexture(viji.video.currentFrame);\n videoMaterial.map = videoTexture;\n }\n videoTexture.needsUpdate = true;\n }\n\n renderer.render(scene, camera);\n}\n```\n\n## Other Libraries\n\nAny library published as an ES module works the same way. Examples:\n\n**Math / utility libraries:**\n\n```javascript\nconst { vec3, mat4 } = await import('https://esm.sh/gl-matrix@3.4.3');\nconst SimplexNoise = (await import('https://esm.sh/simplex-noise@4.0.1')).createNoise3D;\n```\n\n**2D rendering libraries:**\n\n```javascript\nconst PIXI = await import('https://esm.sh/pixi.js@7.3.2');\n```\n\n> [!NOTE]\n> Some libraries expect a DOM environment and may not work in the worker context. Libraries that accept an external canvas or work with pure data (math, noise, data structures) are the best candidates.\n\n## WebGL Context\n\nWhen using a library that creates its own WebGL context (like Three.js or Pixi.js), keep in mind:\n\n- The library consumes the canvas's WebGL context. You cannot also call `viji.useContext('2d')` or `viji.useContext('webgl')` for the same canvas: pick one rendering approach.\n- CV features (face detection, hand tracking, etc.) use additional WebGL contexts internally. Browsers have a limit of 8-16 contexts. Combining a WebGL library with multiple CV features may cause context loss.\n\nSee [Canvas & Context](/native/canvas-context) for details on `useContext()`.\n\n## Tips\n\n- **Pin library versions** in the import URL (e.g., `three@0.160.0`). Unpinned versions may break when the library updates.\n- **Import once at the top level.** Never put `await import()` inside `render()`: it would re-fetch the library every frame.\n- **Check worker compatibility** before choosing a library. Libraries that require `document`, `window`, or `Image()` will not work. `fetch()` is available.\n- **Use `viji.deltaTime`** for animation instead of the library's own timer, to stay frame-rate-independent and synchronized with other Viji features.\n\n## Related\n\n- [Canvas & Context](/native/canvas-context): `useContext('2d')`, `useContext('webgl')`, `useContext('webgl2')`\n- [Native Quick Start](/native/quickstart): building native scenes from scratch\n- [Parameters](/native/parameters): sliders, colors, toggles\n- [Best Practices](/getting-started/best-practices): essential patterns for all renderers"
|
|
1971
|
+
"markdown": "### Key Integration Points\r\n\r\n**Canvas:** Pass `viji.canvas` to the Three.js renderer:\r\n\r\n```javascript\r\nconst renderer = new THREE.WebGLRenderer({\r\n canvas: viji.canvas,\r\n antialias: true\r\n});\r\n```\r\n\r\n**Resize:** Three.js doesn't auto-resize. Check for dimension changes in `render()`:\r\n\r\n```javascript\r\nlet prevWidth = viji.width;\r\nlet prevHeight = viji.height;\r\n\r\nfunction render(viji) {\r\n if (viji.width !== prevWidth || viji.height !== prevHeight) {\r\n renderer.setSize(viji.width, viji.height, false);\r\n camera.aspect = viji.width / viji.height;\r\n camera.updateProjectionMatrix();\r\n prevWidth = viji.width;\r\n prevHeight = viji.height;\r\n }\r\n\r\n renderer.render(scene, camera);\r\n}\r\n```\r\n\r\n> [!IMPORTANT]\r\n> Pass `false` as the third argument to `renderer.setSize()`: this prevents Three.js from setting CSS styles on the canvas, which would fail in the worker environment.\r\n\r\n**No `requestAnimationFrame`:** Viji controls the render loop. Write your animation logic inside `render(viji)` and call `renderer.render(scene, camera)` at the end.\r\n\r\n**Parameters:** All Viji parameter types work as expected:\r\n\r\n```javascript\r\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\r\nconst color = viji.color('#049ef4', { label: 'Color' });\r\n\r\nfunction render(viji) {\r\n cube.rotation.y += speed.value * viji.deltaTime;\r\n material.color.set(color.value);\r\n renderer.render(scene, camera);\r\n}\r\n```\r\n\r\n### Three.js Addons\r\n\r\nImport addons from the Three.js examples directory:\r\n\r\n```javascript\r\nconst THREE = await import('https://esm.sh/three@0.160.0');\r\nconst { OrbitControls } = await import('https://esm.sh/three@0.160.0/examples/jsm/controls/OrbitControls.js');\r\nconst { GLTFLoader } = await import('https://esm.sh/three@0.160.0/examples/jsm/loaders/GLTFLoader.js');\r\nconst { EffectComposer } = await import('https://esm.sh/three@0.160.0/examples/jsm/postprocessing/EffectComposer.js');\r\n```\r\n\r\n> [!NOTE]\r\n> `OrbitControls` requires DOM events and will not work as expected in the worker environment. For camera interaction, use [`viji.pointer`](/native/pointer) to drive the camera manually: it handles both mouse and touch input.\r\n\r\n### Video and Image Textures\r\n\r\nThree.js can consume Viji's video frames and image parameters directly:\r\n\r\n```javascript\r\nconst photo = viji.image(null, { label: 'Texture' });\r\nlet texture = null;\r\n\r\nfunction render(viji) {\r\n if (photo.value && !texture) {\r\n texture = new THREE.CanvasTexture(photo.value);\r\n material.map = texture;\r\n material.needsUpdate = true;\r\n }\r\n\r\n if (viji.video.isConnected && viji.video.currentFrame) {\r\n if (!videoTexture) {\r\n videoTexture = new THREE.CanvasTexture(viji.video.currentFrame);\r\n videoMaterial.map = videoTexture;\r\n }\r\n videoTexture.needsUpdate = true;\r\n }\r\n\r\n renderer.render(scene, camera);\r\n}\r\n```\r\n\r\n## Other Libraries\r\n\r\nAny library published as an ES module works the same way. Examples:\r\n\r\n**Math / utility libraries:**\r\n\r\n```javascript\r\nconst { vec3, mat4 } = await import('https://esm.sh/gl-matrix@3.4.3');\r\nconst SimplexNoise = (await import('https://esm.sh/simplex-noise@4.0.1')).createNoise3D;\r\n```\r\n\r\n**2D rendering libraries:**\r\n\r\n```javascript\r\nconst PIXI = await import('https://esm.sh/pixi.js@7.3.2');\r\n```\r\n\r\n> [!NOTE]\r\n> Some libraries expect a DOM environment and may not work in the worker context. Libraries that accept an external canvas or work with pure data (math, noise, data structures) are the best candidates.\r\n\r\n## WebGL Context\r\n\r\nWhen using a library that creates its own WebGL context (like Three.js or Pixi.js), keep in mind:\r\n\r\n- The library consumes the canvas's WebGL context. You cannot also call `viji.useContext('2d')` or `viji.useContext('webgl')` for the same canvas: pick one rendering approach.\r\n- CV features (face detection, hand tracking, etc.) use additional WebGL contexts internally. Browsers have a limit of 8-16 contexts. Combining a WebGL library with multiple CV features may cause context loss.\r\n\r\nSee [Canvas & Context](/native/canvas-context) for details on `useContext()`.\r\n\r\n## Machine Learning Libraries\r\n\r\nTensorFlow.js and ONNX Runtime Web both work inside scenes. Import the library at the top level, run inference outside `render()` (inference takes longer than a frame), and draw the most recent result each frame. Model files downloaded with `fetch()` are cached by the engine on the host, so a model downloads once and later scene loads read it from disk. Only large static files are cached: small responses such as live API data stay fresh on every call. Note that `import()` module loads are not cached this way; libraries are small compared to models, so this rarely matters.\r\n\r\n**ONNX Runtime Web** (version 1.27 and newer): import the module build from jsdelivr and tell the runtime where to download its WebAssembly files. Both values are URLs on the same CDN:\r\n\r\n```javascript\r\nconst ort = await import('https://cdn.jsdelivr.net/npm/onnxruntime-web@1.27.0/dist/ort.webgpu.min.mjs');\r\nort.env.wasm.wasmPaths = 'https://cdn.jsdelivr.net/npm/onnxruntime-web@1.27.0/dist/';\r\n\r\nconst modelBytes = new Uint8Array(await (await fetch(modelUrl)).arrayBuffer());\r\nconst session = await ort.InferenceSession.create(modelBytes, {\r\n executionProviders: ['webgpu']\r\n});\r\n```\r\n\r\n> [!NOTE]\r\n> Import ONNX Runtime Web from jsdelivr, not esm.sh. The esm.sh build injects Node.js compatibility code that breaks the runtime during startup.\r\n\r\n**TensorFlow.js**: `importScripts('https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@4.22.0/dist/tf.min.js')` loads the classic bundle and exposes the global `tf`. The WebGL backend initializes automatically on first use.\r\n\r\n**Transformers.js**: import the plain-file build and turn off its Cache Storage use. The scene worker cannot access Cache Storage; plain fetch downloads work and are cached by the engine on the host:\r\n\r\n```javascript\r\nconst tf = await import('https://cdn.jsdelivr.net/npm/@huggingface/transformers@3.7.1/dist/transformers.min.js');\r\ntf.env.useBrowserCache = false;\r\n```\r\n\r\n## Tips\r\n\r\n- **Pin library versions** in the import URL (e.g., `three@0.160.0`). Unpinned versions may break when the library updates.\r\n- **Import once at the top level.** Never put `await import()` inside `render()`: it would re-fetch the library every frame.\r\n- **Check worker compatibility** before choosing a library. Libraries that require `document`, `window`, or `Image()` will not work. `fetch()` is available.\r\n- **Use `viji.deltaTime`** for animation instead of the library's own timer, to stay frame-rate-independent and synchronized with other Viji features.\r\n- **Spawning a helper worker?** Build it from a `data:` URL: `new Worker('data:application/javascript,' + encodeURIComponent(workerSource))`. In Chrome, only `data:` workers can use WebGPU and other capabilities that need a secure context. Workers built from `blob:` URLs still run everywhere, they just cannot use those capabilities in Chrome.\r\n\r\n## Related\r\n\r\n- [Canvas & Context](/native/canvas-context): `useContext('2d')`, `useContext('webgl')`, `useContext('webgl2')`\r\n- [Native Quick Start](/native/quickstart): building native scenes from scratch\r\n- [Parameters](/native/parameters): sliders, colors, toggles\r\n- [Best Practices](/getting-started/best-practices): essential patterns for all renderers"
|
|
1947
1972
|
}
|
|
1948
1973
|
]
|
|
1949
1974
|
},
|
|
@@ -9849,7 +9874,7 @@ export const docsApi = {
|
|
|
9849
9874
|
"ai": {
|
|
9850
9875
|
"systemPrompts": {
|
|
9851
9876
|
"base": "# Viji API: Base Reference\n\nThis document is the renderer-agnostic foundation for generating Viji scenes. It is loaded into every AI turn alongside the renderer-specific reference for the renderer in use.\n\n## Architecture\n\nViji scenes run inside a **Web Worker** on an **OffscreenCanvas**. There is no DOM.\n\n- Top-level code runs once when the scene loads (parameter declarations, state, imports).\n- A `render` function (or, for shaders, `void main()`) runs every frame.\n- `fetch()` and `await import()` are available. `window`, `document`, `Image()`, and `localStorage` are not.\n- The global `viji` object exposes everything: canvas, timing, audio, video, computer vision, input, sensors, and parameters. Each renderer also has its own surface (P5's `p5.*` API; the shader's auto-injected uniforms).\n\n## Renderers\n\nThree renderers. Pick by the visual goal, not by perceived difficulty.\n\n| | Native | P5 | Shader |\n|---|---|---|---|\n| Language | JavaScript (Canvas 2D / WebGL) | JavaScript + P5.js 1.9.4 | GLSL fragment shader |\n| Best for | Full control, Three.js, generative art, pixel-perfect Canvas 2D | Creative coding with shapes, colors, transforms; ports of existing P5 sketches | GPU effects, patterns, raymarching, SDF, post-processing |\n| 3D | Yes (WebGL, Three.js via `await import()`) | Yes (`// @renderer p5 webgl`) | Yes (raymarching, SDF) |\n| External libraries | Yes (any ESM via `await import()`) | P5.js built-in | None (GPU only) |\n\n### Decision criteria\n\n- **Per-pixel GPU effects** (raymarching, fractals, distance fields, fullscreen post-processing, patterns that read every pixel): use **Shader**.\n- **3D with Three.js, custom WebGL, pixel-perfect Canvas 2D, or any external ESM library**: use **Native**.\n- **Classic P5 shapes and transforms** (`ellipse`, `line`, `bezier`, color modes, `push`/`pop`), or a port of an existing P5 sketch: use **P5**.\n- **When ambiguous, default to Native**. Audio reactivity, camera/CV, parameters, and input work in every renderer. Pick on visual style, not data sources.\n\n## Scene shape per renderer\n\nEach example below is the minimum viable entry point. Full surface lives in the matching renderer reference.\n\n**Native**:\n```javascript\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nlet angle = 0;\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n angle += speed.value * viji.deltaTime;\n ctx.clearRect(0, 0, viji.width, viji.height);\n // draw with ctx\n}\n```\n\n**P5**:\n```javascript\n// @renderer p5\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' });\nlet angle = 0;\nfunction setup(viji, p5) { p5.colorMode(p5.HSB, 360, 100, 100); }\nfunction render(viji, p5) {\n angle += speed.value * viji.deltaTime;\n p5.background(0);\n // draw with p5.* prefixed calls\n}\n```\n\n**Shader (GLSL)**:\n```glsl\n// @renderer shader\n// @viji-slider:speed label:\"Speed\" default:1.0 min:0.1 max:5.0\n// @viji-accumulator:phase rate:speed\nvoid main() {\n vec2 uv = gl_FragCoord.xy / u_resolution;\n gl_FragColor = vec4(uv, sin(phase) * 0.5 + 0.5, 1.0);\n}\n```\n\n## Cross-cutting rules (apply to every renderer)\n\n1. **Never** touch the DOM. No `window`, `document`, `Image()`, `localStorage`. `fetch()` and `await import()` are available.\n2. **Always** declare parameters at the **top level**, never inside `render()` or `main()`. Parameters declared inside a render call are recreated every frame and break the host UI.\n3. **Always** use `viji.width` / `viji.height` (or `u_resolution` in shaders) for canvas dimensions. Never hardcode pixel sizes.\n4. **Always** use `viji.deltaTime` (or `u_deltaTime` / `@viji-accumulator` in shaders) for animation. Never count frames or assume a fixed frame rate.\n5. **Never** multiply `viji.time` (or `u_time`) by a parameter to drive animation speed. It causes visible jumps when the slider changes. Use a `deltaTime` accumulator at the top level instead. In shaders, use `@viji-accumulator`. This also applies to nested multiplications: never multiply an accumulator by another parameter. Each independent speed gets its own accumulator.\n6. **Never** allocate objects, arrays, or strings inside `render()`. Pre-allocate at the top level and reuse.\n7. **Always** check `viji.audio.isConnected` (or `u_audioVolume > 0.0`) before reading audio data. Check `viji.video.isConnected && viji.video.currentFrame` (or `u_videoConnected`) before reading video. When an input is not connected, degrade gracefully — keep rendering the scene's normal idle/ambient visual. Do **not** draw on-canvas text telling the user to connect or turn on a camera, microphone, or other input; the host provides those controls. (The connection-demo examples draw \"Waiting for…\" text only to illustrate this check itself — don't copy that into real scenes unless the user explicitly asks for an on-canvas prompt.)\n8. **CV features are independent**: each `enableX()` populates only its own subset of fields. Face detection populates `face.bounds` / `center` / `confidence`; face mesh populates `face.landmarks` / `headPose`; emotion detection populates `face.expressions` / `blendshapes`. Fields whose source model is not enabled read as `null` (or `[]` for `landmarks`) — always null-check before reading. CV is expensive and is **not** host-gated, so it is the one feature class that legitimately belongs behind a parameter toggle — but gate by intent: if a CV feature is the scene's defining purpose (the scene is meaningless without it), enable it always-on from module scope; if CV is optional/supplementary, or the scene uses several heavy features, gate each behind a `viji.toggle` (shader: the toggleable `@viji-cv:` form) so the user opts into the cost — and make sure the base visual stands on its own when a toggle is off (no \"enable in parameters\" banner). Each active CV feature uses its own WebGL context; 4+ on low-end hardware can hit context limits, so enable only what you need.\n9. **Always** set `category` on parameters that depend on an external input: `category: 'audio'` for audio-driven controls, `category: 'video'` for video / CV-driven controls, `category: 'interaction'` for mouse / keyboard / touch controls. Omit `category` (defaults to `'general'`) for parameters that work without external input. **Never** add on/off parameter toggles to enable/disable built-in inputs or device features — audio, video, mouse, keyboard, touch, motion/orientation sensors, and external devices are all gated by the host's own controls. Read them in code behind `isConnected` / `isInCanvas` / null checks. A parameter that shapes how an input drives the visual (e.g. a sensitivity slider) is fine; an on/off enable toggle for the input itself is not. If the user explicitly asks for one, advise that the host already provides it before complying. (CV features are the sole exception — see the CV rule above.)\n10. **In P5**: prefix every P5 function and constant with `p5.` (instance mode). Never call `createCanvas()`; Viji creates the canvas.\n11. **In shaders**: never redeclare `precision`, never redeclare built-in uniforms (`u_time`, `u_resolution`, etc.), never redeclare parameter uniforms. Never use the `u_` prefix for your own parameter names; it is reserved for Viji's built-ins.\n\n## Parameter declaration shape (cross-renderer)\n\nAssign each parameter to its own top-level `const` with a unique camelCase\nname (`const speed = viji.slider(1, { label: 'Speed' })`) and read it with\n`.value`. `label` is optional but recommended (falls back to the variable\nname). Native and P5 use JavaScript calls at the top level:\n\n```javascript\nviji.slider(default, { min, max, step, label, group, category }) // .value: number\nviji.color(default, { label }) // .value: '#rrggbb', .rgb, .hsb\nviji.toggle(default, { label }) // .value: boolean\nviji.select(default, { options, label }) // .value: string | number\nviji.number(default, { min, max, step, label }) // .value: number\nviji.text(default, { label, maxLength }) // .value: string\nviji.image(null, { label }) // .value: ImageBitmap | null\nviji.button({ label }) // .value: boolean (true for 1 frame)\nviji.coordinate({ x: 0, y: 0 }, { step, label, group, category }) // .value: { x, y }, range -1..1\n```\n\n**`slider` / `number` `step` default**: when omitted, `step` auto-derives from `min`/`max` as the largest power of 10 giving ~100 positions across the range (e.g., range `0..1` → `0.01`; range `0.1..5` → `0.01`; range `0..100` → `1`). Specify explicitly only when you need a different grid: `step: 1` for integer counts (`{ min: 3, max: 30, step: 1 }`), `step: 0.001` for fine precision.\n\nShaders use GLSL comment directives that compile to typed uniforms:\n\n```glsl\n// @viji-slider:name label:\"Label\" default:1.0 min:0.0 max:5.0 -> uniform float name;\n// @viji-color:name label:\"Color\" default:#ff6600 -> uniform vec3 name;\n// @viji-toggle:name label:\"Toggle\" default:false -> uniform bool name;\n// @viji-select:name label:\"Mode\" default:0 options:[\"A\",\"B\"] -> uniform int name;\n// @viji-number:name label:\"Count\" default:10.0 min:1.0 max:100.0 -> uniform float name;\n// @viji-image:name label:\"Texture\" -> uniform sampler2D name;\n// @viji-button:name label:\"Reset\" -> uniform bool name;\n// @viji-coordinate:name label:\"Pos\" default:[0.0,0.0] -> uniform vec2 name; // range -1..1\n// @viji-accumulator:name rate:speed -> uniform float name;\n```\n\nDirectives use `//` line comments only. Block comments (`/* */`) are not parsed.\n\n## Where to look next\n\nFor the full API surface of the renderer in use, consult the matching reference: Native, P5, or Shader. Each renderer reference includes the complete data shapes for audio, video, CV, input, sensors, external devices, and renderer-specific patterns (e.g. `viji.useContext` for Native, `videoFit` for P5, the complete uniform table for Shader).\n",
|
|
9852
|
-
"native": "# Viji API: Native Renderer Reference\n\nNative scenes run JavaScript on an `OffscreenCanvas` in a Web Worker, with full access to Canvas 2D, WebGL, and WebGL2 contexts. External libraries load via `await import()`. Top-level code runs once; a `render(viji)` function runs every frame.\n\n## Architecture\n\n- The global `viji` object exposes canvas, timing, audio, video, CV, input, sensors, parameters.\n- Call `viji.useContext('2d' | 'webgl' | 'webgl2')` once at the top of `render()` (or top-level) to obtain a rendering context. After the first call, `viji.ctx` / `viji.gl` shortcuts are populated.\n- **Pick one context type for the entire scene.** Calling `useContext` with a different type after the first call returns `null`.\n- There is **no `setup()` function** in Native. All initialization goes at the top level (which supports `await` for dynamic imports).\n- The DOM is unavailable. `fetch()` and `await import()` are available.\n\n## Rules\n\n1. **Never** access `window`, `document`, `Image()`, `localStorage`, or any DOM API. `fetch()` and `await import()` are available.\n2. **Always** declare parameters at the top level as individual `const`s (`const speed = viji.slider(...)`), never inside `render()`. `label` is optional but recommended (falls back to the variable name).\n3. **Always** read parameters via `.value`. Color parameters also expose `.rgb` (`{ r, g, b }` in 0..255) and `.hsb` (`{ h, s, b }` with `h` in 0..360, `s` / `b` in 0..100). Color defaults accept hex (`'#ff6600'`, `'#f60'`), `{ r, g, b }` (0..255), `{ h, s, b }` (0..360 / 0..100), or CSS `'rgb(...)'` / `'hsl(...)'` strings.\n4. **Always** use `viji.width` and `viji.height` for canvas dimensions. Never hardcode pixel sizes.\n5. **Always** use `viji.deltaTime` for animation. Use `viji.time` only for constant-speed oscillations (`Math.sin(viji.time * 2.0)`). For anything driven by a parameter, use a `deltaTime` accumulator.\n6. **Never** multiply `viji.time` by a parameter; it causes visible jumps when the parameter changes. Use a `deltaTime` accumulator at the top level. This also applies to nested multiplications: each independent speed gets its own accumulator.\n\n ```javascript\n // WRONG\n const t = viji.time * speed.value;\n const rot = phase * rotSpeed.value;\n\n // RIGHT\n let phase = 0, rotPhase = 0;\n phase += speed.value * viji.deltaTime;\n rotPhase += speed.value * rotSpeed.value * viji.deltaTime;\n ```\n7. **Never** allocate objects, arrays, or strings inside `render()`. Pre-allocate at the top level and reuse.\n8. **Always** call `viji.useContext()` to get a context. Pick one type and use it for the entire scene.\n9. **Always** check `viji.audio.isConnected` before reading audio data. Check `viji.video.isConnected && viji.video.currentFrame` before drawing video. When not connected, keep rendering the idle visual — don't draw on-canvas \"connect a camera/mic\" instructions.\n10. Enable CV by intent: always-on from module scope when CV is the scene's defining purpose, otherwise gated behind a `viji.toggle` so the user opts into the cost.\n11. **Always** set `category` on input-dependent parameters (`'audio'`, `'video'`, `'interaction'`). Don't add on/off toggles to enable/disable host-gated inputs (audio, video, mouse, keyboard, touch, sensors, devices) — the host controls those. CV is the only exception (it gets a toggle when not always-on).\n12. **Always** type any function parameter that receives the `viji` instance, including `render(viji)` and any helpers that take `viji`. Monaco's TS checker flags API misuse (wrong nesting, typo'd property names) at edit time so the error can be corrected before runtime. Scene code is transpiled via sucrase before execution (`transforms: ['typescript']`), so TS annotations are stripped at runtime — they are safe to include.\n\n ```javascript\n function render(viji: VijiAPI) { ... }\n function applyGlow(viji: VijiAPI, intensity: number) { ... }\n ```\n13. All Viji types are ambient globals — `VijiAPI`, the parameter and config types (`SliderParameter`, `SliderConfig`, ...), input and CV types (`TouchPoint`, `FaceData`, ...). Annotate with them directly; never import, alias, or re-derive them (no `ReturnType<typeof viji.slider>` — the named type exists). For the exact type of any `viji` member, fetch the `native-api-reference` docs page instead of guessing.\n\n## API reference\n\n### Canvas and context\n\n| Member | Type | Description |\n|---|---|---|\n| `viji.canvas` | `OffscreenCanvas` | The canvas |\n| `viji.useContext('2d')` | `OffscreenCanvasRenderingContext2D` | Acquire 2D context |\n| `viji.useContext('webgl')` | `WebGLRenderingContext` | Acquire WebGL 1 |\n| `viji.useContext('webgl2')` | `WebGL2RenderingContext` | Acquire WebGL 2 |\n| `viji.ctx` | `OffscreenCanvasRenderingContext2D` | Shortcut after `useContext('2d')` |\n| `viji.gl` | `WebGLRenderingContext \\| WebGL2RenderingContext` | Shortcut after WebGL context acquired |\n| `viji.width`, `viji.height` | `number` | Current canvas dimensions in pixels |\n\n### Timing\n\n| Member | Type | Description |\n|---|---|---|\n| `viji.time` | `number` | Seconds since scene start |\n| `viji.deltaTime` | `number` | Seconds since last frame |\n| `viji.frameCount` | `number` | Total frames rendered |\n| `viji.fps` | `number` | Target frame rate (host-controlled) |\n\n### Parameters\n\nDeclare at top level. All accept `{ label, description?, group?, category? }`. Category values: `'audio'`, `'video'`, `'interaction'`, `'general'` (default).\n\n```javascript\nviji.slider(default, { min?, max?, step?, label, group?, category? }) // .value: number\nviji.color(default, { label, group?, category? }) // .value: '#rrggbb', .rgb: {r,g,b} 0..255, .hsb: {h:0..360, s/b:0..100}\nviji.toggle(default, { label, group?, category? }) // .value: boolean\nviji.select(default, { options, label, group?, category? }) // .value: string | number\nviji.number(default, { min?, max?, step?, label, group?, category? }) // .value: number\nviji.text(default, { label, group?, category?, maxLength? }) // .value: string\nviji.image(null, { label, group?, category? }) // .value: ImageBitmap | null\nviji.button({ label, description?, group?, category? }) // .value: boolean (true for 1 frame)\nviji.coordinate(default, { step?, label, group?, category? }) // .value: { x, y }, range -1..1\n// slider/number `step` defaults to a power of 10 giving ~100 positions across min..max\n// (e.g., range 0..1 → 0.01, range 0..100 → 1). Specify `step: 1` for integer counts.\n```\n\n### Audio: `viji.audio`\n\nCheck `isConnected` first.\n\n| Member | Type | Notes |\n|---|---|---|\n| `isConnected` | `boolean` | Audio source active |\n| `volume.current`, `volume.peak`, `volume.smoothed` | `number` 0..1 | `smoothed` = 200ms decay envelope of `current` |\n| `bands.low`, `lowMid`, `mid`, `highMid`, `high` | `number` 0..1 | Instant band energies (20-120, 120-400, 400-1600, 1600-6000, 6000-16000 Hz) |\n| `bands.lowSmoothed`, `lowMidSmoothed`, `midSmoothed`, `highMidSmoothed`, `highSmoothed` | `number` 0..1 | 150ms decay envelope siblings (not nested) |\n| `beat.kick`, `snare`, `hat`, `any` | `number` 0..1 | Beat energy curves with a 300ms decay; peak on each detected beat then fall off |\n| `beat.kickSmoothed`, `snareSmoothed`, `hatSmoothed`, `anySmoothed` | `number` 0..1 | Smoother 500ms decay envelopes; use for ambient pulses |\n| `beat.triggers.kick`, `snare`, `hat`, `any` | `boolean` | True for exactly one frame on the matching beat, then auto-resets. OR-accumulated between frames so no beat is lost |\n| `beat.events` | `Array<{type, time, strength}>` | All beats detected since the last frame; `type` is `'kick' \\| 'snare' \\| 'hat'` (never `'any'`); `time` in milliseconds; cleared each frame |\n| `beat.bpm`, `beat.confidence`, `beat.isLocked` | `number`, `number`, `boolean` | `bpm` is `0` when no audio is connected; once audio connects it tracks the detected tempo (clamped to 60..240) with `120` as a fallback before lock-on. `confidence` in 0..1. `isLocked` is `true` on stable tempo lock |\n| `spectral.brightness`, `spectral.flatness` | `number` 0..1 | Spectral centroid; spectral flatness |\n| `getFrequencyData()` | `Uint8Array` | Raw FFT bins 0..255 |\n| `getWaveform()` | `Float32Array` | Time-domain waveform -1..1 |\n\nExternal-device audio (`viji.devices[i].audio`) and host-supplied audio streams (`viji.audioStreams[i]`) use the `AudioStreamAPI` shape: `isConnected`, `volume.{current,peak,smoothed}`, `bands.{low...high}` plus each `*Smoothed` sibling, `spectral.{brightness,flatness}`, `getFrequencyData()`, `getWaveform()`. No `beat`, BPM, triggers, or events on streams: those exist only on the main `viji.audio`.\n\n### Video: `viji.video`\n\nCheck `isConnected && currentFrame` before drawing.\n\n| Member | Type | Notes |\n|---|---|---|\n| `isConnected` | `boolean` | Video source active |\n| `currentFrame` | `OffscreenCanvas \\| ImageBitmap \\| null` | Most recent frame |\n| `frameWidth`, `frameHeight` | `number` | Source frame size |\n| `frameRate` | `number` | Source frame rate |\n| `getFrameData()` | `ImageData \\| null` | Pixel data of `currentFrame` |\n| `cv` | `VideoCVAPI` | Computer-vision surface: see below |\n\nCV data outputs (`analysedFrame`, `getAnalysedFrameData()`, `faces`, `hands`, `pose`, `segmentation`) and CV verbs (`enableFaceDetection`, etc.) all live on `viji.video.cv`, not on `viji.video` directly.\n\n**Aspect ratio.** Camera frames almost never match the canvas aspect. Drawing to `(0, 0, viji.width, viji.height)` stretches video and misaligns CV bounds. Use this helper at module scope:\n\n```javascript\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nconst v = videoFit(viji); // 'cover' or 'contain'\nctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n\n// CV coords are normalized 0..1 to the source frame. Map through v to align with the fitted video:\nviji.video.cv.faces.forEach(face => {\n const bx = v.x + face.bounds.x * v.width;\n const by = v.y + face.bounds.y * v.height;\n const bw = face.bounds.width * v.width;\n const bh = face.bounds.height * v.height;\n ctx.strokeRect(bx, by, bw, bh);\n});\n```\n\nDefault to `'cover'` for live cameras. Use `'contain'` for CV-overlay scenes where bounding boxes near frame edges must stay visible. `viji.video.currentFrame` is the right default for displayed video; reach for `viji.video.cv.analysedFrame` only when sampling pixels at CV-derived positions (segmentation compositing, face-mesh texturing, landmark-anchored color sampling). The common fallback pattern is `viji.video.cv.analysedFrame ?? viji.video.currentFrame` during the brief window before the first CV inference lands.\n\n### Computer Vision: `viji.video.cv`\n\nEach CV feature is independent and populates only its own subset of fields. Enable only what you need (each active feature consumes a separate WebGL context for MediaPipe; 4+ on low-end hardware risks context-limit failures).\n\n```javascript\n// Verbs return Promise<void>. await is optional — without it, the model loads\n// asynchronously while render runs and data fields populate when ready.\n// Safe to call from module scope (always-on CV) or per-frame inside render()\n// gated by a viji.toggle(...) (opt-in CV). Idempotent + reference-counted.\nawait viji.video.cv.enableFaceDetection(true | false); // populates face.bounds, face.center, face.confidence\nawait viji.video.cv.enableFaceMesh(true | false); // populates face.landmarks (468 pts) + face.headPose\nawait viji.video.cv.enableEmotionDetection(true | false); // populates face.expressions + face.blendshapes; also loads landmarker (face.landmarks + face.headPose populated)\nawait viji.video.cv.enableHandTracking(true | false); // populates viji.video.cv.hands[]\nawait viji.video.cv.enablePoseDetection(true | false); // populates viji.video.cv.pose\nawait viji.video.cv.enableBodySegmentation(true | false); // populates viji.video.cv.segmentation\nviji.video.cv.getActiveFeatures(); // CVFeature[]\nviji.video.cv.isProcessing(); // boolean\n```\n\nData outputs on `viji.video.cv`:\n- `analysedFrame: OffscreenCanvas | null`: the exact frame paired with the current CV results. `null` until the first inference lands after a feature is enabled. Use the pattern `viji.video.cv.analysedFrame ?? viji.video.currentFrame` to fall back during the startup window.\n- `getAnalysedFrameData(): ImageData | null`: pixel data of `analysedFrame`, cached and re-extracted only on new CV results.\n\n**`viji.video.cv.faces: FaceData[]`**: `id` always present. Other fields are populated only by their source model and read `null` (or empty array for `landmarks`) otherwise — always null-check before reading.\n- `bounds: {x,y,width,height} | null` (normalized 0..1) — populated by face detection\n- `center: {x,y} | null` (normalized 0..1) — populated by face detection\n- `confidence: number | null` (0..1) — populated by face detection\n- `landmarks: {x,y,z?}[]` (empty `[]` unless face mesh enabled; 468 pts when populated)\n- `headPose: { pitch, yaw, roll } | null` (degrees) — populated by face mesh\n- `expressions: { neutral, happy, sad, angry, surprised, disgusted, fearful } | null` (0..1 each) — populated by emotion detection\n- `blendshapes | null` — populated by emotion detection; 52 ARKit coefficients (`browDownLeft`/`Right`, `browInnerUp`, `browOuterUpLeft`/`Right`, `cheekPuff`, `cheekSquintLeft`/`Right`, `eyeBlinkLeft`/`Right`, `eyeLookDown`/`In`/`Out`/`UpLeft`/`Right`, `eyeSquintLeft`/`Right`, `eyeWideLeft`/`Right`, `jawForward`/`Left`/`Open`/`Right`, `mouthClose`/`DimpleLeft`/`Right`/`FrownLeft`/`Right`/`Funnel`/`Left`/`LowerDownLeft`/`Right`/`PressLeft`/`Right`/`Pucker`/`Right`/`RollLower`/`Upper`/`ShrugLower`/`Upper`/`SmileLeft`/`Right`/`StretchLeft`/`Right`/`UpperUpLeft`/`Right`, `noseSneerLeft`/`Right`, `tongueOut`: all 0..1)\n\nIf you need bounds while only running face mesh, either also enable face detection or compute the bounding box from `face.landmarks` min/max yourself.\n\n**`viji.video.cv.hands: HandData[]`**: `id`, `handedness: 'left' | 'right'`, `confidence`, `bounds`, `landmarks: {x,y,z}[]` (21 points), `palm: {x,y,z}`, `gestures: { fist, openPalm, peace, thumbsUp, thumbsDown, pointing, iLoveYou }` (0..1 confidence each).\n\n**`viji.video.cv.pose: PoseData | null`**: `confidence`, `landmarks: {x,y,z,visibility}[]` (33 points), plus body-part arrays `face`, `torso`, `leftArm`, `rightArm`, `leftLeg`, `rightLeg`.\n\n**`viji.video.cv.segmentation: SegmentationData | null`**: `mask: Uint8Array` (each byte is `0` for background or `1` for person; length = `width * height`), `width`, `height`. Note: the shader `u_segmentationMask` sampler is sampled as a normalized `float` (0.0 = background, 1.0 = person); the JS `mask` byte values are 0/1, not 0/255.\n\n### Input\n\n**Pointer** (unified mouse and touch): `viji.pointer`\n| Member | Type | Notes |\n|---|---|---|\n| `x`, `y` | `number` | Position in pixels |\n| `deltaX`, `deltaY` | `number` | Per-frame movement |\n| `isDown` | `boolean` | Pressed or touching |\n| `wasPressed`, `wasReleased` | `boolean` | One-frame edge triggers |\n| `isInCanvas` | `boolean` | Inside canvas bounds |\n| `type` | `'mouse' \\| 'touch' \\| 'none'` | Active input type |\n\n**Mouse**: `viji.mouse`: `x`, `y`, `isInCanvas`, `isPressed`, `leftButton`, `rightButton`, `middleButton`, `deltaX`, `deltaY`, `wheelDelta`, `wheelX`, `wheelY`, `wasPressed`, `wasReleased`, `wasMoved`.\n\n**Keyboard**: `viji.keyboard`: `isPressed(key)`, `wasPressed(key)`, `wasReleased(key)`, `activeKeys: Set<string>`, `pressedThisFrame`, `releasedThisFrame`, `lastKeyPressed`, `lastKeyReleased`, `shift`, `ctrl`, `alt`, `meta`.\n\n**Touch**: `viji.touches`: `count`, `points: TouchPoint[]`, `started`, `moved`, `ended` (each a `TouchPoint[]`), `primary: TouchPoint | null`. Each `TouchPoint`: `id`, `x`, `y`, `pressure`, `radius`, `radiusX`, `radiusY`, `rotationAngle`, `force`, `isInCanvas`, `deltaX`, `deltaY`, `velocity: {x,y}`, `isNew`, `isActive`, `isEnding`.\n\n### Sensors and external devices\n\n**`viji.device`**: host device motion and orientation:\n- `motion: DeviceMotionData | null`: `acceleration: {x,y,z}` (m/s²), `accelerationIncludingGravity`, `rotationRate: {alpha,beta,gamma}` (deg/s), `interval` (ms).\n- `orientation: DeviceOrientationData | null`: `alpha` (0-360° compass), `beta` (-180..180° front-back tilt), `gamma` (-90..90° left-right tilt), `absolute`.\n\n**`viji.devices`**: `DeviceState[]`, externally connected devices. Each: `id`, `name`, `motion`, `orientation`, `video` (`VideoAPI | null`, same shape as `viji.video` but without CV), `audio` (`AudioStreamAPI | null`, lightweight; no beat or BPM).\n\n### Host-supplied streams\n\n- `viji.videoStreams: VideoAPI[]`: extra video sources from the host compositor. Same shape as `viji.video`, including a per-stream `cv` surface with the same enable verbs and result fields; `cv.analysedFrame` is populated on the main stream only. May be empty.\n- `viji.audioStreams: AudioStreamAPI[]`: extra audio sources from the host. Lightweight shape (no beat or BPM). May be empty.\n\n### External libraries\n\n```javascript\nconst THREE = await import('https://esm.sh/three@0.160.0');\nconst renderer = new THREE.WebGLRenderer({ canvas: viji.canvas, antialias: true });\nrenderer.setSize(viji.width, viji.height, false); // always pass false to skip CSS styling\n```\n\nPin library versions in the import URL. Always pass `viji.canvas` to the renderer. Always pass `false` as the third argument to Three.js `setSize`. Handle resize by comparing `viji.width` / `viji.height` against previous values inside `render()`.\n\n## Template\n\n```javascript\nconst bgColor = viji.color('#1a1a2e', { label: 'Background' });\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' }); // implicit step → 0.01\nconst count = viji.slider(12, { min: 3, max: 30, step: 1, label: 'Count' }); // explicit step for integer counts\n\nlet angle = 0;\n\nfunction render(viji) {\n const ctx = viji.useContext('2d');\n angle += speed.value * viji.deltaTime;\n\n ctx.fillStyle = bgColor.value;\n ctx.fillRect(0, 0, viji.width, viji.height);\n\n const cx = viji.width / 2;\n const cy = viji.height / 2;\n const radius = Math.min(viji.width, viji.height) * 0.3;\n const dotSize = Math.min(viji.width, viji.height) * 0.02;\n const n = Math.floor(count.value);\n\n for (let i = 0; i < n; i++) {\n const a = angle + (i / n) * Math.PI * 2;\n const x = cx + Math.cos(a) * radius;\n const y = cy + Math.sin(a) * radius;\n const hue = (i / n) * 360;\n ctx.beginPath();\n ctx.arc(x, y, dotSize, 0, Math.PI * 2);\n ctx.fillStyle = `hsl(${hue}, 80%, 60%)`;\n ctx.fill();\n }\n}\n```\n",
|
|
9877
|
+
"native": "# Viji API: Native Renderer Reference\r\n\r\nNative scenes run JavaScript on an `OffscreenCanvas` in a Web Worker, with full access to Canvas 2D, WebGL, WebGL2, and WebGPU contexts. External libraries load via `await import()`. Top-level code runs once; a `render(viji)` function runs every frame.\r\n\r\n## Architecture\r\n\r\n- The global `viji` object exposes canvas, timing, audio, video, CV, input, sensors, parameters.\r\n- Call `viji.useContext('2d' | 'webgl' | 'webgl2' | 'webgpu')` once at the top of `render()` (or top-level) to obtain a rendering context. After the first call, the matching shortcut is populated: `viji.ctx` / `viji.gl` / `viji.gpu`.\r\n- **Pick one context type for the entire scene.** Calling `useContext` with a different type after the first call returns `null`.\r\n- **WebGPU**: request the adapter and device at top level. `useContext('webgpu')` returns `null` where unsupported: guard and fall back.\r\n- There is **no `setup()` function** in Native. All initialization goes at the top level (which supports `await` for dynamic imports).\r\n- The DOM is unavailable. `fetch()` and `await import()` are available.\r\n\r\n## Rules\r\n\r\n1. **Never** access `window`, `document`, `Image()`, `localStorage`, or any DOM API. `fetch()` and `await import()` are available.\r\n2. **Always** declare parameters at the top level as individual `const`s (`const speed = viji.slider(...)`), never inside `render()`. `label` is optional but recommended (falls back to the variable name).\r\n3. **Always** read parameters via `.value`. Color parameters also expose `.rgb` (`{ r, g, b }` in 0..255) and `.hsb` (`{ h, s, b }` with `h` in 0..360, `s` / `b` in 0..100). Color defaults accept hex (`'#ff6600'`, `'#f60'`), `{ r, g, b }` (0..255), `{ h, s, b }` (0..360 / 0..100), or CSS `'rgb(...)'` / `'hsl(...)'` strings.\r\n4. **Always** use `viji.width` and `viji.height` for canvas dimensions. Never hardcode pixel sizes.\r\n5. **Always** use `viji.deltaTime` for animation. Use `viji.time` only for constant-speed oscillations (`Math.sin(viji.time * 2.0)`). For anything driven by a parameter, use a `deltaTime` accumulator.\r\n6. **Never** multiply `viji.time` by a parameter; it causes visible jumps when the parameter changes. Use a `deltaTime` accumulator at the top level. This also applies to nested multiplications: each independent speed gets its own accumulator.\r\n\r\n ```javascript\r\n // WRONG\r\n const t = viji.time * speed.value;\r\n const rot = phase * rotSpeed.value;\r\n\r\n // RIGHT\r\n let phase = 0, rotPhase = 0;\r\n phase += speed.value * viji.deltaTime;\r\n rotPhase += speed.value * rotSpeed.value * viji.deltaTime;\r\n ```\r\n7. **Never** allocate objects, arrays, or strings inside `render()`. Pre-allocate at the top level and reuse.\r\n8. **Always** call `viji.useContext()` to get a context. Pick one type and use it for the entire scene.\r\n9. **Always** check `viji.audio.isConnected` before reading audio data. Check `viji.video.isConnected && viji.video.currentFrame` before drawing video. When not connected, keep rendering the idle visual — don't draw on-canvas \"connect a camera/mic\" instructions.\r\n10. Enable CV by intent: always-on from module scope when CV is the scene's defining purpose, otherwise gated behind a `viji.toggle` so the user opts into the cost.\r\n11. **Always** set `category` on input-dependent parameters (`'audio'`, `'video'`, `'interaction'`). Don't add on/off toggles to enable/disable host-gated inputs (audio, video, mouse, keyboard, touch, sensors, devices) — the host controls those. CV is the only exception (it gets a toggle when not always-on).\r\n12. **Always** type any function parameter that receives the `viji` instance, including `render(viji)` and any helpers that take `viji`. Monaco's TS checker flags API misuse (wrong nesting, typo'd property names) at edit time so the error can be corrected before runtime. Scene code is transpiled via sucrase before execution (`transforms: ['typescript']`), so TS annotations are stripped at runtime — they are safe to include.\r\n\r\n ```javascript\r\n function render(viji: VijiAPI) { ... }\r\n function applyGlow(viji: VijiAPI, intensity: number) { ... }\r\n ```\r\n13. All Viji types are ambient globals — `VijiAPI`, the parameter and config types (`SliderParameter`, `SliderConfig`, ...), input and CV types (`TouchPoint`, `FaceData`, ...). Annotate with them directly; never import, alias, or re-derive them (no `ReturnType<typeof viji.slider>` — the named type exists). For the exact type of any `viji` member, fetch the `native-api-reference` docs page instead of guessing.\r\n\r\n## API reference\r\n\r\n### Canvas and context\r\n\r\n| Member | Type | Description |\r\n|---|---|---|\r\n| `viji.canvas` | `OffscreenCanvas` | The canvas |\r\n| `viji.useContext('2d')` | `OffscreenCanvasRenderingContext2D` | Acquire 2D context |\r\n| `viji.useContext('webgl')` | `WebGLRenderingContext` | Acquire WebGL 1 |\r\n| `viji.useContext('webgl2')` | `WebGL2RenderingContext` | Acquire WebGL 2 |\r\n| `viji.useContext('webgpu')` | `GPUCanvasContext \\| null` | Acquire WebGPU |\r\n| `viji.ctx` | `OffscreenCanvasRenderingContext2D` | Shortcut after `useContext('2d')` |\r\n| `viji.gl` | `WebGLRenderingContext \\| WebGL2RenderingContext` | Shortcut after WebGL context acquired |\r\n| `viji.gpu` | `GPUCanvasContext` | Shortcut after `useContext('webgpu')` |\r\n| `viji.width`, `viji.height` | `number` | Current canvas dimensions in pixels |\r\n\r\n### Timing\r\n\r\n| Member | Type | Description |\r\n|---|---|---|\r\n| `viji.time` | `number` | Seconds since scene start |\r\n| `viji.deltaTime` | `number` | Seconds since last frame |\r\n| `viji.frameCount` | `number` | Total frames rendered |\r\n| `viji.fps` | `number` | Target frame rate (host-controlled) |\r\n\r\n### Parameters\r\n\r\nDeclare at top level. All accept `{ label, description?, group?, category? }`. Category values: `'audio'`, `'video'`, `'interaction'`, `'general'` (default).\r\n\r\n```javascript\r\nviji.slider(default, { min?, max?, step?, label, group?, category? }) // .value: number\r\nviji.color(default, { label, group?, category? }) // .value: '#rrggbb', .rgb: {r,g,b} 0..255, .hsb: {h:0..360, s/b:0..100}\r\nviji.toggle(default, { label, group?, category? }) // .value: boolean\r\nviji.select(default, { options, label, group?, category? }) // .value: string | number\r\nviji.number(default, { min?, max?, step?, label, group?, category? }) // .value: number\r\nviji.text(default, { label, group?, category?, maxLength? }) // .value: string\r\nviji.image(null, { label, group?, category? }) // .value: ImageBitmap | null\r\nviji.button({ label, description?, group?, category? }) // .value: boolean (true for 1 frame)\r\nviji.coordinate(default, { step?, label, group?, category? }) // .value: { x, y }, range -1..1\r\n// slider/number `step` defaults to a power of 10 giving ~100 positions across min..max\r\n// (e.g., range 0..1 → 0.01, range 0..100 → 1). Specify `step: 1` for integer counts.\r\n```\r\n\r\n### Audio: `viji.audio`\r\n\r\nCheck `isConnected` first.\r\n\r\n| Member | Type | Notes |\r\n|---|---|---|\r\n| `isConnected` | `boolean` | Audio source active |\r\n| `volume.current`, `volume.peak`, `volume.smoothed` | `number` 0..1 | `smoothed` = 200ms decay envelope of `current` |\r\n| `bands.low`, `lowMid`, `mid`, `highMid`, `high` | `number` 0..1 | Instant band energies (20-120, 120-400, 400-1600, 1600-6000, 6000-16000 Hz) |\r\n| `bands.lowSmoothed`, `lowMidSmoothed`, `midSmoothed`, `highMidSmoothed`, `highSmoothed` | `number` 0..1 | 150ms decay envelope siblings (not nested) |\r\n| `beat.kick`, `snare`, `hat`, `any` | `number` 0..1 | Beat energy curves with a 300ms decay; peak on each detected beat then fall off |\r\n| `beat.kickSmoothed`, `snareSmoothed`, `hatSmoothed`, `anySmoothed` | `number` 0..1 | Smoother 500ms decay envelopes; use for ambient pulses |\r\n| `beat.triggers.kick`, `snare`, `hat`, `any` | `boolean` | True for exactly one frame on the matching beat, then auto-resets. OR-accumulated between frames so no beat is lost |\r\n| `beat.events` | `Array<{type, time, strength}>` | All beats detected since the last frame; `type` is `'kick' \\| 'snare' \\| 'hat'` (never `'any'`); `time` in milliseconds; cleared each frame |\r\n| `beat.bpm`, `beat.confidence`, `beat.isLocked` | `number`, `number`, `boolean` | `bpm` is `0` when no audio is connected; once audio connects it tracks the detected tempo (clamped to 60..240) with `120` as a fallback before lock-on. `confidence` in 0..1. `isLocked` is `true` on stable tempo lock |\r\n| `spectral.brightness`, `spectral.flatness` | `number` 0..1 | Spectral centroid; spectral flatness |\r\n| `getFrequencyData()` | `Uint8Array` | Raw FFT bins 0..255 |\r\n| `getWaveform()` | `Float32Array` | Time-domain waveform -1..1 |\r\n\r\nExternal-device audio (`viji.devices[i].audio`) and host-supplied audio streams (`viji.audioStreams[i]`) use the `AudioStreamAPI` shape: `isConnected`, `volume.{current,peak,smoothed}`, `bands.{low...high}` plus each `*Smoothed` sibling, `spectral.{brightness,flatness}`, `getFrequencyData()`, `getWaveform()`. No `beat`, BPM, triggers, or events on streams: those exist only on the main `viji.audio`.\r\n\r\n### Video: `viji.video`\r\n\r\nCheck `isConnected && currentFrame` before drawing.\r\n\r\n| Member | Type | Notes |\r\n|---|---|---|\r\n| `isConnected` | `boolean` | Video source active |\r\n| `currentFrame` | `OffscreenCanvas \\| ImageBitmap \\| null` | Most recent frame |\r\n| `frameWidth`, `frameHeight` | `number` | Source frame size |\r\n| `frameRate` | `number` | Source frame rate |\r\n| `getFrameData()` | `ImageData \\| null` | Pixel data of `currentFrame` |\r\n| `cv` | `VideoCVAPI` | Computer-vision surface: see below |\r\n\r\nCV data outputs (`analysedFrame`, `getAnalysedFrameData()`, `faces`, `hands`, `pose`, `segmentation`) and CV verbs (`enableFaceDetection`, etc.) all live on `viji.video.cv`, not on `viji.video` directly.\r\n\r\n**Aspect ratio.** Camera frames almost never match the canvas aspect. Drawing to `(0, 0, viji.width, viji.height)` stretches video and misaligns CV bounds. Use this helper at module scope:\r\n\r\n```javascript\r\nfunction videoFit(viji, mode = 'cover') {\r\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\r\n const w = viji.width, h = viji.height;\r\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\r\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\r\n const dw = vw * scale, dh = vh * scale;\r\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\r\n}\r\n\r\nconst v = videoFit(viji); // 'cover' or 'contain'\r\nctx.drawImage(viji.video.currentFrame, v.x, v.y, v.width, v.height);\r\n\r\n// CV coords are normalized 0..1 to the source frame. Map through v to align with the fitted video:\r\nviji.video.cv.faces.forEach(face => {\r\n const bx = v.x + face.bounds.x * v.width;\r\n const by = v.y + face.bounds.y * v.height;\r\n const bw = face.bounds.width * v.width;\r\n const bh = face.bounds.height * v.height;\r\n ctx.strokeRect(bx, by, bw, bh);\r\n});\r\n```\r\n\r\nDefault to `'cover'` for live cameras. Use `'contain'` for CV-overlay scenes where bounding boxes near frame edges must stay visible. `viji.video.currentFrame` is the right default for displayed video; reach for `viji.video.cv.analysedFrame` only when sampling pixels at CV-derived positions (segmentation compositing, face-mesh texturing, landmark-anchored color sampling). The common fallback pattern is `viji.video.cv.analysedFrame ?? viji.video.currentFrame` during the brief window before the first CV inference lands.\r\n\r\n### Computer Vision: `viji.video.cv`\r\n\r\nEach CV feature is independent and populates only its own subset of fields. Enable only what you need (each active feature consumes a separate WebGL context for MediaPipe; 4+ on low-end hardware risks context-limit failures).\r\n\r\n```javascript\r\n// Verbs return Promise<void>. await is optional — without it, the model loads\r\n// asynchronously while render runs and data fields populate when ready.\r\n// Safe to call from module scope (always-on CV) or per-frame inside render()\r\n// gated by a viji.toggle(...) (opt-in CV). Idempotent + reference-counted.\r\nawait viji.video.cv.enableFaceDetection(true | false); // populates face.bounds, face.center, face.confidence\r\nawait viji.video.cv.enableFaceMesh(true | false); // populates face.landmarks (468 pts) + face.headPose\r\nawait viji.video.cv.enableEmotionDetection(true | false); // populates face.expressions + face.blendshapes; also loads landmarker (face.landmarks + face.headPose populated)\r\nawait viji.video.cv.enableHandTracking(true | false); // populates viji.video.cv.hands[]\r\nawait viji.video.cv.enablePoseDetection(true | false); // populates viji.video.cv.pose\r\nawait viji.video.cv.enableBodySegmentation(true | false); // populates viji.video.cv.segmentation\r\nviji.video.cv.getActiveFeatures(); // CVFeature[]\r\nviji.video.cv.isProcessing(); // boolean\r\n```\r\n\r\nData outputs on `viji.video.cv`:\r\n- `analysedFrame: OffscreenCanvas | null`: the exact frame paired with the current CV results. `null` until the first inference lands after a feature is enabled. Use the pattern `viji.video.cv.analysedFrame ?? viji.video.currentFrame` to fall back during the startup window.\r\n- `getAnalysedFrameData(): ImageData | null`: pixel data of `analysedFrame`, cached and re-extracted only on new CV results.\r\n\r\n**`viji.video.cv.faces: FaceData[]`**: `id` always present. Other fields are populated only by their source model and read `null` (or empty array for `landmarks`) otherwise — always null-check before reading.\r\n- `bounds: {x,y,width,height} | null` (normalized 0..1) — populated by face detection\r\n- `center: {x,y} | null` (normalized 0..1) — populated by face detection\r\n- `confidence: number | null` (0..1) — populated by face detection\r\n- `landmarks: {x,y,z?}[]` (empty `[]` unless face mesh enabled; 468 pts when populated)\r\n- `headPose: { pitch, yaw, roll } | null` (degrees) — populated by face mesh\r\n- `expressions: { neutral, happy, sad, angry, surprised, disgusted, fearful } | null` (0..1 each) — populated by emotion detection\r\n- `blendshapes | null` — populated by emotion detection; 52 ARKit coefficients (`browDownLeft`/`Right`, `browInnerUp`, `browOuterUpLeft`/`Right`, `cheekPuff`, `cheekSquintLeft`/`Right`, `eyeBlinkLeft`/`Right`, `eyeLookDown`/`In`/`Out`/`UpLeft`/`Right`, `eyeSquintLeft`/`Right`, `eyeWideLeft`/`Right`, `jawForward`/`Left`/`Open`/`Right`, `mouthClose`/`DimpleLeft`/`Right`/`FrownLeft`/`Right`/`Funnel`/`Left`/`LowerDownLeft`/`Right`/`PressLeft`/`Right`/`Pucker`/`Right`/`RollLower`/`Upper`/`ShrugLower`/`Upper`/`SmileLeft`/`Right`/`StretchLeft`/`Right`/`UpperUpLeft`/`Right`, `noseSneerLeft`/`Right`, `tongueOut`: all 0..1)\r\n\r\nIf you need bounds while only running face mesh, either also enable face detection or compute the bounding box from `face.landmarks` min/max yourself.\r\n\r\n**`viji.video.cv.hands: HandData[]`**: `id`, `handedness: 'left' | 'right'`, `confidence`, `bounds`, `landmarks: {x,y,z}[]` (21 points), `palm: {x,y,z}`, `gestures: { fist, openPalm, peace, thumbsUp, thumbsDown, pointing, iLoveYou }` (0..1 confidence each).\r\n\r\n**`viji.video.cv.pose: PoseData | null`**: `confidence`, `landmarks: {x,y,z,visibility}[]` (33 points), plus body-part arrays `face`, `torso`, `leftArm`, `rightArm`, `leftLeg`, `rightLeg`.\r\n\r\n**`viji.video.cv.segmentation: SegmentationData | null`**: `mask: Uint8Array` (each byte is `0` for background or `1` for person; length = `width * height`), `width`, `height`. Note: the shader `u_segmentationMask` sampler is sampled as a normalized `float` (0.0 = background, 1.0 = person); the JS `mask` byte values are 0/1, not 0/255.\r\n\r\n### Input\r\n\r\n**Pointer** (unified mouse and touch): `viji.pointer`\r\n| Member | Type | Notes |\r\n|---|---|---|\r\n| `x`, `y` | `number` | Position in pixels |\r\n| `deltaX`, `deltaY` | `number` | Per-frame movement |\r\n| `isDown` | `boolean` | Pressed or touching |\r\n| `wasPressed`, `wasReleased` | `boolean` | One-frame edge triggers |\r\n| `isInCanvas` | `boolean` | Inside canvas bounds |\r\n| `type` | `'mouse' \\| 'touch' \\| 'none'` | Active input type |\r\n\r\n**Mouse**: `viji.mouse`: `x`, `y`, `isInCanvas`, `isPressed`, `leftButton`, `rightButton`, `middleButton`, `deltaX`, `deltaY`, `wheelDelta`, `wheelX`, `wheelY`, `wasPressed`, `wasReleased`, `wasMoved`.\r\n\r\n**Keyboard**: `viji.keyboard`: `isPressed(key)`, `wasPressed(key)`, `wasReleased(key)`, `activeKeys: Set<string>`, `pressedThisFrame`, `releasedThisFrame`, `lastKeyPressed`, `lastKeyReleased`, `shift`, `ctrl`, `alt`, `meta`.\r\n\r\n**Touch**: `viji.touches`: `count`, `points: TouchPoint[]`, `started`, `moved`, `ended` (each a `TouchPoint[]`), `primary: TouchPoint | null`. Each `TouchPoint`: `id`, `x`, `y`, `pressure`, `radius`, `radiusX`, `radiusY`, `rotationAngle`, `force`, `isInCanvas`, `deltaX`, `deltaY`, `velocity: {x,y}`, `isNew`, `isActive`, `isEnding`.\r\n\r\n### Sensors and external devices\r\n\r\n**`viji.device`**: host device motion and orientation:\r\n- `motion: DeviceMotionData | null`: `acceleration: {x,y,z}` (m/s²), `accelerationIncludingGravity`, `rotationRate: {alpha,beta,gamma}` (deg/s), `interval` (ms).\r\n- `orientation: DeviceOrientationData | null`: `alpha` (0-360° compass), `beta` (-180..180° front-back tilt), `gamma` (-90..90° left-right tilt), `absolute`.\r\n\r\n**`viji.devices`**: `DeviceState[]`, externally connected devices. Each: `id`, `name`, `motion`, `orientation`, `video` (`VideoAPI | null`, same shape as `viji.video` but without CV), `audio` (`AudioStreamAPI | null`, lightweight; no beat or BPM).\r\n\r\n### Host-supplied streams\r\n\r\n- `viji.videoStreams: VideoAPI[]`: extra video sources from the host compositor. Same shape as `viji.video`, including a per-stream `cv` surface with the same enable verbs and result fields; `cv.analysedFrame` is populated on the main stream only. May be empty.\r\n- `viji.audioStreams: AudioStreamAPI[]`: extra audio sources from the host. Lightweight shape (no beat or BPM). May be empty.\r\n\r\n### External libraries\r\n\r\n```javascript\r\nconst THREE = await import('https://esm.sh/three@0.160.0');\r\nconst renderer = new THREE.WebGLRenderer({ canvas: viji.canvas, antialias: true });\r\nrenderer.setSize(viji.width, viji.height, false); // always pass false to skip CSS styling\r\n```\r\n\r\nPin library versions in the import URL. Always pass `viji.canvas` to the renderer. Always pass `false` as the third argument to Three.js `setSize`. Handle resize by comparing `viji.width` / `viji.height` against previous values inside `render()`. Transformers.js needs `env.useBrowserCache = false` (Cache Storage is unavailable in the scene worker; model downloads are cached by the engine anyway). Load ONNX Runtime Web 1.27+ from jsdelivr (`https://cdn.jsdelivr.net/npm/onnxruntime-web@<version>/dist/ort.webgpu.min.mjs`, and set `ort.env.wasm.wasmPaths` to that `dist/` folder); esm.sh builds of it are broken. Run model inference off the render path: start it at an interval when not already running, keep the latest result, and draw that.\r\n\r\n## Template\r\n\r\n```javascript\r\nconst bgColor = viji.color('#1a1a2e', { label: 'Background' });\r\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' }); // implicit step → 0.01\r\nconst count = viji.slider(12, { min: 3, max: 30, step: 1, label: 'Count' }); // explicit step for integer counts\r\n\r\nlet angle = 0;\r\n\r\nfunction render(viji) {\r\n const ctx = viji.useContext('2d');\r\n angle += speed.value * viji.deltaTime;\r\n\r\n ctx.fillStyle = bgColor.value;\r\n ctx.fillRect(0, 0, viji.width, viji.height);\r\n\r\n const cx = viji.width / 2;\r\n const cy = viji.height / 2;\r\n const radius = Math.min(viji.width, viji.height) * 0.3;\r\n const dotSize = Math.min(viji.width, viji.height) * 0.02;\r\n const n = Math.floor(count.value);\r\n\r\n for (let i = 0; i < n; i++) {\r\n const a = angle + (i / n) * Math.PI * 2;\r\n const x = cx + Math.cos(a) * radius;\r\n const y = cy + Math.sin(a) * radius;\r\n const hue = (i / n) * 360;\r\n ctx.beginPath();\r\n ctx.arc(x, y, dotSize, 0, Math.PI * 2);\r\n ctx.fillStyle = `hsl(${hue}, 80%, 60%)`;\r\n ctx.fill();\r\n }\r\n}\r\n```\r\n",
|
|
9853
9878
|
"p5": "# Viji API: P5 Renderer Reference\n\nP5 scenes use P5.js 1.9.4 inside a Web Worker on an `OffscreenCanvas`. P5 runs in **instance mode**: every P5 function and constant requires the `p5.` prefix. The first line of the scene must be `// @renderer p5` (2D) or `// @renderer p5 webgl` (3D / WEBGL).\n\n## Architecture\n\n- The global `viji` object exposes canvas, timing, audio, video, CV, input, sensors, parameters: identical to Native.\n- The canvas and its rendering context are managed by P5. **Do not call `createCanvas()`.** Viji creates the canvas in the mode declared by the `// @renderer` directive.\n- `viji.useContext()` is **not available** in P5 mode (the context belongs to P5).\n- Top-level code runs once. `function setup(viji, p5)` runs once for configuration. `function render(viji, p5)` runs every frame.\n- The `p5` instance passed to `setup` and `render` is the P5 instance. Use `p5.background`, `p5.fill`, `p5.circle`, `p5.PI`, etc. Unprefixed names will not resolve.\n\n## Rules\n\n1. **Always** add `// @renderer p5` (2D) or `// @renderer p5 webgl` (WEBGL) as the first line.\n2. **Always** use `render(viji, p5)` (not `draw()`) and `setup(viji, p5)` (not `setup()`).\n3. **Always** prefix every P5 function and constant with `p5.`:\n - `background(0)` -> `p5.background(0)`\n - `fill(255)` -> `p5.fill(255)`\n - `PI` -> `p5.PI`, `TWO_PI` -> `p5.TWO_PI`, `HSB` -> `p5.HSB`\n - `createVector(1, 0)` -> `p5.createVector(1, 0)`\n - `map(v, 0, 1, 0, 255)` -> `p5.map(v, 0, 1, 0, 255)`\n - `noise(x)` -> `p5.noise(x)`, `random()` -> `p5.random()`\n4. **Never** call `createCanvas()`. The canvas is created and managed by Viji.\n5. **Never** use `preload()`. Use `viji.image(null, { label })` for images, or `fetch()` inside `setup()`.\n6. **Never** use P5 event callbacks (`mousePressed`, `mouseDragged`, `mouseReleased`, `keyPressed`, `keyReleased`, `keyTyped`, `touchStarted`, `touchMoved`, `touchEnded`). Read state in `render()` via `viji.pointer`, `viji.mouse`, `viji.keyboard`, `viji.touches`. Use `wasPressed` / `wasReleased` for one-frame edges.\n7. **Never** use `loadImage()`, `loadFont()`, `loadJSON()`, `loadModel()`, `loadShader()`. Use `viji.image()` or `fetch()`.\n8. **Never** use `p5.frameRate()`, `p5.save()`, `p5.saveCanvas()`, `p5.saveFrames()`. The host controls frame rate and capture.\n9. **Never** use `createCapture()` or `createVideo()`. Use `viji.video.*`.\n10. **Never** use `p5.dom` or `p5.sound`. Use Viji parameters and `viji.audio.*`.\n11. **Never** access `window`, `document`, `Image()`, `localStorage`. `fetch()` is available.\n12. **Always** declare parameters at the top level as individual `const`s (`const speed = viji.slider(...)`), never inside `render()` or `setup()`. `label` is optional but recommended (falls back to the variable name).\n13. **Always** read parameters via `.value`. Color parameters also expose `.rgb` (matches `colorMode(RGB, 255)`) and `.hsb` (matches `colorMode(HSB, 360, 100, 100)`). Color defaults accept hex, `{ r, g, b }` (0..255), `{ h, s, b }` (0..360 / 0..100), or CSS color strings.\n14. **Always** use `viji.width` and `viji.height` for canvas dimensions. Never hardcode pixel sizes.\n15. **Always** use `viji.deltaTime` for animation:\n ```javascript\n let angle = 0;\n function render(viji, p5) { angle += speed.value * viji.deltaTime; }\n ```\n16. **Never** multiply `viji.time` by a parameter; it causes visible jumps when the parameter changes. Use a `deltaTime` accumulator. This also applies to nested multiplications; each independent speed gets its own accumulator.\n17. **Never** allocate objects, arrays, or strings inside `render()`. Pre-allocate at the top level and reuse.\n18. **Image parameters with P5:** use `.p5` (a `P5Image`) instead of `.value` when passing to `p5.image()`:\n ```javascript\n const photo = viji.image(null, { label: 'Photo' });\n function render(viji, p5) {\n if (photo.value) p5.image(photo.p5, 0, 0, viji.width, viji.height);\n }\n ```\n19. **Video drawing.** In 2D, use `p5.image(viji.video.currentFrame, ...)` or `p5.drawingContext.drawImage(...)`. In WEBGL, use `p5.image(viji.video.currentFrame, ...)` only (`p5.drawingContext` is WebGL, not Canvas 2D). **Always preserve aspect ratio** via the `videoFit` helper below.\n20. `p5.createGraphics(w, h)` is **2D only** (`createGraphics(w, h, p5.WEBGL)` is not supported). It creates an OffscreenCanvas internally.\n21. Fonts: `p5.textFont()` only with CSS generic names (`monospace`, `serif`, `sans-serif`). `loadFont()` is not available.\n22. `p5.tint()` and `p5.blendMode()` work normally.\n23. **Canvas mode is declared by the `// @renderer` directive.** Use `// @renderer p5` for 2D, `// @renderer p5 webgl` for WEBGL. Never call `createCanvas()` or `createCanvas(..., p5.WEBGL)`.\n24. In WEBGL scenes, `p5.drawingContext` is a WebGL context. Use P5 3D drawing primitives or `p5.image()` / textures for images and video.\n25. `p5.pixelDensity()` defaults to 1 in the worker. `p5.loadPixels()` and `p5.pixels[]` work in 2D scenes; WEBGL pixel readback follows standard P5.js rules.\n26. **Always** check `viji.audio.isConnected` before reading audio data.\n27. **Always** check `viji.video.isConnected && viji.video.currentFrame` before drawing video. When not connected, keep rendering the idle visual — don't draw on-canvas \"connect a camera/mic\" instructions.\n28. Enable CV by intent: always-on from module scope when CV is the scene's defining purpose, otherwise gated behind a `viji.toggle`.\n29. **Always** set `category` on input-dependent parameters (`'audio'`, `'video'`, `'interaction'`). Don't add on/off toggles to enable/disable host-gated inputs (audio, video, mouse, keyboard, touch, sensors, devices) — the host controls those. CV is the only exception (it gets a toggle when not always-on).\n30. **Always** type any function parameter that receives the `viji` instance or the `p5` instance, including `setup(viji, p5)`, `render(viji, p5)`, and any helpers that take either. Monaco's TS checker flags API misuse (wrong nesting, typo'd property names) at edit time. Scene code is transpiled via sucrase before execution, so TS annotations are stripped at runtime — safe to include. Note: the p5-instance type is the lowercase namespace `p5` (same name as the conventional parameter; TS resolves the value and the type from separate namespaces, so `p5: p5` is legal and common).\n\n ```javascript\n function setup(viji: VijiAPI, p5: p5) { ... }\n function render(viji: VijiAPI, p5: p5) { ... }\n function videoFit(viji: VijiAPI, mode: 'cover' | 'contain') { ... }\n ```\n31. All Viji types are ambient globals — `VijiAPI`, the parameter and config types (`SliderParameter`, `SliderConfig`, ...), input and CV types (`TouchPoint`, `FaceData`, ...). Annotate with them directly; never import, alias, or re-derive them (no `ReturnType<typeof viji.slider>` — the named type exists). For the exact type of any `viji` member, fetch the `p5-api-reference` docs page instead of guessing.\n\n## Video aspect helper\n\n```javascript\nfunction videoFit(viji, mode = 'cover') {\n const vw = viji.video.frameWidth, vh = viji.video.frameHeight;\n const w = viji.width, h = viji.height;\n if (!vw || !vh) return { x: 0, y: 0, width: 0, height: 0 };\n const scale = mode === 'cover' ? Math.max(w / vw, h / vh) : Math.min(w / vw, h / vh);\n const dw = vw * scale, dh = vh * scale;\n return { x: (w - dw) / 2, y: (h - dh) / 2, width: dw, height: dh };\n}\n\nif (viji.video.isConnected && viji.video.currentFrame) {\n const v = videoFit(viji, 'cover');\n p5.image(viji.video.currentFrame, v.x, v.y, v.width, v.height);\n}\n\n// CV bounds and landmarks are normalized 0..1 to the source frame.\n// Map them through v to align with the drawn video.\n```\n\n## P5 to Viji mapping\n\n| Standard P5.js | Viji P5 equivalent |\n|---|---|\n| `width` / `height` | `viji.width` / `viji.height` |\n| `mouseX` / `mouseY` | `viji.pointer.x` / `viji.pointer.y` (or `viji.mouse.x` / `viji.mouse.y`) |\n| `mouseIsPressed` | `viji.pointer.isDown` |\n| `mouseButton === LEFT` | `viji.mouse.leftButton` |\n| `keyIsPressed` | `viji.keyboard.activeKeys.size > 0` |\n| `keyIsDown(code)` | `viji.keyboard.isPressed('keyName')` |\n| `key` | `viji.keyboard.lastKeyPressed` |\n| `frameCount` | `viji.frameCount` (or a `viji.deltaTime` accumulator) |\n| `frameRate(n)` | Remove. Host controls frame rate. |\n| `createCanvas(w, h)` | Remove. Canvas is provided. |\n| `preload()` | Remove. Use `viji.image()` or `fetch()` in `setup()`. |\n| `loadImage(url)` | `viji.image(null, { label: 'Image' })`; pass `.p5` to `p5.image()`. |\n| `save()` / `saveCanvas()` | Remove. Host handles capture. |\n| `mousePressed()` / `mouseReleased()` callbacks | Check `viji.pointer.wasPressed` / `wasReleased` inside `render()`. |\n| `keyPressed()` / `keyReleased()` callbacks | Check `viji.keyboard.wasPressed(key)` / `wasReleased(key)` inside `render()`. |\n\n## API reference\n\nAll `viji.*` members are identical to Native (same object, same types).\n\n### Canvas and timing\n\n| Member | Type | Notes |\n|---|---|---|\n| `viji.canvas` | `OffscreenCanvas` | Managed by P5 |\n| `viji.width`, `viji.height` | `number` | Canvas dimensions |\n| `viji.time` | `number` | Seconds since scene start |\n| `viji.deltaTime` | `number` | Seconds since last frame |\n| `viji.frameCount` | `number` | Total frames rendered |\n| `viji.fps` | `number` | Target frame rate |\n\n### Parameters\n\n```javascript\nviji.slider(default, { min?, max?, step?, label, group?, category? }) // .value: number\nviji.color(default, { label, group?, category? }) // .value, .rgb, .hsb\nviji.toggle(default, { label, group?, category? }) // .value: boolean\nviji.select(default, { options, label, group?, category? }) // .value: string | number\nviji.number(default, { min?, max?, step?, label, group?, category? }) // .value: number\nviji.text(default, { label, group?, category?, maxLength? }) // .value: string\nviji.image(null, { label, group?, category? }) // .value: ImageBitmap | null, .p5: P5Image\nviji.button({ label, description?, group?, category? }) // .value: boolean (1 frame)\nviji.coordinate(default, { step?, label, group?, category? }) // .value: { x, y }, range -1..1\n// slider/number `step` defaults to a power of 10 giving ~100 positions across min..max\n// (e.g., range 0..1 → 0.01, range 0..100 → 1). Specify `step: 1` for integer counts.\n```\n\n### Audio: `viji.audio`\n\nCheck `isConnected` first. Members:\n\n`isConnected` (boolean); `volume.{current, peak, smoothed}` (0..1; `smoothed` 200ms decay); `bands.{low, lowMid, mid, highMid, high}` (0..1; ranges 20-120, 120-400, 400-1600, 1600-6000, 6000-16000 Hz) and the sibling `*Smoothed` envelopes (150ms decay); `beat.{kick, snare, hat, any}` (0..1, 300ms decay curves; peak on each detected beat); `beat.{kickSmoothed, snareSmoothed, hatSmoothed, anySmoothed}` (500ms decay envelopes); `beat.triggers.{kick, snare, hat, any}` (boolean, true for exactly one frame, OR-accumulated between frames); `beat.events: Array<{type, time, strength}>` (`type` is `'kick' | 'snare' | 'hat'`; `time` ms; cleared each frame); `beat.bpm` (`0` when no audio is connected; once audio connects it tracks the detected tempo clamped to 60..240, with `120` as a fallback before lock-on), `beat.confidence` (0..1), `beat.isLocked` (boolean); `spectral.{brightness, flatness}` (0..1); `getFrequencyData(): Uint8Array` (1024 FFT bins, 0..255); `getWaveform(): Float32Array` (2048 samples, -1..1).\n\nExternal-device audio (`viji.devices[i].audio`) and host streams (`viji.audioStreams[i]`) follow the `AudioStreamAPI` shape: `isConnected`, `volume`, `bands` (instant + smoothed siblings), `spectral`, `getFrequencyData`, `getWaveform`. No beat, BPM, triggers, or events on streams.\n\n### Video: `viji.video`\n\nCheck `isConnected && currentFrame` before drawing. Members:\n\n`isConnected` (boolean); `currentFrame` (`OffscreenCanvas | ImageBitmap | null`); `frameWidth`, `frameHeight`, `frameRate`; `getFrameData(): ImageData | null`; `cv: VideoCVAPI` (CV outputs and verbs live here, not on `viji.video` directly).\n\n### Computer Vision: `viji.video.cv`\n\nEach CV feature is independent and populates only its own subset of fields. Enable only what you need (each active feature uses a separate WebGL context for MediaPipe; 4+ on low-end hardware risks context-limit failures).\n\n```javascript\n// Verbs return Promise<void>. await is optional. Safe to call from module\n// scope (always-on CV) or per-frame inside render() gated by a viji.toggle(...)\n// (opt-in CV). Idempotent + reference-counted.\nawait viji.video.cv.enableFaceDetection(true | false); // face.bounds, face.center, face.confidence\nawait viji.video.cv.enableFaceMesh(true | false); // face.landmarks (468 pts) + face.headPose\nawait viji.video.cv.enableEmotionDetection(true | false); // face.expressions + face.blendshapes; also loads landmarker\nawait viji.video.cv.enableHandTracking(true | false); // viji.video.cv.hands[]\nawait viji.video.cv.enablePoseDetection(true | false); // viji.video.cv.pose\nawait viji.video.cv.enableBodySegmentation(true | false); // viji.video.cv.segmentation\nviji.video.cv.getActiveFeatures(); // CVFeature[]\nviji.video.cv.isProcessing(); // boolean\n```\n\nData outputs on `viji.video.cv`:\n- `analysedFrame: OffscreenCanvas | null`: the frame paired with the current CV results. `null` until the first inference lands. Use `viji.video.cv.analysedFrame ?? viji.video.currentFrame` to fall back during startup.\n- `getAnalysedFrameData(): ImageData | null`: pixel data of `analysedFrame`.\n\n**`faces: FaceData[]`**: `id` always present. Other fields populated only by their source model; otherwise `null` (`landmarks` is `[]` when empty). Always null-check.\n- `bounds: {x,y,width,height} | null` — populated by face detection\n- `center: {x,y} | null` — populated by face detection\n- `confidence: number | null` — populated by face detection\n- `landmarks: {x,y,z?}[]` (empty `[]` unless face mesh enabled)\n- `headPose: {pitch, yaw, roll} | null` — populated by face mesh\n- `expressions: {neutral, happy, sad, angry, surprised, disgusted, fearful} | null` (0..1 each) — populated by emotion detection\n- `blendshapes | null` (52 ARKit coefficients 0..1) — populated by emotion detection\n\nIf you need `bounds` while only running face mesh, either also enable face detection or compute it from `face.landmarks` min/max.\n\n**`hands: HandData[]`**: `id`, `handedness: 'left' | 'right'`, `confidence`, `bounds`, `landmarks` (21 points), `palm`, `gestures` (`fist`, `openPalm`, `peace`, `thumbsUp`, `thumbsDown`, `pointing`, `iLoveYou` 0..1).\n\n**`pose: PoseData | null`**: `confidence`, `landmarks` (33 points), body-part arrays `face`, `torso`, `leftArm`, `rightArm`, `leftLeg`, `rightLeg`.\n\n**`segmentation: SegmentationData | null`**: `mask: Uint8Array` (each byte is `0` for background or `1` for person; length = `width * height`), `width`, `height`.\n\n### Input\n\n- `viji.pointer`: `x`, `y`, `deltaX`, `deltaY`, `isDown`, `wasPressed`, `wasReleased`, `isInCanvas`, `type` (`'mouse' | 'touch' | 'none'`).\n- `viji.mouse`: `x`, `y`, `isInCanvas`, `isPressed`, `leftButton`, `rightButton`, `middleButton`, `deltaX`, `deltaY`, `wheelDelta`, `wheelX`, `wheelY`, `wasPressed`, `wasReleased`, `wasMoved`.\n- `viji.keyboard`: `isPressed(key)`, `wasPressed(key)`, `wasReleased(key)`, `activeKeys`, `pressedThisFrame`, `releasedThisFrame`, `lastKeyPressed`, `lastKeyReleased`, `shift`, `ctrl`, `alt`, `meta`.\n- `viji.touches`: `count`, `points`, `started`, `moved`, `ended`, `primary`. Each `TouchPoint`: `id`, `x`, `y`, `pressure`, `radius`, `radiusX`, `radiusY`, `rotationAngle`, `force`, `isInCanvas`, `deltaX`, `deltaY`, `velocity`, `isNew`, `isActive`, `isEnding`.\n\n### Sensors and external devices\n\n`viji.device.motion`: `acceleration {x,y,z}` (m/s²), `accelerationIncludingGravity`, `rotationRate {alpha,beta,gamma}` (deg/s), `interval` (ms).\n`viji.device.orientation`: `alpha` (0-360°), `beta` (-180..180°), `gamma` (-90..90°), `absolute`.\n\n`viji.devices: DeviceState[]`: each entry: `id`, `name`, `motion`, `orientation`, `video` (`VideoAPI | null`, no CV), `audio` (`AudioStreamAPI | null`, lightweight).\n\n### Host streams\n\n`viji.videoStreams: VideoAPI[]`: extra video sources. Same shape as `viji.video`, including a per-stream `cv` surface (same verbs and result fields; `cv.analysedFrame` is main-stream only). May be empty.\n`viji.audioStreams: AudioStreamAPI[]`: extra audio sources. May be empty. Lightweight shape (no beat / BPM).\n\n## Template\n\n```javascript\n// @renderer p5\n\nconst bgColor = viji.color('#1a1a2e', { label: 'Background' });\nconst speed = viji.slider(1, { min: 0.1, max: 5, label: 'Speed' }); // implicit step → 0.01\nconst count = viji.slider(8, { min: 3, max: 30, step: 1, label: 'Count' }); // explicit step for integer counts\n\nlet angle = 0;\n\nfunction setup(viji, p5) {\n p5.colorMode(p5.HSB, 360, 100, 100);\n}\n\nfunction render(viji, p5) {\n angle += speed.value * viji.deltaTime;\n p5.background(bgColor.value);\n\n const cx = viji.width / 2;\n const cy = viji.height / 2;\n const radius = p5.min(viji.width, viji.height) * 0.3;\n const dotSize = p5.min(viji.width, viji.height) * 0.04;\n const n = p5.floor(count.value);\n\n p5.noStroke();\n for (let i = 0; i < n; i++) {\n const a = angle + (i / n) * p5.TWO_PI;\n const x = cx + p5.cos(a) * radius;\n const y = cy + p5.sin(a) * radius;\n p5.fill((i / n) * 360, 80, 90);\n p5.circle(x, y, dotSize);\n }\n}\n```\n",
|
|
9854
9879
|
"shader": "# Viji API: Shader Renderer Reference\n\nShader scenes are GLSL fragment shaders running on a fullscreen quad. Write helper functions and `void main()`. Precision and all uniforms (built-in plus those generated from `@viji-*` directives) are auto-injected by Viji. The first line must be `// @renderer shader` (or `#version 300 es` followed by `// @renderer shader` for GLSL ES 3.00).\n\n## Architecture\n\n- The shader is **GLSL ES 1.00** by default. Opt into ES 3.00 by making `#version 300 es` the literal first line.\n- Viji auto-injects `precision mediump float;` and every uniform declaration. Never declare them yourself.\n- ES 3.00 requires `out vec4 fragColor;` before `main` and `fragColor = ...` instead of `gl_FragColor`. ES 3.00 uses `texture()` instead of `texture2D()`.\n- If the shader references `fwidth`, Viji auto-injects `#extension GL_OES_standard_derivatives : enable`.\n- The `backbuffer` sampler is auto-enabled when referenced. It carries the previous frame.\n\n## Rules\n\n1. **Always** add `// @renderer shader` as the first line (or as line 2 after `#version 300 es`).\n2. **Never** declare `precision mediump float;` or `precision highp float;`. Viji auto-injects precision.\n3. **Never** redeclare built-in uniforms (`u_time`, `u_resolution`, `u_mouse`, audio / video / CV uniforms, etc.). They are auto-injected.\n4. **Never** redeclare parameter uniforms. They are auto-generated from `@viji-*` directives.\n5. **Never** use the `u_` prefix for your own parameter names. It is reserved for Viji's built-ins. Use descriptive names (`speed`, `colorMix`, `intensity`).\n6. `@viji-*` parameter directives work with `//` comments only. Block comments (`/* */`) are not parsed.\n7. **Always** use `@viji-accumulator` instead of `u_time * speed` for parameter-driven animation. Multiplying `u_time` by a parameter causes visible jumps when sliders change.\n\n ```glsl\n // @viji-slider:speed label:\"Speed\" default:1.0 min:0.1 max:5.0\n // @viji-accumulator:phase rate:speed\n float wave = sin(phase);\n ```\n\n This also applies to **nested** multiplications. Each independent speed needs its own accumulator:\n\n ```glsl\n // @viji-accumulator:phase rate:speed\n // @viji-accumulator:rotPhase rate:rotSpeed\n ```\n8. For `backbuffer` (previous frame): reference it directly in code. Viji auto-detects and enables it. Sample with `texture2D(backbuffer, uv)` (ES 1.00) or `texture(backbuffer, uv)` (ES 3.00).\n9. Remove any `#ifdef GL_ES` / `precision` blocks; Viji handles them.\n10. **Always** set `category` on input-dependent directives: `category:audio`, `category:video`, `category:interaction`. Don't add on/off toggles to enable/disable host-gated inputs (audio, video, mouse, keyboard, touch, sensors, devices) — the host controls those. CV activations are the exception (the `@viji-cv:` directives below).\n11. **CV features must be activated via `// @viji-cv:<featureToken>` directives.** Without the matching `@viji-cv:` activation, CV uniforms (`u_faceCount`, `u_face0*`, `u_handCount`, `u_poseDetected`, `u_segmentationMask`, etc.) read zero. Choose the form by intent: use the bare form (`// @viji-cv:faceDetection`) for always-on detection when CV is the scene's defining purpose, or the toggleable form (`// @viji-cv:faceDetection label:\"Face Detection\" default:false`) — which synthesizes both a host-side toggle parameter and a `bool <featureToken>` shader uniform mirroring its state — when CV is optional or the scene uses several heavy features. The six tokens: `faceDetection`, `faceMesh`, `emotionDetection`, `handTracking`, `poseDetection`, `bodySegmentation`.\n\n## Parameter directives\n\n```glsl\n// @viji-slider:name label:\"Label\" default:1.0 min:0.0 max:5.0 step:0.1\n// -> uniform float name;\n\n// @viji-color:name label:\"Color\" default:#ff6600\n// -> uniform vec3 name; (RGB 0..1)\n// default: accepts #rrggbb, #rgb, vec3(r,g,b) (0..1), rgb(r,g,b) (0..255),\n// hsl(h, s%, l%) (h: 0..360), hsb(h, s, b) (h: 0..360, s/b: 0..100)\n\n// @viji-toggle:name label:\"Toggle\" default:false\n// -> uniform bool name;\n\n// @viji-select:name label:\"Mode\" default:0 options:[\"Solid\",\"Gradient\",\"Noise\"]\n// -> uniform int name; (0-based)\n\n// @viji-number:name label:\"Count\" default:10.0 min:1.0 max:100.0 step:1.0\n// -> uniform float name;\n\n// @viji-image:name label:\"Texture\"\n// -> uniform sampler2D name;\n\n// @viji-button:name label:\"Reset\"\n// -> uniform bool name; (true for one frame on press)\n\n// @viji-coordinate:name label:\"Origin\" default:[0.0,0.0]\n// -> uniform vec2 name; (each component -1..1)\n\n// @viji-accumulator:name rate:speed\n// -> uniform float name; (CPU-side: += rate * deltaTime each frame)\n```\n\nAll directives support `group:\"GroupName\"` and `category:\"audio|video|interaction|general\"`.\n\n## Uniform reference\n\nAll uniforms below are always available. Do not declare them.\n\n### Core\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_resolution` | `vec2` | Canvas width and height in pixels |\n| `u_time` | `float` | Seconds since scene start |\n| `u_deltaTime` | `float` | Seconds since last frame |\n| `u_frame` | `int` | Current frame number |\n| `u_fps` | `float` | Target frame rate (host-controlled) |\n\n### Mouse\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_mouse` | `vec2` | Mouse position in pixels (WebGL coords, bottom-left origin) |\n| `u_mouseInCanvas` | `bool` | Mouse inside canvas |\n| `u_mousePressed` | `bool` | Any button pressed |\n| `u_mouseLeft`, `u_mouseRight`, `u_mouseMiddle` | `bool` | Specific buttons |\n| `u_mouseDelta` | `vec2` | Per-frame movement |\n| `u_mouseWheel` | `float` | Scroll wheel delta |\n| `u_mouseWasPressed`, `u_mouseWasReleased` | `bool` | One-frame edges |\n\n### Pointer (unified mouse / touch)\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_pointer` | `vec2` | Primary input position in pixels (WebGL coords) |\n| `u_pointerDelta` | `vec2` | Primary input movement delta |\n| `u_pointerDown` | `bool` | Primary input active |\n| `u_pointerWasPressed`, `u_pointerWasReleased` | `bool` | One-frame edges |\n| `u_pointerInCanvas` | `bool` | Inside canvas |\n\n### Keyboard\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_keySpace`, `u_keyShift`, `u_keyCtrl`, `u_keyAlt` | `bool` | Modifier and space |\n| `u_keyW`, `u_keyA`, `u_keyS`, `u_keyD` | `bool` | WASD |\n| `u_keyUp`, `u_keyDown`, `u_keyLeft`, `u_keyRight` | `bool` | Arrows |\n| `u_keyboard` | `sampler2D` | Full keyboard state (256x3 LUMINANCE). Row 0 = held, row 1 = pressed-this-frame, row 2 = toggle. Access: `texelFetch(u_keyboard, ivec2(keyCode, row), 0).r` |\n\n### Touch\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_touchCount` | `int` | Active touches (0-5) |\n| `u_touch0`-`u_touch4` | `vec2` | Touch point positions in pixels |\n\n### Audio (scalars)\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_audioVolume`, `u_audioPeak` | `float` | RMS / peak 0..1 |\n| `u_audioVolumeSmoothed` | `float` | 200ms decay envelope of `u_audioVolume` |\n| `u_audioLow`, `u_audioLowMid`, `u_audioMid`, `u_audioHighMid`, `u_audioHigh` | `float` | Band energies 0..1 (20-120, 120-400, 400-1600, 1600-6000, 6000-16000 Hz) |\n| `u_audioLowSmoothed` - `u_audioHighSmoothed` | `float` | 150ms decay envelopes of the bands |\n| `u_audioKick`, `u_audioSnare`, `u_audioHat`, `u_audioAny` | `float` | Beat energy curves with 300ms decay; peak on each detected beat then fall off |\n| `u_audioKickSmoothed` - `u_audioAnySmoothed` | `float` | Smoother 500ms decay envelopes |\n| `u_audioKickTrigger`, `u_audioSnareTrigger`, `u_audioHatTrigger`, `u_audioAnyTrigger` | `bool` | True for exactly one frame on the matching beat, then auto-resets; OR-accumulated between frames |\n| `u_audioBPM`, `u_audioConfidence`, `u_audioIsLocked` | `float`, `float`, `bool` | `u_audioBPM` is `0.0` when no audio is connected; once audio connects it tracks the detected tempo (clamped to 60..240) with `120.0` as a fallback before lock-on. `u_audioConfidence` 0..1. `u_audioIsLocked` true on stable tempo lock |\n| `u_audioBrightness`, `u_audioFlatness` | `float` | Spectral features 0..1 |\n\n### Audio (textures, main source only)\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_audioFFT` | `sampler2D` | FFT spectrum (1024 bins, 0..255) |\n| `u_audioWaveform` | `sampler2D` | Time-domain waveform (-1..1) |\n\nAdditional audio streams and device audio expose scalar uniforms only (no FFT / waveform textures): see \"Streams\" below.\n\n### Video\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_video` | `sampler2D` | Most recent video frame |\n| `u_videoAnalysed` | `sampler2D` | Frame paired with current CV uniforms |\n| `u_videoAnalysedAvailable` | `bool` | True after first CV result lands |\n| `u_videoResolution` | `vec2` | Source frame size in pixels |\n| `u_videoFrameRate` | `float` | Source frame rate |\n| `u_videoConnected` | `bool` | Video source active |\n\n**Aspect handling.** Camera frames almost never match the canvas aspect. Sampling `texture2D(u_video, uv)` with canvas UV stretches the video and misaligns CV uniforms. Use these helpers at the top of every video / CV shader:\n\n```glsl\n// mode: 1 = cover (fills canvas, video edges cropped)\n// 0 = contain (fits video, canvas letterboxed)\nvec2 vijiVideoUV(vec2 uv, int mode) {\n vec2 canvas = u_resolution;\n vec2 video = u_videoResolution;\n if (video.x == 0.0 || video.y == 0.0) return uv;\n float canvasAspect = canvas.x / canvas.y;\n float videoAspect = video.x / video.y;\n vec2 scale = vec2(1.0);\n if (mode == 1) {\n scale = canvasAspect > videoAspect\n ? vec2(1.0, canvasAspect / videoAspect)\n : vec2(videoAspect / canvasAspect, 1.0);\n } else {\n scale = canvasAspect > videoAspect\n ? vec2(videoAspect / canvasAspect, 1.0)\n : vec2(1.0, canvasAspect / videoAspect);\n }\n return (uv - 0.5) / scale + 0.5;\n}\n\nbool vijiInVideo(vec2 uv) {\n return all(greaterThanEqual(uv, vec2(0.0))) && all(lessThanEqual(uv, vec2(1.0)));\n}\n```\n\nDefault to mode `1` (cover) for live cameras. Use `0` (contain) for CV-overlay shaders where features near edges must stay visible. CV uniforms (`u_face0Center`, etc.) share the same bottom-up convention as `videoUV`; compare them directly without flipping.\n\n**`u_video` vs `u_videoAnalysed`.** Default to `u_video` for displayed video. Reach for `u_videoAnalysed` only when sampling pixels at CV-derived positions (segmentation compositing onto the body, sampling skin under face landmarks, face-mesh texturing). For shaders that consume CV uniforms without sampling at CV positions, stay on `u_video` (`u_videoAnalysed` advances only on inference completion and will stutter the displayed video). When `u_videoAnalysed` is the right choice, gate on `u_videoAnalysedAvailable` for the brief startup window:\n\n```glsl\nvec4 source = u_videoAnalysedAvailable\n ? texture2D(u_videoAnalysed, uv)\n : texture2D(u_video, uv);\n```\n\n### CV: face detection\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_faceCount` | `int` | Detected faces (0-1) |\n| `u_face0Bounds` | `vec4` | `(x, y, width, height)` normalized 0..1 |\n| `u_face0Center` | `vec2` | Center normalized 0..1 |\n| `u_face0HeadPose` | `vec3` | `(pitch, yaw, roll)` in degrees |\n| `u_face0Confidence` | `float` | 0..1 |\n| `u_face0Neutral` - `u_face0Fearful` | `float` | 7 expression scores 0..1 |\n\n**52 blendshape uniforms** (all `float`, 0..1, ARKit names prefixed with `u_face0`):\n`BrowDownLeft`, `BrowDownRight`, `BrowInnerUp`, `BrowOuterUpLeft`, `BrowOuterUpRight`, `CheekPuff`, `CheekSquintLeft`, `CheekSquintRight`, `EyeBlinkLeft`, `EyeBlinkRight`, `EyeLookDownLeft`, `EyeLookDownRight`, `EyeLookInLeft`, `EyeLookInRight`, `EyeLookOutLeft`, `EyeLookOutRight`, `EyeLookUpLeft`, `EyeLookUpRight`, `EyeSquintLeft`, `EyeSquintRight`, `EyeWideLeft`, `EyeWideRight`, `JawForward`, `JawLeft`, `JawOpen`, `JawRight`, `MouthClose`, `MouthDimpleLeft`, `MouthDimpleRight`, `MouthFrownLeft`, `MouthFrownRight`, `MouthFunnel`, `MouthLeft`, `MouthLowerDownLeft`, `MouthLowerDownRight`, `MouthPressLeft`, `MouthPressRight`, `MouthPucker`, `MouthRight`, `MouthRollLower`, `MouthRollUpper`, `MouthShrugLower`, `MouthShrugUpper`, `MouthSmileLeft`, `MouthSmileRight`, `MouthStretchLeft`, `MouthStretchRight`, `MouthUpperUpLeft`, `MouthUpperUpRight`, `NoseSneerLeft`, `NoseSneerRight`, `TongueOut`.\n\n### CV: hands\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_handCount` | `int` | Detected hands (0-2) |\n| `u_leftHandPalm`, `u_rightHandPalm` | `vec3` | Palm position `(x, y, z)` |\n| `u_leftHandConfidence`, `u_rightHandConfidence` | `float` | 0..1 |\n| `u_leftHandBounds`, `u_rightHandBounds` | `vec4` | Bounds normalized 0..1 |\n| `u_leftHandFist` - `u_leftHandILoveYou` | `float` | 7 left-hand gesture scores |\n| `u_rightHandFist` - `u_rightHandILoveYou` | `float` | 7 right-hand gesture scores |\n\n### CV: pose\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_poseDetected` | `bool` | Pose detected |\n| `u_poseConfidence` | `float` | 0..1 |\n| `u_nosePosition` | `vec2` | Nose landmark, normalized 0..1 |\n| `u_leftShoulderPosition` … `u_rightAnklePosition` | `vec2` | Joint landmarks (shoulders, elbows, wrists, hips, knees, ankles) |\n\n### CV: body segmentation\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_segmentationMask` | `sampler2D` | 0 = background, 1 = person |\n| `u_segmentationRes` | `vec2` | Mask resolution in pixels |\n\n### Device sensors\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_deviceAcceleration` | `vec3` | Acceleration without gravity (m/s²) |\n| `u_deviceAccelerationGravity` | `vec3` | Acceleration including gravity (m/s²) |\n| `u_deviceRotationRate` | `vec3` | Rotation rate (deg/s) |\n| `u_deviceOrientation` | `vec3` | `(alpha, beta, gamma)` in degrees |\n| `u_deviceOrientationAbsolute` | `bool` | Using magnetometer |\n\n### External devices\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_deviceCount` | `int` | Device video sources (0-8) |\n| `u_externalDeviceCount` | `int` | External devices (0-8) |\n| `u_device0` - `u_device7` | `sampler2D` | Device camera textures |\n| `u_device0Resolution` - `u_device7Resolution` | `vec2` | Per-device resolution |\n| `u_device0Connected` - `u_device7Connected` | `bool` | Per-device connection |\n| `u_device0Acceleration` - `u_device7Acceleration` | `vec3` | Per-device acceleration (no gravity) |\n| `u_device0AccelerationGravity` - `u_device7AccelerationGravity` | `vec3` | Per-device acceleration (with gravity) |\n| `u_device0RotationRate` - `u_device7RotationRate` | `vec3` | Per-device rotation rate |\n| `u_device0Orientation` - `u_device7Orientation` | `vec3` | Per-device orientation |\n\n### Streams (compositor)\n\nVideo streams (host-provided):\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_videoStreamCount` | `int` | Active streams (0-8) |\n| `u_videoStream0` - `u_videoStream7` | `sampler2D` | Stream textures |\n| `u_videoStream0Resolution` - `u_videoStream7Resolution` | `vec2` | Per-stream resolution |\n| `u_videoStream0Connected` - `u_videoStream7Connected` | `bool` | Per-stream connection |\n\nAudio streams (additional sources, including device audio):\n\n| Uniform | Type | Description |\n|---|---|---|\n| `u_audioStreamCount` | `int` | Active streams (0-8) |\n| `u_audioStream0Connected` - `u_audioStream7Connected` | `bool` | Per-slot active |\n| `u_audioStream{i}Volume` | `float` | Volume 0..1 |\n| `u_audioStream{i}Low` - `u_audioStream{i}High` | `float` | Band energies 0..1 |\n| `u_audioStream{i}Brightness`, `u_audioStream{i}Flatness` | `float` | Spectral features 0..1 |\n\n`i` ranges 0..7. **Scalars only**: no `u_audioFFT` / `u_audioWaveform` per stream. Beat / BPM / trigger uniforms are main audio only.\n\n### Backbuffer\n\n| Uniform | Type | Description |\n|---|---|---|\n| `backbuffer` | `sampler2D` | Previous frame (auto-enabled when referenced) |\n\nNo `u_` prefix. RGBA 8-bit, `LINEAR` filtering, `CLAMP_TO_EDGE` wrapping. First frame samples as black. Clears on canvas resize.\n\n## Template\n\n```glsl\n// @renderer shader\n// @viji-slider:speed label:\"Speed\" default:1.0 min:0.1 max:5.0\n// @viji-color:baseColor label:\"Color\" default:#ff6600\n// @viji-accumulator:phase rate:speed\n\nvoid main() {\n vec2 uv = gl_FragCoord.xy / u_resolution;\n\n float wave = sin(uv.x * 10.0 + phase) * 0.5 + 0.5;\n float pulse = 1.0 + u_audioLow * 0.5;\n vec3 color = baseColor * wave * pulse;\n\n gl_FragColor = vec4(color, 1.0);\n}\n```\n"
|
|
9855
9880
|
},
|