mujoco-react 12.1.1 → 12.2.0

package/README.md

@@ -16,19 +16,6 @@ Composable [React Three Fiber](https://docs.pmnd.rs/react-three-fiber) wrapper a
 npm install mujoco-react three @react-three/fiber @react-three/drei
 ```
 
-## Running policies and simulation workloads
-
-Use `const runtime = await api.createRuntime()` after `onReady`, then await
-`runtime.observe()` and `runtime.advance({ steps, controls })`. Physics runs in a
-dedicated worker; commands reject when busy instead of accumulating. The runtime
-supports cancellation, reset, full snapshots, synchronized camera observations,
-and separate physics/capture timing. Use `<CameraView>` for GPU-only previews.
-
-The viewer has an 8 ms / 100-step per-frame budget. Equivalent inline scene
-configs preserve the loaded model. See the [simulation runtime guide](https://dadd.mintlify.app/guides/simulation-runtime)
-for ownership, VLA loops, bundling, and performance limits, or try the
-[worker demo](https://mujoco-react-example.pages.dev/?demo=runtime).
-
 ## Vite Plugin and Type-Safe Names
 
 Use the Vite plugin to generate declaration merging for actuator, sensor, body, joint, site, geom, and keyframe names:
@@ -156,173 +143,6 @@ Use it as a child of `<MujocoCanvas>`:
 </MujocoCanvas>
 ```
 
-## Gaussian Splat Environments
-
-Gaussian splats are visual context; MuJoCo XML remains the source of physics, contacts, and task fixtures. Use visual-only splats when you only need rendered environment context, and add collision proxy metadata when a workflow needs simplified contact geometry.
-
-Use `VisualScenarioEffects` when the same MuJoCo task should render under
-different camera exposure, fog/background, and deterministic material variants:
-
-```tsx
-import { ScenarioLighting, VisualScenarioEffects } from "mujoco-react";
-
-<MujocoCanvas config={sceneConfig}>
-  <VisualScenarioEffects
-    scenario={scenario}
-    materialFilter={({ object }) => object.name.startsWith("prop_")}
-  />
-  <ScenarioLighting preset={scenario.lighting} />
-</MujocoCanvas>;
-```
-
-Use the renderer-agnostic boundary from the main package. If your app stores
-visual scenarios as data, pass the scenario directly; the component resolves the
-splat asset and any paired MJCF collision proxy metadata for you. Visual-only
-splats are valid, and readiness tells you whether a collision proxy is required
-for your training/physics handoff.
-
-```tsx
-import { MujocoCanvas, SplatEnvironment, useSplatSceneConfig } from "mujoco-react";
-
-const splat = useSplatSceneConfig({ sceneConfig, scenario });
-
-<MujocoCanvas config={splat.sceneConfig}>
-  {splat.environment ? (
-    <SplatEnvironment environment={splat.environment} renderer="custom">
-      <MySplatRenderer src={splat.environment.splat.src} />
-    </SplatEnvironment>
-  ) : null}
-</MujocoCanvas>;
-```
-
-When a splat scenario includes paired MJCF collision proxy metadata, render a
-generic wireframe preview from that XML with `SplatCollisionProxyPreview`. The
-component parses MJCF primitives such as planes, boxes, spheres, capsules, and
-mesh placeholders from any fetchable proxy XML; the tabletop example is just one
-possible environment.
-
-```tsx
-import {
-  SplatCollisionProxyPreview,
-  SplatEnvironment,
-  useSplatCollisionProxyGeoms,
-  useSplatSceneConfig,
-} from "mujoco-react";
-
-const splat = useSplatSceneConfig({ sceneConfig, scenario });
-const proxy = splat.environment?.collisionProxy;
-
-<SplatEnvironment
-  environment={splat.environment}
-  collisionProxy={
-    proxy ? <SplatCollisionProxyPreview collisionProxy={proxy} /> : undefined
-  }
-/>;
-```
-
-Use `useSplatCollisionProxyGeoms()` when your app wants to inspect or style the
-proxy primitives itself:
-
-```tsx
-const proxyPreview = useSplatCollisionProxyGeoms({
-  collisionProxy: splat.environment?.collisionProxy,
-});
-
-proxyPreview.geoms.map((geom) => geom.type);
-```
-
-Use `splat.readiness` or `getSplatEnvironmentReadiness(scenario)` to gate
-authoring and import flows. The status distinguishes disabled scenarios,
-missing visual assets, missing collision proxies, unsupported renderer formats,
-and ready environments.
-
-Use `createSplatSceneConfig()` when the same scene composition needs to run
-outside React, such as codegen, import validation, backend handoff metadata, or
-tests:
-
-```tsx
-import { createSplatSceneConfig } from "mujoco-react";
-
-const splat = createSplatSceneConfig({
-  sceneConfig,
-  scenario,
-  renderer: "spark",
-});
-```
-
-Use `createVisualScenarioExecutionContext()` or
-`useVisualScenarioExecutionContext()` when recording rollouts or exporting
-LeRobot/HF Jobs handoff artifacts. It resolves the scenario seed, camera
-exposure/noise/blur/jitter, material randomization, splat source, collision
-proxy, and readiness into one serializable object.
-
-```tsx
-import { createVisualScenarioExecutionContext } from "mujoco-react";
-
-const visualContext = createVisualScenarioExecutionContext({
-  scenario,
-  renderer: "spark",
-  variantId,
-});
-
-writeEpisodeManifest({
-  task,
-  visualExecutionContext: visualContext,
-});
-```
-
-For MuJoCo + 3DGS composition, derive the optional collision environment from
-the same splat metadata and pass the resulting config to `<MujocoCanvas>`:
-
-```tsx
-const sceneConfig = withSplatEnvironment(
-  {
-    src: "/models/xlerobot/",
-    sceneFile: "xlerobot.xml",
-  },
-  kitchenScenario
-);
-```
-
-For first-class Spark rendering, install Spark and import the optional adapter:
-
-```bash
-npm install @sparkjsdev/spark
-```
-
-```tsx
-import {
-  SparkSplatEnvironment,
-  useSparkSplatEnvironment,
-} from "mujoco-react/spark";
-
-function Scene() {
-  const splat = useSparkSplatEnvironment({ sceneConfig, scenario });
-
-  return (
-    <MujocoCanvas config={splat.sceneConfig} gl={{ preserveDrawingBuffer: true }}>
-      {splat.environment ? (
-        <SparkSplatEnvironment hideGroundMeshes {...splat.props} />
-      ) : null}
-      <StatusBadge status={splat.lifecycle.status} error={splat.lifecycle.error} />
-    </MujocoCanvas>
-  );
-}
-```
-
-`SparkSplatEnvironment` currently renders `.spz` assets. Use the renderer-agnostic
-`SplatEnvironment` for `.ply`/`.splat` metadata or when wiring a different renderer.
-Tune live rendering and snapshots separately with `renderTuning` and
-`captureTuning`:
-
-```tsx
-<SparkSplatEnvironment
-  {...splat.props}
-  renderTuning={{ lodSplatScale: 0.75, minSortIntervalMs: 50 }}
-  captureTuning={{ lodSplatScale: 1.4, lodRenderScale: 0.45, maxWarmupFrames: 6 }}
-/>
-```
-
 ## Write Controllers
 
 ```tsx
@@ -470,6 +290,34 @@ function PolicyDriver() {
 
 Use `output: "float64"` when a downstream model expects double precision. Named resources are skipped when absent, so `obs.layout` is the source of truth for the current model.
 
+## Metric RGB-D Capture
+
+Capture aligned RGB and metric depth for browser robotics observations, including
+LIBERO. Use a camera name from your MJCF:
+
+```tsx
+const capture = useCameraFrameCapture({
+  cameraName: "agentview",
+  width: 256,
+  height: 256,
+  mujocoCameraCompatibility: true,
+});
+const frame = await capture.captureRGBD();
+// frame.rgb: Uint8Array, sRGB RGB bytes, top-left origin
+// frame.depth: Float32Array, meters along camera -Z, Infinity for background
+```
+
+Also available as `api.captureCameraRGBD(options)`, the standalone
+`captureCameraRGBD(renderer, scene, camera, options)`, and
+`session.captureRGBD(options)` on reusable camera capture sessions. One render
+produces both arrays; each result owns its buffers and camera matrix snapshots.
+Scene coordinates must be meters. Custom renderers, logarithmic depth, and
+reversed depth fail explicitly. The example app's `?demo=rgbd` route shows
+reference objects and aligned RGB/depth previews.
+
+See [camera capture docs](docs/hooks/use-camera-frame-capture.mdx#metric-rgb-d-capture)
+for projection metadata, moving-camera sessions, and coordinate conventions.
+
 ## WebSocket Control
 
 Stream actuator commands over a WebSocket and send simulation state back. Use a schema validator such as Zod at this boundary because socket messages are untrusted app input (`npm install zod` for this example):
@@ -1405,6 +1253,173 @@ function ClickSelectOverlay() {
 
 See [Click-to-Select](https://dadd.mintlify.app/guides/click-to-select) for the full implementation.
 
+## Gaussian Splat Environments
+
+Gaussian splats are visual context; MuJoCo XML remains the source of physics, contacts, and task fixtures. Use visual-only splats when you only need rendered environment context, and add collision proxy metadata when a workflow needs simplified contact geometry.
+
+Use `VisualScenarioEffects` when the same MuJoCo task should render under
+different camera exposure, fog/background, and deterministic material variants:
+
+```tsx
+import { ScenarioLighting, VisualScenarioEffects } from "mujoco-react";
+
+<MujocoCanvas config={sceneConfig}>
+  <VisualScenarioEffects
+    scenario={scenario}
+    materialFilter={({ object }) => object.name.startsWith("prop_")}
+  />
+  <ScenarioLighting preset={scenario.lighting} />
+</MujocoCanvas>;
+```
+
+Use the renderer-agnostic boundary from the main package. If your app stores
+visual scenarios as data, pass the scenario directly; the component resolves the
+splat asset and any paired MJCF collision proxy metadata for you. Visual-only
+splats are valid, and readiness tells you whether a collision proxy is required
+for your training/physics handoff.
+
+```tsx
+import { MujocoCanvas, SplatEnvironment, useSplatSceneConfig } from "mujoco-react";
+
+const splat = useSplatSceneConfig({ sceneConfig, scenario });
+
+<MujocoCanvas config={splat.sceneConfig}>
+  {splat.environment ? (
+    <SplatEnvironment environment={splat.environment} renderer="custom">
+      <MySplatRenderer src={splat.environment.splat.src} />
+    </SplatEnvironment>
+  ) : null}
+</MujocoCanvas>;
+```
+
+When a splat scenario includes paired MJCF collision proxy metadata, render a
+generic wireframe preview from that XML with `SplatCollisionProxyPreview`. The
+component parses MJCF primitives such as planes, boxes, spheres, capsules, and
+mesh placeholders from any fetchable proxy XML; the tabletop example is just one
+possible environment.
+
+```tsx
+import {
+  SplatCollisionProxyPreview,
+  SplatEnvironment,
+  useSplatCollisionProxyGeoms,
+  useSplatSceneConfig,
+} from "mujoco-react";
+
+const splat = useSplatSceneConfig({ sceneConfig, scenario });
+const proxy = splat.environment?.collisionProxy;
+
+<SplatEnvironment
+  environment={splat.environment}
+  collisionProxy={
+    proxy ? <SplatCollisionProxyPreview collisionProxy={proxy} /> : undefined
+  }
+/>;
+```
+
+Use `useSplatCollisionProxyGeoms()` when your app wants to inspect or style the
+proxy primitives itself:
+
+```tsx
+const proxyPreview = useSplatCollisionProxyGeoms({
+  collisionProxy: splat.environment?.collisionProxy,
+});
+
+proxyPreview.geoms.map((geom) => geom.type);
+```
+
+Use `splat.readiness` or `getSplatEnvironmentReadiness(scenario)` to gate
+authoring and import flows. The status distinguishes disabled scenarios,
+missing visual assets, missing collision proxies, unsupported renderer formats,
+and ready environments.
+
+Use `createSplatSceneConfig()` when the same scene composition needs to run
+outside React, such as codegen, import validation, backend handoff metadata, or
+tests:
+
+```tsx
+import { createSplatSceneConfig } from "mujoco-react";
+
+const splat = createSplatSceneConfig({
+  sceneConfig,
+  scenario,
+  renderer: "spark",
+});
+```
+
+Use `createVisualScenarioExecutionContext()` or
+`useVisualScenarioExecutionContext()` when recording rollouts or exporting
+LeRobot/HF Jobs handoff artifacts. It resolves the scenario seed, camera
+exposure/noise/blur/jitter, material randomization, splat source, collision
+proxy, and readiness into one serializable object.
+
+```tsx
+import { createVisualScenarioExecutionContext } from "mujoco-react";
+
+const visualContext = createVisualScenarioExecutionContext({
+  scenario,
+  renderer: "spark",
+  variantId,
+});
+
+writeEpisodeManifest({
+  task,
+  visualExecutionContext: visualContext,
+});
+```
+
+For MuJoCo + 3DGS composition, derive the optional collision environment from
+the same splat metadata and pass the resulting config to `<MujocoCanvas>`:
+
+```tsx
+const sceneConfig = withSplatEnvironment(
+  {
+    src: "/models/xlerobot/",
+    sceneFile: "xlerobot.xml",
+  },
+  kitchenScenario
+);
+```
+
+For first-class Spark rendering, install Spark and import the optional adapter:
+
+```bash
+npm install @sparkjsdev/spark
+```
+
+```tsx
+import {
+  SparkSplatEnvironment,
+  useSparkSplatEnvironment,
+} from "mujoco-react/spark";
+
+function Scene() {
+  const splat = useSparkSplatEnvironment({ sceneConfig, scenario });
+
+  return (
+    <MujocoCanvas config={splat.sceneConfig} gl={{ preserveDrawingBuffer: true }}>
+      {splat.environment ? (
+        <SparkSplatEnvironment hideGroundMeshes {...splat.props} />
+      ) : null}
+      <StatusBadge status={splat.lifecycle.status} error={splat.lifecycle.error} />
+    </MujocoCanvas>
+  );
+}
+```
+
+`SparkSplatEnvironment` currently renders `.spz` assets. Use the renderer-agnostic
+`SplatEnvironment` for `.ply`/`.splat` metadata or when wiring a different renderer.
+Tune live rendering and snapshots separately with `renderTuning` and
+`captureTuning`:
+
+```tsx
+<SparkSplatEnvironment
+  {...splat.props}
+  renderTuning={{ lodSplatScale: 0.75, minSortIntervalMs: 50 }}
+  captureTuning={{ lodSplatScale: 1.4, lodRenderScale: 0.45, maxWarmupFrames: 6 }}
+/>
+```
+
 ## useFrame Priority
 
 | Priority | Owner | Purpose |
@@ -1434,3 +1449,8 @@ Apache-2.0
 ## Contacts and plasticity
 
 Use MJCF speculative contacts for fast rigid objects and yielding springs for permanent deformation. The existing model/data interface exposes enable flags and plastic state; full snapshots preserve plastic history. See the [guide](docs/guides/engine-features.mdx) and [interactive demos](https://mujoco-react-example.pages.dev/?demo=speculative).
+
+Live splat camera streams use background sorting and reuse prepared splats, so
+robot motion does not force a splat rebuild on every preview frame. IK gizmos
+remain visible in the main view and excluded from captures. One-off and policy
+captures keep snapshot behavior; `renderMode: 'snapshot'` opts a stream into it.