mujoco-react 9.3.0 → 9.5.0

package/README.md

@@ -143,7 +143,7 @@ Use it as a child of `<MujocoCanvas>`:
 
 ## Gaussian Splat Environments
 
-Gaussian splats are visual context; MuJoCo XML remains the source of physics, contacts, and task fixtures. Pair each splat asset with collision proxy metadata so scene variants, rollouts, and datasets preserve both sides of the environment.
+Gaussian splats are visual context; MuJoCo XML remains the source of physics, contacts, and task fixtures. Use visual-only splats when you only need rendered environment context, and add collision proxy metadata when a workflow needs simplified contact geometry.
 
 Use `VisualScenarioEffects` when the same MuJoCo task should render under
 different camera exposure, fog/background, and deterministic material variants:
@@ -162,7 +162,9 @@ import { ScenarioLighting, VisualScenarioEffects } from "mujoco-react";
 
 Use the renderer-agnostic boundary from the main package. If your app stores
 visual scenarios as data, pass the scenario directly; the component resolves the
-splat asset and paired MJCF collision proxy metadata for you.
+splat asset and any paired MJCF collision proxy metadata for you. Visual-only
+splats are valid, and readiness tells you whether a collision proxy is required
+for your training/physics handoff.
 
 ```tsx
 import { MujocoCanvas, SplatEnvironment, useSplatSceneConfig } from "mujoco-react";
@@ -178,13 +180,84 @@ const splat = useSplatSceneConfig({ sceneConfig, scenario });
 </MujocoCanvas>;
 ```
 
+When a splat scenario includes paired MJCF collision proxy metadata, render a
+generic wireframe preview from that XML with `SplatCollisionProxyPreview`. The
+component parses MJCF primitives such as planes, boxes, spheres, capsules, and
+mesh placeholders from any fetchable proxy XML; the tabletop example is just one
+possible environment.
+
+```tsx
+import {
+  SplatCollisionProxyPreview,
+  SplatEnvironment,
+  useSplatCollisionProxyGeoms,
+  useSplatSceneConfig,
+} from "mujoco-react";
+
+const splat = useSplatSceneConfig({ sceneConfig, scenario });
+const proxy = splat.environment?.collisionProxy;
+
+<SplatEnvironment
+  environment={splat.environment}
+  collisionProxy={
+    proxy ? <SplatCollisionProxyPreview collisionProxy={proxy} /> : undefined
+  }
+/>;
+```
+
+Use `useSplatCollisionProxyGeoms()` when your app wants to inspect or style the
+proxy primitives itself:
+
+```tsx
+const proxyPreview = useSplatCollisionProxyGeoms({
+  collisionProxy: splat.environment?.collisionProxy,
+});
+
+proxyPreview.geoms.map((geom) => geom.type);
+```
+
 Use `splat.readiness` or `getSplatEnvironmentReadiness(scenario)` to gate
 authoring and import flows. The status distinguishes disabled scenarios,
-missing splat assets, missing MJCF collision proxies, unsupported Spark formats,
-and ready paired environments.
+missing visual assets, missing collision proxies, unsupported renderer formats,
+and ready environments.
+
+Use `createSplatSceneConfig()` when the same scene composition needs to run
+outside React, such as codegen, import validation, backend handoff metadata, or
+tests:
+
+```tsx
+import { createSplatSceneConfig } from "mujoco-react";
+
+const splat = createSplatSceneConfig({
+  sceneConfig,
+  scenario,
+  renderer: "spark",
+});
+```
+
+Use `createVisualScenarioExecutionContext()` or
+`useVisualScenarioExecutionContext()` when recording rollouts or exporting
+LeRobot/HF Jobs handoff artifacts. It resolves the scenario seed, camera
+exposure/noise/blur/jitter, material randomization, splat source, collision
+proxy, and readiness into one serializable object.
+
+```tsx
+import { createVisualScenarioExecutionContext } from "mujoco-react";
+
+const visualContext = createVisualScenarioExecutionContext({
+  scenario,
+  renderer: "spark",
+  variantId,
+});
 
-For MuJoCo + 3DGS composition, derive the collision environment from the same
-splat metadata and pass the resulting config to `<MujocoCanvas>`:
+writeEpisodeManifest({
+  task,
+  visualExecutionContext: visualContext,
+});
+```
+
+For MuJoCo + 3DGS composition, derive the optional collision environment from
+the same splat metadata and pass the resulting config to `<MujocoCanvas>`:
 
 ```tsx
 const sceneConfig = withSplatEnvironment(
@@ -224,6 +297,16 @@ function Scene() {
 
 `SparkSplatEnvironment` currently renders `.spz` assets. Use the renderer-agnostic
 `SplatEnvironment` for `.ply`/`.splat` metadata or when wiring a different renderer.
+Tune live rendering and snapshots separately with `renderTuning` and
+`captureTuning`:
+
+```tsx
+<SparkSplatEnvironment
+  {...splat.props}
+  renderTuning={{ lodSplatScale: 0.75, minSortIntervalMs: 50 }}
+  captureTuning={{ lodSplatScale: 1.4, lodRenderScale: 0.45, maxWarmupFrames: 6 }}
+/>
+```
 
 ## Write Controllers
 
@@ -552,7 +635,7 @@ interface SceneConfig {
 }
 ```
 
-Use `environmentFiles` to compose reusable physics/collision layers with a robot model. For Gaussian splat scenes, keep the `.spz` as a parallel visual layer and point `environmentFiles` at the paired MJCF proxy scene:
+Use `environmentFiles` to compose reusable physics/collision layers with a robot model. For Gaussian splat scenes, keep the `.spz` as a parallel visual layer and point `environmentFiles` at a paired MJCF proxy scene only when contact geometry is needed:
 
 ```tsx
 const kitchenRobot: SceneConfig = {
@@ -777,6 +860,7 @@ Visualization overlays:
 | `showSites` | `boolean?` | `false` | Site markers |
 | `showJoints` | `boolean?` | `false` | Joint axes |
 | `showContacts` | `boolean?` | `false` | Contact force vectors |
+| `showCameras` | `boolean?` | `false` | MuJoCo camera positions, frustums, and forward rays |
 | `showCOM` | `boolean?` | `false` | Center of mass markers |
 | `showInertia` | `boolean?` | `false` | Inertia ellipsoids |
 | `showTendons` | `boolean?` | `false` | Tendon paths |
@@ -785,6 +869,8 @@ Visualization overlays:
 | `contactColor` | `string?` | `"#ff4444"` | Color for contact force arrows |
 | `comColor` | `string?` | `"#ff0000"` | Color for COM markers |
 
+Camera debug overlays use the live MuJoCo `cam_xpos` / `cam_xmat` frame, so the frustum matches mounted camera captures and follows parent body motion.
+
 ### `<TendonRenderer />`
 
 Renders tendons as tube geometry from wrap paths.
@@ -1066,6 +1152,10 @@ const sequence = await recordMountedCameraFrameSequence(api, {
 sequence.readiness.ready; // true when every requested stream resolved
 sequence.plan.missingKeys; // unresolved task cameras, if requireAll is false
 sequence.cameraSummaries.head.source; // mounted source provenance
+
+const manifest = createMountedCameraFrameSequenceManifest(sequence);
+manifest.streamSummaries.head.complete; // per-camera frame coverage
+manifest.status; // "complete", "partial", or "missing"
 ```
 
 `recordMountedCameraFrameSequence()` requires all requested `cameraKeys` by
@@ -1073,23 +1163,69 @@ default so dataset recording cannot silently omit a camera stream. Set
 `requireAll: false` only for exploratory tooling that can tolerate partial
 camera coverage.
 
+Use `createMountedCameraFrameSequenceManifest()` after recording to persist a
+stable dataset-facing manifest with readiness, source targets, dimensions,
+first/last frame indices, timestamps, and per-camera missing-frame counts.
+
 Inside `<MujocoCanvas>` children, `useMountedCameraSequenceRecorder()` exposes
-the same planning and recording surface with React status/error state.
+the same planning and recording surface with React status/error/result state.
+Use `checkReadiness()` before recording when the UI needs a preflight gate for
+LeRobot camera streams:
+
+```tsx
+function DatasetRecorder() {
+  const recorder = useMountedCameraSequenceRecorder({
+    defaults: { width: 640, height: 480, type: "image/png" },
+    aliases: {
+      head: { cameraName: "head" },
+      left_wrist: { siteName: "left_wrist_camera_optical_frame" },
+      right_wrist: { siteName: "right_wrist_camera_optical_frame" },
+    },
+  });
+
+  async function recordDatasetEpisode() {
+    const cameraKeys = ["head", "left_wrist", "right_wrist"];
+    const readiness = recorder.checkReadiness(cameraKeys);
+    if (!readiness.ready) return;
+
+    await recorder.record({
+      cameraKeys,
+      frames: 16,
+      retainFrames: false,
+      onFrame: ({ frameIndex, cameras }) => {
+        queueLeRobotImages(frameIndex, cameras);
+      },
+    });
+  }
+
+  return (
+    <button disabled={recorder.isRecording} onClick={recordDatasetEpisode}>
+      Record camera streams
+    </button>
+  );
+}
+```
+
+`recorder.readiness` keeps the latest preflight result, and
+`recorder.result?.readiness` keeps the readiness that shipped with the most
+recent recording.
 
 Use `resolveMountedCameraFrameSource()` when dataset feature names need to map
-to named MuJoCo cameras, sites, or bodies before recording. The helper accepts
-the model resource lists plus app-level aliases and returns both the capture
-selector and the mounted-source provenance that should be stored beside the
-dataset:
+to named MuJoCo cameras, sites, or bodies before recording. The helper honors
+aliases first, then prefers camera matches over sites and bodies, then falls
+back to exact body/site names. This keeps streams such as `wrist` mapped to a
+real `<camera name="wrist_cam">` even when the model also has a body named
+`wrist`. It also handles normalized/prefix/suffix matches such as
+`left_wrist` -> `left_wrist_camera_optical_frame`, and token-contained
+imported-model names such as `observation.images.head` ->
+`robot_head_camera`. It returns both the capture selector and the mounted-source
+provenance that should be stored beside the dataset:
 
 ```tsx
 const resolved = resolveMountedCameraFrameSource("head", {
   cameras: api.getCameras(),
   sites: api.getSites(),
   bodies: api.getBodies(),
-  aliases: {
-    head: [{ siteName: "head_camera_rgb_optical_frame" }],
-  },
 });
 
 if (!resolved) throw new Error("head does not resolve to a MuJoCo source");
@@ -1101,6 +1237,10 @@ await api.recordCameraSequence({
 });
 ```
 
+Pass `aliases` when multiple MuJoCo resources could match a dataset stream key
+or when the model uses names that do not share a normalized prefix/suffix with
+the LeRobot camera feature.
+
 ### `useCtrlNoise(config)`
 
 Apply Gaussian noise to controls for robustness testing: