three-mediapipe-rig 0.0.3 → 0.1.0

package/README.md

@@ -1,16 +1,18 @@
 
 ![cover](cover.jpg)
 
-# three-mediapipe-rig
+# Control a skeleton with the webcam
 
-Integrate [Google MediaPipe](https://ai.google.dev/edge/mediapipe/solutions/guide)'s **webcam motion tracking** with [Three.js](https://threejs.org/) skeletal rigs. Load a GLTF/GLB character, bind it, and drive its body, hands, and face from a webcam or video — in just a few lines of code.
+Integrate [Google MediaPipe](https://ai.google.dev/edge/mediapipe/solutions/guide)'s **webcam motion tracking** with [Three.js](https://threejs.org/) skeletal rigs. 
 
-Use your webcam ( or a video ) to drive a skeleton.
+The motion from the webcam will be applied to a skeleton. Angle based so it works with any size skeleton. 
 
 This will run 3 models: face, body, hands. So expect a FPS drop.
 
-LIVE EXAMPLE: https://bandinopla.github.io/three-mediapipe-rig/
-
+Expolore the demos:
+- [Characters](https://bandinopla.github.io/three-mediapipe-rig) 
+- [Hands Demo](https://bandinopla.github.io/three-mediapipe-rig/?demo=hands)
+- [Video to Face Geometry](https://bandinopla.github.io/three-mediapipe-rig/?demo=face-uv-demo)
 ---
 
 ## Table of Contents
@@ -26,6 +28,7 @@ LIVE EXAMPLE: https://bandinopla.github.io/three-mediapipe-rig/
 - [Bone Naming](#bone-naming)
   - [Default bone names](#default-bone-names)
   - [Custom bone map example](#custom-bone-map-example)
+- [Video -> Facial Geometry](#video-to-facial-geometry)
 - [Multiple Characters](#multiple-characters)
 - [Debugging with Video](#debugging-with-video)
 - [Recording](#recording)
@@ -52,41 +55,32 @@ npm install three-mediapipe-rig
 
 ## Quick Start
 
-```ts
-// 1. Create your renderer
-const renderer = new THREE.WebGPURenderer({ antialias: true });
-renderer.setSize(window.innerWidth, window.innerHeight);
-document.body.appendChild(renderer.domElement);
+```ts 
 
-// 2. Initialize the tracker (loads MediaPipe models)
+// 1. Initialize the tracker (loads MediaPipe models)
 await setupTracker({ ...config... })
  
 const rig = scene.getObjectByName("rig")!;
 
-// 3. Bind the rig to the tracker
+// 2. Bind the rig to the tracker
 const binding = tracker.bind(rig);
  
 
-// 4. Start the webcam ( must be initialized by a user triggered event like a click )
-tracker.start();
-
-// 5. Update in your render loop
-const clock = new THREE.Timer();
-renderer.setAnimationLoop((time: number) => {
-  const delta = clock.update(time).getDelta();
+// 3. Start the webcam ( must be initialized by a user triggered event like a click )
+button.addEventListener('click', () => {
+    tracker.start(); //<-- will ask webcam access!
+})
 
-  // 6. update the skeleton...
-  binding?.update(delta);
 
-  renderer.render(scene, camera);
-});
+// 4. Update the rig in your render loop ( this keels the skeleton in-sync)
+binding?.update(delta);
 ```
 
 ### Skeleton
-You can use the skeleton provided in `rig.blend` or use your own and provide a bone name mapping so we know where the bones are in the second argument for the `.bind` method. But pay attention to the bone role of the provided skeleton, as it is the one expected by this module.
+You can use the skeleton provided in `rig.blend` or use your own and provide a bone name mapping so we know where the bones are in the second argument for the `.bind` method. But pay attention to the [**bone roll**](https://docs.blender.org/manual/en/latest/animation/armatures/bones/editing/bone_roll.html) of the provided skeleton, as it is the one expected by this module.
 
 ### Facial Animation
-Media Pipe provides blend shape keys for the face ( estimated from the webcam ). The face it is expected to be a separated mesh, just the face, with blend shape keys named as the ones provided by Media Pipe. See [Blend Shape Keys reference](/face-blendshapekeys.md) You don't have to have all of them, if they are not found, they will be ignored.
+Media Pipe provides blend shape keys for the face ( estimated from the webcam ). The face it is expected to be a separated mesh with a name that starts with "face", with blend shape keys named as the ones provided by Media Pipe. See [Blend Shape Keys reference](/face-blendshapekeys.md) You don't have to have all of them, if they are not found, they will be ignored.
 
 
 ## API
@@ -230,11 +224,18 @@ const myBoneMap: BoneMap = {
   index3L: "LeftHandIndex3",
   // (continue for all fingers)
 };
-
-const binding = tracker.bind(rig, myBoneMap);
 ```
+ 
 
----
+## Video to Facial Geometry
+Media Pipe's facial tracking also provides a facial mesh, if you [download the face mesh object](https://github.com/google-ai-edge/mediapipe/tree/master/mediapipe/modules/face_geometry) you can also make it deform and be textured using the webcam data. This will texture the mesh with the feed from the camera and also will move the vertices to adjust to the facial expressions:
+
+```js
+const face = tracker.faceTracker.bindGeometry( faceMesh );
+
+//... and in your loop
+face.update(delta)
+``` 
 
 ## Multiple Characters