hand-tracking-teleop 0.1.0__tar.gz

hand_tracking_teleop-0.1.0/MANIFEST.in

@@ -0,0 +1,18 @@
+# Include documentation files
+include README.md
+include LICENSE
+include requirements.txt
+
+# Include configuration files
+recursive-include config *.yaml
+
+# Exclude development and build files
+global-exclude *.pyc
+global-exclude *.pyo
+global-exclude __pycache__
+global-exclude .git*
+global-exclude .DS_Store
+prune scripts
+prune tests
+prune .vscode
+prune .idea

hand_tracking_teleop-0.1.0/PKG-INFO

@@ -0,0 +1,436 @@
+Metadata-Version: 2.4
+Name: hand_tracking_teleop
+Version: 0.1.0
+Summary: Hand tracking system for robot teleoperation using MediaPipe
+Home-page: https://github.com/yourusername/hand_tracking_teleop
+Author: Bruk Gebregziabher
+Author-email: Bruk Gebregziabher <bruk@signalbotics.com>
+License: MIT
+Project-URL: Homepage, https://github.com/yourusername/hand_tracking_teleop
+Project-URL: Documentation, https://github.com/yourusername/hand_tracking_teleop#readme
+Project-URL: Repository, https://github.com/yourusername/hand_tracking_teleop
+Project-URL: Bug Tracker, https://github.com/yourusername/hand_tracking_teleop/issues
+Keywords: hand-tracking,teleoperation,robotics,mediapipe,computer-vision
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: opencv-python>=4.5.0
+Requires-Dist: mediapipe>=0.10.0
+Requires-Dist: numpy>=1.20.0
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: vis
+Requires-Dist: matplotlib>=3.5.0; extra == "vis"
+Provides-Extra: rerun
+Requires-Dist: rerun-sdk>=0.18.0; extra == "rerun"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Requires-Dist: black>=23.0; extra == "dev"
+Requires-Dist: flake8>=6.0; extra == "dev"
+Requires-Dist: mypy>=1.0; extra == "dev"
+Requires-Dist: isort>=5.0; extra == "dev"
+Dynamic: author
+Dynamic: home-page
+Dynamic: requires-python
+
+# Hand Tracking Data Collector
+
+A real-time hand tracking system for collecting training data for the dual-arm robot using MediaPipe and Gazebo simulation.
+
+## Overview
+
+This system enables intuitive data collection by tracking human hand movements and mapping them to robot joint configurations. The operator demonstrates tasks by moving their hands while the system records synchronized data from:
+
+- Hand landmark detection (21 points per hand)
+- Robot joint states from Gazebo simulation
+- Robot camera feeds (4 cameras)
+- Timestamps for synchronization
+
+## Quick Start
+
+### 1. Installation
+
+```bash
+# Navigate to the hand tracking collector directory
+cd examples/dual_arm/hand_tracking_collector/
+
+# Install dependencies
+pip install -r requirements.txt
+```
+
+### 2. Test Hand Tracking (Phase 1)
+
+```bash
+# Run the interactive test script (choose visualization mode)
+python scripts/test_hand_tracking.py
+```
+
+**Options:**
+1. 2D hand tracking (OpenCV window) - Classic view
+2. **3D hand tracking (matplotlib) - Shows coordinate axes at key landmarks**
+3. **Combined 2D + 3D view - Best for full overview**
+4. Quick detection test (no display)
+
+**Or use directly in Python:**
+```python
+from hand_tracking.hand_detector import HandDetector
+
+detector = HandDetector()
+
+# Choose mode: '2d', '3d', or 'combined'
+detector.run_interactive(mode='3d')  # 3D visualization with axes
+```
+
+### 3. 3D Visualization
+
+Visualize hand landmarks in interactive 3D space using matplotlib:
+
+```bash
+# Combined 2D + 3D view (Recommended)
+python scripts/visualize_hand_combined.py
+
+# 3D only - dual hand plots
+python scripts/visualize_hand_3d.py
+
+# Snapshot mode - capture single frame
+python scripts/visualize_hand_3d_simple.py
+
+# Test system
+python scripts/test_visualization.py
+```
+
+**Features:**
+- **Coordinate axes at key landmarks** (wrist + fingertips): X=red, Y=green, Z=blue
+- Color-coded finger bones: 🔴 Thumb | 🔵 Index | 💙 Middle | 💚 Ring | 💛 Pinky
+- Interactive controls: Click+drag to rotate, scroll to zoom, right-click to pan
+- Real-time tracking at ~30 FPS
+- Hand confidence scores and statistics
+
+**Tips for best tracking:**
+- Good, even lighting (avoid backlighting)
+- Plain background
+- Keep hands 1-2 feet from camera
+- All fingers visible
+
+### 4. Controls
+
+- **'q'**: Quit hand tracking (2D view)
+- **Close window**: Exit visualization
+- **Mouse drag**: Rotate 3D view
+- **Scroll wheel**: Zoom in/out
+- **Right-click drag**: Pan view
+- **Gestures**: (Future phases will support gesture controls)
+
+## Current Status: Phase 1 Complete ✅
+
+**Phase 1: Basic Hand Tracking**
+- [x] MediaPipe hand detection setup
+- [x] Real-time landmark extraction (21 points per hand)
+- [x] Hand confidence and bounding box calculation
+- [x] Temporal smoothing for stable tracking
+- [x] Interactive 2D visualization
+- [x] Interactive 3D visualization with matplotlib
+- [x] Configuration system
+
+## System Architecture
+
+```
+Human Hands → Camera → MediaPipe → Hand Landmarks
+                                        ↓
+Future: Robot Control ← Joint Mapping ← Hand Poses
+                                        ↓
+Future: Data Collection ← Gazebo Robot ← Joint Commands
+```
+
+## Mathematical Framework
+
+### Camera-to-Robot Coordinate Mapping
+
+The system assumes a **top-view camera** looking down at the hands. The coordinate transformation from camera space to robot workspace is:
+
+**Camera Coordinate System:**
+- X_cam: Left-to-right in camera image (0 to 640 pixels)
+- Y_cam: Top-to-bottom in camera image (0 to 480 pixels)  
+- Z_cam: Depth from camera (MediaPipe estimated depth)
+
+**Robot Coordinate System (TorsoLink frame):**
+- X_robot: Forward/backward relative to robot torso (+X = forward)
+- Y_robot: Left/right relative to robot torso (+Y = left)
+- Z_robot: Up/down relative to robot torso (+Z = up)
+
+**Coordinate Transformation:**
+```
+Δy_robot = -(X_cam/640 - 0.5) × 0.3   [±15cm left/right, inverted]
+Δx_robot = -(Y_cam/480 - 0.5) × 0.2   [±10cm forward/back, inverted]  
+Δz_robot = -Z_cam × 0.2               [0-20cm up/down, inverted]
+
+P_target = P_current + [Δx_robot, Δy_robot, Δz_robot]
+```
+
+**Coordinate System Mapping (with ROS conventions):**
+- Camera X (left-right in image) → Robot -Y (right in image = robot moves right = -Y)
+- Camera Y (top-bottom in image) → Robot -X (down in image = robot moves back = -X)
+- Camera Z (depth from camera) → Robot -Z (closer to camera = robot moves down = -Z)
+
+Where:
+- P_current = Current robot hand position from forward kinematics
+- P_target = Target robot hand position for inverse kinematics
+
+### Hand Landmark Processing
+
+MediaPipe provides 21 landmarks per hand in normalized coordinates [0,1]. The wrist landmark (index 0) is used for position tracking:
+
+```
+P_hand = MediaPipe_landmarks[0] × [640, 480, depth_scale]
+```
+
+### Finger Joint Mapping
+
+The system uses kinematic mapping to convert hand landmarks into robot joint commands. Here are the detailed equations for each controller:
+
+### Finger Mapping (Per Hand Controller)
+
+Each hand has 6 DOF mapped from MediaPipe hand landmarks:
+
+#### Left Hand Controller: `left_hand_controller`
+- `left_thumb_finger_proximal_pitch` (Joint 15)
+- `left_thumb_finger_proximal_yaw` (Joint 16)  
+- `left_index_finger_proximal` (Joint 17)
+- `left_middle_finger_proximal` (Joint 18)
+- `left_ring_finger_proximal` (Joint 19)
+- `left_pinky_finger_proximal` (Joint 20)
+
+#### Right Hand Controller: `right_hand_controller`
+- `right_thumb_finger_proximal_pitch` (Joint 21)
+- `right_thumb_finger_proximal_yaw` (Joint 22)
+- `right_index_finger_proximal` (Joint 23)
+- `right_middle_finger_proximal` (Joint 24)
+- `right_ring_finger_proximal` (Joint 25)
+- `right_pinky_finger_proximal` (Joint 26)
+
+### Finger Curl Calculation
+
+The core finger mapping uses a "curl ratio" calculation based on triangle inequality:
+
+**Distance Calculations:**
+- d₁ = ||P_mcp - P_wrist||₂ (wrist to MCP joint)
+- d₂ = ||P_tip - P_mcp||₂ (MCP joint to fingertip)  
+- d₃ = ||P_tip - P_wrist||₂ (wrist to fingertip direct)
+
+**Expected vs Actual Distance:**
+- d_expected = d₁ + d₂ (if finger was straight)
+- curl_ratio = 1 - (d₃ / d_expected)
+
+**Sensitivity Enhancement:**
+- curl_final = (curl_ratio)^0.7
+- Result clamped to [0, 1] where 0 = straight, 1 = fully curled
+
+### Joint Angle Mapping
+
+Each finger type uses different scaling factors:
+
+#### Thumb (2 DOF):
+- θ_pitch = clamp(curl_final × 3.0, 0.0, 1.5) radians
+- θ_yaw = clamp(curl_final × 2.0, 0.0, 1.5) radians
+
+#### Other Fingers (1 DOF each):
+- θ_joint = clamp(curl_final × 2.5, 0.0, 1.5) radians
+
+### MediaPipe Landmark Indices
+
+The system uses these specific MediaPipe hand landmarks:
+
+- **Wrist**: Landmark 0
+- **Thumb**: MCP=2, Tip=4
+- **Index**: MCP=5, Tip=8  
+- **Middle**: MCP=9, Tip=12
+- **Ring**: MCP=13, Tip=16
+- **Pinky**: MCP=17, Tip=20
+
+### Arm/Torso Controller: `torso_controller`
+
+The torso controller handles 15 DOF (currently disabled for safety):
+
+#### Hand Position Calculation
+
+The 3D hand position P_hand is derived from the wrist landmark (MediaPipe landmark 0):
+
+**Image to World Coordinate Transform:**
+- x_norm = (landmark₀.x / 640) - 0.5  (normalize to [-0.5, 0.5])
+- y_norm = (landmark₀.y / 480) - 0.5  (normalize to [-0.5, 0.5])
+- z_norm = landmark₀.z  (MediaPipe depth estimate)
+
+**Workspace Scaling:**
+- P_hand.x = 2.0 × x_norm × (x_max - x_min) / 2
+- P_hand.y = 2.0 × y_norm × (y_max - y_min) / 2  
+- P_hand.z = z_norm × (z_max - z_min) / 2 + z_min
+
+Where workspace limits: x ∈ [-0.8, 0.8], y ∈ [-0.6, 0.6], z ∈ [0.5, 1.5] meters
+
+#### Hand Orientation Calculation
+
+Hand orientation Ω = [roll, pitch, yaw] is calculated from palm geometry:
+
+**Key Points:**
+- P_wrist = landmark₀ (wrist)
+- P_middle = landmark₉ (middle finger MCP)
+- P_index = landmark₅ (index finger MCP)
+
+**Orientation Vectors:**
+- V_palm = P_middle - P_wrist
+- V_finger = P_index - P_wrist
+
+**Euler Angles:**
+- roll = atan2(V_palm.y, V_palm.x)
+- pitch = atan2(V_palm.z, ||V_palm.xy||₂)
+- yaw = atan2(V_finger.y, V_finger.x)
+
+#### Left Arm (7 DOF):
+Based on calculated hand position P_hand = [x, y, z] and orientation Ω = [roll, pitch, yaw]:
+
+- **Shoulder X**: θ₁ = clamp(2.0 × x, -1.5, 1.5)
+- **Shoulder Z**: θ₂ = clamp(-1.5 × y, -1.0, 1.0)  
+- **Shoulder Y**: θ₃ = 0.5 × yaw
+- **Elbow Y**: θ₄ = clamp(2.0 × (1 - ||P_hand||), -2.0, 0.0)
+- **Wrist Z**: θ₅ = 0.8 × roll
+- **Wrist X**: θ₆ = 0.6 × pitch
+- **Wrist Y**: θ₇ = 0.4 × yaw
+
+#### Right Arm (7 DOF):
+Same equations as left arm, applied to right hand landmarks.
+
+#### Torso (1 DOF):
+- **Torso Z**: θ_torso = 0.0 (currently neutral)
+
+### Smoothing and Limits
+
+The system applies temporal smoothing and joint limits:
+
+**Exponential Moving Average (α = 0.9):**
+- θ_smoothed(t) = α × θ_smoothed(t-1) + (1-α) × θ_raw(t)
+
+**Joint Limits:**
+- Finger joints: θ ∈ [0.0, 1.57] radians (0° to 90°)
+- Shoulder joints: θ ∈ [-3.14, 3.14] radians (±180°)
+- All joints: θ = clamp(θ, θ_min, θ_max)
+
+## Configuration
+
+The system uses YAML configuration files in the `config/` directory:
+
+- `hand_tracking_config.yaml`: Camera and MediaPipe settings
+- `robot_mapping_config.yaml`: Joint mappings and kinematic parameters
+- `data_collection_config.yaml`: Recording and dataset settings
+
+## File Structure
+
+```
+hand_tracking_collector/
+├── README.md                    # This file
+├── requirements.txt             # Python dependencies
+├── config/                      # Configuration files
+│   ├── hand_tracking_config.yaml
+│   ├── robot_mapping_config.yaml
+│   └── data_collection_config.yaml (future)
+├── src/                         # Source code
+│   ├── hand_tracking/           # Phase 1: Hand detection
+│   │   ├── hand_detector.py     ✅ Complete
+│   │   ├── pose_estimator.py    🔄 Future
+│   │   └── gesture_recognizer.py 🔄 Future
+│   ├── robot_interface/         # Phase 2: Robot control
+│   ├── mapping/                 # Phase 3: Kinematic mapping
+│   └── data_collection/         # Phase 4: Data recording
+├── scripts/                     # Utility scripts
+│   ├── test_hand_tracking.py    ✅ Basic 2D tracking test
+│   ├── test_visualization.py    ✅ System check for visualization
+│   ├── visualize_hand_3d.py     ✅ Real-time 3D visualization
+│   ├── visualize_hand_3d_simple.py ✅ Snapshot 3D visualization
+│   ├── visualize_hand_combined.py ✅ Combined 2D+3D visualization
+│   ├── calibrate_system.py      🔄 Future
+│   ├── collect_data.py          🔄 Future
+│   └── validate_dataset.py      🔄 Future
+└── tests/                       # Unit tests
+```
+
+## Hardware Requirements
+
+- **Camera**: USB webcam (1080p recommended)
+- **Computer**: CPU with decent performance (GPU recommended for real-time processing)
+- **Memory**: 4GB+ RAM
+- **Network**: For future Gazebo/ROS2 communication
+
+## Troubleshooting
+
+### Camera Issues
+```bash
+# Check available cameras
+python -c "import cv2; print([i for i in range(10) if cv2.VideoCapture(i).isOpened()])"
+
+# Test specific camera
+python -c "import cv2; cap = cv2.VideoCapture(0); print('Camera OK' if cap.isOpened() else 'Camera failed')"
+```
+
+### MediaPipe Issues
+```bash
+# Verify MediaPipe installation
+python -c "import mediapipe as mp; print('MediaPipe version:', mp.__version__)"
+```
+
+### Performance Issues
+- Lower camera resolution in `hand_tracking_config.yaml`
+- Reduce MediaPipe model complexity to 0 (lite mode)
+- Close other applications using the camera
+
+## Next Phases
+
+### Phase 2: Robot Interface (Coming Next)
+- ROS2 connection to Gazebo robot
+- Joint command publishing
+- Robot state feedback
+- Camera image capture
+
+### Phase 3: Kinematic Mapping
+- Hand-to-arm inverse kinematics
+- Finger-to-gripper mapping
+- User calibration system
+- Safety limits and smoothing
+
+### Phase 4: Data Collection
+- Synchronized recording
+- Episode management
+- LeRobot dataset format
+- Data validation
+
+### Phase 5: Integration & Testing
+- End-to-end system testing
+- Performance optimization
+- User interface improvements
+
+## Contributing
+
+When adding new features:
+
+1. Follow the existing code structure
+2. Add configuration options to YAML files
+3. Include unit tests in the `tests/` directory
+4. Update this README with new functionality
+
+## Dependencies
+
+- **opencv-python**: Camera capture and image processing
+- **mediapipe**: Hand landmark detection
+- **numpy**: Numerical operations
+- **pyyaml**: Configuration file parsing
+- **matplotlib**: Visualization (future phases)
+
+## License
+
+This project is part of the LeRobot ecosystem and follows the same license terms.