@arfuhad/react-native-smart-camera 0.1.2 → 0.1.5

package/CHANGELOG.md

@@ -7,6 +7,77 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.1.5] - 2024-12-24
+
+### Added
+- **WebRTC Video Calling** - Full-featured WebRTC integration for peer-to-peer video calls
+  - `useWebRTC` hook for managing video calls with local/remote streams
+  - `useWebRTCWithDetection` hook combining WebRTC + face detection + eye tracking
+  - `WebRTCManager` class for low-level WebRTC control
+  - Signaling helpers: `createOffer`, `createAnswer`, `setRemoteDescription`, `addIceCandidate`
+  - Media controls: `switchCamera`, `toggleAudio`, `toggleVideo`
+  - Call state tracking: `idle`, `connecting`, `connected`, `disconnected`
+  
+- **WebRTC Types**
+  - `ICEServer`, `PeerConnectionConfig` for connection configuration
+  - `MediaConstraints`, `VideoConstraints`, `AudioConstraints`
+  - `CallState`, `WebRTCConnectionState`, `ICEConnectionState`
+  - `SessionDescription`, `ICECandidateEvent` for signaling
+  - Default configurations: `DEFAULT_ICE_SERVERS`, `DEFAULT_PEER_CONNECTION_CONFIG`
+
+- **Eye Status API** - Updated `useBlinkDetection` hook
+  - Returns real-time `eyeStatus` with `leftEye` and `rightEye` open probability
+  - User-configurable `eyeClosedThreshold` (default 0.5)
+  - `processEyeStatus(faces)` function to process detected faces
+  - `onEyeStatusChange` callback for status updates
+
+### Changed
+- **Breaking**: `useBlinkDetection` now returns `eyeStatus` instead of `blinkCount`
+  - Old: `{ lastBlink, blinkCount, resetCount }`
+  - New: `{ eyeStatus, processEyeStatus, reset }`
+  - Users implement their own blink counting logic using `onEyeStatusChange`
+
+- Updated README with comprehensive WebRTC and eye tracking documentation
+- Example app now includes tabbed interface with Face Detection and Video Call screens
+- Example app integrates with Socket.io signaling server for real video calls
+
+### Deprecated
+- `useSmartCameraWebRTC` - Use `useWebRTC` instead
+- `WebRTCBridge` - Use `WebRTCManager` instead
+
+## [0.1.4] - 2024-12-24
+
+### Added
+- `useFaceDetector` hook matching `react-native-vision-camera-face-detector` API
+- `useFaceDetectorWithCallback` convenience hook
+- All ML Kit contour types support (LEFT_EYEBROW_TOP, LEFT_EYEBROW_BOTTOM, etc.)
+- `autoMode` option for automatic coordinate scaling
+
+### Changed
+- Restructured package as pure React Native module (removed Expo module dependency)
+- Updated native plugin registration for standard React Native autolinking
+- Moved podspec to package root as `react-native-smart-camera.podspec`
+
+### Fixed
+- Fixed duplicate module registration issues
+- Fixed frame processor plugin registration on Android and iOS
+
+## [0.1.3] - 2024-12-24
+
+### Added
+- **Real-time face detection frame processor plugin** for VisionCamera
+  - Android: `FaceDetectorFrameProcessorPlugin` using ML Kit
+  - iOS: `FaceDetectorFrameProcessorPlugin` using ML Kit
+- `isFaceDetectorAvailable()` function to check if the plugin is ready
+
+### Changed
+- Updated `detectFaces` to use VisionCamera's `VisionCameraProxy.initFrameProcessorPlugin` API
+- Added VisionCamera and ML Kit dependencies to iOS podspec
+- Added VisionCamera frame processor support to Android build.gradle
+
+### Fixed
+- Fixed "Property '__detectFaces' doesn't exist" error by implementing proper frame processor plugin
+
 ## [0.1.2] - 2024-12-24
 
 ### Fixed
@@ -50,9 +121,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - iOS (requires Expo Dev Client or Bare Workflow)
 - Android (requires Expo Dev Client or Bare Workflow)
 
-[Unreleased]: https://github.com/arfuhad/react-native-smart-camera/compare/v0.1.2...HEAD
+[Unreleased]: https://github.com/arfuhad/react-native-smart-camera/compare/v0.1.5...HEAD
+[0.1.5]: https://github.com/arfuhad/react-native-smart-camera/compare/v0.1.4...v0.1.5
+[0.1.4]: https://github.com/arfuhad/react-native-smart-camera/compare/v0.1.3...v0.1.4
+[0.1.3]: https://github.com/arfuhad/react-native-smart-camera/compare/v0.1.2...v0.1.3
 [0.1.2]: https://github.com/arfuhad/react-native-smart-camera/compare/v0.1.1...v0.1.2
 [0.1.1]: https://github.com/arfuhad/react-native-smart-camera/compare/v0.1.0...v0.1.1
 [0.1.0]: https://github.com/arfuhad/react-native-smart-camera/releases/tag/v0.1.0
-
-

package/README.md

@@ -1,174 +1,294 @@
-# react-native-smart-camera
+# @arfuhad/react-native-smart-camera
 
-An Expo-compatible native module that provides camera preview via VisionCamera, comprehensive face detection using Google ML Kit, blink detection, and a native WebRTC bridge for video calling and streaming.
+A React Native package providing real-time face detection using VisionCamera + ML Kit, eye tracking for blink detection, and WebRTC video calling capabilities.
 
-## Requirements
+## Features
+
+- **Face Detection** - Real-time face detection using Google ML Kit via VisionCamera frame processor
+- **Eye Tracking** - Track eye open/close probability for blink detection
+- **WebRTC Video Calling** - Full-featured WebRTC integration for peer-to-peer video calls
+- **Expo Compatible** - Works with Expo Dev Client and EAS Build
 
-- Expo SDK ≥ 49
-- Expo Dev Client (required)
-- EAS Build
+## Requirements
 
-> **Important**: This package does NOT work with Expo Go. You must use Expo Dev Client or Bare Workflow.
+- React Native ≥ 0.72
+- Expo SDK ≥ 49 (if using Expo)
+- Expo Dev Client (Expo Go not supported)
 
-| Expo Workflow | Supported | Why |
-|---------------|-----------|-----|
-| Expo Managed (Go) | ❌ No | WebRTC + VisionCamera need native code |
-| Expo Dev Client | ✅ Yes | Allows custom native modules |
-| Expo Bare | ✅ Yes | Full native control |
+| Workflow | Supported | Notes |
+|----------|-----------|-------|
+| Expo Go | ❌ No | Requires native modules |
+| Expo Dev Client | ✅ Yes | Recommended |
+| Bare React Native | ✅ Yes | Full native control |
 
 ## Installation
 
 ```bash
-expo install react-native-smart-camera
-expo prebuild
-expo run:ios
-# or
-expo run:android
-```
+npm install @arfuhad/react-native-smart-camera
+
+# Peer dependencies
+npm install react-native-vision-camera react-native-worklets-core
 
-Or with EAS Build:
+# Optional: For WebRTC video calling
+npm install react-native-webrtc
+```
 
+For Expo:
 ```bash
-eas build --profile development
+npx expo prebuild
+npx expo run:ios  # or run:android
 ```
 
-## Usage
+## Quick Start
 
-### Basic Camera Preview
+### Face Detection with Eye Tracking
 
 ```tsx
-import { SmartCamera } from 'react-native-smart-camera';
+import { Camera, useCameraDevice, useFrameProcessor } from 'react-native-vision-camera';
+import { useRunOnJS } from 'react-native-worklets-core';
+import { useFaceDetector, useBlinkDetection, type Face } from '@arfuhad/react-native-smart-camera';
+
+function FaceDetectionScreen() {
+  const device = useCameraDevice('front');
+  
+  // Face detector
+  const { detectFaces } = useFaceDetector({
+    performanceMode: 'fast',
+    classificationMode: 'all', // Required for eye tracking
+  });
+  
+  // Eye tracking with custom threshold
+  const { eyeStatus, processEyeStatus } = useBlinkDetection({
+    eyeClosedThreshold: 0.3, // Eyes considered closed below this value
+    onEyeStatusChange: (status) => {
+      console.log('Left eye:', status.leftEye.openProbability);
+      console.log('Right eye:', status.rightEye.openProbability);
+    },
+  });
+
+  const handleFaces = useRunOnJS((faces: Face[]) => {
+    processEyeStatus(faces);
+  }, [processEyeStatus]);
+
+  const frameProcessor = useFrameProcessor((frame) => {
+    'worklet';
+    const faces = detectFaces(frame);
+    handleFaces(faces);
+  }, [detectFaces, handleFaces]);
 
-function App() {
   return (
-    <SmartCamera
-      camera="front"
+    <Camera
+      device={device}
+      isActive={true}
+      frameProcessor={frameProcessor}
       style={{ flex: 1 }}
     />
   );
 }
 ```
 
-### Face Detection with Blink Detection
+### WebRTC Video Calling
 
 ```tsx
-import { SmartCamera } from 'react-native-smart-camera';
+import { useWebRTC } from '@arfuhad/react-native-smart-camera';
+import { RTCView } from 'react-native-webrtc';
+import { io } from 'socket.io-client';
 
-function App() {
-  const handleBlink = (event) => {
-    console.log('Blink detected!', event);
+function VideoCallScreen() {
+  const socket = io('http://your-signaling-server:3000');
+  
+  const {
+    localStream,
+    remoteStream,
+    callState,
+    startLocalStream,
+    createPeerConnection,
+    createOffer,
+    createAnswer,
+    setRemoteDescription,
+    addIceCandidate,
+    onIceCandidate,
+    switchCamera,
+    toggleAudio,
+    toggleVideo,
+    cleanup,
+  } = useWebRTC({
+    config: {
+      iceServers: [{ urls: 'stun:stun.l.google.com:19302' }],
+    },
+  });
+
+  // Start a call
+  const startCall = async (peerId: string) => {
+    await startLocalStream();
+    createPeerConnection();
+    
+    onIceCandidate((event) => {
+      if (event.candidate) {
+        socket.emit('ice-candidate', { candidate: event.candidate, to: peerId });
+      }
+    });
+    
+    const offer = await createOffer();
+    socket.emit('offer', { offer, to: peerId });
   };
 
-  const handleFaceDetected = (faces) => {
-    console.log('Faces:', faces);
-  };
+  // Handle incoming offer
+  useEffect(() => {
+    socket.on('offer', async ({ offer, from }) => {
+      await startLocalStream();
+      createPeerConnection();
+      await setRemoteDescription(offer);
+      const answer = await createAnswer();
+      socket.emit('answer', { answer, to: from });
+    });
+
+    socket.on('answer', ({ answer }) => {
+      setRemoteDescription(answer);
+    });
+
+    socket.on('ice-candidate', ({ candidate }) => {
+      addIceCandidate(candidate);
+    });
+  }, []);
 
   return (
-    <SmartCamera
-      camera="front"
-      faceDetection={{
-        enabled: true,
-        performanceMode: 'fast',
-        landmarkMode: 'all',
-        classificationMode: 'all',
-      }}
-      blinkDetection
-      onBlinkDetected={handleBlink}
-      onFaceDetected={handleFaceDetected}
-      style={{ flex: 1 }}
-    />
+    <View style={{ flex: 1 }}>
+      {remoteStream && (
+        <RTCView streamURL={remoteStream.toURL()} style={{ flex: 1 }} />
+      )}
+      {localStream && (
+        <RTCView streamURL={localStream.toURL()} style={styles.localVideo} />
+      )}
+      <View style={styles.controls}>
+        <Button onPress={switchCamera} title="Switch Camera" />
+        <Button onPress={toggleAudio} title="Mute" />
+        <Button onPress={toggleVideo} title="Video Off" />
+      </View>
+    </View>
   );
 }
 ```
 
-### WebRTC Video Calling
+### Combined: Video Call with Face Detection
 
 ```tsx
-import { SmartCamera } from 'react-native-smart-camera';
-import { RTCPeerConnection } from 'react-native-webrtc';
-
-function VideoCall() {
-  const peerConnection = new RTCPeerConnection(config);
-
-  return (
-    <SmartCamera
-      camera="front"
-      webrtc={{
-        enabled: true,
-        peerConnection,
-        mode: 'call',
-      }}
-      style={{ flex: 1 }}
-    />
-  );
+import { useWebRTCWithDetection } from '@arfuhad/react-native-smart-camera';
+
+function VideoCallWithDetection() {
+  const {
+    // WebRTC
+    localStream,
+    remoteStream,
+    startLocalStream,
+    createPeerConnection,
+    // Face detection
+    faces,
+    detectFaces,
+    // Eye tracking
+    eyeStatus,
+    processFrame,
+  } = useWebRTCWithDetection({
+    config: { iceServers: [{ urls: 'stun:stun.l.google.com:19302' }] },
+    faceDetection: { enabled: true, classificationMode: 'all' },
+    eyeTracking: { enabled: true, eyeClosedThreshold: 0.3 },
+  });
+
+  // Use VisionCamera for detection + WebRTC for streaming
 }
 ```
 
 ## API Reference
 
-### SmartCamera Props
+### Hooks
+
+#### `useFaceDetector(options)`
+
+Face detection for VisionCamera frame processor.
 
 ```tsx
-interface SmartCameraProps {
-  // Camera settings
-  camera?: 'front' | 'back';
-  fps?: number;
-  style?: ViewStyle;
-  
-  // Face Detection Options
-  faceDetection?: FaceDetectionConfig;
-  
-  // Blink Detection
-  blinkDetection?: boolean;
-  onBlinkDetected?: (event: BlinkEvent) => void;
-  
-  // Face Detection Callback
-  onFaceDetected?: (faces: Face[]) => void;
-  
-  // WebRTC Configuration
-  webrtc?: WebRTCConfig;
-}
+const { detectFaces } = useFaceDetector({
+  performanceMode: 'fast' | 'accurate',
+  landmarkMode: 'none' | 'all',
+  contourMode: 'none' | 'all',
+  classificationMode: 'none' | 'all',  // Required for eye tracking
+  trackingEnabled: boolean,
+  cameraFacing: 'front' | 'back',
+});
 ```
 
-### Face Detection Options
+#### `useBlinkDetection(options)`
 
-#### Common Options (Frame Processor and Static Images)
+Eye tracking and status monitoring.
 
-| Option | Description | Default | Options |
-|--------|-------------|---------|---------|
-| `performanceMode` | Favor speed or accuracy when detecting faces | `'fast'` | `'fast'`, `'accurate'` |
-| `landmarkMode` | Whether to identify facial landmarks: eyes, ears, nose, cheeks, mouth | `'none'` | `'none'`, `'all'` |
-| `contourMode` | Whether to detect contours of facial features. Contours are detected for only the most prominent face in an image | `'none'` | `'none'`, `'all'` |
-| `classificationMode` | Whether to classify faces into categories such as 'smiling' and 'eyes open' | `'none'` | `'none'`, `'all'` |
-| `minFaceSize` | Sets the smallest desired face size, expressed as the ratio of the width of the head to width of the image | `0.15` | `number` |
-| `trackingEnabled` | Whether to assign faces an ID to track faces across images. Note: Don't enable with contourMode for best performance | `false` | `boolean` |
+```tsx
+const { eyeStatus, processEyeStatus, reset } = useBlinkDetection({
+  enabled: boolean,
+  eyeClosedThreshold: number,  // 0-1, default 0.5
+  onEyeStatusChange: (status: EyeStatusResult) => void,
+});
 
-#### Frame Processor Options
+// EyeStatusResult
+interface EyeStatusResult {
+  leftEye: { openProbability: number; isClosed: boolean };
+  rightEye: { openProbability: number; isClosed: boolean };
+  faceId?: number;
+  timestamp: number;
+}
+```
 
-| Option | Description | Default | Options |
-|--------|-------------|---------|---------|
-| `cameraFacing` | Current active camera | `'front'` | `'front'`, `'back'` |
-| `autoMode` | Handle auto scale (face bounds, contour, landmarks) and rotation on native side. If disabled, results are relative to frame coordinates, not screen/preview. Disable when using Skia Frame Processor | `false` | `boolean` |
-| `windowWidth` | Required when using `autoMode`. Screen width for coordinate scaling | `1.0` | `number` |
-| `windowHeight` | Required when using `autoMode`. Screen height for coordinate scaling | `1.0` | `number` |
+#### `useWebRTC(options)`
+
+WebRTC video calling with signaling helpers.
+
+```tsx
+const {
+  // Streams
+  localStream,
+  remoteStream,
+  
+  // State
+  callState,  // 'idle' | 'connecting' | 'connected' | 'disconnected'
+  isAudioEnabled,
+  isVideoEnabled,
+  currentCamera,
+  
+  // Local stream
+  startLocalStream,
+  stopLocalStream,
+  switchCamera,
+  toggleAudio,
+  toggleVideo,
+  
+  // Peer connection
+  createPeerConnection,
+  closePeerConnection,
+  
+  // Signaling (integrate with your server)
+  createOffer,
+  createAnswer,
+  setRemoteDescription,
+  addIceCandidate,
+  onIceCandidate,
+  
+  // Cleanup
+  cleanup,
+} = useWebRTC({
+  config: PeerConnectionConfig,
+  mediaConstraints: MediaConstraints,
+  onCallStateChange: (state) => void,
+  onRemoteStream: (stream) => void,
+  onError: (error) => void,
+});
+```
 
-#### Static Images
+#### `useWebRTCWithDetection(options)`
 
-| Option | Description | Default | Options |
-|--------|-------------|---------|---------|
-| `image` | Image source for static face detection | - | `number`, `string`, `{ uri: string }` |
+Combined WebRTC + face detection + eye tracking.
 
 ### Types
 
 ```tsx
-interface BlinkEvent {
-  timestamp: number;
-  leftEyeOpen: number;  // 0.0 - 1.0
-  rightEyeOpen: number; // 0.0 - 1.0
-  isBlink: boolean;
-  faceId?: number;
-}
-
 interface Face {
   bounds: { x: number; y: number; width: number; height: number };
   landmarks?: FaceLandmarks;
@@ -179,128 +299,61 @@ interface Face {
   trackingId?: number;
 }
 
-interface WebRTCConfig {
-  enabled: boolean;
-  peerConnection?: RTCPeerConnection;
-  mode?: 'call' | 'stream';
+interface ICEServer {
+  urls: string | string[];
+  username?: string;
+  credential?: string;
 }
-```
-
-### Static Image Detection
-
-```tsx
-import { detectFacesInImage } from 'react-native-smart-camera';
-
-const faces = await detectFacesInImage({
-  image: require('./photo.jpg'),
-  performanceMode: 'accurate',
-  landmarkMode: 'all',
-});
-```
-
-## Hooks
-
-### useSmartCamera
-
-```tsx
-import { useSmartCamera } from 'react-native-smart-camera';
-
-const { 
-  hasPermission,
-  requestPermission,
-  device,
-  switchCamera,
-} = useSmartCamera();
-```
-
-### useFaceDetection
-
-```tsx
-import { useFaceDetection } from 'react-native-smart-camera';
-
-const { faces, isDetecting } = useFaceDetection({
-  performanceMode: 'fast',
-  classificationMode: 'all',
-});
-```
-
-### useBlinkDetection
-
-```tsx
-import { useBlinkDetection } from 'react-native-smart-camera';
-
-const { lastBlink, blinkCount } = useBlinkDetection({
-  debounceMs: 300,
-});
-```
-
-### useSmartCameraWebRTC
 
-```tsx
-import { useSmartCameraWebRTC } from 'react-native-smart-camera';
+interface PeerConnectionConfig {
+  iceServers?: ICEServer[];
+  iceTransportPolicy?: 'all' | 'relay';
+}
 
-const {
-  videoTrack,
-  startStreaming,
-  stopStreaming,
-  switchCamera,
-} = useSmartCameraWebRTC({
-  peerConnection,
-  mode: 'call',
-});
+interface SessionDescription {
+  type: 'offer' | 'answer';
+  sdp: string;
+}
 ```
 
 ## Expo Config Plugin
 
-This package includes an Expo config plugin that automatically configures permissions and native dependencies.
+Add to your `app.json`:
 
 ```json
 {
   "expo": {
-    "plugins": ["react-native-smart-camera"]
+    "plugins": ["@arfuhad/react-native-smart-camera"]
   }
 }
 ```
 
-The plugin automatically adds:
-- Camera permission (iOS & Android)
-- Microphone permission (iOS & Android)
+The plugin automatically configures:
+- Camera and microphone permissions (iOS & Android)
 - Required iOS frameworks
 - Android ProGuard rules for ML Kit
 
-## Architecture
-
-```
-Expo App
- ├── SmartCamera (JS Component)
- ├── VisionCamera (native camera)
- ├── Frame Processor (worklet)
- │     └── ML Kit Face Detection
- ├── Native WebRTC Bridge
- │     ├── iOS (Swift)
- │     └── Android (Kotlin)
- └── react-native-webrtc
-```
-
 ## Peer Dependencies
 
-This package requires the following peer dependencies:
-
 ```json
 {
   "react-native-vision-camera": ">=3.0.0",
-  "react-native-webrtc": ">=118.0.0",
-  "react-native-worklets-core": ">=1.0.0"
+  "react-native-worklets-core": ">=1.0.0",
+  "react-native-webrtc": ">=118.0.0"  // Optional, for video calling
 }
 ```
 
+## Example App
+
+See the `smart-camera-example` directory for a complete example with:
+- Face detection with eye tracking
+- WebRTC video calling with Socket.io signaling
+
 ## Limitations
 
 - ❌ Not supported in Expo Go
-- ❌ Requires Expo Dev Client or Bare Workflow
-- ❌ Native build required (via `expo run:ios/android` or EAS Build)
+- ❌ Requires native build (Expo Dev Client or bare workflow)
 
 ## License
 
 MIT
-