reis-mobile 0.3.6 → 0.5.0

package/skills/mobile-android/references/media/camerax/references/low-light.md

@@ -0,0 +1,124 @@
+This guide covers implementing low-light features using **Night mode
+extensions** and **Low Light Boost (LLB)**.
+
+## Choosing the right tool
+
+| Feature | Used for | Implementation | UX impact |
+|---|---|---|---|
+| **Night mode** | High-quality stills | `ExtensionsManager` | Possibly requires user to hold still for several seconds. |
+| **LLB, AE mode** | Real-time preview or video | `Camera2Interop`, CameraX utility | Hardware drops the frame rate to increase brightness. |
+| **LLB, Play services** | Real-time preview and video | `SurfaceProcessor` | Software-based brightening; maintains a higher frame rate. |
+
+*** ** * ** ***
+
+## Night mode extension
+
+CameraX Extensions provide access to the device's built-in computational
+photography pipeline.
+
+### Basic setup
+
+To set up the extension, initialize the extension manager:
+
+
+```kotlin
+// Use ListenableFuture.await() extension function for coroutine support
+val extensionsManager = ExtensionsManager.getInstanceAsync(context, cameraProvider).await()
+if (extensionsManager.isExtensionAvailable(cameraSelector, ExtensionMode.NIGHT)) {
+    val nightSelector = extensionsManager.getExtensionEnabledCameraSelector(
+        cameraSelector, ExtensionMode.NIGHT
+    )
+    cameraProvider.bindToLifecycle(lifecycleOwner, nightSelector, imageCapture, preview)
+}
+```
+
+<br />
+
+### Comprehensive features
+
+- **Image postview** : Display a low-resolution image immediately while the multi-frame processing occurs.
+
+  ```kotlin
+  val imageCapture = ImageCapture.Builder()
+      .setPostviewEnabled(true)
+      .build()
+  ```
+- **Extension strength** : Let users control the intensity of the night effect.
+
+  ```kotlin
+  // Set the strength of the active extension (e.g. NIGHT mode intensity)
+  val extensionsManager = ExtensionsManager.getInstanceAsync(context, cameraProvider).await()
+  val extensionsControl = extensionsManager.getCameraExtensionsControl(camera.cameraControl)
+  extensionsControl?.setExtensionStrength(strength)
+  ```
+- **Capture progress** : Show a UI progress bar for long exposures.
+
+  ```kotlin
+  // Use the suspend extension function for takePicture to avoid callback boilerplate
+  try {
+      val result = imageCapture.takePicture(outputOptions)
+      // Use result.savedUri or other fields
+  } catch (e: ImageCaptureException) {
+      // Handle capture failure
+  }
+  ```
+
+*** ** * ** ***
+
+## Low-light boost
+
+LLB is designed for preview and video streams where you prefer high frame rates.
+
+### AE mode
+
+The built-in CameraX way to prioritize brightness. It modifies the hardware's
+auto-exposure algorithm.
+
+- **Activation** : Use `CameraControl.enableLowLightBoostAsync`.
+- **Implementation** :
+
+  ```kotlin
+  // Enable Low Light Boost (LLB) natively in CameraX 1.4+
+  camera.cameraControl.enableLowLightBoostAsync(true)
+  ```
+- **Monitoring** : Observe `CameraInfo.lowLightBoostState` to track when the hardware actively applies the enhancement.
+
+### Google Play services LLB
+
+It's a multi-step implementation that uses a session-based `SurfaceProcessor`.
+
+**Dependency** : `com.google.android.gms:play-services-camera-low-light-boost`
+
+To implement Google Play services LLB, follow these core steps:
+
+1. **Initialize client** : `val client = LowLightBoost.getClient`.
+2. **Implement `SurfaceProcessor`** :
+   - **Manage session** : Call `client.createSession`.
+   - **Forward required metadata** : Observe the camera's `TotalCaptureResult` stream and forward every result to the session: `session.processCaptureResult`.
+   - **Provide surface** : Get the input surface from the session, `session.getCameraSurface`, and provide it to the camera's `SurfaceRequest`.
+   - **Lifecycle** : Release the session, `session.release`, when the processor is closed or the `SurfaceRequest` completes.
+3. **Wire using `CameraEffect`** :
+
+   ```kotlin
+   val effect = SimpleCameraEffect(
+       CameraEffect.PREVIEW or CameraEffect.VIDEO_CAPTURE,
+       executor,
+       llbSurfaceProcessor
+   ) { throw it }
+
+   // Add to UseCaseGroup
+   val useCaseGroup = UseCaseGroup.Builder()
+       .addUseCase(preview)
+       .addUseCase(videoCapture)
+       .addEffect(effect)
+       .build()
+   ```
+4. **Scene detection** : Use `session.setSceneDetectorCallback` to receive `boostStrength` updates for real-time UI indicators.
+
+*** ** * ** ***
+
+## Implementation notes
+
+- **Thread safety** : Always handle `ExtensionsManager` and `LowLightBoostClient` initialization asynchronously.
+- **FPS trade-offs**: AE mode LLB often drops the frame rate significantly to increase brightness.
+- **Compatibility** : Extensions, Night Mode, possibly conflict with `ConcurrentCamera`. Always verify support before binding.

package/skills/mobile-android/references/media/camerax/references/mlkit-spatial.md

@@ -0,0 +1,75 @@
+When you use ML Kit for features such as face mesh, object detection, or pose
+detection, the most common failure point is the coordinate disparity between the
+analysis image and the viewfinder UI.
+
+## The mapping mindset
+
+| Dimension | Analysis frame | Viewfinder UI on the screen |
+|---|---|---|
+| Resolution | Fixed, such as 640x480 | Dynamic, such as 1080x2400 |
+| Rotation | 0° for the raw buffer | 90° or 270° in portrait or landscape mode |
+| Origin | Top-left of buffer at (0,0) | Top-left of screen at (0,0) |
+
+*** ** * ** ***
+
+## Follow the implementation guide
+
+### Coordinate transformation matrix
+
+Android provides the `Viewport` and `UseCaseGroup` APIs to calculate the
+transformation matrix automatically. **Don't** calculate aspect ratio scaling
+manually.
+
+
+```kotlin
+val transform = previewView.viewPort?.let { viewPort ->
+    // Use CameraX's built-in coordinate mapper
+    viewPort.getTransformationMatrix(imageProxy.imageInfo.rotationDegrees)
+}
+```
+
+<br />
+
+### Handling the "double rotation" bug
+
+ML Kit results, bounding boxes, are relative to the **rotated buffer**. If the
+device is in portrait, the buffer is often 480x640, landscape, but the screen
+is 1080x1920.
+
+To map the coordinates, use the following workflow:
+
+1. Query `imageProxy.imageInfo.rotationDegrees`.
+2. Pass this rotation to the ML Kit `InputImage`.
+3. Use the `MappingUtils.transformRect` method to map the result `Rect` to the screen.
+
+### Face mesh and pose normalization
+
+For high-precision spatial analysis, for example, "Is the user's hand at a
+specific screen button?", use **normalized coordinates from 0.0 to 1.0**.
+
+
+```kotlin
+// Example: Converting a Pose landmark to a Screen Coordinate
+val screenX = landmark.position.x / analysisWidth * screenWidth
+val screenY = landmark.position.y / analysisHeight * screenHeight
+```
+
+<br />
+
+**Warning** : Always account for **mirrored lenses** . If the `LENS_FACING_FRONT`
+is used, you must flip the X-coordinate: `actualX = screenWidth - screenX`.
+
+### Overlays and canvas clipping
+
+Use a custom `GraphicOverlay` view on top of the `PreviewView`.
+
+- **Buffer lock** : Ensure your `GraphicOverlay` clears its canvas every time a new `ImageAnalysis` frame is processed to prevent "ghosting" of bounding boxes.
+
+*** ** * ** ***
+
+## Spatial pitfalls
+
+- **The "stretched box" bug** : Caused by assuming the Analysis Frame aspect ratio, 4:3, matches the screen aspect ratio, 21:9. Use `PreviewView.SCALE_TYPE_FILL_CENTER` and map coordinates accordingly.
+- **Latency** : If ML processing exceeds 50 ms, the bounding box trails behind the user's face.
+  - **Fix** : Use `ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST` to avoid queuing stale frames.
+- **Sensor vs. display rotation** : On some tablets, the sensor is mounted horizontally. Always query `display.rotation` and `cameraInfo.sensorRotationDegrees`.

package/skills/mobile-android/references/media/camerax/references/modern-apis.md

@@ -0,0 +1,42 @@
+Always prefer these various abstractions over legacy Camera2 or early CameraX
+implementations.
+
+## Compare APIs
+
+| Use case | Legacy or verbose way | Recommendation |
+|---|---|---|
+| QR or face scanning | `ImageAnalysis.Analyzer` and manual ByteBuffer math | **`MlKitAnalyzer`**, which automates coordinate mapping and multi-format support |
+| Post-processing | Custom OpenGL shaders or `SurfaceTexture` | **`Media3Effect`** for composable, declarative effects |
+| Dual camera | Manual binding of two UseCases | **`ConcurrentCamera`**, which provides built-in support in CameraX 1.3 and higher |
+| High dynamic range | Manual bit-depth and profile config | **`DynamicRange`** , which uses `DYNAMIC_RANGE_HLG10` or `SDR` |
+| Zoom and focus | `Camera2Interop` for CameraX-to-Camera2 mapping | **`CameraControl.setZoomRatio`** or **`setLinearZoom`** |
+
+## Hardware awareness
+
+Modern APIs abstract away the complexity of hardware diversity.
+
+- **`CameraSelector`** : Use `DEFAULT_BACK_CAMERA` or `DEFAULT_FRONT_CAMERA` instead of hardcoding camera IDs. Use `filter` if you need specific lens capabilities.
+- **Extensions** : Before enabling advanced modes, such as night, bokeh, and face retouch, use `ExtensionsManager` to query whether the device supports them.
+- **Foldables** : Observe `Lifecycle` and `Viewport` updates to handle posture changes, like a half-opened posture, on foldable devices.
+
+## Required dependencies
+
+Add the following dependencies to your `libs.versions.toml` file:
+
+    # CameraX ML Kit
+
+    androidx-camera-mlkit-vision = { group = "androidx.camera", name =
+    "camera-mlkit-vision", version.ref = "camerax" }
+
+    # Media3 effects
+
+    androidx-media3-effect = { group = "androidx.media3", name = "media3-effect",
+    version.ref = "media3" }
+
+    # Camera extensions
+
+    androidx-camera-extensions = { group = "androidx.camera", name =
+    "camera-extensions", version.ref = "camerax" }
+
+Refer to the official [CameraX Release Notes](https://developer.android.com/jetpack/androidx/releases/camera) for the
+stable versions.

package/skills/mobile-android/references/media/camerax/references/testing.md

@@ -0,0 +1,69 @@
+Automated testing for camera features is notoriously difficult because you
+can't easily mock physical hardware, lighting, or motion. This guide provides
+patterns for reliable, hermetic camera tests.
+
+## Develop a testing mindset
+
+| Challenge | Conventional approach | Camera technical approach |
+|---|---|---|
+| **Frameworks** | `Mockito` or `MockK` | **Fakes over mocks** |
+| **Assertions** | `assertEquals`, `assertTrue` | **Google Truth, `assertThat`** |
+| **Environment** | Implied environments | **Explicit `@RunWith` annotations** |
+| **Async operations** | `Thread.sleep` | **Explicit `timeoutMillis` or `IdlingResource`** |
+
+*** ** * ** ***
+
+## Follow the implementation guide
+
+### Fakes over mocks
+
+**Don't use Mockito.** Relying on mocks for complex, rapidly changing interfaces
+like `ImageProxy` or `CameraInfo` makes tests brittle. Instead, build "Fake"
+implementations that verify state rather than behavior.
+
+
+```kotlin
+// Create a Fake ImageProxy for ML Testing (Fakes over Mocks)
+val fakeImage = FakeImageProxy(w = 640, h = 480)
+
+// Feed the fake buffer into your analyzer
+```
+
+<br />
+
+### Mock camera capabilities
+
+Use `FakeAppConfig` from `androidx.camera:camera-testing` to simulate specific
+hardware constraints in tests, such as a device without a flash.
+
+
+```kotlin
+// Use awaitInstance() extension function for coroutine-based provider retrieval
+val cameraProvider = ProcessCameraProvider.awaitInstance(context)
+```
+
+<br />
+
+### Use Truth assertions
+
+Use Google Truth, `assertThat`, instead of standard JUnit assertions. It
+provides more readable assertion chains and useful failure messages.
+
+### Test asynchronous lifecycles
+
+Camera initialization is asynchronous. Use `IdlingResource` to ensure
+your test waits for the `UseCase` to be bound before asserting.
+
+To test asynchronous lifecycles, use the following pattern:
+
+1. Wrap the `ProcessCameraProvider` initialization in a `CountDownLatch` or `IdlingResource`.
+2. Assert only after the `cameraControl` instance is non-null.
+
+*** ** * ** ***
+
+## Testing pitfalls
+
+- **Resource leaks** : To prevent "Camera in Use" errors, in your `@After` block, call `cameraProvider.unbindAll`.
+- **The "flaky initializer"**: Camera tests often fail on CI because the "Virtual Camera" takes too long to warm up. Use a sufficient explicit timeout for the first initialization.
+- **Permission blockers** : To bypass the system permission dialogs, in your Espresso tests, use `GrantPermissionRule`.
+- **Resolution mismatch** : Tests on emulators often default to 640x480. Ensure your `ResolutionSelector` handles this low-res fallback correctly.

package/skills/mobile-android/references/media/camerax/references/thermals.md

@@ -0,0 +1,85 @@
+Camera operations are among the most power-intensive tasks on mobile devices.
+Without proactive management, the system throttle hardware, drop frames, or
+force-close the camera app.
+
+## The thermal management strategy
+
+| Priority | Strategy | Implementation |
+|---|---|---|
+| **1. Inform** | Use case hints | Provide `StreamUseCase` to allow the OS to optimize hardware. |
+| **2. Monitor** | Thermal state listener | Observe `PowerManager.addThermalStatusListener`. |
+| **3. Act** | Graceful degradation | Dynamically reduce FPS, resolution, or disable demanding effects such as HDR. |
+
+*** ** * ** ***
+
+## Follow the implementation guide
+
+### Stream use case optimization
+
+Android 13 (API level 33) introduced `StreamUseCase`. This is the **single most
+effective** way to tell the hardware how to balance quality versus power.
+
+
+```kotlin
+// In CameraX: Set the hint on your Use Case
+val preview = Preview.Builder()
+    .setTargetName("Preview")
+    .apply {
+        Camera2Interop.Extender(this).setStreamUseCase(
+            CameraMetadata.SCALER_AVAILABLE_STREAM_USE_CASES_VIDEO_CALL.toLong()
+        )
+    }
+    .build()
+```
+
+<br />
+
+Review the following key use cases for stream optimization:
+
+- `PREVIEW`: This option is the default and provides a balanced configuration.
+- `STILL_CAPTURE`: This option provides high-quality capture for short bursts.
+- `VIDEO_RECORD`: This option maintains sustained power and is optimized for encoding.
+- `VIDEO_CALL`: This option minimizes power consumption for long-duration sessions.
+
+### Monitor thermal status
+
+Don't wait for a crash. Monitor the `PowerManager` status and react before
+`THERMAL_STATUS_CRITICAL`.
+
+
+```kotlin
+val powerManager = context.getSystemService(Context.POWER_SERVICE) as PowerManager
+powerManager.addThermalStatusListener { status ->
+    when (status) {
+        PowerManager.THERMAL_STATUS_MODERATE -> {
+            // Signal to UI: "Device is warming up"
+        }
+        PowerManager.THERMAL_STATUS_SEVERE -> {
+            // ACTION: Reduce Frame Rate from 60fps to 30fps
+            // ACTION: Disable HDR or High-Quality Post-processing
+        }
+        PowerManager.THERMAL_STATUS_CRITICAL -> {
+            // ACTION: Close the camera session to prevent hardware damage
+        }
+    }
+}
+```
+
+<br />
+
+### Graceful degradation tiers
+
+| Tier | Action | User impact |
+|---|---|---|
+| **Mild** | Stop background analysis using ML Kit | Minimal |
+| **Moderate** | Cap frame rate to 30 FPS | Noticeable but smooth |
+| **Severe** | Drop resolution from 1080p to 720p | Significant visual change |
+| **Critical** | Shut down the session | App unusable in safe mode |
+
+*** ** * ** ***
+
+## Thermal pitfalls
+
+- **The "double work" bug**: Don't run two high-resolution streams---such as a preview and a video capture stream---at different aspect ratios unless necessary. This forces the image signal processor (ISP) to perform double the scaling work, which generates excessive heat.
+- **Surface overload** : Don't use multi-step `SurfaceProcessor` or `Media3Effect` chains during `THERMAL_STATUS_SEVERE`.
+- **Flash heat** : Flash or torch usage generates high thermal load. Proactively disable the flash if thermal status is `SEVERE`.

package/skills/mobile-android/references/media/camerax/references/wear-os.md

@@ -0,0 +1,76 @@
+Developing camera features for Wear OS is rarely about the watch's own lens, if
+it even has one. It's almost always about creating a **Remote Viewfinder** to
+control the phone's camera.
+
+## The Wear OS remote mindset
+
+| Feature | Phone camera app | Wear OS remote app |
+|---|---|---|
+| **Screen** | Rectangular (Large) | **Circular and less than 2 inches in size** |
+| **Connectivity** | Local Hardware | **Bluetooth or Wi-Fi data layer API** |
+| **Latency** | Direct and less than 20 ms | **Networked, 100 ms to 500 ms** |
+| **Interaction** | Multi-touch gestures | **Rotary input or single taps** |
+
+*** ** * ** ***
+
+## Follow the implementation guide
+
+### The circular UI challenge
+
+Wear OS devices are often round. Standard rectangular layouts clip corner
+buttons.
+
+Follow these blueprint recommendations:
+
+- Use `Horologist` or `Wear Compose` libraries.
+- Use `ScalingLazyColumn` for lists so that items stay within the "safe zone" of the circular display.
+- **Preview scaling**: Crop the center of the rectangular phone viewfinder to fit the circular watch screen.
+
+### Stream the viewfinder
+
+You can't send a raw 60 fps stream over Bluetooth. compress and throttle.
+
+
+```kotlin
+// Example: Sending a viewfinder frame to the watch
+val bitmap = previewView.bitmap // Capture current frame
+if (bitmap != null) {
+    val compressed = compressToJpeg(bitmap, quality = 50)
+    val request = PutDataMapRequest.create("/camera/preview").apply {
+        dataMap.putAsset("image", Asset.createFromBytes(compressed))
+    }
+    Wearable.getDataClient(context).putDataItem(request.asPutDataRequest())
+}
+```
+
+<br />
+
+**Optimization** : Cap the watch preview at **10-15 fps** to preserve battery and
+bandwidth.
+
+### Remote triggers and syncing
+
+Use the `MessageClient` for low-latency commands like "Take Photo" or "Switch
+Camera."
+
+
+```kotlin
+// Watch sends a trigger to the phone
+Wearable.getMessageClient(context).sendMessage(nodeId, "/camera/capture", null)
+```
+
+<br />
+
+### Rotary input support
+
+On devices that support it, use the physical crown, Rotary Input, to control
+**Zoom** or **Exposure**.
+
+*** ** * ** ***
+
+## Wear OS pitfalls
+
+- **Corner clipping** : Placing a **Close** button in the top-right corner of a square layout makes it impossible to tap the button on a round watch display.
+- **Battery drain**: Sustained Bluetooth data transfer, viewfinder sync, drains watch battery. Proactively close the remote app if the phone screen is turned off.
+- **Node discovery** : The phone is possibly connected to multiple "Nodes" (watches, earbuds, or tablets). Ensure your `CapabilityClient` filters for the specific `camera_remote_host` capability.
+- **Disconnect handling**: If the watch disconnects, the phone camera must stop its high-power preview to save energy.

package/skills/mobile-android/references/media/camerax/references/xr.md

@@ -0,0 +1,54 @@
+Developing camera features for XR devices, headsets, and AR glasses requires a
+shift from 2D pixel-pushing to 3D spatial awareness.
+
+## Understand the XR development mindset
+
+| Concept | Mobile focus | **XR focus** |
+|---|---|---|
+| **Input** | Raw camera stream | **Spatial tracking (visual-inertial odometry (VIO) or simultaneous localization and mapping (SLAM))** |
+| **Output** | Screen viewfinder | **Stereo passthrough and occlusion** |
+| **Constraint** | Battery life | **Motion-to-photon latency (less than 20 ms)** |
+
+*** ** * ** ***
+
+## Follow the implementation guide
+
+### API selection
+
+On XR devices, standard `CameraX` implementations are often restricted or
+insufficient. Always use spatial software development kits (SDKs):
+
+- **ARCore**: Use ARCore for plane detection, depth sensing, and motion tracking.
+- **OpenXR**: Use OpenXR as the cross-platform standard for VR and AR rendering and input.
+- **OEM SDKs**: Use manufacturer-specific libraries for hardware-accelerated passthrough.
+
+### Handle spatial passthrough
+
+Unlike a 2D viewport, XR passthrough is often system-managed.
+
+**\[Key requirement\] Frame synchronization**: Synchronize your application's
+frame clock with the headset's head-mounted display (HMD) pose.
+
+```kotlin
+// Example: Querying the spatial pose for the current camera frame
+val headPose = xrSession.getHeadPose(frameTime)
+val projectionMatrix = headPose.getProjectionMatrix(eyeIndex)
+```
+
+<br />
+
+### Manage depth and occlusion
+
+Digital content must respect real-world depth to ensure accurate occlusion.
+
+- **Depth map** : Access raw depth data using `ARCore` or `SurfaceProcessor` to create an occlusion mask.
+- **Hardware buffers** : Use `HardwareBuffer` to share camera frames directly with the GPU without CPU-side copies to minimize latency.
+
+*** ** * ** ***
+
+## XR pitfalls
+
+- **The nausea limit**: Any processing that delays the viewfinder by more than 20 ms causes user sickness. Don't perform image processing on the main thread.
+- **Privacy restrictions** : Some XR devices return a black frame if you attempt to record the "Passthrough" layer. Check `Session.isRecordingSupported`.
+- **Field of view (FOV)**: The camera FOV possibly doesn't match the display FOV. Use the SDK's projection matrixes instead of calculating aspect ratios manually.
+- **Front buffer rendering** : If the device supports it, use `FrontBufferRenderer` for real-time overlays to bypass standard double-buffering latency.