@gyeonghokim/fisheye.js 1.0.2 → 1.1.1

package/README.md

@@ -1,14 +1,18 @@
 # fisheye.js
 
-DEMO: https://gyeonghokim.github.io/fisheye.js/
+DEMO: <https://gyeonghokim.github.io/fisheye.js/>
 
-> Modern fisheye dewarping library for the web using **WebGPU** (general-purpose GPU compute)
+> Modern fisheye undistortion library for the web using **WebGPU** (general-purpose GPU compute)
 
-fisheye.js processes [VideoFrame](https://developer.mozilla.org/en-US/docs/Web/API/VideoFrame)s with **WebGPU compute shaders**—no canvas 2D—and corrects fisheye lens distortion using the **OpenCV fisheye model** (Kannala–Brandt–style polynomial in angle θ with coefficients k1–k4). This is the same model as in [OpenCV’s fisheye module](https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html), not UCM (Unified Camera Model) or a simple radial model.
+fisheye.js processes [VideoFrame](https://developer.mozilla.org/en-US/docs/Web/API/VideoFrame)s with **WebGPU compute shaders**—no canvas 2D—and corrects fisheye lens distortion using the **OpenCV fisheye model** (Kannala–Brandt–style polynomial in angle θ with coefficients k1–k4). This is the same model as in [OpenCV's fisheye module](https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html).
+
+## Learning Docs
+
+See the full educational curriculum in `doc/index.md`.
 
 ## Features
 
-- **WebGPU GPGPU**: Compute-shader pipeline via [TypeGPU](https://www.npmjs.com/package/typegpu); input/output as textures and readback to VideoFrame—no canvas element for dewarping
+- **WebGPU GPGPU**: Compute-shader pipeline via [TypeGPU](https://www.npmjs.com/package/typegpu); input/output as textures and readback to VideoFrame—no canvas element for undistortion
 - **OpenCV fisheye (Kannala–Brandt) model**: Distortion model `θ_d = θ × (1 + k1·θ² + k2·θ⁴ + k3·θ⁶ + k4·θ⁸)` for accurate calibration
 - **WebCodecs**: Built on the [VideoFrame](https://developer.mozilla.org/en-US/docs/Web/API/VideoFrame) API
 - **ESM**: `import { Fisheye } from "@gyeonghokim/fisheye.js"`
@@ -33,7 +37,7 @@ if you installed `@webgpu/types`,
 ```
 
 > Why should I install webgpu types?
-> This library does not render your binary, it just dewarp the VideoFrame.
+> This library does not render your binary, it just undistorts the VideoFrame.
 > You should make **your own YUV renderer**, or you can install `@gyeonghokim/yuv-player`.
 
 in your code,
@@ -41,48 +45,45 @@ in your code,
 ```ts
 import { Fisheye } from "@gyeonghokim/fisheye.js";
 
-const dewarper = new Fisheye({
+const fisheye = new Fisheye({
+  // OpenCV fisheye distortion coefficients (from calibration)
   k1: 0.5,
   k2: 0.0,
   k3: 0.0,
   k4: 0.0,
+
+  // Output configuration
   width: 1920,
   height: 1080,
-  fov: 180, // Field of view in degrees
-  centerX: 0, // X offset of lens center (-1.0 to 1.0)
-  centerY: 0, // Y offset of lens center (-1.0 to 1.0)
-  zoom: 1.0, // Zoom factor
+
+  // Optional: OpenCV camera matrix parameters
+  fx: 1000, // focal length x (pixels)
+  fy: 1000, // focal length y (pixels)
+  cx: 960, // principal point x (pixels)
+  cy: 540, // principal point y (pixels)
+
+  // Optional: Advanced OpenCV parameters
+  balance: 0.0, // 0.0 = all pixels, 1.0 = original FOV
+  fovScale: 1.0, // >1.0 = zoom out, <1.0 = zoom in
+
+  // Optional: Additional features (not part of OpenCV)
+  projection: "rectilinear", // "rectilinear" | "equirectangular"
+  mount: "ceiling", // "ceiling" | "wall" | "desk"
 });
 
-const renderLoop = (timestamp: DOMHighResTimestamp) => {
-  // your render logic
-  const dewarpedVideoFrame: VideoFrame = await dewarper.dewarp(yourVideoFrame);
-  yourYUVPlayer.draw(dewarpedVideoFrame);
+const renderLoop = async (timestamp: DOMHighResTimestamp) => {
+  // Undistort the fisheye frame
+  const undistorted: VideoFrame = await fisheye.undistort(yourVideoFrame);
+  yourYUVPlayer.draw(undistorted);
   requestAnimationFrame(renderLoop);
 };
 ```
 
-## API
-
-### `new Fisheye(options?: FisheyeOptions)`
-
-Creates a new Fisheye dewarper instance.
-
-**Options:**
+## Architecture & Design
 
-- `k1` (number, optional): Fisheye distortion coefficient k1. Typical range: -1.0 to 1.0. Default: `0`.
-- `k2` (number, optional): Fisheye distortion coefficient k2. Default: `0`.
-- `k3` (number, optional): Fisheye distortion coefficient k3. Default: `0`.
-- `k4` (number, optional): Fisheye distortion coefficient k4. Default: `0`.
-- `width` (number, optional): Output frame width. Default: `300`
-- `height` (number, optional): Output frame height. Default: `150`
-- `fov` (number, optional): Field of view in degrees. Default: `180`
-- `centerX` (number, optional): X offset of the lens center (normalized, -1.0 to 1.0). Default: `0`
-- `centerY` (number, optional): Y offset of the lens center (normalized, -1.0 to 1.0). Default: `0`
-- `zoom` (number, optional): Zoom factor. Default: `1.0`
+### Distortion Model: OpenCV Fisheye (Kannala-Brandt)
 
-**Fisheye model (OpenCV fisheye / Kannala–Brandt):**
-We use the same model as OpenCV’s [fisheye module](https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html) (cited there as the “generic camera model” from Kannala & Brandt, 2006). It is a polynomial-in-θ model, not UCM:
+This library uses the **OpenCV fisheye model** (Kannala-Brandt, 2006) for undistortion:
 
 ```
 theta = atan(r)
@@ -90,23 +91,123 @@ theta_d = theta * (1 + k1*theta^2 + k2*theta^4 + k3*theta^6 + k4*theta^8)
 r_d = tan(theta_d)
 ```
 
-### `dewarp(frame: VideoFrame): Promise<VideoFrame>`
+This is the same model as [OpenCV's fisheye module](https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html).
+
+**Important:** OpenCV's `fisheye.undistortImage()` always outputs **rectilinear (perspective) projection only**. It does not provide panoramic or other projection modes.
+
+### Additional Features Beyond OpenCV
+
+For **WebGPU efficiency**, we integrate additional transformations in a **single GPU pass** rather than CPU-side post-processing:
+
+#### 1. **Projection Mode** (`projection`)
+
+- **Purpose**: Controls output coordinate mapping after undistortion
+- **Values**:
+  - `"rectilinear"`: Standard perspective projection (same as OpenCV output)
+  - `"equirectangular"`: Cylindrical/spherical panoramic projection
+- **Implementation**: Applied in GPU shader after undistortion step
+- **OpenCV equivalent**: Would require separate `cv2.warpPerspective()` or custom remapping after `fisheye.undistortImage()`
+
+#### 2. **Mount Orientation** (`mount`)
+
+- **Purpose**: Compensates for camera mounting angle
+- **Values**:
+  - `"ceiling"`: No rotation (0°)
+  - `"wall"`: 90° rotation
+  - `"desk"`: 180° or -90° rotation
+- **Implementation**: Applied as rotation transformation in GPU shader
+- **OpenCV equivalent**: Would require `cv2.getRotationMatrix2D()` + `cv2.warpAffine()` after undistortion
+
+### Why Unified GPU Pipeline?
+
+Traditional OpenCV approach:
+
+```python
+# Step 1: Undistort (GPU/CPU)
+undistorted = cv2.fisheye.undistortImage(img, K, D, Knew)
+
+# Step 2: Projection transform (CPU)
+panorama = cv2.remap(undistorted, custom_map_x, custom_map_y, cv2.INTER_LINEAR)
+
+# Step 3: Rotation (CPU)
+rotated = cv2.warpAffine(panorama, rotation_matrix, size)
+```
+
+**Our approach (single GPU compute shader):**
+
+```typescript
+// All in one GPU pass: undistortion + projection + rotation
+const undistorted = await fisheye.undistort(input);
+```
+
+## API
+
+### `new Fisheye(options?: FisheyeOptions)`
+
+Creates a new Fisheye undistortion instance.
+
+#### OpenCV Fisheye Model Parameters
 
-Dewarps a VideoFrame with fisheye distortion.
+| Parameter  | Type      | Default    | Description                                              |
+| ---------- | --------- | ---------- | -------------------------------------------------------- |
+| `fx`       | `number?` | auto       | Camera matrix K: focal length in x-axis (pixels)         |
+| `fy`       | `number?` | auto       | Camera matrix K: focal length in y-axis (pixels)         |
+| `cx`       | `number?` | `width/2`  | Camera matrix K: principal point x-coordinate (pixels)   |
+| `cy`       | `number?` | `height/2` | Camera matrix K: principal point y-coordinate (pixels)   |
+| `k1`       | `number?` | `0`        | Distortion coefficient k1 (Kannala-Brandt)               |
+| `k2`       | `number?` | `0`        | Distortion coefficient k2 (Kannala-Brandt)               |
+| `k3`       | `number?` | `0`        | Distortion coefficient k3 (Kannala-Brandt)               |
+| `k4`       | `number?` | `0`        | Distortion coefficient k4 (Kannala-Brandt)               |
+| `width`    | `number?` | `300`      | Output image width (OpenCV `new_size.width`)             |
+| `height`   | `number?` | `150`      | Output image height (OpenCV `new_size.height`)           |
+| `balance`  | `number?` | `0.0`      | Balance parameter (0.0 = all pixels, 1.0 = original FOV) |
+| `fovScale` | `number?` | `1.0`      | FOV scale divisor (>1.0 = zoom out, <1.0 = zoom in)      |
+
+**Note:** These parameters exactly match [OpenCV fisheye API](https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html). Use values from `cv2.fisheye.calibrate()` or `cv2.fisheye.estimateNewCameraMatrixForUndistortRectify()`.
+
+#### Additional Features (not part of OpenCV)
+
+| Parameter    | Type                                   | Default         | Description                                                                                             |
+| ------------ | -------------------------------------- | --------------- | ------------------------------------------------------------------------------------------------------- |
+| `projection` | `"rectilinear"` \| `"equirectangular"` | `"rectilinear"` | Output projection mode. `"rectilinear"` = perspective (same as OpenCV), `"equirectangular"` = panoramic |
+| `mount`      | `"ceiling"` \| `"wall"` \| `"desk"`    | `"ceiling"`     | Camera mount orientation for rotation compensation                                                      |
+
+### `undistort(frame: VideoFrame): Promise<VideoFrame>`
+
+Undistorts a VideoFrame with fisheye distortion.
 
 **Parameters:**
 
 - `frame`: Input VideoFrame with fisheye distortion
 
-**Returns:** Promise that resolves to a dewarped VideoFrame
+**Returns:** Promise that resolves to an undistorted VideoFrame
+
+**Differences from OpenCV:**
+
+Unlike OpenCV's `fisheye.undistortImage()` which only outputs rectilinear (perspective) projection, this method performs **all transformations in a single GPU pass** for WebGPU efficiency:
+
+1. **Undistortion** (OpenCV fisheye model)
+2. **Projection** (rectilinear or equirectangular, based on `projection` config)
+3. **Mount rotation** (based on `mount` config)
+
+OpenCV equivalent would require 3 separate operations:
+```python
+# OpenCV: 3 CPU/GPU roundtrips
+undistorted = cv2.fisheye.undistortImage(img, K, D, Knew)
+panorama = cv2.remap(undistorted, map_x, map_y, cv2.INTER_LINEAR)  # if equirectangular
+rotated = cv2.warpAffine(panorama, rotation_matrix, size)  # if ceiling/wall/desk
+
+# fisheye.js: 1 GPU pass - undistortion + projection + rotation
+undistorted = await fisheye.undistort(img)
+```
 
 ### `updateConfig(options: Partial<FisheyeOptions>): void`
 
-Updates the dewarper configuration. You can update any subset of the original options.
+Updates the configuration. You can update any subset of the original options.
 
 ### `destroy(): void`
 
-Cleans up GPU resources. Call this when you're done using the dewarper.
+Cleans up GPU resources. Call this when you're done using the instance.
 
 ## Working with YUV Binary Data
 
@@ -115,21 +216,21 @@ If you receive raw YUV binary data from a camera or server, you can use the `cre
 ```ts
 import { Fisheye, createVideoFrameFromYUV } from "@gyeonghokim/fisheye.js";
 
-const dewarper = new Fisheye({ k1: 0.5, width: 1920, height: 1080 });
+const fisheye = new Fisheye({ k1: 0.5, width: 1920, height: 1080 });
 
 // Example: Receiving NV12 data from a server
 const response = await fetch("/api/camera/frame");
 const yuvBuffer = await response.arrayBuffer();
 
 const frame = createVideoFrameFromYUV(new Uint8Array(yuvBuffer), {
-  format: "NV12",  // YUV format
+  format: "NV12", // YUV format
   width: 1920,
   height: 1080,
-  timestamp: performance.now() * 1000,  // microseconds
+  timestamp: performance.now() * 1000, // microseconds
 });
 
-const dewarpedFrame = await dewarper.dewarp(frame);
-frame.close();  // Don't forget to close the original frame
+const undistorted = await fisheye.undistort(frame);
+frame.close(); // Don't forget to close the original frame
 ```
 
 ### `createVideoFrameFromYUV(data, options)`
@@ -152,13 +253,13 @@ Creates a VideoFrame from YUV binary data.
 
 **Supported YUV Formats:**
 
-| Format | Description | Data Size |
-|--------|-------------|-----------|
-| `I420` | YUV 4:2:0 planar (Y, U, V planes) | width × height × 1.5 |
-| `NV12` | YUV 4:2:0 semi-planar (Y plane, interleaved UV) | width × height × 1.5 |
-| `I420A` | YUV 4:2:0 planar with alpha | width × height × 2.5 |
-| `I422` | YUV 4:2:2 planar | width × height × 2 |
-| `I444` | YUV 4:4:4 planar | width × height × 3 |
+| Format  | Description                                     | Data Size            |
+| ------- | ----------------------------------------------- | -------------------- |
+| `I420`  | YUV 4:2:0 planar (Y, U, V planes)               | width × height × 1.5 |
+| `NV12`  | YUV 4:2:0 semi-planar (Y plane, interleaved UV) | width × height × 1.5 |
+| `I420A` | YUV 4:2:0 planar with alpha                     | width × height × 2.5 |
+| `I422`  | YUV 4:2:2 planar                                | width × height × 2   |
+| `I444`  | YUV 4:4:4 planar                                | width × height × 3   |
 
 ### `calculateYUVDataSize(format, width, height)`
 
@@ -167,7 +268,7 @@ Calculates the expected byte size for YUV data.
 ```ts
 import { calculateYUVDataSize } from "@gyeonghokim/fisheye.js";
 
-const size = calculateYUVDataSize("NV12", 1920, 1080);  // 3110400 bytes
+const size = calculateYUVDataSize("NV12", 1920, 1080); // 3110400 bytes
 ```
 
 ## Development

package/dist/index.d.ts

@@ -121,18 +121,8 @@ export declare interface CreateVideoFrameOptions {
 }
 
 /**
- * Fisheye dewarper using WebGPU via TypeGPU (Pure GPGPU)
- *
- * @example
- * ```ts
- * const dewarper = new Fisheye({
- *   distortion: 0.5,
- *   width: 1920,
- *   height: 1080,
- * });
- *
- * const dewarpedFrame = await dewarper.dewarp(videoFrame);
- * ```
+ * Fisheye undistortion using WebGPU via TypeGPU.
+ * Implements OpenCV fisheye model (Kannala-Brandt).
  */
 export declare class Fisheye {
     private config;
@@ -141,7 +131,7 @@ export declare class Fisheye {
     private inputTexture;
     private outputTexture;
     private bindGroup;
-    private dewarpPipeline;
+    private pipeline;
     private readbackBuffers;
     private readbackIndex;
     private readbackHasData;
@@ -152,115 +142,164 @@ export declare class Fisheye {
     private outputTextureSize;
     private uniformInputWidth;
     private uniformInputHeight;
+    private cachedNewCameraMatrix;
     constructor(options?: FisheyeOptions);
-    /**
-     * Apply default values to options
-     */
     private applyDefaults;
-    /**
-     * Initialize TypeGPU root and resources
-     */
+    /** Undistort a point using Newton's method (OpenCV fisheye inverse). */
+    private undistortPointNormalized;
+    private undistortPixelToNormalized;
+    /** Compute new camera matrix (OpenCV estimateNewCameraMatrixForUndistortRectify). */
+    private computeNewCameraMatrix;
     private initialize;
-    /**
-     * Get uniform data from current configuration
-     */
     private getUniformData;
-    /**
-     * Update uniform buffer with current configuration
-     */
     private updateUniforms;
     private readbackToVideoFrame;
-    /**
-     * Create input texture (TypeGPU; per official docs: sampled + render for .write(image/VideoFrame)).
-     */
     private createInputTexture;
-    /**
-     * Create output storage texture (TypeGPU; type-safe with layout.$)
-     */
     private createOutputTexture;
-    /**
-     * Calculate bytes per row with proper alignment (256-byte alignment for WebGPU)
-     */
     private calculateBytesPerRow;
-    /**
-     * Create or recreate readback buffer for GPU to CPU data transfer
-     */
     private createReadbackBuffer;
-    /**
-     * Dewarp a VideoFrame
-     *
-     * @param frame - Input VideoFrame with fisheye distortion
-     * @returns Dewarped VideoFrame
-     */
-    dewarp(frame: VideoFrame): Promise<VideoFrame>;
-    /**
-     * Update configuration
-     */
+    /** Undistort a VideoFrame. */
+    undistort(frame: VideoFrame): Promise<VideoFrame>;
+    /** Update configuration. */
     updateConfig(options: Partial<FisheyeOptions>): void;
-    /**
-     * Clean up GPU resources
-     */
+    /** Clean up GPU resources. */
     destroy(): void;
 }
 
 /**
- * Internal configuration after applying defaults
+ * Internal configuration with all defaults applied.
+ *
+ * Note: fx, fy, cx, cy are optional because they default to input image dimensions
+ * which are not known until undistort() is called.
  */
-export declare interface FisheyeConfig extends Required<FisheyeOptions> {
+export declare interface FisheyeConfig {
+    /** Camera matrix K: fx. Defaults to input width if not specified. */
+    fx: number | undefined;
+    /** Camera matrix K: fy. Defaults to input width if not specified. */
+    fy: number | undefined;
+    /** Camera matrix K: cx. Defaults to input width / 2 if not specified. */
+    cx: number | undefined;
+    /** Camera matrix K: cy. Defaults to input height / 2 if not specified. */
+    cy: number | undefined;
+    /** Distortion coefficient k1. */
+    k1: number;
+    /** Distortion coefficient k2. */
+    k2: number;
+    /** Distortion coefficient k3. */
+    k3: number;
+    /** Distortion coefficient k4. */
+    k4: number;
+    /** Output width. */
+    width: number;
+    /** Output height. */
+    height: number;
+    /** Balance parameter. */
+    balance: number;
+    /** FOV scale parameter. */
+    fovScale: number;
+    /** Output projection mode. */
+    projection: FisheyeProjection;
+    /** Camera mount position. */
+    mount: FisheyeMount;
 }
 
 /**
- * Options for configuring the Fisheye dewarper
+ * Camera mount position.
+ */
+export declare type FisheyeMount = "ceiling" | "wall" | "desk";
+
+/**
+ * Fisheye undistortion configuration.
+ *
+ * Based on OpenCV fisheye camera model (Kannala-Brandt).
+ * @see https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html
  */
 export declare interface FisheyeOptions {
     /**
-     * Fisheye distortion coefficient k1.
+     * Camera matrix K: focal length x (pixels).
+     * @see https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html
+     */
+    fx?: number;
+    /**
+     * Camera matrix K: focal length y (pixels).
+     * @see https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html
+     */
+    fy?: number;
+    /**
+     * Camera matrix K: principal point x (pixels).
+     * @see https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html
+     */
+    cx?: number;
+    /**
+     * Camera matrix K: principal point y (pixels).
+     * @see https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html
+     */
+    cy?: number;
+    /**
+     * Distortion coefficient k1 (Kannala-Brandt).
+     * @default 0
+     * @see https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html
      */
     k1?: number;
     /**
-     * Fisheye distortion coefficient k2.
+     * Distortion coefficient k2 (Kannala-Brandt).
+     * @default 0
+     * @see https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html
      */
     k2?: number;
     /**
-     * Fisheye distortion coefficient k3.
+     * Distortion coefficient k3 (Kannala-Brandt).
+     * @default 0
+     * @see https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html
      */
     k3?: number;
     /**
-     * Fisheye distortion coefficient k4.
+     * Distortion coefficient k4 (Kannala-Brandt).
+     * @default 0
+     * @see https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html
      */
     k4?: number;
     /**
-     * Canvas width for output
+     * Output image width (pixels).
      * @default 300
+     * @see https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html
      */
     width?: number;
     /**
-     * Canvas height for output
+     * Output image height (pixels).
      * @default 150
+     * @see https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html
      */
     height?: number;
     /**
-     * Field of view in degrees
-     * @default 180
+     * Balance between all pixels vs original FOV (0.0-1.0).
+     * @default 0.0
+     * @see https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html
      */
-    fov?: number;
+    balance?: number;
     /**
-     * X offset of the lens center (normalized, -1.0 to 1.0)
-     * @default 0
+     * FOV scale divisor (>1.0 = zoom out, <1.0 = zoom in).
+     * @default 1.0
+     * @see https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html
      */
-    centerX?: number;
+    fovScale?: number;
     /**
-     * Y offset of the lens center (normalized, -1.0 to 1.0)
-     * @default 0
+     * Output projection mode.
+     * @default "rectilinear"
      */
-    centerY?: number;
+    projection?: FisheyeProjection;
     /**
-     * Zoom factor
-     * @default 1.0
+     * Camera mount position.
+     * @default "ceiling"
      */
-    zoom?: number;
+    mount?: FisheyeMount;
 }
 
+/**
+ * Output projection mode.
+ */
+export declare type FisheyeProjection = "rectilinear" | "equirectangular" | "original";
+
 /**
  * Supported YUV pixel formats for VideoFrame creation
  */