rn-av-binder 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md ADDED
@@ -0,0 +1,130 @@
1
+ # rn-av-binder
2
+
3
+ On-device **image + audio → MP4** composition for React Native — no FFmpeg, no third-party
4
+ media libraries. Uses **AVFoundation** on iOS and **MediaCodec / MediaMuxer / MediaExtractor**
5
+ on Android. Built as an Expo Module for the **New Architecture** (Fabric + JSI).
6
+
7
+ ## Motivation
8
+
9
+ `ffmpeg-kit` was retired in April 2025, leaving React Native apps without a maintained, fully
10
+ on-device way to turn a still image plus an audio track into a playable video. `rn-av-binder`
11
+ is the pure-native replacement for the common "audiogram" use case: it encodes a single static
12
+ image as a 2-frame H.264 video and muxes the original audio in **unmodified** (passthrough, no
13
+ re-encode), so the output duration matches the source audio and the operation is fast and
14
+ lossless on the audio side.
15
+
16
+ ## Installation
17
+
18
+ ```bash
19
+ yarn add rn-av-binder
20
+ npx expo run:ios # or eas build
21
+ ```
22
+
23
+ Add the plugin to your `app.json`, then rebuild the native project:
24
+
25
+ ```json
26
+ {
27
+ "expo": {
28
+ "plugins": ["rn-av-binder"]
29
+ }
30
+ }
31
+ ```
32
+
33
+ > Native linking itself is automatic via Expo Modules autolinking; the plugin entry exists so
34
+ > the package is registered in your config and to allow future native config tweaks.
35
+
36
+ ## API
37
+
38
+ ```ts
39
+ import { createVideo } from "rn-av-binder";
40
+
41
+ createVideo(options, onProgress?): Promise<CreateVideoResult>;
42
+ ```
43
+
44
+ ### `CreateVideoOptions`
45
+
46
+ | Field | Type | Required | Description |
47
+ | ----------- | -------- | -------- | ------------------------------------------------------------------ |
48
+ | `imageUri` | `string` | yes | `file://` URI to a JPEG or PNG image |
49
+ | `audioUri` | `string` | yes | `file://` URI to an AAC, MP3, or M4A audio file |
50
+ | `outputUri` | `string` | no | `file://` URI for the MP4. Defaults to `<cacheDir>/image_audio_<timestamp>.mp4` |
51
+
52
+ ### `CreateVideoResult`
53
+
54
+ | Field | Type | Description |
55
+ | ------------ | -------- | -------------------------------------- |
56
+ | `uri` | `string` | `file://` URI of the created MP4 |
57
+ | `durationMs` | `number` | Duration of the output video, in ms |
58
+
59
+ ### `ProgressEvent`
60
+
61
+ | Field | Type | Description |
62
+ | ---------- | -------- | ------------------------------------ |
63
+ | `progress` | `number` | Encoding progress from `0.0` to `1.0` |
64
+
65
+ The optional `onProgress` callback receives `{ progress }` and fires at `0.0`, `0.5`, and `1.0`.
66
+
67
+ ## Usage
68
+
69
+ ```ts
70
+ import * as ImagePicker from "expo-image-picker";
71
+ import * as DocumentPicker from "expo-document-picker";
72
+ import { createVideo } from "rn-av-binder";
73
+
74
+ async function makeVideo() {
75
+ const image = await ImagePicker.launchImageLibraryAsync({ mediaTypes: "Images" });
76
+ const audio = await DocumentPicker.getDocumentAsync({ type: "audio/*" });
77
+ if (image.canceled || audio.canceled) return;
78
+
79
+ const result = await createVideo(
80
+ { imageUri: image.assets[0].uri, audioUri: audio.assets[0].uri },
81
+ ({ progress }) => console.log(`${Math.round(progress * 100)}%`),
82
+ );
83
+
84
+ console.log(result.uri, `${(result.durationMs / 1000).toFixed(1)}s`);
85
+ }
86
+ ```
87
+
88
+ ## Platform requirements
89
+
90
+ - **iOS** 15.0+
91
+ - **Android** API 26+ (minSdkVersion 26)
92
+ - **Expo** SDK 52+
93
+ - **React Native** 0.76+
94
+ - **New Architecture required** (`newArchEnabled: true`)
95
+
96
+ ## Running the example app
97
+
98
+ The repo includes a standalone example under `example/` that installs the package via
99
+ `"rn-av-binder": "file:.."` — exactly what a consumer does after `yarn add`.
100
+
101
+ ```bash
102
+ # from the package root
103
+ yarn install
104
+ yarn build
105
+
106
+ cd example
107
+ yarn install
108
+ npx expo run:ios # requires Xcode + iOS 15 simulator/device
109
+ npx expo run:android # requires Android Studio + API 26 emulator/device
110
+ ```
111
+
112
+ Pick an image and an audio file, tap **Create Video**, watch the progress bar, and the encoded
113
+ MP4 plays inline on completion.
114
+
115
+ ## Publishing checklist
116
+
117
+ 1. `yarn build` completes with no TypeScript errors; `build/index.js` and `build/index.d.ts` exist.
118
+ 2. `npm pack --dry-run` includes `build/`, `ios/`, `android/`, `expo-module.config.json`,
119
+ `rn-av-binder.podspec`, `app.plugin.js` — and **excludes** `example/`.
120
+ 3. `npm login` (authenticate).
121
+ 4. `npm publish` (runs `prepare` → `yarn build` automatically).
122
+
123
+ ## Known limitations
124
+
125
+ - Single static image only — no slideshow, no transitions.
126
+ - No video-to-video processing; input must be one image + one audio file.
127
+ - Audio is passed through unchanged; the container must support the source audio codec (MP4
128
+ supports AAC/M4A; some exotic codecs may need transcoding, which this library does not do).
129
+ - On the iOS Simulator, camera-roll access can be limited — a physical device may be required
130
+ for picking images.
@@ -0,0 +1,45 @@
1
+ apply plugin: 'com.android.library'
2
+ apply plugin: 'kotlin-android'
3
+
4
+ group = 'expo.modules.imageaudiovideo'
5
+ version = '1.0.0'
6
+
7
+ def expoModulesCorePlugin = new File(project(":expo-modules-core").projectDir.absolutePath, "ExpoModulesCorePlugin.gradle")
8
+ apply from: expoModulesCorePlugin
9
+ applyKotlinExpoModulesCorePlugin()
10
+ useCoreDependencies()
11
+ useExpoPublishing()
12
+
13
+ // If you want to use the managed Android SDK versions from expo-modules-core, set this to true.
14
+ // The Android SDK versions will be bumped from time to time in SDK releases and may introduce
15
+ // breaking changes in your module code. Most of the time, you may like to manage the Android SDK
16
+ // versions yourself.
17
+ def useManagedAndroidSdkVersions = false
18
+ if (useManagedAndroidSdkVersions) {
19
+ useDefaultAndroidSdkVersions()
20
+ } else {
21
+ buildscript {
22
+ // Simple helper that allows the root project to override versions declared by this library.
23
+ ext.safeExtGet = { prop, fallback ->
24
+ rootProject.ext.has(prop) ? rootProject.ext.get(prop) : fallback
25
+ }
26
+ }
27
+ project.android {
28
+ compileSdkVersion safeExtGet("compileSdkVersion", 34)
29
+ defaultConfig {
30
+ minSdkVersion safeExtGet("minSdkVersion", 26)
31
+ targetSdkVersion safeExtGet("targetSdkVersion", 34)
32
+ }
33
+ }
34
+ }
35
+
36
+ android {
37
+ namespace "expo.modules.imageaudiovideo"
38
+ defaultConfig {
39
+ versionCode 1
40
+ versionName "1.0.0"
41
+ }
42
+ lintOptions {
43
+ abortOnError false
44
+ }
45
+ }
@@ -0,0 +1,237 @@
1
+ package expo.modules.imageaudiovideo
2
+
3
+ import android.graphics.Bitmap
4
+ import android.graphics.BitmapFactory
5
+ import android.media.MediaCodec
6
+ import android.media.MediaCodecInfo
7
+ import android.media.MediaExtractor
8
+ import android.media.MediaFormat
9
+ import android.media.MediaMetadataRetriever
10
+ import android.media.MediaMuxer
11
+ import expo.modules.kotlin.modules.Module
12
+ import expo.modules.kotlin.modules.ModuleDefinition
13
+ import kotlinx.coroutines.Dispatchers
14
+ import kotlinx.coroutines.withContext
15
+ import java.io.File
16
+ import java.nio.ByteBuffer
17
+
18
+ class ImageAudioVideoModule : Module() {
19
+ override fun definition() = ModuleDefinition {
20
+ Name("ImageAudioVideo")
21
+ Events("onProgress")
22
+
23
+ AsyncFunction("createVideoAsync") { options: Map<String, String?> ->
24
+ withContext(Dispatchers.IO) {
25
+ createVideo(options)
26
+ }
27
+ }
28
+ }
29
+
30
+ private data class EncodedChunk(val data: ByteArray, val info: MediaCodec.BufferInfo)
31
+
32
+ private fun createVideo(options: Map<String, String?>): Map<String, Any> {
33
+ // ── Step 1 — Resolve paths ────────────────────────────────────────────────
34
+ val imagePath = options["imageUri"]?.takeIf { it.isNotEmpty() }
35
+ ?: throw Exception("ERR_INVALID_IMAGE: imageUri is required")
36
+ val audioPath = options["audioUri"]?.takeIf { it.isNotEmpty() }
37
+ ?: throw Exception("ERR_INVALID_AUDIO: audioUri is required")
38
+ val outputPath = options["outputUri"]?.takeIf { it.isNotEmpty() }
39
+ ?: (appContext.reactContext!!.cacheDir.absolutePath + "/image_audio_${System.currentTimeMillis()}.mp4")
40
+
41
+ File(outputPath).parentFile?.mkdirs()
42
+ File(outputPath).takeIf { it.exists() }?.delete()
43
+ sendEvent("onProgress", mapOf("progress" to 0.0))
44
+
45
+ // ── Step 2 — Load and prepare image ───────────────────────────────────────
46
+ var bitmap = BitmapFactory.decodeFile(imagePath)
47
+ ?: throw Exception("ERR_INVALID_IMAGE: could not decode image at $imagePath")
48
+ if (bitmap.width > 4096 || bitmap.height > 4096) {
49
+ val scale = 4096.0 / maxOf(bitmap.width, bitmap.height)
50
+ bitmap = Bitmap.createScaledBitmap(
51
+ bitmap, (bitmap.width * scale).toInt(), (bitmap.height * scale).toInt(), true
52
+ )
53
+ }
54
+ if (bitmap.config != Bitmap.Config.ARGB_8888) {
55
+ bitmap = bitmap.copy(Bitmap.Config.ARGB_8888, false)
56
+ }
57
+ // Round both dimensions down to the nearest even integer (H.264 requires even dims)
58
+ val width = (bitmap.width / 2) * 2
59
+ val height = (bitmap.height / 2) * 2
60
+ if (width <= 0 || height <= 0) throw Exception("ERR_INVALID_IMAGE: image dimensions are too small")
61
+
62
+ // ── Step 3 — Get audio duration and validate ──────────────────────────────
63
+ val retriever = MediaMetadataRetriever()
64
+ retriever.setDataSource(audioPath)
65
+ val durationMs = retriever.extractMetadata(MediaMetadataRetriever.METADATA_KEY_DURATION)
66
+ ?.toLongOrNull() ?: throw Exception("ERR_INVALID_AUDIO: cannot read audio duration")
67
+ retriever.release()
68
+ if (durationMs <= 0) throw Exception("ERR_INVALID_AUDIO: audio has zero or invalid duration")
69
+ val durationUs = durationMs * 1000L
70
+
71
+ val validator = MediaExtractor()
72
+ validator.setDataSource(audioPath)
73
+ val hasAudio = (0 until validator.trackCount).any {
74
+ validator.getTrackFormat(it).getString(MediaFormat.KEY_MIME)?.startsWith("audio/") == true
75
+ }
76
+ validator.release()
77
+ if (!hasAudio) throw Exception("ERR_NO_AUDIO_TRACK: audio file has no audio track")
78
+
79
+ // ── Step 4 — Configure H.264 encoder ──────────────────────────────────────
80
+ val videoFormat = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, width, height).apply {
81
+ setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Flexible)
82
+ setInteger(MediaFormat.KEY_BIT_RATE, width * height * 2)
83
+ setInteger(MediaFormat.KEY_FRAME_RATE, 1)
84
+ setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1)
85
+ }
86
+ val encoder = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_VIDEO_AVC)
87
+ encoder.configure(videoFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE)
88
+ encoder.start()
89
+
90
+ // ── Step 6 — Encode exactly 2 frames and collect output ───────────────────
91
+ val videoChunks = mutableListOf<EncodedChunk>()
92
+ var outputFormat: MediaFormat? = null
93
+
94
+ fun encodeFrame(yuvData: ByteArray, presentationTimeUs: Long, isLast: Boolean) {
95
+ val inputIndex = encoder.dequeueInputBuffer(10_000L)
96
+ if (inputIndex >= 0) {
97
+ val inputBuffer = encoder.getInputBuffer(inputIndex)!!
98
+ inputBuffer.clear()
99
+ inputBuffer.put(yuvData)
100
+ val flags = if (isLast) MediaCodec.BUFFER_FLAG_END_OF_STREAM else 0
101
+ encoder.queueInputBuffer(inputIndex, 0, yuvData.size, presentationTimeUs, flags)
102
+ }
103
+ // Drain output
104
+ val bufferInfo = MediaCodec.BufferInfo()
105
+ var draining = true
106
+ while (draining) {
107
+ val outputIndex = encoder.dequeueOutputBuffer(bufferInfo, 10_000L)
108
+ when {
109
+ outputIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED -> outputFormat = encoder.outputFormat
110
+ outputIndex >= 0 -> {
111
+ val outputBuffer = encoder.getOutputBuffer(outputIndex)!!
112
+ val chunk = ByteArray(bufferInfo.size)
113
+ outputBuffer.get(chunk)
114
+ videoChunks.add(EncodedChunk(chunk, MediaCodec.BufferInfo().also {
115
+ it.offset = 0
116
+ it.size = bufferInfo.size
117
+ it.presentationTimeUs = bufferInfo.presentationTimeUs
118
+ it.flags = bufferInfo.flags
119
+ }))
120
+ encoder.releaseOutputBuffer(outputIndex, false)
121
+ if (bufferInfo.flags and MediaCodec.BUFFER_FLAG_END_OF_STREAM != 0) draining = false
122
+ }
123
+ else -> draining = false
124
+ }
125
+ }
126
+ }
127
+
128
+ val yuvData = bitmapToYuv420(bitmap, width, height)
129
+ encodeFrame(yuvData, 0L, false)
130
+ encodeFrame(yuvData, durationUs - 1000L, true)
131
+
132
+ encoder.stop()
133
+ encoder.release()
134
+ sendEvent("onProgress", mapOf("progress" to 0.5))
135
+
136
+ // ── Step 7 — Mux video + audio ────────────────────────────────────────────
137
+ val muxer = MediaMuxer(outputPath, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4)
138
+ val videoTrack = muxer.addTrack(
139
+ outputFormat ?: throw Exception("ERR_ENCODE: encoder produced no output format")
140
+ )
141
+
142
+ // Add audio track from extractor
143
+ val extractor = MediaExtractor()
144
+ extractor.setDataSource(audioPath)
145
+ val audioTrackIndex = (0 until extractor.trackCount).first {
146
+ extractor.getTrackFormat(it).getString(MediaFormat.KEY_MIME)?.startsWith("audio/") == true
147
+ }
148
+ extractor.selectTrack(audioTrackIndex)
149
+ val audioTrack = muxer.addTrack(extractor.getTrackFormat(audioTrackIndex))
150
+ muxer.start()
151
+
152
+ // Write video
153
+ val videoBuffer = ByteBuffer.allocate(1024 * 512)
154
+ for (chunk in videoChunks) {
155
+ videoBuffer.clear()
156
+ videoBuffer.put(chunk.data)
157
+ videoBuffer.flip()
158
+ muxer.writeSampleData(videoTrack, videoBuffer, chunk.info)
159
+ }
160
+
161
+ // Write audio (passthrough)
162
+ val audioBuffer = ByteBuffer.allocate(1024 * 512)
163
+ val audioInfo = MediaCodec.BufferInfo()
164
+ while (true) {
165
+ val sampleSize = extractor.readSampleData(audioBuffer, 0)
166
+ if (sampleSize < 0) break
167
+ audioInfo.offset = 0
168
+ audioInfo.size = sampleSize
169
+ audioInfo.presentationTimeUs = extractor.sampleTime
170
+ audioInfo.flags = extractor.sampleFlags
171
+ muxer.writeSampleData(audioTrack, audioBuffer, audioInfo)
172
+ extractor.advance()
173
+ }
174
+
175
+ extractor.release()
176
+ muxer.stop()
177
+ muxer.release()
178
+
179
+ sendEvent("onProgress", mapOf("progress" to 1.0))
180
+ return mapOf("uri" to outputPath, "durationMs" to durationMs)
181
+ }
182
+
183
+ // ── Step 5 — RGB → YUV420 (I420 planar: Y, then U, then V) ──────────────────
184
+ private fun bitmapToYuv420(bitmap: Bitmap, width: Int, height: Int): ByteArray {
185
+ val ySize = width * height
186
+ val cSize = (width / 2) * (height / 2)
187
+ val out = ByteArray(ySize + 2 * cSize)
188
+ val pixels = IntArray(width * height)
189
+ bitmap.getPixels(pixels, 0, width, 0, 0, width, height)
190
+
191
+ // Y plane (full resolution)
192
+ var yIndex = 0
193
+ for (j in 0 until height) {
194
+ for (i in 0 until width) {
195
+ val p = pixels[j * width + i]
196
+ val r = (p shr 16) and 0xFF
197
+ val g = (p shr 8) and 0xFF
198
+ val b = p and 0xFF
199
+ val y = ((66 * r + 129 * g + 25 * b + 128) shr 8) + 16
200
+ out[yIndex++] = clamp(y).toByte()
201
+ }
202
+ }
203
+
204
+ // U and V planes (subsampled — average each 2×2 block)
205
+ var uIndex = ySize
206
+ var vIndex = ySize + cSize
207
+ for (j in 0 until height step 2) {
208
+ for (i in 0 until width step 2) {
209
+ var sumR = 0
210
+ var sumG = 0
211
+ var sumB = 0
212
+ for (dj in 0 until 2) {
213
+ for (di in 0 until 2) {
214
+ val p = pixels[(j + dj) * width + (i + di)]
215
+ sumR += (p shr 16) and 0xFF
216
+ sumG += (p shr 8) and 0xFF
217
+ sumB += p and 0xFF
218
+ }
219
+ }
220
+ val r = sumR / 4
221
+ val g = sumG / 4
222
+ val b = sumB / 4
223
+ val u = ((-38 * r - 74 * g + 112 * b + 128) shr 8) + 128
224
+ val v = ((112 * r - 94 * g - 18 * b + 128) shr 8) + 128
225
+ out[uIndex++] = clamp(u).toByte()
226
+ out[vIndex++] = clamp(v).toByte()
227
+ }
228
+ }
229
+ return out
230
+ }
231
+
232
+ private fun clamp(value: Int): Int = when {
233
+ value < 0 -> 0
234
+ value > 255 -> 255
235
+ else -> value
236
+ }
237
+ }
@@ -0,0 +1,7 @@
1
+ package expo.modules.imageaudiovideo
2
+
3
+ import expo.modules.kotlin.Package
4
+
5
+ class ImageAudioVideoPackage : Package {
6
+ override fun createModules() = listOf(ImageAudioVideoModule())
7
+ }
package/app.plugin.js ADDED
@@ -0,0 +1,8 @@
1
+ const { withPlugins } = require("@expo/config-plugins");
2
+
3
+ /**
4
+ * Expo Config Plugin for rn-av-binder.
5
+ * Pass-through: native linking is handled automatically by Expo Modules autolinking.
6
+ * Exists to satisfy app.json `plugins` array and allow future native config modifications.
7
+ */
8
+ module.exports = (config) => withPlugins(config, []);
@@ -0,0 +1,32 @@
1
+ export interface CreateVideoOptions {
2
+ /** file:// URI to a JPEG or PNG image */
3
+ imageUri: string;
4
+ /** file:// URI to an AAC, MP3, or M4A audio file */
5
+ audioUri: string;
6
+ /**
7
+ * Optional file:// URI for the output MP4.
8
+ * Defaults to <cacheDir>/image_audio_<timestamp>.mp4
9
+ */
10
+ outputUri?: string;
11
+ }
12
+ export interface CreateVideoResult {
13
+ /** file:// URI of the created MP4 */
14
+ uri: string;
15
+ /** Duration of the output video in milliseconds */
16
+ durationMs: number;
17
+ }
18
+ export interface ProgressEvent {
19
+ /** Encoding progress from 0.0 to 1.0 */
20
+ progress: number;
21
+ }
22
+ /**
23
+ * Compose a static image and an audio file into an MP4 video, entirely on-device.
24
+ *
25
+ * @example
26
+ * const result = await createVideo(
27
+ * { imageUri: 'file:///path/to/image.jpg', audioUri: 'file:///path/to/audio.m4a' },
28
+ * ({ progress }) => console.log(`${Math.round(progress * 100)}%`)
29
+ * );
30
+ * console.log(result.uri); // file:///path/to/output.mp4
31
+ */
32
+ export declare function createVideo(options: CreateVideoOptions, onProgress?: (event: ProgressEvent) => void): Promise<CreateVideoResult>;
package/build/index.js ADDED
@@ -0,0 +1,39 @@
1
+ "use strict";
2
+ Object.defineProperty(exports, "__esModule", { value: true });
3
+ exports.createVideo = createVideo;
4
+ const expo_modules_core_1 = require("expo-modules-core");
5
+ const native = (0, expo_modules_core_1.requireNativeModule)("ImageAudioVideo");
6
+ // ─── Public API ───────────────────────────────────────────────────────────────
7
+ /**
8
+ * Compose a static image and an audio file into an MP4 video, entirely on-device.
9
+ *
10
+ * @example
11
+ * const result = await createVideo(
12
+ * { imageUri: 'file:///path/to/image.jpg', audioUri: 'file:///path/to/audio.m4a' },
13
+ * ({ progress }) => console.log(`${Math.round(progress * 100)}%`)
14
+ * );
15
+ * console.log(result.uri); // file:///path/to/output.mp4
16
+ */
17
+ async function createVideo(options, onProgress) {
18
+ const nativeOptions = {
19
+ imageUri: options.imageUri.replace(/^file:\/\//, ""),
20
+ audioUri: options.audioUri.replace(/^file:\/\//, ""),
21
+ outputUri: options.outputUri?.replace(/^file:\/\//, ""),
22
+ };
23
+ let subscription;
24
+ if (onProgress) {
25
+ subscription = native.addListener("onProgress", onProgress);
26
+ }
27
+ try {
28
+ const result = await native.createVideoAsync(nativeOptions);
29
+ return {
30
+ uri: result.uri.startsWith("file://")
31
+ ? result.uri
32
+ : `file://${result.uri}`,
33
+ durationMs: result.durationMs,
34
+ };
35
+ }
36
+ finally {
37
+ subscription?.remove();
38
+ }
39
+ }
@@ -0,0 +1,9 @@
1
+ {
2
+ "platforms": ["apple", "android"],
3
+ "apple": {
4
+ "modules": ["ImageAudioVideoModule"]
5
+ },
6
+ "android": {
7
+ "modules": ["expo.modules.imageaudiovideo.ImageAudioVideoModule"]
8
+ }
9
+ }
@@ -0,0 +1,233 @@
1
+ import ExpoModulesCore
2
+ import AVFoundation
3
+ import UIKit
4
+ import CoreVideo
5
+ import CoreMedia
6
+
7
+ public class ImageAudioVideoModule: Module {
8
+ public func definition() -> ModuleDefinition {
9
+ Name("ImageAudioVideo")
10
+ Events("onProgress")
11
+
12
+ AsyncFunction("createVideoAsync") { (options: [String: String?], promise: Promise) in
13
+ Task {
14
+ do {
15
+ let result = try await self.createVideo(options: options)
16
+ promise.resolve(result)
17
+ } catch {
18
+ promise.reject("ERR_CREATE_VIDEO", error.localizedDescription)
19
+ }
20
+ }
21
+ }
22
+ }
23
+
24
+ // MARK: - Core
25
+
26
+ private func createVideo(options: [String: String?]) async throws -> [String: Any] {
27
+ // ── Step 1 — Resolve paths ────────────────────────────────────────────────
28
+ guard let imagePath = options["imageUri"] ?? nil, !imagePath.isEmpty else {
29
+ throw NSError(domain: "ImageAudioVideo", code: -1,
30
+ userInfo: [NSLocalizedDescriptionKey: "ERR_INVALID_IMAGE: imageUri is required"])
31
+ }
32
+ guard let audioPath = options["audioUri"] ?? nil, !audioPath.isEmpty else {
33
+ throw NSError(domain: "ImageAudioVideo", code: -1,
34
+ userInfo: [NSLocalizedDescriptionKey: "ERR_INVALID_AUDIO: audioUri is required"])
35
+ }
36
+
37
+ let outputPath: String
38
+ if let provided = options["outputUri"] ?? nil, !provided.isEmpty {
39
+ outputPath = provided
40
+ } else {
41
+ let ts = Int(Date().timeIntervalSince1970 * 1000)
42
+ outputPath = NSTemporaryDirectory() + "image_audio_\(ts).mp4"
43
+ }
44
+ let outputURL = URL(fileURLWithPath: outputPath)
45
+
46
+ // Ensure parent directory exists
47
+ let parentDir = outputURL.deletingLastPathComponent()
48
+ if !FileManager.default.fileExists(atPath: parentDir.path) {
49
+ try FileManager.default.createDirectory(at: parentDir, withIntermediateDirectories: true)
50
+ }
51
+ // Delete any pre-existing file (AVAssetWriter fails if it exists)
52
+ try? FileManager.default.removeItem(at: outputURL)
53
+
54
+ sendEvent("onProgress", ["progress": 0.0])
55
+
56
+ // ── Step 2 — Load and prepare image ───────────────────────────────────────
57
+ guard var image = UIImage(contentsOfFile: imagePath) else {
58
+ throw NSError(domain: "ImageAudioVideo", code: -1,
59
+ userInfo: [NSLocalizedDescriptionKey: "ERR_INVALID_IMAGE: could not decode image at \(imagePath)"])
60
+ }
61
+ guard image.cgImage != nil else {
62
+ throw NSError(domain: "ImageAudioVideo", code: -1,
63
+ userInfo: [NSLocalizedDescriptionKey: "ERR_INVALID_IMAGE: image has no backing bitmap"])
64
+ }
65
+
66
+ var rawWidth = image.cgImage!.width
67
+ var rawHeight = image.cgImage!.height
68
+
69
+ // Clamp the larger dimension to 4096
70
+ let maxDim = 4096
71
+ if rawWidth > maxDim || rawHeight > maxDim {
72
+ let scale = Double(maxDim) / Double(max(rawWidth, rawHeight))
73
+ let scaledSize = CGSize(width: CGFloat(Double(rawWidth) * scale),
74
+ height: CGFloat(Double(rawHeight) * scale))
75
+ let renderer = UIGraphicsImageRenderer(size: scaledSize)
76
+ image = renderer.image { _ in
77
+ image.draw(in: CGRect(origin: .zero, size: scaledSize))
78
+ }
79
+ rawWidth = image.cgImage!.width
80
+ rawHeight = image.cgImage!.height
81
+ }
82
+
83
+ // Round both dimensions down to the nearest even integer (H.264 requires even dims)
84
+ let width = (rawWidth / 2) * 2
85
+ let height = (rawHeight / 2) * 2
86
+ guard width > 0 && height > 0 else {
87
+ throw NSError(domain: "ImageAudioVideo", code: -1,
88
+ userInfo: [NSLocalizedDescriptionKey: "ERR_INVALID_IMAGE: image dimensions are too small"])
89
+ }
90
+
91
+ // ── Step 3 — Load audio and get duration ──────────────────────────────────
92
+ let audioURL = URL(fileURLWithPath: audioPath)
93
+ let audioAsset = AVURLAsset(url: audioURL)
94
+ let duration = try await audioAsset.load(.duration)
95
+ if duration == .invalid || CMTimeGetSeconds(duration) <= 0 {
96
+ throw NSError(domain: "ImageAudioVideo", code: -1,
97
+ userInfo: [NSLocalizedDescriptionKey: "ERR_INVALID_AUDIO: audio has zero or invalid duration"])
98
+ }
99
+ let tracks = try await audioAsset.loadTracks(withMediaType: .audio)
100
+ if tracks.isEmpty {
101
+ throw NSError(domain: "ImageAudioVideo", code: -1,
102
+ userInfo: [NSLocalizedDescriptionKey: "ERR_NO_AUDIO_TRACK: audio file has no audio track"])
103
+ }
104
+
105
+ // ── Step 4 — Set up AVAssetWriter ─────────────────────────────────────────
106
+ let writer = try AVAssetWriter(outputURL: outputURL, fileType: .mp4)
107
+
108
+ // ── Step 5 — Video input (H.264, static frames) ───────────────────────────
109
+ let videoSettings: [String: Any] = [
110
+ AVVideoCodecKey: AVVideoCodecType.h264,
111
+ AVVideoWidthKey: NSNumber(value: width),
112
+ AVVideoHeightKey: NSNumber(value: height),
113
+ AVVideoCompressionPropertiesKey: [
114
+ AVVideoAverageBitRateKey: width * height * 2,
115
+ AVVideoProfileLevelKey: AVVideoProfileLevelH264HighAutoLevel,
116
+ ] as [String: Any],
117
+ ]
118
+ let videoInput = AVAssetWriterInput(mediaType: .video, outputSettings: videoSettings)
119
+ videoInput.expectsMediaDataInRealTime = false
120
+
121
+ let adaptor = AVAssetWriterInputPixelBufferAdaptor(
122
+ assetWriterInput: videoInput,
123
+ sourcePixelBufferAttributes: [
124
+ kCVPixelBufferPixelFormatTypeKey as String: Int(kCVPixelFormatType_32ARGB),
125
+ kCVPixelBufferWidthKey as String: width,
126
+ kCVPixelBufferHeightKey as String: height,
127
+ ]
128
+ )
129
+ writer.add(videoInput)
130
+
131
+ // ── Step 6 — Audio input (passthrough, no re-encoding) ────────────────────
132
+ let audioInput = AVAssetWriterInput(mediaType: .audio, outputSettings: nil)
133
+ audioInput.expectsMediaDataInRealTime = false
134
+ writer.add(audioInput)
135
+
136
+ // ── Step 8 — Write exactly 2 video frames ─────────────────────────────────
137
+ guard writer.startWriting() else {
138
+ throw writer.error ?? NSError(domain: "ImageAudioVideo", code: -1,
139
+ userInfo: [NSLocalizedDescriptionKey: "AVAssetWriter failed to start writing"])
140
+ }
141
+ writer.startSession(atSourceTime: .zero)
142
+
143
+ let pixelBuffer = try imageToPixelBuffer(image, width: width, height: height)
144
+
145
+ // Frame 1: presentation time = 0
146
+ while !videoInput.isReadyForMoreMediaData { await Task.yield() }
147
+ adaptor.append(pixelBuffer, withPresentationTime: .zero)
148
+
149
+ // Frame 2: presentation time = duration - 1 tick (avoids single-frame player quirks)
150
+ let lastFrameTime = CMTimeSubtract(duration, CMTime(value: 1, timescale: 600))
151
+ while !videoInput.isReadyForMoreMediaData { await Task.yield() }
152
+ adaptor.append(pixelBuffer, withPresentationTime: lastFrameTime)
153
+
154
+ videoInput.markAsFinished()
155
+ sendEvent("onProgress", ["progress": 0.5])
156
+
157
+ // ── Step 9 — Pipe audio via AVAssetReader (passthrough) ───────────────────
158
+ let reader = try AVAssetReader(asset: audioAsset)
159
+ let audioTrack = tracks.first!
160
+ let readerOutput = AVAssetReaderTrackOutput(track: audioTrack, outputSettings: nil) // nil = compressed passthrough
161
+ reader.add(readerOutput)
162
+ reader.startReading()
163
+
164
+ while let sampleBuffer = readerOutput.copyNextSampleBuffer() {
165
+ while !audioInput.isReadyForMoreMediaData {
166
+ await Task.yield()
167
+ }
168
+ audioInput.append(sampleBuffer)
169
+ }
170
+ audioInput.markAsFinished()
171
+
172
+ // ── Step 10 — Finish and return ───────────────────────────────────────────
173
+ await writer.finishWriting()
174
+ guard writer.status == .completed else {
175
+ throw writer.error ?? NSError(domain: "ImageAudioVideo", code: -1,
176
+ userInfo: [NSLocalizedDescriptionKey: "AVAssetWriter failed with status \(writer.status.rawValue)"])
177
+ }
178
+ sendEvent("onProgress", ["progress": 1.0])
179
+ return [
180
+ "uri": outputURL.path,
181
+ "durationMs": CMTimeGetSeconds(duration) * 1000.0,
182
+ ]
183
+ }
184
+
185
+ // MARK: - Step 7 — Image → CVPixelBuffer helper
186
+
187
+ private func imageToPixelBuffer(_ image: UIImage, width: Int, height: Int) throws -> CVPixelBuffer {
188
+ let attrs: [String: Any] = [
189
+ kCVPixelBufferCGImageCompatibilityKey as String: true,
190
+ kCVPixelBufferCGBitmapContextCompatibilityKey as String: true,
191
+ ]
192
+ var pixelBuffer: CVPixelBuffer?
193
+ let status = CVPixelBufferCreate(
194
+ kCFAllocatorDefault, width, height,
195
+ kCVPixelFormatType_32ARGB, attrs as CFDictionary, &pixelBuffer
196
+ )
197
+ guard status == kCVReturnSuccess, let buffer = pixelBuffer else {
198
+ throw NSError(domain: "ImageAudioVideo", code: -1,
199
+ userInfo: [NSLocalizedDescriptionKey: "ERR_PIXEL_BUFFER: could not allocate pixel buffer"])
200
+ }
201
+
202
+ CVPixelBufferLockBaseAddress(buffer, [])
203
+ defer { CVPixelBufferUnlockBaseAddress(buffer, []) }
204
+
205
+ guard let baseAddress = CVPixelBufferGetBaseAddress(buffer) else {
206
+ throw NSError(domain: "ImageAudioVideo", code: -1,
207
+ userInfo: [NSLocalizedDescriptionKey: "ERR_PIXEL_BUFFER: could not lock pixel buffer"])
208
+ }
209
+
210
+ let colorSpace = CGColorSpaceCreateDeviceRGB()
211
+ let bitmapInfo = CGImageAlphaInfo.noneSkipFirst.rawValue | CGBitmapInfo.byteOrder32Big.rawValue
212
+ guard let context = CGContext(
213
+ data: baseAddress,
214
+ width: width,
215
+ height: height,
216
+ bitsPerComponent: 8,
217
+ bytesPerRow: CVPixelBufferGetBytesPerRow(buffer),
218
+ space: colorSpace,
219
+ bitmapInfo: bitmapInfo
220
+ ) else {
221
+ throw NSError(domain: "ImageAudioVideo", code: -1,
222
+ userInfo: [NSLocalizedDescriptionKey: "ERR_PIXEL_BUFFER: could not create CGContext"])
223
+ }
224
+
225
+ guard let cgImage = image.cgImage else {
226
+ throw NSError(domain: "ImageAudioVideo", code: -1,
227
+ userInfo: [NSLocalizedDescriptionKey: "ERR_INVALID_IMAGE: missing cgImage"])
228
+ }
229
+ context.draw(cgImage, in: CGRect(x: 0, y: 0, width: width, height: height))
230
+
231
+ return buffer
232
+ }
233
+ }
package/package.json ADDED
@@ -0,0 +1,40 @@
1
+ {
2
+ "name": "rn-av-binder",
3
+ "version": "1.0.0",
4
+ "description": "On-device image + audio → MP4 composition for React Native. AVFoundation (iOS) and MediaCodec/MediaMuxer (Android). New Architecture compatible.",
5
+ "main": "build/index.js",
6
+ "types": "build/index.d.ts",
7
+ "author": "mustneerahmadr7@gmail.com",
8
+ "files": [
9
+ "build/",
10
+ "ios/",
11
+ "android/",
12
+ "expo-module.config.json",
13
+ "rn-av-binder.podspec",
14
+ "app.plugin.js"
15
+ ],
16
+ "scripts": {
17
+ "build": "tsc -p tsconfig.build.json",
18
+ "prepare": "npm run build",
19
+ "typecheck": "tsc --noEmit"
20
+ },
21
+ "keywords": [
22
+ "react-native",
23
+ "expo",
24
+ "video",
25
+ "audio",
26
+ "mp4",
27
+ "avfoundation",
28
+ "mediacodec"
29
+ ],
30
+ "license": "MIT",
31
+ "peerDependencies": {
32
+ "expo": ">=52.0.0",
33
+ "expo-modules-core": "*",
34
+ "react-native": ">=0.76.0"
35
+ },
36
+ "devDependencies": {
37
+ "expo-modules-core": "^2.0.0",
38
+ "typescript": "^5.3.0"
39
+ }
40
+ }
@@ -0,0 +1,17 @@
1
+ require 'json'
2
+ package = JSON.parse(File.read(File.join(__dir__, 'package.json')))
3
+
4
+ Pod::Spec.new do |s|
5
+ s.name = 'rn-av-binder'
6
+ s.version = package['version']
7
+ s.summary = package['description']
8
+ s.license = { :type => 'MIT' }
9
+ s.homepage = 'https://github.com/yourname/rn-av-binder'
10
+ s.authors = { 'Author' => 'mustneerahmadr7@gmail.com' }
11
+ s.source = { :git => 'https://github.com/yourname/rn-av-binder.git',
12
+ :tag => s.version.to_s }
13
+ s.platforms = { :ios => '15.0' }
14
+ s.swift_version = '5.9'
15
+ s.source_files = 'ios/**/*.swift'
16
+ s.dependency 'ExpoModulesCore'
17
+ end