whisper.rn 0.3.0-rc.5 → 0.3.0-rc.7

package/README.md

@@ -8,9 +8,12 @@ React Native binding of [whisper.cpp](https://github.com/ggerganov/whisper.cpp).
 
 [whisper.cpp](https://github.com/ggerganov/whisper.cpp): High-performance inference of [OpenAI's Whisper](https://github.com/openai/whisper) automatic speech recognition (ASR) model
 
-<img src="https://user-images.githubusercontent.com/3001525/225511664-8b2ba3ec-864d-4f55-bcb0-447aef168a32.jpeg" width="500" />
+## Screenshots
 
-> Run example with release mode on iPhone 13 Pro Max
+| <img src="https://github.com/mybigday/whisper.rn/assets/3001525/2fea7b2d-c911-44fb-9afc-8efc7b594446" width="300" /> | <img src="https://github.com/mybigday/whisper.rn/assets/3001525/a5005a6c-44f7-4db9-95e8-0fd951a2e147" width="300" /> |
+| :------------------------------------------: | :------------------------------------------: |
+| iOS: Tested on iPhone 13 Pro Max | Android: Tested on Pixel 6 |
+| (tiny.en, Core ML enabled, release mode + archive) | (tiny.en, armv8.2-a+fp16, release mode) |
 
 ## Installation
 
@@ -48,7 +51,6 @@ import { initWhisper } from 'whisper.rn'
 
 const whisperContext = await initWhisper({
   filePath: 'file://.../ggml-tiny.en.bin',
-  isBundleAsset: false, // Set to true if you want to load the model from bundle resources, the filePath will be like `ggml-tiny.en.bin`
 })
 
 const sampleFilePath = 'file://.../sample.wav'
@@ -81,6 +83,44 @@ In Android, you may need to request the microphone permission by [`PermissionAnd
 
 Please visit the [Documentation](docs/) for more details.
 
+## Usage with assets
+
+You can also use the model file / audio file from assets:
+
+```js
+import { initWhisper } from 'whisper.rn'
+
+const whisperContext = await initWhisper({
+  filePath: require('../assets/ggml-tiny.en.bin'),
+})
+
+const { stop, promise } =
+  whisperContext.transcribe(require('../assets/sample.wav'), options)
+
+// ...
+```
+
+This requires editing the `metro.config.js` to support assets:
+
+```js
+// ...
+const defaultAssetExts = require('metro-config/src/defaults/defaults').assetExts
+
+module.exports = {
+  // ...
+  resolver: {
+    // ...
+    assetExts: [
+      ...defaultAssetExts,
+      'bin', // whisper.rn: ggml model binary
+      'mil', // whisper.rn: CoreML model asset
+    ]
+  },
+}
+```
+
+Please note that it will significantly increase the size of the app in release mode.
+
 ## Core ML support
 
 __*Platform: iOS 15.0+, tvOS 15.0+*__
@@ -89,33 +129,50 @@ To use Core ML on iOS, you will need to have the Core ML model files.
 
 The `.mlmodelc` model files is load depend on the ggml model file path. For example, if your ggml model path is `ggml-tiny.en.bin`, the Core ML model path will be `ggml-tiny.en-encoder.mlmodelc`. Please note that the ggml model is still needed as decoder or encoder fallback.
 
-Currently there is no official way to get the Core ML models by URL, you will need to convert Core ML models by yourself. Please see [Core ML Support](https://github.com/ggerganov/whisper.cpp#core-ml-support) of whisper.cpp for more details.
+The Core ML models are hosted here: https://huggingface.co/ggerganov/whisper.cpp/tree/main
+
+If you want to download model at runtime, during the host file is archive, you will need to unzip the file to get the `.mlmodelc` directory, you can use library like [react-native-zip-archive](https://github.com/mockingbot/react-native-zip-archive), or host those individual files to download yourself.
 
-During the `.mlmodelc` is a directory, you will need to download 5 files:
+The `.mlmodelc` is a directory, usually it includes 5 files (3 required):
 
 ```json5
 [
   'model.mil',
-  'metadata.json',
   'coremldata.bin',
   'weights/weight.bin',
-  'analytics/coremldata.bin',
+  // Not required:
+  // 'metadata.json', 'analytics/coremldata.bin',
 ]
 ```
 
-Or just add them to your app's bundle resources, like the example app does, but this would increase the app size significantly.
+Or just use `require` to bundle that in your app, like the example app does, but this would increase the app size significantly.
+
+```js
+const whisperContext = await initWhisper({
+  filePath: require('../assets/ggml-tiny.en.bin')
+  coreMLModelAsset:
+    Platform.OS === 'ios'
+      ? {
+          filename: 'ggml-tiny.en-encoder.mlmodelc',
+          assets: [
+            require('../assets/ggml-tiny.en-encoder.mlmodelc/weights/weight.bin'),
+            require('../assets/ggml-tiny.en-encoder.mlmodelc/model.mil'),
+            require('../assets/ggml-tiny.en-encoder.mlmodelc/coremldata.bin'),
+          ],
+        }
+      : undefined,
+})
+```
+
+In real world, we recommended to split the asset imports into another platform specific file (e.g. `context-opts.ios.js`) to avoid these unused files in the bundle for Android.
 
 ## Run with example
 
-The example app is using [react-native-fs](https://github.com/itinance/react-native-fs) to download the model file and audio file.
+The example app provide a simple UI for testing the functions.
 
-Model: `base.en` in https://huggingface.co/datasets/ggerganov/whisper.cpp  
+Used Whisper model: `tiny.en` in https://huggingface.co/ggerganov/whisper.cpp  
 Sample file: `jfk.wav` in https://github.com/ggerganov/whisper.cpp/tree/master/samples
 
-For test better performance on transcribe, you can run the app in Release mode.
-  - iOS: `yarn example ios --configuration Release`
-  - Android: `yarn example android --mode release`
-
 Please follow the [Development Workflow section of contributing guide](./CONTRIBUTING.md#development-workflow) to run the example app.
 
 ## Mock `whisper.rn`
@@ -130,6 +187,10 @@ jest.mock('whisper.rn', () => require('whisper.rn/jest/mock'))
 
 See the [contributing guide](CONTRIBUTING.md) to learn how to contribute to the repository and the development workflow.
 
+## Troubleshooting
+
+See the [troubleshooting](TROUBLESHOOTING.md) if you encounter any problem while using `whisper.rn`.
+
 ## License
 
 MIT

package/android/src/main/java/com/rnwhisper/Downloader.java

@@ -0,0 +1,83 @@
+package com.rnwhisper;
+
+import android.content.Context;
+
+import java.io.BufferedInputStream;
+import java.io.FileOutputStream;
+import java.io.File;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URL;
+import java.net.URLConnection;
+
+/**
+ * NOTE: This is simple downloader,
+ * the main purpose is supported load assets on RN Debug mode,
+ * so it's a very crude implementation.
+ * 
+ * If you want to use file download in production to load model / audio files,
+ * I would recommend using react-native-fs or expo-file-system to manage the files.
+ */
+public class Downloader {
+  private static Context context;
+
+  public Downloader(Context context) {
+    this.context = context;
+  }
+
+  private String getDir() {
+    String dir = context.getCacheDir().getAbsolutePath() + "/rnwhisper_debug_assets/";
+    File file = new File(dir);
+    if (!file.exists()) {
+      file.mkdirs();
+    }
+    return dir;
+  }
+
+  private boolean fileExists(String filename) {
+    File file = new File(getDir() + filename);
+    return file.exists();
+  }
+
+  public String downloadFile(String urlPath) throws Exception {
+    String filename = urlPath.substring(urlPath.lastIndexOf('/') + 1);
+    if (filename.contains("?")) {
+      filename = filename.substring(0, filename.indexOf("?"));
+    }
+    String filepath = getDir() + filename;
+    if (fileExists(filename)) {
+      return filepath;
+    }
+    try {
+      URL url = new URL(urlPath);
+      URLConnection connection = url.openConnection();
+      connection.connect();
+      InputStream input = new BufferedInputStream(url.openStream());
+      OutputStream output = new FileOutputStream(filepath);
+      byte data[] = new byte[1024];
+      int count;
+      while ((count = input.read(data)) != -1) {
+        output.write(data, 0, count);
+      }
+      output.flush();
+      output.close();
+      input.close();
+    } catch (Exception e) {
+      throw e;
+    }
+    return filepath;
+  }
+
+  private void deleteFile(File fileOrDir) {
+    if (fileOrDir.isDirectory()) {
+      for (File child : fileOrDir.listFiles()) {
+        deleteFile(child);
+      }
+    }
+    fileOrDir.delete();
+  }
+
+  public void clearCache() {
+    deleteFile(new File(getDir()));
+  }
+}

package/android/src/main/java/com/rnwhisper/WhisperContext.java

@@ -26,6 +26,7 @@ import java.io.File;
 import java.io.FileInputStream;
 import java.io.IOException;
 import java.io.InputStream;
+import java.io.PushbackInputStream;
 import java.nio.ByteBuffer;
 import java.nio.ByteOrder;
 import java.nio.ShortBuffer;
@@ -281,10 +282,10 @@ public class WhisperContext {
     eventEmitter.emit(eventName, event);
   }
 
-  public WritableMap transcribeFile(int jobId, String filePath, ReadableMap options) throws IOException, Exception {
+  public WritableMap transcribeInputStream(int jobId, InputStream inputStream, ReadableMap options) throws IOException, Exception {
     this.jobId = jobId;
     isTranscribing = true;
-    float[] audioData = decodeWaveFile(new File(filePath));
+    float[] audioData = decodeWaveFile(inputStream);
     int code = full(jobId, options, audioData, audioData.length);
     isTranscribing = false;
     this.jobId = -1;
@@ -383,14 +384,12 @@ public class WhisperContext {
     freeContext(context);
   }
 
-  public static float[] decodeWaveFile(File file) throws IOException {
+  public static float[] decodeWaveFile(InputStream inputStream) throws IOException {
     ByteArrayOutputStream baos = new ByteArrayOutputStream();
-    try (InputStream inputStream = new FileInputStream(file)) {
-      byte[] buffer = new byte[1024];
-      int bytesRead;
-      while ((bytesRead = inputStream.read(buffer)) != -1) {
-        baos.write(buffer, 0, bytesRead);
-      }
+    byte[] buffer = new byte[1024];
+    int bytesRead;
+    while ((bytesRead = inputStream.read(buffer)) != -1) {
+      baos.write(buffer, 0, bytesRead);
     }
     ByteBuffer byteBuffer = ByteBuffer.wrap(baos.toByteArray());
     byteBuffer.order(ByteOrder.LITTLE_ENDIAN);
@@ -472,6 +471,7 @@ public class WhisperContext {
 
   protected static native long initContext(String modelPath);
   protected static native long initContextWithAsset(AssetManager assetManager, String modelPath);
+  protected static native long initContextWithInputStream(PushbackInputStream inputStream);
   protected static native int fullTranscribe(
     int job_id,
     long context,

package/android/src/main/jni/whisper/Whisper.mk

@@ -1,15 +1,18 @@
 WHISPER_LIB_DIR := $(LOCAL_PATH)/../../../../../cpp
 LOCAL_LDLIBS    := -landroid -llog
 
+# NOTE: If you want to debug the native code, you can uncomment ifneq and endif
+# ifneq ($(APP_OPTIM),debug)
+
 # Make the final output library smaller by only keeping the symbols referenced from the app.
-ifneq ($(APP_OPTIM),debug)
-    LOCAL_CFLAGS += -O3 -DNDEBUG
-    LOCAL_CFLAGS += -fvisibility=hidden -fvisibility-inlines-hidden
-    LOCAL_CFLAGS += -ffunction-sections -fdata-sections
-    LOCAL_LDFLAGS += -Wl,--gc-sections
-    LOCAL_LDFLAGS += -Wl,--exclude-libs,ALL
-    LOCAL_LDFLAGS += -flto
-endif
+LOCAL_CFLAGS += -O3 -DNDEBUG
+LOCAL_CFLAGS += -fvisibility=hidden -fvisibility-inlines-hidden
+LOCAL_CFLAGS += -ffunction-sections -fdata-sections
+LOCAL_LDFLAGS += -Wl,--gc-sections
+LOCAL_LDFLAGS += -Wl,--exclude-libs,ALL
+LOCAL_LDFLAGS += -flto
+
+# endif
 
 LOCAL_CFLAGS    += -DSTDC_HEADERS -std=c11 -I $(WHISPER_LIB_DIR)
 LOCAL_CPPFLAGS  += -std=c++11 -I $(WHISPER_LIB_DIR)

package/android/src/main/jni/whisper/jni.cpp

@@ -20,6 +20,97 @@ static inline int min(int a, int b) {
     return (a < b) ? a : b;
 }
 
+// Load model from input stream (used for drawable / raw resources)
+struct input_stream_context {
+    JNIEnv *env;
+    jobject input_stream;
+};
+
+static size_t input_stream_read(void *ctx, void *output, size_t read_size) {
+    input_stream_context *context = (input_stream_context *)ctx;
+    JNIEnv *env = context->env;
+    jobject input_stream = context->input_stream;
+    jclass input_stream_class = env->GetObjectClass(input_stream);
+
+    jbyteArray buffer = env->NewByteArray(read_size);
+    jint bytes_read = env->CallIntMethod(
+        input_stream,
+        env->GetMethodID(input_stream_class, "read", "([B)I"),
+        buffer
+    );
+
+    if (bytes_read > 0) {
+        env->GetByteArrayRegion(buffer, 0, bytes_read, (jbyte *) output);
+    }
+
+    env->DeleteLocalRef(buffer);
+
+    return bytes_read;
+}
+
+static bool input_stream_is_eof(void *ctx) {
+    input_stream_context *context = (input_stream_context *)ctx;
+    JNIEnv *env = context->env;
+    jobject input_stream = context->input_stream;
+
+    jclass input_stream_class = env->GetObjectClass(input_stream);
+
+    jbyteArray buffer = env->NewByteArray(1);
+    jint bytes_read = env->CallIntMethod(
+        input_stream,
+        env->GetMethodID(input_stream_class, "read", "([B)I"),
+        buffer
+    );
+
+    bool is_eof = (bytes_read == -1);
+    if (!is_eof) {
+        // If we successfully read a byte, "unread" it by pushing it back into the stream.
+        env->CallVoidMethod(
+            input_stream,
+            env->GetMethodID(input_stream_class, "unread", "([BII)V"),
+            buffer,
+            0,
+            1
+        );
+    }
+
+    env->DeleteLocalRef(buffer);
+
+    return is_eof;
+}
+
+static void input_stream_close(void *ctx) {
+    input_stream_context *context = (input_stream_context *)ctx;
+    JNIEnv *env = context->env;
+    jobject input_stream = context->input_stream;
+    jclass input_stream_class = env->GetObjectClass(input_stream);
+    
+    env->CallVoidMethod(
+        input_stream,
+        env->GetMethodID(input_stream_class, "close", "()V")
+    );
+
+    env->DeleteGlobalRef(input_stream);
+}
+
+static struct whisper_context *whisper_init_from_input_stream(
+    JNIEnv *env,
+    jobject input_stream // PushbackInputStream
+) {
+    input_stream_context *context = new input_stream_context;
+    context->env = env;
+    context->input_stream = env->NewGlobalRef(input_stream);
+
+    whisper_model_loader loader = {
+        .context = context,
+        .read = &input_stream_read,
+        .eof = &input_stream_is_eof,
+        .close = &input_stream_close
+    };
+    return whisper_init(&loader);
+}
+
+// Load model from asset
 static size_t asset_read(void *ctx, void *output, size_t read_size) {
     return AAsset_read((AAsset *) ctx, output, read_size);
 }
@@ -81,6 +172,17 @@ Java_com_rnwhisper_WhisperContext_initContextWithAsset(
     return reinterpret_cast<jlong>(context);
 }
 
+JNIEXPORT jlong JNICALL
+Java_com_rnwhisper_WhisperContext_initContextWithInputStream(
+    JNIEnv *env,
+    jobject thiz,
+    jobject input_stream
+) {
+    UNUSED(thiz);
+    struct whisper_context *context = nullptr;
+    context = whisper_init_from_input_stream(env, input_stream);
+    return reinterpret_cast<jlong>(context);
+}
 
 JNIEXPORT jint JNICALL
 Java_com_rnwhisper_WhisperContext_fullTranscribe(

package/android/src/newarch/java/com/rnwhisper/RNWhisperModule.java

@@ -17,17 +17,22 @@ import com.facebook.react.module.annotations.ReactModule;
 
 import java.util.HashMap;
 import java.util.Random;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.PushbackInputStream;
 
 @ReactModule(name = RNWhisperModule.NAME)
 public class RNWhisperModule extends NativeRNWhisperSpec implements LifecycleEventListener {
   public static final String NAME = "RNWhisper";
 
   private ReactApplicationContext reactContext;
+  private Downloader downloader;
 
   public RNWhisperModule(ReactApplicationContext reactContext) {
     super(reactContext);
     reactContext.addLifecycleEventListener(this);
     this.reactContext = reactContext;
+    this.downloader = new Downloader(reactContext);
   }
 
   @Override
@@ -49,19 +54,48 @@ public class RNWhisperModule extends NativeRNWhisperSpec implements LifecycleEve
 
   private HashMap<Integer, WhisperContext> contexts = new HashMap<>();
 
+  private int getResourceIdentifier(String filePath) {
+    int identifier = reactContext.getResources().getIdentifier(
+      filePath,
+      "drawable",
+      reactContext.getPackageName()
+    );
+    if (identifier == 0) {
+      identifier = reactContext.getResources().getIdentifier(
+        filePath,
+        "raw",
+        reactContext.getPackageName()
+      );
+    }
+    return identifier;
+  }
+
   @ReactMethod
-  public void initContext(final String modelPath, final boolean isBundleAsset, final Promise promise) {
+  public void initContext(final ReadableMap options, final Promise promise) {
     new AsyncTask<Void, Void, Integer>() {
       private Exception exception;
 
       @Override
       protected Integer doInBackground(Void... voids) {
         try {
+          String modelPath = options.getString("filePath");
+          boolean isBundleAsset = options.getBoolean("isBundleAsset");
+
+          String modelFilePath = modelPath;
+          if (!isBundleAsset && (modelPath.startsWith("http://") || modelPath.startsWith("https://"))) {
+            modelFilePath = downloader.downloadFile(modelPath);
+          }
+
           long context;
-          if (isBundleAsset) {
-            context = WhisperContext.initContextWithAsset(reactContext.getAssets(), modelPath);
+          int resId = getResourceIdentifier(modelFilePath);
+          if (resId > 0) {
+            context = WhisperContext.initContextWithInputStream(
+              new PushbackInputStream(reactContext.getResources().openRawResource(resId))
+            );
+          } else if (isBundleAsset) {
+            context = WhisperContext.initContextWithAsset(reactContext.getAssets(), modelFilePath);
           } else {
-            context = WhisperContext.initContext(modelPath);
+            context = WhisperContext.initContext(modelFilePath);
           }
           if (context == 0) {
             throw new Exception("Failed to initialize context");
@@ -108,7 +142,26 @@ public class RNWhisperModule extends NativeRNWhisperSpec implements LifecycleEve
       @Override
       protected WritableMap doInBackground(Void... voids) {
         try {
-          return context.transcribeFile((int) jobId, filePath, options);
+          String waveFilePath = filePath;
+
+          if (filePath.startsWith("http://") || filePath.startsWith("https://")) {
+            waveFilePath = downloader.downloadFile(filePath);
+          }
+
+          int resId = getResourceIdentifier(waveFilePath);
+          if (resId > 0) {
+            return context.transcribeInputStream(
+              (int) jobId,
+              reactContext.getResources().openRawResource(resId),
+              options
+            );
+          }
+
+          return context.transcribeInputStream(
+            (int) jobId,
+            new FileInputStream(new File(waveFilePath)),
+            options
+          );
         } catch (Exception e) {
           exception = e;
           return null;
@@ -228,5 +281,6 @@ public class RNWhisperModule extends NativeRNWhisperSpec implements LifecycleEve
       context.release();
     }
     contexts.clear();
+    downloader.clearCache();
   }
 }

package/android/src/oldarch/java/com/rnwhisper/RNWhisperModule.java

@@ -18,17 +18,22 @@ import com.facebook.react.module.annotations.ReactModule;
 
 import java.util.HashMap;
 import java.util.Random;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.PushbackInputStream;
 
 @ReactModule(name = RNWhisperModule.NAME)
 public class RNWhisperModule extends ReactContextBaseJavaModule implements LifecycleEventListener {
   public static final String NAME = "RNWhisper";
 
   private ReactApplicationContext reactContext;
+  private Downloader downloader;
 
   public RNWhisperModule(ReactApplicationContext reactContext) {
     super(reactContext);
     reactContext.addLifecycleEventListener(this);
     this.reactContext = reactContext;
+    this.downloader = new Downloader(reactContext);
   }
 
   @Override
@@ -39,19 +44,48 @@ public class RNWhisperModule extends ReactContextBaseJavaModule implements Lifec
 
   private HashMap<Integer, WhisperContext> contexts = new HashMap<>();
 
+  private int getResourceIdentifier(String filePath) {
+    int identifier = reactContext.getResources().getIdentifier(
+      filePath,
+      "drawable",
+      reactContext.getPackageName()
+    );
+    if (identifier == 0) {
+      identifier = reactContext.getResources().getIdentifier(
+        filePath,
+        "raw",
+        reactContext.getPackageName()
+      );
+    }
+    return identifier;
+  }
+
   @ReactMethod
-  public void initContext(final String modelPath, final boolean isBundleAsset, final Promise promise) {
+  public void initContext(final ReadableMap options, final Promise promise) {
     new AsyncTask<Void, Void, Integer>() {
       private Exception exception;
 
       @Override
       protected Integer doInBackground(Void... voids) {
         try {
+          String modelPath = options.getString("filePath");
+          boolean isBundleAsset = options.getBoolean("isBundleAsset");
+
+          String modelFilePath = modelPath;
+          if (!isBundleAsset && (modelPath.startsWith("http://") || modelPath.startsWith("https://"))) {
+            modelFilePath = downloader.downloadFile(modelPath);
+          }
+
           long context;
-          if (isBundleAsset) {
-            context = WhisperContext.initContextWithAsset(reactContext.getAssets(), modelPath);
+          int resId = getResourceIdentifier(modelFilePath);
+          if (resId > 0) {
+            context = WhisperContext.initContextWithInputStream(
+              new PushbackInputStream(reactContext.getResources().openRawResource(resId))
+            );
+          } else if (isBundleAsset) {
+            context = WhisperContext.initContextWithAsset(reactContext.getAssets(), modelFilePath);
           } else {
-            context = WhisperContext.initContext(modelPath);
+            context = WhisperContext.initContext(modelFilePath);
           }
           if (context == 0) {
             throw new Exception("Failed to initialize context");
@@ -98,7 +132,26 @@ public class RNWhisperModule extends ReactContextBaseJavaModule implements Lifec
       @Override
       protected WritableMap doInBackground(Void... voids) {
         try {
-          return context.transcribeFile(jobId, filePath, options);
+          String waveFilePath = filePath;
+
+          if (filePath.startsWith("http://") || filePath.startsWith("https://")) {
+            waveFilePath = downloader.downloadFile(filePath);
+          }
+
+          int resId = getResourceIdentifier(waveFilePath);
+          if (resId > 0) {
+            return context.transcribeInputStream(
+              (int) jobId,
+              reactContext.getResources().openRawResource(resId),
+              options
+            );
+          }
+
+          return context.transcribeInputStream(
+            (int) jobId,
+            new FileInputStream(new File(waveFilePath)),
+            options
+          );
         } catch (Exception e) {
           exception = e;
           return null;
@@ -217,5 +270,6 @@ public class RNWhisperModule extends ReactContextBaseJavaModule implements Lifec
       context.release();
     }
     contexts.clear();
+    downloader.clearCache();
   }
 }