@novastera-oss/llamarn 0.1.5-beta.3 → 0.2.2

package/README.md

@@ -2,8 +2,6 @@
 
 > ⚠️ **WORK IN PROGRESS**: This package is currently under active development. Community help and feedback are greatly appreciated, especially in the areas mentioned in What Needs Help.
 
-A React Native wrapper for llama.cpp focused on providing a simple, reliable way to run LLMs on mobile devices. This project was inspired by and builds upon the excellent work of [llama.rn](https://github.com/mybigday/llama.rn).
-
 ## Goals
 
 * Provide a thin, reliable wrapper around llama.cpp for React Native
@@ -25,11 +23,12 @@ A React Native wrapper for llama.cpp focused on providing a simple, reliable way
 
 We welcome contributions, especially in these areas:
 
-1. **Android GPU Testing**:
-   * Testing on a variety of Android devices with different GPUs
-   * Verifying OpenCL and Vulkan acceleration on real hardware
-   * Performance profiling and optimization for mobile GPUs
-   * Adding automated GPU detection tests
+1. **Android GPU Testing and Detection**:
+   * Development of reliable GPU detection mechanism in React Native
+   * Implementation of proper backend initialization verification
+   * Creation of robust testing framework for GPU availability
+   * Integration of OpenCL and Vulkan acceleration once detection is stable
+   * Performance benchmarking and optimization for mobile GPUs
 
 2. **CI Improvements**:
    * Adding automated Android GPU tests to CI pipeline
@@ -62,6 +61,73 @@ If you're interested in helping with any of these areas, please check our Contri
 npm install @novastera-oss/llamarn
 ```
 
+## Developer Setup
+
+If you're contributing to the library or running the example project, follow these setup steps:
+
+### Prerequisites
+
+1. Clone the repository and navigate to the project directory
+2. Ensure you have React Native development environment set up for your target platform(s)
+
+### Initial Setup
+
+```sh
+# Install dependencies
+npm install
+
+# Optional if you already had previous version of llamacpp
+npm run clean-llama
+
+# Initialize llama.cpp submodule and dependencies
+npm run setup-llama-cpp
+```
+
+### Android Development
+
+1. Build the native Android libraries:
+```sh
+# Build the external native libraries for Android
+./scripts/build_android_external.sh
+```
+
+2. Run the example project:
+```sh
+cd example
+npm run android
+```
+
+### iOS Development
+
+1. Navigate to the example project and install iOS dependencies:
+```sh
+cd example
+cd ios
+
+# Install CocoaPods dependencies
+bundle exec pod install
+
+# Or if not using Bundler:
+# pod install
+
+cd ..
+```
+
+2. Run the example project:
+```sh
+npm run ios
+```
+
+### Development Notes
+
+- **Android**: The `build_android_external.sh` script compiles llama.cpp for Android architectures and sets up the necessary native libraries. This step is required before running the Android example.
+
+- **iOS**: The iOS setup uses CocoaPods to manage native dependencies. The prebuilt llama.cpp framework is included in the repository.
+
+- **Troubleshooting**: If you encounter build issues, try cleaning your build cache:
+  - Android: `cd android && ./gradlew clean`
+  - iOS: `cd example/ios && rm -rf build && rm Podfile.lock && pod install`
+
 ## Basic Usage
 
 ### Simple Completion
@@ -216,20 +282,20 @@ The module accepts different path formats depending on the platform:
 
 ## About
 
-Part of [Novastera's](https://novastera.com) suite of privacy-focused solutions, this package enables on-device LLM inference with no data leaving the user's device. We're committed to helping developers build AI-powered applications that respect user privacy.
+This library is currently being used in [Novastera's](https://novastera.com) mobile application, demonstrating its capabilities in production environments. We're committed to enabling on-device LLM inference with no data leaving the user's device, helping developers build AI-powered applications that respect user privacy.
 
 ## License
 
-Apache 2.0 © [Novastera](https://novastera.com)
+Apache 2.0
 
 ## Acknowledgments
 
-We extend our heartfelt gratitude to the following projects and communities that made this library possible:
+We acknowledge the following projects and communities that have contributed to the development of this library:
 
-* **[mybigday/llama.rn](https://github.com/mybigday/llama.rn)** - A pioneering React Native binding for llama.cpp that demonstrated the feasibility of on-device LLM inference in mobile applications.
+* **[mybigday/llama.rn](https://github.com/mybigday/llama.rn)** - A foundational React Native binding for llama.cpp that demonstrated the viability of on-device LLM inference in mobile applications.
 
-* **[Zach-Dean-Attractions-io/react-native-pure-cpp-turbo-module-library](https://github.com/Zach-Dean-Attractions-io/react-native-pure-cpp-turbo-module-library)** - A good resource that enabled us to successfully build the Android Turbo Module when the official React Native documentation proved insufficient. This repository provided the missing pieces for proper C++ integration on Android.
+* **[ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp)** - The core C++ library that enables efficient LLM inference, serving as the foundation for this project.
 
-* **[ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp)** - The underlying C++ library that powers all llama.cpp-based projects, providing efficient LLM inference capabilities.
+* The test implementation of the Android Turbo Module ([react-native-pure-cpp-turbo-module-library](https://github.com/Zach-Dean-Attractions-io/react-native-pure-cpp-turbo-module-library)) provided valuable insights for our C++ integration.
 
-Without these contributions to the open-source community, this project would not have been possible. We're proud to continue building upon this foundation and contributing back to the ecosystem.
+These projects have significantly contributed to the open-source ecosystem, and we are committed to building upon their work while maintaining the same spirit of collaboration and innovation.

package/RNLlamaCpp.podspec

@@ -27,11 +27,17 @@ Pod::Spec.new do |s|
                    # If these are compiled directly, their paths need to be relative to the podspec (e.g., "cpp/llama.cpp/common/common.{h,cpp}")
                    "cpp/llama.cpp/common/common.{h,cpp}",
                    "cpp/llama.cpp/common/log.{h,cpp}",
+                   "cpp/llama.cpp/common/arg.{h,cpp}",
                    "cpp/llama.cpp/common/sampling.{h,cpp}",
                    "cpp/llama.cpp/common/chat.{h,cpp}",
+                   "cpp/llama.cpp/common/chat-parser.{h,cpp}",
+                   "cpp/llama.cpp/common/regex-partial.{h,cpp}",
+                   "cpp/llama.cpp/common/console.{h,cpp}",
+                   "cpp/llama.cpp/common/json-partial.{h,cpp}",
                    "cpp/llama.cpp/common/ngram-cache.{h,cpp}",
                    "cpp/llama.cpp/common/json-schema-to-grammar.{h,cpp}",
                    "cpp/llama.cpp/common/speculative.{h,cpp}",
+                   "cpp/llama.cpp/common/llguidance.{h,cpp}",
                    "cpp/llama.cpp/common/*.hpp",
                    "cpp/llama.cpp/common/minja/*.hpp"
   
@@ -52,16 +58,17 @@ Pod::Spec.new do |s|
     "SWIFT_OPTIMIZATION_LEVEL" => "-O",
     "ENABLE_BITCODE" => "NO",
     "DEFINES_MODULE" => "YES",
-    "OTHER_LDFLAGS" => "$(inherited)",
+    "OTHER_LDFLAGS" => "$(inherited) -framework Accelerate -framework Foundation -framework Metal -framework MetalKit",
     # These preprocessor macros ensure TurboModule registration works correctly
     "GCC_PREPROCESSOR_DEFINITIONS" => ["$(inherited)", "RCT_NEW_ARCH_ENABLED=1", 
                                        "__STDC_FORMAT_MACROS=1", # For format macros in C++
                                        "LLAMA_SHARED=1"]         # For llama shared symbols
   }
 
-  # Add user_target_xcconfig to propagate linker flags
+  # Add user_target_xcconfig to propagate linker flags and fix framework issues
   s.user_target_xcconfig = {
-    "OTHER_LDFLAGS" => "$(inherited)"
+    "OTHER_LDFLAGS" => "$(inherited) -framework Accelerate -framework Foundation -framework Metal -framework MetalKit",
+    "FRAMEWORK_SEARCH_PATHS" => "$(inherited) $(PLATFORM_DIR)/Developer/Library/Frameworks"
   }
 
   # Install dependencies for Turbo Modules

package/android/CMakeLists.txt

@@ -57,7 +57,15 @@ add_library(
     ${CPP_DIR}/llama.cpp/common/common.cpp
     ${CPP_DIR}/llama.cpp/common/sampling.cpp
     ${CPP_DIR}/llama.cpp/common/chat.cpp
+    ${CPP_DIR}/llama.cpp/common/chat-parser.cpp
+    ${CPP_DIR}/llama.cpp/common/regex-partial.cpp
+    ${CPP_DIR}/llama.cpp/common/arg.cpp
+    ${CPP_DIR}/llama.cpp/common/console.cpp
+    ${CPP_DIR}/llama.cpp/common/json-partial.cpp
+    ${CPP_DIR}/llama.cpp/common/ngram-cache.cpp
     ${CPP_DIR}/llama.cpp/common/json-schema-to-grammar.cpp
+    ${CPP_DIR}/llama.cpp/common/speculative.cpp
+    ${CPP_DIR}/llama.cpp/common/llguidance.cpp
 )
 
 add_library(

package/android/src/main/cpp/include/llama.h

@@ -4,6 +4,7 @@
 #include "ggml.h"
 #include "ggml-cpu.h"
 #include "ggml-backend.h"
+#include "ggml-opt.h"
 
 #include <stddef.h>
 #include <stdint.h>
@@ -344,7 +345,7 @@ extern "C" {
         float    yarn_beta_fast;   // YaRN low correction dim
         float    yarn_beta_slow;   // YaRN high correction dim
         uint32_t yarn_orig_ctx;    // YaRN original context size
-        float    defrag_thold;     // defragment the KV cache if holes/size > thold, < 0 disabled (default)
+        float    defrag_thold;     // defragment the KV cache if holes/size > thold, <= 0 disabled (default)
 
         ggml_backend_sched_eval_callback cb_eval;
         void * cb_eval_user_data;
@@ -360,10 +361,11 @@ extern "C" {
 
         // Keep the booleans together and at the end of the struct to avoid misalignment during copy-by-value.
         bool embeddings;  // if true, extract embeddings (together with logits)
-        bool offload_kqv; // whether to offload the KQV ops (including the KV cache) to GPU
-        bool flash_attn;  // whether to use flash attention [EXPERIMENTAL]
-        bool no_perf;     // whether to measure performance timings
-        bool op_offload;  // whether to offload host tensor operations to device
+        bool offload_kqv; // offload the KQV ops (including the KV cache) to GPU
+        bool flash_attn;  // use flash attention [EXPERIMENTAL]
+        bool no_perf;     // measure performance timings
+        bool op_offload;  // offload host tensor operations to device
+        bool swa_full;    // use full-size SWA cache (https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
     };
 
     // model quantization parameters
@@ -445,6 +447,10 @@ extern "C" {
                                  size_t    n_paths,
               struct llama_model_params    params);
 
+    LLAMA_API void llama_model_save_to_file(
+            const struct llama_model * model,
+                        const char * path_model);
+
     DEPRECATED(LLAMA_API void llama_free_model(struct llama_model * model),
             "use llama_model_free instead");
 
@@ -465,6 +471,7 @@ extern "C" {
     LLAMA_API int64_t llama_time_us(void);
 
     LLAMA_API size_t llama_max_devices(void);
+    LLAMA_API size_t llama_max_parallel_sequences(void);
 
     LLAMA_API bool llama_supports_mmap       (void);
     LLAMA_API bool llama_supports_mlock      (void);
@@ -602,71 +609,14 @@ extern "C" {
     // KV cache
     //
 
-    // TODO: start using struct llama_kv_cache
-
-    // Information associated with an individual cell in the KV cache view.
-    struct llama_kv_cache_view_cell {
-        // The position for this cell. Takes KV cache shifts into account.
-        // May be negative if the cell is not populated.
-        llama_pos pos;
-    };
-
-    // An updateable view of the KV cache.
-    struct llama_kv_cache_view {
-        // Number of KV cache cells. This will be the same as the context size.
-        int32_t n_cells;
-
-        // Maximum number of sequences that can exist in a cell. It's not an error
-        // if there are more sequences in a cell than this value, however they will
-        // not be visible in the view cells_sequences.
-        int32_t n_seq_max;
-
-        // Number of tokens in the cache. For example, if there are two populated
-        // cells, the first with 1 sequence id in it and the second with 2 sequence
-        // ids then you'll have 3 tokens.
-        int32_t token_count;
-
-        // Number of populated cache cells.
-        int32_t used_cells;
-
-        // Maximum contiguous empty slots in the cache.
-        int32_t max_contiguous;
-
-        // Index to the start of the max_contiguous slot range. Can be negative
-        // when cache is full.
-        int32_t max_contiguous_idx;
-
-        // Information for an individual cell.
-        struct llama_kv_cache_view_cell * cells;
-
-        // The sequences for each cell. There will be n_seq_max items per cell.
-        llama_seq_id * cells_sequences;
-    };
-
-    // Create an empty KV cache view. (use only for debugging purposes)
-    LLAMA_API struct llama_kv_cache_view llama_kv_cache_view_init(const struct llama_context * ctx, int32_t n_seq_max);
-
-    // Free a KV cache view. (use only for debugging purposes)
-    LLAMA_API void llama_kv_cache_view_free(struct llama_kv_cache_view * view);
-
-    // Update the KV cache view structure with the current state of the KV cache. (use only for debugging purposes)
-    // TODO: change signature to llama_kv_cache_view_update(struct llama_kv_cache_view * view, const struct llama_context * ctx)
-    LLAMA_API void llama_kv_cache_view_update(const struct llama_context * ctx, struct llama_kv_cache_view * view);
-
-    ///
-
     // Returns the number of tokens in the KV cache (slow, use only for debug)
     // If a KV cell has multiple sequences assigned to it, it will be counted multiple times
-    LLAMA_API int32_t llama_kv_self_n_tokens(const struct llama_context * ctx);
-
-    DEPRECATED(LLAMA_API int32_t llama_get_kv_cache_token_count(const struct llama_context * ctx),
-            "use llama_kv_self_n_tokens instead");
+    DEPRECATED(LLAMA_API int32_t llama_kv_self_n_tokens(const struct llama_context * ctx),
+               "Use llama_kv_self_seq_pos_max() and llama_kv_self_seq_pos_min() instead (https://github.com/ggml-org/llama.cpp/issues/13793)");
 
     // Returns the number of used KV cells (i.e. have at least one sequence assigned to them)
-    LLAMA_API int32_t llama_kv_self_used_cells(const struct llama_context * ctx);
-
-    DEPRECATED(LLAMA_API int32_t llama_get_kv_cache_used_cells(const struct llama_context * ctx),
-            "use llama_kv_self_used_cells instead");
+    DEPRECATED(LLAMA_API int32_t llama_kv_self_used_cells(const struct llama_context * ctx),
+               "Use llama_kv_self_seq_pos_max() and llama_kv_self_seq_pos_min() instead (https://github.com/ggml-org/llama.cpp/issues/13793)");
 
     // Clear the KV cache - both cell info is erased and KV data is zeroed
     LLAMA_API void llama_kv_self_clear(
@@ -725,10 +675,18 @@ extern "C" {
                        llama_pos   p1,
                              int   d);
 
+    // Returns the smallest position present in the KV cache for the specified sequence
+    // This is typically non-zero only for SWA caches
+    // Return -1 if the sequence is empty
+    LLAMA_API llama_pos llama_kv_self_seq_pos_min(
+            struct llama_context * ctx,
+                    llama_seq_id   seq_id);
+
     // Returns the largest position present in the KV cache for the specified sequence
+    // Return -1 if the sequence is empty
     LLAMA_API llama_pos llama_kv_self_seq_pos_max(
             struct llama_context * ctx,
-                     llama_seq_id   seq_id);
+                    llama_seq_id   seq_id);
 
     // Defragment the KV cache
     // This will be applied:
@@ -742,61 +700,6 @@ extern "C" {
     // Apply the KV cache updates (such as K-shifts, defragmentation, etc.)
     LLAMA_API void llama_kv_self_update(struct llama_context * ctx);
 
-    DEPRECATED(LLAMA_API void llama_kv_cache_clear(
-            struct llama_context * ctx),
-            "use llama_kv_self_clear instead");
-
-    DEPRECATED(LLAMA_API bool llama_kv_cache_seq_rm(
-            struct llama_context * ctx,
-                    llama_seq_id   seq_id,
-                       llama_pos   p0,
-                       llama_pos   p1),
-            "use llama_kv_self_seq_rm instead");
-
-    DEPRECATED(LLAMA_API void llama_kv_cache_seq_cp(
-            struct llama_context * ctx,
-                    llama_seq_id   seq_id_src,
-                    llama_seq_id   seq_id_dst,
-                       llama_pos   p0,
-                       llama_pos   p1),
-            "use llama_kv_self_seq_cp instead");
-
-    DEPRECATED(LLAMA_API void llama_kv_cache_seq_keep(
-            struct llama_context * ctx,
-                    llama_seq_id   seq_id),
-            "use llama_kv_self_seq_keep instead");
-
-    DEPRECATED(LLAMA_API void llama_kv_cache_seq_add(
-            struct llama_context * ctx,
-                    llama_seq_id   seq_id,
-                       llama_pos   p0,
-                       llama_pos   p1,
-                       llama_pos   delta),
-            "use llama_kv_self_seq_add instead");
-
-    DEPRECATED(LLAMA_API void llama_kv_cache_seq_div(
-            struct llama_context * ctx,
-                    llama_seq_id   seq_id,
-                       llama_pos   p0,
-                       llama_pos   p1,
-                             int   d),
-            "use llama_kv_self_seq_div instead");
-
-    DEPRECATED(LLAMA_API llama_pos llama_kv_cache_seq_pos_max(
-            struct llama_context * ctx,
-                    llama_seq_id   seq_id),
-            "use llama_kv_self_seq_pos_max instead");
-
-    DEPRECATED(LLAMA_API void llama_kv_cache_defrag(struct llama_context * ctx),
-            "use llama_kv_self_defrag instead");
-
-    DEPRECATED(LLAMA_API bool llama_kv_cache_can_shift(const struct llama_context * ctx),
-            "use llama_kv_self_can_shift instead");
-
-    DEPRECATED(LLAMA_API void llama_kv_cache_update(struct llama_context * ctx),
-            "use llama_kv_self_update instead");
-
-
     //
     // State / sessions
     //
@@ -938,9 +841,12 @@ extern "C" {
     // Requires KV cache.
     // For encode-decoder contexts, processes the batch using the decoder.
     // Positive return values does not mean a fatal error, but rather a warning.
-    //   0 - success
-    //   1 - could not find a KV slot for the batch (try reducing the size of the batch or increase the context)
-    // < 0 - error. the KV cache state is restored to the state before this call
+    // Upon non-zero return values, the KV cache state is restored to the state before this call
+    //    0 - success
+    //    1 - could not find a KV slot for the batch (try reducing the size of the batch or increase the context)
+    //    2 - aborted
+    //   -1 - invalid input batch
+    // < -1 - error
     LLAMA_API int32_t llama_decode(
             struct llama_context * ctx,
               struct llama_batch   batch);
@@ -1433,6 +1339,37 @@ extern "C" {
     LLAMA_API void                           llama_perf_sampler_print(const struct llama_sampler * chain);
     LLAMA_API void                           llama_perf_sampler_reset(      struct llama_sampler * chain);
 
+    //
+    // training
+    //
+
+    // function that returns whether or not a given tensor contains trainable parameters
+    typedef bool (*llama_opt_param_filter)(const struct ggml_tensor * tensor, void * userdata);
+
+    // always returns true
+    LLAMA_API bool llama_opt_param_filter_all(const struct ggml_tensor * tensor, void * userdata);
+
+    struct llama_opt_params {
+        uint32_t n_ctx_train; // assumed context size post training, use context size specified in llama_context if 0
+
+        llama_opt_param_filter param_filter; // callback for determining which tensors contain trainable parameters
+        void * param_filter_ud;              // userdata for determining which tensors contain trainable parameters
+
+        ggml_opt_get_optimizer_params get_opt_pars; // callback for calculating optimizer parameters
+        void * get_opt_pars_ud;                     // userdata for calculating optimizer parameters
+    };
+
+    LLAMA_API void llama_opt_init(struct llama_context * lctx, struct llama_model * model, struct llama_opt_params lopt_params);
+
+    LLAMA_API void llama_opt_epoch(
+            struct llama_context    * lctx,
+            ggml_opt_dataset_t        dataset,
+            ggml_opt_result_t         result_train,
+            ggml_opt_result_t         result_eval,
+            int64_t                   idata_split,
+            ggml_opt_epoch_callback   callback_train,
+            ggml_opt_epoch_callback   callback_eval);
+
 #ifdef __cplusplus
 }
 #endif

package/cpp/LlamaCppModel.cpp

@@ -35,8 +35,8 @@
 
 namespace facebook::react {
 
-LlamaCppModel::LlamaCppModel(rn_llama_context* rn_ctx)
-    : rn_ctx_(rn_ctx), should_stop_completion_(false), is_predicting_(false) {
+LlamaCppModel::LlamaCppModel(rn_llama_context* rn_ctx, std::shared_ptr<CallInvoker> jsInvoker)
+    : rn_ctx_(rn_ctx), should_stop_completion_(false), is_predicting_(false), jsInvoker_(jsInvoker) {
     initHelpers();
 }
 
@@ -435,11 +435,18 @@ CompletionResult LlamaCppModel::completion(const CompletionOptions& options, std
   rn_ctx_->params.n_predict = options.n_predict;
 
   // Check for a partial callback
-  auto callback_adapter = [&partialCallback, runtime](const std::string& token, bool is_done) -> bool {
+  auto callback_adapter = [&partialCallback, runtime, this](const std::string& token, bool is_done) -> bool {
+    // Check for stop condition first
+    if (should_stop_completion_) {
+      return false; // Signal to stop completion
+    }
+    
     if (partialCallback && runtime && !is_done) {
       partialCallback(*runtime, token.c_str());
     }
-    return true;
+    
+    // Return true to continue, false to stop
+    return !should_stop_completion_;
   };
 
   // Run the completion based on whether we have messages or prompt
@@ -546,7 +553,7 @@ jsi::Value LlamaCppModel::jsonToJsi(jsi::Runtime& rt, const json& j) {
   return jsi::Value::undefined();
 }
 
-// JSI method for completions
+// JSI method for completions (synchronous - kept for compatibility)
 jsi::Value LlamaCppModel::completionJsi(jsi::Runtime& rt, const jsi::Value* args, size_t count) {
   if (count < 1 || !args[0].isObject()) {
     throw jsi::JSError(rt, "completion requires an options object");
@@ -581,6 +588,116 @@ jsi::Value LlamaCppModel::completionJsi(jsi::Runtime& rt, const jsi::Value* args
   }
 }
 
+// JSI method for async completions (recommended approach)
+jsi::Value LlamaCppModel::completionAsyncJsi(jsi::Runtime& rt, const jsi::Value* args, size_t count) {
+  if (count < 1 || !args[0].isObject()) {
+    throw jsi::JSError(rt, "completion requires an options object");
+  }
+
+  if (!jsInvoker_) {
+    // Fallback to synchronous if no CallInvoker available
+    return completionJsi(rt, args, count);
+  }
+
+  // Parse options and callback on the current thread
+  CompletionOptions options;
+  std::shared_ptr<jsi::Function> callbackFn = nullptr;
+  
+  try {
+    options = parseCompletionOptions(rt, args[0].getObject(rt));
+    
+    if (count > 1 && args[1].isObject() && args[1].getObject(rt).isFunction(rt)) {
+      callbackFn = std::make_shared<jsi::Function>(args[1].getObject(rt).getFunction(rt));
+      options.stream = true;
+    }
+  } catch (const std::exception& e) {
+    throw jsi::JSError(rt, e.what());
+  }
+
+  // Create Promise constructor
+  auto Promise = rt.global().getPropertyAsFunction(rt, "Promise");
+  
+  auto executor = jsi::Function::createFromHostFunction(
+    rt,
+    jsi::PropNameID::forAscii(rt, "executor"),
+    2,
+    [this, options, callbackFn](jsi::Runtime& runtime, const jsi::Value& thisValue, const jsi::Value* args, size_t count) -> jsi::Value {
+      
+      auto resolve = std::make_shared<jsi::Function>(args[0].asObject(runtime).asFunction(runtime));
+      auto reject = std::make_shared<jsi::Function>(args[1].asObject(runtime).asFunction(runtime));
+      
+      // Create shared references to runtime and invoker for thread safety
+      auto runtimePtr = &runtime;
+      auto invoker = jsInvoker_;
+      auto selfPtr = shared_from_this(); // This requires LlamaCppModel to inherit from std::enable_shared_from_this
+      
+      // Launch background thread for completion
+      std::thread([selfPtr, options, callbackFn, resolve, reject, runtimePtr, invoker]() {
+        try {
+          // Create callback that schedules token updates on JS thread
+          std::function<void(jsi::Runtime&, const char*)> partialCallback = nullptr;
+          
+          if (callbackFn && invoker) {
+            partialCallback = [callbackFn, invoker, runtimePtr](jsi::Runtime& rt, const char* token) {
+              std::string tokenCopy(token);
+              invoker->invokeAsync([callbackFn, tokenCopy, runtimePtr]() {
+                try {
+                  jsi::Object data(*runtimePtr);
+                  data.setProperty(*runtimePtr, "token", jsi::String::createFromUtf8(*runtimePtr, tokenCopy));
+                  callbackFn->call(*runtimePtr, data);
+                } catch (...) {
+                  // Ignore callback errors
+                }
+              });
+            };
+          }
+          
+          // Run completion
+          CompletionResult result = selfPtr->completion(options, partialCallback, runtimePtr);
+          
+          // Schedule success callback on JS thread
+          invoker->invokeAsync([selfPtr, resolve, result, runtimePtr]() {
+            try {
+              jsi::Object jsResult = selfPtr->completionResultToJsi(*runtimePtr, result);
+              resolve->call(*runtimePtr, jsResult);
+            } catch (const std::exception& e) {
+              // If conversion fails, create a simple error response
+              jsi::Object errorObj(*runtimePtr);
+              errorObj.setProperty(*runtimePtr, "error", jsi::String::createFromUtf8(*runtimePtr, e.what()));
+              resolve->call(*runtimePtr, errorObj);
+            }
+          });
+          
+        } catch (const std::exception& e) {
+          // Schedule error callback on JS thread
+          std::string errorMsg(e.what());
+          invoker->invokeAsync([reject, errorMsg, runtimePtr]() {
+            try {
+              reject->call(*runtimePtr, jsi::String::createFromUtf8(*runtimePtr, errorMsg));
+            } catch (...) {
+              // Ignore rejection errors
+            }
+          });
+        }
+      }).detach();
+      
+      return jsi::Value::undefined();
+    }
+  );
+  
+  return Promise.callAsConstructor(rt, std::move(executor));
+}
+
+// JSI method for stopping completion
+jsi::Value LlamaCppModel::stopCompletionJsi(jsi::Runtime& rt, const jsi::Value* args, size_t count) {
+  try {
+    setShouldStopCompletion(true);
+    return jsi::Value(true);
+  } catch (const std::exception& e) {
+    throw jsi::JSError(rt, e.what());
+  }
+}
+
 jsi::Value LlamaCppModel::tokenizeJsi(jsi::Runtime& rt, const jsi::Value* args, size_t count) {
   if (count < 1 || !args[0].isObject()) {
     throw jsi::JSError(rt, "tokenize requires an options object with 'content' field");
@@ -930,12 +1047,36 @@ jsi::Value LlamaCppModel::get(jsi::Runtime& rt, const jsi::PropNameID& name) {
       });
   }
   else if (nameStr == "completion") {
+    // Use async completion as the default to provide better UX
+    if (jsInvoker_) {
+      return jsi::Function::createFromHostFunction(
+        rt, name, 2,
+        [this](jsi::Runtime& runtime, const jsi::Value& thisValue, const jsi::Value* args, size_t count) {
+          return this->completionAsyncJsi(runtime, args, count);
+        });
+    } else {
+      // Fallback to sync completion if no CallInvoker
+      return jsi::Function::createFromHostFunction(
+        rt, name, 2,
+        [this](jsi::Runtime& runtime, const jsi::Value& thisValue, const jsi::Value* args, size_t count) {
+          return this->completionJsi(runtime, args, count);
+        });
+    }
+  }
+  else if (nameStr == "completionSync") {
     return jsi::Function::createFromHostFunction(
       rt, name, 2,
       [this](jsi::Runtime& runtime, const jsi::Value& thisValue, const jsi::Value* args, size_t count) {
         return this->completionJsi(runtime, args, count);
       });
   }
+  else if (nameStr == "stopCompletion") {
+    return jsi::Function::createFromHostFunction(
+      rt, name, 0,
+      [this](jsi::Runtime& runtime, const jsi::Value& thisValue, const jsi::Value* args, size_t count) {
+        return this->stopCompletionJsi(runtime, args, count);
+      });
+  }
   else if (nameStr == "embedding") {
     return jsi::Function::createFromHostFunction(
       rt, name, 1,
@@ -973,6 +1114,8 @@ std::vector<jsi::PropNameID> LlamaCppModel::getPropertyNames(jsi::Runtime& rt) {
   result.push_back(jsi::PropNameID::forAscii(rt, "tokenize"));
   result.push_back(jsi::PropNameID::forAscii(rt, "detokenize"));
   result.push_back(jsi::PropNameID::forAscii(rt, "completion"));
+  result.push_back(jsi::PropNameID::forAscii(rt, "completionSync"));
+  result.push_back(jsi::PropNameID::forAscii(rt, "stopCompletion"));
   result.push_back(jsi::PropNameID::forAscii(rt, "embedding"));
   result.push_back(jsi::PropNameID::forAscii(rt, "release"));
   result.push_back(jsi::PropNameID::forAscii(rt, "n_vocab"));