@op-engineering/op-sqlite 1.0.10 → 1.0.12

package/README.md

@@ -8,32 +8,17 @@
 </div>
 <br />
 
-OP SQLite embeds the latest version of SQLite and provides a low-level (JSI-backed) API to execute SQL queries.
+OP SQLite embeds the latest version of SQLite and provides a low-level API to execute SQL queries.
 
 **Current SQLite version: 3.44.0**
 
-Created by [@ospfranco](https://twitter.com/ospfranco). Also created `react-native-quick-sqlite`, this is the next version. You can expect a new version once Static Hermes is out.
-
-**Please consider Sponsoring**, none of this work is for free. I pay for it with my time and knowledge. If you are a company in need of help with your React Native/React apps feel free to reach out. I also do a lot of C++ and nowadays Rust.
-
-## Coming up
-
-I will gladly review bug fixes, but in order for me to continue support and add new features, I ask you to sponsor me. Some of the things that can still be done to make this package faster and more complete:
-
-- Prepared statements
-- Inlining functions
-- Batching queries
-- Update hook
+Created by [@ospfranco](https://twitter.com/ospfranco). **Please consider Sponsoring**, none of this work is for free. I pay for it with my time and knowledge. If you are a company in need of help with your React Native/React apps feel free to reach out. I also do a lot of C++ and nowadays Rust.
 
 ## Benchmarks
 
-You can find the [benchmarking code in the example app](https://github.com/OP-Engineering/op-sqlite/blob/main/example/src/Database.ts#L44). Non JSI libraries are not even a contender anymore, you should expect anywhere between a 5x to a 8x improvement over sqlite-storage, sqlite2 and so on. Loading a 300k record database (in milliseconds).
+You can find the [benchmarking code in the example app](https://github.com/OP-Engineering/op-sqlite/blob/main/example/src/Database.ts#L44). You should expect anywhere between a 5x to a 8x improvement over non-JSI packages, and now a 5x to 8x improvement over quick-sqlite and expo-sqlite. Loading a 300k record database (in milliseconds).
 
-| Library      | iPhone 15 Pro | Galaxy S22 |
-| ------------ | ------------- | ---------- |
-| quick-sqlite | 2719ms        | 8851ms     |
-| expo-sqlite  | 2293ms        | 10626ms    |
-| op-sqlite    | 507ms         | 1125ms     |
+![benchmark](benchmark.png)
 
 Memory consumption is also is also 1/4 compared to `react-native-quick-sqlite`. This query used to take 1.2gb of peak memory usage, now runs in 250mbs.
 
@@ -231,7 +216,7 @@ db.executeAsync(
 
 ### Blobs
 
-Blobs are supported via `ArrayBuffer`, you need to be careful about the semantics though. You cannot instanciate an instance of `ArrayBuffer` directly, nor pass a typed array directly. Here is an example:
+Blobs are supported via `ArrayBuffer`, you need to be careful about the semantics though. You cannot instantiate an instance of `ArrayBuffer` directly, nor pass a typed array directly. Here is an example:
 
 ```ts
 db = open({
@@ -296,6 +281,28 @@ const { rowsAffected, commands } = db
   });
 ```
 
+## Update hook
+
+You can subscribe to changes in your database by using an update hook:
+
+```ts
+// Bear in mind: rowId is not your table primary key but the internal rowId sqlite uses
+// to keep track of the table rows
+db.updateHook(({ rowId, table, operation, row = {} }) => {
+  console.warn(`Hook has been called, rowId: ${rowId}, ${table}, ${operation}`);
+  // Will contain the entire row that changed
+  // only on UPDATE and INSERT operations
+  console.warn(JSON.stringify(row, null, 2));
+});
+
+db.execute('INSERT INTO "User" (id, name, age, networth) VALUES(?, ?, ?, ?)', [
+  id,
+  name,
+  age,
+  networth,
+]);
+```
+
 ## Use built-in SQLite
 
 On iOS you can use the embedded SQLite, when running `pod-install` add an environment flag:

package/cpp/ThreadPool.cpp

@@ -2,94 +2,129 @@
 
 namespace opsqlite {
 
-ThreadPool::ThreadPool() : done(false)
-{
-  // This returns the number of threads supported by the system. If the
-  // function can't figure out this information, it returns 0. 0 is not good,
-  // so we create at least 1
-  auto numberOfThreads = std::thread::hardware_concurrency();
-  if (numberOfThreads == 0)
-  {
-    numberOfThreads = 1;
-  }
-
-  for (unsigned i = 0; i < numberOfThreads; ++i)
-  {
-    // The threads will execute the private member `doWork`. Note that we need
-    // to pass a reference to the function (namespaced with the class name) as
-    // the first argument, and the current object as second argument
-    threads.push_back(std::thread(&ThreadPool::doWork, this));
-  }
-}
-
-// The destructor joins all the threads so the program can exit gracefully.
-// This will be executed if there is any exception (e.g. creating the threads)
-ThreadPool::~ThreadPool()
-{
-  // So threads know it's time to shut down
-  done = true;
-
-  // Wake up all the threads, so they can finish and be joined
-  workQueueConditionVariable.notify_all();
-  for (auto &thread : threads)
-  {
-    if (thread.joinable())
+    ThreadPool::ThreadPool() : done(false)
     {
-      thread.join();
+        // This returns the number of threads supported by the system. If the
+        // function can't figure out this information, it returns 0. 0 is not good,
+        // so we create at least 1
+        auto numberOfThreads = std::thread::hardware_concurrency();
+        if (numberOfThreads == 0)
+        {
+            numberOfThreads = 1;
+        }
+        
+        for (unsigned i = 0; i < numberOfThreads; ++i)
+        {
+            // The threads will execute the private member `doWork`. Note that we need
+            // to pass a reference to the function (namespaced with the class name) as
+            // the first argument, and the current object as second argument
+            threads.push_back(std::thread(&ThreadPool::doWork, this));
+        }
     }
-  }
-}
-
-// This function will be called by the server every time there is a request
-// that needs to be processed by the thread pool
-void ThreadPool::queueWork(std::function<void(void)> task)
-{
-  // Grab the mutex
-  std::lock_guard<std::mutex> g(workQueueMutex);
-
-  // Push the request to the queue
-  workQueue.push(task);
-
-  // Notify one thread that there are requests to process
-  workQueueConditionVariable.notify_one();
-}
-
-// Function used by the threads to grab work from the queue
-void ThreadPool::doWork()
-{
-  // Loop while the queue is not destructing
-  while (!done)
-  {
-    std::function<void(void)> task;
 
-    // Create a scope, so we don't lock the queue for longer than necessary
+    // The destructor joins all the threads so the program can exit gracefully.
+    // This will be executed if there is any exception (e.g. creating the threads)
+    ThreadPool::~ThreadPool()
     {
-      std::unique_lock<std::mutex> g(workQueueMutex);
-      workQueueConditionVariable.wait(g, [&]
-                                      {
-        // Only wake up if there are elements in the queue or the program is
-        // shutting down
-        return !workQueue.empty() || done; });
-
-      // If we are shutting down exit witout trying to process more work
-      if (done)
-      {
-        break;
-      }
+        // So threads know it's time to shut down
+        done = true;
+        
+        // Wake up all the threads, so they can finish and be joined
+        workQueueConditionVariable.notify_all();
+        
+        for (auto &thread : threads)
+        {
+            if (thread.joinable())
+            {
+                thread.join();
+            }
+        }
+        
+        threads.clear();
+    }
 
-      task = workQueue.front();
-      workQueue.pop();
+    // This function will be called by the server every time there is a request
+    // that needs to be processed by the thread pool
+    void ThreadPool::queueWork(std::function<void(void)> task)
+    {
+        // Grab the mutex
+        std::lock_guard<std::mutex> g(workQueueMutex);
+        
+        // Push the request to the queue
+        workQueue.push(task);
+        
+        // Notify one thread that there are requests to process
+        workQueueConditionVariable.notify_one();
     }
-    ++busy;
-    task();
-    --busy;
-  }
-}
 
-void ThreadPool::waitFinished() {
-  std::unique_lock<std::mutex> g(workQueueMutex);
-  workQueueConditionVariable.wait(g, [&]{ return workQueue.empty() && (busy == 0); });
-}
+    // Function used by the threads to grab work from the queue
+    void ThreadPool::doWork()
+    {
+        // Loop while the queue is not destructing
+        while (!done)
+        {
+            std::function<void(void)> task;
+            
+            // Create a scope, so we don't lock the queue for longer than necessary
+            {
+                std::unique_lock<std::mutex> g(workQueueMutex);
+                workQueueConditionVariable.wait(g, [&]
+                                                {
+                    // Only wake up if there are elements in the queue or the program is
+                    // shutting down
+                    return !workQueue.empty() || done; });
+                
+                // If we are shutting down exit witout trying to process more work
+                if (done)
+                {
+                    break;
+                }
+                
+                task = workQueue.front();
+                workQueue.pop();
+            }
+            ++busy;
+            task();
+            --busy;
+        }
+    }
 
+    void ThreadPool::waitFinished() {
+        std::unique_lock<std::mutex> g(workQueueMutex);
+        workQueueConditionVariable.wait(g, [&]{ return workQueue.empty() && (busy == 0); });
+    }
 
+    void ThreadPool::restartPool() {
+        // So threads know it's time to shut down
+        done = true;
+        
+        // Wake up all the threads, so they can finish and be joined
+        workQueueConditionVariable.notify_all();
+        
+        for (auto &thread : threads)
+        {
+            if (thread.joinable())
+            {
+                thread.join();
+            }
+        }
+        
+        threads.clear();
+        
+        auto numberOfThreads = std::thread::hardware_concurrency();
+        if (numberOfThreads == 0)
+        {
+            numberOfThreads = 1;
+        }
+        
+        for (unsigned i = 0; i < numberOfThreads; ++i)
+        {
+            // The threads will execute the private member `doWork`. Note that we need
+            // to pass a reference to the function (namespaced with the class name) as
+            // the first argument, and the current object as second argument
+            threads.push_back(std::thread(&ThreadPool::doWork, this));
+        }
+        
+        done = false;
+    }
 }

package/cpp/ThreadPool.h

@@ -13,32 +13,33 @@ namespace opsqlite {
 
 class ThreadPool {
 public:
-  ThreadPool();
-  ~ThreadPool();
-  void queueWork(std::function<void(void)> task);
-  void waitFinished();
-
+    ThreadPool();
+    ~ThreadPool();
+    void queueWork(std::function<void(void)> task);
+    void waitFinished();
+    void restartPool();
+    
 private:
-  unsigned int busy;
-  // This condition variable is used for the threads to wait until there is work
-  // to do
-  std::condition_variable_any workQueueConditionVariable;
-
-  // We store the threads in a vector, so we can later stop them gracefully
-  std::vector<std::thread> threads;
-
-  // Mutex to protect workQueue
-  std::mutex workQueueMutex;
-
-  // Queue of requests waiting to be processed
-  std::queue<std::function<void(void)>> workQueue;
-
-  // This will be set to true when the thread pool is shutting down. This tells
-  // the threads to stop looping and finish
-  bool done;
-
-  // Function used by the threads to grab work from the queue
-  void doWork();
+    unsigned int busy;
+    // This condition variable is used for the threads to wait until there is work
+    // to do
+    std::condition_variable_any workQueueConditionVariable;
+    
+    // We store the threads in a vector, so we can later stop them gracefully
+    std::vector<std::thread> threads;
+    
+    // Mutex to protect workQueue
+    std::mutex workQueueMutex;
+    
+    // Queue of requests waiting to be processed
+    std::queue<std::function<void(void)>> workQueue;
+    
+    // This will be set to true when the thread pool is shutting down. This tells
+    // the threads to stop looping and finish
+    bool done;
+    
+    // Function used by the threads to grab work from the queue
+    void doWork();
 };
 
 }

package/cpp/bindings.cpp

@@ -9,6 +9,7 @@
 #include "macros.h"
 #include <iostream>
 #include "DumbHostObject.h"
+#include <unordered_map>
 
 namespace opsqlite {
 
@@ -16,16 +17,27 @@ namespace jsi = facebook::jsi;
 
 std::string basePath;
 std::shared_ptr<react::CallInvoker> invoker;
-std::shared_ptr<ThreadPool> pool;
+ThreadPool pool;
+std::unordered_map<std::string, std::shared_ptr<jsi::Value>> updateHooks = std::unordered_map<std::string, std::shared_ptr<jsi::Value>>();
+
+
+// React native will try to clean the module on JS context invalidation (CodePush/Hot Reload)
+// The clearState function is called and we use this flag to prevent any ongoing
+// operations from continuing work and can return early
+bool invalidated = false;
 
 void clearState() {
+    invalidated = true;
+    // Will terminate all operations and database connections
     sqliteCloseAll();
+    // We then join all the threads before the context gets invalidated
+    pool.restartPool();
 }
 
 void install(jsi::Runtime &rt, std::shared_ptr<react::CallInvoker> jsCallInvoker, const char *docPath)
 {
+    invalidated = false;
     basePath = std::string(docPath);
-    pool = std::make_shared<ThreadPool>();
     invoker = jsCallInvoker;
     
     auto open = HOSTFN("open", 3) {
@@ -227,21 +239,25 @@ void install(jsi::Runtime &rt, std::shared_ptr<react::CallInvoker> jsCallInvoker
         const jsi::Value &originalParams = args[2];
         
         std::vector<JSVariant> params = toVariantVec(rt, originalParams);
-        
+    
         auto promiseCtr = rt.global().getPropertyAsFunction(rt, "Promise");
         auto promise = promiseCtr.callAsConstructor(rt, HOSTFN("executor", 2) {
             auto resolve = std::make_shared<jsi::Value>(rt, args[0]);
             auto reject = std::make_shared<jsi::Value>(rt, args[1]);
             
             auto task =
-            [&rt, dbName, query, params = std::make_shared<std::vector<JSVariant>>(params), resolve, reject]()
+            [&rt, dbName, query, params = std::move(params), resolve, reject]()
             {
                 try
                 {
                     std::vector<DumbHostObject> results;
                     std::shared_ptr<std::vector<DynamicHostObject>> metadata = std::make_shared<std::vector<DynamicHostObject>>();;
                     
-                    auto status = sqliteExecute(dbName, query, params.get(), &results, metadata);
+                    auto status = sqliteExecute(dbName, query, &params, &results, metadata);
+                    
+                    if(invalidated) {
+                        return;
+                    }
                 
                     invoker->invokeAsync([&rt,
                                            results = std::make_shared<std::vector<DumbHostObject>>(results),
@@ -270,7 +286,7 @@ void install(jsi::Runtime &rt, std::shared_ptr<react::CallInvoker> jsCallInvoker
                 }
             };
             
-            pool->queueWork(task);
+            pool.queueWork(task);
             
             return {};
         }));
@@ -363,7 +379,7 @@ void install(jsi::Runtime &rt, std::shared_ptr<react::CallInvoker> jsCallInvoker
                     });
                 }
             };
-            pool->queueWork(task);
+            pool.queueWork(task);
             
             return {};
         }));
@@ -414,14 +430,54 @@ void install(jsi::Runtime &rt, std::shared_ptr<react::CallInvoker> jsCallInvoker
                     });
                 }
             };
-            pool->queueWork(task);
+            pool.queueWork(task);
             return {};
         }));
         
         return promise;
     });
     
-    
+    auto updateHook = HOSTFN("updateHook", 2)
+    {
+        if (sizeof(args) < 2)
+        {
+            throw jsi::JSError(rt, "[op-sqlite][loadFileAsync] Incorrect parameters: dbName and callback needed");
+            return {};
+        }
+        
+        auto dbName = args[0].asString(rt).utf8(rt);
+        auto callback = std::make_shared<jsi::Value>(rt, args[1]);
+        updateHooks[dbName] = callback;
+        
+        auto hook = [&rt, callback](std::string dbName, std::string tableName, std::string operation, int rowId) {
+            std::vector<JSVariant> params;
+            std::vector<DumbHostObject> results;
+            std::shared_ptr<std::vector<DynamicHostObject>> metadata = std::make_shared<std::vector<DynamicHostObject>>();;
+            
+            if(operation != "DELETE") {
+                std::string query = "SELECT * FROM " + tableName + " where rowid = " + std::to_string(rowId) + ";";
+                sqliteExecute(dbName, query, &params, &results, metadata);
+            }
+            
+            invoker->invokeAsync([&rt, results = std::make_shared<std::vector<DumbHostObject>>(results), callback, tableName = std::move(tableName), operation = std::move(operation), &rowId]
+                                 {
+                auto res = jsi::Object(rt);
+                res.setProperty(rt, "table", jsi::String::createFromUtf8(rt, tableName));
+                res.setProperty(rt, "operation", jsi::String::createFromUtf8(rt, operation));
+                res.setProperty(rt, "rowId", jsi::Value(rowId));
+                if(results->size() != 0) {
+                    res.setProperty(rt, "row", jsi::Object::createFromHostObject(rt, std::make_shared<DumbHostObject>(results->at(0))));
+                }
+                
+                callback->asObject(rt).asFunction(rt).call(rt, res);
+                
+            });
+        };
+        
+        registerUpdateHook(dbName, std::move(hook));
+        
+        return {};
+    });
     
     jsi::Object module = jsi::Object(rt);
     
@@ -435,6 +491,7 @@ void install(jsi::Runtime &rt, std::shared_ptr<react::CallInvoker> jsCallInvoker
     module.setProperty(rt, "executeBatch", std::move(executeBatch));
     module.setProperty(rt, "executeBatchAsync", std::move(executeBatchAsync));
     module.setProperty(rt, "loadFile", std::move(loadFile));
+    module.setProperty(rt, "updateHook", std::move(updateHook));
     
     rt.global().setProperty(rt, "__OPSQLiteProxy", std::move(module));
 }