@op-engineering/op-sqlite 2.0.1 → 2.0.3

package/README.md

@@ -22,6 +22,8 @@ You can find the [benchmarking code in the example app](https://github.com/OP-En
 
 Memory consumption is also 1/4 compared to `react-native-quick-sqlite`. This query used to take 1.2 GB of peak memory usage, and now runs in 250mbs.
 
+You can also turn on [Memory Mapping](#speed) to make your queries even faster by skipping the kernel during I/O and potentially reduce RAM usage, this comes with some disadvantages though. If you want even more speed and you can re-use your queries you can use [Prepared Statements](#prepared-statements).
+
 # Encryption
 
 If you need to encrypt your entire database, there is [`op-sqlcipher`](https://github.com/OP-Engineering/op-sqlcipher), which is a fork of this library that uses [SQLCipher](https://github.com/sqlcipher/sqlcipher). It completely encrypts the database with minimal overhead.
@@ -102,6 +104,25 @@ const largeDb = open({
 });
 ```
 
+# Speed
+
+op-sqlite is already the fastest solution it can be, but it doesn't mean you cannot tweak SQLite to be faster (at the cost of some disadvantages). One possible tweak is turning on [Memory Mapping](https://www.sqlite.org/mmap.html). It allows to read/write to/from the disk without going through the kernel. However, if your queries throw an error your application might crash.
+
+To turn on Memory Mapping, execute the following pragma statement after opening a db:
+
+```ts
+const db = open({
+  name: 'mydb.sqlite',
+});
+
+// 0 turns of memory mapping, any other number enables it with the cache size
+db.execute('PRAGMA mmap_size=268435456');
+```
+
+If you use prepared statements plus memory mapping, you can get to inches of MMKV for the most performance critical queries, here is a simple example writing/reading a single value.
+
+![mmkv comparison](mmkv.png)
+
 # API
 
 ```typescript
@@ -134,7 +155,7 @@ db = {
 }
 ```
 
-### Simple queries
+## Simple queries
 
 The basic query is **synchronous**, it will block rendering on large operations, further below you will find async versions.
 
@@ -162,7 +183,7 @@ try {
 }
 ```
 
-### Multiple statements in a single string
+## Multiple statements in a single string
 
 You can execute multiple statements in a single operation. The API however is not really thought for this use case and the results (and their metadata) will be mangled, so you can discard it.
 
@@ -186,7 +207,7 @@ let t2name = db.execute(
 console.log(t2name.rows?._array[0].name); // outputs "T2"
 ```
 
-### Transactions
+## Transactions
 
 Throwing an error inside the callback will ROLLBACK the transaction.
 
@@ -215,7 +236,7 @@ await db.transaction('myDatabase', (tx) => {
 });
 ```
 
-### Batch operation
+## Batch operation
 
 Batch execution allows the transactional execution of a set of commands
 
@@ -232,7 +253,7 @@ const res = db.executeSqlBatch('myDatabase', commands);
 console.log(`Batch affected ${result.rowsAffected} rows`);
 ```
 
-### Dynamic Column Metadata
+## Dynamic Column Metadata
 
 In some scenarios, dynamic applications may need to get some metadata information about the returned result set.
 
@@ -253,7 +274,7 @@ metadata.forEach((column) => {
 });
 ```
 
-### Async operations
+## Async operations
 
 You might have too much SQL to process and it will cause your application to freeze. There are async versions for some of the operations. This will offload the SQLite processing to a different thread.
 
@@ -267,7 +288,7 @@ db.executeAsync(
 );
 ```
 
-### Blobs
+## Blobs
 
 Blobs are supported via `ArrayBuffer`, you need to be careful about the semantics though. You cannot instantiate an instance of `ArrayBuffer` directly, nor pass a typed array directly. Here is an example:
 
@@ -295,7 +316,22 @@ const result = db.execute('SELECT content FROM BlobTable');
 const finalUint8 = new Uint8Array(result.rows!._array[0].content);
 ```
 
-### Attach or Detach other databases
+## Prepared statements
+
+A lot of the work when executing queries is not iterating through the result set itself but, sometimes, planning the execution. If you have a query which is expensive but you can re-use (even if you have to change the arguments) you can use a `prepared statement`:
+
+```ts
+const statement = db.prepareStatement('SELECT * FROM User WHERE name = ?;');
+statement.bind(['Oscar']);
+let results1 = statement.execute();
+
+statement.bind(['Carlos']);
+let results2 = statement.execute();
+```
+
+You only pay the price of parsing the query once, and each subsequent execution should be faster.
+
+# Attach or Detach other databases
 
 SQLite supports attaching or detaching other database files into your main database connection through an alias.
 You can do any operation you like on this attached database like JOIN results across tables in different schemas, or update data or objects.
@@ -322,7 +358,7 @@ if (!detachResult.status) {
 }
 ```
 
-### Loading SQL Dump Files
+# Loading SQL Dump Files
 
 If you have a SQL dump file, you can load it directly, with low memory consumption:
 
@@ -334,7 +370,7 @@ const { rowsAffected, commands } = db
   });
 ```
 
-## Hooks
+# Hooks
 
 You can subscribe to changes in your database by using an update hook:
 
@@ -396,7 +432,7 @@ db.commitHook(null);
 db.rollbackHook(null);
 ```
 
-## Use built-in SQLite
+# Use built-in SQLite
 
 On iOS you can use the embedded SQLite, when running `pod-install` add an environment flag:
 
@@ -406,11 +442,11 @@ OP_SQLITE_USE_PHONE_VERSION=1 npx pod-install
 
 On Android, it is not possible to link the OS SQLite. It is also a bad idea due to vendor changes, old android bugs, etc. Unfortunately, this means this library will add some megabytes to your app size.
 
-## Enable compile-time options
+# Enable compile-time options
 
 By specifying pre-processor flags, you can enable optional features like FTS5, Geopoly, etc.
 
-### iOS
+## iOS
 
 Add a `post_install` block to your `<PROJECT_ROOT>/ios/Podfile` like so:
 
@@ -429,7 +465,7 @@ end
 Replace the `<SQLITE_FLAGS>` part with the flags you want to add.
 For example, you could add `SQLITE_ENABLE_FTS5=1` to `GCC_PREPROCESSOR_DEFINITIONS` to enable FTS5 in the iOS project.
 
-### Android
+## Android
 
 You can specify flags via `<PROJECT_ROOT>/android/gradle.properties` like so:
 
@@ -437,14 +473,18 @@ You can specify flags via `<PROJECT_ROOT>/android/gradle.properties` like so:
 OPSQLiteFlags="-DSQLITE_ENABLE_FTS5=1"
 ```
 
-## Additional configuration
+# Additional configuration
 
-### App groups (iOS only)
+## App groups (iOS only)
 
 On iOS, the SQLite database can be placed in an app group, in order to make it accessible from other apps in that app group. E.g. for sharing capabilities.
 
 To use an app group, add the app group ID as the value for the `OPSQLite_AppGroup` key in your project's `Info.plist` file. You'll also need to configure the app group in your project settings. (Xcode -> Project Settings -> Signing & Capabilities -> Add Capability -> App Groups)
 
-## License
+# Contribute
+
+You need to have clang-format installed (`brew install clang-format`)
+
+# License
 
 MIT License.

package/android/CMakeLists.txt

@@ -29,8 +29,10 @@ add_library(
   ../cpp/ThreadPool.cpp
   ../cpp/sqlbatchexecutor.h
   ../cpp/sqlbatchexecutor.cpp
-  ../cpp/DynamicHostObject.cpp
-  ../cpp/DynamicHostObject.h
+  ../cpp/SmartHostObject.cpp
+  ../cpp/SmartHostObject.h
+  ../cpp/PreparedStatementHostObject.h
+  ../cpp/PreparedStatementHostObject.cpp
   ../cpp/DumbHostObject.cpp
   ../cpp/DumbHostObject.h
   ../cpp/macros.h

package/cpp/DumbHostObject.cpp

@@ -1,37 +1,42 @@
 #include "DumbHostObject.h"
+#include "SmartHostObject.h"
 #include "utils.h"
 #include <iostream>
 
 namespace opsqlite {
 
-    namespace jsi = facebook::jsi;
-
-    DumbHostObject::DumbHostObject(std::shared_ptr<std::vector<DynamicHostObject>> metadata) {
-        this->metadata = metadata;
-    };
-
-    std::vector<jsi::PropNameID> DumbHostObject::getPropertyNames(jsi::Runtime &rt) {
-        std::vector<jsi::PropNameID> keys;
-        
-        for (auto field : *metadata) {
-            // TODO improve this by generating the propName once on metadata creation
-            keys.push_back(jsi::PropNameID::forAscii(rt, std::get<std::string>(field.fields[0].second)));
-        }
-        
-        return keys;
-    }
+namespace jsi = facebook::jsi;
+
+DumbHostObject::DumbHostObject(
+    std::shared_ptr<std::vector<SmartHostObject>> metadata) {
+  this->metadata = metadata;
+};
+
+std::vector<jsi::PropNameID>
+DumbHostObject::getPropertyNames(jsi::Runtime &rt) {
+  std::vector<jsi::PropNameID> keys;
+
+  for (auto field : *metadata) {
+    // TODO improve this by generating the propName once on metadata creation
+    keys.push_back(jsi::PropNameID::forAscii(
+        rt, std::get<std::string>(field.fields[0].second)));
+  }
 
-    jsi::Value DumbHostObject::get(jsi::Runtime &rt, const jsi::PropNameID &propNameID) {
-        auto name = propNameID.utf8(rt);
-        auto fields = metadata.get();
-        for(int i = 0; i < fields->size(); i++) {
-            auto fieldName = std::get<std::string>(fields->at(i).fields[0].second);
-            if(fieldName == name) {
-                return toJSI(rt, values.at(i));
-            }
-        }
-
-        return {};
+  return keys;
+}
+
+jsi::Value DumbHostObject::get(jsi::Runtime &rt,
+                               const jsi::PropNameID &propNameID) {
+  auto name = propNameID.utf8(rt);
+  auto fields = metadata.get();
+  for (int i = 0; i < fields->size(); i++) {
+    auto fieldName = std::get<std::string>(fields->at(i).fields[0].second);
+    if (fieldName == name) {
+      return toJSI(rt, values.at(i));
     }
+  }
 
+  return {};
 }
+
+} // namespace opsqlite

package/cpp/DumbHostObject.h

@@ -3,31 +3,31 @@
 
 #include <stdio.h>
 
-#include <jsi/jsi.h>
+#include "SmartHostObject.h"
+#include "types.h"
 #include <any>
+#include <jsi/jsi.h>
 #include <vector>
-#include "types.h"
-#include "DynamicHostObject.h"
 
 namespace opsqlite {
 
-    namespace jsi = facebook::jsi;
+namespace jsi = facebook::jsi;
+
+class JSI_EXPORT DumbHostObject : public jsi::HostObject {
+public:
+  DumbHostObject(){};
+
+  DumbHostObject(std::shared_ptr<std::vector<SmartHostObject>> metadata);
 
-    class JSI_EXPORT DumbHostObject: public jsi::HostObject {
-    public:
-        DumbHostObject() {};
-        
-        DumbHostObject(std::shared_ptr<std::vector<DynamicHostObject>> metadata);
+  std::vector<jsi::PropNameID> getPropertyNames(jsi::Runtime &rt);
 
-        std::vector<jsi::PropNameID> getPropertyNames(jsi::Runtime &rt);
+  jsi::Value get(jsi::Runtime &rt, const jsi::PropNameID &propNameID);
 
-        jsi::Value get(jsi::Runtime &rt, const jsi::PropNameID &propNameID);
+  std::vector<JSVariant> values;
 
-        std::vector<JSVariant> values;
-        
-        std::shared_ptr<std::vector<DynamicHostObject>> metadata;
-    };
+  std::shared_ptr<std::vector<SmartHostObject>> metadata;
+};
 
-}
+} // namespace opsqlite
 
 #endif /* DumbHostObject_h */

package/cpp/PreparedStatementHostObject.cpp

@@ -0,0 +1,88 @@
+//
+//  PreparedStatementHostObject.cpp
+//  op-sqlite
+//
+//  Created by Oscar Franco on 5/12/23.
+//
+
+#include "PreparedStatementHostObject.h"
+#include "bridge.h"
+#include "macros.h"
+#include "utils.h"
+
+namespace opsqlite {
+
+namespace jsi = facebook::jsi;
+
+PreparedStatementHostObject::PreparedStatementHostObject(
+    std::string dbName, sqlite3_stmt *statementPtr)
+    : _dbName(dbName), _statement(statementPtr) {}
+
+std::vector<jsi::PropNameID>
+PreparedStatementHostObject::getPropertyNames(jsi::Runtime &rt) {
+  std::vector<jsi::PropNameID> keys;
+
+  //  for (auto field : fields) {
+  //    keys.push_back(jsi::PropNameID::forAscii(rt, field.first));
+  //  }
+
+  return keys;
+}
+
+jsi::Value PreparedStatementHostObject::get(jsi::Runtime &rt,
+                                            const jsi::PropNameID &propNameID) {
+  auto name = propNameID.utf8(rt);
+
+  if (name == "bind") {
+    return HOSTFN("bind", 1) {
+      if (_statement == NULL) {
+        throw std::runtime_error("statement has been freed");
+      }
+
+      std::vector<JSVariant> params;
+
+      const jsi::Value &originalParams = args[0];
+      params = toVariantVec(rt, originalParams);
+
+      std::vector<DumbHostObject> results;
+      std::shared_ptr<std::vector<SmartHostObject>> metadata =
+          std::make_shared<std::vector<SmartHostObject>>();
+
+      sqlite_bind_statement(_statement, &params);
+
+      return {};
+    });
+  }
+
+  if (name == "execute") {
+    return HOSTFN("execute", 1) {
+      if (_statement == NULL) {
+        throw std::runtime_error("statement has been freed");
+      }
+      std::vector<DumbHostObject> results;
+      std::shared_ptr<std::vector<SmartHostObject>> metadata =
+          std::make_shared<std::vector<SmartHostObject>>();
+
+      auto status = sqlite_execute_prepared_statement(_dbName, _statement,
+                                                      &results, metadata);
+
+      if (status.type == SQLiteError) {
+        throw std::runtime_error(status.message);
+      }
+
+      auto jsiResult = createResult(rt, status, &results, metadata);
+      return jsiResult;
+    });
+  }
+
+  return {};
+}
+
+PreparedStatementHostObject::~PreparedStatementHostObject() {
+  if (_statement != NULL) {
+    sqlite3_finalize(_statement);
+    _statement = NULL;
+  }
+}
+
+} // namespace opsqlite

package/cpp/PreparedStatementHostObject.h

@@ -0,0 +1,36 @@
+//
+//  PreparedStatementHostObject.hpp
+//  op-sqlite
+//
+//  Created by Oscar Franco on 5/12/23.
+//
+
+#ifndef PreparedStatementHostObject_h
+#define PreparedStatementHostObject_h
+
+#include <jsi/jsi.h>
+#include <memory>
+#include <sqlite3.h>
+#include <string>
+
+namespace opsqlite {
+namespace jsi = facebook::jsi;
+
+class PreparedStatementHostObject : public jsi::HostObject {
+public:
+  PreparedStatementHostObject(std::string dbName, sqlite3_stmt *statement);
+  virtual ~PreparedStatementHostObject();
+
+  std::vector<jsi::PropNameID> getPropertyNames(jsi::Runtime &rt);
+
+  jsi::Value get(jsi::Runtime &rt, const jsi::PropNameID &propNameID);
+
+private:
+  std::string _dbName;
+  // This shouldn't be de-allocated until sqlite3_finalize is called on it
+  sqlite3_stmt *_statement;
+};
+
+} // namespace opsqlite
+
+#endif /* PreparedStatementHostObject_hpp */

package/cpp/SmartHostObject.cpp

@@ -0,0 +1,33 @@
+#include "SmartHostObject.h"
+#include "utils.h"
+
+namespace opsqlite {
+
+namespace jsi = facebook::jsi;
+
+std::vector<jsi::PropNameID>
+SmartHostObject::getPropertyNames(jsi::Runtime &rt) {
+  std::vector<jsi::PropNameID> keys;
+
+  for (auto field : fields) {
+    keys.push_back(jsi::PropNameID::forAscii(rt, field.first));
+  }
+
+  return keys;
+}
+
+jsi::Value SmartHostObject::get(jsi::Runtime &rt,
+                                const jsi::PropNameID &propNameID) {
+  auto name = propNameID.utf8(rt);
+
+  for (auto field : fields) {
+    auto fieldName = field.first;
+    if (fieldName == name) {
+      return toJSI(rt, field.second);
+    }
+  }
+
+  return {};
+}
+
+} // namespace opsqlite

package/cpp/SmartHostObject.h

@@ -0,0 +1,26 @@
+#ifndef SmartHostObject_h
+#define SmartHostObject_h
+
+#include "types.h"
+#include <any>
+#include <jsi/jsi.h>
+#include <vector>
+
+namespace opsqlite {
+
+namespace jsi = facebook::jsi;
+
+class JSI_EXPORT SmartHostObject : public jsi::HostObject {
+public:
+  SmartHostObject(){};
+
+  std::vector<jsi::PropNameID> getPropertyNames(jsi::Runtime &rt);
+
+  jsi::Value get(jsi::Runtime &rt, const jsi::PropNameID &propNameID);
+
+  std::vector<std::pair<std::string, JSVariant>> fields;
+};
+
+} // namespace opsqlite
+
+#endif /* SmartHostObject_h */