@bullet./paraql 0.1.2 → 0.3.0

package/README.md

@@ -2,35 +2,33 @@
 
 ParaQL is a mad-science experiment combining [libSQL](https://github.com/tursodatabase/libsql) (an open-contribution [SQLite](https://github.com/sqlite/sqlite) fork with native vector support) with [Autobase](https://github.com/holepunchto/autobase) (or to be more precise, it's next-gen iteration [Autobee](https://github.com/holepunchto/autobee)) for massively-parallel multi-writer access.
 
-It logs every write in a per-instance append-only log (oplog) and applies them in deterministic order to a shared view (the database) ensuring no corruption can occur. This is not very fast, however existing writers typically have limited activity and newcomers can simply fast-forward to the latest view (the database), making overall performance acceptable.
+It logs every write in a per-instance append-only log (oplog) and applies them in deterministic order to a shared view (the database) ensuring no corruption can occur. Performance is impressive, it blows vanilla SQLite out of the water, in part due to flushing to disk less often. That's not an issue though, because by the time database write occurs the operation is already in the oplog, so in case crashes or corruption it can simply be reapplied. Peers who haven't written to the database simply fast-forward (download) the latest database version which on a good connection is even faster.
+
+The oplog based design means that disk space used is roughly double (after compaction) that of vanilla SQLite. Operating requirements are considerably higher, because Autobee uses RocksDB as the storage backend and RocksDB doesn't free deleted data immediately as a performance optimization. Periodic compaction, both automatic, and manual, makes this an easily solvable issue.
 
 For a better idea of how ParaQL performs and compares to other solutions see [the benchmark](./BENCHMARK.md).
 
-ParaQL is developed on [Bare](https://github.com/holepunchto/bare) but it should also run on Node. Because Bare is multi-platform, ParaQL should run on recent versions of Android, iOS, macOS, Linux, and Windows.
+ParaQL is developed on [Bare](https://github.com/holepunchto/bare) but it's also tested on Node. Because Bare is multi-platform and supports mobile operating systems as first class citizens, ParaQL runs on recent versions of Android, iOS, macOS, Linux, and Windows.
 
 ParaQL supports encrypting the database with a 256-bit key, both on disk and in transport (meaning remote peers need to know the key to read or write into the database). The local-only temporary files are currently encrypted with unique but not random nonces. This is something we're still working on.
 
-ParaQL also supports compression with zlib's deflate algorithm. While this slows down writing considerably, it reduces disk space used by ~3x or more, so depending on your environment it might be worth considering. Do note that compression affects every instance of the same database.
-
 ParaQL has native support for vector data types and vector search functions with optional indexing. This is courtesy of libSQL and one of the primary reasons ParaQL was made: to support vector similarity search in P2P context.
 
 ## Install
 
 ```shell
-npm i paraql
+npm i @bullet./paraql
 ```
 
 ## Usage
 
 ```javascript
 const Corestore = require("corestore")
-const ParaQL = require("paraql")
+const ParaQL = require("@bullet./paraql")
 
 const store = new Corestore("./paraql")
 const db = new ParaQL(store)
 
-await db.ready()
-
 await db.exec(`
   CREATE TABLE people (id INTEGER PRIMARY KEY, name TEXT);
   INSERT INTO people (name) VALUES ('Alice'), ('Bob');
@@ -42,6 +40,7 @@ for await (const row of select.iterate()) {
   console.log(row)
 }
 
+await select.finalize()
 await db.close()
 ```
 
@@ -62,28 +61,39 @@ Creates a new database or an instance of an existing database if key is provided
 
 ```javascript
 options = {
-  cacheSize: 1024,
   name: "paraql.db",
+  authorize: null,
   keyPair: null,
   encrypted: false,
   encryptionKey: null,
-  compressed: false,
-  compressionLevel: 6,
 }
 ```
 
-`cacheSize` controls how many prepared statements are cached in memory.
-
-The following options should only be used at database creation and set for every instance of the database. Option mismatch will lead to database corruption.
+These options should only be used at database creation and set for every instance of the database. Option mismatch may lead to database corruption and/or sync issues.
 
 `name` is the name of the database file, also used as a prefix for temporary files.
 
+`authorize` is either null or an (asynchronous) authorizer callback with the following signature:
+
+```typescript
+;(
+  key: Buffer,
+  action: ParaQL.AUTHORIZE_ACTION,
+  param1: string | null,
+  param2: string | null,
+  param3: string | null,
+  param4: string | null,
+) => Promise<boolean>
+```
+
+where `key` is the local key of the writer attempting the operation, `action` is numeric action code defined in `ParaQL.AUTHORIZE_ACTION` enum, and params 1 through 4 are [action specific strings (e.g. table name) or null](https://sqlite.org/c3ref/c_alter_table.html).
+
+Do note that operations initiated by initial instance (root node) and all read-only operations are always allowed and authorizer callback is **NOT** invoked for them.
+
 `keyPair`, if provided, is the signing key pair for the local writer in the form `{ publicKey: <32-byte Buffer>, secretKey: <32-byte Buffer> }` .
 
 If `encrypted` is true and `encryptionKey` is provided as 32-byte buffer, it is used to encrypt the database.
 
-`compressed` enables database compression and `compressionLevel` controls how many resources are used for the compression. It should be between 0 and 9 (inclusive).
-
 ### `db.name`
 
 The name of the database file.
@@ -130,11 +140,11 @@ Revoke write access from another instance of the database. `key` should be the l
 
 ### `db.replicate(isInitiatorOrStream)`
 
-Creates a replication stream that can be piped over any streamable transport. `isInitiatorOrStream` can be a boolean indicating whether this instance initiated Noise handshake or another replication stream.
+Creates a replication stream that can be piped over any streamable transport. `isInitiatorOrStream` can be a boolean indicating whether this instance initiated Noise handshake, or another replication stream.
 
 ### `await db.compact()`
 
-Compacts database and removes stale data. This operation is local only and can reduce disk space usage by +3x. You should run this periodically when idle.
+Compacts database and removes stale data. This operation is local only and can reduce disk space usage by up to 30x or more depending on the data stored and settings of the database. Compaction happens automatically but you might want to run this periodically, when idle.
 
 ### `const info = await db.info()`
 
@@ -152,41 +162,19 @@ Get information about disk space usage. Returned object has all properties in by
 
 Execute given SQL statement(s) without checking return values.
 
-This is a convenience method. Unless you need to load a large SQL dump or similar, this method should be avoided as it bloats the disk space requirements.
+This is a convenience method.
 
 ### `const stmt = await db.prepare(sql)`
 
 Prepare SQL statement `stmt` from the first statement in `sql`. If `sql` contains more than one statement tailing statements are discarded.
 
-Prepared statements are cached to improve performance, so calling `db.prepare()` twice with the same `sql` will return previously cached `stmt`.
-
-All prepared statements are finalized when calling `db.close()`.
-
 ### `stmt.sourceSQL`
 
 The SQL string used to initialize this prepared statement.
 
-### `stmt.batching`
-
-Whether this statement is in batching mode.
-
-### `stmt.batch(on)`
-
-Toggle batching mode, `on` is a boolean.
-
-When in batching mode `stmt.run()` queries are queued up in memory and only executed when calling `stmt.flush()`. This can greatly speed up both write and sync performance. However since they aren't executed immediately, you can't query the data until you flush it.
-
-### `const result = await stmt.flush()`
-
-Flush queued up queries writing them to disk.
-
-Returns an object in the form `{ changes: number, lastInsertRowid: number, errors: number }`.
-
 ### `await stmt.finalize()`
 
-Finalize a statement freeing up resources used and clearing batching queue without flushing it.
-
-You should generally not need to call this method in normal usage.
+Finalize a statement freeing up resources used.
 
 ### `const rows = await stmt.all(...params)`
 
@@ -198,16 +186,22 @@ Returns an array of row objects keyed by column name.
 
 Same as `stmt.all()` except it only returns the first row.
 
-### `const result = stmt.run(...params)`
+### `const result = await stmt.run(...params)`
 
 Execute a statement with given params and return an object in the form `{ changes: number, lastInsertRowid: number }`.
 
-In batching mode returns `null`.
-
 ### `for await (const row of stmt.iterate(...params))`
 
 Execute a statement with given params and return rows one by one.
 
+### `const buffer = await db.serialize()`
+
+Return an unencrypted, uncompressed, serialized version of the database that can be (re-)used with any SQLite3 compatible application or library.
+
+### `const db = await ParaQL.deserialize(buffer)`
+
+Initiate a new ParaQL instance with any SQLite3 compatible database contained in the buffer.
+
 ## License
 
 Apache-2.0