@bullet./paraql 0.2.0 → 0.3.0

package/README.md

@@ -2,16 +2,16 @@
 
 ParaQL is a mad-science experiment combining [libSQL](https://github.com/tursodatabase/libsql) (an open-contribution [SQLite](https://github.com/sqlite/sqlite) fork with native vector support) with [Autobase](https://github.com/holepunchto/autobase) (or to be more precise, it's next-gen iteration [Autobee](https://github.com/holepunchto/autobee)) for massively-parallel multi-writer access.
 
-It logs every write in a per-instance append-only log (oplog) and applies them in deterministic order to a shared view (the database) ensuring no corruption can occur. This is not very fast, however existing writers typically have limited activity and newcomers can simply fast-forward to the latest view, making overall performance acceptable.
+It logs every write in a per-instance append-only log (oplog) and applies them in deterministic order to a shared view (the database) ensuring no corruption can occur. Performance is impressive, it blows vanilla SQLite out of the water, in part due to flushing to disk less often. That's not an issue though, because by the time database write occurs the operation is already in the oplog, so in case crashes or corruption it can simply be reapplied. Peers who haven't written to the database simply fast-forward (download) the latest database version which on a good connection is even faster.
+
+The oplog based design means that disk space used is roughly double (after compaction) that of vanilla SQLite. Operating requirements are considerably higher, because Autobee uses RocksDB as the storage backend and RocksDB doesn't free deleted data immediately as a performance optimization. Periodic compaction, both automatic, and manual, makes this an easily solvable issue.
 
 For a better idea of how ParaQL performs and compares to other solutions see [the benchmark](./BENCHMARK.md).
 
-ParaQL is developed on [Bare](https://github.com/holepunchto/bare) but it should also run on Node. Because Bare is multi-platform, ParaQL should run on recent versions of Android, iOS, macOS, Linux, and Windows.
+ParaQL is developed on [Bare](https://github.com/holepunchto/bare) but it's also tested on Node. Because Bare is multi-platform and supports mobile operating systems as first class citizens, ParaQL runs on recent versions of Android, iOS, macOS, Linux, and Windows.
 
 ParaQL supports encrypting the database with a 256-bit key, both on disk and in transport (meaning remote peers need to know the key to read or write into the database). The local-only temporary files are currently encrypted with unique but not random nonces. This is something we're still working on.
 
-ParaQL also supports compression with zlib's deflate algorithm. While this slows down writing considerably, it reduces disk space used by ~3x or more, so depending on your environment it might be worth considering. Do note that compression affects every instance of the same database.
-
 ParaQL has native support for vector data types and vector search functions with optional indexing. This is courtesy of libSQL and one of the primary reasons ParaQL was made: to support vector similarity search in P2P context.
 
 ## Install
@@ -40,6 +40,7 @@ for await (const row of select.iterate()) {
   console.log(row)
 }
 
+await select.finalize()
 await db.close()
 ```
 
@@ -65,8 +66,6 @@ options = {
   keyPair: null,
   encrypted: false,
   encryptionKey: null,
-  compressed: false,
-  compressionLevel: 6,
 }
 ```
 
@@ -95,8 +94,6 @@ Do note that operations initiated by initial instance (root node) and all read-o
 
 If `encrypted` is true and `encryptionKey` is provided as 32-byte buffer, it is used to encrypt the database.
 
-`compressed` enables database compression and `compressionLevel` controls how many resources are used for the compression. It should be between 0 and 9 (inclusive).
-
 ### `db.name`
 
 The name of the database file.
@@ -143,11 +140,11 @@ Revoke write access from another instance of the database. `key` should be the l
 
 ### `db.replicate(isInitiatorOrStream)`
 
-Creates a replication stream that can be piped over any streamable transport. `isInitiatorOrStream` can be a boolean indicating whether this instance initiated Noise handshake or another replication stream.
+Creates a replication stream that can be piped over any streamable transport. `isInitiatorOrStream` can be a boolean indicating whether this instance initiated Noise handshake, or another replication stream.
 
 ### `await db.compact()`
 
-Compacts database and removes stale data. This operation is local only and can reduce disk space usage by up to 15x or more depending on the data stored and settings of the database. You should run this periodically when idle.
+Compacts database and removes stale data. This operation is local only and can reduce disk space usage by up to 30x or more depending on the data stored and settings of the database. Compaction happens automatically but you might want to run this periodically, when idle.
 
 ### `const info = await db.info()`
 
@@ -193,12 +190,6 @@ Same as `stmt.all()` except it only returns the first row.
 
 Execute a statement with given params and return an object in the form `{ changes: number, lastInsertRowid: number }`.
 
-### `await stmt.batch(...params)`
-
-Execute a statement with batches of bind params as a single operation.
-
-When doing large writes and params are known in advance this can speed up sync by an order of magnitude or more.
-
 ### `for await (const row of stmt.iterate(...params))`
 
 Execute a statement with given params and return rows one by one.

package/lib/codecs.js

@@ -1,6 +1,5 @@
 const c = require("compact-encoding")
 const IndexEncoder = require("index-encoder")
-const zlib = require("zlib")
 
 const { OPERATION, PAGE_SIZE } = require("./constants")
 
@@ -96,25 +95,3 @@ module.exports.Operation = {
     return c.decode(op, operation)
   },
 }
-
-module.exports.deflate = async function (buffer, options) {
-  return new Promise((resolve, reject) =>
-    zlib.deflate(buffer, { chunkSize: PAGE_SIZE, ...options }, (err, result) => {
-      if (err) {
-        return reject(result)
-      }
-      resolve(result)
-    }),
-  )
-}
-
-module.exports.inflate = async function (buffer, options) {
-  return new Promise((resolve, reject) =>
-    zlib.inflate(buffer, { chunkSize: PAGE_SIZE, ...options }, (err, result) => {
-      if (err) {
-        return reject(result)
-      }
-      resolve(result)
-    }),
-  )
-}

package/lib/statement.d.ts

@@ -3,7 +3,6 @@ import Buffer from "bare-buffer"
 
 interface ParaQLStatement extends ReadyResource {
   readonly sourceSQL: string
-  readonly batching: boolean
 
   finalize(): Promise<void>
 
@@ -17,8 +16,6 @@ interface ParaQLStatement extends ReadyResource {
 
   run(...params: ParaQLStatement.Parameters): Promise<ParaQLStatement.RunResult | undefined>
 
-  batch(...params: ParaQLStatement.Parameters[]): Promise<void>
-
   iterate<T extends ParaQLStatement.Row = ParaQLStatement.Row>(
     ...params: ParaQLStatement.Parameters
   ): AsyncIterableIterator<T>

package/lib/statement.js

@@ -99,18 +99,6 @@ module.exports = class ParaQLStatement extends ReadyResource {
     return this._db._vfs.run(this._sourceSQL, named, positional)
   }
 
-  async batch(...params) {
-    if (this.closed) throw errors.ALREADY_CLOSED()
-
-    const batch = params.map((p) => {
-      const [named, positional] = splitParameters(p)
-
-      return { sql: this._sourceSQL, named, positional }
-    })
-
-    return this._db._vfs.batch(batch)
-  }
-
   async *iterate(...params) {
     if (this.closed) throw errors.ALREADY_CLOSED()
 

package/lib/vfs.js

@@ -6,7 +6,7 @@ const ReadyResource = require("ready-resource")
 const RocksDB = require("rocksdb-native")
 
 const binding = require("../binding")
-const { PageKey, Operation, deflate, inflate } = require("./codecs")
+const { PageKey, Operation } = require("./codecs")
 const { OPERATION, PAGE_SIZE } = require("./constants")
 const Deferred = require("./deferred")
 const Encryption = require("./encryption")
@@ -19,8 +19,6 @@ module.exports = class ParaVFS extends ReadyResource {
       keyPair = null,
       encrypted = false,
       encryptionKey = null,
-      compressed = false,
-      compressionLevel = 6,
     } = options
 
     super()
@@ -30,8 +28,6 @@ module.exports = class ParaVFS extends ReadyResource {
     this._name = name
 
     this._encryption = null
-    this._compressed = compressed
-    this._compressionLevel = compressionLevel
 
     this.store = store.namespace(name)
     this.bee = new Autobee(this.store, key, {
@@ -76,10 +72,6 @@ module.exports = class ParaVFS extends ReadyResource {
     return !!this._encryption
   }
 
-  get compressed() {
-    return !!this._compressed
-  }
-
   async _open() {
     await this.bee.ready()
 
@@ -114,7 +106,7 @@ module.exports = class ParaVFS extends ReadyResource {
 
   async _apply(nodes, view, host) {
     for (const node of nodes) {
-      const op = Operation.decode(this.compressed ? await inflate(node.value) : node.value)
+      const op = Operation.decode(node.value)
 
       switch (op.type) {
         case OPERATION.DESERIALIZE: {
@@ -180,23 +172,11 @@ module.exports = class ParaVFS extends ReadyResource {
   }
 
   async addWriter(key) {
-    await this.bee.append(
-      this.compressed
-        ? await deflate(Operation.encode({ type: OPERATION.WRITER_ADD, key }), {
-            level: this._compressionLevel,
-          })
-        : Operation.encode({ type: OPERATION.WRITER_ADD, key }),
-    )
+    await this.bee.append(Operation.encode({ type: OPERATION.WRITER_ADD, key }))
   }
 
   async removeWriter(key) {
-    await this.bee.append(
-      this.compressed
-        ? await deflate(Operation.encode({ type: OPERATION.WRITER_DEL, key }), {
-            level: this._compressionLevel,
-          })
-        : Operation.encode({ type: OPERATION.WRITER_DEL, key }),
-    )
+    await this.bee.append(Operation.encode({ type: OPERATION.WRITER_DEL, key }))
   }
 
   replicate(isInitiator) {
@@ -240,11 +220,7 @@ module.exports = class ParaVFS extends ReadyResource {
     this._interactive = new Deferred()
     try {
       const promise = await this.bee.append(
-        this.compressed
-          ? await deflate(Operation.encode({ type: OPERATION.DESERIALIZE, data }), {
-              level: this._compressionLevel,
-            })
-          : Operation.encode({ type: OPERATION.DESERIALIZE, data }),
+        Operation.encode({ type: OPERATION.DESERIALIZE, data }),
       )
       return await this._interactive.promise
         .then(() => promise)
@@ -260,13 +236,7 @@ module.exports = class ParaVFS extends ReadyResource {
   async exec(sql) {
     this._interactive = new Deferred()
     try {
-      const promise = this.bee.append(
-        this.compressed
-          ? await deflate(Operation.encode({ type: OPERATION.EXEC, sql }), {
-              level: this._compressionLevel,
-            })
-          : Operation.encode({ type: OPERATION.EXEC, sql }),
-      )
+      const promise = this.bee.append(Operation.encode({ type: OPERATION.EXEC, sql }))
       await this._interactive.promise
         .then(() => promise)
         .catch(async (err) => {
@@ -282,11 +252,7 @@ module.exports = class ParaVFS extends ReadyResource {
     this._interactive = new Deferred()
     try {
       const promise = this.bee.append(
-        this.compressed
-          ? await deflate(Operation.encode({ type: OPERATION.RUN, sql, named, positional }), {
-              level: this._compressionLevel,
-            })
-          : Operation.encode({ type: OPERATION.RUN, sql, named, positional }),
+        Operation.encode({ type: OPERATION.RUN, sql, named, positional }),
       )
       return await this._interactive.promise
         .then(async (result) => {
@@ -302,20 +268,6 @@ module.exports = class ParaVFS extends ReadyResource {
     }
   }
 
-  async batch(batch) {
-    await this.bee.append(
-      this.compressed
-        ? await Promise.all(
-            batch.map((op) =>
-              deflate(Operation.encode({ type: OPERATION.RUN, ...op }), {
-                level: this._compressionLevel,
-              }),
-            ),
-          )
-        : batch.map((op) => Operation.encode({ type: OPERATION.RUN, ...op })),
-    )
-  }
-
   async _get(name, index) {
     const tmp = name !== this.name
     const key = PageKey.encode([name, index])
@@ -333,10 +285,6 @@ module.exports = class ParaVFS extends ReadyResource {
       value = entry?.value ?? null
     }
 
-    if (value && this.compressed) {
-      value = await inflate(value)
-    }
-
     return value
   }
 
@@ -364,10 +312,6 @@ module.exports = class ParaVFS extends ReadyResource {
 
     this._files.set(name, batch)
 
-    if (this.compressed) {
-      value = await deflate(value, { level: this._compressionLevel })
-    }
-
     if (tmp && this.encrypted) {
       value = this._encryption.encrypt(value, name, index)
     }

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@bullet./paraql",
-  "version": "0.2.0",
-  "description": "Parallel multi-writer relational database with SQL syntax. Native support for encryption, compression, and vector search.",
+  "version": "0.3.0",
+  "description": "Fast, parallel, multi-writer relational database with SQL syntax. Native support for encryption, compression, and vector search.",
   "author": {
     "name": "Tomas Ravinskas",
     "email": "tomas@tomasrav.me",
@@ -49,10 +49,6 @@
     "path": {
       "bare": "bare-path",
       "default": "path"
-    },
-    "zlib": {
-      "bare": "bare-zlib",
-      "default": "zlib"
     }
   },
   "homepage": "https://github.com/getbullet-app/paraql",
@@ -72,11 +68,10 @@
     "vector-search"
   ],
   "dependencies": {
-    "autobee": "^1.0.10",
+    "autobee": "^2.7.2",
     "bare-buffer": "^3.6.2",
     "bare-fs": "^4.7.4",
     "bare-path": "^3.1.1",
-    "bare-zlib": "^1.4.1",
     "compact-encoding": "^3.3.0",
     "index-encoder": "^3.5.0",
     "ready-resource": "^1.2.0",