picovolt 1.3.0 → 1.5.0

package/README.md

@@ -1,7 +1,7 @@
 # PicoVolt (PVDB)
 
 [![CI](https://github.com/MiniJe/picovolt/actions/workflows/ci.yml/badge.svg)](https://github.com/MiniJe/picovolt/actions/workflows/ci.yml)
-[![Version](https://img.shields.io/badge/version-1.1.0-blue.svg)](CHANGELOG.md)
+[![Version](https://img.shields.io/badge/version-1.5.0-blue.svg)](CHANGELOG.md)
 [![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
 ![Status: 1.0 stable](https://img.shields.io/badge/status-1.0%20stable-brightgreen.svg)
 [![GitHub stars](https://img.shields.io/github/stars/MiniJe/picovolt?style=social)](https://github.com/MiniJe/picovolt)
@@ -28,7 +28,7 @@ cache efficiency.
 
 ## Status
 
-The engine is built out across four phases, all implemented, with 103 unit and
+The engine is built out across four phases, all implemented, with over 180 unit and
 integration tests plus doctests passing and a clean `cargo clippy -D warnings` on
 Linux and Windows. Changes are tracked in [CHANGELOG.md](CHANGELOG.md).
 
@@ -52,11 +52,11 @@ Linux and Windows. Changes are tracked in [CHANGELOG.md](CHANGELOG.md).
 | [`storage/compress.rs`](src/storage/compress.rs) | Delta-Z, LEB128 varints, dictionary bit-packing |
 | [`storage/index.rs`](src/storage/index.rs) | in-memory ordered secondary index (value to record addresses; point and range) |
 | [`storage/record.rs`](src/storage/record.rs) | row and record-body serialization with CAS interception |
-| [`storage/vle.rs`](src/storage/vle.rs) | dev directory store, prod mmap monolith, `bake` |
+| [`storage/vle.rs`](src/storage/vle.rs) | dev directory store, owned prod snapshot, streamed reads, `bake` |
 | [`engine/mvcc.rs`](src/engine/mvcc.rs) | transaction clock and snapshot visibility |
 | [`engine/wasm.rs`](src/engine/wasm.rs) | sandboxed `wasmi` extension runtime and the `WasmExec` backend trait |
 | [`engine/interp.rs`](src/engine/interp.rs) | `pv-wasm`: a from-scratch WASM interpreter (integer subset) |
-| [`engine/query.rs`](src/engine/query.rs) | SQL front-end (CREATE/INSERT/UPDATE/DELETE/DROP, `SELECT` with projection, `AS` aliases, `DISTINCT`, aggregates, `GROUP BY`/`HAVING`, `WHERE` predicates incl. `IN`/`BETWEEN`/`IS NULL`/`LIKE`, `BEFORE`, multi-column `ORDER BY`, `LIMIT`) |
+| [`engine/query.rs`](src/engine/query.rs) | SQL front-end (CREATE/INSERT/UPDATE/DELETE/DROP, `SELECT` with projection, `AS` aliases, `DISTINCT`, aggregates, `GROUP BY`/`HAVING`, `WHERE` predicates incl. `IN`/`BETWEEN`/`IS NULL`/`LIKE`, `BEFORE`, multi-column `ORDER BY`, `LIMIT`/`OFFSET`) |
 | [`engine/compliance.rs`](src/engine/compliance.rs) | optional, app-driven usage-policy hook (not a license requirement) |
 | [`db.rs`](src/db.rs) | the `Database` surface that ties it together |
 | [`ffi.rs`](src/ffi.rs) | C ABI (the `capi` feature): a panic-safe, C-callable surface wrapping the engine for Go, Python, and C bindings |
@@ -89,12 +89,15 @@ Linux and Windows. Changes are tracked in [CHANGELOG.md](CHANGELOG.md).
 - **Hardened against untrusted input.** Opening a `.pvdb` or workspace, or running
   a WASM module, validates manifest hashes (no path traversal), bounds-checks CAS
   offsets and page chains (no out-of-bounds reads or infinite loops on a crafted
-  file), and caps WASM resource counts. The decoders are fuzzed (a cross-platform
+  file), and meters WASM instructions, memory, and output. The decoders are fuzzed (a cross-platform
   fuzz-lite test and a [`fuzz/`](fuzz) cargo-fuzz crate), and `cargo audit`
-  reports no advisories. Both run in CI. See [SECURITY.md](SECURITY.md).
+  currently reports no vulnerability failures. Both run in CI. See
+  [SECURITY.md](SECURITY.md).
 
 ## Build
 
+Rust 1.86 or newer is required.
+
 ```sh
 cargo build
 cargo test
@@ -109,14 +112,24 @@ cargo run --release --example repl     # interactive SQL shell (pvsql)
 cargo run --release --example bench    # evaluation harness across modes and workloads
 ```
 
-SQL supported: `CREATE TABLE`, `CREATE INDEX ON t (col)`, `INSERT`,
-`UPDATE ... SET ... WHERE`, `DELETE ... WHERE`, `DROP TABLE`, and
+Install the first-class CLI with `cargo install picovolt`, then use `pv query`,
+`pv inspect`, `pv history`, `pv import`, `pv export`, and `pv bake`. Copyable
+Rust, Python, Go, Node, and browser projects are in [`starters/`](starters/README.md); supported adapters
+are catalogued in [`docs/INTEGRATIONS.md`](docs/INTEGRATIONS.md).
+
+SQL supported: `CREATE TABLE [IF NOT EXISTS]` with `PRIMARY KEY`, `UNIQUE`, and `NOT NULL`,
+`CREATE [UNIQUE] INDEX ON t (col)`, single- and multi-row `INSERT`,
+`UPDATE ... SET ... WHERE`, `DELETE ... WHERE`, `DROP TABLE [IF EXISTS]`, and
 `SELECT [DISTINCT] {* | col [AS alias], ... | COUNT/SUM/MIN/MAX/AVG(...) [AS alias]}
 FROM t [WHERE <pred>] [GROUP BY cols] [HAVING <pred>] [BEFORE tx]
-[ORDER BY col [ASC|DESC], ...] [LIMIT n]`, where `<pred>` combines
+[ORDER BY col [ASC|DESC], ...] [LIMIT n] [OFFSET n]`, where `<pred>` combines
 `col <op> value` (`=`, `!=`, `<`, `<=`, `>`, `>=`, `LIKE`, `NOT LIKE`),
 `col [NOT] IN (...)`, `col [NOT] BETWEEN a AND b`, and `col IS [NOT] NULL` with
 `AND`, `OR`, and parentheses. Integer and decimal values compare by magnitude.
+Two-table equality `INNER`/`LEFT JOIN` queries support qualified references,
+projection, aliases, `DISTINCT`, filters, ordering, and pagination. Rust callers
+can cache `Database::prepare(...)` templates and use
+atomic `Database::transaction(...)` closures with in-memory databases.
 Durability is selectable via `Database::set_durability` (`Fast` OS-cache default,
 or crash-safe `Sync` with fsync and an atomic manifest).
 
@@ -126,9 +139,9 @@ around 33k rows/s, linear), larger-than-RAM reads through a bounded buffer pool
 667-page dataset serves from a 16-page pool), ordered secondary indexes (point
 lookups roughly 11,000 times faster than a scan, plus range predicates), MVCC
 time-travel, opt-in crash-safe durability (`Durability::Sync`), and a fast
-compile-and-publish path (CAS dedup, columnar compression, single-file mmap
-artifacts). Current limits: indexes are in-memory (rebuilt on open) and there is
-no concurrency.
+compile-and-publish path (CAS dedup, columnar compression, memory-mappable
+single-file artifacts). Current limits include no filesystem `BEGIN`/`COMMIT`
+transactions, only basic two-table equality joins, and no concurrent writers.
 
 ## Install and distribution
 
@@ -136,10 +149,12 @@ no concurrency.
 |--------|-----|
 | **Rust** (crates.io) | `cargo add picovolt` |
 | **JavaScript / npm** (WebAssembly, browser and Node) | `npm install picovolt` |
-| **C / Go / Python** (native, via the C ABI) | `cargo build --release --features capi`, then see [`bindings/`](bindings) |
+| **Python** (native wheels) | `python -m pip install picovolt` (after the first PyPI release) |
+| **C / Go** (native, via the C ABI) | `cargo build --release --features capi`, then see [`bindings/`](bindings) |
 | **In-memory** (native, no filesystem) | `Database::open_memory()`, export with `bake_to_bytes()` |
 
-PicoVolt runs in the browser through its in-memory backend. Build the WebAssembly
+PicoVolt runs in the browser through its in-memory backend plus an OPFS persistence
+wrapper and Web Worker endpoint. Build the WebAssembly
 package with `wasm-pack build --target bundler --release -- --features wasm`, then
 `import { Db } from "picovolt"` and run SQL with `db.query(...)`. See
 [src/wasm_api.rs](src/wasm_api.rs) for the JavaScript surface.
@@ -155,9 +170,9 @@ All bindings accept positional `?` parameters
 a familiar surface, drop-in adapters are provided: a `better-sqlite3`-style
 JavaScript API (`import Database from "picovolt/sqlite"`), a Python DB-API 2.0
 module (`import picovolt.dbapi2 as sqlite`), and the Go `database/sql` driver
-([`bindings/go/pvsql`](bindings/go/pvsql)). Shared limits across all of them:
-positional `?` only, no SQL transactions, no JOINs, and `CREATE TABLE` takes
-column names only.
+([`bindings/go/pvsql`](bindings/go/pvsql)). Shared limits include positional `?`
+only and the intentionally compact SQL grammar; JavaScript and in-memory Rust
+also expose rollback-capable transaction wrappers.
 
 ## Server mode
 
@@ -169,13 +184,22 @@ thread over a channel, so the single-threaded core is unchanged.
 ```sh
 cargo build --release --features server
 ./target/release/picovolt-server --memory --addr 127.0.0.1:8080
-curl -s localhost:8080/v1/query -d '{"sql":"SELECT 1 + 1","params":[]}'
+curl -s localhost:8080/v1/query \
+  -H 'Content-Type: application/json' \
+  -d '{"sql":"CREATE TABLE demo (value)","params":[]}'
 ```
 
-Endpoints are `POST /v1/query`, `GET /v1/tx`, and `GET /v1/health`. There is no
-authentication or TLS, so run it behind a reverse proxy. See
+Endpoints are `POST /v1/query`, `GET /v1/tx`, and `GET /v1/health`. Loopback use
+may omit authentication. A non-loopback bind is refused unless a bearer token is
+provided with `--token-file` or `PICOVOLT_SERVER_TOKEN`; send it as
+`Authorization: Bearer ...`. Query bodies, queues, execution time, rows scanned,
+result rows, and response size are bounded. TLS is not built in, so network
+deployments still belong behind a TLS-terminating reverse proxy. See
 [src/bin/server.rs](src/bin/server.rs).
 
+Applications accepting SQL from users can also call `Database::query_with_limits`
+directly and choose their own scan, result, memory, and deadline budgets.
+
 ## Extending PicoVolt
 
 There are two extension paths: sandboxed WebAssembly user-defined functions, and
@@ -187,6 +211,8 @@ native modules built on the public API. Both are documented in
 | | |
 |--|--|
 | Roadmap | [ROADMAP.md](ROADMAP.md) |
+| One-million-download plan | [docs/ROADMAP_1M_DOWNLOADS.md](docs/ROADMAP_1M_DOWNLOADS.md) |
+| Platform and file support | [docs/SUPPORT.md](docs/SUPPORT.md) |
 | Contributing | [CONTRIBUTING.md](CONTRIBUTING.md) |
 | Code of conduct | [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) |
 | Changelog | [CHANGELOG.md](CHANGELOG.md) |

package/browser.js

@@ -0,0 +1,40 @@
+// Durable browser helper backed by the Origin Private File System (OPFS).
+import { Db } from "./picovolt.js";
+
+export class PersistentDb {
+  constructor(name, db) {
+    this.name = name;
+    this.db = db;
+  }
+
+  static async open(name = "picovolt.pvdb") {
+    if (!globalThis.navigator?.storage?.getDirectory) {
+      throw new Error("PicoVolt OPFS persistence is unavailable in this browser/context");
+    }
+    const root = await navigator.storage.getDirectory();
+    const handle = await root.getFileHandle(name, { create: true });
+    const file = await handle.getFile();
+    const bytes = new Uint8Array(await file.arrayBuffer());
+    const db = bytes.length ? Db.fromBytes(bytes) : new Db();
+    return new PersistentDb(name, db);
+  }
+
+  query(sql, params) {
+    const json = params === undefined ? this.db.query(sql) : this.db.query(sql, params);
+    return JSON.parse(json);
+  }
+
+  async save() {
+    const root = await navigator.storage.getDirectory();
+    const handle = await root.getFileHandle(this.name, { create: true });
+    const writable = await handle.createWritable();
+    await writable.write(this.db.export());
+    await writable.close();
+  }
+
+  async close() {
+    await this.save();
+  }
+}
+
+export default PersistentDb;

package/package.json

@@ -2,7 +2,7 @@
   "name": "picovolt",
   "type": "module",
   "description": "PicoVolt (PVDB): a polymorphic embedded database engine in Rust.",
-  "version": "1.3.0",
+  "version": "1.5.0",
   "license": "Apache-2.0",
   "repository": {
     "type": "git",
@@ -13,7 +13,9 @@
     "picovolt.js",
     "picovolt_bg.js",
     "picovolt.d.ts",
-    "sqlite.js"
+    "sqlite.js",
+    "browser.js",
+    "worker.js"
   ],
   "main": "picovolt.js",
   "types": "picovolt.d.ts",
@@ -30,6 +32,8 @@
   ],
   "exports": {
     ".": "./picovolt.js",
-    "./sqlite": "./sqlite.js"
+    "./sqlite": "./sqlite.js",
+    "./browser": "./browser.js",
+    "./worker": "./worker.js"
   }
 }

package/picovolt_bg.js

@@ -135,33 +135,33 @@ export class Db {
     }
 }
 if (Symbol.dispose) Db.prototype[Symbol.dispose] = Db.prototype.free;
-export function __wbg___wbindgen_boolean_get_fa956cfa2d1bd751(arg0) {
+export function __wbg___wbindgen_boolean_get_c9c83ebd41b34df3(arg0) {
     const v = arg0;
     const ret = typeof(v) === 'boolean' ? v : undefined;
     return isLikeNone(ret) ? 0xFFFFFF : ret ? 1 : 0;
 }
-export function __wbg___wbindgen_debug_string_c25d447a39f5578f(arg0, arg1) {
+export function __wbg___wbindgen_debug_string_a57024b9c6e4a48b(arg0, arg1) {
     const ret = debugString(arg1);
     const ptr1 = passStringToWasm0(ret, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc);
     const len1 = WASM_VECTOR_LEN;
     getDataViewMemory0().setInt32(arg0 + 4 * 1, len1, true);
     getDataViewMemory0().setInt32(arg0 + 4 * 0, ptr1, true);
 }
-export function __wbg___wbindgen_is_null_ea9085d691f535d3(arg0) {
+export function __wbg___wbindgen_is_null_7d13f41e1a2d5140(arg0) {
     const ret = arg0 === null;
     return ret;
 }
-export function __wbg___wbindgen_is_undefined_c05833b95a3cf397(arg0) {
+export function __wbg___wbindgen_is_undefined_6cff064c44e0d823(arg0) {
     const ret = arg0 === undefined;
     return ret;
 }
-export function __wbg___wbindgen_number_get_394265ed1e1b84ee(arg0, arg1) {
+export function __wbg___wbindgen_number_get_136b9679cab35cfb(arg0, arg1) {
     const obj = arg1;
     const ret = typeof(obj) === 'number' ? obj : undefined;
     getDataViewMemory0().setFloat64(arg0 + 8 * 1, isLikeNone(ret) ? 0 : ret, true);
     getDataViewMemory0().setInt32(arg0 + 4 * 0, !isLikeNone(ret), true);
 }
-export function __wbg___wbindgen_string_get_b0ca35b86a603356(arg0, arg1) {
+export function __wbg___wbindgen_string_get_d154f1e671052120(arg0, arg1) {
     const obj = arg1;
     const ret = typeof(obj) === 'string' ? obj : undefined;
     var ptr1 = isLikeNone(ret) ? 0 : passStringToWasm0(ret, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc);
@@ -169,14 +169,14 @@ export function __wbg___wbindgen_string_get_b0ca35b86a603356(arg0, arg1) {
     getDataViewMemory0().setInt32(arg0 + 4 * 1, len1, true);
     getDataViewMemory0().setInt32(arg0 + 4 * 0, ptr1, true);
 }
-export function __wbg___wbindgen_throw_344f42d3211c4765(arg0, arg1) {
+export function __wbg___wbindgen_throw_bb96b2010945f0bc(arg0, arg1) {
     throw new Error(getStringFromWasm0(arg0, arg1));
 }
-export function __wbg_call_e3b662382210db98() { return handleError(function (arg0, arg1, arg2, arg3) {
+export function __wbg_call_0f2a9af232c18fd2() { return handleError(function (arg0, arg1, arg2, arg3) {
     const ret = arg0.call(arg1, arg2, arg3);
     return ret;
 }, arguments); }
-export function __wbg_error_a6fa202b58aa1cd3(arg0, arg1) {
+export function __wbg_error_757e9472f8410341(arg0, arg1) {
     let deferred0_0;
     let deferred0_1;
     try {
@@ -187,15 +187,15 @@ export function __wbg_error_a6fa202b58aa1cd3(arg0, arg1) {
         wasm.__wbindgen_free(deferred0_0, deferred0_1, 1);
     }
 }
-export function __wbg_from_13e323c65fc8f464(arg0) {
+export function __wbg_from_74f3d90e0ff11240(arg0) {
     const ret = Array.from(arg0);
     return ret;
 }
-export function __wbg_get_unchecked_6e0ad6d2a41b06f6(arg0, arg1) {
+export function __wbg_get_unchecked_e20b893aeafc3fca(arg0, arg1) {
     const ret = arg0[arg1 >>> 0];
     return ret;
 }
-export function __wbg_instanceof_Uint8Array_309b927aaf7a3fc7(arg0) {
+export function __wbg_instanceof_Uint8Array_f935dbb0aa7cdeed(arg0) {
     let result;
     try {
         result = arg0 instanceof Uint8Array;
@@ -205,11 +205,11 @@ export function __wbg_instanceof_Uint8Array_309b927aaf7a3fc7(arg0) {
     const ret = result;
     return ret;
 }
-export function __wbg_length_1f0964f4a5e2c6d8(arg0) {
+export function __wbg_length_36bd29c6848c2144(arg0) {
     const ret = arg0.length;
     return ret;
 }
-export function __wbg_length_370319915dc99107(arg0) {
+export function __wbg_length_ecfa2c63d3d0d82c(arg0) {
     const ret = arg0.length;
     return ret;
 }
@@ -217,7 +217,7 @@ export function __wbg_new_227d7c05414eb861() {
     const ret = new Error();
     return ret;
 }
-export function __wbg_prototypesetcall_4770620bbe4688a0(arg0, arg1, arg2) {
+export function __wbg_prototypesetcall_de8e0d9553586985(arg0, arg1, arg2) {
     Uint8Array.prototype.set.call(getArrayU8FromWasm0(arg0, arg1), arg2);
 }
 export function __wbg_stack_3b0d974bbf31e44f(arg0, arg1) {

package/sqlite.js

@@ -9,12 +9,14 @@
 //   // [ { id: 1, name: "alice" } ]
 //
 // Limitations: parameters are positional `?` only (named `:id` params are not
-// supported); there are no transactions; blob parameters are unsupported.
+// supported); blob parameters are unsupported.
 
 import { Db } from "./picovolt.js";
 
 function rowToObject(columns, row) {
-  const obj = {};
+  // A null prototype prevents hostile column names such as `__proto__` from
+  // changing the shape or prototype of the returned record.
+  const obj = Object.create(null);
   for (let i = 0; i < columns.length; i++) obj[columns[i]] = row[i];
   return obj;
 }
@@ -63,6 +65,7 @@ class Statement {
 class Database {
   constructor() {
     this._db = new Db();
+    this._inTransaction = false;
   }
 
   prepare(sql) {
@@ -91,8 +94,28 @@ class Database {
     throw new Error("picovolt: pragma is not supported");
   }
 
-  transaction() {
-    throw new Error("picovolt: transactions are not supported");
+  // Wrap a synchronous callback in an atomic unit. A compact PVDB snapshot is
+  // restored if the callback throws, matching better-sqlite3's common pattern.
+  transaction(fn) {
+    if (typeof fn !== "function") throw new TypeError("transaction expects a function");
+    const db = this;
+    function wrapped(...args) {
+      if (db._inTransaction) return fn(...args);
+      const snapshot = db.serialize();
+      db._inTransaction = true;
+      try {
+        return fn(...args);
+      } catch (error) {
+        db._db = Db.fromBytes(snapshot);
+        throw error;
+      } finally {
+        db._inTransaction = false;
+      }
+    }
+    wrapped.deferred = wrapped;
+    wrapped.immediate = wrapped;
+    wrapped.exclusive = wrapped;
+    return wrapped;
   }
 
   close() {

package/worker.js

@@ -0,0 +1,37 @@
+// Module-worker RPC endpoint. Use `new Worker(url, { type: "module" })` and
+// send `{ id, method, ... }`; every response echoes `id` and carries result/error.
+import { PersistentDb } from "./browser.js";
+
+let database;
+
+self.addEventListener("message", async ({ data }) => {
+  const { id, method } = data ?? {};
+  try {
+    let result;
+    switch (method) {
+      case "open":
+        database = await PersistentDb.open(data.name);
+        result = true;
+        break;
+      case "query":
+        if (!database) throw new Error("open the database first");
+        result = database.query(data.sql, data.params);
+        break;
+      case "save":
+        if (!database) throw new Error("open the database first");
+        await database.save();
+        result = true;
+        break;
+      case "close":
+        if (database) await database.close();
+        database = undefined;
+        result = true;
+        break;
+      default:
+        throw new Error(`unknown PicoVolt worker method: ${method}`);
+    }
+    self.postMessage({ id, result });
+  } catch (error) {
+    self.postMessage({ id, error: error instanceof Error ? error.message : String(error) });
+  }
+});