picovolt 1.8.1 → 2.0.0

package/README.md

@@ -3,13 +3,19 @@
 [![CI](https://github.com/MiniJe/picovolt/actions/workflows/ci.yml/badge.svg)](https://github.com/MiniJe/picovolt/actions/workflows/ci.yml)
 [![crates.io](https://img.shields.io/crates/v/picovolt.svg)](https://crates.io/crates/picovolt)
 [![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
-![Status: stable 1.x](https://img.shields.io/badge/status-stable%201.x-brightgreen.svg)
+![Version: 2.0](https://img.shields.io/badge/version-2.0-blue.svg)
 [![GitHub stars](https://img.shields.io/github/stars/MiniJe/picovolt?style=social)](https://github.com/MiniJe/picovolt)
 
-PicoVolt is an embedded database engine written in Rust. Its 1.x public API and
-on-disk format are stable under Semantic Versioning. It is young software and
-has not had an external security audit, so review it and keep backups before
-trusting it with data you cannot regenerate.
+PicoVolt is an embedded database engine written in Rust. **2.0** provides
+concurrent snapshot readers, bounded writer scheduling, and an incremental
+durable commit log. Independent security review and external application trials
+have not been completed. See the [2.0 release ledger](docs/RELEASE_2_0.md)
+for qualification evidence and publication status.
+
+Start with the [2.0 guide for every maintained interface](docs/QUICKSTART_2_0.md)
+for atomic batches, persistence choices, log diagnostics and error recovery.
+The [standalone review prompt](docs/INDEPENDENT_REVIEW_PROMPT.md) defines an
+independent assessment and external trials deferred beyond the 2.0 release.
 
 If PicoVolt is useful to you, consider starring the repository on GitHub. It is
 the simplest way to help others discover the project.
@@ -22,12 +28,13 @@ Virtualization Layer Engine (VLE) that shifts between two on-disk shapes:
 - **Production mode:** a single contiguous, memory-mappable `.pvdb` file produced
   by `pv_bake()`.
 
-New records use a slotted row layout for O(1) appends. Idle pages can be
-transposed into a packed columnar layout for compression and cache efficiency.
+New records use a slotted row layout for O(1) appends. Applications can run
+bounded maintenance steps that transpose immutable non-tail pages into an
+MVCC-preserving columnar layout with packed decimal encoding.
 
 ## Status
 
-The current stable release is exercised by a 240+ test Rust suite plus doctests
+The engine is exercised by Rust unit and integration suites plus doctests
 and maintained-binding integration tests. CI also enforces formatting and
 warning-free Clippy builds on Linux and Windows. Shipped changes are tracked in
 [CHANGELOG.md](CHANGELOG.md), and the remaining work toward 2.0 is tracked in
@@ -54,6 +61,7 @@ warning-free Clippy builds on Linux and Windows. Shipped changes are tracked in
 | [`engine/compliance.rs`](src/engine/compliance.rs) | optional, app-driven usage-policy hook (not a license requirement) |
 | [`enterprise.rs`](src/enterprise.rs) | optional, host-owned audit events and honest capability discovery for fleet integrations |
 | [`db.rs`](src/db.rs) | the `Database` surface that ties it together |
+| [`upgrade.rs`](src/upgrade.rs) | out-of-place format migration with backup and deep verification |
 | [`ffi.rs`](src/ffi.rs) | C ABI (the `capi` feature): a panic-safe, C-callable surface wrapping the engine for Go, Python, and C bindings |
 
 ### Engineering notes
@@ -69,22 +77,27 @@ warning-free Clippy builds on Linux and Windows. Shipped changes are tracked in
   differential test checks `pv-wasm` against `wasmi` to keep it honest. Floats,
   tables, globals, imports, SIMD, and `br_table` are out of scope for `pv-wasm`
   and are rejected rather than mis-run.
-- **Page-backed engine.** Tables are append-only chains of row pages, each header
-  linking to the next. Inserts append to a tail page and write only that page
-  plus an O(tables) manifest, so autocommit is O(1) per insert rather than a
-  whole-table rewrite. Reads stream through a bounded buffer pool
-  ([`storage/cache.rs`](src/storage/cache.rs)), so datasets need not fit in RAM,
-  and opt-in ordered indexes ([`storage/index.rs`](src/storage/index.rs)) turn
-  `WHERE col = value` into a point lookup and range comparisons such as
+- **Page-backed engine.** Tables are append-only chains of hot row pages and
+  optional packed cold pages, each header linking to the next. Inserts append to
+  a row tail. Commit cost also includes catalog/index maintenance, retained-log
+  accounting and the selected durability protocol; it is not uniformly O(1).
+  Logged 2.0 workspaces persist index definitions to reduce catalog rewrites.
+  Reads stream through a
+  bounded buffer pool ([`storage/cache.rs`](src/storage/cache.rs)), so datasets
+  need not fit in RAM, and opt-in ordered indexes
+  ([`storage/index.rs`](src/storage/index.rs)) turn `WHERE col = value` into a
+  point lookup and range comparisons such as
   `WHERE col > v` into an ordered scan rather than a full scan.
 - **Selectable durability.** `Database::set_durability(Durability::Sync)` makes
   each flush `fsync` the data and commit the manifest atomically (write to a temp
   file, `fsync`, then rename). The default `Fast` mode uses the OS cache only:
   fast and durable on a clean exit, but not power-loss-safe.
-- **Crash-recoverable transactions.** Explicit `BEGIN`, `COMMIT`, and
-  `ROLLBACK` group filesystem or in-memory writes. Filesystem transactions keep
-  a synced rollback image and recovery marker; reopening after interruption
-  restores the last committed state before loading the workspace.
+- **Concurrent transactions.** Native `SharedDatabase` exposes independent
+  snapshot readers and bounded FIFO writers. Logged workspaces sync original
+  pages before overwriting them and publish an ordered physical change stream.
+  Reopening rolls back incomplete writes. Format 6 prevents old binaries from
+  bypassing recovery. Existing 1.x images remain readable and migratable.
+  See [the concurrency contract](docs/CONCURRENCY.md) for limits and costs.
 - **Hardened against untrusted input.** Opening a `.pvdb` or workspace, or running
   a WASM module, validates manifest hashes (no path traversal), bounds-checks CAS
   offsets and page chains (no out-of-bounds reads or infinite loops on a crafted
@@ -112,7 +125,8 @@ cargo run --release --example bench    # evaluation harness across modes and wor
 ```
 
 Install the full CLI with `cargo install picovolt --features data-tools`, then use `pv query`,
-`pv inspect`, `pv history`, `pv diff`, `pv import`, `pv export`, and `pv bake`.
+`pv inspect`, `pv history`, `pv diff`, `pv migrate`, `pv compact`, `pv import`,
+`pv export`, and `pv bake`.
 Parquet/SQLite conversion, query explanations, inspection, resumable baking,
 and dataset signing are documented in [Data tools](docs/DATA_TOOLS.md).
 
@@ -146,16 +160,51 @@ databases.
 Durability is selectable via `Database::set_durability` (`Fast` OS-cache default,
 or crash-safe `Sync` with fsync and an atomic manifest).
 
+Native Rust applications can begin adopting the 2.0 concurrency surface through
+`SharedDatabase`. It is a cloneable, bounded worker-thread coordinator with
+explicit read and write transaction handles, FIFO admission, cooperative
+cancellation, and rollback before failed or abandoned writes release the queue.
+The first slice serializes execution and preserves format v5; use clones of one
+coordinator rather than independently opening the same development workspace.
+See [Shared database concurrency](docs/CONCURRENCY.md) for the contract and
+current limits.
+
 Measured results and the methodology are in [BENCHMARKS.md](BENCHMARKS.md). In
 short, PicoVolt is a page-backed engine with O(1) filesystem appends (autocommit
 around 33k rows/s, linear), larger-than-RAM reads through a bounded buffer pool (a
 667-page dataset serves from a 16-page pool), ordered secondary indexes (point
 lookups roughly 6,100 times faster than a scan, plus range predicates), MVCC
 time-travel, opt-in crash-safe durability (`Durability::Sync`), and a fast
-compile-and-publish path (CAS dedup, columnar compression, memory-mappable
-single-file artifacts). Current limits include full-workspace transaction
-backups rather than an incremental WAL, left-deep equality joins rather than a
-general SQL planner, and no concurrent writers.
+compile-and-publish path (CAS dedup, cooperative columnar compression,
+memory-mappable single-file artifacts). Current limits include full-workspace
+transaction backups rather than an incremental WAL, adaptive index access within
+a left-deep equality-join plan rather than a general SQL planner, and no
+concurrent writers.
+
+## Maintenance and migration
+
+Cold-page maintenance is explicit in 1.x so the host retains ownership of
+scheduling and threads:
+
+```sh
+pv compact ./data.pv --max-pages 64
+pv inspect ./data.pv --json
+```
+
+`Database::compact_step(max_pages)` preserves record addresses, indexes, and
+complete MVCC history; it never compacts the mutable tail and leaves a page in
+row form when transposition would not save space. Each pass uses the
+crash-recoverable transaction protocol: allow bounded journal space for logged
+workspaces, or a full rollback image for unlogged workspaces. Baked-image migration is
+out-of-place and deeply verified before publication:
+
+```sh
+pv migrate old.pvdb new.pvdb --dry-run
+pv migrate old.pvdb new.pvdb --backup old.exact-backup.pvdb
+```
+
+The source is never modified and existing destinations are never overwritten.
+See [Migration and compaction](docs/MIGRATION.md).
 
 ## Install and distribution
 
@@ -164,7 +213,7 @@ general SQL planner, and no concurrent writers.
 | **Rust** (crates.io) | `cargo add picovolt` |
 | **JavaScript / npm** (WebAssembly, browser and Node) | `npm install picovolt` |
 | **Python** (native wheels) | `python -m pip install picovolt` |
-| **Go** (`database/sql` and direct API) | `go get github.com/MiniJe/picovolt/bindings/go@latest`, then provide the matching native C ABI library described in [`bindings/go/`](bindings/go) |
+| **Go** (`database/sql` and direct API) | `go get github.com/MiniJe/picovolt/bindings/go/v2@v2.0.0`, then provide the matching native C ABI library described in [`bindings/go/`](bindings/go) |
 | **C** | Download the matching `picovolt-capi-*` bundle from the [latest release](https://github.com/MiniJe/picovolt/releases/latest), or run `cargo build --release --features capi` |
 | **In-memory** (native, no filesystem) | `Database::open_memory()`, export with `bake_to_bytes()` |
 
@@ -233,7 +282,6 @@ native modules built on the public API. Both are documented in
 | | |
 |--|--|
 | Roadmap | [ROADMAP.md](ROADMAP.md) |
-| One-million-download plan | [docs/ROADMAP_1M_DOWNLOADS.md](docs/ROADMAP_1M_DOWNLOADS.md) |
 | Monetization thesis | [docs/MONETIZATION.md](docs/MONETIZATION.md) |
 | Enterprise integration foundation | [docs/ENTERPRISE.md](docs/ENTERPRISE.md) |
 | Platform and file support | [docs/SUPPORT.md](docs/SUPPORT.md) |

package/browser.js

@@ -62,6 +62,11 @@ export class PersistentDb {
     return statement;
   }
 
+  executeMany(sql, rows) {
+    this._assertOpen();
+    return JSON.parse(this.db.executeMany(sql, rows)).mutated;
+  }
+
   async save() {
     this._assertOpen();
     const root = await navigator.storage.getDirectory();

package/package.json

@@ -2,7 +2,7 @@
   "name": "picovolt",
   "type": "module",
   "description": "Embedded SQL database with MVCC history and single-file deployment",
-  "version": "1.8.1",
+  "version": "2.0.0",
   "license": "Apache-2.0",
   "repository": {
     "type": "git",

package/picovolt.d.ts

@@ -20,6 +20,10 @@ export class Db {
      * `... BEFORE tx` time-travel query.
      */
     currentTx(): number;
+    /**
+     * Atomically execute an INSERT/UPDATE/DELETE for an array of parameter arrays.
+     */
+    executeMany(sql: string, rows: any): string;
     /**
      * Export the whole database as a `.pvdb` byte image (a `Uint8Array` in JS).
      */

package/picovolt_bg.js

@@ -45,6 +45,32 @@ export class Db {
         const ret = wasm.db_currentTx(this.__wbg_ptr);
         return ret >>> 0;
     }
+    /**
+     * Atomically execute an INSERT/UPDATE/DELETE for an array of parameter arrays.
+     * @param {string} sql
+     * @param {any} rows
+     * @returns {string}
+     */
+    executeMany(sql, rows) {
+        let deferred3_0;
+        let deferred3_1;
+        try {
+            const ptr0 = passStringToWasm0(sql, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc);
+            const len0 = WASM_VECTOR_LEN;
+            const ret = wasm.db_executeMany(this.__wbg_ptr, ptr0, len0, rows);
+            var ptr2 = ret[0];
+            var len2 = ret[1];
+            if (ret[3]) {
+                ptr2 = 0; len2 = 0;
+                throw takeFromExternrefTable0(ret[2]);
+            }
+            deferred3_0 = ptr2;
+            deferred3_1 = len2;
+            return getStringFromWasm0(ptr2, len2);
+        } finally {
+            wasm.__wbindgen_free(deferred3_0, deferred3_1, 1);
+        }
+    }
     /**
      * Export the whole database as a `.pvdb` byte image (a `Uint8Array` in JS).
      * @returns {Uint8Array}

package/sqlite.js

@@ -94,6 +94,11 @@ class Database {
     return new Statement(this, sql);
   }
 
+  executeMany(sql, rows) {
+    this._assertOpen();
+    return JSON.parse(this._db.executeMany(sql, rows)).mutated;
+  }
+
   // Run one or more `;`-separated statements with no bound parameters.
   exec(sql) {
     this._assertOpen();

package/worker.js

@@ -30,6 +30,10 @@ self.addEventListener("message", async ({ data }) => {
         result = { statementId, parameterCount: statement.parameterCount };
         break;
       }
+      case "executeMany":
+        if (!database) throw new Error("open the database first");
+        result = database.executeMany(data.sql, data.rows);
+        break;
       case "execute": {
         const statement = statements.get(data.statementId);
         if (!statement) throw new Error("unknown PicoVolt prepared statement");