@surrealdb/wasm-native 3.3.4 → 4.0.0-beta.1

package/README.md

@@ -61,12 +61,80 @@ engine.free();
 | Export | Description |
 | --- | --- |
 | `init` | Instantiates the module; must resolve before anything else is called |
-| `SurrealWasmEngine` | One embedded instance: `connect`, `execute`, `notifications`, `export`, `import`, `version`, `free` |
+| `SurrealWasmEngine` | One embedded instance: `connect`, `execute`, `notifications`, `export`, `import`, `version`, `close`, `persisted`, `free` |
 | `ConnectionOptions` | Capabilities, timeouts, and the namespace/database created on new storage |
+| `./worker` | `connect(endpoint, options)`: starts a dedicated worker, hosts the engine in it and returns a `WorkerEngine` with the same methods, for `surrealkv://` |
 | `./module` | The `.wasm` file itself, for a caller that resolves the module source itself |
 
-Storage backends are selected by the endpoint: `mem://` and `indxdb://name`, the latter backed by
-the browser's IndexedDB.
+Storage backends are selected by the endpoint: `mem://` and `surrealkv://name`, the latter backed by
+SurrealKV over the browser's Origin Private File System (OPFS). The two are covered in
+[Persistent storage in the browser](#persistent-storage-in-the-browser).
+
+### Persistent storage in the browser
+
+`surrealkv://name` keeps the database in the origin private file system, in a database called
+`name` that belongs to the origin. `mem://` keeps nothing and works on any thread.
+
+**The module must run in a dedicated worker.** The synchronous file handles that SurrealKV needs
+exist only in dedicated workers, so a `surrealkv://` database cannot be opened from the main
+thread of a page, from a shared worker or from a service worker. `SurrealWasmEngine.connect`
+rejects such a call before it touches storage. The `./worker` export is the bridge: `connect` starts a dedicated worker, loads this module in it
+(`dist/worker.js` beside the loader), connects the engine there and returns a `WorkerEngine` whose
+`execute`, `queryStream`, `notifications`, `export`, `import`, `version`, `persisted` and `close`
+mirror the engine's. Byte payloads are copied or transferred and `ReadableStream`s are transferred,
+so a reader that stops reading still stops the engine's producer.
+
+```ts
+import { connect } from "@surrealdb/wasm-native/worker";
+
+const engine = await connect("surrealkv://my-database");
+// ... engine.execute(request) ...
+await engine.close(); // before the application ends
+```
+
+A caller that hosts the module in its own worker uses `SurrealWasmEngine` directly: it calls `init`
+and `connect` there and carries requests, replies and notifications by message itself. The module needs no threads, no async runtime and no `SharedArrayBuffer`.
+
+**One tab per database.** SurrealKV takes an exclusive lock on the database. A second tab (or
+worker) that connects to the same name waits a few seconds for a holder that is closing and then
+`connect` rejects with a message that says the database is open in another tab or worker. The engine does not retry or take the lock over: the
+page decides, for example by telling the user to close the other tab, or by connecting again later.
+After a reload without a `close()` the browser can take longer than the wait to release the lock,
+so the same rejection can appear for a short time.
+
+**Close the engine when the application ends.** A worker cannot wait for an asynchronous close
+inside a page-unload handler. Call `await engine.close()` as early as the application knows it is
+ending, for example when the page sends a message to the worker. On a `WorkerEngine`, `close()`
+closes the datastore, frees the engine and ends the worker. On a `SurrealWasmEngine`, call
+`await engine.close()` and then `engine.free()`. `close()` closes the SurrealKV tree and then
+releases its files and the lock; it takes longer the more files the database has. A
+`SurrealWasmEngine` that is freed without `close()` closes its datastore on a task of its own, so the
+lock is released shortly after `free()` returns rather than before. A database that was not closed is recovered the next time it is
+opened. Nothing may use the engine after `close()` resolves, other than `free()`.
+
+**Ask for persistent storage from the page.** Unless the origin has been granted persistent
+storage, the browser may evict its data when the disk fills. Only a page can ask: call
+`navigator.storage.persist()` there. Inside the worker, `await SurrealWasmEngine.persisted()`
+reports whether the request was granted; it resolves to `false` until then.
+
+**Data from the browser's earlier storage moves over by itself.** At every `surrealkv://name` connect,
+before the database is opened, the module copies the rows of an earlier browser database called
+`name` into the origin private file system, unless a database `name` exists there already. The
+earlier database is only read: it is never changed or deleted, and the application removes it with
+`indexedDB.deleteDatabase(name)` once it has checked the copy. The copy is verified, and one that
+was interrupted is made again at the next connect, which then opens no database until the copy
+completes. A copy that fails makes `connect` reject, and the database is not opened.
+
+Limits: Chrome is the only browser that is tested. Encryption, checkpoints and the migrators of
+earlier engines are not available in the browser. `Durability` in a browser is an ordering guarantee of
+the file handle's `flush()` and not a promise about a power cut or an operating-system crash. A
+worker has an address space of at most 4 GiB, and the default memtable and block cache sizes
+assume the 1 GiB that is assumed when the memory of the host cannot be read.
+
+The `browser/` directory holds two pages that exercise this in a browser: `smoke.html` drives the
+`./worker` bridge (persistence across close and reopen, the second-tab rejection) and `drop.html`
+frees an engine without closing it and reconnects. Build, serve this directory with any static
+server, and open them in Chrome; each leaves its outcome in `window.__results` or `window.__drop`.
 
 ### Where `init` reads the module from
 
@@ -110,7 +178,7 @@ tick, which a browser does not notice but which keeps a Node or Bun process from
 ## Requirements
 
 - ES modules (`import`) — CommonJS (`require`) is not supported
-- A browser, or any host with `WebAssembly`; `indxdb://` additionally needs IndexedDB
+- A browser, or any host with `WebAssembly`; `surrealkv://` additionally needs a dedicated worker with the origin private file system
 
 ## Contributing
 
@@ -130,6 +198,17 @@ rustup target add wasm32-unknown-unknown
 cargo install wasm-opt --version 0.116.1
 ```
 
+SurrealKV links zstd, which is C code, so the build needs a C compiler and an archiver that target
+WebAssembly. On macOS the compiler and archiver that ship with the system do not, so use LLVM's:
+
+```sh
+brew install llvm
+export CC_wasm32_unknown_unknown="$(brew --prefix llvm)/bin/clang"
+export AR_wasm32_unknown_unknown="$(brew --prefix llvm)/bin/llvm-ar"
+```
+
+On Linux, `clang` and `llvm-ar` serve the same purpose.
+
 `wasm-bindgen-cli` has to match the `wasm-bindgen` version in the workspace lockfile exactly —
 the two share a schema that changes between patch releases. `bun run build` checks this before
 it does anything and prints the `cargo install` line to run, so there is no version to copy from
@@ -147,5 +226,5 @@ The Rust half is a normal member of the Cargo workspace, but it compiles to noth
 the target explicitly:
 
 ```sh
-cargo clippy -p surrealdb-wasm --features kv-mem,kv-indxdb --target wasm32-unknown-unknown
+cargo clippy -p surrealdb-wasm --features kv-mem,kv-surrealkv --target wasm32-unknown-unknown
 ```

package/dist/client.d.ts

@@ -0,0 +1,39 @@
+import type { ConnectionOptions } from "./index.js";
+
+export type { ConnectionOptions };
+
+export type ConnectOptions = {
+	/** Where the engine's options go; see the module's `ConnectionOptions`. */
+	options?: ConnectionOptions;
+	/** The worker script. Defaults to `worker.js` beside this module. */
+	workerUrl?: string | URL;
+	/** Where the worker reads the `.wasm` from. Defaults to the loader's own location. */
+	module?: string | URL;
+};
+
+/** One embedded engine, hosted in a dedicated worker. */
+export declare class WorkerEngine {
+	/** Runs one RPC request. */
+	execute(data: Uint8Array): Promise<Uint8Array>;
+	/** Runs a query, answering with its frames as they are produced. */
+	queryStream(data: Uint8Array): Promise<ReadableStream<Uint8Array>>;
+	/** The live-query notifications of this connection; one stream per engine. */
+	notifications(): ReadableStream<Uint8Array>;
+	export(config?: Uint8Array): Promise<string>;
+	import(input: string): Promise<void>;
+	version(): Promise<string>;
+	/** Whether the browser keeps this origin's storage until the user clears it. */
+	persisted(): Promise<boolean>;
+	/**
+	 * Closes the datastore, frees the engine and ends the worker. When this
+	 * resolves the storage lock is released.
+	 */
+	close(): Promise<void>;
+	[Symbol.asyncDispose](): Promise<void>;
+}
+
+/**
+ * Starts a dedicated worker, loads the engine in it and connects to
+ * `endpoint`.
+ */
+export declare function connect(endpoint: string, options?: ConnectOptions): Promise<WorkerEngine>;

package/dist/client.js

@@ -0,0 +1,102 @@
+// ts/client.ts
+class WorkerEngine {
+  #worker;
+  #next = 0;
+  #pending = new Map;
+  #notifications;
+  #closed = false;
+  #closing;
+  constructor(worker) {
+    this.#worker = worker;
+    worker.onmessage = (event) => {
+      const reply = event.data;
+      const pending = this.#pending.get(reply.id);
+      if (!pending)
+        return;
+      this.#pending.delete(reply.id);
+      if (reply.ok)
+        pending.resolve(reply.value);
+      else
+        pending.reject(new Error(reply.error));
+    };
+    worker.onerror = (event) => this.#fail(new Error(event.message || "The worker failed"));
+    worker.onmessageerror = () => this.#fail(new Error("The worker sent an unreadable message"));
+  }
+  #fail(error) {
+    this.#closed = true;
+    this.#worker.terminate();
+    for (const pending of this.#pending.values())
+      pending.reject(error);
+    this.#pending.clear();
+  }
+  call(request, transfer = []) {
+    if (this.#closed)
+      return Promise.reject(new Error("The engine is closed"));
+    return new Promise((resolve, reject) => {
+      const id = ++this.#next;
+      this.#pending.set(id, { resolve, reject });
+      this.#worker.postMessage({ ...request, id }, transfer);
+    });
+  }
+  execute(data) {
+    return this.call({ op: "execute", data });
+  }
+  queryStream(data) {
+    return this.call({ op: "queryStream", data });
+  }
+  notifications() {
+    if (!this.#notifications)
+      throw new Error("The engine is not connected");
+    return this.#notifications;
+  }
+  export(config) {
+    return this.call({ op: "export", config });
+  }
+  import(input) {
+    return this.call({ op: "import", input });
+  }
+  version() {
+    return this.call({ op: "version" });
+  }
+  persisted() {
+    return this.call({ op: "persisted" });
+  }
+  close() {
+    this.#closing ??= this.#close();
+    return this.#closing;
+  }
+  async#close() {
+    if (this.#closed)
+      return;
+    try {
+      await this.call({ op: "close" });
+    } finally {
+      this.#fail(new Error("The engine is closed"));
+    }
+  }
+  [Symbol.asyncDispose]() {
+    return this.close();
+  }
+  _setNotifications(stream) {
+    this.#notifications = stream;
+  }
+}
+async function connect(endpoint, { options, workerUrl, module } = {}) {
+  const worker = new Worker(workerUrl ?? new URL("./worker.js", import.meta.url), {
+    type: "module"
+  });
+  const engine = new WorkerEngine(worker);
+  try {
+    await engine.call({ op: "init", module: module === undefined ? undefined : String(module) });
+    await engine.call({ op: "connect", endpoint, options });
+    engine._setNotifications(await engine.call({ op: "notifications" }));
+    return engine;
+  } catch (error) {
+    await engine.close().catch(() => {});
+    throw error;
+  }
+}
+export {
+  WorkerEngine,
+  connect
+};

package/dist/index.d.ts

@@ -33,6 +33,7 @@ export type ConnectionOptions = {
         functions?: boolean | string[] | CapabilitiesAllowDenyList;
         network_targets?: boolean | string[] | CapabilitiesAllowDenyList;
         experimental?: boolean | string[] | CapabilitiesAllowDenyList;
+        /** @deprecated Accepted and ignored: the engine has a single execution path. */
         planner_strategy?: "best-effort" | "compute-only" | "all-read-only";
     };
 }
@@ -71,14 +72,33 @@ export class IntoUnderlyingSource {
  * An embedded SurrealDB instance, addressed over the RPC protocol.
  *
  * wasm-bindgen generates a `free()` for this, which is what the JavaScript
- * side calls to close the connection; dropping the engine closes the
+ * side calls to dispose of the connection; dropping the engine closes the
  * datastore's notification channel, which ends any stream handed out by
  * [`Self::notifications`].
+ *
+ * Dropping an engine that [`Self::close`] has not closed also closes its
+ * datastore, on a task of its own, since a drop cannot wait: the storage lock
+ * is released once that task finishes rather than by the time `free()`
+ * returns. A caller that needs the lock released at a known point awaits
+ * `close()` first.
  */
 export class SurrealWasmEngine {
     private constructor();
     free(): void;
     [Symbol.dispose](): void;
+    /**
+     * Closes the datastore: the storage engine is closed and then the files
+     * it holds are released, which frees the database's lock in the origin
+     * private file system so that another tab or worker can open it.
+     *
+     * A worker cannot await an asynchronous close from a page-unload
+     * handler, so the application calls this when it knows that it is ending,
+     * for example on a message from the page, and as early as it can: the
+     * more files the database has, the longer it takes. A database that is
+     * not closed is recovered the next time it is opened. Nothing may use
+     * the engine after this resolves, other than `free()`.
+     */
+    close(): Promise<void>;
     static connect(endpoint: string, opts?: ConnectionOptions | null): Promise<SurrealWasmEngine>;
     /**
      * Every method takes `&self`. With `&mut self` the wasm-bindgen trampoline
@@ -98,6 +118,16 @@ export class SurrealWasmEngine {
      * accumulating behind it.
      */
     notifications(): ReadableStream;
+    /**
+     * Whether the browser keeps this origin's storage until the user clears
+     * it, which is what stops it from evicting a `surrealkv://` database when
+     * the disk fills.
+     *
+     * It is `false` until the page has asked with `navigator.storage.persist()`,
+     * which a worker cannot call. Always `false` for a build without
+     * `surrealkv://` storage, which keeps nothing.
+     */
+    static persisted(): Promise<boolean>;
     /**
      * Runs a query, answering with frames as the executor produces them rather
      * than with every statement's result at once.
@@ -120,17 +150,10 @@ export type InitInput = RequestInfo | URL | Response | BufferSource | WebAssembl
 
 export interface InitOutput {
     readonly memory: WebAssembly.Memory;
-    readonly __wbg_surrealwasmengine_free: (a: number, b: number) => void;
-    readonly surrealwasmengine_connect: (a: number, b: number, c: number) => number;
-    readonly surrealwasmengine_execute: (a: number, b: number, c: number) => number;
-    readonly surrealwasmengine_export: (a: number, b: number, c: number) => number;
-    readonly surrealwasmengine_import: (a: number, b: number, c: number) => number;
-    readonly surrealwasmengine_notifications: (a: number) => number;
-    readonly surrealwasmengine_query_stream: (a: number, b: number, c: number) => number;
-    readonly surrealwasmengine_version: (a: number) => void;
     readonly __wbg_intounderlyingbytesource_free: (a: number, b: number) => void;
     readonly __wbg_intounderlyingsink_free: (a: number, b: number) => void;
     readonly __wbg_intounderlyingsource_free: (a: number, b: number) => void;
+    readonly __wbg_surrealwasmengine_free: (a: number, b: number) => void;
     readonly intounderlyingbytesource_autoAllocateChunkSize: (a: number) => number;
     readonly intounderlyingbytesource_cancel: (a: number) => void;
     readonly intounderlyingbytesource_pull: (a: number, b: number) => number;
@@ -141,11 +164,28 @@ export interface InitOutput {
     readonly intounderlyingsink_write: (a: number, b: number) => number;
     readonly intounderlyingsource_cancel: (a: number) => void;
     readonly intounderlyingsource_pull: (a: number, b: number) => number;
-    readonly __wasm_bindgen_func_elem_29890: (a: number, b: number, c: number, d: number) => void;
-    readonly __wasm_bindgen_func_elem_4538: (a: number, b: number, c: number, d: number) => void;
-    readonly __wasm_bindgen_func_elem_29891: (a: number, b: number, c: number, d: number) => void;
-    readonly __wasm_bindgen_func_elem_4606: (a: number, b: number, c: number, d: number) => void;
-    readonly __wasm_bindgen_func_elem_8031: (a: number, b: number) => void;
+    readonly surrealwasmengine_close: (a: number) => number;
+    readonly surrealwasmengine_connect: (a: number, b: number, c: number) => number;
+    readonly surrealwasmengine_execute: (a: number, b: number, c: number) => number;
+    readonly surrealwasmengine_export: (a: number, b: number, c: number) => number;
+    readonly surrealwasmengine_import: (a: number, b: number, c: number) => number;
+    readonly surrealwasmengine_notifications: (a: number) => number;
+    readonly surrealwasmengine_persisted: () => number;
+    readonly surrealwasmengine_query_stream: (a: number, b: number, c: number) => number;
+    readonly surrealwasmengine_version: (a: number) => void;
+    readonly rust_zstd_wasm_shim_calloc: (a: number, b: number) => number;
+    readonly rust_zstd_wasm_shim_free: (a: number) => void;
+    readonly rust_zstd_wasm_shim_malloc: (a: number) => number;
+    readonly rust_zstd_wasm_shim_memcmp: (a: number, b: number, c: number) => number;
+    readonly rust_zstd_wasm_shim_memcpy: (a: number, b: number, c: number) => number;
+    readonly rust_zstd_wasm_shim_memmove: (a: number, b: number, c: number) => number;
+    readonly rust_zstd_wasm_shim_memset: (a: number, b: number, c: number) => number;
+    readonly rust_zstd_wasm_shim_qsort: (a: number, b: number, c: number, d: number) => void;
+    readonly __wasm_bindgen_func_elem_5798: (a: number, b: number, c: number, d: number) => void;
+    readonly __wasm_bindgen_func_elem_5862: (a: number, b: number, c: number, d: number) => void;
+    readonly __wasm_bindgen_func_elem_46027: (a: number, b: number, c: number) => void;
+    readonly __wasm_bindgen_func_elem_46027_22: (a: number, b: number, c: number) => void;
+    readonly __wasm_bindgen_func_elem_9452: (a: number, b: number) => void;
     readonly __wbindgen_export: (a: number, b: number) => number;
     readonly __wbindgen_export2: (a: number, b: number, c: number, d: number) => number;
     readonly __wbindgen_export3: (a: number) => void;