@rhydb/rhydb-wasm 0.14.0

package/README.md

@@ -0,0 +1,187 @@
+# SILO WASM
+
+This directory builds a browser-usable version of SILO using WebAssembly (WASM). It uses Emscripten to compile the C++ sources to a `.wasm` binary and generates a small JavaScript loader that lets browser code call selected C++ functions.
+
+`CMakeLists.txt` collects SILO core sources, excludes the CLI, HTTP API, tests, and benchmark code, and links the dependencies needed for preprocessing and SaneQL query execution. `src/rhydb_wasm.cpp` is the boundary between JavaScript and C++.
+
+## What It Exposes
+
+The generated JavaScript module exports these embind functions:
+
+- `preprocess(preprocessingConfigPath)`: reads a SILO preprocessing config from Emscripten's virtual filesystem and returns a database handle.
+- `save(handle, outputDirectory)`: writes the in-memory database as SILO's normal processed-state files into the virtual filesystem.
+- `load(stateDirectory)`: loads a saved SILO processed state from the virtual filesystem and returns a database handle.
+- `query(handle, saneqlQuery)`: runs SaneQL and returns the result as NDJSON.
+- `info(handle)`: returns database information as JSON.
+- `dispose(handle)`: releases a database handle.
+
+The module also exposes Emscripten's `FS` runtime API. Browser code uses `FS` to write uploaded files into the virtual filesystem before calling `preprocess` or `load`, and to read files back after calling `save`.
+
+The WASM build supports raw and `.zst` preprocessing inputs. The native `.xz` input path is not included in the WASM target.
+
+Browser queries use a smaller materialization cutoff than native SILO to keep intermediate batches within browser memory limits, especially when projecting full nucleotide sequences.
+
+## Artifacts
+
+Building the WASM target creates these files relative to the SILO repository root:
+
+- `wasm/dist/rhydb_wasm.js`: ES module loader generated by Emscripten. Import this from browser code.
+- `wasm/dist/rhydb_wasm.d.ts`: Autogenerated type declarations.
+- `wasm/dist/rhydb_wasm.wasm`: compiled SILO C++ code and linked dependencies.
+
+All files must be served by the web app. The JavaScript loader fetches the `.wasm` file at runtime.
+
+## Install from NPM
+
+The WASM build is also published as an NPM package.
+
+```sh
+npm install @rhydb/rhydb-wasm
+```
+
+The package is an ES module. Import the loader as a default import:
+
+```js
+import createRhyDbModule from "@rhydb/rhydb-wasm";
+
+const silo = await createRhyDbModule();
+```
+
+The `.wasm` file ships alongside `rhydb_wasm.js` and must be resolvable by your bundler so the loader can fetch it at runtime. The package version matches the SILO release version.
+
+The current build uses pthreads, so the page serving the module must be cross-origin isolated. Pthreads are the POSIX thread API that Emscripten maps to Web Workers, and SILO uses them because its dependencies and query execution are built around threaded native code. Serve it over HTTP with these headers:
+
+```text
+Cross-Origin-Opener-Policy: same-origin
+Cross-Origin-Embedder-Policy: require-corp
+```
+
+For a local example server, run this from the SILO repository root:
+
+```sh
+python3 - <<'PY'
+from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
+
+class Handler(SimpleHTTPRequestHandler):
+    def end_headers(self):
+        self.send_header("Cross-Origin-Opener-Policy", "same-origin")
+        self.send_header("Cross-Origin-Embedder-Policy", "require-corp")
+        super().end_headers()
+
+ThreadingHTTPServer(("127.0.0.1", 8088), Handler).serve_forever()
+PY
+```
+
+## Build Reference
+
+Prerequisites:
+
+- Emscripten SDK activated so `emcmake`, `emcc`, and `em++` are on `PATH`. Install it via
+  [emsdk](https://emscripten.org/docs/getting_started/downloads.html):
+
+  ```sh
+  git clone https://github.com/emscripten-core/emsdk.git
+  cd emsdk && ./emsdk install latest && ./emsdk activate latest
+  source ./emsdk_env.sh
+  ```
+- CMake
+- Ninja
+- uv
+- a native C++ compiler for Conan build tools
+
+From the SILO repository root, build the WASM artifacts:
+
+```sh
+make wasm
+```
+
+The compatibility wrapper `./wasm/scripts/build-wasm.sh` also calls `make wasm`. The Makefile writes the Emscripten Conan profile to `build/wasm/conanprofile-emscripten`, installs dependencies into `build/wasm`, configures CMake with `emcmake`, builds `rhydb_wasm`, and copies the artifacts to `wasm/dist`.
+
+## Tests
+
+`./test` contains a Node.js smoke test (`node:test`) that loads the built module, preprocesses
+the `testBaseData/unitTestDummyDataset` fixture, runs a SaneQL query, and round-trips a saved
+state through `save`/`load`. Run it with:
+
+```sh
+make wasm-test
+```
+
+This builds `rhydb_wasm` first if necessary, then runs `node --test wasm/test/*.test.mjs`.
+
+## Example App
+
+An example app is available in `./example`. Any static web server can be used as long as it sets the headers. Opening the `index.html` directly as a local file is not enough for the pthread-enabled build.
+
+## Usage Reference
+
+Import and initialize the module:
+
+```js
+import createRhydbModule from "./rhydb_wasm.js";
+
+const silo = await createRhydbModule();
+```
+
+This assumes `rhydb_wasm.js` and `rhydb_wasm.wasm` are served from the same URL directory. If a web app serves them from different locations, pass Emscripten's `locateFile` option to tell the loader where to fetch `rhydb_wasm.wasm`.
+
+Write uploaded files into the virtual filesystem and preprocess:
+
+```js
+function mkdirp(path) {
+  let current = "";
+  for (const part of path.split("/").filter(Boolean)) {
+    current += `/${part}`;
+    if (!silo.FS.analyzePath(current).exists) {
+      silo.FS.mkdir(current);
+    }
+  }
+}
+
+mkdirp("/input");
+silo.FS.writeFile("/input/preprocessing_config.yaml", preprocessingConfigBytes);
+silo.FS.writeFile("/input/database_config.yaml", databaseConfigBytes);
+silo.FS.writeFile("/input/reference_genomes.json", referenceGenomeBytes);
+silo.FS.writeFile("/input/sequences.ndjson.zst", ndjsonBytes);
+
+silo.FS.chdir("/input");
+const handle = silo.preprocess("preprocessing_config.yaml");
+```
+
+Query an in-memory database:
+
+```js
+const ndjson = silo.query(handle, "default.groupBy({count:=count()})");
+console.log(ndjson);
+```
+
+Save a processed state and read the files back:
+
+```js
+mkdirp("/state");
+silo.save(handle, "/state");
+
+for (const entry of silo.FS.readdir("/state")) {
+  if (entry !== "." && entry !== "..") {
+    console.log(entry);
+  }
+}
+```
+
+Load an existing processed state:
+
+```js
+mkdirp("/loaded-state");
+// Write the saved SILO state files under /loaded-state first.
+const loadedHandle = silo.load("/loaded-state");
+console.log(silo.info(loadedHandle));
+```
+
+Release memory when a database is no longer needed:
+
+```js
+silo.dispose(handle);
+silo.dispose(loadedHandle);
+```
+
+For a complete browser example that uploads input files, downloads a saved state, uploads that state again, and runs a query, see `example/`.