@supermachine/core 0.5.5 → 0.7.3

package/README.md

@@ -216,6 +216,44 @@ The `"opaque"` default exists because of how `symlink(2)` works: the target is o
 
 A runnable example is in [`examples/06-mount-workspace.mjs`](./examples/06-mount-workspace.mjs).
 
+#### Performance: don't put dependency caches on the mount
+
+A mount is great for source code (host-editable, DAX-fast reads). It is **the wrong place** for `node_modules`, `__pycache__`, `target/`, `.next/`, or anything else that gets thousands of small files written by a build/install step.
+
+Why: every guest file operation on a virtio-fs mount becomes a FUSE message round-trip over vsock — open, write, close, fsync each cost ~µs of in-guest overhead plus ~µs of host-side handling. For one big file (a tarball, an image), that overhead is invisible. For an `npm install` that creates ~10–30 k tiny files, you're paying milliseconds × tens of thousands of round-trips. We've measured the same `npm install` at **3–6 minutes** on a virtio-fs mount vs **10–30 seconds** on a virtio-blk volume backing `node_modules`. Same workload, same data, same VM — the difference is purely FUSE round-trip cost × small-file count.
+
+Block devices (`virtio-blk`, exposed via `volumes`) bypass FUSE entirely — the guest's own filesystem driver writes to a backing file on the host with no per-syscall protocol overhead. Same speed as a "normal" disk inside the guest.
+
+### Pattern 6: npm / pnpm / cargo workflow (mount for source, volume for cache)
+
+The shape that gets you both live host edits and fast installs:
+
+```ts
+const image = await Image.build({
+  ref: "node:20-alpine",
+  // Source tree — live host edits, DAX reads
+  mounts: [{ hostPath: "/Users/me/my-app", guestTag: "src", symlinks: "opaque" }],
+  // Dependency cache — fast writes, persists between bakes via the snapshot
+  volumes: [
+    { name: "node_modules", sizeMib: 1024, mountPoint: "/work/node_modules" },
+  ],
+  // One-time install at bake. Once snapshotted, every restore has
+  // node_modules already populated — no install on the hot path.
+  warmup: async (vm) => {
+    await vm.exec({ argv: ["mount", "-t", "virtiofs", "src", "/work"] });
+    await vm.exec({ argv: ["cp", "-r", "/work/package*.json", "/work/node_modules/.."] });
+    await vm.exec({ argv: ["sh", "-c", "cd /work && npm install"], timeoutMs: 600_000 });
+  },
+});
+```
+
+Three layers, each playing to their strengths:
+- **`mounts`** for source: read-only-ish on the hot path, DAX zero-copy, host edits show up immediately (kqueue invalidation).
+- **`volumes`** for write-heavy caches: native-speed writes, no FUSE overhead, persists across restores.
+- **`warmup`** + bake to do the install once: every restore reuses the baked node_modules; cold install is amortized across all future runs of this snapshot.
+
+A runnable example is in [`examples/07-npm-workspace-fast.mjs`](./examples/07-npm-workspace-fast.mjs).
+
 ---
 
 ## Full API

package/examples/07-npm-workspace-fast.mjs

@@ -0,0 +1,115 @@
+// 07-npm-workspace-fast.mjs — the integrator-recommended pattern
+// for "edit on host, run in VM, install dependencies once."
+//
+// Three layers, each playing to its strength:
+//
+//   1. virtio-fs MOUNT for the source tree — host edits are live
+//      in the guest (DAX-fast reads, kqueue-driven invalidation),
+//      `symlinks: 'opaque'` so npm workspace symlinks work without
+//      letting the host follow guest-supplied targets.
+//
+//   2. virtio-blk VOLUME for `node_modules` — block I/O is
+//      ~native disk speed; bypasses the FUSE per-file round-trip
+//      cost that makes `npm install` 3-6 min on a virtio-fs mount.
+//      Same install on a volume: ~10-30 s.
+//
+//   3. WARMUP runs `npm install` once at bake time. The result
+//      (node_modules populated on the volume) persists across
+//      restores — every subsequent restore has dependencies ready,
+//      no install on the hot path.
+//
+// Run:
+//   node examples/07-npm-workspace-fast.mjs
+
+import { Image } from "@supermachine/core";
+import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+// 1. Build a host-side project that mimics a tiny npm workspace.
+const project = mkdtempSync(join(tmpdir(), "sm-fast-"));
+mkdirSync(project, { recursive: true });
+writeFileSync(
+  join(project, "package.json"),
+  JSON.stringify(
+    {
+      name: "fast-demo",
+      version: "0.0.0",
+      dependencies: { "lodash.chunk": "^4.2.0" },
+    },
+    null,
+    2,
+  ),
+);
+
+// 2. Volume backing file — sparse, 1 GiB cap. Sits alongside the
+//    project for the demo; in production you'd put it under
+//    ~/.local/supermachine/volumes/ or wherever your tenancy boundary
+//    lives.
+const volumePath = join(project, "node_modules.img");
+
+// 3. Bake with a warmup that runs `npm install` once on the volume.
+//    The snapshot captures the *post-install* state; every restore
+//    has node_modules ready instantly.
+const t0 = Date.now();
+const img = await Image.build({
+  ref: "node:20-alpine",
+  name: "npm-workspace-fast-demo",
+  memoryMib: 512,
+  cmd: ["/bin/sh", "-c", "sleep infinity"],
+  mounts: [
+    { hostPath: project, guestTag: "src", symlinks: "opaque" },
+  ],
+  volumes: [
+    { hostPath: volumePath, guestPath: "/work/node_modules", sizeMib: 1024 },
+  ],
+  warmup: async (vm) => {
+    // Mount source RO-ish + format and mount the node_modules volume.
+    await vm.exec({
+      argv: [
+        "sh",
+        "-c",
+        [
+          "mkdir -p /src /work/node_modules",
+          "mount -t virtiofs src /src",
+          // The volume comes up as the next /dev/vdN; on first bake
+          // it's blank, format ext4. Idempotent: existing fs is left
+          // alone by mke2fs -F? No — use `blkid` to skip if already
+          // formatted. For a demo, the bake-time guest always sees
+          // an empty volume.
+          "mkfs.ext4 -F -q /dev/vdb",
+          "mount /dev/vdb /work/node_modules",
+          // Copy package*.json out of the read-mount into the
+          // volume-backed dir so `npm install` writes locally.
+          "cp /src/package.json /work/package.json",
+          "cd /work && npm install --silent --no-fund --no-audit",
+        ].join(" && "),
+      ],
+      timeoutMs: 120_000,
+    });
+  },
+  warmupTag: "v1",
+});
+console.log(`[fast] bake total: ${Date.now() - t0} ms`);
+
+// 4. Run a workload: every restore lands with node_modules already
+//    populated. Re-mount mounts and continue.
+const pool = await img.pool({ min: 1, max: 1, restoreOnRelease: false });
+const vm = await pool.acquire();
+const r = await vm.exec({
+  argv: [
+    "sh",
+    "-c",
+    [
+      "mount -t virtiofs src /src 2>/dev/null || true",
+      "mount /dev/vdb /work/node_modules 2>/dev/null || true",
+      "cd /work && node -e \"console.log(require('lodash.chunk')([1,2,3,4,5], 2))\"",
+    ].join("; "),
+  ],
+  timeoutMs: 10_000,
+});
+console.log(r.stdout.toString());
+
+await vm.release();
+await pool.shutdown();
+rmSync(project, { recursive: true, force: true });

package/examples/README.md

@@ -12,6 +12,7 @@ with the package installed.
 | `04-expose-tcp.mjs` | `listenerRequired: true` bakes, `vm.exposeTcp()`, `TcpForwarder` |
 | `05-spawn-streaming.mjs` | `vm.spawn()` → `ExecChild` (writeStdin / readStdout / wait) |
 | `06-mount-workspace.mjs` | `mounts` + `symlinks` policy: live host dir into guest, npm-workspace symlink pattern |
+| `07-npm-workspace-fast.mjs` | The recommended npm/pnpm/cargo pattern: virtio-fs mount for source + virtio-blk volume for node_modules + warmup-bake for one-time install |
 
 ## Requirements
 

package/index.d.ts

@@ -50,6 +50,40 @@ export interface MountSpec {
   symlinks?: string
 }
 
+/**
+ * A writable virtio-blk volume backed by a host file. Use for
+ * dependency caches (`node_modules`, `target/`, `__pycache__`,
+ * `.next/`) and other write-heavy paths — block I/O is ~native
+ * disk speed, no FUSE round-trip per file. Contrast with
+ * [`MountSpec`] which exposes a host directory via virtio-fs
+ * (right for source trees you want live-editable from the host,
+ * wrong for thousands-of-small-files write workloads).
+ *
+ * The host file is created sparse at `hostPath` if missing
+ * (actual disk usage tracks what the guest writes). Snapshots
+ * capture the mapping, not the volume contents — same backing
+ * file is reused across all restores of this snapshot.
+ */
+export interface VolumeSpec {
+  /**
+   * Host file backing this volume. Created sparse at `sizeMib`
+   * MiB if missing. Persists across restores; delete it to
+   * reset state.
+   */
+  hostPath: string
+  /**
+   * Mount point inside the guest, e.g. `"/var/lib/postgres"`
+   * or `"/work/node_modules"`. The guest needs to actually
+   * mount it (init can do this, or your warmup callback).
+   */
+  guestPath: string
+  /**
+   * Size cap in MiB. Default 1024 (1 GiB). Sparse — actual
+   * disk usage matches what the guest writes.
+   */
+  sizeMib?: number
+}
+
 /**
  * Options for [`Image.build`]. Mirrors `OciImageBuilder` in the
  * Rust crate. All fields optional except `ref` (the OCI image
@@ -114,6 +148,25 @@ export interface BuildOptions {
    * re-bakes.
    */
   mounts?: Array<MountSpec>
+  /**
+   * Writable virtio-blk volumes. Each entry maps a sparse host
+   * file to a guest mount point. Use for dependency caches
+   * (`node_modules`, `target/`) — block I/O is ~native disk
+   * speed and bypasses the FUSE round-trip cost that makes
+   * `mounts` slow for thousands-of-small-files workloads.
+   * Volume bytes persist across restores; only the mapping is
+   * captured in the snapshot.
+   */
+  volumes?: Array<VolumeSpec>
+  /**
+   * Extra kernel cmdline tokens, appended after supermachine's
+   * own defaults (earlycon, console, tsi_hijack, etc.). Power-
+   * user / testing escape hatch — e.g. `["panic=1"]` to force
+   * an immediate panic on warning, or `["init=/nope"]` to
+   * deliberately trigger an early-init kernel panic in a test.
+   * Folded into the bake's input hash so changing this re-bakes.
+   */
+  extraKernelArgs?: Array<string>
   /**
    * Override where to fetch the image bytes. Default behavior
    * (when omitted) treats `ref` as a registry reference and pulls

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@supermachine/core",
-  "version": "0.5.5",
+  "version": "0.7.3",
   "description": "Run any OCI/Docker image as a hardware-isolated microVM. Node/Bun/Deno binding for the supermachine Rust crate.",
   "license": "Apache-2.0",
   "main": "index.js",
@@ -22,7 +22,7 @@
     "arm64"
   ],
   "optionalDependencies": {
-    "@supermachine/core-darwin-arm64": "0.5.5"
+    "@supermachine/core-darwin-arm64": "0.7.3"
   },
   "scripts": {
     "build:rust": "TYPE_DEF_TMP_PATH=$(pwd)/scripts/.napi_type_defs.tmp cargo build --release -p supermachine-napi && cargo build --release --bin supermachine-worker -p supermachine",