sandboxedjs 0.1.49 → 0.1.51

package/README.md

@@ -89,7 +89,7 @@ it alongside their model provider.
 | **Shell** | POSIX `sh` (54 builtins) — pipelines, redirection, here-docs, globbing, brace/parameter/arithmetic/command expansion, functions, `if`/`for`/`while`/`case`/`select`, job control, traps, arrays, `[[ ]]`, `(( ))` |
 | **Coreutils** | 154 programs — `ls cat cp mv rm mkdir grep sed awk find head tail sort uniq wc cut tr tee xargs chmod chown ln du df ps tar gzip base64 sha256sum diff curl wget` and the rest — plus 54 shell builtins |
 | **Node.js** | Its own module engine — `require` *and* `import`, live bindings, `exports` maps, top-level `await`, npm packages, `http`, `fs`, streams, `child_process` |
-| **Python** | CPython 3.13 (Pyodide) on the *same* filesystem, with the real standard library |
+| **Python** | Source-built CPython 3.13/WASM on the *same* filesystem, with the real standard library |
 | **WebAssembly** | A WASI (`preview1`) host — run any `wasm32-wasi` binary from clang, Rust, Zig or TinyGo as an ordinary process, on the same filesystem |
 | **FFmpeg** | `ffmpeg` and `ffprobe` (FFmpeg 5.1) reading and writing container files directly — [optional install](#video-and-audio) |
 | **`/proc`** | Live and synthesised — `ps`, `top`, `free` and `uptime` all read the same source |
@@ -391,7 +391,7 @@ your disk.
 
 ### Python
 
-Python is CPython 3.13, via Pyodide, mounted on the container's filesystem:
+Python is the bundled, source-built CPython 3.13 WebAssembly runtime mounted on the container's filesystem:
 
 ```ts
 await box.exec("python3 -c \"print(open('/etc/hostname').read())\"");
@@ -401,8 +401,10 @@ await box.exec("echo '1 2 3' | python3 -c \"import sys; print(sum(map(int, sys.s
 
 The bundled standard library includes `json`, `re`, `os`, `sys`, `math`, `random`, `hashlib`,
 `binascii`, `struct`, `time`, `collections`, `itertools`, `functools`, `asyncio` and more. It is
-Real CPython, so `sqlite3`, `dataclasses`, `decimal` and `typing` all work. Packages must be built for WebAssembly: `micropip` installs pure-Python wheels, and Pyodide's own distribution covers the rest
-(and only with outbound networking enabled).
+It is real CPython, so `sqlite3`, `dataclasses`, `decimal`, `typing`, `ssl`, and
+`asyncio` work. `pip` installs compatible wheels into the same filesystem and
+creates their `console_scripts` in `/usr/local/bin` (with outbound networking
+enabled). Native extensions still need an Emscripten-compatible build.
 
 ### WebAssembly binaries
 
@@ -701,20 +703,21 @@ Those use dynamic imports, so they only fail if you call them.
 
 ### Python in a browser
 
-Python works lazily in a browser without extra configuration. On first use,
-SandboxedJS loads the matching Pyodide runtime and assets from jsDelivr:
+Python works lazily in a browser without extra configuration. The owned CPython
+runtime and its pthread worker are shipped with the npm package:
 
 ```ts
 const box = await createContainer();
 await box.exec("python3 -c 'import sqlite3; print(sqlite3.sqlite_version)'");
 ```
 
-To self-host Pyodide or use another trusted CDN, override both URLs:
+Pyodide remains available as an explicit compatibility backend:
 
 ```ts
 import { configurePython, createContainer } from "sandboxedjs";
 
 configurePython({
+  backend: "pyodide",
   pyodideURL: "https://cdn.jsdelivr.net/pyodide/v0.28.3/full/pyodide.mjs",
   indexURL: "https://cdn.jsdelivr.net/pyodide/v0.28.3/full/",
 });
@@ -723,8 +726,8 @@ const box = await createContainer();
 await box.exec("python3 -c 'import sqlite3; print(sqlite3.sqlite_version)'");
 ```
 
-Python never falls back to a host interpreter or another runtime. A custom
-Pyodide module and its asset directory must use the same Pyodide version.
+Python never falls back to a host interpreter. When selecting the optional
+Pyodide backend, its module and asset directory must use the same version.
 
 ## Uploading files
 
@@ -972,11 +975,10 @@ Honest list of what does not work:
   now?" and run `npm install && npm run dev` from the shell instead.
 - **No `net`, `tls`, `worker_threads` or `vm`.** `http` and `https` are served by a virtual stack
   that `request()` talks to directly, so servers work; raw sockets do not.
-- **Python is CPython via Pyodide.** The standard library is the real one. A C
-  extension works only if it has been built for WebAssembly — Pyodide ships
-  many, including `numpy`, but an arbitrary wheel from PyPI will not install.
-  Starting an interpreter costs about a second and a half; one is kept per
-  container, and each program runs in its own namespace.
+- **Python is source-built CPython/WASM.** Each program gets its own interpreter
+  process worker. Pure-Python wheels install normally; native extensions must
+  be linked or published for Emscripten. Pydantic 2's `pydantic-core` is not
+  currently available, so use a Pydantic 1 constraint for FastAPI deployments.
 - **No real sockets.** HTTP servers work through the request proxy; raw TCP/UDP does not.
 - **No real processes.** Processes are cooperative async tasks: `kill -9` cannot interrupt a
   tight synchronous loop, and `SIGSTOP` only marks state.

package/dist/index.cjs

@@ -19107,7 +19107,7 @@ async function startNodeWorker(url, workerData) {
     /* webpackIgnore: true */
     specifier
   );
-  const worker = new NodeWorker(url, { workerData });
+  const worker = new NodeWorker(url, { workerData, execArgv: [] });
   return {
     postMessage: (message) => worker.postMessage(message),
     onMessage: (listener) => worker.on("message", listener),
@@ -19376,27 +19376,40 @@ function compareRelease(left, right) {
 function isPreRelease(version) {
   return /(a|b|rc|dev|post)\d*$/i.test(version.replace(/^\d+(\.\d+)*/, ""));
 }
-async function resolvePackage(client, requirement, environment) {
+async function resolvePackageCandidates(client, requirement, environment) {
   const index = await client.json(`https://pypi.org/pypi/${requirement.name}/json`);
   const candidates = Object.keys(index.releases).filter((version) => requirement.specifiers.every((s) => compareVersions(version, s.version, s.op))).sort((a, b) => compareRelease(releaseOf(a), releaseOf(b)));
   const stable = candidates.filter((version) => !isPreRelease(version));
-  const ordered = (stable.length > 0 ? stable : candidates).reverse();
+  const newestFirst = (stable.length > 0 ? stable : candidates).reverse();
+  const probeIndexes = [
+    0,
+    Math.floor(newestFirst.length / 2),
+    Math.floor(newestFirst.length / 4),
+    Math.floor(newestFirst.length * 3 / 4),
+    newestFirst.length - 1
+  ];
+  const probes = [...new Set(probeIndexes)].map((index2) => newestFirst[index2]).filter((v) => Boolean(v));
+  const probeVersions = new Set(probes);
+  const ordered = [...probes, ...newestFirst.filter((version) => !probeVersions.has(version))];
   let sawSourceOnly = false;
+  const resolved = [];
   for (const version of ordered) {
     const files = (index.releases[version] ?? []).filter((file3) => !file3.yanked);
     const wheel = pickWheel(files);
     if (wheel) {
-      return {
+      resolved.push({
         name: requirement.name,
         version,
         url: wheel.url,
         filename: wheel.filename,
         sha256: wheel.digests?.sha256 ?? "",
         requires: []
-      };
+      });
+      continue;
     }
     if (files.some((file3) => file3.packagetype === "sdist")) sawSourceOnly = true;
   }
+  if (resolved.length > 0) return resolved;
   if (ordered.length === 0) {
     throw new Error(
       `no version of ${requirement.name} matches ${describe(requirement)}`
@@ -19424,6 +19437,7 @@ function splitOnce(text2, separator) {
 
 // src/runtime/python/install.ts
 var SITE_PACKAGES = "/usr/lib/python3.13/site-packages";
+var SCRIPTS = "/usr/local/bin";
 function markerEnvironment(pythonVersion) {
   const [major = "3", minor = "13"] = pythonVersion.split(".");
   return {
@@ -19444,18 +19458,39 @@ async function installRequirements(options) {
     const requirement = parseRequirement(text2);
     if (requirement) queue.push({ requirement, extras: requirement.extras });
   }
-  const chosen = /* @__PURE__ */ new Map();
-  const seen = /* @__PURE__ */ new Set();
-  while (queue.length > 0) {
-    const { requirement, extras } = queue.shift();
+  const archives = /* @__PURE__ */ new Map();
+  const unavailable2 = /* @__PURE__ */ new Set();
+  const unsatisfiable = /* @__PURE__ */ new Map();
+  const candidateCache = /* @__PURE__ */ new Map();
+  let candidateAttempts = 0;
+  const MAX_CANDIDATE_ATTEMPTS = 96;
+  const solve = async (pending, state) => {
+    if (pending.length === 0) return state;
+    const [first, ...tail2] = pending;
+    let { requirement, extras } = first;
+    const rest = [];
+    const mergedSpecifiers = [...requirement.specifiers];
+    const mergedExtras = new Set(extras);
+    for (const item of tail2) {
+      if (item.requirement.name === requirement.name && markerApplies(item.requirement.marker, environment, item.extras)) {
+        mergedSpecifiers.push(...item.requirement.specifiers);
+        for (const extra of item.extras) mergedExtras.add(extra);
+      } else {
+        rest.push(item);
+      }
+    }
+    requirement = { ...requirement, specifiers: mergedSpecifiers };
+    extras = [...mergedExtras];
     if (!markerApplies(requirement.marker, environment, extras)) {
-      report.skipped.push({ name: requirement.name, marker: requirement.marker ?? "" });
-      continue;
+      return solve(rest, {
+        ...state,
+        skipped: [...state.skipped, { name: requirement.name, marker: requirement.marker ?? "" }]
+      });
     }
-    const key = `${requirement.name}[${extras.join(",")}]`;
-    if (seen.has(key)) continue;
-    seen.add(key);
-    const already = chosen.get(requirement.name);
+    const solveKey = `${requirement.name}:${requirement.specifiers.map((s) => `${s.op}${s.version}`).sort().join(",")}:${[...extras].sort().join(",")}`;
+    const knownFailure = unsatisfiable.get(solveKey);
+    if (knownFailure) throw knownFailure;
+    const already = state.chosen.get(requirement.name);
     if (already) {
       const satisfied = requirement.specifiers.every(
         (s) => compareVersions(already.version, s.version, s.op)
@@ -19465,38 +19500,152 @@ async function installRequirements(options) {
           `${requirement.name} ${already.version} is already selected but another dependency needs ` + requirement.specifiers.map((s) => `${s.op}${s.version}`).join(", ")
         );
       }
-      continue;
+      return solve(rest, state);
     }
     options.progress.collecting(requirement.name);
-    const resolved = await resolvePackage(options.client, requirement);
-    chosen.set(requirement.name, resolved);
-    options.progress.downloading(resolved.name, resolved.version);
-    const archive = await options.client.bytes(resolved.url);
-    const entries = readZip(archive);
-    const metadata = entries.find((entry) => /\.dist-info\/METADATA$/.test(entry.name));
-    if (metadata) {
-      for (const line of new TextDecoder().decode(metadata.data()).split(/\r?\n/)) {
-        if (!line.startsWith("Requires-Dist:")) continue;
-        const dependency = parseRequirement(line.slice("Requires-Dist:".length));
-        if (dependency) queue.push({ requirement: dependency, extras: [] });
-      }
-    }
-    unpack(options.vfs, options.cred, entries);
-    report.installed.push({ name: resolved.name, version: resolved.version });
-  }
+    if (unavailable2.has(requirement.name)) {
+      throw new Error(`${requirement.name} has no wheel this runtime can use`);
+    }
+    const candidateKey = `${requirement.name}:${requirement.specifiers.map((s) => `${s.op}${s.version}`).join(",")}`;
+    let candidatePromise = candidateCache.get(candidateKey);
+    if (!candidatePromise) {
+      candidatePromise = resolvePackageCandidates(options.client, requirement);
+      candidateCache.set(candidateKey, candidatePromise);
+    }
+    let candidates;
+    try {
+      candidates = await candidatePromise;
+    } catch (error) {
+      const anyKey = `${requirement.name}:`;
+      let anyPromise = candidateCache.get(anyKey);
+      if (!anyPromise) {
+        anyPromise = resolvePackageCandidates(
+          options.client,
+          { ...requirement, specifiers: [] });
+        candidateCache.set(anyKey, anyPromise);
+      }
+      try {
+        await anyPromise;
+      } catch {
+        unavailable2.add(requirement.name);
+      }
+      throw error;
+    }
+    let failure2;
+    for (const resolved of candidates) {
+      try {
+        candidateAttempts += 1;
+        if (candidateAttempts > MAX_CANDIDATE_ATTEMPTS) {
+          throw new Error(
+            "dependency resolution exceeded 96 wheel candidates; add version constraints for packages whose newest generation requires unavailable native WASM extensions"
+          );
+        }
+        let entries = archives.get(resolved.url);
+        if (!entries) {
+          options.progress.downloading(resolved.name, resolved.version);
+          entries = readZip(await options.client.bytes(resolved.url));
+          archives.set(resolved.url, entries);
+        }
+        const dependencies = [];
+        const metadata = entries.find((entry) => /\.dist-info\/METADATA$/.test(entry.name));
+        if (metadata) {
+          for (const line of new TextDecoder().decode(metadata.data()).split(/\r?\n/)) {
+            if (!line.startsWith("Requires-Dist:")) continue;
+            const dependency = parseRequirement(line.slice("Requires-Dist:".length));
+            if (dependency && markerApplies(dependency.marker, environment, extras)) {
+              dependencies.push({ requirement: dependency, extras: dependency.extras });
+            }
+          }
+        }
+        const chosen = new Map(state.chosen);
+        chosen.set(requirement.name, resolved);
+        return await solve([...dependencies, ...rest], {
+          chosen,
+          staged: [...state.staged, ...stageWheel(entries, resolved)],
+          installed: [...state.installed, { name: resolved.name, version: resolved.version }],
+          skipped: state.skipped
+        });
+      } catch (error) {
+        if (error?.message?.startsWith("dependency resolution exceeded")) throw error;
+        failure2 = error;
+      }
+    }
+    const resolutionError = failure2 instanceof Error ? failure2 : new Error(`could not resolve ${requirement.name}`);
+    unsatisfiable.set(solveKey, resolutionError);
+    throw resolutionError;
+  };
+  const solved = await solve(queue, {
+    chosen: /* @__PURE__ */ new Map(),
+    staged: [],
+    installed: [],
+    skipped: []
+  });
+  report.installed = solved.installed;
+  report.skipped = solved.skipped;
+  commit(options.vfs, options.cred, solved.staged);
   return report;
 }
-function unpack(vfs, cred, entries, resolved) {
+function stageWheel(entries, resolved) {
   const staged = [];
   for (const entry of entries) {
     if (entry.isDirectory) continue;
+    const dataPath = /^[^/]+\.data\/(purelib|platlib|scripts)\/(.+)$/.exec(entry.name);
+    if (dataPath) {
+      const [, scheme, relative2] = dataPath;
+      staged.push({
+        path: scheme === "scripts" ? `${SCRIPTS}/${relative2}` : `${SITE_PACKAGES}/${relative2}`,
+        data: entry.data(),
+        ...scheme === "scripts" ? { mode: 493 } : {}
+      });
+      continue;
+    }
     if (/^[^/]+\.data\//.test(entry.name)) continue;
     staged.push({ path: `${SITE_PACKAGES}/${entry.name}`, data: entry.data() });
   }
+  const entryPoints = entries.find((entry) => /\.dist-info\/entry_points\.txt$/.test(entry.name));
+  if (entryPoints) {
+    for (const [name, target] of consoleScripts(new TextDecoder().decode(entryPoints.data()))) {
+      staged.push({ path: `${SCRIPTS}/${name}`, data: new TextEncoder().encode(consoleLauncher(target)), mode: 493 });
+    }
+  }
+  return staged;
+}
+function commit(vfs, cred, staged) {
   const directories = /* @__PURE__ */ new Set();
   for (const file3 of staged) directories.add(file3.path.slice(0, file3.path.lastIndexOf("/")));
   for (const directory2 of directories) vfs.mkdir(directory2, { recursive: true, cred });
-  for (const file3 of staged) vfs.writeFile(file3.path, file3.data, { cred });
+  for (const file3 of staged) {
+    vfs.writeFile(file3.path, file3.data, { cred, ...file3.mode === void 0 ? {} : { mode: file3.mode } });
+    if (file3.mode !== void 0) vfs.chmod(file3.path, file3.mode, cred);
+  }
+}
+function consoleScripts(text2) {
+  const result = [];
+  let section = "";
+  for (const raw of text2.split(/\r?\n/)) {
+    const line = raw.trim();
+    const heading = /^\[([^\]]+)\]$/.exec(line);
+    if (heading) {
+      section = heading[1];
+      continue;
+    }
+    if (section !== "console_scripts" || line === "" || line.startsWith("#")) continue;
+    const assignment = /^([^=\s]+)\s*=\s*([^\s;]+)(?:\s*;.*)?$/.exec(line);
+    if (assignment) result.push([assignment[1], assignment[2].replace(/\s*\[.*\]$/, "")]);
+  }
+  return result;
+}
+function consoleLauncher(target) {
+  const [module, attribute = ""] = target.split(":", 2);
+  return `#!/usr/bin/python3
+import sys
+from importlib import import_module
+entry = import_module(${JSON.stringify(module)})
+for part in ${JSON.stringify(attribute)}.split('.'):
+    if part:
+        entry = getattr(entry, part)
+raise SystemExit(entry())
+`;
 }
 
 // src/runtime/python/backend.ts
@@ -19504,7 +19653,11 @@ function withSitePackages(env2) {
   const existing = env2.PYTHONPATH;
   return {
     ...env2,
-    PYTHONPATH: existing ? `${SITE_PACKAGES}:${existing}` : SITE_PACKAGES
+    PYTHONPATH: existing ? `${SITE_PACKAGES}:${existing}` : SITE_PACKAGES,
+    /* Installed packages live on a host-mounted VFS. Rewriting hundreds of
+     * cache files on every short-lived WASM process is wasted work and can
+     * exhaust the synchronous host-call bridge during large import graphs. */
+    PYTHONDONTWRITEBYTECODE: env2.PYTHONDONTWRITEBYTECODE ?? "1"
   };
 }
 var INTERPRETER_OWNED = /* @__PURE__ */ new Set(["lib", "dev", "proc", "usr"]);
@@ -19594,6 +19747,12 @@ function containerMounts(ctx) {
     }
   } catch {
   }
+  try {
+    if (ctx.vfs.stat(SCRIPTS, { cred: ctx.cred }).isDirectory()) {
+      mounts.push({ guest: SCRIPTS, container: SCRIPTS });
+    }
+  } catch {
+  }
   return mounts;
 }
 
@@ -19647,16 +19806,30 @@ environments. A package with no usable wheel is an error, never a silent skip.`,
         "outbound network access is disabled for this container (enable with network: { allowOutbound: true })"
       );
     }
+    const jsonCache = /* @__PURE__ */ new Map();
+    const bytesCache = /* @__PURE__ */ new Map();
     const client = {
       async json(url) {
-        const response = await fetch(url);
-        if (!response.ok) throw new Error(`${url} answered ${response.status}`);
-        return response.json();
+        let pending = jsonCache.get(url);
+        if (!pending) {
+          pending = fetch(url).then((response) => {
+            if (!response.ok) throw new Error(`${url} answered ${response.status}`);
+            return response.json();
+          });
+          jsonCache.set(url, pending);
+        }
+        return pending;
       },
       async bytes(url) {
-        const response = await fetch(url);
-        if (!response.ok) throw new Error(`${url} answered ${response.status}`);
-        return new Uint8Array(await response.arrayBuffer());
+        let pending = bytesCache.get(url);
+        if (!pending) {
+          pending = fetch(url).then(async (response) => {
+            if (!response.ok) throw new Error(`${url} answered ${response.status}`);
+            return new Uint8Array(await response.arrayBuffer());
+          });
+          bytesCache.set(url, pending);
+        }
+        return pending;
       }
     };
     try {