sandboxedjs 0.2.11 → 0.2.12

package/docs/platform-hardening.md

@@ -0,0 +1,49 @@
+# Platform hardening
+
+SandboxedJs is a virtual OS runtime for browser clients and Node hosts. It is
+not a recording application. Continue developing its own JavaScript module
+engine, virtual kernel, filesystem, network and CPython WebAssembly integration;
+do not replace them with Nodepod or Pyodide.
+
+## Release gates
+
+Completion requires evidence in both a real browser and a supported Node host.
+A successful bundle alone does not establish browser runtime compatibility.
+
+1. Runtime lifecycle: repeated boot, cancellation, child-process termination,
+   disposal and startup failure leave no workers, timers, sockets or mounts.
+2. Isolation: concurrent guests cannot observe each other's environment or
+   filesystem. Worker execution and realm fallback must be tested separately.
+3. Package workloads: install, run, edit and rebuild representative frontend
+   and API projects. Test actual package imports rather than lookalike APIs.
+4. Python: test dotenv loading, imports, subprocesses, HTTP servers and async
+   I/O in the shipped interpreter. Resolve extension setup timeouts before
+   claiming the complete Python suite passes.
+5. Performance: measure cold and warm boot, installation, module loading,
+   rebuild latency, memory after disposal and large console output. Record
+   runtime version, host, workload and repetitions with each result.
+6. Distribution: verify published ESM, CommonJS, worker and static browser
+   assets, including missing assets and unsupported host features.
+
+## Current findings
+
+- Corrected environment support to expose `util.parseEnv`; the previous
+  `process.parseEnv` addition did not match native Node. Added
+  `process.loadEnvFile` and native-Node differential parser fixtures.
+- Typed-array inspection now reads at most 100 entries instead of copying the
+  entire array. DataView logging no longer attempts to iterate a non-iterable.
+- The full test run was interrupted after Python extension setup timed out.
+  Follow-up isolated EPERM on the test's loopback listener and fixed missing
+  setup error handling. All 12 extension tests passed when loopback listening
+  was permitted. This is still not a full release pass.
+- Several Node modules still have stubs: cluster, dgram, diagnostics_channel,
+  domain, http2, inspector, tls, v8, vm and worker_threads. Implement and test
+  usable APIs before marking those modules supported.
+- The Python thread/syscall limitation documented in
+  browser-runtime-architecture.md needs interpreter-image work.
+- Browser CORS, static asset hosting, cross-origin isolation requirements,
+  native addons and platform-specific binaries remain constraints. Arbitrary
+  packages cannot be promised to run solely from this JavaScript runtime.
+
+The platform objective remains open. These gates are an implementation and
+verification backlog, not a claim of complete Node, Python or Linux parity.

package/docs/python/abi.md

@@ -0,0 +1,97 @@
+# `sbx_host_v1` — the host ABI
+
+The contract between compiled guest code and the SandboxedJs kernel.
+
+Everything a guest cannot do for itself — files, descriptors, pipes, readiness,
+time, identity, entropy, and later sockets, processes and signals — crosses this
+boundary and nothing else. Emscripten and WASI adapters translate *into* it;
+they are not the source of truth. That distinction is the point: compiling a
+program to WebAssembly does not give it an operating system, and pretending an
+Emscripten build's own filesystem is the container's is how two divergent copies
+of the same state come to exist.
+
+## Source of truth
+
+`python-runtime/abi/host-v1.json` defines the version, the operation codes, the
+errno values and the capability names. `make -C python-runtime abi` regenerates:
+
+- `python-runtime/abi/sbx_host.h` — the C view.
+- `src/python/host-abi.ts` — the TypeScript view.
+
+Both are committed. Neither is edited by hand: a C file and a TypeScript file
+that disagree about an opcode produce a wrong answer, not a link error.
+
+## Framing
+
+```
+request:   u16 version | u16 op | u32 request_id | u32 generation | u32 length | payload
+response:  u16 version | u16 op | u32 request_id | u32 generation | i32 status | u32 length | payload
+```
+
+Little-endian, fixed width, UTF-8 for text, raw bytes for content, 64-bit file
+offsets. `status` is the operation's result: `>= 0` on success, `-errno` on
+failure, with the canonical (Linux/musl) numbers.
+
+Three rules the transport must keep, each of which was a real defect before it
+was written down:
+
+1. **A transport failure is never a successful empty result.** `read` returning
+   zero bytes means end of input. If losing the host can produce that same
+   answer, input silently vanishes instead of raising.
+2. **A stale answer is not an answer.** `request_id` and `generation` are
+   checked on the way back. A completion belonging to a previous incarnation of
+   a PID must not land on the current process's descriptor table.
+3. **A closed transport stays closed.** Once the host has gone, every later call
+   fails immediately. Without that, a killed process republishes a request over
+   the closed marker and hangs on its way out.
+
+The guest writes a zero into `generation`; the transport stamps the real value,
+because the transport belongs to one process and that is where the authority
+lives. Nothing in guest-controlled data may be trusted as identity.
+
+## Blocking
+
+Only the guest blocks, and the guest never owns shared state.
+
+```
+Python calls read(fd)
+  → libc adapter invokes sbx_host_v1
+  → the process worker parks in Atomics.wait
+  → the kernel thread waits for input on its own event loop
+  → the kernel writes the response and notifies
+  → the worker resumes, read() returns
+```
+
+Keyboard input, network responses and storage completions must reach the
+*kernel*, not arrive as messages the blocked worker would have to process. A
+blocked worker processes nothing.
+
+This requires `SharedArrayBuffer` and `Atomics.wait`, so a browser host must be
+cross-origin isolated, and the guest must never run on the main thread.
+
+## Status
+
+Implemented and covered by `test/python-abi/`: files, descriptors, pipes,
+readiness, time, identity, entropy. Reserved but not implemented: sockets,
+processes, signals, storage, services, threads — `handshake` reports which is
+which, so a guest can tell "not present" from "not implemented" rather than
+discovering it at the first call.
+
+## Known limit
+
+`unlink` and `rename` preserve open-file semantics only for names changed
+*through this ABI*. A name removed by the shell or the Node side cannot be
+intercepted, and an open description will then fail with `ENOENT` rather than
+reading stale bytes. Making that hold everywhere means moving the operations
+into the volume itself, which is M3 work.
+
+## Errno translation is not a formality
+
+The canonical values here are Linux's. Emscripten's musl uses different
+numbers — `ENOENT` is 44 there and 2 here, and 2 is `EACCES`. Passing a kernel
+errno through an Emscripten adapter unchanged turns "no such file" into
+"permission denied", and the symptom is a `PermissionError` carrying the errno
+of an entirely different failure, several layers from the cause. Every adapter
+translates at its own edge; `src/runtime/python/sbxfs.ts` does it by name,
+against the build's own `ERRNO_CODES`, so it cannot drift from the interpreter
+it is loaded into.

package/docs/python/architecture.md

@@ -0,0 +1,94 @@
+# Python on SandboxedJs — architecture
+
+The goal is an operating-system compatibility layer with an owned Python
+distribution on top of it, not an interpreter swap. Replacing Pyodide with a
+differently-built CPython in the same lifecycle would inherit the same problems,
+because the problems are not in the interpreter.
+
+## What "our own Python" means here
+
+A **runtime distribution**, not a new language implementation. SandboxedJs owns
+the OS-facing ABI, the process model, the filesystem integration, the build
+pipeline and the tests. Upstream CPython supplies Python semantics, adapted by a
+maintained patch series. Building from source gives ownership and
+reproducibility; stability comes from process isolation, consistent semantics
+and systematic testing.
+
+## Governing rules
+
+1. One active Python process owns one interpreter instance.
+2. One kernel authority owns shared mutable container resources.
+3. Blocking callers never own the services they are waiting for.
+4. All guest I/O crosses an explicit, versioned boundary.
+5. Processes share files and channels — not Python globals.
+6. Every resource has an owner and a cleanup path.
+
+## Shape
+
+```
+browser app / shell                  kernel owner (volume, process table)
+        │                                     │
+        ├── preview service worker ───────────┤
+        │                                     ├── VFS + descriptors + storage
+        │                                     ├── process supervisor + signals
+        │                                     └── virtual sockets + net policy
+        │                                             ▲
+   python worker A ──── sbx_host_v1 ────────────────── ┤
+   python worker B ──── sbx_host_v1 ────────────────── ┤
+   node workers ────────────────────────────────────── ┘
+```
+
+Kernel services stay on the host that owns them today; Python runs in dedicated
+workers and calls back. Moving the kernel into its own worker is a later
+optimisation with its own migration plan, driven by measured UI responsiveness,
+and is not a prerequisite for any of this.
+
+## Three compatibility targets
+
+| Target | Behaviour | Mechanism |
+|---|---|---|
+| Python development | scripts, REPL, venv, pip, subprocesses, local servers | CPython to Wasm plus this kernel |
+| Scientific / ML | supported packages, model inference | recompiled extensions, dedicated compute services |
+| Linux binaries | existing Linux executables and wheels | optional CPU emulation running a real Linux guest |
+
+The first is the product. The third is a separate execution backend and must
+never silently substitute for the first when an install fails: it changes
+architecture, path semantics, performance and persistence.
+
+## What exists today
+
+- `python-runtime/abi/` — the versioned ABI and its generated bindings.
+- `src/kernel/open-file.ts`, `src/kernel/descriptors.ts` — pathnames, inodes,
+  open-file descriptions and per-process descriptor tables.
+- `src/runtime/python/` — framing, the blocking transport, the kernel-side
+  dispatcher, the typed guest client, the Emscripten filesystem bridge, the
+  release manifest, and the per-process supervisor and worker.
+- `python-runtime/native/probe/` — a C program that exercises the whole path.
+- `python-runtime/scripts/` — the pinned build: fetch and verify, host Python,
+  cross-built CPython, and the packaged release with its manifest.
+- `test/python-abi/`, `test/python-runtime/` — the baseline inventory and the
+  M1 and M2 gates.
+
+### The owned interpreter
+
+`make -C python-runtime fetch python package` produces
+`out/sbx-cpython-<version>-<profile>/`: `python.js`, `python.wasm`,
+`python.data`, and a `runtime.json` naming the ABI it was built against and the
+capabilities the profile actually has. A host selects it with
+`configurePython({ backend: "sbx-cpython-wasm", manifest })`; Pyodide remains
+the default until the parity gates pass.
+
+Two things about the build are worth knowing before reading it:
+
+- Upstream's browser target links a *page* — a classic script that assigns a
+  global `Module` and runs `main` on load. A process worker needs a factory it
+  instantiates once per process, so the final link is repeated with
+  `-sMODULARIZE -sEXPORT_ES6 -sINVOKE_RUN=0`. Only the link changes.
+- Some of what a program asks an operating system for is compiled into libc
+  rather than routed through anything a host can serve. musl's `getpid` under
+  Emscripten returns a constant, so every process would report the same
+  identity however many interpreters were running. `-Wl,--wrap=` is what makes
+  those reachable, and `python-runtime/native/js/library_sbx_posix.js` is where
+  the first of them live.
+
+See [release-gates.md](release-gates.md) for what each milestone has to prove.

package/docs/python/baseline-inventory.md

@@ -0,0 +1,54 @@
+# M0 — baseline inventory
+
+What the *current* Python integration does, measured rather than remembered, so
+the owned runtime has something to be compared against. Every entry is pinned by
+a test in `test/python-abi/baseline.test.ts`, which asserts today's behaviour —
+including the wrong answers, on purpose. Turning one of those assertions around
+is what a milestone landing looks like.
+
+Setup: `src/runtime/cpython.ts` caches **one Pyodide interpreter per container**
+(keyed on the VFS) and runs each program with a fresh globals dict.
+
+## Findings
+
+| # | Behaviour | Verified | Cause | Fixed by |
+|---|---|---|---|---|
+| 1 | A module left in `sys.modules` by one program is importable by the next, unrelated one | yes | fresh globals do not reset `sys.modules` | **fixed in M2** — one interpreter per process |
+| 2 | `os.environ` does **not** leak between programs | yes | the environment is explicitly rebound per run | — (the exception, not the rule) |
+| 3 | `asyncio.start_server` fails | yes | `python-syscalls.ts` replaces parts of `asyncio` and refuses server creation | M6 — virtual sockets |
+| 4 | Every program reports the same `os.getpid()` | yes | one interpreter is one process | **fixed in M2** — a process per program, with `getpid` wrapped to ask the kernel |
+| 5 | `subprocess.run` succeeds, via the container's command table | yes | a bridge, not a process: no fresh interpreter, no descriptor inheritance, no process groups | M3 |
+
+Fresh globals reset none of: `sys.modules`, logging handlers, registered
+callbacks, native extension state, or running tasks. Finding 2 is worth naming
+precisely because it is the one piece of state the integration *does* rebind —
+which is why "each program gets a clean slate" feels true until it isn't.
+
+## What this means for the plan
+
+- Findings 1 and 4 are integration failures, not interpreter failures. A
+  differently-built CPython dropped into the same lifecycle would reproduce both.
+  That is the argument for building the kernel first.
+- Finding 3 is a missing subsystem. No amount of interpreter work reaches it.
+- Finding 5 is the one that reads as working and is not, which makes it the most
+  expensive to leave in place.
+
+## Not yet measured
+
+Concurrency and server behaviour under load, native extension state across
+process exits, and interpreter memory growth over repeated runs. These need the
+process model of M3 before the measurement means anything.
+
+## Known gaps in the owned runtime (M2)
+
+Recorded so they are not rediscovered as surprises. None of them is hidden at
+runtime: each either refuses with a reason or is declared absent in
+`runtime.json`.
+
+| Gap | Symptom | Milestone |
+|---|---|---|
+| No packaging | `pip` is micropip and targets the Pyodide interpreter; under this backend it refuses rather than installing into the wrong interpreter | M4 |
+| No `rlcompleter` / `_pyrepl` in the stdlib image | the REPL prints `warning: can't use pyrepl` and falls back to the basic prompt, which works | M2 follow-up — a `wasm_assets` inclusion |
+| No sockets | anything binding a port, Uvicorn included, cannot start | M6 |
+| No spawn from Python | `subprocess` has no backend on this runtime | M3 |
+| Symlinks | `symlink()` returns ENOSYS through the kernel filesystem | M3 |

package/docs/python/build-on-miss.md

@@ -0,0 +1,198 @@
+# The build-on-miss boundary
+
+What happens when the installer finds no wheel it can use, and how a host may
+turn that into a build without the runtime ever deciding to.
+
+Nothing described here is deployed. `src/runtime/python/build-service.ts`
+defines the shape and ships a local fake; there is no client, no endpoint, and
+no configuration that would produce one.
+
+## The rule
+
+The browser runtime is an installer. It does not compile, and it does not ask
+anyone else to compile on its behalf.
+
+A resolver that reached out to have a wheel built would be performing a remote
+mutation the caller never asked for — spending someone else's compute,
+publishing an artifact — at exactly the moment a user is least able to notice
+it, during a routine `pip install`. So the runtime does one thing: it reports
+the miss precisely enough to act on. `test/python-runtime/build-service.test.ts`
+checks structurally that `resolver.ts`, `install.ts` and `pip-command.ts` do not
+import the build service at all.
+
+## The seam
+
+1. Resolution fails and raises a `ResolutionError` carrying a
+   `ResolutionFailure` — machine-readable, distinguishing
+   `no-compatible-distribution` (building might help) from
+   `conflicting-requirements` (it cannot).
+2. The **host** — application code, not the runtime — may call
+   `buildRequestFor(failure, { recipeRevision, … })`. It is a pure function:
+   it contacts nothing and queues nothing, and returns `null` when building
+   could not help or when no source exists to build from.
+3. The host decides whether to submit that request anywhere.
+
+## Requests converge
+
+A build request promises that its artifact is interchangeable with one built
+anywhere else. That holds only if everything which can change the artifact is
+named in the request, so `idempotencyKey` is *derived* from the requirement,
+the target ABI, the wheel tag and the recipe revision — never invented by a
+client, because a random key makes every retry a new build.
+
+What is deliberately excluded is evidence about *where* the requester saw the
+source: two clients looking at different mirrors of the same release should
+converge on one build. What is deliberately included is the ABI id, so an ABI
+rollover does not serve a wheel the new runtime cannot load, and the recipe
+revision, since the same source built under different rules is a different
+artifact.
+
+## States
+
+`queued`, `resolving`, `building-host-tools`,
+`building-target-dependencies`, `building-wheel`, `testing`, then one of
+`published`, `unsupported`, `failed`, `cancelled`.
+
+The intermediate states are the ones someone waiting would ask about, and they
+fail differently: `building-target-dependencies` failing means a native library
+did not cross compile, which is a different report from the package's own build
+failing.
+
+`unsupported` is terminal and separate from `failed` on purpose. "This package
+cannot work on this platform" is worth caching forever and showing to the user
+as a fact; "this build did not succeed" may be worth retrying. Collapsing them
+makes a permanent answer look transient, and every client retries it forever.
+
+Cancelling something already terminal does not rewrite its outcome: a published
+wheel does not become uncancelled work because someone asked late.
+
+## Security requirements
+
+These are requirements on any implementation, stated here because the request
+carries the policy and an audit should be answerable from the request alone. A
+client-supplied policy is a statement of intent, never a grant of permission —
+a service must enforce its own.
+
+- **Source allowlist.** Fetch only from named hosts; the default is PyPI.
+- **No network during the build.** A build that can reach the network can fetch
+  an unpinned dependency, and then the artifact depends on the day it was
+  built. Everything needed is pinned before the build starts.
+- **Resource limits.** Wall-clock and memory ceilings, so a pathological build
+  cannot occupy a worker indefinitely.
+- **Nothing published before it runs.** A wheel that compiled is not a wheel
+  that works; the `testing` state exists so that publication follows an actual
+  import and exercise in the owned runtime, as `package-cohorts.test.ts` does
+  locally.
+- **Digest verification.** Immutable artifact locations and recorded hashes. A
+  wheel that could be replaced under a URL makes every recorded digest a lie.
+- **Auditability.** The provenance record the wheel already carries — source
+  digest, build tools, native dependencies, ABI, recipe revision, patches — is
+  echoed in the result for clients that will not open the archive.
+
+## Building here: `buildFromSource`
+
+The seam above describes talking to a service. The same boundary also has a
+local implementation, so a host with the toolchain does not need one.
+
+```js
+configurePython({ buildFromSource: true });                       // build here
+configurePython({ buildFromSource: "http://localhost:4180/build" }); // ask a machine
+```
+
+`true` is the default **only** where building can possibly work: a Node
+process, running from a checkout that contains the build pipeline. A browser
+has no compiler and will not get one; a published package has no pipeline. In
+both, attempting a build would replace a clear "no wheel for this package" with
+a hang or a confusing failure, so both keep reporting instead.
+
+When `pip` hits a `no-compatible-distribution` failure for a package that has a
+source distribution, it asks the builder, then retries the install once against
+the index that now contains the wheel. The loop is bounded, because each pass
+can only make progress by adding one package — without a bound, a graph whose
+every member is unbuildable would rebuild forever rather than report.
+
+Two failure classes are deliberately *not* built: conflicting requirements,
+which compiling cannot fix, and packages with no source distribution, which
+have nothing to build from.
+
+### Where the recipe comes from
+
+`python-runtime/scripts/auto_recipe.py` writes one. A recipe states which
+package, where its verified source is, and which backend drives it — and PyPI
+publishes the first two while the source declares the third, so requiring a
+person to transcribe them is what made every new package a small project.
+
+Nothing that affects the artifact is inferred. Compiler flags still come from
+`abi/extension-abi.json`, and a package needing a patch, a native library or a
+build-environment switch still needs those declared by hand. A build backend
+with no adapter is refused **by name** rather than attempted — attempting it
+fails deep inside someone else's build system, where the message is about a
+missing CMake rather than about this pipeline not supporting CMake.
+
+Build requirements a package declares and the lock has never seen are pinned
+at that moment rather than dropped. Dropping them is what made `ujson` fail
+with `ModuleNotFoundError: setuptools_scm` raised from inside its own
+`setup.py` — a message about the package, caused by the generator quietly
+omitting a requirement the package had stated plainly.
+
+## Serving builds to a browser
+
+```bash
+npx sandboxedjs-build-wheels 4180
+```
+
+It serves the wheel index over HTTP and builds on request, binding loopback
+only — a build runs a package's own build system, which is arbitrary code
+execution by design, so exposing it to a network hands that to whoever can
+reach it.
+
+The exchange is a package name in, a verdict out. The wheel itself comes back
+the ordinary way, fetched from the index and checked against its digest like
+every other wheel; a service that returned bytes directly would bypass that.
+
+It is submit-and-poll, not one long request. A build takes minutes, and a
+connection held open that long is dropped somewhere in between — which reads
+to the caller as the service being unreachable while it is in fact working.
+
+## What is not decided here
+
+Whether to run a *hosted* service, where, and who pays for it. That is the
+user's decision, and this project does not own cloud accounts, deployment,
+billing or signing infrastructure.
+
+## What "no wheel" actually means, and what it does not
+
+A package with no compiled extension needs no wheel of ours: `requests`, `rich`
+and everything else pure-Python installs from PyPI unchanged. A package with a
+compiled extension needs one built for `cp313-cp313-emscripten_5_0_6_wasm32`,
+because no such wheel is published anywhere.
+
+There is no compiler inside the container and there is not going to be one. A
+C/C++ toolchain targeting WebAssembly is clang, lld and a sysroot -- hundreds of
+megabytes before a single package is built -- and it would still not cover the
+Fortran in SciPy or the Rust in pydantic-core and cryptography. Shipping that to
+a browser tab to install one package is not a trade worth making, and calling it
+"free" ignores what serving it costs. So the boundary stays where it is: a host
+with the toolchain builds locally, a host without one points at a build service,
+and a host with neither is told plainly that no wheel exists rather than being
+left to wait on a compiler that cannot run.
+
+A local build that fails because the *build machine* lacks a package now says
+so and names what to install, rather than printing the backend's traceback:
+`pip install scipy` reported a `ModuleNotFoundError` for `mesonpy` where it
+meant "install meson-python into the interpreter running this pipeline".
+
+## Standard library coverage
+
+276 of the 290 modules in `sys.stdlib_module_names` import. Of the rest:
+
+- `msvcrt`, `nt`, `winreg`, `winsound` are Windows-only. CPython on Linux or
+  macOS does not have them either; a program importing `msvcrt` unguarded is
+  already broken everywhere but Windows.
+- `tkinter`, `turtle`, `turtledemo`, `idlelib`, `curses`, `readline` need a GUI
+  or a terminal device that a container in a page does not have.
+- `antigravity`, `pydoc_data`, `webbrowser` are absent for no good reason and
+  are cheap to add; they live in the interpreter's data image, so adding them
+  means rebuilding it.
+- `ctypes` is the one real gap. It needs libffi cross-compiled and linked into
+  the image, and packages that import it unconditionally will fail until then.