sandboxedjs 0.2.10 → 0.2.12

package/docs/python/build-on-miss.md

@@ -0,0 +1,198 @@
+# The build-on-miss boundary
+
+What happens when the installer finds no wheel it can use, and how a host may
+turn that into a build without the runtime ever deciding to.
+
+Nothing described here is deployed. `src/runtime/python/build-service.ts`
+defines the shape and ships a local fake; there is no client, no endpoint, and
+no configuration that would produce one.
+
+## The rule
+
+The browser runtime is an installer. It does not compile, and it does not ask
+anyone else to compile on its behalf.
+
+A resolver that reached out to have a wheel built would be performing a remote
+mutation the caller never asked for — spending someone else's compute,
+publishing an artifact — at exactly the moment a user is least able to notice
+it, during a routine `pip install`. So the runtime does one thing: it reports
+the miss precisely enough to act on. `test/python-runtime/build-service.test.ts`
+checks structurally that `resolver.ts`, `install.ts` and `pip-command.ts` do not
+import the build service at all.
+
+## The seam
+
+1. Resolution fails and raises a `ResolutionError` carrying a
+   `ResolutionFailure` — machine-readable, distinguishing
+   `no-compatible-distribution` (building might help) from
+   `conflicting-requirements` (it cannot).
+2. The **host** — application code, not the runtime — may call
+   `buildRequestFor(failure, { recipeRevision, … })`. It is a pure function:
+   it contacts nothing and queues nothing, and returns `null` when building
+   could not help or when no source exists to build from.
+3. The host decides whether to submit that request anywhere.
+
+## Requests converge
+
+A build request promises that its artifact is interchangeable with one built
+anywhere else. That holds only if everything which can change the artifact is
+named in the request, so `idempotencyKey` is *derived* from the requirement,
+the target ABI, the wheel tag and the recipe revision — never invented by a
+client, because a random key makes every retry a new build.
+
+What is deliberately excluded is evidence about *where* the requester saw the
+source: two clients looking at different mirrors of the same release should
+converge on one build. What is deliberately included is the ABI id, so an ABI
+rollover does not serve a wheel the new runtime cannot load, and the recipe
+revision, since the same source built under different rules is a different
+artifact.
+
+## States
+
+`queued`, `resolving`, `building-host-tools`,
+`building-target-dependencies`, `building-wheel`, `testing`, then one of
+`published`, `unsupported`, `failed`, `cancelled`.
+
+The intermediate states are the ones someone waiting would ask about, and they
+fail differently: `building-target-dependencies` failing means a native library
+did not cross compile, which is a different report from the package's own build
+failing.
+
+`unsupported` is terminal and separate from `failed` on purpose. "This package
+cannot work on this platform" is worth caching forever and showing to the user
+as a fact; "this build did not succeed" may be worth retrying. Collapsing them
+makes a permanent answer look transient, and every client retries it forever.
+
+Cancelling something already terminal does not rewrite its outcome: a published
+wheel does not become uncancelled work because someone asked late.
+
+## Security requirements
+
+These are requirements on any implementation, stated here because the request
+carries the policy and an audit should be answerable from the request alone. A
+client-supplied policy is a statement of intent, never a grant of permission —
+a service must enforce its own.
+
+- **Source allowlist.** Fetch only from named hosts; the default is PyPI.
+- **No network during the build.** A build that can reach the network can fetch
+  an unpinned dependency, and then the artifact depends on the day it was
+  built. Everything needed is pinned before the build starts.
+- **Resource limits.** Wall-clock and memory ceilings, so a pathological build
+  cannot occupy a worker indefinitely.
+- **Nothing published before it runs.** A wheel that compiled is not a wheel
+  that works; the `testing` state exists so that publication follows an actual
+  import and exercise in the owned runtime, as `package-cohorts.test.ts` does
+  locally.
+- **Digest verification.** Immutable artifact locations and recorded hashes. A
+  wheel that could be replaced under a URL makes every recorded digest a lie.
+- **Auditability.** The provenance record the wheel already carries — source
+  digest, build tools, native dependencies, ABI, recipe revision, patches — is
+  echoed in the result for clients that will not open the archive.
+
+## Building here: `buildFromSource`
+
+The seam above describes talking to a service. The same boundary also has a
+local implementation, so a host with the toolchain does not need one.
+
+```js
+configurePython({ buildFromSource: true });                       // build here
+configurePython({ buildFromSource: "http://localhost:4180/build" }); // ask a machine
+```
+
+`true` is the default **only** where building can possibly work: a Node
+process, running from a checkout that contains the build pipeline. A browser
+has no compiler and will not get one; a published package has no pipeline. In
+both, attempting a build would replace a clear "no wheel for this package" with
+a hang or a confusing failure, so both keep reporting instead.
+
+When `pip` hits a `no-compatible-distribution` failure for a package that has a
+source distribution, it asks the builder, then retries the install once against
+the index that now contains the wheel. The loop is bounded, because each pass
+can only make progress by adding one package — without a bound, a graph whose
+every member is unbuildable would rebuild forever rather than report.
+
+Two failure classes are deliberately *not* built: conflicting requirements,
+which compiling cannot fix, and packages with no source distribution, which
+have nothing to build from.
+
+### Where the recipe comes from
+
+`python-runtime/scripts/auto_recipe.py` writes one. A recipe states which
+package, where its verified source is, and which backend drives it — and PyPI
+publishes the first two while the source declares the third, so requiring a
+person to transcribe them is what made every new package a small project.
+
+Nothing that affects the artifact is inferred. Compiler flags still come from
+`abi/extension-abi.json`, and a package needing a patch, a native library or a
+build-environment switch still needs those declared by hand. A build backend
+with no adapter is refused **by name** rather than attempted — attempting it
+fails deep inside someone else's build system, where the message is about a
+missing CMake rather than about this pipeline not supporting CMake.
+
+Build requirements a package declares and the lock has never seen are pinned
+at that moment rather than dropped. Dropping them is what made `ujson` fail
+with `ModuleNotFoundError: setuptools_scm` raised from inside its own
+`setup.py` — a message about the package, caused by the generator quietly
+omitting a requirement the package had stated plainly.
+
+## Serving builds to a browser
+
+```bash
+npx sandboxedjs-build-wheels 4180
+```
+
+It serves the wheel index over HTTP and builds on request, binding loopback
+only — a build runs a package's own build system, which is arbitrary code
+execution by design, so exposing it to a network hands that to whoever can
+reach it.
+
+The exchange is a package name in, a verdict out. The wheel itself comes back
+the ordinary way, fetched from the index and checked against its digest like
+every other wheel; a service that returned bytes directly would bypass that.
+
+It is submit-and-poll, not one long request. A build takes minutes, and a
+connection held open that long is dropped somewhere in between — which reads
+to the caller as the service being unreachable while it is in fact working.
+
+## What is not decided here
+
+Whether to run a *hosted* service, where, and who pays for it. That is the
+user's decision, and this project does not own cloud accounts, deployment,
+billing or signing infrastructure.
+
+## What "no wheel" actually means, and what it does not
+
+A package with no compiled extension needs no wheel of ours: `requests`, `rich`
+and everything else pure-Python installs from PyPI unchanged. A package with a
+compiled extension needs one built for `cp313-cp313-emscripten_5_0_6_wasm32`,
+because no such wheel is published anywhere.
+
+There is no compiler inside the container and there is not going to be one. A
+C/C++ toolchain targeting WebAssembly is clang, lld and a sysroot -- hundreds of
+megabytes before a single package is built -- and it would still not cover the
+Fortran in SciPy or the Rust in pydantic-core and cryptography. Shipping that to
+a browser tab to install one package is not a trade worth making, and calling it
+"free" ignores what serving it costs. So the boundary stays where it is: a host
+with the toolchain builds locally, a host without one points at a build service,
+and a host with neither is told plainly that no wheel exists rather than being
+left to wait on a compiler that cannot run.
+
+A local build that fails because the *build machine* lacks a package now says
+so and names what to install, rather than printing the backend's traceback:
+`pip install scipy` reported a `ModuleNotFoundError` for `mesonpy` where it
+meant "install meson-python into the interpreter running this pipeline".
+
+## Standard library coverage
+
+276 of the 290 modules in `sys.stdlib_module_names` import. Of the rest:
+
+- `msvcrt`, `nt`, `winreg`, `winsound` are Windows-only. CPython on Linux or
+  macOS does not have them either; a program importing `msvcrt` unguarded is
+  already broken everywhere but Windows.
+- `tkinter`, `turtle`, `turtledemo`, `idlelib`, `curses`, `readline` need a GUI
+  or a terminal device that a container in a page does not have.
+- `antigravity`, `pydoc_data`, `webbrowser` are absent for no good reason and
+  are cheap to add; they live in the interpreter's data image, so adding them
+  means rebuilding it.
+- `ctypes` is the one real gap. It needs libffi cross-compiled and linked into
+  the image, and packages that import it unconditionally will fail until then.

package/docs/python/compatibility.md

@@ -0,0 +1,206 @@
+# Package compatibility report
+
+What this pipeline has actually built and run, measured on real packages rather
+than on the fixtures written to succeed. Every wheel below was produced by
+`scripts/build_extension.py` from a pinned, hash-verified source, with no
+package-specific code path anywhere in the builder.
+
+ABI for every entry: `sbxabi1-c2637d04695ad927`, tag
+`cp313-cp313-emscripten_5_0_6_wasm32`, Emscripten 5.0.6, CPython 3.13.5.
+
+## Status vocabulary
+
+| status | meaning |
+| --- | --- |
+| `supported-generic` | builds and runs with no recipe beyond identity and source |
+| `supported-recipe` | needs declared configuration (native library, build env, build tool) but no patch |
+| `supported-patched` | needs a visible, tested patch for an assumption false on this target |
+| `blocked-platform-capability` | the platform does not provide what the package needs; no patch can add it |
+| `blocked-toolchain` | the build system or toolchain cannot produce this artifact yet |
+| `not-yet-attempted` | not tried |
+
+## Real packages
+
+| package | version | cohort | status | backend | wheel sha256 (16) | size |
+| --- | --- | --- | --- | --- | --- | --- |
+| typing-extensions | 4.12.2 | 1 pure control | `supported-generic` | — (PyPI pure wheel) | — | — |
+| MarkupSafe | 3.0.2 | 2 setuptools C | `supported-generic` | setuptools | `00619948d4c33812` | 12 KB |
+| msgpack | 1.1.0 | 3 Cython | `supported-recipe` | setuptools | `b5336ed6394b0307` | 62 KB |
+| PyYAML | 6.0.2 | 3 + 5 Cython and native library | `supported-recipe` | setuptools | `23c9ad8ee878f41e` | 129 KB |
+| pydantic-core | 2.46.5 | 4 Rust/PyO3 | `supported-recipe` | pyo3 | `01706c4ed32d7f24` | 1951 KB |
+| pydantic-core | 2.23.2 | 4 Rust/PyO3 | `supported-recipe` | pyo3 | `e1bea0f995650d93` | 1934 KB |
+| numpy | 2.2.6 | 8 scientific stress case | `supported-recipe` | meson | `93ae7230479566b4` | 4917 KB |
+| siphash24 | 1.9 | 6 + 7 Meson, patched subproject | `supported-patched` | meson | `cf672c37b0f8e4cd` | 54 KB |
+| simplejson | 3.19.3 | — | `supported-generic` | — (PyPI pure wheel) | — | — |
+| psutil | 6.1.0 | — | `blocked-platform-capability` | setuptools | — | — |
+
+Eight independent real packages across six build shapes — pure, setuptools C,
+Cython, Cython plus a separately built native library, Rust/PyO3, and Meson via
+meson-python — install and are exercised by
+`test/python-runtime/package-cohorts.test.ts`. Every cohort in the roadmap now
+has a real package in it. That test
+also loads four of the compiled ones into a single interpreter at once, which
+is the only check that proves the shapes coexist rather than merely each
+working alone.
+
+### What each entry adds
+
+**typing-extensions** — the control. No cross build, no platform tag. Proves
+the installer still does the ordinary thing when nothing is compiled.
+
+**MarkupSafe** — one C source, no dependencies, no configuration. The purest
+measurement of the generic setuptools path: the recipe states identity and
+source and nothing else. Its documented pure-Python fallback is why the test
+asserts `markupsafe._speedups` specifically; an extension that failed to build
+would otherwise install and import cleanly.
+
+**msgpack** — a Cython extension with no external library, so a failure here
+would be about generated C rather than about linking. Needs only
+`buildRequires: ["Cython==3.1.4"]`.
+
+**PyYAML** — Cython generates `_yaml.c`, which links the
+libyaml built by the native dependency system. It selects between its pure and
+compiled implementations through its own `PYYAML_FORCE_CYTHON` and
+`PYYAML_FORCE_LIBYAML` variables, which the recipe sets through `buildEnv` —
+build configuration, which belongs in a recipe, as distinct from compiler
+flags, which never do. The test asserts `yaml.CSafeLoader` parsed the document,
+which only exists when the extension built.
+
+**pydantic-core** — Rust, cross compiled with a rebuilt `std` and wasm
+exceptions. Two versions, because pydantic pins its core exactly and resolving
+that pin is part of what the index has to get right.
+
+**simplejson** — recorded for what it shows about policy rather than about
+building. It publishes both C-accelerated platform wheels and a pure
+`py3-none-any` fallback, and the resolver takes the pure one. That is correct:
+a pure wheel is portable and carries no ABI risk. The limitation is real and
+worth stating plainly — **a package that publishes a pure fallback cannot
+currently be accelerated by supplying a compiled wheel, because the pure wheel
+always wins**. Changing that is a policy decision, not an oversight, so no
+compiled simplejson wheel is shipped.
+
+**siphash24** — the densest case, and the only one that needed a patch. Three
+things at once: a frontend that is not setuptools (meson-python, driven with a
+cross file generated from the ABI contract); two dependencies resolved through
+Meson `wrap` files, which Meson satisfies by cloning from git *during the
+build* — disabled here, with the subprojects supplied instead as pinned,
+hash-verified archives recorded in the wheel's provenance; and a patch, because
+one of those subprojects builds its library with `both_libraries`, which always
+produces a shared library. `ld.wasm` has none, and Meson refuses at configure
+time rather than at link time, so the project was unconfigurable until the
+library was made static. The test asserts the first published SipHash-2-4
+reference vector, which comes from outside this repository.
+
+**numpy** — the case this pipeline was built toward, and deliberately the last
+one attempted, so that it drove none of the design. It needed **no patch** and
+no mechanism that did not already exist for smaller packages: Meson through
+meson-python, a cross file generated from the ABI contract, pinned build tools,
+and its own documented `allow-noblas` switch for a platform with no BLAS. It
+built on the first attempt.
+
+Its assertions are numerical rather than structural. NumPy has no pure-Python
+fallback to hide behind, but a miscompiled one imports perfectly well and
+computes wrong answers, so each check has an exactly known result: `a @ a.T`,
+an inverse multiplied back to the identity, a round-trip FFT, a reduction.
+`long double` deserves naming — Meson cannot run a program on the target to
+learn its layout, so the value is derived from the pinned compiler's own
+`__LDBL_MANT_DIG__` (113, little-endian, so `IEEE_QUAD_LE`) and the built
+extension reports the matching 16-byte itemsize. Getting that wrong would have
+been silently wrong arithmetic rather than a build failure.
+
+NumPy's *own* test suite runs inside the runtime, which is the strongest
+evidence in this document because none of it was written here:
+
+| module | result |
+| --- | --- |
+| `numpy._core.tests.test_umath` | 4388 passed, 362 skipped, 2 xfailed, 5 xpassed |
+| `numpy.linalg.tests.test_linalg` | 416 passed, 21 skipped, 2 xfailed |
+| `numpy.fft.tests.test_pocketfft` | 144 passed, 4 skipped |
+
+4,948 upstream tests pass and none fails. Reproduce with `pip install numpy
+pytest hypothesis` in a container, then:
+
+```bash
+python3 -m pytest --pyargs numpy._core.tests.test_umath -q --capture=no
+```
+
+`--capture=no` is required, and the reason is a runtime limitation rather than
+anything to do with NumPy: pytest's default output capture opens a temporary
+file descriptor and reading it back fails with `OSError: [Errno 8] Bad file
+descriptor`, so *no* tests run at all. It belongs with the other pre-existing
+process- and descriptor-level gaps recorded in the project state, not with
+package support.
+
+**psutil** — attempted deliberately as a negative case, and the failure is more
+useful than the success would have been.
+
+## The psutil finding
+
+Command: `python3 scripts/build_extension.py recipes/psutil/recipe.json`
+Source: `psutil-6.1.0.tar.gz`, sha256 `353815f5…c7a`
+
+Earliest unsupported assumption, quoted from the build:
+
+```
+emcc … -DPSUTIL_POSIX=1 -DPSUTIL_OSX=1 … -c psutil/_psutil_common.c
+psutil/_psutil_common.h:131:14: fatal error: 'mach/mach_time.h' file not found
+```
+
+`-DPSUTIL_OSX=1` is the whole finding. `sysconfig` has been pointed at the
+target throughout, so the compiler, its flags and the extension suffix are all
+correct — but psutil's `setup.py` selects its platform implementation from
+`sys.platform`, which reports the *running* interpreter's operating system.
+Under a cross build that is the build machine, so on macOS it compiled the
+Darwin backend and asked for a Mach header.
+
+This is assumption 7 in `cross-build.md` in its most common concrete form, and
+it generalises well beyond psutil: any package that branches on `sys.platform`
+in its build will describe the build machine. It is worth recording that
+`_PYTHON_HOST_PLATFORM` does **not** fix this — that variable changes
+`sysconfig.get_platform()`, not `sys.platform`.
+
+psutil is nevertheless classified `blocked-platform-capability` rather than
+`supported-patched`. A patch could make it select a POSIX backend, but psutil
+reads process and system tables through syscalls Emscripten does not implement;
+the capability is absent from the platform, not from the build. Patching it
+would convert a clear build failure into a runtime one, which is the trade this
+project refuses everywhere else.
+
+## Cohorts
+
+All eight cohorts now have a real package. Cohorts 6 (a package requiring a
+visible cross-compilation patch) and 7 (a PEP 517 backend that is not
+setuptools) are both covered by siphash24; cohort 8 by NumPy.
+
+## Packages with no recipe
+
+Everything above has a recipe checked in, because each was measured
+deliberately. Most packages need none: `pip install <anything>` will generate
+one, build the wheel and install it, when a builder is enabled — see
+`build-on-miss.md`. `ujson` is the worked example, built with no recipe, no
+patch and no hand-pinned tool:
+
+```
+$ pip install ujson
+Collecting ujson
+Building ujson from source (no wheel for this runtime yet)
+Built ujson
+Successfully installed ujson-6.0.0
+```
+
+A package that cannot be built this way still lands in the vocabulary above
+rather than as an opaque failure: a backend with no adapter is
+`blocked-toolchain` and says which backend, and a package with no source
+distribution says that instead.
+
+## Reproducing
+
+```bash
+make -C python-runtime native-deps
+python3 python-runtime/scripts/build_extension.py python-runtime/recipes/<name>/recipe.json
+python3 python-runtime/scripts/build_index.py
+npx vitest run test/python-runtime/package-cohorts.test.ts
+```
+
+Every wheel above rebuilds byte-identically; the digests in this table are
+therefore checkable rather than decorative.