sandboxedjs 0.2.11 → 0.2.13

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (48) hide show
  1. package/README.md +21 -9
  2. package/assets/logo.png +0 -0
  3. package/bin/sandboxedjs-egress.mjs +25 -10
  4. package/dist/index.cjs +87 -3
  5. package/dist/index.js +87 -3
  6. package/dist/python/python.data +140 -141
  7. package/dist/python/python.js +1 -1
  8. package/dist/python/python.wasm +0 -0
  9. package/dist/python/runtime.json +3 -3
  10. package/dist/python-worker.js +9 -0
  11. package/dist/service-worker.js +3 -2
  12. package/docs/agent/COMMANDS.md +85 -0
  13. package/docs/agent/DECISION-TREE.md +84 -0
  14. package/docs/agent/INVARIANTS.md +40 -0
  15. package/docs/agent/LAUNCH-PROMPT.md +37 -0
  16. package/docs/agent/LOOP.md +84 -0
  17. package/docs/agent/README.md +77 -0
  18. package/docs/agent/ROADMAP.md +37 -0
  19. package/docs/agent/STATE.md +93 -0
  20. package/docs/agent/tasks/00-verify-inherited-work.md +36 -0
  21. package/docs/agent/tasks/01-authoritative-metadata.md +34 -0
  22. package/docs/agent/tasks/02-native-dependencies.md +35 -0
  23. package/docs/agent/tasks/03-reproducible-inputs.md +27 -0
  24. package/docs/agent/tasks/04-build-frontends.md +27 -0
  25. package/docs/agent/tasks/05-registry-integration.md +27 -0
  26. package/docs/agent/tasks/06-package-cohorts.md +42 -0
  27. package/docs/agent/tasks/07-build-on-miss-boundary.md +31 -0
  28. package/docs/browser-runtime-architecture.md +132 -0
  29. package/docs/compatibility-implementation-plan.md +98 -0
  30. package/docs/developer-tool-packs.md +134 -0
  31. package/docs/frontend-automation.md +49 -0
  32. package/docs/fullstack-deployment.md +163 -0
  33. package/docs/handoff.md +275 -0
  34. package/docs/original-x64.md +39 -0
  35. package/docs/platform-hardening.md +49 -0
  36. package/docs/python/abi.md +97 -0
  37. package/docs/python/architecture.md +94 -0
  38. package/docs/python/baseline-inventory.md +54 -0
  39. package/docs/python/build-on-miss.md +198 -0
  40. package/docs/python/compatibility.md +206 -0
  41. package/docs/python/cross-build.md +354 -0
  42. package/docs/python/extensions.md +282 -0
  43. package/docs/python/release-gates.md +46 -0
  44. package/docs/python/virtual-sockets-plan.md +331 -0
  45. package/docs/runtime-lifecycle-fixes.md +39 -0
  46. package/docs/server-previews.md +268 -0
  47. package/docs/virtual-browser.md +120 -0
  48. package/package.json +6 -3
@@ -0,0 +1,94 @@
1
+ # Python on SandboxedJs — architecture
2
+
3
+ The goal is an operating-system compatibility layer with an owned Python
4
+ distribution on top of it, not an interpreter swap. Replacing Pyodide with a
5
+ differently-built CPython in the same lifecycle would inherit the same problems,
6
+ because the problems are not in the interpreter.
7
+
8
+ ## What "our own Python" means here
9
+
10
+ A **runtime distribution**, not a new language implementation. SandboxedJs owns
11
+ the OS-facing ABI, the process model, the filesystem integration, the build
12
+ pipeline and the tests. Upstream CPython supplies Python semantics, adapted by a
13
+ maintained patch series. Building from source gives ownership and
14
+ reproducibility; stability comes from process isolation, consistent semantics
15
+ and systematic testing.
16
+
17
+ ## Governing rules
18
+
19
+ 1. One active Python process owns one interpreter instance.
20
+ 2. One kernel authority owns shared mutable container resources.
21
+ 3. Blocking callers never own the services they are waiting for.
22
+ 4. All guest I/O crosses an explicit, versioned boundary.
23
+ 5. Processes share files and channels — not Python globals.
24
+ 6. Every resource has an owner and a cleanup path.
25
+
26
+ ## Shape
27
+
28
+ ```
29
+ browser app / shell kernel owner (volume, process table)
30
+ │ │
31
+ ├── preview service worker ───────────┤
32
+ │ ├── VFS + descriptors + storage
33
+ │ ├── process supervisor + signals
34
+ │ └── virtual sockets + net policy
35
+ │ ▲
36
+ python worker A ──── sbx_host_v1 ────────────────── ┤
37
+ python worker B ──── sbx_host_v1 ────────────────── ┤
38
+ node workers ────────────────────────────────────── ┘
39
+ ```
40
+
41
+ Kernel services stay on the host that owns them today; Python runs in dedicated
42
+ workers and calls back. Moving the kernel into its own worker is a later
43
+ optimisation with its own migration plan, driven by measured UI responsiveness,
44
+ and is not a prerequisite for any of this.
45
+
46
+ ## Three compatibility targets
47
+
48
+ | Target | Behaviour | Mechanism |
49
+ |---|---|---|
50
+ | Python development | scripts, REPL, venv, pip, subprocesses, local servers | CPython to Wasm plus this kernel |
51
+ | Scientific / ML | supported packages, model inference | recompiled extensions, dedicated compute services |
52
+ | Linux binaries | existing Linux executables and wheels | optional CPU emulation running a real Linux guest |
53
+
54
+ The first is the product. The third is a separate execution backend and must
55
+ never silently substitute for the first when an install fails: it changes
56
+ architecture, path semantics, performance and persistence.
57
+
58
+ ## What exists today
59
+
60
+ - `python-runtime/abi/` — the versioned ABI and its generated bindings.
61
+ - `src/kernel/open-file.ts`, `src/kernel/descriptors.ts` — pathnames, inodes,
62
+ open-file descriptions and per-process descriptor tables.
63
+ - `src/runtime/python/` — framing, the blocking transport, the kernel-side
64
+ dispatcher, the typed guest client, the Emscripten filesystem bridge, the
65
+ release manifest, and the per-process supervisor and worker.
66
+ - `python-runtime/native/probe/` — a C program that exercises the whole path.
67
+ - `python-runtime/scripts/` — the pinned build: fetch and verify, host Python,
68
+ cross-built CPython, and the packaged release with its manifest.
69
+ - `test/python-abi/`, `test/python-runtime/` — the baseline inventory and the
70
+ M1 and M2 gates.
71
+
72
+ ### The owned interpreter
73
+
74
+ `make -C python-runtime fetch python package` produces
75
+ `out/sbx-cpython-<version>-<profile>/`: `python.js`, `python.wasm`,
76
+ `python.data`, and a `runtime.json` naming the ABI it was built against and the
77
+ capabilities the profile actually has. A host selects it with
78
+ `configurePython({ backend: "sbx-cpython-wasm", manifest })`; Pyodide remains
79
+ the default until the parity gates pass.
80
+
81
+ Two things about the build are worth knowing before reading it:
82
+
83
+ - Upstream's browser target links a *page* — a classic script that assigns a
84
+ global `Module` and runs `main` on load. A process worker needs a factory it
85
+ instantiates once per process, so the final link is repeated with
86
+ `-sMODULARIZE -sEXPORT_ES6 -sINVOKE_RUN=0`. Only the link changes.
87
+ - Some of what a program asks an operating system for is compiled into libc
88
+ rather than routed through anything a host can serve. musl's `getpid` under
89
+ Emscripten returns a constant, so every process would report the same
90
+ identity however many interpreters were running. `-Wl,--wrap=` is what makes
91
+ those reachable, and `python-runtime/native/js/library_sbx_posix.js` is where
92
+ the first of them live.
93
+
94
+ See [release-gates.md](release-gates.md) for what each milestone has to prove.
@@ -0,0 +1,54 @@
1
+ # M0 — baseline inventory
2
+
3
+ What the *current* Python integration does, measured rather than remembered, so
4
+ the owned runtime has something to be compared against. Every entry is pinned by
5
+ a test in `test/python-abi/baseline.test.ts`, which asserts today's behaviour —
6
+ including the wrong answers, on purpose. Turning one of those assertions around
7
+ is what a milestone landing looks like.
8
+
9
+ Setup: `src/runtime/cpython.ts` caches **one Pyodide interpreter per container**
10
+ (keyed on the VFS) and runs each program with a fresh globals dict.
11
+
12
+ ## Findings
13
+
14
+ | # | Behaviour | Verified | Cause | Fixed by |
15
+ |---|---|---|---|---|
16
+ | 1 | A module left in `sys.modules` by one program is importable by the next, unrelated one | yes | fresh globals do not reset `sys.modules` | **fixed in M2** — one interpreter per process |
17
+ | 2 | `os.environ` does **not** leak between programs | yes | the environment is explicitly rebound per run | — (the exception, not the rule) |
18
+ | 3 | `asyncio.start_server` fails | yes | `python-syscalls.ts` replaces parts of `asyncio` and refuses server creation | M6 — virtual sockets |
19
+ | 4 | Every program reports the same `os.getpid()` | yes | one interpreter is one process | **fixed in M2** — a process per program, with `getpid` wrapped to ask the kernel |
20
+ | 5 | `subprocess.run` succeeds, via the container's command table | yes | a bridge, not a process: no fresh interpreter, no descriptor inheritance, no process groups | M3 |
21
+
22
+ Fresh globals reset none of: `sys.modules`, logging handlers, registered
23
+ callbacks, native extension state, or running tasks. Finding 2 is worth naming
24
+ precisely because it is the one piece of state the integration *does* rebind —
25
+ which is why "each program gets a clean slate" feels true until it isn't.
26
+
27
+ ## What this means for the plan
28
+
29
+ - Findings 1 and 4 are integration failures, not interpreter failures. A
30
+ differently-built CPython dropped into the same lifecycle would reproduce both.
31
+ That is the argument for building the kernel first.
32
+ - Finding 3 is a missing subsystem. No amount of interpreter work reaches it.
33
+ - Finding 5 is the one that reads as working and is not, which makes it the most
34
+ expensive to leave in place.
35
+
36
+ ## Not yet measured
37
+
38
+ Concurrency and server behaviour under load, native extension state across
39
+ process exits, and interpreter memory growth over repeated runs. These need the
40
+ process model of M3 before the measurement means anything.
41
+
42
+ ## Known gaps in the owned runtime (M2)
43
+
44
+ Recorded so they are not rediscovered as surprises. None of them is hidden at
45
+ runtime: each either refuses with a reason or is declared absent in
46
+ `runtime.json`.
47
+
48
+ | Gap | Symptom | Milestone |
49
+ |---|---|---|
50
+ | No packaging | `pip` is micropip and targets the Pyodide interpreter; under this backend it refuses rather than installing into the wrong interpreter | M4 |
51
+ | No `rlcompleter` / `_pyrepl` in the stdlib image | the REPL prints `warning: can't use pyrepl` and falls back to the basic prompt, which works | M2 follow-up — a `wasm_assets` inclusion |
52
+ | No sockets | anything binding a port, Uvicorn included, cannot start | M6 |
53
+ | No spawn from Python | `subprocess` has no backend on this runtime | M3 |
54
+ | Symlinks | `symlink()` returns ENOSYS through the kernel filesystem | M3 |
@@ -0,0 +1,198 @@
1
+ # The build-on-miss boundary
2
+
3
+ What happens when the installer finds no wheel it can use, and how a host may
4
+ turn that into a build without the runtime ever deciding to.
5
+
6
+ Nothing described here is deployed. `src/runtime/python/build-service.ts`
7
+ defines the shape and ships a local fake; there is no client, no endpoint, and
8
+ no configuration that would produce one.
9
+
10
+ ## The rule
11
+
12
+ The browser runtime is an installer. It does not compile, and it does not ask
13
+ anyone else to compile on its behalf.
14
+
15
+ A resolver that reached out to have a wheel built would be performing a remote
16
+ mutation the caller never asked for — spending someone else's compute,
17
+ publishing an artifact — at exactly the moment a user is least able to notice
18
+ it, during a routine `pip install`. So the runtime does one thing: it reports
19
+ the miss precisely enough to act on. `test/python-runtime/build-service.test.ts`
20
+ checks structurally that `resolver.ts`, `install.ts` and `pip-command.ts` do not
21
+ import the build service at all.
22
+
23
+ ## The seam
24
+
25
+ 1. Resolution fails and raises a `ResolutionError` carrying a
26
+ `ResolutionFailure` — machine-readable, distinguishing
27
+ `no-compatible-distribution` (building might help) from
28
+ `conflicting-requirements` (it cannot).
29
+ 2. The **host** — application code, not the runtime — may call
30
+ `buildRequestFor(failure, { recipeRevision, … })`. It is a pure function:
31
+ it contacts nothing and queues nothing, and returns `null` when building
32
+ could not help or when no source exists to build from.
33
+ 3. The host decides whether to submit that request anywhere.
34
+
35
+ ## Requests converge
36
+
37
+ A build request promises that its artifact is interchangeable with one built
38
+ anywhere else. That holds only if everything which can change the artifact is
39
+ named in the request, so `idempotencyKey` is *derived* from the requirement,
40
+ the target ABI, the wheel tag and the recipe revision — never invented by a
41
+ client, because a random key makes every retry a new build.
42
+
43
+ What is deliberately excluded is evidence about *where* the requester saw the
44
+ source: two clients looking at different mirrors of the same release should
45
+ converge on one build. What is deliberately included is the ABI id, so an ABI
46
+ rollover does not serve a wheel the new runtime cannot load, and the recipe
47
+ revision, since the same source built under different rules is a different
48
+ artifact.
49
+
50
+ ## States
51
+
52
+ `queued`, `resolving`, `building-host-tools`,
53
+ `building-target-dependencies`, `building-wheel`, `testing`, then one of
54
+ `published`, `unsupported`, `failed`, `cancelled`.
55
+
56
+ The intermediate states are the ones someone waiting would ask about, and they
57
+ fail differently: `building-target-dependencies` failing means a native library
58
+ did not cross compile, which is a different report from the package's own build
59
+ failing.
60
+
61
+ `unsupported` is terminal and separate from `failed` on purpose. "This package
62
+ cannot work on this platform" is worth caching forever and showing to the user
63
+ as a fact; "this build did not succeed" may be worth retrying. Collapsing them
64
+ makes a permanent answer look transient, and every client retries it forever.
65
+
66
+ Cancelling something already terminal does not rewrite its outcome: a published
67
+ wheel does not become uncancelled work because someone asked late.
68
+
69
+ ## Security requirements
70
+
71
+ These are requirements on any implementation, stated here because the request
72
+ carries the policy and an audit should be answerable from the request alone. A
73
+ client-supplied policy is a statement of intent, never a grant of permission —
74
+ a service must enforce its own.
75
+
76
+ - **Source allowlist.** Fetch only from named hosts; the default is PyPI.
77
+ - **No network during the build.** A build that can reach the network can fetch
78
+ an unpinned dependency, and then the artifact depends on the day it was
79
+ built. Everything needed is pinned before the build starts.
80
+ - **Resource limits.** Wall-clock and memory ceilings, so a pathological build
81
+ cannot occupy a worker indefinitely.
82
+ - **Nothing published before it runs.** A wheel that compiled is not a wheel
83
+ that works; the `testing` state exists so that publication follows an actual
84
+ import and exercise in the owned runtime, as `package-cohorts.test.ts` does
85
+ locally.
86
+ - **Digest verification.** Immutable artifact locations and recorded hashes. A
87
+ wheel that could be replaced under a URL makes every recorded digest a lie.
88
+ - **Auditability.** The provenance record the wheel already carries — source
89
+ digest, build tools, native dependencies, ABI, recipe revision, patches — is
90
+ echoed in the result for clients that will not open the archive.
91
+
92
+ ## Building here: `buildFromSource`
93
+
94
+ The seam above describes talking to a service. The same boundary also has a
95
+ local implementation, so a host with the toolchain does not need one.
96
+
97
+ ```js
98
+ configurePython({ buildFromSource: true }); // build here
99
+ configurePython({ buildFromSource: "http://localhost:4180/build" }); // ask a machine
100
+ ```
101
+
102
+ `true` is the default **only** where building can possibly work: a Node
103
+ process, running from a checkout that contains the build pipeline. A browser
104
+ has no compiler and will not get one; a published package has no pipeline. In
105
+ both, attempting a build would replace a clear "no wheel for this package" with
106
+ a hang or a confusing failure, so both keep reporting instead.
107
+
108
+ When `pip` hits a `no-compatible-distribution` failure for a package that has a
109
+ source distribution, it asks the builder, then retries the install once against
110
+ the index that now contains the wheel. The loop is bounded, because each pass
111
+ can only make progress by adding one package — without a bound, a graph whose
112
+ every member is unbuildable would rebuild forever rather than report.
113
+
114
+ Two failure classes are deliberately *not* built: conflicting requirements,
115
+ which compiling cannot fix, and packages with no source distribution, which
116
+ have nothing to build from.
117
+
118
+ ### Where the recipe comes from
119
+
120
+ `python-runtime/scripts/auto_recipe.py` writes one. A recipe states which
121
+ package, where its verified source is, and which backend drives it — and PyPI
122
+ publishes the first two while the source declares the third, so requiring a
123
+ person to transcribe them is what made every new package a small project.
124
+
125
+ Nothing that affects the artifact is inferred. Compiler flags still come from
126
+ `abi/extension-abi.json`, and a package needing a patch, a native library or a
127
+ build-environment switch still needs those declared by hand. A build backend
128
+ with no adapter is refused **by name** rather than attempted — attempting it
129
+ fails deep inside someone else's build system, where the message is about a
130
+ missing CMake rather than about this pipeline not supporting CMake.
131
+
132
+ Build requirements a package declares and the lock has never seen are pinned
133
+ at that moment rather than dropped. Dropping them is what made `ujson` fail
134
+ with `ModuleNotFoundError: setuptools_scm` raised from inside its own
135
+ `setup.py` — a message about the package, caused by the generator quietly
136
+ omitting a requirement the package had stated plainly.
137
+
138
+ ## Serving builds to a browser
139
+
140
+ ```bash
141
+ npx sandboxedjs-build-wheels 4180
142
+ ```
143
+
144
+ It serves the wheel index over HTTP and builds on request, binding loopback
145
+ only — a build runs a package's own build system, which is arbitrary code
146
+ execution by design, so exposing it to a network hands that to whoever can
147
+ reach it.
148
+
149
+ The exchange is a package name in, a verdict out. The wheel itself comes back
150
+ the ordinary way, fetched from the index and checked against its digest like
151
+ every other wheel; a service that returned bytes directly would bypass that.
152
+
153
+ It is submit-and-poll, not one long request. A build takes minutes, and a
154
+ connection held open that long is dropped somewhere in between — which reads
155
+ to the caller as the service being unreachable while it is in fact working.
156
+
157
+ ## What is not decided here
158
+
159
+ Whether to run a *hosted* service, where, and who pays for it. That is the
160
+ user's decision, and this project does not own cloud accounts, deployment,
161
+ billing or signing infrastructure.
162
+
163
+ ## What "no wheel" actually means, and what it does not
164
+
165
+ A package with no compiled extension needs no wheel of ours: `requests`, `rich`
166
+ and everything else pure-Python installs from PyPI unchanged. A package with a
167
+ compiled extension needs one built for `cp313-cp313-emscripten_5_0_6_wasm32`,
168
+ because no such wheel is published anywhere.
169
+
170
+ There is no compiler inside the container and there is not going to be one. A
171
+ C/C++ toolchain targeting WebAssembly is clang, lld and a sysroot -- hundreds of
172
+ megabytes before a single package is built -- and it would still not cover the
173
+ Fortran in SciPy or the Rust in pydantic-core and cryptography. Shipping that to
174
+ a browser tab to install one package is not a trade worth making, and calling it
175
+ "free" ignores what serving it costs. So the boundary stays where it is: a host
176
+ with the toolchain builds locally, a host without one points at a build service,
177
+ and a host with neither is told plainly that no wheel exists rather than being
178
+ left to wait on a compiler that cannot run.
179
+
180
+ A local build that fails because the *build machine* lacks a package now says
181
+ so and names what to install, rather than printing the backend's traceback:
182
+ `pip install scipy` reported a `ModuleNotFoundError` for `mesonpy` where it
183
+ meant "install meson-python into the interpreter running this pipeline".
184
+
185
+ ## Standard library coverage
186
+
187
+ 276 of the 290 modules in `sys.stdlib_module_names` import. Of the rest:
188
+
189
+ - `msvcrt`, `nt`, `winreg`, `winsound` are Windows-only. CPython on Linux or
190
+ macOS does not have them either; a program importing `msvcrt` unguarded is
191
+ already broken everywhere but Windows.
192
+ - `tkinter`, `turtle`, `turtledemo`, `idlelib`, `curses`, `readline` need a GUI
193
+ or a terminal device that a container in a page does not have.
194
+ - `antigravity`, `pydoc_data`, `webbrowser` are absent for no good reason and
195
+ are cheap to add; they live in the interpreter's data image, so adding them
196
+ means rebuilding it.
197
+ - `ctypes` is the one real gap. It needs libffi cross-compiled and linked into
198
+ the image, and packages that import it unconditionally will fail until then.
@@ -0,0 +1,206 @@
1
+ # Package compatibility report
2
+
3
+ What this pipeline has actually built and run, measured on real packages rather
4
+ than on the fixtures written to succeed. Every wheel below was produced by
5
+ `scripts/build_extension.py` from a pinned, hash-verified source, with no
6
+ package-specific code path anywhere in the builder.
7
+
8
+ ABI for every entry: `sbxabi1-c2637d04695ad927`, tag
9
+ `cp313-cp313-emscripten_5_0_6_wasm32`, Emscripten 5.0.6, CPython 3.13.5.
10
+
11
+ ## Status vocabulary
12
+
13
+ | status | meaning |
14
+ | --- | --- |
15
+ | `supported-generic` | builds and runs with no recipe beyond identity and source |
16
+ | `supported-recipe` | needs declared configuration (native library, build env, build tool) but no patch |
17
+ | `supported-patched` | needs a visible, tested patch for an assumption false on this target |
18
+ | `blocked-platform-capability` | the platform does not provide what the package needs; no patch can add it |
19
+ | `blocked-toolchain` | the build system or toolchain cannot produce this artifact yet |
20
+ | `not-yet-attempted` | not tried |
21
+
22
+ ## Real packages
23
+
24
+ | package | version | cohort | status | backend | wheel sha256 (16) | size |
25
+ | --- | --- | --- | --- | --- | --- | --- |
26
+ | typing-extensions | 4.12.2 | 1 pure control | `supported-generic` | — (PyPI pure wheel) | — | — |
27
+ | MarkupSafe | 3.0.2 | 2 setuptools C | `supported-generic` | setuptools | `00619948d4c33812` | 12 KB |
28
+ | msgpack | 1.1.0 | 3 Cython | `supported-recipe` | setuptools | `b5336ed6394b0307` | 62 KB |
29
+ | PyYAML | 6.0.2 | 3 + 5 Cython and native library | `supported-recipe` | setuptools | `23c9ad8ee878f41e` | 129 KB |
30
+ | pydantic-core | 2.46.5 | 4 Rust/PyO3 | `supported-recipe` | pyo3 | `01706c4ed32d7f24` | 1951 KB |
31
+ | pydantic-core | 2.23.2 | 4 Rust/PyO3 | `supported-recipe` | pyo3 | `e1bea0f995650d93` | 1934 KB |
32
+ | numpy | 2.2.6 | 8 scientific stress case | `supported-recipe` | meson | `93ae7230479566b4` | 4917 KB |
33
+ | siphash24 | 1.9 | 6 + 7 Meson, patched subproject | `supported-patched` | meson | `cf672c37b0f8e4cd` | 54 KB |
34
+ | simplejson | 3.19.3 | — | `supported-generic` | — (PyPI pure wheel) | — | — |
35
+ | psutil | 6.1.0 | — | `blocked-platform-capability` | setuptools | — | — |
36
+
37
+ Eight independent real packages across six build shapes — pure, setuptools C,
38
+ Cython, Cython plus a separately built native library, Rust/PyO3, and Meson via
39
+ meson-python — install and are exercised by
40
+ `test/python-runtime/package-cohorts.test.ts`. Every cohort in the roadmap now
41
+ has a real package in it. That test
42
+ also loads four of the compiled ones into a single interpreter at once, which
43
+ is the only check that proves the shapes coexist rather than merely each
44
+ working alone.
45
+
46
+ ### What each entry adds
47
+
48
+ **typing-extensions** — the control. No cross build, no platform tag. Proves
49
+ the installer still does the ordinary thing when nothing is compiled.
50
+
51
+ **MarkupSafe** — one C source, no dependencies, no configuration. The purest
52
+ measurement of the generic setuptools path: the recipe states identity and
53
+ source and nothing else. Its documented pure-Python fallback is why the test
54
+ asserts `markupsafe._speedups` specifically; an extension that failed to build
55
+ would otherwise install and import cleanly.
56
+
57
+ **msgpack** — a Cython extension with no external library, so a failure here
58
+ would be about generated C rather than about linking. Needs only
59
+ `buildRequires: ["Cython==3.1.4"]`.
60
+
61
+ **PyYAML** — Cython generates `_yaml.c`, which links the
62
+ libyaml built by the native dependency system. It selects between its pure and
63
+ compiled implementations through its own `PYYAML_FORCE_CYTHON` and
64
+ `PYYAML_FORCE_LIBYAML` variables, which the recipe sets through `buildEnv` —
65
+ build configuration, which belongs in a recipe, as distinct from compiler
66
+ flags, which never do. The test asserts `yaml.CSafeLoader` parsed the document,
67
+ which only exists when the extension built.
68
+
69
+ **pydantic-core** — Rust, cross compiled with a rebuilt `std` and wasm
70
+ exceptions. Two versions, because pydantic pins its core exactly and resolving
71
+ that pin is part of what the index has to get right.
72
+
73
+ **simplejson** — recorded for what it shows about policy rather than about
74
+ building. It publishes both C-accelerated platform wheels and a pure
75
+ `py3-none-any` fallback, and the resolver takes the pure one. That is correct:
76
+ a pure wheel is portable and carries no ABI risk. The limitation is real and
77
+ worth stating plainly — **a package that publishes a pure fallback cannot
78
+ currently be accelerated by supplying a compiled wheel, because the pure wheel
79
+ always wins**. Changing that is a policy decision, not an oversight, so no
80
+ compiled simplejson wheel is shipped.
81
+
82
+ **siphash24** — the densest case, and the only one that needed a patch. Three
83
+ things at once: a frontend that is not setuptools (meson-python, driven with a
84
+ cross file generated from the ABI contract); two dependencies resolved through
85
+ Meson `wrap` files, which Meson satisfies by cloning from git *during the
86
+ build* — disabled here, with the subprojects supplied instead as pinned,
87
+ hash-verified archives recorded in the wheel's provenance; and a patch, because
88
+ one of those subprojects builds its library with `both_libraries`, which always
89
+ produces a shared library. `ld.wasm` has none, and Meson refuses at configure
90
+ time rather than at link time, so the project was unconfigurable until the
91
+ library was made static. The test asserts the first published SipHash-2-4
92
+ reference vector, which comes from outside this repository.
93
+
94
+ **numpy** — the case this pipeline was built toward, and deliberately the last
95
+ one attempted, so that it drove none of the design. It needed **no patch** and
96
+ no mechanism that did not already exist for smaller packages: Meson through
97
+ meson-python, a cross file generated from the ABI contract, pinned build tools,
98
+ and its own documented `allow-noblas` switch for a platform with no BLAS. It
99
+ built on the first attempt.
100
+
101
+ Its assertions are numerical rather than structural. NumPy has no pure-Python
102
+ fallback to hide behind, but a miscompiled one imports perfectly well and
103
+ computes wrong answers, so each check has an exactly known result: `a @ a.T`,
104
+ an inverse multiplied back to the identity, a round-trip FFT, a reduction.
105
+ `long double` deserves naming — Meson cannot run a program on the target to
106
+ learn its layout, so the value is derived from the pinned compiler's own
107
+ `__LDBL_MANT_DIG__` (113, little-endian, so `IEEE_QUAD_LE`) and the built
108
+ extension reports the matching 16-byte itemsize. Getting that wrong would have
109
+ been silently wrong arithmetic rather than a build failure.
110
+
111
+ NumPy's *own* test suite runs inside the runtime, which is the strongest
112
+ evidence in this document because none of it was written here:
113
+
114
+ | module | result |
115
+ | --- | --- |
116
+ | `numpy._core.tests.test_umath` | 4388 passed, 362 skipped, 2 xfailed, 5 xpassed |
117
+ | `numpy.linalg.tests.test_linalg` | 416 passed, 21 skipped, 2 xfailed |
118
+ | `numpy.fft.tests.test_pocketfft` | 144 passed, 4 skipped |
119
+
120
+ 4,948 upstream tests pass and none fails. Reproduce with `pip install numpy
121
+ pytest hypothesis` in a container, then:
122
+
123
+ ```bash
124
+ python3 -m pytest --pyargs numpy._core.tests.test_umath -q --capture=no
125
+ ```
126
+
127
+ `--capture=no` is required, and the reason is a runtime limitation rather than
128
+ anything to do with NumPy: pytest's default output capture opens a temporary
129
+ file descriptor and reading it back fails with `OSError: [Errno 8] Bad file
130
+ descriptor`, so *no* tests run at all. It belongs with the other pre-existing
131
+ process- and descriptor-level gaps recorded in the project state, not with
132
+ package support.
133
+
134
+ **psutil** — attempted deliberately as a negative case, and the failure is more
135
+ useful than the success would have been.
136
+
137
+ ## The psutil finding
138
+
139
+ Command: `python3 scripts/build_extension.py recipes/psutil/recipe.json`
140
+ Source: `psutil-6.1.0.tar.gz`, sha256 `353815f5…c7a`
141
+
142
+ Earliest unsupported assumption, quoted from the build:
143
+
144
+ ```
145
+ emcc … -DPSUTIL_POSIX=1 -DPSUTIL_OSX=1 … -c psutil/_psutil_common.c
146
+ psutil/_psutil_common.h:131:14: fatal error: 'mach/mach_time.h' file not found
147
+ ```
148
+
149
+ `-DPSUTIL_OSX=1` is the whole finding. `sysconfig` has been pointed at the
150
+ target throughout, so the compiler, its flags and the extension suffix are all
151
+ correct — but psutil's `setup.py` selects its platform implementation from
152
+ `sys.platform`, which reports the *running* interpreter's operating system.
153
+ Under a cross build that is the build machine, so on macOS it compiled the
154
+ Darwin backend and asked for a Mach header.
155
+
156
+ This is assumption 7 in `cross-build.md` in its most common concrete form, and
157
+ it generalises well beyond psutil: any package that branches on `sys.platform`
158
+ in its build will describe the build machine. It is worth recording that
159
+ `_PYTHON_HOST_PLATFORM` does **not** fix this — that variable changes
160
+ `sysconfig.get_platform()`, not `sys.platform`.
161
+
162
+ psutil is nevertheless classified `blocked-platform-capability` rather than
163
+ `supported-patched`. A patch could make it select a POSIX backend, but psutil
164
+ reads process and system tables through syscalls Emscripten does not implement;
165
+ the capability is absent from the platform, not from the build. Patching it
166
+ would convert a clear build failure into a runtime one, which is the trade this
167
+ project refuses everywhere else.
168
+
169
+ ## Cohorts
170
+
171
+ All eight cohorts now have a real package. Cohorts 6 (a package requiring a
172
+ visible cross-compilation patch) and 7 (a PEP 517 backend that is not
173
+ setuptools) are both covered by siphash24; cohort 8 by NumPy.
174
+
175
+ ## Packages with no recipe
176
+
177
+ Everything above has a recipe checked in, because each was measured
178
+ deliberately. Most packages need none: `pip install <anything>` will generate
179
+ one, build the wheel and install it, when a builder is enabled — see
180
+ `build-on-miss.md`. `ujson` is the worked example, built with no recipe, no
181
+ patch and no hand-pinned tool:
182
+
183
+ ```
184
+ $ pip install ujson
185
+ Collecting ujson
186
+ Building ujson from source (no wheel for this runtime yet)
187
+ Built ujson
188
+ Successfully installed ujson-6.0.0
189
+ ```
190
+
191
+ A package that cannot be built this way still lands in the vocabulary above
192
+ rather than as an opaque failure: a backend with no adapter is
193
+ `blocked-toolchain` and says which backend, and a package with no source
194
+ distribution says that instead.
195
+
196
+ ## Reproducing
197
+
198
+ ```bash
199
+ make -C python-runtime native-deps
200
+ python3 python-runtime/scripts/build_extension.py python-runtime/recipes/<name>/recipe.json
201
+ python3 python-runtime/scripts/build_index.py
202
+ npx vitest run test/python-runtime/package-cohorts.test.ts
203
+ ```
204
+
205
+ Every wheel above rebuilds byte-identically; the digests in this table are
206
+ therefore checkable rather than decorative.