sandboxedjs 0.2.11 → 0.2.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +21 -9
- package/assets/logo.png +0 -0
- package/bin/sandboxedjs-egress.mjs +25 -10
- package/dist/index.cjs +87 -3
- package/dist/index.js +87 -3
- package/dist/python/python.data +140 -141
- package/dist/python/python.js +1 -1
- package/dist/python/python.wasm +0 -0
- package/dist/python/runtime.json +3 -3
- package/dist/python-worker.js +9 -0
- package/dist/service-worker.js +3 -2
- package/docs/agent/COMMANDS.md +85 -0
- package/docs/agent/DECISION-TREE.md +84 -0
- package/docs/agent/INVARIANTS.md +40 -0
- package/docs/agent/LAUNCH-PROMPT.md +37 -0
- package/docs/agent/LOOP.md +84 -0
- package/docs/agent/README.md +77 -0
- package/docs/agent/ROADMAP.md +37 -0
- package/docs/agent/STATE.md +93 -0
- package/docs/agent/tasks/00-verify-inherited-work.md +36 -0
- package/docs/agent/tasks/01-authoritative-metadata.md +34 -0
- package/docs/agent/tasks/02-native-dependencies.md +35 -0
- package/docs/agent/tasks/03-reproducible-inputs.md +27 -0
- package/docs/agent/tasks/04-build-frontends.md +27 -0
- package/docs/agent/tasks/05-registry-integration.md +27 -0
- package/docs/agent/tasks/06-package-cohorts.md +42 -0
- package/docs/agent/tasks/07-build-on-miss-boundary.md +31 -0
- package/docs/browser-runtime-architecture.md +132 -0
- package/docs/compatibility-implementation-plan.md +98 -0
- package/docs/developer-tool-packs.md +134 -0
- package/docs/frontend-automation.md +49 -0
- package/docs/fullstack-deployment.md +163 -0
- package/docs/handoff.md +275 -0
- package/docs/original-x64.md +39 -0
- package/docs/platform-hardening.md +49 -0
- package/docs/python/abi.md +97 -0
- package/docs/python/architecture.md +94 -0
- package/docs/python/baseline-inventory.md +54 -0
- package/docs/python/build-on-miss.md +198 -0
- package/docs/python/compatibility.md +206 -0
- package/docs/python/cross-build.md +354 -0
- package/docs/python/extensions.md +282 -0
- package/docs/python/release-gates.md +46 -0
- package/docs/python/virtual-sockets-plan.md +331 -0
- package/docs/runtime-lifecycle-fixes.md +39 -0
- package/docs/server-previews.md +268 -0
- package/docs/virtual-browser.md +120 -0
- package/package.json +6 -3
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
# Python on SandboxedJs — architecture
|
|
2
|
+
|
|
3
|
+
The goal is an operating-system compatibility layer with an owned Python
|
|
4
|
+
distribution on top of it, not an interpreter swap. Replacing Pyodide with a
|
|
5
|
+
differently-built CPython in the same lifecycle would inherit the same problems,
|
|
6
|
+
because the problems are not in the interpreter.
|
|
7
|
+
|
|
8
|
+
## What "our own Python" means here
|
|
9
|
+
|
|
10
|
+
A **runtime distribution**, not a new language implementation. SandboxedJs owns
|
|
11
|
+
the OS-facing ABI, the process model, the filesystem integration, the build
|
|
12
|
+
pipeline and the tests. Upstream CPython supplies Python semantics, adapted by a
|
|
13
|
+
maintained patch series. Building from source gives ownership and
|
|
14
|
+
reproducibility; stability comes from process isolation, consistent semantics
|
|
15
|
+
and systematic testing.
|
|
16
|
+
|
|
17
|
+
## Governing rules
|
|
18
|
+
|
|
19
|
+
1. One active Python process owns one interpreter instance.
|
|
20
|
+
2. One kernel authority owns shared mutable container resources.
|
|
21
|
+
3. Blocking callers never own the services they are waiting for.
|
|
22
|
+
4. All guest I/O crosses an explicit, versioned boundary.
|
|
23
|
+
5. Processes share files and channels — not Python globals.
|
|
24
|
+
6. Every resource has an owner and a cleanup path.
|
|
25
|
+
|
|
26
|
+
## Shape
|
|
27
|
+
|
|
28
|
+
```
|
|
29
|
+
browser app / shell kernel owner (volume, process table)
|
|
30
|
+
│ │
|
|
31
|
+
├── preview service worker ───────────┤
|
|
32
|
+
│ ├── VFS + descriptors + storage
|
|
33
|
+
│ ├── process supervisor + signals
|
|
34
|
+
│ └── virtual sockets + net policy
|
|
35
|
+
│ ▲
|
|
36
|
+
python worker A ──── sbx_host_v1 ────────────────── ┤
|
|
37
|
+
python worker B ──── sbx_host_v1 ────────────────── ┤
|
|
38
|
+
node workers ────────────────────────────────────── ┘
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Kernel services stay on the host that owns them today; Python runs in dedicated
|
|
42
|
+
workers and calls back. Moving the kernel into its own worker is a later
|
|
43
|
+
optimisation with its own migration plan, driven by measured UI responsiveness,
|
|
44
|
+
and is not a prerequisite for any of this.
|
|
45
|
+
|
|
46
|
+
## Three compatibility targets
|
|
47
|
+
|
|
48
|
+
| Target | Behaviour | Mechanism |
|
|
49
|
+
|---|---|---|
|
|
50
|
+
| Python development | scripts, REPL, venv, pip, subprocesses, local servers | CPython to Wasm plus this kernel |
|
|
51
|
+
| Scientific / ML | supported packages, model inference | recompiled extensions, dedicated compute services |
|
|
52
|
+
| Linux binaries | existing Linux executables and wheels | optional CPU emulation running a real Linux guest |
|
|
53
|
+
|
|
54
|
+
The first is the product. The third is a separate execution backend and must
|
|
55
|
+
never silently substitute for the first when an install fails: it changes
|
|
56
|
+
architecture, path semantics, performance and persistence.
|
|
57
|
+
|
|
58
|
+
## What exists today
|
|
59
|
+
|
|
60
|
+
- `python-runtime/abi/` — the versioned ABI and its generated bindings.
|
|
61
|
+
- `src/kernel/open-file.ts`, `src/kernel/descriptors.ts` — pathnames, inodes,
|
|
62
|
+
open-file descriptions and per-process descriptor tables.
|
|
63
|
+
- `src/runtime/python/` — framing, the blocking transport, the kernel-side
|
|
64
|
+
dispatcher, the typed guest client, the Emscripten filesystem bridge, the
|
|
65
|
+
release manifest, and the per-process supervisor and worker.
|
|
66
|
+
- `python-runtime/native/probe/` — a C program that exercises the whole path.
|
|
67
|
+
- `python-runtime/scripts/` — the pinned build: fetch and verify, host Python,
|
|
68
|
+
cross-built CPython, and the packaged release with its manifest.
|
|
69
|
+
- `test/python-abi/`, `test/python-runtime/` — the baseline inventory and the
|
|
70
|
+
M1 and M2 gates.
|
|
71
|
+
|
|
72
|
+
### The owned interpreter
|
|
73
|
+
|
|
74
|
+
`make -C python-runtime fetch python package` produces
|
|
75
|
+
`out/sbx-cpython-<version>-<profile>/`: `python.js`, `python.wasm`,
|
|
76
|
+
`python.data`, and a `runtime.json` naming the ABI it was built against and the
|
|
77
|
+
capabilities the profile actually has. A host selects it with
|
|
78
|
+
`configurePython({ backend: "sbx-cpython-wasm", manifest })`; Pyodide remains
|
|
79
|
+
the default until the parity gates pass.
|
|
80
|
+
|
|
81
|
+
Two things about the build are worth knowing before reading it:
|
|
82
|
+
|
|
83
|
+
- Upstream's browser target links a *page* — a classic script that assigns a
|
|
84
|
+
global `Module` and runs `main` on load. A process worker needs a factory it
|
|
85
|
+
instantiates once per process, so the final link is repeated with
|
|
86
|
+
`-sMODULARIZE -sEXPORT_ES6 -sINVOKE_RUN=0`. Only the link changes.
|
|
87
|
+
- Some of what a program asks an operating system for is compiled into libc
|
|
88
|
+
rather than routed through anything a host can serve. musl's `getpid` under
|
|
89
|
+
Emscripten returns a constant, so every process would report the same
|
|
90
|
+
identity however many interpreters were running. `-Wl,--wrap=` is what makes
|
|
91
|
+
those reachable, and `python-runtime/native/js/library_sbx_posix.js` is where
|
|
92
|
+
the first of them live.
|
|
93
|
+
|
|
94
|
+
See [release-gates.md](release-gates.md) for what each milestone has to prove.
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# M0 — baseline inventory
|
|
2
|
+
|
|
3
|
+
What the *current* Python integration does, measured rather than remembered, so
|
|
4
|
+
the owned runtime has something to be compared against. Every entry is pinned by
|
|
5
|
+
a test in `test/python-abi/baseline.test.ts`, which asserts today's behaviour —
|
|
6
|
+
including the wrong answers, on purpose. Turning one of those assertions around
|
|
7
|
+
is what a milestone landing looks like.
|
|
8
|
+
|
|
9
|
+
Setup: `src/runtime/cpython.ts` caches **one Pyodide interpreter per container**
|
|
10
|
+
(keyed on the VFS) and runs each program with a fresh globals dict.
|
|
11
|
+
|
|
12
|
+
## Findings
|
|
13
|
+
|
|
14
|
+
| # | Behaviour | Verified | Cause | Fixed by |
|
|
15
|
+
|---|---|---|---|---|
|
|
16
|
+
| 1 | A module left in `sys.modules` by one program is importable by the next, unrelated one | yes | fresh globals do not reset `sys.modules` | **fixed in M2** — one interpreter per process |
|
|
17
|
+
| 2 | `os.environ` does **not** leak between programs | yes | the environment is explicitly rebound per run | — (the exception, not the rule) |
|
|
18
|
+
| 3 | `asyncio.start_server` fails | yes | `python-syscalls.ts` replaces parts of `asyncio` and refuses server creation | M6 — virtual sockets |
|
|
19
|
+
| 4 | Every program reports the same `os.getpid()` | yes | one interpreter is one process | **fixed in M2** — a process per program, with `getpid` wrapped to ask the kernel |
|
|
20
|
+
| 5 | `subprocess.run` succeeds, via the container's command table | yes | a bridge, not a process: no fresh interpreter, no descriptor inheritance, no process groups | M3 |
|
|
21
|
+
|
|
22
|
+
Fresh globals reset none of: `sys.modules`, logging handlers, registered
|
|
23
|
+
callbacks, native extension state, or running tasks. Finding 2 is worth naming
|
|
24
|
+
precisely because it is the one piece of state the integration *does* rebind —
|
|
25
|
+
which is why "each program gets a clean slate" feels true until it isn't.
|
|
26
|
+
|
|
27
|
+
## What this means for the plan
|
|
28
|
+
|
|
29
|
+
- Findings 1 and 4 are integration failures, not interpreter failures. A
|
|
30
|
+
differently-built CPython dropped into the same lifecycle would reproduce both.
|
|
31
|
+
That is the argument for building the kernel first.
|
|
32
|
+
- Finding 3 is a missing subsystem. No amount of interpreter work reaches it.
|
|
33
|
+
- Finding 5 is the one that reads as working and is not, which makes it the most
|
|
34
|
+
expensive to leave in place.
|
|
35
|
+
|
|
36
|
+
## Not yet measured
|
|
37
|
+
|
|
38
|
+
Concurrency and server behaviour under load, native extension state across
|
|
39
|
+
process exits, and interpreter memory growth over repeated runs. These need the
|
|
40
|
+
process model of M3 before the measurement means anything.
|
|
41
|
+
|
|
42
|
+
## Known gaps in the owned runtime (M2)
|
|
43
|
+
|
|
44
|
+
Recorded so they are not rediscovered as surprises. None of them is hidden at
|
|
45
|
+
runtime: each either refuses with a reason or is declared absent in
|
|
46
|
+
`runtime.json`.
|
|
47
|
+
|
|
48
|
+
| Gap | Symptom | Milestone |
|
|
49
|
+
|---|---|---|
|
|
50
|
+
| No packaging | `pip` is micropip and targets the Pyodide interpreter; under this backend it refuses rather than installing into the wrong interpreter | M4 |
|
|
51
|
+
| No `rlcompleter` / `_pyrepl` in the stdlib image | the REPL prints `warning: can't use pyrepl` and falls back to the basic prompt, which works | M2 follow-up — a `wasm_assets` inclusion |
|
|
52
|
+
| No sockets | anything binding a port, Uvicorn included, cannot start | M6 |
|
|
53
|
+
| No spawn from Python | `subprocess` has no backend on this runtime | M3 |
|
|
54
|
+
| Symlinks | `symlink()` returns ENOSYS through the kernel filesystem | M3 |
|
|
@@ -0,0 +1,198 @@
|
|
|
1
|
+
# The build-on-miss boundary
|
|
2
|
+
|
|
3
|
+
What happens when the installer finds no wheel it can use, and how a host may
|
|
4
|
+
turn that into a build without the runtime ever deciding to.
|
|
5
|
+
|
|
6
|
+
Nothing described here is deployed. `src/runtime/python/build-service.ts`
|
|
7
|
+
defines the shape and ships a local fake; there is no client, no endpoint, and
|
|
8
|
+
no configuration that would produce one.
|
|
9
|
+
|
|
10
|
+
## The rule
|
|
11
|
+
|
|
12
|
+
The browser runtime is an installer. It does not compile, and it does not ask
|
|
13
|
+
anyone else to compile on its behalf.
|
|
14
|
+
|
|
15
|
+
A resolver that reached out to have a wheel built would be performing a remote
|
|
16
|
+
mutation the caller never asked for — spending someone else's compute,
|
|
17
|
+
publishing an artifact — at exactly the moment a user is least able to notice
|
|
18
|
+
it, during a routine `pip install`. So the runtime does one thing: it reports
|
|
19
|
+
the miss precisely enough to act on. `test/python-runtime/build-service.test.ts`
|
|
20
|
+
checks structurally that `resolver.ts`, `install.ts` and `pip-command.ts` do not
|
|
21
|
+
import the build service at all.
|
|
22
|
+
|
|
23
|
+
## The seam
|
|
24
|
+
|
|
25
|
+
1. Resolution fails and raises a `ResolutionError` carrying a
|
|
26
|
+
`ResolutionFailure` — machine-readable, distinguishing
|
|
27
|
+
`no-compatible-distribution` (building might help) from
|
|
28
|
+
`conflicting-requirements` (it cannot).
|
|
29
|
+
2. The **host** — application code, not the runtime — may call
|
|
30
|
+
`buildRequestFor(failure, { recipeRevision, … })`. It is a pure function:
|
|
31
|
+
it contacts nothing and queues nothing, and returns `null` when building
|
|
32
|
+
could not help or when no source exists to build from.
|
|
33
|
+
3. The host decides whether to submit that request anywhere.
|
|
34
|
+
|
|
35
|
+
## Requests converge
|
|
36
|
+
|
|
37
|
+
A build request promises that its artifact is interchangeable with one built
|
|
38
|
+
anywhere else. That holds only if everything which can change the artifact is
|
|
39
|
+
named in the request, so `idempotencyKey` is *derived* from the requirement,
|
|
40
|
+
the target ABI, the wheel tag and the recipe revision — never invented by a
|
|
41
|
+
client, because a random key makes every retry a new build.
|
|
42
|
+
|
|
43
|
+
What is deliberately excluded is evidence about *where* the requester saw the
|
|
44
|
+
source: two clients looking at different mirrors of the same release should
|
|
45
|
+
converge on one build. What is deliberately included is the ABI id, so an ABI
|
|
46
|
+
rollover does not serve a wheel the new runtime cannot load, and the recipe
|
|
47
|
+
revision, since the same source built under different rules is a different
|
|
48
|
+
artifact.
|
|
49
|
+
|
|
50
|
+
## States
|
|
51
|
+
|
|
52
|
+
`queued`, `resolving`, `building-host-tools`,
|
|
53
|
+
`building-target-dependencies`, `building-wheel`, `testing`, then one of
|
|
54
|
+
`published`, `unsupported`, `failed`, `cancelled`.
|
|
55
|
+
|
|
56
|
+
The intermediate states are the ones someone waiting would ask about, and they
|
|
57
|
+
fail differently: `building-target-dependencies` failing means a native library
|
|
58
|
+
did not cross compile, which is a different report from the package's own build
|
|
59
|
+
failing.
|
|
60
|
+
|
|
61
|
+
`unsupported` is terminal and separate from `failed` on purpose. "This package
|
|
62
|
+
cannot work on this platform" is worth caching forever and showing to the user
|
|
63
|
+
as a fact; "this build did not succeed" may be worth retrying. Collapsing them
|
|
64
|
+
makes a permanent answer look transient, and every client retries it forever.
|
|
65
|
+
|
|
66
|
+
Cancelling something already terminal does not rewrite its outcome: a published
|
|
67
|
+
wheel does not become uncancelled work because someone asked late.
|
|
68
|
+
|
|
69
|
+
## Security requirements
|
|
70
|
+
|
|
71
|
+
These are requirements on any implementation, stated here because the request
|
|
72
|
+
carries the policy and an audit should be answerable from the request alone. A
|
|
73
|
+
client-supplied policy is a statement of intent, never a grant of permission —
|
|
74
|
+
a service must enforce its own.
|
|
75
|
+
|
|
76
|
+
- **Source allowlist.** Fetch only from named hosts; the default is PyPI.
|
|
77
|
+
- **No network during the build.** A build that can reach the network can fetch
|
|
78
|
+
an unpinned dependency, and then the artifact depends on the day it was
|
|
79
|
+
built. Everything needed is pinned before the build starts.
|
|
80
|
+
- **Resource limits.** Wall-clock and memory ceilings, so a pathological build
|
|
81
|
+
cannot occupy a worker indefinitely.
|
|
82
|
+
- **Nothing published before it runs.** A wheel that compiled is not a wheel
|
|
83
|
+
that works; the `testing` state exists so that publication follows an actual
|
|
84
|
+
import and exercise in the owned runtime, as `package-cohorts.test.ts` does
|
|
85
|
+
locally.
|
|
86
|
+
- **Digest verification.** Immutable artifact locations and recorded hashes. A
|
|
87
|
+
wheel that could be replaced under a URL makes every recorded digest a lie.
|
|
88
|
+
- **Auditability.** The provenance record the wheel already carries — source
|
|
89
|
+
digest, build tools, native dependencies, ABI, recipe revision, patches — is
|
|
90
|
+
echoed in the result for clients that will not open the archive.
|
|
91
|
+
|
|
92
|
+
## Building here: `buildFromSource`
|
|
93
|
+
|
|
94
|
+
The seam above describes talking to a service. The same boundary also has a
|
|
95
|
+
local implementation, so a host with the toolchain does not need one.
|
|
96
|
+
|
|
97
|
+
```js
|
|
98
|
+
configurePython({ buildFromSource: true }); // build here
|
|
99
|
+
configurePython({ buildFromSource: "http://localhost:4180/build" }); // ask a machine
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
`true` is the default **only** where building can possibly work: a Node
|
|
103
|
+
process, running from a checkout that contains the build pipeline. A browser
|
|
104
|
+
has no compiler and will not get one; a published package has no pipeline. In
|
|
105
|
+
both, attempting a build would replace a clear "no wheel for this package" with
|
|
106
|
+
a hang or a confusing failure, so both keep reporting instead.
|
|
107
|
+
|
|
108
|
+
When `pip` hits a `no-compatible-distribution` failure for a package that has a
|
|
109
|
+
source distribution, it asks the builder, then retries the install once against
|
|
110
|
+
the index that now contains the wheel. The loop is bounded, because each pass
|
|
111
|
+
can only make progress by adding one package — without a bound, a graph whose
|
|
112
|
+
every member is unbuildable would rebuild forever rather than report.
|
|
113
|
+
|
|
114
|
+
Two failure classes are deliberately *not* built: conflicting requirements,
|
|
115
|
+
which compiling cannot fix, and packages with no source distribution, which
|
|
116
|
+
have nothing to build from.
|
|
117
|
+
|
|
118
|
+
### Where the recipe comes from
|
|
119
|
+
|
|
120
|
+
`python-runtime/scripts/auto_recipe.py` writes one. A recipe states which
|
|
121
|
+
package, where its verified source is, and which backend drives it — and PyPI
|
|
122
|
+
publishes the first two while the source declares the third, so requiring a
|
|
123
|
+
person to transcribe them is what made every new package a small project.
|
|
124
|
+
|
|
125
|
+
Nothing that affects the artifact is inferred. Compiler flags still come from
|
|
126
|
+
`abi/extension-abi.json`, and a package needing a patch, a native library or a
|
|
127
|
+
build-environment switch still needs those declared by hand. A build backend
|
|
128
|
+
with no adapter is refused **by name** rather than attempted — attempting it
|
|
129
|
+
fails deep inside someone else's build system, where the message is about a
|
|
130
|
+
missing CMake rather than about this pipeline not supporting CMake.
|
|
131
|
+
|
|
132
|
+
Build requirements a package declares and the lock has never seen are pinned
|
|
133
|
+
at that moment rather than dropped. Dropping them is what made `ujson` fail
|
|
134
|
+
with `ModuleNotFoundError: setuptools_scm` raised from inside its own
|
|
135
|
+
`setup.py` — a message about the package, caused by the generator quietly
|
|
136
|
+
omitting a requirement the package had stated plainly.
|
|
137
|
+
|
|
138
|
+
## Serving builds to a browser
|
|
139
|
+
|
|
140
|
+
```bash
|
|
141
|
+
npx sandboxedjs-build-wheels 4180
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
It serves the wheel index over HTTP and builds on request, binding loopback
|
|
145
|
+
only — a build runs a package's own build system, which is arbitrary code
|
|
146
|
+
execution by design, so exposing it to a network hands that to whoever can
|
|
147
|
+
reach it.
|
|
148
|
+
|
|
149
|
+
The exchange is a package name in, a verdict out. The wheel itself comes back
|
|
150
|
+
the ordinary way, fetched from the index and checked against its digest like
|
|
151
|
+
every other wheel; a service that returned bytes directly would bypass that.
|
|
152
|
+
|
|
153
|
+
It is submit-and-poll, not one long request. A build takes minutes, and a
|
|
154
|
+
connection held open that long is dropped somewhere in between — which reads
|
|
155
|
+
to the caller as the service being unreachable while it is in fact working.
|
|
156
|
+
|
|
157
|
+
## What is not decided here
|
|
158
|
+
|
|
159
|
+
Whether to run a *hosted* service, where, and who pays for it. That is the
|
|
160
|
+
user's decision, and this project does not own cloud accounts, deployment,
|
|
161
|
+
billing or signing infrastructure.
|
|
162
|
+
|
|
163
|
+
## What "no wheel" actually means, and what it does not
|
|
164
|
+
|
|
165
|
+
A package with no compiled extension needs no wheel of ours: `requests`, `rich`
|
|
166
|
+
and everything else pure-Python installs from PyPI unchanged. A package with a
|
|
167
|
+
compiled extension needs one built for `cp313-cp313-emscripten_5_0_6_wasm32`,
|
|
168
|
+
because no such wheel is published anywhere.
|
|
169
|
+
|
|
170
|
+
There is no compiler inside the container and there is not going to be one. A
|
|
171
|
+
C/C++ toolchain targeting WebAssembly is clang, lld and a sysroot -- hundreds of
|
|
172
|
+
megabytes before a single package is built -- and it would still not cover the
|
|
173
|
+
Fortran in SciPy or the Rust in pydantic-core and cryptography. Shipping that to
|
|
174
|
+
a browser tab to install one package is not a trade worth making, and calling it
|
|
175
|
+
"free" ignores what serving it costs. So the boundary stays where it is: a host
|
|
176
|
+
with the toolchain builds locally, a host without one points at a build service,
|
|
177
|
+
and a host with neither is told plainly that no wheel exists rather than being
|
|
178
|
+
left to wait on a compiler that cannot run.
|
|
179
|
+
|
|
180
|
+
A local build that fails because the *build machine* lacks a package now says
|
|
181
|
+
so and names what to install, rather than printing the backend's traceback:
|
|
182
|
+
`pip install scipy` reported a `ModuleNotFoundError` for `mesonpy` where it
|
|
183
|
+
meant "install meson-python into the interpreter running this pipeline".
|
|
184
|
+
|
|
185
|
+
## Standard library coverage
|
|
186
|
+
|
|
187
|
+
276 of the 290 modules in `sys.stdlib_module_names` import. Of the rest:
|
|
188
|
+
|
|
189
|
+
- `msvcrt`, `nt`, `winreg`, `winsound` are Windows-only. CPython on Linux or
|
|
190
|
+
macOS does not have them either; a program importing `msvcrt` unguarded is
|
|
191
|
+
already broken everywhere but Windows.
|
|
192
|
+
- `tkinter`, `turtle`, `turtledemo`, `idlelib`, `curses`, `readline` need a GUI
|
|
193
|
+
or a terminal device that a container in a page does not have.
|
|
194
|
+
- `antigravity`, `pydoc_data`, `webbrowser` are absent for no good reason and
|
|
195
|
+
are cheap to add; they live in the interpreter's data image, so adding them
|
|
196
|
+
means rebuilding it.
|
|
197
|
+
- `ctypes` is the one real gap. It needs libffi cross-compiled and linked into
|
|
198
|
+
the image, and packages that import it unconditionally will fail until then.
|
|
@@ -0,0 +1,206 @@
|
|
|
1
|
+
# Package compatibility report
|
|
2
|
+
|
|
3
|
+
What this pipeline has actually built and run, measured on real packages rather
|
|
4
|
+
than on the fixtures written to succeed. Every wheel below was produced by
|
|
5
|
+
`scripts/build_extension.py` from a pinned, hash-verified source, with no
|
|
6
|
+
package-specific code path anywhere in the builder.
|
|
7
|
+
|
|
8
|
+
ABI for every entry: `sbxabi1-c2637d04695ad927`, tag
|
|
9
|
+
`cp313-cp313-emscripten_5_0_6_wasm32`, Emscripten 5.0.6, CPython 3.13.5.
|
|
10
|
+
|
|
11
|
+
## Status vocabulary
|
|
12
|
+
|
|
13
|
+
| status | meaning |
|
|
14
|
+
| --- | --- |
|
|
15
|
+
| `supported-generic` | builds and runs with no recipe beyond identity and source |
|
|
16
|
+
| `supported-recipe` | needs declared configuration (native library, build env, build tool) but no patch |
|
|
17
|
+
| `supported-patched` | needs a visible, tested patch for an assumption false on this target |
|
|
18
|
+
| `blocked-platform-capability` | the platform does not provide what the package needs; no patch can add it |
|
|
19
|
+
| `blocked-toolchain` | the build system or toolchain cannot produce this artifact yet |
|
|
20
|
+
| `not-yet-attempted` | not tried |
|
|
21
|
+
|
|
22
|
+
## Real packages
|
|
23
|
+
|
|
24
|
+
| package | version | cohort | status | backend | wheel sha256 (16) | size |
|
|
25
|
+
| --- | --- | --- | --- | --- | --- | --- |
|
|
26
|
+
| typing-extensions | 4.12.2 | 1 pure control | `supported-generic` | — (PyPI pure wheel) | — | — |
|
|
27
|
+
| MarkupSafe | 3.0.2 | 2 setuptools C | `supported-generic` | setuptools | `00619948d4c33812` | 12 KB |
|
|
28
|
+
| msgpack | 1.1.0 | 3 Cython | `supported-recipe` | setuptools | `b5336ed6394b0307` | 62 KB |
|
|
29
|
+
| PyYAML | 6.0.2 | 3 + 5 Cython and native library | `supported-recipe` | setuptools | `23c9ad8ee878f41e` | 129 KB |
|
|
30
|
+
| pydantic-core | 2.46.5 | 4 Rust/PyO3 | `supported-recipe` | pyo3 | `01706c4ed32d7f24` | 1951 KB |
|
|
31
|
+
| pydantic-core | 2.23.2 | 4 Rust/PyO3 | `supported-recipe` | pyo3 | `e1bea0f995650d93` | 1934 KB |
|
|
32
|
+
| numpy | 2.2.6 | 8 scientific stress case | `supported-recipe` | meson | `93ae7230479566b4` | 4917 KB |
|
|
33
|
+
| siphash24 | 1.9 | 6 + 7 Meson, patched subproject | `supported-patched` | meson | `cf672c37b0f8e4cd` | 54 KB |
|
|
34
|
+
| simplejson | 3.19.3 | — | `supported-generic` | — (PyPI pure wheel) | — | — |
|
|
35
|
+
| psutil | 6.1.0 | — | `blocked-platform-capability` | setuptools | — | — |
|
|
36
|
+
|
|
37
|
+
Eight independent real packages across six build shapes — pure, setuptools C,
|
|
38
|
+
Cython, Cython plus a separately built native library, Rust/PyO3, and Meson via
|
|
39
|
+
meson-python — install and are exercised by
|
|
40
|
+
`test/python-runtime/package-cohorts.test.ts`. Every cohort in the roadmap now
|
|
41
|
+
has a real package in it. That test
|
|
42
|
+
also loads four of the compiled ones into a single interpreter at once, which
|
|
43
|
+
is the only check that proves the shapes coexist rather than merely each
|
|
44
|
+
working alone.
|
|
45
|
+
|
|
46
|
+
### What each entry adds
|
|
47
|
+
|
|
48
|
+
**typing-extensions** — the control. No cross build, no platform tag. Proves
|
|
49
|
+
the installer still does the ordinary thing when nothing is compiled.
|
|
50
|
+
|
|
51
|
+
**MarkupSafe** — one C source, no dependencies, no configuration. The purest
|
|
52
|
+
measurement of the generic setuptools path: the recipe states identity and
|
|
53
|
+
source and nothing else. Its documented pure-Python fallback is why the test
|
|
54
|
+
asserts `markupsafe._speedups` specifically; an extension that failed to build
|
|
55
|
+
would otherwise install and import cleanly.
|
|
56
|
+
|
|
57
|
+
**msgpack** — a Cython extension with no external library, so a failure here
|
|
58
|
+
would be about generated C rather than about linking. Needs only
|
|
59
|
+
`buildRequires: ["Cython==3.1.4"]`.
|
|
60
|
+
|
|
61
|
+
**PyYAML** — Cython generates `_yaml.c`, which links the
|
|
62
|
+
libyaml built by the native dependency system. It selects between its pure and
|
|
63
|
+
compiled implementations through its own `PYYAML_FORCE_CYTHON` and
|
|
64
|
+
`PYYAML_FORCE_LIBYAML` variables, which the recipe sets through `buildEnv` —
|
|
65
|
+
build configuration, which belongs in a recipe, as distinct from compiler
|
|
66
|
+
flags, which never do. The test asserts `yaml.CSafeLoader` parsed the document,
|
|
67
|
+
which only exists when the extension built.
|
|
68
|
+
|
|
69
|
+
**pydantic-core** — Rust, cross compiled with a rebuilt `std` and wasm
|
|
70
|
+
exceptions. Two versions, because pydantic pins its core exactly and resolving
|
|
71
|
+
that pin is part of what the index has to get right.
|
|
72
|
+
|
|
73
|
+
**simplejson** — recorded for what it shows about policy rather than about
|
|
74
|
+
building. It publishes both C-accelerated platform wheels and a pure
|
|
75
|
+
`py3-none-any` fallback, and the resolver takes the pure one. That is correct:
|
|
76
|
+
a pure wheel is portable and carries no ABI risk. The limitation is real and
|
|
77
|
+
worth stating plainly — **a package that publishes a pure fallback cannot
|
|
78
|
+
currently be accelerated by supplying a compiled wheel, because the pure wheel
|
|
79
|
+
always wins**. Changing that is a policy decision, not an oversight, so no
|
|
80
|
+
compiled simplejson wheel is shipped.
|
|
81
|
+
|
|
82
|
+
**siphash24** — the densest case, and the only one that needed a patch. Three
|
|
83
|
+
things at once: a frontend that is not setuptools (meson-python, driven with a
|
|
84
|
+
cross file generated from the ABI contract); two dependencies resolved through
|
|
85
|
+
Meson `wrap` files, which Meson satisfies by cloning from git *during the
|
|
86
|
+
build* — disabled here, with the subprojects supplied instead as pinned,
|
|
87
|
+
hash-verified archives recorded in the wheel's provenance; and a patch, because
|
|
88
|
+
one of those subprojects builds its library with `both_libraries`, which always
|
|
89
|
+
produces a shared library. `ld.wasm` has none, and Meson refuses at configure
|
|
90
|
+
time rather than at link time, so the project was unconfigurable until the
|
|
91
|
+
library was made static. The test asserts the first published SipHash-2-4
|
|
92
|
+
reference vector, which comes from outside this repository.
|
|
93
|
+
|
|
94
|
+
**numpy** — the case this pipeline was built toward, and deliberately the last
|
|
95
|
+
one attempted, so that it drove none of the design. It needed **no patch** and
|
|
96
|
+
no mechanism that did not already exist for smaller packages: Meson through
|
|
97
|
+
meson-python, a cross file generated from the ABI contract, pinned build tools,
|
|
98
|
+
and its own documented `allow-noblas` switch for a platform with no BLAS. It
|
|
99
|
+
built on the first attempt.
|
|
100
|
+
|
|
101
|
+
Its assertions are numerical rather than structural. NumPy has no pure-Python
|
|
102
|
+
fallback to hide behind, but a miscompiled one imports perfectly well and
|
|
103
|
+
computes wrong answers, so each check has an exactly known result: `a @ a.T`,
|
|
104
|
+
an inverse multiplied back to the identity, a round-trip FFT, a reduction.
|
|
105
|
+
`long double` deserves naming — Meson cannot run a program on the target to
|
|
106
|
+
learn its layout, so the value is derived from the pinned compiler's own
|
|
107
|
+
`__LDBL_MANT_DIG__` (113, little-endian, so `IEEE_QUAD_LE`) and the built
|
|
108
|
+
extension reports the matching 16-byte itemsize. Getting that wrong would have
|
|
109
|
+
been silently wrong arithmetic rather than a build failure.
|
|
110
|
+
|
|
111
|
+
NumPy's *own* test suite runs inside the runtime, which is the strongest
|
|
112
|
+
evidence in this document because none of it was written here:
|
|
113
|
+
|
|
114
|
+
| module | result |
|
|
115
|
+
| --- | --- |
|
|
116
|
+
| `numpy._core.tests.test_umath` | 4388 passed, 362 skipped, 2 xfailed, 5 xpassed |
|
|
117
|
+
| `numpy.linalg.tests.test_linalg` | 416 passed, 21 skipped, 2 xfailed |
|
|
118
|
+
| `numpy.fft.tests.test_pocketfft` | 144 passed, 4 skipped |
|
|
119
|
+
|
|
120
|
+
4,948 upstream tests pass and none fails. Reproduce with `pip install numpy
|
|
121
|
+
pytest hypothesis` in a container, then:
|
|
122
|
+
|
|
123
|
+
```bash
|
|
124
|
+
python3 -m pytest --pyargs numpy._core.tests.test_umath -q --capture=no
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
`--capture=no` is required, and the reason is a runtime limitation rather than
|
|
128
|
+
anything to do with NumPy: pytest's default output capture opens a temporary
|
|
129
|
+
file descriptor and reading it back fails with `OSError: [Errno 8] Bad file
|
|
130
|
+
descriptor`, so *no* tests run at all. It belongs with the other pre-existing
|
|
131
|
+
process- and descriptor-level gaps recorded in the project state, not with
|
|
132
|
+
package support.
|
|
133
|
+
|
|
134
|
+
**psutil** — attempted deliberately as a negative case, and the failure is more
|
|
135
|
+
useful than the success would have been.
|
|
136
|
+
|
|
137
|
+
## The psutil finding
|
|
138
|
+
|
|
139
|
+
Command: `python3 scripts/build_extension.py recipes/psutil/recipe.json`
|
|
140
|
+
Source: `psutil-6.1.0.tar.gz`, sha256 `353815f5…c7a`
|
|
141
|
+
|
|
142
|
+
Earliest unsupported assumption, quoted from the build:
|
|
143
|
+
|
|
144
|
+
```
|
|
145
|
+
emcc … -DPSUTIL_POSIX=1 -DPSUTIL_OSX=1 … -c psutil/_psutil_common.c
|
|
146
|
+
psutil/_psutil_common.h:131:14: fatal error: 'mach/mach_time.h' file not found
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
`-DPSUTIL_OSX=1` is the whole finding. `sysconfig` has been pointed at the
|
|
150
|
+
target throughout, so the compiler, its flags and the extension suffix are all
|
|
151
|
+
correct — but psutil's `setup.py` selects its platform implementation from
|
|
152
|
+
`sys.platform`, which reports the *running* interpreter's operating system.
|
|
153
|
+
Under a cross build that is the build machine, so on macOS it compiled the
|
|
154
|
+
Darwin backend and asked for a Mach header.
|
|
155
|
+
|
|
156
|
+
This is assumption 7 in `cross-build.md` in its most common concrete form, and
|
|
157
|
+
it generalises well beyond psutil: any package that branches on `sys.platform`
|
|
158
|
+
in its build will describe the build machine. It is worth recording that
|
|
159
|
+
`_PYTHON_HOST_PLATFORM` does **not** fix this — that variable changes
|
|
160
|
+
`sysconfig.get_platform()`, not `sys.platform`.
|
|
161
|
+
|
|
162
|
+
psutil is nevertheless classified `blocked-platform-capability` rather than
|
|
163
|
+
`supported-patched`. A patch could make it select a POSIX backend, but psutil
|
|
164
|
+
reads process and system tables through syscalls Emscripten does not implement;
|
|
165
|
+
the capability is absent from the platform, not from the build. Patching it
|
|
166
|
+
would convert a clear build failure into a runtime one, which is the trade this
|
|
167
|
+
project refuses everywhere else.
|
|
168
|
+
|
|
169
|
+
## Cohorts
|
|
170
|
+
|
|
171
|
+
All eight cohorts now have a real package. Cohorts 6 (a package requiring a
|
|
172
|
+
visible cross-compilation patch) and 7 (a PEP 517 backend that is not
|
|
173
|
+
setuptools) are both covered by siphash24; cohort 8 by NumPy.
|
|
174
|
+
|
|
175
|
+
## Packages with no recipe
|
|
176
|
+
|
|
177
|
+
Everything above has a recipe checked in, because each was measured
|
|
178
|
+
deliberately. Most packages need none: `pip install <anything>` will generate
|
|
179
|
+
one, build the wheel and install it, when a builder is enabled — see
|
|
180
|
+
`build-on-miss.md`. `ujson` is the worked example, built with no recipe, no
|
|
181
|
+
patch and no hand-pinned tool:
|
|
182
|
+
|
|
183
|
+
```
|
|
184
|
+
$ pip install ujson
|
|
185
|
+
Collecting ujson
|
|
186
|
+
Building ujson from source (no wheel for this runtime yet)
|
|
187
|
+
Built ujson
|
|
188
|
+
Successfully installed ujson-6.0.0
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
A package that cannot be built this way still lands in the vocabulary above
|
|
192
|
+
rather than as an opaque failure: a backend with no adapter is
|
|
193
|
+
`blocked-toolchain` and says which backend, and a package with no source
|
|
194
|
+
distribution says that instead.
|
|
195
|
+
|
|
196
|
+
## Reproducing
|
|
197
|
+
|
|
198
|
+
```bash
|
|
199
|
+
make -C python-runtime native-deps
|
|
200
|
+
python3 python-runtime/scripts/build_extension.py python-runtime/recipes/<name>/recipe.json
|
|
201
|
+
python3 python-runtime/scripts/build_index.py
|
|
202
|
+
npx vitest run test/python-runtime/package-cohorts.test.ts
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
Every wheel above rebuilds byte-identically; the digests in this table are
|
|
206
|
+
therefore checkable rather than decorative.
|