sandboxedjs 0.2.11 → 0.2.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +16 -9
- package/assets/logo.png +0 -0
- package/bin/sandboxedjs-egress.mjs +25 -10
- package/dist/index.cjs +1 -1
- package/dist/index.js +1 -1
- package/dist/service-worker.js +3 -2
- package/docs/agent/COMMANDS.md +85 -0
- package/docs/agent/DECISION-TREE.md +84 -0
- package/docs/agent/INVARIANTS.md +40 -0
- package/docs/agent/LAUNCH-PROMPT.md +37 -0
- package/docs/agent/LOOP.md +84 -0
- package/docs/agent/README.md +77 -0
- package/docs/agent/ROADMAP.md +37 -0
- package/docs/agent/STATE.md +93 -0
- package/docs/agent/tasks/00-verify-inherited-work.md +36 -0
- package/docs/agent/tasks/01-authoritative-metadata.md +34 -0
- package/docs/agent/tasks/02-native-dependencies.md +35 -0
- package/docs/agent/tasks/03-reproducible-inputs.md +27 -0
- package/docs/agent/tasks/04-build-frontends.md +27 -0
- package/docs/agent/tasks/05-registry-integration.md +27 -0
- package/docs/agent/tasks/06-package-cohorts.md +42 -0
- package/docs/agent/tasks/07-build-on-miss-boundary.md +31 -0
- package/docs/browser-runtime-architecture.md +142 -0
- package/docs/compatibility-implementation-plan.md +98 -0
- package/docs/developer-tool-packs.md +134 -0
- package/docs/frontend-automation.md +49 -0
- package/docs/fullstack-deployment.md +163 -0
- package/docs/handoff.md +275 -0
- package/docs/original-x64.md +39 -0
- package/docs/platform-hardening.md +49 -0
- package/docs/python/abi.md +97 -0
- package/docs/python/architecture.md +94 -0
- package/docs/python/baseline-inventory.md +54 -0
- package/docs/python/build-on-miss.md +198 -0
- package/docs/python/compatibility.md +206 -0
- package/docs/python/cross-build.md +354 -0
- package/docs/python/extensions.md +282 -0
- package/docs/python/release-gates.md +46 -0
- package/docs/python/virtual-sockets-plan.md +331 -0
- package/docs/runtime-lifecycle-fixes.md +39 -0
- package/docs/server-previews.md +268 -0
- package/docs/virtual-browser.md +120 -0
- package/package.json +5 -3
|
@@ -0,0 +1,206 @@
|
|
|
1
|
+
# Package compatibility report
|
|
2
|
+
|
|
3
|
+
What this pipeline has actually built and run, measured on real packages rather
|
|
4
|
+
than on the fixtures written to succeed. Every wheel below was produced by
|
|
5
|
+
`scripts/build_extension.py` from a pinned, hash-verified source, with no
|
|
6
|
+
package-specific code path anywhere in the builder.
|
|
7
|
+
|
|
8
|
+
ABI for every entry: `sbxabi1-c2637d04695ad927`, tag
|
|
9
|
+
`cp313-cp313-emscripten_5_0_6_wasm32`, Emscripten 5.0.6, CPython 3.13.5.
|
|
10
|
+
|
|
11
|
+
## Status vocabulary
|
|
12
|
+
|
|
13
|
+
| status | meaning |
|
|
14
|
+
| --- | --- |
|
|
15
|
+
| `supported-generic` | builds and runs with no recipe beyond identity and source |
|
|
16
|
+
| `supported-recipe` | needs declared configuration (native library, build env, build tool) but no patch |
|
|
17
|
+
| `supported-patched` | needs a visible, tested patch for an assumption false on this target |
|
|
18
|
+
| `blocked-platform-capability` | the platform does not provide what the package needs; no patch can add it |
|
|
19
|
+
| `blocked-toolchain` | the build system or toolchain cannot produce this artifact yet |
|
|
20
|
+
| `not-yet-attempted` | not tried |
|
|
21
|
+
|
|
22
|
+
## Real packages
|
|
23
|
+
|
|
24
|
+
| package | version | cohort | status | backend | wheel sha256 (16) | size |
|
|
25
|
+
| --- | --- | --- | --- | --- | --- | --- |
|
|
26
|
+
| typing-extensions | 4.12.2 | 1 pure control | `supported-generic` | — (PyPI pure wheel) | — | — |
|
|
27
|
+
| MarkupSafe | 3.0.2 | 2 setuptools C | `supported-generic` | setuptools | `00619948d4c33812` | 12 KB |
|
|
28
|
+
| msgpack | 1.1.0 | 3 Cython | `supported-recipe` | setuptools | `b5336ed6394b0307` | 62 KB |
|
|
29
|
+
| PyYAML | 6.0.2 | 3 + 5 Cython and native library | `supported-recipe` | setuptools | `23c9ad8ee878f41e` | 129 KB |
|
|
30
|
+
| pydantic-core | 2.46.5 | 4 Rust/PyO3 | `supported-recipe` | pyo3 | `01706c4ed32d7f24` | 1951 KB |
|
|
31
|
+
| pydantic-core | 2.23.2 | 4 Rust/PyO3 | `supported-recipe` | pyo3 | `e1bea0f995650d93` | 1934 KB |
|
|
32
|
+
| numpy | 2.2.6 | 8 scientific stress case | `supported-recipe` | meson | `93ae7230479566b4` | 4917 KB |
|
|
33
|
+
| siphash24 | 1.9 | 6 + 7 Meson, patched subproject | `supported-patched` | meson | `cf672c37b0f8e4cd` | 54 KB |
|
|
34
|
+
| simplejson | 3.19.3 | — | `supported-generic` | — (PyPI pure wheel) | — | — |
|
|
35
|
+
| psutil | 6.1.0 | — | `blocked-platform-capability` | setuptools | — | — |
|
|
36
|
+
|
|
37
|
+
Eight independent real packages across six build shapes — pure, setuptools C,
|
|
38
|
+
Cython, Cython plus a separately built native library, Rust/PyO3, and Meson via
|
|
39
|
+
meson-python — install and are exercised by
|
|
40
|
+
`test/python-runtime/package-cohorts.test.ts`. Every cohort in the roadmap now
|
|
41
|
+
has a real package in it. That test
|
|
42
|
+
also loads four of the compiled ones into a single interpreter at once, which
|
|
43
|
+
is the only check that proves the shapes coexist rather than merely each
|
|
44
|
+
working alone.
|
|
45
|
+
|
|
46
|
+
### What each entry adds
|
|
47
|
+
|
|
48
|
+
**typing-extensions** — the control. No cross build, no platform tag. Proves
|
|
49
|
+
the installer still does the ordinary thing when nothing is compiled.
|
|
50
|
+
|
|
51
|
+
**MarkupSafe** — one C source, no dependencies, no configuration. The purest
|
|
52
|
+
measurement of the generic setuptools path: the recipe states identity and
|
|
53
|
+
source and nothing else. Its documented pure-Python fallback is why the test
|
|
54
|
+
asserts `markupsafe._speedups` specifically; an extension that failed to build
|
|
55
|
+
would otherwise install and import cleanly.
|
|
56
|
+
|
|
57
|
+
**msgpack** — a Cython extension with no external library, so a failure here
|
|
58
|
+
would be about generated C rather than about linking. Needs only
|
|
59
|
+
`buildRequires: ["Cython==3.1.4"]`.
|
|
60
|
+
|
|
61
|
+
**PyYAML** — Cython generates `_yaml.c`, which links the
|
|
62
|
+
libyaml built by the native dependency system. It selects between its pure and
|
|
63
|
+
compiled implementations through its own `PYYAML_FORCE_CYTHON` and
|
|
64
|
+
`PYYAML_FORCE_LIBYAML` variables, which the recipe sets through `buildEnv` —
|
|
65
|
+
build configuration, which belongs in a recipe, as distinct from compiler
|
|
66
|
+
flags, which never do. The test asserts `yaml.CSafeLoader` parsed the document,
|
|
67
|
+
which only exists when the extension built.
|
|
68
|
+
|
|
69
|
+
**pydantic-core** — Rust, cross compiled with a rebuilt `std` and wasm
|
|
70
|
+
exceptions. Two versions, because pydantic pins its core exactly and resolving
|
|
71
|
+
that pin is part of what the index has to get right.
|
|
72
|
+
|
|
73
|
+
**simplejson** — recorded for what it shows about policy rather than about
|
|
74
|
+
building. It publishes both C-accelerated platform wheels and a pure
|
|
75
|
+
`py3-none-any` fallback, and the resolver takes the pure one. That is correct:
|
|
76
|
+
a pure wheel is portable and carries no ABI risk. The limitation is real and
|
|
77
|
+
worth stating plainly — **a package that publishes a pure fallback cannot
|
|
78
|
+
currently be accelerated by supplying a compiled wheel, because the pure wheel
|
|
79
|
+
always wins**. Changing that is a policy decision, not an oversight, so no
|
|
80
|
+
compiled simplejson wheel is shipped.
|
|
81
|
+
|
|
82
|
+
**siphash24** — the densest case, and the only one that needed a patch. Three
|
|
83
|
+
things at once: a frontend that is not setuptools (meson-python, driven with a
|
|
84
|
+
cross file generated from the ABI contract); two dependencies resolved through
|
|
85
|
+
Meson `wrap` files, which Meson satisfies by cloning from git *during the
|
|
86
|
+
build* — disabled here, with the subprojects supplied instead as pinned,
|
|
87
|
+
hash-verified archives recorded in the wheel's provenance; and a patch, because
|
|
88
|
+
one of those subprojects builds its library with `both_libraries`, which always
|
|
89
|
+
produces a shared library. `ld.wasm` has none, and Meson refuses at configure
|
|
90
|
+
time rather than at link time, so the project was unconfigurable until the
|
|
91
|
+
library was made static. The test asserts the first published SipHash-2-4
|
|
92
|
+
reference vector, which comes from outside this repository.
|
|
93
|
+
|
|
94
|
+
**numpy** — the case this pipeline was built toward, and deliberately the last
|
|
95
|
+
one attempted, so that it drove none of the design. It needed **no patch** and
|
|
96
|
+
no mechanism that did not already exist for smaller packages: Meson through
|
|
97
|
+
meson-python, a cross file generated from the ABI contract, pinned build tools,
|
|
98
|
+
and its own documented `allow-noblas` switch for a platform with no BLAS. It
|
|
99
|
+
built on the first attempt.
|
|
100
|
+
|
|
101
|
+
Its assertions are numerical rather than structural. NumPy has no pure-Python
|
|
102
|
+
fallback to hide behind, but a miscompiled one imports perfectly well and
|
|
103
|
+
computes wrong answers, so each check has an exactly known result: `a @ a.T`,
|
|
104
|
+
an inverse multiplied back to the identity, a round-trip FFT, a reduction.
|
|
105
|
+
`long double` deserves naming — Meson cannot run a program on the target to
|
|
106
|
+
learn its layout, so the value is derived from the pinned compiler's own
|
|
107
|
+
`__LDBL_MANT_DIG__` (113, little-endian, so `IEEE_QUAD_LE`) and the built
|
|
108
|
+
extension reports the matching 16-byte itemsize. Getting that wrong would have
|
|
109
|
+
been silently wrong arithmetic rather than a build failure.
|
|
110
|
+
|
|
111
|
+
NumPy's *own* test suite runs inside the runtime, which is the strongest
|
|
112
|
+
evidence in this document because none of it was written here:
|
|
113
|
+
|
|
114
|
+
| module | result |
|
|
115
|
+
| --- | --- |
|
|
116
|
+
| `numpy._core.tests.test_umath` | 4388 passed, 362 skipped, 2 xfailed, 5 xpassed |
|
|
117
|
+
| `numpy.linalg.tests.test_linalg` | 416 passed, 21 skipped, 2 xfailed |
|
|
118
|
+
| `numpy.fft.tests.test_pocketfft` | 144 passed, 4 skipped |
|
|
119
|
+
|
|
120
|
+
4,948 upstream tests pass and none fails. Reproduce with `pip install numpy
|
|
121
|
+
pytest hypothesis` in a container, then:
|
|
122
|
+
|
|
123
|
+
```bash
|
|
124
|
+
python3 -m pytest --pyargs numpy._core.tests.test_umath -q --capture=no
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
`--capture=no` is required, and the reason is a runtime limitation rather than
|
|
128
|
+
anything to do with NumPy: pytest's default output capture opens a temporary
|
|
129
|
+
file descriptor and reading it back fails with `OSError: [Errno 8] Bad file
|
|
130
|
+
descriptor`, so *no* tests run at all. It belongs with the other pre-existing
|
|
131
|
+
process- and descriptor-level gaps recorded in the project state, not with
|
|
132
|
+
package support.
|
|
133
|
+
|
|
134
|
+
**psutil** — attempted deliberately as a negative case, and the failure is more
|
|
135
|
+
useful than the success would have been.
|
|
136
|
+
|
|
137
|
+
## The psutil finding
|
|
138
|
+
|
|
139
|
+
Command: `python3 scripts/build_extension.py recipes/psutil/recipe.json`
|
|
140
|
+
Source: `psutil-6.1.0.tar.gz`, sha256 `353815f5…c7a`
|
|
141
|
+
|
|
142
|
+
Earliest unsupported assumption, quoted from the build:
|
|
143
|
+
|
|
144
|
+
```
|
|
145
|
+
emcc … -DPSUTIL_POSIX=1 -DPSUTIL_OSX=1 … -c psutil/_psutil_common.c
|
|
146
|
+
psutil/_psutil_common.h:131:14: fatal error: 'mach/mach_time.h' file not found
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
`-DPSUTIL_OSX=1` is the whole finding. `sysconfig` has been pointed at the
|
|
150
|
+
target throughout, so the compiler, its flags and the extension suffix are all
|
|
151
|
+
correct — but psutil's `setup.py` selects its platform implementation from
|
|
152
|
+
`sys.platform`, which reports the *running* interpreter's operating system.
|
|
153
|
+
Under a cross build that is the build machine, so on macOS it compiled the
|
|
154
|
+
Darwin backend and asked for a Mach header.
|
|
155
|
+
|
|
156
|
+
This is assumption 7 in `cross-build.md` in its most common concrete form, and
|
|
157
|
+
it generalises well beyond psutil: any package that branches on `sys.platform`
|
|
158
|
+
in its build will describe the build machine. It is worth recording that
|
|
159
|
+
`_PYTHON_HOST_PLATFORM` does **not** fix this — that variable changes
|
|
160
|
+
`sysconfig.get_platform()`, not `sys.platform`.
|
|
161
|
+
|
|
162
|
+
psutil is nevertheless classified `blocked-platform-capability` rather than
|
|
163
|
+
`supported-patched`. A patch could make it select a POSIX backend, but psutil
|
|
164
|
+
reads process and system tables through syscalls Emscripten does not implement;
|
|
165
|
+
the capability is absent from the platform, not from the build. Patching it
|
|
166
|
+
would convert a clear build failure into a runtime one, which is the trade this
|
|
167
|
+
project refuses everywhere else.
|
|
168
|
+
|
|
169
|
+
## Cohorts
|
|
170
|
+
|
|
171
|
+
All eight cohorts now have a real package. Cohorts 6 (a package requiring a
|
|
172
|
+
visible cross-compilation patch) and 7 (a PEP 517 backend that is not
|
|
173
|
+
setuptools) are both covered by siphash24; cohort 8 by NumPy.
|
|
174
|
+
|
|
175
|
+
## Packages with no recipe
|
|
176
|
+
|
|
177
|
+
Everything above has a recipe checked in, because each was measured
|
|
178
|
+
deliberately. Most packages need none: `pip install <anything>` will generate
|
|
179
|
+
one, build the wheel and install it, when a builder is enabled — see
|
|
180
|
+
`build-on-miss.md`. `ujson` is the worked example, built with no recipe, no
|
|
181
|
+
patch and no hand-pinned tool:
|
|
182
|
+
|
|
183
|
+
```
|
|
184
|
+
$ pip install ujson
|
|
185
|
+
Collecting ujson
|
|
186
|
+
Building ujson from source (no wheel for this runtime yet)
|
|
187
|
+
Built ujson
|
|
188
|
+
Successfully installed ujson-6.0.0
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
A package that cannot be built this way still lands in the vocabulary above
|
|
192
|
+
rather than as an opaque failure: a backend with no adapter is
|
|
193
|
+
`blocked-toolchain` and says which backend, and a package with no source
|
|
194
|
+
distribution says that instead.
|
|
195
|
+
|
|
196
|
+
## Reproducing
|
|
197
|
+
|
|
198
|
+
```bash
|
|
199
|
+
make -C python-runtime native-deps
|
|
200
|
+
python3 python-runtime/scripts/build_extension.py python-runtime/recipes/<name>/recipe.json
|
|
201
|
+
python3 python-runtime/scripts/build_index.py
|
|
202
|
+
npx vitest run test/python-runtime/package-cohorts.test.ts
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
Every wheel above rebuilds byte-identically; the digests in this table are
|
|
206
|
+
therefore checkable rather than decorative.
|
|
@@ -0,0 +1,354 @@
|
|
|
1
|
+
# Cross-building native extensions
|
|
2
|
+
|
|
3
|
+
The pipeline that turns a package's own source into a SandboxedJs wheel, and
|
|
4
|
+
the list of things it still assumes about a package — so each of them can be
|
|
5
|
+
removed deliberately rather than discovered by a failure.
|
|
6
|
+
|
|
7
|
+
## What it does
|
|
8
|
+
|
|
9
|
+
`python-runtime/scripts/build_extension.py` takes a recipe and produces a
|
|
10
|
+
wheel tagged `cp313-cp313-emscripten_5_0_6_wasm32` with a `Build-ABI` line
|
|
11
|
+
naming the `abiId` computed from `abi/extension-abi.json`. Three recipe kinds
|
|
12
|
+
exist:
|
|
13
|
+
|
|
14
|
+
| kind | source of truth for *what* to build | covers |
|
|
15
|
+
| --- | --- | --- |
|
|
16
|
+
| `setuptools` | the package's own `setup.py` / `pyproject.toml` | plain C extensions, Cython, anything setuptools builds |
|
|
17
|
+
| `pep517` | the package's declared build backend and hooks | pyproject-only projects, in-tree backends |
|
|
18
|
+
| `meson` | the project's `meson.build`, via meson-python | Meson projects, including NumPy |
|
|
19
|
+
| `pyo3` | the crate's `Cargo.toml` | Rust extensions |
|
|
20
|
+
| `c-extension` | the recipe's own `sources` list | probes only — see the assumptions below |
|
|
21
|
+
|
|
22
|
+
`kind` selects a backend explicitly; nothing is inferred from which files are
|
|
23
|
+
present. A project with both a `setup.py` and a `pyproject.toml` can be built
|
|
24
|
+
either way, and guessing would silently change how a package is built the day
|
|
25
|
+
upstream adds a file. Recipes are schema-validated before anything is fetched
|
|
26
|
+
or compiled, so a malformed one costs a second rather than a Rust build.
|
|
27
|
+
|
|
28
|
+
In every case the *how* — compiler, target, PIC, threads, exceptions,
|
|
29
|
+
extension suffix, wheel tag — comes from `abi/extension-abi.json` and only
|
|
30
|
+
from there. A recipe cannot contribute a compile or link flag. This is not
|
|
31
|
+
tidiness: a recipe that could set its own `-pthread` could produce an artifact
|
|
32
|
+
that links against a main module with a different memory model and then
|
|
33
|
+
corrupts memory, and the mismatch would be invisible in the wheel.
|
|
34
|
+
|
|
35
|
+
## How the setuptools path works
|
|
36
|
+
|
|
37
|
+
`sysconfig` loads its table of build variables from a module named by the
|
|
38
|
+
`_PYTHON_SYSCONFIGDATA_NAME` environment variable, and
|
|
39
|
+
`distutils.command.build_ext` — which setuptools vendors — takes the compiler,
|
|
40
|
+
its flags and the extension suffix from that table. So the package's own
|
|
41
|
+
`setup.py build` runs on the *build machine's* interpreter with `sysconfig`
|
|
42
|
+
answering for the *target*. This is the mechanism CPython uses for its own
|
|
43
|
+
cross builds and the one `crossenv` automates; nothing about it is specific to
|
|
44
|
+
this project.
|
|
45
|
+
|
|
46
|
+
The CPython cross build already writes the target's table
|
|
47
|
+
(`_sysconfigdata__emscripten_wasm32-emscripten.py`). `scripts/crossenv.py`
|
|
48
|
+
copies it and overrides exactly three groups of values, all from the ABI
|
|
49
|
+
contract:
|
|
50
|
+
|
|
51
|
+
* `INCLUDEPY` / `CONFINCLUDEPY` — the shipped table names
|
|
52
|
+
`$prefix/include/python3.13`, where the headers would be if the interpreter
|
|
53
|
+
had been installed. It has not been; they are in the source tree.
|
|
54
|
+
* `LDSHARED` / `BLDSHARED` / `LDCXXSHARED` — the shipped link line has no
|
|
55
|
+
`-sSIDE_MODULE=1`, because CPython adds it in a rule outside `LDSHARED`.
|
|
56
|
+
Without it the artifact is a complete program and `dlopen` cannot load it.
|
|
57
|
+
* `CCSHARED` — the ABI's `-pthread -fwasm-exceptions -fPIC`, which must match
|
|
58
|
+
the main module exactly. `CCSHARED` rather than `CFLAGS` so the flags apply
|
|
59
|
+
to extension objects and not to host tools a package builds during setup.
|
|
60
|
+
|
|
61
|
+
Cython needs no support at all: `cythonize()` runs inside the package's
|
|
62
|
+
`setup.py` on the build machine and hands generated C to the same `build_ext`.
|
|
63
|
+
The only difference between the plain and Cython fixtures is a pinned
|
|
64
|
+
`buildRequires`.
|
|
65
|
+
|
|
66
|
+
## Validation
|
|
67
|
+
|
|
68
|
+
Three checks, in order of how misleading their failure would otherwise be:
|
|
69
|
+
|
|
70
|
+
1. `verify_toolchain` refuses an Emscripten other than the pinned one. A side
|
|
71
|
+
module built by a different Emscripten links and then fails at run time.
|
|
72
|
+
2. `verify_side_module` checks the artifact is a WebAssembly binary, has a
|
|
73
|
+
`dylink` section, and exports `PyInit_<last component of the module name>`.
|
|
74
|
+
A missing dylink section fails at `dlopen` with a message about the file; a
|
|
75
|
+
missing init symbol fails with "dynamic module does not define module
|
|
76
|
+
export function", which reads like a source error and is in fact a link
|
|
77
|
+
setting.
|
|
78
|
+
3. `test/python-runtime/extensions.test.ts` installs each wheel from an HTTP
|
|
79
|
+
index with the container's own `pip` and imports it with CPython's ordinary
|
|
80
|
+
import machinery. This is the only check that proves the ABI, because the
|
|
81
|
+
first two can pass on a module compiled against the wrong headers.
|
|
82
|
+
|
|
83
|
+
## Package-specific assumptions, and how to remove each
|
|
84
|
+
|
|
85
|
+
Each of these is a place the pipeline knows, or requires a recipe to state,
|
|
86
|
+
something a package already states about itself.
|
|
87
|
+
|
|
88
|
+
**1. Target dependencies are not resolved.** *Resolved.* Native target
|
|
89
|
+
dependencies are declared by `dependency.json` manifests under
|
|
90
|
+
`python-runtime/native-deps/` (real libraries) and
|
|
91
|
+
`python-runtime/fixtures/native/` (test ones), built by
|
|
92
|
+
`scripts/native_deps.py` into the profile's sysroot with the ABI's own flags,
|
|
93
|
+
and reached by extensions through the generated cross configuration — so
|
|
94
|
+
`libraries=["yaml"]` in a package's `setup.py` resolves with no path or flag
|
|
95
|
+
in the recipe. A recipe names what it needs in `nativeRequires`, which is a
|
|
96
|
+
separate namespace from `requires` (Python) and `buildRequires` (build
|
|
97
|
+
machine) so that a build-machine library can never satisfy a target link.
|
|
98
|
+
|
|
99
|
+
Reuse is decided by a stamp recording the source digest, profile, exact
|
|
100
|
+
compiler flags and the digest of every installed output — not by whether a
|
|
101
|
+
file exists, which is what let an archive built for one profile satisfy
|
|
102
|
+
another. What remains: only `make` and `configure` build systems are
|
|
103
|
+
supported, and CMake and Meson libraries will need a third; and the libraries
|
|
104
|
+
CPython itself links are still built by the older hand-written
|
|
105
|
+
`scripts/build_dependencies.py` rather than through manifests.
|
|
106
|
+
|
|
107
|
+
**2. Wheel metadata is synthesized from the recipe on the `pyo3` path only.**
|
|
108
|
+
*Resolved for `setuptools`.* That path now runs the package's own `dist_info`
|
|
109
|
+
under the cross environment and ships the `METADATA` it generates, so markers,
|
|
110
|
+
extras and `Requires-Python` are upstream's and a recipe has no `requires`
|
|
111
|
+
field to drift from them. Each wheel records which of the two it used in its
|
|
112
|
+
`WHEEL` file as `Metadata-Source: package|recipe`, so the remaining cases are
|
|
113
|
+
visible in the artifacts rather than only here.
|
|
114
|
+
|
|
115
|
+
A recipe may still *remove* a requirement through `dependencyOverrides`, which
|
|
116
|
+
must carry a `because`, must name a requirement the package actually has, and
|
|
117
|
+
is recorded in the built wheel as `Build-Dropped-Requirement`. *Remove the
|
|
118
|
+
remaining `recipe` cases by:* reading `[project]` from the crate's
|
|
119
|
+
`pyproject.toml` for maturin-built extensions, or by driving maturin itself
|
|
120
|
+
under the cross environment.
|
|
121
|
+
|
|
122
|
+
**3. `pythonRoots` exists only because the `pyo3` path has no `build_py`.** It
|
|
123
|
+
is a hand-written statement of which directory holds the package's Python
|
|
124
|
+
half; the pydantic-core recipe encodes maturin's `python/` convention. *Remove
|
|
125
|
+
by:* reading `[tool.maturin] python-source` from the crate's `pyproject.toml`,
|
|
126
|
+
or by driving maturin itself under the cross environment.
|
|
127
|
+
|
|
128
|
+
**4. The `pyo3` path declares module layout.** `crateName` and `package` state
|
|
129
|
+
where the artifact must sit for `from ._x import ...` to work. The setuptools
|
|
130
|
+
path derives the same thing from the build tree. *Remove by:* same as 3.
|
|
131
|
+
|
|
132
|
+
**5. The `pyo3` recipe hard-codes an unpacked source path.**
|
|
133
|
+
`recipes/pydantic-core/recipe.json` names
|
|
134
|
+
`../../out/ports/pydantic_core-2.46.5`, duplicating the version that appears
|
|
135
|
+
three other times in the same file. *Remove by:* deriving the crate path from
|
|
136
|
+
`source.unpackTo` and the archive's own stem, which `fetch_source` already
|
|
137
|
+
computes.
|
|
138
|
+
|
|
139
|
+
**6. `kind: "c-extension"` is a transcription.** The recipe lists sources the
|
|
140
|
+
package's build system already lists, so the two can disagree silently. It is
|
|
141
|
+
retained only for `fixtures/sbx_c_probe`, which deliberately has no build
|
|
142
|
+
system. *Remove by:* giving the fixture a `setup.py` and deleting the kind and
|
|
143
|
+
`compile_c_module` with it.
|
|
144
|
+
|
|
145
|
+
**7. `setup.py` is assumed to be cross-safe.** *Mitigated, not removable.* A
|
|
146
|
+
`setup.py` that probes the build machine (compiles a test program, runs the
|
|
147
|
+
extension it is building, reads `platform.machine()`) will describe the build
|
|
148
|
+
machine. Nothing detects this; the build succeeds and produces a wrong
|
|
149
|
+
artifact. It is a property of the package, so it cannot be fixed generically —
|
|
150
|
+
but it now has one designated place to be fixed in. A recipe declares
|
|
151
|
+
`patches: {"dir": …}`, and the patch set states the package, the version, the
|
|
152
|
+
digest of the source tree it was written against, and per patch a reason, the
|
|
153
|
+
target fact being substituted, and the test that proves it. Patches apply to a
|
|
154
|
+
staged copy with `git apply` at exact context; a drifted source is refused
|
|
155
|
+
rather than fuzzily patched. `fixtures/sbx_patched_probe` is exactly this
|
|
156
|
+
failure — `platform.machine()` declaring 64-bit pointers for a 32-bit target —
|
|
157
|
+
and its patch is the worked example.
|
|
158
|
+
|
|
159
|
+
**8. Build tools come from PyPI at first build.** *Resolved.*
|
|
160
|
+
`build-tools.lock` pins every build tool — including `setuptools` itself, which
|
|
161
|
+
was previously taken from whatever the build machine had — and records the
|
|
162
|
+
sha256 of every distribution published for that version, so one lock serves any
|
|
163
|
+
build machine. Installation uses `--require-hashes`, which also refuses a tool
|
|
164
|
+
that grows an unpinned dependency. Regenerate with
|
|
165
|
+
`scripts/lock_build_tools.py`; a build reads the lock and never refreshes it.
|
|
166
|
+
|
|
167
|
+
**9. One ABI, no matrix.** `abiId` covers exactly one configuration; there is
|
|
168
|
+
no way to build the same recipe for a second profile. *Remove by:* taking the
|
|
169
|
+
profile from the command line into the wheel's local version segment, once a
|
|
170
|
+
second profile exists to want it.
|
|
171
|
+
|
|
172
|
+
**10. `--build-lib` is assumed to be honoured.** Projects with a custom
|
|
173
|
+
`build` command that ignores it would stage nowhere the collector looks; the
|
|
174
|
+
build "succeeds" and produces no modules. This is caught — the empty-module
|
|
175
|
+
case raises — but only after the fact.
|
|
176
|
+
|
|
177
|
+
## Native target dependencies
|
|
178
|
+
|
|
179
|
+
Three kinds of dependency exist and are deliberately kept in three namespaces,
|
|
180
|
+
because a single list would let a build machine's library satisfy a target
|
|
181
|
+
link — producing a module that links cleanly and then faults:
|
|
182
|
+
|
|
183
|
+
| recipe field | what it is | where it goes |
|
|
184
|
+
| --- | --- | --- |
|
|
185
|
+
| `requires` | Python runtime dependency | installed in the container, imported |
|
|
186
|
+
| `buildRequires` | build-machine tool (Cython, setuptools_scm) | a per-recipe venv on this machine |
|
|
187
|
+
| `nativeRequires` | C library cross compiled for wasm32 | `out/sysroot-<profile>`, linked into the side module |
|
|
188
|
+
|
|
189
|
+
A native dependency is declared by a `dependency.json`:
|
|
190
|
+
|
|
191
|
+
```json
|
|
192
|
+
{
|
|
193
|
+
"name": "yaml", "version": "0.2.5",
|
|
194
|
+
"license": "MIT", "licenseFiles": ["License"],
|
|
195
|
+
"provenance": { "url": "…", "sha256": "…", "unpackDir": "yaml-0.2.5" },
|
|
196
|
+
"profiles": ["dynamic"], "dependsOn": [],
|
|
197
|
+
"build": { "system": "configure", "configureArgs": ["--disable-shared"] },
|
|
198
|
+
"outputs": { "libraries": ["libyaml.a"], "headers": ["yaml.h"] }
|
|
199
|
+
}
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
`name` must be the name a linker is given (`-lyaml`), not the project's title:
|
|
203
|
+
a manifest called `libyaml` would build the right library and leave the
|
|
204
|
+
extension unable to ask for it.
|
|
205
|
+
|
|
206
|
+
`outputs` is checked after the build and recorded in the stamp. A build system
|
|
207
|
+
that quietly produces nothing — a `configure` that disabled the library, a
|
|
208
|
+
`make` that built only tools — otherwise looks like a success until an
|
|
209
|
+
extension fails to link with a message about a missing symbol.
|
|
210
|
+
|
|
211
|
+
Provenance is either `local` (source in this repository, still digested so an
|
|
212
|
+
edit rebuilds) or a pinned `url` and `sha256`, verified before anything is
|
|
213
|
+
unpacked. These are kept out of `sources.lock` on purpose: that file pins what
|
|
214
|
+
the *interpreter* links, and a library that exists for packages should not
|
|
215
|
+
become something the runtime build has to download.
|
|
216
|
+
|
|
217
|
+
`build.host` exists because autotools packages vendor a `config.sub` frozen at
|
|
218
|
+
their release date. libyaml 0.2.5's is from 2018 and rejects
|
|
219
|
+
`wasm32-unknown-emscripten` outright; it accepts `wasm32-unknown-none`, which
|
|
220
|
+
is sufficient because `--host` only tells `configure` that this is a cross
|
|
221
|
+
build — `emconfigure` has already supplied the tools. That is a property of
|
|
222
|
+
the package, so it is declared per dependency rather than worked around for
|
|
223
|
+
everybody.
|
|
224
|
+
|
|
225
|
+
## Reproducible inputs, patches and provenance
|
|
226
|
+
|
|
227
|
+
Builds never run in the source tree. The project is copied to a staging
|
|
228
|
+
directory first, then patched, then built. Two failures follow from doing
|
|
229
|
+
otherwise: a patch applied in place leaves the checkout modified, so a second
|
|
230
|
+
build starts from different source and the patch no longer applies; and a build
|
|
231
|
+
leaves generated files behind — Cython's `.c`, `.egg-info` — which then become
|
|
232
|
+
inputs to the next one.
|
|
233
|
+
|
|
234
|
+
Every input is pinned and verified before use:
|
|
235
|
+
|
|
236
|
+
| input | pinned by | verified by |
|
|
237
|
+
| --- | --- | --- |
|
|
238
|
+
| interpreter and its libraries | `sources.lock` | sha256 before unpacking |
|
|
239
|
+
| build tools (setuptools, Cython) | `build-tools.lock` | `pip --require-hashes` |
|
|
240
|
+
| native target libraries | `dependency.json` provenance | sha256 before unpacking |
|
|
241
|
+
| package source | recipe `source`, or in-repo | sha256, or tree digest |
|
|
242
|
+
| patches | `patches.json` `sourceDigest` | exact-context `git apply` |
|
|
243
|
+
|
|
244
|
+
Wheels are byte-reproducible: every zip member is stamped with the zip epoch
|
|
245
|
+
rather than the build time and RECORD is sorted, so two builds from the same
|
|
246
|
+
inputs produce identical files. This is stronger than the "documented timestamp
|
|
247
|
+
normalization" the task allowed, and it is what makes "did anything actually
|
|
248
|
+
change?" answerable by comparing digests. Confirmed by rebuilding every wheel
|
|
249
|
+
and diffing digests, and by `test/python-runtime/reproducible-builds.test.ts`.
|
|
250
|
+
|
|
251
|
+
Each wheel carries `dist-info/sandboxedjs-provenance.json`, which answers, for
|
|
252
|
+
an artifact held by someone without this repository: which source tree digest
|
|
253
|
+
produced it, which build tools and native libraries went into it, which ABI id
|
|
254
|
+
and recipe revision were used, and which patches were applied. The index
|
|
255
|
+
summarises those fields so they can be compared across wheels without
|
|
256
|
+
downloading any.
|
|
257
|
+
|
|
258
|
+
## Build backends
|
|
259
|
+
|
|
260
|
+
`scripts/backends.py` is the only place a package's build system is known
|
|
261
|
+
about. The orchestrator verifies inputs, selects a backend by the name the
|
|
262
|
+
recipe stated, hands it a `BuildRequest`, and packages the `BuildResult` that
|
|
263
|
+
comes back. It decides nothing else — which is what keeps a package-name check
|
|
264
|
+
from appearing in it, since once an orchestrator is already making build
|
|
265
|
+
decisions, one more looks harmless.
|
|
266
|
+
|
|
267
|
+
A backend receives verified staged source, the locked build-tool interpreter,
|
|
268
|
+
the cross environment, the declared native dependencies, a staging directory
|
|
269
|
+
and a read-only view of the ABI. It returns a wheel-layout tree, the package's
|
|
270
|
+
own `.dist-info`, and evidence recorded in the wheel's provenance. It cannot
|
|
271
|
+
contribute a compiler flag, and the orchestrator digests the ABI contract
|
|
272
|
+
before and after the build — a backend that mutated it would produce a
|
|
273
|
+
correctly-tagged wheel built for a different ABI.
|
|
274
|
+
|
|
275
|
+
The `pep517` backend calls the package's declared hooks the way a frontend
|
|
276
|
+
does, including resolving an in-tree backend through `backend-path`. It does
|
|
277
|
+
**not** use `python -m build`, whose isolation installs whatever versions
|
|
278
|
+
upstream publishes on the day of the build — the unpinned input the rest of
|
|
279
|
+
this pipeline refuses. The environment is assembled from `build-tools.lock`,
|
|
280
|
+
and a build requirement absent from the lock is refused rather than fetched.
|
|
281
|
+
The wheel the hooks produce is unpacked rather than shipped: it carries
|
|
282
|
+
whatever tag the backend chose, and this pipeline tags a wheel for what it
|
|
283
|
+
actually is.
|
|
284
|
+
|
|
285
|
+
Build tools run on an interpreter whose feature version matches the target's.
|
|
286
|
+
Most of the cross build does not care, because `sysconfig` answers for the
|
|
287
|
+
target either way — but `bdist_wheel` composes a tag from the *running*
|
|
288
|
+
interpreter's version and the *target's* ABI tag. Building for 3.13 from 3.14
|
|
289
|
+
yields `('cp314', 'cp313', 'emscripten_5_0_6_wasm32')`, which fails an
|
|
290
|
+
assertion inside setuptools rather than anywhere that names the cause.
|
|
291
|
+
|
|
292
|
+
## Meson
|
|
293
|
+
|
|
294
|
+
`kind: "meson"` drives a project through meson-python's PEP 517 hooks. It is
|
|
295
|
+
its own backend rather than a `pep517` recipe with extra settings because a
|
|
296
|
+
cross file is backend mechanics, not a package quirk — a recipe that passed its
|
|
297
|
+
own `--cross-file` could describe a different target than the wheel is tagged
|
|
298
|
+
for.
|
|
299
|
+
|
|
300
|
+
Meson does not read `sysconfig`; it is told about a target by a file. So the
|
|
301
|
+
same facts the cross table states are stated again in Meson's format, generated
|
|
302
|
+
from `abi/extension-abi.json` by `write_meson_cross_file`. Three settings are
|
|
303
|
+
supplied by the backend because they are facts about the target, not choices:
|
|
304
|
+
|
|
305
|
+
- `needs_exe_wrapper = true`, or Meson believes it can run what it builds and
|
|
306
|
+
every compile-and-run check silently tests the build machine.
|
|
307
|
+
- `--wrap-mode=nodownload`. Meson otherwise resolves a missing subproject by
|
|
308
|
+
cloning it mid-build — an unpinned input arriving over the network at the one
|
|
309
|
+
moment nothing is watching.
|
|
310
|
+
- `-Ddefault_library=static`. There is no such thing as a shared library here;
|
|
311
|
+
`ld.wasm` refuses, and Meson reports it at configure time as a message about
|
|
312
|
+
the linker rather than about the default that reached it.
|
|
313
|
+
- `longdouble_format`, derived by asking the pinned compiler for its
|
|
314
|
+
`__LDBL_MANT_DIG__` and byte order rather than being written down. Meson
|
|
315
|
+
cannot run a program on the target to find out, and a project that inspects
|
|
316
|
+
float layouts — NumPy does — will not configure without it. Deriving it
|
|
317
|
+
matters more than the convenience: a hand-written value that disagreed with
|
|
318
|
+
the toolchain would produce a library that builds, imports, and computes
|
|
319
|
+
wrong answers.
|
|
320
|
+
|
|
321
|
+
A project whose subprojects were previously downloaded supplies them through
|
|
322
|
+
`vendoredSources`: hash-verified archives unpacked at declared paths inside the
|
|
323
|
+
staged tree, recorded in the wheel's provenance because they are compiled into
|
|
324
|
+
it. They cannot escape the staged tree.
|
|
325
|
+
|
|
326
|
+
### Two things that were failures first
|
|
327
|
+
|
|
328
|
+
**Split headers.** A cross build leaves `Include/` in the source tree and the
|
|
329
|
+
generated `pyconfig.h` in the build directory; an *installed* interpreter has
|
|
330
|
+
them in one place, and tools assume the installed shape. `Python.h` includes
|
|
331
|
+
`"pyconfig.h"` in quotes, so the compiler looks beside `Python.h` and then walks
|
|
332
|
+
the `-I` list — and finds the *build machine's* `pyconfig.h` from whichever host
|
|
333
|
+
include directory a tool added. The build then fails with `LONG_BIT definition
|
|
334
|
+
appears wrong for platform`, which reads like a broken toolchain and is in fact
|
|
335
|
+
a 64-bit header describing a 32-bit target. Meson adds such a directory;
|
|
336
|
+
setuptools happens not to. Both now compile against one staged directory
|
|
337
|
+
holding every target header.
|
|
338
|
+
|
|
339
|
+
**Paths in compiled output.** This one recurred three times before the rule was
|
|
340
|
+
clear: *nothing a compiler sees may live at a per-build path.*
|
|
341
|
+
|
|
342
|
+
First the staged include directory, created inside the temporary build
|
|
343
|
+
directory — the compile line carries `-g`, so the varying path reached debug
|
|
344
|
+
information. Then Meson's own `.mesonpy-<random>` build directory, which also
|
|
345
|
+
has to be pinned together with the cross file, because Meson caches the cross
|
|
346
|
+
file's path inside the build directory and a stale one fails on the next run
|
|
347
|
+
with a `FileNotFoundError` about a file nobody asked for.
|
|
348
|
+
|
|
349
|
+
Then the staged *source* directory, which was the subtlest: Cython writes the
|
|
350
|
+
`.pyx` path into its generated C so a traceback can name a line. NumPy exposed
|
|
351
|
+
it — 869 of the 874 files in its wheel matched between builds, and the five
|
|
352
|
+
that differed were its Cython-generated modules, all at identical sizes. The
|
|
353
|
+
staged source is now a stable path too, and `reproducible-builds.test.ts`
|
|
354
|
+
checks that the generated cross configuration carries no temporary path.
|