sandboxedjs 0.2.11 → 0.2.12

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. package/README.md +16 -9
  2. package/assets/logo.png +0 -0
  3. package/bin/sandboxedjs-egress.mjs +25 -10
  4. package/dist/index.cjs +1 -1
  5. package/dist/index.js +1 -1
  6. package/dist/service-worker.js +3 -2
  7. package/docs/agent/COMMANDS.md +85 -0
  8. package/docs/agent/DECISION-TREE.md +84 -0
  9. package/docs/agent/INVARIANTS.md +40 -0
  10. package/docs/agent/LAUNCH-PROMPT.md +37 -0
  11. package/docs/agent/LOOP.md +84 -0
  12. package/docs/agent/README.md +77 -0
  13. package/docs/agent/ROADMAP.md +37 -0
  14. package/docs/agent/STATE.md +93 -0
  15. package/docs/agent/tasks/00-verify-inherited-work.md +36 -0
  16. package/docs/agent/tasks/01-authoritative-metadata.md +34 -0
  17. package/docs/agent/tasks/02-native-dependencies.md +35 -0
  18. package/docs/agent/tasks/03-reproducible-inputs.md +27 -0
  19. package/docs/agent/tasks/04-build-frontends.md +27 -0
  20. package/docs/agent/tasks/05-registry-integration.md +27 -0
  21. package/docs/agent/tasks/06-package-cohorts.md +42 -0
  22. package/docs/agent/tasks/07-build-on-miss-boundary.md +31 -0
  23. package/docs/browser-runtime-architecture.md +142 -0
  24. package/docs/compatibility-implementation-plan.md +98 -0
  25. package/docs/developer-tool-packs.md +134 -0
  26. package/docs/frontend-automation.md +49 -0
  27. package/docs/fullstack-deployment.md +163 -0
  28. package/docs/handoff.md +275 -0
  29. package/docs/original-x64.md +39 -0
  30. package/docs/platform-hardening.md +49 -0
  31. package/docs/python/abi.md +97 -0
  32. package/docs/python/architecture.md +94 -0
  33. package/docs/python/baseline-inventory.md +54 -0
  34. package/docs/python/build-on-miss.md +198 -0
  35. package/docs/python/compatibility.md +206 -0
  36. package/docs/python/cross-build.md +354 -0
  37. package/docs/python/extensions.md +282 -0
  38. package/docs/python/release-gates.md +46 -0
  39. package/docs/python/virtual-sockets-plan.md +331 -0
  40. package/docs/runtime-lifecycle-fixes.md +39 -0
  41. package/docs/server-previews.md +268 -0
  42. package/docs/virtual-browser.md +120 -0
  43. package/package.json +5 -3
@@ -0,0 +1,206 @@
1
+ # Package compatibility report
2
+
3
+ What this pipeline has actually built and run, measured on real packages rather
4
+ than on the fixtures written to succeed. Every wheel below was produced by
5
+ `scripts/build_extension.py` from a pinned, hash-verified source, with no
6
+ package-specific code path anywhere in the builder.
7
+
8
+ ABI for every entry: `sbxabi1-c2637d04695ad927`, tag
9
+ `cp313-cp313-emscripten_5_0_6_wasm32`, Emscripten 5.0.6, CPython 3.13.5.
10
+
11
+ ## Status vocabulary
12
+
13
+ | status | meaning |
14
+ | --- | --- |
15
+ | `supported-generic` | builds and runs with no recipe beyond identity and source |
16
+ | `supported-recipe` | needs declared configuration (native library, build env, build tool) but no patch |
17
+ | `supported-patched` | needs a visible, tested patch for an assumption false on this target |
18
+ | `blocked-platform-capability` | the platform does not provide what the package needs; no patch can add it |
19
+ | `blocked-toolchain` | the build system or toolchain cannot produce this artifact yet |
20
+ | `not-yet-attempted` | not tried |
21
+
22
+ ## Real packages
23
+
24
+ | package | version | cohort | status | backend | wheel sha256 (16) | size |
25
+ | --- | --- | --- | --- | --- | --- | --- |
26
+ | typing-extensions | 4.12.2 | 1 pure control | `supported-generic` | — (PyPI pure wheel) | — | — |
27
+ | MarkupSafe | 3.0.2 | 2 setuptools C | `supported-generic` | setuptools | `00619948d4c33812` | 12 KB |
28
+ | msgpack | 1.1.0 | 3 Cython | `supported-recipe` | setuptools | `b5336ed6394b0307` | 62 KB |
29
+ | PyYAML | 6.0.2 | 3 + 5 Cython and native library | `supported-recipe` | setuptools | `23c9ad8ee878f41e` | 129 KB |
30
+ | pydantic-core | 2.46.5 | 4 Rust/PyO3 | `supported-recipe` | pyo3 | `01706c4ed32d7f24` | 1951 KB |
31
+ | pydantic-core | 2.23.2 | 4 Rust/PyO3 | `supported-recipe` | pyo3 | `e1bea0f995650d93` | 1934 KB |
32
+ | numpy | 2.2.6 | 8 scientific stress case | `supported-recipe` | meson | `93ae7230479566b4` | 4917 KB |
33
+ | siphash24 | 1.9 | 6 + 7 Meson, patched subproject | `supported-patched` | meson | `cf672c37b0f8e4cd` | 54 KB |
34
+ | simplejson | 3.19.3 | — | `supported-generic` | — (PyPI pure wheel) | — | — |
35
+ | psutil | 6.1.0 | — | `blocked-platform-capability` | setuptools | — | — |
36
+
37
+ Eight independent real packages across six build shapes — pure, setuptools C,
38
+ Cython, Cython plus a separately built native library, Rust/PyO3, and Meson via
39
+ meson-python — install and are exercised by
40
+ `test/python-runtime/package-cohorts.test.ts`. Every cohort in the roadmap now
41
+ has a real package in it. That test
42
+ also loads four of the compiled ones into a single interpreter at once, which
43
+ is the only check that proves the shapes coexist rather than merely each
44
+ working alone.
45
+
46
+ ### What each entry adds
47
+
48
+ **typing-extensions** — the control. No cross build, no platform tag. Proves
49
+ the installer still does the ordinary thing when nothing is compiled.
50
+
51
+ **MarkupSafe** — one C source, no dependencies, no configuration. The purest
52
+ measurement of the generic setuptools path: the recipe states identity and
53
+ source and nothing else. Its documented pure-Python fallback is why the test
54
+ asserts `markupsafe._speedups` specifically; an extension that failed to build
55
+ would otherwise install and import cleanly.
56
+
57
+ **msgpack** — a Cython extension with no external library, so a failure here
58
+ would be about generated C rather than about linking. Needs only
59
+ `buildRequires: ["Cython==3.1.4"]`.
60
+
61
+ **PyYAML** — Cython generates `_yaml.c`, which links the
62
+ libyaml built by the native dependency system. It selects between its pure and
63
+ compiled implementations through its own `PYYAML_FORCE_CYTHON` and
64
+ `PYYAML_FORCE_LIBYAML` variables, which the recipe sets through `buildEnv` —
65
+ build configuration, which belongs in a recipe, as distinct from compiler
66
+ flags, which never do. The test asserts `yaml.CSafeLoader` parsed the document,
67
+ which only exists when the extension built.
68
+
69
+ **pydantic-core** — Rust, cross compiled with a rebuilt `std` and wasm
70
+ exceptions. Two versions, because pydantic pins its core exactly and resolving
71
+ that pin is part of what the index has to get right.
72
+
73
+ **simplejson** — recorded for what it shows about policy rather than about
74
+ building. It publishes both C-accelerated platform wheels and a pure
75
+ `py3-none-any` fallback, and the resolver takes the pure one. That is correct:
76
+ a pure wheel is portable and carries no ABI risk. The limitation is real and
77
+ worth stating plainly — **a package that publishes a pure fallback cannot
78
+ currently be accelerated by supplying a compiled wheel, because the pure wheel
79
+ always wins**. Changing that is a policy decision, not an oversight, so no
80
+ compiled simplejson wheel is shipped.
81
+
82
+ **siphash24** — the densest case, and the only one that needed a patch. Three
83
+ things at once: a frontend that is not setuptools (meson-python, driven with a
84
+ cross file generated from the ABI contract); two dependencies resolved through
85
+ Meson `wrap` files, which Meson satisfies by cloning from git *during the
86
+ build* — disabled here, with the subprojects supplied instead as pinned,
87
+ hash-verified archives recorded in the wheel's provenance; and a patch, because
88
+ one of those subprojects builds its library with `both_libraries`, which always
89
+ produces a shared library. `ld.wasm` has none, and Meson refuses at configure
90
+ time rather than at link time, so the project was unconfigurable until the
91
+ library was made static. The test asserts the first published SipHash-2-4
92
+ reference vector, which comes from outside this repository.
93
+
94
+ **numpy** — the case this pipeline was built toward, and deliberately the last
95
+ one attempted, so that it drove none of the design. It needed **no patch** and
96
+ no mechanism that did not already exist for smaller packages: Meson through
97
+ meson-python, a cross file generated from the ABI contract, pinned build tools,
98
+ and its own documented `allow-noblas` switch for a platform with no BLAS. It
99
+ built on the first attempt.
100
+
101
+ Its assertions are numerical rather than structural. NumPy has no pure-Python
102
+ fallback to hide behind, but a miscompiled one imports perfectly well and
103
+ computes wrong answers, so each check has an exactly known result: `a @ a.T`,
104
+ an inverse multiplied back to the identity, a round-trip FFT, a reduction.
105
+ `long double` deserves naming — Meson cannot run a program on the target to
106
+ learn its layout, so the value is derived from the pinned compiler's own
107
+ `__LDBL_MANT_DIG__` (113, little-endian, so `IEEE_QUAD_LE`) and the built
108
+ extension reports the matching 16-byte itemsize. Getting that wrong would have
109
+ been silently wrong arithmetic rather than a build failure.
110
+
111
+ NumPy's *own* test suite runs inside the runtime, which is the strongest
112
+ evidence in this document because none of it was written here:
113
+
114
+ | module | result |
115
+ | --- | --- |
116
+ | `numpy._core.tests.test_umath` | 4388 passed, 362 skipped, 2 xfailed, 5 xpassed |
117
+ | `numpy.linalg.tests.test_linalg` | 416 passed, 21 skipped, 2 xfailed |
118
+ | `numpy.fft.tests.test_pocketfft` | 144 passed, 4 skipped |
119
+
120
+ 4,948 upstream tests pass and none fails. Reproduce with `pip install numpy
121
+ pytest hypothesis` in a container, then:
122
+
123
+ ```bash
124
+ python3 -m pytest --pyargs numpy._core.tests.test_umath -q --capture=no
125
+ ```
126
+
127
+ `--capture=no` is required, and the reason is a runtime limitation rather than
128
+ anything to do with NumPy: pytest's default output capture opens a temporary
129
+ file descriptor and reading it back fails with `OSError: [Errno 8] Bad file
130
+ descriptor`, so *no* tests run at all. It belongs with the other pre-existing
131
+ process- and descriptor-level gaps recorded in the project state, not with
132
+ package support.
133
+
134
+ **psutil** — attempted deliberately as a negative case, and the failure is more
135
+ useful than the success would have been.
136
+
137
+ ## The psutil finding
138
+
139
+ Command: `python3 scripts/build_extension.py recipes/psutil/recipe.json`
140
+ Source: `psutil-6.1.0.tar.gz`, sha256 `353815f5…c7a`
141
+
142
+ Earliest unsupported assumption, quoted from the build:
143
+
144
+ ```
145
+ emcc … -DPSUTIL_POSIX=1 -DPSUTIL_OSX=1 … -c psutil/_psutil_common.c
146
+ psutil/_psutil_common.h:131:14: fatal error: 'mach/mach_time.h' file not found
147
+ ```
148
+
149
+ `-DPSUTIL_OSX=1` is the whole finding. `sysconfig` has been pointed at the
150
+ target throughout, so the compiler, its flags and the extension suffix are all
151
+ correct — but psutil's `setup.py` selects its platform implementation from
152
+ `sys.platform`, which reports the *running* interpreter's operating system.
153
+ Under a cross build that is the build machine, so on macOS it compiled the
154
+ Darwin backend and asked for a Mach header.
155
+
156
+ This is assumption 7 in `cross-build.md` in its most common concrete form, and
157
+ it generalises well beyond psutil: any package that branches on `sys.platform`
158
+ in its build will describe the build machine. It is worth recording that
159
+ `_PYTHON_HOST_PLATFORM` does **not** fix this — that variable changes
160
+ `sysconfig.get_platform()`, not `sys.platform`.
161
+
162
+ psutil is nevertheless classified `blocked-platform-capability` rather than
163
+ `supported-patched`. A patch could make it select a POSIX backend, but psutil
164
+ reads process and system tables through syscalls Emscripten does not implement;
165
+ the capability is absent from the platform, not from the build. Patching it
166
+ would convert a clear build failure into a runtime one, which is the trade this
167
+ project refuses everywhere else.
168
+
169
+ ## Cohorts
170
+
171
+ All eight cohorts now have a real package. Cohorts 6 (a package requiring a
172
+ visible cross-compilation patch) and 7 (a PEP 517 backend that is not
173
+ setuptools) are both covered by siphash24; cohort 8 by NumPy.
174
+
175
+ ## Packages with no recipe
176
+
177
+ Everything above has a recipe checked in, because each was measured
178
+ deliberately. Most packages need none: `pip install <anything>` will generate
179
+ one, build the wheel and install it, when a builder is enabled — see
180
+ `build-on-miss.md`. `ujson` is the worked example, built with no recipe, no
181
+ patch and no hand-pinned tool:
182
+
183
+ ```
184
+ $ pip install ujson
185
+ Collecting ujson
186
+ Building ujson from source (no wheel for this runtime yet)
187
+ Built ujson
188
+ Successfully installed ujson-6.0.0
189
+ ```
190
+
191
+ A package that cannot be built this way still lands in the vocabulary above
192
+ rather than as an opaque failure: a backend with no adapter is
193
+ `blocked-toolchain` and says which backend, and a package with no source
194
+ distribution says that instead.
195
+
196
+ ## Reproducing
197
+
198
+ ```bash
199
+ make -C python-runtime native-deps
200
+ python3 python-runtime/scripts/build_extension.py python-runtime/recipes/<name>/recipe.json
201
+ python3 python-runtime/scripts/build_index.py
202
+ npx vitest run test/python-runtime/package-cohorts.test.ts
203
+ ```
204
+
205
+ Every wheel above rebuilds byte-identically; the digests in this table are
206
+ therefore checkable rather than decorative.
@@ -0,0 +1,354 @@
1
+ # Cross-building native extensions
2
+
3
+ The pipeline that turns a package's own source into a SandboxedJs wheel, and
4
+ the list of things it still assumes about a package — so each of them can be
5
+ removed deliberately rather than discovered by a failure.
6
+
7
+ ## What it does
8
+
9
+ `python-runtime/scripts/build_extension.py` takes a recipe and produces a
10
+ wheel tagged `cp313-cp313-emscripten_5_0_6_wasm32` with a `Build-ABI` line
11
+ naming the `abiId` computed from `abi/extension-abi.json`. Three recipe kinds
12
+ exist:
13
+
14
+ | kind | source of truth for *what* to build | covers |
15
+ | --- | --- | --- |
16
+ | `setuptools` | the package's own `setup.py` / `pyproject.toml` | plain C extensions, Cython, anything setuptools builds |
17
+ | `pep517` | the package's declared build backend and hooks | pyproject-only projects, in-tree backends |
18
+ | `meson` | the project's `meson.build`, via meson-python | Meson projects, including NumPy |
19
+ | `pyo3` | the crate's `Cargo.toml` | Rust extensions |
20
+ | `c-extension` | the recipe's own `sources` list | probes only — see the assumptions below |
21
+
22
+ `kind` selects a backend explicitly; nothing is inferred from which files are
23
+ present. A project with both a `setup.py` and a `pyproject.toml` can be built
24
+ either way, and guessing would silently change how a package is built the day
25
+ upstream adds a file. Recipes are schema-validated before anything is fetched
26
+ or compiled, so a malformed one costs a second rather than a Rust build.
27
+
28
+ In every case the *how* — compiler, target, PIC, threads, exceptions,
29
+ extension suffix, wheel tag — comes from `abi/extension-abi.json` and only
30
+ from there. A recipe cannot contribute a compile or link flag. This is not
31
+ tidiness: a recipe that could set its own `-pthread` could produce an artifact
32
+ that links against a main module with a different memory model and then
33
+ corrupts memory, and the mismatch would be invisible in the wheel.
34
+
35
+ ## How the setuptools path works
36
+
37
+ `sysconfig` loads its table of build variables from a module named by the
38
+ `_PYTHON_SYSCONFIGDATA_NAME` environment variable, and
39
+ `distutils.command.build_ext` — which setuptools vendors — takes the compiler,
40
+ its flags and the extension suffix from that table. So the package's own
41
+ `setup.py build` runs on the *build machine's* interpreter with `sysconfig`
42
+ answering for the *target*. This is the mechanism CPython uses for its own
43
+ cross builds and the one `crossenv` automates; nothing about it is specific to
44
+ this project.
45
+
46
+ The CPython cross build already writes the target's table
47
+ (`_sysconfigdata__emscripten_wasm32-emscripten.py`). `scripts/crossenv.py`
48
+ copies it and overrides exactly three groups of values, all from the ABI
49
+ contract:
50
+
51
+ * `INCLUDEPY` / `CONFINCLUDEPY` — the shipped table names
52
+ `$prefix/include/python3.13`, where the headers would be if the interpreter
53
+ had been installed. It has not been; they are in the source tree.
54
+ * `LDSHARED` / `BLDSHARED` / `LDCXXSHARED` — the shipped link line has no
55
+ `-sSIDE_MODULE=1`, because CPython adds it in a rule outside `LDSHARED`.
56
+ Without it the artifact is a complete program and `dlopen` cannot load it.
57
+ * `CCSHARED` — the ABI's `-pthread -fwasm-exceptions -fPIC`, which must match
58
+ the main module exactly. `CCSHARED` rather than `CFLAGS` so the flags apply
59
+ to extension objects and not to host tools a package builds during setup.
60
+
61
+ Cython needs no support at all: `cythonize()` runs inside the package's
62
+ `setup.py` on the build machine and hands generated C to the same `build_ext`.
63
+ The only difference between the plain and Cython fixtures is a pinned
64
+ `buildRequires`.
65
+
66
+ ## Validation
67
+
68
+ Three checks, in order of how misleading their failure would otherwise be:
69
+
70
+ 1. `verify_toolchain` refuses an Emscripten other than the pinned one. A side
71
+ module built by a different Emscripten links and then fails at run time.
72
+ 2. `verify_side_module` checks the artifact is a WebAssembly binary, has a
73
+ `dylink` section, and exports `PyInit_<last component of the module name>`.
74
+ A missing dylink section fails at `dlopen` with a message about the file; a
75
+ missing init symbol fails with "dynamic module does not define module
76
+ export function", which reads like a source error and is in fact a link
77
+ setting.
78
+ 3. `test/python-runtime/extensions.test.ts` installs each wheel from an HTTP
79
+ index with the container's own `pip` and imports it with CPython's ordinary
80
+ import machinery. This is the only check that proves the ABI, because the
81
+ first two can pass on a module compiled against the wrong headers.
82
+
83
+ ## Package-specific assumptions, and how to remove each
84
+
85
+ Each of these is a place the pipeline knows, or requires a recipe to state,
86
+ something a package already states about itself.
87
+
88
+ **1. Target dependencies are not resolved.** *Resolved.* Native target
89
+ dependencies are declared by `dependency.json` manifests under
90
+ `python-runtime/native-deps/` (real libraries) and
91
+ `python-runtime/fixtures/native/` (test ones), built by
92
+ `scripts/native_deps.py` into the profile's sysroot with the ABI's own flags,
93
+ and reached by extensions through the generated cross configuration — so
94
+ `libraries=["yaml"]` in a package's `setup.py` resolves with no path or flag
95
+ in the recipe. A recipe names what it needs in `nativeRequires`, which is a
96
+ separate namespace from `requires` (Python) and `buildRequires` (build
97
+ machine) so that a build-machine library can never satisfy a target link.
98
+
99
+ Reuse is decided by a stamp recording the source digest, profile, exact
100
+ compiler flags and the digest of every installed output — not by whether a
101
+ file exists, which is what let an archive built for one profile satisfy
102
+ another. What remains: only `make` and `configure` build systems are
103
+ supported, and CMake and Meson libraries will need a third; and the libraries
104
+ CPython itself links are still built by the older hand-written
105
+ `scripts/build_dependencies.py` rather than through manifests.
106
+
107
+ **2. Wheel metadata is synthesized from the recipe on the `pyo3` path only.**
108
+ *Resolved for `setuptools`.* That path now runs the package's own `dist_info`
109
+ under the cross environment and ships the `METADATA` it generates, so markers,
110
+ extras and `Requires-Python` are upstream's and a recipe has no `requires`
111
+ field to drift from them. Each wheel records which of the two it used in its
112
+ `WHEEL` file as `Metadata-Source: package|recipe`, so the remaining cases are
113
+ visible in the artifacts rather than only here.
114
+
115
+ A recipe may still *remove* a requirement through `dependencyOverrides`, which
116
+ must carry a `because`, must name a requirement the package actually has, and
117
+ is recorded in the built wheel as `Build-Dropped-Requirement`. *Remove the
118
+ remaining `recipe` cases by:* reading `[project]` from the crate's
119
+ `pyproject.toml` for maturin-built extensions, or by driving maturin itself
120
+ under the cross environment.
121
+
122
+ **3. `pythonRoots` exists only because the `pyo3` path has no `build_py`.** It
123
+ is a hand-written statement of which directory holds the package's Python
124
+ half; the pydantic-core recipe encodes maturin's `python/` convention. *Remove
125
+ by:* reading `[tool.maturin] python-source` from the crate's `pyproject.toml`,
126
+ or by driving maturin itself under the cross environment.
127
+
128
+ **4. The `pyo3` path declares module layout.** `crateName` and `package` state
129
+ where the artifact must sit for `from ._x import ...` to work. The setuptools
130
+ path derives the same thing from the build tree. *Remove by:* same as 3.
131
+
132
+ **5. The `pyo3` recipe hard-codes an unpacked source path.**
133
+ `recipes/pydantic-core/recipe.json` names
134
+ `../../out/ports/pydantic_core-2.46.5`, duplicating the version that appears
135
+ three other times in the same file. *Remove by:* deriving the crate path from
136
+ `source.unpackTo` and the archive's own stem, which `fetch_source` already
137
+ computes.
138
+
139
+ **6. `kind: "c-extension"` is a transcription.** The recipe lists sources the
140
+ package's build system already lists, so the two can disagree silently. It is
141
+ retained only for `fixtures/sbx_c_probe`, which deliberately has no build
142
+ system. *Remove by:* giving the fixture a `setup.py` and deleting the kind and
143
+ `compile_c_module` with it.
144
+
145
+ **7. `setup.py` is assumed to be cross-safe.** *Mitigated, not removable.* A
146
+ `setup.py` that probes the build machine (compiles a test program, runs the
147
+ extension it is building, reads `platform.machine()`) will describe the build
148
+ machine. Nothing detects this; the build succeeds and produces a wrong
149
+ artifact. It is a property of the package, so it cannot be fixed generically —
150
+ but it now has one designated place to be fixed in. A recipe declares
151
+ `patches: {"dir": …}`, and the patch set states the package, the version, the
152
+ digest of the source tree it was written against, and per patch a reason, the
153
+ target fact being substituted, and the test that proves it. Patches apply to a
154
+ staged copy with `git apply` at exact context; a drifted source is refused
155
+ rather than fuzzily patched. `fixtures/sbx_patched_probe` is exactly this
156
+ failure — `platform.machine()` declaring 64-bit pointers for a 32-bit target —
157
+ and its patch is the worked example.
158
+
159
+ **8. Build tools come from PyPI at first build.** *Resolved.*
160
+ `build-tools.lock` pins every build tool — including `setuptools` itself, which
161
+ was previously taken from whatever the build machine had — and records the
162
+ sha256 of every distribution published for that version, so one lock serves any
163
+ build machine. Installation uses `--require-hashes`, which also refuses a tool
164
+ that grows an unpinned dependency. Regenerate with
165
+ `scripts/lock_build_tools.py`; a build reads the lock and never refreshes it.
166
+
167
+ **9. One ABI, no matrix.** `abiId` covers exactly one configuration; there is
168
+ no way to build the same recipe for a second profile. *Remove by:* taking the
169
+ profile from the command line into the wheel's local version segment, once a
170
+ second profile exists to want it.
171
+
172
+ **10. `--build-lib` is assumed to be honoured.** Projects with a custom
173
+ `build` command that ignores it would stage nowhere the collector looks; the
174
+ build "succeeds" and produces no modules. This is caught — the empty-module
175
+ case raises — but only after the fact.
176
+
177
+ ## Native target dependencies
178
+
179
+ Three kinds of dependency exist and are deliberately kept in three namespaces,
180
+ because a single list would let a build machine's library satisfy a target
181
+ link — producing a module that links cleanly and then faults:
182
+
183
+ | recipe field | what it is | where it goes |
184
+ | --- | --- | --- |
185
+ | `requires` | Python runtime dependency | installed in the container, imported |
186
+ | `buildRequires` | build-machine tool (Cython, setuptools_scm) | a per-recipe venv on this machine |
187
+ | `nativeRequires` | C library cross compiled for wasm32 | `out/sysroot-<profile>`, linked into the side module |
188
+
189
+ A native dependency is declared by a `dependency.json`:
190
+
191
+ ```json
192
+ {
193
+ "name": "yaml", "version": "0.2.5",
194
+ "license": "MIT", "licenseFiles": ["License"],
195
+ "provenance": { "url": "…", "sha256": "…", "unpackDir": "yaml-0.2.5" },
196
+ "profiles": ["dynamic"], "dependsOn": [],
197
+ "build": { "system": "configure", "configureArgs": ["--disable-shared"] },
198
+ "outputs": { "libraries": ["libyaml.a"], "headers": ["yaml.h"] }
199
+ }
200
+ ```
201
+
202
+ `name` must be the name a linker is given (`-lyaml`), not the project's title:
203
+ a manifest called `libyaml` would build the right library and leave the
204
+ extension unable to ask for it.
205
+
206
+ `outputs` is checked after the build and recorded in the stamp. A build system
207
+ that quietly produces nothing — a `configure` that disabled the library, a
208
+ `make` that built only tools — otherwise looks like a success until an
209
+ extension fails to link with a message about a missing symbol.
210
+
211
+ Provenance is either `local` (source in this repository, still digested so an
212
+ edit rebuilds) or a pinned `url` and `sha256`, verified before anything is
213
+ unpacked. These are kept out of `sources.lock` on purpose: that file pins what
214
+ the *interpreter* links, and a library that exists for packages should not
215
+ become something the runtime build has to download.
216
+
217
+ `build.host` exists because autotools packages vendor a `config.sub` frozen at
218
+ their release date. libyaml 0.2.5's is from 2018 and rejects
219
+ `wasm32-unknown-emscripten` outright; it accepts `wasm32-unknown-none`, which
220
+ is sufficient because `--host` only tells `configure` that this is a cross
221
+ build — `emconfigure` has already supplied the tools. That is a property of
222
+ the package, so it is declared per dependency rather than worked around for
223
+ everybody.
224
+
225
+ ## Reproducible inputs, patches and provenance
226
+
227
+ Builds never run in the source tree. The project is copied to a staging
228
+ directory first, then patched, then built. Two failures follow from doing
229
+ otherwise: a patch applied in place leaves the checkout modified, so a second
230
+ build starts from different source and the patch no longer applies; and a build
231
+ leaves generated files behind — Cython's `.c`, `.egg-info` — which then become
232
+ inputs to the next one.
233
+
234
+ Every input is pinned and verified before use:
235
+
236
+ | input | pinned by | verified by |
237
+ | --- | --- | --- |
238
+ | interpreter and its libraries | `sources.lock` | sha256 before unpacking |
239
+ | build tools (setuptools, Cython) | `build-tools.lock` | `pip --require-hashes` |
240
+ | native target libraries | `dependency.json` provenance | sha256 before unpacking |
241
+ | package source | recipe `source`, or in-repo | sha256, or tree digest |
242
+ | patches | `patches.json` `sourceDigest` | exact-context `git apply` |
243
+
244
+ Wheels are byte-reproducible: every zip member is stamped with the zip epoch
245
+ rather than the build time and RECORD is sorted, so two builds from the same
246
+ inputs produce identical files. This is stronger than the "documented timestamp
247
+ normalization" the task allowed, and it is what makes "did anything actually
248
+ change?" answerable by comparing digests. Confirmed by rebuilding every wheel
249
+ and diffing digests, and by `test/python-runtime/reproducible-builds.test.ts`.
250
+
251
+ Each wheel carries `dist-info/sandboxedjs-provenance.json`, which answers, for
252
+ an artifact held by someone without this repository: which source tree digest
253
+ produced it, which build tools and native libraries went into it, which ABI id
254
+ and recipe revision were used, and which patches were applied. The index
255
+ summarises those fields so they can be compared across wheels without
256
+ downloading any.
257
+
258
+ ## Build backends
259
+
260
+ `scripts/backends.py` is the only place a package's build system is known
261
+ about. The orchestrator verifies inputs, selects a backend by the name the
262
+ recipe stated, hands it a `BuildRequest`, and packages the `BuildResult` that
263
+ comes back. It decides nothing else — which is what keeps a package-name check
264
+ from appearing in it, since once an orchestrator is already making build
265
+ decisions, one more looks harmless.
266
+
267
+ A backend receives verified staged source, the locked build-tool interpreter,
268
+ the cross environment, the declared native dependencies, a staging directory
269
+ and a read-only view of the ABI. It returns a wheel-layout tree, the package's
270
+ own `.dist-info`, and evidence recorded in the wheel's provenance. It cannot
271
+ contribute a compiler flag, and the orchestrator digests the ABI contract
272
+ before and after the build — a backend that mutated it would produce a
273
+ correctly-tagged wheel built for a different ABI.
274
+
275
+ The `pep517` backend calls the package's declared hooks the way a frontend
276
+ does, including resolving an in-tree backend through `backend-path`. It does
277
+ **not** use `python -m build`, whose isolation installs whatever versions
278
+ upstream publishes on the day of the build — the unpinned input the rest of
279
+ this pipeline refuses. The environment is assembled from `build-tools.lock`,
280
+ and a build requirement absent from the lock is refused rather than fetched.
281
+ The wheel the hooks produce is unpacked rather than shipped: it carries
282
+ whatever tag the backend chose, and this pipeline tags a wheel for what it
283
+ actually is.
284
+
285
+ Build tools run on an interpreter whose feature version matches the target's.
286
+ Most of the cross build does not care, because `sysconfig` answers for the
287
+ target either way — but `bdist_wheel` composes a tag from the *running*
288
+ interpreter's version and the *target's* ABI tag. Building for 3.13 from 3.14
289
+ yields `('cp314', 'cp313', 'emscripten_5_0_6_wasm32')`, which fails an
290
+ assertion inside setuptools rather than anywhere that names the cause.
291
+
292
+ ## Meson
293
+
294
+ `kind: "meson"` drives a project through meson-python's PEP 517 hooks. It is
295
+ its own backend rather than a `pep517` recipe with extra settings because a
296
+ cross file is backend mechanics, not a package quirk — a recipe that passed its
297
+ own `--cross-file` could describe a different target than the wheel is tagged
298
+ for.
299
+
300
+ Meson does not read `sysconfig`; it is told about a target by a file. So the
301
+ same facts the cross table states are stated again in Meson's format, generated
302
+ from `abi/extension-abi.json` by `write_meson_cross_file`. Three settings are
303
+ supplied by the backend because they are facts about the target, not choices:
304
+
305
+ - `needs_exe_wrapper = true`, or Meson believes it can run what it builds and
306
+ every compile-and-run check silently tests the build machine.
307
+ - `--wrap-mode=nodownload`. Meson otherwise resolves a missing subproject by
308
+ cloning it mid-build — an unpinned input arriving over the network at the one
309
+ moment nothing is watching.
310
+ - `-Ddefault_library=static`. There is no such thing as a shared library here;
311
+ `ld.wasm` refuses, and Meson reports it at configure time as a message about
312
+ the linker rather than about the default that reached it.
313
+ - `longdouble_format`, derived by asking the pinned compiler for its
314
+ `__LDBL_MANT_DIG__` and byte order rather than being written down. Meson
315
+ cannot run a program on the target to find out, and a project that inspects
316
+ float layouts — NumPy does — will not configure without it. Deriving it
317
+ matters more than the convenience: a hand-written value that disagreed with
318
+ the toolchain would produce a library that builds, imports, and computes
319
+ wrong answers.
320
+
321
+ A project whose subprojects were previously downloaded supplies them through
322
+ `vendoredSources`: hash-verified archives unpacked at declared paths inside the
323
+ staged tree, recorded in the wheel's provenance because they are compiled into
324
+ it. They cannot escape the staged tree.
325
+
326
+ ### Two things that were failures first
327
+
328
+ **Split headers.** A cross build leaves `Include/` in the source tree and the
329
+ generated `pyconfig.h` in the build directory; an *installed* interpreter has
330
+ them in one place, and tools assume the installed shape. `Python.h` includes
331
+ `"pyconfig.h"` in quotes, so the compiler looks beside `Python.h` and then walks
332
+ the `-I` list — and finds the *build machine's* `pyconfig.h` from whichever host
333
+ include directory a tool added. The build then fails with `LONG_BIT definition
334
+ appears wrong for platform`, which reads like a broken toolchain and is in fact
335
+ a 64-bit header describing a 32-bit target. Meson adds such a directory;
336
+ setuptools happens not to. Both now compile against one staged directory
337
+ holding every target header.
338
+
339
+ **Paths in compiled output.** This one recurred three times before the rule was
340
+ clear: *nothing a compiler sees may live at a per-build path.*
341
+
342
+ First the staged include directory, created inside the temporary build
343
+ directory — the compile line carries `-g`, so the varying path reached debug
344
+ information. Then Meson's own `.mesonpy-<random>` build directory, which also
345
+ has to be pinned together with the cross file, because Meson caches the cross
346
+ file's path inside the build directory and a stale one fails on the next run
347
+ with a `FileNotFoundError` about a file nobody asked for.
348
+
349
+ Then the staged *source* directory, which was the subtlest: Cython writes the
350
+ `.pyx` path into its generated C so a traceback can name a line. NumPy exposed
351
+ it — 869 of the 874 files in its wheel matched between builds, and the five
352
+ that differed were its Cython-generated modules, all at identical sizes. The
353
+ staged source is now a stable path too, and `reproducible-builds.test.ts`
354
+ checks that the generated cross configuration carries no temporary path.