sandboxedjs 0.2.11 → 0.2.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +16 -9
- package/assets/logo.png +0 -0
- package/bin/sandboxedjs-egress.mjs +25 -10
- package/dist/index.cjs +1 -1
- package/dist/index.js +1 -1
- package/dist/service-worker.js +3 -2
- package/docs/agent/COMMANDS.md +85 -0
- package/docs/agent/DECISION-TREE.md +84 -0
- package/docs/agent/INVARIANTS.md +40 -0
- package/docs/agent/LAUNCH-PROMPT.md +37 -0
- package/docs/agent/LOOP.md +84 -0
- package/docs/agent/README.md +77 -0
- package/docs/agent/ROADMAP.md +37 -0
- package/docs/agent/STATE.md +93 -0
- package/docs/agent/tasks/00-verify-inherited-work.md +36 -0
- package/docs/agent/tasks/01-authoritative-metadata.md +34 -0
- package/docs/agent/tasks/02-native-dependencies.md +35 -0
- package/docs/agent/tasks/03-reproducible-inputs.md +27 -0
- package/docs/agent/tasks/04-build-frontends.md +27 -0
- package/docs/agent/tasks/05-registry-integration.md +27 -0
- package/docs/agent/tasks/06-package-cohorts.md +42 -0
- package/docs/agent/tasks/07-build-on-miss-boundary.md +31 -0
- package/docs/browser-runtime-architecture.md +142 -0
- package/docs/compatibility-implementation-plan.md +98 -0
- package/docs/developer-tool-packs.md +134 -0
- package/docs/frontend-automation.md +49 -0
- package/docs/fullstack-deployment.md +163 -0
- package/docs/handoff.md +275 -0
- package/docs/original-x64.md +39 -0
- package/docs/platform-hardening.md +49 -0
- package/docs/python/abi.md +97 -0
- package/docs/python/architecture.md +94 -0
- package/docs/python/baseline-inventory.md +54 -0
- package/docs/python/build-on-miss.md +198 -0
- package/docs/python/compatibility.md +206 -0
- package/docs/python/cross-build.md +354 -0
- package/docs/python/extensions.md +282 -0
- package/docs/python/release-gates.md +46 -0
- package/docs/python/virtual-sockets-plan.md +331 -0
- package/docs/runtime-lifecycle-fixes.md +39 -0
- package/docs/server-previews.md +268 -0
- package/docs/virtual-browser.md +120 -0
- package/package.json +5 -3
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
# Execution state
|
|
2
|
+
|
|
3
|
+
Keep under roughly 120 lines. Replace stale detail instead of appending.
|
|
4
|
+
|
|
5
|
+
## Status
|
|
6
|
+
|
|
7
|
+
- Phase: `complete`
|
|
8
|
+
- Blocker: none
|
|
9
|
+
- User decision required: none outstanding
|
|
10
|
+
|
|
11
|
+
## Repository layout (restructured)
|
|
12
|
+
|
|
13
|
+
src/
|
|
14
|
+
kernel/ fs/ net/ shell/ the linux-mimicking core
|
|
15
|
+
node/ module shims, commonjs engine, esm transform
|
|
16
|
+
python/ cpython, syscalls, pip, resolver, worker, abi, local-builder
|
|
17
|
+
worker/ pods, sync channel, worker host
|
|
18
|
+
tools/ esbuild, rolldown, ffmpeg hosts
|
|
19
|
+
container/ pkg/ agent/ preview/ hosting/
|
|
20
|
+
docs/ every markdown file: agent/, python/, handoff.md
|
|
21
|
+
python-runtime/ the Python build pipeline (Makefile, scripts, recipes)
|
|
22
|
+
|
|
23
|
+
`src/runtime/` is gone. Moves were done with `git mv` and every relative import
|
|
24
|
+
recomputed programmatically. Four references were strings rather than import
|
|
25
|
+
specifiers -- two `dist/` paths whose depth changed, a `/src/runtime/python/`
|
|
26
|
+
check in config.ts, and a test allowlist -- and those caused 79 failures until
|
|
27
|
+
found. There was no dead code to remove: 2 of 130 source files are unreferenced
|
|
28
|
+
and both are ambient `.d.ts` files tsconfig loads implicitly.
|
|
29
|
+
|
|
30
|
+
## Packages: nothing to pre-build any more
|
|
31
|
+
|
|
32
|
+
`pip install <anything>` now builds a wheel when the index has none.
|
|
33
|
+
|
|
34
|
+
- `scripts/auto_recipe.py` writes a recipe from what PyPI and the source
|
|
35
|
+
already state: identity, verified sdist, and the declared PEP 517 backend.
|
|
36
|
+
Backends with no adapter are refused by name. Build requirements the lock has
|
|
37
|
+
never seen are pinned at that moment rather than dropped.
|
|
38
|
+
- `src/python/local-builder.ts` runs the pipeline; `bin/sandboxedjs-build-wheels.mjs`
|
|
39
|
+
serves the index and builds on request, for browsers that have no compiler.
|
|
40
|
+
- Opt-in, defaulting on only in Node from a checkout. Submit-and-poll, because
|
|
41
|
+
a connection held open for a minutes-long build reads as an unreachable
|
|
42
|
+
service.
|
|
43
|
+
- Verified: `pip install ujson` with no recipe -> generated, built, indexed,
|
|
44
|
+
installed, imported, both locally and through the service.
|
|
45
|
+
|
|
46
|
+
## Two real defects found and fixed
|
|
47
|
+
|
|
48
|
+
- The browser's default wheel index was filtered to the wheels whose bytes are
|
|
49
|
+
inlined in the bundle, so every other built wheel was invisible -- `pip
|
|
50
|
+
install numpy` reported no usable distribution while numpy sat unreferenced
|
|
51
|
+
beside the interpreter. The index now lists all built wheels, served from
|
|
52
|
+
next to the interpreter, with the inlined bytes as an overlay keyed by
|
|
53
|
+
filename.
|
|
54
|
+
- The post-build index read went through pip's memoizing cache and returned the
|
|
55
|
+
pre-build copy, so a package was built twice.
|
|
56
|
+
|
|
57
|
+
## Pre-existing failures, verified not caused by this work
|
|
58
|
+
|
|
59
|
+
11 tests, reproduced identically in a clean worktree at HEAD: `python-syscalls`
|
|
60
|
+
(5), `cpython` (2), `python-abi/baseline` (3), `python-runtime/asyncio` (1).
|
|
61
|
+
Also: pytest's default output capture fails in the container
|
|
62
|
+
(`OSError: [Errno 8] Bad file descriptor`), so upstream suites need
|
|
63
|
+
`--capture=no`.
|
|
64
|
+
|
|
65
|
+
## Unverified
|
|
66
|
+
|
|
67
|
+
`import numpy` in a browser. The embedded browser used for testing blocks
|
|
68
|
+
nested module workers loaded over HTTP, which the runtime needs to start Python
|
|
69
|
+
at all -- `python3 --version` times out there too, and so does a trivial nested
|
|
70
|
+
worker. `pip install numpy` was verified in that browser and returns 0.
|
|
71
|
+
|
|
72
|
+
## Last evidence
|
|
73
|
+
|
|
74
|
+
`npx vitest run`: 610 passed, 24 skipped, 12 failed (the 11 above plus one
|
|
75
|
+
PyPI-network flake that passes in isolation). `npm run typecheck`: pass.
|
|
76
|
+
`npm run build`: pass. All 16 wheels byte-identical across rebuilds.
|
|
77
|
+
|
|
78
|
+
## Compact checkpoint template## Compact checkpoint template## Compact checkpoint template## Compact checkpoint template## Compact checkpoint template## Compact checkpoint template## Compact checkpoint template## Compact checkpoint template## Compact checkpoint template
|
|
79
|
+
|
|
80
|
+
Replace the sections above after each meaningful loop:
|
|
81
|
+
|
|
82
|
+
```text
|
|
83
|
+
Phase: <name>
|
|
84
|
+
Current task: <path>
|
|
85
|
+
Last completed task: <path or none>
|
|
86
|
+
Blocker: <one sentence or none>
|
|
87
|
+
User decision required: <yes/no and exact decision>
|
|
88
|
+
Last evidence:
|
|
89
|
+
- <command>: <pass/fail and decisive result>
|
|
90
|
+
Next action: <one concrete action>
|
|
91
|
+
Files changed this loop: <paths>
|
|
92
|
+
```
|
|
93
|
+
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Task 00: verify inherited generic cross-build work
|
|
2
|
+
|
|
3
|
+
## Purpose
|
|
4
|
+
|
|
5
|
+
Establish that the staged setuptools/Cython work is real and reproducible
|
|
6
|
+
before building new layers on it. Do not expand scope in this task.
|
|
7
|
+
|
|
8
|
+
## Procedure
|
|
9
|
+
|
|
10
|
+
1. Inspect `git status --short`, `git diff --cached --check`, and the staged
|
|
11
|
+
diff limited to the files listed in `STATE.md`.
|
|
12
|
+
2. Confirm recipes cannot provide ABI compiler or linker flags.
|
|
13
|
+
3. Confirm generated artifacts are checked for WebAssembly, `dylink`, and the
|
|
14
|
+
derived `PyInit_*` symbol.
|
|
15
|
+
4. Run the setuptools and Cython fixture builds. Rebuild the wheel index after
|
|
16
|
+
any wheel output changes.
|
|
17
|
+
5. Run `npm run build` only if runtime bundles required by the acceptance test
|
|
18
|
+
are absent or older than relevant inputs.
|
|
19
|
+
6. Run `npx vitest run test/python-runtime/extensions.test.ts`.
|
|
20
|
+
7. Run `npm run typecheck`.
|
|
21
|
+
|
|
22
|
+
## Exit gate
|
|
23
|
+
|
|
24
|
+
- Staged diff check passes.
|
|
25
|
+
- Setuptools C, Cython, and PyO3 wheels install and execute native behavior in
|
|
26
|
+
the owned runtime.
|
|
27
|
+
- Pydantic acceptance remains green.
|
|
28
|
+
- No test passed by weakening or skipping an ABI assertion.
|
|
29
|
+
- `STATE.md` contains exact passing commands and points to task 01.
|
|
30
|
+
|
|
31
|
+
## If it fails
|
|
32
|
+
|
|
33
|
+
Use `DECISION-TREE.md`. Fix only regressions within the inherited work. After
|
|
34
|
+
three failed evidence-driven attempts on the same class, prepare an escalation
|
|
35
|
+
packet instead of redesigning the ABI.
|
|
36
|
+
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Task 01: authoritative wheel metadata
|
|
2
|
+
|
|
3
|
+
## Problem
|
|
4
|
+
|
|
5
|
+
The current builder synthesizes `METADATA` from recipe fields. That duplicates
|
|
6
|
+
the package's dependency declarations and can create a wheel that installs but
|
|
7
|
+
resolves the wrong environment.
|
|
8
|
+
|
|
9
|
+
## Deliverable
|
|
10
|
+
|
|
11
|
+
Generate distribution metadata using the package's build metadata path under
|
|
12
|
+
the controlled build environment. Preserve upstream `Name`, `Version`,
|
|
13
|
+
`Requires-Dist`, extras, Python requirements, entry points, licenses, and other
|
|
14
|
+
install-relevant files. SandboxedJS should add only its wheel tag, generator,
|
|
15
|
+
and `Build-ABI` assertion.
|
|
16
|
+
|
|
17
|
+
Recipe constraints may explicitly override a target-inapplicable dependency,
|
|
18
|
+
but every override must be visible, justified, and tested.
|
|
19
|
+
|
|
20
|
+
## Required tests
|
|
21
|
+
|
|
22
|
+
- A fixture declares a conditional and an extra dependency upstream; neither
|
|
23
|
+
is copied into the recipe.
|
|
24
|
+
- The resulting wheel and generated `index.json` contain the authoritative
|
|
25
|
+
requirements.
|
|
26
|
+
- The resolver respects at least one marker/extras case.
|
|
27
|
+
- A disagreement between recipe identity and upstream identity fails loudly.
|
|
28
|
+
- Existing extension runtime tests remain green.
|
|
29
|
+
|
|
30
|
+
## Exit gate
|
|
31
|
+
|
|
32
|
+
No generic setuptools fixture needs handwritten `requires` metadata, and the
|
|
33
|
+
index is provably derived from the wheel produced from upstream metadata.
|
|
34
|
+
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# Task 02: native target dependencies and sysroot
|
|
2
|
+
|
|
3
|
+
## Problem
|
|
4
|
+
|
|
5
|
+
Extensions that name libraries or include their headers cannot currently
|
|
6
|
+
declare, build, locate, or link those target dependencies generically.
|
|
7
|
+
|
|
8
|
+
## Design requirement
|
|
9
|
+
|
|
10
|
+
Represent native target dependencies separately from Python runtime
|
|
11
|
+
dependencies and host build requirements. Each dependency must identify a
|
|
12
|
+
pinned source, hash, build recipe, output markers, license/provenance, and ABI
|
|
13
|
+
profile. The dynamic profile uses `out/sysroot-dynamic`; incompatible profiles
|
|
14
|
+
must never share object archives.
|
|
15
|
+
|
|
16
|
+
Add sysroot include and library search paths through generated cross-build
|
|
17
|
+
configuration, not ad hoc recipe flags.
|
|
18
|
+
|
|
19
|
+
## Reduced acceptance fixture
|
|
20
|
+
|
|
21
|
+
Before attempting a large external library, create a tiny static target library
|
|
22
|
+
with a header and one function. Build/install it through the dependency system,
|
|
23
|
+
then build a setuptools extension that declares the library normally and calls
|
|
24
|
+
it. The wheel must install and execute the function in the owned runtime.
|
|
25
|
+
|
|
26
|
+
After the reduced fixture passes, add one small real native library already
|
|
27
|
+
compatible with the project architecture. Do not begin with BLAS or NumPy.
|
|
28
|
+
|
|
29
|
+
## Exit gate
|
|
30
|
+
|
|
31
|
+
- Dependency closure is deterministic and cycle/error aware.
|
|
32
|
+
- Rebuilds are skipped only when validated output markers match inputs/profile.
|
|
33
|
+
- Missing dependency and wrong-profile reuse have focused negative tests.
|
|
34
|
+
- Reduced and real-library extensions import and run in owned CPython.
|
|
35
|
+
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# Task 03: reproducible inputs, patches, and provenance
|
|
2
|
+
|
|
3
|
+
## Deliverable
|
|
4
|
+
|
|
5
|
+
Unify source archives, build-tool wheels, native dependencies, and package
|
|
6
|
+
patches under verified acquisition. No release artifact may depend on an
|
|
7
|
+
unpinned `pip install setuptools wheel ...` result.
|
|
8
|
+
|
|
9
|
+
Define a visible patch layout. A patch records package/version, upstream source
|
|
10
|
+
hash, reason, target fact being substituted, and a test. Patch application must
|
|
11
|
+
fail on drift rather than fuzzily succeeding against another source version.
|
|
12
|
+
|
|
13
|
+
Record enough provenance in build output or adjacent index metadata to answer:
|
|
14
|
+
|
|
15
|
+
- which source hash produced this wheel;
|
|
16
|
+
- which build tools and target libraries were used;
|
|
17
|
+
- which ABI ID and recipe revision were used;
|
|
18
|
+
- which patches were applied.
|
|
19
|
+
|
|
20
|
+
## Exit gate
|
|
21
|
+
|
|
22
|
+
- Offline rebuild succeeds from a populated verified cache.
|
|
23
|
+
- A changed hash, unpinned build tool, or drifting patch fails before compile.
|
|
24
|
+
- Two identical clean builds produce equivalent wheel contents, allowing only
|
|
25
|
+
explicitly documented archive timestamp normalization until deterministic
|
|
26
|
+
ZIP output is implemented.
|
|
27
|
+
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# Task 04: build frontend interface
|
|
2
|
+
|
|
3
|
+
## Purpose
|
|
4
|
+
|
|
5
|
+
Turn the current kinds into a stable backend interface rather than accumulating
|
|
6
|
+
branches in `build_extension.py`.
|
|
7
|
+
|
|
8
|
+
## Required interface
|
|
9
|
+
|
|
10
|
+
Each backend receives verified source, host tools, target configuration,
|
|
11
|
+
declared native dependencies, staging directory, and ABI contract. It returns a
|
|
12
|
+
wheel tree or wheel plus build evidence. It cannot mutate the ABI.
|
|
13
|
+
|
|
14
|
+
Retain proven setuptools/Cython and PyO3 behavior. Add another frontend only
|
|
15
|
+
when a selected cohort package requires it; likely candidates include PEP 517
|
|
16
|
+
wheel hooks, Meson, or scikit-build/CMake. Do not implement all frontends
|
|
17
|
+
speculatively.
|
|
18
|
+
|
|
19
|
+
Keep package quirks in patches/recipes and backend mechanics in adapters.
|
|
20
|
+
|
|
21
|
+
## Exit gate
|
|
22
|
+
|
|
23
|
+
- Backend selection is explicit and schema-validated.
|
|
24
|
+
- The generic orchestrator has no package-name checks.
|
|
25
|
+
- Existing fixtures pass through the interface.
|
|
26
|
+
- One pyproject-only package runs through a real isolated PEP 517 path.
|
|
27
|
+
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# Task 05: registry and installer integration
|
|
2
|
+
|
|
3
|
+
## Deliverable
|
|
4
|
+
|
|
5
|
+
Produce an immutable wheel repository that the existing SandboxedJS `pip`
|
|
6
|
+
resolver can consume. Define schema versioning, atomic publication layout,
|
|
7
|
+
digest verification, ABI partitioning, and dependency metadata.
|
|
8
|
+
|
|
9
|
+
The browser runtime remains an installer, not a compiler. A missing native
|
|
10
|
+
wheel should return a structured, actionable classification that a host may use
|
|
11
|
+
later to request a build. It must not perform an implicit remote mutation.
|
|
12
|
+
|
|
13
|
+
## Required tests
|
|
14
|
+
|
|
15
|
+
- Correct ABI wheel wins over incompatible and host-native candidates.
|
|
16
|
+
- Wrong ABI and wrong digest are refused before installation.
|
|
17
|
+
- A failed graph leaves no partial site-packages state.
|
|
18
|
+
- A newly built test wheel can be indexed, served, installed, imported, and
|
|
19
|
+
exercised through the ordinary container command.
|
|
20
|
+
- Index publication cannot expose a wheel before its metadata and content are
|
|
21
|
+
complete.
|
|
22
|
+
|
|
23
|
+
## Exit gate
|
|
24
|
+
|
|
25
|
+
A clean consumer needs only a runtime plus index URL/object; no local source
|
|
26
|
+
tree or compiler is required for supported packages.
|
|
27
|
+
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# Task 06: representative package cohorts
|
|
2
|
+
|
|
3
|
+
## Purpose
|
|
4
|
+
|
|
5
|
+
Measure generality without allowing one package to define the architecture.
|
|
6
|
+
|
|
7
|
+
## Selection method
|
|
8
|
+
|
|
9
|
+
Choose small packages first, pin exact versions and hashes, and record why each
|
|
10
|
+
adds a new capability. Select at least one from each applicable cohort:
|
|
11
|
+
|
|
12
|
+
1. pure wheel control case;
|
|
13
|
+
2. setuptools C extension without external target libraries;
|
|
14
|
+
3. generated C/Cython extension;
|
|
15
|
+
4. Rust/PyO3 extension;
|
|
16
|
+
5. extension using a separately built native target library;
|
|
17
|
+
6. package requiring a visible cross-compilation patch;
|
|
18
|
+
7. pyproject/PEP 517 backend not already covered;
|
|
19
|
+
8. scientific-stack stress case, eventually including NumPy.
|
|
20
|
+
|
|
21
|
+
Do not count existing probes as real-package cohort evidence, but retain them as
|
|
22
|
+
reduced regression tests.
|
|
23
|
+
|
|
24
|
+
## Compatibility report
|
|
25
|
+
|
|
26
|
+
For every attempted package, record exactly one status:
|
|
27
|
+
|
|
28
|
+
- `supported-generic`;
|
|
29
|
+
- `supported-recipe`;
|
|
30
|
+
- `blocked-platform-capability`;
|
|
31
|
+
- `blocked-toolchain`;
|
|
32
|
+
- `not-yet-attempted`.
|
|
33
|
+
|
|
34
|
+
Record build command, source hash, wheel hash, ABI ID, test behavior, peak size
|
|
35
|
+
or memory when material, and the earliest unsupported assumption on failure.
|
|
36
|
+
|
|
37
|
+
## Exit gate
|
|
38
|
+
|
|
39
|
+
At least five independent real packages across four native/build shapes pass
|
|
40
|
+
runtime behavior tests, and failures produce reusable classifications instead
|
|
41
|
+
of package-specific branches.
|
|
42
|
+
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# Task 07: external build-on-miss boundary
|
|
2
|
+
|
|
3
|
+
## Purpose
|
|
4
|
+
|
|
5
|
+
Specify the seam that can later connect the installer to a trusted build
|
|
6
|
+
service without deploying that service or placing a toolchain in the browser.
|
|
7
|
+
|
|
8
|
+
## Deliverable
|
|
9
|
+
|
|
10
|
+
Define versioned request/status/result schemas. A request includes normalized
|
|
11
|
+
requirement, source/index evidence, target ABI ID, policy/recipe revision, and
|
|
12
|
+
an idempotency key. A result includes immutable wheel/index locations, hashes,
|
|
13
|
+
provenance, logs summary, compatibility classification, and terminal state.
|
|
14
|
+
|
|
15
|
+
States should cover queued, resolving, building host tools, building target
|
|
16
|
+
dependencies, building wheel, testing, published, unsupported, failed, and
|
|
17
|
+
cancelled. Repeated identical requests must converge on one cached artifact.
|
|
18
|
+
|
|
19
|
+
Security requirements include source allowlists/policy, resource limits,
|
|
20
|
+
network isolation, no untrusted artifact publication before runtime tests,
|
|
21
|
+
digest verification, and auditability.
|
|
22
|
+
|
|
23
|
+
## Exit gate
|
|
24
|
+
|
|
25
|
+
- Schemas and a local fake implementation are tested.
|
|
26
|
+
- Installer/host integration can report a buildable miss without automatically
|
|
27
|
+
triggering external work.
|
|
28
|
+
- Cached success, unsupported result, transient failure, cancellation, and ABI
|
|
29
|
+
rollover are covered.
|
|
30
|
+
- No cloud service is deployed without explicit user authorization.
|
|
31
|
+
|
|
@@ -0,0 +1,142 @@
|
|
|
1
|
+
# Browser runtime architecture
|
|
2
|
+
|
|
3
|
+
The current implementation is a JavaScript/POSIX compatibility runtime with
|
|
4
|
+
CPython compiled to WebAssembly. It is not a Linux kernel or an x86 emulator.
|
|
5
|
+
Keep this distinction in the public API and product claims.
|
|
6
|
+
|
|
7
|
+
## Existing execution paths
|
|
8
|
+
|
|
9
|
+
- JavaScript: the module engine and worker pods implement a subset of Node APIs.
|
|
10
|
+
- Python: CPython runs in a process worker and calls the shared virtual kernel.
|
|
11
|
+
- WASI: compiled wasm32-wasi programs use the existing WASI host.
|
|
12
|
+
- Browser preview: a service worker routes HTTP into the virtual network.
|
|
13
|
+
- Native Linux ELF executables and native Node addons: unsupported by these paths.
|
|
14
|
+
|
|
15
|
+
Browser execution needs suitable worker URLs and, for blocking worker calls,
|
|
16
|
+
SharedArrayBuffer and cross-origin isolation. Static hosting is still needed to
|
|
17
|
+
serve application and runtime assets. It is not a computation backend.
|
|
18
|
+
Python build-on-miss is different: its local builder requires Node/toolchains,
|
|
19
|
+
and its remote builder uses another machine. Prebuilt compatible wheels avoid
|
|
20
|
+
that backend. Not every PyPI package has such a wheel.
|
|
21
|
+
|
|
22
|
+
## Changes in this pass
|
|
23
|
+
|
|
24
|
+
The npm installer now overlaps up to six sibling package downloads while keeping
|
|
25
|
+
filesystem mutations ordered. Installers created through `forCwd` share their
|
|
26
|
+
metadata, archive and platform-detail caches within that installer family.
|
|
27
|
+
Transient metadata/archive failures are evicted so retries can recover.
|
|
28
|
+
An extracted package no longer hides an incomplete dependency tree, and optional
|
|
29
|
+
dependencies override duplicate required entries.
|
|
30
|
+
|
|
31
|
+
Python startup now cleans resources after worker creation or message-send
|
|
32
|
+
failures, and honors cancellation that arrives before or during worker creation.
|
|
33
|
+
|
|
34
|
+
Caches remain in memory; there is no persistent offline package store, lockfile
|
|
35
|
+
implementation, global hoisting, or measured real-project performance claim.
|
|
36
|
+
Browser bundling and the static import guard are checks of build compatibility,
|
|
37
|
+
not substitutes for interactive browser acceptance testing.
|
|
38
|
+
|
|
39
|
+
## Native Linux direction (not implemented)
|
|
40
|
+
|
|
41
|
+
Use a separately loaded browser machine emulator with its own Linux kernel and
|
|
42
|
+
disk image if running unchanged Linux packages is required. Keep it optional:
|
|
43
|
+
ordinary JavaScript/Python projects should not download an entire Linux image.
|
|
44
|
+
Expose the same high-level exec, filesystem transfer and preview operations, but
|
|
45
|
+
report the selected backend and its limitations explicitly. The emulator owns
|
|
46
|
+
its filesystem; transfer files explicitly instead of pretending it shares the
|
|
47
|
+
JavaScript runtime's synchronous in-memory volume.
|
|
48
|
+
|
|
49
|
+
This requires choosing the CPU architecture, emulator, kernel/root filesystem,
|
|
50
|
+
network bridge and image distribution terms. Native addon compatibility then
|
|
51
|
+
comes from running actual Node inside the guest Linux system. It brings larger
|
|
52
|
+
assets and different startup/performance tradeoffs, and does not promise every
|
|
53
|
+
Linux package will work. No such backend is included in this change.
|
|
54
|
+
|
|
55
|
+
MIT licensing of this project can avoid a runtime subscription. It does not
|
|
56
|
+
promise free CDN hosting, registry availability or third-party services forever.
|
|
57
|
+
Audit the separate distribution terms of any kernel, disk image and packages
|
|
58
|
+
before redistributing them.
|
|
59
|
+
|
|
60
|
+
## Extension implementation
|
|
61
|
+
|
|
62
|
+
The subsequent implementation adds ELF dispatch, external WASI command packs,
|
|
63
|
+
three ordered backend tiers, a translator output cache, and optional local Git
|
|
64
|
+
and embedded SQL adapters. See [Developer tool packs](developer-tool-packs.md)
|
|
65
|
+
for the working APIs and explicit limits. Translation and emulator engines
|
|
66
|
+
remain external and are not included.
|
|
67
|
+
|
|
68
|
+
## Verified behaviour, and the gaps that remain
|
|
69
|
+
|
|
70
|
+
A verification harness in the browser test project (`npm run verify`, driving
|
|
71
|
+
real Chrome over the published package) exercises the container boot, Node,
|
|
72
|
+
CPython, an npm install from the registry, both x86-64 ELF tiers, WASI command
|
|
73
|
+
packs, Git, PostgreSQL, an in-container HTTP server, and the frontend
|
|
74
|
+
automation subset. All of it passes in a real browser.
|
|
75
|
+
|
|
76
|
+
One gap is closed and one remains.
|
|
77
|
+
|
|
78
|
+
## Outbound HTTP for Python
|
|
79
|
+
|
|
80
|
+
A browser cannot open a raw TCP socket, so nothing inside the container can
|
|
81
|
+
speak TLS to a real server: outbound traffic has to leave at the HTTP layer,
|
|
82
|
+
through the page's `fetch`. The container therefore serves a loopback HTTP
|
|
83
|
+
endpoint, advertised to the guest as `SBX_HTTP_EGRESS`, which takes an
|
|
84
|
+
*absolute* URL in the request line the way a proxy does and performs the
|
|
85
|
+
transfer on the guest's behalf. `https://` targets are named in that request
|
|
86
|
+
line rather than tunnelled with CONNECT, precisely so the guest never begins a
|
|
87
|
+
TLS handshake it cannot finish; the real request leaves the page over HTTPS.
|
|
88
|
+
|
|
89
|
+
The startup hook points `urllib.request` and `http.client` at it, so the
|
|
90
|
+
standard library reaches the network without changes to a program. The same
|
|
91
|
+
outbound policy every other exit applies is applied here, and because a refusal
|
|
92
|
+
comes back as a real HTTP response it carries the reason -- a guest is told that
|
|
93
|
+
outbound access is disabled and where to enable it, rather than being handed an
|
|
94
|
+
errno it renders as "connection refused".
|
|
95
|
+
|
|
96
|
+
Libraries that manage their own TLS rather than going through `http.client` --
|
|
97
|
+
`urllib3`, and so `requests` -- still cannot egress. Their sockets would need to
|
|
98
|
+
carry a TLS handshake this transport cannot terminate. A page's `fetch` is also
|
|
99
|
+
subject to CORS, so a cross-origin target must send the headers that let the
|
|
100
|
+
page read the response; that is a browser rule, not a container policy.
|
|
101
|
+
|
|
102
|
+
## Blocking syscalls stop every thread
|
|
103
|
+
|
|
104
|
+
Threads in this runtime are real and run in parallel: a worker thread happily
|
|
105
|
+
burns CPU while the main thread sits in `time.sleep`, which releases the GIL.
|
|
106
|
+
What they cannot do is make a syscall while another thread is inside one.
|
|
107
|
+
|
|
108
|
+
The cause is in the interpreter image, not in this package. It is built with
|
|
109
|
+
`-sPROXY_TO_PTHREAD` over a JavaScript-backed filesystem, so every syscall made
|
|
110
|
+
on a thread is forwarded to the single thread that owns that filesystem -- the
|
|
111
|
+
generated glue is full of `if (ENVIRONMENT_IS_PTHREAD) return
|
|
112
|
+
proxyToMainThread(...)`. That thread answers a host call by blocking in
|
|
113
|
+
`Atomics.wait`. While it is blocked it cannot service anybody else's proxied
|
|
114
|
+
syscall, so one thread parked in a blocking read freezes every other thread's
|
|
115
|
+
I/O until it returns.
|
|
116
|
+
|
|
117
|
+
`asyncio.to_thread` is the case that meets this head-on: the loop parks in a
|
|
118
|
+
blocking `select`, and waking it requires the worker thread to make a syscall
|
|
119
|
+
that the parked thread would have to service. Neither completes.
|
|
120
|
+
`to_thread` now raises immediately, naming the limitation, instead of hanging
|
|
121
|
+
until the process is killed. That is a mitigation, not a fix.
|
|
122
|
+
|
|
123
|
+
Three earlier explanations of this were wrong and are recorded here so the next
|
|
124
|
+
person does not re-derive them: it is not a single control slot in the transport
|
|
125
|
+
(threads do run concurrently), it is not the file-scope statics in `sbx_call`
|
|
126
|
+
(they are downstream of the proxying), and it is not a missing per-thread
|
|
127
|
+
channel on its own (a channel per thread does nothing while the syscalls are
|
|
128
|
+
still funnelled to one thread).
|
|
129
|
+
|
|
130
|
+
Fixing it is a change to how the runtime image is built, and there are two
|
|
131
|
+
routes, neither small:
|
|
132
|
+
|
|
133
|
+
- Stack switching. Suspend the wasm stack on a blocking call and return to the
|
|
134
|
+
event loop, instead of holding the thread in `Atomics.wait`. This is what the
|
|
135
|
+
Pyodide backend did through `run_sync`, and it is why that backend did not
|
|
136
|
+
have this problem. It needs JSPI and a transport rewritten around suspension.
|
|
137
|
+
- Unproxied per-thread syscalls. Give each thread its own channel and host
|
|
138
|
+
server and stop routing syscalls through one thread. This fights the
|
|
139
|
+
JavaScript filesystem the image is built on, since that filesystem lives on
|
|
140
|
+
one thread by construction.
|
|
141
|
+
|
|
142
|
+
Both are runtime-image work measured in days, not a patch plus a rebuild.
|
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
# Compatibility implementation plan
|
|
2
|
+
|
|
3
|
+
## Decisions and evidence
|
|
4
|
+
|
|
5
|
+
Keep the owned JavaScript runtime and CPython WebAssembly distribution. No
|
|
6
|
+
Nodepod, Pyodide, v86, or replacement VM is required for the first milestones.
|
|
7
|
+
The timeout, missing Node APIs and Python syscall deadlock are separate issues.
|
|
8
|
+
|
|
9
|
+
The extension acceptance suite serves local wheels using node:http. In the
|
|
10
|
+
restricted development environment listen(127.0.0.1) fails with EPERM. Its
|
|
11
|
+
setup promise previously had no error handler, so setup waited for the hook
|
|
12
|
+
timeout. The test now rejects immediately and teardown handles failed startup.
|
|
13
|
+
With loopback listening permitted, all 12 extension acceptance tests passed in
|
|
14
|
+
34.19 seconds of test execution (37.26 seconds total). This includes C, Cython,
|
|
15
|
+
Rust/PyO3, Meson and Pydantic's compiled core. This establishes the Node-hosted
|
|
16
|
+
fixture path, not browser acceptance or arbitrary PyPI compatibility.
|
|
17
|
+
|
|
18
|
+
The Python build uses PROXY_TO_PTHREAD. backend.ts sets SBX_SERIAL_HOST_CALLS
|
|
19
|
+
and refuses asyncio.to_thread because proxied blocking syscalls serialize the
|
|
20
|
+
filesystem-owning thread. Removing the refusal alone is not a fix.
|
|
21
|
+
|
|
22
|
+
## Alternatives
|
|
23
|
+
|
|
24
|
+
| Approach | Useful for | Main constraint | Decision |
|
|
25
|
+
| --- | --- | --- | --- |
|
|
26
|
+
| Owned JavaScript builtins | Node utility and lifecycle APIs | Must reproduce observable semantics | First choice |
|
|
27
|
+
| Compile source to Wasm | C/C++/Rust extensions and tools | ABI, dependencies and host calls must match | Preferred native-code route |
|
|
28
|
+
| JSPI suspension | Freeing the worker event loop during a blocking host call | Runtime support, suspension boundaries, dynamic modules and pthread interaction | Prototype before adopting |
|
|
29
|
+
| Asyncify | Suspension where JSPI is unavailable | Instrumentation cost and dynamic-module compatibility | Compare as fallback |
|
|
30
|
+
| Per-thread syscall channels | Concurrent Python I/O | Must also remove synchronous proxy bottleneck and preserve fd semantics | Prototype alongside suspension |
|
|
31
|
+
| WasmFS integration | Moving filesystem work away from JS-only ownership | Existing virtual FS needs a backend; not a flag-only change | Assess with thread prototype |
|
|
32
|
+
| Host-native executables | Compatible tools on Node servers | Host OS/architecture, process permissions, unavailable in browsers | Optional explicit host capability |
|
|
33
|
+
| Owned instruction interpreter | Selected binaries without source | ISA, loader, syscalls, linking, signals and threads | Defer until a concrete binary needs it |
|
|
34
|
+
| Binary-to-Wasm translation | Repeated execution of supported binaries | Same ABI needs plus translation correctness and cache invalidation | Later, reuse existing backend interface |
|
|
35
|
+
|
|
36
|
+
## Ordered implementation milestones
|
|
37
|
+
|
|
38
|
+
### 1. Make extension acceptance trustworthy
|
|
39
|
+
|
|
40
|
+
Run the complete local wheel suite with working loopback networking. Separate
|
|
41
|
+
setup, download, install, link, import and execution failures. Fix each observed
|
|
42
|
+
failure with a focused regression. Preserve failures; do not raise timeouts to
|
|
43
|
+
make deadlocks disappear. Repeat in the browser using static fixture hosting.
|
|
44
|
+
|
|
45
|
+
Exit: ordinary imports execute compiled C/C++/Rust fixtures; missing/incompatible
|
|
46
|
+
artifacts produce actionable failures, and shutdown releases all resources.
|
|
47
|
+
|
|
48
|
+
### 2. Replace Node stubs in dependency order
|
|
49
|
+
|
|
50
|
+
Inventory actual package imports. Start with diagnostics_channel and utility
|
|
51
|
+
APIs, followed by worker_threads backed by guest workers, then virtual network
|
|
52
|
+
APIs. Separate virtual TCP from external browser networking. cluster/domain,
|
|
53
|
+
http2, tls, inspector, v8 and vm each need an explicit supported surface rather
|
|
54
|
+
than a catch-all function that pretends to implement them. A JS vm substitute
|
|
55
|
+
must not claim a security boundary it does not provide.
|
|
56
|
+
|
|
57
|
+
Exit per module: native-Node differential tests, error/cancellation tests,
|
|
58
|
+
cross-guest isolation, and a real package that consumes that API in Node and
|
|
59
|
+
browser hosts. Mark unsupported operations explicitly.
|
|
60
|
+
|
|
61
|
+
### 3. Prove concurrent Python I/O before rebuilding production
|
|
62
|
+
|
|
63
|
+
Build a minimal C/pthread fixture against the same Emscripten toolchain. One
|
|
64
|
+
thread blocks on a host read while another writes; add cancellation and a
|
|
65
|
+
timer on the owning JS worker. Compare an asynchronous proxy/suspension path
|
|
66
|
+
with unproxied per-thread host calls. Channels alone cannot repair serialized
|
|
67
|
+
proxy dispatch. Feature-detect JSPI; test fallback behavior explicitly.
|
|
68
|
+
|
|
69
|
+
Measure idle CPU, wakeup latency, memory, download size and throughput for each
|
|
70
|
+
variant. Select only after the fixture passes. Then rebuild CPython and all
|
|
71
|
+
affected extension fixtures, preserving one shared descriptor authority.
|
|
72
|
+
|
|
73
|
+
Exit: asyncio.to_thread, run_in_executor, concurrent socket read/write,
|
|
74
|
+
cancellation, subprocess interaction and dynamic extensions work together.
|
|
75
|
+
Remove the current guard only after these pass in supported browsers and Node.
|
|
76
|
+
|
|
77
|
+
### 4. Expand portable binaries
|
|
78
|
+
|
|
79
|
+
Use source builds targeting the existing Wasm ABI first. Track toolchain and
|
|
80
|
+
ABI fingerprints in artifacts. If an essential source-unavailable binary
|
|
81
|
+
remains, define one ISA and a static executable subset for an owned interpreter;
|
|
82
|
+
reject unsupported instructions and syscalls deterministically. Bound memory,
|
|
83
|
+
execution and cancellation. Do not start with dynamic Linux package parity.
|
|
84
|
+
|
|
85
|
+
Exit: selected real binaries have reproducible correctness tests and measured
|
|
86
|
+
startup, memory and throughput. Translation caching follows correctness.
|
|
87
|
+
|
|
88
|
+
## Sources for the suspension prototypes
|
|
89
|
+
|
|
90
|
+
- Emscripten pthread proxying and blocking rules:
|
|
91
|
+
https://emscripten.org/docs/porting/pthreads.html
|
|
92
|
+
- Emscripten Asyncify and JSPI integration:
|
|
93
|
+
https://emscripten.org/docs/porting/asyncify.html
|
|
94
|
+
- V8 JSPI boundary model:
|
|
95
|
+
https://v8.dev/blog/jspi-newapi
|
|
96
|
+
|
|
97
|
+
These sources describe mechanisms; they do not establish that this CPython
|
|
98
|
+
image or its extensions already support them. The prototypes are required.
|