sandboxedjs 0.2.11 → 0.2.13

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (48) hide show
  1. package/README.md +21 -9
  2. package/assets/logo.png +0 -0
  3. package/bin/sandboxedjs-egress.mjs +25 -10
  4. package/dist/index.cjs +87 -3
  5. package/dist/index.js +87 -3
  6. package/dist/python/python.data +140 -141
  7. package/dist/python/python.js +1 -1
  8. package/dist/python/python.wasm +0 -0
  9. package/dist/python/runtime.json +3 -3
  10. package/dist/python-worker.js +9 -0
  11. package/dist/service-worker.js +3 -2
  12. package/docs/agent/COMMANDS.md +85 -0
  13. package/docs/agent/DECISION-TREE.md +84 -0
  14. package/docs/agent/INVARIANTS.md +40 -0
  15. package/docs/agent/LAUNCH-PROMPT.md +37 -0
  16. package/docs/agent/LOOP.md +84 -0
  17. package/docs/agent/README.md +77 -0
  18. package/docs/agent/ROADMAP.md +37 -0
  19. package/docs/agent/STATE.md +93 -0
  20. package/docs/agent/tasks/00-verify-inherited-work.md +36 -0
  21. package/docs/agent/tasks/01-authoritative-metadata.md +34 -0
  22. package/docs/agent/tasks/02-native-dependencies.md +35 -0
  23. package/docs/agent/tasks/03-reproducible-inputs.md +27 -0
  24. package/docs/agent/tasks/04-build-frontends.md +27 -0
  25. package/docs/agent/tasks/05-registry-integration.md +27 -0
  26. package/docs/agent/tasks/06-package-cohorts.md +42 -0
  27. package/docs/agent/tasks/07-build-on-miss-boundary.md +31 -0
  28. package/docs/browser-runtime-architecture.md +132 -0
  29. package/docs/compatibility-implementation-plan.md +98 -0
  30. package/docs/developer-tool-packs.md +134 -0
  31. package/docs/frontend-automation.md +49 -0
  32. package/docs/fullstack-deployment.md +163 -0
  33. package/docs/handoff.md +275 -0
  34. package/docs/original-x64.md +39 -0
  35. package/docs/platform-hardening.md +49 -0
  36. package/docs/python/abi.md +97 -0
  37. package/docs/python/architecture.md +94 -0
  38. package/docs/python/baseline-inventory.md +54 -0
  39. package/docs/python/build-on-miss.md +198 -0
  40. package/docs/python/compatibility.md +206 -0
  41. package/docs/python/cross-build.md +354 -0
  42. package/docs/python/extensions.md +282 -0
  43. package/docs/python/release-gates.md +46 -0
  44. package/docs/python/virtual-sockets-plan.md +331 -0
  45. package/docs/runtime-lifecycle-fixes.md +39 -0
  46. package/docs/server-previews.md +268 -0
  47. package/docs/virtual-browser.md +120 -0
  48. package/package.json +6 -3
@@ -0,0 +1,34 @@
1
+ # Task 01: authoritative wheel metadata
2
+
3
+ ## Problem
4
+
5
+ The current builder synthesizes `METADATA` from recipe fields. That duplicates
6
+ the package's dependency declarations and can create a wheel that installs but
7
+ resolves the wrong environment.
8
+
9
+ ## Deliverable
10
+
11
+ Generate distribution metadata using the package's build metadata path under
12
+ the controlled build environment. Preserve upstream `Name`, `Version`,
13
+ `Requires-Dist`, extras, Python requirements, entry points, licenses, and other
14
+ install-relevant files. SandboxedJS should add only its wheel tag, generator,
15
+ and `Build-ABI` assertion.
16
+
17
+ Recipe constraints may explicitly override a target-inapplicable dependency,
18
+ but every override must be visible, justified, and tested.
19
+
20
+ ## Required tests
21
+
22
+ - A fixture declares a conditional and an extra dependency upstream; neither
23
+ is copied into the recipe.
24
+ - The resulting wheel and generated `index.json` contain the authoritative
25
+ requirements.
26
+ - The resolver respects at least one marker/extras case.
27
+ - A disagreement between recipe identity and upstream identity fails loudly.
28
+ - Existing extension runtime tests remain green.
29
+
30
+ ## Exit gate
31
+
32
+ No generic setuptools fixture needs handwritten `requires` metadata, and the
33
+ index is provably derived from the wheel produced from upstream metadata.
34
+
@@ -0,0 +1,35 @@
1
+ # Task 02: native target dependencies and sysroot
2
+
3
+ ## Problem
4
+
5
+ Extensions that name libraries or include their headers cannot currently
6
+ declare, build, locate, or link those target dependencies generically.
7
+
8
+ ## Design requirement
9
+
10
+ Represent native target dependencies separately from Python runtime
11
+ dependencies and host build requirements. Each dependency must identify a
12
+ pinned source, hash, build recipe, output markers, license/provenance, and ABI
13
+ profile. The dynamic profile uses `out/sysroot-dynamic`; incompatible profiles
14
+ must never share object archives.
15
+
16
+ Add sysroot include and library search paths through generated cross-build
17
+ configuration, not ad hoc recipe flags.
18
+
19
+ ## Reduced acceptance fixture
20
+
21
+ Before attempting a large external library, create a tiny static target library
22
+ with a header and one function. Build/install it through the dependency system,
23
+ then build a setuptools extension that declares the library normally and calls
24
+ it. The wheel must install and execute the function in the owned runtime.
25
+
26
+ After the reduced fixture passes, add one small real native library already
27
+ compatible with the project architecture. Do not begin with BLAS or NumPy.
28
+
29
+ ## Exit gate
30
+
31
+ - Dependency closure is deterministic and cycle/error aware.
32
+ - Rebuilds are skipped only when validated output markers match inputs/profile.
33
+ - Missing dependency and wrong-profile reuse have focused negative tests.
34
+ - Reduced and real-library extensions import and run in owned CPython.
35
+
@@ -0,0 +1,27 @@
1
+ # Task 03: reproducible inputs, patches, and provenance
2
+
3
+ ## Deliverable
4
+
5
+ Unify source archives, build-tool wheels, native dependencies, and package
6
+ patches under verified acquisition. No release artifact may depend on an
7
+ unpinned `pip install setuptools wheel ...` result.
8
+
9
+ Define a visible patch layout. A patch records package/version, upstream source
10
+ hash, reason, target fact being substituted, and a test. Patch application must
11
+ fail on drift rather than fuzzily succeeding against another source version.
12
+
13
+ Record enough provenance in build output or adjacent index metadata to answer:
14
+
15
+ - which source hash produced this wheel;
16
+ - which build tools and target libraries were used;
17
+ - which ABI ID and recipe revision were used;
18
+ - which patches were applied.
19
+
20
+ ## Exit gate
21
+
22
+ - Offline rebuild succeeds from a populated verified cache.
23
+ - A changed hash, unpinned build tool, or drifting patch fails before compile.
24
+ - Two identical clean builds produce equivalent wheel contents, allowing only
25
+ explicitly documented archive timestamp normalization until deterministic
26
+ ZIP output is implemented.
27
+
@@ -0,0 +1,27 @@
1
+ # Task 04: build frontend interface
2
+
3
+ ## Purpose
4
+
5
+ Turn the current kinds into a stable backend interface rather than accumulating
6
+ branches in `build_extension.py`.
7
+
8
+ ## Required interface
9
+
10
+ Each backend receives verified source, host tools, target configuration,
11
+ declared native dependencies, staging directory, and ABI contract. It returns a
12
+ wheel tree or wheel plus build evidence. It cannot mutate the ABI.
13
+
14
+ Retain proven setuptools/Cython and PyO3 behavior. Add another frontend only
15
+ when a selected cohort package requires it; likely candidates include PEP 517
16
+ wheel hooks, Meson, or scikit-build/CMake. Do not implement all frontends
17
+ speculatively.
18
+
19
+ Keep package quirks in patches/recipes and backend mechanics in adapters.
20
+
21
+ ## Exit gate
22
+
23
+ - Backend selection is explicit and schema-validated.
24
+ - The generic orchestrator has no package-name checks.
25
+ - Existing fixtures pass through the interface.
26
+ - One pyproject-only package runs through a real isolated PEP 517 path.
27
+
@@ -0,0 +1,27 @@
1
+ # Task 05: registry and installer integration
2
+
3
+ ## Deliverable
4
+
5
+ Produce an immutable wheel repository that the existing SandboxedJS `pip`
6
+ resolver can consume. Define schema versioning, atomic publication layout,
7
+ digest verification, ABI partitioning, and dependency metadata.
8
+
9
+ The browser runtime remains an installer, not a compiler. A missing native
10
+ wheel should return a structured, actionable classification that a host may use
11
+ later to request a build. It must not perform an implicit remote mutation.
12
+
13
+ ## Required tests
14
+
15
+ - Correct ABI wheel wins over incompatible and host-native candidates.
16
+ - Wrong ABI and wrong digest are refused before installation.
17
+ - A failed graph leaves no partial site-packages state.
18
+ - A newly built test wheel can be indexed, served, installed, imported, and
19
+ exercised through the ordinary container command.
20
+ - Index publication cannot expose a wheel before its metadata and content are
21
+ complete.
22
+
23
+ ## Exit gate
24
+
25
+ A clean consumer needs only a runtime plus index URL/object; no local source
26
+ tree or compiler is required for supported packages.
27
+
@@ -0,0 +1,42 @@
1
+ # Task 06: representative package cohorts
2
+
3
+ ## Purpose
4
+
5
+ Measure generality without allowing one package to define the architecture.
6
+
7
+ ## Selection method
8
+
9
+ Choose small packages first, pin exact versions and hashes, and record why each
10
+ adds a new capability. Select at least one from each applicable cohort:
11
+
12
+ 1. pure wheel control case;
13
+ 2. setuptools C extension without external target libraries;
14
+ 3. generated C/Cython extension;
15
+ 4. Rust/PyO3 extension;
16
+ 5. extension using a separately built native target library;
17
+ 6. package requiring a visible cross-compilation patch;
18
+ 7. pyproject/PEP 517 backend not already covered;
19
+ 8. scientific-stack stress case, eventually including NumPy.
20
+
21
+ Do not count existing probes as real-package cohort evidence, but retain them as
22
+ reduced regression tests.
23
+
24
+ ## Compatibility report
25
+
26
+ For every attempted package, record exactly one status:
27
+
28
+ - `supported-generic`;
29
+ - `supported-recipe`;
30
+ - `blocked-platform-capability`;
31
+ - `blocked-toolchain`;
32
+ - `not-yet-attempted`.
33
+
34
+ Record build command, source hash, wheel hash, ABI ID, test behavior, peak size
35
+ or memory when material, and the earliest unsupported assumption on failure.
36
+
37
+ ## Exit gate
38
+
39
+ At least five independent real packages across four native/build shapes pass
40
+ runtime behavior tests, and failures produce reusable classifications instead
41
+ of package-specific branches.
42
+
@@ -0,0 +1,31 @@
1
+ # Task 07: external build-on-miss boundary
2
+
3
+ ## Purpose
4
+
5
+ Specify the seam that can later connect the installer to a trusted build
6
+ service without deploying that service or placing a toolchain in the browser.
7
+
8
+ ## Deliverable
9
+
10
+ Define versioned request/status/result schemas. A request includes normalized
11
+ requirement, source/index evidence, target ABI ID, policy/recipe revision, and
12
+ an idempotency key. A result includes immutable wheel/index locations, hashes,
13
+ provenance, logs summary, compatibility classification, and terminal state.
14
+
15
+ States should cover queued, resolving, building host tools, building target
16
+ dependencies, building wheel, testing, published, unsupported, failed, and
17
+ cancelled. Repeated identical requests must converge on one cached artifact.
18
+
19
+ Security requirements include source allowlists/policy, resource limits,
20
+ network isolation, no untrusted artifact publication before runtime tests,
21
+ digest verification, and auditability.
22
+
23
+ ## Exit gate
24
+
25
+ - Schemas and a local fake implementation are tested.
26
+ - Installer/host integration can report a buildable miss without automatically
27
+ triggering external work.
28
+ - Cached success, unsupported result, transient failure, cancellation, and ABI
29
+ rollover are covered.
30
+ - No cloud service is deployed without explicit user authorization.
31
+
@@ -0,0 +1,132 @@
1
+ # Browser runtime architecture
2
+
3
+ The current implementation is a JavaScript/POSIX compatibility runtime with
4
+ CPython compiled to WebAssembly. It is not a Linux kernel or an x86 emulator.
5
+ Keep this distinction in the public API and product claims.
6
+
7
+ ## Existing execution paths
8
+
9
+ - JavaScript: the module engine and worker pods implement a subset of Node APIs.
10
+ - Python: CPython runs in a process worker and calls the shared virtual kernel.
11
+ - WASI: compiled wasm32-wasi programs use the existing WASI host.
12
+ - Browser preview: a service worker routes HTTP into the virtual network.
13
+ - Native Linux ELF executables and native Node addons: unsupported by these paths.
14
+
15
+ Browser execution needs suitable worker URLs and, for blocking worker calls,
16
+ SharedArrayBuffer and cross-origin isolation. Static hosting is still needed to
17
+ serve application and runtime assets. It is not a computation backend.
18
+ Python build-on-miss is different: its local builder requires Node/toolchains,
19
+ and its remote builder uses another machine. Prebuilt compatible wheels avoid
20
+ that backend. Not every PyPI package has such a wheel.
21
+
22
+ ## Changes in this pass
23
+
24
+ The npm installer now overlaps up to six sibling package downloads while keeping
25
+ filesystem mutations ordered. Installers created through `forCwd` share their
26
+ metadata, archive and platform-detail caches within that installer family.
27
+ Transient metadata/archive failures are evicted so retries can recover.
28
+ An extracted package no longer hides an incomplete dependency tree, and optional
29
+ dependencies override duplicate required entries.
30
+
31
+ Python startup now cleans resources after worker creation or message-send
32
+ failures, and honors cancellation that arrives before or during worker creation.
33
+
34
+ Caches remain in memory; there is no persistent offline package store, lockfile
35
+ implementation, global hoisting, or measured real-project performance claim.
36
+ Browser bundling and the static import guard are checks of build compatibility,
37
+ not substitutes for interactive browser acceptance testing.
38
+
39
+ ## Native Linux direction (not implemented)
40
+
41
+ Use a separately loaded browser machine emulator with its own Linux kernel and
42
+ disk image if running unchanged Linux packages is required. Keep it optional:
43
+ ordinary JavaScript/Python projects should not download an entire Linux image.
44
+ Expose the same high-level exec, filesystem transfer and preview operations, but
45
+ report the selected backend and its limitations explicitly. The emulator owns
46
+ its filesystem; transfer files explicitly instead of pretending it shares the
47
+ JavaScript runtime's synchronous in-memory volume.
48
+
49
+ This requires choosing the CPU architecture, emulator, kernel/root filesystem,
50
+ network bridge and image distribution terms. Native addon compatibility then
51
+ comes from running actual Node inside the guest Linux system. It brings larger
52
+ assets and different startup/performance tradeoffs, and does not promise every
53
+ Linux package will work. No such backend is included in this change.
54
+
55
+ MIT licensing of this project can avoid a runtime subscription. It does not
56
+ promise free CDN hosting, registry availability or third-party services forever.
57
+ Audit the separate distribution terms of any kernel, disk image and packages
58
+ before redistributing them.
59
+
60
+ ## Extension implementation
61
+
62
+ The subsequent implementation adds ELF dispatch, external WASI command packs,
63
+ three ordered backend tiers, a translator output cache, and optional local Git
64
+ and embedded SQL adapters. See [Developer tool packs](developer-tool-packs.md)
65
+ for the working APIs and explicit limits. Translation and emulator engines
66
+ remain external and are not included.
67
+
68
+ ## Verified behaviour, and the gaps that remain
69
+
70
+ A verification harness in the browser test project (`npm run verify`, driving
71
+ real Chrome over the published package) exercises the container boot, Node,
72
+ CPython, an npm install from the registry, both x86-64 ELF tiers, WASI command
73
+ packs, Git, PostgreSQL, an in-container HTTP server, and the frontend
74
+ automation subset. All of it passes in a real browser.
75
+
76
+ One gap is closed and one remains.
77
+
78
+ ## Outbound HTTP for Python
79
+
80
+ A browser cannot open a raw TCP socket, so nothing inside the container can
81
+ speak TLS to a real server: outbound traffic has to leave at the HTTP layer,
82
+ through the page's `fetch`. The container therefore serves a loopback HTTP
83
+ endpoint, advertised to the guest as `SBX_HTTP_EGRESS`, which takes an
84
+ *absolute* URL in the request line the way a proxy does and performs the
85
+ transfer on the guest's behalf. `https://` targets are named in that request
86
+ line rather than tunnelled with CONNECT, precisely so the guest never begins a
87
+ TLS handshake it cannot finish; the real request leaves the page over HTTPS.
88
+
89
+ The startup hook points `urllib.request` and `http.client` at it, so the
90
+ standard library reaches the network without changes to a program. The same
91
+ outbound policy every other exit applies is applied here, and because a refusal
92
+ comes back as a real HTTP response it carries the reason -- a guest is told that
93
+ outbound access is disabled and where to enable it, rather than being handed an
94
+ errno it renders as "connection refused".
95
+
96
+ Libraries that manage their own TLS rather than going through `http.client` --
97
+ `urllib3`, and so `requests` -- still cannot egress. Their sockets would need to
98
+ carry a TLS handshake this transport cannot terminate. A page's `fetch` is also
99
+ subject to CORS, so a cross-origin target must send the headers that let the
100
+ page read the response; that is a browser rule, not a container policy.
101
+
102
+ ## Python thread offloads
103
+
104
+ The bundled pthread interpreter supports standard `asyncio.to_thread`,
105
+ `run_in_executor`, and AnyIO's thread pool, including synchronous FastAPI routes.
106
+ These execute on real Python worker threads, preserving context propagation,
107
+ exceptions, and cancellation of the awaiting task.
108
+
109
+ Native socket calls release the GIL and send requests directly from each pthread
110
+ to the process supervisor over a private BroadcastChannel. Every outstanding
111
+ request owns a shared response buffer, so a waiting interpreter cannot prevent
112
+ another thread from completing network I/O. The supervisor still applies the
113
+ same descriptor ownership and outbound network policy.
114
+
115
+ Emscripten filesystem operations still proxy through the interpreter thread.
116
+ While other Python threads exist, the asyncio selector limits each wait to
117
+ 10 ms and briefly yields, allowing those queued operations to run. This prevents
118
+ the old deadlock between a sleeping selector and a worker trying to wake it.
119
+ It adds polling overhead while the thread pool is alive.
120
+
121
+ Deploy the matching `python-worker.js` and the complete `python/` runtime directory
122
+ together. The worker detects the native bridge before enabling this behavior.
123
+ Older custom interpreter images retain the previous explicit `to_thread` error
124
+ and inline AnyIO fallback rather than silently entering a known deadlock.
125
+
126
+ Regression coverage includes context variables, exceptions, cancellation, file
127
+ I/O, simultaneous thread HTTP calls into an asyncio server, executor shutdown,
128
+ and FastAPI synchronous endpoints. The browser fixture also offloads a file read
129
+ and an outbound HTTPX request from a FastAPI streaming endpoint. This does not
130
+ certify arbitrary native extensions or indefinite blocking filesystem operations;
131
+ thread churn and long-running cleanup soak remain release gates. Browser egress
132
+ restrictions still apply independently of threading.
@@ -0,0 +1,98 @@
1
+ # Compatibility implementation plan
2
+
3
+ ## Decisions and evidence
4
+
5
+ Keep the owned JavaScript runtime and CPython WebAssembly distribution. No
6
+ Nodepod, Pyodide, v86, or replacement VM is required for the first milestones.
7
+ The timeout, missing Node APIs and Python syscall deadlock are separate issues.
8
+
9
+ The extension acceptance suite serves local wheels using node:http. In the
10
+ restricted development environment listen(127.0.0.1) fails with EPERM. Its
11
+ setup promise previously had no error handler, so setup waited for the hook
12
+ timeout. The test now rejects immediately and teardown handles failed startup.
13
+ With loopback listening permitted, all 12 extension acceptance tests passed in
14
+ 34.19 seconds of test execution (37.26 seconds total). This includes C, Cython,
15
+ Rust/PyO3, Meson and Pydantic's compiled core. This establishes the Node-hosted
16
+ fixture path, not browser acceptance or arbitrary PyPI compatibility.
17
+
18
+ The Python build uses PROXY_TO_PTHREAD. backend.ts sets SBX_SERIAL_HOST_CALLS
19
+ and refuses asyncio.to_thread because proxied blocking syscalls serialize the
20
+ filesystem-owning thread. Removing the refusal alone is not a fix.
21
+
22
+ ## Alternatives
23
+
24
+ | Approach | Useful for | Main constraint | Decision |
25
+ | --- | --- | --- | --- |
26
+ | Owned JavaScript builtins | Node utility and lifecycle APIs | Must reproduce observable semantics | First choice |
27
+ | Compile source to Wasm | C/C++/Rust extensions and tools | ABI, dependencies and host calls must match | Preferred native-code route |
28
+ | JSPI suspension | Freeing the worker event loop during a blocking host call | Runtime support, suspension boundaries, dynamic modules and pthread interaction | Prototype before adopting |
29
+ | Asyncify | Suspension where JSPI is unavailable | Instrumentation cost and dynamic-module compatibility | Compare as fallback |
30
+ | Per-thread syscall channels | Concurrent Python I/O | Must also remove synchronous proxy bottleneck and preserve fd semantics | Prototype alongside suspension |
31
+ | WasmFS integration | Moving filesystem work away from JS-only ownership | Existing virtual FS needs a backend; not a flag-only change | Assess with thread prototype |
32
+ | Host-native executables | Compatible tools on Node servers | Host OS/architecture, process permissions, unavailable in browsers | Optional explicit host capability |
33
+ | Owned instruction interpreter | Selected binaries without source | ISA, loader, syscalls, linking, signals and threads | Defer until a concrete binary needs it |
34
+ | Binary-to-Wasm translation | Repeated execution of supported binaries | Same ABI needs plus translation correctness and cache invalidation | Later, reuse existing backend interface |
35
+
36
+ ## Ordered implementation milestones
37
+
38
+ ### 1. Make extension acceptance trustworthy
39
+
40
+ Run the complete local wheel suite with working loopback networking. Separate
41
+ setup, download, install, link, import and execution failures. Fix each observed
42
+ failure with a focused regression. Preserve failures; do not raise timeouts to
43
+ make deadlocks disappear. Repeat in the browser using static fixture hosting.
44
+
45
+ Exit: ordinary imports execute compiled C/C++/Rust fixtures; missing/incompatible
46
+ artifacts produce actionable failures, and shutdown releases all resources.
47
+
48
+ ### 2. Replace Node stubs in dependency order
49
+
50
+ Inventory actual package imports. Start with diagnostics_channel and utility
51
+ APIs, followed by worker_threads backed by guest workers, then virtual network
52
+ APIs. Separate virtual TCP from external browser networking. cluster/domain,
53
+ http2, tls, inspector, v8 and vm each need an explicit supported surface rather
54
+ than a catch-all function that pretends to implement them. A JS vm substitute
55
+ must not claim a security boundary it does not provide.
56
+
57
+ Exit per module: native-Node differential tests, error/cancellation tests,
58
+ cross-guest isolation, and a real package that consumes that API in Node and
59
+ browser hosts. Mark unsupported operations explicitly.
60
+
61
+ ### 3. Prove concurrent Python I/O before rebuilding production
62
+
63
+ Build a minimal C/pthread fixture against the same Emscripten toolchain. One
64
+ thread blocks on a host read while another writes; add cancellation and a
65
+ timer on the owning JS worker. Compare an asynchronous proxy/suspension path
66
+ with unproxied per-thread host calls. Channels alone cannot repair serialized
67
+ proxy dispatch. Feature-detect JSPI; test fallback behavior explicitly.
68
+
69
+ Measure idle CPU, wakeup latency, memory, download size and throughput for each
70
+ variant. Select only after the fixture passes. Then rebuild CPython and all
71
+ affected extension fixtures, preserving one shared descriptor authority.
72
+
73
+ Exit: asyncio.to_thread, run_in_executor, concurrent socket read/write,
74
+ cancellation, subprocess interaction and dynamic extensions work together.
75
+ Remove the current guard only after these pass in supported browsers and Node.
76
+
77
+ ### 4. Expand portable binaries
78
+
79
+ Use source builds targeting the existing Wasm ABI first. Track toolchain and
80
+ ABI fingerprints in artifacts. If an essential source-unavailable binary
81
+ remains, define one ISA and a static executable subset for an owned interpreter;
82
+ reject unsupported instructions and syscalls deterministically. Bound memory,
83
+ execution and cancellation. Do not start with dynamic Linux package parity.
84
+
85
+ Exit: selected real binaries have reproducible correctness tests and measured
86
+ startup, memory and throughput. Translation caching follows correctness.
87
+
88
+ ## Sources for the suspension prototypes
89
+
90
+ - Emscripten pthread proxying and blocking rules:
91
+ https://emscripten.org/docs/porting/pthreads.html
92
+ - Emscripten Asyncify and JSPI integration:
93
+ https://emscripten.org/docs/porting/asyncify.html
94
+ - V8 JSPI boundary model:
95
+ https://v8.dev/blog/jspi-newapi
96
+
97
+ These sources describe mechanisms; they do not establish that this CPython
98
+ image or its extensions already support them. The prototypes are required.
@@ -0,0 +1,134 @@
1
+ # Optional developer tools and binary backends
2
+
3
+ This is an extension mechanism and two limited developer-tool adapters, not
4
+ universal Linux compatibility. The core has no new third-party dependencies,
5
+ paid endpoint, account requirement, or automatic downloads.
6
+
7
+ ## Working developer tools
8
+
9
+ Install engines in the **host application**, then pass them to the adapters:
10
+
11
+ ```sh
12
+ npm install sandboxedjs isomorphic-git @electric-sql/pglite
13
+ ```
14
+
15
+ ```js
16
+ import git from 'isomorphic-git';
17
+ import { PGlite } from '@electric-sql/pglite';
18
+ import { createContainer, createGitCommand, createSqlCommand } from 'sandboxedjs';
19
+
20
+ const box = await createContainer({ cwd: '/app' });
21
+ const db = new PGlite();
22
+ box.kernel.installCommand(createGitCommand(git));
23
+ box.kernel.installCommand(createSqlCommand(db));
24
+
25
+ await box.exec('git init');
26
+ await box.fs.writeFile('/app/hello.txt', 'hello');
27
+ await box.exec('git add hello.txt');
28
+ await box.exec('git commit -m first', {
29
+ env: { GIT_AUTHOR_NAME: 'Your Name', GIT_AUTHOR_EMAIL: 'you@example.com' },
30
+ });
31
+ console.log(await box.exec('git log'));
32
+ console.log(await box.exec('sql -c "SELECT 42 AS answer"'));
33
+
34
+ // The application owns the database lifetime and persistence configuration.
35
+ await db.close();
36
+ box.dispose();
37
+ ```
38
+
39
+ Git supports `init`, `add PATH...`, `commit -m MESSAGE`, `status`, and `log`.
40
+ These are local operations backed by isomorphic-git and the credential-aware
41
+ container filesystem. Clone, fetch, push, branches, checkout and full Git CLI
42
+ compatibility are not implemented. No CORS proxy is configured implicitly.
43
+
44
+ `sql -c SQL` and `sql -f FILE` print result rows as JSON. This is an embedded
45
+ SQL adapter, not `psql`, a PostgreSQL server, or a TCP endpoint for `pg` clients.
46
+ Database storage belongs to PGlite, separately from the container filesystem.
47
+ Use its persistence options in the host; container snapshots do not capture it.
48
+ The supplied database API does not provide hard query cancellation; a timeout
49
+ is not a transaction rollback. Do not share the same database across tenants.
50
+
51
+ Both adapters are trusted host integrations. This pass validated real Git
52
+ commits with isomorphic-git 1.42.2 and real SQL with PGlite 0.5.8 on Node,
53
+ not a browser UI. Their upstream engines
54
+ support browsers; host asset loading must still be configured and verified.
55
+
56
+ ## Precompiled command packs
57
+
58
+ Packs live outside the core. A host can load their manifests and artifact bytes
59
+ from its own storage and call:
60
+
61
+ ```js
62
+ import { installWasmCommands } from 'sandboxedjs';
63
+ installWasmCommands(box.kernel, [
64
+ { name: 'my-tool', bytes: wasmBytes, sha256: trustedManifest.sha256 },
65
+ ]);
66
+ await box.exec('my-tool --help');
67
+ ```
68
+
69
+ Supply a wasm32-wasi command compatible with this runtime's WASI preview1 host.
70
+ Hashes detect changed artifact bytes; trust in the manifest comes from the host.
71
+ Names, collisions, hashes and Wasm validity are checked before commands are
72
+ installed. No network fetch, registry, package archive or license acceptance is
73
+ hidden in this API. WASI imports and behaviors still need compatibility checks;
74
+ valid Wasm alone does not prove that a command can execute here.
75
+
76
+ ## ELF execution and fallback
77
+
78
+ The kernel now recognizes executable ELF files instead of treating them as
79
+ shell scripts. It consults `box.kernel.binaries` in this order:
80
+
81
+ 1. `compatibility`: exact binary hashes mapped to tested ports.
82
+ 2. `translation`: a supplied compiler producing compatible WASI commands.
83
+ 3. `emulation`: a supplied emulator backend.
84
+
85
+ `createWasmCompatibilityBackend(id, entries)` registers mappings of
86
+ `{ elfSha256, wasm: { name, bytes, sha256 } }`. Matching content rather than a
87
+ command name prevents silently substituting a port for a different version.
88
+
89
+ `createTranslationBackend(translator, maxCacheBytes?)` wraps a compiler with a
90
+ 32 MiB default in-memory LRU output cache. The compiler supplies `id`,
91
+ `supports(info)` and `translate(request, signal)`. Give it a versioned identity.
92
+ Return null only when the input is unsupported. Thrown errors are surfaced.
93
+ This wrapper **does not contain an ELF compiler**. Its checks use a fixture
94
+ translator, not a claim that native instructions were translated in the test.
95
+ Ordinary elfconv/Emscripten output cannot be assumed to match our WASI ABI.
96
+
97
+ An emulator implements `BinaryBackend`, with `tier: 'emulation'` and
98
+ `prepare(request, signal)`. Register it with `box.kernel.binaries.register()`;
99
+ the returned function unregisters it. `list()` reports installed providers.
100
+ An original experimental x86-64 translator and interpreter are now available
101
+ through `createOriginalX64Backends()`. They support a small freestanding subset;
102
+ see [Original engines](original-x64.md). No Linux image is shipped.
103
+
104
+ Preparation returns either `{ supported: false, reason }` or
105
+ `{ supported: true, program: { run(ctx) } }`. Preparation must not execute guest
106
+ code or modify guest files. Backends receive an AbortSignal and must honor it.
107
+ After `run()` starts, its exit code is final: retrying on another backend could
108
+ duplicate writes or other effects. A native program with no supported backend
109
+ exits 126 with a diagnostic. Hard interruption requires a backend worker; the
110
+ existing in-realm WASI runner cannot interrupt a tight compute loop.
111
+
112
+ ## Package priorities
113
+
114
+ | Priority | Tools | Approach and current status |
115
+ | --- | --- | --- |
116
+ | 1 | Git | Optional isomorphic-git adapter; local subset implemented |
117
+ | 1 | PostgreSQL SQL | Optional PGlite adapter implemented; embedded SQL only |
118
+ | 2 | jq, ripgrep, SQLite CLI, diff/patch | Candidate separate WASI packs; not bundled or validated here |
119
+ | 3 | C/C++ compiler and build tools | Separate large toolchain pack; compilation and subprocess support need work |
120
+ | Later | Redis-compatible services | Evaluate a separate engine and exact protocol/command coverage |
121
+ | Separate project | Docker Engine | Requires Linux kernel facilities; not a small WASI shim |
122
+
123
+ Broad binary translation and emulation coverage remain substantive follow-up work.
124
+ Do not advertise arbitrary ELF, Docker, complete PostgreSQL service compatibility,
125
+ or WebContainer performance parity based on this extension layer.
126
+
127
+ ## Cost and licensing
128
+
129
+ SandboxedJs remains MIT. Upstream PGlite and isomorphic-git publish permissive
130
+ licenses, but retain the notices and check the specific versions you distribute.
131
+ A compatibility layer does not remove a tool's license obligations or make
132
+ enterprise software free. Host-supplied packs let users choose their own tools
133
+ and licenses. Hosting, compilation and network services have real resource costs;
134
+ this API does not require a paid provider or promise free third-party hosting.
@@ -0,0 +1,49 @@
1
+ # Frontend browser automation
2
+
3
+ `createFrontendPlaywright(options?)` provides a Playwright-shaped API that runs
4
+ entirely in the page hosting SandboxedJs. It drives a same-origin `<iframe>`
5
+ using the browser the user already has. It does **not** bundle, download or
6
+ launch Chromium, and it is not Playwright.
7
+
8
+ ```js
9
+ import { createFrontendPlaywright } from 'sandboxedjs';
10
+
11
+ const pw = createFrontendPlaywright({ mount: document.querySelector('#tests') });
12
+ const browser = await pw.chromium.launch();
13
+ const page = await browser.newPage();
14
+
15
+ await page.goto('/preview/index.html');
16
+ await page.getByRole('textbox', { name: 'Email' }).fill('a@example.com');
17
+ await page.getByRole('button', { name: 'Submit' }).click();
18
+ await pw.expect(page.getByText('Thanks')).toBeVisible();
19
+
20
+ await pw.dispose();
21
+ ```
22
+
23
+ ## Supported
24
+
25
+ `page`: `goto`, `setContent`, `title`, `content`, `evaluate`,
26
+ `setDefaultTimeout`, `close`; locators via `locator`, `getByTestId`,
27
+ `getByText`, and `getByRole` for `button`, `textbox`, `link`, `checkbox`
28
+ (plus any explicit `role=` attribute).
29
+
30
+ `locator`: `first`, `nth`, `locator`, `count`, `isVisible`, `textContent`,
31
+ `innerText`, `inputValue`, `getAttribute`, `click`, `fill`, `waitFor`.
32
+ Locators are strict: more than one match throws unless you narrow with
33
+ `first()`/`nth()`. Actions wait for the element to be visible and enabled.
34
+
35
+ `expect(locator)`: `toBeVisible`, `toHaveText`.
36
+
37
+ ## Not supported
38
+
39
+ `screenshot`, `route`/network interception, cross-origin navigation, browser
40
+ launch options, and `firefox`/`webkit` all throw
41
+ `FrontendAutomationUnsupported`. Clicks and typing are dispatched DOM events,
42
+ not OS-level trusted input; `capabilities.trustedEvents` is `false`. `goto`
43
+ returns `null` rather than a Response. `evaluate` uses `eval` inside the frame
44
+ and fails where the page's CSP forbids it.
45
+
46
+ Pass `resolveUrl` to map a container's virtual server URL to the same-origin
47
+ preview URL served by the SandboxedJs service worker. Inspect
48
+ `pw.capabilities` before relying on any behavior; treat this as a labeled
49
+ subset for smoke-testing your own preview output, not a Chromium replacement.