sandboxedjs 0.2.10 → 0.2.12

package/docs/agent/LAUNCH-PROMPT.md

@@ -0,0 +1,37 @@
+# Prompt to launch or resume Claude Opus
+
+Copy the text below into the coding task. Do not paste the rest of this folder
+into the prompt; the agent should read files on demand.
+
+```text
+You are the sole implementation agent for the SandboxedJS native Python
+package platform. The project controller is:
+
+agent-projects/python-package-platform/README.md
+
+Begin by reading only README.md, STATE.md, and the current task named in
+STATE.md. Follow LOOP.md. Execute commands and make decisions from their actual
+outputs. Continue through tasks while gates pass and context remains healthy.
+Keep STATE.md current so another task can resume without chat history.
+
+Preserve all pre-existing and staged work. Do not commit, publish, deploy,
+discard changes, weaken ABI validation, or change the ABI contract unless I
+explicitly authorize it. Use focused tests before broad suites. Keep long build
+logs in /tmp and bring only decisive excerpts into context.
+
+If a stop condition in LOOP.md occurs, stop with a compact escalation packet.
+The goal is a general package platform; NumPy is a later acceptance case, not a
+package-specific architecture target.
+
+Do not merely review or propose a plan. Work autonomously: inspect, edit,
+execute focused commands, diagnose their outputs, verify exit gates, update
+STATE.md, and advance to the next task. Do not ask for confirmation between
+ordinary in-scope steps. Continue until a documented stop condition requires my
+decision or all roadmap gates are complete.
+```
+
+## Execution policy
+
+This workflow assumes one Claude Opus agent. Do not delegate work to other
+models or create subagents. Use the stop conditions and ask the user for one
+concrete decision only when necessary.

package/docs/agent/LOOP.md

@@ -0,0 +1,84 @@
+# Agent execution loop
+
+Use this loop exactly. It is designed for a smaller model that can act well
+when each decision is grounded in the previous command's output.
+
+## 1. Orient
+
+Read `README.md`, `STATE.md`, and the current task only. Inspect `git status
+--short` before editing. Treat all pre-existing changes as user-owned.
+
+State the current hypothesis in one sentence. A hypothesis must predict an
+observable result, for example: "the linker cannot find `libxml2` because the
+dynamic sysroot is absent from target link flags."
+
+## 2. Choose the cheapest discriminating command
+
+Run the smallest command that can disprove the hypothesis. Prefer, in order:
+
+1. static inspection of one relevant file;
+2. one unit test or one fixture build;
+3. one runtime import test;
+4. the focused suite;
+5. the full suite only at a milestone gate.
+
+Never rerun a long build unchanged. If a command failed, inspect its decisive
+error first and change either the hypothesis or the implementation.
+
+## 3. Classify the result
+
+Use `DECISION-TREE.md`. Every failure must be classified before code changes.
+If there is not enough evidence, gather more evidence rather than guessing.
+
+## 4. Make one coherent change
+
+Change the smallest architectural seam that removes the demonstrated failure
+class. Do not add a package-name conditional to the generic builder. A package
+patch belongs in an explicit recipe patch directory, with a comment explaining
+the upstream assumption it replaces.
+
+Use `apply_patch` for hand edits. Do not overwrite files with shell redirects.
+Do not alter generated ABI files directly.
+
+## 5. Verify in layers
+
+Run the focused check that failed. If it passes, run the task's exit gate.
+Compilation is never the final gate for an extension: install and import it in
+the owned CPython runtime and exercise at least one native behavior.
+
+## 6. Record and advance
+
+Update `STATE.md` with compact evidence. If the task exit gate passes, set its
+checkboxes, point `Current task` to the next task in `ROADMAP.md`, and continue
+if budget remains.
+
+## Stop conditions
+
+Stop and request a user decision when:
+
+- the next action requires credentials, deployment, paid infrastructure, or
+  publishing outside the repository;
+- fulfilling the task requires changing the ABI contract rather than fixing a
+  consumer of it;
+- an upstream license is unclear or incompatible;
+- a destructive action, history rewrite, or discard of existing work appears
+  necessary;
+- the same failure class survives three evidence-driven attempts;
+- a design choice changes public API or materially increases shipped runtime
+  size without an already-stated budget.
+
+When stopping, preserve logs and state the smallest exact decision needed.
+
+## Escalation packet for the user
+
+When a stop condition is reached, prepare a compact packet containing:
+
+- intended invariant;
+- exact command;
+- final 80-150 relevant log lines;
+- files and functions involved;
+- three attempted hypotheses and what disproved each;
+- the decision required.
+
+Do not send entire build logs or the full repository history. Wait for the
+user's decision, record it in `STATE.md`, and then resume this same loop.

package/docs/agent/README.md

@@ -0,0 +1,77 @@
+# SandboxedJS native Python package platform
+
+This folder is an executable work specification for a coding agent. Its goal is
+not to port NumPy. Its goal is to make native Python packages a platform
+capability of SandboxedJS, with NumPy eventually serving as one demanding
+acceptance case.
+
+## Start here
+
+At the beginning of every turn, read only these files:
+
+1. `README.md` (this file).
+2. `STATE.md`.
+3. The one task file named by `STATE.md`.
+
+Do not reread every document on every turn. Open `INVARIANTS.md`,
+`DECISION-TREE.md`, or `COMMANDS.md` only when the current task points to them
+or an observed failure requires them. This is intentional: the folder is also
+a context and token budget.
+
+Then execute the loop in `LOOP.md`. Continue until the current task's exit gate
+passes or a stop condition requires the user or a stronger model.
+
+## Product objective
+
+The eventual user experience is:
+
+```sh
+pip install some-package
+```
+
+The implementation may obtain a pure wheel, install an already-built
+SandboxedJS wheel, or ask an external build system to create and cache a wheel.
+It must never install a host-native artifact by relabeling it, silently omit a
+dependency, or claim support because compilation alone succeeded.
+
+## Current baseline
+
+The working tree already contains staged, uncommitted work produced in an
+earlier session:
+
+- a generic setuptools cross environment;
+- package-owned `setup.py` and `pyproject.toml` builds;
+- C and Cython fixtures;
+- the existing PyO3 path;
+- side-module validation and runtime import tests;
+- `docs/python/cross-build.md`, which records ten known assumptions.
+
+Preserve those staged changes. Do not unstage, discard, rewrite wholesale, or
+commit them unless the user explicitly requests it. First verify them and
+record the result in `STATE.md`.
+
+## Scope boundary
+
+This project owns:
+
+- reproducible cross-build inputs;
+- build/host/target dependency separation;
+- package build-backend execution;
+- ABI-valid wheel construction and metadata;
+- wheel indexing, resolution, installation, and runtime verification;
+- explicit recipes and patches for irreducibly package-specific behavior;
+- compatibility evidence.
+
+This project does not own, unless the user separately authorizes it:
+
+- a hosted public build service;
+- cloud accounts, deployment, billing, or package signing infrastructure;
+- pretending raw sockets, subprocesses, GPU APIs, or unavailable operating
+  system capabilities exist;
+- broad unrelated cleanup of SandboxedJS.
+
+## Definition of success
+
+Success is a general pipeline demonstrated by several independent package
+shapes, not one famous package. The final gate is described in `ROADMAP.md`.
+

package/docs/agent/ROADMAP.md

@@ -0,0 +1,37 @@
+# Capability roadmap
+
+Advance only when the current task's exit gate passes. Each task must leave a
+focused regression test.
+
+| Order | Task | Capability produced |
+| --- | --- | --- |
+| 00 | `tasks/00-verify-inherited-work.md` | trusted baseline for the staged generic builder |
+| 01 | `tasks/01-authoritative-metadata.md` | wheel metadata derived from the package |
+| 02 | `tasks/02-native-dependencies.md` | declared target libraries and profile sysroot |
+| 03 | `tasks/03-reproducible-inputs.md` | hashed sources, build tools, patches, provenance |
+| 04 | `tasks/04-build-frontends.md` | explicit backend interface beyond setuptools/PyO3 |
+| 05 | `tasks/05-registry-integration.md` | reproducible build output consumable by `pip` |
+| 06 | `tasks/06-package-cohorts.md` | evidence across independent package shapes |
+| 07 | `tasks/07-build-on-miss-boundary.md` | local protocol/spec for an eventual build service |
+
+## Platform completion gate
+
+The platform milestone is complete when all of the following are true:
+
+- Pure wheels continue to install directly and transactionally.
+- Three native language/build shapes work without package-name branches.
+- At least one extension links a separately built native target library.
+- At least one package needs and cleanly carries a visible target patch.
+- Wheel metadata and dependency resolution come from authoritative package
+  metadata plus explicit target constraints.
+- Every indexed artifact is source-pinned, hash-verifiable, ABI-identified,
+  installable, importable, and behavior-tested in the owned runtime.
+- A compatibility report distinguishes supported, recipe-required,
+  platform-blocked, and not-yet-attempted packages.
+- The external build-on-miss interface is specified without embedding a
+  compiler toolchain in the browser runtime.
+
+NumPy may be attempted after tasks 00-05. It is evidence for native dependency,
+build-probe, memory, and scientific-stack behavior; it is not itself the
+definition of platform completion.
+

package/docs/agent/STATE.md

@@ -0,0 +1,93 @@
+# Execution state
+
+Keep under roughly 120 lines. Replace stale detail instead of appending.
+
+## Status
+
+- Phase: `complete`
+- Blocker: none
+- User decision required: none outstanding
+
+## Repository layout (restructured)
+
+    src/
+      kernel/ fs/ net/ shell/     the linux-mimicking core
+      node/      module shims, commonjs engine, esm transform
+      python/    cpython, syscalls, pip, resolver, worker, abi, local-builder
+      worker/    pods, sync channel, worker host
+      tools/     esbuild, rolldown, ffmpeg hosts
+      container/ pkg/ agent/ preview/ hosting/
+    docs/        every markdown file: agent/, python/, handoff.md
+    python-runtime/  the Python build pipeline (Makefile, scripts, recipes)
+
+`src/runtime/` is gone. Moves were done with `git mv` and every relative import
+recomputed programmatically. Four references were strings rather than import
+specifiers -- two `dist/` paths whose depth changed, a `/src/runtime/python/`
+check in config.ts, and a test allowlist -- and those caused 79 failures until
+found. There was no dead code to remove: 2 of 130 source files are unreferenced
+and both are ambient `.d.ts` files tsconfig loads implicitly.
+
+## Packages: nothing to pre-build any more
+
+`pip install <anything>` now builds a wheel when the index has none.
+
+- `scripts/auto_recipe.py` writes a recipe from what PyPI and the source
+  already state: identity, verified sdist, and the declared PEP 517 backend.
+  Backends with no adapter are refused by name. Build requirements the lock has
+  never seen are pinned at that moment rather than dropped.
+- `src/python/local-builder.ts` runs the pipeline; `bin/sandboxedjs-build-wheels.mjs`
+  serves the index and builds on request, for browsers that have no compiler.
+- Opt-in, defaulting on only in Node from a checkout. Submit-and-poll, because
+  a connection held open for a minutes-long build reads as an unreachable
+  service.
+- Verified: `pip install ujson` with no recipe -> generated, built, indexed,
+  installed, imported, both locally and through the service.
+
+## Two real defects found and fixed
+
+- The browser's default wheel index was filtered to the wheels whose bytes are
+  inlined in the bundle, so every other built wheel was invisible -- `pip
+  install numpy` reported no usable distribution while numpy sat unreferenced
+  beside the interpreter. The index now lists all built wheels, served from
+  next to the interpreter, with the inlined bytes as an overlay keyed by
+  filename.
+- The post-build index read went through pip's memoizing cache and returned the
+  pre-build copy, so a package was built twice.
+
+## Pre-existing failures, verified not caused by this work
+
+11 tests, reproduced identically in a clean worktree at HEAD: `python-syscalls`
+(5), `cpython` (2), `python-abi/baseline` (3), `python-runtime/asyncio` (1).
+Also: pytest's default output capture fails in the container
+(`OSError: [Errno 8] Bad file descriptor`), so upstream suites need
+`--capture=no`.
+
+## Unverified
+
+`import numpy` in a browser. The embedded browser used for testing blocks
+nested module workers loaded over HTTP, which the runtime needs to start Python
+at all -- `python3 --version` times out there too, and so does a trivial nested
+worker. `pip install numpy` was verified in that browser and returns 0.
+
+## Last evidence
+
+`npx vitest run`: 610 passed, 24 skipped, 12 failed (the 11 above plus one
+PyPI-network flake that passes in isolation). `npm run typecheck`: pass.
+`npm run build`: pass. All 16 wheels byte-identical across rebuilds.
+
+## Compact checkpoint template## Compact checkpoint template## Compact checkpoint template## Compact checkpoint template## Compact checkpoint template## Compact checkpoint template## Compact checkpoint template## Compact checkpoint template## Compact checkpoint template
+
+Replace the sections above after each meaningful loop:
+
+```text
+Phase: <name>
+Current task: <path>
+Last completed task: <path or none>
+Blocker: <one sentence or none>
+User decision required: <yes/no and exact decision>
+Last evidence:
+- <command>: <pass/fail and decisive result>
+Next action: <one concrete action>
+Files changed this loop: <paths>
+```
+

package/docs/agent/tasks/00-verify-inherited-work.md

@@ -0,0 +1,36 @@
+# Task 00: verify inherited generic cross-build work
+
+## Purpose
+
+Establish that the staged setuptools/Cython work is real and reproducible
+before building new layers on it. Do not expand scope in this task.
+
+## Procedure
+
+1. Inspect `git status --short`, `git diff --cached --check`, and the staged
+   diff limited to the files listed in `STATE.md`.
+2. Confirm recipes cannot provide ABI compiler or linker flags.
+3. Confirm generated artifacts are checked for WebAssembly, `dylink`, and the
+   derived `PyInit_*` symbol.
+4. Run the setuptools and Cython fixture builds. Rebuild the wheel index after
+   any wheel output changes.
+5. Run `npm run build` only if runtime bundles required by the acceptance test
+   are absent or older than relevant inputs.
+6. Run `npx vitest run test/python-runtime/extensions.test.ts`.
+7. Run `npm run typecheck`.
+
+## Exit gate
+
+- Staged diff check passes.
+- Setuptools C, Cython, and PyO3 wheels install and execute native behavior in
+  the owned runtime.
+- Pydantic acceptance remains green.
+- No test passed by weakening or skipping an ABI assertion.
+- `STATE.md` contains exact passing commands and points to task 01.
+
+## If it fails
+
+Use `DECISION-TREE.md`. Fix only regressions within the inherited work. After
+three failed evidence-driven attempts on the same class, prepare an escalation
+packet instead of redesigning the ABI.
+

package/docs/agent/tasks/01-authoritative-metadata.md

@@ -0,0 +1,34 @@
+# Task 01: authoritative wheel metadata
+
+## Problem
+
+The current builder synthesizes `METADATA` from recipe fields. That duplicates
+the package's dependency declarations and can create a wheel that installs but
+resolves the wrong environment.
+
+## Deliverable
+
+Generate distribution metadata using the package's build metadata path under
+the controlled build environment. Preserve upstream `Name`, `Version`,
+`Requires-Dist`, extras, Python requirements, entry points, licenses, and other
+install-relevant files. SandboxedJS should add only its wheel tag, generator,
+and `Build-ABI` assertion.
+
+Recipe constraints may explicitly override a target-inapplicable dependency,
+but every override must be visible, justified, and tested.
+
+## Required tests
+
+- A fixture declares a conditional and an extra dependency upstream; neither
+  is copied into the recipe.
+- The resulting wheel and generated `index.json` contain the authoritative
+  requirements.
+- The resolver respects at least one marker/extras case.
+- A disagreement between recipe identity and upstream identity fails loudly.
+- Existing extension runtime tests remain green.
+
+## Exit gate
+
+No generic setuptools fixture needs handwritten `requires` metadata, and the
+index is provably derived from the wheel produced from upstream metadata.
+

package/docs/agent/tasks/02-native-dependencies.md

@@ -0,0 +1,35 @@
+# Task 02: native target dependencies and sysroot
+
+## Problem
+
+Extensions that name libraries or include their headers cannot currently
+declare, build, locate, or link those target dependencies generically.
+
+## Design requirement
+
+Represent native target dependencies separately from Python runtime
+dependencies and host build requirements. Each dependency must identify a
+pinned source, hash, build recipe, output markers, license/provenance, and ABI
+profile. The dynamic profile uses `out/sysroot-dynamic`; incompatible profiles
+must never share object archives.
+
+Add sysroot include and library search paths through generated cross-build
+configuration, not ad hoc recipe flags.
+
+## Reduced acceptance fixture
+
+Before attempting a large external library, create a tiny static target library
+with a header and one function. Build/install it through the dependency system,
+then build a setuptools extension that declares the library normally and calls
+it. The wheel must install and execute the function in the owned runtime.
+
+After the reduced fixture passes, add one small real native library already
+compatible with the project architecture. Do not begin with BLAS or NumPy.
+
+## Exit gate
+
+- Dependency closure is deterministic and cycle/error aware.
+- Rebuilds are skipped only when validated output markers match inputs/profile.
+- Missing dependency and wrong-profile reuse have focused negative tests.
+- Reduced and real-library extensions import and run in owned CPython.
+

package/docs/agent/tasks/03-reproducible-inputs.md

@@ -0,0 +1,27 @@
+# Task 03: reproducible inputs, patches, and provenance
+
+## Deliverable
+
+Unify source archives, build-tool wheels, native dependencies, and package
+patches under verified acquisition. No release artifact may depend on an
+unpinned `pip install setuptools wheel ...` result.
+
+Define a visible patch layout. A patch records package/version, upstream source
+hash, reason, target fact being substituted, and a test. Patch application must
+fail on drift rather than fuzzily succeeding against another source version.
+
+Record enough provenance in build output or adjacent index metadata to answer:
+
+- which source hash produced this wheel;
+- which build tools and target libraries were used;
+- which ABI ID and recipe revision were used;
+- which patches were applied.
+
+## Exit gate
+
+- Offline rebuild succeeds from a populated verified cache.
+- A changed hash, unpinned build tool, or drifting patch fails before compile.
+- Two identical clean builds produce equivalent wheel contents, allowing only
+  explicitly documented archive timestamp normalization until deterministic
+  ZIP output is implemented.
+

package/docs/agent/tasks/04-build-frontends.md

@@ -0,0 +1,27 @@
+# Task 04: build frontend interface
+
+## Purpose
+
+Turn the current kinds into a stable backend interface rather than accumulating
+branches in `build_extension.py`.
+
+## Required interface
+
+Each backend receives verified source, host tools, target configuration,
+declared native dependencies, staging directory, and ABI contract. It returns a
+wheel tree or wheel plus build evidence. It cannot mutate the ABI.
+
+Retain proven setuptools/Cython and PyO3 behavior. Add another frontend only
+when a selected cohort package requires it; likely candidates include PEP 517
+wheel hooks, Meson, or scikit-build/CMake. Do not implement all frontends
+speculatively.
+
+Keep package quirks in patches/recipes and backend mechanics in adapters.
+
+## Exit gate
+
+- Backend selection is explicit and schema-validated.
+- The generic orchestrator has no package-name checks.
+- Existing fixtures pass through the interface.
+- One pyproject-only package runs through a real isolated PEP 517 path.
+

package/docs/agent/tasks/05-registry-integration.md

@@ -0,0 +1,27 @@
+# Task 05: registry and installer integration
+
+## Deliverable
+
+Produce an immutable wheel repository that the existing SandboxedJS `pip`
+resolver can consume. Define schema versioning, atomic publication layout,
+digest verification, ABI partitioning, and dependency metadata.
+
+The browser runtime remains an installer, not a compiler. A missing native
+wheel should return a structured, actionable classification that a host may use
+later to request a build. It must not perform an implicit remote mutation.
+
+## Required tests
+
+- Correct ABI wheel wins over incompatible and host-native candidates.
+- Wrong ABI and wrong digest are refused before installation.
+- A failed graph leaves no partial site-packages state.
+- A newly built test wheel can be indexed, served, installed, imported, and
+  exercised through the ordinary container command.
+- Index publication cannot expose a wheel before its metadata and content are
+  complete.
+
+## Exit gate
+
+A clean consumer needs only a runtime plus index URL/object; no local source
+tree or compiler is required for supported packages.
+

package/docs/agent/tasks/06-package-cohorts.md

@@ -0,0 +1,42 @@
+# Task 06: representative package cohorts
+
+## Purpose
+
+Measure generality without allowing one package to define the architecture.
+
+## Selection method
+
+Choose small packages first, pin exact versions and hashes, and record why each
+adds a new capability. Select at least one from each applicable cohort:
+
+1. pure wheel control case;
+2. setuptools C extension without external target libraries;
+3. generated C/Cython extension;
+4. Rust/PyO3 extension;
+5. extension using a separately built native target library;
+6. package requiring a visible cross-compilation patch;
+7. pyproject/PEP 517 backend not already covered;
+8. scientific-stack stress case, eventually including NumPy.
+
+Do not count existing probes as real-package cohort evidence, but retain them as
+reduced regression tests.
+
+## Compatibility report
+
+For every attempted package, record exactly one status:
+
+- `supported-generic`;
+- `supported-recipe`;
+- `blocked-platform-capability`;
+- `blocked-toolchain`;
+- `not-yet-attempted`.
+
+Record build command, source hash, wheel hash, ABI ID, test behavior, peak size
+or memory when material, and the earliest unsupported assumption on failure.
+
+## Exit gate
+
+At least five independent real packages across four native/build shapes pass
+runtime behavior tests, and failures produce reusable classifications instead
+of package-specific branches.
+

package/docs/agent/tasks/07-build-on-miss-boundary.md

@@ -0,0 +1,31 @@
+# Task 07: external build-on-miss boundary
+
+## Purpose
+
+Specify the seam that can later connect the installer to a trusted build
+service without deploying that service or placing a toolchain in the browser.
+
+## Deliverable
+
+Define versioned request/status/result schemas. A request includes normalized
+requirement, source/index evidence, target ABI ID, policy/recipe revision, and
+an idempotency key. A result includes immutable wheel/index locations, hashes,
+provenance, logs summary, compatibility classification, and terminal state.
+
+States should cover queued, resolving, building host tools, building target
+dependencies, building wheel, testing, published, unsupported, failed, and
+cancelled. Repeated identical requests must converge on one cached artifact.
+
+Security requirements include source allowlists/policy, resource limits,
+network isolation, no untrusted artifact publication before runtime tests,
+digest verification, and auditability.
+
+## Exit gate
+
+- Schemas and a local fake implementation are tested.
+- Installer/host integration can report a buildable miss without automatically
+  triggering external work.
+- Cached success, unsupported result, transient failure, cancellation, and ABI
+  rollover are covered.
+- No cloud service is deployed without explicit user authorization.
+