marin-shellbox 0.2.151.dev37846966940__tar.gz → 0.2.153.dev37866254792__tar.gz

marin_shellbox-0.2.151.dev37846966940/README.md → marin_shellbox-0.2.153.dev37866254792/PKG-INFO

@@ -1,3 +1,23 @@
+Metadata-Version: 2.5
+Name: marin-shellbox
+Version: 0.2.153.dev37866254792
+Summary: Shell execution machines with local and remote backends
+Requires-Python: >=3.12
+Requires-Dist: bubblewrap-bin==0.11.0; sys_platform == 'linux'
+Requires-Dist: httpx>=0.27
+Provides-Extra: daytona
+Requires-Dist: daytona==0.200.2; extra == 'daytona'
+Requires-Dist: marin-rigging>=0.2.0; extra == 'daytona'
+Provides-Extra: gvisor
+Provides-Extra: iris
+Requires-Dist: marin-iris==0.2.153.dev37866254792; extra == 'iris'
+Provides-Extra: qemu
+Requires-Dist: quicksand-qemu<0.6,>=0.5.12; extra == 'qemu'
+Provides-Extra: shellsim
+Requires-Dist: marin-rigging>=0.2.0; extra == 'shellsim'
+Requires-Dist: shellsim<0.2,>=0.1.29; extra == 'shellsim'
+Description-Content-Type: text/markdown
+
 # Marin Shellbox (0.1)
 
 `marin-shellbox` provides three Harbor import paths:
@@ -88,8 +108,7 @@ these settings to its resource flags. Storage limits require a Docker storage
 driver that supports `--storage-opt size`, and GPU allocation requires a GPU
 runtime. Daytona applies CPU, memory, storage, and GPU settings through its SDK.
 It rounds memory and storage up to whole GiB. Iris applies CPU, memory, and storage settings. QEMU applies CPU and
-memory settings. Backends reject resource overrides that they cannot apply, except that the local backend ignores
-`memory_mb`.
+memory settings. Backends reject resource overrides that they cannot apply.
 
 `Command.user` selects the execution user for one command. Docker accepts a
 username or UID string. An omitted user retains the image's user. ShellSim and
@@ -157,9 +176,11 @@ With the `qemu` extra, `quicksand_qemu.get_bin_dir()` gives the QEMU executable
 ```python
 from pathlib import Path
 from shellbox.backends.local.machine import LocalMachineFactory
+from shellbox.backends.local.python_environment import build_python_environment
 from shellbox.machine import Command, HostImage, MachineSpec
 
-factory = LocalMachineFactory(bin_dirs=(Path("/opt/grader-venv/bin"),))
+environment = build_python_environment(Path("/var/tmp/grader-env"), Path("grader.lock"), "3.12.13")
+factory = LocalMachineFactory(read_only=(environment.root,), bin_dirs=(environment.bin_dir,))
 machine = await factory.create(MachineSpec(HostImage(), workdir="/app"))
 try:
     result = await machine.run(Command(("python3", "/tests/grade.py")))
@@ -167,16 +188,20 @@ finally:
     await machine.close()
 ```
 
-- Each machine has a root directory of its own, which `create` makes empty under the host's temporary directory and `close` removes. Commands see it as `/`, with the host's `/usr`, `/bin`, `/sbin`, `/lib*`, `/etc`, `/opt`, `/sys` and `/run/systemd/resolve` mounted read-only over it, together with each venv in `bin_dirs` and the Python installation its `python3` links to. Every other path, including `/app`, `/tests`, `/tmp` and `HOME`, is the machine's own, so paths that the host process uses, such as an Iris task's `/app`, are never touched. No other host file is visible.
+- Each machine has a root directory of its own, which `create` makes empty under the host's temporary directory and `close` removes. Commands see it as `/`, with the host's `/usr`, `/bin`, `/sbin`, `/lib*`, `/etc`, `/opt`, `/sys` and `/run/systemd/resolve` mounted read-only over it, together with each directory in `read_only` at its own host path. Every other path, including `/app`, `/tests`, `/tmp` and `HOME`, is the machine's own, so paths that the host process uses, such as an Iris task's `/app`, are never touched. No other host file is visible.
 - Machines are independent, so a process may run any number of them at once.
 - Files persist in the machine's root between commands. Each command gets new PID, IPC and UTS namespaces and a fresh `/proc`, `/dev` and `/dev/shm`; a command's processes end when it exits or times out. Under `NetworkPolicy.DENY` it also gets a network namespace with only loopback, which refuses TCP and UDP to the host and beyond; under `ALLOW` it uses the host network.
 - `upload` copies into the machine's root and rejects targets in the read-only directories. `download` reads the machine's root or the read-only host directories; a missing path raises `RuntimeError`, as the container backends do.
-- Commands never inherit the host environment, which can hold cluster credentials. Each command gets `PATH` with `bin_dirs` ahead of the standard system directories, `HOME=/home/shellbox`, `LANG=C.UTF-8`, and the host's `PYTHONHASHSEED` if set, then `MachineSpec.env` and `Command.env`.
+- Commands never inherit the host environment, which can hold cluster credentials. Each command gets `PATH` with `bin_dirs` ahead of the standard system directories, `HOME=/home/shellbox`, `LANG=C.UTF-8`, and `PYTHONHASHSEED` when the factory has a `hash_seed`, then `MachineSpec.env` and `Command.env`.
 - `/bin/sh` enters the working directory and starts the program, so a missing program exits with status 127, as in the container backends. Output beyond `output_limit_bytes` is discarded while the command runs.
 - Commands have no capabilities and run with `no_new_privs`, so setuid programs cannot raise their privileges. They have no core files, files of at most 1 GiB, and, for a command with a timeout, at most the timeout multiplied by the usable CPUs plus 5 seconds of CPU time per process. `RLIMIT_NPROC` counts all of a user's tasks on the host, so the backend limits the command's user to the host's task count when the command starts plus 256; the kernel does not apply it to root.
-- `memory_mb` is ignored: an address-space limit breaks NumPy and SymPy allocations. CPU, storage, and GPU requests raise `UnsupportedMachineSpec`.
+- The backend enforces no memory limit, since an address-space limit breaks NumPy and SymPy allocations. `memory_mb`, CPU, storage, and GPU requests raise `UnsupportedMachineSpec`.
 - `Command.user` keeps the process's user when it is omitted or names that user. Another user requires a root process; the command then runs with that account's UID and primary group, no supplementary groups and no capabilities. A user without a host account raises `UnsupportedMachineSpec`.
 
+The factory mounts exactly the directories in `read_only` and infers nothing from `bin_dirs`, which only sets `PATH`. The caller must keep each `read_only` directory self-contained. A symlink whose target lies outside the mounted directories does not resolve in the sandbox. A uv venv, for example, links `python` to an interpreter elsewhere on the host, so mounting the venv alone leaves `python3` missing. The factory raises `ValueError` for a `read_only` path that is not an existing directory.
+
+`build_python_environment(root, lock, python_version)` builds such a directory for Python. It installs a uv-managed CPython of the given full version under `root/python`, creates a venv at `root/venv` from it, and runs `uv pip sync --require-hashes` with the lock, a requirements file with hashes such as `uv pip compile --generate-hashes` writes. Every file and symlink of the environment lies under `root`, and the venv refers to its interpreter by absolute path, so the root must stay where it was built. It returns a `PythonEnvironment` whose `root` goes in `read_only` and whose `bin_dir` goes in `bin_dirs`. Concurrent builders of one root on a host take an exclusive `flock` on a file beside it, so the environment is built once. A completion marker records a finished build; a root without one is a failed or interrupted build and is removed and rebuilt. A finished root built from another lock or Python version raises `ValueError`, and a failed uv command raises `UvError` with uv's error output. The builder needs `uv` on `PATH` and network access to download the interpreter and packages.
+
 The factory checks once, when it is created, that `bwrap` can build a sandbox, and raises `SandboxUnavailable` with each candidate's error when it cannot. It tries the bundled binary, then any `bwrap` on `PATH`, or only the `bwrap` path it is given. Run as root, `bwrap` creates its namespaces without a user namespace, which needs `CAP_SYS_ADMIN` and `CAP_NET_ADMIN`, no AppArmor confinement, and a seccomp filter that allows `pivot_root`. Container runtimes withhold all three by default. On Kubernetes, the container's `securityContext` needs `capabilities.add: [SYS_ADMIN, NET_ADMIN]` and `appArmorProfile: {type: Unconfined}`, and no `RuntimeDefault` seccomp profile. Run as another user, `bwrap` needs unprivileged user namespaces; Ubuntu restricts them to programs whose AppArmor profile allows them, including `/usr/bin/bwrap`, so there the factory uses the system binary.
 
 ## ShellSim backend

marin_shellbox-0.2.151.dev37846966940/PKG-INFO → marin_shellbox-0.2.153.dev37866254792/README.md

@@ -1,23 +1,3 @@
-Metadata-Version: 2.5
-Name: marin-shellbox
-Version: 0.2.151.dev37846966940
-Summary: Shell execution machines with local and remote backends
-Requires-Python: >=3.12
-Requires-Dist: bubblewrap-bin==0.11.0; sys_platform == 'linux'
-Requires-Dist: httpx>=0.27
-Provides-Extra: daytona
-Requires-Dist: daytona==0.200.2; extra == 'daytona'
-Requires-Dist: marin-rigging>=0.2.0; extra == 'daytona'
-Provides-Extra: gvisor
-Provides-Extra: iris
-Requires-Dist: marin-iris==0.2.151.dev37846966940; extra == 'iris'
-Provides-Extra: qemu
-Requires-Dist: quicksand-qemu<0.6,>=0.5.12; extra == 'qemu'
-Provides-Extra: shellsim
-Requires-Dist: marin-rigging>=0.2.0; extra == 'shellsim'
-Requires-Dist: shellsim<0.2,>=0.1.29; extra == 'shellsim'
-Description-Content-Type: text/markdown
-
 # Marin Shellbox (0.1)
 
 `marin-shellbox` provides three Harbor import paths:
@@ -108,8 +88,7 @@ these settings to its resource flags. Storage limits require a Docker storage
 driver that supports `--storage-opt size`, and GPU allocation requires a GPU
 runtime. Daytona applies CPU, memory, storage, and GPU settings through its SDK.
 It rounds memory and storage up to whole GiB. Iris applies CPU, memory, and storage settings. QEMU applies CPU and
-memory settings. Backends reject resource overrides that they cannot apply, except that the local backend ignores
-`memory_mb`.
+memory settings. Backends reject resource overrides that they cannot apply.
 
 `Command.user` selects the execution user for one command. Docker accepts a
 username or UID string. An omitted user retains the image's user. ShellSim and
@@ -177,9 +156,11 @@ With the `qemu` extra, `quicksand_qemu.get_bin_dir()` gives the QEMU executable
 ```python
 from pathlib import Path
 from shellbox.backends.local.machine import LocalMachineFactory
+from shellbox.backends.local.python_environment import build_python_environment
 from shellbox.machine import Command, HostImage, MachineSpec
 
-factory = LocalMachineFactory(bin_dirs=(Path("/opt/grader-venv/bin"),))
+environment = build_python_environment(Path("/var/tmp/grader-env"), Path("grader.lock"), "3.12.13")
+factory = LocalMachineFactory(read_only=(environment.root,), bin_dirs=(environment.bin_dir,))
 machine = await factory.create(MachineSpec(HostImage(), workdir="/app"))
 try:
     result = await machine.run(Command(("python3", "/tests/grade.py")))
@@ -187,16 +168,20 @@ finally:
     await machine.close()
 ```
 
-- Each machine has a root directory of its own, which `create` makes empty under the host's temporary directory and `close` removes. Commands see it as `/`, with the host's `/usr`, `/bin`, `/sbin`, `/lib*`, `/etc`, `/opt`, `/sys` and `/run/systemd/resolve` mounted read-only over it, together with each venv in `bin_dirs` and the Python installation its `python3` links to. Every other path, including `/app`, `/tests`, `/tmp` and `HOME`, is the machine's own, so paths that the host process uses, such as an Iris task's `/app`, are never touched. No other host file is visible.
+- Each machine has a root directory of its own, which `create` makes empty under the host's temporary directory and `close` removes. Commands see it as `/`, with the host's `/usr`, `/bin`, `/sbin`, `/lib*`, `/etc`, `/opt`, `/sys` and `/run/systemd/resolve` mounted read-only over it, together with each directory in `read_only` at its own host path. Every other path, including `/app`, `/tests`, `/tmp` and `HOME`, is the machine's own, so paths that the host process uses, such as an Iris task's `/app`, are never touched. No other host file is visible.
 - Machines are independent, so a process may run any number of them at once.
 - Files persist in the machine's root between commands. Each command gets new PID, IPC and UTS namespaces and a fresh `/proc`, `/dev` and `/dev/shm`; a command's processes end when it exits or times out. Under `NetworkPolicy.DENY` it also gets a network namespace with only loopback, which refuses TCP and UDP to the host and beyond; under `ALLOW` it uses the host network.
 - `upload` copies into the machine's root and rejects targets in the read-only directories. `download` reads the machine's root or the read-only host directories; a missing path raises `RuntimeError`, as the container backends do.
-- Commands never inherit the host environment, which can hold cluster credentials. Each command gets `PATH` with `bin_dirs` ahead of the standard system directories, `HOME=/home/shellbox`, `LANG=C.UTF-8`, and the host's `PYTHONHASHSEED` if set, then `MachineSpec.env` and `Command.env`.
+- Commands never inherit the host environment, which can hold cluster credentials. Each command gets `PATH` with `bin_dirs` ahead of the standard system directories, `HOME=/home/shellbox`, `LANG=C.UTF-8`, and `PYTHONHASHSEED` when the factory has a `hash_seed`, then `MachineSpec.env` and `Command.env`.
 - `/bin/sh` enters the working directory and starts the program, so a missing program exits with status 127, as in the container backends. Output beyond `output_limit_bytes` is discarded while the command runs.
 - Commands have no capabilities and run with `no_new_privs`, so setuid programs cannot raise their privileges. They have no core files, files of at most 1 GiB, and, for a command with a timeout, at most the timeout multiplied by the usable CPUs plus 5 seconds of CPU time per process. `RLIMIT_NPROC` counts all of a user's tasks on the host, so the backend limits the command's user to the host's task count when the command starts plus 256; the kernel does not apply it to root.
-- `memory_mb` is ignored: an address-space limit breaks NumPy and SymPy allocations. CPU, storage, and GPU requests raise `UnsupportedMachineSpec`.
+- The backend enforces no memory limit, since an address-space limit breaks NumPy and SymPy allocations. `memory_mb`, CPU, storage, and GPU requests raise `UnsupportedMachineSpec`.
 - `Command.user` keeps the process's user when it is omitted or names that user. Another user requires a root process; the command then runs with that account's UID and primary group, no supplementary groups and no capabilities. A user without a host account raises `UnsupportedMachineSpec`.
 
+The factory mounts exactly the directories in `read_only` and infers nothing from `bin_dirs`, which only sets `PATH`. The caller must keep each `read_only` directory self-contained. A symlink whose target lies outside the mounted directories does not resolve in the sandbox. A uv venv, for example, links `python` to an interpreter elsewhere on the host, so mounting the venv alone leaves `python3` missing. The factory raises `ValueError` for a `read_only` path that is not an existing directory.
+
+`build_python_environment(root, lock, python_version)` builds such a directory for Python. It installs a uv-managed CPython of the given full version under `root/python`, creates a venv at `root/venv` from it, and runs `uv pip sync --require-hashes` with the lock, a requirements file with hashes such as `uv pip compile --generate-hashes` writes. Every file and symlink of the environment lies under `root`, and the venv refers to its interpreter by absolute path, so the root must stay where it was built. It returns a `PythonEnvironment` whose `root` goes in `read_only` and whose `bin_dir` goes in `bin_dirs`. Concurrent builders of one root on a host take an exclusive `flock` on a file beside it, so the environment is built once. A completion marker records a finished build; a root without one is a failed or interrupted build and is removed and rebuilt. A finished root built from another lock or Python version raises `ValueError`, and a failed uv command raises `UvError` with uv's error output. The builder needs `uv` on `PATH` and network access to download the interpreter and packages.
+
 The factory checks once, when it is created, that `bwrap` can build a sandbox, and raises `SandboxUnavailable` with each candidate's error when it cannot. It tries the bundled binary, then any `bwrap` on `PATH`, or only the `bwrap` path it is given. Run as root, `bwrap` creates its namespaces without a user namespace, which needs `CAP_SYS_ADMIN` and `CAP_NET_ADMIN`, no AppArmor confinement, and a seccomp filter that allows `pivot_root`. Container runtimes withhold all three by default. On Kubernetes, the container's `securityContext` needs `capabilities.add: [SYS_ADMIN, NET_ADMIN]` and `appArmorProfile: {type: Unconfined}`, and no `RuntimeDefault` seccomp profile. Run as another user, `bwrap` needs unprivileged user namespaces; Ubuntu restricts them to programs whose AppArmor profile allows them, including `/usr/bin/bwrap`, so there the factory uses the system binary.
 
 ## ShellSim backend

{marin_shellbox-0.2.151.dev37846966940 → marin_shellbox-0.2.153.dev37866254792}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "marin-shellbox"
-version = "0.2.151.dev37846966940"
+version = "0.2.153.dev37866254792"
 description = "Shell execution machines with local and remote backends"
 readme = "README.md"
 requires-python = ">=3.12"
@@ -16,7 +16,7 @@ shellsim = ["shellsim>=0.1.29,<0.2", "marin-rigging>=0.2.0"]
 gvisor = []
 daytona = ["daytona==0.200.2", "marin-rigging>=0.2.0"]
 iris = [
-    "marin-iris==0.2.151.dev37846966940",
+    "marin-iris==0.2.153.dev37866254792",
 ]
 
 [project.scripts]

{marin_shellbox-0.2.151.dev37846966940 → marin_shellbox-0.2.153.dev37866254792}/src/shellbox/backends/local/machine.py

@@ -97,63 +97,11 @@ def _copy(source: Path, target: Path) -> None:
     shutil.copy2(source, target)
 
 
-def _interpreter_roots(bin_dirs: Iterable[Path]) -> tuple[str, ...]:
-    """Directories to read for the programs in ``bin_dirs``: each venv and the Python it links to, or the directory."""
-    roots: list[str] = []
-    for directory in bin_dirs:
-        venv = directory.parent
-        if (venv / "pyvenv.cfg").exists():
-            roots += [str(venv), str(Path(os.path.realpath(directory / "python3")).parent.parent)]
-        else:
-            roots.append(str(directory))
-    return tuple(roots)
-
-
-MAX_LINK_HOPS = 40
-
-
-def _link_chain(path: Path) -> list[tuple[str, str]]:
-    """Every symlink met while resolving ``path``, as ``(link, target)`` pairs in resolution order."""
-    links: list[tuple[str, str]] = []
-    parts = list(path.parts[1:])
-    current = PurePosixPath("/")
-    while parts:
-        current = current / parts.pop(0)
-        if not os.path.islink(current):
-            continue
-        if len(links) >= MAX_LINK_HOPS:
-            raise RuntimeError(f"{path} has more than {MAX_LINK_HOPS} symlink hops")
-        target = os.readlink(current)
-        links.append((str(current), target))
-        resolved = PurePosixPath(os.path.normpath(current.parent / target))
-        parts = [*resolved.parts[1:], *parts]
-        current = PurePosixPath("/")
-    return links
-
-
-def _interpreter_links(bin_dirs: Iterable[Path], roots: Iterable[str]) -> tuple[tuple[str, str], ...]:
-    """Symlinks on the way from each venv's ``python3`` to its interpreter that lie outside the mounted ``roots``.
-
-    uv links a venv's interpreter through a minor-version alias directory beside the install it resolves to;
-    the sandbox mounts the resolved install, so it must recreate the alias for the venv's link to resolve.
-    """
-    mounted = tuple(map(PurePosixPath, (*SYSTEM_DIRECTORIES, *roots)))
-    links = []
-    for directory in bin_dirs:
-        if not (directory.parent / "pyvenv.cfg").exists():
-            continue
-        for link, target in _link_chain(directory / "python3"):
-            if not _within(PurePosixPath(link), mounted) and (link, target) not in links:
-                links.append((link, target))
-    return tuple(links)
-
-
 def _sandbox_argv(
     bwrap: Path,
     *,
     root: Path,
     read_only: Iterable[str],
-    links: Iterable[tuple[str, str]] = (),
     network: NetworkPolicy,
     account: pwd.struct_passwd | None,
 ) -> list[str]:
@@ -177,9 +125,7 @@ def _sandbox_argv(
             argv += ["--ro-bind-try", path, path]
     argv += ["--dev", "/dev", "--tmpfs", SHARED_MEMORY, "--proc", "/proc"]
     for path in read_only:
-        argv += ["--ro-bind-try", path, path]
-    for link, target in links:
-        argv += ["--symlink", target, link]
+        argv += ["--ro-bind", path, path]
     if account is not None:
         argv += ["setpriv", f"--reuid={account.pw_uid}", f"--regid={account.pw_gid}", "--clear-groups", "--"]
     return argv
@@ -295,13 +241,11 @@ class LocalMachine:
         bwrap: Path,
         root: Path,
         read_only: tuple[str, ...],
-        links: tuple[tuple[str, str], ...],
         environment: dict[str, str],
     ):
         self.spec = spec
         self.root = root
         self.read_only = read_only
-        self._links = links
         self._bwrap = bwrap
         self._environment = environment
         self._closed = False
@@ -332,7 +276,6 @@ class LocalMachine:
             self._bwrap,
             root=self.root,
             read_only=self.read_only,
-            links=self._links,
             network=self.spec.network,
             account=_command_account(command.user),
         )
@@ -406,12 +349,16 @@ class LocalMachineFactory:
     """Run trusted commands on this host in bubblewrap sandboxes, any number of machines at a time.
 
     A machine's commands see the host's system directories (``/usr``, ``/etc``, ``/opt`` and the like)
-    and the venvs and interpreters of ``bin_dirs`` read-only. Every other path, including ``/tmp`` and
-    ``HOME``, lies in a root directory of the machine's own that starts empty and is removed by
-    ``close``; no other host file is visible. Commands never inherit the host's environment: they get
-    ``bin_dirs`` ahead of a standard ``PATH``, ``HOME``, ``LANG``, ``PYTHONHASHSEED`` when the factory has
-    a ``hash_seed``, and the spec's and command's variables. A spec with ``memory_mb`` is rejected, since
-    the backend enforces no memory limit.
+    and each directory in ``read_only`` at its own path, read-only. Every other path, including ``/tmp``
+    and ``HOME``, lies in a root directory of the machine's own that starts empty and is removed by
+    ``close``; no other host file is visible. The caller keeps each ``read_only`` directory self-contained:
+    a symlink that leads out of the mounted directories does not resolve in the sandbox.
+    ``build_python_environment`` builds such a directory for a Python interpreter and its packages.
+
+    Commands never inherit the host's environment: they get ``bin_dirs`` ahead of a standard ``PATH``,
+    ``HOME``, ``LANG``, ``PYTHONHASHSEED`` when the factory has a ``hash_seed``, and the spec's and
+    command's variables. ``bin_dirs`` only sets ``PATH``; a directory outside the mounted ones is not
+    visible to commands. A spec with ``memory_mb`` is rejected, since the backend enforces no memory limit.
 
     ``bwrap`` names the executable to use. By default the factory takes the first of the bundled bwrap
     and any bwrap on ``PATH`` that can build a sandbox on this host, and raises ``SandboxUnavailable``
@@ -420,12 +367,22 @@ class LocalMachineFactory:
 
     backend: Backend = Backend.LOCAL
 
-    def __init__(self, *, bin_dirs: tuple[Path, ...] = (), bwrap: Path | None = None, hash_seed: str | None = None):
+    def __init__(
+        self,
+        *,
+        read_only: tuple[Path, ...] = (),
+        bin_dirs: tuple[Path, ...] = (),
+        bwrap: Path | None = None,
+        hash_seed: str | None = None,
+    ):
+        self.read_only = tuple(directory.absolute() for directory in read_only)
+        missing = [str(directory) for directory in self.read_only if not directory.is_dir()]
+        if missing:
+            raise ValueError(f"Read-only directories do not exist: {', '.join(missing)}")
         self.bin_dirs = tuple(directory.absolute() for directory in bin_dirs)
         self.hash_seed = hash_seed
         self.bwrap = _working_bwrap(_bwrap_candidates(bwrap))
-        self._read_only = _interpreter_roots(self.bin_dirs)
-        self._links = _interpreter_links(self.bin_dirs, self._read_only)
+        self._read_only = tuple(map(str, self.read_only))
         logger.info("Local backend sandboxes commands with %s", self.bwrap)
 
     async def create(self, spec: MachineSpec) -> LocalMachine:
@@ -449,6 +406,4 @@ class LocalMachineFactory:
         }
         if self.hash_seed is not None:
             environment["PYTHONHASHSEED"] = self.hash_seed
-        return LocalMachine(
-            spec, bwrap=self.bwrap, root=root, read_only=self._read_only, links=self._links, environment=environment
-        )
+        return LocalMachine(spec, bwrap=self.bwrap, root=root, read_only=self._read_only, environment=environment)

marin_shellbox-0.2.153.dev37866254792/src/shellbox/backends/local/python_environment.py

@@ -0,0 +1,118 @@
+# Copyright The Marin Authors
+# SPDX-License-Identifier: Apache-2.0
+
+"""Build self-contained Python environments for local machines to mount read-only.
+
+An environment root holds a uv-managed CPython and a venv made from it, so every file and symlink the
+interpreter needs lies under the root. A ``LocalMachineFactory`` given the root in ``read_only`` and
+the venv's ``bin`` directory in ``bin_dirs`` runs that interpreter with no other host directory mounted.
+"""
+
+import fcntl
+import hashlib
+import os
+import shlex
+import shutil
+import subprocess
+from dataclasses import dataclass
+from pathlib import Path
+
+PYTHON_DIRECTORY = "python"
+VENV_DIRECTORY = "venv"
+COMPLETE_MARKER = ".complete"
+
+
+@dataclass(frozen=True)
+class PythonEnvironment:
+    """A venv at ``root/venv`` whose interpreter is installed at ``root/python``.
+
+    The venv links to its interpreter by absolute path, so the root must stay where it was built.
+    """
+
+    root: Path
+
+    @property
+    def bin_dir(self) -> Path:
+        return self.root / VENV_DIRECTORY / "bin"
+
+    @property
+    def python(self) -> Path:
+        return self.bin_dir / "python3"
+
+
+def _marker_text(lock: Path, python_version: str) -> str:
+    return f"python {python_version}\nlock sha256 {hashlib.sha256(lock.read_bytes()).hexdigest()}\n"
+
+
+class UvError(RuntimeError):
+    """A uv command failed; the message holds the command and uv's error output."""
+
+
+def _uv(*args: str, environment: dict[str, str] | None = None) -> None:
+    command = ["uv", *args]
+    result = subprocess.run(
+        command,
+        stderr=subprocess.PIPE,
+        text=True,
+        env=None if environment is None else {**os.environ, **environment},
+        check=False,
+    )
+    if result.returncode != 0:
+        raise UvError(f"{shlex.join(command)} exited with status {result.returncode}:\n{result.stderr}")
+
+
+def _build(environment: PythonEnvironment, lock: Path, python_version: str) -> None:
+    install_dir = {"UV_PYTHON_INSTALL_DIR": str(environment.root / PYTHON_DIRECTORY)}
+    # --no-bin keeps uv from linking the interpreter into the user's executable directory.
+    _uv("python", "install", "--no-config", "--no-bin", python_version, environment=install_dir)
+    _uv(
+        "venv",
+        "--no-config",
+        "--managed-python",
+        "--no-python-downloads",
+        "--python",
+        python_version,
+        str(environment.root / VENV_DIRECTORY),
+        environment=install_dir,
+    )
+    # Copying overrides a UV_LINK_MODE of symlink, which would point installed files into uv's cache.
+    _uv(
+        "pip",
+        "sync",
+        "--no-config",
+        "--python",
+        str(environment.python),
+        "--require-hashes",
+        "--link-mode",
+        "copy",
+        str(lock),
+    )
+
+
+def build_python_environment(root: Path, lock: Path, python_version: str) -> PythonEnvironment:
+    """Build a venv under ``root`` with CPython ``python_version`` and the packages of the hash lock ``lock``.
+
+    ``python_version`` is a full version such as ``3.12.13``, which uv downloads into ``root``. ``lock`` is a
+    requirements file with hashes for every package, as ``uv pip compile --generate-hashes`` writes.
+
+    Concurrent callers on one host build the root once: an exclusive ``flock`` beside it serializes
+    builders, and a completion marker records a finished build. A root without the marker is a failed
+    or interrupted build and is rebuilt. A finished root built from another lock or version raises
+    ``ValueError``. A failed uv command raises ``UvError`` with uv's error output.
+    """
+    root = root.absolute()
+    environment = PythonEnvironment(root)
+    expected = _marker_text(lock, python_version)
+    root.parent.mkdir(parents=True, exist_ok=True)
+    with (root.parent / f"{root.name}.lock").open("w") as guard:
+        fcntl.flock(guard, fcntl.LOCK_EX)
+        marker = root / COMPLETE_MARKER
+        if marker.exists():
+            if marker.read_text() != expected:
+                raise ValueError(f"{root} holds an environment built from other inputs:\n{marker.read_text()}")
+            return environment
+        if root.exists():
+            shutil.rmtree(root)
+        _build(environment, lock, python_version)
+        marker.write_text(expected)
+    return environment

{marin_shellbox-0.2.151.dev37846966940 → marin_shellbox-0.2.153.dev37866254792}/src/shellbox/backends/shellsim/machine.py

@@ -31,6 +31,7 @@ DEFAULT_CPU_LIMIT = 10_000_000_000
 DEFAULT_DISK_LIMIT = 256 * 1024 * 1024
 DEFAULT_OUTPUT_LIMIT = 128 * 1024 * 1024
 DEFAULT_MEMORY_MB = 256
+SIMULATOR_PANIC_EXIT_CODE = 1
 ENV_NAME = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\Z")
 
 
@@ -45,7 +46,16 @@ class ShellSimShellSession:
     async def execute(
         self, command: str, *, wait: float = 120, output_limit_bytes: int = DEFAULT_MACHINE_OUTPUT_LIMIT_BYTES
     ) -> ShellUpdate:
-        result = await self.machine._run_source(command)
+        try:
+            result = await self.machine._run_source(command)
+        except shellsim.SimulationError as error:
+            output = str(error).encode()
+            return ShellUpdate(
+                output[:output_limit_bytes],
+                ShellStatus.COMPLETED,
+                SIMULATOR_PANIC_EXIT_CODE,
+                len(output) > output_limit_bytes,
+            )
         output = result.stdout + result.stderr
         return ShellUpdate(
             output[:output_limit_bytes],
@@ -123,6 +133,17 @@ class ShellSimMachine:
         except TimeoutError:
             self._closed = True
             return Result(None, b"", b"", False, False, ExitReason.TIMED_OUT)
+        except shellsim.SimulationError as error:
+            stderr = str(error).encode()
+            limit = command.output_limit_bytes
+            return Result(
+                SIMULATOR_PANIC_EXIT_CODE,
+                b"",
+                stderr[:limit],
+                False,
+                len(stderr) > limit,
+                ExitReason.EXITED,
+            )
         limit = command.output_limit_bytes
         return Result(
             result.returncode,

{marin_shellbox-0.2.151.dev37846966940 → marin_shellbox-0.2.153.dev37866254792}/tests/test_local_machine.py

@@ -7,7 +7,6 @@ import asyncio
 import os
 import shutil
 import socket
-import sys
 import tarfile
 import tempfile
 import uuid
@@ -16,6 +15,7 @@ from pathlib import Path
 
 import pytest
 from shellbox.backends.local.machine import NPROC_HEADROOM, LocalMachine, LocalMachineFactory, SandboxUnavailable
+from shellbox.backends.local.python_environment import PythonEnvironment, UvError, build_python_environment
 from shellbox.machine import (
     Command,
     DockerImage,
@@ -66,6 +66,13 @@ release.set()
 print(started)
 """
 SPEC = MachineSpec(HostImage(), workdir="/app")
+PYTHON_VERSION = "3.12.13"
+# A hash lock as ``uv pip compile --generate-hashes`` writes it, for a small pure-Python package.
+LOCK = """\
+six==1.17.0 \\
+    --hash=sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274 \\
+    --hash=sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81
+"""
 
 
 def make_factory(**options) -> LocalMachineFactory:
@@ -80,10 +87,23 @@ def factory() -> LocalMachineFactory:
     return make_factory()
 
 
+@pytest.fixture(scope="session")
+def python_environment(tmp_path_factory) -> PythonEnvironment:
+    lock = tmp_path_factory.mktemp("lock") / "requirements.lock"
+    lock.write_text(LOCK)
+    try:
+        return build_python_environment(tmp_path_factory.mktemp("environment") / "root", lock, PYTHON_VERSION)
+    except UvError as error:
+        # uv reports an unreachable or failed fetch this way; any other failure is a bug in the builder.
+        if not str(error).startswith("uv python install") or "Failed to download" not in str(error):
+            raise
+        pytest.skip(f"uv could not download a managed CPython {PYTHON_VERSION}: {error}")
+
+
 @pytest.fixture
-def python_factory() -> LocalMachineFactory:
-    """A factory whose commands find this test's Python, from a venv whose interpreter may lie anywhere."""
-    return make_factory(bin_dirs=(Path(sys.executable).parent,))
+def python_factory(python_environment) -> LocalMachineFactory:
+    """A factory whose commands find python3 in a self-contained environment that it mounts read-only."""
+    return make_factory(read_only=(python_environment.root,), bin_dirs=(python_environment.bin_dir,))
 
 
 @pytest.fixture
@@ -145,7 +165,7 @@ def test_bin_dirs_win_path_lookup_and_host_environment_is_not_inherited(tmp_path
     fake = bin_dir / "python3"
     fake.write_text("#!/bin/sh\necho fake-python\n")
     fake.chmod(0o755)
-    factory = make_factory(bin_dirs=(bin_dir,))
+    factory = make_factory(read_only=(bin_dir,), bin_dirs=(bin_dir,))
 
     async def scenario(machine: LocalMachine) -> tuple[Result, Result]:
         return await machine.run(Command(("python3",))), await machine.run(Command(("env",)))
@@ -337,26 +357,22 @@ def test_a_transfer_does_not_follow_a_symlink_out_of_the_machine_root(tmp_path,
     assert not (outside_dir / "planted.txt").exists() and not (tmp_path / "stolen.txt").exists()
 
 
-def test_a_venv_linked_through_an_alias_directory_runs_its_own_interpreter(tmp_path):
-    # uv links a venv's python through ``cpython-3.12-<platform>``, an alias symlink beside the resolved install.
-    interpreter = Path(os.path.realpath(sys.executable))
-    real = tmp_path / "real"
-    (real / "bin").mkdir(parents=True)
-    (real / "bin" / "python3").symlink_to(interpreter)
-    (tmp_path / "alias").symlink_to(real, target_is_directory=True)
-    venv = tmp_path / "venv"
-    (venv / "bin").mkdir(parents=True)
-    (venv / "bin" / "python").symlink_to(tmp_path / "alias" / "bin" / "python3")
-    (venv / "bin" / "python3").symlink_to("python")
-    (venv / "pyvenv.cfg").write_text(f"home = {tmp_path / 'alias' / 'bin'}\nversion = 3\n")
-    factory = make_factory(bin_dirs=(venv / "bin",))
-
-    async def scenario() -> Result:
-        machine = await factory.create(MachineSpec(HostImage(), workdir="/app"))
-        try:
-            return await machine.run(Command(("python3", "-c", "import sys; print(sys.prefix)")))
-        finally:
-            await machine.close()
-
-    result = asyncio.run(scenario())
-    assert result.stdout.decode().strip() == str(venv), result.stderr.decode()
+def test_a_python_environment_runs_its_own_interpreter_and_locked_packages(python_environment, python_factory):
+    script = "import sys, six; print(sys.prefix); print(sys.base_prefix)"
+    result = run_command(python_factory, Command(("python3", "-c", script)))
+    assert result.exit_code == 0, result.stderr.decode()
+    prefix, base_prefix = map(Path, result.stdout.decode().split())
+    assert prefix == python_environment.bin_dir.parent
+    assert base_prefix.is_relative_to(python_environment.root)
+
+
+def test_every_symlink_in_a_python_environment_resolves_inside_its_root(python_environment):
+    root = python_environment.root
+    links = [
+        Path(directory, name)
+        for directory, directories, files in os.walk(root)
+        for name in (*directories, *files)
+        if os.path.islink(Path(directory, name))
+    ]
+    assert python_environment.python in links
+    assert [link for link in links if not Path(os.path.realpath(link)).is_relative_to(root)] == []

marin_shellbox-0.2.153.dev37866254792/tests/test_shellsim_machine.py

@@ -0,0 +1,106 @@
+# Copyright The Marin Authors
+# SPDX-License-Identifier: Apache-2.0
+
+"""Behavior checks for the simulated shell and verifier boundary."""
+
+import asyncio
+from pathlib import Path
+
+from shellbox.backends.shellsim.machine import ShellSimMachineFactory
+from shellbox.machine import Command, ExitReason, MachineSpec, ShellSimBuiltins, ShellStatus
+
+PANIC_COMMAND = "cat /workspace/numbers.txt | python3 -c 'import sys; print(sum(int(line) for line in sys.stdin))'"
+PANIC_OUTPUT_LIMIT_BYTES = 64
+
+
+def test_shell_state_and_one_shot_command() -> None:
+    async def check() -> None:
+        machine = await ShellSimMachineFactory().create(MachineSpec(ShellSimBuiltins()))
+        try:
+            shell = await machine.open_shell()
+            first = await shell.execute(
+                'cd /tmp; export ANSWER=hello; answer() { echo "$ANSWER"; }; echo file >/workspace/result'
+            )
+            assert first.status is ShellStatus.COMPLETED
+            second = await shell.execute("pwd; answer")
+            assert second.output == b"/tmp\nhello\n"
+
+            verifier = await machine.run(Command(("sh", "-c", "pwd; cat /workspace/result")))
+            assert verifier.stdout == b"/workspace\nfile\n"
+            after = await shell.execute("pwd; answer")
+            assert after.output == b"/tmp\nhello\n"
+        finally:
+            await machine.close()
+
+    asyncio.run(check())
+
+
+def test_file_transfer_and_bounded_output(tmp_path: Path) -> None:
+    async def check() -> None:
+        machine = await ShellSimMachineFactory().create(MachineSpec(ShellSimBuiltins()))
+        try:
+            source = tmp_path / "source"
+            source.mkdir()
+            (source / "data.txt").write_text("payload")
+            await machine.upload(source, "/workspace/imported")
+            shell = await machine.open_shell()
+            result = await shell.execute("cat /workspace/imported/data.txt; printf 'x'", output_limit_bytes=4)
+            assert result.output == b"payl"
+            assert result.truncated
+
+            target = tmp_path / "downloaded"
+            target.mkdir()
+            await machine.download("/workspace/imported", target)
+            assert (target / "data.txt").read_text() == "payload"
+        finally:
+            await machine.close()
+
+    asyncio.run(check())
+
+
+def test_run_simulator_panic_returns_failed_result(tmp_path: Path) -> None:
+    async def check() -> None:
+        machine = await ShellSimMachineFactory().create(MachineSpec(ShellSimBuiltins()))
+        try:
+            source = tmp_path / "numbers.txt"
+            source.write_text("1\n2\n")
+            await machine.upload(source, "/workspace/numbers.txt")
+
+            result = await machine.run(Command(("sh", "-c", PANIC_COMMAND), output_limit_bytes=PANIC_OUTPUT_LIMIT_BYTES))
+            assert result.reason is ExitReason.EXITED
+            assert result.exit_code == 1
+            assert result.stdout == b""
+            assert result.stderr.startswith(b"shellsim operation panicked")
+            assert len(result.stderr) == PANIC_OUTPUT_LIMIT_BYTES
+            assert result.stderr_truncated
+
+            followup = await machine.run(Command(("cat", "/workspace/numbers.txt")))
+            assert (followup.exit_code, followup.stdout) == (0, b"1\n2\n")
+        finally:
+            await machine.close()
+
+    asyncio.run(check())
+
+
+def test_shell_session_simulator_panic_returns_failed_update(tmp_path: Path) -> None:
+    async def check() -> None:
+        machine = await ShellSimMachineFactory().create(MachineSpec(ShellSimBuiltins()))
+        try:
+            source = tmp_path / "numbers.txt"
+            source.write_text("1\n2\n")
+            await machine.upload(source, "/workspace/numbers.txt")
+            shell = await machine.open_shell()
+
+            update = await shell.execute(PANIC_COMMAND, output_limit_bytes=PANIC_OUTPUT_LIMIT_BYTES)
+            assert update.status is ShellStatus.COMPLETED
+            assert update.exit_code == 1
+            assert update.output.startswith(b"shellsim operation panicked")
+            assert len(update.output) == PANIC_OUTPUT_LIMIT_BYTES
+            assert update.truncated
+
+            followup = await shell.execute("cat /workspace/numbers.txt")
+            assert (followup.exit_code, followup.output) == (0, b"1\n2\n")
+        finally:
+            await machine.close()
+
+    asyncio.run(check())

marin_shellbox-0.2.151.dev37846966940/tests/test_shellsim_machine.py

@@ -1,55 +0,0 @@
-# Copyright The Marin Authors
-# SPDX-License-Identifier: Apache-2.0
-
-"""Behavior checks for the simulated shell and verifier boundary."""
-
-import asyncio
-from pathlib import Path
-
-from shellbox.backends.shellsim.machine import ShellSimMachineFactory
-from shellbox.machine import Command, MachineSpec, ShellSimBuiltins, ShellStatus
-
-
-def test_shell_state_and_one_shot_command() -> None:
-    async def check() -> None:
-        machine = await ShellSimMachineFactory().create(MachineSpec(ShellSimBuiltins()))
-        try:
-            shell = await machine.open_shell()
-            first = await shell.execute(
-                'cd /tmp; export ANSWER=hello; answer() { echo "$ANSWER"; }; echo file >/workspace/result'
-            )
-            assert first.status is ShellStatus.COMPLETED
-            second = await shell.execute("pwd; answer")
-            assert second.output == b"/tmp\nhello\n"
-
-            verifier = await machine.run(Command(("sh", "-c", "pwd; cat /workspace/result")))
-            assert verifier.stdout == b"/workspace\nfile\n"
-            after = await shell.execute("pwd; answer")
-            assert after.output == b"/tmp\nhello\n"
-        finally:
-            await machine.close()
-
-    asyncio.run(check())
-
-
-def test_file_transfer_and_bounded_output(tmp_path: Path) -> None:
-    async def check() -> None:
-        machine = await ShellSimMachineFactory().create(MachineSpec(ShellSimBuiltins()))
-        try:
-            source = tmp_path / "source"
-            source.mkdir()
-            (source / "data.txt").write_text("payload")
-            await machine.upload(source, "/workspace/imported")
-            shell = await machine.open_shell()
-            result = await shell.execute("cat /workspace/imported/data.txt; printf 'x'", output_limit_bytes=4)
-            assert result.output == b"payl"
-            assert result.truncated
-
-            target = tmp_path / "downloaded"
-            target.mkdir()
-            await machine.download("/workspace/imported", target)
-            assert (target / "data.txt").read_text() == "payload"
-        finally:
-            await machine.close()
-
-    asyncio.run(check())