pynterp 0.1.0__tar.gz

pynterp-0.1.0/.gitignore

@@ -0,0 +1,6 @@
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.ruff_cache/
+.venv/
+.todo-loop/

pynterp-0.1.0/PKG-INFO

@@ -0,0 +1,5 @@
+Metadata-Version: 2.4
+Name: pynterp
+Version: 0.1.0
+Summary: AST-walk interpreter for a meaningful Python subset
+Requires-Python: >=3.10

pynterp-0.1.0/README.md

@@ -0,0 +1,135 @@
+# pynterp
+
+`pynterp` is an AST-walk interpreter for a substantial, tested subset of Python.
+
+## Project focus
+
+- readable interpreter internals (AST + explicit runtime scopes)
+- deterministic execution through explicit environments
+- measured behavior against CPython tests
+
+## Compatibility
+
+- host runtime: CPython `3.10+` (`requires-python >=3.10`)
+- language support tracks modern CPython AST features, including:
+  - structural pattern matching (`match` / `case`)
+  - exception groups (`except*`)
+  - generic type parameter syntax and runtime typing objects
+  - template strings (`TemplateStr`) when running on Python `3.14+`
+
+## Quickstart
+
+```pycon
+>>> from pynterp import Interpreter
+>>> interpreter = Interpreter()
+>>> env = interpreter.make_default_env()
+>>> run_result = interpreter.run("""
+... print("Hello, World!")
+... """, env=env)
+Hello, World!
+```
+
+## What is implemented
+
+### Core execution model
+
+- explicit env execution: `Interpreter.run(source, env=...)` requires a dict
+- no implicit inheritance of host globals/builtins
+- `RunResult` return model captures uncaught exceptions without forcing immediate raise
+- interpreter-aware handling for `globals()`, `locals()`, `vars()`, `dir()`, `eval()`, and `exec()`
+
+### Statement support
+
+- assignment forms: `=`, annotated assignment, augmented assignment, destructuring/starred targets
+- control flow: `if`, `while`, `for`, `break`, `continue`, loop `else`
+- function forms: `def`, `async def`, `return`, `global`, `nonlocal`
+- class definitions with metaclass namespace handling (`__prepare__`) and private-name mangling
+- exception handling: `try/except/else/finally`, `raise`, `raise ... from ...`, `try/except*`
+- context managers: `with`, `async with`
+- pattern matching: value/singleton/sequence/mapping/class/or/as patterns with guards
+- imports: `import`, `from ... import ...`, optional relative import support
+- type alias statement (`type X = ...`) with type parameter support
+
+### Expression support
+
+- scalar and operator forms: constants, names, unary/binary/bool ops, chained comparisons, conditional expressions, walrus
+- calls with CPython-like argument binding checks (`*args`, `**kwargs`, duplicate-key errors)
+- containers and indexing: list/tuple/set/dict literals, starred unpacking, attributes, subscripts, slices
+- string forms: f-strings and template strings (runtime-dependent for 3.14 features)
+- functional forms: lambdas, comprehensions, generator expressions, `yield`, `yield from`, `await`
+
+### Functions, scopes, and runtime semantics
+
+- lexical scoping with closure cells and `nonlocal` behavior
+- descriptor-correct method binding (`UserFunction` + `BoundMethod`)
+- zero-argument `super()` support via `__class__` closure behavior
+- function metadata and interoperability: defaults/kwdefaults mutation, annotations, signatures, weakrefs, pickling paths
+- class/generic metadata wiring including `__qualname__`, `__orig_bases__`, and `__type_params__` where applicable
+
+### Async and generator runtime
+
+- generator execution path mirrors statement/expression semantics in generator mode
+- async function execution with awaitable protocol handling
+- interpreted async generator object with `asend`, `athrow`, and `aclose` behavior
+
+### Environment controls and hardening
+
+- allowlist-based import control (`allowed_imports`)
+- safe builtins surface (for example no implicit `open`)
+- guarded reflection pivots via wrapped `getattr`/`setattr`/`delattr`/`hasattr`
+- blocked high-risk attributes include names such as `__mro__`, `__subclasses__`, `__globals__`, frame globals/locals, and related pivots
+
+This is in-process hardening, not an OS sandbox.
+Out of scope: CPU/memory/time quotas and process/kernel isolation.
+
+### Interpreted module loading
+
+- optional interpreted package imports via `InterpretedModuleLoader`
+- interpreted modules can import each other through the interpreter runtime
+- the project can interpret code that imports and runs `pynterp` itself (see bootstrap tests)
+
+### Tests in this repo
+
+- `~711` tests across CLI behavior, semantics, security hardening, keyword binding, `super()` semantics, bootstrap/self-interpretation checks, and probe correctness
+- large dedicated security suite exercises reflective escape attempts and descriptor rebound edge cases
+
+## CPython compatibility probe
+
+Probe script: [`scripts/cpython_pynterp_probe.py`](scripts/cpython_pynterp_probe.py)
+
+Reproducible module-mode probe command (from script header):
+
+```bash
+uv run python scripts/cpython_pynterp_probe.py \
+  --cpython-root /tmp/cpython-3.14 \
+  --python-exe /tmp/cpython-3.14/python.exe \
+  --basis tests \
+  --mode module \
+  --strict-worker-match \
+  --json-out /tmp/pynterp-probe-tests-module.json
+```
+
+Default unsupported source filters used by the probe classifier:
+
+- `__import__`
+- `__dict__`
+- `__code__`
+
+Snapshot baseline (February 27, 2026; CPython `origin/3.14`, `module` mode, `basis=tests`):
+
+- total test files: `762`
+- applicable files: `515` (`67.59%`)
+- not applicable files: `247`
+- declared individual tests in applicable files: `10,761`
+- discovered+run individual tests: `12,560`
+- estimated individual tests total: `15,339`
+- individual pass: `9,406`
+- individual skip: `1,239`
+- individual fail: `4,694`
+- pass rate (individual): `61.32%`
+- individual pass+skip rate: `69.40%`
+
+`script` mode (`__name__ == "__main__"`) is much lower and mainly useful for diagnosing entry-point assumptions (same run, `--mode script`):
+
+- individual pass rate: `4.11%`
+- individual pass+skip rate: `7.92%`

pynterp-0.1.0/pyproject.toml

@@ -0,0 +1,42 @@
+[build-system]
+requires = ["hatchling>=1.27.0"]
+build-backend = "hatchling.build"
+
+[project]
+name = "pynterp"
+version = "0.1.0"
+description = "AST-walk interpreter for a meaningful Python subset"
+requires-python = ">=3.10"
+dependencies = []
+
+[dependency-groups]
+dev = [
+  "ipython>=8.38.0",
+  "pytest>=8.4.0",
+  "ruff>=0.13.0",
+]
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/pynterp"]
+
+[tool.hatch.build.targets.sdist]
+include = [
+  "src/pynterp",
+  "tests",
+  "pyproject.toml",
+  "README.md",
+]
+
+[tool.uv]
+default-groups = ["dev"]
+
+[tool.pytest.ini_options]
+addopts = "-ra"
+testpaths = ["tests"]
+
+[tool.ruff]
+target-version = "py310"
+line-length = 100
+
+[tool.ruff.lint]
+select = ["F", "I"]

pynterp-0.1.0/src/pynterp/__init__.py

@@ -0,0 +1,4 @@
+from .core import RunResult
+from .main import Interpreter
+
+__all__ = ["Interpreter", "RunResult"]

pynterp-0.1.0/src/pynterp/__main__.py

@@ -0,0 +1,57 @@
+import argparse
+import sys
+import traceback
+from pathlib import Path
+
+from .main import Interpreter
+
+
+def _build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="python -m pynterp",
+        usage="python -m pynterp <script.py>",
+    )
+    parser.add_argument("script")
+    return parser
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = _build_parser()
+    args_list = sys.argv[1:] if argv is None else argv
+    try:
+        args = parser.parse_args(args_list)
+    except SystemExit as exc:
+        return int(exc.code)
+
+    script_path = Path(args.script).resolve()
+    if not script_path.is_file():
+        print(f"pynterp: script not found: {script_path}", file=sys.stderr)
+        return 2
+
+    source = script_path.read_text()
+    interpreter = Interpreter()
+    env = interpreter.make_default_env(
+        env={
+            "__file__": str(script_path),
+            "__package__": None,
+        },
+        name="__main__",
+    )
+    result = interpreter.run(source, env=env, filename=str(script_path))
+    if result.exception is not None:
+        exc = result.exception
+        if isinstance(exc, SystemExit):
+            code = exc.code
+            if code is None:
+                return 0
+            if isinstance(code, int):
+                return code
+            print(code, file=sys.stderr)
+            return 1
+        traceback.print_exception(exc, file=sys.stderr)
+        return 1
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

pynterp-0.1.0/src/pynterp/code.py

@@ -0,0 +1,150 @@
+from __future__ import annotations
+
+import ast
+import symtable
+from typing import Dict, Set
+
+from .symtable_utils import _table_frees
+
+
+def _build_symtable(source: str, filename: str) -> symtable.SymbolTable:
+    try:
+        return symtable.symtable(source, filename, "exec")
+    except TypeError as exc:
+        # CPython 3.14's Lib/symtable.py passes module=<...> as a keyword-only
+        # argument to _symtable.symtable(). Older host runtimes reject that call.
+        if "_symtable.symtable() takes no keyword arguments" not in str(exc):
+            raise
+        import _symtable
+
+        raw_table = _symtable.symtable(source, filename, "exec")
+        new_symbol_table = getattr(symtable, "_newSymbolTable", None)
+        if callable(new_symbol_table):
+            return new_symbol_table(raw_table, filename)
+        raise
+
+
+class ScopeInfo:
+    """
+    Per-function scope info needed for runtime name resolution.
+    """
+
+    def __init__(self, table: symtable.Function, cellvars: Set[str]):
+        self.table = table
+        self.locals: Set[str] = set(table.get_locals())
+        self.frees: Set[str] = set(table.get_frees())
+        self.cellvars: Set[str] = set(cellvars)
+        self.declared_globals: Set[str] = {
+            s.get_name() for s in table.get_symbols() if s.is_declared_global()
+        }
+
+
+class ModuleCode:
+    """
+    Holds:
+      - parsed AST
+      - symtable tree
+      - a mapping from (type,name,lineno) -> table
+      - computed cellvars for each function table
+    """
+
+    def __init__(self, source: str, filename: str = "<pynterp>"):
+        self.source = source
+        self.filename = filename
+        self.tree = ast.parse(source, filename=filename, mode="exec")
+        self.sym_root = _build_symtable(source, filename)
+
+        self._tables_by_key: Dict[tuple[str, str, int], list[symtable.SymbolTable]] = {}
+        self._cellvars_by_id: Dict[int, Set[str]] = {}
+        self._lambda_occurrence_by_location: Dict[tuple[int, int], int] = {}
+
+        self._index_tables(self.sym_root)
+        self._index_lambda_occurrences()
+        self._compute_cellvars(self.sym_root)
+
+    def _index_tables(self, table: symtable.SymbolTable) -> None:
+        key = (table.get_type(), table.get_name(), table.get_lineno())
+        self._tables_by_key.setdefault(key, []).append(table)
+        for child in table.get_children():
+            self._index_tables(child)
+
+    def _compute_cellvars(self, table: symtable.SymbolTable) -> None:
+        for child in table.get_children():
+            self._compute_cellvars(child)
+
+        if table.get_type() == "function":
+            assert isinstance(table, symtable.Function)
+            locals_ = set(table.get_locals())
+            child_frees: Set[str] = set()
+            for child in table.get_children():
+                child_frees |= _table_frees(child)
+            self._cellvars_by_id[table.get_id()] = locals_ & child_frees
+        else:
+            self._cellvars_by_id[table.get_id()] = set()
+
+    def _index_lambda_occurrences(self) -> None:
+        lambda_counts_by_line: Dict[int, int] = {}
+
+        class Visitor(ast.NodeVisitor):
+            def _visit_lambda_default_exprs(self, node: ast.Lambda) -> None:
+                for default in node.args.defaults or []:
+                    self.visit(default)
+                for kw_default in getattr(node.args, "kw_defaults", []) or []:
+                    if kw_default is not None:
+                        self.visit(kw_default)
+
+            def visit_Lambda(self, node: ast.Lambda) -> None:
+                # Match symtable ordering: lambda defaults are analyzed in the
+                # enclosing scope before the lambda scope itself is registered.
+                self._visit_lambda_default_exprs(node)
+                index = lambda_counts_by_line.get(node.lineno, 0)
+                lambda_counts_by_line[node.lineno] = index + 1
+                self_outer._lambda_occurrence_by_location[(node.lineno, node.col_offset)] = index
+                self.visit(node.body)
+
+        self_outer = self
+        Visitor().visit(self.tree)
+
+    def lookup_function_table(
+        self, fn_node: ast.FunctionDef | ast.AsyncFunctionDef
+    ) -> symtable.Function:
+        key = ("function", fn_node.name, fn_node.lineno)
+        tables = self._tables_by_key.get(key, [])
+        if not tables:
+            raise RuntimeError(
+                f"Could not find symtable for function {fn_node.name!r} at line {fn_node.lineno}"
+            )
+        if len(tables) > 1:
+            raise RuntimeError(
+                f"Ambiguous symtable match for function {fn_node.name!r} at line {fn_node.lineno}"
+            )
+        tbl = tables[0]
+        if not isinstance(tbl, symtable.Function):
+            raise RuntimeError("lookup_function_table returned non-function table")
+        return tbl
+
+    def lookup_lambda_table(self, lambda_node: ast.Lambda) -> symtable.Function:
+        key = ("function", "lambda", lambda_node.lineno)
+        tables = self._tables_by_key.get(key, [])
+        if not tables:
+            raise RuntimeError(
+                "Could not find symtable for lambda at "
+                f"line {lambda_node.lineno}:{lambda_node.col_offset}"
+            )
+        if len(tables) == 1:
+            tbl = tables[0]
+        else:
+            location = (lambda_node.lineno, lambda_node.col_offset)
+            index = self._lambda_occurrence_by_location.get(location)
+            if index is None or index >= len(tables):
+                raise RuntimeError(
+                    "Ambiguous symtable match for lambda at "
+                    f"line {lambda_node.lineno}:{lambda_node.col_offset}"
+                )
+            tbl = tables[index]
+        if not isinstance(tbl, symtable.Function):
+            raise RuntimeError("lookup_lambda_table returned non-function table")
+        return tbl
+
+    def scope_info_for(self, fn_table: symtable.Function) -> ScopeInfo:
+        return ScopeInfo(fn_table, self._cellvars_by_id.get(fn_table.get_id(), set()))

pynterp-0.1.0/src/pynterp/common.py

@@ -0,0 +1,44 @@
+from __future__ import annotations
+
+from typing import Any
+
+UNBOUND = object()
+NO_DEFAULT = object()
+
+
+class ControlFlowSignal(BaseException):
+    """Internal non-user exceptions used for control flow (return/break/continue)."""
+
+
+class ReturnSignal(ControlFlowSignal):
+    def __init__(self, value: Any):
+        self.value = value
+
+
+class BreakSignal(ControlFlowSignal):
+    pass
+
+
+class ContinueSignal(ControlFlowSignal):
+    pass
+
+
+class AwaitRequest:
+    """Internal value used to hand awaitables from the AST runner to async trampoline."""
+
+    __slots__ = ("awaitable",)
+
+    def __init__(self, awaitable: Any):
+        self.awaitable = awaitable
+
+
+class Cell:
+    """A tiny closure cell."""
+
+    __slots__ = ("value",)
+
+    def __init__(self, value: Any = UNBOUND):
+        self.value = value
+
+    def __repr__(self) -> str:
+        return "<Cell UNBOUND>" if self.value is UNBOUND else f"<Cell {self.value!r}>"

pynterp-0.1.0/src/pynterp/core.py

@@ -0,0 +1,192 @@
+import ast
+from dataclasses import dataclass
+from pathlib import Path
+from types import ModuleType
+from typing import Any, Dict, Iterator, Optional, Set
+
+from pynterp.lib import (
+    InterpretedModuleLoader,
+    import_safe_stdlib_module,
+    make_safe_env,
+)
+from pynterp.lib.compat import maybe_patch_runtime_module
+
+from .code import ModuleCode
+from .scopes import ModuleScope, RuntimeScope
+
+
+@dataclass(slots=True)
+class RunResult:
+    globals: dict[str, Any]
+    exception: BaseException | None = None
+
+    @property
+    def ok(self) -> bool:
+        return self.exception is None
+
+    def raise_for_exception(self) -> None:
+        if self.exception is not None:
+            raise self.exception
+
+
+class InterpreterCore:
+    def __init__(
+        self, allowed_imports: Optional[Set[str]] = None, allow_relative_imports: bool = False
+    ):
+        """
+        allowed_imports:
+          - None  -> allow any import (NOT secure)
+          - set() -> block all imports
+          - {"math", "json"} -> allow only these roots (and their submodules)
+        """
+        self.allowed_imports = None if allowed_imports is None else set(allowed_imports)
+        self.allow_relative_imports = bool(allow_relative_imports)
+
+    # ----- restricted import -----
+
+    def _is_allowed_module(self, name: str) -> bool:
+        if self.allowed_imports is None:
+            return True
+        if not name:
+            return False
+        for allowed in self.allowed_imports:
+            if (
+                name == allowed
+                or name.startswith(allowed + ".")
+                or name.split(".", 1)[0] == allowed
+            ):
+                return True
+        return False
+
+    def _restricted_import(self, name, globals=None, locals=None, fromlist=(), level=0):
+        if level and not self.allow_relative_imports:
+            raise ImportError("relative imports are not supported by this interpreter")
+        if not self._is_allowed_module(name):
+            raise ImportError(f"import of '{name}' is not allowed")
+        module = import_safe_stdlib_module(name)
+        # Match __import__ behavior: without fromlist, return the top-level package.
+        if not fromlist and "." in name:
+            return self._adapt_runtime_value(import_safe_stdlib_module(name.split(".", 1)[0]))
+        return self._adapt_runtime_value(module)
+
+    def _adapt_runtime_value(self, value: Any) -> Any:
+        return maybe_patch_runtime_module(value)
+
+    def _import(self, name: str, scope: RuntimeScope, fromlist=(), level=0):
+        imp = scope.builtins.get("__import__")
+        if imp is None or not callable(imp):
+            raise ImportError("__import__ is not available in this environment")
+        value = imp(name, scope.globals, scope.globals, fromlist, level)
+        return self._adapt_runtime_value(value)
+
+    def make_default_env(
+        self,
+        env: Optional[dict] = None,
+        *,
+        name: str = "__main__",
+        package_root: str | Path | None = None,
+        package_name: str = "pynterp",
+    ) -> dict:
+        if env is None:
+            base: Dict[str, Any] = {}
+        elif isinstance(env, dict):
+            base = dict(env)
+        else:
+            raise TypeError("env must be dict or None")
+
+        loader: InterpretedModuleLoader | None = None
+        importer = self._restricted_import
+        if package_root is not None:
+            loader = InterpretedModuleLoader(
+                self,
+                package_name=package_name,
+                package_root=package_root,
+                fallback_importer=self._restricted_import,
+            )
+            importer = loader.import_module
+
+        out = make_safe_env(importer, env=base, name=name)
+        if loader is not None:
+            out.setdefault("__module_loader__", loader)
+        return out
+
+    # ----- run -----
+
+    def run(self, source: str, env: dict, filename: str = "<pynterp>") -> RunResult:
+        """
+        Execute `source` in a fresh AST interpreter module environment.
+
+        Returns a RunResult with globals and any uncaught exception.
+        """
+        if not isinstance(env, dict):
+            raise TypeError("env must be dict")
+        globals_dict = env
+
+        raw_builtins = globals_dict.get("__builtins__", {})
+        if raw_builtins is None:
+            builtins_dict: Dict[str, Any] = {}
+        elif isinstance(raw_builtins, dict):
+            builtins_dict = raw_builtins
+        elif isinstance(raw_builtins, ModuleType):
+            builtins_dict = raw_builtins.__dict__
+        else:
+            raise TypeError("__builtins__ must be dict, module, or None")
+
+        try:
+            code = ModuleCode(source, filename)
+            scope = ModuleScope(code, globals_dict, builtins_dict)
+            self.exec_module(code.tree, scope)
+        except BaseException as exc:
+            return RunResult(globals=globals_dict, exception=exc)
+        return RunResult(globals=globals_dict)
+
+    def run_or_raise(self, source: str, env: dict, filename: str = "<pynterp>") -> dict:
+        result = self.run(source, env, filename)
+        result.raise_for_exception()
+        return result.globals
+
+    # ----- dispatch (normal) -----
+
+    def exec_module(self, node: ast.Module, scope: RuntimeScope) -> None:
+        for stmt in node.body:
+            self.exec_stmt(stmt, scope)
+
+    def exec_block(self, stmts: list[ast.stmt], scope: RuntimeScope) -> None:
+        for stmt in stmts:
+            self.exec_stmt(stmt, scope)
+
+    def exec_stmt(self, node: ast.AST, scope: RuntimeScope) -> None:
+        m = getattr(self, f"exec_{node.__class__.__name__}", None)
+        if m is None:
+            raise NotImplementedError(f"Statement not supported: {node.__class__.__name__}")
+        m(node, scope)
+
+    def eval_expr(self, node: ast.AST, scope: RuntimeScope) -> Any:
+        m = getattr(self, f"eval_{node.__class__.__name__}", None)
+        if m is None:
+            raise NotImplementedError(f"Expression not supported: {node.__class__.__name__}")
+        return m(node, scope)
+
+    # ----- dispatch (generator-mode) -----
+    # These are Python generators so that `yield` in interpreted code maps to real Python yield.
+
+    def g_exec_block(self, stmts: list[ast.stmt], scope: RuntimeScope) -> Iterator[Any]:
+        for stmt in stmts:
+            yield from self.g_exec_stmt(stmt, scope)
+
+    def g_exec_stmt(self, node: ast.AST, scope: RuntimeScope) -> Iterator[Any]:
+        m = getattr(self, f"g_exec_{node.__class__.__name__}", None)
+        if m is None:
+            # fallback: run a non-yielding statement
+            self.exec_stmt(node, scope)
+            return
+        yield from m(node, scope)
+        if False:
+            yield None  # keeps it a generator in all branches
+
+    def g_eval_expr(self, node: ast.AST, scope: RuntimeScope) -> Iterator[Any]:
+        m = getattr(self, f"g_eval_{node.__class__.__name__}", None)
+        if m is None:
+            return self.eval_expr(node, scope)
+        val = yield from m(node, scope)
+        return val