pyrust-engine 0.6.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- pyrust_engine-0.6.0/LICENSE +21 -0
- pyrust_engine-0.6.0/PKG-INFO +473 -0
- pyrust_engine-0.6.0/README.md +458 -0
- pyrust_engine-0.6.0/pyproject.toml +23 -0
- pyrust_engine-0.6.0/pyrust_engine/__init__.py +5 -0
- pyrust_engine-0.6.0/pyrust_engine/__main__.py +60 -0
- pyrust_engine-0.6.0/pyrust_engine/cache.py +116 -0
- pyrust_engine-0.6.0/pyrust_engine/compiler.py +812 -0
- pyrust_engine-0.6.0/pyrust_engine/profiling.py +45 -0
- pyrust_engine-0.6.0/pyrust_engine/py.typed +0 -0
- pyrust_engine-0.6.0/pyrust_engine/runtime.py +400 -0
- pyrust_engine-0.6.0/pyrust_engine.egg-info/PKG-INFO +473 -0
- pyrust_engine-0.6.0/pyrust_engine.egg-info/SOURCES.txt +22 -0
- pyrust_engine-0.6.0/pyrust_engine.egg-info/dependency_links.txt +1 -0
- pyrust_engine-0.6.0/pyrust_engine.egg-info/requires.txt +7 -0
- pyrust_engine-0.6.0/pyrust_engine.egg-info/top_level.txt +1 -0
- pyrust_engine-0.6.0/setup.cfg +4 -0
- pyrust_engine-0.6.0/tests/test_engine.py +135 -0
- pyrust_engine-0.6.0/tests/test_fuzz.py +266 -0
- pyrust_engine-0.6.0/tests/test_v2.py +267 -0
- pyrust_engine-0.6.0/tests/test_v3.py +190 -0
- pyrust_engine-0.6.0/tests/test_v4.py +107 -0
- pyrust_engine-0.6.0/tests/test_v5.py +249 -0
- pyrust_engine-0.6.0/tests/test_v6.py +225 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Suraj Mhaske
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,473 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: pyrust-engine
|
|
3
|
+
Version: 0.6.0
|
|
4
|
+
Summary: Make Python code faster with one decorator: compiles numeric Python to Rust, with exact Python fallback
|
|
5
|
+
License-Expression: MIT
|
|
6
|
+
Requires-Python: >=3.10
|
|
7
|
+
Description-Content-Type: text/markdown
|
|
8
|
+
License-File: LICENSE
|
|
9
|
+
Provides-Extra: arrays
|
|
10
|
+
Requires-Dist: numpy>=1.24; extra == "arrays"
|
|
11
|
+
Provides-Extra: benchmark
|
|
12
|
+
Requires-Dist: numpy>=1.24; extra == "benchmark"
|
|
13
|
+
Requires-Dist: numba>=0.59; extra == "benchmark"
|
|
14
|
+
Dynamic: license-file
|
|
15
|
+
|
|
16
|
+
# PyRust Engine
|
|
17
|
+
|
|
18
|
+
[](https://github.com/surajmhaske02/pyrust-engine/actions/workflows/test.yml)
|
|
19
|
+

|
|
20
|
+

|
|
21
|
+
[](LICENSE)
|
|
22
|
+
|
|
23
|
+
**Make your Python code faster by adding one line.**
|
|
24
|
+
|
|
25
|
+
Put `@optimize` above a function. PyRust turns that function into Rust, compiles it and runs it as native machine code. You keep writing normal Python, and **your answers never change**: anything PyRust can't run safely runs as normal Python instead.
|
|
26
|
+
|
|
27
|
+
- ✍️ **Write plain Python.** No new language, no build step, no rewrite.
|
|
28
|
+
- ⚡ **Get real speed on calculation-heavy code:** math functions, arithmetic, bit operations, helper functions, NumPy arrays and repeated work run about **5× to 180× faster** in our tests on Linux, macOS and Windows.
|
|
29
|
+
- 🛟 **Never get a wrong answer.** Big integers, math errors and anything unsupported fall back to normal Python with the exact same result.
|
|
30
|
+
- 🔍 **See what's happening.** `fn.explain()` and a one-time warning tell you which line can't be sped up, and why.
|
|
31
|
+
|
|
32
|
+
📖 [API reference](docs/API.md) · 📓 [Try it in a notebook](examples/pyrust_vs_python.ipynb) · 🤝 [Contributing](CONTRIBUTING.md) · 📝 [Changelog](CHANGELOG.md)
|
|
33
|
+
|
|
34
|
+
Good fits: scientific and engineering formulas, finance and statistics, simulations, games and physics, signal processing, hashing and checksums, and number-heavy data processing.
|
|
35
|
+
|
|
36
|
+
```python
|
|
37
|
+
import math
|
|
38
|
+
from pyrust_engine import optimize
|
|
39
|
+
|
|
40
|
+
def distance(x: float, y: float) -> float:
|
|
41
|
+
return math.sqrt(x * x + y * y) # helpers are compiled together with their caller
|
|
42
|
+
|
|
43
|
+
@optimize
|
|
44
|
+
def total_distance(n: int) -> float:
|
|
45
|
+
total = 0.0
|
|
46
|
+
for i in range(n):
|
|
47
|
+
total += distance(float(i), float(n - i))
|
|
48
|
+
return total
|
|
49
|
+
|
|
50
|
+
@optimize
|
|
51
|
+
def checksum(n: int) -> int:
|
|
52
|
+
h = 2166136261
|
|
53
|
+
for i in range(n):
|
|
54
|
+
h = ((h ^ i) * 16777619) & (2 ** 32 - 1) # integer math and bit operations
|
|
55
|
+
return h
|
|
56
|
+
|
|
57
|
+
total_distance(300_000) # first call compiles (~0.1 s), then runs as native code
|
|
58
|
+
total_distance(300_000) # ~68x faster than plain Python in our benchmark
|
|
59
|
+
checksum(300_000) # same answer as plain Python, at native speed
|
|
60
|
+
print(total_distance.explain()) # does it compile? if not, which line and why
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
> **Status: experimental prototype.** Not a general Python compiler and not production-hardened. Tests pass in CI on Linux, Windows and macOS (Python 3.10, 3.12, 3.13), and benchmarks are published for all three.
|
|
64
|
+
|
|
65
|
+
## Comparison at a glance
|
|
66
|
+
|
|
67
|
+
| | Plain Python | **PyRust 0.6** |
|
|
68
|
+
|---|---|---|
|
|
69
|
+
| Integer loop (500k) | 43.0 ms · 1× | **1.20 ms · 35.9×** |
|
|
70
|
+
| Float recurrence (300k) | 21.8 ms · 1× | **1.82 ms · 12.0×** |
|
|
71
|
+
| Array kernel, read (100k float64) | 12.9 ms · 1× | **0.174 ms · 73.8×** |
|
|
72
|
+
| Write into an output array (100k) · *new in 0.6* | 44.1 ms · 1× | **0.247 ms · 179×** |
|
|
73
|
+
| Recursive `fib(25)` · *new in 0.6* | 9.71 ms · 1× | **0.642 ms · 15.1×** |
|
|
74
|
+
| `math.sqrt` / `sin` / `log` loop (300k) | 52.2 ms · 1× | **4.92 ms · 10.6×** |
|
|
75
|
+
| Helper function called in a loop (300k) | 57.7 ms · 1× | **0.848 ms · 68.1×** |
|
|
76
|
+
| Bit-mixing hash with `^ & << >> **` (300k) | 94.8 ms · 1× | **0.754 ms · 126×** |
|
|
77
|
+
| Tiny `add(a, b)` per call | 0.069 µs · 1× | **0.059 µs · ≈1×** (runs as plain Python) |
|
|
78
|
+
| JSON → array pipeline | 9.20 ms · 1× | **6.72 ms · 1.37×** |
|
|
79
|
+
| Big integers / overflow / math errors | Exact | **Exact (replays in Python)** |
|
|
80
|
+
| Unsupported code | Runs | **Runs in Python, with a warning that says which line** |
|
|
81
|
+
| Extra runtime dependencies | None | **None** (NumPy for arrays) |
|
|
82
|
+
| First call (compile) | None | 0.10–0.16 s (0.02 s from disk cache) |
|
|
83
|
+
|
|
84
|
+
<sub>"time · speedup vs Python", measured by CI on a GitHub Actions `ubuntu-latest` runner (x86_64, Python 3.13, Rust 1.99). Rows marked *new in 0.6* ran as plain Python (1×) in 0.5 and earlier. macOS and Windows numbers are under [Speed on every platform](#speed-on-every-platform).</sub>
|
|
85
|
+
|
|
86
|
+
## Version comparison
|
|
87
|
+
|
|
88
|
+
| | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | **0.6 (current)** |
|
|
89
|
+
|---|---|---|---|---|---|---|
|
|
90
|
+
| Types | `int` | + `float`, `float64` arrays | same | same | same | **+ `int64` arrays** |
|
|
91
|
+
| Arithmetic | `+ - *` | + `// % /` | + unary `+` | same | + `**`, `& \| ^ ~ << >>`, float `//` `%` | same |
|
|
92
|
+
| Builtins and `math` | — | `len`, `float` | same | same | + `abs min max int pow`, `math.*` | same |
|
|
93
|
+
| Calling other functions | — | — | — | — | Yes | **+ recursion** |
|
|
94
|
+
| Arrays | — | read | read | read | read | **read and write** |
|
|
95
|
+
| Comparisons | single | + `and/or/not` | + chained, truthiness | same | same | **+ exact int vs float** |
|
|
96
|
+
| Other syntax | `for`, `if` | + `while`, `break`, `continue` | + `elif` | same | + ternary, `while True`, globals | **+ functions returning `None`** |
|
|
97
|
+
| Tiny `add(a, b)` per call | 3.3 µs | 5.3 µs | 2.0 µs | ≈ Python | ≈ Python | ≈ Python |
|
|
98
|
+
| Developer tools | `report()` | + `prepare`, `reset`, profiler | same | + `native_loop_free` | + `explain()`, warnings, `check` | **+ type hints, API docs** |
|
|
99
|
+
| Automated tests | 12 | 34 | 47 | 56 | 89 | **112** |
|
|
100
|
+
| CI platforms passing | — | — | Linux | all three | all three | all three **+ package build** |
|
|
101
|
+
| Published benchmarks | Linux | Linux | Linux | Linux | Linux | **Linux, macOS, Windows** |
|
|
102
|
+
|
|
103
|
+
---
|
|
104
|
+
|
|
105
|
+
## What makes Python slow, and what PyRust does about it
|
|
106
|
+
|
|
107
|
+
Python is slow for a few specific reasons. PyRust removes some of them for the code it compiles. Others need a different tool, and it's better to know which is which.
|
|
108
|
+
|
|
109
|
+
| Why Python is slow | Example | PyRust 0.6 |
|
|
110
|
+
|---|---|---|
|
|
111
|
+
| Every operation is interpreted one bytecode at a time | `for` / `while` loops | ✅ Compiled to native machine code |
|
|
112
|
+
| Every number is a heap object with a type check | `total += i * i` | ✅ Plain 64-bit ints and floats in CPU registers |
|
|
113
|
+
| Function calls are expensive | a helper called inside a loop, recursion | ✅ Helpers and recursive functions are compiled into the same native code (0.5, 0.6) |
|
|
114
|
+
| Math functions go through Python objects | `math.sqrt(x)`, `x ** 0.5` | ✅ Native calls, same results (0.5) |
|
|
115
|
+
| Bit manipulation creates new int objects | hashing, `x ^ (x >> 13)` | ✅ Native bit operations (0.5) |
|
|
116
|
+
| Indexing arrays goes through NumPy objects | `float(a[i])`, `out[i] = x` in a loop | ✅ Reads and writes array memory directly (writes in 0.6) |
|
|
117
|
+
| Crossing into native code has a fixed cost | tiny functions called millions of times | ⚠️ ~2 µs per native call; tiny loop-free functions skip it |
|
|
118
|
+
| The GIL stops threads running Python in parallel | CPU-bound threads | ⚠️ Native calls release the GIL (ctypes), but there are no parallel loops yet |
|
|
119
|
+
| Lists, dicts, strings and objects are dynamic | text processing, JSON, classes | ❌ Not compiled. Use the right library, PyPy or CPython's JIT |
|
|
120
|
+
| I/O waits (network, disk, database) | web requests | ❌ Not a CPU problem: use `asyncio` or threads |
|
|
121
|
+
|
|
122
|
+
## Why it exists
|
|
123
|
+
|
|
124
|
+
Most "make Python fast" tools ask you to give something up: a new interpreter, a build step, a different integer model, or a heavy dependency. PyRust takes a narrower bet:
|
|
125
|
+
|
|
126
|
+
- **Correctness first.** Integers stay arbitrary-precision. If a 64-bit operation would overflow, divide by zero, index out of bounds or hit an invalid `range` step, the native call stops and **the original Python function is replayed**, so you get exactly Python's result or Python's exception. Nothing wraps silently.
|
|
127
|
+
- **Opt-in, per function.** Decorate the hot loop, leave everything else alone. Unsupported code, inputs or environments run as normal Python and the reason is recorded.
|
|
128
|
+
- **Light footprint.** Pure-Python package with no runtime dependencies (NumPy only for array kernels). Uses `rustc` directly through `ctypes`: no PyO3, Maturin, crates or C compiler toolchain beyond Rust's own linker.
|
|
129
|
+
|
|
130
|
+
## How it compares
|
|
131
|
+
|
|
132
|
+
### Speed
|
|
133
|
+
|
|
134
|
+

|
|
135
|
+
|
|
136
|
+
| Workload | Python | **PyRust 0.6** | PyRust vs Python |
|
|
137
|
+
|---|---:|---:|---:|
|
|
138
|
+
| Integer loop (`//`, `%`, branches; n = 500k) | 43.0 ms | **1.20 ms** | **35.9× faster** |
|
|
139
|
+
| Float recurrence (n = 300k) | 21.8 ms | **1.82 ms** | **12.0× faster** |
|
|
140
|
+
| Array kernel, sum of squares (100k float64) | 12.9 ms | **0.174 ms** | **73.8× faster** |
|
|
141
|
+
| Smoothing filter writing an output array (100k) | 44.1 ms | **0.247 ms** | **179× faster** |
|
|
142
|
+
| Recursive `fib(25)` | 9.71 ms | **0.642 ms** | **15.1× faster** |
|
|
143
|
+
| `math.sqrt(x) * math.sin(x) + math.log(x)` loop (n = 300k) | 52.2 ms | **4.92 ms** | **10.6× faster** |
|
|
144
|
+
| Helper `distance(x, y)` called inside a loop (n = 300k) | 57.7 ms | **0.848 ms** | **68.1× faster** |
|
|
145
|
+
| Bit-mixing hash with `^ & << >> **` (n = 300k) | 94.8 ms | **0.754 ms** | **126× faster** |
|
|
146
|
+
| Tiny function `add(a, b)`, per call (default) | 0.069 µs | **0.059 µs** | **same speed** (plain Python) |
|
|
147
|
+
| Tiny function `add(a, b)`, forced native | 0.068 µs | 4.00 µs | 59× slower |
|
|
148
|
+
| JSON → NumPy → kernel → JSON pipeline | 9.20 ms | **6.72 ms** | **1.37× faster** |
|
|
149
|
+
|
|
150
|
+
<sub>Median of 7 warm runs, wrapper and argument conversion included. Measured by CI on a GitHub Actions `ubuntu-latest` runner: Linux x86_64, CPython 3.13.18, Rust 1.99.1, NumPy 2.5.4. Shared CI machines are noisy, so treat these as approximate. Raw samples: [`benchmarks/results-v0.6.0-ci-linux.json`](benchmarks/results-v0.6.0-ci-linux.json); older runs are in [`benchmarks/`](benchmarks/). Workloads are in [`examples/workloads.py`](examples/workloads.py). Reproduce with `python -m examples.benchmark_suite`.</sub>
|
|
151
|
+
|
|
152
|
+
### Speed on every platform
|
|
153
|
+
|
|
154
|
+
Speedup over plain Python, same CI run, Python 3.13 and Rust 1.99 on each GitHub Actions runner:
|
|
155
|
+
|
|
156
|
+
| Workload | Linux x86_64 | macOS Apple Silicon (arm64) | Windows x86_64 |
|
|
157
|
+
|---|---:|---:|---:|
|
|
158
|
+
| Integer loop | 35.9× | 39.5× | 37.5× |
|
|
159
|
+
| Float recurrence | 12.0× | 5.3× | 8.9× |
|
|
160
|
+
| Array kernel (read) | 73.8× | 37.8× | 69.7× |
|
|
161
|
+
| Write into an output array | 179× | 128× | 152× |
|
|
162
|
+
| Recursive `fib(25)` | 15.1× | 17.0× | 19.9× |
|
|
163
|
+
| `math` functions loop | 10.6× | 15.3× | 15.5× |
|
|
164
|
+
| Helper function in a loop | 68.1× | 87.2× | 61.6× |
|
|
165
|
+
| Bit-mixing hash | 126× | 60.0× | 131× |
|
|
166
|
+
| Tiny `add(a, b)` (runs as Python) | ≈1× | ≈1× | ≈1× |
|
|
167
|
+
| JSON → array pipeline | 1.37× | 1.35× | 1.31× |
|
|
168
|
+
|
|
169
|
+
<sub>Raw samples: [Linux](benchmarks/results-v0.6.0-ci-linux.json), [macOS](benchmarks/results-v0.6.0-ci-macos.json), [Windows](benchmarks/results-v0.6.0-ci-windows.json). Differences between platforms mostly reflect how fast plain Python is on each machine.</sub>
|
|
170
|
+
|
|
171
|
+
**How to read this, honestly:**
|
|
172
|
+
|
|
173
|
+
- PyRust gives **large speedups on number-crunching code** on all three platforms: 36–40× on integer loops, 60–131× on bit operations and 128–179× when writing arrays.
|
|
174
|
+
- **New in 0.6:** writing into arrays and recursion now compile. In 0.5 those rows ran as plain Python (1×).
|
|
175
|
+
- Math-heavy code gains less (10–15×), because `sqrt`, `sin` and `log` cost the same in both; PyRust only removes the overhead around them. Float recurrences gain 5–12× for the same reason.
|
|
176
|
+
- **Tiny functions don't get slower.** Loop-free functions run as plain Python (details in [Tiny functions](#tiny-functions-the-problem-and-the-fix)). A function *with* a loop but a very small input still pays about 2–4 µs per call.
|
|
177
|
+
- **End-to-end gains shrink** when parsing and conversion dominate: the JSON pipeline only improved about 1.35×.
|
|
178
|
+
|
|
179
|
+
### Startup cost
|
|
180
|
+
|
|
181
|
+
| Workload | Linux: first call incl. compile | Linux: from disk cache | macOS: compile / cache | Windows: compile / cache |
|
|
182
|
+
|---|---:|---:|---:|---:|
|
|
183
|
+
| Integer loop | 0.113 s | 0.021 s | 0.196 s / 0.027 s | 0.366 s / 0.051 s |
|
|
184
|
+
| Recursive `fib` | 0.098 s | 0.020 s | 0.134 s / 0.026 s | 0.296 s / 0.048 s |
|
|
185
|
+
| Array write | 0.124 s | 0.020 s | 0.176 s / 0.029 s | 0.334 s / 0.051 s |
|
|
186
|
+
|
|
187
|
+
PyRust caches compiled libraries on disk across processes. Use `fn.prepare()` (or `prepare(background=True)`) before latency-sensitive work.
|
|
188
|
+
|
|
189
|
+
### Design trade-offs
|
|
190
|
+
|
|
191
|
+
This table compares how each tool is designed. It is not a benchmark.
|
|
192
|
+
|
|
193
|
+
| | **PyRust** | Numba | Cython | PyPy | Codon |
|
|
194
|
+
|---|---|---|---|---|---|
|
|
195
|
+
| How you use it | Decorator on a function | Decorator on a function | Separate build step (`.pyx` or annotated `.py`) | Different Python interpreter | Separate compiler |
|
|
196
|
+
| Integer overflow | **Exact Python result** (replays in Python) | Wraps at 64 bits | Python ints exact; C-typed ints follow C rules (optional overflow checks) | Exact | Wraps at 64 bits |
|
|
197
|
+
| Unsupported code | Runs as normal Python, reason recorded | Compile error | Compiles via the C API (slower paths) | Runs everything | Compile error |
|
|
198
|
+
| Scope | Numeric subset: ints, floats, `math`, helper calls, read-only arrays | Large NumPy-oriented subset | Full Python | Full Python | Large Python-like subset |
|
|
199
|
+
| Runtime dependencies | None (NumPy for arrays) | numba + llvmlite | None after build | Whole runtime | Codon runtime |
|
|
200
|
+
| Toolchain needed | `rustc` | none extra | C compiler | none extra | Codon |
|
|
201
|
+
| Maturity | Experimental | Mature | Mature | Mature | Active project |
|
|
202
|
+
|
|
203
|
+
**Pick PyRust** when you want Python-exact results (including big integers and math errors), a safety net that never changes your answers and no compiled dependency stack, and your hot spots are numeric loops, math and helper functions. **Pick Numba or NumPy** for maximum speed on array maths. **Pick Cython** when you need to compile large or object-heavy code. **Pick PyPy** to speed up a whole pure-Python application.
|
|
204
|
+
|
|
205
|
+
## Tiny functions: the problem and the fix
|
|
206
|
+
|
|
207
|
+
**The problem (0.3.0 and earlier).** `add(a, b)` took 2.08 µs through PyRust against 0.034 µs in plain Python: 61× slower. The Rust code isn't the slow part. The add itself takes about a nanosecond. The cost is the trip from Python into native code on every call:
|
|
208
|
+
|
|
209
|
+
| Step per call | What happens |
|
|
210
|
+
|---|---|
|
|
211
|
+
| Wrapper checks | Code/defaults unchanged? Builtins not rebound? Library loaded? |
|
|
212
|
+
| Argument checks | Each argument must be an exact `int` that fits 64 bits, or an exact `float` |
|
|
213
|
+
| `ctypes` conversion | Python objects → C values, plus an output buffer for the result |
|
|
214
|
+
| Native call | Status code back, then the result is read out of the buffer |
|
|
215
|
+
| Metrics | Timers and counters updated under a lock |
|
|
216
|
+
|
|
217
|
+
Together that is about 2 µs. A loop of 500,000 iterations easily pays it back. A single addition never can.
|
|
218
|
+
|
|
219
|
+
**The fix (0.4.0).** At decoration time, PyRust checks the function's AST. A function with **no `for` or `while` loop** does a fixed, tiny amount of work, so native code can't win. For those functions `@optimize` returns a plain Python function (a clone of the original with `report()`, `prepare()`, `reset()` and `rust_source()` attached). There is no wrapper, so a call costs exactly what the original costs, and results are the original's by construction. Since 0.5, a loop-free function that calls another function (for example `return helper_sum(n) + 1`) is compiled normally, because the call can hide real work.
|
|
220
|
+
|
|
221
|
+
```python
|
|
222
|
+
@optimize
|
|
223
|
+
def add(a, b):
|
|
224
|
+
return a + b
|
|
225
|
+
|
|
226
|
+
add(2, 3) # plain Python speed
|
|
227
|
+
add.report()['dispatch'] # 'python-direct'
|
|
228
|
+
add.report()['reason'] # 'loop-free function: a native call costs more than ...'
|
|
229
|
+
|
|
230
|
+
@optimize(native_loop_free=True) # force Rust anyway (e.g. to test the native path)
|
|
231
|
+
def add_native(a, b):
|
|
232
|
+
return a + b
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
Trade-off: a loop-free function has no wrapper, so `report()` can't count its calls (`call_counting` is `False`).
|
|
236
|
+
|
|
237
|
+
**What still costs ~2 µs.** A function that contains a loop but is called with tiny inputs (say `n = 3`), or a tiny function called millions of times from a Python loop. Best practice: decorate the function that **contains the hot loop**. Since 0.5, small helpers it calls are compiled into the same native code, so they cost nothing extra. The remaining fix is on the [roadmap](#roadmap): a CPython extension call path instead of `ctypes`.
|
|
238
|
+
|
|
239
|
+
## Limitations
|
|
240
|
+
|
|
241
|
+
- **Numeric subset only.** `int` (64-bit, overflow-checked), `float`, 1-D `float64` / `int64` arrays, `math` functions and calls to other numeric functions (including recursion). No lists, dicts, strings, tuples, classes or generators: those run as plain Python.
|
|
242
|
+
- **`min` / `max` over mixed int and float values** aren't compiled, because Python returns the winning argument with its own type.
|
|
243
|
+
- **Recursion** needs a return annotation (`-> int` / `-> float`) and runs natively up to 300 levels deep; deeper calls replay in Python.
|
|
244
|
+
- **Per-call overhead of ~2–4 µs** for native calls. Loop-free functions avoid it; loops with tiny inputs don't.
|
|
245
|
+
- **First call compiles** (about 0.1–0.4 s per function depending on the OS, 0.02–0.05 s from the disk cache). Use `fn.prepare()` ahead of time.
|
|
246
|
+
- **Arrays:** only 1-D, C-contiguous `float64` / `int64`. Writing an array copies it once per call (copy-on-write, so failures can be replayed exactly). Only the decorated function itself can write arrays, not helpers it calls.
|
|
247
|
+
- **Needs `rustc` on `PATH`** on every machine that should run natively (no prebuilt wheels).
|
|
248
|
+
- **Native loops can't be interrupted** by Ctrl-C until they finish or hit `max_iterations`.
|
|
249
|
+
- **No parallel loops.** One call uses one CPU core.
|
|
250
|
+
- **Benchmarks come from GitHub's CI machines** (Linux x86_64, macOS arm64, Windows x86_64), which are shared and noisy. There are no numbers for ARM Linux yet.
|
|
251
|
+
- **Experimental.** Not production-hardened, not a sandbox: only run trusted code.
|
|
252
|
+
|
|
253
|
+
## Correctness evidence
|
|
254
|
+
|
|
255
|
+
- **112 automated tests**, passing in CI on Linux, Windows and macOS with Python 3.10, 3.12 and 3.13, covering semantics, recursion (depth and budget limits), array writes (failure halfway through, overlapping and read-only arrays), `int64` arrays, exact int/float comparisons at the 2**53 and 2**63 boundaries, every 0.5 operator and `math` function at its edge cases (NaN, ±0.0, ±inf, overflow, domain errors, huge exponents and shifts), calls between functions, rebinding of globals and helpers, overflow replay, exceptions, NaN, ±0.0, infinities, array layouts, bounds, cache corruption and recovery, cross-process locking, background compilation, threads and profiling.
|
|
256
|
+
- **Generated-program fuzzing:** 4,000 random programs (2,000 integer + 2,000 float, using `while`, `break`, `continue`, `elif`, `//`, `%`, `/`), **32,000 calls compared with CPython, 0 mismatches**. Floats are compared by `repr`, so NaN and signed zero must match exactly. Every generated program must compile natively, so the test can't pass by quietly falling back to Python.
|
|
257
|
+
- **0.5 fuzzing:** a second generator adds `**`, `& | ^ ~ << >>`, `abs`, `min`, `max`, `pow`, ternaries, float `//` `%` `**`, `int()`, `math.floor/ceil` and nine `math` functions. Its programs run on all three operating systems in CI and must match CPython exactly, including which exception is raised.
|
|
258
|
+
- **Fuzzer sensitivity:** deliberately planted bugs (wrong modulo sign; `a / b` rewritten as `a * (1/b)`) were caught immediately.
|
|
259
|
+
|
|
260
|
+
Details and limitations: [`VALIDATION.md`](VALIDATION.md).
|
|
261
|
+
|
|
262
|
+
## Install
|
|
263
|
+
|
|
264
|
+
Requires **Python 3.10+** and **Rust** (`rustc` on `PATH`, from <https://rustup.rs>). Not yet on PyPI.
|
|
265
|
+
|
|
266
|
+
```sh
|
|
267
|
+
git clone https://github.com/surajmhaske02/pyrust-engine.git
|
|
268
|
+
cd pyrust-engine
|
|
269
|
+
python -m pip install ".[arrays]" # NumPy for array kernels
|
|
270
|
+
python -m unittest discover -s tests -v # 112 tests
|
|
271
|
+
python -m examples.benchmark_suite # add ".[benchmark]" for Numba comparison
|
|
272
|
+
```
|
|
273
|
+
|
|
274
|
+
Without Rust, everything still works in plain Python; `report()` says why.
|
|
275
|
+
|
|
276
|
+
## Usage
|
|
277
|
+
|
|
278
|
+
### Integer and float kernels
|
|
279
|
+
|
|
280
|
+
Save functions in a `.py` file (source must be inspectable). Annotations pick native types: unannotated or `int` means signed 64-bit integer; `float` means f64.
|
|
281
|
+
|
|
282
|
+
```python
|
|
283
|
+
from pyrust_engine import optimize
|
|
284
|
+
|
|
285
|
+
@optimize
|
|
286
|
+
def recurrence(n: int, value: float) -> float:
|
|
287
|
+
for i in range(n):
|
|
288
|
+
value = (value * 1.000001 + 0.0001) / 1.000002
|
|
289
|
+
return value
|
|
290
|
+
```
|
|
291
|
+
|
|
292
|
+
Arguments must be exact `int` (fitting 64 bits) or exact `float`. Anything else, including `bool`, int subclasses or an int passed to a `float` parameter, runs in Python. A local variable keeps one type: start a float accumulator at `0.0`, not `0`.
|
|
293
|
+
|
|
294
|
+
### NumPy arrays: read and write
|
|
295
|
+
|
|
296
|
+
```python
|
|
297
|
+
import numpy as np
|
|
298
|
+
from pyrust_engine import optimize, F64Array, I64Array
|
|
299
|
+
|
|
300
|
+
@optimize
|
|
301
|
+
def energy(values: F64Array) -> float: # read
|
|
302
|
+
total = 0.0
|
|
303
|
+
for i in range(len(values)):
|
|
304
|
+
x = float(values[i])
|
|
305
|
+
total += x * x
|
|
306
|
+
return total
|
|
307
|
+
|
|
308
|
+
@optimize
|
|
309
|
+
def smooth(values: F64Array, out: F64Array): # write into an output array (0.6)
|
|
310
|
+
n = len(values)
|
|
311
|
+
for i in range(n):
|
|
312
|
+
left = float(values[i - 1]) if i > 0 else float(values[i])
|
|
313
|
+
right = float(values[i + 1]) if i < n - 1 else float(values[i])
|
|
314
|
+
out[i] = (left + 2.0 * float(values[i]) + right) / 4.0
|
|
315
|
+
|
|
316
|
+
@optimize
|
|
317
|
+
def histogram(labels: I64Array, counts: I64Array, bins: int): # int64 arrays (0.6)
|
|
318
|
+
for i in range(len(labels)):
|
|
319
|
+
counts[int(labels[i]) % bins] += 1
|
|
320
|
+
|
|
321
|
+
data = np.random.default_rng(0).standard_normal(100_000)
|
|
322
|
+
out = np.empty_like(data)
|
|
323
|
+
energy(data); smooth(data, out)
|
|
324
|
+
```
|
|
325
|
+
|
|
326
|
+
Accepted arrays: exact `ndarray`, 1-D, C-contiguous, aligned, native-endian `float64` (`F64Array`) or `int64` (`I64Array`). Others run in Python. Read with `float(a[i])` / `int(a[i])`; write with `a[i] = value`, `a[i] += value`, `-=`, `*=` (and `/=` for floats). Negative indices work; out-of-bounds access raises Python's `IndexError` via replay.
|
|
327
|
+
|
|
328
|
+
Writes go to a private copy that is copied back only when the whole native call succeeds, so if anything fails halfway, Python re-runs the function on your original data and you see exactly what plain Python would do. Read-only arrays, and written arrays that share memory with another argument, run as plain Python.
|
|
329
|
+
|
|
330
|
+
### Recursion
|
|
331
|
+
|
|
332
|
+
```python
|
|
333
|
+
@optimize
|
|
334
|
+
def fib(n: int) -> int: # the return annotation is required for recursion
|
|
335
|
+
if n < 2:
|
|
336
|
+
return n
|
|
337
|
+
return fib(n - 1) + fib(n - 2)
|
|
338
|
+
|
|
339
|
+
fib(30) # ~15-20x faster than plain Python in our benchmarks
|
|
340
|
+
```
|
|
341
|
+
|
|
342
|
+
Mutual recursion works too. Native recursion goes up to 300 levels deep, and each call counts toward `max_iterations`; deeper or longer runs replay in Python with the same result (including `RecursionError`).
|
|
343
|
+
|
|
344
|
+
### Supported syntax
|
|
345
|
+
|
|
346
|
+
| Construct | Native support |
|
|
347
|
+
|---|---|
|
|
348
|
+
| Parameters | `int` (default), `float`, `F64Array`, `I64Array` |
|
|
349
|
+
| Integer ops | `+ - *`, unary `- + ~`, `//` and `%` with Python floor semantics, `**` (non-negative exponent), `& \| ^ << >>`; all overflow-checked. `int / int` gives a float like Python |
|
|
350
|
+
| Float ops | `+ - * / // % **`, unary `- +`, using CPython's exact rules for signed zero, NaN, infinities and errors; mixed int/float promotes like Python; no fast-math |
|
|
351
|
+
| Builtins | `abs`, `min(a, b, ...)`, `max(a, b, ...)`, `int(x)`, `float(x)`, `pow(a, b)`, `len(array)` |
|
|
352
|
+
| `math` | `sqrt exp log log2 log10 sin cos tan asin acos atan sinh cosh tanh fabs`, `floor ceil trunc` (return int), `isnan isinf isfinite` (in conditions), `pi e tau inf nan`. Works as `math.sqrt(x)` or `from math import sqrt` |
|
|
353
|
+
| Function calls | Calls to other plain or `@optimize` functions with int/float/array arguments, compiled into the same library. Recursion and mutual recursion with a `-> int` / `-> float` annotation |
|
|
354
|
+
| Globals | Module-level `int` / `float` constants (if one is reassigned later, calls fall back to Python) |
|
|
355
|
+
| Loops | `for` over `range(stop)`, `range(start, stop)`, `range(start, stop, step)` (negative steps too); `while`; `break`; `continue` |
|
|
356
|
+
| Branches | `if` / `elif` / `else`, early `return` |
|
|
357
|
+
| Conditions | Comparisons incl. chained `a < b < c`; exact int vs float (`i < x`); `and`, `or`, `not`; truthiness `if n:`; `while True`; `a if cond else b` |
|
|
358
|
+
| Locals | Assignment, annotated assignment, `+= -= *= //= %= /=` |
|
|
359
|
+
| Arrays | `float(a[i])`, `int(a[i])`, `a[i] = v`, `a[i] += v` (`-= *= /=`), `len(a)`; negative indices; bounds-checked |
|
|
360
|
+
| Returns | One scalar type (`int` or `float`), or no value (`None`) |
|
|
361
|
+
|
|
362
|
+
These run as normal Python instead: lists, dicts, strings, tuples, closures, generators, classes, keyword arguments in calls. Variables first assigned inside a loop aren't treated as assigned after it, which is conservative by design. When something isn't supported, `fn.explain()` and the fallback warning give the line and the code.
|
|
363
|
+
|
|
364
|
+
### Control and diagnostics
|
|
365
|
+
|
|
366
|
+
```python
|
|
367
|
+
fn.prepare() # compile now
|
|
368
|
+
fn.prepare(background=True) # compile in a thread; calls use Python meanwhile
|
|
369
|
+
fn.reset() # clear compile state and retry
|
|
370
|
+
fn.report() # calls, native/fallback counts, timings, fallback_reasons
|
|
371
|
+
fn.rust_source() # inspect the generated Rust
|
|
372
|
+
fn.explain() # human-readable: compiles or not, which line and why (0.5)
|
|
373
|
+
fn.python(...) # call the original function
|
|
374
|
+
|
|
375
|
+
@optimize(fallback=False) # raise OptimizationError instead of falling back (good for benchmarks)
|
|
376
|
+
@optimize(max_iterations=10**9) # native loop budget (default 10,000,000)
|
|
377
|
+
@optimize(native_loop_free=True) # compile loop-free functions too (default: run them as plain Python)
|
|
378
|
+
@optimize(warn_on_fallback=False) # silence PyRustFallbackWarning (default: warn once per reason)
|
|
379
|
+
```
|
|
380
|
+
|
|
381
|
+
When a call runs as plain Python, PyRust warns once per reason, for example:
|
|
382
|
+
|
|
383
|
+
```text
|
|
384
|
+
PyRustFallbackWarning: simulate ran as plain Python: simulate, line 14: unsupported expression: `[x, y]`
|
|
385
|
+
```
|
|
386
|
+
|
|
387
|
+
`fn.explain()` prints the same information on demand:
|
|
388
|
+
|
|
389
|
+
```text
|
|
390
|
+
simulate: compiles to Rust.
|
|
391
|
+
parameters: n: int, dt: float
|
|
392
|
+
returns: float
|
|
393
|
+
compiled together with: step, energy
|
|
394
|
+
```
|
|
395
|
+
|
|
396
|
+
> **Iteration budget:** a call that exceeds `max_iterations` discards its native work and **replays the whole call in Python**, losing the speedup. With 12M iterations: Python 0.450 s, default budget 0.384 s, `max_iterations=10**9` 0.015 s. Raise it for known large loops; watch `report()['fallback_reasons']`.
|
|
397
|
+
|
|
398
|
+
### Find candidates in existing code
|
|
399
|
+
|
|
400
|
+
```sh
|
|
401
|
+
python -m pyrust_engine check path/to/module.py # which functions compile, and why not (0.5)
|
|
402
|
+
python -m pyrust_engine profile path/to/script.py arg1 arg2 # runs the script once (trusted code only)
|
|
403
|
+
python -m pyrust_engine cache --days 30 # list old cache entries (add --delete)
|
|
404
|
+
```
|
|
405
|
+
|
|
406
|
+
`check` loads the file (without running its `if __name__ == "__main__":` block) and explains every function in it; add `--json` for tools. The profiler ranks functions in the entry script by time and reports which ones fit the native subset. Neither rewrites your code.
|
|
407
|
+
|
|
408
|
+
## How it works
|
|
409
|
+
|
|
410
|
+
```
|
|
411
|
+
Python source ─► AST check & type inference ─► Rust source (checked i64 / f64 ops)
|
|
412
|
+
─► rustc --crate-type cdylib (cached, hashed, locked) ─► ctypes call
|
|
413
|
+
─► status 0: return native result
|
|
414
|
+
─► status ≠ 0 (overflow, ÷0, bounds, budget, no return): replay original Python
|
|
415
|
+
```
|
|
416
|
+
|
|
417
|
+
The supported subset has no side effects, so replaying a call is always safe. Generated Rust never contains raw user text: identifiers are mapped to internal names and only allowed AST nodes are translated.
|
|
418
|
+
|
|
419
|
+
## Safety
|
|
420
|
+
|
|
421
|
+
- Run **trusted code only**. Compiled libraries run inside your process; this is not a sandbox.
|
|
422
|
+
- Keep the cache private (it must be owned by you and not group/world-writable on POSIX).
|
|
423
|
+
- Long native loops can't be interrupted by Ctrl-C until they finish or hit the iteration budget.
|
|
424
|
+
- Don't mutate arrays, function code or builtins concurrently with a running call.
|
|
425
|
+
|
|
426
|
+
## Project layout
|
|
427
|
+
|
|
428
|
+
```
|
|
429
|
+
pyrust_engine/ compiler.py (AST → Rust), runtime.py (decorator, dispatch, metrics, typing),
|
|
430
|
+
cache.py (disk cache, locks), profiling.py, __main__.py (CLI), py.typed
|
|
431
|
+
tests/ test_engine.py, test_v2.py … test_v6.py, test_fuzz.py (generated programs)
|
|
432
|
+
examples/ pyrust_vs_python.ipynb (try it), benchmark_suite.py, workloads.py
|
|
433
|
+
benchmarks/ raw benchmark results (0.2.0 – 0.6.0; 0.6.0 for Linux, macOS and Windows)
|
|
434
|
+
docs/ API.md (reference), benchmark chart and the script that generates it
|
|
435
|
+
.github/ CI (tests on 3 OSes + package build), PyPI publish workflow, issue templates
|
|
436
|
+
```
|
|
437
|
+
|
|
438
|
+
Regenerate the chart after a new benchmark run:
|
|
439
|
+
|
|
440
|
+
```sh
|
|
441
|
+
python -m examples.benchmark_suite --output benchmarks/my-results.json
|
|
442
|
+
python docs/make_chart.py benchmarks/my-results.json docs/benchmark-speedup.svg
|
|
443
|
+
```
|
|
444
|
+
|
|
445
|
+
## What's missing for developers (v0.6)
|
|
446
|
+
|
|
447
|
+
Done in 0.6: array writes, `int64` arrays, recursion, exact int/float comparisons, type hints (`py.typed`), the [API reference](docs/API.md), the [contributor guide](CONTRIBUTING.md) with issue templates, an updated [notebook](examples/pyrust_vs_python.ipynb), GitHub Releases, and benchmarks for macOS and Windows. These gaps remain:
|
|
448
|
+
|
|
449
|
+
| Missing | Why it matters to a developer |
|
|
450
|
+
|---|---|
|
|
451
|
+
| **Not on PyPI yet** | The package builds and passes `twine check` in CI, and a publish workflow is ready, but the PyPI project still has to be set up by its owner. Until then, install from GitHub. |
|
|
452
|
+
| **Needs Rust installed on every machine** | PyRust compiles *your* functions, so `rustc` is needed wherever they should run natively. Without it, code still runs as plain Python. |
|
|
453
|
+
| **Lists, tuples, dicts and strings** | Code that builds lists or returns several values still runs as plain Python. |
|
|
454
|
+
| **2-D arrays and other dtypes** | Only 1-D `float64` / `int64` arrays are compiled. |
|
|
455
|
+
| **Call overhead** | Each native call costs about 2–4 µs, which matters for small inputs called very often. |
|
|
456
|
+
|
|
457
|
+
## Roadmap
|
|
458
|
+
|
|
459
|
+
What needs improving, most impactful first:
|
|
460
|
+
|
|
461
|
+
1. **Lower native call overhead (2–4 µs → well under 1 µs).** Replace the `ctypes` call with a small CPython extension entry point, and generate a specialised argument checker per function.
|
|
462
|
+
2. **Typed lists and tuples** (`list[float]` arguments, local tuples, multiple return values).
|
|
463
|
+
3. **2-D arrays** and `float32` / `int32` / `bool` element types.
|
|
464
|
+
4. **Parallel loops** (`prange`) that use every CPU core, since native code already releases the GIL.
|
|
465
|
+
5. **Adaptive dispatch.** Measure native vs Python time for the first calls and route small inputs (e.g. `n = 3`) to Python automatically.
|
|
466
|
+
6. **PyPI release** (waiting on the one-time PyPI setup; see `.github/workflows/publish.yml`).
|
|
467
|
+
7. **Benchmarks on ARM Linux**, and against Cython, PyPy and Codon.
|
|
468
|
+
|
|
469
|
+
See [`CHANGELOG.md`](CHANGELOG.md) for release history.
|
|
470
|
+
|
|
471
|
+
## License
|
|
472
|
+
|
|
473
|
+
[MIT](LICENSE) © 2026 Suraj Mhaske
|