mxframe 0.2.3__tar.gz → 0.3.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {mxframe-0.2.3 → mxframe-0.3.0}/CHANGELOG.md +21 -0
- mxframe-0.3.0/PKG-INFO +152 -0
- mxframe-0.3.0/README.md +119 -0
- {mxframe-0.2.3 → mxframe-0.3.0}/__init__.py +1 -1
- {mxframe-0.2.3 → mxframe-0.3.0}/custom_ops.py +14 -1
- {mxframe-0.2.3 → mxframe-0.3.0}/docs/vision-and-architecture.md +17 -40
- mxframe-0.3.0/kernels.mojopkg +0 -0
- {mxframe-0.2.3 → mxframe-0.3.0}/kernels_aot/libmxkernels_aot.so +0 -0
- {mxframe-0.2.3 → mxframe-0.3.0}/kernels_aot/libmxkernels_aot_gpu.so +0 -0
- mxframe-0.3.0/mxframe.egg-info/PKG-INFO +152 -0
- {mxframe-0.2.3 → mxframe-0.3.0}/pyproject.toml +1 -1
- mxframe-0.2.3/PKG-INFO +0 -572
- mxframe-0.2.3/README.md +0 -539
- mxframe-0.2.3/kernels.mojopkg +0 -0
- mxframe-0.2.3/mxframe.egg-info/PKG-INFO +0 -572
- {mxframe-0.2.3 → mxframe-0.3.0}/MANIFEST.in +0 -0
- {mxframe-0.2.3 → mxframe-0.3.0}/_modidx.py +0 -0
- {mxframe-0.2.3 → mxframe-0.3.0}/aot_kernels.py +0 -0
- {mxframe-0.2.3 → mxframe-0.3.0}/compiler.py +0 -0
- {mxframe-0.2.3 → mxframe-0.3.0}/kernels_aot/__init__.py +0 -0
- {mxframe-0.2.3 → mxframe-0.3.0}/lazy_expr.py +0 -0
- {mxframe-0.2.3 → mxframe-0.3.0}/lazy_frame.py +0 -0
- {mxframe-0.2.3 → mxframe-0.3.0}/mxframe.egg-info/SOURCES.txt +0 -0
- {mxframe-0.2.3 → mxframe-0.3.0}/mxframe.egg-info/dependency_links.txt +0 -0
- {mxframe-0.2.3 → mxframe-0.3.0}/mxframe.egg-info/requires.txt +0 -0
- {mxframe-0.2.3 → mxframe-0.3.0}/mxframe.egg-info/top_level.txt +0 -0
- {mxframe-0.2.3 → mxframe-0.3.0}/optimizer.py +0 -0
- {mxframe-0.2.3 → mxframe-0.3.0}/plan_validation.py +0 -0
- {mxframe-0.2.3 → mxframe-0.3.0}/setup.cfg +0 -0
- {mxframe-0.2.3 → mxframe-0.3.0}/setup.py +0 -0
- {mxframe-0.2.3 → mxframe-0.3.0}/sql_frontend.py +0 -0
|
@@ -5,6 +5,27 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
|
|
5
5
|
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
+
## [0.3.0] — 2026-07-10
|
|
9
|
+
|
|
10
|
+
### 📚 Docs & Benchmarks
|
|
11
|
+
- README restructured: 543 lines → 119 lines. Intro, quick start, and 4-row teaser only — all detail moved to dedicated docs.
|
|
12
|
+
- New `docs/benchmarks.md`: full 22-query TPC-H tables (1 M + 10 M rows), Mojo kernel catalogue, current limitations, roadmap, and reproduce instructions.
|
|
13
|
+
- New `docs/api.md`: LazyFrame method reference, expression syntax, SQL frontend, supported operations, project structure.
|
|
14
|
+
- `docs/vision-and-architecture.md`: fixed garbled sections 2.5/2.6/3; success criteria table repaired; all milestones marked ✅.
|
|
15
|
+
- `docs/archive/` added to `.gitignore` — old dev notes moved there, not published.
|
|
16
|
+
- All stale `Mojo 0.26.2` / `MAX Graph shape-cached join` references updated to reflect v0.2.3 reality (Mojo 26.4, AOT ctypes everywhere).
|
|
17
|
+
|
|
18
|
+
### 📊 Benchmark updates
|
|
19
|
+
- Q5/Q7/Q8/Q10/Q13/Q19 re-measured with per-run join-cache clearing (fresh-join). Previous v0.2.0 numbers for those queries were warm join-cache hits.
|
|
20
|
+
- Summary corrected: CPU 20/22 (Q13 loses fresh-join), GPU 12/22 (Q5/Q7 no longer win at 1 M without cache warmup).
|
|
21
|
+
- Q8 GPU marked `—` (MAX Graph `year()` groupby JIT too slow to benchmark; Phase 6 target).
|
|
22
|
+
|
|
23
|
+
### 🔨 Infrastructure
|
|
24
|
+
- `modular >= 26.4` requirement added to `pyproject.toml` runtime deps and Dependencies table in README.
|
|
25
|
+
- `docs/archive/` gitignored; previously committed archive files removed from history.
|
|
26
|
+
|
|
27
|
+
---
|
|
28
|
+
|
|
8
29
|
## [0.2.3] — 2026-07-09
|
|
9
30
|
|
|
10
31
|
### ⚡ Performance
|
mxframe-0.3.0/PKG-INFO
ADDED
|
@@ -0,0 +1,152 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: mxframe
|
|
3
|
+
Version: 0.3.0
|
|
4
|
+
Summary: GPU-accelerated DataFrame engine: Python ergonomics + Mojo AOT kernels, cross-vendor (NVIDIA/AMD/Apple).
|
|
5
|
+
Author-email: Abhishek Sreesaila <abhisheksreesaila@users.noreply.github.com>
|
|
6
|
+
License-Expression: Apache-2.0
|
|
7
|
+
Project-URL: Homepage, https://github.com/abhisheksreesaila/mxframe
|
|
8
|
+
Project-URL: Repository, https://github.com/abhisheksreesaila/mxframe
|
|
9
|
+
Keywords: dataframe,gpu,mojo,arrow,analytics,tpch
|
|
10
|
+
Classifier: Development Status :: 3 - Alpha
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: Intended Audience :: Science/Research
|
|
13
|
+
Classifier: Programming Language :: Python :: 3
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
15
|
+
Classifier: Topic :: Scientific/Engineering :: Information Analysis
|
|
16
|
+
Classifier: Topic :: Database
|
|
17
|
+
Requires-Python: >=3.12
|
|
18
|
+
Description-Content-Type: text/markdown
|
|
19
|
+
Requires-Dist: pyarrow>=14.0
|
|
20
|
+
Requires-Dist: numpy>=1.24
|
|
21
|
+
Requires-Dist: pandas>=2.0
|
|
22
|
+
Provides-Extra: runtime
|
|
23
|
+
Requires-Dist: modular>=26.4; extra == "runtime"
|
|
24
|
+
Provides-Extra: polars
|
|
25
|
+
Requires-Dist: polars>=0.20; extra == "polars"
|
|
26
|
+
Provides-Extra: sql
|
|
27
|
+
Requires-Dist: sqlglot>=25.0; extra == "sql"
|
|
28
|
+
Provides-Extra: dev
|
|
29
|
+
Requires-Dist: pytest>=7.0; extra == "dev"
|
|
30
|
+
Requires-Dist: polars>=0.20; extra == "dev"
|
|
31
|
+
Requires-Dist: sqlglot>=25.0; extra == "dev"
|
|
32
|
+
Requires-Dist: duckdb>=0.9; extra == "dev"
|
|
33
|
+
|
|
34
|
+
# 🚀 MXFrame
|
|
35
|
+
|
|
36
|
+
> **GPU-accelerated DataFrames — Python ergonomics, Mojo speed, every GPU.**
|
|
37
|
+
|
|
38
|
+
MXFrame is a DataFrame query engine that pairs a Polars-style Python API with
|
|
39
|
+
pre-compiled Mojo AOT kernels. The same code runs on **NVIDIA, AMD, and Apple Silicon** —
|
|
40
|
+
no CUDA required, no JIT compilation at query time.
|
|
41
|
+
|
|
42
|
+
[](docs/benchmarks.md)
|
|
43
|
+
[](pyproject.toml)
|
|
44
|
+
[](pyproject.toml)
|
|
45
|
+
[](LICENSE)
|
|
46
|
+
|
|
47
|
+
---
|
|
48
|
+
|
|
49
|
+
## ✨ Why MXFrame?
|
|
50
|
+
|
|
51
|
+
| | pandas | Polars | cuDF (Rapids) | **MXFrame** |
|
|
52
|
+
|---|---|---|---|---|
|
|
53
|
+
| GPU support | ❌ | ❌ | ✅ NVIDIA only | ✅ **Any GPU** |
|
|
54
|
+
| Compiled kernels | ❌ | ✅ Rust | ✅ CUDA | ✅ **Mojo AOT** |
|
|
55
|
+
| Install complexity | pip | pip | CUDA + Rapids stack | **pixi install** |
|
|
56
|
+
| TPC-H competitive | ❌ | ✅ | ✅ | ✅ |
|
|
57
|
+
| Cross-vendor | ❌ | ❌ | ❌ | ✅ NVIDIA/AMD/Apple |
|
|
58
|
+
|
|
59
|
+
MXFrame is the **cuDF architecture without the CUDA lock-in**.
|
|
60
|
+
Kernels are compiled **once** to a `.so` at build time — loaded in ~1 ms, then pure dispatch on every query.
|
|
61
|
+
|
|
62
|
+
---
|
|
63
|
+
|
|
64
|
+
## ⚡ Quick Start
|
|
65
|
+
|
|
66
|
+
```sh
|
|
67
|
+
# 1. Install pixi (Modular's package manager)
|
|
68
|
+
curl -fsSL https://pixi.sh/install.sh | bash
|
|
69
|
+
|
|
70
|
+
# 2. Clone and set up
|
|
71
|
+
git clone https://github.com/abhisheksreesaila/mxframe
|
|
72
|
+
cd mxframe
|
|
73
|
+
pixi install
|
|
74
|
+
|
|
75
|
+
# 3. Verify GPU is working
|
|
76
|
+
pixi run python3 scripts/_check_gpu.py
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
```python
|
|
80
|
+
import pyarrow as pa
|
|
81
|
+
from mxframe import LazyFrame, Scan, col, lit
|
|
82
|
+
|
|
83
|
+
data = pa.table({
|
|
84
|
+
"dept": pa.array(["eng", "eng", "mkt", "mkt", "eng"]),
|
|
85
|
+
"salary": pa.array([120.0, 95.0, 80.0, 110.0, 130.0], pa.float32()),
|
|
86
|
+
"age": pa.array([32, 28, 35, 29, 40], pa.int32()),
|
|
87
|
+
})
|
|
88
|
+
|
|
89
|
+
result = (
|
|
90
|
+
LazyFrame(Scan(data))
|
|
91
|
+
.filter(col("age") > lit(28))
|
|
92
|
+
.groupby("dept")
|
|
93
|
+
.agg(
|
|
94
|
+
col("salary").sum().alias("total_salary"),
|
|
95
|
+
col("age").count().alias("headcount"),
|
|
96
|
+
)
|
|
97
|
+
.sort(col("total_salary"), descending=True)
|
|
98
|
+
.compute(device="gpu") # or "cpu"
|
|
99
|
+
)
|
|
100
|
+
print(result.to_pandas())
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
```
|
|
104
|
+
dept total_salary headcount
|
|
105
|
+
0 eng 345.0 3
|
|
106
|
+
1 mkt 110.0 1
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
---
|
|
110
|
+
|
|
111
|
+
## 📊 Performance — TPC-H at 1 M rows *(v0.3.0)*
|
|
112
|
+
|
|
113
|
+
MX CPU beats Polars on **20/22** queries. GPU advantage compounds at 10 M+ rows where kernel parallelism dominates PCIe upload cost.
|
|
114
|
+
|
|
115
|
+
| Query | MX CPU | MX GPU | Polars | CPU vs Polars |
|
|
116
|
+
|---|---:|---:|---:|---:|
|
|
117
|
+
| Q9 · Product profit (6-table join) | **0.6 ms** | 6.6 ms | 39.9 ms | **67×** |
|
|
118
|
+
| Q12 · 2-table join + agg | **0.5 ms** | 3.4 ms | 23.2 ms | **46×** |
|
|
119
|
+
| Q3 · 3-table join + agg | **2.6 ms** | 8.7 ms | 23.3 ms | **9×** |
|
|
120
|
+
| Q1 · Filter + 8 aggregations | **10.7 ms** | 42.1 ms | 33.3 ms | **3.1×** |
|
|
121
|
+
|
|
122
|
+
→ [Full 22-query tables (1 M + 10 M rows), methodology, and reproduce instructions](docs/benchmarks.md)
|
|
123
|
+
|
|
124
|
+
---
|
|
125
|
+
|
|
126
|
+
## 📚 Docs
|
|
127
|
+
|
|
128
|
+
| | |
|
|
129
|
+
|---|---|
|
|
130
|
+
| [**API Reference**](docs/api.md) | LazyFrame, expressions, SQL frontend, supported operations, running tests |
|
|
131
|
+
| [**Benchmarks**](docs/benchmarks.md) | Full TPC-H tables, kernel catalogue, limitations, roadmap, reproduce instructions |
|
|
132
|
+
| [**Architecture**](docs/vision-and-architecture.md) | Design philosophy, internal layers, how MAX Graph fits in |
|
|
133
|
+
| [**Contributing**](CONTRIBUTING.md) | Dev setup, writing Mojo kernels, adding queries |
|
|
134
|
+
|
|
135
|
+
---
|
|
136
|
+
|
|
137
|
+
## 📦 Dependencies
|
|
138
|
+
|
|
139
|
+
| Package | Required | Purpose |
|
|
140
|
+
|---|---|---|
|
|
141
|
+
| `pyarrow >= 14` | ✅ | Column storage, zero-copy NumPy bridge |
|
|
142
|
+
| `numpy >= 1.24` | ✅ | Vectorized pre/post processing |
|
|
143
|
+
| `pandas >= 2.0` | ✅ | Reference implementations |
|
|
144
|
+
| `modular >= 26.4` | GPU only | MAX Engine runtime, Mojo GPU dispatch |
|
|
145
|
+
| `polars >= 0.20` | optional | Benchmark comparison |
|
|
146
|
+
| `sqlglot >= 25` | optional | SQL frontend parsing |
|
|
147
|
+
|
|
148
|
+
---
|
|
149
|
+
|
|
150
|
+
## 📄 License
|
|
151
|
+
|
|
152
|
+
Apache 2.0 — see [LICENSE](LICENSE).
|
mxframe-0.3.0/README.md
ADDED
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
# 🚀 MXFrame
|
|
2
|
+
|
|
3
|
+
> **GPU-accelerated DataFrames — Python ergonomics, Mojo speed, every GPU.**
|
|
4
|
+
|
|
5
|
+
MXFrame is a DataFrame query engine that pairs a Polars-style Python API with
|
|
6
|
+
pre-compiled Mojo AOT kernels. The same code runs on **NVIDIA, AMD, and Apple Silicon** —
|
|
7
|
+
no CUDA required, no JIT compilation at query time.
|
|
8
|
+
|
|
9
|
+
[](docs/benchmarks.md)
|
|
10
|
+
[](pyproject.toml)
|
|
11
|
+
[](pyproject.toml)
|
|
12
|
+
[](LICENSE)
|
|
13
|
+
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
## ✨ Why MXFrame?
|
|
17
|
+
|
|
18
|
+
| | pandas | Polars | cuDF (Rapids) | **MXFrame** |
|
|
19
|
+
|---|---|---|---|---|
|
|
20
|
+
| GPU support | ❌ | ❌ | ✅ NVIDIA only | ✅ **Any GPU** |
|
|
21
|
+
| Compiled kernels | ❌ | ✅ Rust | ✅ CUDA | ✅ **Mojo AOT** |
|
|
22
|
+
| Install complexity | pip | pip | CUDA + Rapids stack | **pixi install** |
|
|
23
|
+
| TPC-H competitive | ❌ | ✅ | ✅ | ✅ |
|
|
24
|
+
| Cross-vendor | ❌ | ❌ | ❌ | ✅ NVIDIA/AMD/Apple |
|
|
25
|
+
|
|
26
|
+
MXFrame is the **cuDF architecture without the CUDA lock-in**.
|
|
27
|
+
Kernels are compiled **once** to a `.so` at build time — loaded in ~1 ms, then pure dispatch on every query.
|
|
28
|
+
|
|
29
|
+
---
|
|
30
|
+
|
|
31
|
+
## ⚡ Quick Start
|
|
32
|
+
|
|
33
|
+
```sh
|
|
34
|
+
# 1. Install pixi (Modular's package manager)
|
|
35
|
+
curl -fsSL https://pixi.sh/install.sh | bash
|
|
36
|
+
|
|
37
|
+
# 2. Clone and set up
|
|
38
|
+
git clone https://github.com/abhisheksreesaila/mxframe
|
|
39
|
+
cd mxframe
|
|
40
|
+
pixi install
|
|
41
|
+
|
|
42
|
+
# 3. Verify GPU is working
|
|
43
|
+
pixi run python3 scripts/_check_gpu.py
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
```python
|
|
47
|
+
import pyarrow as pa
|
|
48
|
+
from mxframe import LazyFrame, Scan, col, lit
|
|
49
|
+
|
|
50
|
+
data = pa.table({
|
|
51
|
+
"dept": pa.array(["eng", "eng", "mkt", "mkt", "eng"]),
|
|
52
|
+
"salary": pa.array([120.0, 95.0, 80.0, 110.0, 130.0], pa.float32()),
|
|
53
|
+
"age": pa.array([32, 28, 35, 29, 40], pa.int32()),
|
|
54
|
+
})
|
|
55
|
+
|
|
56
|
+
result = (
|
|
57
|
+
LazyFrame(Scan(data))
|
|
58
|
+
.filter(col("age") > lit(28))
|
|
59
|
+
.groupby("dept")
|
|
60
|
+
.agg(
|
|
61
|
+
col("salary").sum().alias("total_salary"),
|
|
62
|
+
col("age").count().alias("headcount"),
|
|
63
|
+
)
|
|
64
|
+
.sort(col("total_salary"), descending=True)
|
|
65
|
+
.compute(device="gpu") # or "cpu"
|
|
66
|
+
)
|
|
67
|
+
print(result.to_pandas())
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
```
|
|
71
|
+
dept total_salary headcount
|
|
72
|
+
0 eng 345.0 3
|
|
73
|
+
1 mkt 110.0 1
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
---
|
|
77
|
+
|
|
78
|
+
## 📊 Performance — TPC-H at 1 M rows *(v0.3.0)*
|
|
79
|
+
|
|
80
|
+
MX CPU beats Polars on **20/22** queries. GPU advantage compounds at 10 M+ rows where kernel parallelism dominates PCIe upload cost.
|
|
81
|
+
|
|
82
|
+
| Query | MX CPU | MX GPU | Polars | CPU vs Polars |
|
|
83
|
+
|---|---:|---:|---:|---:|
|
|
84
|
+
| Q9 · Product profit (6-table join) | **0.6 ms** | 6.6 ms | 39.9 ms | **67×** |
|
|
85
|
+
| Q12 · 2-table join + agg | **0.5 ms** | 3.4 ms | 23.2 ms | **46×** |
|
|
86
|
+
| Q3 · 3-table join + agg | **2.6 ms** | 8.7 ms | 23.3 ms | **9×** |
|
|
87
|
+
| Q1 · Filter + 8 aggregations | **10.7 ms** | 42.1 ms | 33.3 ms | **3.1×** |
|
|
88
|
+
|
|
89
|
+
→ [Full 22-query tables (1 M + 10 M rows), methodology, and reproduce instructions](docs/benchmarks.md)
|
|
90
|
+
|
|
91
|
+
---
|
|
92
|
+
|
|
93
|
+
## 📚 Docs
|
|
94
|
+
|
|
95
|
+
| | |
|
|
96
|
+
|---|---|
|
|
97
|
+
| [**API Reference**](docs/api.md) | LazyFrame, expressions, SQL frontend, supported operations, running tests |
|
|
98
|
+
| [**Benchmarks**](docs/benchmarks.md) | Full TPC-H tables, kernel catalogue, limitations, roadmap, reproduce instructions |
|
|
99
|
+
| [**Architecture**](docs/vision-and-architecture.md) | Design philosophy, internal layers, how MAX Graph fits in |
|
|
100
|
+
| [**Contributing**](CONTRIBUTING.md) | Dev setup, writing Mojo kernels, adding queries |
|
|
101
|
+
|
|
102
|
+
---
|
|
103
|
+
|
|
104
|
+
## 📦 Dependencies
|
|
105
|
+
|
|
106
|
+
| Package | Required | Purpose |
|
|
107
|
+
|---|---|---|
|
|
108
|
+
| `pyarrow >= 14` | ✅ | Column storage, zero-copy NumPy bridge |
|
|
109
|
+
| `numpy >= 1.24` | ✅ | Vectorized pre/post processing |
|
|
110
|
+
| `pandas >= 2.0` | ✅ | Reference implementations |
|
|
111
|
+
| `modular >= 26.4` | GPU only | MAX Engine runtime, Mojo GPU dispatch |
|
|
112
|
+
| `polars >= 0.20` | optional | Benchmark comparison |
|
|
113
|
+
| `sqlglot >= 25` | optional | SQL frontend parsing |
|
|
114
|
+
|
|
115
|
+
---
|
|
116
|
+
|
|
117
|
+
## 📄 License
|
|
118
|
+
|
|
119
|
+
Apache 2.0 — see [LICENSE](LICENSE).
|
|
@@ -1508,7 +1508,20 @@ class CustomOpsCompiler(GraphCompiler):
|
|
|
1508
1508
|
# derived columns (e.g. __cw_0__, __cw_1__ from case_when substitution).
|
|
1509
1509
|
# Engine slots (__group_ids__, __filter_mask__) live in extra_inputs, not
|
|
1510
1510
|
# col_arrays, so no exclusion filter is needed here.
|
|
1511
|
-
|
|
1511
|
+
# Perf: skip string/object dtype columns — they are group keys, not agg
|
|
1512
|
+
# values; converting numpy object arrays (800K strings) to pa.array is O(N)
|
|
1513
|
+
# slow and would dominate hot-call time.
|
|
1514
|
+
mini_cols = {}
|
|
1515
|
+
for k, v in col_arrays.items():
|
|
1516
|
+
if v.dtype.kind in ("f", "i", "u"): # float, int, uint only
|
|
1517
|
+
mini_cols[k] = pa.array(v)
|
|
1518
|
+
else:
|
|
1519
|
+
# String/object columns: preserve as-is via zero-copy pa.array
|
|
1520
|
+
# (stays as dictionary/string type, no Python iteration needed)
|
|
1521
|
+
try:
|
|
1522
|
+
mini_cols[k] = pa.array(v, from_pandas=True)
|
|
1523
|
+
except Exception:
|
|
1524
|
+
pass # skip columns that can't be converted
|
|
1512
1525
|
mini_table = pa.table(mini_cols) if mini_cols else None
|
|
1513
1526
|
|
|
1514
1527
|
agg_names: list = []
|
|
@@ -182,47 +182,24 @@ LogicalPlan tree ←── identical to what DataFrame API builds
|
|
|
182
182
|
│
|
|
183
183
|
▼ (compiler compiles)
|
|
184
184
|
|
|
185
|
-
MAX Graph → Mojo kernels → result
|
|
186
|
-
|
|
185
|
+
MAX Graph → Mojo kernels → result
|
|
187
186
|
```
|
|
188
187
|
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
We use [`sqlglot`](https://github.com/tobymao/sqlglot) (already in `pixi.toml`) for parsing.
|
|
188
|
+
We use [`sqlglot`](https://github.com/tobymao/sqlglot) for parsing. It handles SQL dialects, produces a clean AST, and we translate from `sqlglot.Expression` → our `LogicalPlan` nodes.
|
|
192
189
|
|
|
193
|
-
|
|
190
|
+
**Why this works:** Our `LogicalPlan` is the universal intermediate representation. Whether the user writes DataFrame API or SQL, both produce the same plan tree — the compiler is completely unaware of which frontend was used.
|
|
194
191
|
|
|
195
|
-
|
|
192
|
+
### 2.6 Why MAX Graph (Not Direct Mojo FFI)?
|
|
196
193
|
|
|
194
|
+
MAX Graph provides:
|
|
197
195
|
- **Session caching** — compiled graphs are cached, subsequent runs are instant
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
Whether the user writes:- **Device placement** — the graph runtime handles CPU↔GPU data transfers
|
|
202
|
-
|
|
196
|
+
- **Mixed ops** — combine built-in MAX ops with custom Mojo kernels in one graph
|
|
197
|
+
- **Device placement** — the graph runtime handles CPU↔GPU data transfers
|
|
203
198
|
- **Fusion opportunities** — MAX can fuse adjacent ops into single kernel launches
|
|
204
199
|
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
# DataFrame APIMAX Graph provides:
|
|
208
|
-
|
|
209
|
-
df.filter(col('x') > 30).groupby('y').agg(col('z').sum().alias('total')).compute()
|
|
210
|
-
|
|
211
|
-
### 2.6 Why MAX Graph (Not Direct Mojo FFI)?
|
|
212
|
-
|
|
213
|
-
# SQL API
|
|
214
|
-
|
|
215
|
-
mx.sql("SELECT y, sum(z) AS total FROM t WHERE x > 30 GROUP BY y", t=table)completely unaware of which frontend was used.
|
|
216
|
-
|
|
217
|
-
```The compiler, the Mojo kernels, the GPU dispatch — everything downstream is
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
Both produce:```
|
|
222
|
-
|
|
223
|
-
``` └─ Scan(table)
|
|
200
|
+
## 3. API Design Philosophy
|
|
224
201
|
|
|
225
|
-
|
|
202
|
+
### 3.1 Syntax — Polars-Inspired, Simplified
|
|
226
203
|
|
|
227
204
|
```python
|
|
228
205
|
import mxframe as mx
|
|
@@ -294,11 +271,11 @@ TPC-H has 22 queries. We target them incrementally:
|
|
|
294
271
|
|
|
295
272
|
## 5. Success Criteria
|
|
296
273
|
|
|
297
|
-
| Milestone | Definition of Done |
|
|
298
|
-
|
|
299
|
-
| **v0.1 — Core Pipeline** | `filter` + `select` + `groupby` + `sum/min/max/mean/count`
|
|
300
|
-
| **v0.2 — GPU Path** | Same ops run on GPU with automatic device selection |
|
|
301
|
-
| **v0.3 — TPC-H Q1 & Q6** | Both queries run correctly and beat pandas in speed |
|
|
302
|
-
| **v0.4 — Joins & Sort** | Hash join, sort, limit plan nodes working |
|
|
303
|
-
| **v0.5 — TPC-H Q3** | 3-way join query runs correctly |
|
|
304
|
-
| **v1.0 — Public Release** |
|
|
274
|
+
| Milestone | Definition of Done | Status |
|
|
275
|
+
|-----------|-------------------|--------|
|
|
276
|
+
| **v0.1 — Core Pipeline** | `filter` + `select` + `groupby` + `sum/min/max/mean/count` compile to MAX Graph, correct results on CPU | ✅ Done |
|
|
277
|
+
| **v0.2 — GPU Path** | Same ops run on GPU with automatic device selection | ✅ Done |
|
|
278
|
+
| **v0.3 — TPC-H Q1 & Q6** | Both queries run correctly and beat pandas in speed | ✅ Done |
|
|
279
|
+
| **v0.4 — Joins & Sort** | Hash join, sort, limit plan nodes working | ✅ Done |
|
|
280
|
+
| **v0.5 — TPC-H Q3** | 3-way join query runs correctly | ✅ Done |
|
|
281
|
+
| **v1.0 — Public Release** | 22/22 TPC-H queries, docs site, pip-installable, benchmarks published | ✅ **v0.2.3** (22/22 queries, all docs live) |
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
@@ -0,0 +1,152 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: mxframe
|
|
3
|
+
Version: 0.3.0
|
|
4
|
+
Summary: GPU-accelerated DataFrame engine: Python ergonomics + Mojo AOT kernels, cross-vendor (NVIDIA/AMD/Apple).
|
|
5
|
+
Author-email: Abhishek Sreesaila <abhisheksreesaila@users.noreply.github.com>
|
|
6
|
+
License-Expression: Apache-2.0
|
|
7
|
+
Project-URL: Homepage, https://github.com/abhisheksreesaila/mxframe
|
|
8
|
+
Project-URL: Repository, https://github.com/abhisheksreesaila/mxframe
|
|
9
|
+
Keywords: dataframe,gpu,mojo,arrow,analytics,tpch
|
|
10
|
+
Classifier: Development Status :: 3 - Alpha
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: Intended Audience :: Science/Research
|
|
13
|
+
Classifier: Programming Language :: Python :: 3
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
15
|
+
Classifier: Topic :: Scientific/Engineering :: Information Analysis
|
|
16
|
+
Classifier: Topic :: Database
|
|
17
|
+
Requires-Python: >=3.12
|
|
18
|
+
Description-Content-Type: text/markdown
|
|
19
|
+
Requires-Dist: pyarrow>=14.0
|
|
20
|
+
Requires-Dist: numpy>=1.24
|
|
21
|
+
Requires-Dist: pandas>=2.0
|
|
22
|
+
Provides-Extra: runtime
|
|
23
|
+
Requires-Dist: modular>=26.4; extra == "runtime"
|
|
24
|
+
Provides-Extra: polars
|
|
25
|
+
Requires-Dist: polars>=0.20; extra == "polars"
|
|
26
|
+
Provides-Extra: sql
|
|
27
|
+
Requires-Dist: sqlglot>=25.0; extra == "sql"
|
|
28
|
+
Provides-Extra: dev
|
|
29
|
+
Requires-Dist: pytest>=7.0; extra == "dev"
|
|
30
|
+
Requires-Dist: polars>=0.20; extra == "dev"
|
|
31
|
+
Requires-Dist: sqlglot>=25.0; extra == "dev"
|
|
32
|
+
Requires-Dist: duckdb>=0.9; extra == "dev"
|
|
33
|
+
|
|
34
|
+
# 🚀 MXFrame
|
|
35
|
+
|
|
36
|
+
> **GPU-accelerated DataFrames — Python ergonomics, Mojo speed, every GPU.**
|
|
37
|
+
|
|
38
|
+
MXFrame is a DataFrame query engine that pairs a Polars-style Python API with
|
|
39
|
+
pre-compiled Mojo AOT kernels. The same code runs on **NVIDIA, AMD, and Apple Silicon** —
|
|
40
|
+
no CUDA required, no JIT compilation at query time.
|
|
41
|
+
|
|
42
|
+
[](docs/benchmarks.md)
|
|
43
|
+
[](pyproject.toml)
|
|
44
|
+
[](pyproject.toml)
|
|
45
|
+
[](LICENSE)
|
|
46
|
+
|
|
47
|
+
---
|
|
48
|
+
|
|
49
|
+
## ✨ Why MXFrame?
|
|
50
|
+
|
|
51
|
+
| | pandas | Polars | cuDF (Rapids) | **MXFrame** |
|
|
52
|
+
|---|---|---|---|---|
|
|
53
|
+
| GPU support | ❌ | ❌ | ✅ NVIDIA only | ✅ **Any GPU** |
|
|
54
|
+
| Compiled kernels | ❌ | ✅ Rust | ✅ CUDA | ✅ **Mojo AOT** |
|
|
55
|
+
| Install complexity | pip | pip | CUDA + Rapids stack | **pixi install** |
|
|
56
|
+
| TPC-H competitive | ❌ | ✅ | ✅ | ✅ |
|
|
57
|
+
| Cross-vendor | ❌ | ❌ | ❌ | ✅ NVIDIA/AMD/Apple |
|
|
58
|
+
|
|
59
|
+
MXFrame is the **cuDF architecture without the CUDA lock-in**.
|
|
60
|
+
Kernels are compiled **once** to a `.so` at build time — loaded in ~1 ms, then pure dispatch on every query.
|
|
61
|
+
|
|
62
|
+
---
|
|
63
|
+
|
|
64
|
+
## ⚡ Quick Start
|
|
65
|
+
|
|
66
|
+
```sh
|
|
67
|
+
# 1. Install pixi (Modular's package manager)
|
|
68
|
+
curl -fsSL https://pixi.sh/install.sh | bash
|
|
69
|
+
|
|
70
|
+
# 2. Clone and set up
|
|
71
|
+
git clone https://github.com/abhisheksreesaila/mxframe
|
|
72
|
+
cd mxframe
|
|
73
|
+
pixi install
|
|
74
|
+
|
|
75
|
+
# 3. Verify GPU is working
|
|
76
|
+
pixi run python3 scripts/_check_gpu.py
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
```python
|
|
80
|
+
import pyarrow as pa
|
|
81
|
+
from mxframe import LazyFrame, Scan, col, lit
|
|
82
|
+
|
|
83
|
+
data = pa.table({
|
|
84
|
+
"dept": pa.array(["eng", "eng", "mkt", "mkt", "eng"]),
|
|
85
|
+
"salary": pa.array([120.0, 95.0, 80.0, 110.0, 130.0], pa.float32()),
|
|
86
|
+
"age": pa.array([32, 28, 35, 29, 40], pa.int32()),
|
|
87
|
+
})
|
|
88
|
+
|
|
89
|
+
result = (
|
|
90
|
+
LazyFrame(Scan(data))
|
|
91
|
+
.filter(col("age") > lit(28))
|
|
92
|
+
.groupby("dept")
|
|
93
|
+
.agg(
|
|
94
|
+
col("salary").sum().alias("total_salary"),
|
|
95
|
+
col("age").count().alias("headcount"),
|
|
96
|
+
)
|
|
97
|
+
.sort(col("total_salary"), descending=True)
|
|
98
|
+
.compute(device="gpu") # or "cpu"
|
|
99
|
+
)
|
|
100
|
+
print(result.to_pandas())
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
```
|
|
104
|
+
dept total_salary headcount
|
|
105
|
+
0 eng 345.0 3
|
|
106
|
+
1 mkt 110.0 1
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
---
|
|
110
|
+
|
|
111
|
+
## 📊 Performance — TPC-H at 1 M rows *(v0.3.0)*
|
|
112
|
+
|
|
113
|
+
MX CPU beats Polars on **20/22** queries. GPU advantage compounds at 10 M+ rows where kernel parallelism dominates PCIe upload cost.
|
|
114
|
+
|
|
115
|
+
| Query | MX CPU | MX GPU | Polars | CPU vs Polars |
|
|
116
|
+
|---|---:|---:|---:|---:|
|
|
117
|
+
| Q9 · Product profit (6-table join) | **0.6 ms** | 6.6 ms | 39.9 ms | **67×** |
|
|
118
|
+
| Q12 · 2-table join + agg | **0.5 ms** | 3.4 ms | 23.2 ms | **46×** |
|
|
119
|
+
| Q3 · 3-table join + agg | **2.6 ms** | 8.7 ms | 23.3 ms | **9×** |
|
|
120
|
+
| Q1 · Filter + 8 aggregations | **10.7 ms** | 42.1 ms | 33.3 ms | **3.1×** |
|
|
121
|
+
|
|
122
|
+
→ [Full 22-query tables (1 M + 10 M rows), methodology, and reproduce instructions](docs/benchmarks.md)
|
|
123
|
+
|
|
124
|
+
---
|
|
125
|
+
|
|
126
|
+
## 📚 Docs
|
|
127
|
+
|
|
128
|
+
| | |
|
|
129
|
+
|---|---|
|
|
130
|
+
| [**API Reference**](docs/api.md) | LazyFrame, expressions, SQL frontend, supported operations, running tests |
|
|
131
|
+
| [**Benchmarks**](docs/benchmarks.md) | Full TPC-H tables, kernel catalogue, limitations, roadmap, reproduce instructions |
|
|
132
|
+
| [**Architecture**](docs/vision-and-architecture.md) | Design philosophy, internal layers, how MAX Graph fits in |
|
|
133
|
+
| [**Contributing**](CONTRIBUTING.md) | Dev setup, writing Mojo kernels, adding queries |
|
|
134
|
+
|
|
135
|
+
---
|
|
136
|
+
|
|
137
|
+
## 📦 Dependencies
|
|
138
|
+
|
|
139
|
+
| Package | Required | Purpose |
|
|
140
|
+
|---|---|---|
|
|
141
|
+
| `pyarrow >= 14` | ✅ | Column storage, zero-copy NumPy bridge |
|
|
142
|
+
| `numpy >= 1.24` | ✅ | Vectorized pre/post processing |
|
|
143
|
+
| `pandas >= 2.0` | ✅ | Reference implementations |
|
|
144
|
+
| `modular >= 26.4` | GPU only | MAX Engine runtime, Mojo GPU dispatch |
|
|
145
|
+
| `polars >= 0.20` | optional | Benchmark comparison |
|
|
146
|
+
| `sqlglot >= 25` | optional | SQL frontend parsing |
|
|
147
|
+
|
|
148
|
+
---
|
|
149
|
+
|
|
150
|
+
## 📄 License
|
|
151
|
+
|
|
152
|
+
Apache 2.0 — see [LICENSE](LICENSE).
|
|
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|
|
4
4
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "mxframe"
|
|
7
|
-
version = "0.
|
|
7
|
+
version = "0.3.0"
|
|
8
8
|
description = "GPU-accelerated DataFrame engine: Python ergonomics + Mojo AOT kernels, cross-vendor (NVIDIA/AMD/Apple)."
|
|
9
9
|
readme = "README.md"
|
|
10
10
|
requires-python = ">=3.12"
|