mxframe 0.2.3__tar.gz → 0.3.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (31) hide show
  1. {mxframe-0.2.3 → mxframe-0.3.0}/CHANGELOG.md +21 -0
  2. mxframe-0.3.0/PKG-INFO +152 -0
  3. mxframe-0.3.0/README.md +119 -0
  4. {mxframe-0.2.3 → mxframe-0.3.0}/__init__.py +1 -1
  5. {mxframe-0.2.3 → mxframe-0.3.0}/custom_ops.py +14 -1
  6. {mxframe-0.2.3 → mxframe-0.3.0}/docs/vision-and-architecture.md +17 -40
  7. mxframe-0.3.0/kernels.mojopkg +0 -0
  8. {mxframe-0.2.3 → mxframe-0.3.0}/kernels_aot/libmxkernels_aot.so +0 -0
  9. {mxframe-0.2.3 → mxframe-0.3.0}/kernels_aot/libmxkernels_aot_gpu.so +0 -0
  10. mxframe-0.3.0/mxframe.egg-info/PKG-INFO +152 -0
  11. {mxframe-0.2.3 → mxframe-0.3.0}/pyproject.toml +1 -1
  12. mxframe-0.2.3/PKG-INFO +0 -572
  13. mxframe-0.2.3/README.md +0 -539
  14. mxframe-0.2.3/kernels.mojopkg +0 -0
  15. mxframe-0.2.3/mxframe.egg-info/PKG-INFO +0 -572
  16. {mxframe-0.2.3 → mxframe-0.3.0}/MANIFEST.in +0 -0
  17. {mxframe-0.2.3 → mxframe-0.3.0}/_modidx.py +0 -0
  18. {mxframe-0.2.3 → mxframe-0.3.0}/aot_kernels.py +0 -0
  19. {mxframe-0.2.3 → mxframe-0.3.0}/compiler.py +0 -0
  20. {mxframe-0.2.3 → mxframe-0.3.0}/kernels_aot/__init__.py +0 -0
  21. {mxframe-0.2.3 → mxframe-0.3.0}/lazy_expr.py +0 -0
  22. {mxframe-0.2.3 → mxframe-0.3.0}/lazy_frame.py +0 -0
  23. {mxframe-0.2.3 → mxframe-0.3.0}/mxframe.egg-info/SOURCES.txt +0 -0
  24. {mxframe-0.2.3 → mxframe-0.3.0}/mxframe.egg-info/dependency_links.txt +0 -0
  25. {mxframe-0.2.3 → mxframe-0.3.0}/mxframe.egg-info/requires.txt +0 -0
  26. {mxframe-0.2.3 → mxframe-0.3.0}/mxframe.egg-info/top_level.txt +0 -0
  27. {mxframe-0.2.3 → mxframe-0.3.0}/optimizer.py +0 -0
  28. {mxframe-0.2.3 → mxframe-0.3.0}/plan_validation.py +0 -0
  29. {mxframe-0.2.3 → mxframe-0.3.0}/setup.cfg +0 -0
  30. {mxframe-0.2.3 → mxframe-0.3.0}/setup.py +0 -0
  31. {mxframe-0.2.3 → mxframe-0.3.0}/sql_frontend.py +0 -0
@@ -5,6 +5,27 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
5
5
 
6
6
  ---
7
7
 
8
+ ## [0.3.0] — 2026-07-10
9
+
10
+ ### 📚 Docs & Benchmarks
11
+ - README restructured: 543 lines → 119 lines. Intro, quick start, and 4-row teaser only — all detail moved to dedicated docs.
12
+ - New `docs/benchmarks.md`: full 22-query TPC-H tables (1 M + 10 M rows), Mojo kernel catalogue, current limitations, roadmap, and reproduce instructions.
13
+ - New `docs/api.md`: LazyFrame method reference, expression syntax, SQL frontend, supported operations, project structure.
14
+ - `docs/vision-and-architecture.md`: fixed garbled sections 2.5/2.6/3; success criteria table repaired; all milestones marked ✅.
15
+ - `docs/archive/` added to `.gitignore` — old dev notes moved there, not published.
16
+ - All stale `Mojo 0.26.2` / `MAX Graph shape-cached join` references updated to reflect v0.2.3 reality (Mojo 26.4, AOT ctypes everywhere).
17
+
18
+ ### 📊 Benchmark updates
19
+ - Q5/Q7/Q8/Q10/Q13/Q19 re-measured with per-run join-cache clearing (fresh-join). Previous v0.2.0 numbers for those queries were warm join-cache hits.
20
+ - Summary corrected: CPU 20/22 (Q13 loses fresh-join), GPU 12/22 (Q5/Q7 no longer win at 1 M without cache warmup).
21
+ - Q8 GPU marked `—` (MAX Graph `year()` groupby JIT too slow to benchmark; Phase 6 target).
22
+
23
+ ### 🔨 Infrastructure
24
+ - `modular >= 26.4` requirement added to `pyproject.toml` runtime deps and Dependencies table in README.
25
+ - `docs/archive/` gitignored; previously committed archive files removed from history.
26
+
27
+ ---
28
+
8
29
  ## [0.2.3] — 2026-07-09
9
30
 
10
31
  ### ⚡ Performance
mxframe-0.3.0/PKG-INFO ADDED
@@ -0,0 +1,152 @@
1
+ Metadata-Version: 2.4
2
+ Name: mxframe
3
+ Version: 0.3.0
4
+ Summary: GPU-accelerated DataFrame engine: Python ergonomics + Mojo AOT kernels, cross-vendor (NVIDIA/AMD/Apple).
5
+ Author-email: Abhishek Sreesaila <abhisheksreesaila@users.noreply.github.com>
6
+ License-Expression: Apache-2.0
7
+ Project-URL: Homepage, https://github.com/abhisheksreesaila/mxframe
8
+ Project-URL: Repository, https://github.com/abhisheksreesaila/mxframe
9
+ Keywords: dataframe,gpu,mojo,arrow,analytics,tpch
10
+ Classifier: Development Status :: 3 - Alpha
11
+ Classifier: Intended Audience :: Developers
12
+ Classifier: Intended Audience :: Science/Research
13
+ Classifier: Programming Language :: Python :: 3
14
+ Classifier: Programming Language :: Python :: 3.12
15
+ Classifier: Topic :: Scientific/Engineering :: Information Analysis
16
+ Classifier: Topic :: Database
17
+ Requires-Python: >=3.12
18
+ Description-Content-Type: text/markdown
19
+ Requires-Dist: pyarrow>=14.0
20
+ Requires-Dist: numpy>=1.24
21
+ Requires-Dist: pandas>=2.0
22
+ Provides-Extra: runtime
23
+ Requires-Dist: modular>=26.4; extra == "runtime"
24
+ Provides-Extra: polars
25
+ Requires-Dist: polars>=0.20; extra == "polars"
26
+ Provides-Extra: sql
27
+ Requires-Dist: sqlglot>=25.0; extra == "sql"
28
+ Provides-Extra: dev
29
+ Requires-Dist: pytest>=7.0; extra == "dev"
30
+ Requires-Dist: polars>=0.20; extra == "dev"
31
+ Requires-Dist: sqlglot>=25.0; extra == "dev"
32
+ Requires-Dist: duckdb>=0.9; extra == "dev"
33
+
34
+ # 🚀 MXFrame
35
+
36
+ > **GPU-accelerated DataFrames — Python ergonomics, Mojo speed, every GPU.**
37
+
38
+ MXFrame is a DataFrame query engine that pairs a Polars-style Python API with
39
+ pre-compiled Mojo AOT kernels. The same code runs on **NVIDIA, AMD, and Apple Silicon** —
40
+ no CUDA required, no JIT compilation at query time.
41
+
42
+ [![TPC-H](https://img.shields.io/badge/TPC--H-22%2F22%20queries-brightgreen)](docs/benchmarks.md)
43
+ [![Python](https://img.shields.io/badge/python-3.12%2B-blue)](pyproject.toml)
44
+ [![Platform](https://img.shields.io/badge/platform-linux--x86__64-lightgrey)](pyproject.toml)
45
+ [![License](https://img.shields.io/badge/license-Apache%202.0-blue)](LICENSE)
46
+
47
+ ---
48
+
49
+ ## ✨ Why MXFrame?
50
+
51
+ | | pandas | Polars | cuDF (Rapids) | **MXFrame** |
52
+ |---|---|---|---|---|
53
+ | GPU support | ❌ | ❌ | ✅ NVIDIA only | ✅ **Any GPU** |
54
+ | Compiled kernels | ❌ | ✅ Rust | ✅ CUDA | ✅ **Mojo AOT** |
55
+ | Install complexity | pip | pip | CUDA + Rapids stack | **pixi install** |
56
+ | TPC-H competitive | ❌ | ✅ | ✅ | ✅ |
57
+ | Cross-vendor | ❌ | ❌ | ❌ | ✅ NVIDIA/AMD/Apple |
58
+
59
+ MXFrame is the **cuDF architecture without the CUDA lock-in**.
60
+ Kernels are compiled **once** to a `.so` at build time — loaded in ~1 ms, then pure dispatch on every query.
61
+
62
+ ---
63
+
64
+ ## ⚡ Quick Start
65
+
66
+ ```sh
67
+ # 1. Install pixi (Modular's package manager)
68
+ curl -fsSL https://pixi.sh/install.sh | bash
69
+
70
+ # 2. Clone and set up
71
+ git clone https://github.com/abhisheksreesaila/mxframe
72
+ cd mxframe
73
+ pixi install
74
+
75
+ # 3. Verify GPU is working
76
+ pixi run python3 scripts/_check_gpu.py
77
+ ```
78
+
79
+ ```python
80
+ import pyarrow as pa
81
+ from mxframe import LazyFrame, Scan, col, lit
82
+
83
+ data = pa.table({
84
+ "dept": pa.array(["eng", "eng", "mkt", "mkt", "eng"]),
85
+ "salary": pa.array([120.0, 95.0, 80.0, 110.0, 130.0], pa.float32()),
86
+ "age": pa.array([32, 28, 35, 29, 40], pa.int32()),
87
+ })
88
+
89
+ result = (
90
+ LazyFrame(Scan(data))
91
+ .filter(col("age") > lit(28))
92
+ .groupby("dept")
93
+ .agg(
94
+ col("salary").sum().alias("total_salary"),
95
+ col("age").count().alias("headcount"),
96
+ )
97
+ .sort(col("total_salary"), descending=True)
98
+ .compute(device="gpu") # or "cpu"
99
+ )
100
+ print(result.to_pandas())
101
+ ```
102
+
103
+ ```
104
+ dept total_salary headcount
105
+ 0 eng 345.0 3
106
+ 1 mkt 110.0 1
107
+ ```
108
+
109
+ ---
110
+
111
+ ## 📊 Performance — TPC-H at 1 M rows *(v0.3.0)*
112
+
113
+ MX CPU beats Polars on **20/22** queries. GPU advantage compounds at 10 M+ rows where kernel parallelism dominates PCIe upload cost.
114
+
115
+ | Query | MX CPU | MX GPU | Polars | CPU vs Polars |
116
+ |---|---:|---:|---:|---:|
117
+ | Q9 · Product profit (6-table join) | **0.6 ms** | 6.6 ms | 39.9 ms | **67×** |
118
+ | Q12 · 2-table join + agg | **0.5 ms** | 3.4 ms | 23.2 ms | **46×** |
119
+ | Q3 · 3-table join + agg | **2.6 ms** | 8.7 ms | 23.3 ms | **9×** |
120
+ | Q1 · Filter + 8 aggregations | **10.7 ms** | 42.1 ms | 33.3 ms | **3.1×** |
121
+
122
+ → [Full 22-query tables (1 M + 10 M rows), methodology, and reproduce instructions](docs/benchmarks.md)
123
+
124
+ ---
125
+
126
+ ## 📚 Docs
127
+
128
+ | | |
129
+ |---|---|
130
+ | [**API Reference**](docs/api.md) | LazyFrame, expressions, SQL frontend, supported operations, running tests |
131
+ | [**Benchmarks**](docs/benchmarks.md) | Full TPC-H tables, kernel catalogue, limitations, roadmap, reproduce instructions |
132
+ | [**Architecture**](docs/vision-and-architecture.md) | Design philosophy, internal layers, how MAX Graph fits in |
133
+ | [**Contributing**](CONTRIBUTING.md) | Dev setup, writing Mojo kernels, adding queries |
134
+
135
+ ---
136
+
137
+ ## 📦 Dependencies
138
+
139
+ | Package | Required | Purpose |
140
+ |---|---|---|
141
+ | `pyarrow >= 14` | ✅ | Column storage, zero-copy NumPy bridge |
142
+ | `numpy >= 1.24` | ✅ | Vectorized pre/post processing |
143
+ | `pandas >= 2.0` | ✅ | Reference implementations |
144
+ | `modular >= 26.4` | GPU only | MAX Engine runtime, Mojo GPU dispatch |
145
+ | `polars >= 0.20` | optional | Benchmark comparison |
146
+ | `sqlglot >= 25` | optional | SQL frontend parsing |
147
+
148
+ ---
149
+
150
+ ## 📄 License
151
+
152
+ Apache 2.0 — see [LICENSE](LICENSE).
@@ -0,0 +1,119 @@
1
+ # 🚀 MXFrame
2
+
3
+ > **GPU-accelerated DataFrames — Python ergonomics, Mojo speed, every GPU.**
4
+
5
+ MXFrame is a DataFrame query engine that pairs a Polars-style Python API with
6
+ pre-compiled Mojo AOT kernels. The same code runs on **NVIDIA, AMD, and Apple Silicon** —
7
+ no CUDA required, no JIT compilation at query time.
8
+
9
+ [![TPC-H](https://img.shields.io/badge/TPC--H-22%2F22%20queries-brightgreen)](docs/benchmarks.md)
10
+ [![Python](https://img.shields.io/badge/python-3.12%2B-blue)](pyproject.toml)
11
+ [![Platform](https://img.shields.io/badge/platform-linux--x86__64-lightgrey)](pyproject.toml)
12
+ [![License](https://img.shields.io/badge/license-Apache%202.0-blue)](LICENSE)
13
+
14
+ ---
15
+
16
+ ## ✨ Why MXFrame?
17
+
18
+ | | pandas | Polars | cuDF (Rapids) | **MXFrame** |
19
+ |---|---|---|---|---|
20
+ | GPU support | ❌ | ❌ | ✅ NVIDIA only | ✅ **Any GPU** |
21
+ | Compiled kernels | ❌ | ✅ Rust | ✅ CUDA | ✅ **Mojo AOT** |
22
+ | Install complexity | pip | pip | CUDA + Rapids stack | **pixi install** |
23
+ | TPC-H competitive | ❌ | ✅ | ✅ | ✅ |
24
+ | Cross-vendor | ❌ | ❌ | ❌ | ✅ NVIDIA/AMD/Apple |
25
+
26
+ MXFrame is the **cuDF architecture without the CUDA lock-in**.
27
+ Kernels are compiled **once** to a `.so` at build time — loaded in ~1 ms, then pure dispatch on every query.
28
+
29
+ ---
30
+
31
+ ## ⚡ Quick Start
32
+
33
+ ```sh
34
+ # 1. Install pixi (Modular's package manager)
35
+ curl -fsSL https://pixi.sh/install.sh | bash
36
+
37
+ # 2. Clone and set up
38
+ git clone https://github.com/abhisheksreesaila/mxframe
39
+ cd mxframe
40
+ pixi install
41
+
42
+ # 3. Verify GPU is working
43
+ pixi run python3 scripts/_check_gpu.py
44
+ ```
45
+
46
+ ```python
47
+ import pyarrow as pa
48
+ from mxframe import LazyFrame, Scan, col, lit
49
+
50
+ data = pa.table({
51
+ "dept": pa.array(["eng", "eng", "mkt", "mkt", "eng"]),
52
+ "salary": pa.array([120.0, 95.0, 80.0, 110.0, 130.0], pa.float32()),
53
+ "age": pa.array([32, 28, 35, 29, 40], pa.int32()),
54
+ })
55
+
56
+ result = (
57
+ LazyFrame(Scan(data))
58
+ .filter(col("age") > lit(28))
59
+ .groupby("dept")
60
+ .agg(
61
+ col("salary").sum().alias("total_salary"),
62
+ col("age").count().alias("headcount"),
63
+ )
64
+ .sort(col("total_salary"), descending=True)
65
+ .compute(device="gpu") # or "cpu"
66
+ )
67
+ print(result.to_pandas())
68
+ ```
69
+
70
+ ```
71
+ dept total_salary headcount
72
+ 0 eng 345.0 3
73
+ 1 mkt 110.0 1
74
+ ```
75
+
76
+ ---
77
+
78
+ ## 📊 Performance — TPC-H at 1 M rows *(v0.3.0)*
79
+
80
+ MX CPU beats Polars on **20/22** queries. GPU advantage compounds at 10 M+ rows where kernel parallelism dominates PCIe upload cost.
81
+
82
+ | Query | MX CPU | MX GPU | Polars | CPU vs Polars |
83
+ |---|---:|---:|---:|---:|
84
+ | Q9 · Product profit (6-table join) | **0.6 ms** | 6.6 ms | 39.9 ms | **67×** |
85
+ | Q12 · 2-table join + agg | **0.5 ms** | 3.4 ms | 23.2 ms | **46×** |
86
+ | Q3 · 3-table join + agg | **2.6 ms** | 8.7 ms | 23.3 ms | **9×** |
87
+ | Q1 · Filter + 8 aggregations | **10.7 ms** | 42.1 ms | 33.3 ms | **3.1×** |
88
+
89
+ → [Full 22-query tables (1 M + 10 M rows), methodology, and reproduce instructions](docs/benchmarks.md)
90
+
91
+ ---
92
+
93
+ ## 📚 Docs
94
+
95
+ | | |
96
+ |---|---|
97
+ | [**API Reference**](docs/api.md) | LazyFrame, expressions, SQL frontend, supported operations, running tests |
98
+ | [**Benchmarks**](docs/benchmarks.md) | Full TPC-H tables, kernel catalogue, limitations, roadmap, reproduce instructions |
99
+ | [**Architecture**](docs/vision-and-architecture.md) | Design philosophy, internal layers, how MAX Graph fits in |
100
+ | [**Contributing**](CONTRIBUTING.md) | Dev setup, writing Mojo kernels, adding queries |
101
+
102
+ ---
103
+
104
+ ## 📦 Dependencies
105
+
106
+ | Package | Required | Purpose |
107
+ |---|---|---|
108
+ | `pyarrow >= 14` | ✅ | Column storage, zero-copy NumPy bridge |
109
+ | `numpy >= 1.24` | ✅ | Vectorized pre/post processing |
110
+ | `pandas >= 2.0` | ✅ | Reference implementations |
111
+ | `modular >= 26.4` | GPU only | MAX Engine runtime, Mojo GPU dispatch |
112
+ | `polars >= 0.20` | optional | Benchmark comparison |
113
+ | `sqlglot >= 25` | optional | SQL frontend parsing |
114
+
115
+ ---
116
+
117
+ ## 📄 License
118
+
119
+ Apache 2.0 — see [LICENSE](LICENSE).
@@ -1,6 +1,6 @@
1
1
  """mxframe - GPU-accelerated DataFrames with MAX Engine"""
2
2
 
3
- __version__ = "0.2.3"
3
+ __version__ = "0.3.0"
4
4
 
5
5
  # Lazy expression layer
6
6
  from .lazy_expr import Expr, col, lit, when, row_number
@@ -1508,7 +1508,20 @@ class CustomOpsCompiler(GraphCompiler):
1508
1508
  # derived columns (e.g. __cw_0__, __cw_1__ from case_when substitution).
1509
1509
  # Engine slots (__group_ids__, __filter_mask__) live in extra_inputs, not
1510
1510
  # col_arrays, so no exclusion filter is needed here.
1511
- mini_cols = {k: pa.array(v) for k, v in col_arrays.items()}
1511
+ # Perf: skip string/object dtype columns — they are group keys, not agg
1512
+ # values; converting numpy object arrays (800K strings) to pa.array is O(N)
1513
+ # slow and would dominate hot-call time.
1514
+ mini_cols = {}
1515
+ for k, v in col_arrays.items():
1516
+ if v.dtype.kind in ("f", "i", "u"): # float, int, uint only
1517
+ mini_cols[k] = pa.array(v)
1518
+ else:
1519
+ # String/object columns: preserve as-is via zero-copy pa.array
1520
+ # (stays as dictionary/string type, no Python iteration needed)
1521
+ try:
1522
+ mini_cols[k] = pa.array(v, from_pandas=True)
1523
+ except Exception:
1524
+ pass # skip columns that can't be converted
1512
1525
  mini_table = pa.table(mini_cols) if mini_cols else None
1513
1526
 
1514
1527
  agg_names: list = []
@@ -182,47 +182,24 @@ LogicalPlan tree ←── identical to what DataFrame API builds
182
182
  │
183
183
  ▼ (compiler compiles)
184
184
 
185
- MAX Graph → Mojo kernels → result### 3.1 Syntax — Polars-Inspired, Simplified
186
-
185
+ MAX Graph → Mojo kernels → result
187
186
  ```
188
187
 
189
- ## 3. API Design Philosophy
190
-
191
- We use [`sqlglot`](https://github.com/tobymao/sqlglot) (already in `pixi.toml`) for parsing.
188
+ We use [`sqlglot`](https://github.com/tobymao/sqlglot) for parsing. It handles SQL dialects, produces a clean AST, and we translate from `sqlglot.Expression` → our `LogicalPlan` nodes.
192
189
 
193
- It handles SQL dialects, produces a clean AST, and we only need to write a---
190
+ **Why this works:** Our `LogicalPlan` is the universal intermediate representation. Whether the user writes DataFrame API or SQL, both produce the same plan tree — the compiler is completely unaware of which frontend was used.
194
191
 
195
- translator from `sqlglot.Expression` → our `LogicalPlan` nodes.
192
+ ### 2.6 Why MAX Graph (Not Direct Mojo FFI)?
196
193
 
194
+ MAX Graph provides:
197
195
  - **Session caching** — compiled graphs are cached, subsequent runs are instant
198
-
199
- **Why this works:** Our `LogicalPlan` is the universal intermediate representation.- **Mixed ops** — combine built-in MAX ops with custom Mojo kernels in one graph
200
-
201
- Whether the user writes:- **Device placement** — the graph runtime handles CPU↔GPU data transfers
202
-
196
+ - **Mixed ops** — combine built-in MAX ops with custom Mojo kernels in one graph
197
+ - **Device placement** — the graph runtime handles CPU↔GPU data transfers
203
198
  - **Fusion opportunities** — MAX can fuse adjacent ops into single kernel launches
204
199
 
205
- ```python- **Automatic memory management** — tensors flow through the graph, no manual alloc/free
206
-
207
- # DataFrame APIMAX Graph provides:
208
-
209
- df.filter(col('x') > 30).groupby('y').agg(col('z').sum().alias('total')).compute()
210
-
211
- ### 2.6 Why MAX Graph (Not Direct Mojo FFI)?
212
-
213
- # SQL API
214
-
215
- mx.sql("SELECT y, sum(z) AS total FROM t WHERE x > 30 GROUP BY y", t=table)completely unaware of which frontend was used.
216
-
217
- ```The compiler, the Mojo kernels, the GPU dispatch — everything downstream is
218
-
219
-
220
-
221
- Both produce:```
222
-
223
- ``` └─ Scan(table)
200
+ ## 3. API Design Philosophy
224
201
 
225
- Aggregate(group_by=[col('y')], aggs=[sum(col('z')).alias('total')]) └─ Filter(predicate=col('x') > lit(30))
202
+ ### 3.1 Syntax — Polars-Inspired, Simplified
226
203
 
227
204
  ```python
228
205
  import mxframe as mx
@@ -294,11 +271,11 @@ TPC-H has 22 queries. We target them incrementally:
294
271
 
295
272
  ## 5. Success Criteria
296
273
 
297
- | Milestone | Definition of Done |
298
- |-----------|-------------------|
299
- | **v0.1 — Core Pipeline** | `filter` + `select` + `groupby` + `sum/min/max/mean/count` all compile to MAX Graph and produce correct results on CPU |
300
- | **v0.2 — GPU Path** | Same ops run on GPU with automatic device selection |
301
- | **v0.3 — TPC-H Q1 & Q6** | Both queries run correctly and beat pandas in speed |
302
- | **v0.4 — Joins & Sort** | Hash join, sort, limit plan nodes working |
303
- | **v0.5 — TPC-H Q3** | 3-way join query runs correctly |
304
- | **v1.0 — Public Release** | 10+ TPC-H queries, docs site, pip-installable, benchmarks published |
274
+ | Milestone | Definition of Done | Status |
275
+ |-----------|-------------------|--------|
276
+ | **v0.1 — Core Pipeline** | `filter` + `select` + `groupby` + `sum/min/max/mean/count` compile to MAX Graph, correct results on CPU | ✅ Done |
277
+ | **v0.2 — GPU Path** | Same ops run on GPU with automatic device selection | ✅ Done |
278
+ | **v0.3 — TPC-H Q1 & Q6** | Both queries run correctly and beat pandas in speed | ✅ Done |
279
+ | **v0.4 — Joins & Sort** | Hash join, sort, limit plan nodes working | ✅ Done |
280
+ | **v0.5 — TPC-H Q3** | 3-way join query runs correctly | ✅ Done |
281
+ | **v1.0 — Public Release** | 22/22 TPC-H queries, docs site, pip-installable, benchmarks published | ✅ **v0.2.3** (22/22 queries, all docs live) |
Binary file
@@ -0,0 +1,152 @@
1
+ Metadata-Version: 2.4
2
+ Name: mxframe
3
+ Version: 0.3.0
4
+ Summary: GPU-accelerated DataFrame engine: Python ergonomics + Mojo AOT kernels, cross-vendor (NVIDIA/AMD/Apple).
5
+ Author-email: Abhishek Sreesaila <abhisheksreesaila@users.noreply.github.com>
6
+ License-Expression: Apache-2.0
7
+ Project-URL: Homepage, https://github.com/abhisheksreesaila/mxframe
8
+ Project-URL: Repository, https://github.com/abhisheksreesaila/mxframe
9
+ Keywords: dataframe,gpu,mojo,arrow,analytics,tpch
10
+ Classifier: Development Status :: 3 - Alpha
11
+ Classifier: Intended Audience :: Developers
12
+ Classifier: Intended Audience :: Science/Research
13
+ Classifier: Programming Language :: Python :: 3
14
+ Classifier: Programming Language :: Python :: 3.12
15
+ Classifier: Topic :: Scientific/Engineering :: Information Analysis
16
+ Classifier: Topic :: Database
17
+ Requires-Python: >=3.12
18
+ Description-Content-Type: text/markdown
19
+ Requires-Dist: pyarrow>=14.0
20
+ Requires-Dist: numpy>=1.24
21
+ Requires-Dist: pandas>=2.0
22
+ Provides-Extra: runtime
23
+ Requires-Dist: modular>=26.4; extra == "runtime"
24
+ Provides-Extra: polars
25
+ Requires-Dist: polars>=0.20; extra == "polars"
26
+ Provides-Extra: sql
27
+ Requires-Dist: sqlglot>=25.0; extra == "sql"
28
+ Provides-Extra: dev
29
+ Requires-Dist: pytest>=7.0; extra == "dev"
30
+ Requires-Dist: polars>=0.20; extra == "dev"
31
+ Requires-Dist: sqlglot>=25.0; extra == "dev"
32
+ Requires-Dist: duckdb>=0.9; extra == "dev"
33
+
34
+ # 🚀 MXFrame
35
+
36
+ > **GPU-accelerated DataFrames — Python ergonomics, Mojo speed, every GPU.**
37
+
38
+ MXFrame is a DataFrame query engine that pairs a Polars-style Python API with
39
+ pre-compiled Mojo AOT kernels. The same code runs on **NVIDIA, AMD, and Apple Silicon** —
40
+ no CUDA required, no JIT compilation at query time.
41
+
42
+ [![TPC-H](https://img.shields.io/badge/TPC--H-22%2F22%20queries-brightgreen)](docs/benchmarks.md)
43
+ [![Python](https://img.shields.io/badge/python-3.12%2B-blue)](pyproject.toml)
44
+ [![Platform](https://img.shields.io/badge/platform-linux--x86__64-lightgrey)](pyproject.toml)
45
+ [![License](https://img.shields.io/badge/license-Apache%202.0-blue)](LICENSE)
46
+
47
+ ---
48
+
49
+ ## ✨ Why MXFrame?
50
+
51
+ | | pandas | Polars | cuDF (Rapids) | **MXFrame** |
52
+ |---|---|---|---|---|
53
+ | GPU support | ❌ | ❌ | ✅ NVIDIA only | ✅ **Any GPU** |
54
+ | Compiled kernels | ❌ | ✅ Rust | ✅ CUDA | ✅ **Mojo AOT** |
55
+ | Install complexity | pip | pip | CUDA + Rapids stack | **pixi install** |
56
+ | TPC-H competitive | ❌ | ✅ | ✅ | ✅ |
57
+ | Cross-vendor | ❌ | ❌ | ❌ | ✅ NVIDIA/AMD/Apple |
58
+
59
+ MXFrame is the **cuDF architecture without the CUDA lock-in**.
60
+ Kernels are compiled **once** to a `.so` at build time — loaded in ~1 ms, then pure dispatch on every query.
61
+
62
+ ---
63
+
64
+ ## ⚡ Quick Start
65
+
66
+ ```sh
67
+ # 1. Install pixi (Modular's package manager)
68
+ curl -fsSL https://pixi.sh/install.sh | bash
69
+
70
+ # 2. Clone and set up
71
+ git clone https://github.com/abhisheksreesaila/mxframe
72
+ cd mxframe
73
+ pixi install
74
+
75
+ # 3. Verify GPU is working
76
+ pixi run python3 scripts/_check_gpu.py
77
+ ```
78
+
79
+ ```python
80
+ import pyarrow as pa
81
+ from mxframe import LazyFrame, Scan, col, lit
82
+
83
+ data = pa.table({
84
+ "dept": pa.array(["eng", "eng", "mkt", "mkt", "eng"]),
85
+ "salary": pa.array([120.0, 95.0, 80.0, 110.0, 130.0], pa.float32()),
86
+ "age": pa.array([32, 28, 35, 29, 40], pa.int32()),
87
+ })
88
+
89
+ result = (
90
+ LazyFrame(Scan(data))
91
+ .filter(col("age") > lit(28))
92
+ .groupby("dept")
93
+ .agg(
94
+ col("salary").sum().alias("total_salary"),
95
+ col("age").count().alias("headcount"),
96
+ )
97
+ .sort(col("total_salary"), descending=True)
98
+ .compute(device="gpu") # or "cpu"
99
+ )
100
+ print(result.to_pandas())
101
+ ```
102
+
103
+ ```
104
+ dept total_salary headcount
105
+ 0 eng 345.0 3
106
+ 1 mkt 110.0 1
107
+ ```
108
+
109
+ ---
110
+
111
+ ## 📊 Performance — TPC-H at 1 M rows *(v0.3.0)*
112
+
113
+ MX CPU beats Polars on **20/22** queries. GPU advantage compounds at 10 M+ rows where kernel parallelism dominates PCIe upload cost.
114
+
115
+ | Query | MX CPU | MX GPU | Polars | CPU vs Polars |
116
+ |---|---:|---:|---:|---:|
117
+ | Q9 · Product profit (6-table join) | **0.6 ms** | 6.6 ms | 39.9 ms | **67×** |
118
+ | Q12 · 2-table join + agg | **0.5 ms** | 3.4 ms | 23.2 ms | **46×** |
119
+ | Q3 · 3-table join + agg | **2.6 ms** | 8.7 ms | 23.3 ms | **9×** |
120
+ | Q1 · Filter + 8 aggregations | **10.7 ms** | 42.1 ms | 33.3 ms | **3.1×** |
121
+
122
+ → [Full 22-query tables (1 M + 10 M rows), methodology, and reproduce instructions](docs/benchmarks.md)
123
+
124
+ ---
125
+
126
+ ## 📚 Docs
127
+
128
+ | | |
129
+ |---|---|
130
+ | [**API Reference**](docs/api.md) | LazyFrame, expressions, SQL frontend, supported operations, running tests |
131
+ | [**Benchmarks**](docs/benchmarks.md) | Full TPC-H tables, kernel catalogue, limitations, roadmap, reproduce instructions |
132
+ | [**Architecture**](docs/vision-and-architecture.md) | Design philosophy, internal layers, how MAX Graph fits in |
133
+ | [**Contributing**](CONTRIBUTING.md) | Dev setup, writing Mojo kernels, adding queries |
134
+
135
+ ---
136
+
137
+ ## 📦 Dependencies
138
+
139
+ | Package | Required | Purpose |
140
+ |---|---|---|
141
+ | `pyarrow >= 14` | ✅ | Column storage, zero-copy NumPy bridge |
142
+ | `numpy >= 1.24` | ✅ | Vectorized pre/post processing |
143
+ | `pandas >= 2.0` | ✅ | Reference implementations |
144
+ | `modular >= 26.4` | GPU only | MAX Engine runtime, Mojo GPU dispatch |
145
+ | `polars >= 0.20` | optional | Benchmark comparison |
146
+ | `sqlglot >= 25` | optional | SQL frontend parsing |
147
+
148
+ ---
149
+
150
+ ## 📄 License
151
+
152
+ Apache 2.0 — see [LICENSE](LICENSE).
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
4
4
 
5
5
  [project]
6
6
  name = "mxframe"
7
- version = "0.2.3"
7
+ version = "0.3.0"
8
8
  description = "GPU-accelerated DataFrame engine: Python ergonomics + Mojo AOT kernels, cross-vendor (NVIDIA/AMD/Apple)."
9
9
  readme = "README.md"
10
10
  requires-python = ">=3.12"