mxframe 0.2.3__tar.gz → 0.5.0__tar.gz

mxframe-0.5.0/CHANGELOG.md

@@ -0,0 +1,160 @@
+# 📋 Changelog
+
+All notable changes to MXFrame are documented here.  
+Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
+
+---
+
+## [0.5.0] — 2026-07-11
+
+Full narrative: [v0.5.0 release notes](RELEASE_NOTES_0.5.0.md).
+
+### ⚡ Native GPU UTF-8 predicates
+- Added AOT `utf8_startswith_mask_gpu`, `utf8_contains_mask_gpu`, `utf8_equal_mask_gpu`, and packed-literal `utf8_isin_mask_gpu` kernels over Arrow validity, offsets, and byte buffers.
+- Supports sliced arrays, nulls, empty strings/patterns, duplicate literals, and multibyte UTF-8 while matching Arrow filter semantics.
+- GPU expression dispatch covers string `==`, `!=`, `startswith`, literal `contains`, and non-null literal `isin`; unsupported/null-membership cases retain Arrow fallback.
+- Added permanent CPU/GPU Phase 6 tests and `pixi run bench-utf8` comparisons against PyArrow and Polars.
+- At 1M rows, packed `isin` and `contains` beat local Arrow and Polars; exact equality remains faster in Polars' optimized CPU path, so no universal string-performance claim is made.
+
+### ⚡ Weak-query GPU fusion
+- Q6 dispatches `range5_sum_product_f32_gpu` instead of materializing five masks and a product array.
+- Q4 uses `semi_range_group_count_i32_gpu` for fused qualifying-key marking, semi-join probing, and grouped counts.
+- Q13 uses `key_match_counts_i32_gpu` to emit one compact count per customer without materializing a left join.
+- Q21 uses `single_late_supplier_counts_i32_gpu` for exact `(order, supplier)` deduplication and deterministic compact supplier counts.
+- Representative 1M/10M GPU results versus Polars: Q4 **3.4/28.6 vs 20.2/105.4 ms**; Q13 **1.8/28.5 vs 26.5/442.0 ms**; Q21 **6.6/49.8 vs 30.2/99.1 ms**.
+
+### 📊 Bounded 22-query benchmark
+- Every query and engine now runs in its own process, releasing Arrow, Pandas, and CUDA allocations between measurements.
+- Added per-engine timeouts, five-query progress groups, merged CSV output, and optional synchronized `cudf.pandas` workers.
+- MX CPU beat Polars on **22/22 at 1M** and **18/22 at 10M**.
+- MX GPU beat Polars on **15/17 comparable paths at 1M** and **14/17 at 10M**.
+- Q8 GPU remains `N/A:JIT`; Q15/Q17/Q18/Q20 GPU workers exceeded the bounded timeout.
+- cuDF was unavailable locally, so RAPIDS remains `N/A` and no RAPIDS parity claim is made.
+
+### 🎬 Visualization and identity
+- Q4, Q6, Q12, Q13, Q14, and Q21 visualize their logical rewrite, dispatch boundary, and step-by-step Mojo kernel algorithm.
+- Added a cool-gray MX mark whose rising X stroke is shaped as orange lightning.
+- Verified responsive rendering and zero horizontal overflow in Playwright.
+
+### Remaining performance-parity work
+- Replace Q8's JIT path and fix Q15/Q17/Q18/Q20 GPU timeouts.
+- Improve Q1 and Q14 GPU pipelines at 10M.
+- Run synchronized 1M/10M benchmarks on a compatible RAPIDS/cuDF environment.
+
+### Remaining feature-parity work
+- Native GPU UTF-8 dictionary construction and string/date gather.
+- GPU windows and additional semi/anti/cross/as-of/range joins.
+- Broader regex/string, datetime, nested-type, reshape, SQL, streaming, and I/O pushdown support.
+- Cross-generation NVIDIA testing plus AMD and Apple Silicon validation.
+
+---
+
+## [0.4.0] — 2026-07-11
+
+### 🔗 String and composite joins
+- Join keys from both tables now use one shared Arrow dictionary before dispatch, so equal strings receive identical dense `int32` IDs regardless of encounter order.
+- Multi-column joins jointly densify shared per-column codes, avoiding mixed-radix overflow and supporting mixed string/integer composite keys.
+- SQL null semantics are preserved: null keys receive side-specific IDs and never match, including left joins.
+- Compact nonnegative integer joins retain their direct `int32` fast path; sparse and non-integer keys are safely densified.
+- Fixed GPU upload-cache aliasing by retaining each cached NumPy host owner, preventing recycled host pointers from returning stale device buffers during joined-table gather.
+- String encoding runs in Arrow on CPU; resulting IDs use the Mojo AOT CPU/GPU join kernels. Native GPU UTF-8 dictionary construction remains future work.
+
+---
+
+## [0.3.0] — 2026-07-10
+
+### 📚 Docs & Benchmarks
+- README restructured: 543 lines → 119 lines. Intro, quick start, and 4-row teaser only — all detail moved to dedicated docs.
+- New `docs/benchmarks.md`: full 22-query TPC-H tables (1 M + 10 M rows), Mojo kernel catalogue, current limitations, roadmap, and reproduce instructions.
+- New `docs/api.md`: LazyFrame method reference, expression syntax, SQL frontend, supported operations, project structure.
+- `docs/vision-and-architecture.md`: fixed garbled sections 2.5/2.6/3; success criteria table repaired; all milestones marked ✅.
+- `docs/archive/` added to `.gitignore` — old dev notes moved there, not published.
+- All stale `Mojo 0.26.2` / `MAX Graph shape-cached join` references updated to reflect v0.2.3 reality (Mojo 26.4, AOT ctypes everywhere).
+
+### 📊 Benchmark updates
+- Q5/Q7/Q8/Q10/Q13/Q19 re-measured with per-run join-cache clearing (fresh-join). Previous v0.2.0 numbers for those queries were warm join-cache hits.
+- Summary corrected: CPU 20/22 (Q13 loses fresh-join), GPU 12/22 (Q5/Q7 no longer win at 1 M without cache warmup).
+- Q8 GPU marked `—` (MAX Graph `year()` groupby JIT too slow to benchmark; Phase 6 target).
+
+### 🔨 Infrastructure
+- `modular >= 26.4` requirement added to `pyproject.toml` runtime deps and Dependencies table in README.
+- `docs/archive/` gitignored; previously committed archive files removed from history.
+
+---
+
+## [0.2.3] — 2026-07-09
+
+### ⚡ Performance
+- **Phase 4 — Device-resident GPU join pipeline** (`join_count_i32_gpu` + `join_scatter_i32_gpu`):
+  GPU inner joins now bypass MAX Graph JIT entirely. Cold-start join latency drops from ~300 ms (MAX session init + JIT) to ~70 ms. `AOTKernelsGPU.hash_join()` caches key arrays via `_cached_upload`; `AOTKernelsGPU.gather_table()` gathers all numeric columns with a single shared device-resident index upload.
+- **Phase 5 — GPU sort + top-k** (`sort_topk_i32_gpu`): `ORDER BY … LIMIT k` fuses sort and limit in a single AOT bitonic sort kernel, downloading only `k × 4` bytes instead of `N × 4`. `_apply_post_ops_custom` peek-ahead passes `topk=k` directly to the sort kernel.
+- **Phase 3 — Vectorised rank mapping**: `_encode_sort_key` replaces the per-unique-value Python `for` loop with a single vectorised NumPy scatter (`rank_of[sorted_order] = np.arange(n_unique)`). ~1.5× speedup for sort-key encoding at 5 M rows.
+
+### 🛠 Fixed
+- **GPU sort+limit crash**: `_apply_sort_custom` was calling `_gpu_gather_table` which referenced `gather_f32` as a MAX Graph custom op — that kernel was never registered in the mojopkg. Any `ORDER BY` on `device="gpu"` would crash. Now routes through `_aot_gpu.gather_table` (AOT ctypes, no MAX Graph).
+- `_apply_sort_custom` CPU path now uses `np.argpartition` (O(N) partial sort) when `topk < N`, instead of a full O(N log N) argsort.
+
+### 🔨 Infrastructure
+- **Mojo/MAX 26.4 migration**: all `kernels/*.mojo` and `kernels_aot/*.mojo` migrated from 26.2 API (`fn` → `def`, `from tensor import` → `from extensibility import`, `DeviceContextPtr` → `DeviceContext`, `enqueue_function_experimental` → `enqueue_function`).
+- **`kernels/_atomic.mojo`** (new): drop-in replacement for the missing `std.os.atomic.Atomic`. Implements `fetch_add`, `min`, `max`, `compare_exchange` via MLIR `llvm.atomicrmw` / `llvm.cmpxchg` instructions — portable across NVIDIA and AMD; works in both `mojo package` (JIT) and `mojo build --emit shared-lib` (AOT) contexts.
+- All compiled artifacts rebuilt with Mojo 1.0.0b2 (modular 26.4): `kernels.mojopkg`, `libmxkernels_aot.so`, `libmxkernels_aot_gpu.so`.
+
+---
+
+## [0.2.2] — 2026-07-08
+
+### 🔨 Infrastructure
+- First release targeting **Mojo/MAX 26.4** (`modular = "26.4.*"` in `pixi.toml`).
+- **`kernels/_atomic.mojo`** shim introduced to replace the removed `std.os.atomic` module.
+- All kernel sources migrated to 26.4 syntax; compiled artefacts rebuilt.
+- `pyproject.toml` `runtime` extra pinned to `modular>=26.4` to prevent 26.2 wheels from being installed by Colab/pip users.
+
+---
+
+## [0.2.1] — 2026-06-xx
+
+### 🛠 Fixed
+- TestPyPI re-upload to validate wheel packaging. No functional changes.
+
+---
+
+## [0.2.0] — 2026-05-xx
+
+### ✨ Added
+- **Phase 1 — GPU buffer cache + masked global min/max**:
+  - `masked_global_min_f32_gpu` / `masked_global_max_f32_gpu` Mojo AOT kernels.
+  - `AOTKernelsGPU` device-buffer cache (`_cached_upload`, eviction, `clear_buf_cache`) — eliminates redundant PCIe uploads on repeated hot queries.
+  - `group_mean_f32` GPU kernel (sum + count on GPU, divide on CPU).
+- **Phase 2 — Masked global aggregation for any expression**:
+  - `_compute_masked_global_agg` extended to handle any inner expression via `_eval_expr_arrow` (e.g. `a*(1-b)`, `case_when`, `startswith`, `isin`) — SUM reduction stays GPU-resident.
+  - Join-only plans emit `"join_mojo_shortcut"` provenance (not `"pyarrow_shortcut"`) — Q19 restructured to single-pass join+filter+global_agg.
+- **Phase 3 — GPU integer key encoding**:
+  - `group_encode_i32_gpu` open-addressing hash-table kernel for integer group keys; replaces PyArrow `dictionary_encode` for integer-keyed group-by on GPU.
+- **Audit tool** (`scripts/audit_gpu_paths.py`): classifies each of 22 TPC-H queries as `GPU-CLEAN` / `MOJO-CPU` / `FALLBACK` via `last_compile_provenance["path"]`.
+
+### ⚡ Performance
+- **22/22 TPC-H queries GPU-CLEAN** (100% Mojo-backed on GPU, zero PyArrow fallbacks in the reduction and join hot paths).
+
+---
+
+## [0.1.2] — 2026-04-22
+
+### 🛠 Fixed
+- `pip install "mxframe[runtime]"` now gives a working install. The `[runtime]` extra pulls `modular>=25.5`, which provides the `max.engine` / `max.driver` modules that `mxframe.compiler` imports at module load. (Pixi users get `modular` from the Modular conda channel and should install `mxframe` without the extra.)
+- `sqlglot` import in `mxframe.sql_frontend` is now lazy: `import mxframe` no longer requires the `[sql]` extra. Calling `mxframe.sql(…)` without `sqlglot` installed raises a clear `ImportError` with the install hint.
+- `__init__.py` `__version__` now matches `pyproject.toml`.
+
+---
+
+## [0.1.1] — 2026-04-22
+
+### 🛠 Fixed
+- Packaging: `readme` now points to `README.md` so PyPI shows the full project page with benchmarks and quickstart (was accidentally showing the internal vision/architecture doc).
+- CI: `AOTKernelsGPU` init no longer hard-fails on CPU-only runners (`CUDA driver unavailable` is caught, `self._aot_gpu` falls back to `None`).
+- CPU-only import and compute path validated in a clean venv without `modular`.
+
+---
+
+## [0.1.0] — 2026-04-21
+
+Initial public release.

mxframe-0.5.0/PKG-INFO

@@ -0,0 +1,188 @@
+Metadata-Version: 2.4
+Name: mxframe
+Version: 0.5.0
+Summary: GPU-accelerated DataFrame engine with Python ergonomics and Mojo AOT kernels.
+Author-email: Abhishek Sreesaila <abhisheksreesaila@users.noreply.github.com>
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/abhisheksreesaila/mxframe
+Project-URL: Repository, https://github.com/abhisheksreesaila/mxframe
+Keywords: dataframe,gpu,mojo,arrow,analytics,tpch
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Database
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: pyarrow>=14.0
+Requires-Dist: numpy>=1.24
+Requires-Dist: pandas>=2.0
+Provides-Extra: runtime
+Requires-Dist: modular>=26.4; extra == "runtime"
+Provides-Extra: polars
+Requires-Dist: polars>=0.20; extra == "polars"
+Provides-Extra: sql
+Requires-Dist: sqlglot>=25.0; extra == "sql"
+Provides-Extra: dev
+Requires-Dist: build>=1.2; extra == "dev"
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: polars>=0.20; extra == "dev"
+Requires-Dist: sqlglot>=25.0; extra == "dev"
+Requires-Dist: duckdb>=0.9; extra == "dev"
+Requires-Dist: twine>=6.0; extra == "dev"
+
+<img src="visualize/mx-spark.svg" alt="MXFrame MX lightning mark" width="72">
+
+# 🚀 MXFrame
+
+> **GPU-accelerated DataFrames — Python ergonomics, Mojo AOT kernels.**
+
+MXFrame is a DataFrame query engine that pairs a Polars-style Python API with pre-compiled Mojo AOT kernels. The current GPU build is validated on **NVIDIA**; AMD and Apple Silicon validation remain on the roadmap.
+
+[![TPC-H](https://img.shields.io/badge/TPC--H-22%2F22%20queries-brightgreen)](docs/benchmarks.md)
+[![Python](https://img.shields.io/badge/python-3.12%2B-blue)](pyproject.toml)
+[![Platform](https://img.shields.io/badge/platform-linux--x86__64-lightgrey)](pyproject.toml)
+[![License](https://img.shields.io/badge/license-Apache%202.0-blue)](LICENSE)
+
+---
+
+## ✨ Why MXFrame?
+
+| | pandas | Polars | cuDF (RAPIDS) | **MXFrame** |
+|---|---|---|---|---|
+| GPU support | ❌ | ❌ | ✅ NVIDIA | ✅ **NVIDIA validated** |
+| Compiled kernels | ❌ | ✅ Rust | ✅ CUDA | ✅ **Mojo AOT** |
+| Install complexity | pip | pip | CUDA + RAPIDS stack | **pip or pixi** |
+| TPC-H coverage | reference | ✅ | ✅ | ✅ **22/22** |
+| Portable kernel design | ❌ | CPU | CUDA | ✅ NVIDIA today; AMD/Apple planned |
+
+MXFrame uses a portable Mojo kernel architecture rather than CUDA-specific source. Kernels compile once to shared libraries and dispatch directly at runtime.
+
+### What changed in v0.4 and v0.5
+
+- **v0.4.0 — correct string/composite joins:** shared dictionaries assign identical dense IDs across both join sides; mixed composite keys and SQL null semantics are preserved.
+- **v0.5.0 — native GPU UTF-8 predicates:** `startswith`, literal `contains`, string equality/inequality, and packed non-null literal `isin` operate directly on Arrow offsets and byte buffers.
+- **v0.5.0 — weak-query fusion:** dedicated compact GPU paths remove expanded intermediates from Q4, Q6, Q13, and Q21.
+- **v0.5.0 — bounded benchmarking and visualization:** every query/engine runs in an isolated process; Q4, Q6, Q12, Q13, Q14, and Q21 have step-by-step kernel visualizations.
+
+See [v0.5.0 release notes](RELEASE_NOTES_0.5.0.md) for measured gains, limitations, and the parity roadmap.
+
+---
+
+## ⚡ Quick Start
+
+```sh
+# 1. Install pixi (Modular's package manager)
+curl -fsSL https://pixi.sh/install.sh | bash
+
+# 2. Clone and set up
+git clone https://github.com/abhisheksreesaila/mxframe
+cd mxframe
+pixi install
+
+# 3. Verify GPU is working
+pixi run python3 scripts/_check_gpu.py
+```
+
+```python
+import pyarrow as pa
+from mxframe import LazyFrame, Scan, col, lit
+
+data = pa.table({
+    "dept":   pa.array(["eng", "eng", "mkt", "mkt", "eng"]),
+    "salary": pa.array([120.0, 95.0, 80.0, 110.0, 130.0], pa.float32()),
+    "age":    pa.array([32, 28, 35, 29, 40], pa.int32()),
+})
+
+result = (
+    LazyFrame(Scan(data))
+    .filter(col("age") > lit(28))
+    .groupby("dept")
+    .agg(
+        col("salary").sum().alias("total_salary"),
+        col("age").count().alias("headcount"),
+    )
+    .sort(col("total_salary"), descending=True)
+    .compute(device="gpu")   # or "cpu"
+)
+print(result.to_pandas())
+```
+
+```
+  dept  total_salary  headcount
+0  eng         345.0          3
+1  mkt         110.0          1
+```
+
+---
+
+## 📊 Performance — bounded v0.5.0 benchmark
+
+RTX 3090, warm median of three runs. Every query and engine ran in its own process so Arrow, Pandas, and CUDA allocations were released between measurements.
+
+- **1M:** MX CPU beat Polars on **22/22**; MX GPU beat Polars on **15/17 comparable** paths.
+- **10M:** MX CPU beat Polars on **18/22**; MX GPU beat Polars on **14/17 comparable** paths.
+- Q8 GPU remains `N/A:JIT`; Q15, Q17, Q18, and Q20 GPU workers exceeded the bounded timeout.
+- cuDF was unavailable on this machine, so RAPIDS remains `N/A` and no RAPIDS parity claim is made.
+
+### 1M representative queries
+
+| Query | MX CPU | MX GPU | Polars | Best MX speedup |
+|---|---:|---:|---:|---:|
+| Q4 · Order priority | 12.8 ms | **3.4 ms** | 20.2 ms | **5.9× GPU** |
+| Q6 · Discounted revenue | **6.7 ms** | 6.8 ms | 10.8 ms | **1.6× CPU** |
+| Q12 · Shipping modes | **0.6 ms** | 6.1 ms | 26.6 ms | **44.3× CPU** |
+| Q13 · Customer distribution | 20.4 ms | **1.8 ms** | 26.5 ms | **14.7× GPU** |
+| Q21 · Waiting suppliers | 27.1 ms | **6.6 ms** | 30.2 ms | **4.6× GPU** |
+
+### 10M representative queries
+
+| Query | MX CPU | MX GPU | Polars | Best MX speedup |
+|---|---:|---:|---:|---:|
+| Q4 · Order priority | 265.1 ms | **28.6 ms** | 105.4 ms | **3.7× GPU** |
+| Q9 · Product profit | **2.0 ms** | 16.5 ms | 109.7 ms | **54.9× CPU** |
+| Q12 · Shipping modes | **1.5 ms** | 8.2 ms | 128.0 ms | **85.3× CPU** |
+| Q13 · Customer distribution | 368.8 ms | **28.5 ms** | 442.0 ms | **15.5× GPU** |
+| Q21 · Waiting suppliers | 1068.7 ms | **49.8 ms** | 99.1 ms | **2.0× GPU** |
+
+Full matrices: [1M CSV](scripts/bench_results_1M.csv) · [10M CSV](scripts/bench_results_10M.csv) · [methodology and kernel catalogue](docs/benchmarks.md)
+
+## What remains for parity
+
+**Performance parity:** replace Q8's JIT path; fix Q15/Q17/Q18/Q20 GPU timeouts; improve Q1 and Q14 GPU execution; run synchronized 1M/10M measurements on a compatible RAPIDS/cuDF environment.
+
+**Feature parity:** native GPU UTF-8 dictionary construction and string/date gather; GPU windows; semi/anti/as-of/range joins; broader string/regex, datetime, nested-type, reshape, SQL, streaming, and I/O pushdown coverage.
+
+---
+
+## 📚 Docs
+
+| | |
+|---|---|
+| [**v0.5.0 Release Notes**](RELEASE_NOTES_0.5.0.md) | v0.4/v0.5 gains, benchmark evidence, known gaps, and parity roadmap |
+| [**API Reference**](docs/api.md) | LazyFrame, expressions, SQL frontend, supported operations, running tests |
+| [**Benchmarks**](docs/benchmarks.md) | Full TPC-H tables, kernel catalogue, limitations, roadmap, reproduce instructions |
+| [**Architecture**](docs/vision-and-architecture.md) | Design philosophy, internal layers, how MAX Graph fits in |
+| [**Kernel Visualizer**](visualize/mxframe_pipeline.html) | Interactive plans, dispatch boundaries, GPU threads, and output masks |
+| [**Contributing**](CONTRIBUTING.md) | Dev setup, writing Mojo kernels, adding queries |
+
+---
+
+## 📦 Dependencies
+
+| Package | Required | Purpose |
+|---|---|---|
+| `pyarrow >= 14` | ✅ | Column storage, zero-copy NumPy bridge |
+| `numpy >= 1.24` | ✅ | Vectorized pre/post processing |
+| `pandas >= 2.0` | ✅ | Reference implementations |
+| `modular >= 26.4` | GPU only | MAX Engine runtime, Mojo GPU dispatch |
+| `polars >= 0.20` | optional | Benchmark comparison |
+| `sqlglot >= 25` | optional | SQL frontend parsing |
+
+---
+
+## 📄 License
+
+Apache 2.0 — see [LICENSE](LICENSE).

mxframe-0.5.0/README.md

@@ -0,0 +1,153 @@
+<img src="visualize/mx-spark.svg" alt="MXFrame MX lightning mark" width="72">
+
+# 🚀 MXFrame
+
+> **GPU-accelerated DataFrames — Python ergonomics, Mojo AOT kernels.**
+
+MXFrame is a DataFrame query engine that pairs a Polars-style Python API with pre-compiled Mojo AOT kernels. The current GPU build is validated on **NVIDIA**; AMD and Apple Silicon validation remain on the roadmap.
+
+[![TPC-H](https://img.shields.io/badge/TPC--H-22%2F22%20queries-brightgreen)](docs/benchmarks.md)
+[![Python](https://img.shields.io/badge/python-3.12%2B-blue)](pyproject.toml)
+[![Platform](https://img.shields.io/badge/platform-linux--x86__64-lightgrey)](pyproject.toml)
+[![License](https://img.shields.io/badge/license-Apache%202.0-blue)](LICENSE)
+
+---
+
+## ✨ Why MXFrame?
+
+| | pandas | Polars | cuDF (RAPIDS) | **MXFrame** |
+|---|---|---|---|---|
+| GPU support | ❌ | ❌ | ✅ NVIDIA | ✅ **NVIDIA validated** |
+| Compiled kernels | ❌ | ✅ Rust | ✅ CUDA | ✅ **Mojo AOT** |
+| Install complexity | pip | pip | CUDA + RAPIDS stack | **pip or pixi** |
+| TPC-H coverage | reference | ✅ | ✅ | ✅ **22/22** |
+| Portable kernel design | ❌ | CPU | CUDA | ✅ NVIDIA today; AMD/Apple planned |
+
+MXFrame uses a portable Mojo kernel architecture rather than CUDA-specific source. Kernels compile once to shared libraries and dispatch directly at runtime.
+
+### What changed in v0.4 and v0.5
+
+- **v0.4.0 — correct string/composite joins:** shared dictionaries assign identical dense IDs across both join sides; mixed composite keys and SQL null semantics are preserved.
+- **v0.5.0 — native GPU UTF-8 predicates:** `startswith`, literal `contains`, string equality/inequality, and packed non-null literal `isin` operate directly on Arrow offsets and byte buffers.
+- **v0.5.0 — weak-query fusion:** dedicated compact GPU paths remove expanded intermediates from Q4, Q6, Q13, and Q21.
+- **v0.5.0 — bounded benchmarking and visualization:** every query/engine runs in an isolated process; Q4, Q6, Q12, Q13, Q14, and Q21 have step-by-step kernel visualizations.
+
+See [v0.5.0 release notes](RELEASE_NOTES_0.5.0.md) for measured gains, limitations, and the parity roadmap.
+
+---
+
+## ⚡ Quick Start
+
+```sh
+# 1. Install pixi (Modular's package manager)
+curl -fsSL https://pixi.sh/install.sh | bash
+
+# 2. Clone and set up
+git clone https://github.com/abhisheksreesaila/mxframe
+cd mxframe
+pixi install
+
+# 3. Verify GPU is working
+pixi run python3 scripts/_check_gpu.py
+```
+
+```python
+import pyarrow as pa
+from mxframe import LazyFrame, Scan, col, lit
+
+data = pa.table({
+    "dept":   pa.array(["eng", "eng", "mkt", "mkt", "eng"]),
+    "salary": pa.array([120.0, 95.0, 80.0, 110.0, 130.0], pa.float32()),
+    "age":    pa.array([32, 28, 35, 29, 40], pa.int32()),
+})
+
+result = (
+    LazyFrame(Scan(data))
+    .filter(col("age") > lit(28))
+    .groupby("dept")
+    .agg(
+        col("salary").sum().alias("total_salary"),
+        col("age").count().alias("headcount"),
+    )
+    .sort(col("total_salary"), descending=True)
+    .compute(device="gpu")   # or "cpu"
+)
+print(result.to_pandas())
+```
+
+```
+  dept  total_salary  headcount
+0  eng         345.0          3
+1  mkt         110.0          1
+```
+
+---
+
+## 📊 Performance — bounded v0.5.0 benchmark
+
+RTX 3090, warm median of three runs. Every query and engine ran in its own process so Arrow, Pandas, and CUDA allocations were released between measurements.
+
+- **1M:** MX CPU beat Polars on **22/22**; MX GPU beat Polars on **15/17 comparable** paths.
+- **10M:** MX CPU beat Polars on **18/22**; MX GPU beat Polars on **14/17 comparable** paths.
+- Q8 GPU remains `N/A:JIT`; Q15, Q17, Q18, and Q20 GPU workers exceeded the bounded timeout.
+- cuDF was unavailable on this machine, so RAPIDS remains `N/A` and no RAPIDS parity claim is made.
+
+### 1M representative queries
+
+| Query | MX CPU | MX GPU | Polars | Best MX speedup |
+|---|---:|---:|---:|---:|
+| Q4 · Order priority | 12.8 ms | **3.4 ms** | 20.2 ms | **5.9× GPU** |
+| Q6 · Discounted revenue | **6.7 ms** | 6.8 ms | 10.8 ms | **1.6× CPU** |
+| Q12 · Shipping modes | **0.6 ms** | 6.1 ms | 26.6 ms | **44.3× CPU** |
+| Q13 · Customer distribution | 20.4 ms | **1.8 ms** | 26.5 ms | **14.7× GPU** |
+| Q21 · Waiting suppliers | 27.1 ms | **6.6 ms** | 30.2 ms | **4.6× GPU** |
+
+### 10M representative queries
+
+| Query | MX CPU | MX GPU | Polars | Best MX speedup |
+|---|---:|---:|---:|---:|
+| Q4 · Order priority | 265.1 ms | **28.6 ms** | 105.4 ms | **3.7× GPU** |
+| Q9 · Product profit | **2.0 ms** | 16.5 ms | 109.7 ms | **54.9× CPU** |
+| Q12 · Shipping modes | **1.5 ms** | 8.2 ms | 128.0 ms | **85.3× CPU** |
+| Q13 · Customer distribution | 368.8 ms | **28.5 ms** | 442.0 ms | **15.5× GPU** |
+| Q21 · Waiting suppliers | 1068.7 ms | **49.8 ms** | 99.1 ms | **2.0× GPU** |
+
+Full matrices: [1M CSV](scripts/bench_results_1M.csv) · [10M CSV](scripts/bench_results_10M.csv) · [methodology and kernel catalogue](docs/benchmarks.md)
+
+## What remains for parity
+
+**Performance parity:** replace Q8's JIT path; fix Q15/Q17/Q18/Q20 GPU timeouts; improve Q1 and Q14 GPU execution; run synchronized 1M/10M measurements on a compatible RAPIDS/cuDF environment.
+
+**Feature parity:** native GPU UTF-8 dictionary construction and string/date gather; GPU windows; semi/anti/as-of/range joins; broader string/regex, datetime, nested-type, reshape, SQL, streaming, and I/O pushdown coverage.
+
+---
+
+## 📚 Docs
+
+| | |
+|---|---|
+| [**v0.5.0 Release Notes**](RELEASE_NOTES_0.5.0.md) | v0.4/v0.5 gains, benchmark evidence, known gaps, and parity roadmap |
+| [**API Reference**](docs/api.md) | LazyFrame, expressions, SQL frontend, supported operations, running tests |
+| [**Benchmarks**](docs/benchmarks.md) | Full TPC-H tables, kernel catalogue, limitations, roadmap, reproduce instructions |
+| [**Architecture**](docs/vision-and-architecture.md) | Design philosophy, internal layers, how MAX Graph fits in |
+| [**Kernel Visualizer**](visualize/mxframe_pipeline.html) | Interactive plans, dispatch boundaries, GPU threads, and output masks |
+| [**Contributing**](CONTRIBUTING.md) | Dev setup, writing Mojo kernels, adding queries |
+
+---
+
+## 📦 Dependencies
+
+| Package | Required | Purpose |
+|---|---|---|
+| `pyarrow >= 14` | ✅ | Column storage, zero-copy NumPy bridge |
+| `numpy >= 1.24` | ✅ | Vectorized pre/post processing |
+| `pandas >= 2.0` | ✅ | Reference implementations |
+| `modular >= 26.4` | GPU only | MAX Engine runtime, Mojo GPU dispatch |
+| `polars >= 0.20` | optional | Benchmark comparison |
+| `sqlglot >= 25` | optional | SQL frontend parsing |
+
+---
+
+## 📄 License
+
+Apache 2.0 — see [LICENSE](LICENSE).

{mxframe-0.2.3 → mxframe-0.5.0}/__init__.py

@@ -1,6 +1,6 @@
 """mxframe - GPU-accelerated DataFrames with MAX Engine"""
 
-__version__ = "0.2.3"
+__version__ = "0.5.0"
 
 # Lazy expression layer
 from .lazy_expr import Expr, col, lit, when, row_number
@@ -13,17 +13,41 @@ from .lazy_frame import (
     DeviceType,
 )
 
-# Compiler
-from .compiler import GraphCompiler
-
 # Optimizer
 from .optimizer import PlanOptimizer, OptimizationResult, optimize_plan
 
 # Plan validation
 from .plan_validation import PlanValidationError, validate_plan, validate_plan_or_raise
 
-# Custom ops compiler
-from .custom_ops import CustomOpsCompiler, KERNELS_PATH, clear_cache
+_RUNTIME_EXPORTS = {"GraphCompiler", "CustomOpsCompiler", "KERNELS_PATH"}
+
+
+def _runtime_import_error() -> ImportError:
+    return ImportError(
+        "MXFrame computation requires the Modular MAX runtime. "
+        "Install with: pip install 'mxframe[runtime]'"
+    )
+
+
+def __getattr__(name: str):
+    if name not in _RUNTIME_EXPORTS:
+        raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+    try:
+        if name == "GraphCompiler":
+            from .compiler import GraphCompiler
+            return GraphCompiler
+        from .custom_ops import CustomOpsCompiler, KERNELS_PATH
+        return CustomOpsCompiler if name == "CustomOpsCompiler" else KERNELS_PATH
+    except ImportError as exc:
+        raise _runtime_import_error() from exc
+
+
+def clear_cache() -> None:
+    try:
+        from .custom_ops import clear_cache as _clear_cache
+    except ImportError as exc:
+        raise _runtime_import_error() from exc
+    _clear_cache()
 
 # SQL frontend (optional; requires `sqlglot`).
 # Imported lazily so `pip install mxframe` works without the sql extra.
@@ -59,6 +83,11 @@ def warmup(device: str = "auto") -> float:
     import pyarrow as _pa
     import numpy as _np
 
+    try:
+        from .custom_ops import CustomOpsCompiler
+    except ImportError as exc:
+        raise _runtime_import_error() from exc
+
     t0 = _t.perf_counter()
 
     # Step 1: session init (cached -- effectively free on repeat calls)