mxframe 0.3.0__tar.gz → 0.6.0__tar.gz

mxframe-0.6.0/CHANGELOG.md

@@ -0,0 +1,246 @@
+# 📋 Changelog
+
+All notable changes to MXFrame are documented here.  
+Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
+
+---
+
+## [Unreleased]
+
+## [0.6.0] — 2026-10-14
+
+The CPU engine is now a native Mojo executor: one call per query stage runs the scan, filter,
+joins and aggregate over Arrow memory, compiled ahead of time. Engine-only TPC-H and a seeded
+random-query suite are the benchmarks, with every result checked against DuckDB. Numbers and the
+story behind each step: `docs/optimizations.md`.
+
+### 🧠 Native Mojo executor (F-030, F-031, M2a–M2n)
+- One native call (`libmxexec_cpu.so`, over the Arrow C Data Interface) runs a whole plan stage: a
+  bytecode interpreter for predicates and expressions, group-by specialized at compile time
+  (direct-mapped, compacted or hashed keys), and joins fused into the scan as lookups (star joins
+  in one pass, semi / anti joins without a DISTINCT, nested stars).
+- Native DISTINCT, ORDER BY ... LIMIT (per-worker heaps), WHERE-only filters, `str_slice` and
+  utf8 `isin`, COUNT(DISTINCT) on sorted runs.
+- One thread per physical core; fewer workers for small inputs; no `pyarrow.compute` or pandas
+  import on the query path; the Mojo runtime loads once.
+
+### ⚡ Closing the gaps to DuckDB (F-043, F-046)
+- Millions of groups aggregate into one shared output behind a per-worker cache; keys double as
+  ORDER BY tie-breaks; a runtime join filter for large semi-join partners.
+- Dictionary strings stay as codes through joins, single-table filters, SELECT DISTINCT and
+  COUNT(DISTINCT), and are decoded only for the rows that reach the output.
+- COUNT(DISTINCT) on unsorted keys: a byte map for small key x value ranges, hash partitions by
+  key range or by (key, value) pair, so few keys no longer stall on one worker.
+- WHERE + ORDER BY + LIMIT in one pass (`mx_filter_topk`).
+
+### 📥 Parquet, SQL and planning
+- `mx.scan_parquet` / `mx.read_parquet_table` with projection and filter pushdown and dictionary
+  strings (F-010, F-017); `mx.sql` front end on sqlglot (joins of every side, aliases, GROUP BY
+  keys by alias, F-041); `device="auto"` planner with explain and fallback (F-005).
+- Fixed: a group-by on an aliased column (F-040); a SQL `SELECT` alias on a group key (F-041).
+
+### 📊 Benchmarks
+- Engine-only TPC-H scoreboard (`scripts/bench_user_felt.py --mode engine`, also `--shuffle`d
+  rows) and a seeded random-query suite (`scripts/bench_random/`) rendered for MXFrame, Polars and
+  DuckDB, all checked against DuckDB. DuckDB's lazy pyarrow import is now staged like every
+  engine's import (it had inflated DuckDB's times).
+
+
+### 🔗 Join engine: lean, reordered, late-materialized (F-018)
+- **New `join_engine.py`.** A chain of inner joins is flattened into leaves (the inputs, filtered and cut to the columns the query reads) and key edges, joined in an order chosen from the leaves' real post-filter row counts (small filtered sides first, the larger side probes and the smaller builds), and carried as row-index vectors: a join gathers only the key columns it compares and composes the indices, every output column is gathered once at the end. Output column names and order are exactly those of the joins as written; left joins are barriers (their two sides are executed on their own). A `Filter` comparing columns of two leaves between joins (TPC-H Q5) is applied as soon as its leaves meet.
+- **Projection pushdown through joins:** each `Join` produces only the columns read above it (through `Filter`, `Project`, `Sort`, `Aggregate`, `Distinct`).
+- **Optimizer:** a filter is now optimized before the Parquet sources are read, so a predicate moved below a join lands on its `ParquetScan` and is applied while decoding (F-017). A disjunction across both sides of a join (`(a.x = 1 AND b.y = 2) OR (a.x = 3 AND b.y = 4)`, Q7, Q19) pushes what it implies to each side (`a.x IN (1, 3)`); the exact predicate stays above the join.
+- **Join keys:** a composite integer key joins on its widest column and checks the others on the matching pairs (dictionary-encoding 11M `(partkey, suppkey)` pairs took 2 s); sparse integer keys up to 32x the row count use the direct-address join instead of hashing every row into a dictionary; the prefilter kernel `libmxjoin_cpu.so` (`kernels_aot/joinsemi_cpu.mojo`) drops probe rows whose key is not in a small build side with a cache-resident bitmap before the table join (456K orders against 60M lineitem rows: 0.57 s to 0.25 s); NumPy passes over long key arrays run on several threads.
+- Strings of a small joined side (`n_name` of 25 nations over millions of rows) are gathered as dictionary codes when the consumer above is a filter, projection or aggregate.
+- A one-shot gather after a GPU join runs on the CPU (0.12 s against 0.5 s for 3.3M rows x 6 columns: the GPU one uploads its sources and downloads the result).
+- **Fixed:** the composite group id of two or more high-cardinality keys was `int32` and wrapped around, merging distinct groups (`Distinct` on `(l_orderkey, l_suppkey)` dropped 3,080 of 6M rows, TPC-H Q21 was wrong); it is `int64`, with a rank-fold fallback beyond 2^62 combinations. `count(col)` of a right-side column after a left join counts unmatched rows as 0 (regression test). `np_arrow.take_rows` no longer imports pandas for string columns.
+- **Optimizer, projection pushdown:** a new pass `prune_unused_columns` drops the outputs of a `Project` that its parent does not read. `with_columns` passes every input column through, so after a join it named every column of both tables, and `resolve_plan` read them all from Parquet (TPC-H Q8 SF10: 16 lineitem columns, 5.4 s of reading instead of 5 columns, 0.8 s).
+- **Join output limit:** the join kernels index their output with int32 offsets; a many-to-many join with more than 2,147,483,647 output rows (two 46,500-row tables on one key) wrote through wrapped offsets and segfaulted in `join_scatter`. It now raises `MemoryError` before any allocation. (The GPU join's prefix sum is int32 as well and is not guarded.)
+- New `MXFRAME_JOIN_TRACE=1` prints one line per join step (rows in and out, time of the key, encode, join and compose stages).
+
+### ✅ Correctness against DuckDB: counts, nulls, TPC-H queries (F-011)
+- **Counts** are exact `int64` on every path (grouped and global, fused and legacy, empty results); the fused path used to return float64 and the GPU count kernel saturates at 2^24 per group (the host now counts labels).
+- **Nulls follow SQL.** An aggregate that reads a column with nulls runs on Arrow's hash aggregate: a null group key is its own group, `sum/mean/min/max/count` skip null values, a comparison with null is not true, `AND`/`OR` are three-valued, a null `CASE` condition takes `ELSE`. Sorting with null keys puts them last (it raised `IndexError`).
+- A `Sort/Limit/Distinct/Tail` below a `Filter/Project/Aggregate` is applied first (a `Limit` under an `Aggregate` was ignored).
+- Join keys of different integer (or float) widths are cast to the wider type instead of raising; the join-result cache is keyed by device; the GPU gather no longer rounds `float64` columns to `float32` (it moves the 8-byte words) and defers nulls / other widths to Arrow.
+- A numeric literal next to a `float32` column is compared in `float32` on every path (`l_discount <= 0.07` dropped the rows equal to 0.07 on the unfused paths); `case_when` values are `float64` on the CPU; `min`/`max` of a group of only ±inf returns that infinity on the legacy AOT paths.
+- New `Expr.year_of_days()` for day-number date columns (`year()` reads YYYYMMDD integers).
+- **TPC-H benchmark queries** (`scripts/benchmark_tpch.py`) follow the TPC-H text and use day-number dates (Q4, Q5, Q7, Q8, Q10, Q15, Q20 returned no rows on real Parquet; Q12's date constants were 5 days off; Q19 lacked the ship mode/instruction filter, Q13 the comment filter, Q21 the EXISTS clause). Q5 and Q8 need the supplier table: `supplier=` keyword. The pandas and Polars versions changed with them.
+- `scripts/gen_tpch_parquet.py` casts DECIMAL to `float32` through `float64` (nearest); the direct cast truncated 0.05 to 0.049999997 and TPC-H Q6 came out 28% low. Regenerated data lives in `data/tpch_sf{1,10}_exact`.
+- **GPU memory pool:** `mxframe.aot_kernels` defaults `MODULAR_DEVICE_CONTEXT_MEMORY_MANAGER_SIZE_PERCENT` to 25 (the Mojo runtime otherwise reserves ~90% of free GPU memory at the first kernel-side buffer and our own allocations ran out at SF10). It is process-wide, respects a value you set, and is documented in the README. `hash_join` raises instead of returning an empty join if its prefix sum is 0 over non-zero match counts.
+- Join outputs never hold two columns of one name: a right-side column whose name is taken is renamed `<name>_right` (`_right2`, ...), for inner and left joins; the optimizer's filter pushdown uses the same names.
+- `tests/test_tpch_duckdb.py`: all 22 queries against DuckDB (official SQL over the same Parquet files) on `cpu`, `gpu` and `auto`, plus the pandas/Polars baselines.
+
+### 🩺 Kernel health: GPU crashes, hangs and per-call overhead (F-012)
+- **GPU grouped aggregates** (`group_sum/min/max/count`): grid-stride kernels sized to the GPU with a block-private table in dynamic shared memory sized to the real group count (up to 12,288 groups) or straight global atomics beyond it. `group_min`/`group_max` no longer fault above 8,192 groups. The big win was in `_atomic.mojo`: atomics were `seq_cst`, which made NVPTX put a fence around every atomic; they are `monotonic` now (CUDA `atomicAdd` semantics). 30M rows, resident input: 8 to 4,096 groups went from 44 to 610 ms down to about 1 to 3 ms.
+- Skewed high-cardinality group-bys no longer collapse: a float `atomicrmw fadd` lowers to a compare-exchange loop, so a hot group cost O(k^2) retries (10M rows / 100k groups with one key at 25%: 40 s). The global path for sum/count now uses native `red.global.add.f32` and combines equal groups within a warp first (`match.any`); Zipf(1.3) is within 1.5x of uniform.
+- GPU wrappers bind the CUDA context to the calling thread, so the Router's GPU worker thread can call them.
+- **GPU `group_encode_i32`** no longer hangs (spin-wait on plain loads could read a stale L1 line) and accepts every int32 key including `INT32_MIN`. The Q21 hash-table kernel had the same spin-wait and was fixed the same way.
+- **GPU sort** is stable, correct for any length (the old network skipped out-of-range partners, so any non-power-of-two length could come back out of order), ascending or descending, and about 2.6x faster (64-bit key|row words, shared-memory tiles). Multi-key composite sort keys no longer overflow int32 (also fixed on the CPU path).
+- **GPU hash join** runs its prefix sums and the right-side argsort on the device.
+- **Device-resident chaining:** the `AOTKernelsGPU` wrappers take `DeviceArray` inputs (`DeviceArray.borrow(ptr, dtype, n)` for residency-tracker or stager pointers) and `device_out=True` to leave results on the device. NumPy signatures are unchanged.
+- **CPU `sort_indices`** is a stable parallel LSD radix sort for int32/int64/float32/float64 keys (10M rows: about 60 ms, was O(n^2) and hung at 10,000 rows); `unique_mask`, `prefix_sum_count` and `filter_gather_*` are parallel. Kernel ABI version 3 (these gained the trailing `threads` argument).
+- Auto mode no longer treats the GPU sort as broken; the GPU calibration probe for `group_encode` is enabled and the min/max group-count limit is gone.
+
+### ⚡ Multi-core CPU kernels with float64 accumulation (F-002)
+- CPU AOT kernels (`libmxkernels_aot.so`) now run on all cores: grouped sum/min/max/mean/count, masked global sum/min/max/sum-product (SIMD), join build/probe/scatter, gather, and group-key composition. Grouped kernels use per-worker partial arrays merged in parallel; the join build is parallel by key range.
+- Sums and means accumulate in **float64** and return float64 (counts are exact). At TPC-H SF10 the float32 accumulators were off by up to 4.6% (counts by 42%); Q1-style grouped sums now match DuckDB summing the same float32 values as doubles to ~1e-14 relative.
+- Compound aggregate expressions (`price * (1 - discount)`) are evaluated in float64 on the CPU path.
+- New thread budget: env `MXFRAME_CPU_THREADS` (default: all cores), `mxframe.set_cpu_threads(n)` / `mxframe.get_cpu_threads()`, and a per-call `threads=` on every `AOTKernels` method. The library keeps no global state, so several Python threads can call kernels at once.
+- The exported symbols gained a trailing `threads` argument (ABI version 2); a stale `.so` is rejected with a clear error. `device="gpu"` is untouched.
+
+---
+
+## [0.5.0] — 2026-07-11
+
+Full narrative: [v0.5.0 release notes](RELEASE_NOTES_0.5.0.md).
+
+### ⚡ Native GPU UTF-8 predicates
+- Added AOT `utf8_startswith_mask_gpu`, `utf8_contains_mask_gpu`, `utf8_equal_mask_gpu`, and packed-literal `utf8_isin_mask_gpu` kernels over Arrow validity, offsets, and byte buffers.
+- Supports sliced arrays, nulls, empty strings/patterns, duplicate literals, and multibyte UTF-8 while matching Arrow filter semantics.
+- GPU expression dispatch covers string `==`, `!=`, `startswith`, literal `contains`, and non-null literal `isin`; unsupported/null-membership cases retain Arrow fallback.
+- Added permanent CPU/GPU Phase 6 tests and `pixi run bench-utf8` comparisons against PyArrow and Polars.
+- At 1M rows, packed `isin` and `contains` beat local Arrow and Polars; exact equality remains faster in Polars' optimized CPU path, so no universal string-performance claim is made.
+
+### ⚡ Weak-query GPU fusion
+- Q6 dispatches `range5_sum_product_f32_gpu` instead of materializing five masks and a product array.
+- Q4 uses `semi_range_group_count_i32_gpu` for fused qualifying-key marking, semi-join probing, and grouped counts.
+- Q13 uses `key_match_counts_i32_gpu` to emit one compact count per customer without materializing a left join.
+- Q21 uses `single_late_supplier_counts_i32_gpu` for exact `(order, supplier)` deduplication and deterministic compact supplier counts.
+- Representative 1M/10M GPU results versus Polars: Q4 **3.4/28.6 vs 20.2/105.4 ms**; Q13 **1.8/28.5 vs 26.5/442.0 ms**; Q21 **6.6/49.8 vs 30.2/99.1 ms**.
+
+### 📊 Bounded 22-query benchmark
+- Every query and engine now runs in its own process, releasing Arrow, Pandas, and CUDA allocations between measurements.
+- Added per-engine timeouts, five-query progress groups, merged CSV output, and optional synchronized `cudf.pandas` workers.
+- MX CPU beat Polars on **22/22 at 1M** and **18/22 at 10M**.
+- MX GPU beat Polars on **15/17 comparable paths at 1M** and **14/17 at 10M**.
+- Q8 GPU remains `N/A:JIT`; Q15/Q17/Q18/Q20 GPU workers exceeded the bounded timeout.
+- cuDF was unavailable locally, so RAPIDS remains `N/A` and no RAPIDS parity claim is made.
+
+### 🎬 Visualization and identity
+- Q4, Q6, Q12, Q13, Q14, and Q21 visualize their logical rewrite, dispatch boundary, and step-by-step Mojo kernel algorithm.
+- Added a cool-gray MX mark whose rising X stroke is shaped as orange lightning.
+- Verified responsive rendering and zero horizontal overflow in Playwright.
+
+### Remaining performance-parity work
+- Replace Q8's JIT path and fix Q15/Q17/Q18/Q20 GPU timeouts.
+- Improve Q1 and Q14 GPU pipelines at 10M.
+- Run synchronized 1M/10M benchmarks on a compatible RAPIDS/cuDF environment.
+
+### Remaining feature-parity work
+- Native GPU UTF-8 dictionary construction and string/date gather.
+- GPU windows and additional semi/anti/cross/as-of/range joins.
+- Broader regex/string, datetime, nested-type, reshape, SQL, streaming, and I/O pushdown support.
+- Cross-generation NVIDIA testing plus AMD and Apple Silicon validation.
+
+---
+
+## [0.4.0] — 2026-07-11
+
+### 🔗 String and composite joins
+- Join keys from both tables now use one shared Arrow dictionary before dispatch, so equal strings receive identical dense `int32` IDs regardless of encounter order.
+- Multi-column joins jointly densify shared per-column codes, avoiding mixed-radix overflow and supporting mixed string/integer composite keys.
+- SQL null semantics are preserved: null keys receive side-specific IDs and never match, including left joins.
+- Compact nonnegative integer joins retain their direct `int32` fast path; sparse and non-integer keys are safely densified.
+- Fixed GPU upload-cache aliasing by retaining each cached NumPy host owner, preventing recycled host pointers from returning stale device buffers during joined-table gather.
+- String encoding runs in Arrow on CPU; resulting IDs use the Mojo AOT CPU/GPU join kernels. Native GPU UTF-8 dictionary construction remains future work.
+
+---
+
+## [0.3.0] — 2026-07-10
+
+### 📚 Docs & Benchmarks
+- README restructured: 543 lines → 119 lines. Intro, quick start, and 4-row teaser only — all detail moved to dedicated docs.
+- New `docs/benchmarks.md`: full 22-query TPC-H tables (1 M + 10 M rows), Mojo kernel catalogue, current limitations, roadmap, and reproduce instructions.
+- New `docs/api.md`: LazyFrame method reference, expression syntax, SQL frontend, supported operations, project structure.
+- `docs/vision-and-architecture.md`: fixed garbled sections 2.5/2.6/3; success criteria table repaired; all milestones marked ✅.
+- `docs/archive/` added to `.gitignore` — old dev notes moved there, not published.
+- All stale `Mojo 0.26.2` / `MAX Graph shape-cached join` references updated to reflect v0.2.3 reality (Mojo 26.4, AOT ctypes everywhere).
+
+### 📊 Benchmark updates
+- Q5/Q7/Q8/Q10/Q13/Q19 re-measured with per-run join-cache clearing (fresh-join). Previous v0.2.0 numbers for those queries were warm join-cache hits.
+- Summary corrected: CPU 20/22 (Q13 loses fresh-join), GPU 12/22 (Q5/Q7 no longer win at 1 M without cache warmup).
+- Q8 GPU marked `—` (MAX Graph `year()` groupby JIT too slow to benchmark; Phase 6 target).
+
+### 🔨 Infrastructure
+- `modular >= 26.4` requirement added to `pyproject.toml` runtime deps and Dependencies table in README.
+- `docs/archive/` gitignored; previously committed archive files removed from history.
+
+---
+
+## [0.2.3] — 2026-07-09
+
+### ⚡ Performance
+- **Phase 4 — Device-resident GPU join pipeline** (`join_count_i32_gpu` + `join_scatter_i32_gpu`):
+  GPU inner joins now bypass MAX Graph JIT entirely. Cold-start join latency drops from ~300 ms (MAX session init + JIT) to ~70 ms. `AOTKernelsGPU.hash_join()` caches key arrays via `_cached_upload`; `AOTKernelsGPU.gather_table()` gathers all numeric columns with a single shared device-resident index upload.
+- **Phase 5 — GPU sort + top-k** (`sort_topk_i32_gpu`): `ORDER BY … LIMIT k` fuses sort and limit in a single AOT bitonic sort kernel, downloading only `k × 4` bytes instead of `N × 4`. `_apply_post_ops_custom` peek-ahead passes `topk=k` directly to the sort kernel.
+- **Phase 3 — Vectorised rank mapping**: `_encode_sort_key` replaces the per-unique-value Python `for` loop with a single vectorised NumPy scatter (`rank_of[sorted_order] = np.arange(n_unique)`). ~1.5× speedup for sort-key encoding at 5 M rows.
+
+### 🛠 Fixed
+- **GPU sort+limit crash**: `_apply_sort_custom` was calling `_gpu_gather_table` which referenced `gather_f32` as a MAX Graph custom op — that kernel was never registered in the mojopkg. Any `ORDER BY` on `device="gpu"` would crash. Now routes through `_aot_gpu.gather_table` (AOT ctypes, no MAX Graph).
+- `_apply_sort_custom` CPU path now uses `np.argpartition` (O(N) partial sort) when `topk < N`, instead of a full O(N log N) argsort.
+
+### 🔨 Infrastructure
+- **Mojo/MAX 26.4 migration**: all `kernels/*.mojo` and `kernels_aot/*.mojo` migrated from 26.2 API (`fn` → `def`, `from tensor import` → `from extensibility import`, `DeviceContextPtr` → `DeviceContext`, `enqueue_function_experimental` → `enqueue_function`).
+- **`kernels/_atomic.mojo`** (new): drop-in replacement for the missing `std.os.atomic.Atomic`. Implements `fetch_add`, `min`, `max`, `compare_exchange` via MLIR `llvm.atomicrmw` / `llvm.cmpxchg` instructions — portable across NVIDIA and AMD; works in both `mojo package` (JIT) and `mojo build --emit shared-lib` (AOT) contexts.
+- All compiled artifacts rebuilt with Mojo 1.0.0b2 (modular 26.4): `kernels.mojopkg`, `libmxkernels_aot.so`, `libmxkernels_aot_gpu.so`.
+
+---
+
+## [0.2.2] — 2026-07-08
+
+### 🔨 Infrastructure
+- First release targeting **Mojo/MAX 26.4** (`modular = "26.4.*"` in `pixi.toml`).
+- **`kernels/_atomic.mojo`** shim introduced to replace the removed `std.os.atomic` module.
+- All kernel sources migrated to 26.4 syntax; compiled artefacts rebuilt.
+- `pyproject.toml` `runtime` extra pinned to `modular>=26.4` to prevent 26.2 wheels from being installed by Colab/pip users.
+
+---
+
+## [0.2.1] — 2026-06-xx
+
+### 🛠 Fixed
+- TestPyPI re-upload to validate wheel packaging. No functional changes.
+
+---
+
+## [0.2.0] — 2026-05-xx
+
+### ✨ Added
+- **Phase 1 — GPU buffer cache + masked global min/max**:
+  - `masked_global_min_f32_gpu` / `masked_global_max_f32_gpu` Mojo AOT kernels.
+  - `AOTKernelsGPU` device-buffer cache (`_cached_upload`, eviction, `clear_buf_cache`) — eliminates redundant PCIe uploads on repeated hot queries.
+  - `group_mean_f32` GPU kernel (sum + count on GPU, divide on CPU).
+- **Phase 2 — Masked global aggregation for any expression**:
+  - `_compute_masked_global_agg` extended to handle any inner expression via `_eval_expr_arrow` (e.g. `a*(1-b)`, `case_when`, `startswith`, `isin`) — SUM reduction stays GPU-resident.
+  - Join-only plans emit `"join_mojo_shortcut"` provenance (not `"pyarrow_shortcut"`) — Q19 restructured to single-pass join+filter+global_agg.
+- **Phase 3 — GPU integer key encoding**:
+  - `group_encode_i32_gpu` open-addressing hash-table kernel for integer group keys; replaces PyArrow `dictionary_encode` for integer-keyed group-by on GPU.
+- **Audit tool** (`scripts/audit_gpu_paths.py`): classifies each of 22 TPC-H queries as `GPU-CLEAN` / `MOJO-CPU` / `FALLBACK` via `last_compile_provenance["path"]`.
+
+### ⚡ Performance
+- **22/22 TPC-H queries GPU-CLEAN** (100% Mojo-backed on GPU, zero PyArrow fallbacks in the reduction and join hot paths).
+
+---
+
+## [0.1.2] — 2026-04-22
+
+### 🛠 Fixed
+- `pip install "mxframe[runtime]"` now gives a working install. The `[runtime]` extra pulls `modular>=25.5`, which provides the `max.engine` / `max.driver` modules that `mxframe.compiler` imports at module load. (Pixi users get `modular` from the Modular conda channel and should install `mxframe` without the extra.)
+- `sqlglot` import in `mxframe.sql_frontend` is now lazy: `import mxframe` no longer requires the `[sql]` extra. Calling `mxframe.sql(…)` without `sqlglot` installed raises a clear `ImportError` with the install hint.
+- `__init__.py` `__version__` now matches `pyproject.toml`.
+
+---
+
+## [0.1.1] — 2026-04-22
+
+### 🛠 Fixed
+- Packaging: `readme` now points to `README.md` so PyPI shows the full project page with benchmarks and quickstart (was accidentally showing the internal vision/architecture doc).
+- CI: `AOTKernelsGPU` init no longer hard-fails on CPU-only runners (`CUDA driver unavailable` is caught, `self._aot_gpu` falls back to `None`).
+- CPU-only import and compute path validated in a clean venv without `modular`.
+
+---
+
+## [0.1.0] — 2026-04-21
+
+Initial public release.

mxframe-0.6.0/MANIFEST.in

@@ -0,0 +1,13 @@
+include pyproject.toml
+include setup.py
+include README.md
+include CHANGELOG.md
+include docs/vision-and-architecture.md
+# every module of the package (the repo root is the `mxframe` package); a hand-kept list went stale
+include *.py
+include kernels.mojoc
+recursive-include kernels_aot *.py *.so
+prune llm-ctx
+prune scripts
+prune kernels
+global-exclude __pycache__ *.py[cod]

mxframe-0.6.0/PKG-INFO

@@ -0,0 +1,349 @@
+Metadata-Version: 2.4
+Name: mxframe
+Version: 0.6.0
+Summary: GPU-accelerated DataFrame engine with Python ergonomics and Mojo AOT kernels.
+Author-email: Abhishek Sreesaila <abhisheksreesaila@users.noreply.github.com>
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/abhisheksreesaila/mxframe
+Project-URL: Repository, https://github.com/abhisheksreesaila/mxframe
+Keywords: dataframe,gpu,mojo,arrow,analytics,tpch
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Database
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: pyarrow>=14.0
+Requires-Dist: numpy>=1.24
+Requires-Dist: pandas>=2.0
+Provides-Extra: runtime
+Requires-Dist: modular>=26.6; extra == "runtime"
+Provides-Extra: polars
+Requires-Dist: polars>=0.20; extra == "polars"
+Provides-Extra: sql
+Requires-Dist: sqlglot>=25.0; extra == "sql"
+Provides-Extra: dev
+Requires-Dist: build>=1.2; extra == "dev"
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: polars>=0.20; extra == "dev"
+Requires-Dist: sqlglot>=25.0; extra == "dev"
+Requires-Dist: duckdb>=0.9; extra == "dev"
+Requires-Dist: twine>=6.0; extra == "dev"
+
+<img src="visualize/mx-spark.svg" alt="MXFrame MX lightning mark" width="72">
+
+# 🚀 MXFrame
+
+> **GPU-accelerated DataFrames — Python ergonomics, Mojo AOT kernels.**
+
+MXFrame is a DataFrame query engine that pairs a Polars-style Python API with pre-compiled Mojo AOT kernels. The current GPU build is validated on **NVIDIA**; AMD and Apple Silicon validation remain on the roadmap.
+
+[![TPC-H](https://img.shields.io/badge/TPC--H-22%2F22%20queries-brightgreen)](docs/benchmarks.md)
+[![Python](https://img.shields.io/badge/python-3.12%2B-blue)](pyproject.toml)
+[![Platform](https://img.shields.io/badge/platform-linux--x86__64-lightgrey)](pyproject.toml)
+[![License](https://img.shields.io/badge/license-Apache%202.0-blue)](LICENSE)
+
+---
+
+## ✨ Why MXFrame?
+
+| | pandas | Polars | cuDF (RAPIDS) | **MXFrame** |
+|---|---|---|---|---|
+| GPU support | ❌ | ❌ | ✅ NVIDIA | ✅ **NVIDIA validated** |
+| Compiled kernels | ❌ | ✅ Rust | ✅ CUDA | ✅ **Mojo AOT** |
+| Install complexity | pip | pip | CUDA + RAPIDS stack | **pip or pixi** |
+| TPC-H coverage | reference | ✅ | ✅ | ✅ **22/22** |
+| Portable kernel design | ❌ | CPU | CUDA | ✅ NVIDIA today; AMD/Apple planned |
+
+MXFrame uses a portable Mojo kernel architecture rather than CUDA-specific source. Kernels compile once to shared libraries and dispatch directly at runtime.
+
+### What changed in v0.4, v0.5 and v0.6
+
+- **v0.6.0 — native Mojo CPU executor:** each query stage (scan, filter, joins, group-by, DISTINCT, top-N, COUNT(DISTINCT)) runs in one ahead-of-time compiled Mojo call over Arrow memory. Engine-only TPC-H SF10 totals 3.15 s, against DuckDB 5.08 s and Polars 14.40 s (without Q21, DuckDB is lower: 2.27 s against 2.41 s). See [How the CPU engine got fast](#-how-the-cpu-engine-got-fast-v060-2026-10-14) and the [v0.6.0 release notes](RELEASE_NOTES_0.6.0.md).
+
+- **v0.4.0 — correct string/composite joins:** shared dictionaries assign identical dense IDs across both join sides; mixed composite keys and SQL null semantics are preserved.
+- **v0.5.0 — native GPU UTF-8 predicates:** `startswith`, literal `contains`, string equality/inequality, and packed non-null literal `isin` operate directly on Arrow offsets and byte buffers.
+- **v0.5.0 — weak-query fusion:** dedicated compact GPU paths remove expanded intermediates from Q4, Q6, Q13, and Q21.
+- **v0.5.0 — bounded benchmarking and visualization:** every query/engine runs in an isolated process; Q4, Q6, Q12, Q13, Q14, and Q21 have step-by-step kernel visualizations.
+
+See [v0.5.0 release notes](RELEASE_NOTES_0.5.0.md) for measured gains, limitations, and the parity roadmap.
+
+---
+
+## 🧠 How `device="auto"` works
+
+Give MXFrame a CPU and a GPU and `compute(device="auto")` decides where each query runs
+from costs it measures on your machine. First scans run on the CPU while the table's
+columns copy to the GPU in the background; after that, queries split their rows so the
+GPU and every CPU core work at the same time. On TPC-H SF10 Q1 the split finishes the
+aggregation in 25 ms, against 57 ms on the CPU alone and 39 ms on the GPU alone.
+
+On the final user-felt TPC-H run, `auto` is within 10% of the better of `cpu` and `gpu` on
+22/22 queries at SF10, and on 21/22 at SF1 (median of 4 runs; single runs 17–21). Each query ran in a fresh process from Parquet on disk with
+no warm-up, and every result matched DuckDB. On the five SF10 queries where the GPU wins from a
+cold start (Q5, Q7, Q9, Q18, Q21), `auto` stays within 5% of it. At SF1, where CUDA start-up
+would cost more than the query, it stays on the CPU.
+
+➡️ **[Read how it works](docs/auto-mode.md)** · [visual version](https://htmlpreview.github.io/?https://github.com/abhisheksreesaila/mxframe/blob/main/docs/auto-mode.html)
+
+---
+
+## ⚡ Quick Start
+
+```sh
+# 1. Install pixi (Modular's package manager)
+curl -fsSL https://pixi.sh/install.sh | bash
+
+# 2. Clone and set up
+git clone https://github.com/abhisheksreesaila/mxframe
+cd mxframe
+pixi install
+
+# 3. Verify GPU is working
+pixi run python3 scripts/_check_gpu.py
+```
+
+```python
+import pyarrow as pa
+from mxframe import LazyFrame, Scan, col, lit
+
+data = pa.table({
+    "dept":   pa.array(["eng", "eng", "mkt", "mkt", "eng"]),
+    "salary": pa.array([120.0, 95.0, 80.0, 110.0, 130.0], pa.float32()),
+    "age":    pa.array([32, 28, 35, 29, 40], pa.int32()),
+})
+
+result = (
+    LazyFrame(Scan(data))
+    .filter(col("age") > lit(28))
+    .groupby("dept")
+    .agg(
+        col("salary").sum().alias("total_salary"),
+        col("age").count().alias("headcount"),
+    )
+    .sort(col("total_salary"), descending=True)
+    .compute(device="gpu")   # or "cpu"
+)
+print(result.to_pandas())
+```
+
+```
+  dept  total_salary  headcount
+0  eng         345.0          3
+1  mkt         110.0          1
+```
+
+**GPU memory.** Importing `mxframe.aot_kernels` sets `MODULAR_DEVICE_CONTEXT_MEMORY_MANAGER_SIZE_PERCENT=25` unless you have set it yourself. Left at its default, the Mojo runtime reserves about 90% of the free GPU memory the first time a kernel allocates a scratch buffer, which starves mxframe's own allocations at TPC-H SF10 and every other process on the GPU. The variable is process-wide: other MAX / Mojo users in the same Python process get the 25% pool too. Set it to a larger value before importing mxframe if a kernel needs more scratch than a quarter of the GPU.
+
+---
+
+## 📈 How the CPU engine got fast (v0.6.0, 2026-10-14)
+
+Over three days of work, the engine went from Python glue around kernels to Mojo running each
+query stage in one native call. Python plans the query. Mojo runs the filters, joins, group-bys,
+DISTINCT and gathers directly on Arrow memory.
+
+**How we measure (the headline): engine-only, first run.**
+- Loading data is staging. Each engine first reads the query's columns from Parquet into its own
+  in-memory form, outside the clock. The official TPC-H power test, the H2O db-benchmark and
+  ClickBench do the same.
+- The clock covers the query and its materialized result only.
+- Each query runs in a fresh process, and only the **first** run counts: no warm-up and no
+  reused results.
+- Every result is checked against DuckDB.
+
+![MXFrame total per milestone against Polars and DuckDB](docs/img/progress.svg)
+
+| total of 22 queries, engine-only | MXFrame (CPU) | Polars | DuckDB |
+|---|---:|---:|---:|
+| SF10 | **3.15 s** | 14.40 s | 5.08 s |
+| SF1 | **0.47 s** | 1.31 s | 0.63 s |
+| SF10, rows shuffled | **4.97 s** | 15.06 s | 5.55 s |
+| SF1, rows shuffled | **0.61 s** | 1.41 s | 0.66 s |
+
+**Where it stands, honestly:**
+
+- **Much of MXFrame's lead over DuckDB comes from Q21.**
+  - Q21 at SF10 takes 0.74 s for MXFrame against 2.81 s for DuckDB on stored rows, and 1.34 s
+    against 2.77 s on shuffled rows.
+  - **Leaving Q21 out, DuckDB's SF10 total is lower:**
+    - stored rows: 2.27 s against MXFrame's 2.41 s;
+    - shuffled rows: 2.79 s against 3.63 s.
+  - At SF1 the two are even: 0.39 s each on stored rows, and DuckDB 0.42 s against 0.46 s
+    shuffled.
+- **Against DuckDB, per query:**
+  - DuckDB is faster on 12 of the 22 TPC-H queries at SF10.
+  - The largest gaps are joins: Q3, Q5, Q8, Q11, Q12, Q17 and Q22 run at 0.44–0.69x DuckDB's speed.
+  - MXFrame is faster on the other 10. It leads most on Q1, Q13 and Q21 (2.1–3.8x).
+- **Against Polars:**
+  - TPC-H: MXFrame is faster on 20 of the 21 verified queries at SF10 (Polars is 3% faster on Q16;
+    Polars' Q1 result is wrong) and on 11 of 22 at SF1.
+  - Random queries: faster at SF10 (43 of 50, 3.8x geometric mean) and on shuffled SF1 (97 of
+    99, 3.1x).
+- **MXFrame's results equal DuckDB's** on every query: the 22 TPC-H queries (stock and shuffled)
+  and the 250 random queries.
+
+**Beyond the 22 queries.** Every number above also has to hold on queries and data the engine was
+not tuned for.
+- A seeded generator writes random queries over the TPC-H tables. They cover filters, group-bys,
+  joins, top-N, DISTINCT, count-distinct and HAVING.
+- Each query runs in all three engines, on the stored row order and on shuffled rows, and every
+  result is checked against DuckDB.
+
+Reports:
+- random queries: [SF1](docs/bench/random-sf1-m2s.md) ·
+  [SF1 shuffled](docs/bench/random-sf1-shuffled-m2s.md) · [SF10](docs/bench/random-sf10-m2s.md);
+- TPC-H: [stored rows](docs/bench/m2s-engine.md) ·
+  [shuffled rows](docs/bench/m2s-engine-shuffled.md).
+
+MXFrame's speed against DuckDB on random queries, for all three runs. Each figure is the geometric
+mean of DuckDB time / MXFrame time; above 1x means MXFrame is faster. The last column shows the
+same SF10 figure before and after the F-046 work (F-044 run → this run):
+
+| query shape | SF1 | SF1, rows shuffled | SF10 | SF10, before → after F-046 |
+|---|---:|---:|---:|---:|
+| filter + group-by | 1.12x | 1.72x | 1.84x | 1.86 → 1.84x |
+| global aggregate | 0.68x | 1.16x | 1.26x | 1.33 → 1.26x |
+| joins | 0.88x | 1.13x | 0.96x | 0.64 → 0.96x |
+| top-N rows | 0.58x | 0.98x | 0.90x | 0.41 → 0.90x |
+| SELECT DISTINCT | 0.57x | 0.99x | 0.62x | 0.43 → 0.62x |
+| COUNT(DISTINCT) | 0.84x | 1.11x | 0.89x | 0.31 → 0.89x |
+| **all** | **0.83x** | **1.21x** | **1.09x** | **0.75 → 1.09x** |
+
+- MXFrame is now ahead overall at SF10 and on shuffled SF1.
+- It leads on filter + group-by in every run. On shuffled SF1 it leads on every shape except
+  SELECT DISTINCT and top-N, which are about even.
+- At SF1 and SF10 it trails on SELECT DISTINCT, COUNT(DISTINCT), top-N and joins, and at SF1 on
+  global aggregates.
+- SF1 queries take a few milliseconds each, so the SF1 columns move with machine load.
+
+Each optimization is explained in **[docs/optimizations.md](docs/optimizations.md)** with a
+diagram, its before/after numbers, and the Mojo feature it relies on (or a plain statement when it
+is ordinary engineering). Caveats:
+
+- **Numbers come from `docs/bench/m2s-engine*.md` and `random-*-m2s.md`, all on one build.**
+  - Shuffled TPC-H and the three random suites ran 01:21–01:50 on 2026-10-14. They started after
+    the load average had stayed under 2 for 5 minutes.
+  - The first stored-rows TPC-H pass overlapped a scheduled snapshot cleanup, and every engine
+    came out slower. It was discarded.
+  - The stored-rows run shown here was rerun at 02:05, after 5 minutes under load 1. The only
+    processes sampled during it besides the benchmark were a browser at about a quarter of one
+    core and, in the last sample, one short process at 98% of one core.
+- **DuckDB is timed without its pyarrow import.**
+  - DuckDB imports pyarrow inside its first `.arrow()` call (about 70 ms per process).
+  - That import is now staging, like MXFrame's own import.
+  - Earlier versions of this page counted it, which overstated MXFrame's lead over DuckDB.
+- **MXFrame warms its kernels during staging.** `import mxframe` warms the kernel libraries in the
+  background, and that finishes during the untimed load.
+- **Polars' Q1 at SF10** takes 2.7 s and differs from DuckDB's result, so it is not counted as a
+  win. Leaving Q1 out, the SF10 totals are MXFrame 3.09 s, Polars 11.68 s and DuckDB 4.94 s.
+- **Import time and the Parquet read are reported separately.**
+  - The earlier scoreboards below include them, and predate this work.
+  - MXFrame still reads Parquet through PyArrow (F-021).
+
+## 📊 Earlier scoreboards (2026-10-12, before the Mojo executor)
+
+**User-felt (secondary).** These include staging: each engine runs each query in a fresh
+process. The data starts as Parquet on disk, and the clock covers read + query + result with no warm-up. Every result is checked against DuckDB.
+Ryzen 9 5900X, RTX 3090.
+
+| sum of 22 queries, ms | pandas | Polars | DuckDB | MXFrame `auto` |
+|---|---:|---:|---:|---:|
+| SF1 | 6,394 | **2,014** | 3,336 | 4,479 |
+| SF10 | 82,081 | 17,075 | **10,394** | 28,892 |
+
+MXFrame `auto` beats pandas on 14/22 (SF1) and 19/22 (SF10). It beats DuckDB on 7/22 at SF1.
+It is behind Polars on most queries and behind DuckDB on every SF10 query. The main causes are
+the PyArrow Parquet decoder (F-021) and join-heavy queries (F-007). Full tables and the honest
+losses: [docs/benchmarks.md](docs/benchmarks.md#final-user-felt-scoreboard-2026-10-12-f-009).
+
+**Engine only.** A second run takes file reading out of the clock: each engine first loads the
+query's columns into its own in-memory form, untimed, and the clock covers query + result.
+
+| sum of queries, ms | pandas | Polars | DuckDB | MXFrame `auto` |
+|---|---:|---:|---:|---:|
+| SF1 (22) | 4,499 | **1,391** | 2,299 | 2,777 |
+| SF10 (21) | 66,199 | 12,146 | **6,716** | 22,511 |
+
+Roughly half of MXFrame's gap to Polars is the engine and half is the Parquet read. Leaving out
+Q12 and Q19, `auto` against Polars is 2.1x with the read and 1.8x without it at SF1, and 1.9x
+against 1.6x at SF10. Q12 and Q19 run slower on in-memory Arrow tables than through MXFrame's own
+Parquet scan; the cause is not yet profiled (F-025).
+[Details](docs/benchmarks.md#engine-only-scoreboard-2026-10-12-f-024).
+
+The section below is the older warm-run benchmark on synthetic data: 1 warm-up, then the median
+of 3 runs. Its numbers are not comparable with the table above.
+
+> ⚠️ **Not a fair comparison; kept only as history.** MXFrame's result caches (joins, group encodings, input conversions) were cleared between queries but not between the warm-up and the timed runs, so the timed runs reused work done in the warm-up (hence Q9 in 0.6 ms); Polars and pandas recomputed every run. The data came from our own synthetic generator, with some tables smaller than the label. Use the user-felt and engine-only scoreboards above.
+
+### Historical: 📊 Performance — bounded v0.5.0 benchmark
+
+RTX 3090, warm median of three runs. Every query and engine ran in its own process so Arrow, Pandas, and CUDA allocations were released between measurements.
+
+- **1M:** MX CPU beat Polars on **22/22**; MX GPU beat Polars on **15/17 comparable** paths.
+- **10M:** MX CPU beat Polars on **18/22**; MX GPU beat Polars on **14/17 comparable** paths.
+- Q8 GPU remains `N/A:JIT`; Q15, Q17, Q18, and Q20 GPU workers exceeded the bounded timeout.
+- cuDF was unavailable on this machine, so RAPIDS remains `N/A` and no RAPIDS parity claim is made.
+
+### 1M representative queries
+
+| Query | MX CPU | MX GPU | Polars | Best MX speedup |
+|---|---:|---:|---:|---:|
+| Q4 · Order priority | 12.8 ms | **3.4 ms** | 20.2 ms | **5.9× GPU** |
+| Q6 · Discounted revenue | **6.7 ms** | 6.8 ms | 10.8 ms | **1.6× CPU** |
+| Q12 · Shipping modes | **0.6 ms** | 6.1 ms | 26.6 ms | **44.3× CPU** |
+| Q13 · Customer distribution | 20.4 ms | **1.8 ms** | 26.5 ms | **14.7× GPU** |
+| Q21 · Waiting suppliers | 27.1 ms | **6.6 ms** | 30.2 ms | **4.6× GPU** |
+
+### 10M representative queries
+
+| Query | MX CPU | MX GPU | Polars | Best MX speedup |
+|---|---:|---:|---:|---:|
+| Q4 · Order priority | 265.1 ms | **28.6 ms** | 105.4 ms | **3.7× GPU** |
+| Q9 · Product profit | **2.0 ms** | 16.5 ms | 109.7 ms | **54.9× CPU** |
+| Q12 · Shipping modes | **1.5 ms** | 8.2 ms | 128.0 ms | **85.3× CPU** |
+| Q13 · Customer distribution | 368.8 ms | **28.5 ms** | 442.0 ms | **15.5× GPU** |
+| Q21 · Waiting suppliers | 1068.7 ms | **49.8 ms** | 99.1 ms | **2.0× GPU** |
+
+Full matrices: [1M CSV](scripts/bench_results_1M.csv) · [10M CSV](scripts/bench_results_10M.csv) · [methodology and kernel catalogue](docs/benchmarks.md)
+
+## What remains for parity
+
+**Performance parity:** replace Q8's JIT path; fix Q15/Q17/Q18/Q20 GPU timeouts; improve Q1 and Q14 GPU execution; run synchronized 1M/10M measurements on a compatible RAPIDS/cuDF environment.
+
+**Feature parity:** native GPU UTF-8 dictionary construction and string/date gather; GPU windows; as-of/range joins (semi and anti joins exist since 2026-10-13: `how="semi"` / `how="anti"`); broader string/regex, datetime, nested-type, reshape, SQL, streaming, and I/O pushdown coverage.
+
+---
+
+## 📚 Docs
+
+| | |
+|---|---|
+| [**v0.5.0 Release Notes**](RELEASE_NOTES_0.5.0.md) | v0.4/v0.5 gains, benchmark evidence, known gaps, and parity roadmap |
+| [**API Reference**](docs/api.md) | LazyFrame, expressions, SQL frontend, supported operations, running tests |
+| [**Benchmarks**](docs/benchmarks.md) | Full TPC-H tables, kernel catalogue, limitations, roadmap, reproduce instructions |
+| [**Architecture**](docs/vision-and-architecture.md) | Design philosophy, internal layers, how MAX Graph fits in |
+| [**Kernel Visualizer**](visualize/mxframe_pipeline.html) | Interactive plans, dispatch boundaries, GPU threads, and output masks |
+| [**Contributing**](CONTRIBUTING.md) | Dev setup, writing Mojo kernels, adding queries |
+
+---
+
+## 📦 Dependencies
+
+| Package | Required | Purpose |
+|---|---|---|
+| `pyarrow >= 14` | ✅ | Column storage, zero-copy NumPy bridge |
+| `numpy >= 1.24` | ✅ | Vectorized pre/post processing |
+| `pandas >= 2.0` | ✅ | Reference implementations |
+| `modular >= 26.4` | GPU only | MAX Engine runtime, Mojo GPU dispatch |
+| `polars >= 0.20` | optional | Benchmark comparison |
+| `sqlglot >= 25` | optional | SQL frontend parsing |
+
+---
+
+## 📄 License
+
+Apache 2.0 — see [LICENSE](LICENSE).