mlx_commander 0.2.0__tar.gz

mlx_commander-0.2.0/.gitignore

@@ -0,0 +1,58 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# Virtual environments
+.venv/
+venv/
+ENV/
+env/
+
+# macOS system files
+.DS_Store
+.AppleDouble
+.LSOverride
+._*
+
+# Temporary test/scratch files & local datasets
+*.scpt
+func-calling-singleturn.json
+mlx_dataset/
+mlx_out*/
+
+# Unit test / coverage
+.coverage
+.coverage.*
+.pytest_cache/
+htmlcov/
+
+# IDE & Editor configs
+.idea/
+.vscode/
+*.swp
+*.swo
+*~

mlx_commander-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 tomkubik
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

mlx_commander-0.2.0/PKG-INFO

@@ -0,0 +1,370 @@
+Metadata-Version: 2.4
+Name: mlx_commander
+Version: 0.2.0
+Summary: Convert Hugging Face datasets into Apple MLX formats, configure parameters, and run training and fine-tuning via interactive TUI and CLI
+Project-URL: Homepage, https://github.com/tomkubik/mlx-commander
+Project-URL: Repository, https://github.com/tomkubik/mlx-commander.git
+Author: Tom Kubik
+License-Expression: MIT
+License-File: LICENSE
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Provides-Extra: all
+Requires-Dist: datasets>=2.14.0; extra == 'all'
+Requires-Dist: duckdb>=0.9.0; extra == 'all'
+Requires-Dist: mcp>=1.0.0; extra == 'all'
+Requires-Dist: pyarrow>=12.0.0; extra == 'all'
+Requires-Dist: pylance>=0.10.0; extra == 'all'
+Provides-Extra: arrow
+Requires-Dist: pyarrow>=12.0.0; extra == 'arrow'
+Provides-Extra: duckdb
+Requires-Dist: duckdb>=0.9.0; extra == 'duckdb'
+Provides-Extra: lance
+Requires-Dist: pylance>=0.10.0; extra == 'lance'
+Provides-Extra: mcp
+Requires-Dist: mcp>=1.0.0; extra == 'mcp'
+Provides-Extra: parquet
+Requires-Dist: pyarrow>=12.0.0; extra == 'parquet'
+Description-Content-Type: text/markdown
+
+# MLX Commander 🚀
+
+A fast, persistent dual-panel TUI (Norton Commander style) & CLI converter for preparing Hugging Face datasets into Apple Silicon MLX fine-tuning formats (`mlx-lm`).
+
+Built entirely with Python's standard library `curses` with zero mandatory dependencies and zero pre-compiled binaries.
+
+![Orthodox TUI which makes MLX defaults explicit](docs/images/screenshot1.png)
+
+*Orthodox TUI which makes MLX defaults explicit*
+
+---
+
+## 🌟 Key Features
+
+- **Persistent Multi-Panel TUI (Norton Commander style)**: Full keyboard navigation (`Tab` to switch panels, `↑`/`↓` to navigate, `Enter` to edit/open dropdowns, `F2` for Finder, `F5` to convert).
+- **AI Agent Skill & TUI Pre-Population**: Coding agents (Antigravity, Claude, Cursor) can inspect dataset schemas, pre-populate format, column mappings, and splits, and launch the TUI for split-second visual confirmation.
+- **macOS Terminal.app Spawner**: Seamless handoff from non-interactive agent environments to an interactive TUI window via AppleScript.
+- **Model Context Protocol (MCP) Server**: Native stdio MCP server exposing dataset inspection, TUI launching, and headless conversions to Claude Desktop and Cursor.
+- **Machine-Readable Manifest (`mlx_manifest.json`)**: Emits structured output with file paths, row counts, and copy-paste `mlx_lm.lora` commands for automated downstream pipelines.
+- **Multi-File Selection & Dataset Merging**: Select multiple dataset files at once (e.g. combining pre-split `train.jsonl` and `test.jsonl`). Verifies that all files have identical column schemas and merges them so you can randomize fresh Train / Validation / Test sets from scratch with a custom seed.
+- **Multi-Column Concatenation**: Tap `Space` to multi-select and order columns from the original dataset (e.g. `instruction + input`) to concatenate them seamlessly with `\n\n`.
+- **Live Reactive Preview**: Sample records format in real time as you change target formats or adjust column mappings.
+- **Instantaneous ESC Response**: Curses escape delay configured to 25ms (< 1 frame), making modal dismissal instantaneous while preserving arrow and function keys.
+- **Native macOS Cocoa Finder Picker**: Seamlessly select dataset folders or files via native macOS dialogs (compiled on the fly in `/tmp` with zero checked-in binaries).
+- **Supported MLX Formats**:
+  1. **Text Format**: `{"text": "..."}` — Causal LM / pre-training (single column, concatenated columns, or custom template).
+  2. **Chat / Messages Format**: `{"messages": [{"role": "system|user|assistant", "content": "..."}]}` — Supports message lists (standard role/content or ShareGPT `from`/`value`), or separate role columns.
+  3. **Prompt & Completion Format**: `{"prompt": "...", "completion": "..."}` — Q&A / instruction fine-tuning (`mlx_lm.lora --mask-prompt` compatible).
+  4. **DPO / Preference Format**: `{"prompt": "...", "chosen": "...", "rejected": "..."}` — Direct Preference Optimization.
+- **Flexible Data Loader**: Parquet (`.parquet`), Arrow (`.arrow`), Hugging Face `save_to_disk` directories, JSONL (`.jsonl`), JSON arrays (`.json`), CSV (`.csv`), TSV (`.tsv`), SQLite (`.sqlite`, `.db`), and WebDataset (`.tar`).
+- **Deterministic Splits & Random Seed**: Customizable Train / Validation / Test percentages with 100% reproducible shuffling via random seed.
+- **Ready-to-Use `mlx_lm.lora` Command**: Generates the exact training command ready to copy-paste.
+- **CLI Wizard & Headless Modes**: Run line-by-line via `--wizard` or fully automated via headless CLI flags.
+
+![Inspired by Norton Commander, with a classic color scheme available in TUI](docs/images/screenshot2.png)
+
+*Inspired by Norton Commander, with a classic color scheme available in TUI*
+
+---
+
+## 📦 Quick Start
+
+### 1. Launch MLX Commander (Default)
+
+Launch the interactive dashboard using any of these equivalent commands:
+
+```bash
+# Recommended for local repository execution (Primary):
+python3 mlx_commander.py
+
+# Or via secondary compatibility alias:
+python3 run.py
+
+# Or as a Python package module:
+python3 -m mlx_commander
+
+# Or zero-install directly via uvx from GitHub:
+uvx --from git+https://github.com/tomkubik/mlx_commander.git mlx_commander
+
+# Or via PyPI (once published):
+uvx mlx_commander
+```
+
+You can also pass arguments directly (e.g. pre-loading a dataset or multiple files):
+```bash
+python3 mlx_commander.py -d /path/to/my_hf_dataset
+# Or combine multiple files:
+python3 mlx_commander.py -d train.jsonl test.jsonl
+# (python3 run.py accepts all the same arguments)
+```
+
+### 2. Line-by-Line Wizard Mode
+
+For SSH sessions or non-curses environments:
+
+```bash
+python3 mlx_commander.py --wizard
+```
+
+### 3. Direct Command-Line Conversion (Automated / Headless)
+
+You can pass all options via flags for direct scripted conversions:
+
+```bash
+python3 mlx_commander.py \
+  --dataset /path/to/my_hf_dataset \
+  --format prompt_completion \
+  --prompt-col instruction \
+  --completion-col output \
+  --output ./mlx_data \
+  --train 80 \
+  --valid 10 \
+  --test 10 \
+  --seed 42
+```
+
+---
+
+## 🖥️ Command Line Reference
+
+```
+usage: mlx_commander [-h] [-v] [-d DATASET [DATASET ...]]
+                     [-f {text,chat,prompt_completion,dpo}]
+                     [-o OUTPUT] [--train TRAIN] [--valid VALID] [--test TEST]
+                     [--seed SEED] [--keep-splits] [--mapping MAPPING]
+                     [--text-col TEXT_COL] [--text-template TEXT_TEMPLATE]
+                     [--prompt-col PROMPT_COL] [--completion-col COMPLETION_COL]
+                     [--messages-col MESSAGES_COL] [--user-col USER_COL]
+                     [--assistant-col ASSISTANT_COL] [--system-col SYSTEM_COL]
+                     [--chosen-col CHOSEN_COL] [--rejected-col REJECTED_COL]
+                     [--commander] [--wizard]
+```
+
+### Key Flags:
+
+| Flag | Description |
+|---|---|
+| `-d`, `--dataset` | Path to HF dataset directory or file on disk (`.arrow`, `.parquet`, `.jsonl`, `.json`, `.csv`). |
+| `-f`, `--format` | Target MLX format (`text`, `chat`, `prompt_completion`, `dpo`). |
+| `-o`, `--output` | Destination directory where `train.jsonl`, `valid.jsonl`, and `test.jsonl` are saved. |
+| `--train` | Percentage of data for training (e.g. `80.0`). |
+| `--valid` | Percentage of data for validation (e.g. `10.0`). |
+| `--test` | Percentage of data for test (e.g. `10.0`, or `0` to omit). |
+| `--seed` | Integer random seed for reproducible random shuffling. |
+| `--keep-splits` | Preserve existing dataset splits without re-splitting. |
+| `--text-col` | Column to use as `text` for `text` format. |
+| `--text-template`| Template string with `{column_name}` variables for `text` format. |
+| `--prompt-col` | Column to map to `prompt`. |
+| `--completion-col` | Column to map to `completion`. |
+| `--messages-col` | Column containing conversation turns list for `chat` format. |
+| `--user-col` | Column for user turn in multi-column `chat` format. |
+| `--assistant-col`| Column for assistant turn in multi-column `chat` format. |
+| `--system-col` | Column for system prompt in multi-column `chat` format. |
+| `--chosen-col` | Column for preferred response in `dpo` format. |
+| `--rejected-col`| Column for dispreferred response in `dpo` format. |
+| `--manifest-file`| Custom file path where machine-readable `mlx_manifest.json` will be saved. |
+| `--prefill-state`| Pre-populate TUI state from a JSON string or path to JSON file. |
+| `--spawn-terminal`| Launch interactive TUI in an external macOS Terminal window. |
+| `--mcp` | Start Model Context Protocol (MCP) server over stdio. |
+| `--tui` | Force launch full-screen curses TUI. |
+| `--no-tui`, `--cli` | Run line-by-line CLI wizard instead of curses TUI. |
+
+---
+
+## 🤖 AI Agent Integration & MCP Support
+
+MLX Commander is designed for the modern AI agent era (**Antigravity**, **Claude Desktop**, **Cursor**, **Zed**, **Cline**). 
+
+Instead of an agent interrogating users with 10 sequential chat prompts or guessing schemas blindly, agents can **inspect schemas, formulate recommended settings, and launch MLX Commander with pre-populated values**. 
+
+The user gets a 3-second tactile review with live JSONL preview in the Norton Commander TUI, presses **[F5 Convert]**, and hands control back to the agent with a machine-readable manifest.
+
+### 🔄 The End-to-End Workflow
+
+```mermaid
+sequenceDiagram
+    autonumber
+    actor User as User (Human Developer)
+    participant Agent as AI Agent (Antigravity / Claude / Cursor)
+    participant TUI as MLX Commander TUI (macOS Terminal)
+    actor MLX as MLX Engine (mlx_lm.lora)
+
+    User->>Agent: "Convert dataset.parquet and fine-tune Llama 3 on it."
+    Agent->>Agent: Inspects schema, picks target format, maps columns & splits
+    Agent->>TUI: Launches TUI with pre-populated arguments (--tui --spawn-terminal)
+    Note over User,TUI: TUI pops up in macOS Terminal with fields pre-filled & live preview rendered.<br/>User reviews with arrow keys, presses [F5 Convert].
+    TUI->>TUI: Converts dataset, writes mlx_dataset/ & mlx_manifest.json
+    TUI-->>Agent: Closes window & returns exit code 0
+    Agent->>Agent: Reads mlx_manifest.json (split counts, paths, lora command)
+    Agent->>User: "Dataset converted (8,000 train / 1,000 valid / 1,000 test). Starting LoRA training..."
+    Agent->>MLX: Executes mlx_lm.lora training run
+```
+
+---
+
+### The Three Architectural Hand-offs
+
+#### 1. Hand-off 1: TUI State Pre-Population
+Agents can pre-populate every field of `CommanderState` via CLI flags or a JSON payload:
+- **Via CLI Flags**:
+  ```bash
+  mlx_commander --tui --spawn-terminal \
+    --dataset "./data.parquet" \
+    --format chat \
+    --messages-col conversations \
+    --train 85 --valid 15 \
+    --output "./mlx_dataset"
+  ```
+- **Via JSON (`--prefill-state`)**:
+  ```bash
+  mlx_commander --tui --spawn-terminal \
+    --prefill-state '{"dataset": "./data.parquet", "format": "prompt_completion", "prompt_col": "question", "completion_col": "answer", "train": 80, "valid": 20}'
+  ```
+When launched with pre-fill data, the TUI opens directly with focus on the mappings panel and renders the reactive JSONL preview immediately.
+
+#### 2. Hand-off 2: Machine-Readable Manifest Handshake (`mlx_manifest.json`)
+Every conversion automatically outputs `output_dir/mlx_manifest.json` (or to a custom path specified with `--manifest-file <path>`):
+
+```json
+{
+  "status": "success",
+  "format": "prompt_completion",
+  "source_path": "/path/to/source.parquet",
+  "output_dir": "/path/to/mlx_dataset",
+  "files": {
+    "train": {
+      "path": "/path/to/mlx_dataset/train.jsonl",
+      "filename": "train.jsonl",
+      "records": 8000,
+      "size_bytes": 1048576
+    },
+    "valid": {
+      "path": "/path/to/mlx_dataset/valid.jsonl",
+      "filename": "valid.jsonl",
+      "records": 1000,
+      "size_bytes": 131072
+    },
+    "test": {
+      "path": "/path/to/mlx_dataset/test.jsonl",
+      "filename": "test.jsonl",
+      "records": 1000,
+      "size_bytes": 131072
+    }
+  },
+  "splits": { "train": 8000, "valid": 1000, "test": 1000 },
+  "total_records": 10000,
+  "seed_used": 42,
+  "mlx_lora_command": "mlx_lm.lora --model mlx-community/Llama-3.2-3B-Instruct-4bit --train --data /path/to/mlx_dataset --mask-prompt --iters 600 --batch-size 4",
+  "manifest_path": "/path/to/mlx_dataset/mlx_manifest.json"
+}
+```
+
+- **Exit Code 0**: Conversion succeeded; manifest written.
+- **Exit Code 130**: User cancelled/closed the TUI without converting. If `--manifest-file` was set, writes `{"status": "cancelled"}`.
+
+#### 3. Hand-off 3: macOS Terminal.app Spawner
+When invoked by background agent runners (such as IDE extensions, subshells, or MCP daemons) without an active TTY:
+- Passing `--spawn-terminal` (or auto-detected on macOS in non-interactive sessions) executes the TUI in a dedicated macOS `Terminal.app` window via AppleScript.
+- The calling process blocks synchronously until the user converts or exits, then unblocks and returns the exit code and manifest.
+
+---
+
+### Model Context Protocol (MCP) Server
+
+MLX Commander includes a built-in MCP server that works over `stdio`.
+
+#### 1. Claude Desktop Setup (`claude_desktop_config.json`):
+```json
+{
+  "mcpServers": {
+    "mlx_commander": {
+      "command": "python3",
+      "args": ["-m", "mlx_commander", "--mcp"]
+    }
+  }
+}
+```
+*Or via zero-install `uvx`:*
+```json
+{
+  "mcpServers": {
+    "mlx_commander": {
+      "command": "uvx",
+      "args": ["--from", "git+https://github.com/tomkubik/mlx_commander.git", "--with", "mcp", "mlx_commander", "--mcp"]
+    }
+  }
+}
+```
+
+#### 2. Cursor Setup (`.cursor/mcp.json`):
+```json
+{
+  "mcpServers": {
+    "mlx_commander": {
+      "command": "python3",
+      "args": ["-m", "mlx_commander", "--mcp"]
+    }
+  }
+}
+```
+
+#### 3. Exposed MCP Tools:
+
+| MCP Tool | Description | Arguments |
+|---|---|---|
+| `inspect_dataset` | Inspects columns, total rows, split names, sample records, and auto-detects candidate mappings. | `dataset_path: str` |
+| `launch_conversion_tui` | Pre-populates and opens the TUI in macOS Terminal.app for user review. Returns conversion manifest. | `dataset_path`, `format`, `prompt_col`, `completion_col`, `messages_col`, `train_pct`, `valid_pct`, `test_pct`, `output_dir` |
+| `convert_dataset_headless` | Runs direct headless conversion in background without opening TUI. Returns conversion manifest. | Same arguments as `launch_conversion_tui` |
+
+#### 4. Exposed MCP Prompt:
+- **`prepare_dataset_for_mlx`**: Instructs the model on the optimal workflow to inspect the schema, formulate column mappings, and launch the conversion TUI.
+
+---
+
+### Agent Skill (`SKILL.md`)
+
+A standardized skill specification is included in the repository:
+- Skill path: [`.agents/skills/mlx-dataset-prep/SKILL.md`](.agents/skills/mlx-dataset-prep/SKILL.md)
+
+AI agents that support skill discovery (like **Antigravity**) automatically read this file when users ask to convert datasets or fine-tune models with Apple MLX.
+
+---
+
+## 🛠️ Step-by-Step Wizard Walkthrough
+
+1. **Step 1: Dataset Source**: Select your dataset folder or file on your drive. The tool validates the file, inspects column names, row counts, and existing splits.
+2. **Step 2: MLX Format**: Choose your target format (`Text`, `Chat / Messages`, `Prompt & Completion`, `DPO / Preference`).
+3. **Step 3: Column Mapping**: Match dataset columns to MLX fields or enter a formatting template. The tool automatically detects candidate columns.
+4. **Step 4: Splitting & Seed**: Configure Train / Valid / Test percentages. A random seed is automatically generated, and you can accept it or provide your own.
+5. **Step 5: Output & Preview**: Specify the destination folder, review a live preview of the formatted JSONL lines, and confirm to write the files.
+6. **Step 6: Ready to Fine-Tune**: Review written file sizes, row counts, and copy the generated `mlx_lm.lora` fine-tuning command.
+
+---
+
+## 🚀 Running Fine-Tuning with Apple MLX
+
+Once your dataset is converted, fine-tune an LLM on Apple Silicon with `mlx-lm`:
+
+```bash
+mlx_lm.lora \
+    --model mlx-community/Llama-3.2-3B-Instruct-4bit \
+    --train \
+    --data ./mlx_dataset \
+    --mask-prompt \
+    --iters 600 \
+    --batch-size 4
+```
+
+---
+
+## 🧪 Running Unit Tests
+
+Run the test suite with Python's built-in `unittest`:
+
+```bash
+.venv/bin/python -m unittest discover -s tests -p "test_*.py" -v
+```