soup-cli 0.1.0__tar.gz

soup_cli-0.1.0/.claude/settings.json

@@ -0,0 +1,27 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(git status*)",
+      "Bash(git diff*)",
+      "Bash(git log*)",
+      "Bash(git add*)",
+      "Bash(git commit*)",
+      "Bash(git push*)",
+      "Bash(git branch*)",
+      "Bash(git checkout*)",
+      "Bash(git stash*)",
+      "Bash(git remote*)",
+      "Bash(ruff check*)",
+      "Bash(ruff format*)",
+      "Bash(python -m ruff*)",
+      "Bash(pytest*)",
+      "Bash(python -m pytest*)",
+      "Bash(pip install*)",
+      "Bash(pip list*)",
+      "Bash(soup *)",
+      "Bash(python -m soup_cli*)",
+      "Bash(cd /c/Users/tokmo/peder/Soup && python -m pytest*)",
+      "Bash(cd /c/Users/tokmo/peder/Soup && ruff check*)"
+    ]
+  }
+}

soup_cli-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,31 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - run: pip install ruff
+      - run: ruff check soup_cli/ tests/
+
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.9", "3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: pip install -e ".[dev]"
+      - run: pytest tests/ -v --tb=short

soup_cli-0.1.0/.gitignore

@@ -0,0 +1,39 @@
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+dist/
+build/
+*.egg
+.eggs/
+
+# Virtual env
+.venv/
+venv/
+env/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Training outputs
+output/
+checkpoints/
+wandb/
+
+# Soup local data
+.soup/
+*.db
+
+# Secrets
+.env
+*.key
+
+# Internal plan (not for repo)
+plan.md

soup_cli-0.1.0/CLAUDE.md

@@ -0,0 +1,97 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Build & Development Commands
+
+```bash
+# Install in dev mode (editable + test deps)
+pip install -e ".[dev]"
+
+# Run all tests
+pytest tests/ -v --tb=short
+
+# Run a single test file
+pytest tests/test_config.py -v
+
+# Run a single test
+pytest tests/test_data.py::test_detect_alpaca_format -v
+
+# Lint
+ruff check soup_cli/ tests/
+
+# Lint with auto-fix
+ruff check --fix soup_cli/ tests/
+```
+
+## Architecture
+
+Soup is a CLI-first tool for LLM fine-tuning. The core flow:
+
+```
+soup train --config soup.yaml
+  → config/loader.py    (YAML → Pydantic SoupConfig)
+  → utils/gpu.py        (detect CUDA/MPS/CPU, estimate batch size)
+  → data/loader.py      (load file or HF dataset → normalize format)
+  → trainer/sft.py      (load model → quantize → apply LoRA → train)
+  → monitoring/callback.py + display.py  (live Rich dashboard)
+  → experiment/tracker.py  (log run + metrics to SQLite)
+  → save LoRA adapter to output/
+```
+
+**Config system:** `config/schema.py` is the single source of truth. All YAML fields are validated by Pydantic models (`SoupConfig` → `TrainingConfig` → `LoraConfig`, `DataConfig`). Templates (chat/code/medical) live as YAML strings in this file.
+
+**Data pipeline:** `data/loader.py` handles local files (JSONL/JSON/CSV/Parquet) and HuggingFace datasets. `data/formats.py` auto-detects and normalizes alpaca/sharegpt/chatml formats into a unified `{"messages": [...]}` structure. Also supports reverse conversion via `messages_to_format()`.
+
+**Trainer:** `trainer/sft.py` (`SFTTrainerWrapper`) and `trainer/dpo.py` (`DPOTrainerWrapper`) wrap HuggingFace's SFTTrainer/DPOTrainer with auto quantization (BitsAndBytes), LoRA (PEFT), and batch size estimation. Heavy ML imports are lazy (inside methods) so CLI stays fast for non-training commands.
+
+**Monitoring:** `monitoring/callback.py` is a HuggingFace `TrainerCallback` that streams metrics to `monitoring/display.py` (Rich Live panel at 2Hz) and optionally to the experiment tracker.
+
+**Experiment tracking:** `experiment/tracker.py` (`ExperimentTracker`) stores runs, per-step metrics, and eval results in SQLite at `~/.soup/experiments.db`. Automatically integrated into `soup train`. Commands: `soup runs`, `soup runs show`, `soup runs compare`, `soup runs delete`.
+
+**Data tools:** `commands/data.py` provides inspect, validate, convert (between alpaca/sharegpt/chatml), merge, dedup (MinHash via datasketch), and stats (extended statistics with plotext histograms).
+
+**Eval:** `commands/eval.py` wraps lm-evaluation-harness for model evaluation on standard benchmarks (mmlu, gsm8k, etc.) with results saved to the experiment tracker.
+
+## Code Conventions
+
+- **Line length:** 100 chars (ruff enforced)
+- **Linter:** ruff with E, F, I, N, W rules
+- **Config validation:** Always Pydantic v2 (BaseModel + Field)
+- **CLI framework:** Typer with `rich_markup_mode="rich"`
+- **Output:** Use `rich.console.Console` — never bare `print()`
+- **Lazy imports:** Heavy deps (torch, transformers, peft, datasketch, lm_eval, plotext) are imported inside functions, not at module level
+- **Variable naming:** Avoid single-letter names (ruff E741) — use `entry`, `part`, `length` instead of `l`
+- **Testing:** Rich Panel objects must be rendered via `Console(file=StringIO())` for string assertions, not `str(panel)`
+
+## Git Workflow
+
+- Repo: https://github.com/MakazhanAlpamys/Soup
+- Branch: `main`
+- CI: GitHub Actions runs ruff lint + pytest on Python 3.9/3.11/3.12
+- Always run `ruff check soup_cli/ tests/` before committing
+- Always run `pytest tests/ -v` before committing
+
+## Tests
+
+Test suite (~147 tests) lives in `tests/`:
+
+| File | Covers |
+|---|---|
+| `test_config.py` | Config loading, validation, defaults |
+| `test_data.py` | Format detection, conversion, validation |
+| `test_gpu.py` | GPU detection, batch size estimation |
+| `test_cli.py` | CLI commands basic validation |
+| `test_tracker.py` | SQLite experiment tracker |
+| `test_runs.py` | `soup runs` CLI commands |
+| `test_data_tools.py` | Data convert/merge/dedup/stats commands |
+| `test_eval.py` | Eval command |
+| `test_smoke_train.py` | Full pipeline smoke tests (GPU) |
+| `test_chat.py` | Chat command, `_detect_base_model` |
+| `test_push.py` | Push command, `_format_size`, `_generate_model_card` |
+| `test_init.py` | Init command, templates, overwrite logic |
+| `test_callback.py` | `SoupTrainerCallback` (mock-based) |
+| `test_display.py` | `TrainingDisplay` rendering |
+| `test_loader.py` | Data loading (JSONL/JSON/CSV, edge cases) |
+| `test_validator.py` | `validate_and_stats`, `extended_stats`, `_percentile` |
+| `test_formats.py` | Reverse conversion, round-trips, edge cases |

soup_cli-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Soup Team
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

soup_cli-0.1.0/PKG-INFO

@@ -0,0 +1,346 @@
+Metadata-Version: 2.4
+Name: soup-cli
+Version: 0.1.0
+Summary: Fine-tune LLMs in one command. No SSH, no config hell.
+Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
+Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
+Project-URL: Issues, https://github.com/MakazhanAlpamys/Soup/issues
+Author: Soup Team
+License-Expression: MIT
+License-File: LICENSE
+Keywords: fine-tuning,llm,lora,machine-learning,qlora
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Requires-Dist: accelerate>=0.25.0
+Requires-Dist: bitsandbytes>=0.41.0
+Requires-Dist: datasets>=2.14.0
+Requires-Dist: huggingface-hub>=0.16.0
+Requires-Dist: peft>=0.7.0
+Requires-Dist: plotext>=5.2.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: torch>=2.0.0
+Requires-Dist: transformers>=4.36.0
+Requires-Dist: trl>=0.7.0
+Requires-Dist: typer>=0.9.0
+Provides-Extra: data
+Requires-Dist: datasketch>=1.6.0; extra == 'data'
+Provides-Extra: dev
+Requires-Dist: pytest-cov>=4.0; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Requires-Dist: ruff>=0.1.0; extra == 'dev'
+Provides-Extra: eval
+Requires-Dist: lm-eval>=0.4.0; extra == 'eval'
+Provides-Extra: ui
+Requires-Dist: fastapi>=0.104.0; extra == 'ui'
+Requires-Dist: uvicorn>=0.24.0; extra == 'ui'
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <img src="soup.png" alt="Soup" width="200">
+</p>
+
+<h1 align="center">Soup</h1>
+
+<p align="center">
+  <strong>Fine-tune LLMs in one command. No SSH, no config hell.</strong>
+</p>
+
+<p align="center">
+  <a href="#quick-start">Quick Start</a> &middot;
+  <a href="#features">Features</a> &middot;
+  <a href="#data-tools">Data Tools</a> &middot;
+  <a href="#experiment-tracking">Tracking</a> &middot;
+  <a href="#model-evaluation">Eval</a> &middot;
+  <a href="#all-commands">Commands</a>
+</p>
+
+<p align="center">
+  <img src="https://img.shields.io/badge/python-3.9%2B-blue" alt="Python 3.9+">
+  <img src="https://img.shields.io/badge/license-MIT-green" alt="MIT License">
+  <img src="https://img.shields.io/badge/tests-147%20passed-brightgreen" alt="Tests">
+  <a href="https://github.com/MakazhanAlpamys/Soup/actions"><img src="https://github.com/MakazhanAlpamys/Soup/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
+</p>
+
+---
+
+Soup turns the pain of LLM fine-tuning into a simple workflow. One config, one command, done.
+
+```bash
+pip install git+https://github.com/MakazhanAlpamys/Soup.git
+soup init --template chat
+soup train
+```
+
+## Why Soup?
+
+Training LLMs is still painful. Even experienced teams spend 30-50% of their time fighting infrastructure instead of improving models. Soup fixes that.
+
+- **Zero SSH.** Never SSH into a broken GPU box again.
+- **One config.** A simple YAML file is all you need.
+- **Auto everything.** Batch size, GPU detection, quantization — handled.
+- **Works locally.** Train on your own GPU with QLoRA. No cloud required.
+
+## Quick Start
+
+### 1. Install
+
+```bash
+# From GitHub (recommended for now):
+pip install git+https://github.com/MakazhanAlpamys/Soup.git
+
+# From PyPI (coming soon):
+# pip install soup-cli
+```
+
+### 2. Create config
+
+```bash
+# Interactive wizard
+soup init
+
+# Or use a template
+soup init --template chat    # conversational fine-tune
+soup init --template code    # code generation
+soup init --template medical # domain expert
+```
+
+### 3. Train
+
+```bash
+soup train --config soup.yaml
+```
+
+That's it. Soup handles LoRA setup, quantization, batch size, monitoring, and checkpoints.
+
+### 4. Test your model
+
+```bash
+soup chat --model ./output
+```
+
+### 5. Push to HuggingFace
+
+```bash
+soup push --model ./output --repo your-username/my-model
+```
+
+## Config Example
+
+```yaml
+base: meta-llama/Llama-3.1-8B-Instruct
+task: sft
+
+data:
+  train: ./data/train.jsonl
+  format: alpaca
+  val_split: 0.1
+
+training:
+  epochs: 3
+  lr: 2e-5
+  batch_size: auto
+  lora:
+    r: 64
+    alpha: 16
+  quantization: 4bit
+
+output: ./output
+```
+
+## DPO Training
+
+Train with preference data using Direct Preference Optimization:
+
+```yaml
+base: meta-llama/Llama-3.1-8B-Instruct
+task: dpo
+
+data:
+  train: ./data/preferences.jsonl
+  format: dpo
+
+training:
+  epochs: 3
+  dpo_beta: 0.1
+  lora:
+    r: 64
+    alpha: 16
+  quantization: 4bit
+```
+
+## Chat with your model
+
+```bash
+# Chat with a LoRA adapter (auto-detects base model)
+soup chat --model ./output
+
+# Specify base model explicitly
+soup chat --model ./output --base meta-llama/Llama-3.1-8B-Instruct
+
+# Adjust generation
+soup chat --model ./output --temperature 0.3 --max-tokens 256
+```
+
+## Push to HuggingFace
+
+```bash
+# Upload model to HF Hub
+soup push --model ./output --repo your-username/my-model
+
+# Make it private
+soup push --model ./output --repo your-username/my-model --private
+```
+
+## Data Formats
+
+Soup supports these formats (auto-detected):
+
+**Alpaca:**
+```json
+{"instruction": "Explain gravity", "input": "", "output": "Gravity is..."}
+```
+
+**ShareGPT:**
+```json
+{"conversations": [{"from": "human", "value": "Hi"}, {"from": "gpt", "value": "Hello!"}]}
+```
+
+**ChatML:**
+```json
+{"messages": [{"role": "user", "content": "Hi"}, {"role": "assistant", "content": "Hello!"}]}
+```
+
+**DPO (preference pairs):**
+```json
+{"prompt": "Explain gravity", "chosen": "Gravity is a force...", "rejected": "I don't know"}
+```
+
+## Data Tools
+
+```bash
+# Inspect a dataset
+soup data inspect ./data/train.jsonl
+
+# Validate format
+soup data validate ./data/train.jsonl --format alpaca
+
+# Convert between formats
+soup data convert ./data/train.jsonl --to sharegpt --output converted.jsonl
+
+# Merge multiple datasets
+soup data merge data1.jsonl data2.jsonl --output merged.jsonl --shuffle
+
+# Remove near-duplicates (requires: pip install 'soup-cli[data]')
+soup data dedup ./data/train.jsonl --threshold 0.8
+
+# Extended statistics (length distribution, token counts, languages)
+soup data stats ./data/train.jsonl
+```
+
+## Experiment Tracking
+
+Every `soup train` run is automatically tracked in a local SQLite database (`~/.soup/experiments.db`).
+
+```bash
+# List all training runs
+soup runs
+
+# Show detailed info + loss curve for a run
+soup runs show run_20260223_143052_a1b2
+
+# Compare two runs side by side
+soup runs compare run_1 run_2
+
+# Delete a run
+soup runs delete run_1
+```
+
+## Model Evaluation
+
+Evaluate models on standard benchmarks using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness):
+
+```bash
+# Install eval dependencies
+pip install 'soup-cli[eval]'
+
+# Evaluate on benchmarks
+soup eval --model ./output --benchmarks mmlu,gsm8k,hellaswag
+
+# Link results to a training run
+soup eval --model ./output --benchmarks mmlu --run-id run_20260223_143052_a1b2
+```
+
+## Features
+
+| Feature | Status |
+|---|---|
+| LoRA / QLoRA fine-tuning | ✅ |
+| SFT (Supervised Fine-Tune) | ✅ |
+| DPO (Direct Preference Optimization) | ✅ |
+| Auto batch size | ✅ |
+| Auto GPU detection (CUDA/MPS/CPU) | ✅ |
+| Live terminal dashboard | ✅ |
+| Alpaca / ShareGPT / ChatML / DPO formats | ✅ |
+| HuggingFace datasets support | ✅ |
+| Interactive model chat | ✅ |
+| Push to HuggingFace Hub | ✅ |
+| Experiment tracking (SQLite) | ✅ |
+| Data tools (convert, merge, dedup, stats) | ✅ |
+| Model evaluation (lm-eval) | ✅ |
+| Web dashboard | 🔜 |
+| Cloud mode (BYOG) | 🔜 |
+
+## All Commands
+
+```
+soup init [--template chat|code|medical]      Create soup.yaml config
+soup train --config soup.yaml [--dry-run]     Start training
+soup chat --model ./output                    Interactive chat with model
+soup push --model ./output --repo user/name   Upload to HuggingFace Hub
+soup data inspect <path>                      View dataset stats
+soup data validate <path> --format alpaca     Check format
+soup data convert <path> --to chatml          Convert between formats
+soup data merge data1.jsonl data2.jsonl       Combine datasets
+soup data dedup <path> --threshold 0.8        Remove duplicates (MinHash)
+soup data stats <path>                        Extended statistics
+soup runs                                     List all training runs
+soup runs show <run_id>                       Detailed run info + loss graph
+soup runs compare <run_1> <run_2>             Compare two runs
+soup runs delete <run_id>                     Remove a run
+soup eval --model ./output --benchmarks mmlu  Evaluate on benchmarks
+soup version                                  Show version
+```
+
+## Requirements
+
+- Python 3.9+
+- GPU with CUDA (recommended) or Apple Silicon (MPS) or CPU (slow)
+- 8 GB+ VRAM for 7B models with QLoRA
+
+## Development
+
+```bash
+git clone https://github.com/MakazhanAlpamys/Soup.git
+cd Soup
+pip install -e ".[dev]"
+
+# Lint
+ruff check soup_cli/ tests/
+
+# Run unit tests (fast, no GPU needed — 147 tests)
+pytest tests/ -v
+
+# Run smoke tests (downloads tiny model, runs real training)
+pytest tests/ -m smoke -v
+```
+
+## License
+
+MIT