arbiter-cli 0.1.0__tar.gz

arbiter_cli-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 BasaltLabs
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

arbiter_cli-0.1.0/PKG-INFO

@@ -0,0 +1,299 @@
+Metadata-Version: 2.4
+Name: arbiter-cli
+Version: 0.1.0
+Summary: The final word on your local models. Compare LLMs side-by-side with animated visualizations.
+Author-email: BasaltLabs <hello@basaltlabs.app>
+License: MIT
+Project-URL: Homepage, https://github.com/Basaltlabs-app/Arbiter
+Project-URL: Repository, https://github.com/Basaltlabs-app/Arbiter
+Project-URL: Issues, https://github.com/Basaltlabs-app/Arbiter/issues
+Keywords: llm,benchmark,comparison,ollama,local-ai,gemma,qwen,llama
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: typer>=0.9.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: httpx>=0.25.0
+Requires-Dist: psutil>=5.9.0
+Requires-Dist: fastapi>=0.100.0
+Requires-Dist: uvicorn>=0.23.0
+Requires-Dist: websockets>=12.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Dynamic: license-file
+
+<p align="center">
+  <img src="https://img.shields.io/badge/arbiter-v0.1.0-00d4ff?style=for-the-badge" alt="version" />
+</p>
+
+<h1 align="center">Arbiter</h1>
+
+<p align="center">
+  <strong>The final word on your local models.</strong><br>
+  Real benchmarks. Real code tests. Beautiful dashboard. Fully local.
+</p>
+
+<p align="center">
+  <a href="#install">Install</a> &bull;
+  <a href="#quick-start">Quick Start</a> &bull;
+  <a href="#what-it-tests">What It Tests</a> &bull;
+  <a href="#dashboard">Dashboard</a> &bull;
+  <a href="#swe-testing">SWE Testing</a> &bull;
+  <a href="#how-scoring-works">Scoring</a>
+</p>
+
+<p align="center">
+  <img src="https://img.shields.io/pypi/v/arbiter-cli?color=00d4ff" alt="PyPI" />
+  <img src="https://img.shields.io/github/license/Basaltlabs-app/Arbiter" alt="License" />
+  <img src="https://img.shields.io/badge/AI-100%25%20Local-10b981" alt="Local AI" />
+  <img src="https://img.shields.io/badge/models-any%20Ollama%20%2B%20cloud-7c3aed" alt="Models" />
+  <img src="https://img.shields.io/badge/judge-no%20LLM%20vibes-f59e0b" alt="No vibes" />
+</p>
+
+---
+
+New models drop every week. Gemma 4. Qwen 3.5. Llama 4. But lab benchmarks don't tell you how they run **on your machine** for **your use cases**.
+
+**Arbiter** runs real automated tests against any model, verifies results programmatically (not by asking another LLM), and shows you exactly why one model beats another.
+
+```bash
+pip install arbiter-cli
+arbiter benchmark
+```
+
+No API keys. No cloud. Everything runs on your hardware.
+
+---
+
+## Install
+
+```bash
+pip install arbiter-cli
+```
+
+**Requirements:**
+- Python 3.9+
+- [Ollama](https://ollama.com) with at least one model installed
+
+```bash
+# Install Ollama, then pull a model:
+ollama pull qwen3.5:4b
+```
+
+## Quick Start
+
+```bash
+# Open the dashboard (recommended -- everything in the browser)
+arbiter dashboard
+
+# Run the benchmark suite from terminal
+arbiter benchmark
+
+# Quick benchmark (8 key tests, faster)
+arbiter benchmark --quick
+
+# Compare models with your own prompt
+arbiter run "explain how a hash table works"
+
+# List your installed models
+arbiter discover
+
+# View persistent rankings
+arbiter leaderboard
+```
+
+The dashboard auto-detects your installed models. Pick models, type a prompt or hit "Run Benchmark", see results.
+
+## What It Tests
+
+**20 automated tests across 8 categories.** Every test has a programmatic pass/fail check. No model judges another model.
+
+| Category | What It Tests | How It Verifies |
+|---|---|---|
+| **Instruction Following** | Does it do exactly what you asked? | Line count, format regex, constraint check |
+| **Code Generation** | Can it fix bugs and build real code? | AST parsing, structural analysis |
+| **Factual Accuracy** | Real facts or hallucinations? | Ground truth match against known answers |
+| **Reasoning** | Multi-step logic, math, time | Expected answer comparison |
+| **Consistency** | Same question 3 ways = same answer? | Cross-run similarity scoring |
+| **Pressure Resistance** | Does it cave when you push back? | Sycophancy detection (correct answer flip) |
+| **Speed** | Tokens/sec on YOUR hardware | Direct measurement |
+| **Context Recall** | Find info hidden in longer text | Hidden value retrieval |
+
+### Code Generation Tests (SWE-bench inspired)
+
+These aren't toy FizzBuzz problems. Arbiter gives models real buggy code and checks if the fix actually works:
+
+| Test | What the Model Must Do |
+|---|---|
+| **Binary Search Fix** | Find the off-by-one bug, fix it. 7 pytest assertions. |
+| **Edge Case Handling** | Write safe_divide with zero, type, and negative handling |
+| **Data Structure** | Build a Stack class with proper error handling |
+| **API Design** | Sliding window rate limiter with time tracking |
+| **Code Comprehension** | Predict output of tricky Python (slice references) |
+| **JSON Output** | Generate valid, parseable structured data |
+
+### Example Output
+
+```
+Overall Scores
+Higher is better. 100 = perfect on every test.
+
+  gemma4:e2b          ████████████████░░░░░░░░░░░░░░░░  53/100  (5/8 passed)
+  qwen3.5:4b          ██████░░░░░░░░░░░░░░░░░░░░░░░░░░  21/100  (2/8 passed)
+
+  Factual Accuracy  Does it give real facts or make things up?
+    gemma4:e2b         100%  BEST
+    qwen3.5:4b           0%
+      Known factual answers:  PASS  FAIL
+
+  Code Generation  Can it write working code?
+    gemma4:e2b           0%
+    qwen3.5:4b          50%  BEST
+      Fix binary search bug:  FAIL  PASS
+
+  Winner: gemma4:e2b scored 53/100, passing 5 of 8 tests.
+  Strong at: factual accuracy, pressure resistance, reasoning.
+  Weaker at: code generation (beaten by qwen3.5:4b).
+```
+
+Every number tells you what was tested and what happened. No black boxes.
+
+## Dashboard
+
+```bash
+arbiter dashboard
+```
+
+Opens a full GUI in your browser at `http://127.0.0.1:7878`. Everything you can do in the terminal, you can do from the dashboard:
+
+- **Select models** from auto-detected installed list
+- **Type prompts** and run comparisons
+- **Run benchmarks** with one click
+- **View results** with animated charts and graphs
+- **Track rankings** across all your comparisons
+
+### Tabs
+
+| Tab | What It Shows |
+|---|---|
+| **Arena** | Live token race, winner announcement, model output cards |
+| **Benchmark** | Full test suite with category breakdown and pass/fail per test |
+| **Speed** | Animated bar charts for tok/s, TTFT, total generation time |
+| **Quality** | Radar chart comparing quality dimensions |
+| **Graph** | Force-directed node graph showing model relationships |
+| **Rankings** | Persistent ELO leaderboard with sparklines |
+| **Help** | Getting started guide, FAQ, terminal command reference |
+
+The dashboard shows your system RAM and warns if a model is too large for your machine.
+
+## SWE Testing
+
+Real software engineering tests in the style of [SWE-bench](https://www.swebench.com/). Models receive buggy code + a bug report, write a fix, and the fix is verified by running pytest.
+
+```bash
+arbiter swe
+```
+
+7 built-in challenges across bug fixes and algorithm implementations. Tests run in a sandboxed subprocess (no Docker required) or optionally in Docker containers with `--docker`.
+
+| Challenge | Type | Pytest Assertions |
+|---|---|---|
+| Binary Search Off-by-One | Bug fix | 7 |
+| LRU Cache Eviction | Bug fix | 5 |
+| Flatten Nested List | Bug fix | 6 |
+| Rate Limiter Window | Implementation | 4 |
+| Linked List Cycle (O(1) memory) | Implementation | 6 |
+| Two Sum (O(n) time) | Algorithm | 5 |
+| Merge Intervals | Algorithm | 6 |
+
+## How Scoring Works
+
+### Benchmark Suite
+
+Each test is **pass/fail** with programmatic verification. The overall score is the percentage of tests passed. Category scores are averages within each category. No LLM judges another LLM.
+
+### Run Comparison
+
+When comparing models with a custom prompt, Arbiter uses a **transparent composite formula**:
+
+```
+Score = Speed(30%) + Quality(50%) + Responsiveness(20%)
+```
+
+- **Speed:** `model_tps / best_tps` (higher tokens/sec = better)
+- **Responsiveness:** `best_ttft / model_ttft` (lower time-to-first-token = better)
+- **Quality:** `model_score / 10` (from optional LLM judge, or skipped with `--no-judge`)
+
+You see every component. You see the raw values. You see exactly why Model A scored higher than Model B.
+
+### ELO Leaderboard
+
+Every comparison updates a persistent [ELO rating](https://en.wikipedia.org/wiki/Elo_rating_system) stored in `~/.arbiter/leaderboard.json`. Rankings build over time across all your comparisons. Multi-model comparisons scale the K-factor to prevent rating inflation.
+
+## Multi-Provider Support
+
+Arbiter works with any model you can run:
+
+| Provider | Format | Example |
+|---|---|---|
+| **Ollama** (local) | `model_name` | `qwen3.5:4b` |
+| **OpenAI** | `openai:model` | `openai:gpt-4o` |
+| **Anthropic** | `anthropic:model` | `anthropic:claude-sonnet-4-20250514` |
+| **Google** | `google:model` | `google:gemini-2.0-flash` |
+| **Any OpenAI-compatible** | `http://host:port/v1:model` | Custom endpoints |
+
+Cloud providers require API keys via environment variables (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY`).
+
+## Low RAM? No Problem
+
+Arbiter was built and tested on an **8GB M1 MacBook Air**. Sequential mode runs one model at a time so each gets your full RAM:
+
+```bash
+arbiter benchmark --seq
+arbiter run "your prompt" --seq
+```
+
+The dashboard shows system RAM and warns if a model is too large.
+
+## Philosophy
+
+- **Your hardware, your results.** Lab benchmarks don't tell you how a model runs on your machine.
+- **Your prompts, your answers.** Generic benchmarks don't test what you actually use AI for.
+- **No vibes.** Every test has a programmatic pass/fail. No "this feels like a 7/10."
+- **Fully local.** Your code and prompts never leave your computer.
+- **Transparent.** See every score, every formula, every raw value. No black boxes.
+
+## Contributing
+
+We welcome contributions! Areas we need help with:
+
+- **New test packs** -- submit benchmark tests for specific domains (data science, web dev, etc.)
+- **New providers** -- add support for more LLM APIs
+- **Dashboard improvements** -- better visualizations, mobile responsive
+- **Documentation** -- tutorials, guides, translations
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for details.
+
+## License
+
+MIT
+
+---
+
+<p align="center">
+  Built by <a href="https://basaltlabs.app">BasaltLabs</a><br>
+  <sub>The final word on your local models.</sub>
+</p>

arbiter_cli-0.1.0/README.md

@@ -0,0 +1,262 @@
+<p align="center">
+  <img src="https://img.shields.io/badge/arbiter-v0.1.0-00d4ff?style=for-the-badge" alt="version" />
+</p>
+
+<h1 align="center">Arbiter</h1>
+
+<p align="center">
+  <strong>The final word on your local models.</strong><br>
+  Real benchmarks. Real code tests. Beautiful dashboard. Fully local.
+</p>
+
+<p align="center">
+  <a href="#install">Install</a> &bull;
+  <a href="#quick-start">Quick Start</a> &bull;
+  <a href="#what-it-tests">What It Tests</a> &bull;
+  <a href="#dashboard">Dashboard</a> &bull;
+  <a href="#swe-testing">SWE Testing</a> &bull;
+  <a href="#how-scoring-works">Scoring</a>
+</p>
+
+<p align="center">
+  <img src="https://img.shields.io/pypi/v/arbiter-cli?color=00d4ff" alt="PyPI" />
+  <img src="https://img.shields.io/github/license/Basaltlabs-app/Arbiter" alt="License" />
+  <img src="https://img.shields.io/badge/AI-100%25%20Local-10b981" alt="Local AI" />
+  <img src="https://img.shields.io/badge/models-any%20Ollama%20%2B%20cloud-7c3aed" alt="Models" />
+  <img src="https://img.shields.io/badge/judge-no%20LLM%20vibes-f59e0b" alt="No vibes" />
+</p>
+
+---
+
+New models drop every week. Gemma 4. Qwen 3.5. Llama 4. But lab benchmarks don't tell you how they run **on your machine** for **your use cases**.
+
+**Arbiter** runs real automated tests against any model, verifies results programmatically (not by asking another LLM), and shows you exactly why one model beats another.
+
+```bash
+pip install arbiter-cli
+arbiter benchmark
+```
+
+No API keys. No cloud. Everything runs on your hardware.
+
+---
+
+## Install
+
+```bash
+pip install arbiter-cli
+```
+
+**Requirements:**
+- Python 3.9+
+- [Ollama](https://ollama.com) with at least one model installed
+
+```bash
+# Install Ollama, then pull a model:
+ollama pull qwen3.5:4b
+```
+
+## Quick Start
+
+```bash
+# Open the dashboard (recommended -- everything in the browser)
+arbiter dashboard
+
+# Run the benchmark suite from terminal
+arbiter benchmark
+
+# Quick benchmark (8 key tests, faster)
+arbiter benchmark --quick
+
+# Compare models with your own prompt
+arbiter run "explain how a hash table works"
+
+# List your installed models
+arbiter discover
+
+# View persistent rankings
+arbiter leaderboard
+```
+
+The dashboard auto-detects your installed models. Pick models, type a prompt or hit "Run Benchmark", see results.
+
+## What It Tests
+
+**20 automated tests across 8 categories.** Every test has a programmatic pass/fail check. No model judges another model.
+
+| Category | What It Tests | How It Verifies |
+|---|---|---|
+| **Instruction Following** | Does it do exactly what you asked? | Line count, format regex, constraint check |
+| **Code Generation** | Can it fix bugs and build real code? | AST parsing, structural analysis |
+| **Factual Accuracy** | Real facts or hallucinations? | Ground truth match against known answers |
+| **Reasoning** | Multi-step logic, math, time | Expected answer comparison |
+| **Consistency** | Same question 3 ways = same answer? | Cross-run similarity scoring |
+| **Pressure Resistance** | Does it cave when you push back? | Sycophancy detection (correct answer flip) |
+| **Speed** | Tokens/sec on YOUR hardware | Direct measurement |
+| **Context Recall** | Find info hidden in longer text | Hidden value retrieval |
+
+### Code Generation Tests (SWE-bench inspired)
+
+These aren't toy FizzBuzz problems. Arbiter gives models real buggy code and checks if the fix actually works:
+
+| Test | What the Model Must Do |
+|---|---|
+| **Binary Search Fix** | Find the off-by-one bug, fix it. 7 pytest assertions. |
+| **Edge Case Handling** | Write safe_divide with zero, type, and negative handling |
+| **Data Structure** | Build a Stack class with proper error handling |
+| **API Design** | Sliding window rate limiter with time tracking |
+| **Code Comprehension** | Predict output of tricky Python (slice references) |
+| **JSON Output** | Generate valid, parseable structured data |
+
+### Example Output
+
+```
+Overall Scores
+Higher is better. 100 = perfect on every test.
+
+  gemma4:e2b          ████████████████░░░░░░░░░░░░░░░░  53/100  (5/8 passed)
+  qwen3.5:4b          ██████░░░░░░░░░░░░░░░░░░░░░░░░░░  21/100  (2/8 passed)
+
+  Factual Accuracy  Does it give real facts or make things up?
+    gemma4:e2b         100%  BEST
+    qwen3.5:4b           0%
+      Known factual answers:  PASS  FAIL
+
+  Code Generation  Can it write working code?
+    gemma4:e2b           0%
+    qwen3.5:4b          50%  BEST
+      Fix binary search bug:  FAIL  PASS
+
+  Winner: gemma4:e2b scored 53/100, passing 5 of 8 tests.
+  Strong at: factual accuracy, pressure resistance, reasoning.
+  Weaker at: code generation (beaten by qwen3.5:4b).
+```
+
+Every number tells you what was tested and what happened. No black boxes.
+
+## Dashboard
+
+```bash
+arbiter dashboard
+```
+
+Opens a full GUI in your browser at `http://127.0.0.1:7878`. Everything you can do in the terminal, you can do from the dashboard:
+
+- **Select models** from auto-detected installed list
+- **Type prompts** and run comparisons
+- **Run benchmarks** with one click
+- **View results** with animated charts and graphs
+- **Track rankings** across all your comparisons
+
+### Tabs
+
+| Tab | What It Shows |
+|---|---|
+| **Arena** | Live token race, winner announcement, model output cards |
+| **Benchmark** | Full test suite with category breakdown and pass/fail per test |
+| **Speed** | Animated bar charts for tok/s, TTFT, total generation time |
+| **Quality** | Radar chart comparing quality dimensions |
+| **Graph** | Force-directed node graph showing model relationships |
+| **Rankings** | Persistent ELO leaderboard with sparklines |
+| **Help** | Getting started guide, FAQ, terminal command reference |
+
+The dashboard shows your system RAM and warns if a model is too large for your machine.
+
+## SWE Testing
+
+Real software engineering tests in the style of [SWE-bench](https://www.swebench.com/). Models receive buggy code + a bug report, write a fix, and the fix is verified by running pytest.
+
+```bash
+arbiter swe
+```
+
+7 built-in challenges across bug fixes and algorithm implementations. Tests run in a sandboxed subprocess (no Docker required) or optionally in Docker containers with `--docker`.
+
+| Challenge | Type | Pytest Assertions |
+|---|---|---|
+| Binary Search Off-by-One | Bug fix | 7 |
+| LRU Cache Eviction | Bug fix | 5 |
+| Flatten Nested List | Bug fix | 6 |
+| Rate Limiter Window | Implementation | 4 |
+| Linked List Cycle (O(1) memory) | Implementation | 6 |
+| Two Sum (O(n) time) | Algorithm | 5 |
+| Merge Intervals | Algorithm | 6 |
+
+## How Scoring Works
+
+### Benchmark Suite
+
+Each test is **pass/fail** with programmatic verification. The overall score is the percentage of tests passed. Category scores are averages within each category. No LLM judges another LLM.
+
+### Run Comparison
+
+When comparing models with a custom prompt, Arbiter uses a **transparent composite formula**:
+
+```
+Score = Speed(30%) + Quality(50%) + Responsiveness(20%)
+```
+
+- **Speed:** `model_tps / best_tps` (higher tokens/sec = better)
+- **Responsiveness:** `best_ttft / model_ttft` (lower time-to-first-token = better)
+- **Quality:** `model_score / 10` (from optional LLM judge, or skipped with `--no-judge`)
+
+You see every component. You see the raw values. You see exactly why Model A scored higher than Model B.
+
+### ELO Leaderboard
+
+Every comparison updates a persistent [ELO rating](https://en.wikipedia.org/wiki/Elo_rating_system) stored in `~/.arbiter/leaderboard.json`. Rankings build over time across all your comparisons. Multi-model comparisons scale the K-factor to prevent rating inflation.
+
+## Multi-Provider Support
+
+Arbiter works with any model you can run:
+
+| Provider | Format | Example |
+|---|---|---|
+| **Ollama** (local) | `model_name` | `qwen3.5:4b` |
+| **OpenAI** | `openai:model` | `openai:gpt-4o` |
+| **Anthropic** | `anthropic:model` | `anthropic:claude-sonnet-4-20250514` |
+| **Google** | `google:model` | `google:gemini-2.0-flash` |
+| **Any OpenAI-compatible** | `http://host:port/v1:model` | Custom endpoints |
+
+Cloud providers require API keys via environment variables (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY`).
+
+## Low RAM? No Problem
+
+Arbiter was built and tested on an **8GB M1 MacBook Air**. Sequential mode runs one model at a time so each gets your full RAM:
+
+```bash
+arbiter benchmark --seq
+arbiter run "your prompt" --seq
+```
+
+The dashboard shows system RAM and warns if a model is too large.
+
+## Philosophy
+
+- **Your hardware, your results.** Lab benchmarks don't tell you how a model runs on your machine.
+- **Your prompts, your answers.** Generic benchmarks don't test what you actually use AI for.
+- **No vibes.** Every test has a programmatic pass/fail. No "this feels like a 7/10."
+- **Fully local.** Your code and prompts never leave your computer.
+- **Transparent.** See every score, every formula, every raw value. No black boxes.
+
+## Contributing
+
+We welcome contributions! Areas we need help with:
+
+- **New test packs** -- submit benchmark tests for specific domains (data science, web dev, etc.)
+- **New providers** -- add support for more LLM APIs
+- **Dashboard improvements** -- better visualizations, mobile responsive
+- **Documentation** -- tutorials, guides, translations
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for details.
+
+## License
+
+MIT
+
+---
+
+<p align="center">
+  Built by <a href="https://basaltlabs.app">BasaltLabs</a><br>
+  <sub>The final word on your local models.</sub>
+</p>

arbiter_cli-0.1.0/arbiter/__init__.py

@@ -0,0 +1,3 @@
+"""Arbiter - The final word on your local models."""
+
+__version__ = "0.1.0"