cupel 0.1.0__tar.gz

cupel-0.1.0/.gitignore

@@ -0,0 +1,170 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+src/lab/
+hyperland/lab/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+.claude
+prototype/
+requirements/
+.DS_Store
+eval-results/
+eval-set.json

cupel-0.1.0/PKG-INFO

@@ -0,0 +1,471 @@
+Metadata-Version: 2.4
+Name: cupel
+Version: 0.1.0
+Summary: separates precious LLMs from base LLMs. works with any OpenAI/Anthropic compatible API
+Project-URL: Homepage, https://github.com/tolitius/cupel
+Project-URL: Repository, https://github.com/tolitius/cupel
+Author: tolitius
+License: MIT
+Keywords: benchmark,eval,leaderboard,llm,local-llm
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.11
+Requires-Dist: fastapi>=0.110
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: requests>=2.31
+Requires-Dist: rich>=13.0
+Requires-Dist: uvicorn>=0.27
+Description-Content-Type: text/markdown
+
+# bench
+
+> run prompts against local LLMs, then let a judge score them
+
+bench sends the same set of prompts to multiple models running on any OpenAI-compatible API,<br/>
+collects responses (with thinking separated from the answer), and then scores them using a configurable judge — which can be another local model, Claude, GPT, or anything with an API.
+
+- [what it does](#what-it-does)
+- [install](#install)
+- [run it](#run-it)
+  - [collect responses](#collect-responses)
+  - [run specific prompts](#run-specific-prompts)
+  - [score them](#score-them)
+  - [judge with Claude](#judge-with-claude)
+  - [judge with a local model](#judge-with-a-local-model)
+- [config](#config)
+  - [models and inference](#models-and-inference)
+  - [judge](#judge)
+  - [endpoint and keys](#endpoint-and-keys)
+- [the eval set](#the-eval-set)
+  - [prompts](#prompts)
+  - [multi-turn prompts](#multi-turn-prompts)
+  - [writing your own](#writing-your-own)
+- [thinking models](#thinking-models)
+- [license](#license)
+
+# what it does
+
+you have a few local models. you want to know which one is actually good at the things _you_ care about — not MMLU, not HumanEval, but your questions, your domain, your work.
+
+bench lets you:
+
+1. **define prompts** with scoring rubrics in a JSON file
+2. **run** them against multiple models in one shot
+3. **judge** the responses with any model (local or API)
+4. **compare** scores side by side with response times
+
+the judge and the models under test don't have to be on the same server.<br/>
+you can collect responses locally and score them with Claude later, or vice versa.
+
+# install
+
+```bash
+$ git clone https://github.com/tolitius/bench.git
+$ cd bench
+$ pip install .
+```
+
+this installs three dependencies: `requests`, `pyyaml`, `rich`
+
+alternatively:
+```bash
+$ pip install -r requirements.txt
+```
+
+if you skip the install and just run `python eval.py`, it will tell you what's missing:
+
+```
+  missing required packages: pyyaml
+  python: /usr/local/bin/python3.12 (3.12.4)
+
+  install with one of:
+    /usr/local/bin/python3.12 -m pip install .                        # from repo root
+    /usr/local/bin/python3.12 -m pip install -r requirements.txt      # alternative
+    /usr/local/bin/python3.12 -m pip install pyyaml
+```
+
+it shows the exact python path so you don't end up installing packages into the wrong interpreter.
+
+# run it
+
+bench has two commands: `run` and `judge`
+
+## collect responses
+
+```bash
+$ python eval.py run
+```
+```
+  config:    config.yml
+  .env:      .env
+  endpoint:  localhost:8000
+  eval set:  My Eval Set (12 prompts)
+  models:    Qwen3.5-35B-A3B-4bit, Qwen3.5-9B-bf16, Qwen3.5-27B-8bit
+  thinking:  model default
+  output:    ./eval-results
+
+  ⚡ Local LLM Eval  @ localhost:8000
+   #  Prompt                          Q35B·4b    Q9B·bf16   Q27B·8b
+   1  Image Description               19.65s     68.97s     32.4s
+   2  Explain a Sorting Algorithm     16.65s     67.06s     28.1s
+   ...
+   Done                               12/12      12/12     12/12
+
+  Saved: eval-results/eval_Qwen3.5-35B-A3B-4bit_20260321_194308.json
+  Saved: eval-results/eval_Qwen3.5-9B-bf16_20260321_194308.json
+  Saved: eval-results/eval_Qwen3.5-27B-8bit_20260321_194308.json
+
+✅ Run complete. To score:
+  python eval.py judge eval-results/eval_*.json
+```
+
+each result JSON has the prompt, the model's response, thinking (if any), and timing.<br/>
+no scoring happens here — just collection.
+
+## run specific prompts
+
+don't want to wait through all 22 prompts to retest the 3 you just changed? use `--prompts`:
+
+```bash
+$ python eval.py run --prompts 18-22              # range
+$ python eval.py run --prompts 21,22              # specific IDs
+$ python eval.py run --prompts 1,18-22            # mix of both
+```
+
+works with `--models` too:
+
+```bash
+$ python eval.py run --prompts 21,22 --models gpt-oss-120b-MXFP4-Q8
+```
+
+## score them
+
+```bash
+$ python eval.py judge eval-results/eval_*.json
+```
+```
+  judge:     Qwen3.5-27B-8bit @ localhost:8000
+  scoring:   3 file(s), 12 prompts each
+
+  ⚖ Scoring  @ localhost:8000
+   #  Prompt                          Q35B·4b        Q9B·bf16       Q27B·8b
+   1  Image Description               3 (19.65s)     2 (68.97s)     3 (32.4s)
+   2  Explain a Sorting Algorithm     2 (16.65s)     1 (67.06s)     3 (28.1s)
+   ...
+   Score                              38/51          24/51          41/51
+
+  Summary: eval-results/scoring_Qwen3.5-27B-8bit_20260321.md
+
+  Qwen3.5-35B-A3B-4bit              38/51
+  Qwen3.5-9B-bf16                   24/51
+  Qwen3.5-27B-8bit                  41/51
+
+✅ Judging complete.
+```
+
+the judge sends each response + its rubric to the judge model, which returns a 0-3 score:
+
+| score | meaning |
+| --- | --- |
+| 0 | wrong or hallucinated |
+| 1 | partially correct |
+| 2 | correct but shallow |
+| 3 | correct and insightful |
+
+scores are written back into the result JSONs and a markdown summary is generated.
+
+## judge with Claude
+
+point the judge at Anthropic's API directly:
+
+```yaml
+# config.yml
+judge:
+  model: claude-sonnet-4-20250514
+  api_url: https://api.anthropic.com/v1/messages
+  api_key_env: ANTHROPIC_API_KEY
+```
+```bash
+# .env
+ANTHROPIC_API_KEY=sk-ant-...
+```
+```bash
+$ python eval.py judge eval-results/eval_*.json
+```
+
+bench detects `api.anthropic.com` and uses the correct auth header (`x-api-key`) and response format automatically.
+
+## judge with a local model
+
+```yaml
+# config.yml
+judge:
+  model: Qwen3.5-27B-8bit
+```
+
+when `api_url` is omitted, the judge uses the same endpoint from `.env` as the eval run.
+
+you can also override the judge model from the command line:
+
+```bash
+$ python eval.py judge eval-results/eval_*.json --judge-model Qwen3.5-122B-A10B-4bit
+```
+
+or point at a completely different server:
+
+```bash
+$ python eval.py judge eval-results/eval_*.json \
+    --judge-model gpt-4o \
+    --judge-url https://api.openai.com/v1/chat/completions \
+    --judge-key-env OPENAI_API_KEY
+```
+
+# config
+
+## models and inference
+
+```yaml
+# config.yml
+
+models:
+  - Qwen3.5-35B-A3B-4bit
+  - Qwen3.5-9B-bf16
+  - Qwen3.5-122B-A10B-4bit
+  - Qwen3.5-27B-8bit
+  - Nemotron-Cascade-2-30B-A3B-8bit
+
+eval_set: eval-set.json
+output_dir: ./eval-results
+temperature: 0
+max_tokens: 16384
+thinking: null          # null = model default, 0 = off, 4096 = explicit budget
+```
+
+`max_tokens` covers _both_ thinking and the response.<br/>
+Qwen3.5 models can burn 2-4K tokens thinking before answering, so 16384 gives plenty of room.
+
+## judge
+
+```yaml
+# config.yml
+
+judge:
+  model: claude-sonnet-4-20250514
+  api_url: https://api.anthropic.com/v1/messages
+  api_key_env: ANTHROPIC_API_KEY
+```
+
+| field | what it does | default |
+| --- | --- | --- |
+| `model` | which model scores responses | _(required)_ |
+| `api_url` | endpoint for the judge | reuses `LLM_API_URL` from `.env` |
+| `api_key_env` | env var name holding the key | reuses `LLM_API_KEY` from `.env` |
+
+some examples:
+
+```yaml
+# local (same server as eval)
+judge:
+  model: Qwen3.5-27B-8bit
+
+# OpenAI
+judge:
+  model: gpt-4o
+  api_url: https://api.openai.com/v1/chat/completions
+  api_key_env: OPENAI_API_KEY
+
+# Anthropic (direct)
+judge:
+  model: claude-sonnet-4-20250514
+  api_url: https://api.anthropic.com/v1/messages
+  api_key_env: ANTHROPIC_API_KEY
+```
+
+## endpoint and keys
+
+```bash
+# .env
+
+LLM_API_URL=http://localhost:8000/v1/chat/completions
+LLM_API_KEY=2893692
+
+# only if judge uses a different API:
+# OPENAI_API_KEY=sk-proj-...
+# ANTHROPIC_API_KEY=sk-ant-...
+```
+
+works with any OpenAI-compatible server: oMLX, Ollama, LM Studio, vLLM, SGLang, llama.cpp
+
+env vars override `.env` file values, so you can do one-off overrides:
+```bash
+$ LLM_API_URL=http://other-server:8000/v1/chat/completions python eval.py run
+```
+
+# the eval set
+
+## prompts
+
+the repo ships with a sample `eval-set-sample.json` to show the format. the idea is that you write your own — prompts that test what _you_ actually care about.
+
+good prompts for a personal eval set:
+
+| category | example |
+| --- | --- |
+| multimodal | show an image, ask "what is this?" |
+| code reading | paste a function, ask what it does and how to improve it |
+| system design | "design an X that does Y given these constraints" |
+| debugging | describe symptoms, ask for root cause in priority order |
+| domain knowledge | something only someone in your field would know well |
+| math / estimation | a calculation with a verifiable answer |
+| architecture | "compare X vs Y, when would you choose each?" |
+| business logic | a tricky edge case from your actual work |
+| assistant competence | give structured data about multiple entities, test if the model keeps them straight |
+| tool calling | define tools in a system prompt, test if the model emits correct JSON calls |
+
+a few principles that make prompts work well for eval:
+
+- **self-contained** — no hidden context, no prior conversation
+- **verifiable** — you can tell if the answer is right or wrong
+- **no tools needed** — no web search, no code execution (unless testing tool-call _format_)
+- **stable** — the answer shouldn't change next month
+
+each prompt includes a rubric that the judge uses:
+
+```json
+{
+  "id": 1,
+  "category": "math_estimation",
+  "title": "Estimate Memory from Quantization",
+  "prompt": "A model has 7 billion parameters. Estimate the memory footprint for FP16, 8-bit, and 4-bit.",
+  "rubric": {
+    "3": "FP16: ~14GB, 8-bit: ~7GB, 4-bit: ~3.5GB. Shows the math.",
+    "2": "Correct for 2 of 3, or all correct but no explanation.",
+    "1": "Gets the direction right but wrong numbers.",
+    "0": "Wrong math or doesn't understand quantization."
+  }
+}
+```
+
+## multi-turn prompts
+
+not every capability shows up in a single Q&A turn. some things — tool calling, context shifts, follow-up questions — need a conversation.
+
+multi-turn prompts use `turns` instead of `prompt`:
+
+```json
+{
+  "id": 21,
+  "category": "assistant_competence",
+  "title": "Tool Calling — School Status Check",
+  "turns": [
+    {
+      "messages": [
+        {"role": "system", "content": "You are an assistant with these tools: ..."},
+        {"role": "user", "content": "How are both kids doing?"}
+      ]
+    },
+    {
+      "inject_after": [
+        {"role": "user", "content": "Tool results:\n\nget_grades(\"phoebe\") => ..."}
+      ],
+      "messages": []
+    },
+    {
+      "messages": [
+        {"role": "user", "content": "Now check last week's attendance too."}
+      ]
+    }
+  ],
+  "rubric": {
+    "3": "Turn 1 emits correct tool calls as JSON. Turn 2 synthesizes results accurately. ...",
+    "2": "...",
+    "1": "...",
+    "0": "..."
+  }
+}
+```
+
+each turn has:
+
+- **`messages`** — added to history before calling the model. if the last message is from a user, the model responds.
+- **`inject_after`** — added to history _after_ the model responds (or immediately if there are no messages). if the injected messages end with a user message, the model responds again.
+
+this lets you simulate tool-call workflows: the model emits tool calls, you inject fake results, the model synthesizes — all scored by the judge at the end.
+
+the judge sees the full conversation transcript (all turns, all model responses) when scoring.
+
+**when to use multi-turn vs single-turn:**
+
+a model that aces single-turn knowledge questions can still fail badly as an assistant. multi-turn prompts catch things like:
+
+- can it emit tool calls in the format you asked for, or does it fabricate data?
+- can it stay coherent across turns without mixing up entities?
+- does it follow workflow rules (check health _before_ investigating, don't rollback without approval)?
+
+if you're evaluating models for assistant / agent use, you probably want both.
+
+## writing your own
+
+create an `eval-set.json` following the same structure. copy `eval-set-sample.json` as a starting point:
+
+```bash
+$ cp eval-set-sample.json eval-set.json
+```
+
+add, remove, or replace prompts:
+
+```json
+{
+  "id": 4,
+  "category": "debugging",
+  "title": "Connection Pool Exhaustion",
+  "prompt": "A web service starts returning 503s after 2 hours under steady load. DB connections are not being returned to the pool. What are the most likely causes in priority order?",
+  "rubric": {
+    "3": "Identifies: unclosed connections in error paths, missing finally/context manager, connection leak on timeout. Correct priority.",
+    "2": "Gets top cause right but weak prioritization.",
+    "1": "Some valid causes but misses the leak pattern.",
+    "0": "Doesn't understand connection pooling."
+  }
+}
+```
+
+one prompt can use an image (for vision models). set `image_filename` in config.yml and place the image in the repo root or use `--image-dir`.
+
+prompts can be single-turn (a `prompt` string) or multi-turn (a `turns` array) — see [multi-turn prompts](#multi-turn-prompts) above.<br/>
+when a model returns native tool calls (OpenAI `tool_calls` format), bench captures them as JSON in the response so the judge can see what the model tried to do.
+
+# thinking models
+
+Qwen3.5, DeepSeek, and other reasoning models emit `<think>...</think>` blocks before their answer.<br/>
+bench handles this automatically:
+
+- the thinking is **separated** from the response and stored in a `thinking` field
+- only the **response after thinking** is sent to the judge
+- if thinking consumes all tokens and no answer is produced, it's flagged as an error
+- `thinking: null` in config means "let models think naturally" — this is the recommended default
+- some servers (oMLX, Ollama) leak think tags into the content field — bench strips them
+
+if you want to explicitly disable thinking:
+
+```yaml
+thinking: 0
+```
+
+or set a budget:
+
+```yaml
+thinking: 4096
+```
+
+# license
+
+Copyright © 2026 tolitius
+
+Distributed under the Eclipse Public License either version 1.0 or (at
+your option) any later version.