nano-eval 0.2.4__tar.gz

nano_eval-0.2.4/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Thomas Børstad
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

nano_eval-0.2.4/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.4
+Name: nano-eval
+Version: 0.2.4
+Summary: Nano Eval - A minimal tool for verifying VLMs/LLMs across frameworks
+Author-email: Thomas Børstad <tboerstad@users.noreply.github.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/tboerstad/nano-eval
+Project-URL: Repository, https://github.com/tboerstad/nano-eval
+Project-URL: Issues, https://github.com/tboerstad/nano-eval/issues
+Keywords: llm,evaluation,benchmark,openai,api,vllm,tgi
+Classifier: Development Status :: 4 - Beta
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: click>=8.0.0
+Requires-Dist: datasets>=2.0.0
+Requires-Dist: httpx>=0.23.0
+Requires-Dist: pillow>=9.0.0
+Requires-Dist: tqdm>=4.62.0
+Requires-Dist: typing-extensions>=4.0.0
+Requires-Dist: multiprocess<=0.70.17
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Requires-Dist: pre-commit>=3.0.0; extra == "dev"
+Requires-Dist: respx>=0.20.0; extra == "dev"
+Requires-Dist: ty>=0.0.1; extra == "dev"
+Dynamic: license-file
+
+**nano-eval** is a minimal tool for measuring the quality of a text or vision model.
+
+## Quickstart
+
+```bash
+uvx nano-eval -t text -t vision --base-url http://localhost:8000/v1 --max-samples 100
+
+# prints:
+Task    Accuracy  Samples  Duration
+------  --------  -------  --------
+text      84.3%      100       45s
+vision    71.8%      100       38s
+```
+
+> **Note:** This tool is for eyeballing the accuracy of a model. One use case is comparing accuracy between inference frameworks (e.g., vLLM vs SGLang vs MAX running the same model).
+
+## Supported Types
+
+| Type | Dataset | Description |
+|------|---------|-------------|
+| `text` | gsm8k_cot_llama | Grade school math with chain-of-thought (8-shot) |
+| `vision` | HuggingFaceM4/ChartQA | Chart question answering with images |
+
+## Usage
+
+```
+$ nano-eval --help
+Usage: nano-eval [OPTIONS]
+
+  Evaluate LLMs on standardized tasks via OpenAI-compatible APIs.
+
+  Example: nano-eval -t text --base-url http://localhost:8000/v1
+
+Options:
+  -t, --type [text|vision]        Type to evaluate (can be repeated)
+                                  [required]
+  --base-url TEXT                 OpenAI-compatible API endpoint  [required]
+  --model TEXT                    Model name; auto-detected if endpoint serves
+                                  one model
+  --api-key TEXT                  Bearer token for API authentication
+  --max-concurrent INTEGER        [default: 8]
+  --extra-request-params TEXT     API params as key=value,...  [default:
+                                  temperature=0,max_tokens=256,seed=42]
+  --max-samples INTEGER           If provided, limit samples per task
+  --output-path PATH              Write results.json and sample logs to this
+                                  directory
+  --log-samples                   Save per-sample results as JSONL (requires
+                                  --output-path)
+  --seed INTEGER                  Controls sample order  [default: 42]
+  -v, --verbose                   Increase verbosity (up to -vv)
+  --version                       Show the version and exit.
+  --help                          Show this message and exit.
+```
+
+### Python API
+
+```python
+import asyncio
+from nano_eval import evaluate, EvalResult
+
+result: EvalResult = asyncio.run(evaluate(
+    types=["text"],
+    base_url="http://localhost:8000/v1",
+    model="google/gemma-3-4b-it",
+    max_samples=100,
+))
+text_result = result["results"]["text"]
+print(f"Accuracy: {text_result['metrics']['exact_match']:.1%}")
+```
+
+
+This tool is inspired and borrows from: [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness). Please check it out
+
+## Example Output
+
+When using `--output-path`, a `results.json` file is generated:
+
+```json
+{
+  "config": {
+    "max_samples": 37,
+    "model": "google/gemma-3-4b-it"
+  },
+  "framework_version": "0.2.1",
+  "results": {
+    "text": {
+      "elapsed_seconds": 28.45,
+      "metrics": {
+        "exact_match": 0.7837837837837838,
+        "exact_match_stderr": 0.06861056852129647
+      },
+      "num_samples": 37,
+      "samples_hash": "12a1e9404db6afe810290a474d69cfebdaffefd0b56e48ac80e1fec0f286d659",
+      "task": "gsm8k_cot_llama",
+      "task_type": "text"
+    }
+  },
+  "total_seconds": 28.45
+}
+```

nano_eval-0.2.4/README.md

@@ -0,0 +1,100 @@
+**nano-eval** is a minimal tool for measuring the quality of a text or vision model.
+
+## Quickstart
+
+```bash
+uvx nano-eval -t text -t vision --base-url http://localhost:8000/v1 --max-samples 100
+
+# prints:
+Task    Accuracy  Samples  Duration
+------  --------  -------  --------
+text      84.3%      100       45s
+vision    71.8%      100       38s
+```
+
+> **Note:** This tool is for eyeballing the accuracy of a model. One use case is comparing accuracy between inference frameworks (e.g., vLLM vs SGLang vs MAX running the same model).
+
+## Supported Types
+
+| Type | Dataset | Description |
+|------|---------|-------------|
+| `text` | gsm8k_cot_llama | Grade school math with chain-of-thought (8-shot) |
+| `vision` | HuggingFaceM4/ChartQA | Chart question answering with images |
+
+## Usage
+
+```
+$ nano-eval --help
+Usage: nano-eval [OPTIONS]
+
+  Evaluate LLMs on standardized tasks via OpenAI-compatible APIs.
+
+  Example: nano-eval -t text --base-url http://localhost:8000/v1
+
+Options:
+  -t, --type [text|vision]        Type to evaluate (can be repeated)
+                                  [required]
+  --base-url TEXT                 OpenAI-compatible API endpoint  [required]
+  --model TEXT                    Model name; auto-detected if endpoint serves
+                                  one model
+  --api-key TEXT                  Bearer token for API authentication
+  --max-concurrent INTEGER        [default: 8]
+  --extra-request-params TEXT     API params as key=value,...  [default:
+                                  temperature=0,max_tokens=256,seed=42]
+  --max-samples INTEGER           If provided, limit samples per task
+  --output-path PATH              Write results.json and sample logs to this
+                                  directory
+  --log-samples                   Save per-sample results as JSONL (requires
+                                  --output-path)
+  --seed INTEGER                  Controls sample order  [default: 42]
+  -v, --verbose                   Increase verbosity (up to -vv)
+  --version                       Show the version and exit.
+  --help                          Show this message and exit.
+```
+
+### Python API
+
+```python
+import asyncio
+from nano_eval import evaluate, EvalResult
+
+result: EvalResult = asyncio.run(evaluate(
+    types=["text"],
+    base_url="http://localhost:8000/v1",
+    model="google/gemma-3-4b-it",
+    max_samples=100,
+))
+text_result = result["results"]["text"]
+print(f"Accuracy: {text_result['metrics']['exact_match']:.1%}")
+```
+
+
+This tool is inspired and borrows from: [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness). Please check it out
+
+## Example Output
+
+When using `--output-path`, a `results.json` file is generated:
+
+```json
+{
+  "config": {
+    "max_samples": 37,
+    "model": "google/gemma-3-4b-it"
+  },
+  "framework_version": "0.2.1",
+  "results": {
+    "text": {
+      "elapsed_seconds": 28.45,
+      "metrics": {
+        "exact_match": 0.7837837837837838,
+        "exact_match_stderr": 0.06861056852129647
+      },
+      "num_samples": 37,
+      "samples_hash": "12a1e9404db6afe810290a474d69cfebdaffefd0b56e48ac80e1fec0f286d659",
+      "task": "gsm8k_cot_llama",
+      "task_type": "text"
+    }
+  },
+  "total_seconds": 28.45
+}
+```

nano_eval-0.2.4/core.py

@@ -0,0 +1,328 @@
+"""
+Core utilities for nano-eval.
+
+Responsibilities:
+- APIConfig: endpoint, model, concurrency, timeout
+- Sample/Task: minimal task abstraction (generator + scorer)
+- complete(): async batch chat completions (OpenAI-compatible)
+- run_task(): evaluate a Task, return TaskResult
+- _normalize(): text normalization for comparison
+- _encode_image(): PIL→base64; rejects remote URLs
+"""
+
+from __future__ import annotations
+
+import asyncio
+import base64
+import hashlib
+import logging
+import math
+import re
+import time
+from collections.abc import Callable
+from contextlib import contextmanager
+from dataclasses import dataclass, field
+from io import BytesIO
+from pathlib import Path
+from typing import Any
+
+import datasets.config as ds_config
+import httpx
+from PIL import Image
+from tqdm.asyncio import tqdm_asyncio
+from typing_extensions import NotRequired, TypedDict
+
+logger = logging.getLogger(__name__)
+
+
+class Metrics(TypedDict):
+    exact_match: float
+    exact_match_stderr: float
+
+
+class LoggedSample(TypedDict):
+    sample_id: int
+    target: str
+    prompt: str
+    response: str
+    exact_match: float
+
+
+class TaskResult(TypedDict):
+    elapsed_seconds: float
+    metrics: Metrics
+    num_samples: int
+    samples: NotRequired[list[LoggedSample]]
+    samples_hash: str
+    task: str
+    task_type: str
+
+
+@dataclass(frozen=True)
+class TextPrompt:
+    """Text-only prompt (simple string or pre-formatted messages)."""
+
+    text: str | list[dict[str, str]]
+
+
+@dataclass(frozen=True)
+class VisionPrompt:
+    """Multimodal prompt with text and images."""
+
+    text: str
+    images: list[Any]
+
+
+Input = TextPrompt | VisionPrompt
+
+
+@dataclass
+class Sample:
+    """A single evaluation sample: prompt + expected target."""
+
+    prompt: Input
+    target: str
+
+
+@dataclass(frozen=True)
+class Task:
+    """Minimal task definition: a loader of samples + a scoring function."""
+
+    name: str
+    task_type: str  # "text" or "vision"
+    samples: Callable[[int | None, int | None], list[Sample]]  # (max_samples, seed)
+    score: Callable[[str, str], float]  # (response, target) -> score
+
+
+@dataclass
+class APIConfig:
+    """API configuration."""
+
+    url: str
+    model: str
+    api_key: str = ""
+    max_concurrent: int = 8
+    timeout: int = 300
+    gen_kwargs: dict[str, Any] = field(default_factory=dict)
+
+
+async def _request(
+    client: httpx.AsyncClient,
+    url: str,
+    payload: dict[str, Any],
+) -> str:
+    """Single request. Raises RuntimeError on failure."""
+    resp = await client.post(url, json=payload)
+    if resp.is_success:
+        return resp.json()["choices"][0]["message"]["content"]
+    raise RuntimeError(f"Request failed: {resp.text}")
+
+
+async def complete(
+    prompts: list[Input],
+    config: APIConfig,
+    progress_desc: str = "Running evals",
+) -> list[str]:
+    """
+    Run batch of chat completions.
+
+    Args:
+        prompts: List of prompts (TextPrompt or VisionPrompt)
+        config: API configuration (includes gen_kwargs for temperature, max_tokens, etc.)
+
+    Returns:
+        List of response strings
+    """
+    headers = {"Content-Type": "application/json"}
+    if config.api_key:
+        headers["Authorization"] = f"Bearer {config.api_key}"
+
+    async with httpx.AsyncClient(
+        limits=httpx.Limits(max_connections=config.max_concurrent),
+        timeout=httpx.Timeout(config.timeout),
+        headers=headers,
+        trust_env=True,
+    ) as client:
+        tasks: list[asyncio.Task[str]] = []
+        for prompt in prompts:
+            if isinstance(prompt, VisionPrompt):
+                messages = _build_vision_message(prompt.text, prompt.images)
+            elif isinstance(prompt.text, list):
+                messages = prompt.text
+            else:
+                messages = [{"role": "user", "content": prompt.text}]
+
+            payload: dict[str, Any] = {
+                "model": config.model,
+                "messages": messages,
+                **config.gen_kwargs,
+            }
+
+            tasks.append(asyncio.create_task(_request(client, config.url, payload)))
+
+        try:
+            return list(
+                await tqdm_asyncio.gather(*tasks, desc=progress_desc, leave=False)
+            )
+        except BaseException:
+            # On any failure (including Ctrl-C), cancel all pending tasks and await
+            # them to properly "retrieve" their exceptions. Without this, Python logs
+            # "Task exception was never retrieved" for each concurrent task that failed.
+            # Using BaseException (not Exception) ensures cleanup runs on KeyboardInterrupt.
+            for task in tasks:
+                task.cancel()
+            await asyncio.gather(*tasks, return_exceptions=True)
+            raise
+
+
+def _build_vision_message(text: str, images: list[Any]) -> list[dict[str, Any]]:
+    """Build OpenAI vision API message."""
+    content: list[dict[str, Any]] = []
+    for img in images:
+        if b64 := _encode_image(img):
+            content.append(
+                {
+                    "type": "image_url",
+                    "image_url": {"url": f"data:image/png;base64,{b64}"},
+                }
+            )
+    content.append({"type": "text", "text": text.replace("<image>", "").strip()})
+    return [{"role": "user", "content": content}]
+
+
+def _encode_image(image: Any) -> str:
+    """Encode PIL image to base64, or pass through string."""
+    if isinstance(image, str):
+        if image.startswith("http"):
+            raise ValueError("Remote image URLs are not supported.")
+        return image
+
+    if isinstance(image, Image.Image):
+        try:
+            # Convert to RGB if needed to avoid save errors with CMYK/palette modes
+            if image.mode not in ("RGB", "L"):
+                image = image.convert("RGB")
+            buf = BytesIO()
+            image.save(buf, format="PNG")
+            return base64.b64encode(buf.getvalue()).decode()
+        except Exception as e:
+            raise ValueError(f"Failed to encode image: {e}") from e
+
+    raise TypeError(f"Unsupported image type: {type(image).__name__}")
+
+
+def _normalize(text: str) -> str:
+    """Normalize text for comparison."""
+    text = re.sub(r"[$,]", "", text)
+    text = re.sub(r"(?s).*#### ", "", text)
+    text = re.sub(r"\.$", "", text)
+    return text.lower().strip()
+
+
+def _prompt_to_str(prompt: Input) -> str:
+    """Extract text from prompt (handles TextPrompt and VisionPrompt)."""
+    if isinstance(prompt, VisionPrompt):
+        return prompt.text
+    if isinstance(prompt.text, list):
+        return "\n".join(f"{m['role']}: {m['content']}" for m in prompt.text)
+    return prompt.text
+
+
+def compute_samples_hash(samples: list[Sample]) -> str:
+    """Compute SHA256 hash for all samples in a task (includes image data)."""
+    hasher = hashlib.sha256()
+    for s in samples:
+        hasher.update(_prompt_to_str(s.prompt).encode())
+        if isinstance(s.prompt, VisionPrompt):
+            for img in s.prompt.images:
+                hasher.update(_encode_image(img).encode())
+        hasher.update(s.target.encode())
+    return hasher.hexdigest()
+
+
+async def run_task(
+    task: Task,
+    config: APIConfig,
+    max_samples: int | None = None,
+    seed: int | None = None,
+) -> TaskResult:
+    """
+    Evaluate a task: collect samples, run inference, compute scores.
+
+    Args:
+        task: Task definition with samples loader and scoring function
+        config: API configuration (includes gen_kwargs for temperature, max_tokens, etc.)
+        max_samples: Optional limit on number of samples
+        seed: Optional seed for shuffling sample order
+
+    Returns:
+        TaskResult with metrics, sample count, elapsed time, and per-sample data
+    """
+    samples = task.samples(max_samples, seed)
+    samples_hash = compute_samples_hash(samples)
+    prompts = [s.prompt for s in samples]
+
+    logger.info(
+        f"Starting {task.task_type} ({task.name}) eval: "
+        f"{len(samples)} samples, up to {config.max_concurrent} concurrent requests"
+    )
+    t0 = time.perf_counter()
+    desc = "Running vision eval" if task.task_type == "vision" else "Running text eval"
+    responses = await complete(prompts, config, desc)
+    elapsed = time.perf_counter() - t0
+
+    scores = [task.score(r, s.target) for r, s in zip(responses, samples)]
+    n = len(samples)
+    accuracy = sum(scores) / n if n else 0.0
+    stderr = math.sqrt(accuracy * (1 - accuracy) / (n - 1)) if n > 1 else 0.0
+
+    logger.debug(f"{task.name}: accuracy={accuracy:.4f}±{stderr:.4f} ({elapsed:.2f}s)")
+
+    # Always collect per-sample data for optional JSONL export (negligible overhead)
+    logged_samples: list[LoggedSample] = [
+        LoggedSample(
+            sample_id=i,
+            target=s.target,
+            prompt=_prompt_to_str(s.prompt),
+            response=r,
+            exact_match=score,
+        )
+        for i, (s, r, score) in enumerate(zip(samples, responses, scores))
+    ]
+    return TaskResult(
+        elapsed_seconds=round(elapsed, 2),
+        metrics=Metrics(exact_match=accuracy, exact_match_stderr=stderr),
+        num_samples=n,
+        samples=logged_samples,
+        samples_hash=samples_hash,
+        task=task.name,
+        task_type=task.task_type,
+    )
+
+
+@contextmanager
+def offline_if_cached(dataset: str, revision: str):
+    """Context manager: enable HF offline mode if dataset is cached (avoids HEAD requests).
+
+    Yields:
+        Tuple of (cached: bool, hf_home: Path) where cached indicates if dataset
+        is in cache and hf_home is the HuggingFace cache directory.
+    """
+    from huggingface_hub.constants import HF_HOME, HF_HUB_CACHE
+
+    hub_path = (
+        Path(HF_HUB_CACHE)
+        / f"datasets--{dataset.replace('/', '--')}"
+        / "snapshots"
+        / revision
+    )
+    cached = hub_path.is_dir()
+
+    if cached:
+        old = ds_config.HF_HUB_OFFLINE
+        ds_config.HF_HUB_OFFLINE = True
+    try:
+        yield cached, Path(HF_HOME)
+    finally:
+        if cached:
+            ds_config.HF_HUB_OFFLINE = old