evalmetry 1.0.0__tar.gz → 1.1.1__tar.gz

{evalmetry-1.0.0 → evalmetry-1.1.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: evalmetry
-Version: 1.0.0
+Version: 1.1.1
 Summary: A research toolkit for measuring and analyzing model behavior.
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/minguinho26/evalmetry
@@ -52,9 +52,13 @@ Evalmetry evaluates Hugging Face language models with lm-eval and collects inter
 
 Model evaluation supports one process on one CUDA GPU. CPU execution is for test fixtures and verification tools. Reports and saved-data readers do not require a GPU.
 
+## Evidence and research example
+
+Read the [1.1.0 verification report](https://github.com/minguinho26/evalmetry/blob/v1.1.1/docs/verification.md) and the [prediction-depth research showcase](https://github.com/minguinho26/evalmetry/blob/v1.1.1/docs/showcase.md). Generation parity is established at the same actual batch size. Generation is not batch-invariant.
+
 ## Install
 
-Install from PyPI in your Python environment:
+Install from PyPI:
 
 ```bash
 pip install evalmetry
@@ -62,8 +66,7 @@ pip install evalmetry
 
 Evalmetry requires Python >=3.10. Use a compatible CUDA build of PyTorch. The Python import and CLI are both `evalmetry`.
 
-To install this release explicitly, use `pip install evalmetry==1.0.0`.
-For development, clone this repository and run `python -m pip install -e .`.
+To install this release explicitly, use `pip install evalmetry==1.1.1`. For development, clone this repository and run `python -m pip install -e .`.
 
 ## Quick start
 
@@ -81,14 +84,16 @@ Use `report` to generate a report from that run:
 evalmetry report results/quickstart --output report/quickstart
 ```
 
-The first run downloads the model and dataset if needed. Reusing a run directory resumes its recorded configuration; use a different directory for a different experiment.
+The first run downloads the model and dataset if needed. Reusing a run directory resumes its recorded configuration. A different experiment needs a different directory.
 
 Other commands: `collect-research-data` adds optional tensors to a completed run, `debug` reads a saved module trace, and `module-stats` reads saved statistics. Traces and statistics must be enabled during collection. Use `evalmetry <command> --help` for options.
 
 ## Reading results
 
-Evaluation results, manifests and collected signals are written under the selected output directory. Read saved signals with `evalmetry.load_signals(run_dir)`; `evalmetry.describe_schema()` describes the columns.
+Evaluation results, manifests and collected signals are written under the selected output directory. Read saved signals with `evalmetry.load_signals(run_dir)`. `evalmetry.describe_schema()` describes the columns.
 
 ## License
 
 Evalmetry is released under the MIT License.
+
+This project is built on top of [EleutherAI's lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness).

{evalmetry-1.0.0 → evalmetry-1.1.1}/README.md

@@ -13,9 +13,13 @@ Evalmetry evaluates Hugging Face language models with lm-eval and collects inter
 
 Model evaluation supports one process on one CUDA GPU. CPU execution is for test fixtures and verification tools. Reports and saved-data readers do not require a GPU.
 
+## Evidence and research example
+
+Read the [1.1.0 verification report](https://github.com/minguinho26/evalmetry/blob/v1.1.1/docs/verification.md) and the [prediction-depth research showcase](https://github.com/minguinho26/evalmetry/blob/v1.1.1/docs/showcase.md). Generation parity is established at the same actual batch size. Generation is not batch-invariant.
+
 ## Install
 
-Install from PyPI in your Python environment:
+Install from PyPI:
 
 ```bash
 pip install evalmetry
@@ -23,8 +27,7 @@ pip install evalmetry
 
 Evalmetry requires Python >=3.10. Use a compatible CUDA build of PyTorch. The Python import and CLI are both `evalmetry`.
 
-To install this release explicitly, use `pip install evalmetry==1.0.0`.
-For development, clone this repository and run `python -m pip install -e .`.
+To install this release explicitly, use `pip install evalmetry==1.1.1`. For development, clone this repository and run `python -m pip install -e .`.
 
 ## Quick start
 
@@ -42,14 +45,16 @@ Use `report` to generate a report from that run:
 evalmetry report results/quickstart --output report/quickstart
 ```
 
-The first run downloads the model and dataset if needed. Reusing a run directory resumes its recorded configuration; use a different directory for a different experiment.
+The first run downloads the model and dataset if needed. Reusing a run directory resumes its recorded configuration. A different experiment needs a different directory.
 
 Other commands: `collect-research-data` adds optional tensors to a completed run, `debug` reads a saved module trace, and `module-stats` reads saved statistics. Traces and statistics must be enabled during collection. Use `evalmetry <command> --help` for options.
 
 ## Reading results
 
-Evaluation results, manifests and collected signals are written under the selected output directory. Read saved signals with `evalmetry.load_signals(run_dir)`; `evalmetry.describe_schema()` describes the columns.
+Evaluation results, manifests and collected signals are written under the selected output directory. Read saved signals with `evalmetry.load_signals(run_dir)`. `evalmetry.describe_schema()` describes the columns.
 
 ## License
 
 Evalmetry is released under the MIT License.
+
+This project is built on top of [EleutherAI's lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness).

{evalmetry-1.0.0 → evalmetry-1.1.1}/evalmetry/backend.py

@@ -30,6 +30,7 @@ Because that one is a copy, `check_upstream_source()` guards it: CI fails when l
 from __future__ import annotations
 
 import ast
+import contextlib
 import hashlib
 import inspect
 import os
@@ -579,7 +580,7 @@ Without this the backend behaves exactly like `HFLM`.
         from tqdm import tqdm
 
         results: list[str] = []
-        with self._recorder.session():
+        with self._recorder.session(), self._resolved_auto_batch_size():
             for instance in tqdm(requests, disable=disable_tqdm,
                                  desc="Running generate_until requests (traced)"):
                 self._recorder.expect_generation(
@@ -596,6 +597,34 @@ Without this the backend behaves exactly like `HFLM`.
                 results.extend(output)
         return results
 
+    @contextlib.contextmanager
+    def _resolved_auto_batch_size(self):
+        """Pin `batch_size="auto"` to one probed value for the duration of a generate loop.
+
+        The loop above calls `super().generate_until()` once per document, and lm-eval re-runs
+        `_detect_batch_size()` on every entry to that method while `batch_size` reads "auto".
+        The probed value is then discarded, because a one-instance call batches one instance
+        either way - so the probe is pure overhead, paid once per document. Measured on
+        SmolLM2-135M over 10 gsm8k documents at bfloat16 on an RTX A5000: the traced pass takes
+        8:07 with the probe repeated and 0:16 with it resolved once, against 0:17 for the same
+        pass at an explicit batch of 1. All three produce identical generations.
+
+        The probe is kept rather than forced to 1: it still decides what the scoring pass of a
+        mixed run may use, and skipping it would change behaviour rather than only its cost.
+
+        `HFLM.batch_size` is a read-only property over `batch_size_per_gpu`, so the pin is
+        written there and restored afterwards, leaving the model as this found it.
+        """
+        if self.batch_size != "auto":
+            yield
+            return
+        original = self.batch_size_per_gpu
+        self.batch_size_per_gpu = self._detect_batch_size()
+        try:
+            yield
+        finally:
+            self.batch_size_per_gpu = original
+
     def _model_generate(self, context, max_length, stop, **generation_kwargs):
         """Note the prompt length, then keep the token ids that came out.
 

{evalmetry-1.0.0 → evalmetry-1.1.1}/evalmetry/debug.py

@@ -3,7 +3,7 @@
 This is a debugging aid, off by default, and deliberately separate from `recorder.py`.
 The recorder exists to capture *signals* and is driven by what the reducers ask for; this exists to capture *what happened*, must keep working when the recorder is not recording (a batch-size probe, a document already on disk), and must survive the process that produced it.
 
-CLI options are available through `evalmetry debug --help`; additional review tasks are in REVIEW_CPU.md and REVIEW_GPU.md.
+CLI options are available through `evalmetry debug --help`.
 Three things are worth repeating here, because they are what the code looks strange without.
 
 **`always_call=True` is not used, on purpose.**

{evalmetry-1.0.0 → evalmetry-1.1.1}/evalmetry/main.py

@@ -25,6 +25,7 @@ import re
 import sys
 import dataclasses
 import functools
+import time
 from contextlib import contextmanager
 from dataclasses import dataclass
 from datetime import date, datetime, timezone
@@ -35,7 +36,7 @@ from .debug import TRACE_BUFFER_EVENTS
 from .storage import FIXED_SETTINGS, SAMPLING_SEED, SCHEMA_VERSION
 
 #: Version of this tool, recorded in every manifest.
-TOOL_VERSION = "0.1.0"
+TOOL_VERSION = "1.1.1"
 
 #: Default number of documents collected per correctness group.
 DEFAULT_COLLECT_LIMIT = 500
@@ -142,6 +143,7 @@ class RunConfig:
         tasks: task names.
         include_path: directories containing local benchmark YAML, JSONL and scoring code.
             Paths inside YAML resolve relative to that YAML. See examples/custom_benchmarks.
+        num_fewshot: how many examples precede each document. It has no default because there is no value that is right for every task: lm-eval gives each task its own, and silently substituting one here would score gsm8k at 0-shot while the task itself asks for 5. A measurement has to say what it measured.
         save_attention / save_hidden: opt-in signals, both collected in a second pass because correctness is not known during the first one.
         debug: module tracing, off by default.
             Absent from `identity` and `config_hash`: it changes how long a run takes, and with `sync` or `stop_on_nonfinite` whether it finishes, but not what the signals mean.
@@ -149,7 +151,7 @@ class RunConfig:
 
     model_args: str
     tasks: list[str]
-    num_fewshot: int = 0
+    num_fewshot: int
     limit: int | None = None
     batch_size: int | str = 1
     adapter: str | None = None
@@ -182,7 +184,7 @@ class RunConfig:
         """The lm-eval model argument string parsed into a dict.
 
         Example:
-            >>> RunConfig("pretrained=Qwen/Qwen3-8B,dtype=bfloat16", []).model_kwargs()
+            >>> RunConfig("pretrained=Qwen/Qwen3-8B,dtype=bfloat16", [], num_fewshot=0).model_kwargs()
             {'pretrained': 'Qwen/Qwen3-8B', 'dtype': 'bfloat16'}
         """
         parsed: dict[str, str] = {}
@@ -261,7 +263,7 @@ class RunConfig:
         Only a convenience for browsing; `report` reads the manifest and never parses this path.
 
         Example:
-            >>> RunConfig("pretrained=Qwen/Qwen3-8B", ["xnli_ko"]).default_output("ab12cd34")
+            >>> RunConfig("pretrained=Qwen/Qwen3-8B", ["xnli_ko"], num_fewshot=0).default_output("ab12cd34")
             'results/xnli_ko/Qwen__Qwen3-8B/2026-01-31-ab12cd34'   # date varies
         """
         task = "+".join(sorted(self.tasks)) or "unknown-task"
@@ -672,6 +674,60 @@ def _resolved_revision(lm: Any) -> str:
 # --------------------------------------------------------------------------
 
 
+class PhaseTimer:
+    """How long each phase of a run took, recorded on every run rather than on request.
+
+    Not behind a flag. A run that cannot say how long its evaluation took cannot be
+    compared against another one, and reading a clock costs nothing beside the work it
+    measures - so the choice to record it is not the user's to make and not theirs to
+    forget. The numbers land in the manifest under `timings`.
+
+    CUDA queues kernels asynchronously, so a naive `perf_counter` pair reports how long
+    the work took to *launch*. Every boundary therefore synchronises first. That is done
+    at phase boundaries only, where a handful of syncs disappear into minutes of work;
+    `--debug-sync` is the per-module variant and exists to attribute time inside a trace,
+    not to measure a run.
+
+    Phases do not have to cover the whole run and may be absent: a run without a
+    collection pass simply has no `collection` key. `end_to_end` always spans the object's
+    lifetime, so it is not the sum of the others.
+    """
+
+    def __init__(self) -> None:
+        self._phases: dict[str, float] = {}
+        self._started = time.perf_counter()
+
+    @staticmethod
+    def _synchronize() -> None:
+        """Wait for queued CUDA work, when there is any to wait for.
+
+        Skipped unless a context already exists, so timing a CPU run does not initialise
+        CUDA just to measure nothing. Without torch at all there is no device work in
+        flight, so there is nothing to wait for either.
+        """
+        try:
+            import torch
+        except ImportError:
+            return
+        if torch.cuda.is_available() and torch.cuda.is_initialized():
+            torch.cuda.synchronize()
+
+    @contextmanager
+    def phase(self, name: str):
+        """Time one phase, recording it even if the phase raises."""
+        self._synchronize()
+        started = time.perf_counter()
+        try:
+            yield
+        finally:
+            self._synchronize()
+            self._phases[name] = round(time.perf_counter() - started, 3)
+
+    def finish(self) -> dict[str, float]:
+        self._synchronize()
+        return {**self._phases, "end_to_end": round(time.perf_counter() - self._started, 3)}
+
+
 @contextmanager
 def _tracing(config: RunConfig, model: Any, run_dir: str, name: str = "trace"):
     """Hook the model for the duration, or do nothing at all.
@@ -795,6 +851,7 @@ def cmd_run(config: RunConfig) -> str:
         The run directory.
     """
     validate_single_gpu_execution(config)
+    timer = PhaseTimer()
 
     import lm_eval
 
@@ -811,7 +868,8 @@ def cmd_run(config: RunConfig) -> str:
     unknown = set(config.signals) - {"logit_lens", "similarity"}
     if unknown:
         raise ValueError(f"unknown signals: {sorted(unknown)}")
-    lm = load_model(config)
+    with timer.phase("model_load"):
+        lm = load_model(config)
     adapter = resolve_model_adapter(config, lm)
     environment = check_environment(lm.model)
 
@@ -866,9 +924,10 @@ def cmd_run(config: RunConfig) -> str:
         lm.attach_recorder(recorder)
 
     # 3. 평가와 judge 채점을 완료하고 문서별 결과를 저장한다.
-    results, samples = _evaluate_and_save_samples(
-        config, lm, writer, run_dir, task_manager, evaluation_tasks
-    )
+    with timer.phase("evaluate"):
+        results, samples = _evaluate_and_save_samples(
+            config, lm, writer, run_dir, task_manager, evaluation_tasks
+        )
 
     # Read before the collection pass, which may switch the model to eager attention: the manifest should say what the scored pass actually ran with.
     attn_implementation = getattr(lm.model.config, "_attn_implementation", "unknown")
@@ -888,7 +947,7 @@ def cmd_run(config: RunConfig) -> str:
     if config.save_attention or config.save_hidden or any(h.pass_name == "collection" for h in config.resolved_hooks):
         # Its own trace: the collection pass forces eager attention and materialises a
         # (heads, seq, seq) map per block, which is the heaviest thing this tool does.
-        with _tracing(config, lm.model, run_dir, name="collection"):
+        with timer.phase("collection"), _tracing(config, lm.model, run_dir, name="collection"):
             collection_counts = _collect_research_data(
                 config, lm, adapter, writer, run_dir, hidden_layers
             )
@@ -920,6 +979,7 @@ def cmd_run(config: RunConfig) -> str:
             "evaluation_completed": True,
             "generation_kwargs_source": "lm-eval task config",
             "debug": config.debug.manifest_entry(),
+            "timings": timer.finish(),
         },
     )
     storage.write_results(run_dir, manifest, results.get("results", {}), writer.signal_files())
@@ -1103,6 +1163,10 @@ def _collection_config_from_manifest(
         model_config=saved_model.get("config", {}),
         signals=(),
         tasks=list(manifest["tasks"]),
+        # Carried from the manifest rather than defaulted: collection replays the prompts the
+        # evaluation stored, so this does not build them, but a config that disagreed with the
+        # run it describes would be a trap for anyone reading it back.
+        num_fewshot=manifest["num_fewshot"],
         batch_size=1,                      # collection is always unbatched
         adapter=saved_model.get("adapter") if saved_model else manifest.get("model_type"),
         save_attention=args.save_attention,
@@ -1121,11 +1185,13 @@ def cmd_collect_research_data(args: argparse.Namespace) -> str:
 
     Model and tokenizer settings come from the run's manifest, never from the command line: the two passes have to produce identical input tokens, so letting them be re-specified would be a way to get that wrong.
     """
+    timer = PhaseTimer()
     run_dir = args.run_dir
     manifest = storage.read_manifest(run_dir)
     config = _collection_config_from_manifest(args, manifest)
     saved_model = manifest.get("custom_model", {})
-    lm = load_model(config)
+    with timer.phase("model_load"):
+        lm = load_model(config)
     if saved_model and config.model_provenance != saved_model:
         raise ValueError("custom model implementation/config/checkpoint changed since evaluation")
     from .models import resolve_model_adapter
@@ -1137,7 +1203,7 @@ def cmd_collect_research_data(args: argparse.Namespace) -> str:
         else None
     )
     writer = storage.RunWriter(run_dir)
-    with _tracing(config, lm.model, run_dir, name="collection"):
+    with timer.phase("collection"), _tracing(config, lm.model, run_dir, name="collection"):
         counts = _collect_research_data(
             config, lm, adapter, writer, run_dir, hidden_layers,
             write_steps=not manifest.get("internal_signals_available", True),
@@ -1156,6 +1222,7 @@ def cmd_collect_research_data(args: argparse.Namespace) -> str:
     storage.write_results(run_dir, payload["manifest"], payload["results"], writer.signal_files())
     collection_metadata = {
         "collection_counts": counts,
+        "collection_timings": timer.finish(),
         "collection_attn_implementation": getattr(
             lm.model.config, "_attn_implementation", "unknown"),
         "options": options,
@@ -1328,7 +1395,10 @@ def build_parser() -> argparse.ArgumentParser:
     run.add_argument("--tasks", required=True, help="comma separated lm-eval task names")
     run.add_argument("--include-path", action="append", default=[],
                      help="local benchmark directory; repeat for multiple bundles")
-    run.add_argument("--num-fewshot", type=int, default=0)
+    run.add_argument("--num-fewshot", type=int, required=True,
+                     help="examples before each document. Required: lm-eval gives each "
+                          "task its own default and this overrides it for every task in "
+                          "the run, so the run has to state which number it used")
     run.add_argument("--limit", type=int, default=None, help="cap on the number of documents")
     run.add_argument("--batch-size", type=parse_batch_size, default=1,
                      help='positive integer, or "auto" to let lm-eval find the '

{evalmetry-1.0.0 → evalmetry-1.1.1}/evalmetry/module_stats.py

@@ -985,8 +985,19 @@ def resolve_database(path: str, pass_name: str = "trace") -> Path:
         return given
     if (given / "statistics.sqlite").is_file():
         return given / "statistics.sqlite"
-    found = sorted((given / "debug" / "module_stats" / pass_name).glob("*/statistics.sqlite"))
+    root = given / "debug" / "module_stats"
+    found = sorted((root / pass_name).glob("*/statistics.sqlite"))
     if not found:
+        # "nothing was recorded" and "recorded, but not this pass" need different advice.
+        # A run with --module-stats but no collection pass is not misconfigured, so telling
+        # its owner to re-run with --module-stats sends them to change a flag already set.
+        recorded = sorted(directory.name for directory in root.glob("*")
+                          if any(directory.glob("*/statistics.sqlite")))
+        if recorded:
+            raise FileNotFoundError(
+                f"no {pass_name} module statistics under {path}; this run recorded: "
+                f"{', '.join(recorded)}. The collection pass is written only when the run "
+                f"also collects (--save-attention, --save-hidden or a collection hook).")
         raise FileNotFoundError(f"no module statistics under {path}; run with --module-stats")
     return found[-1]
 

{evalmetry-1.0.0 → evalmetry-1.1.1}/evalmetry.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: evalmetry
-Version: 1.0.0
+Version: 1.1.1
 Summary: A research toolkit for measuring and analyzing model behavior.
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/minguinho26/evalmetry
@@ -52,9 +52,13 @@ Evalmetry evaluates Hugging Face language models with lm-eval and collects inter
 
 Model evaluation supports one process on one CUDA GPU. CPU execution is for test fixtures and verification tools. Reports and saved-data readers do not require a GPU.
 
+## Evidence and research example
+
+Read the [1.1.0 verification report](https://github.com/minguinho26/evalmetry/blob/v1.1.1/docs/verification.md) and the [prediction-depth research showcase](https://github.com/minguinho26/evalmetry/blob/v1.1.1/docs/showcase.md). Generation parity is established at the same actual batch size. Generation is not batch-invariant.
+
 ## Install
 
-Install from PyPI in your Python environment:
+Install from PyPI:
 
 ```bash
 pip install evalmetry
@@ -62,8 +66,7 @@ pip install evalmetry
 
 Evalmetry requires Python >=3.10. Use a compatible CUDA build of PyTorch. The Python import and CLI are both `evalmetry`.
 
-To install this release explicitly, use `pip install evalmetry==1.0.0`.
-For development, clone this repository and run `python -m pip install -e .`.
+To install this release explicitly, use `pip install evalmetry==1.1.1`. For development, clone this repository and run `python -m pip install -e .`.
 
 ## Quick start
 
@@ -81,14 +84,16 @@ Use `report` to generate a report from that run:
 evalmetry report results/quickstart --output report/quickstart
 ```
 
-The first run downloads the model and dataset if needed. Reusing a run directory resumes its recorded configuration; use a different directory for a different experiment.
+The first run downloads the model and dataset if needed. Reusing a run directory resumes its recorded configuration. A different experiment needs a different directory.
 
 Other commands: `collect-research-data` adds optional tensors to a completed run, `debug` reads a saved module trace, and `module-stats` reads saved statistics. Traces and statistics must be enabled during collection. Use `evalmetry <command> --help` for options.
 
 ## Reading results
 
-Evaluation results, manifests and collected signals are written under the selected output directory. Read saved signals with `evalmetry.load_signals(run_dir)`; `evalmetry.describe_schema()` describes the columns.
+Evaluation results, manifests and collected signals are written under the selected output directory. Read saved signals with `evalmetry.load_signals(run_dir)`. `evalmetry.describe_schema()` describes the columns.
 
 ## License
 
 Evalmetry is released under the MIT License.
+
+This project is built on top of [EleutherAI's lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness).

{evalmetry-1.0.0 → evalmetry-1.1.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "evalmetry"
-version = "1.0.0"
+version = "1.1.1"
 description = "A research toolkit for measuring and analyzing model behavior."
 readme = "README.md"
 license = "MIT"