evalmetry 1.0.0__tar.gz

evalmetry-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Evalmetry contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

evalmetry-1.0.0/PKG-INFO

@@ -0,0 +1,94 @@
+Metadata-Version: 2.4
+Name: evalmetry
+Version: 1.0.0
+Summary: A research toolkit for measuring and analyzing model behavior.
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/minguinho26/evalmetry
+Project-URL: Repository, https://github.com/minguinho26/evalmetry
+Project-URL: Issues, https://github.com/minguinho26/evalmetry/issues
+Project-URL: Releases, https://github.com/minguinho26/evalmetry/releases
+Keywords: lm-eval,language-models,evaluation,interpretability
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: lm-eval[hf]==0.4.13
+Requires-Dist: torch>=2.1
+Requires-Dist: transformers==5.16.1
+Requires-Dist: pyarrow>=14
+Requires-Dist: pandas>=2.0
+Requires-Dist: safetensors>=0.4
+Requires-Dist: matplotlib>=3.7
+Requires-Dist: sentencepiece
+Requires-Dist: protobuf
+Requires-Dist: pyyaml
+Requires-Dist: tqdm
+Provides-Extra: pinned-latest
+Requires-Dist: lm-eval==0.4.13; extra == "pinned-latest"
+Requires-Dist: transformers==5.16.1; extra == "pinned-latest"
+Requires-Dist: accelerate; extra == "pinned-latest"
+Requires-Dist: pyarrow==25.0.1; extra == "pinned-latest"
+Requires-Dist: pandas==3.0.5; extra == "pinned-latest"
+Requires-Dist: safetensors==0.8.0; extra == "pinned-latest"
+Requires-Dist: matplotlib==3.11.1; extra == "pinned-latest"
+Requires-Dist: sentencepiece; extra == "pinned-latest"
+Requires-Dist: protobuf; extra == "pinned-latest"
+Dynamic: license-file
+
+# Evalmetry
+
+A research toolkit for measuring and analyzing model behavior.
+
+Evalmetry evaluates Hugging Face language models with lm-eval and collects internal signals from the same forward passes used for scoring.
+
+## Features
+
+- Benchmark scores with per-layer logit lens and layer similarity.
+- Optional attention, hidden states, custom hooks and module statistics.
+- Custom benchmarks, LLM judge scoring and modified-model adapters.
+- Resumable runs, saved-data readers and comparison reports.
+
+Model evaluation supports one process on one CUDA GPU. CPU execution is for test fixtures and verification tools. Reports and saved-data readers do not require a GPU.
+
+## Install
+
+Install from PyPI in your Python environment:
+
+```bash
+pip install evalmetry
+```
+
+Evalmetry requires Python >=3.10. Use a compatible CUDA build of PyTorch. The Python import and CLI are both `evalmetry`.
+
+To install this release explicitly, use `pip install evalmetry==1.0.0`.
+For development, clone this repository and run `python -m pip install -e .`.
+
+## Quick start
+
+Use `run` to evaluate eight documents and save the default signals:
+
+```bash
+evalmetry run --model-args pretrained=Qwen/Qwen3-0.6B,dtype=bfloat16,device=cuda \
+    --tasks arc_easy --num-fewshot 0 --limit 8 --batch-size 1 \
+    --output results/quickstart
+```
+
+Use `report` to generate a report from that run:
+
+```bash
+evalmetry report results/quickstart --output report/quickstart
+```
+
+The first run downloads the model and dataset if needed. Reusing a run directory resumes its recorded configuration; use a different directory for a different experiment.
+
+Other commands: `collect-research-data` adds optional tensors to a completed run, `debug` reads a saved module trace, and `module-stats` reads saved statistics. Traces and statistics must be enabled during collection. Use `evalmetry <command> --help` for options.
+
+## Reading results
+
+Evaluation results, manifests and collected signals are written under the selected output directory. Read saved signals with `evalmetry.load_signals(run_dir)`; `evalmetry.describe_schema()` describes the columns.
+
+## License
+
+Evalmetry is released under the MIT License.

evalmetry-1.0.0/README.md

@@ -0,0 +1,55 @@
+# Evalmetry
+
+A research toolkit for measuring and analyzing model behavior.
+
+Evalmetry evaluates Hugging Face language models with lm-eval and collects internal signals from the same forward passes used for scoring.
+
+## Features
+
+- Benchmark scores with per-layer logit lens and layer similarity.
+- Optional attention, hidden states, custom hooks and module statistics.
+- Custom benchmarks, LLM judge scoring and modified-model adapters.
+- Resumable runs, saved-data readers and comparison reports.
+
+Model evaluation supports one process on one CUDA GPU. CPU execution is for test fixtures and verification tools. Reports and saved-data readers do not require a GPU.
+
+## Install
+
+Install from PyPI in your Python environment:
+
+```bash
+pip install evalmetry
+```
+
+Evalmetry requires Python >=3.10. Use a compatible CUDA build of PyTorch. The Python import and CLI are both `evalmetry`.
+
+To install this release explicitly, use `pip install evalmetry==1.0.0`.
+For development, clone this repository and run `python -m pip install -e .`.
+
+## Quick start
+
+Use `run` to evaluate eight documents and save the default signals:
+
+```bash
+evalmetry run --model-args pretrained=Qwen/Qwen3-0.6B,dtype=bfloat16,device=cuda \
+    --tasks arc_easy --num-fewshot 0 --limit 8 --batch-size 1 \
+    --output results/quickstart
+```
+
+Use `report` to generate a report from that run:
+
+```bash
+evalmetry report results/quickstart --output report/quickstart
+```
+
+The first run downloads the model and dataset if needed. Reusing a run directory resumes its recorded configuration; use a different directory for a different experiment.
+
+Other commands: `collect-research-data` adds optional tensors to a completed run, `debug` reads a saved module trace, and `module-stats` reads saved statistics. Traces and statistics must be enabled during collection. Use `evalmetry <command> --help` for options.
+
+## Reading results
+
+Evaluation results, manifests and collected signals are written under the selected output directory. Read saved signals with `evalmetry.load_signals(run_dir)`; `evalmetry.describe_schema()` describes the columns.
+
+## License
+
+Evalmetry is released under the MIT License.

evalmetry-1.0.0/evalmetry/__init__.py

@@ -0,0 +1,27 @@
+"""Per-layer internal signal collection alongside an lm-eval benchmark run.
+
+Scoring is lm-eval's, unchanged.
+This package only pulls extra research signals out of the same forward passes and writes them in a fixed, self-describing format.
+
+The reading entry point is `load_signals`; `describe_schema` prints what the columns mean.
+"""
+
+from .storage import SCHEMA_VERSION, describe_schema, load_signals, read_manifest, read_sample_metrics
+
+__all__ = ["SCHEMA_VERSION", "describe_schema", "load_signals", "read_manifest", "read_sample_metrics"]
+
+from .hooks import HookSpec, HookContext
+
+__all__ += ["HookSpec", "HookContext"]
+
+from .models import ModelBundle, fingerprint_files
+
+__all__ += ["ModelBundle", "fingerprint_files", "RunConfig", "run"]
+
+
+def __getattr__(name):
+    """Load evaluation entrypoints lazily so `python -m evalmetry.main` works."""
+    if name in {"RunConfig", "run"}:
+        from .main import RunConfig, cmd_run
+        return {"RunConfig": RunConfig, "run": cmd_run}[name]
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")