eval-framework 0.2.0__py3-none-any.whl

eval_framework-0.2.0.dist-info/METADATA

@@ -0,0 +1,514 @@
+Metadata-Version: 2.3
+Name: eval-framework
+Version: 0.2.0
+Summary: Evalulation Framework
+Author: Aleph Alpha Research
+License:                                  Apache License
+                                    Version 2.0, January 2004
+                                 http://www.apache.org/licenses/
+         
+            TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+         
+            1. Definitions.
+         
+               "License" shall mean the terms and conditions for use, reproduction,
+               and distribution as defined by Sections 1 through 9 of this document.
+         
+               "Licensor" shall mean the copyright owner or entity authorized by
+               the copyright owner that is granting the License.
+         
+               "Legal Entity" shall mean the union of the acting entity and all
+               other entities that control, are controlled by, or are under common
+               control with that entity. For the purposes of this definition,
+               "control" means (i) the power, direct or indirect, to cause the
+               direction or management of such entity, whether by contract or
+               otherwise, or (ii) ownership of fifty percent (50%) or more of the
+               outstanding shares, or (iii) beneficial ownership of such entity.
+         
+               "You" (or "Your") shall mean an individual or Legal Entity
+               exercising permissions granted by this License.
+         
+               "Source" form shall mean the preferred form for making modifications,
+               including but not limited to software source code, documentation
+               source, and configuration files.
+         
+               "Object" form shall mean any form resulting from mechanical
+               transformation or translation of a Source form, including but
+               not limited to compiled object code, generated documentation,
+               and conversions to other media types.
+         
+               "Work" shall mean the work of authorship, whether in Source or
+               Object form, made available under the License, as indicated by a
+               copyright notice that is included in or attached to the work
+               (an example is provided in the Appendix below).
+         
+               "Derivative Works" shall mean any work, whether in Source or Object
+               form, that is based on (or derived from) the Work and for which the
+               editorial revisions, annotations, elaborations, or other modifications
+               represent, as a whole, an original work of authorship. For the purposes
+               of this License, Derivative Works shall not include works that remain
+               separable from, or merely link (or bind by name) to the interfaces of,
+               the Work and Derivative Works thereof.
+         
+               "Contribution" shall mean any work of authorship, including
+               the original version of the Work and any modifications or additions
+               to that Work or Derivative Works thereof, that is intentionally
+               submitted to Licensor for inclusion in the Work by the copyright owner
+               or by an individual or Legal Entity authorized to submit on behalf of
+               the copyright owner. For the purposes of this definition, "submitted"
+               means any form of electronic, verbal, or written communication sent
+               to the Licensor or its representatives, including but not limited to
+               communication on electronic mailing lists, source code control systems,
+               and issue tracking systems that are managed by, or on behalf of, the
+               Licensor for the purpose of discussing and improving the Work, but
+               excluding communication that is conspicuously marked or otherwise
+               designated in writing by the copyright owner as "Not a Contribution."
+         
+               "Contributor" shall mean Licensor and any individual or Legal Entity
+               on behalf of whom a Contribution has been received by Licensor and
+               subsequently incorporated within the Work.
+         
+            2. Grant of Copyright License. Subject to the terms and conditions of
+               this License, each Contributor hereby grants to You a perpetual,
+               worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+               copyright license to reproduce, prepare Derivative Works of,
+               publicly display, publicly perform, sublicense, and distribute the
+               Work and such Derivative Works in Source or Object form.
+         
+            3. Grant of Patent License. Subject to the terms and conditions of
+               this License, each Contributor hereby grants to You a perpetual,
+               worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+               (except as stated in this section) patent license to make, have made,
+               use, offer to sell, sell, import, and otherwise transfer the Work,
+               where such license applies only to those patent claims licensable
+               by such Contributor that are necessarily infringed by their
+               Contribution(s) alone or by combination of their Contribution(s)
+               with the Work to which such Contribution(s) was submitted. If You
+               institute patent litigation against any entity (including a
+               cross-claim or counterclaim in a lawsuit) alleging that the Work
+               or a Contribution incorporated within the Work constitutes direct
+               or contributory patent infringement, then any patent licenses
+               granted to You under this License for that Work shall terminate
+               as of the date such litigation is filed.
+         
+            4. Redistribution. You may reproduce and distribute copies of the
+               Work or Derivative Works thereof in any medium, with or without
+               modifications, and in Source or Object form, provided that You
+               meet the following conditions:
+         
+               (a) You must give any other recipients of the Work or
+                   Derivative Works a copy of this License; and
+         
+               (b) You must cause any modified files to carry prominent notices
+                   stating that You changed the files; and
+         
+               (c) You must retain, in the Source form of any Derivative Works
+                   that You distribute, all copyright, patent, trademark, and
+                   attribution notices from the Source form of the Work,
+                   excluding those notices that do not pertain to any part of
+                   the Derivative Works; and
+         
+               (d) If the Work includes a "NOTICE" text file as part of its
+                   distribution, then any Derivative Works that You distribute must
+                   include a readable copy of the attribution notices contained
+                   within such NOTICE file, excluding those notices that do not
+                   pertain to any part of the Derivative Works, in at least one
+                   of the following places: within a NOTICE text file distributed
+                   as part of the Derivative Works; within the Source form or
+                   documentation, if provided along with the Derivative Works; or,
+                   within a display generated by the Derivative Works, if and
+                   wherever such third-party notices normally appear. The contents
+                   of the NOTICE file are for informational purposes only and
+                   do not modify the License. You may add Your own attribution
+                   notices within Derivative Works that You distribute, alongside
+                   or as an addendum to the NOTICE text from the Work, provided
+                   that such additional attribution notices cannot be construed
+                   as modifying the License.
+         
+               You may add Your own copyright statement to Your modifications and
+               may provide additional or different license terms and conditions
+               for use, reproduction, or distribution of Your modifications, or
+               for any such Derivative Works as a whole, provided Your use,
+               reproduction, and distribution of the Work otherwise complies with
+               the conditions stated in this License.
+         
+            5. Submission of Contributions. Unless You explicitly state otherwise,
+               any Contribution intentionally submitted for inclusion in the Work
+               by You to the Licensor shall be under the terms and conditions of
+               this License, without any additional terms or conditions.
+               Notwithstanding the above, nothing herein shall supersede or modify
+               the terms of any separate license agreement you may have executed
+               with Licensor regarding such Contributions.
+         
+            6. Trademarks. This License does not grant permission to use the trade
+               names, trademarks, service marks, or product names of the Licensor,
+               except as required for reasonable and customary use in describing the
+               origin of the Work and reproducing the content of the NOTICE file.
+         
+            7. Disclaimer of Warranty. Unless required by applicable law or
+               agreed to in writing, Licensor provides the Work (and each
+               Contributor provides its Contributions) on an "AS IS" BASIS,
+               WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+               implied, including, without limitation, any warranties or conditions
+               of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+               PARTICULAR PURPOSE. You are solely responsible for determining the
+               appropriateness of using or redistributing the Work and assume any
+               risks associated with Your exercise of permissions under this License.
+         
+            8. Limitation of Liability. In no event and under no legal theory,
+               whether in tort (including negligence), contract, or otherwise,
+               unless required by applicable law (such as deliberate and grossly
+               negligent acts) or agreed to in writing, shall any Contributor be
+               liable to You for damages, including any direct, indirect, special,
+               incidental, or consequential damages of any character arising as a
+               result of this License or out of the use or inability to use the
+               Work (including but not limited to damages for loss of goodwill,
+               work stoppage, computer failure or malfunction, or any and all
+               other commercial damages or losses), even if such Contributor
+               has been advised of the possibility of such damages.
+         
+            9. Accepting Warranty or Additional Liability. While redistributing
+               the Work or Derivative Works thereof, You may choose to offer,
+               and charge a fee for, acceptance of support, warranty, indemnity,
+               or other liability obligations and/or rights consistent with this
+               License. However, in accepting such obligations, You may act only
+               on Your own behalf and on Your sole responsibility, not on behalf
+               of any other Contributor, and only if You agree to indemnify,
+               defend, and hold each Contributor harmless for any liability
+               incurred by, or claims asserted against, such Contributor by reason
+               of your accepting any such warranty or additional liability.
+         
+            END OF TERMS AND CONDITIONS
+         
+            APPENDIX: How to apply the Apache License to your work.
+         
+               To apply the Apache License to your work, attach the following
+               boilerplate notice, with the fields enclosed by brackets "[]"
+               replaced with your own identifying information. (Don't include
+               the brackets!)  The text should be enclosed in the appropriate
+               comment syntax for the file format. We also recommend that a
+               file or class name and description of purpose be included on the
+               same "printed page" as the copyright notice for easier
+               identification within third-party archives.
+         
+            Copyright [yyyy] [name of copyright owner]
+         
+            Licensed under the Apache License, Version 2.0 (the "License");
+            you may not use this file except in compliance with the License.
+            You may obtain a copy of the License at
+         
+                http://www.apache.org/licenses/LICENSE-2.0
+         
+            Unless required by applicable law or agreed to in writing, software
+            distributed under the License is distributed on an "AS IS" BASIS,
+            WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+            See the License for the specific language governing permissions and
+            limitations under the License.
+Classifier: Operating System :: OS Independent
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Typing :: Typed
+Requires-Dist: pyyaml>=6.0.1,<7
+Requires-Dist: xmltodict>=0.13.0,<0.16
+Requires-Dist: pydantic>=2.7,<3
+Requires-Dist: datasets>=2.19.1,<4
+Requires-Dist: sacrebleu>=2.4.3,<3
+Requires-Dist: pycountry>=24.6.1,<25
+Requires-Dist: nltk>=3.9.1,<4
+Requires-Dist: types-pyyaml>=6.0.12.20240917,<7
+Requires-Dist: psutil>=6.1,<7
+Requires-Dist: python-dotenv>=1.0.1,<2
+Requires-Dist: lingua-language-detector>=2.0.2,<3
+Requires-Dist: google-crc32c>=1.5.0,<2
+Requires-Dist: kubernetes>=31.0.0,<32
+Requires-Dist: langdetect>=1.0.9,<2
+Requires-Dist: spacy>=3.8.3,<4
+Requires-Dist: jsonschema>=4.23.0,<5
+Requires-Dist: mysql-connector-python>=9.0.0,<10
+Requires-Dist: psycopg2-binary>=2.9.9,<3
+Requires-Dist: sympy>=1.13.1,<2
+Requires-Dist: llm-sandbox[docker]>=0.1.8,<0.2
+Requires-Dist: jsonlines>=4,<5
+Requires-Dist: lxml>=6,<7
+Requires-Dist: python-iso639>=2025.2.18
+Requires-Dist: wandb>=0.21.1,<1
+Requires-Dist: torch
+Requires-Dist: accelerate ; extra == 'accelerate'
+Requires-Dist: eval-framework[determined,api,openai,transformers,accelerate,vllm,comet,optional,mistral] ; extra == 'all'
+Requires-Dist: aleph-alpha-client>=10,<11 ; extra == 'api'
+Requires-Dist: unbabel-comet>=2.2.6,<3 ; extra == 'comet'
+Requires-Dist: determined>=0.38,<0.39 ; extra == 'determined'
+Requires-Dist: tensorboard==2.19.0 ; extra == 'determined'
+Requires-Dist: mistral-common>=1.7,<2 ; extra == 'mistral'
+Requires-Dist: huggingface-hub>=0.33.2,<0.34 ; extra == 'mistral'
+Requires-Dist: eval-framework[vllm] ; extra == 'mistral'
+Requires-Dist: openai>=1.62,<2 ; extra == 'openai'
+Requires-Dist: tiktoken>=0.9,<0.10 ; extra == 'openai'
+Requires-Dist: transformers>=4.45.2,<5 ; extra == 'optional'
+Requires-Dist: jinja2>=3.1.6,<4 ; extra == 'optional'
+Requires-Dist: transformers>=4.45.2,<5 ; extra == 'transformers'
+Requires-Dist: torch>=2.5,<3 ; extra == 'transformers'
+Requires-Dist: vllm>=0.8.5,<0.9 ; extra == 'vllm'
+Requires-Dist: torch>=2.5,<3 ; extra == 'vllm'
+Requires-Python: >=3.12, <3.13
+Project-URL: repository, https://github.com/Aleph-Alpha-Research/eval-framework
+Provides-Extra: accelerate
+Provides-Extra: all
+Provides-Extra: api
+Provides-Extra: comet
+Provides-Extra: determined
+Provides-Extra: mistral
+Provides-Extra: openai
+Provides-Extra: optional
+Provides-Extra: transformers
+Provides-Extra: vllm
+Description-Content-Type: text/markdown
+
+# Aleph Alpha Eval-Framework
+
+> **Comprehensive LLM evaluation at scale** - A production-ready framework for evaluating large language models across 90+ benchmarks.
+
+## Features
+
+- 90+ Benchmarks: Covers reasoning, knowledge, coding, long-context, and safety tasks.
+- Custom Benchmarks: Easily add new benchmarks with minimal code using the BaseTask class.
+- Distributed Evaluation: Integration with Determined AI for scalable distributed evaluation.
+- Docker Support: Pre-configured Dockerfiles for local and distributed setups.
+- Flexible Model Integration: Supports models loaded via HuggingFace Transformers or custom implementations using the BaseLLM class.
+- Custom Metrics: Easily define new metrics using the BaseMetric class.
+- Rich Outputs: Generates JSON results, plots, and detailed analysis reports.
+- Perturbation Testing: Robustness analysis with configurable perturbation types and probabilities.
+- Statistical Analysis: Includes confidence intervals and significance testing for reliable comparisons.
+- LLM-as-a-Judge: Evaluation using LLM judges.
+
+![eval-framework](docs/eval-framework.png "eval-framework")
+
+## Benchmark Coverage & Task Categories
+
+### Core Capabilities
+
+| **Reasoning** | **Knowledge** | **Coding** | **Long Context** |
+|---------------|---------------|------------|------------------|
+| MMLU (57 subjects) | TriviaQA | HumanEval | InfiniteBench |
+| SQuAD v1/v2 | MBPP |
+| ARC | Natural Questions | CodeT5 | ZeroSCROLLS |
+| HellaSwag | QuAC | Programming | QuALITY |
+| Winogrande | COPA | Debugging  |
+
+### Languages & Domains
+
+| **Multilingual** | **Specialized** | **Safety & Bias** | **Efficiency** |
+|------------------|-----------------|-------------------|----------------|
+| WMT Translation | Legal (CaseHold) | TruthfulQA | Token counting |
+| FLORES-200 | Winogender | Latency metrics |
+| Multilingual MMLU | Medical (MedQA) | Stereotype detection | Memory usage |
+| German/Finnish tasks | Scientific (SciQ) | Harmful content | Cost analysis |
+
+### Completion
+
+Tasks focused on logical reasoning, text distillation, instruction following, and output control. Examples include:
+- **AIME 2024:** Logical Reasoning (Math)
+- **DUC Abstractive:** Text Distillation (Extraction)
+- **Custom Data: Complaint Summarization:** Text Distillation (Summarization)
+
+### Loglikelihoods
+
+Tasks emphasizing classification, reasoning, and open QA. Examples include:
+- **Abstract Reasoning Challenge (ARC):** Classification
+- **Casehold:** Open QA
+
+### Long-Context
+
+Tasks designed for long-context scenarios, including QA, summarization, and aggregation. Examples include:
+- **InfiniteBench_CodeDebug:** Programming
+- **ZeroSCROLLS GovReport:** QA (Government)
+
+### Metrics
+
+Evaluation metrics include:
+- **Completion Metrics:** Accuracy, Bleu, F1, Rouge
+- **Loglikelihood Metrics:** Accuracy Loglikelihood, Probability Mass
+- **LLM Metrics:** Chatbot Style Judge, Instruction Judge
+- **Efficiency Metrics:** Bytes per Sequence Position
+
+For the full list of tasks and metrics, see [Detailed Task Table](docs/benchmarks_and_metrics.md).
+
+## Quick Start
+
+The codebase is tested and compatible with Python 3.12 and PyTorch 2.5.
+You will also need the appropriate CUDA dependencies and version installed on your system for GPU support.
+
+The easiest way to get started is by installing the library via `pip` and use it as an external dependency.
+```
+pip install eval_framework
+```
+
+There are optional extras available to unlock specific features of the library:
+- `mistral` for inference on Mistral models
+- `transformers` for inference using the transformers library
+- `api` for inference using the aleph-alpha client.
+- `vllm` for inference via VLLM
+- `determined` for running jobs via determined
+- `comet` for the COMET metric
+
+As a short hand, the `all` extra installs all of the above.
+
+For development, you can instead install it directly from the repository instead, please first install
+ [uv](https://docs.astral.sh/uv/getting-started/installation/)
+
+To install the project with all optional extras use
+```bash
+uv sync --all-extras
+```
+
+We provide custom groups to control optional extras.
+- `cpu`: Use the CPU backend for torch
+- `cu124`: Use the CUDA 12.4 backend
+- `flash_attn`: Install `flash_attn` with correct handling of build isolation
+
+Thus, the following will setup the project with `flash_attn` and CUDA 12.4
+```bash
+uv sync --all-extras --group flash_attn --group cu124
+```
+
+There is also a pre-commit hook to help with development:
+```
+uv run pre-commit install
+```
+
+After installation, task documentation can be generated with `uv run python src/eval_framework/utils/generate_task_docs.py` (see [docs/installation.md(docs/installation.md)) for more details.
+
+## Getting Started
+
+### Understanding the Evaluation Framework
+
+Eval-Framework provides a unified interface for evaluating language models across diverse benchmarks. The framework follows this interaction model:
+
+1. **Define Your Model** - Specify which model to evaluate (HuggingFace, API, or custom)
+2. **Choose Your Task** - Select from 150+ available benchmarks or create custom ones
+3. **Configure Evaluation** - Set parameters like few-shot examples, sample count, and output format
+4. **Run Evaluation** - Execute locally via CLI/script or distribute via Determined AI
+5. **Analyze Results** - Review detailed JSON outputs, metrics, and generated reports
+
+### Core Components
+
+- **Models**: Defined via [`BaseLLM`](docs/evaluate_huggingface_model.md) interface (HuggingFace, OpenAI, custom APIs)
+- **Tasks**: Inherit from [`BaseTask`](docs/add_new_benchmark_guide.md) (completion, loglikelihood, or LLM-judge based)
+- **Metrics**: Automatic scoring via [`BaseMetric`](docs/benchmarks_and_metrics.md) classes
+- **Formatters**: Handle prompt construction and model-specific formatting
+- **Results**: Structured outputs with sample-level details and aggregated statistics
+
+### Your First Evaluation
+
+1. **Install the framework** (see Quick Start above)
+```
+pip install eval_framework[transformers]
+```
+
+2. **Create and run your first evaluation using HuggingFace model**:
+
+   ```python
+    from pathlib import Path
+
+    from eval_framework.llm.huggingface import HFLLM
+    from eval_framework.main import main
+    from eval_framework.tasks.eval_config import EvalConfig
+    from template_formatting.formatter import HFFormatter
+
+    # Define your model
+    class MyHuggingFaceModel(HFLLM):
+        LLM_NAME = "microsoft/DialoGPT-medium"
+        DEFAULT_FORMATTER = partial(HFFormatter, "microsoft/DialoGPT-medium")
+
+    if __name__ == "__main__":
+        # Initialize your model
+        llm = MyHuggingFaceModel()
+
+        # Running evaluation on GSM8K task using 5 few-shot examples and 10 samples
+        config = EvalConfig(
+            output_dir=Path("./eval_results"),
+            num_fewshot=5,
+            num_samples=10,
+            task_name="GSM8K",
+            llm_class=MyHuggingFaceModel,
+        )
+
+        # Run evaluation and get results
+        results = main(llm=llm, config=config)
+   ```
+
+3. **Review results** - Check `./eval_results/` for detailed outputs and use our [results guide](docs/understanding_results_guide.md) to interpret them
+
+### Next Steps
+
+- **Use CLI interface**: See [CLI usage guide](docs/cli_usage.md) for command-line evaluation options
+- **Evaluate HuggingFace models**: Follow our [HuggingFace evaluation guide](docs/evaluate_huggingface_model.md)
+- **Create custom benchmarks**: Follow our [benchmark creation guide](docs/add_new_benchmark_guide.md)
+- **Scale your evaluations**: Use [Determined AI integration](docs/using_determined.md) for distributed evaluation
+- **Understand your results**: Read our [results interpretation guide](docs/understanding_results_guide.md)
+
+### Example CLI Usage
+
+To evaluate a single benchmark locally, you can use the following command:
+
+```bash
+eval_framework \
+    --models src/eval_framework/llm/models.py \
+    --llm-name Smollm135MInstruct \
+    --task-name "GSM8K" \
+    --output-dir ./eval \
+    --num-fewshot 5 \
+    --num-samples 10
+```
+
+For more detailed CLI usage instructions, see the [CLI Usage Guide](docs/cli_usage.md).
+
+## Documentation
+
+### Getting Started
+
+- **[CLI Usage Guide](docs/cli_usage.md)** - Detailed instructions for using the command-line interface
+- **[Evaluating HuggingFace Models](docs/evaluate_huggingface_model.md)** - Complete guide for evaluating HuggingFace models
+- **[Understanding Results](docs/understanding_results_guide.md)** - How to read and interpret evaluation results
+
+### Advanced Usage
+
+- **[Adding New Benchmarks](docs/add_new_benchmark_guide.md)** - Complete guide with practical examples for adding new benchmarks
+- **[Benchmarks and Metrics](docs/benchmarks_and_metrics.md)** - Comprehensive overview of all available benchmarks and evaluation metrics
+- **[Overview of Dataloading](docs/overview_dataloading.md)** - Explanation of dataloading and task/sample/message structure
+
+### Scaling & Production
+
+- **[Using Determined](docs/using_determined.md)** - Guide for distributed evaluation using Determined AI
+- **[Controlling Upload Results](docs/controlling_upload_results.md)** - How to manage and control the upload of evaluation results
+
+### Citation
+
+If you use `eval-framework` in your research, please cite:
+
+```bibtex
+@software{eval_framework,
+  title={Aleph Alpha Eval Framework},
+  year={2025},
+  url={https://github.com/Aleph-Alpha-Research/eval-framework}
+}
+```
+
+### License
+
+This project is licensed under the [Apache License 2.0](LICENSE).
+
+<br><br>
+---
+
+This project has received funding from the European Union’s Digital Europe Programme under grant agreement No. 101195233 (OpenEuroLLM).
+
+The contents of this publication are the sole responsibility of the OpenEuroLLM consortium and do not necessarily reflect the opinion of the European Union.
+
+<p align="center">
+  <img src="docs/OELLM_1.png" alt="OELLM 1" width="100" style="margin-right: 50px;"/>
+  <img src="docs/OELLM_2.png" alt="OELLM 2" width="350"/>
+</p>