ScandEval 16.9.0__tar.gz → 16.10.0__tar.gz

{scandeval-16.9.0 → scandeval-16.10.0}/.github/ISSUE_TEMPLATE/benchmark_dataset_request.yaml

@@ -24,6 +24,7 @@ body:
     label: Dataset languages
     description: What languages is the dataset in?
     options:
+      - label: Albanian
       - label: Bulgarian
       - label: Bosnian
       - label: Catalan

{scandeval-16.9.0 → scandeval-16.10.0}/.github/ISSUE_TEMPLATE/model_evaluation_request.yaml

@@ -24,6 +24,7 @@ body:
       - label: Scandinavian languages (Danish, Faroese, Icelandic, Norwegian, Swedish)
       - label: Slavic languages (Bulgarian, Bosnian, Croatian, Czech, Polish, Serbian, Slovak, Slovenian, Ukrainian)
       - label: West Germanic languages (Dutch, English, German)
+      - label: Albanian
       - label: Greek
       - label: Hungarian
   validations:

{scandeval-16.9.0 → scandeval-16.10.0}/.pre-commit-config.yaml

@@ -8,9 +8,9 @@ repos:
     hooks:
       - id: end-of-file-fixer
       - id: trailing-whitespace
-      - id: debug-statements
+      # - id: debug-statements
 -   repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.14.9
+    rev: v0.14.10
     hooks:
       - id: ruff
         args:
@@ -34,7 +34,7 @@ repos:
     hooks:
     -   id: nbstripout
 -   repo: https://github.com/facebook/pyrefly-pre-commit
-    rev: 0.46.0
+    rev: 0.46.2
     hooks:
     -   id: pyrefly-check
         name: Pyrefly (type checking)

{scandeval-16.9.0 → scandeval-16.10.0}/CHANGELOG.md

@@ -7,6 +7,31 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+## [v16.10.0] - 2025-12-30
+
+### Added
+
+- Added support for Albanian 🇦🇱! This includes the sentiment classification dataset
+  MMS-sq, the linguistic acceptability dataset ScaLA-sq, the named entity recognition
+  dataset WikiANN-sq, the reading comprehension dataset MultiWikiQA-sq, the
+  summarisation dataset LR-Sum-sq, the knowledge dataset Global-MMLU-Lite-sq,
+  and the common-sense reasoning dataset Winogrande-sq. This was contributed by
+  @oliverkinch ✨
+- Added the Dutch common sense reasoning dataset COPA-NL, which is part of the Dutch
+  [DUMB benchmark](https://github.com/wietsedv/dumb). This was contributed by @tvosch ✨
+- Added new task for simplification and Dutch simplification dataset [Duidelijke
+  Taal](http://hdl.handle.net/10032/tm-a2-y8). dataset. This was contributed by
+  @simonevanbruggen ✨
+- Added multi-node support with Ray as a backend in this case. This was contributed by
+  @tvosch ✨
+- Added metadata for the Gemini-3 models.
+
+### Fixed
+
+- Fixed an issue with evaluations of LiteLLM models where asyncio event loops weren't
+  closed properly, leading to a buildup of file descriptors and eventually a "too many
+  open files" error.
+
 ## [v16.9.0] - 2025-12-16
 
 ### Added
@@ -15,6 +40,12 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
   [liu-nlp/swedish-facts-v1](https://huggingface.co/datasets/liu-nlp/swedish-facts-v1)
   dataset. This was contributed by @oliverkinch ✨
 
+### Changed
+
+- When benchmarking generative models, we now use their generation parameters as
+  specified in the `generation_config.json` file in the model repository on the Hugging
+  Face Hub, if it exists. We log this to the user if verbose mode is enabled.
+
 ### Fixed
 
 - When a model has registered the number of parameters wrongly within their safetensors

{scandeval-16.9.0 → scandeval-16.10.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ScandEval
-Version: 16.9.0
+Version: 16.10.0
 Summary: The robust European language model benchmark.
 Project-URL: Repository, https://github.com/EuroEval/EuroEval
 Project-URL: Issues, https://github.com/EuroEval/EuroEval/issues
@@ -50,6 +50,7 @@ Requires-Dist: pydantic>=2.6.0
 Requires-Dist: pyinfer>=0.0.3
 Requires-Dist: python-dotenv>=1.0.1
 Requires-Dist: rouge-score>=0.1.2
+Requires-Dist: sacrebleu>=2.5.1
 Requires-Dist: sacremoses>=0.1.1
 Requires-Dist: scikit-learn==1.6.1
 Requires-Dist: sentencepiece>=0.1.96
@@ -62,11 +63,13 @@ Requires-Dist: transformers[mistral-common]>=4.56.0
 Provides-Extra: all
 Requires-Dist: bitsandbytes>=0.43.1; (platform_system == 'Linux') and extra == 'all'
 Requires-Dist: fbgemm-gpu>=1.0.0; (platform_system == 'Linux') and extra == 'all'
+Requires-Dist: ray>=2.53.0; (platform_system == 'Linux') and extra == 'all'
 Requires-Dist: timm>=1.0.19; extra == 'all'
 Requires-Dist: vllm[flashinfer]==0.11.0; (platform_system == 'Linux') and extra == 'all'
 Provides-Extra: generative
 Requires-Dist: bitsandbytes>=0.43.1; (platform_system == 'Linux') and extra == 'generative'
 Requires-Dist: fbgemm-gpu>=1.0.0; (platform_system == 'Linux') and extra == 'generative'
+Requires-Dist: ray>=2.53.0; (platform_system == 'Linux') and extra == 'generative'
 Requires-Dist: timm>=1.0.19; extra == 'generative'
 Requires-Dist: vllm[flashinfer]==0.11.0; (platform_system == 'Linux') and extra == 'generative'
 Description-Content-Type: text/markdown
@@ -575,6 +578,20 @@ A huge thank you to all the contributors who have helped make this project a suc
         alt="Contributor avatar for mrkowalski"
     />
 </a>
+<a href="https://github.com/simonevanbruggen">
+    <img
+        src="https://avatars.githubusercontent.com/u/24842609"
+        width=50
+        alt="Contributor avatar for simonevanbruggen"
+    />
+</a>
+<a href="https://github.com/tvosch">
+    <img
+        src="https://avatars.githubusercontent.com/u/110661769"
+        width=50
+        alt="Contributor avatar for tvosch"
+    />
+</a>
 
 ### Contribute to EuroEval
 

{scandeval-16.9.0 → scandeval-16.10.0}/README.md

@@ -502,6 +502,20 @@ A huge thank you to all the contributors who have helped make this project a suc
         alt="Contributor avatar for mrkowalski"
     />
 </a>
+<a href="https://github.com/simonevanbruggen">
+    <img
+        src="https://avatars.githubusercontent.com/u/24842609"
+        width=50
+        alt="Contributor avatar for simonevanbruggen"
+    />
+</a>
+<a href="https://github.com/tvosch">
+    <img
+        src="https://avatars.githubusercontent.com/u/110661769"
+        width=50
+        alt="Contributor avatar for tvosch"
+    />
+</a>
 
 ### Contribute to EuroEval