ScandEval 16.8.0__tar.gz → 16.10.0__tar.gz

{scandeval-16.8.0 → scandeval-16.10.0}/.github/ISSUE_TEMPLATE/benchmark_dataset_request.yaml

@@ -24,6 +24,7 @@ body:
     label: Dataset languages
     description: What languages is the dataset in?
     options:
+      - label: Albanian
       - label: Bulgarian
       - label: Bosnian
       - label: Catalan

{scandeval-16.8.0 → scandeval-16.10.0}/.github/ISSUE_TEMPLATE/model_evaluation_request.yaml

@@ -20,11 +20,13 @@ body:
     options:
       - label: Baltic languages (Latvian, Lithuanian)
       - label: Finnic languages (Estonian, Finnish)
-      - label: Greek
       - label: Romance languages (Catalan, French, Italian, Portuguese, Romanian, Spanish)
       - label: Scandinavian languages (Danish, Faroese, Icelandic, Norwegian, Swedish)
-      - label: Slavic languages (Bulgarian, Bosnian, Croatian, Czech, Hungarian, Polish, Serbian, Slovak, Slovenian, Ukrainian)
+      - label: Slavic languages (Bulgarian, Bosnian, Croatian, Czech, Polish, Serbian, Slovak, Slovenian, Ukrainian)
       - label: West Germanic languages (Dutch, English, German)
+      - label: Albanian
+      - label: Greek
+      - label: Hungarian
   validations:
     required: true
 - type: dropdown

{scandeval-16.8.0 → scandeval-16.10.0}/.pre-commit-config.yaml

@@ -8,9 +8,9 @@ repos:
     hooks:
       - id: end-of-file-fixer
       - id: trailing-whitespace
-      - id: debug-statements
+      # - id: debug-statements
 -   repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.14.6
+    rev: v0.14.10
     hooks:
       - id: ruff
         args:
@@ -34,13 +34,13 @@ repos:
     hooks:
     -   id: nbstripout
 -   repo: https://github.com/facebook/pyrefly-pre-commit
-    rev: 0.0.1
+    rev: 0.46.2
     hooks:
-    -   id: pyrefly-typecheck-system
+    -   id: pyrefly-check
         name: Pyrefly (type checking)
         pass_filenames: true
 -   repo: https://github.com/DavidAnson/markdownlint-cli2
-    rev: v0.19.1
+    rev: v0.20.0
     hooks:
     -   id: markdownlint-cli2
         args:

{scandeval-16.8.0 → scandeval-16.10.0}/CHANGELOG.md

@@ -7,6 +7,60 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+## [v16.10.0] - 2025-12-30
+
+### Added
+
+- Added support for Albanian 🇦🇱! This includes the sentiment classification dataset
+  MMS-sq, the linguistic acceptability dataset ScaLA-sq, the named entity recognition
+  dataset WikiANN-sq, the reading comprehension dataset MultiWikiQA-sq, the
+  summarisation dataset LR-Sum-sq, the knowledge dataset Global-MMLU-Lite-sq,
+  and the common-sense reasoning dataset Winogrande-sq. This was contributed by
+  @oliverkinch ✨
+- Added the Dutch common sense reasoning dataset COPA-NL, which is part of the Dutch
+  [DUMB benchmark](https://github.com/wietsedv/dumb). This was contributed by @tvosch ✨
+- Added new task for simplification and Dutch simplification dataset [Duidelijke
+  Taal](http://hdl.handle.net/10032/tm-a2-y8). dataset. This was contributed by
+  @simonevanbruggen ✨
+- Added multi-node support with Ray as a backend in this case. This was contributed by
+  @tvosch ✨
+- Added metadata for the Gemini-3 models.
+
+### Fixed
+
+- Fixed an issue with evaluations of LiteLLM models where asyncio event loops weren't
+  closed properly, leading to a buildup of file descriptors and eventually a "too many
+  open files" error.
+
+## [v16.9.0] - 2025-12-16
+
+### Added
+
+- Added the Swedish factual knowledge dataset SwedishFacts, which is based on the
+  [liu-nlp/swedish-facts-v1](https://huggingface.co/datasets/liu-nlp/swedish-facts-v1)
+  dataset. This was contributed by @oliverkinch ✨
+
+### Changed
+
+- When benchmarking generative models, we now use their generation parameters as
+  specified in the `generation_config.json` file in the model repository on the Hugging
+  Face Hub, if it exists. We log this to the user if verbose mode is enabled.
+
+### Fixed
+
+- When a model has registered the number of parameters wrongly within their safetensors
+  files, we collect all the potential parameter counts from the safetensors file and
+  pick the largest one.
+- We now pinned vLLM to v0.11.0, as all future versions (up to and including v0.12.0)
+  have breaking changes regarding loading of Mistral models. We aim to unpin this when a
+  new vLLM version fixes this.
+- Removed mentions of `hf_transfer` and the associated environment variable
+  `HF_HUB_ENABLE_HF_TRANSFER`, since this has been removed from the `transformers`
+  library now.
+- Marked the `PleIAs/Pleias-3b-Preview` as requiring the `TRITON_ATTN` backend over the
+  default `FLASHINFER` backend, as the model architecture is currently not supported by
+  the default backend.
+
 ## [v16.8.0] - 2025-11-25
 
 ### Added
@@ -2735,8 +2789,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ### Deprecated
 
-- Deprecated support for evaluating finetuned models, as the package was primarily used to
-  benchmark pretrained models anyway, and the change in datasets means that many
+- Deprecated support for evaluating finetuned models, as the package was primarily used
+  to benchmark pretrained models anyway, and the change in datasets means that many
   finetuned models would have been trained on (part of) the test sets, resulting in
   artificially large scores. For evaluation of finetuned models, please check out the
   `aiai_eval` Python package instead (under development).

{scandeval-16.8.0 → scandeval-16.10.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ScandEval
-Version: 16.8.0
+Version: 16.10.0
 Summary: The robust European language model benchmark.
 Project-URL: Repository, https://github.com/EuroEval/EuroEval
 Project-URL: Issues, https://github.com/EuroEval/EuroEval/issues
@@ -39,6 +39,7 @@ Requires-Dist: evaluate>=0.4.1
 Requires-Dist: huggingface-hub>=0.30.1
 Requires-Dist: levenshtein>=0.24.0
 Requires-Dist: litellm>=1.75.6
+Requires-Dist: mistral-common[soundfile]
 Requires-Dist: more-itertools>=10.5.0
 Requires-Dist: numpy>=2.0.0
 Requires-Dist: ollama>=0.5.1
@@ -49,6 +50,7 @@ Requires-Dist: pydantic>=2.6.0
 Requires-Dist: pyinfer>=0.0.3
 Requires-Dist: python-dotenv>=1.0.1
 Requires-Dist: rouge-score>=0.1.2
+Requires-Dist: sacrebleu>=2.5.1
 Requires-Dist: sacremoses>=0.1.1
 Requires-Dist: scikit-learn==1.6.1
 Requires-Dist: sentencepiece>=0.1.96
@@ -61,13 +63,15 @@ Requires-Dist: transformers[mistral-common]>=4.56.0
 Provides-Extra: all
 Requires-Dist: bitsandbytes>=0.43.1; (platform_system == 'Linux') and extra == 'all'
 Requires-Dist: fbgemm-gpu>=1.0.0; (platform_system == 'Linux') and extra == 'all'
+Requires-Dist: ray>=2.53.0; (platform_system == 'Linux') and extra == 'all'
 Requires-Dist: timm>=1.0.19; extra == 'all'
-Requires-Dist: vllm[flashinfer]>=0.11.0; (platform_system == 'Linux') and extra == 'all'
+Requires-Dist: vllm[flashinfer]==0.11.0; (platform_system == 'Linux') and extra == 'all'
 Provides-Extra: generative
 Requires-Dist: bitsandbytes>=0.43.1; (platform_system == 'Linux') and extra == 'generative'
 Requires-Dist: fbgemm-gpu>=1.0.0; (platform_system == 'Linux') and extra == 'generative'
+Requires-Dist: ray>=2.53.0; (platform_system == 'Linux') and extra == 'generative'
 Requires-Dist: timm>=1.0.19; extra == 'generative'
-Requires-Dist: vllm[flashinfer]>=0.11.0; (platform_system == 'Linux') and extra == 'generative'
+Requires-Dist: vllm[flashinfer]==0.11.0; (platform_system == 'Linux') and extra == 'generative'
 Description-Content-Type: text/markdown
 
 <!-- This disables the requirement that the first line is a top-level heading -->
@@ -574,6 +578,20 @@ A huge thank you to all the contributors who have helped make this project a suc
         alt="Contributor avatar for mrkowalski"
     />
 </a>
+<a href="https://github.com/simonevanbruggen">
+    <img
+        src="https://avatars.githubusercontent.com/u/24842609"
+        width=50
+        alt="Contributor avatar for simonevanbruggen"
+    />
+</a>
+<a href="https://github.com/tvosch">
+    <img
+        src="https://avatars.githubusercontent.com/u/110661769"
+        width=50
+        alt="Contributor avatar for tvosch"
+    />
+</a>
 
 ### Contribute to EuroEval
 

{scandeval-16.8.0 → scandeval-16.10.0}/README.md

@@ -502,6 +502,20 @@ A huge thank you to all the contributors who have helped make this project a suc
         alt="Contributor avatar for mrkowalski"
     />
 </a>
+<a href="https://github.com/simonevanbruggen">
+    <img
+        src="https://avatars.githubusercontent.com/u/24842609"
+        width=50
+        alt="Contributor avatar for simonevanbruggen"
+    />
+</a>
+<a href="https://github.com/tvosch">
+    <img
+        src="https://avatars.githubusercontent.com/u/110661769"
+        width=50
+        alt="Contributor avatar for tvosch"
+    />
+</a>
 
 ### Contribute to EuroEval