ScandEval 16.7.1__tar.gz → 16.9.0__tar.gz

{scandeval-16.7.1 → scandeval-16.9.0}/.github/ISSUE_TEMPLATE/benchmark_dataset_request.yaml

@@ -26,6 +26,7 @@ body:
     options:
       - label: Bulgarian
       - label: Bosnian
+      - label: Catalan
       - label: Croatian
       - label: Czech
       - label: Danish
@@ -37,6 +38,7 @@ body:
       - label: French
       - label: German
       - label: Greek
+      - label: Hungarian
       - label: Icelandic
       - label: Italian
       - label: Latvian
@@ -44,6 +46,7 @@ body:
       - label: Norwegian (Bokmål or Nynorsk)
       - label: Polish
       - label: Portuguese
+      - label: Romanian
       - label: Serbian
       - label: Slovak
       - label: Slovenian

{scandeval-16.7.1 → scandeval-16.9.0}/.github/ISSUE_TEMPLATE/model_evaluation_request.yaml

@@ -20,11 +20,12 @@ body:
     options:
       - label: Baltic languages (Latvian, Lithuanian)
       - label: Finnic languages (Estonian, Finnish)
-      - label: Hellenic languages (Greek)
-      - label: Romance languages (French, Italian, Portuguese, Spanish)
+      - label: Romance languages (Catalan, French, Italian, Portuguese, Romanian, Spanish)
       - label: Scandinavian languages (Danish, Faroese, Icelandic, Norwegian, Swedish)
       - label: Slavic languages (Bulgarian, Bosnian, Croatian, Czech, Polish, Serbian, Slovak, Slovenian, Ukrainian)
       - label: West Germanic languages (Dutch, English, German)
+      - label: Greek
+      - label: Hungarian
   validations:
     required: true
 - type: dropdown

{scandeval-16.7.1 → scandeval-16.9.0}/.pre-commit-config.yaml

@@ -10,7 +10,7 @@ repos:
       - id: trailing-whitespace
       - id: debug-statements
 -   repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.14.5
+    rev: v0.14.9
     hooks:
       - id: ruff
         args:
@@ -33,18 +33,14 @@ repos:
     rev: 0.8.2
     hooks:
     -   id: nbstripout
--   repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v1.18.2
+-   repo: https://github.com/facebook/pyrefly-pre-commit
+    rev: 0.46.0
     hooks:
-    -   id: mypy
-        args:
-          - --install-types
-          - --non-interactive
-          - --ignore-missing-imports
-          - --show-error-codes
-          - --check-untyped-defs
+    -   id: pyrefly-check
+        name: Pyrefly (type checking)
+        pass_filenames: true
 -   repo: https://github.com/DavidAnson/markdownlint-cli2
-    rev: v0.19.0
+    rev: v0.20.0
     hooks:
     -   id: markdownlint-cli2
         args:

{scandeval-16.7.1 → scandeval-16.9.0}/CHANGELOG.md

@@ -7,6 +7,71 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+## [v16.9.0] - 2025-12-16
+
+### Added
+
+- Added the Swedish factual knowledge dataset SwedishFacts, which is based on the
+  [liu-nlp/swedish-facts-v1](https://huggingface.co/datasets/liu-nlp/swedish-facts-v1)
+  dataset. This was contributed by @oliverkinch ✨
+
+### Fixed
+
+- When a model has registered the number of parameters wrongly within their safetensors
+  files, we collect all the potential parameter counts from the safetensors file and
+  pick the largest one.
+- We now pinned vLLM to v0.11.0, as all future versions (up to and including v0.12.0)
+  have breaking changes regarding loading of Mistral models. We aim to unpin this when a
+  new vLLM version fixes this.
+- Removed mentions of `hf_transfer` and the associated environment variable
+  `HF_HUB_ENABLE_HF_TRANSFER`, since this has been removed from the `transformers`
+  library now.
+- Marked the `PleIAs/Pleias-3b-Preview` as requiring the `TRITON_ATTN` backend over the
+  default `FLASHINFER` backend, as the model architecture is currently not supported by
+  the default backend.
+
+## [v16.8.0] - 2025-11-25
+
+### Added
+
+- Added support for Romanian 🇷🇴! This includes the sentiment classification dataset
+  RoSent, the linguistic acceptability dataset ScaLA-ro, the named entity recognition
+  dataset RoNEC, the reading comprehension dataset MultiWikiQA-ro, the summarisation
+  dataset SumO-Ro, the knowledge dataset Global-MMLU-ro, and the common-sense
+  reasoning dataset Winogrande-ro. This was contributed by @oliverkinch ✨
+- Added support for Hungarian 🇭🇺! This includes the sentiment classification dataset
+  HuSST, the linguistic acceptability dataset ScaLA-hu, the named entity recognition
+  dataset SzegedNER, the reading comprehension dataset MultiWikiQA-hu, the
+  summarisation dataset HunSum, the knowledge dataset MMLU-hu, and the common-sense
+  reasoning dataset Winogrande-hu. This was contributed by @oliverkinch ✨
+- Added support for Catalan! This includes the sentiment classification dataset
+  GuiaCat, the linguistic acceptability dataset ScaLA-ca, the named entity recognition
+  dataset WikiANN-ca, the reading comprehension dataset MultiWikiQA-ca, the summarisation
+  dataset DACSA-ca, the knowledge dataset MMLU-ca, and the common-sense reasoning dataset
+  Winogrande-ca. This was contributed by @oliverkinch ✨
+- Added Spanish summarisation dataset DACSA-es as an unofficial dataset.
+- Added Lithuanian sentiment classification dataset Atsiliepimai to replace the now
+  unofficial Lithuanian Emotions dataset. This was contributed by @oliverkinch ✨
+- Added new `--custom-datasets-file` (`custom_datasets_file` in the `Benchmarker` API)
+  argument, which can be used to specify a custom Python file containing custom dataset
+  definitions. It defaults to `custom_datasets.py` in the current working directory.
+
+### Changed
+
+- When evaluating models with the `--debug` flag (`debug=True` in the `Benchmarker`
+  API), we now include the full model inputs and outputs in the JSON file stored to the
+  current working directory, where we previously only included the model outputs.
+
+### Fixed
+
+- When encountering rate limits for API inference models, we ended up waiting 10 seconds
+  for each request, which was unnecessarily long. We now only wait 10 seconds for each
+  batch of requests.
+- Uses the `FLASH_ATTN` backend with vLLM for Gemma-3-1b and Gemma-3n models now and the
+  `TRITON_ATTN` with the other Gemma-3 models, as their architecture is currently not
+  supported by the default `FLASHINFER` backend. Note that this can always be changed
+  manually with the `VLLM_ATTENTION_BACKEND` environment variable.
+
 ## [v16.7.1] - 2025-11-18
 
 ### Fixed
@@ -23,7 +88,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 - Added support for Bosnian 🇧🇦! This includes the sentiment classification dataset
   MMS-bs, the named entity recognition dataset WikiANN-bs, the reading comprehension
-  dataset MultiWikiQA-bs, and the summarisation dataset LR-Sum-bs.
+  dataset MultiWikiQA-bs, and the summarisation dataset LR-Sum-bs. This was contributed
+  by @oliverkinch ✨
 - Now allows the 'low', 'medium' and 'high' reasoning effort parameters for the GPT-OSS
   models, which can be set by appending `#low`, `#medium` or `#high` to the model ID.
 
@@ -65,7 +131,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 - Added support for Croatian 🇭🇷! This includes the sentiment classification dataset
   MMS-hr, the linguistic acceptability dataset ScaLA-hr, the named entity recognition
   dataset WikiANN-hr, the reading comprehension dataset MultiWikiQA-hr, the knowledge
-  dataset MMLU-hr, and the common-sense reasoning dataset Winogrande-hr.
+  dataset MMLU-hr, and the common-sense reasoning dataset Winogrande-hr. This was
+  contributed by @oliverkinch ✨
 - Added a system dependency check for `nvcc` in the `VLLMModel.__init__` method to
   ensure the CUDA Toolkit is installed. Raises an error with installation instructions
   if NVCC is not available in the system PATH.
@@ -86,11 +153,6 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ### Added
 
-- Added support for Slovene 🇸🇮! This includes the sentiment classification dataset
-  Sentinews, the linguistic acceptability dataset ScaLA-sl, the named entity recognition
-  dataset ssj500k-NER, the reading comprehension
-  dataset MultiWikiQA-sl, the knowledge dataset MMLU-sl, and the common-sense reasoning
-  dataset Winogrande-sl.
 - Added better support for evaluating on custom datasets, by allowing `DatasetConfig`
   objects directly in the `Benchmarker.benchmark` method. We also support custom
   datasets with the CLI, by simply defining the desired `DatasetConfig`s in a
@@ -99,6 +161,11 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
   with the new `source` argument. This argument can both be the Hugging Face Hub ID of
   the dataset or a dictionary with 'train', 'val' and 'test', and values the paths to
   the CSV files.
+- Added support for Slovene 🇸🇮! This includes the sentiment classification dataset
+  Sentinews, the linguistic acceptability dataset ScaLA-sl, the named entity recognition
+  dataset ssj500k-NER, the reading comprehension
+  dataset MultiWikiQA-sl, the knowledge dataset MMLU-sl, and the common-sense reasoning
+  dataset Winogrande-sl. This was contributed by @oliverkinch ✨
 - Added support for Serbian 🇷🇸! This includes the sentiment classification dataset
   MMS-sr, the linguistic acceptability dataset ScaLA-sr, the named entity recognition
   dataset UNER-sr, the reading comprehension dataset MultiWikiQA-sr, the summarisation
@@ -2691,8 +2758,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ### Deprecated
 
-- Deprecated support for evaluating finetuned models, as the package was primarily used to
-  benchmark pretrained models anyway, and the change in datasets means that many
+- Deprecated support for evaluating finetuned models, as the package was primarily used
+  to benchmark pretrained models anyway, and the change in datasets means that many
   finetuned models would have been trained on (part of) the test sets, resulting in
   artificially large scores. For evaluation of finetuned models, please check out the
   `aiai_eval` Python package instead (under development).

{scandeval-16.7.1 → scandeval-16.9.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ScandEval
-Version: 16.7.1
+Version: 16.9.0
 Summary: The robust European language model benchmark.
 Project-URL: Repository, https://github.com/EuroEval/EuroEval
 Project-URL: Issues, https://github.com/EuroEval/EuroEval/issues
@@ -39,6 +39,7 @@ Requires-Dist: evaluate>=0.4.1
 Requires-Dist: huggingface-hub>=0.30.1
 Requires-Dist: levenshtein>=0.24.0
 Requires-Dist: litellm>=1.75.6
+Requires-Dist: mistral-common[soundfile]
 Requires-Dist: more-itertools>=10.5.0
 Requires-Dist: numpy>=2.0.0
 Requires-Dist: ollama>=0.5.1
@@ -62,12 +63,12 @@ Provides-Extra: all
 Requires-Dist: bitsandbytes>=0.43.1; (platform_system == 'Linux') and extra == 'all'
 Requires-Dist: fbgemm-gpu>=1.0.0; (platform_system == 'Linux') and extra == 'all'
 Requires-Dist: timm>=1.0.19; extra == 'all'
-Requires-Dist: vllm[flashinfer]>=0.11.0; (platform_system == 'Linux') and extra == 'all'
+Requires-Dist: vllm[flashinfer]==0.11.0; (platform_system == 'Linux') and extra == 'all'
 Provides-Extra: generative
 Requires-Dist: bitsandbytes>=0.43.1; (platform_system == 'Linux') and extra == 'generative'
 Requires-Dist: fbgemm-gpu>=1.0.0; (platform_system == 'Linux') and extra == 'generative'
 Requires-Dist: timm>=1.0.19; extra == 'generative'
-Requires-Dist: vllm[flashinfer]>=0.11.0; (platform_system == 'Linux') and extra == 'generative'
+Requires-Dist: vllm[flashinfer]==0.11.0; (platform_system == 'Linux') and extra == 'generative'
 Description-Content-Type: text/markdown
 
 <!-- This disables the requirement that the first line is a top-level heading -->
@@ -92,7 +93,7 @@ ______________________________________________________________________
 [![Second paper](https://img.shields.io/badge/arXiv-2406.13469-b31b1b.svg)](https://arxiv.org/abs/2406.13469)
 [![License](https://img.shields.io/github/license/EuroEval/EuroEval)](https://github.com/EuroEval/EuroEval/blob/main/LICENSE)
 [![LastCommit](https://img.shields.io/github/last-commit/EuroEval/EuroEval)](https://github.com/EuroEval/EuroEval/commits/main)
-[![Code Coverage](https://img.shields.io/badge/Coverage-69%25-yellow.svg)](https://github.com/EuroEval/EuroEval/tree/main/tests)
+[![Code Coverage](https://img.shields.io/badge/Coverage-70%25-yellow.svg)](https://github.com/EuroEval/EuroEval/tree/main/tests)
 [![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-2.0-4baaaa.svg)](https://github.com/EuroEval/EuroEval/blob/main/CODE_OF_CONDUCT.md)
 
 ## Maintainer

{scandeval-16.7.1 → scandeval-16.9.0}/README.md

@@ -20,7 +20,7 @@ ______________________________________________________________________
 [![Second paper](https://img.shields.io/badge/arXiv-2406.13469-b31b1b.svg)](https://arxiv.org/abs/2406.13469)
 [![License](https://img.shields.io/github/license/EuroEval/EuroEval)](https://github.com/EuroEval/EuroEval/blob/main/LICENSE)
 [![LastCommit](https://img.shields.io/github/last-commit/EuroEval/EuroEval)](https://github.com/EuroEval/EuroEval/commits/main)
-[![Code Coverage](https://img.shields.io/badge/Coverage-69%25-yellow.svg)](https://github.com/EuroEval/EuroEval/tree/main/tests)
+[![Code Coverage](https://img.shields.io/badge/Coverage-70%25-yellow.svg)](https://github.com/EuroEval/EuroEval/tree/main/tests)
 [![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-2.0-4baaaa.svg)](https://github.com/EuroEval/EuroEval/blob/main/CODE_OF_CONDUCT.md)
 
 ## Maintainer

{scandeval-16.7.1 → scandeval-16.9.0}/docs/datasets/bosnian.md

@@ -9,8 +9,8 @@ information about what these constitute.
 ### MMS-bs
 
 This dataset was published in [this paper](https://doi.org/10.48550/arXiv.2306.07902).
-The corpus consists of 79 manually selected datasets from over 350 datasets reported in the
-scientific literature based on strict quality criteria.
+The corpus consists of 79 manually selected datasets from over 350 datasets reported in
+the scientific literature based on strict quality criteria.
 
 The original dataset contains a single split with 36,183 Bosnian samples.
 We use 1,024 / 256 / 2,048 samples for our training, validation, and test splits,