EuroEval 15.6.0__tar.gz → 15.6.1__tar.gz

{euroeval-15.6.0 → euroeval-15.6.1}/CHANGELOG.md

@@ -10,6 +10,18 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 
 
+## [v15.6.1] - 2025-04-14
+### Changed
+- Added more info about SQuAD-nl in the documentation. This was contributed by
+  [@Rijgersberg](https://github.com/Rijgersberg) ✨
+
+### Fixed
+- The "E" option for the Norwegian NorCommonSenseQA dataset was not included in the
+  refactor in v15.6.0, leading to evaluation errors. This has been fixed now.
+- The number of few-shot examples for FoSent was not reduced to 5 again during the
+  refactor in v15.6.0, leading to evaluation errors. This has been fixed now.
+
+
 ## [v15.6.0] - 2025-04-13
 ### Added
 - We now support specifying custom inference providers when benchmarking via the Hugging

{euroeval-15.6.0 → euroeval-15.6.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: EuroEval
-Version: 15.6.0
+Version: 15.6.1
 Summary: The robust European language model benchmark.
 Project-URL: Repository, https://github.com/EuroEval/EuroEval
 Project-URL: Issues, https://github.com/EuroEval/EuroEval/issues
@@ -237,6 +237,7 @@ A huge thank you to all the contributors who have helped make this project a suc
 <a href="https://github.com/ThomasKluiters"><img src="https://avatars.githubusercontent.com/u/8137941" width=50 alt="Contributor avatar for ThomasKluiters"/></a>
 <a href="https://github.com/BramVanroy"><img src="https://avatars.githubusercontent.com/u/2779410" width=50 alt="Contributor avatar for BramVanroy"/></a>
 <a href="https://github.com/peregilk"><img src="https://avatars.githubusercontent.com/u/9079808" width=50 alt="Contributor avatar for peregilk"/></a>
+<a href="https://github.com/Rijgersberg"><img src="https://avatars.githubusercontent.com/u/8604946" width=50 alt="Contributor avatar for Rijgersberg"/></a>
 
 ### Special Thanks
 - Thanks to [Google](https://google.com/) for sponsoring Gemini credits as part of their

{euroeval-15.6.0 → euroeval-15.6.1}/README.md

@@ -161,6 +161,7 @@ A huge thank you to all the contributors who have helped make this project a suc
 <a href="https://github.com/ThomasKluiters"><img src="https://avatars.githubusercontent.com/u/8137941" width=50 alt="Contributor avatar for ThomasKluiters"/></a>
 <a href="https://github.com/BramVanroy"><img src="https://avatars.githubusercontent.com/u/2779410" width=50 alt="Contributor avatar for BramVanroy"/></a>
 <a href="https://github.com/peregilk"><img src="https://avatars.githubusercontent.com/u/9079808" width=50 alt="Contributor avatar for peregilk"/></a>
+<a href="https://github.com/Rijgersberg"><img src="https://avatars.githubusercontent.com/u/8604946" width=50 alt="Contributor avatar for Rijgersberg"/></a>
 
 ### Special Thanks
 - Thanks to [Google](https://google.com/) for sponsoring Gemini credits as part of their

{euroeval-15.6.0 → euroeval-15.6.1}/docs/datasets/dutch.md

@@ -310,12 +310,14 @@ Here are a few examples from the training split:
 This dataset is published
 [here](https://huggingface.co/datasets/GroNLP/squad-nl-v2.0) and is a machine translated
 dataset of the English [SQuAD](https://aclanthology.org/D16-1264/) and
-[XQuAD](https://aclanthology.org/2020.acl-main.421/) datasets. Google Translate was used
-to translate the original datasets to Dutch.
-
-These are based on English Wikipedia articles and the questions and answers are written
-by crowdworkers. It is not clear how the translations were done, this information is
-pending from the authors.
+[XQuAD](https://aclanthology.org/2020.acl-main.421/) datasets, created for the
+Dutch-language [DUMB](https://dumbench.nl/) benchmark. Google Translate was used to
+translate the original datasets to Dutch. The test data
+[was manually corrected](https://aclanthology.org/2023.emnlp-main.447/) by eight BSc
+students as part of their thesis work.
+
+The original SQuAD and XQuAD datasets are based on English Wikipedia articles and the
+questions and answers are written by crowdworkers.
 
 Here are a few examples from the training split:
 

{euroeval-15.6.0 → euroeval-15.6.1}/makefile

@@ -81,21 +81,8 @@ test:  ## Run tests
 tree:  ## Print directory tree
 	@tree -a --gitignore -I .git .
 
-lint:  ## Lint the project
-	uv run ruff check . --fix --unsafe-fixes
-
-format:  ## Format the project
-	uv run ruff format .
-
-type-check:  ## Type-check the project
-	@uv run mypy . \
-		--install-types \
-		--non-interactive \
-		--ignore-missing-imports \
-		--show-error-codes \
-		--check-untyped-defs
-
-check: lint format type-check  ## Lint, format, and type-check the code
+check:  ## Lint, format, and type-check the code
+	@uv run pre-commit run --all-files
 
 bump-major:
 	@uv run python -m src.scripts.versioning --major

{euroeval-15.6.0 → euroeval-15.6.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "EuroEval"
-version = "15.6.0"
+version = "15.6.1"
 description = "The robust European language model benchmark."
 readme = "README.md"
 authors = [

{euroeval-15.6.0 → euroeval-15.6.1}/src/euroeval/data_models.py

@@ -388,8 +388,10 @@ class DatasetConfig:
             language.
         _prompt_label_mapping (optional):
             A mapping from the labels to another phrase which is used as a substitute
-            for the label in few-shot evaluation. Defaults to the template for the task
-            and language.
+            for the label in few-shot evaluation. If "auto" then the mapping will be set
+            to a 1:1 mapping between the labels and themselves. If None then the mapping
+            will be set to the default mapping for the task and language. Defaults to
+            None.
         unofficial (optional):
             Whether the dataset is unofficial. Defaults to False.
     """
@@ -405,7 +407,7 @@ class DatasetConfig:
     _num_few_shot_examples: int | None = None
     _max_generated_tokens: int | None = None
     _labels: list[str] | None = None
-    _prompt_label_mapping: dict[str, str] | None = None
+    _prompt_label_mapping: dict[str, str] | t.Literal["auto"] | None = None
     unofficial: bool = False
 
     @property
@@ -475,7 +477,9 @@ class DatasetConfig:
     @property
     def prompt_label_mapping(self) -> dict[str, str]:
         """Mapping from English labels to localised labels."""
-        if self._prompt_label_mapping is not None:
+        if self._prompt_label_mapping == "auto":
+            return {label: label for label in self.labels}
+        elif self._prompt_label_mapping is not None:
             return self._prompt_label_mapping
 
         main_language = self.languages[0]

{euroeval-15.6.0 → euroeval-15.6.1}/src/euroeval/dataset_configs/faroese.py

@@ -12,6 +12,7 @@ FOSENT_CONFIG = DatasetConfig(
     huggingface_id="EuroEval/fosent",
     task=SENT,
     languages=[FO],
+    _num_few_shot_examples=5,
 )
 
 SCALA_FO_CONFIG = DatasetConfig(

{euroeval-15.6.0 → euroeval-15.6.1}/src/euroeval/dataset_configs/norwegian.py

@@ -83,6 +83,7 @@ NOR_COMMON_SENSE_QA_CONFIG = DatasetConfig(
     huggingface_id="EuroEval/nor-common-sense-qa",
     task=COMMON_SENSE,
     languages=[NB, NN, NO],
+    _labels=["a", "b", "c", "d", "e"],
 )
 
 
@@ -105,7 +106,6 @@ NORGLM_MULTI_QA = DatasetConfig(
     huggingface_id="EuroEval/norglm-multi-qa",
     task=RC,
     languages=[NB, NN, NO],
-    _num_few_shot_examples=2,
     unofficial=True,
 )
 

{euroeval-15.6.0 → euroeval-15.6.1}/uv.lock

@@ -906,7 +906,7 @@ wheels = [
 
 [[package]]
 name = "euroeval"
-version = "15.6.0"
+version = "15.6.1"
 source = { editable = "." }
 dependencies = [
     { name = "accelerate" },