EuroEval 15.7.0__tar.gz → 15.7.2__tar.gz

{euroeval-15.7.0 → euroeval-15.7.2}/.pre-commit-config.yaml

@@ -8,9 +8,9 @@ repos:
     hooks:
       - id: end-of-file-fixer
       - id: trailing-whitespace
-      # - id: debug-statements
+      - id: debug-statements
 -   repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.11.7
+    rev: v0.11.8
     hooks:
       - id: ruff
         args:

{euroeval-15.7.0 → euroeval-15.7.2}/CHANGELOG.md

@@ -10,6 +10,34 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 
 
+## [v15.7.2] - 2025-05-02
+### Fixed
+- Now does not check if a model exists if it has already been evaluated. This is an
+  issue when evaluating Ollama models, if the Ollama server is not running.
+- When evaluating instruction-tuned models on text classification tasks, the chat
+  template sometimes ends with special symbols, such as a newline, which can change the
+  tokenisation of the generated label. When we are evaluating the model using logprobs
+  we are thus looking for the wrong label in these cases. We now take this into account,
+  and log it to the user if the labels are not found, to avoid confusion.
+- Finnish datasets were not included in the default "all" dataset list, which is the
+  default used when no datasets are specified. This has been fixed now.
+- Temporarily disabled HellaSwag-fi, as there is an issue with the labels in the test
+  split, causing errors during evaluation. We will re-enable in a future release, when
+  this has been fixed.
+
+
+## [v15.7.1] - 2025-04-29
+### Changed
+- Marked the DBRD Dutch sentiment classification as official, as the quality is
+  substantially better than the previous Dutch Social.
+
+### Fixed
+- Fixed an issue with NER evaluation of instruction-tuned models, which was caused by
+  the "O" label mistakenly being included in the prompt template, causing an error
+  during evaluation. No evaluations were affected by this, only that some evaluations
+  could not be run.
+
+
 ## [v15.7.0] - 2025-04-28
 ### Added
 - Added support for Finnish 🇫🇮! This includes the Finnish part of the reading

{euroeval-15.7.0 → euroeval-15.7.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: EuroEval
-Version: 15.7.0
+Version: 15.7.2
 Summary: The robust European language model benchmark.
 Project-URL: Repository, https://github.com/EuroEval/EuroEval
 Project-URL: Issues, https://github.com/EuroEval/EuroEval/issues

{euroeval-15.7.0 → euroeval-15.7.2}/docs/datasets/dutch.md

@@ -7,68 +7,7 @@ information about what these constitute.
 
 ## Sentiment Classification
 
-### Dutch Social
-
-This dataset consists of Dutch tweets annotated with sentiment labels. It is not sure
-how the sentiment labels were assigned, this information is pending from the authors.
-
-The original full dataset consists of 162,805 / 54,269 / 54,268 samples for training,
-validation and testing, respectively (so 271,342 samples used in total). We use a 1,024
-/ 256 / 1,024 split for training, validation and testing, respectively. All the new
-splits are subsets of the original splits.
-
-Here are a few examples from the training split:
-
-```json
-{
-  "text": 'Novak Djokovic positief getest op coronavirus na eigen tennistoernooi\n\nhttps://t.co/U7VOcjANh9',
-  "label": 'positive'
-}
-```
-```json
-{
-  "text": "via @NYTimes  https://t.co/IjbCWIwYvR",
-  "label": "neutral"
-}
-```
-```json
-{
-  "text": "@backinflow 30 min Corona tijd....",
-  "label": "negative"
-}
-```
-
-When evaluating generative models, we use the following setup (see the
-[methodology](/methodology) for more information on how these are used):
-
-- Number of few-shot examples: 12
-- Prefix prompt:
-  ```
-  Hieronder staan tweets en hun sentiment, dat 'positief', 'neutraal' of 'negatief' kan zijn.
-  ```
-- Base prompt template:
-  ```
-  Tweet: {text}
-  Sentiment: {label}
-  ```
-- Instruction-tuned prompt template:
-  ```
-  Tweet: {text}
-
-  Classificeer het sentiment in de tweet. Antwoord met 'positief', 'neutraal' of 'negatief'.
-  ```
-- Label mapping:
-    - `positive` ➡️ `positief`
-    - `neutral` ➡️ `neutraal`
-    - `negative` ➡️ `negatief`
-
-You can evaluate this dataset directly as follows:
-
-```bash
-$ euroeval --model <model-id> --dataset dutch-social
-```
-
-### Unofficial: DBRD
+### DBRD
 
 This dataset was published in [this paper](https://doi.org/10.48550/arXiv.1910.00896)
 and features Dutch book reviews from [Hebban.nl](https://www.hebban.nl), annotated with

{euroeval-15.7.0 → euroeval-15.7.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "EuroEval"
-version = "15.7.0"
+version = "15.7.2"
 description = "The robust European language model benchmark."
 readme = "README.md"
 authors = [

{euroeval-15.7.0 → euroeval-15.7.2}/src/euroeval/benchmark_config_factory.py

@@ -238,7 +238,7 @@ def prepare_languages(
             The default language codes of the languages to include.
 
     Returns:
-        The prepared model or dataset languages.
+        The prepared dataset languages.
     """
     # Create a dictionary that maps languages to their associated language objects
     language_mapping = get_all_languages()

{euroeval-15.7.0 → euroeval-15.7.2}/src/euroeval/benchmark_modules/litellm.py

@@ -1,11 +1,8 @@
 """Generative models from an inference API, using the LiteLLM framework."""
 
 import collections.abc as c
-import itertools as it
-import json
 import logging
 import os
-import random
 import re
 import typing as t
 from functools import cached_property, partial
@@ -60,6 +57,7 @@ from ..exceptions import (
     NeedsEnvironmentVariable,
     NeedsExtraInstalled,
 )
+from ..generation_utils import apply_prompt, extract_few_shot_examples
 from ..task_group_utils import (
     question_answering,
     sequence_classification,
@@ -943,14 +941,22 @@ class LiteLLMModel(BenchmarkModule):
             )
 
         if self.benchmark_config.few_shot:
-            few_shot_examples = self._extract_few_shot_examples(
-                dataset=dataset, task=task, itr_idx=itr_idx
+            few_shot_examples = extract_few_shot_examples(
+                dataset=dataset, dataset_config=self.dataset_config, itr_idx=itr_idx
             )
         else:
             few_shot_examples = list()
 
         dataset["test"] = dataset["test"].map(
-            partial(self._apply_prompt, few_shot_examples=few_shot_examples, task=task),
+            partial(
+                apply_prompt,
+                few_shot_examples=few_shot_examples,
+                model_config=self.model_config,
+                dataset_config=self.dataset_config,
+                instruction_model=True,
+                always_populate_text_field=False,
+                tokenizer=None,
+            ),
             batched=True,
             load_from_cache_file=False,
             keep_in_memory=True,
@@ -958,253 +964,6 @@ class LiteLLMModel(BenchmarkModule):
 
         return dataset
 
-    def _extract_few_shot_examples(
-        self, dataset: DatasetDict, task: Task, itr_idx: int
-    ) -> list[dict[str, t.Any]]:
-        """Extract few-shot examples from a dataset.
-
-        This will always extract the examples from the training split.
-
-        We ensure that the few-shot examples are unique by picking them one at a time.
-
-        Args:
-            dataset:
-                The dataset to extract the few-shot examples from.
-            task:
-                The task that is being benchmarked.
-            itr_idx:
-                The index of the dataset in the iterator.
-
-        Returns:
-            The few-shot examples.
-        """
-        random_seed = 4242 + itr_idx
-        num_few_shots = self.dataset_config.num_few_shot_examples
-        few_shot_examples: list[dict[str, t.Any]] = list()
-        shuffled_train = dataset["train"].shuffle(seed=random_seed)
-
-        match task.task_group:
-            case (
-                TaskGroup.SEQUENCE_CLASSIFICATION
-                | TaskGroup.MULTIPLE_CHOICE_CLASSIFICATION
-            ):
-                labels = it.cycle(self.dataset_config.labels)
-                while (
-                    len(few_shot_examples) < num_few_shots and len(shuffled_train) > 0
-                ):
-                    label = next(labels)
-                    possible_examples = shuffled_train.filter(
-                        lambda x: x["label"].lower() == label.lower()
-                    )
-                    if len(possible_examples) == 0:
-                        continue
-                    example = possible_examples.select(range(1))[0]
-                    few_shot_examples.append(example)
-                    shuffled_train = shuffled_train.filter(
-                        lambda x: x["text"] != example["text"]
-                    )
-
-            case TaskGroup.TEXT_TO_TEXT:
-                while (
-                    len(few_shot_examples) < num_few_shots and len(shuffled_train) > 0
-                ):
-                    example = shuffled_train.select(range(1))[0]
-                    few_shot_examples.append(example)
-                    shuffled_train = shuffled_train.filter(
-                        lambda x: x["text"] != example["text"]
-                    )
-
-            case TaskGroup.TOKEN_CLASSIFICATION:
-                labels = it.cycle(
-                    [
-                        label.lower()
-                        for label in self.dataset_config.labels
-                        if label.lower().startswith("b-")
-                    ]
-                )
-                while (
-                    len(few_shot_examples) < num_few_shots and len(shuffled_train) > 0
-                ):
-                    label = next(labels)
-                    possible_examples = shuffled_train.filter(
-                        lambda x: label in [tag.lower() for tag in x["labels"]]
-                    )
-                    if len(possible_examples) == 0:
-                        continue
-                    example = possible_examples.select(range(1))[0]
-                    few_shot_examples.append(example)
-                    shuffled_train = shuffled_train.filter(
-                        lambda x: x["tokens"] != example["tokens"]
-                    )
-
-            case TaskGroup.QUESTION_ANSWERING:
-                # Locate the maximum number of tokens that constitutes a short example
-                for max_num_tokens in [512, 1024, 2048, 4096, 8192]:
-                    train_with_short_examples = dataset["train"].filter(
-                        lambda example: len(example["context"]) < max_num_tokens
-                    )
-                    num_short_examples = len(train_with_short_examples)
-                    if num_short_examples >= self.dataset_config.num_few_shot_examples:
-                        break
-                else:
-                    raise InvalidBenchmark(
-                        "Could not find enough short examples for few-shot learning."
-                    )
-
-                shuffled_train = train_with_short_examples.shuffle(seed=random_seed)
-                while (
-                    len(few_shot_examples) < num_few_shots and len(shuffled_train) > 0
-                ):
-                    example = shuffled_train.select(range(1))[0]
-                    few_shot_examples.append(example)
-                    shuffled_train = shuffled_train.filter(
-                        lambda x: x["context"] != example["context"]
-                    )
-
-            case _:
-                raise NotImplementedError(f"Unsupported task group: {task.task_group}.")
-
-        random.seed(random_seed)
-        random.shuffle(few_shot_examples)
-        return few_shot_examples
-
-    def _apply_prompt(
-        self,
-        examples: dict[str, t.Any],
-        few_shot_examples: list[dict[str, t.Any]],
-        task: Task,
-    ) -> dict[str, t.Any]:
-        """Apply prompt template to an example, potentially with few-shot examples.
-
-        Args:
-            examples:
-                The examples to apply the few-shot examples to.
-            few_shot_examples:
-                The few-shot examples to apply.
-            task:
-                The task that is being benchmarked.
-
-        Returns:
-            The example with the few-shot examples applied.
-        """
-
-        def create_prompt(**kwargs: str) -> tuple[str, str]:
-            """Create a prompt from the given keyword arguments.
-
-            Args:
-                kwargs:
-                    The keyword arguments to use in the prompt.
-
-            Returns:
-                A pair (prompt, label), where "label" is an empty string if the model is
-                not instruction tuned (as in this case it is included in the prompt).
-            """
-            label_key = "label" if "label" in kwargs else "target_text"
-            label = kwargs.pop(label_key)
-            label_mapping = self.dataset_config.prompt_label_mapping
-            label = label_mapping.get(label, label)
-            prompt = self.dataset_config.instruction_prompt.format(**kwargs)
-            return prompt, label
-
-        match task.task_group:
-            case (
-                TaskGroup.SEQUENCE_CLASSIFICATION
-                | TaskGroup.MULTIPLE_CHOICE_CLASSIFICATION
-            ):
-                few_shot_sections = [
-                    create_prompt(
-                        text=example["text"].replace("\n", " ").strip(),
-                        label=example["label"].replace("\n", " ").strip(),
-                    )
-                    for example in few_shot_examples
-                ]
-                new_sections = [
-                    create_prompt(text=text.replace("\n", " ").strip(), label="")
-                    for text in examples["text"]
-                ]
-
-            case TaskGroup.TEXT_TO_TEXT:
-                few_shot_sections = [
-                    create_prompt(
-                        text=example["text"].replace("\n", " ").strip(),
-                        target_text=example["target_text"].replace("\n", " ").strip(),
-                    )
-                    for example in few_shot_examples
-                ]
-                new_sections = [
-                    create_prompt(text=text.replace("\n", " ").strip(), target_text="")
-                    for text in examples["text"]
-                ]
-
-            case TaskGroup.TOKEN_CLASSIFICATION:
-
-                def create_label(example: dict) -> str:
-                    prompt_labels = self.dataset_config.prompt_label_mapping.values()
-                    labels: dict[str, list[str]] = {
-                        prompt_label: list() for prompt_label in prompt_labels
-                    }
-                    for token, label in zip(example["tokens"], example["labels"]):
-                        label = label.lower()
-                        if label == "o":
-                            continue
-                        prompt_label = self.dataset_config.prompt_label_mapping[label]
-                        if label.startswith("b-"):
-                            labels[prompt_label].append(token)
-                        elif label.startswith("i-"):
-                            labels[prompt_label][-1] += " " + token
-                    return json.dumps(labels, ensure_ascii=False)
-
-                few_shot_sections = [
-                    create_prompt(
-                        text=" ".join(example["tokens"]).replace("\n", " ").strip(),
-                        label=create_label(example=example),
-                    )
-                    for example in few_shot_examples
-                ]
-                new_sections = [
-                    create_prompt(
-                        text=" ".join(tokens).replace("\n", " ").strip(), label=""
-                    )
-                    for tokens in examples["tokens"]
-                ]
-
-            case TaskGroup.QUESTION_ANSWERING:
-                few_shot_sections = [
-                    create_prompt(
-                        text=example["context"].replace("\n", " ").strip(),
-                        question=example["question"].replace("\n", " ").strip(),
-                        label=example["answers"]["text"][0].replace("\n", " "),
-                    )
-                    for example in few_shot_examples
-                ]
-                new_sections = [
-                    create_prompt(
-                        text=context.replace("\n", " ").strip(),
-                        question=question.replace("\n", " ").strip(),
-                        label="",
-                    )
-                    for context, question in zip(
-                        examples["context"], examples["question"]
-                    )
-                ]
-
-            case _:
-                raise NotImplementedError(f"Unsupported task group: {task.task_group}.")
-
-        few_shot_messages = [
-            dict(role=role, content=content)
-            for prompt, label in few_shot_sections
-            for role, content in [("user", prompt), ("assistant", label)]
-        ]
-
-        messages_list = [
-            few_shot_messages + [dict(role="user", content=prompt)]
-            for prompt, _ in new_sections
-        ]
-
-        examples["messages"] = messages_list
-        return examples
-
 
 def raise_if_wrong_params(
     model_config: ModelConfig, allowed_params: dict[str, list[str]]
@@ -1248,6 +1007,10 @@ def try_download_ollama_model(model_id: str) -> bool:
 
     Returns:
         Whether the model was downloaded successfully.
+
+    Raises:
+        InvalidModel:
+            If Ollama is not running or the model cannot be downloaded.
     """
     if not (model_id.startswith("ollama/") or model_id.startswith("ollama_chat/")):
         return False
@@ -1262,11 +1025,17 @@ def try_download_ollama_model(model_id: str) -> bool:
             level=logging.WARNING,
         )
 
-    downloaded_ollama_models: list[str] = [
-        model_obj.model
-        for model_obj in ollama.list().models
-        if model_obj.model is not None
-    ]
+    try:
+        downloaded_ollama_models: list[str] = [
+            model_obj.model
+            for model_obj in ollama.list().models
+            if model_obj.model is not None
+        ]
+    except ConnectionError:
+        raise InvalidModel(
+            "Ollama does not seem to be running, so we cannot evaluate the model "
+            f"{model_id!r}. Please make sure that Ollama is running and try again."
+        )
 
     ollama_model_id = "/".join(model_id.split("/")[1:])
     if ollama_model_id not in downloaded_ollama_models: