EuroEval 16.4.0__py3-none-any.whl → 16.5.0__py3-none-any.whl

{euroeval-16.4.0.dist-info → euroeval-16.5.0.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: EuroEval
-Version: 16.4.0
+Version: 16.5.0
 Summary: The robust European language model benchmark.
 Project-URL: Repository, https://github.com/EuroEval/EuroEval
 Project-URL: Issues, https://github.com/EuroEval/EuroEval/issues
@@ -92,7 +92,7 @@ ______________________________________________________________________
 [![Second paper](https://img.shields.io/badge/arXiv-2406.13469-b31b1b.svg)](https://arxiv.org/abs/2406.13469)
 [![License](https://img.shields.io/github/license/EuroEval/EuroEval)](https://github.com/EuroEval/EuroEval/blob/main/LICENSE)
 [![LastCommit](https://img.shields.io/github/last-commit/EuroEval/EuroEval)](https://github.com/EuroEval/EuroEval/commits/main)
-[![Code Coverage](https://img.shields.io/badge/Coverage-70%25-yellow.svg)](https://github.com/EuroEval/EuroEval/tree/main/tests)
+[![Code Coverage](https://img.shields.io/badge/Coverage-76%25-yellowgreen.svg)](https://github.com/EuroEval/EuroEval/tree/main/tests)
 [![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-2.0-4baaaa.svg)](https://github.com/EuroEval/EuroEval/blob/main/CODE_OF_CONDUCT.md)
 
 ## Maintainer
@@ -113,7 +113,7 @@ when an evaluation requires a certain extra dependency, and how you install it.
 
 ## Quickstart
 
-### Benchmarking from the Command Line
+### Benchmarking from the command line
 
 The easiest way to benchmark pretrained models is via the command line interface. After
 having installed the package, you can benchmark your favorite model like so:
@@ -160,7 +160,7 @@ See all the arguments and options available for the `euroeval` command by typing
 euroeval --help
 ```
 
-### Benchmarking from a Script
+### Benchmarking from a script
 
 In a script, the syntax is similar to the command line interface. You simply initialise
 an object of the `Benchmarker` class, and call this benchmark object with your favorite
@@ -168,15 +168,19 @@ model:
 
 ```python
 >>> from euroeval import Benchmarker
->>> benchmark = Benchmarker()
->>> benchmark(model="<model-id>")
+>>> benchmarker = Benchmarker()
+>>> benchmarker.benchmark(model="<model-id>")
 ```
 
 To benchmark on a specific task and/or language, you simply specify the `task` or
 `language` arguments, shown here with same example as above:
 
 ```python
->>> benchmark(model="<model-id>", task="sentiment-classification", language="da")
+>>> benchmarker.benchmark(
+...     model="<model-id>",
+...     task="sentiment-classification",
+...     language="da",
+... )
 ```
 
 If you want to benchmark a subset of all the models on the Hugging Face Hub, you can
@@ -184,10 +188,61 @@ simply leave out the `model` argument. In this example, we're benchmarking all D
 models on the Danish sentiment classification task:
 
 ```python
->>> benchmark(task="sentiment-classification", language="da")
+>>> benchmarker.benchmark(task="sentiment-classification", language="da")
 ```
 
-### Benchmarking in an Offline Environment
+### Benchmarking from Docker
+
+A Dockerfile is provided in the repo, which can be downloaded and run, without needing
+to clone the repo and installing from source. This can be fetched programmatically by
+running the following:
+
+```bash
+wget https://raw.githubusercontent.com/EuroEval/EuroEval/main/Dockerfile.cuda
+```
+
+Next, to be able to build the Docker image, first ensure that the NVIDIA Container
+Toolkit is
+[installed](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installation)
+and
+[configured](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#configuring-docker).
+Ensure that the the CUDA version stated at the top of the Dockerfile matches the CUDA
+version installed (which you can check using `nvidia-smi`). After that, we build the
+image as follows:
+
+```bash
+docker build --pull -t euroeval -f Dockerfile.cuda .
+```
+
+With the Docker image built, we can now evaluate any model as follows:
+
+```bash
+docker run -e args="<euroeval-arguments>" --gpus 1 --name euroeval --rm euroeval
+```
+
+Here `<euroeval-arguments>` consists of the arguments added to the `euroeval` CLI
+argument. This could for instance be `--model <model-id> --task
+sentiment-classification`.
+
+## Benchmarking custom inference APIs
+
+If the model you want to benchmark is hosted by a custom inference provider, such as a
+[vLLM server](https://docs.vllm.ai/en/stable/), then this is also supported in EuroEval.
+When benchmarking, you simply have to set the `--api-base` argument (`api_base` when
+using the `Benchmarker` API) to the URL of the inference API, and optionally the
+`--api-key` argument (`api_key`) to the API key, if authentication is required.
+
+When benchmarking models hosted on a custom inference API, the model ID
+(`--model`/`model`) should be the model name as registered on the inference server,
+potentially with a required prefix, depending on the type of inference server used. For
+instance, if the model is hosted on a vLLM server, the model ID should be prefixed with
+`hosted_vllm/`, and if the model is hosted on an Ollama server, the model ID should be
+prefixed with `ollama_chat/`. See the full list of possible inference providers as well
+as their corresponding prefixes in the [LiteLLM
+documentation](https://docs.litellm.ai/docs/providers/), as EuroEval uses LiteLLM to
+handle evaluation of inference APIs in general.
+
+## Benchmarking in an offline environment
 
 If you need to benchmark in an offline environment, you need to download the models,
 datasets and metrics beforehand. This can be done by adding the `--download-only`
@@ -202,7 +257,7 @@ euroeval --model <model-id> --task sentiment-classification --language da --down
 Or from a script:
 
 ```python
->>> benchmark(
+>>> benchmarker.benchmark(
 ... model="<model-id>",
 ... task="sentiment-classification",
 ... language="da",
@@ -210,44 +265,139 @@ Or from a script:
 ... )
 ```
 
-Please note: Offline benchmarking of adapter models is not currently supported. An
-internet connection will be required during evaluation. If offline support is important
-to you, please consider [opening an issue](https://github.com/EuroEval/EuroEval/issues).
+Please note: Offline benchmarking of adapter models is not currently supported, meaning
+that we still require an internet connection during the evaluation of these. If offline
+support of adapters is important to you, please consider [opening an
+issue](https://github.com/EuroEval/EuroEval/issues).
 
-### Benchmarking from Docker
+## Benchmarking custom datasets
 
-A Dockerfile is provided in the repo, which can be downloaded and run, without needing
-to clone the repo and installing from source. This can be fetched programmatically by
-running the following:
+If you want to benchmark models on your own custom dataset, this is also possible.
+First, you need to set up your dataset to be compatible with EuroEval. This means
+splitting up your dataset in a training, validation and test split, and ensuring that
+the column names are correct. We use `text` as the column name for the input text, and
+the output column name depends on the type of task:
 
-```bash
-wget https://raw.githubusercontent.com/EuroEval/EuroEval/main/Dockerfile.cuda
+- **Text or multiple-choice classification**: `label`
+- **Token classification**: `labels`
+- **Reading comprehension**: `answers`
+- **Free-form text generation**: `target_text`
+
+Text and multiple-choice classification tasks are by far the most common. Next, you
+store your three dataset splits as three different CSV files with the desired two
+columns. Finally, you create a file called `custom_datasets.py` script in which you
+define the associated `DatasetConfig` objects for your dataset. Here is an example of a
+simple text classification dataset with two classes:
+
+```python
+from euroeval import DatasetConfig, TEXT_CLASSIFICATION
+from euroeval.languages import ENGLISH
+
+MY_CONFIG = DatasetConfig(
+    name="my-dataset",
+    source=dict(train="train.csv", val="val.csv", test="test.csv"),
+    task=TEXT_CLASSIFICATION,
+    languages=[ENGLISH],
+    _labels=["positive", "negative"],
+)
 ```
 
-Next, to be able to build the Docker image, first ensure that the NVIDIA Container
-Toolkit is
-[installed](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installation)
-and
-[configured](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#configuring-docker).
-Ensure that the the CUDA version stated at the top of the Dockerfile matches the CUDA
-version installed (which you can check using `nvidia-smi`). After that, we build the
-image as follows:
+You can then benchmark your custom dataset by simply running
 
 ```bash
-docker build --pull -t euroeval -f Dockerfile.cuda .
+euroeval --dataset my-dataset --model <model-id>
 ```
 
-With the Docker image built, we can now evaluate any model as follows:
+You can also run the benchmark from a Python script, by simply providing your custom
+dataset configuration directly into the `benchmark` method:
 
-```bash
-docker run -e args="<euroeval-arguments>" --gpus 1 --name euroeval --rm euroeval
+```python
+from euroeval import Benchmarker
+
+benchmarker = Benchmarker()
+benchmarker.benchmark(model="<model-id>", dataset=MY_CONFIG)
 ```
 
-Here `<euroeval-arguments>` consists of the arguments added to the `euroeval` CLI
-argument. This could for instance be `--model <model-id> --task
-sentiment-classification`.
+We have included three convenience tasks to make it easier to set up custom datasets:
+
+- `TEXT_CLASSIFICATION`, which is used for text classification tasks. This requires you
+  to set the `_labels` argument in the `DatasetConfig`, and requires the columns `text`
+  and `label` to be present in the dataset.
+- `MULTIPLE_CHOICE`, which is used for multiple-choice classification tasks. This
+  also requires you to set the `_labels` argument in the `DatasetConfig`. Note that for
+  multiple choice tasks, you need to set up your `text` column to also list all the
+  choices, and all the samples should have the same number of choices. This requires the
+  columns `text` and `label` to be present in the dataset.
+- `TOKEN_CLASSIFICATION`, which is used when classifying individual tokens in a text.
+  This also require you to set the `_labels` argument in the `DatasetConfig`. This
+  requires the columns `tokens` and `labels` to be present in the dataset, where
+  `tokens` is a list of tokens/words in the text, and `labels` is a list of the
+  corresponding labels for each token (so the two lists have the same length).
+
+On top of these three convenience tasks, there are of course also the tasks that we use
+in the official benchmark, which you can use if you want to use one of these tasks with
+your own bespoke dataset:
+
+- `LA`, for linguistic acceptability datasets.
+- `NER`, for named entity recognition datasets with the standard BIO tagging scheme.
+- `RC`, for reading comprehension datasets in the SQuAD format.
+- `SENT`, for sentiment classification datasets.
+- `SUMM`, for text summarisation datasets.
+- `KNOW`, for multiple-choice knowledge datasets (e.g., MMLU).
+- `MCRC`, for multiple-choice reading comprehension datasets (e.g., Belebele).
+- `COMMON_SENSE`, for multiple-choice common-sense reasoning datasets (e.g., HellaSwag).
+
+These can all be imported from `euroeval.tasks` module.
+
+### Creating your own custom task
+
+You are of course also free to define your own task from scratch, which allows you to
+customise the prompts used when evaluating generative models, for instance. Here is an
+example of a custom free-form text generation task, where the goal for the model is to
+generate a SQL query based on a natural language input:
+
+```python
+from euroeval import DatasetConfig
+from euroeval.data_models import Task, PromptConfig
+from euroeval.enums import TaskGroup, ModelType
+from euroeval.languages import ENGLISH
+from euroeval.metrics import rouge_l_metric
+
+sql_generation_task = Task(
+    name="sql-generation",
+    task_group=TaskGroup.TEXT_TO_TEXT,
+    template_dict={
+        ENGLISH: PromptConfig(
+            default_prompt_prefix="The following are natural language texts and their "
+            "corresponding SQL queries.",
+            default_prompt_template="Natural language query: {text}\nSQL query: "
+            "{target_text}",
+            default_instruction_prompt="Generate the SQL query for the following "
+            "natural language query:\n{text!r}",
+            default_prompt_label_mapping=dict(),
+        ),
+    },
+    metrics=[rouge_l_metric],
+    default_num_few_shot_examples=3,
+    default_max_generated_tokens=256,
+    default_allowed_model_types=[ModelType.GENERATIVE],
+)
+
+MY_SQL_DATASET = DatasetConfig(
+    name="my-sql-dataset",
+    source=dict(train="train.csv", val="val.csv", test="test.csv"),
+    task=sql_generation_task,
+    languages=[ENGLISH],
+)
+```
+
+Again, with this you can benchmark your custom dataset by simply running
+
+```bash
+euroeval --dataset my-sql-dataset --model <model-id>
+```
 
-### Reproducing the datasets
+## Reproducing the evaluation datasets
 
 All datasets used in this project are generated using the scripts located in the
 [src/scripts](src/scripts) folder. To reproduce a dataset, run the corresponding script
@@ -379,6 +529,13 @@ A huge thank you to all the contributors who have helped make this project a suc
         alt="Contributor avatar for slowwavesleep"
     />
 </a>
+<a href="https://github.com/mrkowalski">
+    <img
+        src="https://avatars.githubusercontent.com/u/6357044"
+        width=50
+        alt="Contributor avatar for mrkowalski"
+    />
+</a>
 
 ### Contribute to EuroEval
 
@@ -390,7 +547,7 @@ contributing new datasets, your help makes this project better for everyone.
 - **Adding datasets**: If you're interested in adding a new dataset to EuroEval, we have
   a [dedicated guide](NEW_DATASET_GUIDE.md) with step-by-step instructions.
 
-### Special Thanks
+### Special thanks
 
 - Thanks to [Google](https://google.com/) for sponsoring Gemini credits as part of their
   [Google Cloud for Researchers Program](https://cloud.google.com/edu/researchers).
@@ -401,7 +558,7 @@ contributing new datasets, your help makes this project better for everyone.
 - Thanks to [UWV](https://www.uwv.nl/) and [KU
   Leuven](https://www.arts.kuleuven.be/ling/ccl) for sponsoring the Azure OpenAI
   credits used to evaluate GPT-4-turbo in Dutch.
-- Thanks to [Miðeind](https://mideind.is/english.html) for sponsoring the OpenAI
+- Thanks to [Miðeind](https://mideind.is/en) for sponsoring the OpenAI
   credits used to evaluate GPT-4-turbo in Icelandic and Faroese.
 - Thanks to [CHC](https://chc.au.dk/) for sponsoring the OpenAI credits used to
   evaluate GPT-4-turbo in German.

euroeval-16.5.0.dist-info/RECORD

@@ -0,0 +1,81 @@
+euroeval/__init__.py,sha256=MuVVOnGU3IJIROBmNkrFSCfnLaeqAFaI-gBaf78_Zr0,4118
+euroeval/benchmark_config_factory.py,sha256=evOZWuK5tBZkiVx5l7hxO5XPBbD-tmxMUZdcEsmBX4o,8689
+euroeval/benchmarker.py,sha256=ejvAhL9XNHYwIYhotIZMmvvYURCnsrMP5W2YhA2RI38,50615
+euroeval/caching_utils.py,sha256=lLUbkpDdJZy4xodIpwIz5d-WNKGuszbr_d9dyiJ5kZc,2591
+euroeval/callbacks.py,sha256=l8f6Zr8EoHfVFsI1ZnMUK0Y8uZB00Nvaz_I6XDn6avE,2515
+euroeval/cli.py,sha256=4g0tOOO9ItEAVEBVqDWvkKk12hys1rmpYk9zMRIH7Xw,12638
+euroeval/constants.py,sha256=XeEQ6nyvgVwhPu9ASe2BN2uO9URlcK03A76VbHyFit0,2959
+euroeval/data_loading.py,sha256=2CYnvjMVvJOUaDhwKueh3gYIdIwyAsCBonD9uciv_RU,6896
+euroeval/data_models.py,sha256=kHZ_90jtAq8ea2LQwhTc_UTW0ACcznxmLaJF10ZN0Wg,29127
+euroeval/enums.py,sha256=SeFek-Lre2Q5sxbP5svqjDZFZR2vlJhg9dkRH4JvU1g,3436
+euroeval/exceptions.py,sha256=wE5jg1ReR_ULtdFj5_E96QXzgi1AEnR2tYjyMLXF1R0,5115
+euroeval/finetuning.py,sha256=ggaoREQb7GA7LRpa1dORVLF170ubLZnX3j7RKm2vG5s,11804
+euroeval/generation.py,sha256=GiXsWlPhF19d85kA4bS1RZXEO0y-z7ENu5dH6fiQ_aQ,12637
+euroeval/generation_utils.py,sha256=6TepIaKjnt-7ViajDwCvAZmkmO3rSj3IvRx_tuje5lM,18285
+euroeval/languages.py,sha256=ABZu-en6eyrpbKsJK7pCYyhZIT_We4raLAFxTuiZVzg,37299
+euroeval/logging_utils.py,sha256=iq9K2-7URgeHOUJCYCC9k8vRAz4YmBf4f44GpCVEGcc,8802
+euroeval/model_cache.py,sha256=Jz7pIgzWav-29jqajxBkA-Nm0jDN6npmBDx5JgR_CgE,9304
+euroeval/model_config.py,sha256=fxHfgpw-9vj3hwke28DguVGvG9TU06nkTXT0V6KAMpQ,2761
+euroeval/model_loading.py,sha256=Ru44ONwCMEquM48T-pDWCnZq2V7mFKhu7L80OXtX4co,2340
+euroeval/scores.py,sha256=9a1XtppFbp8GJFc9JdThGxqBY0YUE7-92oyrlxScjNk,3281
+euroeval/speed_benchmark.py,sha256=VUOvauc9tuAegThNT2g1a-Z1l7DEmKq57dHI4t16o5A,4068
+euroeval/tasks.py,sha256=rEzL8Tixux6kHdeMQxOGCHSi-HK2B1BDFeYVuOq7QCU,5955
+euroeval/tokenisation_utils.py,sha256=KccepQZaAtwUI_K1LLHjz8R47O4Q7HUthkSShFl-m40,21551
+euroeval/types.py,sha256=dJFhHEPpTaDWWNUx1bayG7w9dVOspBdM1JT1Pr9EzhI,2951
+euroeval/utils.py,sha256=_l8vSXzXytA-BXvVREBC2sfgRB_xzfkfmPX-q7JnF_U,15287
+euroeval/benchmark_modules/__init__.py,sha256=TNO-sNDwlXE-LMFXfwwqjQqUy55gywSmwRBcoPUFuaU,236
+euroeval/benchmark_modules/base.py,sha256=PKRbPhh7mh-s9IPxn893G9v6MUCmMMJJZkhVD1rrwr0,11286
+euroeval/benchmark_modules/fresh.py,sha256=h4TPJlJK6xxxyhAXURr0T9gk4Khm3WyujnKBDFc3sCE,10806
+euroeval/benchmark_modules/hf.py,sha256=4dFXgLuSi48NPftd8mLd5a4tcF8r28w9jnntKKXHBGw,45935
+euroeval/benchmark_modules/litellm.py,sha256=LoGfj9Zp-S8YhKjufbDy3FvFdrqlLFj_eBKtPhvVjpM,66685
+euroeval/benchmark_modules/vllm.py,sha256=BjVcotncijq2YcDIZL-JSgtap6uiE4k4PTda187NlGw,47352
+euroeval/dataset_configs/__init__.py,sha256=8RXDhAowzRcKFYQwoDTumzpsCp5ab2n0_09_mlt0mdI,2361
+euroeval/dataset_configs/bulgarian.py,sha256=OVoDPTRdU-lVq-xUka7-Ct20h2jbs8HV43KBxRQenIE,1284
+euroeval/dataset_configs/czech.py,sha256=ghv2yNw839G-utll8PQRSjyKYbM5gfoQhFKy664GTCI,1562
+euroeval/dataset_configs/danish.py,sha256=LEKs04vK2KnV0CYheT7FeS-g3iHBvf2bQxyl0D_LbTg,3293
+euroeval/dataset_configs/dutch.py,sha256=HB1O7IxQUyOxLg7g0tqcCci1MHaKtZJiFlRJZo2jPr4,3107
+euroeval/dataset_configs/english.py,sha256=nc9nGwxf1tHVMUhQeND61yJbpTO4rJaAusPZlstqtq0,2817
+euroeval/dataset_configs/estonian.py,sha256=bWiKA_dJ7WUE8Z_1YZnSewhi4ZdCQBGJZ7pQxkCwMcU,2757
+euroeval/dataset_configs/faroese.py,sha256=13qYwXonDPWG9Av5MY_NBNTRDglPVKz5_mbz7ZCJ_mo,1247
+euroeval/dataset_configs/finnish.py,sha256=eyAMoQE43R-jKD3IZS0iqD2SZOPxWLqjmXzmm00tqPs,2444
+euroeval/dataset_configs/french.py,sha256=z6cGY0J7TgXjqmkOLOxQE6ADO1EEPoMF1sdk2n2USe0,2611
+euroeval/dataset_configs/german.py,sha256=CXW8_6CMMtrqrq85tDB6gY_fja_FATOqN5glNJI6efU,2858
+euroeval/dataset_configs/greek.py,sha256=BLdhfBIG7ABzoZWvtI_VlInpb0SYLK36nhHN33LPVo0,1475
+euroeval/dataset_configs/icelandic.py,sha256=G2Ibe6oF1NknkQmHqLpoHlysW_8f-0G53DJAGAlLkzQ,3552
+euroeval/dataset_configs/italian.py,sha256=qhjAQChnQanzs7EyN1DSAJ4OOU41HAlWqWntQOtbWCw,2761
+euroeval/dataset_configs/latvian.py,sha256=wbwIDieq5Lplng5Jzx9LEqq4d8b5LnNOyCUmT64b4bA,1928
+euroeval/dataset_configs/lithuanian.py,sha256=NaU5uTeHFaXgWHHXsxOzG5LO7KuTT5yXHqdjhIJIVKo,1498
+euroeval/dataset_configs/norwegian.py,sha256=skKKs4V4-zbd-1lpVUaxKXAjTMpBM6SAU5HZ8kcQ2mI,5454
+euroeval/dataset_configs/polish.py,sha256=nN_NT8cUK2iv1L_zO_aCYOk2R7ACSDZgvI7e0hIaFAM,2074
+euroeval/dataset_configs/portuguese.py,sha256=m9lEeVtI_yNvIdTIEOn3HFK_ilY2tn3-acC981hjZFM,2401
+euroeval/dataset_configs/serbian.py,sha256=yE_aoQRU8qiJqSD-iEt6o_M4rSkGahLX8qz2RRYDvbo,1411
+euroeval/dataset_configs/slovak.py,sha256=7JQj2GlRj1giACKqVZ6QJxiFBzP9byoowm0DjPaZ5u8,1252
+euroeval/dataset_configs/spanish.py,sha256=FhHSsFemau8mMkqF5VCSViRBZEcnSGcDeUUO9318XDM,2745
+euroeval/dataset_configs/swedish.py,sha256=QAN-cy818OojTQZdDRCPbMB_HZO_xZGoTb3CR7j9JmU,3071
+euroeval/dataset_configs/ukrainian.py,sha256=spbCmCOU27jOfz6FZxqCIfVmDN5l8H-7VCl-k-8eAIo,1527
+euroeval/metrics/__init__.py,sha256=qkELjrnBkuO9WzeQJZQRyXpZg_WclUByHswAc6Il7Ns,199
+euroeval/metrics/base.py,sha256=dUBby-ZzettMjdcjek6rw0JTZMuScX4cQ2Rd6untKHY,2525
+euroeval/metrics/huggingface.py,sha256=0j8AITealZiRNQU1K1apvRMvz8YFH0AL_xHyrvXX-qo,7159
+euroeval/metrics/llm_as_a_judge.py,sha256=ohEQfkevuixt_vjjb0oMfKus2SfewIm9FRl8TpNlLXs,9729
+euroeval/metrics/pipeline.py,sha256=xGCA7N1F4cLKOIeXP9SGAZvrWToREwAVb_gR5iBMQIU,10825
+euroeval/metrics/speed.py,sha256=G5hEQcrtqxF070ZZwLDh61iZnq2CSW2o6ZM7zR4lOTY,1298
+euroeval/prompt_templates/__init__.py,sha256=HN6Qspqm10ik6RKoPBJsvM-Nng9sywQojZbtbCqj4Z8,475
+euroeval/prompt_templates/classification.py,sha256=QuZh6hTMaqMYTsoruAhwjVP9381zzlQmDIwSeyGnav0,10121
+euroeval/prompt_templates/linguistic_acceptability.py,sha256=QOP0X0mpuPGmUUp9FkjsdMGfkK9FWSvh-Mx2gE7Xju0,12753
+euroeval/prompt_templates/multiple_choice.py,sha256=pzdWzGye0cV1bVWl_GdOupm1cdwT_BCKPNd3Ltx_FM4,10547
+euroeval/prompt_templates/named_entity_recognition.py,sha256=cUrKwDTJ3ztIvlZBo18Xccst7gqsN8guEdnKPFMYpOI,23942
+euroeval/prompt_templates/reading_comprehension.py,sha256=kjGh6rOlqTaUiTln9hco5BFjOeZpWcFP0kywOP6Y1Os,13240
+euroeval/prompt_templates/sentiment_classification.py,sha256=t17NxR2KQ2f1VWjRybHQ4LetwTawtMV_-5VobXrjSHM,13877
+euroeval/prompt_templates/summarization.py,sha256=6wYnUe10UT-sP3pFYJxfXXgAN1_SYIHyZi0uR6fis5A,8916
+euroeval/prompt_templates/token_classification.py,sha256=8Uw34mN2xQ_5es-nz7vCK-GgDg_oE-zsAzPJPzAxFrQ,15531
+euroeval/task_group_utils/__init__.py,sha256=CorGVkixkoEDOQuDsrOGlTmF1zmM0wnGHs8psWTfD28,72
+euroeval/task_group_utils/multiple_choice_classification.py,sha256=m0jRF8ifm4iJsBwvLsPN7kIS_5HpnOy6XMAOaX09AcY,7085
+euroeval/task_group_utils/question_answering.py,sha256=kAceiNVcRwASjhlvOCPXj0uX_MjX_TMsAr7fkVgTFJo,28130
+euroeval/task_group_utils/sequence_classification.py,sha256=vEV7b7DhiLaj2qz67u3k_rGdteC1yknC2RC0lnLzBKY,16520
+euroeval/task_group_utils/text_to_text.py,sha256=Y9f27-I_ie0zRBmOwleObZ5u8B0lzvYmbJ0zH9DqI1U,5415
+euroeval/task_group_utils/token_classification.py,sha256=_u8Ks4FK1oKB8Ifyu_I2NMW-z1GyJ7Mml5Z_edEaV_Q,17312
+euroeval-16.5.0.dist-info/METADATA,sha256=bBhULf3_umIesaQrluir1G4cbhYNm_0qQAOxwG7L7d8,22112
+euroeval-16.5.0.dist-info/WHEEL,sha256=qtCwoSJWgHk21S1Kb4ihdzI2rlJ1ZKaIurTj_ngOhyQ,87
+euroeval-16.5.0.dist-info/entry_points.txt,sha256=-mtBu-10bFWeZ2bS32gVK6-s-LNCQLxvnNUPBLd5ud4,87
+euroeval-16.5.0.dist-info/licenses/LICENSE,sha256=guvz_zBHgkQSY_QiUU0Bkc1k-L_PFZuLjIPfuKne2OY,1080
+euroeval-16.5.0.dist-info/RECORD,,

euroeval-16.4.0.dist-info/RECORD

@@ -1,75 +0,0 @@
-euroeval/__init__.py,sha256=Ci1Sta9hl-v-ZPwJ1qqAVpzvj-vVgZZbQQuP5Qopc4o,3956
-euroeval/benchmark_config_factory.py,sha256=x1HfK8kDVxN14PPHxonsDv0vhkdrexsMJfKaXhO9WQQ,8540
-euroeval/benchmarker.py,sha256=M_2KV0f41RmCiRLcQLEIACt1TcL7QqvH48ds0ebJCG8,49705
-euroeval/caching_utils.py,sha256=AkR0TLY9EHbqv3TrhtCmpEGsm0DWZSLEfR2fRHq1S3E,2587
-euroeval/callbacks.py,sha256=l8f6Zr8EoHfVFsI1ZnMUK0Y8uZB00Nvaz_I6XDn6avE,2515
-euroeval/cli.py,sha256=yb_Gw3TrouBiUmeQIZF0705zio8UPFACUDOzSB3CCfo,9316
-euroeval/constants.py,sha256=XAdsdSE4bAOUeW2o5qmMlfqRmsXZUNIKlEZrbxBPdLk,2845
-euroeval/data_loading.py,sha256=r2GtvH2fAPapE9Idyu8W27n3YXD2Bgw8Qt88vdDn0DQ,4751
-euroeval/data_models.py,sha256=j3gdzLSxgr3FakBIOqvVGZ5K5cXb4RrCMOkJc8J8Zmc,28007
-euroeval/enums.py,sha256=SeFek-Lre2Q5sxbP5svqjDZFZR2vlJhg9dkRH4JvU1g,3436
-euroeval/exceptions.py,sha256=5kQ-YvHyFO3aaA-zfOTaS07LRFH8xlSqlOiATvnIObY,5116
-euroeval/finetuning.py,sha256=t3VqkuRVqRxcpHhSzU4nF4npvLDnjNzPJqGqG-L6ifk,11764
-euroeval/generation.py,sha256=epv2QPHTxzoBmq5OFQtolvuvJ6ce4FkdD03NTYdKFZk,12579
-euroeval/generation_utils.py,sha256=3mI-T9imk433VsvbwCy71Zzv2XOdm-l1SH-IiFfSd9M,18285
-euroeval/languages.py,sha256=G2cJI8lDT7eOFHxNR9opJ6zWjdxFDwm8P8HY_4WKFI4,33815
-euroeval/logging_utils.py,sha256=iq9K2-7URgeHOUJCYCC9k8vRAz4YmBf4f44GpCVEGcc,8802
-euroeval/model_cache.py,sha256=S_8ZtLaliTiUEvQAVw_DJ1qk5PWUO5-eE04hGScCj_o,9246
-euroeval/model_config.py,sha256=fxHfgpw-9vj3hwke28DguVGvG9TU06nkTXT0V6KAMpQ,2761
-euroeval/model_loading.py,sha256=mVh05sPENBBOIUkd_rwXqbBd13YvF_tOVZ8XGtguNzw,2338
-euroeval/scores.py,sha256=tlLfmI6Pgm1d_odubfyFcGLoB6Mxgfw3Yl7POzFv9l8,3235
-euroeval/speed_benchmark.py,sha256=k9xEF7jPAMrEBcZdykilQ6eJMGhFW1eUGuhQco9470M,4034
-euroeval/tasks.py,sha256=EzEWFDo_0ffabBFiRu-mw80jENUioE8D_VEn_Dsv-F8,4703
-euroeval/tokenisation_utils.py,sha256=rytsJy4mNEqeSdGzXsvVU4OShveeHOOlbaQOJDsX4S0,21275
-euroeval/types.py,sha256=_iVy-RwiCGu9TNX2sfyJTdCvXy1akNGTCywAo-YpBqU,2815
-euroeval/utils.py,sha256=VJrbEFXr6ZCJIHiMT7M7Y84ZYl0LHe2uhIz4eePciAw,15235
-euroeval/benchmark_modules/__init__.py,sha256=TNO-sNDwlXE-LMFXfwwqjQqUy55gywSmwRBcoPUFuaU,236
-euroeval/benchmark_modules/base.py,sha256=PeOqhfrc9iqyRz1aDHFBiTpWcwU5zDXo5pB_CD8W4VI,11199
-euroeval/benchmark_modules/fresh.py,sha256=h4TPJlJK6xxxyhAXURr0T9gk4Khm3WyujnKBDFc3sCE,10806
-euroeval/benchmark_modules/hf.py,sha256=enj88OY2XELdNgLnqeRPXvX2ATgcm6fjQpSYpBhmgzI,46274
-euroeval/benchmark_modules/litellm.py,sha256=VNFIOJU8TJNrifHtfBILh1MeS7tehqztxH5WoPLr5fc,66581
-euroeval/benchmark_modules/vllm.py,sha256=dm19gYG-MR63V8YpZBM1iOQ1c7xbFRzo9NuDWHG3q-Y,46952
-euroeval/dataset_configs/__init__.py,sha256=zvyH0onXIDtm8lHDVRSzk7J0_mJFU0H6WnLueaxM7WE,2157
-euroeval/dataset_configs/czech.py,sha256=9IDYKg1aoikMXIqQo2tYTQHf2WmQEujkNTyF0a8c9c8,2134
-euroeval/dataset_configs/danish.py,sha256=nkw1poFOJGpQJFB9HYC6bdlNzUR5pXxYacvZs4GrK4Y,5522
-euroeval/dataset_configs/dutch.py,sha256=CDr0oQnmDxeNloZ6iTGYPcNqPM5R9N8Z4aTKEE0C2MU,5408
-euroeval/dataset_configs/english.py,sha256=2sJQPM4rZSYlwE5v4AiDm95Boq-_53AzdOt5cL_drJs,4628
-euroeval/dataset_configs/estonian.py,sha256=fC5TUGpd6u22DUxoETBLA7EThwqsPDU54gXTzWtFCHk,3369
-euroeval/dataset_configs/faroese.py,sha256=sFC25nwlPtnl6hwkPOxPkwVggPGTjw167YhSBnLl1EA,3039
-euroeval/dataset_configs/finnish.py,sha256=DwrhwluoV4rmW8m2E5gWTfvHZ1XKRQG_3KU7wSOqM40,4281
-euroeval/dataset_configs/french.py,sha256=MIZUAn2rNwasb41DC92q6vMwRBem1Fw4D7Hj1cLFlfs,4611
-euroeval/dataset_configs/german.py,sha256=D-Yuz0pGf8pOEfMmTibXfk0k0QGjA4nEyAmea4TnCh0,5021
-euroeval/dataset_configs/icelandic.py,sha256=qX-szARxqzJ9l-h0k5iXirC5StpW_B3BOakZQ14zmpM,5797
-euroeval/dataset_configs/italian.py,sha256=GaN7u2NgAOrpe7n--CsmpQm_n-PCVsTN-wV78UKtQco,4895
-euroeval/dataset_configs/latvian.py,sha256=8hb32_YD_nQHn4kRVfwiGRDoJHF8M00ZdcwuD5ozJwU,2647
-euroeval/dataset_configs/lithuanian.py,sha256=Gv3ta3Gs7xknZ_h_dVWY7YN43UfQzLkJPnYnQcuBguU,1998
-euroeval/dataset_configs/norwegian.py,sha256=VcNftTvOJMCQEJvDFe3iixKbr8cjE3C6oHG4Jp4HET4,7636
-euroeval/dataset_configs/polish.py,sha256=wiorGf4Z06WLPYAa5blD8F2qDaEWUr4MgVShkkVfVo4,3563
-euroeval/dataset_configs/portuguese.py,sha256=TsjJMGJc_wExE_9TMJiQuxhN9BylXcHTXRFaCmkE4Gg,3980
-euroeval/dataset_configs/slovak.py,sha256=Dc9ai2VW-ckQk7trglL2w1Ki0NECsr1RMXQPYBAN6OU,1759
-euroeval/dataset_configs/spanish.py,sha256=VQHQiRsTLlen1zBKgbmRiXSB--b89WofXgFxeIgMR1o,4793
-euroeval/dataset_configs/swedish.py,sha256=pNd-O9cU-4_9gkQU-EFVzsjri6Jg-0taVkzQYdFT6Lw,5257
-euroeval/metrics/__init__.py,sha256=qkELjrnBkuO9WzeQJZQRyXpZg_WclUByHswAc6Il7Ns,199
-euroeval/metrics/base.py,sha256=dUBby-ZzettMjdcjek6rw0JTZMuScX4cQ2Rd6untKHY,2525
-euroeval/metrics/huggingface.py,sha256=eCbL-jIj5WYAVRSYdbBWDzDoakIPl6_rSvBqLZhXO-E,6736
-euroeval/metrics/llm_as_a_judge.py,sha256=br-pIyzhgrfDXZb6K0GuSUAyczLnrc7pFugW1DYwK6w,9721
-euroeval/metrics/pipeline.py,sha256=xGCA7N1F4cLKOIeXP9SGAZvrWToREwAVb_gR5iBMQIU,10825
-euroeval/metrics/speed.py,sha256=G5hEQcrtqxF070ZZwLDh61iZnq2CSW2o6ZM7zR4lOTY,1298
-euroeval/prompt_templates/__init__.py,sha256=HWMZpybxs2xHPnVeJ43893conARahIVLWNXeRhXEGZw,357
-euroeval/prompt_templates/linguistic_acceptability.py,sha256=Q-GyoS_c_iM-wQ8aXTywRTdxl1kUF0WEzHWh40hsk3s,10098
-euroeval/prompt_templates/multiple_choice.py,sha256=p6Dt3EMaASyqFHOjxdisFnh7OOVi-roCyKalSPwp5Dc,8220
-euroeval/prompt_templates/named_entity_recognition.py,sha256=3yEr1GHk0UbubsTwDSK928QssgYO0mnMfOgVmlDT2HI,19066
-euroeval/prompt_templates/reading_comprehension.py,sha256=0eYnJOfk8u9Zv_Xj6VtDLoQwvfe5_jjzAWGAksRMO6Y,10338
-euroeval/prompt_templates/sentiment_classification.py,sha256=Xg90BzCHQEmgTImn9zqI9Z48nW1paGQ-4AWYCxoUJxk,11027
-euroeval/prompt_templates/summarization.py,sha256=ypyJRX2R5CyCFjJnM3iE5J4OrvLZBwXr7WdBLv8qMKQ,7391
-euroeval/task_group_utils/__init__.py,sha256=CorGVkixkoEDOQuDsrOGlTmF1zmM0wnGHs8psWTfD28,72
-euroeval/task_group_utils/multiple_choice_classification.py,sha256=tAFQOM_iZwyknbOcZfw6_71lUSbcB5OlY0gOkNfUBAY,7051
-euroeval/task_group_utils/question_answering.py,sha256=vr5gjIJxqqvbj0DYNSEdN0Ek9fkZ5maCAt7FKqzo-Xs,27695
-euroeval/task_group_utils/sequence_classification.py,sha256=_kYgAIF2LABZ-nate3O6s7vlfI2RGHVtpNPjaMIHLDk,16450
-euroeval/task_group_utils/text_to_text.py,sha256=ibSOiP_wpEyGYQh7uEeTjOp-ojLJsEcJT1W7IWOBfk8,5381
-euroeval/task_group_utils/token_classification.py,sha256=hFiO29eSX_KtqbjJM4jy37jmyhfhfnWj3WTpNvh_vQk,17208
-euroeval-16.4.0.dist-info/METADATA,sha256=ot4RNMLDwwJR2UIk20k59E7MsBOXlIqJPYI9xc_XUP8,15365
-euroeval-16.4.0.dist-info/WHEEL,sha256=qtCwoSJWgHk21S1Kb4ihdzI2rlJ1ZKaIurTj_ngOhyQ,87
-euroeval-16.4.0.dist-info/entry_points.txt,sha256=-mtBu-10bFWeZ2bS32gVK6-s-LNCQLxvnNUPBLd5ud4,87
-euroeval-16.4.0.dist-info/licenses/LICENSE,sha256=guvz_zBHgkQSY_QiUU0Bkc1k-L_PFZuLjIPfuKne2OY,1080
-euroeval-16.4.0.dist-info/RECORD,,