evalscope 0.5.5__tar.gz → 0.6.0__tar.gz

{evalscope-0.5.5 → evalscope-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: evalscope
-Version: 0.5.5
+Version: 0.6.0
 Summary: EvalScope: Lightweight LLMs Evaluation Framework
 Home-page: https://github.com/modelscope/evalscope
 Author: ModelScope team
@@ -19,7 +19,7 @@ Requires-Dist: torch
 Requires-Dist: absl-py
 Requires-Dist: accelerate
 Requires-Dist: cachetools
-Requires-Dist: datasets<3.0.0,>=2.18.0
+Requires-Dist: datasets<=3.0.1,>=3.0.0
 Requires-Dist: editdistance
 Requires-Dist: jsonlines
 Requires-Dist: matplotlib
@@ -52,8 +52,9 @@ Requires-Dist: ms-opencompass>=0.1.1; extra == "opencompass"
 Provides-Extra: vlmeval
 Requires-Dist: ms-vlmeval>=0.0.5; extra == "vlmeval"
 Provides-Extra: rag
-Requires-Dist: ragas; extra == "rag"
 Requires-Dist: mteb>=0.14.16; extra == "rag"
+Requires-Dist: ragas<0.3,>=0.2.3; extra == "rag"
+Requires-Dist: webdataset>0.2.0; extra == "rag"
 Provides-Extra: inner
 Requires-Dist: absl-py; extra == "inner"
 Requires-Dist: accelerate; extra == "inner"
@@ -86,7 +87,7 @@ Requires-Dist: torch; extra == "all"
 Requires-Dist: absl-py; extra == "all"
 Requires-Dist: accelerate; extra == "all"
 Requires-Dist: cachetools; extra == "all"
-Requires-Dist: datasets<3.0.0,>=2.18.0; extra == "all"
+Requires-Dist: datasets<=3.0.1,>=3.0.0; extra == "all"
 Requires-Dist: editdistance; extra == "all"
 Requires-Dist: jsonlines; extra == "all"
 Requires-Dist: matplotlib; extra == "all"
@@ -116,14 +117,18 @@ Requires-Dist: jieba; extra == "all"
 Requires-Dist: rouge-chinese; extra == "all"
 Requires-Dist: ms-opencompass>=0.1.1; extra == "all"
 Requires-Dist: ms-vlmeval>=0.0.5; extra == "all"
-Requires-Dist: ragas; extra == "all"
 Requires-Dist: mteb>=0.14.16; extra == "all"
+Requires-Dist: ragas<0.3,>=0.2.3; extra == "all"
+Requires-Dist: webdataset>0.2.0; extra == "all"
 
-English | [简体中文](README_zh.md)
 
 
 ![](docs/en/_static/images/evalscope_logo.png)
 
+<p align="center">
+    English | <a href="README_zh.md">简体中文</a>
+</p>
+
 <p align="center">
 <a href="https://badge.fury.io/py/evalscope"><img src="https://badge.fury.io/py/evalscope.svg" alt="PyPI version" height="18"></a>
 <a href="https://pypi.org/project/evalscope"><img alt="PyPI - Downloads" src="https://static.pepy.tech/badge/evalscope">
@@ -132,7 +137,7 @@ English | [简体中文](README_zh.md)
     <img src='https://readthedocs.org/projects/evalscope-en/badge/?version=latest' alt='Documentation Status' />
 </a>
 <br>
- <a href="https://evalscope.readthedocs.io/en/latest/"><span style="font-size: 16px;">📖 Documents</span></a> &nbsp | &nbsp<a href="https://evalscope.readthedocs.io/zh-cn/latest/"><span style="font-size: 16px;"> 📖  中文文档</span></a>
+ <a href="https://evalscope.readthedocs.io/en/latest/">📖 Documents</a>
 <p>
 
 
@@ -146,7 +151,7 @@ English | [简体中文](README_zh.md)
 - [Offline Evaluation](#offline-evaluation)
 - [Arena Mode](#arena-mode)
 - [Model Serving Performance Evaluation](#Model-Serving-Performance-Evaluation)
-- [Leaderboard](#leaderboard)
+
 
 ## 📝 Introduction
 
@@ -172,6 +177,8 @@ The architecture includes the following modules:
 
 
 ## 🎉 News
+- 🔥 **[2024.10.31]** The best practice for evaluating Multimodal-RAG has been updated, please check the [📖 Blog](https://evalscope.readthedocs.io/zh-cn/latest/blog/RAG/multimodal_RAG.html#multimodal-rag) for more details.
+- 🔥 **[2024.10.23]** Supports multimodal RAG evaluation, including the assessment of image-text retrieval using [CLIP_Benchmark](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/clip_benchmark.html), and extends [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html) to support end-to-end multimodal metrics evaluation.
 - 🔥 **[2024.10.8]** Support for RAG evaluation, including independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html).
 - 🔥 **[2024.09.18]** Our documentation has been updated to include a blog module, featuring some technical research and discussions related to evaluations. We invite you to [📖 read it](https://evalscope.readthedocs.io/en/refact_readme/blog/index.html).
 - 🔥 **[2024.09.12]** Support for LongWriter evaluation, which supports 10,000+ word generation. You can use the benchmark [LongBench-Write](evalscope/third_party/longbench_write/README.md) to measure the long output quality as well as the output length.
@@ -368,11 +375,6 @@ A stress testing tool that focuses on large language models and can be customize
 Refer to : Model Serving Performance Evaluation [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test.html)
 
 
-## Leaderboard
-The LLM Leaderboard aims to provide an objective and comprehensive evaluation standard and platform to help researchers and developers understand and compare the performance of models on various tasks on ModelScope.
-
-Refer to : [Leaderboard](https://modelscope.cn/leaderboard/58/ranking?type=free)
-
 
 ## TO-DO List
 - [x] RAG evaluation

{evalscope-0.5.5 → evalscope-0.6.0}/README.md

@@ -1,8 +1,11 @@
-English | [简体中文](README_zh.md)
 
 
 ![](docs/en/_static/images/evalscope_logo.png)
 
+<p align="center">
+    English | <a href="README_zh.md">简体中文</a>
+</p>
+
 <p align="center">
 <a href="https://badge.fury.io/py/evalscope"><img src="https://badge.fury.io/py/evalscope.svg" alt="PyPI version" height="18"></a>
 <a href="https://pypi.org/project/evalscope"><img alt="PyPI - Downloads" src="https://static.pepy.tech/badge/evalscope">
@@ -11,7 +14,7 @@ English | [简体中文](README_zh.md)
     <img src='https://readthedocs.org/projects/evalscope-en/badge/?version=latest' alt='Documentation Status' />
 </a>
 <br>
- <a href="https://evalscope.readthedocs.io/en/latest/"><span style="font-size: 16px;">📖 Documents</span></a> &nbsp | &nbsp<a href="https://evalscope.readthedocs.io/zh-cn/latest/"><span style="font-size: 16px;"> 📖  中文文档</span></a>
+ <a href="https://evalscope.readthedocs.io/en/latest/">📖 Documents</a>
 <p>
 
 
@@ -25,7 +28,7 @@ English | [简体中文](README_zh.md)
 - [Offline Evaluation](#offline-evaluation)
 - [Arena Mode](#arena-mode)
 - [Model Serving Performance Evaluation](#Model-Serving-Performance-Evaluation)
-- [Leaderboard](#leaderboard)
+
 
 ## 📝 Introduction
 
@@ -51,6 +54,8 @@ The architecture includes the following modules:
 
 
 ## 🎉 News
+- 🔥 **[2024.10.31]** The best practice for evaluating Multimodal-RAG has been updated, please check the [📖 Blog](https://evalscope.readthedocs.io/zh-cn/latest/blog/RAG/multimodal_RAG.html#multimodal-rag) for more details.
+- 🔥 **[2024.10.23]** Supports multimodal RAG evaluation, including the assessment of image-text retrieval using [CLIP_Benchmark](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/clip_benchmark.html), and extends [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html) to support end-to-end multimodal metrics evaluation.
 - 🔥 **[2024.10.8]** Support for RAG evaluation, including independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html).
 - 🔥 **[2024.09.18]** Our documentation has been updated to include a blog module, featuring some technical research and discussions related to evaluations. We invite you to [📖 read it](https://evalscope.readthedocs.io/en/refact_readme/blog/index.html).
 - 🔥 **[2024.09.12]** Support for LongWriter evaluation, which supports 10,000+ word generation. You can use the benchmark [LongBench-Write](evalscope/third_party/longbench_write/README.md) to measure the long output quality as well as the output length.
@@ -247,11 +252,6 @@ A stress testing tool that focuses on large language models and can be customize
 Refer to : Model Serving Performance Evaluation [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test.html)
 
 
-## Leaderboard
-The LLM Leaderboard aims to provide an objective and comprehensive evaluation standard and platform to help researchers and developers understand and compare the performance of models on various tasks on ModelScope.
-
-Refer to : [Leaderboard](https://modelscope.cn/leaderboard/58/ranking?type=free)
-
 
 ## TO-DO List
 - [x] RAG evaluation

{evalscope-0.5.5 → evalscope-0.6.0}/evalscope/backend/opencompass/tasks/eval_datasets.py

@@ -50,12 +50,12 @@ with read_base():
     from opencompass.configs.datasets.nq.nq_gen_c788f6 import nq_datasets
     from opencompass.configs.datasets.triviaqa.triviaqa_gen_2121ce import triviaqa_datasets
     from opencompass.configs.datasets.cmb.cmb_gen_dfb5c4 import cmb_datasets
-    from opencompass.configs.datasets.bbh.bbh_gen_5b92b0 import bbh_datasets
 
     # Note: to be supported
     # from opencompass.configs.datasets.flores.flores_gen_806ede import flores_datasets
     # from opencompass.configs.datasets.TheoremQA.TheoremQA_5shot_gen_6f0af8 import TheoremQA_datasets
     # from opencompass.configs.datasets.commonsenseqa.commonsenseqa_gen_c946f2 import commonsenseqa_datasets
+    # from opencompass.configs.datasets.bbh.bbh_gen_5b92b0 import bbh_datasets
 
 
 datasets = []

evalscope-0.6.0/evalscope/backend/rag_eval/__init__.py

@@ -0,0 +1,4 @@
+from evalscope.backend.rag_eval.utils.embedding import EmbeddingModel
+from evalscope.backend.rag_eval.utils.llm import LLM, LocalLLM, ChatOpenAI
+from evalscope.backend.rag_eval.utils.clip import VisionModel
+from evalscope.backend.rag_eval.backend_manager import RAGEvalBackendManager

{evalscope-0.5.5 → evalscope-0.6.0}/evalscope/backend/rag_eval/backend_manager.py

@@ -24,45 +24,57 @@ class RAGEvalBackendManager(BackendManager):
         else:
             logger.error(f"Please install `{module_name}` first")
 
-    def run_mteb(self):
+    @staticmethod
+    def run_mteb(model_args, eval_args):
         from evalscope.backend.rag_eval.cmteb import ModelArguments, EvalArguments
         from evalscope.backend.rag_eval.cmteb import one_stage_eval, two_stage_eval
 
-        if len(self.model_args) > 2:
+        if len(model_args) > 2:
             raise ValueError("Not support multiple models yet")
 
         # Convert arguments to dictionary
-        model_args_list = [ModelArguments(**args).to_dict() for args in self.model_args]
-        eval_args = EvalArguments(**self.eval_args).to_dict()
+        model_args_list = [ModelArguments(**args).to_dict() for args in model_args]
+        eval_args = EvalArguments(**eval_args).to_dict()
 
         if len(model_args_list) == 1:
             one_stage_eval(model_args_list[0], eval_args)
         else:  # len(model_args_list) == 2
             two_stage_eval(model_args_list[0], model_args_list[1], eval_args)
 
-    def run_ragas(self):
-        from evalscope.backend.rag_eval.ragas import rag_eval, testset_generation
+    @staticmethod
+    def run_ragas(testset_args, eval_args):
+        from evalscope.backend.rag_eval.ragas import rag_eval
+        from evalscope.backend.rag_eval.ragas.tasks import generate_testset
         from evalscope.backend.rag_eval.ragas import (
             TestsetGenerationArguments,
             EvaluationArguments,
         )
 
-        if self.testset_args is not None:
-            testset_generation(TestsetGenerationArguments(**self.testset_args))
-        if self.eval_args is not None:
-            rag_eval(EvaluationArguments(**self.eval_args))
+        if testset_args is not None:
+            generate_testset(TestsetGenerationArguments(**testset_args))
+        if eval_args is not None:
+            rag_eval(EvaluationArguments(**eval_args))
+
+    @staticmethod
+    def run_clip_benchmark(args):
+        from evalscope.backend.rag_eval.clip_benchmark import Arguments, evaluate
+
+        evaluate(Arguments(**args))
 
     def run(self, *args, **kwargs):
         tool = self.config_d.pop("tool")
         if tool.lower() == "mteb":
             self._check_env("mteb")
-            self.model_args = self.config_d["model"]
-            self.eval_args = self.config_d["eval"]
-            self.run_mteb()
+            model_args = self.config_d["model"]
+            eval_args = self.config_d["eval"]
+            self.run_mteb(model_args, eval_args)
         elif tool.lower() == "ragas":
             self._check_env("ragas")
-            self.testset_args = self.config_d.get("testset_generation", None)
-            self.eval_args = self.config_d.get("eval", None)
-            self.run_ragas()
+            testset_args = self.config_d.get("testset_generation", None)
+            eval_args = self.config_d.get("eval", None)
+            self.run_ragas(testset_args, eval_args)
+        elif tool.lower() == "clip_benchmark":
+            self._check_env("webdataset")
+            self.run_clip_benchmark(self.config_d["eval"])
         else:
             raise ValueError(f"Unknown tool: {tool}")

evalscope-0.6.0/evalscope/backend/rag_eval/clip_benchmark/__init__.py

@@ -0,0 +1,2 @@
+from evalscope.backend.rag_eval.clip_benchmark.task_template import evaluate
+from evalscope.backend.rag_eval.clip_benchmark.arguments import Arguments

evalscope-0.6.0/evalscope/backend/rag_eval/clip_benchmark/arguments.py

@@ -0,0 +1,34 @@
+from dataclasses import dataclass, field
+from typing import List, Dict
+
+
+@dataclass
+class Arguments:
+    # fmt: off
+    """
+    A dataclass to store and manage the arguments for the model configuration and data processing.
+    """
+    """
+    For CLIP model support, you can use the following fields:
+        model_name: str
+        revision: str = "master"
+        hub: str = "modelscope"
+        
+    For API VLM model support, you can use the following fields, (image caption only):
+        model_name="gpt-4o-mini"
+        api_base: str = "",
+        api_key: Optional[str] = None
+        prompt: str = None
+    """
+    models: List[Dict] = field(default_factory=dict)  # List of paths to the pre-trained models or model identifiers
+    dataset_name: List[str] = field(default_factory=list)  # List of dataset names to be used
+    data_dir: str = None  # Root directory where the datasets are stored
+    split: str = "test"  # Split of the dataset to be used (e.g., 'train', 'validation', 'test')
+    task: str = None
+    batch_size: int = 128  # Batch size for data loading
+    num_workers: int = 1  # Number of workers for data loading
+    verbose: bool = True  # Flag to enable verbose logging
+    output_dir: str = "outputs"  # Directory where the outputs (e.g., predictions, logs) will be saved
+    cache_dir: str = "cache"  # Directory where the dataset cache will be stored
+    skip_existing: bool = False  # Flag to skip processing if outputs already exist
+    limit: int = None # Limit the number of samples to be processed

evalscope-0.6.0/evalscope/backend/rag_eval/clip_benchmark/dataset_builder.py

@@ -0,0 +1,277 @@
+import os
+import torch
+from torch.utils.data import DataLoader, Dataset as TorchDataset
+from evalscope.utils.logger import get_logger
+
+
+logger = get_logger()
+
+
+def build_dataset(
+    dataset_name,
+    root=None,
+    transform=None,
+    split="test",
+    wds_cache_dir=None,
+    **kwargs,
+):
+    """
+    Main function to use in order to build a dataset instance,
+
+    dataset_name: str
+        name of the dataset
+
+    root: str
+        root folder where the dataset is downloaded and stored. can be shared among datasets.
+
+    transform: torchvision transform applied to images
+
+    split: str
+        split to use, depending on the dataset can have different options.
+        In general, `train` and `test` are available.
+        For specific splits, please look at the corresponding dataset.
+
+    custom_classname_file: str or None
+        Custom classname file where keys are dataset names and values are list of classnames.
+
+    custom_template_file: str or None
+        Custom template file where keys are dataset names and values are list of prompts, or dicts
+        where keys are classnames and values are class-specific prompts.
+
+    """
+
+    if dataset_name == "dummy":
+        ds = Dummy()
+    elif dataset_name == "custom":
+        ds = build_custom_dataset(dataset_name, data_dir=root, transform=transform)
+    else:
+        # WebDataset support using `webdataset` library
+        ds = build_wds_dataset(
+            dataset_name,
+            transform=transform,
+            split=split,
+            data_dir=root,
+            cache_dir=wds_cache_dir,
+        )
+
+    return ds
+
+
+class Dummy:
+
+    def __init__(self):
+        self.classes = ["blank image", "noisy image"]
+
+    def __getitem__(self, i):
+        return torch.zeros(3, 224, 224), 0
+
+    def __len__(self):
+        return 1
+
+
+class DatasetWrapper(TorchDataset):
+    def __init__(self, dataset, transform=None, image_key="image", text_key="query"):
+        self.dataset = dataset
+        self.transform = transform
+        self.image_key = image_key
+        self.text_key = text_key
+
+    def __len__(self):
+        return len(self.dataset)
+
+    def __getitem__(self, idx):
+        item = self.dataset[idx]
+
+        # 加载图像
+        image = item[self.image_key]
+        if self.transform is not None:
+            image = self.transform(image, return_tensors="pt")
+
+        # 获取查询列表
+        query = item[self.text_key]
+        if isinstance(query, str):
+            query = [query]
+
+        return image, query
+
+
+def get_dataset_default_task(dataset):
+    if dataset in (
+        "custom",
+        "muge",
+        "flickr30k",
+        "flickr8k",
+        "mscoco_captions",
+        "mscoco_captions2017",
+        "multilingual_mscoco_captions",
+        "flickr30k-200",
+        "crossmodal3600",
+        "xtd200",
+    ):
+        return "zeroshot_retrieval"
+    else:
+        return "zeroshot_classification"
+
+
+def get_dataloader(dataset_name, dataset, batch_size, num_workers):
+    if dataset_name == "custom":
+        dataloader = DataLoader(
+            dataset,
+            batch_size=batch_size,
+            shuffle=False,
+            num_workers=num_workers,
+            collate_fn=image_captions_collate_fn,
+        )
+    else:
+        dataloader = DataLoader(
+            dataset.batched(batch_size),
+            batch_size=None,
+            shuffle=False,
+            num_workers=num_workers,
+        )
+    return dataloader
+
+
+def image_captions_collate_fn(batch):
+    transposed = list(zip(*batch))
+    imgs = transposed[0]
+    texts = transposed[1]
+    return imgs, texts
+
+
+def build_custom_dataset(dataset_name, data_dir, transform=None):
+    from datasets import load_dataset, Features, Image, Sequence, Value
+
+    qrels_ds = load_dataset(
+        "json",
+        data_files=os.path.join(data_dir, "image_queries.jsonl"),
+        features=Features(
+            {"image_path": Image(decode=True), "query": Sequence(Value("string"))}
+        ),
+        split="train",
+    )
+
+    dataset = DatasetWrapper(
+        qrels_ds, transform, image_key="image_path", text_key="query"
+    )
+    return dataset
+
+
+def build_wds_dataset(
+    dataset_name, transform, split="test", data_dir="root", cache_dir=None
+):
+    """
+    Load a dataset in WebDataset format. Either local paths or HTTP URLs can be specified.
+    Expected file structure is:
+    ```
+    data_dir/
+        train/
+            nshards.txt
+            0.tar
+            1.tar
+            ...
+        test/
+            nshards.txt
+            0.tar
+            1.tar
+            ...
+        classnames.txt
+        zeroshot_classification_templates.txt
+        dataset_type.txt
+    ```
+    Classnames and templates are required for zeroshot classification, while dataset type
+    (equal to "retrieval") is required for zeroshot retrieval datasets.
+
+    You can use the `clip_benchmark_export_wds` or corresponding API
+    (`clip_benchmark.webdataset_builder.convert_dataset`) to convert datasets to this format.
+
+    Set `cache_dir` to a path to cache the dataset, otherwise, no caching will occur.
+    """
+    import webdataset as wds
+
+    def read_txt(fname):
+        if "://" in fname:
+            stream = os.popen("curl -L -s --fail '%s'" % fname, "r")
+            value = stream.read()
+            if stream.close():
+                raise FileNotFoundError("Failed to retreive data")
+        else:
+            with open(fname, "r") as file:
+                value = file.read()
+        return value
+
+    if not data_dir:
+        data_dir = f"https://modelscope.cn/datasets/clip-benchmark/wds_{dataset_name}/resolve/master"
+
+    # Git LFS files have a different file path to access the raw data than other files
+    if data_dir.startswith("https://modelscope.cn/datasets"):
+        *split_url_head, _, url_path = data_dir.split("/", 7)
+        url_head = "/".join(split_url_head)
+        metadata_dir = "/".join([url_head, "resolve", url_path])
+        tardata_dir = "/".join([url_head, "resolve", url_path])
+    else:
+        metadata_dir = tardata_dir = data_dir
+    # Get number of shards
+    nshards_fname = os.path.join(metadata_dir, split, "nshards.txt")
+    nshards = int(
+        read_txt(nshards_fname)
+    )  # Do not catch FileNotFound, nshards.txt should be mandatory
+    
+    # Get dataset type (classification or retrieval)
+    type_fname = os.path.join(metadata_dir, "dataset_type.txt")
+    try:
+        dataset_type = read_txt(type_fname).strip().lower()
+    except FileNotFoundError:
+        dataset_type = "classification"
+    
+    filepattern = os.path.join(tardata_dir, split, "{0..%d}.tar" % (nshards - 1))
+    # Load webdataset (support WEBP, PNG, and JPG for now)
+    if not cache_dir or not isinstance(cache_dir, str):
+        cache_dir = None
+    else:
+        os.makedirs(cache_dir, exist_ok=True)
+    dataset = wds.WebDataset(
+        filepattern,
+        cache_dir=cache_dir,
+        nodesplitter=lambda src: src,
+        shardshuffle=False,
+        verbose=True,
+    ).decode(
+        wds.autodecode.ImageHandler("pil", extensions=["webp", "png", "jpg", "jpeg"])
+    )
+    
+    # Load based on classification or retrieval task
+    if dataset_type == "retrieval":
+        dataset = dataset.to_tuple(["webp", "png", "jpg", "jpeg"], "txt").map_tuple(
+            transform, str.splitlines
+        )
+        dataset.classes = dataset.templates = None
+    else:
+        label_type = (
+            "npy" if dataset_type == "multilabel" else "cls"
+        )  # Special case for multilabel
+        dataset = dataset.to_tuple(
+            ["webp", "png", "jpg", "jpeg"], label_type
+        ).map_tuple(transform, None)
+        # Get class names if present
+        classnames_fname = os.path.join(metadata_dir, "classnames.txt")
+        try:
+            dataset.classes = [
+                line.strip() for line in read_txt(classnames_fname).splitlines()
+            ]
+        except FileNotFoundError:
+            logger.warning("WARNING: classnames.txt not found")
+            dataset.classes = None
+        # Get zeroshot classification templates if present
+        templates_fname = os.path.join(
+            metadata_dir, "zeroshot_classification_templates.txt"
+        )
+        try:
+            dataset.templates = [
+                line.strip() for line in read_txt(templates_fname).splitlines()
+            ]
+        except FileNotFoundError:
+            logger.warning("WARNING: zeroshot_classification_templates.txt not found")
+            dataset.templates = None
+
+    return dataset

evalscope-0.6.0/evalscope/backend/rag_eval/clip_benchmark/task_template.py

@@ -0,0 +1,119 @@
+import os
+import torch
+import json
+from itertools import product
+
+from evalscope.backend.rag_eval.clip_benchmark.dataset_builder import (
+    build_dataset,
+    get_dataset_default_task,
+    get_dataloader,
+)
+from evalscope.backend.rag_eval.clip_benchmark.tasks import (
+    zeroshot_classification,
+    zeroshot_retrieval,
+    image_caption,
+)
+from evalscope.backend.rag_eval.clip_benchmark.arguments import Arguments
+from evalscope.backend.rag_eval.utils.clip import VisionModel
+from evalscope.utils.logger import get_logger
+
+logger = get_logger()
+
+
+def evaluate(args: Arguments):
+    models = args.models
+    dataset_names = args.dataset_name
+    data_dir = args.data_dir
+    split = args.split
+    batch_size = args.batch_size
+    num_workers = args.num_workers
+    verbose = args.verbose
+    input_task = args.task
+    output_dir = args.output_dir
+    cache_dir = args.cache_dir
+    skip_existing = args.skip_existing
+    limit = args.limit
+
+    # Iterate over model and dataset combinations
+    for model_cfg, dataset_name in product(models, dataset_names):
+        task = input_task or get_dataset_default_task(dataset_name)
+        model_name = os.path.basename(model_cfg["model_name"])
+
+        output_path = os.path.join(output_dir, model_name)
+        os.makedirs(output_path, exist_ok=True)
+        output_file = os.path.join(output_path, f"{dataset_name}_{task}.json")
+
+        # Skip evaluation if the result already exists and skip_existing is True
+        if os.path.exists(output_file) and skip_existing:
+            if verbose:
+                logger.info(f"Skip {output_dir}, exists already.")
+            return
+
+        # Determine device (CPU or GPU)
+        device = "cuda" if torch.cuda.is_available() else "cpu"
+        model_cfg["device"] = device
+        # Initialize the model
+        model = VisionModel.load(**model_cfg)
+
+        # Build the dataset
+        dataset = build_dataset(
+            dataset_name=dataset_name,
+            root=data_dir,
+            transform=model.transform,
+            split=split,
+            wds_cache_dir=f"{cache_dir}/{dataset_name}",
+        )
+
+        # Create the dataloader
+        dataloader = get_dataloader(dataset_name, dataset, batch_size, num_workers)
+
+        # Evaluate based on the task
+        if task == "zeroshot_classification":
+            zeroshot_templates = (
+                dataset.templates if hasattr(dataset, "templates") else None
+            )
+            if verbose:
+                logger.info(f"Zero-shot templates: {zeroshot_templates}")
+            classnames = dataset.classes if hasattr(dataset, "classes") else None
+            assert (
+                zeroshot_templates is not None and classnames is not None
+            ), "Dataset does not support classification"
+            metrics = zeroshot_classification.evaluate(
+                model,
+                dataloader,
+                classnames,
+                zeroshot_templates,
+                device=device,
+                verbose=verbose,
+                limit=limit,
+            )
+        elif task == "zeroshot_retrieval":
+            metrics = zeroshot_retrieval.evaluate(
+                model, dataloader, recall_k_list=[5], device=device, limit=limit
+            )
+        elif task == "image_caption":
+            output_path = os.path.join(output_path, dataset_name, "retrieval_data")
+            metrics = image_caption.evaluate(
+                model, dataloader, limit=limit, output_path=output_path
+            )
+
+        # Prepare dump data
+        dump = {
+            "dataset": dataset_name,
+            "model": model_name,
+            "task": task,
+            "metrics": metrics,
+        }
+
+        if verbose:
+            logger.info(f"Evaluation results: {dump}")
+
+        # Write the results to output file
+        if verbose:
+            logger.info(f"Dump results to: {output_file}")
+        with open(output_file, "w") as f:
+            json.dump(dump, f)
+
+
+if __name__ == "__main__":
+    evaluate()