evalscope 0.5.4__tar.gz → 0.5.5__tar.gz

{evalscope-0.5.4 → evalscope-0.5.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: evalscope
-Version: 0.5.4
+Version: 0.5.5
 Summary: EvalScope: Lightweight LLMs Evaluation Framework
 Home-page: https://github.com/modelscope/evalscope
 Author: ModelScope team
@@ -24,7 +24,7 @@ Requires-Dist: editdistance
 Requires-Dist: jsonlines
 Requires-Dist: matplotlib
 Requires-Dist: modelscope[framework]
-Requires-Dist: nltk
+Requires-Dist: nltk>=3.9
 Requires-Dist: openai
 Requires-Dist: pandas
 Requires-Dist: plotly
@@ -34,7 +34,7 @@ Requires-Dist: pyyaml
 Requires-Dist: regex
 Requires-Dist: requests
 Requires-Dist: requests-toolbelt
-Requires-Dist: rouge-score
+Requires-Dist: rouge-score>=0.1.0
 Requires-Dist: sacrebleu
 Requires-Dist: scikit-learn
 Requires-Dist: seaborn
@@ -48,9 +48,12 @@ Requires-Dist: transformers_stream_generator
 Requires-Dist: jieba
 Requires-Dist: rouge-chinese
 Provides-Extra: opencompass
-Requires-Dist: ms-opencompass>=0.1.0; extra == "opencompass"
+Requires-Dist: ms-opencompass>=0.1.1; extra == "opencompass"
 Provides-Extra: vlmeval
 Requires-Dist: ms-vlmeval>=0.0.5; extra == "vlmeval"
+Provides-Extra: rag
+Requires-Dist: ragas; extra == "rag"
+Requires-Dist: mteb>=0.14.16; extra == "rag"
 Provides-Extra: inner
 Requires-Dist: absl-py; extra == "inner"
 Requires-Dist: accelerate; extra == "inner"
@@ -88,7 +91,7 @@ Requires-Dist: editdistance; extra == "all"
 Requires-Dist: jsonlines; extra == "all"
 Requires-Dist: matplotlib; extra == "all"
 Requires-Dist: modelscope[framework]; extra == "all"
-Requires-Dist: nltk; extra == "all"
+Requires-Dist: nltk>=3.9; extra == "all"
 Requires-Dist: openai; extra == "all"
 Requires-Dist: pandas; extra == "all"
 Requires-Dist: plotly; extra == "all"
@@ -98,7 +101,7 @@ Requires-Dist: pyyaml; extra == "all"
 Requires-Dist: regex; extra == "all"
 Requires-Dist: requests; extra == "all"
 Requires-Dist: requests-toolbelt; extra == "all"
-Requires-Dist: rouge-score; extra == "all"
+Requires-Dist: rouge-score>=0.1.0; extra == "all"
 Requires-Dist: sacrebleu; extra == "all"
 Requires-Dist: scikit-learn; extra == "all"
 Requires-Dist: seaborn; extra == "all"
@@ -111,8 +114,10 @@ Requires-Dist: transformers>=4.33; extra == "all"
 Requires-Dist: transformers_stream_generator; extra == "all"
 Requires-Dist: jieba; extra == "all"
 Requires-Dist: rouge-chinese; extra == "all"
-Requires-Dist: ms-opencompass>=0.1.0; extra == "all"
+Requires-Dist: ms-opencompass>=0.1.1; extra == "all"
 Requires-Dist: ms-vlmeval>=0.0.5; extra == "all"
+Requires-Dist: ragas; extra == "all"
+Requires-Dist: mteb>=0.14.16; extra == "all"
 
 English | [简体中文](README_zh.md)
 
@@ -145,30 +150,11 @@ English | [简体中文](README_zh.md)
 
 ## 📝 Introduction
 
-Large Model (including Large Language Models, Multi-modal Large Language Models) evaluation has become a critical process for assessing and improving LLMs. To better support the evaluation of large models, we propose the EvalScope framework.
-
-### Framework Features
-- **Benchmark Datasets**: Preloaded with several commonly used test benchmarks, including MMLU, CMMLU, C-Eval, GSM8K, ARC, HellaSwag, TruthfulQA, MATH, HumanEval, etc.
-- **Evaluation Metrics**: Implements various commonly used evaluation metrics.
-- **Model Access**: A unified model access mechanism that is compatible with the Generate and Chat interfaces of multiple model families.
-- **Automated Evaluation**: Includes automatic evaluation of objective questions and complex task evaluation using expert models.
-- **Evaluation Reports**: Automatically generates evaluation reports.
-- **Arena Mode**: Used for comparisons between models and objective evaluation of models, supporting various evaluation modes, including:
-  - **Single mode**: Scoring a single model.
-  - **Pairwise-baseline mode**: Comparing against a baseline model.
-  - **Pairwise (all) mode**: Pairwise comparison among all models.
-- **Visualization Tools**: Provides intuitive displays of evaluation results.
-- **Model Performance Evaluation**: Offers a performance testing tool for model inference services and detailed statistics, see [Model Performance Evaluation Documentation](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test.html).
-- **OpenCompass Integration**: Supports OpenCompass as the evaluation backend, providing advanced encapsulation and task simplification, allowing for easier task submission for evaluation.
-- **VLMEvalKit Integration**: Supports VLMEvalKit as the evaluation backend, facilitating the initiation of multi-modal evaluation tasks, supporting various multi-modal models and datasets.
-- **Full-Link Support**: Through seamless integration with the [ms-swift](https://github.com/modelscope/ms-swift) training framework, provides a one-stop development process for model training, model deployment, model evaluation, and report viewing, enhancing user development efficiency.
-
-
-<details><summary>Overall Architecture</summary>
+EvalScope is the official model evaluation and performance benchmarking framework launched by the [ModelScope](https://modelscope.cn/) community. It comes with built-in common benchmarks and evaluation metrics, such as MMLU, CMMLU, C-Eval, GSM8K, ARC, HellaSwag, TruthfulQA, MATH, and HumanEval. EvalScope supports various types of model evaluations, including LLMs, multimodal LLMs, embedding models, and reranker models. It is also applicable to multiple evaluation scenarios, such as end-to-end RAG evaluation, arena mode, and model inference performance stress testing. Moreover, with the seamless integration of the ms-swift training framework, evaluations can be initiated with a single click, providing full end-to-end support from model training to evaluation 🚀
 
 <p align="center">
   <img src="docs/en/_static/images/evalscope_framework.png" width="70%">
-  <br>Fig 1. EvalScope Framework.
+  <br>EvalScope Framework.
 </p>
 
 The architecture includes the following modules:
@@ -178,14 +164,15 @@ The architecture includes the following modules:
     - **Native**: EvalScope’s own **default evaluation framework**, supporting various evaluation modes, including single model evaluation, arena mode, baseline model comparison mode, etc.
     - **OpenCompass**: Supports [OpenCompass](https://github.com/open-compass/opencompass) as the evaluation backend, providing advanced encapsulation and task simplification, allowing you to submit tasks for evaluation more easily.
     - **VLMEvalKit**: Supports [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) as the evaluation backend, enabling easy initiation of multi-modal evaluation tasks, supporting various multi-modal models and datasets.
+    - **RAGEval**: Supports RAG evaluation, supporting independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html).
     - **ThirdParty**: Other third-party evaluation tasks, such as ToolBench.
 4. **Performance Evaluator**: Model performance evaluation, responsible for measuring model inference service performance, including performance testing, stress testing, performance report generation, and visualization.
 5. **Evaluation Report**: The final generated evaluation report summarizes the model's performance, which can be used for decision-making and further model optimization.
 6. **Visualization**: Visualization results help users intuitively understand evaluation results, facilitating analysis and comparison of different model performances.
-</details>
 
 
 ## 🎉 News
+- 🔥 **[2024.10.8]** Support for RAG evaluation, including independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html).
 - 🔥 **[2024.09.18]** Our documentation has been updated to include a blog module, featuring some technical research and discussions related to evaluations. We invite you to [📖 read it](https://evalscope.readthedocs.io/en/refact_readme/blog/index.html).
 - 🔥 **[2024.09.12]** Support for LongWriter evaluation, which supports 10,000+ word generation. You can use the benchmark [LongBench-Write](evalscope/third_party/longbench_write/README.md) to measure the long output quality as well as the output length.
 - 🔥 **[2024.08.30]** Support for custom dataset evaluations, including text datasets and multimodal image-text datasets.
@@ -356,9 +343,10 @@ run_task(task_cfg=your_task_cfg)
 ## Evaluation Backend
 EvalScope supports using third-party evaluation frameworks to initiate evaluation tasks, which we call Evaluation Backend. Currently supported Evaluation Backend includes:
 - **Native**: EvalScope's own **default evaluation framework**, supporting various evaluation modes including single model evaluation, arena mode, and baseline model comparison mode.
-- [OpenCompass](https://github.com/open-compass/opencompass): Initiate OpenCompass evaluation tasks through EvalScope. Lightweight, easy to customize, supports seamless integration with the LLM fine-tuning framework ms-swift. [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/opencompass_backend.html)
-- [VLMEvalKit](https://github.com/open-compass/VLMEvalKit): Initiate VLMEvalKit multimodal evaluation tasks through EvalScope. Supports various multimodal models and datasets, and offers seamless integration with the LLM fine-tuning framework ms-swift. [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/vlmevalkit_backend.html)
-- **ThirdParty**: The third-party task, e.g. [ToolBench](https://evalscope.readthedocs.io/en/latest/third_party/toolbench.html), you can contribute your own evaluation task to EvalScope as third-party backend.
+- [OpenCompass](https://github.com/open-compass/opencompass): Initiate OpenCompass evaluation tasks through EvalScope. Lightweight, easy to customize, supports seamless integration with the LLM fine-tuning framework ms-swift. [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/opencompass_backend.html)
+- [VLMEvalKit](https://github.com/open-compass/VLMEvalKit): Initiate VLMEvalKit multimodal evaluation tasks through EvalScope. Supports various multimodal models and datasets, and offers seamless integration with the LLM fine-tuning framework ms-swift. [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/vlmevalkit_backend.html)
+- **RAGEval**: Initiate RAG evaluation tasks through EvalScope, supporting independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html): [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/index.html)
+- **ThirdParty**: Third-party evaluation tasks, such as [ToolBench](https://evalscope.readthedocs.io/en/latest/third_party/toolbench.html) and [LongBench-Write](https://evalscope.readthedocs.io/en/latest/third_party/longwriter.html).
 
 ## Custom Dataset Evaluation
 EvalScope supports custom dataset evaluation. For detailed information, please refer to the Custom Dataset Evaluation [📖User Guide](https://evalscope.readthedocs.io/en/latest/advanced_guides/custom_dataset.html)
@@ -387,6 +375,8 @@ Refer to : [Leaderboard](https://modelscope.cn/leaderboard/58/ranking?type=free)
 
 
 ## TO-DO List
+- [x] RAG evaluation
+- [x] VLM evaluation
 - [x] Agents evaluation
 - [x] vLLM
 - [ ] Distributed evaluating
@@ -398,3 +388,7 @@ Refer to : [Leaderboard](https://modelscope.cn/leaderboard/58/ranking?type=free)
 - [ ] Auto-reviewer
   - [ ] Qwen-max
 
+
+## Star History
+
+[![Star History Chart](https://api.star-history.com/svg?repos=modelscope/evalscope&type=Date)](https://star-history.com/#modelscope/evalscope&Date)

{evalscope-0.5.4 → evalscope-0.5.5}/README.md

@@ -29,30 +29,11 @@ English | [简体中文](README_zh.md)
 
 ## 📝 Introduction
 
-Large Model (including Large Language Models, Multi-modal Large Language Models) evaluation has become a critical process for assessing and improving LLMs. To better support the evaluation of large models, we propose the EvalScope framework.
-
-### Framework Features
-- **Benchmark Datasets**: Preloaded with several commonly used test benchmarks, including MMLU, CMMLU, C-Eval, GSM8K, ARC, HellaSwag, TruthfulQA, MATH, HumanEval, etc.
-- **Evaluation Metrics**: Implements various commonly used evaluation metrics.
-- **Model Access**: A unified model access mechanism that is compatible with the Generate and Chat interfaces of multiple model families.
-- **Automated Evaluation**: Includes automatic evaluation of objective questions and complex task evaluation using expert models.
-- **Evaluation Reports**: Automatically generates evaluation reports.
-- **Arena Mode**: Used for comparisons between models and objective evaluation of models, supporting various evaluation modes, including:
-  - **Single mode**: Scoring a single model.
-  - **Pairwise-baseline mode**: Comparing against a baseline model.
-  - **Pairwise (all) mode**: Pairwise comparison among all models.
-- **Visualization Tools**: Provides intuitive displays of evaluation results.
-- **Model Performance Evaluation**: Offers a performance testing tool for model inference services and detailed statistics, see [Model Performance Evaluation Documentation](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test.html).
-- **OpenCompass Integration**: Supports OpenCompass as the evaluation backend, providing advanced encapsulation and task simplification, allowing for easier task submission for evaluation.
-- **VLMEvalKit Integration**: Supports VLMEvalKit as the evaluation backend, facilitating the initiation of multi-modal evaluation tasks, supporting various multi-modal models and datasets.
-- **Full-Link Support**: Through seamless integration with the [ms-swift](https://github.com/modelscope/ms-swift) training framework, provides a one-stop development process for model training, model deployment, model evaluation, and report viewing, enhancing user development efficiency.
-
-
-<details><summary>Overall Architecture</summary>
+EvalScope is the official model evaluation and performance benchmarking framework launched by the [ModelScope](https://modelscope.cn/) community. It comes with built-in common benchmarks and evaluation metrics, such as MMLU, CMMLU, C-Eval, GSM8K, ARC, HellaSwag, TruthfulQA, MATH, and HumanEval. EvalScope supports various types of model evaluations, including LLMs, multimodal LLMs, embedding models, and reranker models. It is also applicable to multiple evaluation scenarios, such as end-to-end RAG evaluation, arena mode, and model inference performance stress testing. Moreover, with the seamless integration of the ms-swift training framework, evaluations can be initiated with a single click, providing full end-to-end support from model training to evaluation 🚀
 
 <p align="center">
   <img src="docs/en/_static/images/evalscope_framework.png" width="70%">
-  <br>Fig 1. EvalScope Framework.
+  <br>EvalScope Framework.
 </p>
 
 The architecture includes the following modules:
@@ -62,14 +43,15 @@ The architecture includes the following modules:
     - **Native**: EvalScope’s own **default evaluation framework**, supporting various evaluation modes, including single model evaluation, arena mode, baseline model comparison mode, etc.
     - **OpenCompass**: Supports [OpenCompass](https://github.com/open-compass/opencompass) as the evaluation backend, providing advanced encapsulation and task simplification, allowing you to submit tasks for evaluation more easily.
     - **VLMEvalKit**: Supports [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) as the evaluation backend, enabling easy initiation of multi-modal evaluation tasks, supporting various multi-modal models and datasets.
+    - **RAGEval**: Supports RAG evaluation, supporting independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html).
     - **ThirdParty**: Other third-party evaluation tasks, such as ToolBench.
 4. **Performance Evaluator**: Model performance evaluation, responsible for measuring model inference service performance, including performance testing, stress testing, performance report generation, and visualization.
 5. **Evaluation Report**: The final generated evaluation report summarizes the model's performance, which can be used for decision-making and further model optimization.
 6. **Visualization**: Visualization results help users intuitively understand evaluation results, facilitating analysis and comparison of different model performances.
-</details>
 
 
 ## 🎉 News
+- 🔥 **[2024.10.8]** Support for RAG evaluation, including independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html).
 - 🔥 **[2024.09.18]** Our documentation has been updated to include a blog module, featuring some technical research and discussions related to evaluations. We invite you to [📖 read it](https://evalscope.readthedocs.io/en/refact_readme/blog/index.html).
 - 🔥 **[2024.09.12]** Support for LongWriter evaluation, which supports 10,000+ word generation. You can use the benchmark [LongBench-Write](evalscope/third_party/longbench_write/README.md) to measure the long output quality as well as the output length.
 - 🔥 **[2024.08.30]** Support for custom dataset evaluations, including text datasets and multimodal image-text datasets.
@@ -240,9 +222,10 @@ run_task(task_cfg=your_task_cfg)
 ## Evaluation Backend
 EvalScope supports using third-party evaluation frameworks to initiate evaluation tasks, which we call Evaluation Backend. Currently supported Evaluation Backend includes:
 - **Native**: EvalScope's own **default evaluation framework**, supporting various evaluation modes including single model evaluation, arena mode, and baseline model comparison mode.
-- [OpenCompass](https://github.com/open-compass/opencompass): Initiate OpenCompass evaluation tasks through EvalScope. Lightweight, easy to customize, supports seamless integration with the LLM fine-tuning framework ms-swift. [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/opencompass_backend.html)
-- [VLMEvalKit](https://github.com/open-compass/VLMEvalKit): Initiate VLMEvalKit multimodal evaluation tasks through EvalScope. Supports various multimodal models and datasets, and offers seamless integration with the LLM fine-tuning framework ms-swift. [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/vlmevalkit_backend.html)
-- **ThirdParty**: The third-party task, e.g. [ToolBench](https://evalscope.readthedocs.io/en/latest/third_party/toolbench.html), you can contribute your own evaluation task to EvalScope as third-party backend.
+- [OpenCompass](https://github.com/open-compass/opencompass): Initiate OpenCompass evaluation tasks through EvalScope. Lightweight, easy to customize, supports seamless integration with the LLM fine-tuning framework ms-swift. [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/opencompass_backend.html)
+- [VLMEvalKit](https://github.com/open-compass/VLMEvalKit): Initiate VLMEvalKit multimodal evaluation tasks through EvalScope. Supports various multimodal models and datasets, and offers seamless integration with the LLM fine-tuning framework ms-swift. [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/vlmevalkit_backend.html)
+- **RAGEval**: Initiate RAG evaluation tasks through EvalScope, supporting independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html): [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/index.html)
+- **ThirdParty**: Third-party evaluation tasks, such as [ToolBench](https://evalscope.readthedocs.io/en/latest/third_party/toolbench.html) and [LongBench-Write](https://evalscope.readthedocs.io/en/latest/third_party/longwriter.html).
 
 ## Custom Dataset Evaluation
 EvalScope supports custom dataset evaluation. For detailed information, please refer to the Custom Dataset Evaluation [📖User Guide](https://evalscope.readthedocs.io/en/latest/advanced_guides/custom_dataset.html)
@@ -271,6 +254,8 @@ Refer to : [Leaderboard](https://modelscope.cn/leaderboard/58/ranking?type=free)
 
 
 ## TO-DO List
+- [x] RAG evaluation
+- [x] VLM evaluation
 - [x] Agents evaluation
 - [x] vLLM
 - [ ] Distributed evaluating
@@ -282,3 +267,7 @@ Refer to : [Leaderboard](https://modelscope.cn/leaderboard/58/ranking?type=free)
 - [ ] Auto-reviewer
   - [ ] Qwen-max
 
+
+## Star History
+
+[![Star History Chart](https://api.star-history.com/svg?repos=modelscope/evalscope&type=Date)](https://star-history.com/#modelscope/evalscope&Date)

{evalscope-0.5.4 → evalscope-0.5.5}/evalscope/backend/opencompass/tasks/eval_datasets.py

@@ -7,7 +7,7 @@ with read_base():
     from opencompass.configs.datasets.agieval.agieval_gen_64afd3 import agieval_datasets
     from opencompass.configs.datasets.GaokaoBench.GaokaoBench_gen_5cfe9e import GaokaoBench_datasets
     from opencompass.configs.datasets.humaneval.humaneval_gen_8e312c import humaneval_datasets
-    from opencompass.configs.datasets.mbpp.deprecated_mbpp_gen_1e1056 import mbpp_datasets
+    from opencompass.configs.datasets.mbpp.mbpp_gen_830460 import mbpp_datasets
     from opencompass.configs.datasets.CLUE_C3.CLUE_C3_gen_8c358f import C3_datasets
     from opencompass.configs.datasets.CLUE_CMRC.CLUE_CMRC_gen_1bd3c8 import CMRC_datasets
     from opencompass.configs.datasets.CLUE_DRCD.CLUE_DRCD_gen_1bd3c8 import DRCD_datasets
@@ -45,7 +45,7 @@ with read_base():
     from opencompass.configs.datasets.piqa.piqa_gen_1194eb import piqa_datasets
     from opencompass.configs.datasets.siqa.siqa_gen_e78df3 import siqa_datasets
     from opencompass.configs.datasets.strategyqa.strategyqa_gen_1180a7 import strategyqa_datasets
-    from opencompass.configs.datasets.winogrande.deprecated_winogrande_gen_a9ede5 import winogrande_datasets
+    from opencompass.configs.datasets.winogrande.winogrande_gen_458220 import winogrande_datasets
     from opencompass.configs.datasets.obqa.obqa_gen_9069e4 import obqa_datasets
     from opencompass.configs.datasets.nq.nq_gen_c788f6 import nq_datasets
     from opencompass.configs.datasets.triviaqa.triviaqa_gen_2121ce import triviaqa_datasets

evalscope-0.5.5/evalscope/backend/rag_eval/__init__.py

@@ -0,0 +1,3 @@
+from evalscope.backend.rag_eval.utils.embedding import EmbeddingModel
+from evalscope.backend.rag_eval.utils.llm import LLM
+from evalscope.backend.rag_eval.backend_manager import RAGEvalBackendManager

evalscope-0.5.5/evalscope/backend/rag_eval/backend_manager.py

@@ -0,0 +1,68 @@
+import os
+from typing import Optional, Union
+from evalscope.utils import is_module_installed, get_valid_list
+from evalscope.backend.base import BackendManager
+from evalscope.utils.logger import get_logger
+
+
+logger = get_logger()
+
+
+class RAGEvalBackendManager(BackendManager):
+    def __init__(self, config: Union[str, dict], **kwargs):
+        """BackendManager for VLM Evaluation Kit
+
+        Args:
+            config (Union[str, dict]): the configuration yaml-file or the configuration dictionary
+        """
+        super().__init__(config, **kwargs)
+
+    @staticmethod
+    def _check_env(module_name: str):
+        if is_module_installed(module_name):
+            logger.info(f"Check `{module_name}` Installed")
+        else:
+            logger.error(f"Please install `{module_name}` first")
+
+    def run_mteb(self):
+        from evalscope.backend.rag_eval.cmteb import ModelArguments, EvalArguments
+        from evalscope.backend.rag_eval.cmteb import one_stage_eval, two_stage_eval
+
+        if len(self.model_args) > 2:
+            raise ValueError("Not support multiple models yet")
+
+        # Convert arguments to dictionary
+        model_args_list = [ModelArguments(**args).to_dict() for args in self.model_args]
+        eval_args = EvalArguments(**self.eval_args).to_dict()
+
+        if len(model_args_list) == 1:
+            one_stage_eval(model_args_list[0], eval_args)
+        else:  # len(model_args_list) == 2
+            two_stage_eval(model_args_list[0], model_args_list[1], eval_args)
+
+    def run_ragas(self):
+        from evalscope.backend.rag_eval.ragas import rag_eval, testset_generation
+        from evalscope.backend.rag_eval.ragas import (
+            TestsetGenerationArguments,
+            EvaluationArguments,
+        )
+
+        if self.testset_args is not None:
+            testset_generation(TestsetGenerationArguments(**self.testset_args))
+        if self.eval_args is not None:
+            rag_eval(EvaluationArguments(**self.eval_args))
+
+    def run(self, *args, **kwargs):
+        tool = self.config_d.pop("tool")
+        if tool.lower() == "mteb":
+            self._check_env("mteb")
+            self.model_args = self.config_d["model"]
+            self.eval_args = self.config_d["eval"]
+            self.run_mteb()
+        elif tool.lower() == "ragas":
+            self._check_env("ragas")
+            self.testset_args = self.config_d.get("testset_generation", None)
+            self.eval_args = self.config_d.get("eval", None)
+            self.run_ragas()
+        else:
+            raise ValueError(f"Unknown tool: {tool}")

evalscope-0.5.5/evalscope/backend/rag_eval/cmteb/__init__.py

@@ -0,0 +1,4 @@
+from evalscope.backend.rag_eval.cmteb.tasks import *
+from evalscope.backend.rag_eval.cmteb.base import *
+from evalscope.backend.rag_eval.cmteb.arguments import ModelArguments, EvalArguments
+from evalscope.backend.rag_eval.cmteb.task_template import one_stage_eval, two_stage_eval

evalscope-0.5.5/evalscope/backend/rag_eval/cmteb/arguments.py

@@ -0,0 +1,59 @@
+from dataclasses import dataclass, field
+from typing import List, Optional, Union, Dict, Any
+
+
+@dataclass
+class ModelArguments:
+    # Arguments for embeding model: sentence transformer or cross encoder
+    model_name_or_path: str = ""  # model name or path
+    is_cross_encoder: bool = False  # whether the model is a cross encoder
+    # pooling mode: Either “cls”, “lasttoken”, “max”, “mean”, “mean_sqrt_len_tokens”, or “weightedmean”.
+    pooling_mode: Optional[str] = None
+    max_seq_length: int = 512  # max sequence length
+    # prompt for llm based model
+    prompt: str = ""
+    # model kwargs
+    model_kwargs: dict = field(default_factory=lambda: {"torch_dtype": "auto"})
+    # config kwargs
+    config_kwargs: Dict[str, Any] = field(default_factory=dict)
+    # encode kwargs
+    encode_kwargs: dict = field(
+        default_factory=lambda: {"show_progress_bar": True, "batch_size": 32}
+    )
+    hub: str = "modelscope"  # modelscope or huggingface
+
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "model_name_or_path": self.model_name_or_path,
+            "is_cross_encoder": self.is_cross_encoder,
+            "pooling_mode": self.pooling_mode,
+            "max_seq_length": self.max_seq_length,
+            "prompt": self.prompt,
+            "model_kwargs": self.model_kwargs,
+            "config_kwargs": self.config_kwargs,
+            "encode_kwargs": self.encode_kwargs,
+            "hub": self.hub,
+        }
+
+
+@dataclass
+class EvalArguments:
+    # Evaluation
+    tasks: List[str] = field(default_factory=list)  # task names
+    verbosity: int = 2  # verbosity level 0-3
+    output_folder: str = "outputs"  # output folder
+    overwrite_results: bool = True  # overwrite results
+    limits: Optional[int] = None  # limit number of samples
+    hub: str = "modelscope"  # modelscope or huggingface
+    top_k: int = 5
+
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "tasks": self.tasks,
+            "verbosity": self.verbosity,
+            "output_folder": self.output_folder,
+            "overwrite_results": self.overwrite_results,
+            "limits": self.limits,
+            "hub": self.hub,
+            "top_k": 5,
+        }

evalscope-0.5.5/evalscope/backend/rag_eval/cmteb/base.py

@@ -0,0 +1,89 @@
+from collections import defaultdict
+from typing import List
+from mteb import AbsTask
+from datasets import DatasetDict
+from modelscope import MsDataset
+import datasets
+from evalscope.backend.rag_eval.cmteb.tasks import CLS_DICT, CLS_RETRIEVAL
+
+__all__ = ["TaskBase"]
+
+
+
+class TaskBase:
+
+    @staticmethod
+    def get_tasks(task_names, **kwargs) -> List[AbsTask]:
+
+        return [TaskBase.get_task(task_name, **kwargs) for task_name in task_names]
+
+    @staticmethod
+    def get_task(task_name, **kwargs) -> AbsTask:
+
+        if task_name not in CLS_DICT:
+            from mteb.overview import TASKS_REGISTRY
+
+            task_cls = TASKS_REGISTRY[task_name]
+            if task_cls.metadata.type != "Retrieval":
+                task_cls.load_data = load_data
+        else:
+            task_cls = CLS_DICT[task_name]
+            task_cls.load_data = load_data
+        # init task instance
+        task_instance = task_cls()
+        return task_instance
+
+
+def load_data(self, **kwargs):
+    """Load dataset from the hub, compatible with ModelScope and Hugging Face."""
+    if self.data_loaded:
+        return
+
+    limits = kwargs.get("limits", None)
+    hub = kwargs.get("hub", "modelscope")
+    name = self.metadata_dict.get("name")
+    path = self.metadata_dict["dataset"].get("path")
+
+    assert path is not None, "Path must be specified in dataset"
+
+    # Loading the dataset based on the source hub
+    if hub == "modelscope":
+        import re
+
+        path = re.sub(r"^mteb/", "MTEB/", path)
+        dataset = MsDataset.load(path)
+    else:
+        dataset = datasets.load_dataset(**self.metadata_dict["dataset"])  # type: ignore
+
+    if limits is not None:
+        dataset = {
+            split: dataset[split].select(range(min(limits, len(dataset[split]))))
+            for split in dataset.keys()
+        }
+
+    if name in CLS_RETRIEVAL:
+        self.corpus, self.queries, self.relevant_docs = load_retrieval_data(
+            dataset,
+            path,
+            self.metadata_dict["eval_splits"],
+        )
+
+    self.dataset = dataset
+    self.dataset_transform()
+    self.data_loaded = True
+
+
+def load_retrieval_data(dataset, dataset_name: str, eval_splits: list) -> tuple:
+    eval_split = eval_splits[0]
+    qrels = MsDataset.load(dataset_name + "-qrels")[eval_split]
+
+    corpus = {e["id"]: {"text": e["text"]} for e in dataset["corpus"]}
+    queries = {e["id"]: e["text"] for e in dataset["queries"]}
+    relevant_docs = defaultdict(dict)
+    for e in qrels:
+        relevant_docs[e["qid"]][e["pid"]] = e["score"]
+
+    corpus = DatasetDict({eval_split: corpus})
+    queries = DatasetDict({eval_split: queries})
+    relevant_docs = DatasetDict({eval_split: relevant_docs})
+    return corpus, queries, relevant_docs

evalscope-0.5.5/evalscope/backend/rag_eval/cmteb/task_template.py

@@ -0,0 +1,83 @@
+import os
+import mteb
+from evalscope.backend.rag_eval import EmbeddingModel
+from evalscope.backend.rag_eval import cmteb
+from mteb.task_selection import results_to_dataframe
+from evalscope.utils.logger import get_logger
+
+logger = get_logger()
+
+
+def show_results(output_folder, model, results):
+    model_name = model.mteb_model_meta.model_name_as_path()
+    revision = model.mteb_model_meta.revision
+
+    results_df = results_to_dataframe({model_name: {revision: results}})
+
+    save_path = os.path.join(
+        output_folder,
+        model_name,
+        revision,
+    )
+    logger.info(f"Evaluation results:\n{results_df.to_markdown()}")
+    logger.info(f"Evaluation results saved in {os.path.abspath(save_path)}")
+
+
+def one_stage_eval(
+    model_args,
+    eval_args,
+) -> None:
+    # load model
+    model = EmbeddingModel.load(**model_args)
+
+    # load task first to update instructions
+    tasks = cmteb.TaskBase.get_tasks(task_names=eval_args["tasks"])
+    evaluation = mteb.MTEB(tasks=tasks)
+
+    # run evaluation
+    results = evaluation.run(model, **eval_args)
+
+    # save and log results
+    show_results(eval_args["output_folder"], model, results)
+
+
+def two_stage_eval(
+    model1_args,
+    model2_args,
+    eval_args,
+) -> None:
+    """a two-stage run with the second stage reading results saved from the first stage."""
+    # load model
+    dual_encoder = EmbeddingModel.load(**model1_args)
+    cross_encoder = EmbeddingModel.load(**model2_args)
+
+    first_stage_path = f"{eval_args['output_folder']}/stage1"
+    second_stage_path = f"{eval_args['output_folder']}/stage2"
+
+    tasks = cmteb.TaskBase.get_tasks(task_names=eval_args["tasks"])
+    for task in tasks:
+        evaluation = mteb.MTEB(tasks=[task])
+
+        # stage 1: run dual encoder
+        evaluation.run(
+            dual_encoder,
+            save_predictions=True,
+            output_folder=first_stage_path,
+            overwrite_results=True,
+            hub=eval_args["hub"],
+            limits=eval_args["limits"],
+        )
+        # stage 2: run cross encoder
+        results = evaluation.run(
+            cross_encoder,
+            top_k=eval_args["top_k"],
+            save_predictions=True,
+            output_folder=second_stage_path,
+            previous_results=f"{first_stage_path}/{task.metadata.name}_default_predictions.json",
+            overwrite_results=True,
+            hub=eval_args["hub"],
+            limits=eval_args["limits"],
+        )
+
+        # save and log results
+        show_results(second_stage_path, cross_encoder, results)