YESciEval 0.1.0__tar.gz

yescieval-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 XXX
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

yescieval-0.1.0/PKG-INFO

@@ -0,0 +1,58 @@
+Metadata-Version: 2.3
+Name: YESciEval
+Version: 0.1.0
+Summary: YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering.
+License: MIT
+Author: Hamed Babaei Giglou
+Author-email: hamedbabaeigiglou@gmail.com
+Requires-Python: >=3.10,<4.0.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: numpy
+Requires-Dist: openai
+Requires-Dist: pandas
+Requires-Dist: peft
+Requires-Dist: pre-commit
+Requires-Dist: pydantic
+Requires-Dist: torch
+Requires-Dist: transformers
+Project-URL: Homepage, https://yescieval.readthedocs.io/
+Project-URL: Repository, https://github.com/sciknoworg/YESciEval/
+Description-Content-Type: text/markdown
+
+<div align="center">
+  <img src="images/logo.png" width="50%" height="30%"/>
+</div>
+
+<div align="center">
+
+
+[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+
+</div>
+
+
+## 📋 What is the YESciEval?
+
+
+Large Language Models (LLMs) drive scientific question-answering on modern search engines, yet their evaluation robustness remains underexplored. We introduce **YESciEval**, an open-source framework that combines fine-grained rubric-based assessment with reinforcement learning to mitigate optimism bias in LLM evaluators. The framework is presented as f ollows:
+
+
+We release multidisciplinary scienceQ&A datasets, including adversarial variants, with evaluation scores from multiple LLMs. Independent of proprietary models and human feedback, our approach enables scalable, cost-free evaluation. By advancing reliable LLM-as-a-judge models, this work supports AI alignment and fosters robust, transparent evaluation essential for scientific inquiry and artificial general intelligence.
+
+## 📃 License
+
+This work is licensed under a [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT).
+
+
+
+
+

yescieval-0.1.0/README.md

@@ -0,0 +1,31 @@
+<div align="center">
+  <img src="images/logo.png" width="50%" height="30%"/>
+</div>
+
+<div align="center">
+
+
+[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+
+</div>
+
+
+## 📋 What is the YESciEval?
+
+
+Large Language Models (LLMs) drive scientific question-answering on modern search engines, yet their evaluation robustness remains underexplored. We introduce **YESciEval**, an open-source framework that combines fine-grained rubric-based assessment with reinforcement learning to mitigate optimism bias in LLM evaluators. The framework is presented as f ollows:
+
+
+We release multidisciplinary scienceQ&A datasets, including adversarial variants, with evaluation scores from multiple LLMs. Independent of proprietary models and human feedback, our approach enables scalable, cost-free evaluation. By advancing reliable LLM-as-a-judge models, this work supports AI alignment and fosters robust, transparent evaluation essential for scientific inquiry and artificial general intelligence.
+
+## 📃 License
+
+This work is licensed under a [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT).
+
+
+
+

yescieval-0.1.0/pyproject.toml

@@ -0,0 +1,35 @@
+[tool.poetry]
+name = "YESciEval"
+
+version = "0.1.0"
+
+description = "YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering."
+authors = ["Hamed Babaei Giglou <hamedbabaeigiglou@gmail.com>"]
+license = "MIT License"
+readme = "README.md"
+homepage = "https://yescieval.readthedocs.io/"
+repository = "https://github.com/sciknoworg/YESciEval/"
+include = ["images/logo.png"]
+
+[tool.poetry.dependencies]
+python = ">=3.10,<4.0.0"
+pre-commit="*"
+transformers="*"
+torch="*"
+peft="*"
+openai="*"
+pandas="*"
+numpy="*"
+pydantic="*"
+
+
+[tool.poetry.dev-dependencies]
+ruff = "*"
+pre-commit = "*"
+setuptools = "*"
+wheel = "*"
+twine = "*"
+
+[build-system]
+requires = ["poetry-core>=1.0.0"]
+build-backend = "poetry.core.masonry.api"

yescieval-0.1.0/yescieval/__init__.py

@@ -0,0 +1,27 @@
+
+__version__ = "0.1.0"
+
+from .base import Rubric, Parser
+from .rubric import (Informativeness, Correctness, Completeness, Coherence, Relevancy,
+                    Integration, Cohesion, Readability, Conciseness)
+from .judge import AutoJudge, AskAutoJudge, BioASAutoJudge
+from .parser import GPTParser
+#
+# __all__ = [
+#     "Rubric",
+#     "Informativeness",
+#     "Correctness",
+#     "Completeness",
+#     "Coherence",
+#     "Relevancy",
+#     "Integration",
+#     "Cohesion",
+#     "Readability",
+#     "Conciseness",
+#     "Parser",
+#     "AutoJudge",
+#     "AskAutoJudge",
+#     "BioASAutoJudge"
+# ]
+
+#

yescieval-0.1.0/yescieval/base/__init__.py

@@ -0,0 +1,10 @@
+from .rubric import Rubric
+from .parser import Parser, RubricLikertScale
+from .judge import Judge
+
+__all__ = [
+    "Rubric",
+    "Parser",
+    "RubricLikertScale",
+    "Judge"
+]

yescieval-0.1.0/yescieval/base/judge.py

@@ -0,0 +1,16 @@
+from abc import ABC
+from typing import Dict, Any
+from . import Parser, Rubric
+
+
+class Judge(ABC):
+
+    def from_pretrained(self, model_id:str, device: str="auto", token:str =""):
+        self.model, self.tokenizer = self._from_pretrained(model_id=model_id, device=device, token=token)
+
+    def judge(self, rubric: Rubric, max_new_tokens: int=150) -> Dict[str, Dict[str, str]]:
+        pass
+
+    def _from_pretrained(self, model_id: str, device: str = "auto", token: str = "") -> [Any, Any]:
+        pass
+

yescieval-0.1.0/yescieval/base/parser.py

@@ -0,0 +1,28 @@
+from abc import ABC, abstractmethod
+from typing import Dict, Any
+from pydantic import BaseModel, Field
+
+
+class RubricLikertScale(BaseModel):
+    rating: int = Field(..., ge=1, le=5, description="Rating from 1 to 5")
+    rationale: str = Field(..., description="Textual explanation for the rating")
+
+
+class Parser(ABC):
+    """
+    Abstract base class for parsing model outputs into structured characteristic evaluations.
+
+    Each characteristic maps to a CharacteristicScore with a rating and rationale.
+    """
+    @abstractmethod
+    def parse(self, raw_output: str) -> Any:
+        """
+        Parse the raw model output into structured characteristic evaluations.
+
+        Args:
+            raw_output (str): The text generated by the model.
+
+        Returns:
+            Dict[str, CharacteristicScore]: Mapping from characteristic name to its score and rationale.
+        """
+        return raw_output

yescieval-0.1.0/yescieval/base/rubric.py

@@ -0,0 +1,39 @@
+from abc import ABC
+from pydantic import BaseModel
+from typing import Dict, List
+
+
+
+class Rubric(BaseModel, ABC):
+    """
+    Abstract base class for evaluation rubrics.
+    Subclasses must implement `verbalize`.
+    """
+    system_prompt_template: str
+    papers: Dict[str, str]
+    question: str
+    answer: str
+    user_prompt_template: str = ("Evaluate and rate the quality of the following scientific synthesis "
+                             "according to the characteristics given in the system prompt.\n"
+                             "\n<scientific-synthesis>{answer}</scientific-synthesis>\n"
+                             "\n<research-question>{question}</research-question>\n"
+                             "\n<paper-titles-and-abstracts>\n{content}</paper-titles-and-abstracts>\n\n###")
+
+    def render_papers(self) -> str:
+        paper_content = ""
+        for idx, (title, abstract) in enumerate(self.papers.items()):
+            paper_content += f"{idx + 1}. " + title + "\n\n" + abstract + "\n\n"
+        return paper_content
+
+    def verbalize(self):
+        return self.user_prompt_template.format(answer=self.answer,
+                                                question=self.question,
+                                                content=self.render_papers())
+
+    def instruct(self) -> List[Dict[str, str]]:
+        message = [
+            {"role": "system", "content":  self.system_prompt_template},
+            {"role": "user", "content": self.verbalize()},
+        ]
+        return message
+

yescieval-0.1.0/yescieval/judge/__init__.py

@@ -0,0 +1,7 @@
+from .judges import AutoJudge, AskAutoJudge, BioASAutoJudge
+
+__all__ = [
+    "AutoJudge",
+    "AskAutoJudge",
+    "BioASAutoJudge"
+]

yescieval-0.1.0/yescieval/judge/judges.py

@@ -0,0 +1,51 @@
+from ..base import Judge, Rubric
+from typing import Dict
+
+from transformers import AutoTokenizer, AutoModelForCausalLM
+from peft import PeftModel, PeftConfig
+import torch
+
+
+
+class AutoJudge(Judge):
+
+    def _from_pretrained(self, model_id:str, device:str="auto", token:str =""):
+        config = PeftConfig.from_pretrained(model_id)
+        base_model_name = config.base_model_name_or_path
+        tokenizer = AutoTokenizer.from_pretrained(base_model_name,
+                                                  padding_side="left",
+                                                  token=token)
+        tokenizer.pad_token = tokenizer.eos_token
+        base_model = AutoModelForCausalLM.from_pretrained(
+            base_model_name,
+            torch_dtype=torch.float32,
+            device_map=device,
+            token=token
+        )
+        model = PeftModel.from_pretrained(base_model, model_id)
+        return model, tokenizer
+
+    def evaluate(self, rubric: Rubric, max_new_tokens: int=150) -> Dict[str, Dict[str, str]]:
+        inputs = self.tokenizer.apply_chat_template(rubric.instruct(),
+                                                    add_generation_prompt=True,
+                                                    return_dict=True,
+                                                    return_tensors="pt")
+        inputs.to(self.model.device)
+        outputs = self.model.generate(**inputs,
+                                      max_new_tokens=max_new_tokens,
+                                      pad_token_id=self.tokenizer.eos_token_id)
+        evaluation = self.tokenizer.decode(outputs[0][len(inputs["input_ids"][0]):], skip_special_tokens=True)
+        return evaluation
+
+
+class AskAutoJudge(AutoJudge):
+    def from_pretrained(self, model_id:str="SciKnowOrg/YESciEval-ASK-Llama-3.1-8B",
+                         device:str="auto",
+                         token:str =""):
+        return super()._from_pretrained(model_id=model_id, device=device, token=token)
+
+class BioASAutoJudge(AutoJudge):
+    def from_pretrained(self, model_id: str = "SciKnowOrg/YESciEval-BioASQ-Llama-3.1-8B",
+                         device: str = "auto",
+                         token: str = ""):
+        return super()._from_pretrained(model_id=model_id, device=device, token=token)

yescieval-0.1.0/yescieval/parser/__init__.py

@@ -0,0 +1,3 @@
+from .parsers import GPTParser
+
+__all__ = ["GPTParser"]

yescieval-0.1.0/yescieval/parser/parsers.py

@@ -0,0 +1,61 @@
+
+from ..base import Parser, RubricLikertScale
+import time
+from openai import OpenAI
+
+class GPTParser(Parser):
+    """
+    Abstract base class for parsing model outputs into structured characteristic evaluations.
+
+    Each characteristic maps to a CharacteristicScore with a rating and rationale.
+    """
+    def __init__(self, openai_key:str, parser_model:str="gpt-4o-mini"):
+        self.client = OpenAI(api_key=openai_key)
+        self.parser_model = parser_model
+
+    def parse(self, raw_output: str) -> RubricLikertScale:
+        """
+        Parse the raw model output into structured characteristic evaluations.
+
+        Args:
+            raw_output (str): The text generated by the model.
+
+        Returns:
+            Dict[str, CharacteristicScore]: Mapping from characteristic name to its score and rationale.
+        """
+        functions = [
+            {
+                "name": "evaluate_characteristic",
+                "description": "Extracting the exact `rating` and `rationale` from the given text.",
+                "parameters": {
+                    "type": "object",
+                    "properties": {
+                        "rating": {
+                            "type": "number",
+                            "description": "A numerical rating assigned to the characteristic in the text.",
+                            "minimum": 1,
+                            "maximum": 5
+                        },
+                        "rationale": {
+                            "type": "string",
+                            "description": "The explanation for the assigned rating."
+                        }
+                    },
+                    "required": ["rating", "rationale"]
+                }
+            }
+        ]
+        while True:
+            try:
+                completion = self.client.chat.completions.create(
+                    model=self.parser_model,
+                    messages=[{"role": "user", "content": raw_output}],
+                    functions=functions
+                )
+                parsed_output = eval(completion.choices[0].message.function_call.arguments)
+                break
+            except Exception as e:
+                print(f"Error {e}")
+                time.sleep(3)
+
+        return RubricLikertScale(rating=parsed_output['rating'], rationale=parsed_output['rationale'])

yescieval-0.1.0/yescieval/rubric/__init__.py

@@ -0,0 +1,7 @@
+from .informativeness import Informativeness, Correctness, Completeness
+from .structural import Coherence, Relevancy, Integration
+from .stylistic import Cohesion, Readability, Conciseness
+
+__all__ = ["Informativeness", "Correctness", "Completeness",
+           "Coherence", "Relevancy", "Integration",
+           "Cohesion", "Readability", "Conciseness"]

yescieval-0.1.0/yescieval/rubric/informativeness.py

@@ -0,0 +1,161 @@
+from ..base import Rubric
+
+correctness_prompt = """<Context> 
+Scientific synthesis generation involves creating a concise, coherent, and integrated summary from a collection of scientific texts (such as research paper titles and abstracts) that addresses a specific research question. Unlike general text summarization, which may focus on extracting or abstracting key points from a single text or multiple texts on a broad topic, scientific synthesis is more specialized. It requires:
+
+- Understanding and Addressing a Specific Research Question: The synthesis must specifically answer a research question, requiring a deep understanding of the subject matter and the ability to extract and integrate relevant information from various sources.
+- Use of Scientific Literature: The process involves synthesizing information from scientific literature, such as research papers, focusing on the given titles and abstracts. This requires not only summarizing these texts but also evaluating their relevance, correctness, and completeness in the context of the research question.
+- Synthesis Format: The synthesis output should be concisely presented in a single paragraph of not more than 200 words. This format requires distilling and integrating diverse scientific insights into a coherent and comprehensive summary that addresses the research question directly. The single-paragraph format emphasizes the importance of concise and integrated communication of complex information.
+- Synthesize vs. Summarize: The goal is to synthesize—meaning to combine elements to form a coherent whole—rather than just summarize each source individually. This involves integration, cohesion, and coherence of information from multiple sources, presenting it in a way that produces new insights or understanding in response to the research question.
+- Referencing Source Material: Each claim or piece of information in the synthesis must be traceable to the source material (the abstracts), ensuring the synthesis's accuracy and reliability.
+- Adherence to Quality Characteristics: It should be possible to evaluate the synthesis quality based on correctness characteristic, ensuring it effectively communicates the synthesized information.
+
+In essence, scientific synthesis generation is a complex task that goes beyond simply summarizing texts; it involves critically analyzing, integrating, and presenting scientific information from multiple sources to succinctly answer a targeted research question, adhering to high standards of clarity, reliability, and insightfulness.
+</Context>
+
+<Role>
+You are tasked as a scientific syntheses quality evaluator.
+</Role>
+
+<Task-Description>
+A user will provide you with a synthesis which has been generated as an answer to a research question using the titles and abstracts of relevant research works.  You will also be provided with the research question and the paper titles+abstracts of the relevant works that were synthesized. You must use the evaluation characteristic listed below to evaluate a given scientific synthesis. The general objective is that a synthesis should succinctly address the research question by synthesizing only the content from the provided abstracts, while also referencing the source abstract for each claim.
+</Task-Description>
+
+<Evaluation-Characteristics>
+1. Correctness: is the information in the answer a correct representation of the content of the provided abstracts?
+</Evaluation-Characteristics>
+
+<Rating-Scale>
+For a given characteristic, rate the quality from 1 (very bad) to 5 (very good). Follow the guidelines specified below for each rating per evaluation characteristic.
+
+1. Correctness
+Rating 1. Very bad: The synthesis consistently misrepresents or inaccurately portrays the content of the provided abstracts, showing a significant deviation from the original sources.
+Rating 2. Bad: The synthesis contains several inaccuracies or misinterpretations of the source abstracts.
+Rating 3. Moderate: The synthesis accurately represents most of the content from the provided abstracts but may contain minor errors.
+Rating 4. Good: The synthesis provides an accurate representation of the content from the provided abstracts with minor exceptions.
+Rating 5. Very good: The information in the synthesis is an accurate and faithful representation of the content from the provided abstracts, without any factual errors or misinterpretations.
+</Rating-Scale>
+
+<Response-Format>
+For each characteristic rate the quality from 1 (very bad) to 5 (very good).  Provide a short rationale for each rating. 
+Return your response in JSON format: {characteristic : {‘rating’ : ‘’, ‘rationale’ : ‘’}}
+
+<Example-Response>
+{
+  "Correctness": {"rating": "4", "rationale": "The synthesis represents the content of the provided abstract, but with minor inrelevant information."}
+}
+</Example-Response>
+</Response-Format>
+
+<Note>
+Your evaluation should be based solely on the content of the provided synthesis and abstracts. Ensure your rationale is objective and backed by specific examples from the provided material.
+</Note>"""
+class Correctness(Rubric):
+    system_prompt_template: str = correctness_prompt
+
+completeness_prompt = """<Context> 
+Scientific synthesis generation involves creating a concise, coherent, and integrated summary from a collection of scientific texts (such as research paper titles and abstracts) that addresses a specific research question. Unlike general text summarization, which may focus on extracting or abstracting key points from a single text or multiple texts on a broad topic, scientific synthesis is more specialized. It requires:
+
+- Understanding and Addressing a Specific Research Question: The synthesis must specifically answer a research question, requiring a deep understanding of the subject matter and the ability to extract and integrate relevant information from various sources.
+- Use of Scientific Literature: The process involves synthesizing information from scientific literature, such as research papers, focusing on the given titles and abstracts. This requires not only summarizing these texts but also evaluating their relevance, correctness, and completeness in the context of the research question.
+- Synthesis Format: The synthesis output should be concisely presented in a single paragraph of not more than 200 words. This format requires distilling and integrating diverse scientific insights into a coherent and comprehensive summary that addresses the research question directly. The single-paragraph format emphasizes the importance of concise and integrated communication of complex information.
+- Synthesize vs. Summarize: The goal is to synthesize—meaning to combine elements to form a coherent whole—rather than just summarize each source individually. This involves integration, cohesion, and coherence of information from multiple sources, presenting it in a way that produces new insights or understanding in response to the research question.
+- Referencing Source Material: Each claim or piece of information in the synthesis must be traceable to the source material (the abstracts), ensuring the synthesis's accuracy and reliability.
+- Adherence to Quality Characteristics: It should be possible to evaluate the synthesis quality based on completeness characteristic, ensuring it effectively communicates the synthesized information.
+
+In essence, scientific synthesis generation is a complex task that goes beyond simply summarizing texts; it involves critically analyzing, integrating, and presenting scientific information from multiple sources to succinctly answer a targeted research question, adhering to high standards of clarity, reliability, and insightfulness.
+</Context>
+
+<Role>
+You are tasked as a scientific syntheses quality evaluator.
+</Role>
+
+<Task-Description>
+A user will provide you with a synthesis which has been generated as an answer to a research question using the titles and abstracts of relevant research works.  You will also be provided with the research question and the paper titles+abstracts of the relevant works that were synthesized. You must use the evaluation characteristic listed below to evaluate a given scientific synthesis. The general objective is that a synthesis should succinctly address the research question by synthesizing only the content from the provided abstracts, while also referencing the source abstract for each claim.
+</Task-Description>
+
+<Evaluation-Characteristics>
+1. Completeness: is the answer a comprehensive encapsulation of the relevant information in the provided abstracts?
+</Evaluation-Characteristics>
+
+<Rating-Scale>
+For a given characteristic, rate the quality from 1 (very bad) to 5 (very good). Follow the guidelines specified below for each rating per evaluation characteristic.
+
+1. Completeness
+Rating 1. Very bad: The synthesis omits most of the relevant information, failing to capture the essential points or details from the provided abstracts.
+Rating 2. Bad: Significant portions of relevant information from the provided abstracts are missing.
+Rating 3. Moderate: The synthesis captures a fair amount of the relevant information, though it may overlook some details.
+Rating 4. Good: The synthesis includes almost all relevant information, missing only minor details.
+Rating 5. Very good: The synthesis comprehensively encapsulates all relevant information from the provided abstracts, leaving no pertinent details or points unaddressed.
+</Rating-Scale>
+
+<Response-Format>
+For each characteristic rate the quality from 1 (very bad) to 5 (very good).  Provide a short rationale for each rating. 
+Return your response in JSON format: {characteristic : {‘rating’ : ‘’, ‘rationale’ : ‘’}}
+
+<Example-Response>
+{
+  "Completeness": {"rating": "4", "rationale": "Only minor details are missing in the synthesis."}
+}
+</Example-Response>
+</Response-Format>
+
+<Note>
+Your evaluation should be based solely on the content of the provided synthesis and abstracts. Ensure your rationale is objective and backed by specific examples from the provided material.
+</Note>"""
+class Completeness(Rubric):
+    system_prompt_template: str = completeness_prompt
+
+informativeness_prompt = """<Context> 
+Scientific synthesis generation involves creating a concise, coherent, and integrated summary from a collection of scientific texts (such as research paper titles and abstracts) that addresses a specific research question. Unlike general text summarization, which may focus on extracting or abstracting key points from a single text or multiple texts on a broad topic, scientific synthesis is more specialized. It requires:
+
+- Understanding and Addressing a Specific Research Question: The synthesis must specifically answer a research question, requiring a deep understanding of the subject matter and the ability to extract and integrate relevant information from various sources.
+- Use of Scientific Literature: The process involves synthesizing information from scientific literature, such as research papers, focusing on the given titles and abstracts. This requires not only summarizing these texts but also evaluating their relevance, correctness, and completeness in the context of the research question.
+- Synthesis Format: The synthesis output should be concisely presented in a single paragraph of not more than 200 words. This format requires distilling and integrating diverse scientific insights into a coherent and comprehensive summary that addresses the research question directly. The single-paragraph format emphasizes the importance of concise and integrated communication of complex information.
+- Synthesize vs. Summarize: The goal is to synthesize—meaning to combine elements to form a coherent whole—rather than just summarize each source individually. This involves integration, cohesion, and coherence of information from multiple sources, presenting it in a way that produces new insights or understanding in response to the research question.
+- Referencing Source Material: Each claim or piece of information in the synthesis must be traceable to the source material (the abstracts), ensuring the synthesis's accuracy and reliability.
+- Adherence to Quality Characteristics: It should be possible to evaluate the synthesis quality based on informativeness characteristic, ensuring it effectively communicates the synthesized information.
+
+In essence, scientific synthesis generation is a complex task that goes beyond simply summarizing texts; it involves critically analyzing, integrating, and presenting scientific information from multiple sources to succinctly answer a targeted research question, adhering to high standards of clarity, reliability, and insightfulness.
+</Context>
+
+<Role>
+You are tasked as a scientific syntheses quality evaluator.
+</Role>
+
+<Task-Description>
+A user will provide you with a synthesis which has been generated as an answer to a research question using the titles and abstracts of relevant research works.  You will also be provided with the research question and the paper titles+abstracts of the relevant works that were synthesized. You must use the evaluation characteristic listed below to evaluate a given scientific synthesis. The general objective is that a synthesis should succinctly address the research question by synthesizing only the content from the provided abstracts, while also referencing the source abstract for each claim.
+</Task-Description>
+
+<Evaluation-Characteristics>
+1. Informativeness: is the answer a useful and informative reply to the question?
+</Evaluation-Characteristics>
+
+<Rating-Scale>
+For a given characteristic, rate the quality from 1 (very bad) to 5 (very good). Follow the guidelines specified below for each rating per evaluation characteristic.
+
+1. Informativeness
+Rating 1. Very bad: The synthesis offers no valuable insights or useful information in response to the research question, lacking depth and utility.
+Rating 2. Bad: The answer provides limited new insights or useful information in response to the research question.
+Rating 3. Moderate: The answer is somewhat informative, offering insights or useful information but not in a comprehensive or detailed manner.
+Rating 4. Good: The answer is informative and insightful, providing comprehensive information in response to the research question.
+Rating 5. Very good: The synthesis is highly informative, providing valuable insights and detailed information that thoroughly addresses the research question.
+</Rating-Scale>
+
+<Response-Format>
+For each characteristic rate the quality from 1 (very bad) to 5 (very good).  Provide a short rationale for each rating. 
+Return your response in JSON format: {characteristic : {‘rating’ : ‘’, ‘rationale’ : ‘’}}
+
+<Example-Response>
+{
+  "Informativeness": {"rating": "4", "rationale": "Most information is informative for the research question."}
+}
+</Example-Response>
+</Response-Format>
+
+<Note>
+Your evaluation should be based solely on the content of the provided synthesis and abstracts. Ensure your rationale is objective and backed by specific examples from the provided material.
+</Note>"""
+class Informativeness(Rubric):
+    system_prompt_template: str = informativeness_prompt
+

yescieval-0.1.0/yescieval/rubric/structural.py

@@ -0,0 +1,160 @@
+from ..base import Rubric
+
+coherence_prompt = """<Context> 
+Scientific synthesis generation involves creating a concise, coherent, and integrated summary from a collection of scientific texts (such as research paper titles and abstracts) that addresses a specific research question. Unlike general text summarization, which may focus on extracting or abstracting key points from a single text or multiple texts on a broad topic, scientific synthesis is more specialized. It requires:
+
+- Understanding and Addressing a Specific Research Question: The synthesis must specifically answer a research question, requiring a deep understanding of the subject matter and the ability to extract and integrate relevant information from various sources.
+- Use of Scientific Literature: The process involves synthesizing information from scientific literature, such as research papers, focusing on the given titles and abstracts. This requires not only summarizing these texts but also evaluating their relevance, correctness, and completeness in the context of the research question.
+- Synthesis Format: The synthesis output should be concisely presented in a single paragraph of not more than 200 words. This format requires distilling and integrating diverse scientific insights into a coherent and comprehensive summary that addresses the research question directly. The single-paragraph format emphasizes the importance of concise and integrated communication of complex information.
+- Synthesize vs. Summarize: The goal is to synthesize—meaning to combine elements to form a coherent whole—rather than just summarize each source individually. This involves integration, cohesion, and coherence of information from multiple sources, presenting it in a way that produces new insights or understanding in response to the research question.
+- Referencing Source Material: Each claim or piece of information in the synthesis must be traceable to the source material (the abstracts), ensuring the synthesis's accuracy and reliability.
+- Adherence to Quality Characteristics: It should be possible to evaluate the synthesis quality based on coherence characteristic, ensuring it effectively communicates the synthesized information.
+
+In essence, scientific synthesis generation is a complex task that goes beyond simply summarizing texts; it involves critically analyzing, integrating, and presenting scientific information from multiple sources to succinctly answer a targeted research question, adhering to high standards of clarity, reliability, and insightfulness.
+</Context>
+
+<Role>
+You are tasked as a scientific syntheses quality evaluator.
+</Role>
+
+<Task-Description>
+A user will provide you with a synthesis which has been generated as an answer to a research question using the titles and abstracts of relevant research works.  You will also be provided with the research question and the paper titles+abstracts of the relevant works that were synthesized. You must use the evaluation characteristic listed below to evaluate a given scientific synthesis. The general objective is that a synthesis should succinctly address the research question by synthesizing only the content from the provided abstracts, while also referencing the source abstract for each claim.
+</Task-Description>
+
+<Evaluation-Characteristics>
+1. Coherence: are the ideas connected in a sound and logical manner?
+</Evaluation-Characteristics>
+
+<Rating-Scale>
+For a given characteristic, rate the quality from 1 (very bad) to 5 (very good). Follow the guidelines specified below for each rating per evaluation characteristic.
+
+1. Coherence
+Rating 1. Very bad: The synthesis lacks logical connection between ideas, leading to a narrative that is confusing and difficult to follow.
+Rating 2. Bad: The ideas are not always logically connected, leading to a somewhat confusing narrative.
+Rating 3. Moderate: The ideas are logically connected for the most part, but the narrative could be strengthened for better clarity.
+Rating 4. Good: The ideas are logically and soundly connected, offering a clear and understandable narrative.
+Rating 5. Very good: The ideas within the synthesis are connected in a logical and sound manner, forming a coherent and compelling narrative that is easy to follow.
+</Rating-Scale>
+
+<Response-Format>
+For each characteristic rate the quality from 1 (very bad) to 5 (very good).  Provide a short rationale for each rating. 
+Return your response in JSON format: {characteristic : {‘rating’ : ‘’, ‘rationale’ : ‘’}}
+
+<Example-Response>
+{
+  "Coherence": {"rating": "4", "rationale": "The ideation is soundly connected with clear narrative."}
+}
+</Example-Response>
+</Response-Format>
+
+<Note>
+Your evaluation should be based solely on the content of the provided synthesis and abstracts. Ensure your rationale is objective and backed by specific examples from the provided material.
+</Note>"""
+class Coherence(Rubric):
+    system_prompt_template: str = coherence_prompt
+
+integration_prompt = """<Context> 
+Scientific synthesis generation involves creating a concise, coherent, and integrated summary from a collection of scientific texts (such as research paper titles and abstracts) that addresses a specific research question. Unlike general text summarization, which may focus on extracting or abstracting key points from a single text or multiple texts on a broad topic, scientific synthesis is more specialized. It requires:
+
+- Understanding and Addressing a Specific Research Question: The synthesis must specifically answer a research question, requiring a deep understanding of the subject matter and the ability to extract and integrate relevant information from various sources.
+- Use of Scientific Literature: The process involves synthesizing information from scientific literature, such as research papers, focusing on the given titles and abstracts. This requires not only summarizing these texts but also evaluating their relevance, correctness, and completeness in the context of the research question.
+- Synthesis Format: The synthesis output should be concisely presented in a single paragraph of not more than 200 words. This format requires distilling and integrating diverse scientific insights into a coherent and comprehensive summary that addresses the research question directly. The single-paragraph format emphasizes the importance of concise and integrated communication of complex information.
+- Synthesize vs. Summarize: The goal is to synthesize—meaning to combine elements to form a coherent whole—rather than just summarize each source individually. This involves integration, cohesion, and coherence of information from multiple sources, presenting it in a way that produces new insights or understanding in response to the research question.
+- Referencing Source Material: Each claim or piece of information in the synthesis must be traceable to the source material (the abstracts), ensuring the synthesis's accuracy and reliability.
+- Adherence to Quality Characteristics: It should be possible to evaluate the synthesis quality based on integration characteristic, ensuring it effectively communicates the synthesized information.
+
+In essence, scientific synthesis generation is a complex task that goes beyond simply summarizing texts; it involves critically analyzing, integrating, and presenting scientific information from multiple sources to succinctly answer a targeted research question, adhering to high standards of clarity, reliability, and insightfulness.
+</Context>
+
+<Role>
+You are tasked as a scientific syntheses quality evaluator.
+</Role>
+
+<Task-Description>
+A user will provide you with a synthesis which has been generated as an answer to a research question using the titles and abstracts of relevant research works.  You will also be provided with the research question and the paper titles+abstracts of the relevant works that were synthesized. You must use the evaluation characteristic listed below to evaluate a given scientific synthesis. The general objective is that a synthesis should succinctly address the research question by synthesizing only the content from the provided abstracts, while also referencing the source abstract for each claim.
+</Task-Description>
+
+<Evaluation-Characteristics>
+1. Integration: are the sources structurally and linguistically well-integrated, using appropriate markers of provenance/quotation and logical connectors for each reference? In addition, are the sources integrated as a single paragraph?
+</Evaluation-Characteristics>
+
+<Rating-Scale>
+For a given characteristic, rate the quality from 1 (very bad) to 5 (very good). Follow the guidelines specified below for each rating per evaluation characteristic.
+
+1. Integration
+Rating 1. Very Bad: The synthesis fails to integrate the sources in any meaningful way. It lacks appropriate markers, connectors, or transitions between ideas and fails to combine the information into a single, cohesive paragraph.
+Rating 2. Bad: The sources are somewhat integrated but inconsistently. The use of markers and connectors is sporadic or inappropriately applied, and the information is presented in multiple paragraphs without a clear unifying structure.
+Rating 3. Moderate: The sources are integrated into a coherent manner within one or multiple paragraphs. The transitions or connections could be smoother, and the text would benefit from better paragraph structure to enhance clarity and unity.
+Rating 4. Good: The sources are well-integrated, using appropriate markers and connectors to create a seamless narrative. The information is effectively organized into a single paragraph, showing a clear, unified approach.
+Rating 5. Very Good: The synthesis seamlessly integrates information from the various sources, using appropriate markers and connectors to create a smooth and unified narrative. All information is skillfully condensed into a single, well-structured paragraph, exemplifying excellent integration.
+</Rating-Scale>
+
+<Response-Format>
+For each characteristic rate the quality from 1 (very bad) to 5 (very good).  Provide a short rationale for each rating. 
+Return your response in JSON format: {characteristic : {‘rating’ : ‘’, ‘rationale’ : ‘’}}
+
+<Example-Response>
+{
+  "Integration": {"rating": "4", "rationale": "Almost all sources are well-integrated with approriate connectors."}
+}
+</Example-Response>
+</Response-Format>
+
+<Note>
+Your evaluation should be based solely on the content of the provided synthesis and abstracts. Ensure your rationale is objective and backed by specific examples from the provided material.
+</Note>"""
+class Integration(Rubric):
+    system_prompt_template: str = integration_prompt
+
+relevancy_prompt = """<Context> 
+Scientific synthesis generation involves creating a concise, coherent, and integrated summary from a collection of scientific texts (such as research paper titles and abstracts) that addresses a specific research question. Unlike general text summarization, which may focus on extracting or abstracting key points from a single text or multiple texts on a broad topic, scientific synthesis is more specialized. It requires:
+
+- Understanding and Addressing a Specific Research Question: The synthesis must specifically answer a research question, requiring a deep understanding of the subject matter and the ability to extract and integrate relevant information from various sources.
+- Use of Scientific Literature: The process involves synthesizing information from scientific literature, such as research papers, focusing on the given titles and abstracts. This requires not only summarizing these texts but also evaluating their relevance, correctness, and completeness in the context of the research question.
+- Synthesis Format: The synthesis output should be concisely presented in a single paragraph of not more than 200 words. This format requires distilling and integrating diverse scientific insights into a coherent and comprehensive summary that addresses the research question directly. The single-paragraph format emphasizes the importance of concise and integrated communication of complex information.
+- Synthesize vs. Summarize: The goal is to synthesize—meaning to combine elements to form a coherent whole—rather than just summarize each source individually. This involves integration, cohesion, and coherence of information from multiple sources, presenting it in a way that produces new insights or understanding in response to the research question.
+- Referencing Source Material: Each claim or piece of information in the synthesis must be traceable to the source material (the abstracts), ensuring the synthesis's accuracy and reliability.
+- Adherence to Quality Characteristics: It should be possible to evaluate the synthesis quality based on relevancy characteristic, ensuring it effectively communicates the synthesized information.
+
+In essence, scientific synthesis generation is a complex task that goes beyond simply summarizing texts; it involves critically analyzing, integrating, and presenting scientific information from multiple sources to succinctly answer a targeted research question, adhering to high standards of clarity, reliability, and insightfulness.
+</Context>
+
+<Role>
+You are tasked as a scientific syntheses quality evaluator.
+</Role>
+
+<Task-Description>
+A user will provide you with a synthesis which has been generated as an answer to a research question using the titles and abstracts of relevant research works.  You will also be provided with the research question and the paper titles+abstracts of the relevant works that were synthesized. You must use the evaluation characteristic listed below to evaluate a given scientific synthesis. The general objective is that a synthesis should succinctly address the research question by synthesizing only the content from the provided abstracts, while also referencing the source abstract for each claim.
+</Task-Description>
+
+<Evaluation-Characteristics>
+1. Relevancy: is the information in the answer relevant to the question?
+</Evaluation-Characteristics>
+
+<Rating-Scale>
+For a given characteristic, rate the quality from 1 (very bad) to 5 (very good). Follow the guidelines specified below for each rating per evaluation characteristic.
+
+1. Relevancy
+Rating 1. Very bad: The information provided does not relate to the research question, showing a lack of understanding or connection to the topic.
+Rating 2. Bad: The information occasionally relates to the research question but lacks direct and consistent relevance.
+Rating 3. Moderate: The information is generally related to the research question, with occasional lapses in direct relevance.
+Rating 4. Good: The information is consistently relevant to the research question, with only minor exceptions.
+Rating 5. Very good: The synthesis is directly and consistently relevant to the research question, demonstrating a deep understanding of the topic and its nuances.
+</Rating-Scale>
+
+<Response-Format>
+For each characteristic rate the quality from 1 (very bad) to 5 (very good).  Provide a short rationale for each rating. 
+Return your response in JSON format: {characteristic : {‘rating’ : ‘’, ‘rationale’ : ‘’}}
+
+<Example-Response>
+{
+  "Relevancy": {"rating": "4", "rationale": "Most information is relevant, but there is a minor detail that seems out of scope."}
+}
+</Example-Response>
+</Response-Format>
+
+<Note>
+Your evaluation should be based solely on the content of the provided synthesis and abstracts. Ensure your rationale is objective and backed by specific examples from the provided material.
+</Note>"""
+class Relevancy(Rubric):
+    system_prompt_template: str = relevancy_prompt

yescieval-0.1.0/yescieval/rubric/stylistic.py

@@ -0,0 +1,163 @@
+from ..base import Rubric
+
+cohesion_prompt = """<Context> 
+Scientific synthesis generation involves creating a concise, coherent, and integrated summary from a collection of scientific texts (such as research paper titles and abstracts) that addresses a specific research question. Unlike general text summarization, which may focus on extracting or abstracting key points from a single text or multiple texts on a broad topic, scientific synthesis is more specialized. It requires:
+
+- Understanding and Addressing a Specific Research Question: The synthesis must specifically answer a research question, requiring a deep understanding of the subject matter and the ability to extract and integrate relevant information from various sources.
+- Use of Scientific Literature: The process involves synthesizing information from scientific literature, such as research papers, focusing on the given titles and abstracts. This requires not only summarizing these texts but also evaluating their relevance, correctness, and completeness in the context of the research question.
+- Synthesis Format: The synthesis output should be concisely presented in a single paragraph of not more than 200 words. This format requires distilling and integrating diverse scientific insights into a coherent and comprehensive summary that addresses the research question directly. The single-paragraph format emphasizes the importance of concise and integrated communication of complex information.
+- Synthesize vs. Summarize: The goal is to synthesize—meaning to combine elements to form a coherent whole—rather than just summarize each source individually. This involves integration, cohesion, and coherence of information from multiple sources, presenting it in a way that produces new insights or understanding in response to the research question.
+- Referencing Source Material: Each claim or piece of information in the synthesis must be traceable to the source material (the abstracts), ensuring the synthesis's accuracy and reliability.
+- Adherence to Quality Characteristics: It should be possible to evaluate the synthesis quality based on cohesion characteristic, ensuring it effectively communicates the synthesized information.
+
+In essence, scientific synthesis generation is a complex task that goes beyond simply summarizing texts; it involves critically analyzing, integrating, and presenting scientific information from multiple sources to succinctly answer a targeted research question, adhering to high standards of clarity, reliability, and insightfulness.
+</Context>
+
+<Role>
+You are tasked as a scientific syntheses quality evaluator.
+</Role>
+
+<Task-Description>
+A user will provide you with a synthesis which has been generated as an answer to a research question using the titles and abstracts of relevant research works.  You will also be provided with the research question and the paper titles+abstracts of the relevant works that were synthesized. You must use the evaluation characteristic listed below to evaluate a given scientific synthesis. The general objective is that a synthesis should succinctly address the research question by synthesizing only the content from the provided abstracts, while also referencing the source abstract for each claim.
+</Task-Description>
+
+<Evaluation-Characteristics>
+1. Cohesion: are the sentences connected appropriately such that the resulting synthesis is cohesive?
+</Evaluation-Characteristics>
+
+<Rating-Scale>
+For a given characteristic, rate the quality from 1 (very bad) to 5 (very good). Follow the guidelines specified below for each rating per evaluation characteristic.
+
+1. Cohesion
+Rating 1. Very bad: The sentences within the synthesis are disconnected, resulting in a disjointed and fragmented narrative.
+Rating 2. Bad: There are attempts at connecting sentences, but the synthesis often feels disjointed.
+Rating 3. Moderate: The sentences are connected in a way that the synthesis is mostly cohesive, with some areas of improvement.
+Rating 4. Good: The synthesis is cohesive, with sentences well-connected to form a unified narrative.
+Rating 5. Very good: The synthesis is highly cohesive, with all sentences and paragraphs logically connected, facilitating a clear and coherent narrative flow.
+</Rating-Scale>
+
+<Response-Format>
+For each characteristic rate the quality from 1 (very bad) to 5 (very good).  Provide a short rationale for each rating. 
+Return your response in JSON format: {characteristic : {‘rating’ : ‘’, ‘rationale’ : ‘’}}
+
+<Example-Response>
+{
+  "Cohesion": {"rating": "4", "rationale": "Almost all sentences are connected appropriately and form a cohesive narrative."}
+}
+</Example-Response>
+</Response-Format>
+
+<Note>
+Your evaluation should be based solely on the content of the provided synthesis and abstracts. Ensure your rationale is objective and backed by specific examples from the provided material.
+</Note>"""
+
+class Cohesion(Rubric):
+    system_prompt_template: str = cohesion_prompt
+
+
+conciseness_prompt = """<Context> 
+Scientific synthesis generation involves creating a concise, coherent, and integrated summary from a collection of scientific texts (such as research paper titles and abstracts) that addresses a specific research question. Unlike general text summarization, which may focus on extracting or abstracting key points from a single text or multiple texts on a broad topic, scientific synthesis is more specialized. It requires:
+
+- Understanding and Addressing a Specific Research Question: The synthesis must specifically answer a research question, requiring a deep understanding of the subject matter and the ability to extract and integrate relevant information from various sources.
+- Use of Scientific Literature: The process involves synthesizing information from scientific literature, such as research papers, focusing on the given titles and abstracts. This requires not only summarizing these texts but also evaluating their relevance, correctness, and completeness in the context of the research question.
+- Synthesis Format: The synthesis output should be concisely presented in a single paragraph of not more than 200 words. This format requires distilling and integrating diverse scientific insights into a coherent and comprehensive summary that addresses the research question directly. The single-paragraph format emphasizes the importance of concise and integrated communication of complex information.
+- Synthesize vs. Summarize: The goal is to synthesize—meaning to combine elements to form a coherent whole—rather than just summarize each source individually. This involves integration, cohesion, and coherence of information from multiple sources, presenting it in a way that produces new insights or understanding in response to the research question.
+- Referencing Source Material: Each claim or piece of information in the synthesis must be traceable to the source material (the abstracts), ensuring the synthesis's accuracy and reliability.
+- Adherence to Quality Characteristics: It should be possible to evaluate the synthesis quality based on conciseness characteristic, ensuring it effectively communicates the synthesized information.
+
+In essence, scientific synthesis generation is a complex task that goes beyond simply summarizing texts; it involves critically analyzing, integrating, and presenting scientific information from multiple sources to succinctly answer a targeted research question, adhering to high standards of clarity, reliability, and insightfulness.
+</Context>
+
+<Role>
+You are tasked as a scientific syntheses quality evaluator.
+</Role>
+
+<Task-Description>
+A user will provide you with a synthesis which has been generated as an answer to a research question using the titles and abstracts of relevant research works.  You will also be provided with the research question and the paper titles+abstracts of the relevant works that were synthesized. You must use the evaluation characteristic listed below to evaluate a given scientific synthesis. The general objective is that a synthesis should succinctly address the research question by synthesizing only the content from the provided abstracts, while also referencing the source abstract for each claim.
+</Task-Description>
+
+<Evaluation-Characteristics>
+1. Conciseness: is the answer short and clear, without redundant statements?
+</Evaluation-Characteristics>
+
+<Rating-Scale>
+For a given characteristic, rate the quality from 1 (very bad) to 5 (very good). Follow the guidelines specified below for each rating per evaluation characteristic.
+
+1. Conciseness
+Rating 1. Very Bad: The synthesis is verbose and cluttered with redundant or irrelevant information, significantly detracting from its clarity and focus.
+Rating 2. Bad: The synthesis includes some redundant or irrelevant statements, detracting from its clarity.
+Rating 3. Moderate: The synthesis is relatively clear and to the point, but could be more concise by eliminating a few redundant elements.
+Rating 4. Good: The synthesis is concise and to the point, with virtually no redundant statements or unnecessary information.
+Rating 5. Very Good: The synthesis is precisely concise, delivering information clearly and directly without any superfluous details or redundancy, enhancing its clarity and impact.
+</Rating-Scale>
+
+<Response-Format>
+For each characteristic rate the quality from 1 (very bad) to 5 (very good).  Provide a short rationale for each rating. 
+Return your response in JSON format: {characteristic : {‘rating’ : ‘’, ‘rationale’ : ‘’}}
+
+<Example-Response>
+{
+  "Conciseness": {"rating": "4", "rationale": "The synthesis contains almost no unnecessary information or redundant statements and is straight to the point."}
+}
+</Example-Response>
+</Response-Format>
+
+<Note>
+Your evaluation should be based solely on the content of the provided synthesis and abstracts. Ensure your rationale is objective and backed by specific examples from the provided material.
+</Note>"""
+class Conciseness(Rubric):
+    system_prompt_template: str = conciseness_prompt
+
+readability_prompt = """<Context> 
+Scientific synthesis generation involves creating a concise, coherent, and integrated summary from a collection of scientific texts (such as research paper titles and abstracts) that addresses a specific research question. Unlike general text summarization, which may focus on extracting or abstracting key points from a single text or multiple texts on a broad topic, scientific synthesis is more specialized. It requires:
+
+- Understanding and Addressing a Specific Research Question: The synthesis must specifically answer a research question, requiring a deep understanding of the subject matter and the ability to extract and integrate relevant information from various sources.
+- Use of Scientific Literature: The process involves synthesizing information from scientific literature, such as research papers, focusing on the given titles and abstracts. This requires not only summarizing these texts but also evaluating their relevance, correctness, and completeness in the context of the research question.
+- Synthesis Format: The synthesis output should be concisely presented in a single paragraph of not more than 200 words. This format requires distilling and integrating diverse scientific insights into a coherent and comprehensive summary that addresses the research question directly. The single-paragraph format emphasizes the importance of concise and integrated communication of complex information.
+- Synthesize vs. Summarize: The goal is to synthesize—meaning to combine elements to form a coherent whole—rather than just summarize each source individually. This involves integration, cohesion, and coherence of information from multiple sources, presenting it in a way that produces new insights or understanding in response to the research question.
+- Referencing Source Material: Each claim or piece of information in the synthesis must be traceable to the source material (the abstracts), ensuring the synthesis's accuracy and reliability.
+- Adherence to Quality Characteristics: It should be possible to evaluate the synthesis quality based on readability characteristic, ensuring it effectively communicates the synthesized information.
+
+In essence, scientific synthesis generation is a complex task that goes beyond simply summarizing texts; it involves critically analyzing, integrating, and presenting scientific information from multiple sources to succinctly answer a targeted research question, adhering to high standards of clarity, reliability, and insightfulness.
+</Context>
+
+<Role>
+You are tasked as a scientific syntheses quality evaluator.
+</Role>
+
+<Task-Description>
+A user will provide you with a synthesis which has been generated as an answer to a research question using the titles and abstracts of relevant research works.  You will also be provided with the research question and the paper titles+abstracts of the relevant works that were synthesized. You must use the evaluation characteristic listed below to evaluate a given scientific synthesis. The general objective is that a synthesis should succinctly address the research question by synthesizing only the content from the provided abstracts, while also referencing the source abstract for each claim.
+</Task-Description>
+
+<Evaluation-Characteristics>
+1. Readability: does the answer follow appropriate style and structure conventions for academic writing, particularly for readability?
+</Evaluation-Characteristics>
+
+<Rating-Scale>
+For a given characteristic, rate the quality from 1 (very bad) to 5 (very good). Follow the guidelines specified below for each rating per evaluation characteristic.
+
+1. Readability
+Rating 1. Very bad: The synthesis is poorly written, with pervasive issues in style, structure, and language use, making it difficult to understand.
+Rating 2. Bad: The text has noticeable issues with style, structure, or language use, affecting clarity.
+Rating 3. Moderate: The synthesis follows appropriate conventions and uses language correctly, with minor issues in style or structure.
+Rating 4. Good: The text is well-structured and easy to read, with language that is appropriately used and only minor stylistic improvements needed.
+Rating 5. Very good: The synthesis is exceptionally well-written, following stylistic and structural conventions with precise language use, making it accessible and easy to read.
+</Rating-Scale>
+
+<Response-Format>
+For each characteristic rate the quality from 1 (very bad) to 5 (very good).  Provide a short rationale for each rating. 
+Return your response in JSON format: {characteristic : {‘rating’ : ‘’, ‘rationale’ : ‘’}}
+
+<Example-Response>
+{
+  "Readability": {"rating": "4", "rationale": "The synthesis follows academic writing conventions almost perfectly and displays appropriate style."}
+}
+</Example-Response>
+</Response-Format>
+
+<Note>
+Your evaluation should be based solely on the content of the provided synthesis and abstracts. Ensure your rationale is objective and backed by specific examples from the provided material.
+</Note>"""
+class Readability(Rubric):
+    system_prompt_template: str = readability_prompt
+