judgeval 0.0.32__py3-none-any.whl → 0.0.33__py3-none-any.whl

judgeval/scorers/judgeval_scorers/local_implementations/faithfulness/prompts.py

@@ -1,268 +0,0 @@
-from typing import List, Optional
-from pydantic import BaseModel, Field
-
-
-class FaithfulnessVerdict(BaseModel):
-    verdict: str
-    reason: Optional[str] = Field(default=None)
-
-
-class Verdicts(BaseModel):
-    verdicts: List[FaithfulnessVerdict]
-
-
-class Truths(BaseModel):
-    truths: List[str]
-
-
-class Claims(BaseModel):
-    claims: List[str]
-
-
-class Reason(BaseModel):
-    reason: str
-
-
-class FaithfulnessTemplate:
-    @staticmethod
-    def find_claims(text):
-        return f"""==== TASK INSTRUCTIONS ====
-You will be provided with a passage of text. Based on the text, your task is to generate a comprehensive list of ALL CLAIMS that can be inferred from the text.  
-For every claim that you derive from the text, provide the source of the claim via quoting the original text. Please try to extract EVERY CLAIM that is in the original text; priortize generating the most claims rather than being concise. 
-You should NOT include any prior knowledge, and take the text at face value when extracting claims.
-
-==== FORMATTING YOUR ANSWER ====
-Please return your answer in JSON format, with the "claims" key as a list of JSON objects with keys "claim" and "quote". No words or explanation beyond the output JSON is needed.
-
-==== EXAMPLES ====
-
----- START OF EXAMPLE 1 ----
-Example Text: 
-"Einstein won the nobel prize in 1968 for his discovery of the photoelectric effect."
-
-Example JSON: 
-{{
-    "claims": [
-        {{
-             "claim": "Einstein won the nobel prize for his discovery of the photoelectric effect.",
-             "quote": "Einstein won the nobel prize in 1968 for his discovery of the photoelectric effect."
-        }},
-        {{
-             "claim": "Einstein won the nobel prize in 1968.",
-             "quote": "Einstein won the nobel prize in 1968 for his discovery of the photoelectric effect."
-        }}
-    ]  
-}}
----- END OF EXAMPLE 1 ----
-
----- START OF EXAMPLE 2 ----
-Example Text: "The Wright brothers successfully flew the first powered airplane on December 17, 1903, in Kitty Hawk, North Carolina."
-
-{{
-    "claims": [
-        {{
-            "claim": "The Wright brothers flew the first powered airplane.",
-            "quote": "The Wright brothers successfully flew the first powered airplane on December 17, 1903, in Kitty Hawk, North Carolina."
-        }},
-        {{
-            "claim": "The Wright brothers made their flight in Kitty Hawk, North Carolina.",
-            "quote": "The Wright brothers successfully flew the first powered airplane on December 17, 1903, in Kitty Hawk, North Carolina."
-        }},
-        {{
-            "claim": "The first powered airplane flight occurred on December 17, 1903.",
-            "quote": "The Wright brothers successfully flew the first powered airplane on December 17, 1903, in Kitty Hawk, North Carolina."
-        }}
-    ]
-}}
----- END OF EXAMPLE 2 ----
-
----- START OF EXAMPLE 3 ----
-Example Text:
-"The Great Wall of China was built over many centuries by different Chinese dynasties. Construction began more than 2,000 years ago during the Warring States period. The most famous sections were built during the Ming Dynasty. The wall stretches for thousands of miles across northern China and was primarily built for military defense."
-
-Example JSON:
-{{
-    "claims": [
-        {{
-            "claim": "The Great Wall of China was built by multiple Chinese dynasties",
-            "quote": "The Great Wall of China was built over many centuries by different Chinese dynasties."
-        }},
-        {{
-            "claim": "Construction of the Great Wall began over 2,000 years ago",
-            "quote": "Construction began more than 2,000 years ago during the Warring States period."
-        }},
-        {{
-            "claim": "Construction started during the Warring States period",
-            "quote": "Construction began more than 2,000 years ago during the Warring States period."
-        }},
-        {{
-            "claim": "The most well-known parts of the wall were constructed during the Ming Dynasty",
-            "quote": "The most famous sections were built during the Ming Dynasty."
-        }},
-        {{
-            "claim": "The Great Wall extends for thousands of miles",
-            "quote": "The wall stretches for thousands of miles across northern China"
-        }},
-        {{
-            "claim": "The wall is located in northern China",
-            "quote": "The wall stretches for thousands of miles across northern China"
-        }},
-        {{
-            "claim": "The Great Wall was constructed for defensive military purposes",
-            "quote": "was primarily built for military defense."
-        }}
-    ]
-}}
----- END OF EXAMPLE 3 ----
-
-==== END OF EXAMPLES ====
-
-==== YOUR TURN ====
-
-Example Text:
-{text}
-
-JSON:
-"""
-
-    @staticmethod
-    def create_verdicts(claims, retrieval_context):
-        return f"""==== TASK INSTRUCTIONS ====
-You will be provided with a list of claims from an LLM's output text, accompanied by the retrieval documents that the LLM used to generate the output. 
-I'm pretty sure that many of the claims are factually contradictory to the retrieval context, but I want you to double check that I'm right. 
-For each claim, choose one of ("yes", "no", or "idk") to represent whether the claim is correct based on the retrieval context. 
-YOU SHOULD be very scrutinous--if any part of the claim is contradicted by the retrieval context, you should choose "no". Think really hard about finding the contradictions, since they can be subtle!
-
-Choose 'no' if the retrieval context CONTRADICTS the claims. YOU SHOULD NEVER USE YOUR PRIOR KNOWLEDGE IN YOUR JUDGMENT.
-Claims made using vague, suggestive, or speculative language such as 'may have', 'possibility due to', do NOT count as a contradiction.
-Claims that are fuzzy based on lack of information MUST BE ANSWERED with 'idk'.
-
-==== FORMATTING YOUR ANSWER ====
-Please return your answer in JSON format, with the 'verdicts' key as a list of JSON objects. Each JSON object should have 2 fields: 'verdict' and 'reason'. 
-The 'verdict' key should be either 'yes', 'no', or 'idk', which states WHETHER THE CLAIM AGREES with the context. 
-The 'reason' key should be a string explaining why the claim is 'yes', 'no', or 'idk'. Make specific reference to the retrieval context to justify your verdict.
-
-==== EXAMPLES ====
----- START OF EXAMPLE 1 ----
-Example retrieval contexts: "Einstein won the Nobel Prize for his discovery of the photoelectric effect. Einstein won the Nobel Prize in 1968. Einstein is a German Scientist."
-Example claims: ["Barack Obama is a caucasian male.", "Zurich is a city in London", "Einstein won the Nobel Prize for the discovery of the photoelectric effect which may have contributed to his fame.", "Einstein won the Nobel Prize in 1969 for his discovery of the photoelectric effect.", "Einstein was a Germen chef."]
-
-Example JSON:
-{{
-    "verdicts": [
-        {{
-            "verdict": "idk",
-            "reason": "The claim about Barack Obama's ethnicity cannot be verified from the given retrieval context as it contains no information about Barack Obama."
-        }},
-        {{
-            "verdict": "idk", 
-            "reason": "The claim about Zurich being a city in London cannot be verified from the given retrieval context as it contains no information about Zurich or London."
-        }},
-        {{
-            "verdict": "yes",
-            "reason": "The retrieval context confirms that Einstein won the Nobel Prize for discovering the photoelectric effect."
-        }},
-        {{
-            "verdict": "no",
-            "reason": "The actual output claims Einstein won the Nobel Prize in 1969, which is untrue as the retrieval context states it is 1968 instead."
-        }},
-        {{
-            "verdict": "no",
-            "reason": "The actual output claims Einstein is a Germen chef, which is not correct as the retrieval context states he was a German scientist instead."
-        }}
-    ]  
-}}
----- END OF EXAMPLE 1 ----
----- START OF EXAMPLE 2 ----
-Example retrieval contexts: "The Great Wall of China was built over many centuries by different Chinese dynasties. Construction began more than 2,000 years ago. The wall stretches for thousands of miles across China's northern borders. Most of the existing wall was built during the Ming Dynasty."
-Example claims: ["The Great Wall was built in a single year.", "The Great Wall may have taken centuries to complete.", "The Great Wall was built by the Romans.", "The Great Wall is located in China's northern region.", "The Great Wall is 100 meters long."]
-
-Example JSON:
-{{
-    "verdicts": [
-        {{
-            "verdict": "no",
-            "reason": "The claim that the Great Wall was built in a single year directly contradicts the retrieval context, which states it was built over many centuries."
-        }},
-        {{
-            "verdict": "yes",
-            "reason": "The retrieval context confirms that the Great Wall was built over many centuries by different Chinese dynasties."
-        }},
-        {{
-            "verdict": "no",
-            "reason": "The claim states the Romans built the wall, which contradicts the retrieval context that specifies it was built by Chinese dynasties."
-        }},
-        {{
-            "verdict": "yes",
-            "reason": "The retrieval context explicitly states that the wall stretches across China's northern borders."
-        }},
-        {{
-            "verdict": "no",
-            "reason": "The claim that the wall is 100 meters long contradicts the retrieval context which states it stretches for thousands of miles."
-        }}
-    ]
-}}
----- END OF EXAMPLE 2 ----
-==== END OF EXAMPLES ====
-
-==== YOUR TURN ====
-Retrieval Contexts:
-{retrieval_context}
-
-Claims:
-{claims}
-
-JSON:
-"""
-
-    @staticmethod
-    def justify_reason(score, contradictions):
-        return f"""==== TASK INSTRUCTIONS ====
-You will be provided with a list of contradictions and a faithfulness score. 
-The list of contradictions will be references to statements made by a RAG generator that contradicted one or more document(s) from the retrieval context. 
-- To clarify, the LLM produced an `actual output` that contained claims that contradicted the `retrieval context` used to produce the output.
-The faithfulness score is a 0 - 1 float (1 is highest) indicating how factually consistent the RAG generator's output is to the retrieval context.
-
-Your task is to CLEARLY and CONCISELY summarize the contradictions to justify the score. 
-If there are no contradictions, just say something positive with an upbeat encouraging tone (but don't overdo it otherwise it gets annoying). 
-Your reason MUST use information from the contradictions in your reason.
-
-
-==== FORMATTING YOUR ANSWER ====
-Please make sure to only return in JSON format, with the 'reason' key providing the reason.
-
-Example JSON:
-{{
-    "reason": "The score is <faithfulness_score> because <your_reason>."
-}}
-
-==== EXAMPLE ====
-Example Contradictions:
-[
-    {{
-        "verdict": "no", 
-        "reason": "The output claims Marie Curie was born in Russia, but the context clearly states she was born in Warsaw, Poland."
-    }},
-    {{
-        "verdict": "no",
-        "reason": "The output states Marie Curie discovered uranium, but the context indicates she discovered radium and polonium."
-    }}
-]
-
-Example Faithfulness Score:
-0.60
-
-Example Response:
-{{
-    "reason": "The score is 0.60 because the output made two significant factual errors: incorrectly stating Marie Curie was born in Russia instead of Poland, and wrongly attributing the discovery of uranium to her instead of radium and polonium."
-}}
-
-==== YOUR TURN ====
-Faithfulness Score:
-{score}
-
-Contradictions:
-{contradictions}
-
-JSON:
-"""

judgeval/scorers/judgeval_scorers/local_implementations/hallucination/__init__.py

@@ -1,3 +0,0 @@
-from judgeval.scorers.judgeval_scorers.local_implementations.hallucination.hallucination_scorer import HallucinationScorer
-
-__all__ = ["HallucinationScorer"]

judgeval/scorers/judgeval_scorers/local_implementations/hallucination/hallucination_scorer.py

@@ -1,264 +0,0 @@
-"""
-Metric that evaluates hallucinations in model outputs
-
-The hallucination metric determines whether your LLM generates factually correct information by comparing 
-the actual_output to the provided context.
-
-If you're looking to evaluate hallucination for a RAG system, refer to the faithfulness metric instead.
-
-The HallucinationMetric uses an LLM to determine, for each context in contexts, whether there are any 
-contradictions to the actual_output.
-
-Although extremely similar to the FaithfulnessMetric, the HallucinationMetric is calculated differently 
-since it uses contexts as the source of truth instead. Since contexts is the ideal segment of your 
-knowledge base relevant to a specific input, the degree of hallucination can be measured by the degree 
-of which the contexts is disagreed upon.
-
-Faithfulness is measuring the number of statements in output that agree with contexts.
-Hallucination is measuring the fraction of contexts that agree with output (do not contradict == agree)
-"""
-
-from typing import Optional, Union, List
-
-from judgeval.constants import APIScorer
-from judgeval.scorers.utils import (
-    get_or_create_event_loop,
-    scorer_progress_meter,
-    create_verbose_logs,
-    parse_response_json,
-    check_example_params,
-)
-from judgeval.scorers import JudgevalScorer
-from judgeval.judges import JudgevalJudge
-from judgeval.judges.utils import create_judge
-from judgeval.data import Example, ExampleParams
-from judgeval.scorers.judgeval_scorers.local_implementations.hallucination.prompts import *
-
-
-required_params = [
-    ExampleParams.INPUT,
-    ExampleParams.ACTUAL_OUTPUT,
-    ExampleParams.CONTEXT,
-]
-
-
-class HallucinationScorer(JudgevalScorer):
-    def __init__(
-        self,
-        threshold: float = 0.5,
-        model: Optional[Union[str, JudgevalJudge]] = None,
-        include_reason: bool = True,
-        async_mode: bool = False,
-        strict_mode: bool = False,
-        verbose_mode: bool = False,
-    ):
-        super().__init__(
-            score_type=APIScorer.HALLUCINATION,
-            threshold=1 if strict_mode else threshold,
-            evaluation_model=None,
-            include_reason=include_reason,
-            async_mode=async_mode,
-            strict_mode=strict_mode,
-            verbose_mode=verbose_mode
-        )
-        self.model, self.using_native_model = create_judge(model)
-        self.evaluation_model = self.model.get_model_name()
-
-    def score_example(
-        self,
-        example: Example,
-        _show_indicator: bool = True,
-    ) -> float:
-        check_example_params(example, required_params, self)
-
-        with scorer_progress_meter(self, display_meter=_show_indicator):
-            if self.async_mode:
-                loop = get_or_create_event_loop()
-                loop.run_until_complete(
-                    self.a_score_example(example, _show_indicator=False)
-                )
-            else:
-                self.verdicts: List[HallucinationVerdict] = (
-                    self._generate_verdicts(
-                        example.actual_output, example.context
-                    )
-                )
-                self.score = self._calculate_score()
-                self.reason = self._generate_reason()
-                self.success = self.score <= self.threshold
-                self.verbose_logs = create_verbose_logs(
-                    self,
-                    steps=[
-                        f"Verdicts:\n{[v.model_dump() for v in self.verdicts]}",
-                        f"Score: {self.score}\nReason: {self.reason}",
-                    ],
-                )
-
-                return self.score
-
-    async def a_score_example(
-        self,
-        example: Example,
-        _show_indicator: bool = True,
-    ) -> float:
-        check_example_params(example, required_params, self)
-
-        with scorer_progress_meter(
-            self, async_mode=True, display_meter=_show_indicator
-        ):
-            self.verdicts: List[HallucinationVerdict] = (
-                await self._a_generate_verdicts(
-                    example.actual_output, example.context
-                )
-            )
-            self.score = self._calculate_score()
-            self.reason = await self._a_generate_reason()
-            self.success = self.score <= self.threshold
-            self.verbose_logs = create_verbose_logs(
-                self,
-                steps=[
-                    f"Verdicts:\n{[v.model_dump() for v in self.verdicts]}",
-                    f"Score: {self.score}\nReason: {self.reason}",
-                ],
-            )
-
-            return self.score
-
-    async def _a_generate_reason(self):
-        if self.include_reason is False:
-            return None
-
-        contradictions = []
-        for verdict in self.verdicts:
-            if verdict.verdict.strip().lower() == "no":
-                contradictions.append(verdict.reason)
-
-        prompt: dict = HallucinationTemplate.generate_reason(
-            contradictions=contradictions,
-            score=format(self.score, ".2f"),
-        )
-
-        if self.using_native_model:
-            res = await self.model.a_generate(prompt)
-            data = parse_response_json(res, self)
-            return data["reason"]
-        else:
-            try:
-                res: Reason = await self.model.a_generate(prompt, schema=Reason)
-                return res.reason
-            except TypeError:
-                res = await self.model.a_generate(prompt)
-                data = parse_response_json(res, self)
-                return data["reason"]
-
-    def _generate_reason(self):
-        if self.include_reason is False:
-            return None
-
-        factual_alignments = []
-        contradictions = []
-        for verdict in self.verdicts:
-            if verdict.verdict.strip().lower() == "no":
-                contradictions.append(verdict.reason)
-
-        prompt: dict = HallucinationTemplate.generate_reason(
-            factual_alignments=factual_alignments,
-            contradictions=contradictions,
-            score=format(self.score, ".2f"),
-        )
-
-        if self.using_native_model:
-            res = self.model.generate(prompt)
-            data = parse_response_json(res, self)
-            return data["reason"]
-        else:
-            try:
-                res: Reason = self.model.generate(prompt, schema=Reason)
-                return res.reason
-            except TypeError:
-                res = self.model.generate(prompt)
-                data = parse_response_json(res, self)
-                return data["reason"]
-
-    async def _a_generate_verdicts(
-        self, actual_output: str, contexts: List[str]
-    ) -> List[HallucinationVerdict]:
-        verdicts: List[HallucinationVerdict] = []
-        prompt = HallucinationTemplate.generate_verdicts(
-            actual_output=actual_output, contexts=contexts
-        )
-        if self.using_native_model:
-            res = await self.model.a_generate(prompt)
-            data = parse_response_json(res, self)
-            verdicts = [
-                HallucinationVerdict(**item) for item in data["verdicts"]
-            ]
-            return verdicts
-        else:
-            try:
-                res: Verdicts = await self.model.a_generate(
-                    prompt, schema=Verdicts
-                )
-                verdicts = [item for item in res.verdicts]
-                return verdicts
-            except TypeError:
-                res = await self.model.a_generate(prompt)
-                data = parse_response_json(res, self)
-                verdicts = [
-                    HallucinationVerdict(**item) for item in data["verdicts"]
-                ]
-                return verdicts
-
-    def _generate_verdicts(
-        self, actual_output: str, contexts: List[str]
-    ) -> List[HallucinationVerdict]:
-        verdicts: List[HallucinationVerdict] = []
-        prompt = HallucinationTemplate.generate_verdicts(
-            actual_output=actual_output, contexts=contexts
-        )
-        if self.using_native_model:
-            res = self.model.generate(prompt)
-            data = parse_response_json(res, self)
-            verdicts = [
-                HallucinationVerdict(**item) for item in data["verdicts"]
-            ]
-            return verdicts
-        else:
-            try:
-                res: Verdicts = self.model.generate(prompt, schema=Verdicts)
-                verdicts = [item for item in res.verdicts]
-                return verdicts
-            except TypeError:
-                res = self.model.generate(prompt)
-                data = parse_response_json(res, self)
-                verdicts = [
-                    HallucinationVerdict(**item) for item in data["verdicts"]
-                ]
-                return verdicts
-
-    def _calculate_score(self) -> float:
-        number_of_verdicts = len(self.verdicts)
-        if number_of_verdicts == 0:
-            return 0
-
-        hallucination_count = 0
-        for verdict in self.verdicts:
-            if verdict.verdict.strip().lower() == "no":
-                hallucination_count += 1
-
-        score = hallucination_count / number_of_verdicts
-        return 1 if self.strict_mode and score > self.threshold else score
-
-    def _success_check(self) -> bool:
-        if self.error is not None:
-            self.success = False
-        else:
-            try:
-                self.success = self.score <= self.threshold
-            except:
-                self.success = False
-        return self.success
-
-    @property
-    def __name__(self):
-        return "Hallucination"

judgeval/scorers/judgeval_scorers/local_implementations/hallucination/prompts.py

@@ -1,104 +0,0 @@
-from typing import List
-from pydantic import BaseModel
-
-
-class HallucinationVerdict(BaseModel):
-    verdict: str
-    reason: str
-
-
-class Verdicts(BaseModel):
-    verdicts: List[HallucinationVerdict]
-
-
-class Reason(BaseModel):
-    reason: str
-
-
-class HallucinationTemplate:
-    @staticmethod
-    def generate_verdicts(actual_output, contexts):
-        return f"""==== TASK INSTRUCTIONS ====
-You will be provided with an `actual output` (the response of an LLM to a particular query) and `contexts` (ground truth contextual information from a knowledge base).
-Your task is to take each context in contexts and determine whether the `actual output` factually agrees with the context.
-
-Additional notes:
-You should NOT use any prior knowledge you have in your decision making process; take each context at face value. 
-Since you will determine a verdict for EACH context, the number of 'verdicts' is EXACTLY EQUAL TO the number of contexts. 
-You should be lenient in your judgment when the actual output lacks detail with respect to the context segment; you should ONLY provide a 'no' answer if the context contradicts the actual output.
-
-==== FORMATTING INSTRUCTIONS ====
-You should return a JSON object with a key 'verdicts', which is a list of JSON objects. Each JSON object corresponds to a context in `contexts`, and should have 2 fields: 'verdict' and 'reason'. 
-The 'verdict' key should be EXACTLY one of 'yes' or 'no', representing whether the `actual output` factually agrees with the context segment. 
-The 'reason' is the justification for the verdict. If your verdict is 'no', try to provide a correction in the reason. 
-
-==== EXAMPLE ====
-Example contexts: ["Einstein won the Nobel Prize for his discovery of the photoelectric effect.", "Einstein won the Nobel Prize in 1968."]
-Example actual output: "Einstein won the Nobel Prize in 1969 for his discovery of the photoelectric effect."
-
-Example:
-{{
-    "verdicts": [
-        {{
-            "verdict": "yes",
-            "reason": "The actual output agrees with the provided context which states that Einstein won the Nobel Prize for his discovery of the photoelectric effect."
-        }},
-        {{
-            "verdict": "no",
-            "reason": "The actual output contradicts the provided context which states that Einstein won the Nobel Prize in 1968, not 1969."
-        }}
-    ]  
-}}
-
-==== YOUR TURN ====
-Contexts:
-{contexts}
-
-Actual Output:
-{actual_output}
-
-JSON:
-"""
-
-    @staticmethod
-    def generate_reason(contradictions, score):
-        return f"""==== TASK INSTRUCTIONS ====
-An LLM has been provided with a list of `contexts` (ground truth contextual information from a knowledge base) and `actual output` (the response of an LLM to a particular query). 
-You will be provided with a list of `contradictions`, which are factual discrepancies between the context segments and the actual output. 
-Additionally, you will be provided with a hallucination score, which is a float (0 - 1, where 0 is the best score) indicating the fraction of context segments that contradict the actual output.
-
-Your task is to provide a CLEAR and CONCISE reason for the hallucination score. 
-If the hallucination score is 0 (no contradictions), you should instead respond with a positive remark with an upbeat encouraging tone (but don't overblow the kind attitude).
-        
-==== FORMATTING INSTRUCTIONS ====
-Please make sure to only return in JSON format, with the 'reason' key providing the reason.
-Example JSON:
-{{
-    "reason": "The score is <hallucination_score> because <your_reason>."
-}}
-
-==== EXAMPLE ====
-Example Contradictions:
-[
-    "The actual output claims Einstein won the Nobel Prize in 1969, which contradicts the context stating he won it in 1968.",
-    "The actual output states Einstein was a chemist, but the context indicates he was a physicist.",
-    "The actual output claims Einstein was born in Switzerland, while the context states he was born in Germany."
-]
-
-Example Hallucination Score: 
-0.75
-
-Example Response:
-{{
-    "reason": "The score is 0.75 because the actual output made multiple factual errors: incorrectly stating Einstein's Nobel Prize year (1969 vs 1968), his profession (chemist vs physicist), and birthplace (Switzerland vs Germany)."
-}}
-
-==== YOUR TURN ====
-Contradictions:
-{contradictions}
-
-Hallucination Score:
-{score}
-
-JSON:
-"""