deepeval 4.3.2__tar.gz → 4.3.3__tar.gz

{deepeval-4.3.2 → deepeval-4.3.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: deepeval
-Version: 4.3.2
+Version: 4.3.3
 Summary: The LLM Evaluation Framework
 License: Apache-2.0
 License-File: LICENSE.md
@@ -114,7 +114,7 @@ Description-Content-Type: text/markdown
 
 **DeepEval** is a simple-to-use, open-source LLM evaluation framework, for evaluating large-language model systems. It is similar to Pytest but specialized for unit testing LLM apps. DeepEval incorporates the latest research to run evals via metrics such as G-Eval, task completion, answer relevancy, hallucination, etc., which use LLM-as-a-judge and other NLP models that run **locally on your machine**.
 
-Whether you're building AI agents, multi-turn chatbots, voice agents, RAG pipelines, or anything else, implemented via LangChain or OpenAI, DeepEval has you covered. With it, you can easily evaluate:
+Whether you're building AI agents, voice agents, multi-turn chatbots, RAG pipelines, or anything else, implemented via LangChain or OpenAI, DeepEval has you covered. With it, you can easily evaluate:
 
 - **LLM apps end-to-end** as black boxes
 - **Complete AI agent trajectories** across every decision and action
@@ -155,17 +155,6 @@ Use these evaluations to determine the optimal models, prompts, and architecture
 
     </details>
 
-  - <details>
-    <summary><b>Multi-Turn Metrics</b></summary>
-
-    - [Knowledge Retention](https://deepeval.com/docs/metrics-knowledge-retention) — evaluate whether the chatbot retains factual information throughout a conversation
-    - [Conversation Completeness](https://deepeval.com/docs/metrics-conversation-completeness) — measure whether the chatbot satisfies user needs throughout a conversation
-    - [Turn Relevancy](https://deepeval.com/docs/metrics-turn-relevancy) — evaluate whether the chatbot generates consistently relevant responses throughout a conversation
-    - [Turn Faithfulness](https://deepeval.com/docs/metrics-turn-faithfulness) — check if the chatbot's responses are factually grounded in retrieval context across turns
-    - [Role Adherence](https://deepeval.com/docs/metrics-role-adherence) — evaluate whether the chatbot adheres to its assigned role throughout a conversation
-
-    </details>
-
   - <details>
     <summary><b>Voice Agent Metrics</b></summary>
 
@@ -177,6 +166,17 @@ Use these evaluations to determine the optimal models, prompts, and architecture
 
     </details>
 
+  - <details>
+    <summary><b>Multi-Turn Metrics</b></summary>
+
+    - [Knowledge Retention](https://deepeval.com/docs/metrics-knowledge-retention) — evaluate whether the chatbot retains factual information throughout a conversation
+    - [Conversation Completeness](https://deepeval.com/docs/metrics-conversation-completeness) — measure whether the chatbot satisfies user needs throughout a conversation
+    - [Turn Relevancy](https://deepeval.com/docs/metrics-turn-relevancy) — evaluate whether the chatbot generates consistently relevant responses throughout a conversation
+    - [Turn Faithfulness](https://deepeval.com/docs/metrics-turn-faithfulness) — check if the chatbot's responses are factually grounded in retrieval context across turns
+    - [Role Adherence](https://deepeval.com/docs/metrics-role-adherence) — evaluate whether the chatbot adheres to its assigned role throughout a conversation
+
+    </details>
+
   - <details>
     <summary><b>RAG Metrics</b></summary>
 
@@ -223,7 +223,7 @@ Use these evaluations to determine the optimal models, prompts, and architecture
 
 - 🎯 Supports both end-to-end and component-level LLM evaluation.
 - 🧩 Build your own custom metrics that are automatically integrated with DeepEval's ecosystem.
-- 🔮 Generate both single and multi-turn synthetic datasets for evaluation.
+- 🔮 Generate single and multi-turn synthetic datasets, and simulate live voice calls against your voice agent.
 - 🔗 Integrates seamlessly with **ANY** CI/CD environment.
 - 🧬 Optimize prompts automatically based on evaluation results.
 - 🏆 Easily benchmark **ANY** LLM on popular LLM benchmarks in [under 10 lines of code.](https://deepeval.com/docs/benchmarks-introduction?utm_source=GitHub), including MMLU, HellaSwag, DROP, BIG-Bench Hard, TruthfulQA, HumanEval, GSM8K.
@@ -232,7 +232,7 @@ Use these evaluations to determine the optimal models, prompts, and architecture
 
 # 🔌 Integrations
 
-DeepEval plugs into any LLM framework — OpenAI Agents, LangChain, CrewAI, and more. The same traces can go to **Confident AI** once you're live, where evals run on every production trace.
+DeepEval plugs into any LLM framework — OpenAI Agents, LangChain, CrewAI, and more — and into voice stacks like Vapi, ElevenLabs, LiveKit, and Pipecat. The same traces can go to **Confident AI** once you're live, where evals run on every production trace.
 
 ## Frameworks
 
@@ -249,13 +249,20 @@ DeepEval plugs into any LLM framework — OpenAI Agents, LangChain, CrewAI, and
 - [OpenAI Agents](https://www.deepeval.com/integrations/frameworks/openai-agents?utm_source=GitHub) — evaluate OpenAI Agents end-to-end in under a minute
 - [LlamaIndex](https://www.deepeval.com/integrations/frameworks/llamaindex?utm_source=GitHub) — evaluate AI agents and RAG applications built with LlamaIndex
 
+## Voice Agent Connectors
+
+- [Vapi](https://www.deepeval.com/integrations/voice-agent-connectors/vapi?utm_source=GitHub) — simulate calls to a Vapi assistant over its WebSocket transport
+- [ElevenLabs](https://www.deepeval.com/integrations/voice-agent-connectors/elevenlabs?utm_source=GitHub) — simulate calls to an ElevenLabs conversational agent
+- [LiveKit](https://www.deepeval.com/integrations/voice-agent-connectors/livekit?utm_source=GitHub) — join your agent's LiveKit room and trade audio over WebRTC
+- [Pipecat](https://www.deepeval.com/integrations/voice-agent-connectors/pipecat?utm_source=GitHub) — connect to your Pipecat pipeline over its WebSocket transport
+
 ## ☁️ Platform + Ecosystem
 
 [Confident AI](https://www.confident-ai.com?utm_source=deepeval&utm_medium=github&utm_content=platform_section&ref_page=github/readme) is AI monitoring for applications in production. DeepEval tests what you know to test before you ship; Confident AI finds what you didn't, and it's model- and framework-agnostic.
 
 - Failing traces surface before anyone has written an eval for them, clustered into failure modes, with user sentiment and behavior across traces.
 - Failing traces route to the right annotators and become goldens, so production issues turn into your next eval.
-- For enterprise teams, one system for AI quality: governance as release gates and native red teaming.
+- For enterprise teams, one system for AI quality: evals, monitoring, human annotation, and voice-native observability, with governance as release gates.
 - Prefer to stay in your editor? Inspect failing traces and pull the datasets built from them in Claude Code or Cursor through Confident AI's [MCP server](https://github.com/confident-ai/confident-mcp-server).
 
 <p align="center">
@@ -266,7 +273,7 @@ DeepEval plugs into any LLM framework — OpenAI Agents, LangChain, CrewAI, and
 
 # 🤖 Vibe-Coder QuickStart
 
-Want your coding agent to add evals and fix failures for you? Install the DeepEval skill, point it at your AI agent, multi-turn chatbot, voice agent, RAG pipeline, or any other LLM app, and ask it to generate a dataset, write the eval suite, run `deepeval test run`, and iterate on the failing metrics.
+Want your coding agent to add evals and fix failures for you? Install the DeepEval skill, point it at your AI agent, voice agent, multi-turn chatbot, RAG pipeline, or any other LLM app, and ask it to generate a dataset, write the eval suite, run `deepeval test run`, and iterate on the failing metrics.
 
 [Start with the 5-minute vibe-coder guide](https://deepeval.com/docs/vibe-coder-quickstart?utm_source=GitHub).
 
@@ -670,7 +677,7 @@ print(answer_relevancy_metric.score)
 print(answer_relevancy_metric.reason)
 ```
 
-Note that some metrics are built for specific use cases, such as AI agents, multi-turn chatbots, voice agents, RAG, and more. Make sure to use our docs to pick the right one for your use case.
+Note that some metrics are built for specific use cases, such as AI agents, voice agents, multi-turn chatbots, RAG, and more. Make sure to use our docs to pick the right one for your use case.
 
 ## A Note on Env Variables (.env / .env.local)
 

{deepeval-4.3.2 → deepeval-4.3.3}/README.md

@@ -61,7 +61,7 @@
 
 **DeepEval** is a simple-to-use, open-source LLM evaluation framework, for evaluating large-language model systems. It is similar to Pytest but specialized for unit testing LLM apps. DeepEval incorporates the latest research to run evals via metrics such as G-Eval, task completion, answer relevancy, hallucination, etc., which use LLM-as-a-judge and other NLP models that run **locally on your machine**.
 
-Whether you're building AI agents, multi-turn chatbots, voice agents, RAG pipelines, or anything else, implemented via LangChain or OpenAI, DeepEval has you covered. With it, you can easily evaluate:
+Whether you're building AI agents, voice agents, multi-turn chatbots, RAG pipelines, or anything else, implemented via LangChain or OpenAI, DeepEval has you covered. With it, you can easily evaluate:
 
 - **LLM apps end-to-end** as black boxes
 - **Complete AI agent trajectories** across every decision and action
@@ -102,17 +102,6 @@ Use these evaluations to determine the optimal models, prompts, and architecture
 
     </details>
 
-  - <details>
-    <summary><b>Multi-Turn Metrics</b></summary>
-
-    - [Knowledge Retention](https://deepeval.com/docs/metrics-knowledge-retention) — evaluate whether the chatbot retains factual information throughout a conversation
-    - [Conversation Completeness](https://deepeval.com/docs/metrics-conversation-completeness) — measure whether the chatbot satisfies user needs throughout a conversation
-    - [Turn Relevancy](https://deepeval.com/docs/metrics-turn-relevancy) — evaluate whether the chatbot generates consistently relevant responses throughout a conversation
-    - [Turn Faithfulness](https://deepeval.com/docs/metrics-turn-faithfulness) — check if the chatbot's responses are factually grounded in retrieval context across turns
-    - [Role Adherence](https://deepeval.com/docs/metrics-role-adherence) — evaluate whether the chatbot adheres to its assigned role throughout a conversation
-
-    </details>
-
   - <details>
     <summary><b>Voice Agent Metrics</b></summary>
 
@@ -124,6 +113,17 @@ Use these evaluations to determine the optimal models, prompts, and architecture
 
     </details>
 
+  - <details>
+    <summary><b>Multi-Turn Metrics</b></summary>
+
+    - [Knowledge Retention](https://deepeval.com/docs/metrics-knowledge-retention) — evaluate whether the chatbot retains factual information throughout a conversation
+    - [Conversation Completeness](https://deepeval.com/docs/metrics-conversation-completeness) — measure whether the chatbot satisfies user needs throughout a conversation
+    - [Turn Relevancy](https://deepeval.com/docs/metrics-turn-relevancy) — evaluate whether the chatbot generates consistently relevant responses throughout a conversation
+    - [Turn Faithfulness](https://deepeval.com/docs/metrics-turn-faithfulness) — check if the chatbot's responses are factually grounded in retrieval context across turns
+    - [Role Adherence](https://deepeval.com/docs/metrics-role-adherence) — evaluate whether the chatbot adheres to its assigned role throughout a conversation
+
+    </details>
+
   - <details>
     <summary><b>RAG Metrics</b></summary>
 
@@ -170,7 +170,7 @@ Use these evaluations to determine the optimal models, prompts, and architecture
 
 - 🎯 Supports both end-to-end and component-level LLM evaluation.
 - 🧩 Build your own custom metrics that are automatically integrated with DeepEval's ecosystem.
-- 🔮 Generate both single and multi-turn synthetic datasets for evaluation.
+- 🔮 Generate single and multi-turn synthetic datasets, and simulate live voice calls against your voice agent.
 - 🔗 Integrates seamlessly with **ANY** CI/CD environment.
 - 🧬 Optimize prompts automatically based on evaluation results.
 - 🏆 Easily benchmark **ANY** LLM on popular LLM benchmarks in [under 10 lines of code.](https://deepeval.com/docs/benchmarks-introduction?utm_source=GitHub), including MMLU, HellaSwag, DROP, BIG-Bench Hard, TruthfulQA, HumanEval, GSM8K.
@@ -179,7 +179,7 @@ Use these evaluations to determine the optimal models, prompts, and architecture
 
 # 🔌 Integrations
 
-DeepEval plugs into any LLM framework — OpenAI Agents, LangChain, CrewAI, and more. The same traces can go to **Confident AI** once you're live, where evals run on every production trace.
+DeepEval plugs into any LLM framework — OpenAI Agents, LangChain, CrewAI, and more — and into voice stacks like Vapi, ElevenLabs, LiveKit, and Pipecat. The same traces can go to **Confident AI** once you're live, where evals run on every production trace.
 
 ## Frameworks
 
@@ -196,13 +196,20 @@ DeepEval plugs into any LLM framework — OpenAI Agents, LangChain, CrewAI, and
 - [OpenAI Agents](https://www.deepeval.com/integrations/frameworks/openai-agents?utm_source=GitHub) — evaluate OpenAI Agents end-to-end in under a minute
 - [LlamaIndex](https://www.deepeval.com/integrations/frameworks/llamaindex?utm_source=GitHub) — evaluate AI agents and RAG applications built with LlamaIndex
 
+## Voice Agent Connectors
+
+- [Vapi](https://www.deepeval.com/integrations/voice-agent-connectors/vapi?utm_source=GitHub) — simulate calls to a Vapi assistant over its WebSocket transport
+- [ElevenLabs](https://www.deepeval.com/integrations/voice-agent-connectors/elevenlabs?utm_source=GitHub) — simulate calls to an ElevenLabs conversational agent
+- [LiveKit](https://www.deepeval.com/integrations/voice-agent-connectors/livekit?utm_source=GitHub) — join your agent's LiveKit room and trade audio over WebRTC
+- [Pipecat](https://www.deepeval.com/integrations/voice-agent-connectors/pipecat?utm_source=GitHub) — connect to your Pipecat pipeline over its WebSocket transport
+
 ## ☁️ Platform + Ecosystem
 
 [Confident AI](https://www.confident-ai.com?utm_source=deepeval&utm_medium=github&utm_content=platform_section&ref_page=github/readme) is AI monitoring for applications in production. DeepEval tests what you know to test before you ship; Confident AI finds what you didn't, and it's model- and framework-agnostic.
 
 - Failing traces surface before anyone has written an eval for them, clustered into failure modes, with user sentiment and behavior across traces.
 - Failing traces route to the right annotators and become goldens, so production issues turn into your next eval.
-- For enterprise teams, one system for AI quality: governance as release gates and native red teaming.
+- For enterprise teams, one system for AI quality: evals, monitoring, human annotation, and voice-native observability, with governance as release gates.
 - Prefer to stay in your editor? Inspect failing traces and pull the datasets built from them in Claude Code or Cursor through Confident AI's [MCP server](https://github.com/confident-ai/confident-mcp-server).
 
 <p align="center">
@@ -213,7 +220,7 @@ DeepEval plugs into any LLM framework — OpenAI Agents, LangChain, CrewAI, and
 
 # 🤖 Vibe-Coder QuickStart
 
-Want your coding agent to add evals and fix failures for you? Install the DeepEval skill, point it at your AI agent, multi-turn chatbot, voice agent, RAG pipeline, or any other LLM app, and ask it to generate a dataset, write the eval suite, run `deepeval test run`, and iterate on the failing metrics.
+Want your coding agent to add evals and fix failures for you? Install the DeepEval skill, point it at your AI agent, voice agent, multi-turn chatbot, RAG pipeline, or any other LLM app, and ask it to generate a dataset, write the eval suite, run `deepeval test run`, and iterate on the failing metrics.
 
 [Start with the 5-minute vibe-coder guide](https://deepeval.com/docs/vibe-coder-quickstart?utm_source=GitHub).
 
@@ -617,7 +624,7 @@ print(answer_relevancy_metric.score)
 print(answer_relevancy_metric.reason)
 ```
 
-Note that some metrics are built for specific use cases, such as AI agents, multi-turn chatbots, voice agents, RAG, and more. Make sure to use our docs to pick the right one for your use case.
+Note that some metrics are built for specific use cases, such as AI agents, voice agents, multi-turn chatbots, RAG, and more. Make sure to use our docs to pick the right one for your use case.
 
 ## A Note on Env Variables (.env / .env.local)
 

deepeval-4.3.3/deepeval/_version.py

@@ -0,0 +1 @@
+__version__: str = "4.3.3"

{deepeval-4.3.2 → deepeval-4.3.3}/deepeval/dataset/dataset.py

@@ -476,9 +476,11 @@ class EvaluationDataset:
             retrieval_context = reconstruct_retrieval_context(
                 json_obj.get(retrieval_context_key_name)
             )
-            tools_called_data = json_obj.get(tools_called_key_name, [])
+            # A key that is present but null (as save_as writes for goldens
+            # without tools) means no tools, same as a missing key.
+            tools_called_data = json_obj.get(tools_called_key_name) or []
             tools_called = [ToolCall(**tool) for tool in tools_called_data]
-            expected_tools_data = json_obj.get(expected_tools_key_name, [])
+            expected_tools_data = json_obj.get(expected_tools_key_name) or []
             expected_tools = [ToolCall(**tool) for tool in expected_tools_data]
             # additional_metadata = json_obj.get(addtional_metadata_key_name)
 

{deepeval-4.3.2 → deepeval-4.3.3}/deepeval/dataset/golden.py

@@ -384,6 +384,9 @@ class ConversationalGolden(BaseModel):
             if re.search(pattern, self.user_description) is not None:
                 self.multimodal = True
                 return self
+        if self.context and any(re.search(pattern, c) for c in self.context):
+            self.multimodal = True
+            return self
         if self.turns:
             for turn in self.turns:
                 if re.search(pattern, turn.content) is not None:

{deepeval-4.3.2 → deepeval-4.3.3}/deepeval/dataset/utils.py

@@ -32,6 +32,7 @@ def strip_local_expectation_fields(golden_bodies: List[Dict[str, Any]]):
         for field in _LOCAL_ONLY_EXPECTATION_FIELDS:
             expectations.pop(field, None)
 
+
 # RetrievedContextData declares an @model_serializer, so a plain model_dump
 # flattens it and a save/load round-trip loses the source. Serialize each item
 # to a namespaced, parseable marker instead, and reconstruct it on load.

{deepeval-4.3.2 → deepeval-4.3.3}/deepeval/evaluate/evaluate.py

@@ -22,6 +22,7 @@ from deepeval.evaluate.utils import (
     validate_evaluate_inputs,
 )
 from deepeval.evaluate.console_report import EvaluationConsoleReport
+from deepeval.evaluate.local_store import save_results_locally
 from deepeval.dataset import Golden
 from deepeval.prompt import Prompt
 from deepeval.test_case.utils import (
@@ -346,12 +347,18 @@ def evaluate(
 
         if not metrics and not has_expectations:
             # Classifier-only run: classifications are local-only for now, so
-            # there is no test run to finalize or upload.
-            return EvaluationResult(
+            # there is no test run to finalize or upload; only save locally.
+            result = EvaluationResult(
                 test_results=test_results,
                 confident_link=None,
                 test_run_id=None,
             )
+            save_results_locally(
+                result,
+                results_folder=display_config.results_folder,
+                results_subfolder=display_config.results_subfolder,
+            )
+            return result
 
         global_test_run_manager.save_test_run(TEMP_FILE_PATH)
 

{deepeval-4.3.2 → deepeval-4.3.3}/deepeval/evaluate/expectations.py

@@ -176,8 +176,7 @@ class _ExpectationEvaluator:
             for condition, verdict in zip(conditions, self.verdicts)
         ]
         return (
-            _REASON_INSTRUCTIONS
-            + f"Score:\n{self.score}\n\n"
+            _REASON_INSTRUCTIONS + f"Score:\n{self.score}\n\n"
             f"Requirements:\n{json.dumps(verdicts, ensure_ascii=False)}\n\n"
             "JSON:\n"
         )

{deepeval-4.3.2 → deepeval-4.3.3}/deepeval/evaluate/local_store.py

@@ -16,6 +16,8 @@ import sys
 from pathlib import Path
 from typing import Optional
 
+from pydantic import BaseModel
+
 from deepeval.test_run.test_run import TestRun, TestRunEncoder
 from deepeval.utils import is_read_only_env
 
@@ -92,10 +94,10 @@ def resolve_target_dir(
     return base
 
 
-def resolve_test_run_path(target_dir: Path) -> Path:
-    """Resolve the exact `test_run_*.json` path inside `target_dir`.
+def resolve_test_run_path(target_dir: Path, prefix: str = "test_run") -> Path:
+    """Resolve the exact `<prefix>_*.json` path inside `target_dir`.
 
-    Base name: `test_run_<YYYYMMDD_HHMMSS>.json` — matches the existing
+    Base name: `<prefix>_<YYYYMMDD_HHMMSS>.json` — matches the existing
     `DEEPEVAL_RESULTS_FOLDER` timestamp format byte-for-byte, just with the
     `.json` extension the original code forgot.
 
@@ -104,13 +106,13 @@ def resolve_test_run_path(target_dir: Path) -> Path:
     `_acquire_lock(target_dir)` when racing writers are possible.
     """
     ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
-    candidate = target_dir / f"test_run_{ts}.json"
+    candidate = target_dir / f"{prefix}_{ts}.json"
     if not candidate.exists():
         return candidate
 
     n = 2
     while True:
-        candidate = target_dir / f"test_run_{ts}_{n}.json"
+        candidate = target_dir / f"{prefix}_{ts}_{n}.json"
         if not candidate.exists():
             return candidate
         n += 1
@@ -142,9 +144,10 @@ def write_rolling_test_run(test_run: TestRun) -> Optional[Path]:
 
 def write_test_run(
     target_dir: Path,
-    test_run: TestRun,
+    test_run: BaseModel,
+    prefix: str = "test_run",
 ) -> Path:
-    """Write `test_run` to `target_dir` as `test_run_<YYYYMMDD_HHMMSS>.json`.
+    """Write `test_run` to `target_dir` as `<prefix>_<YYYYMMDD_HHMMSS>.json`.
 
     Uses `TestRunEncoder` (and `model_dump(by_alias=True, exclude_none=True)`)
     so the serialized form matches the `.deepeval/.temp_test_run_data.json`
@@ -158,16 +161,38 @@ def write_test_run(
     if portalocker is not None:
         lock_path = target_dir / _LOCK_FILENAME
         with portalocker.Lock(str(lock_path), mode="w"):
-            path = resolve_test_run_path(target_dir)
+            path = resolve_test_run_path(target_dir, prefix)
             _dump(test_run, path)
     else:  # pragma: no cover - portalocker is pinned in requirements
-        path = resolve_test_run_path(target_dir)
+        path = resolve_test_run_path(target_dir, prefix)
         _dump(test_run, path)
 
     return path
 
 
-def _dump(test_run: TestRun, path: Path) -> None:
+def save_results_locally(
+    results: BaseModel,
+    results_folder: Optional[str],
+    results_subfolder: Optional[str] = None,
+) -> Optional[Path]:
+    """Write `results` to the results folder, for runs with no `TestRun` to
+    finalize (classifier-only). Failures warn instead of raising."""
+    target_dir = resolve_target_dir(results_folder, results_subfolder)
+    if target_dir is None:
+        return None
+    try:
+        path = write_test_run(target_dir, results, prefix="classifier_run")
+    except Exception as e:
+        print(
+            f"Warning: failed to save classifier results to {target_dir}: {e}",
+            file=sys.stderr,
+        )
+        return None
+    print(f"Classifier results saved at {path}")
+    return path
+
+
+def _dump(test_run: BaseModel, path: Path) -> None:
     try:
         body = test_run.model_dump(by_alias=True, exclude_none=True)
     except AttributeError:

{deepeval-4.3.2 → deepeval-4.3.3}/deepeval/metrics/conversational_g_eval/conversational_g_eval.py

@@ -1,13 +1,12 @@
 """A slightly modified tailored version of the LLM evaluated metric based on the GEval framework: https://arxiv.org/pdf/2303.16634.pdf"""
 
-from openai.types.chat.chat_completion import ChatCompletion
-from typing import Optional, List, Tuple, Union, Dict, Type
+from typing import Optional, List, Tuple, Union, Type
 from rich.console import Console
-import math
 from deepeval.metrics import BaseConversationalMetric
 from deepeval.metrics.g_eval.utils import (
     MetricPullResponse,
     Rubric,
+    calculate_weighted_summed_score,
     construct_conversational_g_eval_turn_params_string,
     construct_non_turns_test_case_string,
     format_rubrics,
@@ -327,7 +326,7 @@ class ConversationalGEval(BaseConversationalMetric):
             schema_cls=cgschema.ReasonScore,
             strict_mode=self.strict_mode,
             top_logprobs=self.top_logprobs,
-            weighted_score_fn=self.generate_weighted_summed_score,
+            weighted_score_fn=calculate_weighted_summed_score,
             audio=audio,
             model=self._scoring_model,
         )
@@ -342,53 +341,11 @@ class ConversationalGEval(BaseConversationalMetric):
             schema_cls=cgschema.ReasonScore,
             strict_mode=self.strict_mode,
             top_logprobs=self.top_logprobs,
-            weighted_score_fn=self.generate_weighted_summed_score,
+            weighted_score_fn=calculate_weighted_summed_score,
             audio=audio,
             model=self._scoring_model,
         )
 
-    def generate_weighted_summed_score(
-        self, raw_score: int, raw_response: ChatCompletion
-    ) -> Union[int, float]:
-        generated_logprobs = raw_response.choices[0].logprobs.content
-        # First, locate the token that we care for logprobs, i.e., the token matching the score
-        score_logprobs = None
-        for token_logprobs in generated_logprobs:
-            if token_logprobs.token == str(raw_score):
-                score_logprobs = token_logprobs
-                break
-        # Then, calculate the score based on the logprobs
-        token_linear_probability: Dict[int, float] = {}
-        sum_linear_probability = 0
-        # Filter out tokens with <1% linear probability, i.e., logprobs < math.log(0.01)
-        min_logprob = math.log(0.01)
-        for token_logprob in score_logprobs.top_logprobs:
-            logprob = token_logprob.logprob
-
-            # Filter out low probability tokens
-            if logprob < min_logprob:
-                continue
-            # Filter out non-decimal token to prevent errors in later int(token) conversion
-            if not token_logprob.token.isdecimal():
-                continue
-
-            # Calculate the linear probability
-            linear_prob = math.exp(logprob)
-            token_score = int(token_logprob.token)
-            if token_linear_probability.get(token_score):
-                token_linear_probability[token_score] += linear_prob
-            else:
-                token_linear_probability[token_score] = linear_prob
-            sum_linear_probability += linear_prob
-
-        sum_of_weighted_scores = 0.0
-        for score, prob in token_linear_probability.items():
-            sum_of_weighted_scores += score * prob
-
-        # Scale the sum of linear probability to 1
-        weighted_summed_score = sum_of_weighted_scores / sum_linear_probability
-        return weighted_summed_score
-
     def number_evaluation_steps(self):
         evaluation_steps = """"""
         for index, string in enumerate(self.evaluation_steps, start=1):

{deepeval-4.3.2 → deepeval-4.3.3}/deepeval/metrics/g_eval/g_eval.py

@@ -1,6 +1,7 @@
 """LLM evaluated metric based on the GEval framework: https://arxiv.org/pdf/2303.16634.pdf"""
 
 import asyncio
+from functools import partial
 from rich.console import Console
 from typing import Dict, Optional, List, Tuple, Union, Type
 from deepeval.errors import MissingTestCaseParamsError
@@ -355,7 +356,10 @@ class GEval(BaseMetric):
             schema_cls=gschema.ReasonScore,
             strict_mode=self.strict_mode,
             top_logprobs=self.top_logprobs,
-            weighted_score_fn=calculate_weighted_summed_score,
+            weighted_score_fn=partial(
+                calculate_weighted_summed_score,
+                score_max=self.score_range[1],
+            ),
         )
 
     def _evaluate(
@@ -373,7 +377,10 @@ class GEval(BaseMetric):
             schema_cls=gschema.ReasonScore,
             strict_mode=self.strict_mode,
             top_logprobs=self.top_logprobs,
-            weighted_score_fn=calculate_weighted_summed_score,
+            weighted_score_fn=partial(
+                calculate_weighted_summed_score,
+                score_max=self.score_range[1],
+            ),
         )
 
     def upload(self):

{deepeval-4.3.2 → deepeval-4.3.3}/deepeval/metrics/g_eval/utils.py

@@ -337,55 +337,74 @@ def construct_test_case_string(
     return text
 
 
+# Candidates under 1% probability are ignored.
+_MIN_LOGPROB = math.log(0.01)
+
+
+def _find_score_span(tokens, raw_score: int) -> Optional[Tuple[int, int]]:
+    """Start and length of the last run of tokens spelling ``raw_score``. The
+    reason may repeat its digits, and some tokenizers split "10" into "1", "0".
+    """
+    score_str = str(raw_score)
+    for start in reversed(range(len(tokens))):
+        for length in range(len(score_str), 0, -1):
+            span = tokens[start : start + length]
+            spelled = "".join(t.token for t in span)
+            if len(span) == length and spelled == score_str:
+                return start, length
+    return None
+
+
+def _is_prefix_of_larger_score(token: str, score_max: int) -> bool:
+    """Whether ``token`` could be the first digit of a larger score, like "1"
+    of 10."""
+    return any(
+        str(score).startswith(token)
+        for score in range(int(token) + 1, score_max + 1)
+    )
+
+
 def calculate_weighted_summed_score(
-    raw_score: int, raw_response: ChatCompletion
+    raw_score: int,
+    raw_response: ChatCompletion,
+    score_max: int = 10,
 ) -> Union[int, float]:
-    try:
-        generated_logprobs = raw_response.choices[0].logprobs.content
-        # First, locate the final token matching the score. The reasoning may
-        # contain the same token before the model emits its final score.
-        score_logprobs = None
-        for token_logprobs in reversed(generated_logprobs):
-            if token_logprobs.token == str(raw_score):
-                score_logprobs = token_logprobs
-                break
-        # Then, calculate the score based on the logprobs
-        token_linear_probability: Dict[int, float] = {}
-        sum_linear_probability = 0
-        # Filter out tokens with <1% linear probability, i.e., logprobs < math.log(0.01)
-        min_logprob = math.log(0.01)
-        for token_logprob in score_logprobs.top_logprobs:
-            logprob = token_logprob.logprob
-
-            # Filter out low probability tokens
-            if logprob < min_logprob:
-                continue
-            # Filter out non-decimal token to prevent errors in later int(token) conversion
-            if not token_logprob.token.isdecimal():
-                continue
-
-            # Calculate the linear probability
-            linear_prob = math.exp(logprob)
-            token_score = int(token_logprob.token)
-            if token_linear_probability.get(token_score):
-                token_linear_probability[token_score] += linear_prob
-            else:
-                token_linear_probability[token_score] = linear_prob
-            sum_linear_probability += linear_prob
-
-        sum_of_weighted_scores = 0.0
-        for score, prob in token_linear_probability.items():
-            sum_of_weighted_scores += score * prob
-
-        # If all tokens were filtered out, fall back to the raw score
-        if sum_linear_probability == 0:
-            return raw_score
-
-        # Scale the sum of linear probability to 1
-        weighted_summed_score = sum_of_weighted_scores / sum_linear_probability
-        return weighted_summed_score
-    except Exception:
-        raise
+    """Average the scores the judge considered, weighted by their probability.
+    Falls back to ``raw_score`` when there is nothing to weight."""
+    tokens = raw_response.choices[0].logprobs.content
+    span = _find_score_span(tokens, raw_score)
+    if span is None:
+        return raw_score
+    start, length = span
+    score_token = tokens[start]
+
+    weighted_sum = 0.0
+    total_prob = 0.0
+    for candidate in score_token.top_logprobs:
+        if candidate.logprob < _MIN_LOGPROB or not candidate.token.isdecimal():
+            continue
+        if int(candidate.token) > score_max:
+            continue
+
+        if candidate.token == score_token.token:
+            # The emitted score, including the rest of its span.
+            score = raw_score
+            logprob = candidate.logprob + sum(
+                t.logprob for t in tokens[start + 1 : start + length]
+            )
+        elif _is_prefix_of_larger_score(candidate.token, score_max):
+            continue
+        else:
+            score = int(candidate.token)
+            logprob = candidate.logprob
+
+        prob = math.exp(logprob)
+        weighted_sum += score * prob
+        total_prob += prob
+
+    if total_prob == 0:
+        return raw_score
+    return weighted_sum / total_prob
 
 
 def number_evaluation_steps(evaluation_steps: List[str]) -> str: