clear-eval 2.0.1__tar.gz → 2.0.3__tar.gz

clear_eval-2.0.3/PKG-INFO

@@ -0,0 +1,248 @@
+Metadata-Version: 2.4
+Name: clear_eval
+Version: 2.0.3
+Summary: A python API sdk facilitating Error Analysis via LLM-as-a-Judge
+Author-email: Lilach Eden <lilache@il.ibm.com>, Asaf Yehudai <asaf.yehudai@ibm.com>
+License: Apache-2.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: python-dotenv
+Requires-Dist: langchain
+Requires-Dist: langgraph
+Requires-Dist: langchain_openai
+Requires-Dist: langchain_ibm
+Requires-Dist: ibm_watsonx_ai>=1.3.42
+Requires-Dist: openai
+Requires-Dist: pyyaml
+Requires-Dist: pandas
+Requires-Dist: tqdm
+Requires-Dist: pyyaml
+Requires-Dist: streamlit
+Requires-Dist: matplotlib
+Requires-Dist: seaborn
+Requires-Dist: langchain_community
+Requires-Dist: numpy
+Requires-Dist: pyarrow>=22.0.0
+Requires-Dist: nest-asyncio
+Requires-Dist: nicegui>=3.7.1
+Requires-Dist: plotly
+Requires-Dist: networkx
+Requires-Dist: litellm>=1.83.0
+Requires-Dist: nest-asyncio
+Requires-Dist: scipy
+Requires-Dist: scikit-learn
+Requires-Dist: botocore
+Provides-Extra: tool-calls
+Requires-Dist: agent-lifecycle-toolkit; extra == "tool-calls"
+Dynamic: license-file
+
+# CLEAR: Comprehensive LLM Error Analysis and Reporting
+
+**CLEAR** is an open-source toolkit for **LLM error analysis** using an LLM-as-a-Judge approach.
+
+[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/)
+[![License](https://img.shields.io/badge/license-Apache%202.0-green.svg)](LICENSE)
+[![PyPI](https://img.shields.io/pypi/v/clear-eval.svg)](https://pypi.org/project/clear-eval/)
+
+---
+
+## What is CLEAR?
+
+CLEAR provides systematic error analysis for LLM-based systems. It combines automated LLM-as-a-judge evaluation with interactive dashboards to help you:
+- Identify recurring error patterns across your dataset
+- Quantify issue frequencies and severity
+- Drill down into specific failure cases
+- Prioritize improvements based on data-driven insights
+
+CLEAR operates in two phases:
+
+1. **Analysis** — Generates per-instance textual feedback, identifies system-level error categories, and quantifies their frequencies.
+2. **Interactive Dashboard** — Explore aggregate visualizations, apply dynamic filters, and drill down into individual failure examples.
+
+---
+
+## Two Analysis Modes
+
+### LLM Analysis
+
+Evaluate standard LLM outputs — generation quality, correctness, and recurring error patterns. Provide a CSV with prompts and responses, and CLEAR will score each instance, generate textual critiques, and surface system-level issues.
+
+- **Input:** CSV with model inputs and responses
+- **Output:** Per-record scores, evaluation text, aggregated issue categories
+- **Dashboard:** Streamlit-based interactive explorer
+
+> **[LLM Analysis Guide →](docs/llm-analysis.md)**
+
+### Agentic Analysis
+
+Evaluate multi-agent system trajectories — step-by-step agent interactions and full trajectory analysis.
+Supports traces from LangGraph, CrewAI, and other frameworks via MLflow or Langfuse.
+
+- **Input:** Raw JSON traces or preprocessed trajectory CSVs (each trace captures one complete agent task execution)
+- **Output:** Per-step CLEAR analysis, trajectory-level scores, rubric evaluations
+- **Dashboard:** NiceGUI-based workflow visualization with path and temporal analysis
+
+> **[Agentic Workflows Guide →](src/clear_eval/agentic/README.md)**
+
+---
+
+## Installation
+
+**Requires Python 3.10+**
+
+#### Option 1: pip
+
+```bash
+pip install clear-eval
+```
+
+#### Option 2: From source (for development)
+
+```bash
+git clone https://github.com/IBM/CLEAR.git
+cd CLEAR
+python3 -m venv .venv
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+pip install -e .
+```
+
+Verify the installation:
+
+```bash
+run-clear-eval-analysis --help
+```
+
+---
+
+## Quick Start
+
+### 1. Set your provider credentials
+
+CLEAR requires a supported LLM provider. Set the appropriate environment variables for your provider (e.g., `OPENAI_API_KEY` for OpenAI). Adjust `--provider` and `--eval-model-name` in the commands below to match your setup. See [Provider Configuration](#provider-configuration) for details.
+
+### 2. LLM Analysis
+
+This evaluates GSM8K math problem responses and surfaces recurring quality issues:
+
+```bash
+run-clear-eval-analysis --provider openai --eval-model-name gpt-4o
+```
+
+Results are saved to `results/gsm8k/sample_output/`. View them:
+
+```bash
+run-clear-eval-dashboard
+```
+
+> **[Full LLM Analysis Guide →](docs/llm-analysis.md)** — input formats, CLI arguments, configuration, Python API, and external judges.
+
+### 3. Agentic Analysis
+
+These two modes are independent — this section does not require step 2.
+
+Run CLEAR on sample agent traces (3 traces, each capturing one complete agent task execution, ~2 minutes):
+
+```bash
+run-clear-agentic-eval \
+    --data-dir src/clear_eval/sample_data/agentic/research_agent_traces/mlflow \
+    --results-dir my_smoke_test_results \
+    --from-raw-traces true \
+    --agent-framework langgraph \
+    --observability-framework mlflow \
+    --run-name smoke_test \
+    --max-files 3 \
+    --eval-model-name gpt-4o \
+    --provider openai
+```
+
+View pre-computed results (all 20 traces) without re-running:
+
+```bash
+run-clear-agentic-dashboard
+# Upload: src/clear_eval/sample_data/agentic/research_agent_results/mlflow/my_experiment/unified_ui_results.zip
+```
+
+> **[Full Agentic Guide →](src/clear_eval/agentic/README.md)** — trace generation, configuration, output structure, and dashboard features.
+
+---
+
+## Provider Configuration
+
+CLEAR uses [LiteLLM](https://docs.litellm.ai/docs/providers) as its inference backend, supporting 100+ LLM providers (OpenAI, Anthropic, WatsonX, AWS Bedrock, Google Vertex AI, and more).
+
+**Parameters:**
+
+| Parameter | CLI Flag | Description |
+|-----------|----------|-------------|
+| `provider` | `--provider` | LiteLLM provider name (e.g., `openai`, `anthropic`, `bedrock`, `vertex_ai`) |
+| `eval_model_name` | `--eval-model-name` | Model identifier (e.g., `gpt-4o`, `claude-3-5-sonnet-20241022`) |
+| `eval_model_params` | `--eval-model-params` | Additional model parameters as JSON (e.g., `{"temperature": 0}`) |
+| `endpoint_url` | `--endpoint-url` | Custom endpoint URL for local/self-hosted models |
+
+**Cloud provider example:**
+
+```bash
+export OPENAI_API_KEY="..."
+run-clear-eval-analysis --provider openai --eval-model-name gpt-4o
+```
+
+**Local model example (vLLM, llama.cpp, Ollama, etc.):**
+
+```bash
+run-clear-eval-analysis \
+    --provider openai \
+    --eval-model-name my-local-model \
+    --endpoint-url http://localhost:8000/v1
+```
+
+No credentials are needed when using `--endpoint-url` with a local server.
+
+Set the required environment variables for your provider according to [LiteLLM's documentation](https://docs.litellm.ai/docs/providers).
+
+---
+
+## Documentation
+
+| Guide | Description |
+|-------|-------------|
+| [**Agentic Workflows Guide**](src/clear_eval/agentic/README.md) | Multi-agent evaluation — trace preprocessing, step-by-step and trajectory analysis, configuration reference |
+| [**Agentic Dashboard Guide**](docs/agentic/dashboard.md) | Dashboard features — workflow view, node analysis, trajectory explorer, path and temporal analysis |
+| [**LLM Analysis Guide**](docs/llm-analysis.md) | Full pipeline reference — input formats, CLI arguments, configuration, and external judges |
+
+---
+
+## Citation
+
+If you use CLEAR in your research, please cite the relevant paper(s):
+
+**LLM Analysis** ([AAAI 2026](https://ojs.aaai.org/index.php/AAAI/article/view/42398)):
+```bibtex
+@inproceedings{yehudai2026clear,
+  title={CLEAR: Error analysis via llm-as-a-judge made easy},
+  author={Yehudai, Asaf and Eden, Lilach and Perlitz, Yotam and Bar-Haim, Roy and Shmueli-Scheuer, Michal},
+  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
+  volume={40},
+  number={48},
+  pages={41736--41738},
+  year={2026}
+}
+```
+
+**Agentic Analysis** (ACL 2026, to appear — [preprint](https://arxiv.org/abs/2605.22608)):
+```bibtex
+@article{yehudai2026agentic,
+  title={Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents},
+  author={Yehudai, Asaf and Eden, Lilach and Shmueli-Scheuer, Michal},
+  journal={arXiv preprint arXiv:2605.22608},
+  year={2026}
+}
+```
+
+---
+
+## License
+
+Apache 2.0 — see [LICENSE](LICENSE) for details.

clear_eval-2.0.3/README.md

@@ -0,0 +1,207 @@
+# CLEAR: Comprehensive LLM Error Analysis and Reporting
+
+**CLEAR** is an open-source toolkit for **LLM error analysis** using an LLM-as-a-Judge approach.
+
+[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/)
+[![License](https://img.shields.io/badge/license-Apache%202.0-green.svg)](LICENSE)
+[![PyPI](https://img.shields.io/pypi/v/clear-eval.svg)](https://pypi.org/project/clear-eval/)
+
+---
+
+## What is CLEAR?
+
+CLEAR provides systematic error analysis for LLM-based systems. It combines automated LLM-as-a-judge evaluation with interactive dashboards to help you:
+- Identify recurring error patterns across your dataset
+- Quantify issue frequencies and severity
+- Drill down into specific failure cases
+- Prioritize improvements based on data-driven insights
+
+CLEAR operates in two phases:
+
+1. **Analysis** — Generates per-instance textual feedback, identifies system-level error categories, and quantifies their frequencies.
+2. **Interactive Dashboard** — Explore aggregate visualizations, apply dynamic filters, and drill down into individual failure examples.
+
+---
+
+## Two Analysis Modes
+
+### LLM Analysis
+
+Evaluate standard LLM outputs — generation quality, correctness, and recurring error patterns. Provide a CSV with prompts and responses, and CLEAR will score each instance, generate textual critiques, and surface system-level issues.
+
+- **Input:** CSV with model inputs and responses
+- **Output:** Per-record scores, evaluation text, aggregated issue categories
+- **Dashboard:** Streamlit-based interactive explorer
+
+> **[LLM Analysis Guide →](docs/llm-analysis.md)**
+
+### Agentic Analysis
+
+Evaluate multi-agent system trajectories — step-by-step agent interactions and full trajectory analysis.
+Supports traces from LangGraph, CrewAI, and other frameworks via MLflow or Langfuse.
+
+- **Input:** Raw JSON traces or preprocessed trajectory CSVs (each trace captures one complete agent task execution)
+- **Output:** Per-step CLEAR analysis, trajectory-level scores, rubric evaluations
+- **Dashboard:** NiceGUI-based workflow visualization with path and temporal analysis
+
+> **[Agentic Workflows Guide →](src/clear_eval/agentic/README.md)**
+
+---
+
+## Installation
+
+**Requires Python 3.10+**
+
+#### Option 1: pip
+
+```bash
+pip install clear-eval
+```
+
+#### Option 2: From source (for development)
+
+```bash
+git clone https://github.com/IBM/CLEAR.git
+cd CLEAR
+python3 -m venv .venv
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+pip install -e .
+```
+
+Verify the installation:
+
+```bash
+run-clear-eval-analysis --help
+```
+
+---
+
+## Quick Start
+
+### 1. Set your provider credentials
+
+CLEAR requires a supported LLM provider. Set the appropriate environment variables for your provider (e.g., `OPENAI_API_KEY` for OpenAI). Adjust `--provider` and `--eval-model-name` in the commands below to match your setup. See [Provider Configuration](#provider-configuration) for details.
+
+### 2. LLM Analysis
+
+This evaluates GSM8K math problem responses and surfaces recurring quality issues:
+
+```bash
+run-clear-eval-analysis --provider openai --eval-model-name gpt-4o
+```
+
+Results are saved to `results/gsm8k/sample_output/`. View them:
+
+```bash
+run-clear-eval-dashboard
+```
+
+> **[Full LLM Analysis Guide →](docs/llm-analysis.md)** — input formats, CLI arguments, configuration, Python API, and external judges.
+
+### 3. Agentic Analysis
+
+These two modes are independent — this section does not require step 2.
+
+Run CLEAR on sample agent traces (3 traces, each capturing one complete agent task execution, ~2 minutes):
+
+```bash
+run-clear-agentic-eval \
+    --data-dir src/clear_eval/sample_data/agentic/research_agent_traces/mlflow \
+    --results-dir my_smoke_test_results \
+    --from-raw-traces true \
+    --agent-framework langgraph \
+    --observability-framework mlflow \
+    --run-name smoke_test \
+    --max-files 3 \
+    --eval-model-name gpt-4o \
+    --provider openai
+```
+
+View pre-computed results (all 20 traces) without re-running:
+
+```bash
+run-clear-agentic-dashboard
+# Upload: src/clear_eval/sample_data/agentic/research_agent_results/mlflow/my_experiment/unified_ui_results.zip
+```
+
+> **[Full Agentic Guide →](src/clear_eval/agentic/README.md)** — trace generation, configuration, output structure, and dashboard features.
+
+---
+
+## Provider Configuration
+
+CLEAR uses [LiteLLM](https://docs.litellm.ai/docs/providers) as its inference backend, supporting 100+ LLM providers (OpenAI, Anthropic, WatsonX, AWS Bedrock, Google Vertex AI, and more).
+
+**Parameters:**
+
+| Parameter | CLI Flag | Description |
+|-----------|----------|-------------|
+| `provider` | `--provider` | LiteLLM provider name (e.g., `openai`, `anthropic`, `bedrock`, `vertex_ai`) |
+| `eval_model_name` | `--eval-model-name` | Model identifier (e.g., `gpt-4o`, `claude-3-5-sonnet-20241022`) |
+| `eval_model_params` | `--eval-model-params` | Additional model parameters as JSON (e.g., `{"temperature": 0}`) |
+| `endpoint_url` | `--endpoint-url` | Custom endpoint URL for local/self-hosted models |
+
+**Cloud provider example:**
+
+```bash
+export OPENAI_API_KEY="..."
+run-clear-eval-analysis --provider openai --eval-model-name gpt-4o
+```
+
+**Local model example (vLLM, llama.cpp, Ollama, etc.):**
+
+```bash
+run-clear-eval-analysis \
+    --provider openai \
+    --eval-model-name my-local-model \
+    --endpoint-url http://localhost:8000/v1
+```
+
+No credentials are needed when using `--endpoint-url` with a local server.
+
+Set the required environment variables for your provider according to [LiteLLM's documentation](https://docs.litellm.ai/docs/providers).
+
+---
+
+## Documentation
+
+| Guide | Description |
+|-------|-------------|
+| [**Agentic Workflows Guide**](src/clear_eval/agentic/README.md) | Multi-agent evaluation — trace preprocessing, step-by-step and trajectory analysis, configuration reference |
+| [**Agentic Dashboard Guide**](docs/agentic/dashboard.md) | Dashboard features — workflow view, node analysis, trajectory explorer, path and temporal analysis |
+| [**LLM Analysis Guide**](docs/llm-analysis.md) | Full pipeline reference — input formats, CLI arguments, configuration, and external judges |
+
+---
+
+## Citation
+
+If you use CLEAR in your research, please cite the relevant paper(s):
+
+**LLM Analysis** ([AAAI 2026](https://ojs.aaai.org/index.php/AAAI/article/view/42398)):
+```bibtex
+@inproceedings{yehudai2026clear,
+  title={CLEAR: Error analysis via llm-as-a-judge made easy},
+  author={Yehudai, Asaf and Eden, Lilach and Perlitz, Yotam and Bar-Haim, Roy and Shmueli-Scheuer, Michal},
+  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
+  volume={40},
+  number={48},
+  pages={41736--41738},
+  year={2026}
+}
+```
+
+**Agentic Analysis** (ACL 2026, to appear — [preprint](https://arxiv.org/abs/2605.22608)):
+```bibtex
+@article{yehudai2026agentic,
+  title={Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents},
+  author={Yehudai, Asaf and Eden, Lilach and Shmueli-Scheuer, Michal},
+  journal={arXiv preprint arXiv:2605.22608},
+  year={2026}
+}
+```
+
+---
+
+## License
+
+Apache 2.0 — see [LICENSE](LICENSE) for details.

{clear_eval-2.0.1 → clear_eval-2.0.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "clear_eval"
-version = "2.0.1"
+version = "2.0.3"
 description = "A python API sdk facilitating Error Analysis via LLM-as-a-Judge"
 readme = "README.md"
 authors = [{ name = "Lilach Eden", email = "lilache@il.ibm.com"},

{clear_eval-2.0.1 → clear_eval-2.0.3}/src/clear_eval/agentic/dashboard/agentic_workflow_dashboard.py

@@ -2174,7 +2174,8 @@ def main_page():
                 # Try to get dimensions from parsed_evaluation first (preferred structure)
                 step_quality = parsed_eval.get("step_quality_dimensions", {})
                 trajectory_dims = parsed_eval.get("trajectory_dimensions", {})
-                
+                custom_dims = parsed_eval.get("dimensions", {})
+
                 # Collect step quality dimension scores
                 for criterion, details in step_quality.items():
                     score = details.get("score")
@@ -2183,7 +2184,7 @@ def main_page():
                             all_dimension_scores[f"Step Quality: {criterion}"].append(float(score))
                         except (ValueError, TypeError):
                             print(f"⚠️  Warning: Could not convert score to float for {criterion}: {score}")
-                
+
                 # Collect trace dimension scores
                 for criterion, details in trajectory_dims.items():
                     score = details.get("score")
@@ -2192,9 +2193,19 @@ def main_page():
                             all_dimension_scores[f"Trace: {criterion}"].append(float(score))
                         except (ValueError, TypeError):
                             print(f"⚠️  Warning: Could not convert score to float for {criterion}: {score}")
-                
+
+                # Collect custom flat dimension scores
+                if custom_dims and not step_quality and not trajectory_dims:
+                    for criterion, details in custom_dims.items():
+                        score = details.get("score")
+                        if score is not None:
+                            try:
+                                all_dimension_scores[criterion].append(float(score))
+                            except (ValueError, TypeError):
+                                print(f"⚠️  Warning: Could not convert score to float for {criterion}: {score}")
+
                 # Fallback: if no dimensions found in parsed_evaluation, try dimension_scores directly
-                if not step_quality and not trajectory_dims and dimension_scores:
+                if not step_quality and not trajectory_dims and not custom_dims and dimension_scores:
                     for criterion, score in dimension_scores.items():
                         if score is not None:
                             try:
@@ -3083,7 +3094,8 @@ def main_page():
                 # Separate step quality and trace dimensions
                 step_quality = parsed_eval.get("step_quality_dimensions", {})
                 trajectory_dims = parsed_eval.get("trajectory_dimensions", {})
-                
+                custom_dims = parsed_eval.get("dimensions", {})
+
                 if step_quality:
                     ui.html('<div style="font-weight: 600; color: #1E293B; margin: 12px 0 8px 0; font-size: 15px;">Step Quality Dimensions</div>')
                     for criterion, details in step_quality.items():
@@ -3091,7 +3103,7 @@ def main_page():
                         justification = details.get("justification", "No justification provided")
                         score_color = score_to_hex(score)
                         definition = get_dimension_definition(criterion)
-                        
+
                         with ui.row().classes("w-full items-center gap-2 mb-2"):
                             with ui.expansion(f"{criterion}: {score:.2f}", icon="check_circle").classes("flex-1").style(f"border-left: 4px solid {score_color};"):
                                 ui.html(f"""
@@ -3101,7 +3113,7 @@ def main_page():
                                 """)
                             with ui.button(icon="info", on_click=lambda: None).props("flat dense round size=sm").classes("text-blue-500"):
                                 ui.tooltip(definition).classes("bg-slate-800 text-white text-sm max-w-md")
-                
+
                 if trajectory_dims:
                     ui.html('<div style="font-weight: 600; color: #1E293B; margin: 20px 0 8px 0; font-size: 15px;">Trace-Level Dimensions</div>')
                     for criterion, details in trajectory_dims.items():
@@ -3109,7 +3121,7 @@ def main_page():
                         justification = details.get("justification", "No justification provided")
                         score_color = score_to_hex(score)
                         definition = get_dimension_definition(criterion)
-                        
+
                         with ui.row().classes("w-full items-center gap-2 mb-2"):
                             with ui.expansion(f"{criterion}: {score:.2f}", icon="check_circle").classes("flex-1").style(f"border-left: 4px solid {score_color};"):
                                 ui.html(f"""
@@ -3120,6 +3132,21 @@ def main_page():
                             with ui.button(icon="info", on_click=lambda: None).props("flat dense round size=sm").classes("text-blue-500"):
                                 ui.tooltip(definition).classes("bg-slate-800 text-white text-sm max-w-md")
 
+                if custom_dims and not step_quality and not trajectory_dims:
+                    ui.html('<div style="font-weight: 600; color: #1E293B; margin: 12px 0 8px 0; font-size: 15px;">Evaluation Dimensions</div>')
+                    for criterion, details in custom_dims.items():
+                        score = float(details.get("score", 0))
+                        justification = details.get("justification", "No justification provided")
+                        score_color = score_to_hex(score)
+
+                        with ui.row().classes("w-full items-center gap-2 mb-2"):
+                            with ui.expansion(f"{criterion}: {score:.2f}", icon="check_circle").classes("flex-1").style(f"border-left: 4px solid {score_color};"):
+                                ui.html(f"""
+                                    <div style="padding: 8px; background: #FAFAFA; border-radius: 6px; color: #475569; font-size: 13px; line-height: 1.5;">
+                                        {justification}
+                                    </div>
+                                """)
+
     def render_rubric_evaluation(rubric_eval):
         """Render the rubric-based trace evaluation."""
         # Overall score card

{clear_eval-2.0.1 → clear_eval-2.0.3}/src/clear_eval/agentic/dashboard/launch_dashboard.py

@@ -9,6 +9,18 @@ import sys
 from pathlib import Path
 
 
+def _is_port_available(host: str, port: int) -> bool:
+    """Check if a port is available for binding."""
+    import socket
+    try:
+        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+            s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+            s.bind((host, port))
+            return True
+    except OSError:
+        return False
+
+
 def main():
     parser = argparse.ArgumentParser(description="Launch the Agentic Workflow Dashboard")
     parser.add_argument("--port", type=int, default=8080, help="Port to run the dashboard on (default: 8080)")
@@ -16,6 +28,12 @@ def main():
     parser.add_argument("--no-open", action="store_true", help="Don't auto-open browser")
     args = parser.parse_args()
 
+    if not _is_port_available(args.host, args.port):
+        print(f"Error: Port {args.port} is already in use.")
+        print(f"Either stop the process using port {args.port}, or use a different port:")
+        print(f"  run-clear-agentic-dashboard --port {args.port + 1}")
+        sys.exit(1)
+
     # Ensure the server shuts down cleanly on termination signals,
     # so the port is released immediately instead of staying in TIME_WAIT.
     def _handle_signal(signum, frame):

{clear_eval-2.0.1 → clear_eval-2.0.3}/src/clear_eval/agentic/pipeline/argument_definitions.py

@@ -10,7 +10,7 @@ reusable argument groups that can be composed based on each script's needs.
 """
 
 import argparse
-from clear_eval.args import str2bool
+from clear_eval.args import str2bool, parse_dict
 
 
 def add_agentic_pipeline_args(parser: argparse.ArgumentParser) -> None:
@@ -132,6 +132,14 @@ def add_full_trajectory_args(parser: argparse.ArgumentParser) -> None:
         type=int,
         help="Model context window size (for full trajectory)"
     )
+    group.add_argument(
+        "--full-trace-evaluation-criteria",
+        type=parse_dict,
+        default=None,
+        help="JSON dict of evaluation criteria for full trace evaluation. "
+             "Replaces all default dimensions when set. "
+             "Example: '{\"Correctness\": \"Accurate results\", \"Efficiency\": \"No redundant steps\"}'"
+    )
     # Note: --max-files is defined in add_agentic_pipeline_args() (shared across pipelines)
     # Do not add it here to avoid conflicts
 

{clear_eval-2.0.1 → clear_eval-2.0.3}/src/clear_eval/agentic/pipeline/full_traces_evaluation/clear_analysis/base_clear_runner.py

@@ -44,6 +44,7 @@ class BaseClearRunner(ABC):
         output_dir: Path,
         inference_config: InferenceConfig,
         overwrite: bool = False,
+        predefined_issues: list = None,
     ):
         """
         Initialize CLEAR runner.
@@ -53,11 +54,13 @@ class BaseClearRunner(ABC):
             output_dir: Base directory for CLEAR analysis outputs
             inference_config: LLM inference configuration
             overwrite: Whether to overwrite existing CLEAR results
+            predefined_issues: Predefined issues list to skip issue discovery
         """
         self.eval_results_dir = Path(eval_results_dir)
         self.output_dir = Path(output_dir)
         self.inference_config = inference_config
         self.overwrite = overwrite
+        self.predefined_issues = predefined_issues
 
     @abstractmethod
     def get_source_name(self) -> str:
@@ -219,6 +222,9 @@ class BaseClearRunner(ABC):
                 "endpoint_url": self.inference_config.endpoint_url,
             }
 
+            if self.predefined_issues:
+                analysis_kwargs["predefined_issues"] = self.predefined_issues
+
             run_clear_eval_aggregation(**analysis_kwargs)
             logger.info(f"Completed: {group_key}")
             return True