traceeval-cli 0.1.0__tar.gz → 0.2.0__tar.gz

{traceeval_cli-0.1.0 → traceeval_cli-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: traceeval-cli
-Version: 0.1.0
+Version: 0.2.0
 Summary: CI/CD and Evaluation Infrastructure for Autonomous Agents (Agentic Engineering)
 Project-URL: Repository, https://github.com/tej007-awesome/TraceEval
 Project-URL: Issues, https://github.com/tej007-awesome/TraceEval/issues
@@ -19,6 +19,8 @@ Requires-Dist: pydantic>=2.0.0
 Requires-Dist: rich>=13.7.0
 Requires-Dist: typer>=0.9.0
 Provides-Extra: dev
+Requires-Dist: opentelemetry-exporter-otlp-proto-common>=1.20.0; extra == 'dev'
+Requires-Dist: opentelemetry-sdk>=1.20.0; extra == 'dev'
 Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
 Requires-Dist: pytest>=8.0.0; extra == 'dev'
 Requires-Dist: ruff==0.11.2; extra == 'dev'
@@ -96,6 +98,11 @@ Perfect for pre-deployment CI/CD gating. Dynamically spawns your agent, captures
 ```bash
 traceeval run --case sample_data/case_01.json --pipeline examples.reference_agent:process_refund_success --export report.json
 ```
+**Mode C: Evaluate an OpenTelemetry (OTel) Trace**
+Audit OpenTelemetry traces adhering to GenAI semantic conventions.
+```bash
+traceeval run --case sample_data/case_01.json --otel-trace tests/fixtures/otel/refund_happy.json
+```
 
 *(Tip: Add `--verbose` right after `traceeval` to view detailed middleware logs!)*
 
@@ -140,13 +147,46 @@ TraceEval decouples the **Ingestion Layer** from the **Evaluation Engine** using
 
 ---
 
+## Evaluating OpenTelemetry Traces
+
+TraceEval natively ingests OpenTelemetry (OTel) JSON traces adhering to the **OTel GenAI Semantic Conventions**.
+
+```bash
+traceeval run --case sample_data/case_01.json --otel-trace tests/fixtures/otel/refund_happy.json
+```
+
+### Attributes Ingested
+TraceEval inspects the following GenAI span attributes:
+- `gen_ai.operation.name`: Identifies span types (`invoke_agent`, `chat`, `execute_tool`).
+- `gen_ai.agent.name` & `gen_ai.conversation.id`: Extracts agent skills and session metadata.
+- `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`: Tracks per-model token consumption.
+- `gen_ai.tool.name` & `gen_ai.tool.call.arguments`: Maps tool trajectories and parameters.
+- `gen_ai.output.messages`: Captures the agent's final text response.
+
+### Cost Computation & Custom Pricing
+Session token costs are automatically computed by matching `gen_ai.request.model` and token counts against model pricing rates. If pricing is missing for an unknown model, the cost check fails (`"cost could not be verified"`) to prevent unmonitored financial risk.
+
+You can supply custom pricing via `--pricing my_pricing.json`:
+```json
+{
+  "my-custom-model": {
+    "input_usd_per_1k": 0.00015,
+    "output_usd_per_1k": 0.0006
+  }
+}
+```
+
+> **Privacy Note:** Standard OTel instrumentation often redacts `gen_ai.tool.call.arguments` and `gen_ai.output.messages` to comply with privacy policies. When arguments are omitted, TraceEval outputs non-blocking warnings, and argument matching defaults to `{}`.
+
+---
+
 ## Roadmap
 
-v0 ships the core EDD Schema, Trajectory Validator, BYOJ Engine, and Live Pipeline Hook. Planned for v1:
+v0.2 adds OpenTelemetry GenAI trace ingestion with computed cost and specific failure reasons. Planned next:
 
-- **OpenTelemetry trace ingestion:** Adapters to ingest native OTel spans from LangGraph, OpenAI Swarm, Claude SDK, and raw MCP servers — so you can point TraceEval at real production traces without converting them by hand.
-- **Live budget guard:** Real-time token-cost interception during agent execution, not just post-run checking.
-- **Offline mock judge mode:** Deterministic stub judge for CI pipelines that cannot call an external LLM (air-gapped environments, cost-sensitive PR checks).
+- **Flexible argument matching and forbidden tools:** match tool arguments exactly, partially, or not at all (for privacy-redacted traces), and fail any trajectory that calls a denied tool.
+- **More reliable LLM judge:** per-rubric-item verdicts, multiple samples, retries, and an offline mock judge for CI pipelines that can't call an external LLM.
+- **Meta-evaluation benchmark:** a labelled set of good and bad traces that measures how accurately TraceEval itself catches failures.
 
 To track granular progress, see our [GitHub Issues](https://github.com/tej007-awesome/TraceEval/issues).
 

{traceeval_cli-0.1.0 → traceeval_cli-0.2.0}/README.md

@@ -70,6 +70,11 @@ Perfect for pre-deployment CI/CD gating. Dynamically spawns your agent, captures
 ```bash
 traceeval run --case sample_data/case_01.json --pipeline examples.reference_agent:process_refund_success --export report.json
 ```
+**Mode C: Evaluate an OpenTelemetry (OTel) Trace**
+Audit OpenTelemetry traces adhering to GenAI semantic conventions.
+```bash
+traceeval run --case sample_data/case_01.json --otel-trace tests/fixtures/otel/refund_happy.json
+```
 
 *(Tip: Add `--verbose` right after `traceeval` to view detailed middleware logs!)*
 
@@ -114,13 +119,46 @@ TraceEval decouples the **Ingestion Layer** from the **Evaluation Engine** using
 
 ---
 
+## Evaluating OpenTelemetry Traces
+
+TraceEval natively ingests OpenTelemetry (OTel) JSON traces adhering to the **OTel GenAI Semantic Conventions**.
+
+```bash
+traceeval run --case sample_data/case_01.json --otel-trace tests/fixtures/otel/refund_happy.json
+```
+
+### Attributes Ingested
+TraceEval inspects the following GenAI span attributes:
+- `gen_ai.operation.name`: Identifies span types (`invoke_agent`, `chat`, `execute_tool`).
+- `gen_ai.agent.name` & `gen_ai.conversation.id`: Extracts agent skills and session metadata.
+- `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`: Tracks per-model token consumption.
+- `gen_ai.tool.name` & `gen_ai.tool.call.arguments`: Maps tool trajectories and parameters.
+- `gen_ai.output.messages`: Captures the agent's final text response.
+
+### Cost Computation & Custom Pricing
+Session token costs are automatically computed by matching `gen_ai.request.model` and token counts against model pricing rates. If pricing is missing for an unknown model, the cost check fails (`"cost could not be verified"`) to prevent unmonitored financial risk.
+
+You can supply custom pricing via `--pricing my_pricing.json`:
+```json
+{
+  "my-custom-model": {
+    "input_usd_per_1k": 0.00015,
+    "output_usd_per_1k": 0.0006
+  }
+}
+```
+
+> **Privacy Note:** Standard OTel instrumentation often redacts `gen_ai.tool.call.arguments` and `gen_ai.output.messages` to comply with privacy policies. When arguments are omitted, TraceEval outputs non-blocking warnings, and argument matching defaults to `{}`.
+
+---
+
 ## Roadmap
 
-v0 ships the core EDD Schema, Trajectory Validator, BYOJ Engine, and Live Pipeline Hook. Planned for v1:
+v0.2 adds OpenTelemetry GenAI trace ingestion with computed cost and specific failure reasons. Planned next:
 
-- **OpenTelemetry trace ingestion:** Adapters to ingest native OTel spans from LangGraph, OpenAI Swarm, Claude SDK, and raw MCP servers — so you can point TraceEval at real production traces without converting them by hand.
-- **Live budget guard:** Real-time token-cost interception during agent execution, not just post-run checking.
-- **Offline mock judge mode:** Deterministic stub judge for CI pipelines that cannot call an external LLM (air-gapped environments, cost-sensitive PR checks).
+- **Flexible argument matching and forbidden tools:** match tool arguments exactly, partially, or not at all (for privacy-redacted traces), and fail any trajectory that calls a denied tool.
+- **More reliable LLM judge:** per-rubric-item verdicts, multiple samples, retries, and an offline mock judge for CI pipelines that can't call an external LLM.
+- **Meta-evaluation benchmark:** a labelled set of good and bad traces that measures how accurately TraceEval itself catches failures.
 
 To track granular progress, see our [GitHub Issues](https://github.com/tej007-awesome/TraceEval/issues).
 

{traceeval_cli-0.1.0 → traceeval_cli-0.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "traceeval-cli"
-version = "0.1.0"
+version = "0.2.0"
 description = "CI/CD and Evaluation Infrastructure for Autonomous Agents (Agentic Engineering)"
 authors = [
     { name = "Tejas Rajesh", email = "tejasrajesh05@gmail.com" }
@@ -38,7 +38,9 @@ Issues = "https://github.com/tej007-awesome/TraceEval/issues"
 dev = [
     "pytest>=8.0.0",
     "pytest-asyncio>=0.23.0",
-    "ruff==0.11.2"
+    "ruff==0.11.2",
+    "opentelemetry-sdk>=1.20.0",
+    "opentelemetry-exporter-otlp-proto-common>=1.20.0",
 ]
 
 [tool.ruff]

traceeval_cli-0.2.0/scripts/generate_otel_fixtures.py

@@ -0,0 +1,119 @@
+import json
+import time
+from pathlib import Path
+
+from google.protobuf.json_format import MessageToDict
+from opentelemetry.exporter.otlp.proto.common._internal.trace_encoder import encode_spans
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import SimpleSpanProcessor
+from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
+
+FIXTURES_DIR = Path("tests/fixtures/otel")
+
+
+def generate_trace_data(include_content: bool = True) -> dict:
+    provider = TracerProvider()
+    exporter = InMemorySpanExporter()
+    provider.add_span_processor(SimpleSpanProcessor(exporter))
+    tracer = provider.get_tracer("traceeval.reference_agent")
+
+    with tracer.start_as_current_span("invoke_agent refund-processor") as root:
+        root.set_attribute("gen_ai.operation.name", "invoke_agent")
+        root.set_attribute("gen_ai.agent.name", "refund-processor")
+        root.set_attribute("gen_ai.conversation.id", "sess_otel_001")
+        time.sleep(0.002)
+
+        # Step 1: LLM initial decision
+        with tracer.start_as_current_span("chat gpt-4o-mini") as chat1:
+            chat1.set_attribute("gen_ai.operation.name", "chat")
+            chat1.set_attribute("gen_ai.request.model", "gpt-4o-mini")
+            chat1.set_attribute("gen_ai.usage.input_tokens", 150)
+            chat1.set_attribute("gen_ai.usage.output_tokens", 30)
+            time.sleep(0.002)
+
+        # Step 2: Tool call 1 (lookup_order)
+        with tracer.start_as_current_span("execute_tool lookup_order") as tool1:
+            tool1.set_attribute("gen_ai.operation.name", "execute_tool")
+            tool1.set_attribute("gen_ai.tool.name", "lookup_order")
+            tool1.set_attribute("gen_ai.tool.call.id", "call_lookup_001")
+            if include_content:
+                tool1.set_attribute(
+                    "gen_ai.tool.call.arguments", json.dumps({"order_id": "4521"})
+                )
+            time.sleep(0.002)
+
+        # Step 3: LLM reasoning after lookup
+        with tracer.start_as_current_span("chat gpt-4o-mini") as chat2:
+            chat2.set_attribute("gen_ai.operation.name", "chat")
+            chat2.set_attribute("gen_ai.request.model", "gpt-4o-mini")
+            chat2.set_attribute("gen_ai.usage.input_tokens", 220)
+            chat2.set_attribute("gen_ai.usage.output_tokens", 35)
+            time.sleep(0.002)
+
+        # Step 4: Tool call 2 (check_duplicate_charge)
+        with tracer.start_as_current_span("execute_tool check_duplicate_charge") as tool2:
+            tool2.set_attribute("gen_ai.operation.name", "execute_tool")
+            tool2.set_attribute("gen_ai.tool.name", "check_duplicate_charge")
+            tool2.set_attribute("gen_ai.tool.call.id", "call_check_dup_002")
+            if include_content:
+                tool2.set_attribute(
+                    "gen_ai.tool.call.arguments", json.dumps({"order_id": "4521"})
+                )
+            time.sleep(0.002)
+
+        # Step 5: LLM reasoning after check
+        with tracer.start_as_current_span("chat gpt-4o-mini") as chat3:
+            chat3.set_attribute("gen_ai.operation.name", "chat")
+            chat3.set_attribute("gen_ai.request.model", "gpt-4o-mini")
+            chat3.set_attribute("gen_ai.usage.input_tokens", 310)
+            chat3.set_attribute("gen_ai.usage.output_tokens", 40)
+            time.sleep(0.002)
+
+        # Step 6: Tool call 3 (issue_refund)
+        with tracer.start_as_current_span("execute_tool issue_refund") as tool3:
+            tool3.set_attribute("gen_ai.operation.name", "execute_tool")
+            tool3.set_attribute("gen_ai.tool.name", "issue_refund")
+            tool3.set_attribute("gen_ai.tool.call.id", "call_refund_003")
+            if include_content:
+                tool3.set_attribute(
+                    "gen_ai.tool.call.arguments",
+                    json.dumps({"order_id": "4521", "amount": "full"}),
+                )
+            time.sleep(0.002)
+
+        # Step 7: Final response chat
+        with tracer.start_as_current_span("chat gpt-4o-mini") as chat4:
+            chat4.set_attribute("gen_ai.operation.name", "chat")
+            chat4.set_attribute("gen_ai.request.model", "gpt-4o-mini")
+            chat4.set_attribute("gen_ai.usage.input_tokens", 420)
+            chat4.set_attribute("gen_ai.usage.output_tokens", 55)
+            if include_content:
+                chat4.set_attribute(
+                    "gen_ai.output.messages",
+                    "I have verified the duplicate charge for order #4521. A full refund has been issued.",
+                )
+            time.sleep(0.002)
+
+    spans = sorted(exporter.get_finished_spans(), key=lambda s: s.start_time)
+    pb_request = encode_spans(spans)
+    return MessageToDict(pb_request)
+
+
+def generate_fixtures():
+    FIXTURES_DIR.mkdir(parents=True, exist_ok=True)
+
+    happy_data = generate_trace_data(include_content=True)
+    happy_path = FIXTURES_DIR / "refund_happy.json"
+    with open(happy_path, "w", encoding="utf-8") as f:
+        json.dump(happy_data, f, indent=2)
+    print(f"Generated {happy_path}")
+
+    no_args_data = generate_trace_data(include_content=False)
+    no_args_path = FIXTURES_DIR / "refund_no_args.json"
+    with open(no_args_path, "w", encoding="utf-8") as f:
+        json.dump(no_args_data, f, indent=2)
+    print(f"Generated {no_args_path}")
+
+
+if __name__ == "__main__":
+    generate_fixtures()

{traceeval_cli-0.1.0 → traceeval_cli-0.2.0}/src/traceeval/cli.py

@@ -1,3 +1,4 @@
+import os
 import sys
 import asyncio
 import typer
@@ -41,35 +42,51 @@ def main(
 def run(
     case_file: Path = typer.Option(..., "--case", "-c", help="Path to EDDTestCase JSON"),
     trace_file: Optional[Path] = typer.Option(None, "--trace", "-t", help="Path to static AgentTrace JSON"),
+    otel_trace: Optional[Path] = typer.Option(None, "--otel-trace", help="Path to OpenTelemetry trace JSON"),
     pipeline: Optional[str] = typer.Option(None, "--pipeline", "-p", help="Live agent function (e.g. 'examples.reference_agent:process_refund')"),
+    pricing: Optional[Path] = typer.Option(None, "--pricing", help="Path to custom JSON pricing file"),
     export_path: Optional[str] = typer.Option(None, "--export", "-e", help="Path to save the JSON EvaluationResult"),
     max_cost: float = typer.Option(0.10, "--max-cost", help="Maximum allowable session budget in USD"),
-    score_threshold: float = typer.Option(0.8, "--score-threshold", help="Minimum score threshold for intent/correctness"),
+    score_threshold: float = typer.Option(0.8, "--score-threshold", help="Minimum score (0-1) that every non-null judge dimension must meet"),
 ):
-    """Run a TraceEval evaluation against a static trace or a live agent pipeline."""
-    import os
-    if "OPENAI_API_KEY" not in os.environ and settings.llm_base_url is None:
+    """Run a TraceEval evaluation against a static trace, an OTel trace, or a live agent pipeline."""
+    has_api_key = bool(settings.llm_api_key or os.environ.get("OPENAI_API_KEY"))
+    has_base_url = bool(settings.llm_base_url)
+    if not (has_api_key or has_base_url):
         console.print("\n[bold red]Configuration Error:[/bold red]")
         console.print("Either LLM_API_KEY (or OPENAI_API_KEY) or LLM_BASE_URL must be configured.")
         raise typer.Exit(code=1)
 
-    if not trace_file and not pipeline:
-        console.print("[bold red]Error:[/bold red] You must provide either a static --trace file or a live --pipeline hook.")
+    provided = [x for x in [trace_file, otel_trace, pipeline] if x is not None]
+    if len(provided) != 1:
+        console.print(
+            "[bold red]Error:[/bold red] Exactly one of --trace, --otel-trace, or --pipeline must be provided."
+        )
         raise typer.Exit(code=1)
-    
+
     try:
         case = load_test_case(case_file)
-        
-        # Determine Execution Mode (Static vs Live)
+
+        # Determine Execution Mode (Static vs OTel vs Live)
         if pipeline:
             console.print(f"[dim]Mode: Live Pipeline execution ({pipeline})[/dim]")
             trace = run_live_pipeline(pipeline, case)
         elif trace_file:
             console.print(f"[dim]Mode: Static Batch execution ({trace_file})[/dim]")
             trace = load_trace(trace_file)
+        elif otel_trace:
+            console.print(f"[dim]Mode: OTel trace ({otel_trace})[/dim]")
+            from traceeval.loaders.otel import load_otel_trace
+            from traceeval.pricing import DEFAULT_PRICING, load_pricing
+
+            pricing_table = load_pricing(pricing) if pricing else DEFAULT_PRICING
+            otel_result = load_otel_trace(otel_trace, pricing=pricing_table)
+            for warning in otel_result.warnings:
+                console.print(f"[yellow]Warning:[/yellow] {warning}")
+            trace = otel_result.trace
         else:
-            raise ValueError("No trace file or pipeline provided.")
-            
+            raise ValueError("No valid trace or pipeline provided.")
+
     except Exception as e:
         console.print(f"[bold red]Ingestion Error:[/bold red] {e}")
         raise typer.Exit(code=1)
@@ -83,12 +100,12 @@ def run(
 
     # Render Terminal Output
     render_result(result)
-    
+
     # Handle Export
     if export_path:
         export_to_json(result, export_path)
         console.print(f"\n[dim]Report successfully exported to {export_path}[/dim]")
-    
+
     if not result.passed:
         raise typer.Exit(code=1)
 

{traceeval_cli-0.1.0 → traceeval_cli-0.2.0}/src/traceeval/core/config.py

@@ -1,41 +1,50 @@
 import os
 from typing import Optional
-from pydantic_settings import BaseSettings, SettingsConfigDict
+
+from openai import AsyncOpenAI
 from pydantic import Field, SecretStr
+from pydantic_settings import BaseSettings, SettingsConfigDict
+
 
 class Settings(BaseSettings):
-    """
-    Defensive configuration manager. 
+    """Defensive configuration manager.
+
     Loads from .env and guarantees required keys are present at startup.
     """
+
     llm_api_key: Optional[SecretStr] = Field(
-        default=None, 
+        default=None,
         alias="LLM_API_KEY",
-        description="API Key for the Bring-Your-Own-Judge LLM provider"
+        description="API Key for the Bring-Your-Own-Judge LLM provider",
     )
     llm_base_url: Optional[str] = Field(
-        default=None, 
+        default=None,
         alias="LLM_BASE_URL",
-        description="Custom base URL for the LLM provider (e.g. Ollama, vLLM)"
+        description="Custom base URL for the LLM provider (e.g. Ollama, vLLM)",
     )
     llm_model_name: str = Field(
-        default="gpt-4o-mini", 
+        default="gpt-4o-mini",
         alias="LLM_MODEL_NAME",
-        description="Model name to target for the judge evaluations"
+        description="Model name to target for the judge evaluations",
     )
 
-    model_config = SettingsConfigDict(
-        env_file=".env",
-        env_file_encoding="utf-8",
-        extra="ignore"
-    )
+    model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore")
+
 
 # Instantiate the settings.
 settings = Settings()
 
-# Inject into environment variables for OpenAI SDK
-if settings.llm_api_key is not None:
-    os.environ["OPENAI_API_KEY"] = settings.llm_api_key.get_secret_value()
 
-if settings.llm_base_url is not None:
-    os.environ["OPENAI_BASE_URL"] = settings.llm_base_url
+def get_judge_client() -> AsyncOpenAI:
+    """Return an AsyncOpenAI client configured explicitly from settings or environment."""
+    api_key = (
+        settings.llm_api_key.get_secret_value()
+        if settings.llm_api_key
+        else os.environ.get("OPENAI_API_KEY")
+    )
+    base_url = settings.llm_base_url
+
+    if not api_key and base_url:
+        api_key = "not-needed"
+
+    return AsyncOpenAI(api_key=api_key, base_url=base_url)

{traceeval_cli-0.1.0 → traceeval_cli-0.2.0}/src/traceeval/core/schema.py

@@ -5,9 +5,14 @@ from enum import Enum
 
 class TrajectoryMode(str, Enum):
     """How strict the agent's tool execution path must be evaluated."""
-    EXACT = "EXACT"           
-    IN_ORDER = "IN_ORDER"     
-    ANY_ORDER = "ANY_ORDER"  
+    EXACT = "EXACT"
+    IN_ORDER = "IN_ORDER"
+    ANY_ORDER = "ANY_ORDER"
+
+class CheckResult(BaseModel):
+    """Outcome of a single deterministic gate, with human-readable reasons on failure."""
+    passed: bool
+    reasons: List[str] = []
 
 class GoldenRecord(BaseModel):
     meta_id: str
@@ -47,6 +52,7 @@ class AgentTrace(BaseModel):
     executed_tools: List[ToolCall]
     final_output: str
     total_token_cost_usd: float = Field(ge=0.0)
+    cost_complete: bool = True
 
 class EvaluationDimensionScore(BaseModel):
     """Scores mapped directly to the 5 dimensions of Vibe Coding Evaluation."""
@@ -62,4 +68,5 @@ class EvaluationResult(BaseModel):
     case_id: str
     passed: bool
     scores: EvaluationDimensionScore
-    trace_summary: AgentTrace
+    trace_summary: AgentTrace
+    failures: List[str] = []