traceval 0.2.2__tar.gz → 0.2.3__tar.gz

{traceval-0.2.2 → traceval-0.2.3}/CHANGELOG.md

@@ -5,6 +5,18 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.2.3] - 2026-07-02
+
+### Fixed
+- **Adapter tool detection no longer depends on demo tool names.** The Langfuse and OTel adapters matched span names against the demo agent's tool vocabulary (`order_lookup`, `stripe_lookup`, `kb_search`), so real exports with any other tool names silently classified tool spans as `other`, breaking `tool_error` labeling, `tool_sequence`/`no_tool_loop` generation, and cluster signatures. Langfuse SPANs now classify via `metadata.tool`, user-supplied `--tool-span-names` globs, or a documented input+output/error heuristic; OTel uses GenAI semantic-convention attributes with attribute-only fallbacks. Detection is proven vocabulary-free by new `create_ticket` fixtures. Surfaced by external review, found by reading; a real export would have found it in seconds.
+- Sub-millisecond latencies render as `<0.1` in the run summary instead of `0.0`.
+
+### Added
+- `traceval ingest --tool-span-names`: comma-separated name globs marking spans as tool calls (replaces the Langfuse heuristic, adds an OTel fallback).
+- `docs/targets.md`: the exact run-against-my-agent contract (HTTP request/response shape, callable return shapes, timeout and failure behavior) with an executable FastAPI example.
+- `docs/extending.md`: custom outcome rules, redaction hook, judge configuration as implemented, and an honest `traceval.yaml` reference.
+- `docs/formats.md`: complete annotated generic-format examples (success and tool-failure lines that ingest as-is) and a required-vs-optional field table; Langfuse/OTel sections rewritten to match the new heuristics.
+
 ## [0.2.2] - 2026-07-02
 
 ### Fixed

{traceval-0.2.2 → traceval-0.2.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: traceval
-Version: 0.2.2
+Version: 0.2.3
 Summary: Trace-to-Eval Compiler
 License: MIT
 License-File: LICENSE
@@ -106,6 +106,8 @@ Report written to analysis/report.html
 
 `analysis/report.html` is the single-file page shown in the screenshot above. Pass `--evals evals/` to overlay eval coverage per cluster, and `--rules my_rules.py` to add your own labeling rules. To view it over HTTP instead of `file://`, `traceval serve analysis` starts a stdlib localhost server and prints the report URL.
 
+Custom labeling rules, the redaction hook, and judge configuration are documented in [docs/extending.md](docs/extending.md).
+
 ### 3. Generate
 
 ```bash
@@ -131,19 +133,19 @@ traceval Run Summary
 ┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━┓
 ┃ Case ID              ┃ Cluster    ┃ Outcome ┃ Latency (ms) ┃
 ┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━┩
-│ c_0c422a7a__case_001 │ c_0c422a7a │  PASS   │          0.0 │
-│ c_1e5d0942__case_002 │ c_1e5d0942 │  PASS   │          0.0 │
-│ c_2c881177__case_003 │ c_2c881177 │  PASS   │          0.0 │
-│ c_361535b0__case_004 │ c_361535b0 │  PASS   │          0.0 │
-│ c_9a8a4644__case_005 │ c_9a8a4644 │  PASS   │          0.0 │
-│ c_d30af83a__case_006 │ c_d30af83a │  PASS   │          0.0 │
-│ c_d3f3b631__case_007 │ c_d3f3b631 │  PASS   │          0.0 │
-│ c_e834c13c__case_008 │ c_e834c13c │  PASS   │          0.0 │
+│ c_0c422a7a__case_001 │ c_0c422a7a │  PASS   │         <0.1 │
+│ c_1e5d0942__case_002 │ c_1e5d0942 │  PASS   │         <0.1 │
+│ c_2c881177__case_003 │ c_2c881177 │  PASS   │         <0.1 │
+│ c_361535b0__case_004 │ c_361535b0 │  PASS   │         <0.1 │
+│ c_9a8a4644__case_005 │ c_9a8a4644 │  PASS   │         <0.1 │
+│ c_d30af83a__case_006 │ c_d30af83a │  PASS   │         <0.1 │
+│ c_d3f3b631__case_007 │ c_d3f3b631 │  PASS   │         <0.1 │
+│ c_e834c13c__case_008 │ c_e834c13c │  PASS   │         <0.1 │
 └──────────────────────┴────────────┴─────────┴──────────────┘
 Total: 8 | Passed: 8 | Failed: 0 | Errored: 0
 ```
 
-The target is an HTTP URL or a `module:function` callable. Checks cover `exact`, `contains_any`, `not_contains`, `regex`, `json_schema`, `tool_sequence`, `no_tool_loop`, and `judge`. Run reports land in `<evals_dir>/runs/` (override with `--runs-dir`); pass `--compare <previous report>` to print regressions and improvements between runs. The exit code is nonzero when any case fails.
+The target is an HTTP URL or a `module:function` callable; the exact request/response contract, with a copy-pasteable FastAPI example, is in [docs/targets.md](docs/targets.md). Checks cover `exact`, `contains_any`, `not_contains`, `regex`, `json_schema`, `tool_sequence`, `no_tool_loop`, and `judge`. Run reports land in `<evals_dir>/runs/` (override with `--runs-dir`); pass `--compare <previous report>` to print regressions and improvements between runs. The exit code is nonzero when any case fails.
 
 ### 5. Calibrate the judge
 
@@ -191,7 +193,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-      - uses: theramkm/traceval@v0.2.2
+      - uses: theramkm/traceval@v0.2.3
         with:
           evals-dir: evals/
           target: myapp.agent:invoke_agent   # or an HTTP URL

{traceval-0.2.2 → traceval-0.2.3}/README.md

@@ -89,6 +89,8 @@ Report written to analysis/report.html
 
 `analysis/report.html` is the single-file page shown in the screenshot above. Pass `--evals evals/` to overlay eval coverage per cluster, and `--rules my_rules.py` to add your own labeling rules. To view it over HTTP instead of `file://`, `traceval serve analysis` starts a stdlib localhost server and prints the report URL.
 
+Custom labeling rules, the redaction hook, and judge configuration are documented in [docs/extending.md](docs/extending.md).
+
 ### 3. Generate
 
 ```bash
@@ -114,19 +116,19 @@ traceval Run Summary
 ┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━┓
 ┃ Case ID              ┃ Cluster    ┃ Outcome ┃ Latency (ms) ┃
 ┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━┩
-│ c_0c422a7a__case_001 │ c_0c422a7a │  PASS   │          0.0 │
-│ c_1e5d0942__case_002 │ c_1e5d0942 │  PASS   │          0.0 │
-│ c_2c881177__case_003 │ c_2c881177 │  PASS   │          0.0 │
-│ c_361535b0__case_004 │ c_361535b0 │  PASS   │          0.0 │
-│ c_9a8a4644__case_005 │ c_9a8a4644 │  PASS   │          0.0 │
-│ c_d30af83a__case_006 │ c_d30af83a │  PASS   │          0.0 │
-│ c_d3f3b631__case_007 │ c_d3f3b631 │  PASS   │          0.0 │
-│ c_e834c13c__case_008 │ c_e834c13c │  PASS   │          0.0 │
+│ c_0c422a7a__case_001 │ c_0c422a7a │  PASS   │         <0.1 │
+│ c_1e5d0942__case_002 │ c_1e5d0942 │  PASS   │         <0.1 │
+│ c_2c881177__case_003 │ c_2c881177 │  PASS   │         <0.1 │
+│ c_361535b0__case_004 │ c_361535b0 │  PASS   │         <0.1 │
+│ c_9a8a4644__case_005 │ c_9a8a4644 │  PASS   │         <0.1 │
+│ c_d30af83a__case_006 │ c_d30af83a │  PASS   │         <0.1 │
+│ c_d3f3b631__case_007 │ c_d3f3b631 │  PASS   │         <0.1 │
+│ c_e834c13c__case_008 │ c_e834c13c │  PASS   │         <0.1 │
 └──────────────────────┴────────────┴─────────┴──────────────┘
 Total: 8 | Passed: 8 | Failed: 0 | Errored: 0
 ```
 
-The target is an HTTP URL or a `module:function` callable. Checks cover `exact`, `contains_any`, `not_contains`, `regex`, `json_schema`, `tool_sequence`, `no_tool_loop`, and `judge`. Run reports land in `<evals_dir>/runs/` (override with `--runs-dir`); pass `--compare <previous report>` to print regressions and improvements between runs. The exit code is nonzero when any case fails.
+The target is an HTTP URL or a `module:function` callable; the exact request/response contract, with a copy-pasteable FastAPI example, is in [docs/targets.md](docs/targets.md). Checks cover `exact`, `contains_any`, `not_contains`, `regex`, `json_schema`, `tool_sequence`, `no_tool_loop`, and `judge`. Run reports land in `<evals_dir>/runs/` (override with `--runs-dir`); pass `--compare <previous report>` to print regressions and improvements between runs. The exit code is nonzero when any case fails.
 
 ### 5. Calibrate the judge
 
@@ -174,7 +176,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-      - uses: theramkm/traceval@v0.2.2
+      - uses: theramkm/traceval@v0.2.3
         with:
           evals-dir: evals/
           target: myapp.agent:invoke_agent   # or an HTTP URL

traceval-0.2.3/docs/extending.md

@@ -0,0 +1,97 @@
+# Extension points
+
+traceval has three: custom outcome rules, a redaction hook, and judge
+configuration.
+
+## Custom outcome rules (`traceval analyze --rules my_rules.py`)
+
+The file must expose a module-level `RULES` list. Each entry is either a
+`Rule` object from `traceval.analyze.outcomes` or a bare callable with the
+signature `(Trace) -> Outcome | None`. Return `None` to pass; the first
+rule that returns an `Outcome` wins. **User rules run before the
+built-ins**, so they can override any built-in label.
+
+```python
+# my_rules.py
+from traceval.analyze.outcomes import Rule
+from traceval.model import Outcome, Trace
+
+
+def escalation(trace: Trace) -> Outcome | None:
+    if trace.final_output and "escalate to human" in trace.final_output.lower():
+        return Outcome(
+            label="bad_output",
+            reason="agent escalated instead of resolving",
+            rule_id="R_USER_ESCALATION",
+            labeled_by="user_rule",
+        )
+    return None
+
+
+RULES = [Rule("R_USER_ESCALATION", "Escalations count as failures", escalation)]
+```
+
+```bash
+traceval analyze traces.db --rules my_rules.py -o analysis
+```
+
+`Outcome` requires `label` (one of `success`, `tool_error`,
+`validation_error`, `loop`, `timeout`, `bad_output`, `unknown`), `reason`,
+and `labeled_by`; `rule_id` is optional but shows up in reports, so set it.
+Built-in rules run afterwards in this order: `R_TOOL_ERROR`, `R_LLM_ERROR`,
+`R_LOOP`, `R_TIMEOUT`, `R_VALIDATION`, `R_EMPTY_OUTPUT`,
+`R_DEFAULT_SUCCESS`, `R_UNKNOWN`.
+
+## Redaction hook (`traceval generate --redact-hook module:function`)
+
+A `str -> str` function applied to case inputs and reference outputs before
+they are written to YAML, after the built-in scrubbers (emails, credit
+cards, phone numbers, API tokens). The module is imported from the current
+working directory.
+
+```python
+# my_redactions.py
+import re
+
+
+def scrub(text: str) -> str:
+    return re.sub(r"ACC-\d{6}", "[REDACTED_ACCOUNT]", text)
+```
+
+```bash
+traceval generate traces.db -o evals --include-failures --redact-hook my_redactions:scrub
+```
+
+## Judge configuration (`traceval run --judge ...`)
+
+| Value | Behavior |
+| --- | --- |
+| `fake` (default) | Deterministic offline judge: keyword-overlap heuristic, never gates a run at the generated `min_score`. Use it in CI without API keys. |
+| `<model>` | `OpenAICompatJudge` against `https://api.openai.com/v1` with that model, e.g. `--judge gpt-4o-mini`. |
+| `<model>:<base_url>` | Any OpenAI-compatible endpoint, e.g. `--judge llama3:http://localhost:11434/v1`. |
+
+API keys come from the environment: `OPENAI_API_KEY` first, then
+`GEMINI_API_KEY`. If only `GEMINI_API_KEY` is set and the base URL is the
+OpenAI default, traceval automatically routes to Gemini's OpenAI-compatible
+endpoint (`https://generativelanguage.googleapis.com/v1beta/openai`) with
+model `gemini-2.5-flash`.
+
+Judge calls are budget-capped at 200 per run (hardcoded); calls beyond the
+budget score 0.0 with an explanatory reason. Validate any real judge with
+`traceval calibrate` before trusting its scores.
+
+## `traceval.yaml` reference
+
+`traceval generate` writes this scaffold next to the suite. Honest status
+of every key:
+
+| Key | Default | Consumed today? |
+| --- | --- | --- |
+| `schema_version` | `"1"` | No, informational. |
+| `target.default_url` | `http://localhost:8000/agent` | **Yes**: used as the target when `traceval run` is invoked without `--target`. |
+| `target.timeout_s` | `30` | No: the HTTP timeout is fixed at 30s in code. |
+| `judge.default_provider` | `fake` | No: the CLI's `--judge` default (`fake`) applies instead. |
+| `judge.max_judge_calls` | `200` | No: the 200-call budget is hardcoded. |
+
+The unconsumed keys document intended configuration surface; treat them as
+reserved.

{traceval-0.2.2 → traceval-0.2.3}/docs/formats.md

@@ -22,11 +22,41 @@ All trace adapters transform incoming logs into the `Trace` schema (`src/traceva
 
 ## 1. Generic format (`generic`)
 
-A line-by-line JSONL file where each line is a raw JSON string validating directly against our canonical `Trace` model.
+A line-by-line JSONL file where each line is a raw JSON string validating directly against our canonical `Trace` model. This is the format to convert to when your backend is not natively supported; the two examples below are complete and ingest as-is.
+
+### Required vs optional fields
+
+| Field | Required | Notes |
+| --- | --- | --- |
+| `trace_id` | yes | Unique string. |
+| `source` | yes | Free-form origin label, e.g. `"generic"`. |
+| `started_at` | yes | ISO 8601 datetime. |
+| `task_input` | yes | The user request that started the trace. |
+| `steps` | yes | May be `[]`. Each step needs `index` and `kind` (`llm`, `tool`, `retrieval`, `other`); `llm` steps need an `llm` object with `span_id` and `input_messages`, `tool` steps need a `tool` object with `span_id`, `name`, `arguments_json`. |
+| `ended_at` | no | Missing/`null` triggers the built-in timeout rule. |
+| `final_output` | no | `null`/empty triggers the empty-output rule. |
+| `metadata` | no | `dict[str, str]`. |
+| `schema_version`, `outcome` | no | `outcome` is filled by `traceval analyze`; supply it only to pre-label. |
+
+### Example: success trace (one llm step, one tool step)
+
+```json
+{"trace_id": "tr-ok-1", "source": "generic", "started_at": "2026-07-01T12:00:00Z", "ended_at": "2026-07-01T12:00:02Z", "task_input": "Where is order 88421?", "final_output": "Your order 88421 is in transit.", "steps": [{"index": 0, "kind": "llm", "llm": {"span_id": "s1", "model": "gpt-4o-mini", "input_messages": [{"role": "user", "content": "Where is order 88421?"}], "output_message": {"role": "assistant", "content": "Let me look that up."}}}, {"index": 1, "kind": "tool", "tool": {"span_id": "s2", "name": "order_lookup", "arguments_json": "{\"order_id\": \"88421\"}", "output": "status: in_transit", "latency_ms": 142.0}}]}
+```
+
+Annotations: `arguments_json` is a raw JSON *string* (never a parsed object), so nothing is lost in translation. `output_message`, `model`, token counts, and `latency_ms` are all optional.
+
+### Example: failure trace (tool step with `error` set)
+
+```json
+{"trace_id": "tr-fail-1", "source": "generic", "started_at": "2026-07-01T12:05:00Z", "ended_at": "2026-07-01T12:05:01Z", "task_input": "Refund order 88421", "final_output": "Error: refund service unavailable.", "steps": [{"index": 0, "kind": "tool", "tool": {"span_id": "s3", "name": "refund_api", "arguments_json": "{\"order_id\": \"88421\"}", "output": null, "error": "HTTP 503 Service Unavailable"}}]}
+```
+
+A non-null `tool.error` makes the built-in `R_TOOL_ERROR` rule label the trace `tool_error`, which is what routes it into a failure cluster and, with `--include-failures`, into a regression case.
 
 ### Assumptions & Heuristics
-- Direct structural validation.
-- Lines failing to parse are logged as warnings and skipped.
+- Direct structural validation, no field inference.
+- Lines failing to parse are logged as warnings and skipped; the ingest never aborts.
 
 ---
 
@@ -40,10 +70,14 @@ OTel traces are ingested from flat lists of JSON span logs (e.g. OTLP export for
   - `gen_ai.system`
   - `gen_ai.prompt`
   - `gen_ai.completion`
-- **Tool Call**: Spans containing:
-  - `gen_ai.tool.name`
-  - `gen_ai.tool.arguments`
-  - Or span name matching `order_lookup`, `stripe_lookup`, or `kb_search`.
+- **Tool Call**: Spans matching any of these signals:
+  - `gen_ai.tool.name` present in attributes (primary, per GenAI semantic conventions)
+  - `gen_ai.tool.arguments` present in attributes
+  - `gen_ai.operation.name` attribute equal to `"execute_tool"`
+  - `tool.name` present in attributes
+  - Span name matches a user-supplied glob from `traceval ingest --tool-span-names` (comma-separated, e.g. `"*_lookup,tool_*"`)
+
+  There is no built-in tool-name list; detection never depends on a specific tool vocabulary.
 - **Other**: All other spans are categorized as `other`.
 
 ### Attribute Translations
@@ -89,7 +123,12 @@ Langfuse exports traces as JSON objects with nested lists of observations (of ty
   - `llm.prompt_tokens` $\leftarrow$ `usage.promptTokens`
   - `llm.completion_tokens` $\leftarrow$ `usage.completionTokens`
   - `llm.error` $\leftarrow$ `statusMessage` when `level == "ERROR"`
-- **SPAN** $\rightarrow$ `ToolCall` (if name is order/stripe/kb lookup or `metadata.tool` matches):
+- **SPAN** $\rightarrow$ `ToolCall` when, in priority order:
+  1. the observation's `metadata.tool` is set (explicit marker, always wins), or
+  2. `traceval ingest --tool-span-names` globs were supplied and the observation `name` matches one (globs replace the heuristic below), or
+  3. default heuristic: the SPAN recorded an `input` AND either an `output` or an error signal (`level == "ERROR"` or `statusMessage` set). Failed tool calls often produce no output, which is why an error counts as the second signal.
+
+  SPANs matching none of these become `other` steps. Field mapping:
   - `tool.name` $\leftarrow$ Observation `name`
   - `tool.arguments_json` $\leftarrow$ Observation `input` (serialized to JSON)
   - `tool.output` $\leftarrow$ Observation `output` (stringified)

traceval-0.2.3/docs/targets.md

@@ -0,0 +1,91 @@
+# Running the suite against your agent: the target contract
+
+`traceval run <evals_dir> --target <target>` accepts two target forms. Both
+are resolved once per session; an unresolvable target prints one clear
+`ERROR:` line at the top of the output, records a `target_resolution` entry
+in the run report's `errors` list, and the run exits nonzero.
+
+## HTTP target
+
+Pass any `http://` or `https://` URL. For every case, traceval sends:
+
+```
+POST <your url>
+Content-Type: application/json
+
+{"input": "<the case's task text>"}
+```
+
+with a 30 second timeout. Non-2xx responses fail the case.
+
+The JSON response is interpreted as follows:
+
+| Field | Meaning |
+| --- | --- |
+| `output` / `final_output` / `response` | The agent's answer, checked in that priority order; the first present key wins. |
+| any other key | Fallback: if none of the three keys exist, the first non-`tool_calls` value is stringified and used as the output. A non-object response body is stringified whole. |
+| `tool_calls` (optional) | List of tool invocations, either `{"name": "..."}` objects or plain strings. Only the names are kept. |
+
+`tool_calls` exists so the `tool_sequence` check (did the agent call the
+recorded tools, in order) and the `no_tool_loop` check (did it avoid calling
+the same tool 3+ times consecutively) have something to score. If your
+endpoint omits it, generated `tool_sequence` checks will fail; either return
+the names or delete those checks from the case YAML.
+
+If the URL is unreachable (connection refused, invalid URL) traceval prints
+the one-line `ERROR:` at session start; cases still run and fail
+individually so the run report stays complete.
+
+### Minimal FastAPI implementation
+
+```python
+# my_agent.py
+from fastapi import FastAPI
+from pydantic import BaseModel
+
+app = FastAPI()
+
+
+class AgentInput(BaseModel):
+    input: str
+
+
+@app.post("/agent")
+def run_agent(payload: AgentInput) -> dict:
+    # Call your real agent here.
+    answer = f"You asked: {payload.input}"
+    return {
+        "output": answer,
+        "tool_calls": [{"name": "kb_search"}],
+    }
+```
+
+```bash
+uvicorn my_agent:app --port 8000 &
+traceval run evals/ --target http://127.0.0.1:8000/agent --judge fake
+```
+
+## Callable target
+
+Pass `module:function` (one colon). traceval inserts the current working
+directory into `sys.path`, imports the module, and calls the function with
+the case's task text as a single string argument:
+
+```python
+def invoke_agent(input_text: str) -> dict: ...
+```
+
+Accepted return shapes:
+
+- a dict, interpreted exactly like the HTTP response above
+  (`output`/`final_output`/`response` priority, optional `tool_calls`)
+- an object with an `.output` attribute (and optional `.tool_calls`)
+- anything else, stringified whole and used as the output
+
+```bash
+traceval run evals/ --target myapp.agent:invoke_agent --judge fake
+```
+
+Because the working directory is importable, `myapp/agent.py` in your repo
+root works without installation. A bad module path or missing attribute
+produces the same one-line `ERROR:` plus a self-describing run report.

{traceval-0.2.2 → traceval-0.2.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "traceval"
-version = "0.2.2"
+version = "0.2.3"
 description = "Trace-to-Eval Compiler"
 readme = "README.md"
 requires-python = ">=3.11"

traceval-0.2.3/src/traceval/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.2.3"

{traceval-0.2.2 → traceval-0.2.3}/src/traceval/cli.py

@@ -25,15 +25,28 @@ def ingest(
     path: str,
     format: str = typer.Option("auto", help="auto|otel|langfuse|langsmith|generic"),
     output: str = typer.Option("traces.db", "-o", help="SQLite database output path"),
+    tool_span_names: str = typer.Option(
+        None,
+        "--tool-span-names",
+        help=(
+            "Comma-separated name globs (e.g. '*_lookup,tool_*') that mark "
+            "spans as tool calls; replaces the built-in heuristic for "
+            "Langfuse SPANs and adds a fallback for OTel spans"
+        ),
+    ),
     json_output: bool = typer.Option(
         False, "--json", help="Print a machine-readable JSON summary to stdout"
     ),
 ) -> None:
     """Ingest trace logs into SQLite database."""
+    globs = None
+    if tool_span_names:
+        globs = [g.strip() for g in tool_span_names.split(",") if g.strip()]
+
     db = TraceStore(output)
     try:
         ok_count, span_count, warn_count, log_file = ingest_file(
-            Path(path), db, format_name=format
+            Path(path), db, format_name=format, tool_span_globs=globs
         )
         if json_output:
             typer.echo(

{traceval-0.2.2 → traceval-0.2.3}/src/traceval/compile/templates/conftest.py.jinja

@@ -201,11 +201,12 @@ def pytest_sessionfinish(session, exitstatus):
 
         for r in _results_accumulator:
             outcome_str = "[bold green]PASS[/bold green]" if r["passed"] else "[bold red]FAIL[/bold red]"
+            latency_str = "<0.1" if r["latency_ms"] < 0.1 else f"{r['latency_ms']:.1f}"
             table.add_row(
                 r["case_id"],
                 r["cluster"],
                 outcome_str,
-                f"{r['latency_ms']:.1f}"
+                latency_str
             )
 
         console.print(table)

{traceval-0.2.2 → traceval-0.2.3}/src/traceval/ingest/__init__.py

@@ -58,6 +58,7 @@ def ingest_file(
     store: TraceStore,
     format_name: str = "auto",
     log_path: Path | None = None,
+    tool_span_globs: list[str] | None = None,
 ) -> tuple[int, int, int, Path]:
     if format_name == "auto":
         format_name = detect_format(path)
@@ -66,7 +67,7 @@ def ingest_file(
     if not adapter_cls:
         raise ValueError(f"Unknown format: {format_name}")
 
-    adapter = adapter_cls()
+    adapter = adapter_cls(tool_span_globs=tool_span_globs)
 
     if log_path is None:
         db_p = Path(store.db_path)

{traceval-0.2.2 → traceval-0.2.3}/src/traceval/ingest/base.py

@@ -8,6 +8,8 @@ from traceval.model import Trace
 class Adapter(Protocol):
     format_name: ClassVar[str]
 
+    def __init__(self, tool_span_globs: list[str] | None = None) -> None: ...
+
     def detect(self, first_lines: list[str]) -> bool: ...
 
     def parse(self, path: Path) -> Iterator[Trace]: ...

{traceval-0.2.2 → traceval-0.2.3}/src/traceval/ingest/generic.py

@@ -13,6 +13,10 @@ logger = logging.getLogger(__name__)
 class GenericAdapter(Adapter):
     format_name: ClassVar[str] = "generic"
 
+    def __init__(self, tool_span_globs: list[str] | None = None) -> None:
+        # Generic traces declare step kinds explicitly; globs are unused.
+        self.tool_span_globs = tool_span_globs
+
     def detect(self, first_lines: list[str]) -> bool:
         if not first_lines:
             return False

{traceval-0.2.2 → traceval-0.2.3}/src/traceval/ingest/langfuse.py

@@ -1,3 +1,4 @@
+import fnmatch
 import json
 import logging
 from collections.abc import Iterator
@@ -25,6 +26,9 @@ def parse_iso_datetime(dt_str: str | None) -> datetime | None:
 class LangfuseAdapter(Adapter):
     format_name: ClassVar[str] = "langfuse"
 
+    def __init__(self, tool_span_globs: list[str] | None = None) -> None:
+        self.tool_span_globs = tool_span_globs
+
     def detect(self, first_lines: list[str]) -> bool:
         if not first_lines:
             return False
@@ -142,11 +146,30 @@ class LangfuseAdapter(Adapter):
                             )
 
                         elif obs_type == "SPAN":
-                            # Check if tool span
-                            is_tool = (
-                                name in ["order_lookup", "stripe_lookup", "kb_search"]
-                                or obs.get("metadata", {}).get("tool") is not None
-                            )
+                            # Tool classification, in priority order:
+                            # 1. explicit metadata.tool marker (always wins)
+                            # 2. user globs (--tool-span-names), which REPLACE
+                            #    the built-in heuristic when provided
+                            # 3. heuristic: a SPAN that recorded an input and
+                            #    either an output or an error signal is a tool
+                            #    call (failed tools often produce no output,
+                            #    so an ERROR level counts as second signal)
+                            metadata = obs.get("metadata") or {}
+                            if metadata.get("tool") is not None:
+                                is_tool = True
+                            elif self.tool_span_globs is not None:
+                                is_tool = any(
+                                    fnmatch.fnmatch(name, pattern)
+                                    for pattern in self.tool_span_globs
+                                )
+                            else:
+                                has_error_signal = (
+                                    level == "ERROR"
+                                    or obs.get("statusMessage") is not None
+                                )
+                                is_tool = obs.get("input") is not None and (
+                                    obs.get("output") is not None or has_error_signal
+                                )
                             if is_tool:
                                 obs_input = obs.get("input")
                                 if isinstance(obs_input, (dict, list)):

{traceval-0.2.2 → traceval-0.2.3}/src/traceval/ingest/langsmith.py

@@ -25,6 +25,10 @@ def parse_iso_datetime(dt_str: str | None) -> datetime | None:
 class LangsmithAdapter(Adapter):
     format_name: ClassVar[str] = "langsmith"
 
+    def __init__(self, tool_span_globs: list[str] | None = None) -> None:
+        # LangSmith runs carry an explicit run_type; globs are unused.
+        self.tool_span_globs = tool_span_globs
+
     def detect(self, first_lines: list[str]) -> bool:
         if not first_lines:
             return False

{traceval-0.2.2 → traceval-0.2.3}/src/traceval/ingest/otel.py

@@ -1,3 +1,4 @@
+import fnmatch
 import json
 import logging
 from collections.abc import Iterator
@@ -26,6 +27,9 @@ def parse_iso_datetime(dt_str: str | None) -> datetime | None:
 class OtelAdapter(Adapter):
     format_name: ClassVar[str] = "otel"
 
+    def __init__(self, tool_span_globs: list[str] | None = None) -> None:
+        self.tool_span_globs = tool_span_globs
+
     def detect(self, first_lines: list[str]) -> bool:
         if not first_lines:
             return False
@@ -170,13 +174,27 @@ class OtelAdapter(Adapter):
                             )
                         )
 
-                    # Check if Tool call
+                    # Check if Tool call. Primary signal: GenAI semantic
+                    # convention attributes. Fallbacks are attribute- or
+                    # user-glob-based only, never tool-name lists.
                     elif (
                         "gen_ai.tool.name" in span_attrs
                         or "gen_ai.tool.arguments" in span_attrs
-                        or name in ["order_lookup", "stripe_lookup", "kb_search"]
+                        or span_attrs.get("gen_ai.operation.name") == "execute_tool"
+                        or "tool.name" in span_attrs
+                        or (
+                            self.tool_span_globs is not None
+                            and any(
+                                fnmatch.fnmatch(name, pattern)
+                                for pattern in self.tool_span_globs
+                            )
+                        )
                     ):
-                        tool_name = span_attrs.get("gen_ai.tool.name") or name
+                        tool_name = (
+                            span_attrs.get("gen_ai.tool.name")
+                            or span_attrs.get("tool.name")
+                            or name
+                        )
                         args_json = span_attrs.get("gen_ai.tool.arguments") or "{}"
                         tool_call = ToolCall(
                             span_id=span["span_id"],

{traceval-0.2.2 → traceval-0.2.3}/tests/fixtures/README.md

@@ -16,3 +16,14 @@ This directory contains synthetic logs and outputs used for testing traceval ada
 10. **tr-010 (Simple success)**: Direct assistant output response.
 11. **tr-011 (Latency timeout)**: Completed but exceeds runtime duration bounds.
 12. **tr-012 (Success stripe lookup)**: Simple tool path resolving successfully.
+
+## Backend export fixtures (otel_spans / langfuse_export / langsmith_runs)
+
+Each backend file carries the same five stories (success with llm+tool,
+tool error, llm error, timeout, tool loop) plus one corrupt line to test
+warning handling. The Langfuse and OTel files additionally contain a
+sixth trace (`lf-006` / `otel-006`) whose tool span is named
+`create_ticket`, a name the demo agent never uses: it proves tool
+detection relies on the documented signals (Langfuse input/output
+heuristic, OTel gen_ai semantic-convention attributes), not on a tool
+vocabulary.

{traceval-0.2.2 → traceval-0.2.3}/tests/fixtures/langfuse_export.jsonl

@@ -3,4 +3,5 @@
 {"id": "lf-003", "name": "agent_run", "timestamp": "2026-07-01T12:02:00Z", "input": "Get user data.", "output": null, "metadata": {}, "observations": [{"id": "obs-4", "type": "GENERATION", "name": "llm_call", "startTime": "2026-07-01T12:02:01Z", "endTime": "2026-07-01T12:02:02Z", "level": "ERROR", "statusMessage": "API Error: Invalid credentials"}]}
 {"id": "lf-004", "name": "agent_run", "timestamp": "2026-07-01T12:03:00Z", "input": "Run infinite process.", "output": null, "metadata": {}, "observations": []}
 {"id": "lf-005", "name": "agent_run", "timestamp": "2026-07-01T12:04:00Z", "input": "Find information.", "output": null, "metadata": {}, "observations": [{"id": "obs-5", "type": "SPAN", "name": "kb_search", "startTime": "2026-07-01T12:04:01Z", "endTime": "2026-07-01T12:04:02Z", "input": {"query": "refund"}, "output": "None"}, {"id": "obs-6", "type": "SPAN", "name": "kb_search", "startTime": "2026-07-01T12:04:03Z", "endTime": "2026-07-01T12:04:04Z", "input": {"query": "refund"}, "output": "None"}, {"id": "obs-7", "type": "SPAN", "name": "kb_search", "startTime": "2026-07-01T12:04:05Z", "endTime": "2026-07-01T12:04:06Z", "input": {"query": "refund"}, "output": "None"}]}
+{"id": "lf-006", "name": "agent_run", "timestamp": "2026-07-01T12:06:00Z", "input": "Open a support ticket for my broken keyboard", "output": "Ticket TCK-778 created.", "metadata": {}, "observations": [{"id": "obs-6a", "type": "SPAN", "name": "create_ticket", "startTime": "2026-07-01T12:06:01Z", "endTime": "2026-07-01T12:06:02Z", "input": {"summary": "broken keyboard"}, "output": "TCK-778"}]}
 INVALID_LINE_CORRUPT_JSON

{traceval-0.2.2 → traceval-0.2.3}/tests/fixtures/otel_spans.jsonl

@@ -10,4 +10,6 @@
 {"trace_id": "otel-005", "span_id": "tool-5-1", "parent_span_id": "root-5", "name": "kb_search", "start_time": "2026-07-01T12:04:01Z", "end_time": "2026-07-01T12:04:02Z", "attributes": {"gen_ai.tool.name": "kb_search", "gen_ai.tool.arguments": "{\"query\": \"refund\"}"}}
 {"trace_id": "otel-005", "span_id": "tool-5-2", "parent_span_id": "root-5", "name": "kb_search", "start_time": "2026-07-01T12:04:03Z", "end_time": "2026-07-01T12:04:04Z", "attributes": {"gen_ai.tool.name": "kb_search", "gen_ai.tool.arguments": "{\"query\": \"refund\"}"}}
 {"trace_id": "otel-005", "span_id": "tool-5-3", "parent_span_id": "root-5", "name": "kb_search", "start_time": "2026-07-01T12:04:05Z", "end_time": "2026-07-01T12:04:06Z", "attributes": {"gen_ai.tool.name": "kb_search", "gen_ai.tool.arguments": "{\"query\": \"refund\"}"}}
+{"trace_id": "otel-006", "span_id": "root-6", "parent_span_id": null, "name": "agent_run", "start_time": "2026-07-01T12:06:00Z", "end_time": "2026-07-01T12:06:03Z", "attributes": {"gen_ai.task_input": "Open a support ticket for my broken keyboard", "gen_ai.final_output": "Ticket TCK-778 created."}}
+{"trace_id": "otel-006", "span_id": "span-6a", "parent_span_id": "root-6", "name": "create_ticket", "start_time": "2026-07-01T12:06:01Z", "end_time": "2026-07-01T12:06:02Z", "attributes": {"gen_ai.tool.name": "create_ticket", "gen_ai.tool.arguments": "{\"summary\": \"broken keyboard\"}", "gen_ai.tool.output": "TCK-778"}}
 INVALID_LINE_CORRUPT_JSON

{traceval-0.2.2 → traceval-0.2.3}/tests/test_phase2.py

@@ -29,13 +29,13 @@ def test_otel_ingest(tmp_path):
         log_path=log_path,
     )
 
-    # 5 traces are defined in our otel_spans.jsonl + 1 invalid line
-    assert ok_count == 5
+    # 6 traces are defined in our otel_spans.jsonl + 1 invalid line
+    assert ok_count == 6
     assert warn_count == 1
     assert log_file == log_path
 
     traces = list(store.list_traces())
-    assert len(traces) == 5
+    assert len(traces) == 6
 
     # Check trace 1 detail: otel-001 has 1 llm step and 1 tool step
     t1 = next(t for t in traces if t.trace_id == "otel-001")
@@ -70,12 +70,12 @@ def test_langfuse_ingest(tmp_path):
         log_path=log_path,
     )
 
-    # 5 traces + 1 invalid line
-    assert ok_count == 5
+    # 6 traces + 1 invalid line
+    assert ok_count == 6
     assert warn_count == 1
 
     traces = list(store.list_traces())
-    assert len(traces) == 5
+    assert len(traces) == 6
 
     t1 = next(t for t in traces if t.trace_id == "lf-001")
     assert len(t1.steps) == 2
@@ -133,7 +133,55 @@ def test_robustness_on_shuffled_inputs(tmp_path):
         store,
         format_name="otel",
     )
-    # Reconstructs all 5 traces
-    assert ok_count == 5
+    # Reconstructs all 6 traces
+    assert ok_count == 6
     assert warn_count == 0
     store.close()
+
+
+def test_real_tool_names_detected(tmp_path):
+    # Tool detection must not depend on the demo agent's tool vocabulary:
+    # the create_ticket spans in both fixtures classify as tools via the
+    # documented signals (Langfuse input/output heuristic; OTel gen_ai
+    # semantic-convention attributes), never a name list.
+    for fixture, fmt, trace_id in [
+        ("langfuse_export.jsonl", "langfuse", "lf-006"),
+        ("otel_spans.jsonl", "otel", "otel-006"),
+    ]:
+        store = TraceStore(tmp_path / f"{fmt}.db")
+        ingest_file(FIXTURES_DIR / fixture, store, format_name=fmt)
+        traces = list(store.list_traces())
+        store.close()
+
+        trace = next(t for t in traces if t.trace_id == trace_id)
+        tool_steps = [s for s in trace.steps if s.kind == "tool"]
+        assert len(tool_steps) == 1, f"{fmt}: create_ticket span not detected"
+        assert tool_steps[0].tool.name == "create_ticket"
+
+
+def test_tool_span_globs_override(tmp_path):
+    # User globs replace the built-in Langfuse heuristic: with a glob that
+    # matches nothing, the create_ticket SPAN (no metadata.tool) must NOT
+    # classify as a tool; with a matching glob it must.
+    store = TraceStore(tmp_path / "none.db")
+    ingest_file(
+        FIXTURES_DIR / "langfuse_export.jsonl",
+        store,
+        format_name="langfuse",
+        tool_span_globs=["nothing_matches_*"],
+    )
+    trace = next(t for t in store.list_traces() if t.trace_id == "lf-006")
+    store.close()
+    assert all(s.kind != "tool" for s in trace.steps)
+
+    store = TraceStore(tmp_path / "match.db")
+    ingest_file(
+        FIXTURES_DIR / "langfuse_export.jsonl",
+        store,
+        format_name="langfuse",
+        tool_span_globs=["create_*"],
+    )
+    trace = next(t for t in store.list_traces() if t.trace_id == "lf-006")
+    store.close()
+    assert [s.kind for s in trace.steps] == ["tool"]
+    assert trace.steps[0].tool.name == "create_ticket"

{traceval-0.2.2 → traceval-0.2.3}/uv.lock

@@ -1037,7 +1037,7 @@ wheels = [
 
 [[package]]
 name = "traceval"
-version = "0.2.2"
+version = "0.2.3"
 source = { editable = "." }
 dependencies = [
     { name = "httpx" },

traceval-0.2.2/src/traceval/__init__.py

@@ -1 +0,0 @@
-__version__ = "0.2.2"