traceval 0.2.1__tar.gz → 0.2.3__tar.gz

{traceval-0.2.1 → traceval-0.2.3}/CHANGELOG.md

@@ -5,6 +5,24 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.2.3] - 2026-07-02
+
+### Fixed
+- **Adapter tool detection no longer depends on demo tool names.** The Langfuse and OTel adapters matched span names against the demo agent's tool vocabulary (`order_lookup`, `stripe_lookup`, `kb_search`), so real exports with any other tool names silently classified tool spans as `other`, breaking `tool_error` labeling, `tool_sequence`/`no_tool_loop` generation, and cluster signatures. Langfuse SPANs now classify via `metadata.tool`, user-supplied `--tool-span-names` globs, or a documented input+output/error heuristic; OTel uses GenAI semantic-convention attributes with attribute-only fallbacks. Detection is proven vocabulary-free by new `create_ticket` fixtures. Surfaced by external review, found by reading; a real export would have found it in seconds.
+- Sub-millisecond latencies render as `<0.1` in the run summary instead of `0.0`.
+
+### Added
+- `traceval ingest --tool-span-names`: comma-separated name globs marking spans as tool calls (replaces the Langfuse heuristic, adds an OTel fallback).
+- `docs/targets.md`: the exact run-against-my-agent contract (HTTP request/response shape, callable return shapes, timeout and failure behavior) with an executable FastAPI example.
+- `docs/extending.md`: custom outcome rules, redaction hook, judge configuration as implemented, and an honest `traceval.yaml` reference.
+- `docs/formats.md`: complete annotated generic-format examples (success and tool-failure lines that ingest as-is) and a required-vs-optional field table; Langfuse/OTel sections rewritten to match the new heuristics.
+
+## [0.2.2] - 2026-07-02
+
+### Fixed
+- A run in which zero cases execute (e.g. unresolvable target) now writes a self-describing run report with an `errors` section instead of writing nothing; `--json` never reports `null`. Added a single clear top-level error line on target-resolution failure. Added `make test` and CONTRIBUTING as the canonical dev commands. Reported via external review of a failed-target invocation.
+- Run report schema additions (existing fields unchanged): `summary.errored` counts cases that never executed due to setup/collection errors; top-level `errors` lists `{stage, detail}` entries for `target_resolution`, `collection`, and `setup` failures, with identical details deduplicated into one entry carrying a `count`; `results` is `[]` rather than absent when nothing executed. The terminal summary now shows `Errored: n`.
+
 ## [0.2.1] - 2026-07-02
 
 ### Added

traceval-0.2.3/CONTRIBUTING.md

@@ -0,0 +1,22 @@
+# Contributing to traceval
+
+## Setup
+
+```bash
+git clone https://github.com/theramkm/traceval.git
+cd traceval
+uv sync          # installs the package and dev dependencies (same as CI)
+```
+
+## Development loop
+
+```bash
+make test        # pytest -q
+make lint        # ruff check, ruff format --check, mypy
+make demo        # end-to-end smoke: healthy agent passes, buggy agent fails
+make all         # lint + test
+```
+
+CI runs the same commands on Python 3.11, 3.12, and 3.13, plus a
+wheel-based demo smoke job, and enforces 85% coverage. Keep all of it
+green; add a test for every behavior change.

traceval-0.2.3/Makefile

@@ -0,0 +1,14 @@
+.PHONY: test lint demo all
+
+test:
+	uv run pytest -q
+
+lint:
+	uv run ruff check src tests examples
+	uv run ruff format --check src tests examples
+	uv run mypy src/traceval
+
+demo:
+	uv run traceval demo -o /tmp/traceval-demo --force
+
+all: lint test

{traceval-0.2.1 → traceval-0.2.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: traceval
-Version: 0.2.1
+Version: 0.2.3
 Summary: Trace-to-Eval Compiler
 License: MIT
 License-File: LICENSE
@@ -106,6 +106,8 @@ Report written to analysis/report.html
 
 `analysis/report.html` is the single-file page shown in the screenshot above. Pass `--evals evals/` to overlay eval coverage per cluster, and `--rules my_rules.py` to add your own labeling rules. To view it over HTTP instead of `file://`, `traceval serve analysis` starts a stdlib localhost server and prints the report URL.
 
+Custom labeling rules, the redaction hook, and judge configuration are documented in [docs/extending.md](docs/extending.md).
+
 ### 3. Generate
 
 ```bash
@@ -131,19 +133,19 @@ traceval Run Summary
 ┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━┓
 ┃ Case ID              ┃ Cluster    ┃ Outcome ┃ Latency (ms) ┃
 ┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━┩
-│ c_0c422a7a__case_001 │ c_0c422a7a │  PASS   │          0.0 │
-│ c_1e5d0942__case_002 │ c_1e5d0942 │  PASS   │          0.0 │
-│ c_2c881177__case_003 │ c_2c881177 │  PASS   │          0.0 │
-│ c_361535b0__case_004 │ c_361535b0 │  PASS   │          0.0 │
-│ c_9a8a4644__case_005 │ c_9a8a4644 │  PASS   │          0.0 │
-│ c_d30af83a__case_006 │ c_d30af83a │  PASS   │          0.0 │
-│ c_d3f3b631__case_007 │ c_d3f3b631 │  PASS   │          0.0 │
-│ c_e834c13c__case_008 │ c_e834c13c │  PASS   │          0.0 │
+│ c_0c422a7a__case_001 │ c_0c422a7a │  PASS   │         <0.1 │
+│ c_1e5d0942__case_002 │ c_1e5d0942 │  PASS   │         <0.1 │
+│ c_2c881177__case_003 │ c_2c881177 │  PASS   │         <0.1 │
+│ c_361535b0__case_004 │ c_361535b0 │  PASS   │         <0.1 │
+│ c_9a8a4644__case_005 │ c_9a8a4644 │  PASS   │         <0.1 │
+│ c_d30af83a__case_006 │ c_d30af83a │  PASS   │         <0.1 │
+│ c_d3f3b631__case_007 │ c_d3f3b631 │  PASS   │         <0.1 │
+│ c_e834c13c__case_008 │ c_e834c13c │  PASS   │         <0.1 │
 └──────────────────────┴────────────┴─────────┴──────────────┘
-Total: 8 | Passed: 8 | Failed: 0
+Total: 8 | Passed: 8 | Failed: 0 | Errored: 0
 ```
 
-The target is an HTTP URL or a `module:function` callable. Checks cover `exact`, `contains_any`, `not_contains`, `regex`, `json_schema`, `tool_sequence`, `no_tool_loop`, and `judge`. Run reports land in `<evals_dir>/runs/` (override with `--runs-dir`); pass `--compare <previous report>` to print regressions and improvements between runs. The exit code is nonzero when any case fails.
+The target is an HTTP URL or a `module:function` callable; the exact request/response contract, with a copy-pasteable FastAPI example, is in [docs/targets.md](docs/targets.md). Checks cover `exact`, `contains_any`, `not_contains`, `regex`, `json_schema`, `tool_sequence`, `no_tool_loop`, and `judge`. Run reports land in `<evals_dir>/runs/` (override with `--runs-dir`); pass `--compare <previous report>` to print regressions and improvements between runs. The exit code is nonzero when any case fails.
 
 ### 5. Calibrate the judge
 
@@ -191,7 +193,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-      - uses: theramkm/traceval@v0.2.1
+      - uses: theramkm/traceval@v0.2.3
         with:
           evals-dir: evals/
           target: myapp.agent:invoke_agent   # or an HTTP URL
@@ -200,6 +202,11 @@ jobs:
 
 Inputs: `evals-dir` and `target` (required); `judge`, `compare`, `only`, `runs-dir`, `traceval-version`, `python-version` (optional). For a real LLM judge, set `judge: openai` and pass `OPENAI_API_KEY` (or `GEMINI_API_KEY`) via `env:` from your repository secrets.
 
+## Development
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for setup.
+Run the test suite with `make test` and the full gate set with `make lint`.
+
 ## Honest Limitations
 
 * **Side-Effect Free**: traceval assertions evaluate input/output matches. It does not attempt to replay side effects (e.g., updating database records) on mock tools.

{traceval-0.2.1 → traceval-0.2.3}/README.md

@@ -89,6 +89,8 @@ Report written to analysis/report.html
 
 `analysis/report.html` is the single-file page shown in the screenshot above. Pass `--evals evals/` to overlay eval coverage per cluster, and `--rules my_rules.py` to add your own labeling rules. To view it over HTTP instead of `file://`, `traceval serve analysis` starts a stdlib localhost server and prints the report URL.
 
+Custom labeling rules, the redaction hook, and judge configuration are documented in [docs/extending.md](docs/extending.md).
+
 ### 3. Generate
 
 ```bash
@@ -114,19 +116,19 @@ traceval Run Summary
 ┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━┓
 ┃ Case ID              ┃ Cluster    ┃ Outcome ┃ Latency (ms) ┃
 ┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━┩
-│ c_0c422a7a__case_001 │ c_0c422a7a │  PASS   │          0.0 │
-│ c_1e5d0942__case_002 │ c_1e5d0942 │  PASS   │          0.0 │
-│ c_2c881177__case_003 │ c_2c881177 │  PASS   │          0.0 │
-│ c_361535b0__case_004 │ c_361535b0 │  PASS   │          0.0 │
-│ c_9a8a4644__case_005 │ c_9a8a4644 │  PASS   │          0.0 │
-│ c_d30af83a__case_006 │ c_d30af83a │  PASS   │          0.0 │
-│ c_d3f3b631__case_007 │ c_d3f3b631 │  PASS   │          0.0 │
-│ c_e834c13c__case_008 │ c_e834c13c │  PASS   │          0.0 │
+│ c_0c422a7a__case_001 │ c_0c422a7a │  PASS   │         <0.1 │
+│ c_1e5d0942__case_002 │ c_1e5d0942 │  PASS   │         <0.1 │
+│ c_2c881177__case_003 │ c_2c881177 │  PASS   │         <0.1 │
+│ c_361535b0__case_004 │ c_361535b0 │  PASS   │         <0.1 │
+│ c_9a8a4644__case_005 │ c_9a8a4644 │  PASS   │         <0.1 │
+│ c_d30af83a__case_006 │ c_d30af83a │  PASS   │         <0.1 │
+│ c_d3f3b631__case_007 │ c_d3f3b631 │  PASS   │         <0.1 │
+│ c_e834c13c__case_008 │ c_e834c13c │  PASS   │         <0.1 │
 └──────────────────────┴────────────┴─────────┴──────────────┘
-Total: 8 | Passed: 8 | Failed: 0
+Total: 8 | Passed: 8 | Failed: 0 | Errored: 0
 ```
 
-The target is an HTTP URL or a `module:function` callable. Checks cover `exact`, `contains_any`, `not_contains`, `regex`, `json_schema`, `tool_sequence`, `no_tool_loop`, and `judge`. Run reports land in `<evals_dir>/runs/` (override with `--runs-dir`); pass `--compare <previous report>` to print regressions and improvements between runs. The exit code is nonzero when any case fails.
+The target is an HTTP URL or a `module:function` callable; the exact request/response contract, with a copy-pasteable FastAPI example, is in [docs/targets.md](docs/targets.md). Checks cover `exact`, `contains_any`, `not_contains`, `regex`, `json_schema`, `tool_sequence`, `no_tool_loop`, and `judge`. Run reports land in `<evals_dir>/runs/` (override with `--runs-dir`); pass `--compare <previous report>` to print regressions and improvements between runs. The exit code is nonzero when any case fails.
 
 ### 5. Calibrate the judge
 
@@ -174,7 +176,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-      - uses: theramkm/traceval@v0.2.1
+      - uses: theramkm/traceval@v0.2.3
         with:
           evals-dir: evals/
           target: myapp.agent:invoke_agent   # or an HTTP URL
@@ -183,6 +185,11 @@ jobs:
 
 Inputs: `evals-dir` and `target` (required); `judge`, `compare`, `only`, `runs-dir`, `traceval-version`, `python-version` (optional). For a real LLM judge, set `judge: openai` and pass `OPENAI_API_KEY` (or `GEMINI_API_KEY`) via `env:` from your repository secrets.
 
+## Development
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for setup.
+Run the test suite with `make test` and the full gate set with `make lint`.
+
 ## Honest Limitations
 
 * **Side-Effect Free**: traceval assertions evaluate input/output matches. It does not attempt to replay side effects (e.g., updating database records) on mock tools.

traceval-0.2.3/docs/extending.md

@@ -0,0 +1,97 @@
+# Extension points
+
+traceval has three: custom outcome rules, a redaction hook, and judge
+configuration.
+
+## Custom outcome rules (`traceval analyze --rules my_rules.py`)
+
+The file must expose a module-level `RULES` list. Each entry is either a
+`Rule` object from `traceval.analyze.outcomes` or a bare callable with the
+signature `(Trace) -> Outcome | None`. Return `None` to pass; the first
+rule that returns an `Outcome` wins. **User rules run before the
+built-ins**, so they can override any built-in label.
+
+```python
+# my_rules.py
+from traceval.analyze.outcomes import Rule
+from traceval.model import Outcome, Trace
+
+
+def escalation(trace: Trace) -> Outcome | None:
+    if trace.final_output and "escalate to human" in trace.final_output.lower():
+        return Outcome(
+            label="bad_output",
+            reason="agent escalated instead of resolving",
+            rule_id="R_USER_ESCALATION",
+            labeled_by="user_rule",
+        )
+    return None
+
+
+RULES = [Rule("R_USER_ESCALATION", "Escalations count as failures", escalation)]
+```
+
+```bash
+traceval analyze traces.db --rules my_rules.py -o analysis
+```
+
+`Outcome` requires `label` (one of `success`, `tool_error`,
+`validation_error`, `loop`, `timeout`, `bad_output`, `unknown`), `reason`,
+and `labeled_by`; `rule_id` is optional but shows up in reports, so set it.
+Built-in rules run afterwards in this order: `R_TOOL_ERROR`, `R_LLM_ERROR`,
+`R_LOOP`, `R_TIMEOUT`, `R_VALIDATION`, `R_EMPTY_OUTPUT`,
+`R_DEFAULT_SUCCESS`, `R_UNKNOWN`.
+
+## Redaction hook (`traceval generate --redact-hook module:function`)
+
+A `str -> str` function applied to case inputs and reference outputs before
+they are written to YAML, after the built-in scrubbers (emails, credit
+cards, phone numbers, API tokens). The module is imported from the current
+working directory.
+
+```python
+# my_redactions.py
+import re
+
+
+def scrub(text: str) -> str:
+    return re.sub(r"ACC-\d{6}", "[REDACTED_ACCOUNT]", text)
+```
+
+```bash
+traceval generate traces.db -o evals --include-failures --redact-hook my_redactions:scrub
+```
+
+## Judge configuration (`traceval run --judge ...`)
+
+| Value | Behavior |
+| --- | --- |
+| `fake` (default) | Deterministic offline judge: keyword-overlap heuristic, never gates a run at the generated `min_score`. Use it in CI without API keys. |
+| `<model>` | `OpenAICompatJudge` against `https://api.openai.com/v1` with that model, e.g. `--judge gpt-4o-mini`. |
+| `<model>:<base_url>` | Any OpenAI-compatible endpoint, e.g. `--judge llama3:http://localhost:11434/v1`. |
+
+API keys come from the environment: `OPENAI_API_KEY` first, then
+`GEMINI_API_KEY`. If only `GEMINI_API_KEY` is set and the base URL is the
+OpenAI default, traceval automatically routes to Gemini's OpenAI-compatible
+endpoint (`https://generativelanguage.googleapis.com/v1beta/openai`) with
+model `gemini-2.5-flash`.
+
+Judge calls are budget-capped at 200 per run (hardcoded); calls beyond the
+budget score 0.0 with an explanatory reason. Validate any real judge with
+`traceval calibrate` before trusting its scores.
+
+## `traceval.yaml` reference
+
+`traceval generate` writes this scaffold next to the suite. Honest status
+of every key:
+
+| Key | Default | Consumed today? |
+| --- | --- | --- |
+| `schema_version` | `"1"` | No, informational. |
+| `target.default_url` | `http://localhost:8000/agent` | **Yes**: used as the target when `traceval run` is invoked without `--target`. |
+| `target.timeout_s` | `30` | No: the HTTP timeout is fixed at 30s in code. |
+| `judge.default_provider` | `fake` | No: the CLI's `--judge` default (`fake`) applies instead. |
+| `judge.max_judge_calls` | `200` | No: the 200-call budget is hardcoded. |
+
+The unconsumed keys document intended configuration surface; treat them as
+reserved.

{traceval-0.2.1 → traceval-0.2.3}/docs/formats.md

@@ -22,11 +22,41 @@ All trace adapters transform incoming logs into the `Trace` schema (`src/traceva
 
 ## 1. Generic format (`generic`)
 
-A line-by-line JSONL file where each line is a raw JSON string validating directly against our canonical `Trace` model.
+A line-by-line JSONL file where each line is a raw JSON string validating directly against our canonical `Trace` model. This is the format to convert to when your backend is not natively supported; the two examples below are complete and ingest as-is.
+
+### Required vs optional fields
+
+| Field | Required | Notes |
+| --- | --- | --- |
+| `trace_id` | yes | Unique string. |
+| `source` | yes | Free-form origin label, e.g. `"generic"`. |
+| `started_at` | yes | ISO 8601 datetime. |
+| `task_input` | yes | The user request that started the trace. |
+| `steps` | yes | May be `[]`. Each step needs `index` and `kind` (`llm`, `tool`, `retrieval`, `other`); `llm` steps need an `llm` object with `span_id` and `input_messages`, `tool` steps need a `tool` object with `span_id`, `name`, `arguments_json`. |
+| `ended_at` | no | Missing/`null` triggers the built-in timeout rule. |
+| `final_output` | no | `null`/empty triggers the empty-output rule. |
+| `metadata` | no | `dict[str, str]`. |
+| `schema_version`, `outcome` | no | `outcome` is filled by `traceval analyze`; supply it only to pre-label. |
+
+### Example: success trace (one llm step, one tool step)
+
+```json
+{"trace_id": "tr-ok-1", "source": "generic", "started_at": "2026-07-01T12:00:00Z", "ended_at": "2026-07-01T12:00:02Z", "task_input": "Where is order 88421?", "final_output": "Your order 88421 is in transit.", "steps": [{"index": 0, "kind": "llm", "llm": {"span_id": "s1", "model": "gpt-4o-mini", "input_messages": [{"role": "user", "content": "Where is order 88421?"}], "output_message": {"role": "assistant", "content": "Let me look that up."}}}, {"index": 1, "kind": "tool", "tool": {"span_id": "s2", "name": "order_lookup", "arguments_json": "{\"order_id\": \"88421\"}", "output": "status: in_transit", "latency_ms": 142.0}}]}
+```
+
+Annotations: `arguments_json` is a raw JSON *string* (never a parsed object), so nothing is lost in translation. `output_message`, `model`, token counts, and `latency_ms` are all optional.
+
+### Example: failure trace (tool step with `error` set)
+
+```json
+{"trace_id": "tr-fail-1", "source": "generic", "started_at": "2026-07-01T12:05:00Z", "ended_at": "2026-07-01T12:05:01Z", "task_input": "Refund order 88421", "final_output": "Error: refund service unavailable.", "steps": [{"index": 0, "kind": "tool", "tool": {"span_id": "s3", "name": "refund_api", "arguments_json": "{\"order_id\": \"88421\"}", "output": null, "error": "HTTP 503 Service Unavailable"}}]}
+```
+
+A non-null `tool.error` makes the built-in `R_TOOL_ERROR` rule label the trace `tool_error`, which is what routes it into a failure cluster and, with `--include-failures`, into a regression case.
 
 ### Assumptions & Heuristics
-- Direct structural validation.
-- Lines failing to parse are logged as warnings and skipped.
+- Direct structural validation, no field inference.
+- Lines failing to parse are logged as warnings and skipped; the ingest never aborts.
 
 ---
 
@@ -40,10 +70,14 @@ OTel traces are ingested from flat lists of JSON span logs (e.g. OTLP export for
   - `gen_ai.system`
   - `gen_ai.prompt`
   - `gen_ai.completion`
-- **Tool Call**: Spans containing:
-  - `gen_ai.tool.name`
-  - `gen_ai.tool.arguments`
-  - Or span name matching `order_lookup`, `stripe_lookup`, or `kb_search`.
+- **Tool Call**: Spans matching any of these signals:
+  - `gen_ai.tool.name` present in attributes (primary, per GenAI semantic conventions)
+  - `gen_ai.tool.arguments` present in attributes
+  - `gen_ai.operation.name` attribute equal to `"execute_tool"`
+  - `tool.name` present in attributes
+  - Span name matches a user-supplied glob from `traceval ingest --tool-span-names` (comma-separated, e.g. `"*_lookup,tool_*"`)
+
+  There is no built-in tool-name list; detection never depends on a specific tool vocabulary.
 - **Other**: All other spans are categorized as `other`.
 
 ### Attribute Translations
@@ -89,7 +123,12 @@ Langfuse exports traces as JSON objects with nested lists of observations (of ty
   - `llm.prompt_tokens` $\leftarrow$ `usage.promptTokens`
   - `llm.completion_tokens` $\leftarrow$ `usage.completionTokens`
   - `llm.error` $\leftarrow$ `statusMessage` when `level == "ERROR"`
-- **SPAN** $\rightarrow$ `ToolCall` (if name is order/stripe/kb lookup or `metadata.tool` matches):
+- **SPAN** $\rightarrow$ `ToolCall` when, in priority order:
+  1. the observation's `metadata.tool` is set (explicit marker, always wins), or
+  2. `traceval ingest --tool-span-names` globs were supplied and the observation `name` matches one (globs replace the heuristic below), or
+  3. default heuristic: the SPAN recorded an `input` AND either an `output` or an error signal (`level == "ERROR"` or `statusMessage` set). Failed tool calls often produce no output, which is why an error counts as the second signal.
+
+  SPANs matching none of these become `other` steps. Field mapping:
   - `tool.name` $\leftarrow$ Observation `name`
   - `tool.arguments_json` $\leftarrow$ Observation `input` (serialized to JSON)
   - `tool.output` $\leftarrow$ Observation `output` (stringified)

traceval-0.2.3/docs/targets.md

@@ -0,0 +1,91 @@
+# Running the suite against your agent: the target contract
+
+`traceval run <evals_dir> --target <target>` accepts two target forms. Both
+are resolved once per session; an unresolvable target prints one clear
+`ERROR:` line at the top of the output, records a `target_resolution` entry
+in the run report's `errors` list, and the run exits nonzero.
+
+## HTTP target
+
+Pass any `http://` or `https://` URL. For every case, traceval sends:
+
+```
+POST <your url>
+Content-Type: application/json
+
+{"input": "<the case's task text>"}
+```
+
+with a 30 second timeout. Non-2xx responses fail the case.
+
+The JSON response is interpreted as follows:
+
+| Field | Meaning |
+| --- | --- |
+| `output` / `final_output` / `response` | The agent's answer, checked in that priority order; the first present key wins. |
+| any other key | Fallback: if none of the three keys exist, the first non-`tool_calls` value is stringified and used as the output. A non-object response body is stringified whole. |
+| `tool_calls` (optional) | List of tool invocations, either `{"name": "..."}` objects or plain strings. Only the names are kept. |
+
+`tool_calls` exists so the `tool_sequence` check (did the agent call the
+recorded tools, in order) and the `no_tool_loop` check (did it avoid calling
+the same tool 3+ times consecutively) have something to score. If your
+endpoint omits it, generated `tool_sequence` checks will fail; either return
+the names or delete those checks from the case YAML.
+
+If the URL is unreachable (connection refused, invalid URL) traceval prints
+the one-line `ERROR:` at session start; cases still run and fail
+individually so the run report stays complete.
+
+### Minimal FastAPI implementation
+
+```python
+# my_agent.py
+from fastapi import FastAPI
+from pydantic import BaseModel
+
+app = FastAPI()
+
+
+class AgentInput(BaseModel):
+    input: str
+
+
+@app.post("/agent")
+def run_agent(payload: AgentInput) -> dict:
+    # Call your real agent here.
+    answer = f"You asked: {payload.input}"
+    return {
+        "output": answer,
+        "tool_calls": [{"name": "kb_search"}],
+    }
+```
+
+```bash
+uvicorn my_agent:app --port 8000 &
+traceval run evals/ --target http://127.0.0.1:8000/agent --judge fake
+```
+
+## Callable target
+
+Pass `module:function` (one colon). traceval inserts the current working
+directory into `sys.path`, imports the module, and calls the function with
+the case's task text as a single string argument:
+
+```python
+def invoke_agent(input_text: str) -> dict: ...
+```
+
+Accepted return shapes:
+
+- a dict, interpreted exactly like the HTTP response above
+  (`output`/`final_output`/`response` priority, optional `tool_calls`)
+- an object with an `.output` attribute (and optional `.tool_calls`)
+- anything else, stringified whole and used as the output
+
+```bash
+traceval run evals/ --target myapp.agent:invoke_agent --judge fake
+```
+
+Because the working directory is importable, `myapp/agent.py` in your repo
+root works without installation. A bad module path or missing attribute
+produces the same one-line `ERROR:` plus a self-describing run report.

{traceval-0.2.1 → traceval-0.2.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "traceval"
-version = "0.2.1"
+version = "0.2.3"
 description = "Trace-to-Eval Compiler"
 readme = "README.md"
 requires-python = ">=3.11"

traceval-0.2.3/src/traceval/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.2.3"

{traceval-0.2.1 → traceval-0.2.3}/src/traceval/cli.py

@@ -25,15 +25,28 @@ def ingest(
     path: str,
     format: str = typer.Option("auto", help="auto|otel|langfuse|langsmith|generic"),
     output: str = typer.Option("traces.db", "-o", help="SQLite database output path"),
+    tool_span_names: str = typer.Option(
+        None,
+        "--tool-span-names",
+        help=(
+            "Comma-separated name globs (e.g. '*_lookup,tool_*') that mark "
+            "spans as tool calls; replaces the built-in heuristic for "
+            "Langfuse SPANs and adds a fallback for OTel spans"
+        ),
+    ),
     json_output: bool = typer.Option(
         False, "--json", help="Print a machine-readable JSON summary to stdout"
     ),
 ) -> None:
     """Ingest trace logs into SQLite database."""
+    globs = None
+    if tool_span_names:
+        globs = [g.strip() for g in tool_span_names.split(",") if g.strip()]
+
     db = TraceStore(output)
     try:
         ok_count, span_count, warn_count, log_file = ingest_file(
-            Path(path), db, format_name=format
+            Path(path), db, format_name=format, tool_span_globs=globs
         )
         if json_output:
             typer.echo(