valcore 0.2.6__tar.gz → 0.2.7__tar.gz

{valcore-0.2.6 → valcore-0.2.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: valcore
-Version: 0.2.6
+Version: 0.2.7
 Summary: A small, self-contained local tool for developing, improving, and running agentic evaluations.
 Author-email: Duncan McKinnon <dkmckinnon@outlook.com>
 License-Expression: Apache-2.0
@@ -98,6 +98,32 @@ The website docs explain the concepts the other four surfaces assume — label s
 frozen versions, run kinds, agreement metrics, installation, credentials, portable
 packages, CI, and Logfire. The app links there instead of maintaining a second copy.
 
+### Run an agent and evaluate its responses
+
+In **Runs → New run**, choose **Agent** to generate responses for a dataset,
+or **Agent then evaluators** to select a dataset, an agent version, and one or
+more evaluator versions. The workflow saves the agent responses automatically,
+then runs the evaluators concurrently against the original row data plus the new
+response columns. Each evaluator has its own run, progress, and results, linked
+from the agent run.
+
+Failed agent rows are skipped by the evaluators. If the agent run fails or is
+cancelled, queued evaluators do not start. A failing evaluator does not stop the
+others. Execution continues when you leave the page; keep the valcore server
+running. Concurrency is configured per run.
+
+The same workflow is available without a server:
+
+```bash
+valcore run pipeline support-agent --dataset support-cases \
+  --evaluator accuracy --evaluator safety \
+  --version v2 --evaluator-version accuracy v3 \
+  --concurrency 4 --watch --json
+```
+
+The CLI waits for completion, evaluates concurrently, and emits one JSON summary.
+See the bundled CLI reference for version selection and failure behavior.
+
 ## Quickstart
 
 Store your gateway API key, then start the app:
@@ -232,6 +258,7 @@ source of truth (`scripts/sync_command_table.py --check` fails CI if the two dri
 | `valcore serve` | Serve the web UI and API (`--port`, `--host`, `--no-browser`). |
 | `valcore list <evaluators\|datasets\|runs\|agents\|derivations>` | List resources as a table or, with `--json`, as JSON. |
 | `valcore run agent <agent>` | Run an agent over a dataset, one row, ad-hoc inputs, or a literal prompt. |
+| `valcore run pipeline <agent> --dataset <dataset> --evaluator <evaluator>` | Run an agent, save responses, and run one or more evaluators concurrently. |
 | `valcore run evaluator <evaluator> --dataset <dataset>` | Run an evaluator version over a dataset. |
 | `valcore run experiment <evaluator> --dataset <dataset>` | Run an evaluator through `pydantic_evals.Dataset.evaluate`. |
 | `valcore export <evaluator>` | Export an evaluator (and, with `--dataset`, a dataset) as a Python script or, with `--format json`, a portable eval package. |

{valcore-0.2.6 → valcore-0.2.7}/README.md

@@ -70,6 +70,32 @@ The website docs explain the concepts the other four surfaces assume — label s
 frozen versions, run kinds, agreement metrics, installation, credentials, portable
 packages, CI, and Logfire. The app links there instead of maintaining a second copy.
 
+### Run an agent and evaluate its responses
+
+In **Runs → New run**, choose **Agent** to generate responses for a dataset,
+or **Agent then evaluators** to select a dataset, an agent version, and one or
+more evaluator versions. The workflow saves the agent responses automatically,
+then runs the evaluators concurrently against the original row data plus the new
+response columns. Each evaluator has its own run, progress, and results, linked
+from the agent run.
+
+Failed agent rows are skipped by the evaluators. If the agent run fails or is
+cancelled, queued evaluators do not start. A failing evaluator does not stop the
+others. Execution continues when you leave the page; keep the valcore server
+running. Concurrency is configured per run.
+
+The same workflow is available without a server:
+
+```bash
+valcore run pipeline support-agent --dataset support-cases \
+  --evaluator accuracy --evaluator safety \
+  --version v2 --evaluator-version accuracy v3 \
+  --concurrency 4 --watch --json
+```
+
+The CLI waits for completion, evaluates concurrently, and emits one JSON summary.
+See the bundled CLI reference for version selection and failure behavior.
+
 ## Quickstart
 
 Store your gateway API key, then start the app:
@@ -204,6 +230,7 @@ source of truth (`scripts/sync_command_table.py --check` fails CI if the two dri
 | `valcore serve` | Serve the web UI and API (`--port`, `--host`, `--no-browser`). |
 | `valcore list <evaluators\|datasets\|runs\|agents\|derivations>` | List resources as a table or, with `--json`, as JSON. |
 | `valcore run agent <agent>` | Run an agent over a dataset, one row, ad-hoc inputs, or a literal prompt. |
+| `valcore run pipeline <agent> --dataset <dataset> --evaluator <evaluator>` | Run an agent, save responses, and run one or more evaluators concurrently. |
 | `valcore run evaluator <evaluator> --dataset <dataset>` | Run an evaluator version over a dataset. |
 | `valcore run experiment <evaluator> --dataset <dataset>` | Run an evaluator through `pydantic_evals.Dataset.evaluate`. |
 | `valcore export <evaluator>` | Export an evaluator (and, with `--dataset`, a dataset) as a Python script or, with `--format json`, a portable eval package. |

{valcore-0.2.6 → valcore-0.2.7}/src/valcore/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.6'
-__version_tuple__ = version_tuple = (0, 2, 6)
+__version__ = version = '0.2.7'
+__version_tuple__ = version_tuple = (0, 2, 7)
 
 __commit_id__ = commit_id = None

{valcore-0.2.6 → valcore-0.2.7}/src/valcore/api/routes/runs.py

@@ -12,7 +12,7 @@ from datetime import UTC, datetime
 from typing import Annotated
 
 from fastapi import APIRouter, Depends
-from pydantic import BaseModel, ConfigDict
+from pydantic import BaseModel, ConfigDict, Field
 from pydantic_ai import Agent as PydanticAgent
 from sse_starlette.sse import EventSourceResponse
 
@@ -33,6 +33,7 @@ from valcore.models import (
     check_dataset_compatibility,
     find_matching_label_set,
 )
+from valcore.pipeline import create_pipeline, prepare_evaluators
 from valcore.runner import RunEvent, execute_run
 from valcore.settings import is_local_cli_model
 from valcore.store import Store
@@ -78,7 +79,8 @@ class RunCreate(BaseModel):
     kind: RunKind
     version_id: str
     dataset_id: str
-    concurrency: int | None = None
+    concurrency: int | None = Field(default=None, ge=1, le=64)
+    evaluator_version_ids: list[str] = Field(default_factory=list)
     # DERIVE creates its own derivation, so this must be absent; for EVAL it names the
     # response-column contract to read; VALIDATION rejects it until scoped labels exist.
     derivation_id: str | None = None
@@ -110,6 +112,8 @@ class RunOut(BaseModel):
     error: str | None
     cancel_requested: bool
     derivation_id: str | None
+    parent_run_id: str | None
+    evaluator_run_ids: list[str]
 
 
 class ResultRow(BaseModel):
@@ -176,8 +180,12 @@ class RunCoverageOut(BaseModel):
 def _run_out(store: Store, run: Run) -> RunOut:
     """Build a run response including its optional derivation contract."""
     link = store.get_run_derivation(run.id)
+    parent_id, child_ids = store.run_dependencies(run.id)
     return RunOut(
-        **run.model_dump(), derivation_id=link.derivation_id if link is not None else None
+        **run.model_dump(),
+        derivation_id=link.derivation_id if link is not None else None,
+        parent_run_id=parent_id,
+        evaluator_run_ids=child_ids,
     )
 
 
@@ -203,7 +211,11 @@ async def _run_to_completion(
     model, which reaches an already-logged-in CLI on this machine rather than the gateway.
     """
 
+    _, child_ids = await asyncio.to_thread(store.run_dependencies, run_id)
+
     async def on_event(event: RunEvent) -> None:
+        if child_ids and event.type in ("finished", "error"):
+            return
         bus.publish(run_id, {"type": event.type, "run_id": run_id, "payload": event.payload})
 
     try:
@@ -235,9 +247,38 @@ async def _run_to_completion(
         )
         bus.publish(run_id, {"type": "error", "run_id": run_id, "payload": {"error": str(exc)}})
     finally:
+        if child_ids:
+            await _start_dependent_runs(store, run_id, agent_factory)
+            finished = await asyncio.to_thread(store.get_run, run_id)
+            bus.publish(
+                run_id,
+                {
+                    "type": "finished",
+                    "run_id": run_id,
+                    "payload": {"status": finished.status.value, "metrics": finished.metrics},
+                },
+            )
         bus.close(run_id)
 
 
+async def _start_dependent_runs(
+    store: Store,
+    parent_id: str,
+    agent_factory: AgentFactory | None,
+) -> None:
+    """Save successful responses before launching independent evaluator tasks."""
+    children = await asyncio.to_thread(prepare_evaluators, store, parent_id)
+    for child in children:
+        if child.status is RunStatus.PENDING and not child.cancel_requested:
+            _launch_run(store, child.id, agent_factory)
+        else:
+            bus.publish(
+                child.id,
+                {"type": "finished", "run_id": child.id, "payload": {"status": child.status.value}},
+            )
+            bus.close(child.id)
+
+
 def _launch_run(
     store: Store,
     run_id: str,
@@ -305,6 +346,10 @@ async def create_run(body: RunCreate, store: StoreDep, agent_factory: AgentFacto
     the user has to go and read. Passing ``body.kind`` keeps an EVAL run permitted when the
     label spaces differ, which is the whole point of prescribing them.
     """
+    if body.evaluator_version_ids and body.kind is not RunKind.DERIVE:
+        raise ContractError("Follow-up evaluators require an agent run.")
+    if len(set(body.evaluator_version_ids)) != len(body.evaluator_version_ids):
+        raise ContractError("Select each evaluator version only once.")
     if body.kind is RunKind.DERIVE:
         if body.experiment:
             raise ContractError("A derive run cannot use the evaluator experiment engine.")
@@ -315,6 +360,16 @@ async def create_run(body: RunCreate, store: StoreDep, agent_factory: AgentFacto
             config.require_gateway_key()
         dataset = store.get_dataset(body.dataset_id)
         check_agent_dataset_compatibility(version, dataset)
+        if body.evaluator_version_ids:
+            run = create_pipeline(
+                store,
+                version.id,
+                dataset.id,
+                body.evaluator_version_ids,
+                body.concurrency or _DEFAULT_CONCURRENCY,
+            )
+            _launch_run(store, run.id, agent_factory)
+            return _run_out(store, run)
         run = store.create_run(
             kind=body.kind,
             version_id=body.version_id,
@@ -501,6 +556,12 @@ async def list_run_results(
     if run.kind is RunKind.DERIVE:
         raise ContractError("Derive runs do not have evaluator results.")
     rows_by_id = {row.id: row for row in store.list_rows(run.dataset_id)}
+    link = store.get_run_derivation(id)
+    derived_data = (
+        {row.row_id: row.data for row in store.derived_rows(link.derivation_id)}
+        if link is not None and link.role is DerivationRole.READS
+        else {}
+    )
 
     version = store.get_version(run.version_id)
     label_sets = store.list_label_sets(run.dataset_id)
@@ -539,7 +600,7 @@ async def list_run_results(
                 result_id=result.id,
                 row_id=result.row_id,
                 idx=row.idx if row is not None else -1,
-                data=row.data if row is not None else {},
+                data=derived_data.get(result.row_id, row.data if row is not None else {}),
                 output=result.output,
                 score_value=result.score_value,
                 agreement=result.agreement,
@@ -604,7 +665,13 @@ async def run_events(id: str, store: StoreDep) -> EventSourceResponse:
 @router.post("/{id}/cancel", response_model=RunOut)
 async def cancel_run(id: str, store: StoreDep) -> RunOut:
     """Flag a run for cancellation; the background task transitions it to ``CANCELLED``."""
-    return _run_out(store, store.request_cancel(id))
+    run = store.request_cancel(id)
+    parent_id, _ = store.run_dependencies(id)
+    if parent_id and run.status is RunStatus.PENDING:
+        run = store.update_run_status(id, RunStatus.CANCELLED, finished_at=datetime.now(UTC))
+        bus.publish(id, {"type": "finished", "run_id": id, "payload": {"status": "cancelled"}})
+        bus.close(id)
+    return _run_out(store, run)
 
 
 @router.post("/{id}/retry-failed", response_model=RunOut)
@@ -624,6 +691,11 @@ async def retry_failed(id: str, store: StoreDep, agent_factory: AgentFactoryDep)
     existing = store.get_run(id)
     if existing.kind is RunKind.DERIVE:
         raise ContractError("Derive runs only support whole-dataset execution.")
+    parent_id, _ = store.run_dependencies(id)
+    if parent_id and store.get_run_derivation(id) is None:
+        raise ContractError(
+            "This evaluator has no completed agent responses; start a new workflow."
+        )
     version = store.get_version(existing.version_id)
     if not is_local_cli_model(version.model):
         config.require_gateway_key()

{valcore-0.2.6 → valcore-0.2.7}/src/valcore/cli/main.py

@@ -71,11 +71,13 @@ from valcore.models import (
     RunStatus,
     ScoreKind,
     check_agent_dataset_compatibility,
+    check_dataset_compatibility,
     label_set_fields_from_schema,
     validate_agent_version,
     validate_version,
 )
 from valcore.paths import config_path
+from valcore.pipeline import create_pipeline, prepare_evaluators
 from valcore.runner import RunEvent, execute_run
 from valcore.settings import (
     LOCAL_CLI_NAMES,
@@ -244,6 +246,13 @@ def list_(ctx: click.Context, kind: str, as_json: bool) -> None:
         rows = []
         for run in store.list_runs():
             row = run.model_dump()
+            parent_id, child_ids = store.run_dependencies(run.id)
+            link = store.get_run_derivation(run.id)
+            row.update(
+                parent_run_id=parent_id,
+                evaluator_run_ids=child_ids,
+                derivation_id=link.derivation_id if link else None,
+            )
             row["accuracy"] = run.metrics.get("accuracy") if run.metrics else None
             rows.append(row)
         emit(
@@ -1136,6 +1145,131 @@ async def _drive_run(store: Store, run_id: str, watch: bool) -> Run:
     return run
 
 
+@run_group.command("pipeline")
+@click.argument("agent_ref")
+@click.option("--dataset", "dataset_ref", required=True, help="Dataset to run the agent over.")
+@click.option("--version", "version_name", default=None, help="Agent version (default: active).")
+@click.option(
+    "--evaluator",
+    "evaluator_refs",
+    multiple=True,
+    required=True,
+    help="Evaluator name or id prefix; repeat for multiple evaluators.",
+)
+@click.option(
+    "--evaluator-version",
+    "version_overrides",
+    type=(str, str),
+    multiple=True,
+    metavar="EVALUATOR VERSION",
+    help="Override a selected evaluator's active version.",
+)
+@click.option(
+    "--concurrency", type=click.IntRange(1, 64), default=None, help="Max concurrent rows per run."
+)
+@click.option("--json", "as_json", is_flag=True, help="Emit one JSON workflow summary.")
+@click.option("--watch", is_flag=True, help="Print per-row progress with run ids to stderr.")
+@click.pass_context
+def run_pipeline(
+    ctx: click.Context,
+    agent_ref: str,
+    dataset_ref: str,
+    version_name: str | None,
+    evaluator_refs: tuple[str, ...],
+    version_overrides: tuple[tuple[str, str], ...],
+    concurrency: int | None,
+    as_json: bool,
+    watch: bool,
+) -> None:
+    """Run an agent, save its responses, then evaluate them concurrently."""
+    store = _store(ctx)
+    agent = resolve_agent(store, agent_ref)
+    version = resolve_agent_version(store, agent, version_name)
+    dataset = resolve_dataset(store, dataset_ref)
+    evaluators = [resolve_evaluator(store, ref) for ref in evaluator_refs]
+    overrides = {}
+    for ref, name in version_overrides:
+        evaluator = resolve_evaluator(store, ref)
+        if evaluator.id not in {item.id for item in evaluators}:
+            raise ContractError(f"Version override {ref!r} must also be selected with --evaluator.")
+        if evaluator.id in overrides:
+            raise ContractError(f"Multiple version overrides for {ref!r}.")
+        overrides[evaluator.id] = name
+    versions = [resolve_version(store, item, overrides.get(item.id)) for item in evaluators]
+    workers = concurrency if concurrency is not None else get_settings().default_concurrency
+    parent = create_pipeline(store, version.id, dataset.id, [item.id for item in versions], workers)
+
+    async def drive(run_id: str) -> Run:
+        async def on_event(event: RunEvent) -> None:
+            if watch and event.type == "row":
+                payload = event.payload
+                click.echo(
+                    f"{run_id} {payload['row_id']} "
+                    f"{'ok' if payload['success'] else 'error'} "
+                    f"score={payload.get('score_value')}",
+                    err=True,
+                )
+
+        try:
+            return await execute_run(store, run_id, on_event=on_event)
+        except Exception as exc:  # noqa: BLE001 — isolate failures between evaluator runs
+            return store.update_run_status(
+                run_id,
+                RunStatus.FAILED,
+                error=str(exc),
+                finished_at=datetime.now(UTC),
+            )
+
+    async def workflow() -> list[Run]:
+        finished = await drive(parent.id)
+        children = await asyncio.to_thread(prepare_evaluators, store, parent.id)
+        completed = await asyncio.gather(
+            *[
+                drive(child.id)
+                for child in children
+                if child.status is RunStatus.PENDING and not child.cancel_requested
+            ]
+        )
+        by_id = {child.id: child for child in completed}
+        return [finished, *[by_id.get(child.id, child) for child in children]]
+
+    finished = asyncio.run(workflow())
+    link = store.get_run_derivation(parent.id)
+    derivation = {
+        "id": link.derivation_id,
+        "state": store.derivation_state(link.derivation_id).value,
+    }
+
+    def summary(run: Run) -> dict:
+        return {
+            "run_id": run.id,
+            "version_id": run.version_id,
+            "kind": run.kind.value,
+            "status": run.status.value,
+            "error": run.error,
+            "metrics": run.metrics,
+            "results": [result.model_dump(mode="json") for result in store.list_results(run.id)],
+        }
+
+    if as_json:
+        emit(
+            {
+                "agent_run": summary(finished[0]),
+                "evaluator_runs": [summary(run) for run in finished[1:]],
+                "derivation": derivation,
+            },
+            True,
+        )
+    else:
+        click.echo(f"derivation {derivation['id']} ({derivation['state']})")
+        for run in finished:
+            _emit_run(store, run, False)
+            if run.error:
+                click.echo(run.error, err=True)
+    if any(run.status is not RunStatus.COMPLETED for run in finished):
+        ctx.exit(1)
+
+
 @run_group.command("evaluator")
 @click.argument("evaluator")
 @click.option("--dataset", required=True, help="Dataset to score.")
@@ -1243,6 +1377,13 @@ async def _drive_experiment(store: Store, run_id: str) -> Run:
 @click.argument("evaluator")
 @click.option("--dataset", required=True, help="Dataset to score.")
 @click.option("--version", "version_name", default=None, help="Version name (default: active).")
+@click.option(
+    "--kind",
+    type=click.Choice(["validation", "eval"]),
+    default="validation",
+    help="Validate against labels or score rows (default: validation).",
+)
+@click.option("--derivation", "derivation_ref", default=None, help="Saved responses to score.")
 @click.option("--concurrency", type=int, default=None, help="Max concurrent rows.")
 @click.option("--json", "as_json", is_flag=True, help="Emit JSON results to stdout.")
 @click.pass_context
@@ -1253,6 +1394,8 @@ def experiment_cmd(
     version_name: str | None,
     concurrency: int | None,
     as_json: bool,
+    kind: str,
+    derivation_ref: str | None,
 ) -> None:
     """Run an evaluator version over a dataset via ``pydantic_evals.Dataset.evaluate``.
 
@@ -1271,8 +1414,23 @@ def experiment_cmd(
         config_module.require_gateway_key()
     ds = resolve_dataset(store, dataset)
 
+    derivation = (
+        resolve_derivation(store, derivation_ref, dataset_id=ds.id)
+        if derivation_ref is not None
+        else None
+    )
+    check_dataset_compatibility(
+        ver,
+        store.get_dataset(ds.id),
+        store.list_label_sets(ds.id),
+        kind=RunKind(kind),
+        derivation=derivation,
+    )
+
     workers = concurrency if concurrency is not None else get_settings().default_concurrency
-    created = store.create_run(RunKind.VALIDATION, ver.id, ds.id, workers)
+    created = store.create_run(RunKind(kind), ver.id, ds.id, workers)
+    if derivation is not None:
+        store.link_run_derivation(created.id, derivation.id, DerivationRole.READS)
 
     finished = asyncio.run(_drive_experiment(store, created.id))
     if finished.status is RunStatus.FAILED:

{valcore-0.2.6 → valcore-0.2.7}/src/valcore/models.py

@@ -546,6 +546,14 @@ class DerivationStatus(SQLModel, table=True):
     state: DerivationState
 
 
+class RunDependency(SQLModel, table=True):
+    """A queued evaluator run waits for this agent run to produce its input."""
+
+    id: str = Field(default_factory=lambda: uuid4().hex, primary_key=True)
+    parent_run_id: str = Field(index=True)
+    child_run_id: str = Field(unique=True, index=True)
+
+
 class RunDerivation(SQLModel, table=True):
     """Links a run to the derivation it fills or reads.
 

valcore-0.2.7/src/valcore/pipeline.py

@@ -0,0 +1,83 @@
+"""Shared workflow transitions for the CLI and API; neither transport owns the contract."""
+
+from datetime import UTC, datetime
+
+from valcore import config
+from valcore.errors import ContractError
+from valcore.models import (
+    DatasetDerivation,
+    DerivationRole,
+    DerivationState,
+    Run,
+    RunKind,
+    RunStatus,
+    check_agent_dataset_compatibility,
+    check_dataset_compatibility,
+)
+from valcore.settings import is_local_cli_model
+from valcore.store import Store
+
+
+def create_pipeline(
+    store: Store,
+    agent_version_id: str,
+    dataset_id: str,
+    evaluator_version_ids: list[str],
+    concurrency: int,
+) -> Run:
+    """Preflight the entire contract before atomically freezing and queuing its runs."""
+    if not evaluator_version_ids:
+        raise ContractError("Select at least one evaluator.")
+    if len(set(evaluator_version_ids)) != len(evaluator_version_ids):
+        raise ContractError("Select each evaluator version only once.")
+    if not 1 <= concurrency <= 64:
+        raise ContractError("Concurrency must be between 1 and 64.")
+    agent = store.get_agent_version(agent_version_id)
+    dataset = store.get_dataset(dataset_id)
+    check_agent_dataset_compatibility(agent, dataset)
+    contract = DatasetDerivation(
+        dataset_id=dataset.id,
+        agent_version_id=agent.id,
+        response_columns=agent.response_columns,
+    )
+    if not is_local_cli_model(agent.model):
+        config.require_gateway_key()
+    for id in evaluator_version_ids:
+        evaluator = store.get_version(id)
+        if not is_local_cli_model(evaluator.model):
+            config.require_gateway_key()
+        check_dataset_compatibility(evaluator, dataset, [], kind=RunKind.EVAL, derivation=contract)
+    return store.create_agent_pipeline(agent.id, dataset.id, evaluator_version_ids, concurrency)
+
+
+def prepare_evaluators(store: Store, parent_id: str) -> list[Run]:
+    """Attach completed responses, or terminate queued runs when their input cannot be produced.
+
+    The caller owns execution and event delivery. This transition is shared so command-line
+    and web workflows make identical decisions about failed rows and upstream failures.
+    """
+    parent = store.get_run(parent_id)
+    _, child_ids = store.run_dependencies(parent_id)
+    try:
+        if parent.status not in (RunStatus.COMPLETED, RunStatus.COMPLETED_WITH_ERRORS):
+            raise ContractError(f"Agent run {parent.status.value}; evaluation was not started.")
+        link = store.get_run_derivation(parent_id)
+        if link is None:
+            raise ContractError("Agent run has no response derivation.")
+        if store.derivation_state(link.derivation_id) is DerivationState.STAGED:
+            store.save_staged_derivation(link.derivation_id)
+        for child_id in child_ids:
+            child = store.get_run(child_id)
+            if child.cancel_requested or child.status is not RunStatus.PENDING:
+                continue
+            store.link_run_derivation(child_id, link.derivation_id, DerivationRole.READS)
+    except Exception as exc:  # noqa: BLE001 — no queued evaluation may be stranded
+        for child_id in child_ids:
+            child = store.get_run(child_id)
+            if child.status is not RunStatus.PENDING:
+                continue
+            status = (
+                RunStatus.CANCELLED if parent.status is RunStatus.CANCELLED else RunStatus.FAILED
+            )
+            store.update_run_status(child_id, status, error=str(exc), finished_at=datetime.now(UTC))
+    return [store.get_run(id) for id in child_ids]

{valcore-0.2.6 → valcore-0.2.7}/src/valcore/skills/use-valcore/reference.md

@@ -33,6 +33,7 @@ command here first, then regenerate with `scripts/sync_command_table.py`.
 | `valcore serve` | Serve the web UI and API (`--port`, `--host`, `--no-browser`). |
 | `valcore list <evaluators\|datasets\|runs\|agents\|derivations>` | List resources as a table or, with `--json`, as JSON. |
 | `valcore run agent <agent>` | Run an agent over a dataset, one row, ad-hoc inputs, or a literal prompt. |
+| `valcore run pipeline <agent> --dataset <dataset> --evaluator <evaluator>` | Run an agent, save responses, and run one or more evaluators concurrently. |
 | `valcore run evaluator <evaluator> --dataset <dataset>` | Run an evaluator version over a dataset. |
 | `valcore run experiment <evaluator> --dataset <dataset>` | Run an evaluator through `pydantic_evals.Dataset.evaluate`. |
 | `valcore export <evaluator>` | Export an evaluator (and, with `--dataset`, a dataset) as a Python script or, with `--format json`, a portable eval package. |
@@ -146,7 +147,9 @@ has no `--watch` and no cancellation, because `Dataset.evaluate` offers neither.
 | `--concurrency INTEGER` | Max concurrent rows. |
 | `--json` | Emit JSON results to stdout. |
 
-Always runs as `validation` kind; there is no `--kind` here.
+Also accepts `--kind [validation|eval]` (default: `validation`) and
+`--derivation REF` to score saved agent responses with `--kind eval`.
+Validation cannot target a derivation, matching ordinary evaluator runs.
 
 ### `valcore run agent AGENT`
 
@@ -159,6 +162,52 @@ An agent may have no input fields or prompt template. In that case ad-hoc `--inp
 is sent as plain text, and dataset rows are sent as JSON after the agent's instructions.
 Running without any inputs sends an empty request. Dependency mappings remain optional.
 
+### `valcore run pipeline AGENT --dataset DATASET --evaluator EVALUATOR`
+
+The CLI equivalent of **Runs → Agent then evaluators**. Runs the selected agent over
+the dataset, automatically saves its response derivation, then starts all selected
+evaluators concurrently against the original columns plus the new response columns.
+The CLI works directly with the local database; no running API server is required.
+
+| Option | Meaning |
+|---|---|
+| `--dataset DATASET` | Required dataset name or unique id prefix. |
+| `--version TEXT` | Agent version name; defaults to active. |
+| `--evaluator EVALUATOR` | Required, repeatable evaluator name or unique id prefix. |
+| `--evaluator-version EVALUATOR VERSION` | Repeatable version override for a selected evaluator; otherwise active. |
+| `--concurrency INTEGER` | Concurrent rows per run, 1–64; defaults to configured concurrency. |
+| `--json` | One JSON summary on stdout, including agent/evaluator run ids, versions, statuses, errors, metrics, evaluator results, and derivation id/state. |
+| `--watch` | Per-row progress with run ids on stderr; safe alongside `--json`. |
+
+```bash
+valcore run pipeline support-agent --dataset support-cases \
+  --version v2 \
+  --evaluator accuracy --evaluator safety \
+  --evaluator-version accuracy v3 \
+  --concurrency 4 --watch --json
+```
+
+Every selection and required column is checked before runs are created. Versions freeze
+when queued. Gateway credentials are required only for versions using a gateway model;
+an entirely local workflow needs no gateway key.
+
+The process stays in the foreground until the agent and evaluators finish. It does not
+detach, but evaluator runs execute asynchronously with each other. Results and workflow
+relationships persist in the local database and appear in the UI's Runs view.
+`valcore list runs --json` includes `parent_run_id`, `evaluator_run_ids`, and
+`derivation_id`, so an agent can discover the same relationships later.
+
+Failed agent rows are skipped by evaluators; a failed or cancelled agent prevents its
+queued evaluations from starting. An evaluator failure does not stop its siblings.
+Exit code is 0 only if every run completes without errors; otherwise 1. With `--json`,
+execution failures still emit the complete summary before exiting 1. Invalid selections
+fail before execution with an error on stderr.
+
+Pipelines use ordinary `eval` runs. To validate human labels use `run evaluator
+--kind validation`; to record a Logfire experiment use `run experiment`. You can also
+run the steps separately with `run agent --dataset DATASET --save` and `run evaluator
+--kind eval --derivation REF`.
+
 ### `valcore agent derivation {list|save|discard}`
 
 `list` includes staged entries and marks their state. Saved derivations can also be listed with