runapprentice 0.3.0__tar.gz → 0.4.1__tar.gz

{runapprentice-0.3.0 → runapprentice-0.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: runapprentice
-Version: 0.3.0
+Version: 0.4.1
 Summary: Cut LLM cost without losing quality: capture traffic, build golden datasets, and optimize prompts.
 Project-URL: Homepage, https://runapprentice.com
 Project-URL: Documentation, https://docs.runapprentice.com
@@ -31,7 +31,7 @@ Description-Content-Type: text/markdown
 
 The Python SDK for [Apprentice](https://runapprentice.com), a tool that cuts frontier-LLM cost
 on repeatable tasks. You give it a dataset of inputs and correct outputs for one task (uploaded
-as a CSV, or captured live from a LangChain app with one callback). Apprentice runs prompt
+as a CSV, or captured live from an OpenAI or LangChain app). Apprentice runs prompt
 optimization (DSPy GEPA) against the verified rows (gold plus silver), or fine-tunes a small open model on
 your Mac, and reports the score change on a held-out slice, so the gain is measured, never
 claimed. Gated, eval-verified takeover of production traffic is still in development.
@@ -89,7 +89,54 @@ print(report.baseline_score, "->", report.optimized_score)
 
 # 4. Pull the optimized prompt back into your code (versioned)
 best = client.prompts.get("ticket-triage")
-print(best.version, best.text)
+resp = OpenAI().chat.completions.create(   # from openai import OpenAI
+    model="gpt-5-mini",
+    messages=best.messages(input=ticket_text),
+)
+```
+
+`datasets.upload()` replaces prior rows for this task where the source is an upload and
+the tier is silver or raw. Gold rows and SDK-captured traces stay intact. Omitting `prompt`
+or passing `None` clears the task's stored baseline prompt. Check
+`DatasetStatus.replaced_rows` to see how many rows were removed.
+
+`best.text` is instruction text, not a template. It holds no input placeholder and often
+contains literal JSON braces, so `best.text.format(input=...)` raises `KeyError` and
+`best.text.replace("{input}", ...)` drops the ticket with no error at all. `messages()`
+renders the message shape the backend recorded when it scored this version. Match the rest
+of the request too if you want the reported score to describe your call: JSON metrics were
+scored with `response_format={"type": "json_object"}` (on the Responses API, that is
+`text={"format": {"type": "json_object"}}`), on the model in
+`report.detail["student_model"]`. Versions optimized before 0.4.0 carry no recorded shape
+and fall back to the task's current baseline prompt. The keyword names are the artifact's `input_variables`:
+`input` for a plain task, `question` and `context` for RAG, your own names if you
+registered a template.
+
+## Live capture: OpenAI (one wrapper)
+
+```python
+from openai import OpenAI
+from runapprentice.openai import wrap
+
+openai = wrap(OpenAI(), client, task="ticket-triage")
+
+response = openai.responses.create(
+    model="gpt-5-mini",
+    input="Triage this ticket.",
+)
+# create() and parse() are captured, on chat.completions and responses alike
+# (parse posts on its own, so patching create alone would miss it).
+```
+
+Both sync `OpenAI` and async `AsyncOpenAI` clients work. Capture is fail-open and
+returns every OpenAI result unchanged. Calls with `stream=True` and calls through
+`responses.stream(...)` pass through uncaptured because reading them would consume
+the caller's stream.
+
+**PII redaction** runs in your process, before anything is transmitted:
+
+```python
+openai = wrap(OpenAI(), client, task="ticket-triage", redact=lambda s: my_scrubber(s))
 ```
 
 ## Live capture: LangChain (one line, zero code changes)
@@ -226,7 +273,7 @@ count and the minimum).
 |---|---|---|
 | `Apprentice(api_key=, base_url=, timeout=)` | client | `ApprenticeError` on bad config |
 | `client.tasks.create(name, metric="auto")` | dict (`created` bool) | `ApprenticeError` on HTTP failure |
-| `client.datasets.upload(task, path= or rows=, input_col=, output_col=, prompt=)` | `DatasetStatus` | same; also if both/neither of path/rows |
+| `client.datasets.upload(task, path= or rows=, input_col=, output_col=, prompt=)` | `DatasetStatus` including `replaced_rows` | same; also if both/neither of path/rows |
 | `client.datasets.status(task)` | `DatasetStatus(gold, silver, raw, ready_for_optimization)` | same |
 | `client.prompts.register(task, template)` | dict | `ImportError` if a LangChain template is passed without the `[langchain]` extra |
 | `client.prompts.to_langchain(task= or artifact=)` | LangChain prompt | `ImportError` without the `[langchain]` extra |
@@ -247,10 +294,12 @@ count and the minimum).
 | `client.flush_captures(timeout=2.0)` | None (blocks until buffered traces deliver or timeout; no-op in sync mode) | nothing by design |
 | `client.close()` | None (flushes and closes transport) | nothing by design |
 | `job.report()` | typed optimize, train, or generate report; raw dict for future shapes | if job has no report |
-| `client.prompts.get(task, version=None)` | `PromptVersion(version, text, score)` | 404 if never optimized |
+| `client.prompts.get(task, version=None)` | `PromptVersion(version, text, score, baseline_prompt)` | 404 if never optimized |
+| `PromptVersion.messages(user_prompt=None, **inputs)` | `list[dict]` reproducing the scored call | `ValueError` if a keyword is not one of the artifact's `input_variables` |
 | `client.prompts.history(task)` | `list[PromptVersion]` | `ApprenticeError` on API failure |
 | `client.feedback(trace_id, good=, score=, note=)` | None | 404 unknown trace; `ApprenticeError` if pending captures cannot drain in time, or on API failure |
 | `client.post_trace_failopen(record)` | trace_id or None | **never raises** |
+| `wrap(openai_client, client, task, redact=None)` | the same OpenAI client, with chat completions and responses captured | **never raises from capture; OpenAI call errors pass through unchanged** |
 | `ApprenticeCallback(task, client, redact=None)` | LangChain callback | **never raises into your chain** |
 
 ## Data tiers (how your rows are treated)

{runapprentice-0.3.0 → runapprentice-0.4.1}/README.md

@@ -2,7 +2,7 @@
 
 The Python SDK for [Apprentice](https://runapprentice.com), a tool that cuts frontier-LLM cost
 on repeatable tasks. You give it a dataset of inputs and correct outputs for one task (uploaded
-as a CSV, or captured live from a LangChain app with one callback). Apprentice runs prompt
+as a CSV, or captured live from an OpenAI or LangChain app). Apprentice runs prompt
 optimization (DSPy GEPA) against the verified rows (gold plus silver), or fine-tunes a small open model on
 your Mac, and reports the score change on a held-out slice, so the gain is measured, never
 claimed. Gated, eval-verified takeover of production traffic is still in development.
@@ -60,7 +60,54 @@ print(report.baseline_score, "->", report.optimized_score)
 
 # 4. Pull the optimized prompt back into your code (versioned)
 best = client.prompts.get("ticket-triage")
-print(best.version, best.text)
+resp = OpenAI().chat.completions.create(   # from openai import OpenAI
+    model="gpt-5-mini",
+    messages=best.messages(input=ticket_text),
+)
+```
+
+`datasets.upload()` replaces prior rows for this task where the source is an upload and
+the tier is silver or raw. Gold rows and SDK-captured traces stay intact. Omitting `prompt`
+or passing `None` clears the task's stored baseline prompt. Check
+`DatasetStatus.replaced_rows` to see how many rows were removed.
+
+`best.text` is instruction text, not a template. It holds no input placeholder and often
+contains literal JSON braces, so `best.text.format(input=...)` raises `KeyError` and
+`best.text.replace("{input}", ...)` drops the ticket with no error at all. `messages()`
+renders the message shape the backend recorded when it scored this version. Match the rest
+of the request too if you want the reported score to describe your call: JSON metrics were
+scored with `response_format={"type": "json_object"}` (on the Responses API, that is
+`text={"format": {"type": "json_object"}}`), on the model in
+`report.detail["student_model"]`. Versions optimized before 0.4.0 carry no recorded shape
+and fall back to the task's current baseline prompt. The keyword names are the artifact's `input_variables`:
+`input` for a plain task, `question` and `context` for RAG, your own names if you
+registered a template.
+
+## Live capture: OpenAI (one wrapper)
+
+```python
+from openai import OpenAI
+from runapprentice.openai import wrap
+
+openai = wrap(OpenAI(), client, task="ticket-triage")
+
+response = openai.responses.create(
+    model="gpt-5-mini",
+    input="Triage this ticket.",
+)
+# create() and parse() are captured, on chat.completions and responses alike
+# (parse posts on its own, so patching create alone would miss it).
+```
+
+Both sync `OpenAI` and async `AsyncOpenAI` clients work. Capture is fail-open and
+returns every OpenAI result unchanged. Calls with `stream=True` and calls through
+`responses.stream(...)` pass through uncaptured because reading them would consume
+the caller's stream.
+
+**PII redaction** runs in your process, before anything is transmitted:
+
+```python
+openai = wrap(OpenAI(), client, task="ticket-triage", redact=lambda s: my_scrubber(s))
 ```
 
 ## Live capture: LangChain (one line, zero code changes)
@@ -197,7 +244,7 @@ count and the minimum).
 |---|---|---|
 | `Apprentice(api_key=, base_url=, timeout=)` | client | `ApprenticeError` on bad config |
 | `client.tasks.create(name, metric="auto")` | dict (`created` bool) | `ApprenticeError` on HTTP failure |
-| `client.datasets.upload(task, path= or rows=, input_col=, output_col=, prompt=)` | `DatasetStatus` | same; also if both/neither of path/rows |
+| `client.datasets.upload(task, path= or rows=, input_col=, output_col=, prompt=)` | `DatasetStatus` including `replaced_rows` | same; also if both/neither of path/rows |
 | `client.datasets.status(task)` | `DatasetStatus(gold, silver, raw, ready_for_optimization)` | same |
 | `client.prompts.register(task, template)` | dict | `ImportError` if a LangChain template is passed without the `[langchain]` extra |
 | `client.prompts.to_langchain(task= or artifact=)` | LangChain prompt | `ImportError` without the `[langchain]` extra |
@@ -218,10 +265,12 @@ count and the minimum).
 | `client.flush_captures(timeout=2.0)` | None (blocks until buffered traces deliver or timeout; no-op in sync mode) | nothing by design |
 | `client.close()` | None (flushes and closes transport) | nothing by design |
 | `job.report()` | typed optimize, train, or generate report; raw dict for future shapes | if job has no report |
-| `client.prompts.get(task, version=None)` | `PromptVersion(version, text, score)` | 404 if never optimized |
+| `client.prompts.get(task, version=None)` | `PromptVersion(version, text, score, baseline_prompt)` | 404 if never optimized |
+| `PromptVersion.messages(user_prompt=None, **inputs)` | `list[dict]` reproducing the scored call | `ValueError` if a keyword is not one of the artifact's `input_variables` |
 | `client.prompts.history(task)` | `list[PromptVersion]` | `ApprenticeError` on API failure |
 | `client.feedback(trace_id, good=, score=, note=)` | None | 404 unknown trace; `ApprenticeError` if pending captures cannot drain in time, or on API failure |
 | `client.post_trace_failopen(record)` | trace_id or None | **never raises** |
+| `wrap(openai_client, client, task, redact=None)` | the same OpenAI client, with chat completions and responses captured | **never raises from capture; OpenAI call errors pass through unchanged** |
 | `ApprenticeCallback(task, client, redact=None)` | LangChain callback | **never raises into your chain** |
 
 ## Data tiers (how your rows are treated)

{runapprentice-0.3.0 → runapprentice-0.4.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "runapprentice"
-version = "0.3.0"
+version = "0.4.1"
 description = "Cut LLM cost without losing quality: capture traffic, build golden datasets, and optimize prompts."
 readme = "README.md"
 requires-python = ">=3.11"

{runapprentice-0.3.0 → runapprentice-0.4.1}/src/runapprentice/cli.py

@@ -243,6 +243,14 @@ def _optimize(args: argparse.Namespace, client: Apprentice) -> int:
         metric=args.metric,
     ).wait()
     payload = job.info.model_dump(mode="json")
+    if job.info.status != "succeeded":
+        # Exit nonzero, or a caller that checks the exit code (every script and
+        # every agent) reads a failed run as a shipped one.
+        return _fail(
+            f"optimize {job.info.status}: {job.info.error or 'no error reported'}",
+            bool(args.json),
+            payload=payload,
+        )
     return _emit(payload, _report_line(job.info.report), bool(args.json))
 
 
@@ -505,9 +513,11 @@ def _emit(payload: dict[str, Any], text: str, as_json: bool) -> int:
     return 0
 
 
-def _fail(message: str, as_json: bool) -> int:
+def _fail(message: str, as_json: bool, payload: dict[str, Any] | None = None) -> int:
     if as_json:
-        print(json.dumps({"error": message}, sort_keys=True), file=sys.stderr)
+        # Keep the job fields alongside the error so a caller parsing --json still
+        # gets the job id and status it needs to investigate.
+        print(json.dumps({**(payload or {}), "error": message}, sort_keys=True), file=sys.stderr)
     else:
         print(message, file=sys.stderr)
     return 1

{runapprentice-0.3.0 → runapprentice-0.4.1}/src/runapprentice/client.py

@@ -16,6 +16,7 @@ import os
 import time
 import uuid
 from pathlib import Path
+from urllib.parse import quote
 from typing import Any, Literal
 
 import httpx
@@ -164,18 +165,18 @@ class Apprentice:
         return Job(self, JobInfo.model_validate(data))
 
     def jobs(self, task: str) -> list[JobSummary]:
-        data = self._request("GET", f"/v1/tasks/{task}/jobs")
+        data = self._request("GET", f"/v1/tasks/{quote(task, safe='')}/jobs")
         return [JobSummary.model_validate(item) for item in data]
 
     def cancel_job(self, job_id: str) -> dict[str, Any]:
         return dict(self._request("POST", f"/v1/jobs/{job_id}/cancel"))
 
     def local_runs(self, task: str) -> list[LocalRun]:
-        data = self._request("GET", f"/v1/tasks/{task}/local-runs")
+        data = self._request("GET", f"/v1/tasks/{quote(task, safe='')}/local-runs")
         return [LocalRun.model_validate(item) for item in data]
 
     def train(self, task: str) -> Job:
-        data = self._request("POST", f"/v1/tasks/{task}/train", json={})
+        data = self._request("POST", f"/v1/tasks/{quote(task, safe='')}/train", json={})
         return Job(self, JobInfo.model_validate(data))
 
     def optimize_local(
@@ -422,10 +423,15 @@ class _DatasetsAPI:
         output_col: str = "output",
         prompt: str | None = None,
     ) -> DatasetStatus:
-        """Upload rows.
+        """Replace uploaded, unverified rows for a task.
 
         Rows may use legacy `input`/`output` or structured `inputs`/`output`.
         LCEL auto-capture of prompt variables is future work; v1 stays explicit.
+
+        This is a replace, not an append: existing uploaded silver and raw rows are
+        removed first, and `replaced_rows` in the response reports how many. Gold
+        rows and captured traces survive. Omitting `prompt` (or passing `None`)
+        clears the task's stored baseline prompt.
         """
         if (path is None) == (rows is None):
             raise ApprenticeError("provide exactly one of path= or rows=")
@@ -436,13 +442,13 @@ class _DatasetsAPI:
         payload_rows = [_dataset_row_payload(r, input_col, output_col) for r in rows]
         data = self._client._request(
             "POST",
-            f"/v1/datasets/{task}/rows",
+            f"/v1/datasets/{quote(task, safe='')}/rows",
             json={"rows": payload_rows, "prompt": prompt},
         )
         return DatasetStatus.model_validate(data)
 
     def status(self, task: str) -> DatasetStatus:
-        data = self._client._request("GET", f"/v1/datasets/{task}/status")
+        data = self._client._request("GET", f"/v1/datasets/{quote(task, safe='')}/status")
         return DatasetStatus.model_validate(data)
 
     def rows(self, task: str, tier: str, limit: int = 200) -> list[DatasetRow]:
@@ -451,7 +457,7 @@ class _DatasetsAPI:
         while True:
             data = self._client._request(
                 "GET",
-                f"/v1/datasets/{task}/rows?tier={tier}&limit={limit}&offset={offset}",
+                f"/v1/datasets/{quote(task, safe='')}/rows?tier={tier}&limit={limit}&offset={offset}",
             )
             batch = [DatasetRow.model_validate(item) for item in data.get("rows", [])]
             rows.extend(batch)
@@ -476,21 +482,21 @@ class _PromptsAPI:
 
     def register(self, task: str, template: Any) -> dict[str, Any]:
         artifact = template if isinstance(template, dict) else _serialize_langchain_prompt(template)
-        return dict(self._client._request("POST", f"/v1/templates/{task}", json=artifact))
+        return dict(self._client._request("POST", f"/v1/templates/{quote(task, safe='')}", json=artifact))
 
     def to_langchain(self, task: str | None = None, artifact: dict[str, Any] | None = None) -> Any:
         if artifact is None:
             if task is None:
                 raise ApprenticeError("provide task= or artifact=")
-            artifact = dict(self._client._request("GET", f"/v1/templates/{task}"))
+            artifact = dict(self._client._request("GET", f"/v1/templates/{quote(task, safe='')}"))
         return _artifact_to_langchain(artifact)
 
     def get(self, task: str, version: int | None = None) -> PromptVersion:
-        path = f"/v1/prompts/{task}" + (f"?version={version}" if version is not None else "")
+        path = f"/v1/prompts/{quote(task, safe='')}" + (f"?version={version}" if version is not None else "")
         return PromptVersion.model_validate(self._client._request("GET", path))
 
     def history(self, task: str) -> list[PromptVersion]:
-        data = self._client._request("GET", f"/v1/prompts/{task}/history")
+        data = self._client._request("GET", f"/v1/prompts/{quote(task, safe='')}/history")
         return [PromptVersion.model_validate(p) for p in data]
 
 

runapprentice-0.4.1/src/runapprentice/models.py

@@ -0,0 +1,327 @@
+"""Pydantic models for the Apprentice SDK. Mirrors the control-plane API contract."""
+
+from __future__ import annotations
+
+from datetime import datetime
+import re
+from typing import Any
+
+from pydantic import BaseModel, Field, SkipValidation, ValidationError, model_validator
+
+
+class DatasetRow(BaseModel):
+    id: str | None = None
+    input: str
+    inputs: dict[str, str] | None = None
+    output: str
+    tier: str | None = None
+    source: str | None = None
+    flag_label: str | None = None
+    feedback_score: float | None = None
+    expert_authored: bool | None = None
+    created_at: datetime | None = None
+    metadata: dict[str, Any] = Field(default_factory=dict)
+
+
+class DatasetStatus(BaseModel):
+    task: str
+    gold: int = 0
+    silver: int = 0
+    raw: int = 0
+    ready_for_optimization: bool = False
+    replaced_rows: int = 0
+
+
+class TraceRecord(BaseModel):
+    trace_id: str | None = Field(default=None, pattern=r"^[0-9a-f]{32}$")
+    task: str
+    input: str
+    inputs: dict[str, str] | None = None
+    output: str
+    model: str | None = None
+    latency_ms: float | None = None
+    prompt_tokens: int | None = None
+    completion_tokens: int | None = None
+    error: str | None = None
+    metadata: dict[str, Any] = Field(default_factory=dict)
+
+
+class HoldoutEvidenceRow(BaseModel):
+    row_id: str | None = None
+    tier: str | None = None
+    bucket: str | None = None
+    input: str | None = None
+    expected: str | None = None
+    baseline_output: str | None = None
+    baseline_score: float | None = None
+    optimized_output: str | None = None
+    optimized_score: float | None = None
+    # Set when the row's 0 came from an evaluation error, not a wrong answer.
+    baseline_error: str | None = None
+    optimized_error: str | None = None
+
+
+class OptimizationReport(BaseModel):
+    task: str | None = None
+    baseline_score: float | None = None
+    optimized_score: float | None = None
+    examples_used: int | None = None
+    baseline_prompt: str | None = None
+    baseline_from_user: bool | None = None
+    candidates_explored: int | None = None
+    duplicates_dropped: int | None = None
+    dataset_fingerprint: str | None = None
+    dataset_size: int | None = None
+    optimized_prompt: str | None = None
+    executable_prompt: dict[str, Any] | None = None
+    optimized_template: dict[str, Any] | None = None
+    template_safe: bool | None = None
+    template_placeholder_changes: dict[str, list[str]] | None = None
+    eval_metric: str | None = None
+    metric_tier: str | None = None
+    metric_version: str | None = None
+    judge_estimate: bool | None = None
+    judge_model: str | None = None
+    judge_cache: dict[str, int] | None = None
+    score_explanation: str | None = None
+    worst_examples: list[dict[str, Any]] | None = None
+    failure_count: int | None = None
+    holdout_evidence: list[HoldoutEvidenceRow] | None = None
+    improved_count: int | None = None
+    regressed_count: int | None = None
+    unchanged_count: int | None = None
+    duration_seconds: float | None = None
+    cost_usd: float | None = None
+    input_tokens: int | None = None
+    output_tokens: int | None = None
+    cached_input_tokens: int | None = None
+    reasoning_tokens: int | None = None
+    cache_replayed_calls: int | None = None
+    llm_student_model: str | None = None
+    llm_reflection_model: str | None = None
+    llm_judge_model: str | None = None
+    llm_fallback_used: bool | None = None
+    detail: dict[str, Any] = Field(default_factory=dict)
+
+
+class TrainingReport(BaseModel):
+    task: str | None = None
+    model_id: str | None = None
+    base_model: str | None = None
+    adapter_ref: str | None = None
+    champion_score: float | None = None
+    challenger_score: float | None = None
+    eval_metric: str | None = None
+    metric_tier: str | None = None
+    holdout_size: int | None = None
+    status: str | None = None
+    reason: str | None = None
+    config: dict[str, Any] | None = None
+
+
+class GenerationReport(BaseModel):
+    rows_inserted: int | None = None
+    rows_rejected: int | None = None
+    reject_reasons: dict[str, int] | None = None
+    model: str | None = None
+    llm_fallback_used: bool | None = None
+    input_tokens: int | None = None
+    output_tokens: int | None = None
+    samples: list[dict[str, Any]] | None = None
+    chunks_truncated: int | None = None
+    response_failures: int | None = None
+    duplicate_inputs: int | None = None
+    executor_requested: str | None = None
+    executor_actual_models: list[str] | None = None
+    executor_fallback_used: bool | None = None
+
+
+JobReport = OptimizationReport | TrainingReport | GenerationReport | dict[str, Any]
+
+
+class JobInfo(BaseModel):
+    job_id: str
+    task: str  # task ID; use task_name when calling task-scoped client methods
+    task_name: str | None = None
+    kind: str
+    status: str
+    created_at: datetime | None = None
+    error: str | None = None
+    report: SkipValidation[JobReport] | None = None
+    progress: dict[str, Any] | None = None
+    queue_position: int | None = None
+    baseline_score: float | None = None
+    optimized_score: float | None = None
+    baseline_from_user: bool | None = None
+    generation_session_id: str | None = None
+    round: int | None = None
+    predecessor_job_id: str | None = None
+    phase: str | None = None
+    guidance: str | None = None
+
+    @model_validator(mode="before")
+    @classmethod
+    def parse_report_for_kind(cls, value: Any) -> Any:
+        if not isinstance(value, dict) or not isinstance(value.get("report"), dict):
+            return value
+        kind = value.get("kind")
+        if kind == "optimize":
+            report_type: type[BaseModel] = OptimizationReport
+        elif kind == "train":
+            report_type = TrainingReport
+        elif kind == "generate":
+            report_type = GenerationReport
+        else:
+            return value
+        try:
+            report = report_type.model_validate(value["report"])
+        except ValidationError:
+            return value
+        return {**value, "report": report}
+
+
+class JobSummary(BaseModel):
+    job_id: str
+    task_name: str | None = None
+    status: str
+    kind: str
+    created_at: datetime | None = None
+    baseline_score: float | None = None
+    optimized_score: float | None = None
+
+
+class LocalRun(BaseModel):
+    run_id: str
+    task: str | None = None
+    task_id: str | None = None
+    status: str = "registered"
+    eval_tier: str | None = None
+    base_model: str | None = None
+    dataset_row_count: int | None = None
+    seed: int | None = None
+    baseline_score: float | None = None
+    final_score: float | None = None
+    held_out_rows: int | None = None
+    adapter_sha256: str | None = None
+    mlx_lm_version: str | None = None
+    error_message: str | None = None
+    created_at: datetime | None = None
+    completed_at: datetime | None = None
+
+
+class OptimizationPreview(BaseModel):
+    eval_metric: str
+    metric_tier: str
+    requested_metric: str
+    recommended_metric: str
+    metric_reason: str
+    rag_candidate: bool
+    rows_checked: int
+    rows_source: str
+    guidance: dict[str, Any]
+    estimate: dict[str, Any]
+
+
+# LangChain roles (as stored in the artifact) -> chat-completions roles.
+_CHAT_ROLE = {"human": "user", "ai": "assistant", "system": "system"}
+_FORMAT_TOKEN = re.compile(r"\{\{|\}\}|\{(?P<name>[A-Za-z_][A-Za-z0-9_]*)\}")
+
+
+def _fill(template: str, values: dict[str, str]) -> str:
+    """Substitute only declared variables, never raise.
+
+    Mirrors the backend's renderer: `{{`/`}}` are literal braces, and any other
+    brace run (a JSON example in the instruction, an unknown name) is left alone.
+    str.format would raise KeyError on the JSON braces these prompts routinely carry.
+    """
+
+    def repl(match: re.Match[str]) -> str:
+        token = match.group(0)
+        if token in ("{{", "}}"):
+            return token[0]
+        name = match.group("name")
+        return str(values[name]) if name in values else token
+
+    return _FORMAT_TOKEN.sub(repl, template)
+
+
+class PromptVersion(BaseModel):
+    task: str
+    version: int
+    text: str
+    score: float | None = None
+    # The message shape this version's score was measured with. Frozen server-side
+    # when the version was stored, so it does not move when the task baseline does.
+    artifact: dict[str, Any] | None = None
+    baseline_prompt: str | None = None
+
+    # `user_prompt` is positional-only: a task is free to have an input variable of
+    # that name, and a named parameter would swallow it and make the prompt unrunnable.
+    def messages(self, user_prompt: str | None = None, /, **inputs: Any) -> list[dict[str, str]]:
+        """Chat messages that reproduce the call this version's score was measured with.
+
+        `text` alone is not runnable: it is instruction text with no input placeholder
+        and often literal JSON braces, so `text.format(...)` raises KeyError and
+        substituting into it drops the input with no error at all. This renders the
+        artifact the backend recorded when it scored the prompt, so the messages you
+        send are the messages that produced `score`.
+
+        Pass the task inputs as keywords. A first positional argument replaces the user
+        message, which changes the call away from the scored one, so `score` no longer
+        describes it. It is positional so that a task may have an input named
+        `user_prompt` without the two colliding.
+        """
+        if self.artifact is not None and user_prompt is None:
+            # A template whose partial_variables supply every variable is complete on
+            # its own, so no keywords is the right call there, not a mistake.
+            return self._render_artifact(inputs)
+        if not inputs:
+            raise ValueError(
+                "pass the task inputs as keyword arguments, e.g. messages(input=text)"
+            )
+        return self._render_user_prompt(user_prompt, inputs)
+
+    def _render_artifact(self, inputs: dict[str, Any]) -> list[dict[str, str]]:
+        artifact = self.artifact or {}
+        declared = list(artifact.get("input_variables") or [])
+        unknown = [name for name in inputs if name not in declared]
+        if unknown:
+            raise ValueError(
+                f"task '{self.task}' takes {declared or 'no inputs'}, not "
+                f"{unknown}. Passing a name the prompt does not use would drop it "
+                "from the call."
+            )
+        values = {**(artifact.get("partial_variables") or {}), **inputs}
+        missing = [name for name in declared if name not in values]
+        if missing:
+            raise ValueError(f"task '{self.task}' also needs {missing} to build the call.")
+        return [
+            {
+                "role": _CHAT_ROLE.get(message["role"], message["role"]),
+                "content": _fill(message["template"], values),
+            }
+            for message in artifact.get("messages") or []
+        ]
+
+    def _render_user_prompt(
+        self, user_prompt: str | None, inputs: dict[str, Any]
+    ) -> list[dict[str, str]]:
+        """Fallback: no recorded artifact (a version stored before they existed), or
+        the caller supplied their own user message."""
+        template = user_prompt if user_prompt is not None else self.baseline_prompt
+        if template is None:
+            raise ValueError(
+                f"task '{self.task}' has no recorded prompt shape to carry the input. "
+                "Re-run optimize, or pass a user prompt positionally, "
+                "e.g. messages('...{input}...', input=text)."
+            )
+        for name in inputs:
+            if "{" + name + "}" not in template:
+                raise ValueError(
+                    f"user prompt has no {{{name}}} placeholder, so '{name}' would be "
+                    "dropped from the call. Add the placeholder or fix the keyword name."
+                )
+        return [
+            {"role": "system", "content": self.text},
+            {"role": "user", "content": _fill(template, inputs)},
+        ]