@orangekame3/pi-qcaleval 0.0.1 → 0.0.2-beta.11

package/README.md

@@ -4,32 +4,85 @@ A [pi](https://github.com/badlogic/pi-mono) extension that sends superconducting
 qubit calibration plots to a dedicated calibration-evaluation VLM and returns a
 structured diagnosis. The default model is
 [NVIDIA Ising-Calibration](https://build.nvidia.com/nvidia/ising-calibration-1.5-31b/modelcard)
-served by vLLM.
+served by vLLM, and the questions are the ones it was trained and benchmarked
+on: the six questions of
+[QCalEval](https://arxiv.org/abs/2604.25884), asked the way the benchmark asks
+them.
 
 The planner model keeps the conversation; `qcal_evaluate` is the tool it calls
 whenever a calibration figure needs a fit-quality judgement, parameter
-extraction, or a success verdict. The extension has no QDash dependency, so it
+extraction, or a status verdict. The extension has no QDash dependency, so it
 works from the pi CLI and from the QDash Copilot runtime alike.
 
 ## Tool
 
-`qcal_evaluate(context, question?, images?, max_images?)` returns a JSON object with
-`technical_description`, `experimental_conclusion`, `experimental_significance`,
-`fit_quality`, `parameter_extraction`, and `success`. The parsed object is also
-stored in the tool result's `details.diagnosis`; when the model did not answer
-with JSON, `details.raw` holds its text instead.
+`qcal_evaluate(context, question?, task_name?, knowledge?, criteria?, images?, max_images?, use_examples?, examples?, questions?)`
+asks the evaluation model six questions, each in its own single-turn request
+(no system prompt), with the figure at the position the benchmark's templates
+put it, and returns the parsed answers as `details.diagnosis`:
+
+| key | the benchmark's question | parsed answer |
+| --- | --- | --- |
+| `technical_description` | Q1 Describe the figure in JSON | `{plot_type, x_axis, y_axis, main_features}`, or the text when not JSON |
+| `experimental_conclusion` | Q2 Classify the outcome | `{classification: expected \| suboptimal \| anomalous \| apparatus, reason}` |
+| `experimental_significance` | Q3 What does this result imply | text |
+| `fit_reliability` | Q4 Is the fit reliable for extraction | `{assessment: reliable \| unreliable \| no_fit, reason}` |
+| `parameter_extraction` | Q5 Extract parameters in JSON | the object, or the text when not JSON |
+| `calibration_diagnosis` | Q6 Determine the status | `{status: CODE, suggested_range, notes}` |
+
+plus `success`: whether the status is `SUCCESS` (or, when the status question
+was not asked, whether the outcome is `expected`). `details.answers` keeps every
+answer as returned, `details.questions` the questions asked, `details.prompt`
+every user turn the model received (with `[figure]` where an image was), and
+`details.calls` each question's call with its duration, stop reason and output
+tokens. A call that fails is retried once, as the benchmark retries API
+failures; nothing is re-asked for its content. Questions that got no answer are
+listed in `details.missing_keys`. `questions` asks a subset.
+
+`knowledge` is the experiment background the benchmark prepends to Q2–Q6: a few
+sentences on what the experiment sweeps and what a successful result looks like.
+`criteria` fills the family-specific parts of the templates: `guidance` (the
+points Q3 should explain), `parameters` (the Q5 schema, name to type),
+`statuses` (the Q6 decision criteria, code to criterion, `SUCCESS` first) and
+`range_format`. A host that keeps such knowledge per task fills both in from
+`task_name`, so the planner only names the task. Without them the templates use
+generic guidance, ask for whatever parameters the plot determines, and offer
+`SUCCESS` / `SUBOPTIMAL` / `NO_SIGNAL` / `ANOMALOUS`.
+
+`context` is what the background cannot know (the task, the target, a swept
+range); the model reads values off the plot, so do not pass fitted results. A
+`question` is added to Q3 as one more point to address.
 
 `images` are base64 image blocks. A host may fill them in from the most recent
 figures of the conversation, so the planner does not have to pass image bytes;
 `max_images` (default 1, up to 4) says how many of the newest figures to supply,
 for example 2 when an expected reference figure precedes the measured result.
+The last figure is the one judged; earlier ones are shown as references.
+
+### Labeled examples (in-context)
+
+With `use_examples`, the host supplies `examples`: each is one or more figures
+with the answers an expert validated (`scenario`, `analysis`, `parameters`,
+`status`, `suggested_range`). They are laid out as the benchmark's few-shot
+split lays them out, examples first and the query figure last, and only for the
+three questions the benchmark uses them for: Q3 shows each example's
+`analysis` under `Scenario "<scenario>":`, Q5 shows one example's
+`parameters`, Q6 shows each example's `Status:` and `Suggested range:`. Up to
+six examples and six example figures are shown; the rest are dropped in order.
 
 ## Configuration
 
-| Variable | Purpose |
-|----------|---------|
-| `QCALEVAL_MODEL` | `provider/modelId` of the evaluation model in pi's `models.json` (default: `vllm/nvidia/Ising-Calibration-1.5-31B-NVFP4`) |
-| `QCALEVAL_MAX_TOKENS` | Output budget for one evaluation (default: `2048`) |
+No setting is required. The evaluation model is resolved in this order:
+
+1. `QCALEVAL_MODEL` (`provider/modelId`), an explicit override for pi CLI use;
+2. a configured calibration VLM, recognised by its id (`Ising-Calibration`);
+3. the conversation's own model, when it accepts images. The result is then
+   marked `fallback: true` in `details`, so a host without an Ising deployment
+   still gets a diagnosis from the vision model it already runs.
+
+The output budget per answer follows the model's `maxTokens` (at most 4096). The
+benchmark decodes greedily; set the model's temperature to 0 for reproducible
+answers.
 
 The provider must be declared in pi's `models.json` with its base URL and
 credential, for example:
@@ -47,8 +100,9 @@ credential, for example:
 }
 ```
 
-Only the plot, a short system prompt, and the `context` text are sent, so the
-evaluation fits comfortably in a 16k-token deployment.
+Each request carries one plot, the background and one question, so an
+evaluation fits comfortably in a 16k-token deployment; the six requests run at
+once.
 
 ## Installation
 

package/extensions/qcaleval.ts

@@ -2,13 +2,15 @@
  * pi-qcaleval: evaluate calibration plots with a dedicated calibration VLM.
  *
  * The extension has no QDash dependency. It takes a plot (plus the experiment
- * context the planner already knows) and asks the evaluation model, which is
- * configured separately from the conversation's model, for a diagnosis. The
- * planner decides what to do with it; whether a calibration loop continues is
- * the host's policy, not this tool's.
+ * background and criteria the host supplies for the task, and the context the
+ * planner already knows) and asks the evaluation model the six QCalEval
+ * questions, each as its own single-turn request the way the benchmark and
+ * the Ising-Calibration training data lay them out. The planner decides what
+ * to do with the diagnosis; whether a calibration loop continues is the
+ * host's policy, not this tool's.
  */
 
-import type { Api, AssistantMessage, ImageContent, Model } from "@earendil-works/pi-ai";
+import type { Api, AssistantMessage, ImageContent, Model, Usage } from "@earendil-works/pi-ai";
 import type {
   AgentToolResult,
   ExtensionAPI,
@@ -16,20 +18,29 @@ import type {
 } from "@earendil-works/pi-coding-agent";
 import { Type, type Static } from "typebox";
 
-/** `provider/modelId`; the model id itself may contain slashes (nvidia/...). */
-export const DEFAULT_MODEL = "vllm/nvidia/Ising-Calibration-1.5-31B-NVFP4";
-
 /** Figures per evaluation; a 16k-token deployment holds a few plots comfortably. */
 export const MAX_IMAGES = 4;
 
+/** Labeled example plots shown before the query, and the figures they may hold in total. */
+export const MAX_EXAMPLES = 6;
+export const MAX_EXAMPLE_IMAGES = 6;
+
+/** Model ids of models fine-tuned for calibration plots; preferred when configured. */
+export const CALIBRATION_MODEL_PATTERN = /ising-calibration/i;
+
+/** Output budget when the model declares none; one answer is a few hundred tokens. */
+const DEFAULT_MAX_TOKENS = 2048;
+const MAX_OUTPUT_TOKENS = 4096;
+
 export interface ModelSpec {
   provider: string;
   modelId: string;
 }
 
-/** Parse `provider/modelId`, splitting at the first slash only. */
-export function parseModelSpec(value: string | undefined): ModelSpec {
-  const spec = (value ?? "").trim() || DEFAULT_MODEL;
+/** Parse an explicit `provider/modelId` override; the model id may contain slashes (nvidia/...). */
+export function parseModelSpec(value: string | undefined): ModelSpec | undefined {
+  const spec = (value ?? "").trim();
+  if (!spec) return undefined;
   const slash = spec.indexOf("/");
   if (slash <= 0 || slash === spec.length - 1) {
     throw new Error(`QCALEVAL_MODEL must be provider/modelId, got "${spec}"`);
@@ -37,49 +48,383 @@ export function parseModelSpec(value: string | undefined): ModelSpec {
   return { provider: spec.slice(0, slash), modelId: spec.slice(slash + 1) };
 }
 
-/** Categories the Ising-Calibration model card describes; asked for in this order. */
-export const DIAGNOSIS_SECTIONS = [
+/** The slice of pi's model registry the resolution needs. */
+export interface ModelLookup {
+  find(provider: string, modelId: string): Model<Api> | undefined;
+  getAvailable(): Model<Api>[];
+}
+
+export type EvaluationModel =
+  | { model: Model<Api>; fallback: false }
+  | { model: Model<Api>; fallback: true }
+  | { model?: undefined; error: string };
+
+/**
+ * Pick the model that judges the plot, without any host-specific setting:
+ *
+ * 1. `QCALEVAL_MODEL` (`provider/modelId`), an explicit override for the pi CLI;
+ * 2. a configured calibration VLM, recognised by its id (Ising-Calibration);
+ * 3. the conversation's own model, when it accepts images.
+ *
+ * A host that declares the calibration model in its models.json gets it
+ * everywhere; one that does not (no Ising deployment) still gets a diagnosis
+ * from the vision model it already runs, marked as a fallback.
+ */
+export function resolveEvaluationModel(
+  registry: ModelLookup,
+  current: Model<Api> | undefined,
+  override: string | undefined,
+): EvaluationModel {
+  const spec = parseModelSpec(override);
+  if (spec) {
+    const model = registry.find(spec.provider, spec.modelId);
+    return model
+      ? { model, fallback: false }
+      : { error: `model ${spec.provider}/${spec.modelId} is not configured (QCALEVAL_MODEL)` };
+  }
+  const calibration = registry.getAvailable().find((model) => CALIBRATION_MODEL_PATTERN.test(model.id));
+  if (calibration) return { model: calibration, fallback: false };
+  if (current?.input.includes("image")) return { model: current, fallback: true };
+  return {
+    error:
+      "no evaluation model: declare a calibration VLM (Ising-Calibration) in models.json, or run with a vision model",
+  };
+}
+
+/** Output budget for one answer, within what the model allows. */
+export function outputBudget(model: Pick<Model<Api>, "maxTokens">): number {
+  return model.maxTokens > 0 ? Math.min(model.maxTokens, MAX_OUTPUT_TOKENS) : DEFAULT_MAX_TOKENS;
+}
+
+// ---------------------------------------------------------------------------
+// The six questions
+// ---------------------------------------------------------------------------
+
+/**
+ * The QCalEval questions, keyed as the benchmark keys its responses, in the
+ * order the paper numbers them (Q1–Q6). Each is asked in its own request.
+ */
+export const QUESTIONS = [
   "technical_description",
   "experimental_conclusion",
   "experimental_significance",
-  "fit_quality",
+  "fit_reliability",
   "parameter_extraction",
-  "success",
+  "calibration_diagnosis",
+] as const;
+export type Question = (typeof QUESTIONS)[number];
+
+/** Questions that take labeled demonstrations (the benchmark's ICL mode). */
+export const ICL_QUESTIONS: ReadonlySet<Question> = new Set<Question>([
+  "experimental_significance",
+  "parameter_extraction",
+  "calibration_diagnosis",
+]);
+
+/** The four outcomes of Q2, as the benchmark words them, with the key the diagnosis uses. */
+export const OUTCOMES = [
+  { key: "expected", option: "Expected behavior", meaning: "Experiment produced usable calibration data" },
+  {
+    key: "suboptimal",
+    option: "Suboptimal parameters",
+    meaning: "Working but needs parameter adjustment within this experiment",
+  },
+  {
+    key: "anomalous",
+    option: "Anomalous behavior",
+    meaning: "Requires upstream recalibration or shows uncontrollable quantum effects",
+  },
+  { key: "apparatus", option: "Apparatus issue", meaning: "No meaningful signal — measurement system misconfigured" },
+] as const;
+export type Outcome = (typeof OUTCOMES)[number]["key"];
+
+/** The three answers of Q4, with the key the diagnosis uses. */
+export const FIT_ASSESSMENTS = [
+  { key: "reliable", option: "Reliable" },
+  { key: "unreliable", option: "Unreliable" },
+  { key: "no_fit", option: "No fit" },
 ] as const;
+export type FitAssessment = (typeof FIT_ASSESSMENTS)[number]["key"];
+
+/** What Q3 asks the model to explain when the host gives no task-specific points. */
+export const DEFAULT_GUIDANCE = [
+  "What the observed behavior indicates about the calibration",
+  "Whether the measurement is valid for its purpose (sweep range, resolution, signal quality)",
+  "What calibration step follows",
+] as const;
+
+/** Q6 statuses when the host gives none; a task's own criteria replace them. */
+export const DEFAULT_STATUSES: Readonly<Record<string, string>> = {
+  SUCCESS: "The plot shows the expected behavior described above and the result is usable",
+  SUBOPTIMAL: "Usable data, but a parameter of this experiment (range, amplitude, resolution) needs adjustment",
+  NO_SIGNAL: "Flat or random data with no usable signal",
+  ANOMALOUS: "Behavior that calls for upstream recalibration or shows uncontrolled effects",
+};
+export const DEFAULT_RANGE_FORMAT = "(<min>, <max>) of the swept parameter";
+
+/** Task-specific pieces of the question templates, supplied by the host. */
+export interface Criteria {
+  /** Q3 "Explain:" bullets. */
+  guidance?: ReadonlyArray<string>;
+  /** Q5 schema: parameter name to its type or description. */
+  parameters?: Readonly<Record<string, string>>;
+  /** Q6 decision criteria: status code to the criterion; SUCCESS first. */
+  statuses?: Readonly<Record<string, string>>;
+  /** Q6 suggested-range format, e.g. "(<min 1/alpha>, <max 1/alpha>)". */
+  range_format?: string;
+}
+
+export interface Figure {
+  data: string;
+  mimeType: string;
+}
+
+/** One labeled example plot: its figure(s) and the answers an expert validated. */
+export interface LabeledExample {
+  images: ReadonlyArray<Figure>;
+  /** What the example shows, e.g. "drag_failure_no_signal" or a short title. */
+  scenario?: string;
+  /** Q3 answer for the example. */
+  analysis?: string;
+  /** Q5 answer for the example: JSON text or an object. */
+  parameters?: string | Readonly<Record<string, unknown>>;
+  /** Q6 status code of the example. */
+  status?: string;
+  /** Q6 suggested range of the example; "N/A" or empty for a success. */
+  suggested_range?: string;
+}
+
+export type UserContentBlock =
+  | { type: "text"; text: string }
+  | { type: "image"; data: string; mimeType: string };
 
-export const SYSTEM_PROMPT = [
-  "You evaluate superconducting qubit calibration experiment plots.",
-  "Answer with one JSON object and nothing else, with these keys:",
-  '  "technical_description": what the plot shows (axes, traces, fit);',
-  '  "experimental_conclusion": what the data says about the experiment;',
-  '  "experimental_significance": what it means for the calibration;',
-  '  "fit_quality": "good" | "marginal" | "poor" and why;',
-  '  "parameter_extraction": extracted values with units, or null;',
-  '  "success": true | false, whether the experiment achieved its goal.',
-  "Base every statement on the plot and the given context only.",
-].join("\n");
+export interface PromptInput {
+  /** The experiment background: what is swept and what a successful result looks like. */
+  background?: string;
+  /** The host's context: task, target, anything the background cannot know. */
+  context: string;
+  /** A specific point to address in the analysis (Q3). */
+  question?: string;
+  criteria?: Criteria;
+  /** The figures; the last one is the measured result to judge. */
+  images: ReadonlyArray<Figure>;
+  examples?: ReadonlyArray<LabeledExample>;
+}
+
+/** The marker the templates use for the figure to judge, as the benchmark's prompts do. */
+const IMAGE = "<image>";
+
+function paragraphs(...parts: Array<string | undefined>): string {
+  return parts.filter((part): part is string => Boolean(part && part.trim())).join("\n\n");
+}
+
+function opening(input: PromptInput, withBackground = true): string {
+  const context = input.context.trim();
+  return paragraphs(
+    withBackground ? input.background?.trim() : undefined,
+    context ? `Experiment context: ${context}` : undefined,
+  );
+}
+
+/** Keep examples in order within the example and figure budgets; later ones are dropped. */
+export function fitExamples(examples: ReadonlyArray<LabeledExample>): LabeledExample[] {
+  const kept: LabeledExample[] = [];
+  let figures = 0;
+  for (const example of examples) {
+    if (kept.length >= MAX_EXAMPLES) break;
+    const labeled = Boolean(example.analysis?.trim() || example.status?.trim() || example.parameters);
+    if (!labeled || !example.images?.length) continue;
+    if (figures + example.images.length > MAX_EXAMPLE_IMAGES) break;
+    figures += example.images.length;
+    kept.push(example);
+  }
+  return kept;
+}
+
+function parametersText(value: LabeledExample["parameters"]): string | undefined {
+  if (value === undefined || value === null) return undefined;
+  if (typeof value === "string") return value.trim() || undefined;
+  return JSON.stringify(value);
+}
+
+function statusList(statuses: Readonly<Record<string, string>>): string {
+  return Object.entries(statuses)
+    .map(([code, criterion]) => `- ${code}: ${criterion}`)
+    .join("\n");
+}
 
 /**
- * Build the single user turn sent to the evaluation model. With several
- * figures, the model is told how many follow so it reads them in the order
- * the context describes (expected reference first, measured result last).
+ * The text of one question's prompt, with `<image>` where each figure goes:
+ * the example figures in order, then the figure to judge. Templates follow the
+ * QCalEval dataset's prompts; the host's criteria fill the family-specific parts.
  */
-export function buildUserPrompt(
-  context: string,
-  question: string | undefined,
-  imageCount = 1,
-): string {
-  const lines = ["Experiment context:", context.trim() || "(none)"];
-  if (imageCount > 1) {
-    lines.push(
-      "",
-      `${imageCount} figures follow, in the order the context describes; the last one is the measured result to judge.`,
+export function questionTemplate(question: Question, input: PromptInput): { text: string; images: Figure[] } {
+  const criteria = input.criteria ?? {};
+  const examples = fitExamples(input.examples ?? []);
+  const exampleImages: Figure[] = [];
+  const demo = (example: LabeledExample): string => {
+    exampleImages.push(...example.images);
+    return example.images.map(() => IMAGE).join(" ");
+  };
+  const specific = input.question?.trim();
+  let text: string;
+
+  switch (question) {
+    case "technical_description": {
+      text = [
+        `Describe the figure ${IMAGE} in JSON format.`,
+        "",
+        "Required fields:",
+        "{",
+        '  "plot_type": "scatter" | "line" | "heatmap" | "histogram",',
+        '  "x_axis": {"label": string, "scale": "linear" | "log", "range": [min, max]},',
+        '  "y_axis": {"label": string, "scale": "linear" | "log", "range": [min, max]},',
+        '  "main_features": string',
+        "}",
+      ].join("\n");
+      break;
+    }
+    case "experimental_conclusion": {
+      text = paragraphs(
+        opening(input),
+        `Based on what you observe in the data ${IMAGE}, classify the experimental outcome.`,
+        ["Options:", ...OUTCOMES.map((o) => `- ${o.option}: ${o.meaning}`)].join("\n"),
+        "Provide your answer as:\nClassification: <your choice>\nReason: <brief explanation>",
+      );
+      break;
+    }
+    case "experimental_significance": {
+      const shown = examples.filter((example) => example.analysis?.trim());
+      if (shown.length > 0) {
+        text = paragraphs(
+          opening(input),
+          "Here are example analyses for each scenario type in this experiment family:",
+          ...shown.map(
+            (example, index) =>
+              `${demo(example)} Scenario "${example.scenario?.trim() || `example ${index + 1}`}":\n${example.analysis!.trim()}`,
+          ),
+          specific ? `Also address: ${specific}` : undefined,
+          `Now analyze this chart:\n\n${IMAGE}`,
+        );
+      } else {
+        const guidance = criteria.guidance?.filter((line) => line.trim()).length
+          ? criteria.guidance.filter((line) => line.trim())
+          : DEFAULT_GUIDANCE;
+        text = paragraphs(
+          opening(input),
+          `What does this result ${IMAGE} imply?`,
+          ["Explain:", ...guidance.map((line) => `- ${line.trim()}`), ...(specific ? [`- ${specific}`] : [])].join(
+            "\n",
+          ),
+          "Provide your assessment.",
+        );
+      }
+      break;
+    }
+    case "fit_reliability": {
+      text = paragraphs(
+        opening(input),
+        `Assess whether the fit to the data in this plot ${IMAGE} is reliable for parameter extraction.`,
+        ["Options:", ...FIT_ASSESSMENTS.map((f) => `- ${f.option}`)].join("\n"),
+        "Provide your answer as:\nAssessment: <your choice>\nReason: <brief explanation>",
+      );
+      break;
+    }
+    case "parameter_extraction": {
+      const shown = examples.find((example) => parametersText(example.parameters));
+      const schema = Object.entries(criteria.parameters ?? {});
+      if (shown) {
+        text = paragraphs(
+          opening(input),
+          "Here is an example of parameter extraction from a similar chart:",
+          `${demo(shown)} Extracted parameters: ${parametersText(shown.parameters)}`,
+          `Now extract parameters from this chart:\n\n${IMAGE}`,
+        );
+      } else if (schema.length > 0) {
+        text = paragraphs(
+          opening(input),
+          `Extract the following parameters from this plot ${IMAGE}.`,
+          `Report in JSON format:\n{${schema.map(([name, type]) => `"${name}": ${type}`).join(", ")}}\nUse "Unreliable" for a value the plot does not determine.`,
+        );
+      } else {
+        text = paragraphs(
+          opening(input),
+          `Extract the physical parameters this plot ${IMAGE} determines.`,
+          'Report in JSON format, one key per parameter with its value and unit, e.g. {"<parameter>": <value>, "<parameter>_unit": "<unit>"}. Use "Unreliable" for a value the plot does not determine.',
+        );
+      }
+      break;
+    }
+    case "calibration_diagnosis": {
+      const statuses = criteria.statuses && Object.keys(criteria.statuses).length > 0 ? criteria.statuses : DEFAULT_STATUSES;
+      const range = criteria.range_format?.trim() || DEFAULT_RANGE_FORMAT;
+      const shown = examples.filter((example) => example.status?.trim());
+      if (shown.length > 0) {
+        text = paragraphs(
+          opening(input),
+          `Classify this chart given the following labeled examples.\n\nDECISION CRITERIA\n${statusList(statuses)}`,
+          ...shown.map(
+            (example) =>
+              `${demo(example)} Status: ${example.status!.trim().toUpperCase()}\nSuggested range: ${example.suggested_range?.trim() || "N/A"}`,
+          ),
+          `Now classify this chart:\n\n${IMAGE}`,
+        );
+      } else {
+        text = paragraphs(
+          opening(input),
+          `Evaluate the image ${IMAGE} and determine the experiment status.`,
+          `DECISION CRITERIA\n${statusList(statuses)}`,
+          `When the status is not SUCCESS, provide a SPECIFIC suggested ${range}.`,
+          `The response MUST follow this exact format:\n\nStatus: <one of the listed statuses>\nSuggested range: ${range} (or "N/A" if SUCCESS)\nNotes: <1-3 sentences explaining your reasoning>`,
+        );
+      }
+      break;
+    }
+  }
+
+  // A reference figure ahead of the measured one (a task review opens with
+  // the expected plot) is shown first, as the benchmark shows its examples.
+  const query = input.images[input.images.length - 1];
+  const references = input.images.slice(0, -1);
+  if (references.length > 0) {
+    text = paragraphs(
+      `Reference figure${references.length === 1 ? "" : "s"} for comparison (expected result); the last figure is the measured result to judge:\n${references.map(() => IMAGE).join(" ")}`,
+      text,
     );
   }
-  if (question?.trim()) lines.push("", "Specific question:", question.trim());
-  return lines.join("\n");
+  return { text, images: [...references, ...exampleImages, ...(query ? [query] : [])] };
+}
+
+/**
+ * One question's user turn as content blocks: the template text with each
+ * `<image>` replaced by the next figure, images as native blocks. `prompt` is
+ * the same turn as text with `[figure]` where an image was, for the record.
+ */
+export function buildQuestionContent(
+  question: Question,
+  input: PromptInput,
+): { content: UserContentBlock[]; prompt: string } {
+  const { text, images } = questionTemplate(question, input);
+  const content: UserContentBlock[] = [];
+  const parts = text.split(IMAGE);
+  parts.forEach((part, index) => {
+    if (part.trim()) content.push({ type: "text", text: part.trim() });
+    const figure = images[index];
+    if (index < parts.length - 1 && figure) content.push({ type: "image", data: figure.data, mimeType: figure.mimeType });
+  });
+  const prompt = text.replace(/<image>/g, "[figure]").replace(/\n{3,}/g, "\n\n").trim();
+  return { content, prompt };
 }
 
+/** The prompts of several questions as one text, headed by the question key, for the record. */
+export function promptRecord(prompts: ReadonlyArray<{ question: Question; prompt: string }>): string {
+  return prompts.map(({ question, prompt }) => `## ${question}\n${prompt}`).join("\n\n");
+}
+
+// ---------------------------------------------------------------------------
+// Answers
+// ---------------------------------------------------------------------------
+
 /** Pull the text blocks of the assistant answer, dropping thinking. */
 export function answerText(message: AssistantMessage): string {
   return message.content
@@ -89,10 +434,11 @@ export function answerText(message: AssistantMessage): string {
 }
 
 /**
- * Parse the model's JSON object, tolerating a code fence, a "thought" preamble
- * or trailing commentary around it: the first balanced `{...}` is taken.
+ * Parse the first JSON object in an answer, tolerating a code fence, a
+ * "thought" preamble or trailing commentary around it: the first balanced
+ * `{...}` is taken.
  */
-export function parseDiagnosis(text: string): Record<string, unknown> | undefined {
+export function parseJsonObject(text: string): Record<string, unknown> | undefined {
   const start = text.indexOf("{");
   if (start < 0) return undefined;
   let depth = 0;
@@ -133,19 +479,263 @@ function parseObject(json: string): Record<string, unknown> | undefined {
   return undefined;
 }
 
+/** The value after `Label:` on its own line (markdown emphasis tolerated), or undefined. */
+export function labeledLine(text: string, label: string): string | undefined {
+  const match = text.match(new RegExp(`^[ \\t]*[*_#]*${label}[*_]*[ \\t]*:[*_ \\t]*(.+?)[ \\t]*$`, "im"));
+  return match?.[1]?.replace(/[*_]+$/, "").trim() || undefined;
+}
+
+/** Everything after `Label:` to the end of the text, across lines. */
+function labeledRest(text: string, label: string): string | undefined {
+  const match = text.match(new RegExp(`^[ \\t]*[*_#]*${label}[*_]*[ \\t]*:[*_ \\t]*([\\s\\S]*)$`, "im"));
+  return match?.[1]?.trim() || undefined;
+}
+
+export function parseOutcome(text: string): Outcome | null {
+  const line = (labeledLine(text, "Classification") ?? text).toLowerCase();
+  for (const outcome of OUTCOMES) {
+    if (line.includes(outcome.key) || line.includes(outcome.option.toLowerCase())) return outcome.key;
+  }
+  return null;
+}
+
+export function parseFitAssessment(text: string): FitAssessment | null {
+  const line = (labeledLine(text, "Assessment") ?? text).toLowerCase();
+  if (/\bno fit\b|\bno_fit\b/.test(line)) return "no_fit";
+  if (/\bunreliable\b/.test(line)) return "unreliable";
+  if (/\breliable\b/.test(line)) return "reliable";
+  return null;
+}
+
+/** A status code as the benchmark normalises it: trimmed, upper-cased, spaces as underscores. */
+export function normalizeStatus(value: string | undefined): string | null {
+  const code = (value ?? "")
+    .replace(/[`*]+/g, "")
+    .trim()
+    .split(/\s*\(|\s+[—–-]\s+/)[0]
+    ?.trim()
+    .toUpperCase()
+    .replace(/[\s-]+/g, "_");
+  return code && /^[A-Z][A-Z0-9_]*$/.test(code) ? code : null;
+}
+
+/** The parsed value of one answer, in the shape the diagnosis stores for that question. */
+export function parseAnswer(question: Question, text: string): unknown {
+  switch (question) {
+    case "technical_description":
+      return parseJsonObject(text) ?? text;
+    case "experimental_conclusion":
+      return { classification: parseOutcome(text), reason: labeledRest(text, "Reason") ?? "" };
+    case "experimental_significance":
+      return text;
+    case "fit_reliability":
+      return { assessment: parseFitAssessment(text), reason: labeledRest(text, "Reason") ?? "" };
+    case "parameter_extraction": {
+      const object = parseJsonObject(text);
+      if (object) return object;
+      return text.trim().toLowerCase() === "null" ? null : text;
+    }
+    case "calibration_diagnosis": {
+      const range = labeledLine(text, "Suggested range");
+      return {
+        status: normalizeStatus(labeledLine(text, "Status")),
+        suggested_range: range && !/^n\/?a$/i.test(range) ? range : null,
+        notes: labeledRest(text, "Notes") ?? "",
+      };
+    }
+  }
+}
+
+/**
+ * The diagnosis from the parsed answers, plus `success`: the Q6 status is
+ * SUCCESS; without a Q6 answer, the Q2 outcome is expected; null when neither
+ * was answered.
+ */
+export function assembleDiagnosis(answers: Partial<Record<Question, unknown>>): Record<string, unknown> {
+  const diagnosis: Record<string, unknown> = {};
+  for (const question of QUESTIONS) {
+    if (question in answers) diagnosis[question] = answers[question];
+  }
+  const status = (answers.calibration_diagnosis as { status?: unknown } | undefined)?.status;
+  const outcome = (answers.experimental_conclusion as { classification?: unknown } | undefined)?.classification;
+  diagnosis.success =
+    typeof status === "string"
+      ? status === "SUCCESS"
+      : answers.calibration_diagnosis === undefined && typeof outcome === "string"
+        ? outcome === "expected"
+        : null;
+  return diagnosis;
+}
+
+/** One model call made for an answer, for the record: how long it took and how it ended. */
+export interface CallRecord {
+  question: Question;
+  duration_ms: number;
+  stop_reason: string;
+  output_tokens?: number;
+}
+
+/** The answers of one evaluation, read for the record. */
+export interface Answers {
+  diagnosis: Record<string, unknown>;
+  /** Every answered question's text, as returned. */
+  answers: Partial<Record<Question, string>>;
+  questions: Question[];
+  /** Questions asked that got no answer (a model error, or an empty reply). */
+  missing: Question[];
+  /** Model calls made, retries included. */
+  attempts: number;
+  stopReason: string;
+  usage?: Usage;
+  calls: CallRecord[];
+  /** The error of the last failed call, when every call for some question failed. */
+  errors: Partial<Record<Question, string>>;
+}
+
+/** Token usage of two calls together, so a retried answer is billed as both. */
+export function addUsage(a: Usage | undefined, b: Usage | undefined): Usage | undefined {
+  if (!a || !b) return a ?? b;
+  const sum = (x: number | undefined, y: number | undefined) => (x ?? 0) + (y ?? 0);
+  return {
+    ...a,
+    input: sum(a.input, b.input),
+    output: sum(a.output, b.output),
+    cacheRead: sum(a.cacheRead, b.cacheRead),
+    cacheWrite: sum(a.cacheWrite, b.cacheWrite),
+    ...(a.reasoning === undefined && b.reasoning === undefined
+      ? {}
+      : { reasoning: sum(a.reasoning, b.reasoning) }),
+    totalTokens: sum(a.totalTokens, b.totalTokens),
+    cost: {
+      input: sum(a.cost?.input, b.cost?.input),
+      output: sum(a.cost?.output, b.cost?.output),
+      cacheRead: sum(a.cost?.cacheRead, b.cost?.cacheRead),
+      cacheWrite: sum(a.cost?.cacheWrite, b.cost?.cacheWrite),
+      total: sum(a.cost?.total, b.cost?.total),
+    },
+  };
+}
+
+/**
+ * Ask every question, each in its own request, all at once. A call that
+ * fails (a transport or server error) is retried once, as the benchmark
+ * retries API failures; an answer is never re-asked for its content. Each
+ * call's duration and stop reason are kept, so a slow evaluation can be
+ * traced to the question that ran long.
+ */
+export async function askQuestions(
+  questions: ReadonlyArray<Question>,
+  call: (question: Question) => Promise<AssistantMessage>,
+  clock: () => number = Date.now,
+): Promise<Answers> {
+  const calls: CallRecord[] = [];
+  const usages: Usage[] = [];
+  const ask = async (question: Question): Promise<AssistantMessage> => {
+    const started = clock();
+    const message = await call(question);
+    calls.push({
+      question,
+      duration_ms: Math.max(0, clock() - started),
+      stop_reason: message.stopReason,
+      ...(message.usage ? { output_tokens: message.usage.output } : {}),
+    });
+    if (message.usage) usages.push(message.usage);
+    return message;
+  };
+  const results = await Promise.all(
+    questions.map(async (question) => {
+      let message = await ask(question);
+      if (message.stopReason === "error") message = await ask(question);
+      return { question, message };
+    }),
+  );
+  const answers: Partial<Record<Question, string>> = {};
+  const parsed: Partial<Record<Question, unknown>> = {};
+  const errors: Partial<Record<Question, string>> = {};
+  const missing: Question[] = [];
+  for (const { question, message } of results) {
+    const text = message.stopReason === "error" ? "" : answerText(message);
+    if (!text) {
+      missing.push(question);
+      if (message.stopReason === "error") errors[question] = message.errorMessage ?? "model error";
+      continue;
+    }
+    answers[question] = text;
+    parsed[question] = parseAnswer(question, text);
+  }
+  // Calls finish in any order; the record lists them by question.
+  calls.sort((a, b) => QUESTIONS.indexOf(a.question) - QUESTIONS.indexOf(b.question));
+  const stops = calls.map((c) => c.stop_reason);
+  const stopReason = stops.includes("length") ? "length" : missing.length === questions.length && stops.length ? stops[stops.length - 1]! : "stop";
+  return {
+    diagnosis: assembleDiagnosis(parsed),
+    answers,
+    questions: [...questions],
+    missing,
+    attempts: calls.length,
+    stopReason,
+    usage: usages.reduce<Usage | undefined>((sum, usage) => addUsage(sum, usage), undefined),
+    calls,
+    errors,
+  };
+}
+
+// ---------------------------------------------------------------------------
+// The tool
+// ---------------------------------------------------------------------------
+
 const imageSchema = Type.Object({
   data: Type.String({ description: "Base64 image bytes (no data: prefix)" }),
   mimeType: Type.String({ description: "image/png or image/jpeg" }),
 });
 
+const exampleSchema = Type.Object({
+  images: Type.Array(imageSchema, { minItems: 1 }),
+  scenario: Type.Optional(Type.String({ description: "What the example shows, in a few words" })),
+  analysis: Type.Optional(Type.String({ description: "The expert's analysis of the example (its experimental_significance answer)" })),
+  parameters: Type.Optional(
+    Type.Union([Type.String(), Type.Record(Type.String(), Type.Unknown())], {
+      description: "The parameters read from the example plot, as JSON",
+    }),
+  ),
+  status: Type.Optional(Type.String({ description: "The example's status code (its calibration_diagnosis answer)" })),
+  suggested_range: Type.Optional(Type.String({ description: "The example's suggested range, or N/A" })),
+});
+
+const criteriaSchema = Type.Object({
+  guidance: Type.Optional(Type.Array(Type.String(), { description: "Points the analysis should explain" })),
+  parameters: Type.Optional(
+    Type.Record(Type.String(), Type.String(), { description: "Parameters to extract: name to type, e.g. float" }),
+  ),
+  statuses: Type.Optional(
+    Type.Record(Type.String(), Type.String(), { description: "Decision criteria: status code to criterion, SUCCESS first" }),
+  ),
+  range_format: Type.Optional(Type.String({ description: "Format of the suggested range for a failed status" })),
+});
+
+const questionSchema = Type.Union(QUESTIONS.map((question) => Type.Literal(question)));
+
 const parameters = Type.Object({
   context: Type.String({
     description:
-      "Experiment context: task name, target qubit/coupling, parameters, and what a successful result looks like",
+      "Experiment context the background cannot know: task name, target qubit/coupling, swept range, previous values",
   }),
   question: Type.Optional(
-    Type.String({ description: "A specific question to answer about the plot" }),
+    Type.String({ description: "A specific point the analysis should address, on top of the standard questions" }),
   ),
+  task_name: Type.Optional(
+    Type.String({
+      description:
+        "The calibration task's name as the host knows it (e.g. CheckRabi). Lets the host supply the background and criteria for the experiment.",
+    }),
+  ),
+  knowledge: Type.Optional(
+    Type.String({
+      description:
+        "Normally omitted: the host supplies the experiment background for `task_name` (what is swept, what a successful result looks like). Fill only to override it with text you were given.",
+    }),
+  ),
+  criteria: Type.Optional(criteriaSchema),
   images: Type.Optional(
     Type.Array(imageSchema, {
       description:
@@ -160,18 +750,59 @@ const parameters = Type.Object({
         "How many of the newest figures the host should supply when `images` is omitted (default 1). Use 2 when an expected reference figure precedes the measured one, and describe their order in `context`.",
     }),
   ),
+  use_examples: Type.Optional(
+    Type.Boolean({
+      description:
+        "Set true only when the user asked to judge against labeled reference examples: the host then supplies its expert-labeled example plots for `task_name` (in-context examples). Default false.",
+    }),
+  ),
+  examples: Type.Optional(
+    Type.Array(exampleSchema, {
+      maxItems: MAX_EXAMPLES,
+      description:
+        "Normally omitted: the host supplies labeled examples when `use_examples` is true. Each is one or more real figures with the expert's answers.",
+    }),
+  ),
+  questions: Type.Optional(
+    Type.Array(questionSchema, {
+      minItems: 1,
+      description: "The questions to ask (default: all six, in order).",
+    }),
+  ),
 });
 
 /** Stored on the tool result; the host reads `diagnosis` without reparsing text. */
 export interface EvaluationDetails {
-  model: ModelSpec;
+  /** The model that judged the plot; absent when none could be resolved. */
+  model?: ModelSpec;
+  /** True when no calibration VLM was configured and the conversation's model judged instead. */
+  fallback?: boolean;
+  /** The parsed answers by question, plus `success`. */
   diagnosis?: Record<string, unknown> | null;
-  /** The model's answer when it was not the requested JSON. */
-  raw?: string;
+  /** Every answered question's text, as returned. */
+  answers?: Partial<Record<Question, string>>;
+  /** The questions asked, in order. */
+  questions?: Question[];
+  /** Questions asked that got no answer. */
+  missing_keys?: Question[];
+  /** Model calls made, retries included. */
+  attempts?: number;
+  /** "length" when a call hit the output budget; otherwise how the calls ended. */
+  stop_reason?: string;
+  /** Every model call made, by question: duration, stop reason, output tokens. */
+  calls?: CallRecord[];
+  /** The user turns the model was sent, one per question, so the host can show what it saw. */
+  prompt?: string;
+  /** How many labeled examples were shown. */
+  examples_used?: number;
 }
 
-function failure(spec: ModelSpec, text: string): AgentToolResult<EvaluationDetails> {
-  return { content: [{ type: "text", text: `qcal_evaluate: ${text}` }], details: { model: spec }, isError: true };
+function failure(spec: ModelSpec | undefined, text: string): AgentToolResult<EvaluationDetails> {
+  return {
+    content: [{ type: "text", text: `qcal_evaluate: ${text}` }],
+    details: spec ? { model: spec } : {},
+    isError: true,
+  };
 }
 
 export default function qcalevalExtension(pi: ExtensionAPI) {
@@ -179,11 +810,13 @@ export default function qcalevalExtension(pi: ExtensionAPI) {
     name: "qcal_evaluate",
     label: "Evaluate Calibration Plot",
     description:
-      "Send a calibration experiment plot to the calibration-evaluation VLM and return its diagnosis: technical description, conclusion, significance, fit quality, extracted parameters, and success.",
+      "Send a calibration experiment plot to the calibration-evaluation VLM and return its diagnosis: technical description, outcome classification, significance, fit reliability, extracted parameters, and calibration status.",
     promptSnippet: "Diagnose a calibration plot with the dedicated calibration VLM",
     promptGuidelines: [
       "Use qcal_evaluate for every calibration figure you need to judge, instead of judging it yourself.",
-      "Pass the task name, target, and expected result as context so the diagnosis is specific.",
+      "Pass the task name and target as context so the diagnosis is specific; do not pass fitted values, the model reads the plot.",
+      "Pass task_name when you know it: the host then supplies the background and criteria for that task.",
+      "Set use_examples true only when the user asked for a judgement against labeled reference examples.",
       "Treat the returned diagnosis as the evaluation of record; add your own reasoning only on top of it.",
     ],
     parameters,
@@ -194,11 +827,10 @@ export default function qcalevalExtension(pi: ExtensionAPI) {
       _onUpdate: unknown,
       ctx: ExtensionToolContext,
     ): Promise<AgentToolResult<EvaluationDetails>> {
-      const spec = parseModelSpec(process.env.QCALEVAL_MODEL);
-      const model: Model<Api> | undefined = ctx.modelRegistry.find(spec.provider, spec.modelId);
-      if (!model) {
-        return failure(spec, `model ${spec.provider}/${spec.modelId} is not configured (QCALEVAL_MODEL)`);
-      }
+      const resolved = resolveEvaluationModel(ctx.modelRegistry, ctx.model, process.env.QCALEVAL_MODEL);
+      if (!resolved.model) return failure(undefined, resolved.error);
+      const { model, fallback } = resolved;
+      const spec: ModelSpec = { provider: model.provider, modelId: model.id };
       const images: ImageContent[] = (params.images ?? []).map((image) => ({
         type: "image",
         data: image.data,
@@ -208,39 +840,59 @@ export default function qcalevalExtension(pi: ExtensionAPI) {
         return failure(spec, "no plot was given. Pass the figure in `images` or fetch it first.");
       }
 
-      const message = await ctx.modelRegistry.complete(
-        model,
-        {
-          systemPrompt: SYSTEM_PROMPT,
-          messages: [
+      const requested = new Set<string>(params.questions ?? []);
+      const questions: Question[] = requested.size > 0 ? QUESTIONS.filter((question) => requested.has(question)) : [...QUESTIONS];
+      const input: PromptInput = {
+        background: params.knowledge,
+        context: params.context,
+        question: params.question,
+        criteria: params.criteria,
+        images: images.map((image) => ({ data: image.data, mimeType: image.mimeType })),
+        examples: params.examples ?? [],
+      };
+      const turns = new Map(questions.map((question) => [question, buildQuestionContent(question, input)]));
+      const examplesUsed = fitExamples(input.examples ?? []).length;
+      const answers = await askQuestions(
+        questions,
+        (question) =>
+          ctx.modelRegistry.complete(
+            model,
             {
-              role: "user",
-              content: [
+              messages: [
                 {
-                  type: "text",
-                  text: buildUserPrompt(params.context, params.question, images.length),
+                  role: "user",
+                  content: turns.get(question)!.content.map((block) =>
+                    block.type === "text"
+                      ? { type: "text" as const, text: block.text }
+                      : { type: "image" as const, data: block.data, mimeType: block.mimeType },
+                  ),
+                  timestamp: Date.now(),
                 },
-                ...images,
               ],
-              timestamp: Date.now(),
             },
-          ],
-        },
-        { signal, maxTokens: Number(process.env.QCALEVAL_MAX_TOKENS ?? 2048) },
+            { signal, maxTokens: outputBudget(model) },
+          ),
       );
-      if (message.stopReason === "error") {
-        return failure(spec, message.errorMessage ?? "model error");
+      if (answers.missing.length === questions.length) {
+        const error = Object.values(answers.errors)[0];
+        return failure(spec, error ?? "the model answered none of the questions");
       }
-      const text = answerText(message);
-      const diagnosis = parseDiagnosis(text);
       return {
-        content: [{ type: "text", text: diagnosis ? JSON.stringify(diagnosis, null, 2) : text }],
+        content: [{ type: "text", text: JSON.stringify(answers.diagnosis, null, 2) }],
         details: {
           model: spec,
-          diagnosis: diagnosis ?? null,
-          ...(diagnosis ? {} : { raw: text }),
+          ...(fallback ? { fallback: true } : {}),
+          diagnosis: answers.diagnosis,
+          answers: answers.answers,
+          questions: answers.questions,
+          ...(answers.missing.length > 0 ? { missing_keys: answers.missing } : {}),
+          attempts: answers.attempts,
+          stop_reason: answers.stopReason,
+          calls: answers.calls,
+          prompt: promptRecord(questions.map((question) => ({ question, prompt: turns.get(question)!.prompt }))),
+          ...(examplesUsed ? { examples_used: examplesUsed } : {}),
         },
-        ...(message.usage ? { usage: message.usage } : {}),
+        ...(answers.usage ? { usage: answers.usage } : {}),
       };
     },
   });

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@orangekame3/pi-qcaleval",
-  "version": "0.0.1",
+  "version": "0.0.2-beta.11",
   "description": "pi extension that sends calibration plots to a calibration-evaluation VLM such as NVIDIA Ising-Calibration and returns a structured diagnosis",
   "license": "Apache-2.0",
   "repository": {

package/skills/qcaleval/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: qcaleval
-description: Evaluate superconducting qubit calibration plots with the dedicated calibration VLM through the pi-qcaleval extension. Use when a calibration figure (Rabi, Ramsey, chevron, T1/T2, readout, spectroscopy, two-qubit tune-up) needs a fit-quality judgement, parameter extraction, or a success verdict.
+description: Evaluate superconducting qubit calibration plots with the dedicated calibration VLM through the pi-qcaleval extension. Use when a calibration figure (Rabi, Ramsey, chevron, T1/T2, readout, spectroscopy, two-qubit tune-up) needs an outcome classification, a fit-reliability judgement, parameter extraction, or a status verdict.
 ---
 
 <!-- Whether a calibration loop continues after an evaluation is the host's policy. This skill describes the tool, not when to stop. -->
@@ -8,28 +8,50 @@ description: Evaluate superconducting qubit calibration plots with the dedicated
 # Calibration plot evaluation
 
 `qcal_evaluate` sends a plot to a calibration-evaluation model (NVIDIA
-Ising-Calibration by default) and returns a JSON diagnosis with
-`technical_description`, `experimental_conclusion`, `experimental_significance`,
-`fit_quality`, `parameter_extraction`, and `success`.
+Ising-Calibration by default) and asks it the six QCalEval questions, each in
+its own request. The diagnosis it returns has one entry per question:
+`technical_description`, `experimental_conclusion` (outcome: expected /
+suboptimal / anomalous / apparatus), `experimental_significance`,
+`fit_reliability` (reliable / unreliable / no_fit), `parameter_extraction`,
+and `calibration_diagnosis` (a status code, a suggested range, notes), plus
+`success` (the status is SUCCESS).
 
 ## Preferred tools
 
-- `qcal_evaluate`: judge one or more calibration plots. Always give `context`:
-  the task name, the target (qubit or coupling), the swept parameter, and what a
-  successful result looks like. Add `question` when a specific point matters
-  (for example "is the chevron centred on the current frequency?").
+- `qcal_evaluate`: judge one or more calibration plots. Give `context` for what
+  the background cannot know: the task name, the target (qubit or coupling),
+  the swept range. Do not put fitted values in it; the model reads the plot.
+  Add `question` when a specific point matters (for example "is the chevron
+  centred on the current frequency?"); it is added to the significance question.
+- Pass `task_name` whenever you know the task (for example `CheckRabi`): the
+  host then supplies the experiment background and the task's criteria (the
+  parameters to extract, the status codes), which is more reliable than a
+  summary you write yourself.
 - When the conversation holds an expected reference figure followed by the
   measured result (a task review opens this way), pass `max_images: 2` and say
   in `context` that the first figure is the reference and the last the
   measurement. The host then supplies both; the default is the newest figure only.
+- Pass `questions` to ask a subset, for example `["calibration_diagnosis"]`
+  when only the status matters.
+
+## Labeled examples (in-context)
+
+- When the user asks to judge a plot against the labeled reference examples
+  (in-context examples, few-shot), pass `use_examples: true` with `task_name`.
+  The host supplies its expert-labeled example plots for that task, and the
+  model sees them before the figure to judge, in the layout of the QCalEval
+  few-shot benchmark: each example's status for the status question, each
+  example's analysis for the significance question, one example's values for
+  parameter extraction. Leave it off otherwise.
 
 ## How to use the result
 
 - Treat the diagnosis as the evaluation of record; reason on top of it instead
   of re-judging the plot yourself.
-- Report `fit_quality` and `success` verbatim, then the extracted parameters.
-- When `success` is false, say what the diagnosis identified as the cause before
-  proposing a next step.
-- The model only sees the plot and the context you pass. If a conclusion needs
-  data that is not in the plot (previous values, device limits), fetch it and
-  include it in `context`.
+- Report the status, the fit reliability and the outcome verbatim, then the
+  extracted parameters and the suggested range when there is one.
+- When the status is not SUCCESS, say what the notes and the outcome reason
+  identified as the cause before proposing a next step.
+- The model only sees the plot, the background and the context you pass. If a
+  conclusion needs data that is not in the plot (previous values, device
+  limits), fetch it and include it in `context`.