@orangekame3/pi-qcaleval 0.0.2-beta.10 → 0.0.2-beta.11

package/README.md

@@ -4,40 +4,71 @@ A [pi](https://github.com/badlogic/pi-mono) extension that sends superconducting
 qubit calibration plots to a dedicated calibration-evaluation VLM and returns a
 structured diagnosis. The default model is
 [NVIDIA Ising-Calibration](https://build.nvidia.com/nvidia/ising-calibration-1.5-31b/modelcard)
-served by vLLM.
+served by vLLM, and the questions are the ones it was trained and benchmarked
+on: the six questions of
+[QCalEval](https://arxiv.org/abs/2604.25884), asked the way the benchmark asks
+them.
 
 The planner model keeps the conversation; `qcal_evaluate` is the tool it calls
 whenever a calibration figure needs a fit-quality judgement, parameter
-extraction, or a success verdict. The extension has no QDash dependency, so it
+extraction, or a status verdict. The extension has no QDash dependency, so it
 works from the pi CLI and from the QDash Copilot runtime alike.
 
 ## Tool
 
-`qcal_evaluate(context, question?, task_name?, knowledge?, images?, max_images?)` returns a JSON object with
-`technical_description`, `experimental_conclusion`, `experimental_significance`,
-`fit_quality`, `parameter_extraction`, and `success`. The parsed object is also
-stored in the tool result's `details.diagnosis`; when the model did not answer
-with JSON, or not with all of it, `details.raw` holds its text and
-`details.missing_keys` the keys it lacks.
-
-On a chat-completions API (vLLM, SGLang, llama.cpp) the request carries a
-`response_format` with the diagnosis schema, so decoding is constrained to the
-six keys; a server that refuses it is asked plainly. An answer that still
-lacks keys is asked for once more and the more complete one kept. The details
-record `attempts` (1 or 2), `structured` (whether decoding was constrained),
-and `stop_reason`.
+`qcal_evaluate(context, question?, task_name?, knowledge?, criteria?, images?, max_images?, use_examples?, examples?, questions?)`
+asks the evaluation model six questions, each in its own single-turn request
+(no system prompt), with the figure at the position the benchmark's templates
+put it, and returns the parsed answers as `details.diagnosis`:
+
+| key | the benchmark's question | parsed answer |
+| --- | --- | --- |
+| `technical_description` | Q1 Describe the figure in JSON | `{plot_type, x_axis, y_axis, main_features}`, or the text when not JSON |
+| `experimental_conclusion` | Q2 Classify the outcome | `{classification: expected \| suboptimal \| anomalous \| apparatus, reason}` |
+| `experimental_significance` | Q3 What does this result imply | text |
+| `fit_reliability` | Q4 Is the fit reliable for extraction | `{assessment: reliable \| unreliable \| no_fit, reason}` |
+| `parameter_extraction` | Q5 Extract parameters in JSON | the object, or the text when not JSON |
+| `calibration_diagnosis` | Q6 Determine the status | `{status: CODE, suggested_range, notes}` |
+
+plus `success`: whether the status is `SUCCESS` (or, when the status question
+was not asked, whether the outcome is `expected`). `details.answers` keeps every
+answer as returned, `details.questions` the questions asked, `details.prompt`
+every user turn the model received (with `[figure]` where an image was), and
+`details.calls` each question's call with its duration, stop reason and output
+tokens. A call that fails is retried once, as the benchmark retries API
+failures; nothing is re-asked for its content. Questions that got no answer are
+listed in `details.missing_keys`. `questions` asks a subset.
+
+`knowledge` is the experiment background the benchmark prepends to Q2–Q6: a few
+sentences on what the experiment sweeps and what a successful result looks like.
+`criteria` fills the family-specific parts of the templates: `guidance` (the
+points Q3 should explain), `parameters` (the Q5 schema, name to type),
+`statuses` (the Q6 decision criteria, code to criterion, `SUCCESS` first) and
+`range_format`. A host that keeps such knowledge per task fills both in from
+`task_name`, so the planner only names the task. Without them the templates use
+generic guidance, ask for whatever parameters the plot determines, and offer
+`SUCCESS` / `SUBOPTIMAL` / `NO_SIGNAL` / `ANOMALOUS`.
+
+`context` is what the background cannot know (the task, the target, a swept
+range); the model reads values off the plot, so do not pass fitted results. A
+`question` is added to Q3 as one more point to address.
 
 `images` are base64 image blocks. A host may fill them in from the most recent
 figures of the conversation, so the planner does not have to pass image bytes;
 `max_images` (default 1, up to 4) says how many of the newest figures to supply,
 for example 2 when an expected reference figure precedes the measured result.
+The last figure is the one judged; earlier ones are shown as references.
+
+### Labeled examples (in-context)
 
-`knowledge` is the host's reference for the experiment: what a good result
-looks like, the questions a reviewer asks, and the common failure patterns. A
-host that keeps such knowledge per task fills it in from `task_name`, so the
-planner only names the task; the text is sent before the context. The exact
-user turn the model received (reference, context, question) is stored in
-`details.prompt`, so a host can show what the model saw.
+With `use_examples`, the host supplies `examples`: each is one or more figures
+with the answers an expert validated (`scenario`, `analysis`, `parameters`,
+`status`, `suggested_range`). They are laid out as the benchmark's few-shot
+split lays them out, examples first and the query figure last, and only for the
+three questions the benchmark uses them for: Q3 shows each example's
+`analysis` under `Scenario "<scenario>":`, Q5 shows one example's
+`parameters`, Q6 shows each example's `Status:` and `Suggested range:`. Up to
+six examples and six example figures are shown; the rest are dropped in order.
 
 ## Configuration
 
@@ -49,7 +80,9 @@ No setting is required. The evaluation model is resolved in this order:
    marked `fallback: true` in `details`, so a host without an Ising deployment
    still gets a diagnosis from the vision model it already runs.
 
-The output budget follows the model's `maxTokens` (at most 4096).
+The output budget per answer follows the model's `maxTokens` (at most 4096). The
+benchmark decodes greedily; set the model's temperature to 0 for reproducible
+answers.
 
 The provider must be declared in pi's `models.json` with its base URL and
 credential, for example:
@@ -67,8 +100,9 @@ credential, for example:
 }
 ```
 
-Only the plot, a short system prompt, and the `context` text are sent, so the
-evaluation fits comfortably in a 16k-token deployment.
+Each request carries one plot, the background and one question, so an
+evaluation fits comfortably in a 16k-token deployment; the six requests run at
+once.
 
 ## Installation
 

package/extensions/qcaleval.ts

@@ -2,11 +2,12 @@
  * pi-qcaleval: evaluate calibration plots with a dedicated calibration VLM.
  *
  * The extension has no QDash dependency. It takes a plot (plus the experiment
- * context the planner already knows, and the host's reference knowledge for
- * the task when the host supplies one) and asks the evaluation model, which is
- * configured separately from the conversation's model, for a diagnosis. The
- * planner decides what to do with it; whether a calibration loop continues is
- * the host's policy, not this tool's.
+ * background and criteria the host supplies for the task, and the context the
+ * planner already knows) and asks the evaluation model the six QCalEval
+ * questions, each as its own single-turn request the way the benchmark and
+ * the Ising-Calibration training data lay them out. The planner decides what
+ * to do with the diagnosis; whether a calibration loop continues is the
+ * host's policy, not this tool's.
  */
 
 import type { Api, AssistantMessage, ImageContent, Model, Usage } from "@earendil-works/pi-ai";
@@ -21,13 +22,13 @@ import { Type, type Static } from "typebox";
 export const MAX_IMAGES = 4;
 
 /** Labeled example plots shown before the query, and the figures they may hold in total. */
-export const MAX_EXAMPLES = 4;
-export const MAX_EXAMPLE_IMAGES = 4;
+export const MAX_EXAMPLES = 6;
+export const MAX_EXAMPLE_IMAGES = 6;
 
 /** Model ids of models fine-tuned for calibration plots; preferred when configured. */
 export const CALIBRATION_MODEL_PATTERN = /ising-calibration/i;
 
-/** Output budget when the model declares none; a diagnosis is a few hundred tokens. */
+/** Output budget when the model declares none; one answer is a few hundred tokens. */
 const DEFAULT_MAX_TOKENS = 2048;
 const MAX_OUTPUT_TOKENS = 4096;
 
@@ -90,152 +91,139 @@ export function resolveEvaluationModel(
   };
 }
 
-/** Output budget for one evaluation, within what the model allows. */
+/** Output budget for one answer, within what the model allows. */
 export function outputBudget(model: Pick<Model<Api>, "maxTokens">): number {
   return model.maxTokens > 0 ? Math.min(model.maxTokens, MAX_OUTPUT_TOKENS) : DEFAULT_MAX_TOKENS;
 }
 
-/** Categories the Ising-Calibration model card describes; asked for in this order. */
-export const DIAGNOSIS_SECTIONS = [
+// ---------------------------------------------------------------------------
+// The six questions
+// ---------------------------------------------------------------------------
+
+/**
+ * The QCalEval questions, keyed as the benchmark keys its responses, in the
+ * order the paper numbers them (Q1–Q6). Each is asked in its own request.
+ */
+export const QUESTIONS = [
   "technical_description",
   "experimental_conclusion",
   "experimental_significance",
-  "fit_quality",
+  "fit_reliability",
   "parameter_extraction",
-  "success",
+  "calibration_diagnosis",
 ] as const;
+export type Question = (typeof QUESTIONS)[number];
 
-/**
- * The diagnosis as a JSON schema: the six keys, all required. Sent as the
- * response format to OpenAI-compatible servers that constrain decoding to a
- * schema (vLLM, SGLang, llama.cpp). Asked for JSON in the prompt alone, the
- * calibration model now and then closes the object after the first key or
- * nests quotes that break it; constrained decoding rules both out.
- */
-export const DIAGNOSIS_SCHEMA = {
-  type: "object",
-  properties: {
-    technical_description: { type: "string" },
-    experimental_conclusion: { type: "string" },
-    experimental_significance: { type: "string" },
-    fit_quality: { type: "string" },
-    parameter_extraction: {
-      anyOf: [{ type: "object", additionalProperties: true }, { type: "string" }, { type: "null" }],
-    },
-    success: { type: "boolean" },
+/** Questions that take labeled demonstrations (the benchmark's ICL mode). */
+export const ICL_QUESTIONS: ReadonlySet<Question> = new Set<Question>([
+  "experimental_significance",
+  "parameter_extraction",
+  "calibration_diagnosis",
+]);
+
+/** The four outcomes of Q2, as the benchmark words them, with the key the diagnosis uses. */
+export const OUTCOMES = [
+  { key: "expected", option: "Expected behavior", meaning: "Experiment produced usable calibration data" },
+  {
+    key: "suboptimal",
+    option: "Suboptimal parameters",
+    meaning: "Working but needs parameter adjustment within this experiment",
   },
-  required: [...DIAGNOSIS_SECTIONS],
-  additionalProperties: false,
-} as const;
+  {
+    key: "anomalous",
+    option: "Anomalous behavior",
+    meaning: "Requires upstream recalibration or shows uncontrollable quantum effects",
+  },
+  { key: "apparatus", option: "Apparatus issue", meaning: "No meaningful signal — measurement system misconfigured" },
+] as const;
+export type Outcome = (typeof OUTCOMES)[number]["key"];
 
-/** `response_format` of a chat completion constrained to the diagnosis schema. */
-export const RESPONSE_FORMAT = {
-  type: "json_schema",
-  json_schema: { name: "calibration_diagnosis", schema: DIAGNOSIS_SCHEMA },
-} as const;
+/** The three answers of Q4, with the key the diagnosis uses. */
+export const FIT_ASSESSMENTS = [
+  { key: "reliable", option: "Reliable" },
+  { key: "unreliable", option: "Unreliable" },
+  { key: "no_fit", option: "No fit" },
+] as const;
+export type FitAssessment = (typeof FIT_ASSESSMENTS)[number]["key"];
 
-/**
- * Whether the model's API takes `response_format` in the request body. Pi
- * merges `samplingParams` into the body of chat completions only; the
- * responses API spells the format differently and other APIs ignore it.
- */
-export function canConstrain(model: Pick<Model<Api>, "api">): boolean {
-  return model.api === "openai-completions";
-}
+/** What Q3 asks the model to explain when the host gives no task-specific points. */
+export const DEFAULT_GUIDANCE = [
+  "What the observed behavior indicates about the calibration",
+  "Whether the measurement is valid for its purpose (sweep range, resolution, signal quality)",
+  "What calibration step follows",
+] as const;
 
-/** The diagnosis keys an answer lacks; all six when there is no diagnosis. */
-export function missingKeys(diagnosis: Record<string, unknown> | undefined): string[] {
-  if (!diagnosis) return [...DIAGNOSIS_SECTIONS];
-  return DIAGNOSIS_SECTIONS.filter((key) => !(key in diagnosis));
+/** Q6 statuses when the host gives none; a task's own criteria replace them. */
+export const DEFAULT_STATUSES: Readonly<Record<string, string>> = {
+  SUCCESS: "The plot shows the expected behavior described above and the result is usable",
+  SUBOPTIMAL: "Usable data, but a parameter of this experiment (range, amplitude, resolution) needs adjustment",
+  NO_SIGNAL: "Flat or random data with no usable signal",
+  ANOMALOUS: "Behavior that calls for upstream recalibration or shows uncontrolled effects",
+};
+export const DEFAULT_RANGE_FORMAT = "(<min>, <max>) of the swept parameter";
+
+/** Task-specific pieces of the question templates, supplied by the host. */
+export interface Criteria {
+  /** Q3 "Explain:" bullets. */
+  guidance?: ReadonlyArray<string>;
+  /** Q5 schema: parameter name to its type or description. */
+  parameters?: Readonly<Record<string, string>>;
+  /** Q6 decision criteria: status code to the criterion; SUCCESS first. */
+  statuses?: Readonly<Record<string, string>>;
+  /** Q6 suggested-range format, e.g. "(<min 1/alpha>, <max 1/alpha>)". */
+  range_format?: string;
 }
 
-export const SYSTEM_PROMPT = [
-  "You evaluate superconducting qubit calibration experiment plots.",
-  "Answer with one JSON object and nothing else, with these keys:",
-  '  "technical_description": what the plot shows (axes, traces, fit);',
-  '  "experimental_conclusion": what the data says about the experiment;',
-  '  "experimental_significance": what it means for the calibration;',
-  '  "fit_quality": "good" | "marginal" | "poor" and why;',
-  '  "parameter_extraction": extracted values with units, or null;',
-  '  "success": true | false, whether the experiment achieved its goal.',
-  "Base every statement on the plot, the reference, and the given context only.",
-].join("\n");
+export interface Figure {
+  data: string;
+  mimeType: string;
+}
 
-/** One labeled example: its figure(s) followed by the expert's diagnosis. */
+/** One labeled example plot: its figure(s) and the answers an expert validated. */
 export interface LabeledExample {
-  label: string;
-  images: ReadonlyArray<{ data: string; mimeType: string }>;
+  images: ReadonlyArray<Figure>;
+  /** What the example shows, e.g. "drag_failure_no_signal" or a short title. */
+  scenario?: string;
+  /** Q3 answer for the example. */
+  analysis?: string;
+  /** Q5 answer for the example: JSON text or an object. */
+  parameters?: string | Readonly<Record<string, unknown>>;
+  /** Q6 status code of the example. */
+  status?: string;
+  /** Q6 suggested range of the example; "N/A" or empty for a success. */
+  suggested_range?: string;
 }
 
 export type UserContentBlock =
   | { type: "text"; text: string }
   | { type: "image"; data: string; mimeType: string };
 
-/**
- * The single user turn sent to the evaluation model, as the QCalEval papers
- * lay it out: the reference for the experiment, then each labeled example as
- * its figure followed by its diagnosis, then the context and question, then
- * the figure to judge. Everything is one ordered content array; images are
- * native blocks, never placeholders in the text.
- *
- * `prompt` is the same turn as text, with `[figure]` where an image was, for
- * the record of what the model saw.
- */
-export function buildUserContent(
-  context: string,
-  question: string | undefined,
-  images: ReadonlyArray<{ data: string; mimeType: string }>,
-  knowledge?: string,
-  examples: ReadonlyArray<LabeledExample> = [],
-): { content: UserContentBlock[]; prompt: string; examplesUsed: number } {
-  const content: UserContentBlock[] = [];
-  const shown = fitExamples(examples);
-  const text = (lines: string[]) => {
-    content.push({ type: "text", text: lines.join("\n") });
-  };
-  const image = (figure: { data: string; mimeType: string }) => {
-    content.push({ type: "image", data: figure.data, mimeType: figure.mimeType });
-  };
+export interface PromptInput {
+  /** The experiment background: what is swept and what a successful result looks like. */
+  background?: string;
+  /** The host's context: task, target, anything the background cannot know. */
+  context: string;
+  /** A specific point to address in the analysis (Q3). */
+  question?: string;
+  criteria?: Criteria;
+  /** The figures; the last one is the measured result to judge. */
+  images: ReadonlyArray<Figure>;
+  examples?: ReadonlyArray<LabeledExample>;
+}
 
-  const opening: string[] = [];
-  if (knowledge?.trim()) {
-    opening.push("Reference for this experiment (expected plot, outcomes, parameters):", knowledge.trim(), "");
-  }
-  if (shown.length > 0) {
-    opening.push(
-      `${shown.length} labeled example${shown.length === 1 ? "" : "s"} of this experiment follow, each figure followed by the diagnosis an expert validated. Use them as the standard for the figure to judge, which comes last.`,
-    );
-  }
-  if (opening.length > 0) text(opening);
+/** The marker the templates use for the figure to judge, as the benchmark's prompts do. */
+const IMAGE = "<image>";
 
-  shown.forEach((example, index) => {
-    text(["", `Example ${index + 1} of ${shown.length}:`]);
-    for (const figure of example.images) image(figure);
-    text([`Diagnosis of example ${index + 1}:`, example.label.trim()]);
-  });
+function paragraphs(...parts: Array<string | undefined>): string {
+  return parts.filter((part): part is string => Boolean(part && part.trim())).join("\n\n");
+}
 
-  const closing: string[] = [];
-  if (shown.length > 0) closing.push("", "Now the figure to judge, evaluated independently of the examples.");
-  closing.push("Experiment context:", context.trim() || "(none)");
-  if (images.length > 1) {
-    closing.push(
-      "",
-      `${images.length} figures follow, in the order the context describes; the last one is the measured result to judge.`,
-    );
-  }
-  if (question?.trim()) closing.push("", "Specific question:", question.trim());
-  // Repeated here because a long reference ahead of the plot pulls the model
-  // towards prose; the last instruction it reads is the output format.
-  closing.push("", "Answer with the JSON object described in the system prompt, and nothing else.");
-  text(closing);
-  for (const figure of images) image(figure);
-
-  const prompt = content
-    .map((block) => (block.type === "text" ? block.text : "[figure]"))
-    .join("\n")
-    .replace(/\n{3,}/g, "\n\n")
-    .trim();
-  return { content, prompt, examplesUsed: shown.length };
+function opening(input: PromptInput, withBackground = true): string {
+  const context = input.context.trim();
+  return paragraphs(
+    withBackground ? input.background?.trim() : undefined,
+    context ? `Experiment context: ${context}` : undefined,
+  );
 }
 
 /** Keep examples in order within the example and figure budgets; later ones are dropped. */
@@ -244,7 +232,8 @@ export function fitExamples(examples: ReadonlyArray<LabeledExample>): LabeledExa
   let figures = 0;
   for (const example of examples) {
     if (kept.length >= MAX_EXAMPLES) break;
-    if (!example.label?.trim() || !example.images?.length) continue;
+    const labeled = Boolean(example.analysis?.trim() || example.status?.trim() || example.parameters);
+    if (!labeled || !example.images?.length) continue;
     if (figures + example.images.length > MAX_EXAMPLE_IMAGES) break;
     figures += example.images.length;
     kept.push(example);
@@ -252,22 +241,190 @@ export function fitExamples(examples: ReadonlyArray<LabeledExample>): LabeledExa
   return kept;
 }
 
-/** The user turn as text only (no examples), kept for callers that render the prompt. */
-export function buildUserPrompt(
-  context: string,
-  question: string | undefined,
-  imageCount = 1,
-  knowledge?: string,
-): string {
-  const placeholder = { data: "", mimeType: "image/png" };
-  return buildUserContent(context, question, Array(imageCount).fill(placeholder), knowledge)
-    .content.filter((block) => block.type === "text")
-    .map((block) => (block as { text: string }).text)
-    .join("\n")
-    .replace(/\n{3,}/g, "\n\n")
-    .trim();
+function parametersText(value: LabeledExample["parameters"]): string | undefined {
+  if (value === undefined || value === null) return undefined;
+  if (typeof value === "string") return value.trim() || undefined;
+  return JSON.stringify(value);
+}
+
+function statusList(statuses: Readonly<Record<string, string>>): string {
+  return Object.entries(statuses)
+    .map(([code, criterion]) => `- ${code}: ${criterion}`)
+    .join("\n");
+}
+
+/**
+ * The text of one question's prompt, with `<image>` where each figure goes:
+ * the example figures in order, then the figure to judge. Templates follow the
+ * QCalEval dataset's prompts; the host's criteria fill the family-specific parts.
+ */
+export function questionTemplate(question: Question, input: PromptInput): { text: string; images: Figure[] } {
+  const criteria = input.criteria ?? {};
+  const examples = fitExamples(input.examples ?? []);
+  const exampleImages: Figure[] = [];
+  const demo = (example: LabeledExample): string => {
+    exampleImages.push(...example.images);
+    return example.images.map(() => IMAGE).join(" ");
+  };
+  const specific = input.question?.trim();
+  let text: string;
+
+  switch (question) {
+    case "technical_description": {
+      text = [
+        `Describe the figure ${IMAGE} in JSON format.`,
+        "",
+        "Required fields:",
+        "{",
+        '  "plot_type": "scatter" | "line" | "heatmap" | "histogram",',
+        '  "x_axis": {"label": string, "scale": "linear" | "log", "range": [min, max]},',
+        '  "y_axis": {"label": string, "scale": "linear" | "log", "range": [min, max]},',
+        '  "main_features": string',
+        "}",
+      ].join("\n");
+      break;
+    }
+    case "experimental_conclusion": {
+      text = paragraphs(
+        opening(input),
+        `Based on what you observe in the data ${IMAGE}, classify the experimental outcome.`,
+        ["Options:", ...OUTCOMES.map((o) => `- ${o.option}: ${o.meaning}`)].join("\n"),
+        "Provide your answer as:\nClassification: <your choice>\nReason: <brief explanation>",
+      );
+      break;
+    }
+    case "experimental_significance": {
+      const shown = examples.filter((example) => example.analysis?.trim());
+      if (shown.length > 0) {
+        text = paragraphs(
+          opening(input),
+          "Here are example analyses for each scenario type in this experiment family:",
+          ...shown.map(
+            (example, index) =>
+              `${demo(example)} Scenario "${example.scenario?.trim() || `example ${index + 1}`}":\n${example.analysis!.trim()}`,
+          ),
+          specific ? `Also address: ${specific}` : undefined,
+          `Now analyze this chart:\n\n${IMAGE}`,
+        );
+      } else {
+        const guidance = criteria.guidance?.filter((line) => line.trim()).length
+          ? criteria.guidance.filter((line) => line.trim())
+          : DEFAULT_GUIDANCE;
+        text = paragraphs(
+          opening(input),
+          `What does this result ${IMAGE} imply?`,
+          ["Explain:", ...guidance.map((line) => `- ${line.trim()}`), ...(specific ? [`- ${specific}`] : [])].join(
+            "\n",
+          ),
+          "Provide your assessment.",
+        );
+      }
+      break;
+    }
+    case "fit_reliability": {
+      text = paragraphs(
+        opening(input),
+        `Assess whether the fit to the data in this plot ${IMAGE} is reliable for parameter extraction.`,
+        ["Options:", ...FIT_ASSESSMENTS.map((f) => `- ${f.option}`)].join("\n"),
+        "Provide your answer as:\nAssessment: <your choice>\nReason: <brief explanation>",
+      );
+      break;
+    }
+    case "parameter_extraction": {
+      const shown = examples.find((example) => parametersText(example.parameters));
+      const schema = Object.entries(criteria.parameters ?? {});
+      if (shown) {
+        text = paragraphs(
+          opening(input),
+          "Here is an example of parameter extraction from a similar chart:",
+          `${demo(shown)} Extracted parameters: ${parametersText(shown.parameters)}`,
+          `Now extract parameters from this chart:\n\n${IMAGE}`,
+        );
+      } else if (schema.length > 0) {
+        text = paragraphs(
+          opening(input),
+          `Extract the following parameters from this plot ${IMAGE}.`,
+          `Report in JSON format:\n{${schema.map(([name, type]) => `"${name}": ${type}`).join(", ")}}\nUse "Unreliable" for a value the plot does not determine.`,
+        );
+      } else {
+        text = paragraphs(
+          opening(input),
+          `Extract the physical parameters this plot ${IMAGE} determines.`,
+          'Report in JSON format, one key per parameter with its value and unit, e.g. {"<parameter>": <value>, "<parameter>_unit": "<unit>"}. Use "Unreliable" for a value the plot does not determine.',
+        );
+      }
+      break;
+    }
+    case "calibration_diagnosis": {
+      const statuses = criteria.statuses && Object.keys(criteria.statuses).length > 0 ? criteria.statuses : DEFAULT_STATUSES;
+      const range = criteria.range_format?.trim() || DEFAULT_RANGE_FORMAT;
+      const shown = examples.filter((example) => example.status?.trim());
+      if (shown.length > 0) {
+        text = paragraphs(
+          opening(input),
+          `Classify this chart given the following labeled examples.\n\nDECISION CRITERIA\n${statusList(statuses)}`,
+          ...shown.map(
+            (example) =>
+              `${demo(example)} Status: ${example.status!.trim().toUpperCase()}\nSuggested range: ${example.suggested_range?.trim() || "N/A"}`,
+          ),
+          `Now classify this chart:\n\n${IMAGE}`,
+        );
+      } else {
+        text = paragraphs(
+          opening(input),
+          `Evaluate the image ${IMAGE} and determine the experiment status.`,
+          `DECISION CRITERIA\n${statusList(statuses)}`,
+          `When the status is not SUCCESS, provide a SPECIFIC suggested ${range}.`,
+          `The response MUST follow this exact format:\n\nStatus: <one of the listed statuses>\nSuggested range: ${range} (or "N/A" if SUCCESS)\nNotes: <1-3 sentences explaining your reasoning>`,
+        );
+      }
+      break;
+    }
+  }
+
+  // A reference figure ahead of the measured one (a task review opens with
+  // the expected plot) is shown first, as the benchmark shows its examples.
+  const query = input.images[input.images.length - 1];
+  const references = input.images.slice(0, -1);
+  if (references.length > 0) {
+    text = paragraphs(
+      `Reference figure${references.length === 1 ? "" : "s"} for comparison (expected result); the last figure is the measured result to judge:\n${references.map(() => IMAGE).join(" ")}`,
+      text,
+    );
+  }
+  return { text, images: [...references, ...exampleImages, ...(query ? [query] : [])] };
+}
+
+/**
+ * One question's user turn as content blocks: the template text with each
+ * `<image>` replaced by the next figure, images as native blocks. `prompt` is
+ * the same turn as text with `[figure]` where an image was, for the record.
+ */
+export function buildQuestionContent(
+  question: Question,
+  input: PromptInput,
+): { content: UserContentBlock[]; prompt: string } {
+  const { text, images } = questionTemplate(question, input);
+  const content: UserContentBlock[] = [];
+  const parts = text.split(IMAGE);
+  parts.forEach((part, index) => {
+    if (part.trim()) content.push({ type: "text", text: part.trim() });
+    const figure = images[index];
+    if (index < parts.length - 1 && figure) content.push({ type: "image", data: figure.data, mimeType: figure.mimeType });
+  });
+  const prompt = text.replace(/<image>/g, "[figure]").replace(/\n{3,}/g, "\n\n").trim();
+  return { content, prompt };
+}
+
+/** The prompts of several questions as one text, headed by the question key, for the record. */
+export function promptRecord(prompts: ReadonlyArray<{ question: Question; prompt: string }>): string {
+  return prompts.map(({ question, prompt }) => `## ${question}\n${prompt}`).join("\n\n");
 }
 
+// ---------------------------------------------------------------------------
+// Answers
+// ---------------------------------------------------------------------------
+
 /** Pull the text blocks of the assistant answer, dropping thinking. */
 export function answerText(message: AssistantMessage): string {
   return message.content
@@ -277,10 +434,11 @@ export function answerText(message: AssistantMessage): string {
 }
 
 /**
- * Parse the model's JSON object, tolerating a code fence, a "thought" preamble
- * or trailing commentary around it: the first balanced `{...}` is taken.
+ * Parse the first JSON object in an answer, tolerating a code fence, a
+ * "thought" preamble or trailing commentary around it: the first balanced
+ * `{...}` is taken.
  */
-export function parseDiagnosis(text: string): Record<string, unknown> | undefined {
+export function parseJsonObject(text: string): Record<string, unknown> | undefined {
   const start = text.indexOf("{");
   if (start < 0) return undefined;
   let depth = 0;
@@ -321,32 +479,117 @@ function parseObject(json: string): Record<string, unknown> | undefined {
   return undefined;
 }
 
-/** One answer of the evaluation model, read for the record. */
-export interface Answer {
-  text: string;
-  diagnosis?: Record<string, unknown>;
-  /** Diagnosis keys the answer lacks; empty when it is complete. */
-  missing: string[];
-  /** Model calls made for this answer: 1, or 2 after an incomplete or refused first one. */
-  attempts: number;
-  /** Whether decoding was constrained to the diagnosis schema. */
-  structured: boolean;
-  stopReason: AssistantMessage["stopReason"];
-  usage?: Usage;
+/** The value after `Label:` on its own line (markdown emphasis tolerated), or undefined. */
+export function labeledLine(text: string, label: string): string | undefined {
+  const match = text.match(new RegExp(`^[ \\t]*[*_#]*${label}[*_]*[ \\t]*:[*_ \\t]*(.+?)[ \\t]*$`, "im"));
+  return match?.[1]?.replace(/[*_]+$/, "").trim() || undefined;
 }
 
-function readAnswer(message: AssistantMessage, structured: boolean): Answer {
-  const text = answerText(message);
-  const diagnosis = parseDiagnosis(text);
-  return {
-    text,
-    diagnosis,
-    missing: missingKeys(diagnosis),
-    attempts: 1,
-    structured,
-    stopReason: message.stopReason,
-    usage: message.usage,
-  };
+/** Everything after `Label:` to the end of the text, across lines. */
+function labeledRest(text: string, label: string): string | undefined {
+  const match = text.match(new RegExp(`^[ \\t]*[*_#]*${label}[*_]*[ \\t]*:[*_ \\t]*([\\s\\S]*)$`, "im"));
+  return match?.[1]?.trim() || undefined;
+}
+
+export function parseOutcome(text: string): Outcome | null {
+  const line = (labeledLine(text, "Classification") ?? text).toLowerCase();
+  for (const outcome of OUTCOMES) {
+    if (line.includes(outcome.key) || line.includes(outcome.option.toLowerCase())) return outcome.key;
+  }
+  return null;
+}
+
+export function parseFitAssessment(text: string): FitAssessment | null {
+  const line = (labeledLine(text, "Assessment") ?? text).toLowerCase();
+  if (/\bno fit\b|\bno_fit\b/.test(line)) return "no_fit";
+  if (/\bunreliable\b/.test(line)) return "unreliable";
+  if (/\breliable\b/.test(line)) return "reliable";
+  return null;
+}
+
+/** A status code as the benchmark normalises it: trimmed, upper-cased, spaces as underscores. */
+export function normalizeStatus(value: string | undefined): string | null {
+  const code = (value ?? "")
+    .replace(/[`*]+/g, "")
+    .trim()
+    .split(/\s*\(|\s+[—–-]\s+/)[0]
+    ?.trim()
+    .toUpperCase()
+    .replace(/[\s-]+/g, "_");
+  return code && /^[A-Z][A-Z0-9_]*$/.test(code) ? code : null;
+}
+
+/** The parsed value of one answer, in the shape the diagnosis stores for that question. */
+export function parseAnswer(question: Question, text: string): unknown {
+  switch (question) {
+    case "technical_description":
+      return parseJsonObject(text) ?? text;
+    case "experimental_conclusion":
+      return { classification: parseOutcome(text), reason: labeledRest(text, "Reason") ?? "" };
+    case "experimental_significance":
+      return text;
+    case "fit_reliability":
+      return { assessment: parseFitAssessment(text), reason: labeledRest(text, "Reason") ?? "" };
+    case "parameter_extraction": {
+      const object = parseJsonObject(text);
+      if (object) return object;
+      return text.trim().toLowerCase() === "null" ? null : text;
+    }
+    case "calibration_diagnosis": {
+      const range = labeledLine(text, "Suggested range");
+      return {
+        status: normalizeStatus(labeledLine(text, "Status")),
+        suggested_range: range && !/^n\/?a$/i.test(range) ? range : null,
+        notes: labeledRest(text, "Notes") ?? "",
+      };
+    }
+  }
+}
+
+/**
+ * The diagnosis from the parsed answers, plus `success`: the Q6 status is
+ * SUCCESS; without a Q6 answer, the Q2 outcome is expected; null when neither
+ * was answered.
+ */
+export function assembleDiagnosis(answers: Partial<Record<Question, unknown>>): Record<string, unknown> {
+  const diagnosis: Record<string, unknown> = {};
+  for (const question of QUESTIONS) {
+    if (question in answers) diagnosis[question] = answers[question];
+  }
+  const status = (answers.calibration_diagnosis as { status?: unknown } | undefined)?.status;
+  const outcome = (answers.experimental_conclusion as { classification?: unknown } | undefined)?.classification;
+  diagnosis.success =
+    typeof status === "string"
+      ? status === "SUCCESS"
+      : answers.calibration_diagnosis === undefined && typeof outcome === "string"
+        ? outcome === "expected"
+        : null;
+  return diagnosis;
+}
+
+/** One model call made for an answer, for the record: how long it took and how it ended. */
+export interface CallRecord {
+  question: Question;
+  duration_ms: number;
+  stop_reason: string;
+  output_tokens?: number;
+}
+
+/** The answers of one evaluation, read for the record. */
+export interface Answers {
+  diagnosis: Record<string, unknown>;
+  /** Every answered question's text, as returned. */
+  answers: Partial<Record<Question, string>>;
+  questions: Question[];
+  /** Questions asked that got no answer (a model error, or an empty reply). */
+  missing: Question[];
+  /** Model calls made, retries included. */
+  attempts: number;
+  stopReason: string;
+  usage?: Usage;
+  calls: CallRecord[];
+  /** The error of the last failed call, when every call for some question failed. */
+  errors: Partial<Record<Question, string>>;
 }
 
 /** Token usage of two calls together, so a retried answer is billed as both. */
@@ -374,66 +617,125 @@ export function addUsage(a: Usage | undefined, b: Usage | undefined): Usage | un
 }
 
 /**
- * Ask the model for the diagnosis, in at most two calls. The first call is
- * constrained to the schema when the API allows it; a server that refuses the
- * response format (an older vLLM, a gateway) gets the plain request instead.
- * An answer that lacks diagnosis keys is asked for once more, since the model
- * usually completes the object on a second sample, and the more complete of
- * the two answers is kept. Usage counts every call made.
+ * Ask every question, each in its own request, all at once. A call that
+ * fails (a transport or server error) is retried once, as the benchmark
+ * retries API failures; an answer is never re-asked for its content. Each
+ * call's duration and stop reason are kept, so a slow evaluation can be
+ * traced to the question that ran long.
  */
-export async function askForDiagnosis(
-  call: (structured: boolean) => Promise<AssistantMessage>,
-  constrain: boolean,
-): Promise<Answer | { error: string }> {
-  let structured = constrain;
-  let first = await call(structured);
-  let attempts = 1;
-  let usage: Usage | undefined = first.usage;
-  if (first.stopReason === "error" && structured) {
-    structured = false;
-    first = await call(structured);
-    attempts += 1;
-    usage = addUsage(usage, first.usage);
+export async function askQuestions(
+  questions: ReadonlyArray<Question>,
+  call: (question: Question) => Promise<AssistantMessage>,
+  clock: () => number = Date.now,
+): Promise<Answers> {
+  const calls: CallRecord[] = [];
+  const usages: Usage[] = [];
+  const ask = async (question: Question): Promise<AssistantMessage> => {
+    const started = clock();
+    const message = await call(question);
+    calls.push({
+      question,
+      duration_ms: Math.max(0, clock() - started),
+      stop_reason: message.stopReason,
+      ...(message.usage ? { output_tokens: message.usage.output } : {}),
+    });
+    if (message.usage) usages.push(message.usage);
+    return message;
+  };
+  const results = await Promise.all(
+    questions.map(async (question) => {
+      let message = await ask(question);
+      if (message.stopReason === "error") message = await ask(question);
+      return { question, message };
+    }),
+  );
+  const answers: Partial<Record<Question, string>> = {};
+  const parsed: Partial<Record<Question, unknown>> = {};
+  const errors: Partial<Record<Question, string>> = {};
+  const missing: Question[] = [];
+  for (const { question, message } of results) {
+    const text = message.stopReason === "error" ? "" : answerText(message);
+    if (!text) {
+      missing.push(question);
+      if (message.stopReason === "error") errors[question] = message.errorMessage ?? "model error";
+      continue;
+    }
+    answers[question] = text;
+    parsed[question] = parseAnswer(question, text);
   }
-  if (first.stopReason === "error") return { error: first.errorMessage ?? "model error" };
-  const answer = { ...readAnswer(first, structured), attempts, usage };
-  if (answer.missing.length === 0 || attempts === 2) return answer;
-  const second = await call(structured);
-  const retried = second.stopReason === "error" ? undefined : readAnswer(second, structured);
-  const kept = retried && retried.missing.length <= answer.missing.length ? retried : answer;
-  return { ...kept, attempts: 2, usage: addUsage(first.usage, second.usage) };
+  // Calls finish in any order; the record lists them by question.
+  calls.sort((a, b) => QUESTIONS.indexOf(a.question) - QUESTIONS.indexOf(b.question));
+  const stops = calls.map((c) => c.stop_reason);
+  const stopReason = stops.includes("length") ? "length" : missing.length === questions.length && stops.length ? stops[stops.length - 1]! : "stop";
+  return {
+    diagnosis: assembleDiagnosis(parsed),
+    answers,
+    questions: [...questions],
+    missing,
+    attempts: calls.length,
+    stopReason,
+    usage: usages.reduce<Usage | undefined>((sum, usage) => addUsage(sum, usage), undefined),
+    calls,
+    errors,
+  };
 }
 
+// ---------------------------------------------------------------------------
+// The tool
+// ---------------------------------------------------------------------------
+
 const imageSchema = Type.Object({
   data: Type.String({ description: "Base64 image bytes (no data: prefix)" }),
   mimeType: Type.String({ description: "image/png or image/jpeg" }),
 });
 
 const exampleSchema = Type.Object({
-  label: Type.String({ description: "The expert's diagnosis of the example figure(s)" }),
   images: Type.Array(imageSchema, { minItems: 1 }),
+  scenario: Type.Optional(Type.String({ description: "What the example shows, in a few words" })),
+  analysis: Type.Optional(Type.String({ description: "The expert's analysis of the example (its experimental_significance answer)" })),
+  parameters: Type.Optional(
+    Type.Union([Type.String(), Type.Record(Type.String(), Type.Unknown())], {
+      description: "The parameters read from the example plot, as JSON",
+    }),
+  ),
+  status: Type.Optional(Type.String({ description: "The example's status code (its calibration_diagnosis answer)" })),
+  suggested_range: Type.Optional(Type.String({ description: "The example's suggested range, or N/A" })),
 });
 
+const criteriaSchema = Type.Object({
+  guidance: Type.Optional(Type.Array(Type.String(), { description: "Points the analysis should explain" })),
+  parameters: Type.Optional(
+    Type.Record(Type.String(), Type.String(), { description: "Parameters to extract: name to type, e.g. float" }),
+  ),
+  statuses: Type.Optional(
+    Type.Record(Type.String(), Type.String(), { description: "Decision criteria: status code to criterion, SUCCESS first" }),
+  ),
+  range_format: Type.Optional(Type.String({ description: "Format of the suggested range for a failed status" })),
+});
+
+const questionSchema = Type.Union(QUESTIONS.map((question) => Type.Literal(question)));
+
 const parameters = Type.Object({
   context: Type.String({
     description:
-      "Experiment context: task name, target qubit/coupling, parameters, and what a successful result looks like",
+      "Experiment context the background cannot know: task name, target qubit/coupling, swept range, previous values",
   }),
   question: Type.Optional(
-    Type.String({ description: "A specific question to answer about the plot" }),
+    Type.String({ description: "A specific point the analysis should address, on top of the standard questions" }),
   ),
   task_name: Type.Optional(
     Type.String({
       description:
-        "The calibration task's name as the host knows it (e.g. RandomizedBenchmarking). Lets the host supply its reference knowledge for the experiment.",
+        "The calibration task's name as the host knows it (e.g. CheckRabi). Lets the host supply the background and criteria for the experiment.",
     }),
   ),
   knowledge: Type.Optional(
     Type.String({
       description:
-        "Normally omitted: the host supplies the reference for `task_name` (expected result, review questions, failure patterns). Fill only to override it with text you were given.",
+        "Normally omitted: the host supplies the experiment background for `task_name` (what is swept, what a successful result looks like). Fill only to override it with text you were given.",
     }),
   ),
+  criteria: Type.Optional(criteriaSchema),
   images: Type.Optional(
     Type.Array(imageSchema, {
       description:
@@ -458,7 +760,13 @@ const parameters = Type.Object({
     Type.Array(exampleSchema, {
       maxItems: MAX_EXAMPLES,
       description:
-        "Normally omitted: the host supplies labeled examples when `use_examples` is true. Each is one or more real figures followed by the expert's diagnosis.",
+        "Normally omitted: the host supplies labeled examples when `use_examples` is true. Each is one or more real figures with the expert's answers.",
+    }),
+  ),
+  questions: Type.Optional(
+    Type.Array(questionSchema, {
+      minItems: 1,
+      description: "The questions to ask (default: all six, in order).",
     }),
   ),
 });
@@ -469,23 +777,24 @@ export interface EvaluationDetails {
   model?: ModelSpec;
   /** True when no calibration VLM was configured and the conversation's model judged instead. */
   fallback?: boolean;
+  /** The parsed answers by question, plus `success`. */
   diagnosis?: Record<string, unknown> | null;
-  /** The model's answer when it was not the requested JSON, or not all of it. */
-  raw?: string;
-  /** Diagnosis keys the parsed answer lacks, when it is incomplete. */
-  missing_keys?: string[];
-  /** Model calls made: 1, or 2 after an incomplete or refused first answer. */
+  /** Every answered question's text, as returned. */
+  answers?: Partial<Record<Question, string>>;
+  /** The questions asked, in order. */
+  questions?: Question[];
+  /** Questions asked that got no answer. */
+  missing_keys?: Question[];
+  /** Model calls made, retries included. */
   attempts?: number;
-  /** Whether decoding was constrained to the diagnosis schema. */
-  structured?: boolean;
-  /** Why the model stopped, as pi reports it. */
+  /** "length" when a call hit the output budget; otherwise how the calls ended. */
   stop_reason?: string;
-  /** The user turn the model was sent (reference, examples, context, question), so the host can show what it saw. */
+  /** Every model call made, by question: duration, stop reason, output tokens. */
+  calls?: CallRecord[];
+  /** The user turns the model was sent, one per question, so the host can show what it saw. */
   prompt?: string;
-  /** How many labeled examples were shown before the figure. */
+  /** How many labeled examples were shown. */
   examples_used?: number;
-  /** The system prompt the model was sent, so the host can show the whole request. */
-  system_prompt?: string;
 }
 
 function failure(spec: ModelSpec | undefined, text: string): AgentToolResult<EvaluationDetails> {
@@ -501,12 +810,12 @@ export default function qcalevalExtension(pi: ExtensionAPI) {
     name: "qcal_evaluate",
     label: "Evaluate Calibration Plot",
     description:
-      "Send a calibration experiment plot to the calibration-evaluation VLM and return its diagnosis: technical description, conclusion, significance, fit quality, extracted parameters, and success.",
+      "Send a calibration experiment plot to the calibration-evaluation VLM and return its diagnosis: technical description, outcome classification, significance, fit reliability, extracted parameters, and calibration status.",
     promptSnippet: "Diagnose a calibration plot with the dedicated calibration VLM",
     promptGuidelines: [
       "Use qcal_evaluate for every calibration figure you need to judge, instead of judging it yourself.",
-      "Pass the task name, target, and expected result as context so the diagnosis is specific.",
-      "Pass task_name when you know it: the host then supplies its reference knowledge for that task.",
+      "Pass the task name and target as context so the diagnosis is specific; do not pass fitted values, the model reads the plot.",
+      "Pass task_name when you know it: the host then supplies the background and criteria for that task.",
       "Set use_examples true only when the user asked for a judgement against labeled reference examples.",
       "Treat the returned diagnosis as the evaluation of record; add your own reasoning only on top of it.",
     ],
@@ -531,23 +840,28 @@ export default function qcalevalExtension(pi: ExtensionAPI) {
         return failure(spec, "no plot was given. Pass the figure in `images` or fetch it first.");
       }
 
-      const { content, prompt, examplesUsed } = buildUserContent(
-        params.context,
-        params.question,
-        images.map((image) => ({ data: image.data, mimeType: image.mimeType })),
-        params.knowledge,
-        params.examples ?? [],
-      );
-      const answer = await askForDiagnosis(
-        (structured) =>
+      const requested = new Set<string>(params.questions ?? []);
+      const questions: Question[] = requested.size > 0 ? QUESTIONS.filter((question) => requested.has(question)) : [...QUESTIONS];
+      const input: PromptInput = {
+        background: params.knowledge,
+        context: params.context,
+        question: params.question,
+        criteria: params.criteria,
+        images: images.map((image) => ({ data: image.data, mimeType: image.mimeType })),
+        examples: params.examples ?? [],
+      };
+      const turns = new Map(questions.map((question) => [question, buildQuestionContent(question, input)]));
+      const examplesUsed = fitExamples(input.examples ?? []).length;
+      const answers = await askQuestions(
+        questions,
+        (question) =>
           ctx.modelRegistry.complete(
             model,
             {
-              systemPrompt: SYSTEM_PROMPT,
               messages: [
                 {
                   role: "user",
-                  content: content.map((block) =>
+                  content: turns.get(question)!.content.map((block) =>
                     block.type === "text"
                       ? { type: "text" as const, text: block.text }
                       : { type: "image" as const, data: block.data, mimeType: block.mimeType },
@@ -556,33 +870,29 @@ export default function qcalevalExtension(pi: ExtensionAPI) {
                 },
               ],
             },
-            {
-              signal,
-              maxTokens: outputBudget(model),
-              ...(structured ? { samplingParams: { response_format: RESPONSE_FORMAT } } : {}),
-            },
+            { signal, maxTokens: outputBudget(model) },
           ),
-        canConstrain(model),
       );
-      if ("error" in answer) return failure(spec, answer.error);
-      const { text, diagnosis, missing } = answer;
-      const complete = diagnosis !== undefined && missing.length === 0;
+      if (answers.missing.length === questions.length) {
+        const error = Object.values(answers.errors)[0];
+        return failure(spec, error ?? "the model answered none of the questions");
+      }
       return {
-        content: [{ type: "text", text: diagnosis ? JSON.stringify(diagnosis, null, 2) : text }],
+        content: [{ type: "text", text: JSON.stringify(answers.diagnosis, null, 2) }],
         details: {
           model: spec,
           ...(fallback ? { fallback: true } : {}),
-          diagnosis: diagnosis ?? null,
-          ...(complete ? {} : { raw: text }),
-          ...(diagnosis && !complete ? { missing_keys: missing } : {}),
-          attempts: answer.attempts,
-          structured: answer.structured,
-          stop_reason: answer.stopReason,
-          prompt,
-          system_prompt: SYSTEM_PROMPT,
+          diagnosis: answers.diagnosis,
+          answers: answers.answers,
+          questions: answers.questions,
+          ...(answers.missing.length > 0 ? { missing_keys: answers.missing } : {}),
+          attempts: answers.attempts,
+          stop_reason: answers.stopReason,
+          calls: answers.calls,
+          prompt: promptRecord(questions.map((question) => ({ question, prompt: turns.get(question)!.prompt }))),
           ...(examplesUsed ? { examples_used: examplesUsed } : {}),
         },
-        ...(answer.usage ? { usage: answer.usage } : {}),
+        ...(answers.usage ? { usage: answers.usage } : {}),
       };
     },
   });

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@orangekame3/pi-qcaleval",
-  "version": "0.0.2-beta.10",
+  "version": "0.0.2-beta.11",
   "description": "pi extension that sends calibration plots to a calibration-evaluation VLM such as NVIDIA Ising-Calibration and returns a structured diagnosis",
   "license": "Apache-2.0",
   "repository": {

package/skills/qcaleval/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: qcaleval
-description: Evaluate superconducting qubit calibration plots with the dedicated calibration VLM through the pi-qcaleval extension. Use when a calibration figure (Rabi, Ramsey, chevron, T1/T2, readout, spectroscopy, two-qubit tune-up) needs a fit-quality judgement, parameter extraction, or a success verdict.
+description: Evaluate superconducting qubit calibration plots with the dedicated calibration VLM through the pi-qcaleval extension. Use when a calibration figure (Rabi, Ramsey, chevron, T1/T2, readout, spectroscopy, two-qubit tune-up) needs an outcome classification, a fit-reliability judgement, parameter extraction, or a status verdict.
 ---
 
 <!-- Whether a calibration loop continues after an evaluation is the host's policy. This skill describes the tool, not when to stop. -->
@@ -8,40 +8,50 @@ description: Evaluate superconducting qubit calibration plots with the dedicated
 # Calibration plot evaluation
 
 `qcal_evaluate` sends a plot to a calibration-evaluation model (NVIDIA
-Ising-Calibration by default) and returns a JSON diagnosis with
-`technical_description`, `experimental_conclusion`, `experimental_significance`,
-`fit_quality`, `parameter_extraction`, and `success`.
+Ising-Calibration by default) and asks it the six QCalEval questions, each in
+its own request. The diagnosis it returns has one entry per question:
+`technical_description`, `experimental_conclusion` (outcome: expected /
+suboptimal / anomalous / apparatus), `experimental_significance`,
+`fit_reliability` (reliable / unreliable / no_fit), `parameter_extraction`,
+and `calibration_diagnosis` (a status code, a suggested range, notes), plus
+`success` (the status is SUCCESS).
 
 ## Preferred tools
 
-- `qcal_evaluate`: judge one or more calibration plots. Always give `context`:
-  the task name, the target (qubit or coupling), the swept parameter, and what a
-  successful result looks like. Add `question` when a specific point matters
-  (for example "is the chevron centred on the current frequency?").
+- `qcal_evaluate`: judge one or more calibration plots. Give `context` for what
+  the background cannot know: the task name, the target (qubit or coupling),
+  the swept range. Do not put fitted values in it; the model reads the plot.
+  Add `question` when a specific point matters (for example "is the chevron
+  centred on the current frequency?"); it is added to the significance question.
 - Pass `task_name` whenever you know the task (for example `CheckRabi`): the
-  host then supplies its own reference knowledge for that experiment, which is
-  more reliable than a summary you write yourself. Keep `context` for what the
-  reference cannot know: the target, the swept range, and previous values.
+  host then supplies the experiment background and the task's criteria (the
+  parameters to extract, the status codes), which is more reliable than a
+  summary you write yourself.
 - When the conversation holds an expected reference figure followed by the
   measured result (a task review opens this way), pass `max_images: 2` and say
   in `context` that the first figure is the reference and the last the
   measurement. The host then supplies both; the default is the newest figure only.
+- Pass `questions` to ask a subset, for example `["calibration_diagnosis"]`
+  when only the status matters.
 
 ## Labeled examples (in-context)
 
 - When the user asks to judge a plot against the labeled reference examples
   (in-context examples, few-shot), pass `use_examples: true` with `task_name`.
   The host supplies its expert-labeled example plots for that task, and the
-  model sees each example figure followed by its diagnosis before the figure
-  to judge, as in the QCalEval papers. Leave it off otherwise.
+  model sees them before the figure to judge, in the layout of the QCalEval
+  few-shot benchmark: each example's status for the status question, each
+  example's analysis for the significance question, one example's values for
+  parameter extraction. Leave it off otherwise.
 
 ## How to use the result
 
 - Treat the diagnosis as the evaluation of record; reason on top of it instead
   of re-judging the plot yourself.
-- Report `fit_quality` and `success` verbatim, then the extracted parameters.
-- When `success` is false, say what the diagnosis identified as the cause before
-  proposing a next step.
-- The model only sees the plot and the context you pass. If a conclusion needs
-  data that is not in the plot (previous values, device limits), fetch it and
-  include it in `context`.
+- Report the status, the fit reliability and the outcome verbatim, then the
+  extracted parameters and the suggested range when there is one.
+- When the status is not SUCCESS, say what the notes and the outcome reason
+  identified as the cause before proposing a next step.
+- The model only sees the plot, the background and the context you pass. If a
+  conclusion needs data that is not in the plot (previous values, device
+  limits), fetch it and include it in `context`.