@orangekame3/pi-qcaleval 0.0.0-stage → 0.0.2-beta.10

package/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright yyyy OQTOPUS Team
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

package/README.md

@@ -1,3 +1,98 @@
-# Temporary Holding Version
+# pi-qcaleval
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+A [pi](https://github.com/badlogic/pi-mono) extension that sends superconducting
+qubit calibration plots to a dedicated calibration-evaluation VLM and returns a
+structured diagnosis. The default model is
+[NVIDIA Ising-Calibration](https://build.nvidia.com/nvidia/ising-calibration-1.5-31b/modelcard)
+served by vLLM.
+
+The planner model keeps the conversation; `qcal_evaluate` is the tool it calls
+whenever a calibration figure needs a fit-quality judgement, parameter
+extraction, or a success verdict. The extension has no QDash dependency, so it
+works from the pi CLI and from the QDash Copilot runtime alike.
+
+## Tool
+
+`qcal_evaluate(context, question?, task_name?, knowledge?, images?, max_images?)` returns a JSON object with
+`technical_description`, `experimental_conclusion`, `experimental_significance`,
+`fit_quality`, `parameter_extraction`, and `success`. The parsed object is also
+stored in the tool result's `details.diagnosis`; when the model did not answer
+with JSON, or not with all of it, `details.raw` holds its text and
+`details.missing_keys` the keys it lacks.
+
+On a chat-completions API (vLLM, SGLang, llama.cpp) the request carries a
+`response_format` with the diagnosis schema, so decoding is constrained to the
+six keys; a server that refuses it is asked plainly. An answer that still
+lacks keys is asked for once more and the more complete one kept. The details
+record `attempts` (1 or 2), `structured` (whether decoding was constrained),
+and `stop_reason`.
+
+`images` are base64 image blocks. A host may fill them in from the most recent
+figures of the conversation, so the planner does not have to pass image bytes;
+`max_images` (default 1, up to 4) says how many of the newest figures to supply,
+for example 2 when an expected reference figure precedes the measured result.
+
+`knowledge` is the host's reference for the experiment: what a good result
+looks like, the questions a reviewer asks, and the common failure patterns. A
+host that keeps such knowledge per task fills it in from `task_name`, so the
+planner only names the task; the text is sent before the context. The exact
+user turn the model received (reference, context, question) is stored in
+`details.prompt`, so a host can show what the model saw.
+
+## Configuration
+
+No setting is required. The evaluation model is resolved in this order:
+
+1. `QCALEVAL_MODEL` (`provider/modelId`), an explicit override for pi CLI use;
+2. a configured calibration VLM, recognised by its id (`Ising-Calibration`);
+3. the conversation's own model, when it accepts images. The result is then
+   marked `fallback: true` in `details`, so a host without an Ising deployment
+   still gets a diagnosis from the vision model it already runs.
+
+The output budget follows the model's `maxTokens` (at most 4096).
+
+The provider must be declared in pi's `models.json` with its base URL and
+credential, for example:
+
+```json
+{
+  "providers": {
+    "vllm": {
+      "baseUrl": "https://llm.example/v1",
+      "api": "openai-completions",
+      "apiKey": "$VLLM_API_KEY",
+      "models": [{ "id": "nvidia/Ising-Calibration-1.5-31B-NVFP4", "contextWindow": 16384, "maxTokens": 2048, "input": ["text", "image"] }]
+    }
+  }
+}
+```
+
+Only the plot, a short system prompt, and the `context` text are sent, so the
+evaluation fits comfortably in a 16k-token deployment.
+
+## Installation
+
+```bash
+pi install npm:@orangekame3/pi-qcaleval
+```
+
+Releases are tagged `vX.Y.Z` and published to npm as `latest`; every push to
+`main` also publishes a `beta` prerelease (`npm:@orangekame3/pi-qcaleval@beta`)
+for hosts that want an unreleased build. Installing from git
+(`pi install git:github.com/orangekame3/pi-qcaleval`) also works.
+
+For development against the QDash Agent Runtime, clone this repository under
+`agent-runtime/extensions/` in the QDash checkout and start the runtime with
+`compose.extensions.yaml`; see the QDash Copilot architecture docs.
+
+## Development
+
+```bash
+bun install
+bun run check   # tsc --noEmit
+bun run test    # node --test
+```
+
+## License
+
+Apache-2.0

package/extensions/qcaleval.ts

@@ -0,0 +1,589 @@
+/**
+ * pi-qcaleval: evaluate calibration plots with a dedicated calibration VLM.
+ *
+ * The extension has no QDash dependency. It takes a plot (plus the experiment
+ * context the planner already knows, and the host's reference knowledge for
+ * the task when the host supplies one) and asks the evaluation model, which is
+ * configured separately from the conversation's model, for a diagnosis. The
+ * planner decides what to do with it; whether a calibration loop continues is
+ * the host's policy, not this tool's.
+ */
+
+import type { Api, AssistantMessage, ImageContent, Model, Usage } from "@earendil-works/pi-ai";
+import type {
+  AgentToolResult,
+  ExtensionAPI,
+  ExtensionToolContext,
+} from "@earendil-works/pi-coding-agent";
+import { Type, type Static } from "typebox";
+
+/** Figures per evaluation; a 16k-token deployment holds a few plots comfortably. */
+export const MAX_IMAGES = 4;
+
+/** Labeled example plots shown before the query, and the figures they may hold in total. */
+export const MAX_EXAMPLES = 4;
+export const MAX_EXAMPLE_IMAGES = 4;
+
+/** Model ids of models fine-tuned for calibration plots; preferred when configured. */
+export const CALIBRATION_MODEL_PATTERN = /ising-calibration/i;
+
+/** Output budget when the model declares none; a diagnosis is a few hundred tokens. */
+const DEFAULT_MAX_TOKENS = 2048;
+const MAX_OUTPUT_TOKENS = 4096;
+
+export interface ModelSpec {
+  provider: string;
+  modelId: string;
+}
+
+/** Parse an explicit `provider/modelId` override; the model id may contain slashes (nvidia/...). */
+export function parseModelSpec(value: string | undefined): ModelSpec | undefined {
+  const spec = (value ?? "").trim();
+  if (!spec) return undefined;
+  const slash = spec.indexOf("/");
+  if (slash <= 0 || slash === spec.length - 1) {
+    throw new Error(`QCALEVAL_MODEL must be provider/modelId, got "${spec}"`);
+  }
+  return { provider: spec.slice(0, slash), modelId: spec.slice(slash + 1) };
+}
+
+/** The slice of pi's model registry the resolution needs. */
+export interface ModelLookup {
+  find(provider: string, modelId: string): Model<Api> | undefined;
+  getAvailable(): Model<Api>[];
+}
+
+export type EvaluationModel =
+  | { model: Model<Api>; fallback: false }
+  | { model: Model<Api>; fallback: true }
+  | { model?: undefined; error: string };
+
+/**
+ * Pick the model that judges the plot, without any host-specific setting:
+ *
+ * 1. `QCALEVAL_MODEL` (`provider/modelId`), an explicit override for the pi CLI;
+ * 2. a configured calibration VLM, recognised by its id (Ising-Calibration);
+ * 3. the conversation's own model, when it accepts images.
+ *
+ * A host that declares the calibration model in its models.json gets it
+ * everywhere; one that does not (no Ising deployment) still gets a diagnosis
+ * from the vision model it already runs, marked as a fallback.
+ */
+export function resolveEvaluationModel(
+  registry: ModelLookup,
+  current: Model<Api> | undefined,
+  override: string | undefined,
+): EvaluationModel {
+  const spec = parseModelSpec(override);
+  if (spec) {
+    const model = registry.find(spec.provider, spec.modelId);
+    return model
+      ? { model, fallback: false }
+      : { error: `model ${spec.provider}/${spec.modelId} is not configured (QCALEVAL_MODEL)` };
+  }
+  const calibration = registry.getAvailable().find((model) => CALIBRATION_MODEL_PATTERN.test(model.id));
+  if (calibration) return { model: calibration, fallback: false };
+  if (current?.input.includes("image")) return { model: current, fallback: true };
+  return {
+    error:
+      "no evaluation model: declare a calibration VLM (Ising-Calibration) in models.json, or run with a vision model",
+  };
+}
+
+/** Output budget for one evaluation, within what the model allows. */
+export function outputBudget(model: Pick<Model<Api>, "maxTokens">): number {
+  return model.maxTokens > 0 ? Math.min(model.maxTokens, MAX_OUTPUT_TOKENS) : DEFAULT_MAX_TOKENS;
+}
+
+/** Categories the Ising-Calibration model card describes; asked for in this order. */
+export const DIAGNOSIS_SECTIONS = [
+  "technical_description",
+  "experimental_conclusion",
+  "experimental_significance",
+  "fit_quality",
+  "parameter_extraction",
+  "success",
+] as const;
+
+/**
+ * The diagnosis as a JSON schema: the six keys, all required. Sent as the
+ * response format to OpenAI-compatible servers that constrain decoding to a
+ * schema (vLLM, SGLang, llama.cpp). Asked for JSON in the prompt alone, the
+ * calibration model now and then closes the object after the first key or
+ * nests quotes that break it; constrained decoding rules both out.
+ */
+export const DIAGNOSIS_SCHEMA = {
+  type: "object",
+  properties: {
+    technical_description: { type: "string" },
+    experimental_conclusion: { type: "string" },
+    experimental_significance: { type: "string" },
+    fit_quality: { type: "string" },
+    parameter_extraction: {
+      anyOf: [{ type: "object", additionalProperties: true }, { type: "string" }, { type: "null" }],
+    },
+    success: { type: "boolean" },
+  },
+  required: [...DIAGNOSIS_SECTIONS],
+  additionalProperties: false,
+} as const;
+
+/** `response_format` of a chat completion constrained to the diagnosis schema. */
+export const RESPONSE_FORMAT = {
+  type: "json_schema",
+  json_schema: { name: "calibration_diagnosis", schema: DIAGNOSIS_SCHEMA },
+} as const;
+
+/**
+ * Whether the model's API takes `response_format` in the request body. Pi
+ * merges `samplingParams` into the body of chat completions only; the
+ * responses API spells the format differently and other APIs ignore it.
+ */
+export function canConstrain(model: Pick<Model<Api>, "api">): boolean {
+  return model.api === "openai-completions";
+}
+
+/** The diagnosis keys an answer lacks; all six when there is no diagnosis. */
+export function missingKeys(diagnosis: Record<string, unknown> | undefined): string[] {
+  if (!diagnosis) return [...DIAGNOSIS_SECTIONS];
+  return DIAGNOSIS_SECTIONS.filter((key) => !(key in diagnosis));
+}
+
+export const SYSTEM_PROMPT = [
+  "You evaluate superconducting qubit calibration experiment plots.",
+  "Answer with one JSON object and nothing else, with these keys:",
+  '  "technical_description": what the plot shows (axes, traces, fit);',
+  '  "experimental_conclusion": what the data says about the experiment;',
+  '  "experimental_significance": what it means for the calibration;',
+  '  "fit_quality": "good" | "marginal" | "poor" and why;',
+  '  "parameter_extraction": extracted values with units, or null;',
+  '  "success": true | false, whether the experiment achieved its goal.',
+  "Base every statement on the plot, the reference, and the given context only.",
+].join("\n");
+
+/** One labeled example: its figure(s) followed by the expert's diagnosis. */
+export interface LabeledExample {
+  label: string;
+  images: ReadonlyArray<{ data: string; mimeType: string }>;
+}
+
+export type UserContentBlock =
+  | { type: "text"; text: string }
+  | { type: "image"; data: string; mimeType: string };
+
+/**
+ * The single user turn sent to the evaluation model, as the QCalEval papers
+ * lay it out: the reference for the experiment, then each labeled example as
+ * its figure followed by its diagnosis, then the context and question, then
+ * the figure to judge. Everything is one ordered content array; images are
+ * native blocks, never placeholders in the text.
+ *
+ * `prompt` is the same turn as text, with `[figure]` where an image was, for
+ * the record of what the model saw.
+ */
+export function buildUserContent(
+  context: string,
+  question: string | undefined,
+  images: ReadonlyArray<{ data: string; mimeType: string }>,
+  knowledge?: string,
+  examples: ReadonlyArray<LabeledExample> = [],
+): { content: UserContentBlock[]; prompt: string; examplesUsed: number } {
+  const content: UserContentBlock[] = [];
+  const shown = fitExamples(examples);
+  const text = (lines: string[]) => {
+    content.push({ type: "text", text: lines.join("\n") });
+  };
+  const image = (figure: { data: string; mimeType: string }) => {
+    content.push({ type: "image", data: figure.data, mimeType: figure.mimeType });
+  };
+
+  const opening: string[] = [];
+  if (knowledge?.trim()) {
+    opening.push("Reference for this experiment (expected plot, outcomes, parameters):", knowledge.trim(), "");
+  }
+  if (shown.length > 0) {
+    opening.push(
+      `${shown.length} labeled example${shown.length === 1 ? "" : "s"} of this experiment follow, each figure followed by the diagnosis an expert validated. Use them as the standard for the figure to judge, which comes last.`,
+    );
+  }
+  if (opening.length > 0) text(opening);
+
+  shown.forEach((example, index) => {
+    text(["", `Example ${index + 1} of ${shown.length}:`]);
+    for (const figure of example.images) image(figure);
+    text([`Diagnosis of example ${index + 1}:`, example.label.trim()]);
+  });
+
+  const closing: string[] = [];
+  if (shown.length > 0) closing.push("", "Now the figure to judge, evaluated independently of the examples.");
+  closing.push("Experiment context:", context.trim() || "(none)");
+  if (images.length > 1) {
+    closing.push(
+      "",
+      `${images.length} figures follow, in the order the context describes; the last one is the measured result to judge.`,
+    );
+  }
+  if (question?.trim()) closing.push("", "Specific question:", question.trim());
+  // Repeated here because a long reference ahead of the plot pulls the model
+  // towards prose; the last instruction it reads is the output format.
+  closing.push("", "Answer with the JSON object described in the system prompt, and nothing else.");
+  text(closing);
+  for (const figure of images) image(figure);
+
+  const prompt = content
+    .map((block) => (block.type === "text" ? block.text : "[figure]"))
+    .join("\n")
+    .replace(/\n{3,}/g, "\n\n")
+    .trim();
+  return { content, prompt, examplesUsed: shown.length };
+}
+
+/** Keep examples in order within the example and figure budgets; later ones are dropped. */
+export function fitExamples(examples: ReadonlyArray<LabeledExample>): LabeledExample[] {
+  const kept: LabeledExample[] = [];
+  let figures = 0;
+  for (const example of examples) {
+    if (kept.length >= MAX_EXAMPLES) break;
+    if (!example.label?.trim() || !example.images?.length) continue;
+    if (figures + example.images.length > MAX_EXAMPLE_IMAGES) break;
+    figures += example.images.length;
+    kept.push(example);
+  }
+  return kept;
+}
+
+/** The user turn as text only (no examples), kept for callers that render the prompt. */
+export function buildUserPrompt(
+  context: string,
+  question: string | undefined,
+  imageCount = 1,
+  knowledge?: string,
+): string {
+  const placeholder = { data: "", mimeType: "image/png" };
+  return buildUserContent(context, question, Array(imageCount).fill(placeholder), knowledge)
+    .content.filter((block) => block.type === "text")
+    .map((block) => (block as { text: string }).text)
+    .join("\n")
+    .replace(/\n{3,}/g, "\n\n")
+    .trim();
+}
+
+/** Pull the text blocks of the assistant answer, dropping thinking. */
+export function answerText(message: AssistantMessage): string {
+  return message.content
+    .map((block) => (block.type === "text" ? block.text : ""))
+    .join("")
+    .trim();
+}
+
+/**
+ * Parse the model's JSON object, tolerating a code fence, a "thought" preamble
+ * or trailing commentary around it: the first balanced `{...}` is taken.
+ */
+export function parseDiagnosis(text: string): Record<string, unknown> | undefined {
+  const start = text.indexOf("{");
+  if (start < 0) return undefined;
+  let depth = 0;
+  let inString = false;
+  for (let i = start; i < text.length; i++) {
+    const ch = text[i];
+    if (inString) {
+      if (ch === "\\") i++;
+      else if (ch === '"') inString = false;
+      continue;
+    }
+    if (ch === '"') inString = true;
+    else if (ch === "{") depth++;
+    else if (ch === "}" && --depth === 0) {
+      return parseObject(text.slice(start, i + 1));
+    }
+  }
+  return undefined;
+}
+
+/**
+ * Parse one JSON object. The model writes LaTeX such as `\pi` inside strings,
+ * which is an invalid JSON escape, so a failed strict parse is retried with
+ * every backslash that does not start a JSON escape doubled.
+ */
+function parseObject(json: string): Record<string, unknown> | undefined {
+  for (const candidate of [json, json.replace(/\\(?!["\\/bfnrtu])/g, "\\\\")]) {
+    try {
+      const parsed: unknown = JSON.parse(candidate);
+      if (typeof parsed === "object" && parsed !== null && !Array.isArray(parsed)) {
+        return parsed as Record<string, unknown>;
+      }
+      return undefined;
+    } catch {
+      // Try the lenient candidate next.
+    }
+  }
+  return undefined;
+}
+
+/** One answer of the evaluation model, read for the record. */
+export interface Answer {
+  text: string;
+  diagnosis?: Record<string, unknown>;
+  /** Diagnosis keys the answer lacks; empty when it is complete. */
+  missing: string[];
+  /** Model calls made for this answer: 1, or 2 after an incomplete or refused first one. */
+  attempts: number;
+  /** Whether decoding was constrained to the diagnosis schema. */
+  structured: boolean;
+  stopReason: AssistantMessage["stopReason"];
+  usage?: Usage;
+}
+
+function readAnswer(message: AssistantMessage, structured: boolean): Answer {
+  const text = answerText(message);
+  const diagnosis = parseDiagnosis(text);
+  return {
+    text,
+    diagnosis,
+    missing: missingKeys(diagnosis),
+    attempts: 1,
+    structured,
+    stopReason: message.stopReason,
+    usage: message.usage,
+  };
+}
+
+/** Token usage of two calls together, so a retried answer is billed as both. */
+export function addUsage(a: Usage | undefined, b: Usage | undefined): Usage | undefined {
+  if (!a || !b) return a ?? b;
+  const sum = (x: number | undefined, y: number | undefined) => (x ?? 0) + (y ?? 0);
+  return {
+    ...a,
+    input: sum(a.input, b.input),
+    output: sum(a.output, b.output),
+    cacheRead: sum(a.cacheRead, b.cacheRead),
+    cacheWrite: sum(a.cacheWrite, b.cacheWrite),
+    ...(a.reasoning === undefined && b.reasoning === undefined
+      ? {}
+      : { reasoning: sum(a.reasoning, b.reasoning) }),
+    totalTokens: sum(a.totalTokens, b.totalTokens),
+    cost: {
+      input: sum(a.cost?.input, b.cost?.input),
+      output: sum(a.cost?.output, b.cost?.output),
+      cacheRead: sum(a.cost?.cacheRead, b.cost?.cacheRead),
+      cacheWrite: sum(a.cost?.cacheWrite, b.cost?.cacheWrite),
+      total: sum(a.cost?.total, b.cost?.total),
+    },
+  };
+}
+
+/**
+ * Ask the model for the diagnosis, in at most two calls. The first call is
+ * constrained to the schema when the API allows it; a server that refuses the
+ * response format (an older vLLM, a gateway) gets the plain request instead.
+ * An answer that lacks diagnosis keys is asked for once more, since the model
+ * usually completes the object on a second sample, and the more complete of
+ * the two answers is kept. Usage counts every call made.
+ */
+export async function askForDiagnosis(
+  call: (structured: boolean) => Promise<AssistantMessage>,
+  constrain: boolean,
+): Promise<Answer | { error: string }> {
+  let structured = constrain;
+  let first = await call(structured);
+  let attempts = 1;
+  let usage: Usage | undefined = first.usage;
+  if (first.stopReason === "error" && structured) {
+    structured = false;
+    first = await call(structured);
+    attempts += 1;
+    usage = addUsage(usage, first.usage);
+  }
+  if (first.stopReason === "error") return { error: first.errorMessage ?? "model error" };
+  const answer = { ...readAnswer(first, structured), attempts, usage };
+  if (answer.missing.length === 0 || attempts === 2) return answer;
+  const second = await call(structured);
+  const retried = second.stopReason === "error" ? undefined : readAnswer(second, structured);
+  const kept = retried && retried.missing.length <= answer.missing.length ? retried : answer;
+  return { ...kept, attempts: 2, usage: addUsage(first.usage, second.usage) };
+}
+
+const imageSchema = Type.Object({
+  data: Type.String({ description: "Base64 image bytes (no data: prefix)" }),
+  mimeType: Type.String({ description: "image/png or image/jpeg" }),
+});
+
+const exampleSchema = Type.Object({
+  label: Type.String({ description: "The expert's diagnosis of the example figure(s)" }),
+  images: Type.Array(imageSchema, { minItems: 1 }),
+});
+
+const parameters = Type.Object({
+  context: Type.String({
+    description:
+      "Experiment context: task name, target qubit/coupling, parameters, and what a successful result looks like",
+  }),
+  question: Type.Optional(
+    Type.String({ description: "A specific question to answer about the plot" }),
+  ),
+  task_name: Type.Optional(
+    Type.String({
+      description:
+        "The calibration task's name as the host knows it (e.g. RandomizedBenchmarking). Lets the host supply its reference knowledge for the experiment.",
+    }),
+  ),
+  knowledge: Type.Optional(
+    Type.String({
+      description:
+        "Normally omitted: the host supplies the reference for `task_name` (expected result, review questions, failure patterns). Fill only to override it with text you were given.",
+    }),
+  ),
+  images: Type.Optional(
+    Type.Array(imageSchema, {
+      description:
+        "Normally omitted: the host supplies the newest figures of the conversation. Fill only with real base64 image bytes you were given; never with placeholders or descriptions.",
+    }),
+  ),
+  max_images: Type.Optional(
+    Type.Integer({
+      minimum: 1,
+      maximum: MAX_IMAGES,
+      description:
+        "How many of the newest figures the host should supply when `images` is omitted (default 1). Use 2 when an expected reference figure precedes the measured one, and describe their order in `context`.",
+    }),
+  ),
+  use_examples: Type.Optional(
+    Type.Boolean({
+      description:
+        "Set true only when the user asked to judge against labeled reference examples: the host then supplies its expert-labeled example plots for `task_name` (in-context examples). Default false.",
+    }),
+  ),
+  examples: Type.Optional(
+    Type.Array(exampleSchema, {
+      maxItems: MAX_EXAMPLES,
+      description:
+        "Normally omitted: the host supplies labeled examples when `use_examples` is true. Each is one or more real figures followed by the expert's diagnosis.",
+    }),
+  ),
+});
+
+/** Stored on the tool result; the host reads `diagnosis` without reparsing text. */
+export interface EvaluationDetails {
+  /** The model that judged the plot; absent when none could be resolved. */
+  model?: ModelSpec;
+  /** True when no calibration VLM was configured and the conversation's model judged instead. */
+  fallback?: boolean;
+  diagnosis?: Record<string, unknown> | null;
+  /** The model's answer when it was not the requested JSON, or not all of it. */
+  raw?: string;
+  /** Diagnosis keys the parsed answer lacks, when it is incomplete. */
+  missing_keys?: string[];
+  /** Model calls made: 1, or 2 after an incomplete or refused first answer. */
+  attempts?: number;
+  /** Whether decoding was constrained to the diagnosis schema. */
+  structured?: boolean;
+  /** Why the model stopped, as pi reports it. */
+  stop_reason?: string;
+  /** The user turn the model was sent (reference, examples, context, question), so the host can show what it saw. */
+  prompt?: string;
+  /** How many labeled examples were shown before the figure. */
+  examples_used?: number;
+  /** The system prompt the model was sent, so the host can show the whole request. */
+  system_prompt?: string;
+}
+
+function failure(spec: ModelSpec | undefined, text: string): AgentToolResult<EvaluationDetails> {
+  return {
+    content: [{ type: "text", text: `qcal_evaluate: ${text}` }],
+    details: spec ? { model: spec } : {},
+    isError: true,
+  };
+}
+
+export default function qcalevalExtension(pi: ExtensionAPI) {
+  pi.registerTool<typeof parameters, EvaluationDetails>({
+    name: "qcal_evaluate",
+    label: "Evaluate Calibration Plot",
+    description:
+      "Send a calibration experiment plot to the calibration-evaluation VLM and return its diagnosis: technical description, conclusion, significance, fit quality, extracted parameters, and success.",
+    promptSnippet: "Diagnose a calibration plot with the dedicated calibration VLM",
+    promptGuidelines: [
+      "Use qcal_evaluate for every calibration figure you need to judge, instead of judging it yourself.",
+      "Pass the task name, target, and expected result as context so the diagnosis is specific.",
+      "Pass task_name when you know it: the host then supplies its reference knowledge for that task.",
+      "Set use_examples true only when the user asked for a judgement against labeled reference examples.",
+      "Treat the returned diagnosis as the evaluation of record; add your own reasoning only on top of it.",
+    ],
+    parameters,
+    async execute(
+      _toolCallId: string,
+      params: Static<typeof parameters>,
+      signal: AbortSignal | undefined,
+      _onUpdate: unknown,
+      ctx: ExtensionToolContext,
+    ): Promise<AgentToolResult<EvaluationDetails>> {
+      const resolved = resolveEvaluationModel(ctx.modelRegistry, ctx.model, process.env.QCALEVAL_MODEL);
+      if (!resolved.model) return failure(undefined, resolved.error);
+      const { model, fallback } = resolved;
+      const spec: ModelSpec = { provider: model.provider, modelId: model.id };
+      const images: ImageContent[] = (params.images ?? []).map((image) => ({
+        type: "image",
+        data: image.data,
+        mimeType: image.mimeType,
+      }));
+      if (images.length === 0) {
+        return failure(spec, "no plot was given. Pass the figure in `images` or fetch it first.");
+      }
+
+      const { content, prompt, examplesUsed } = buildUserContent(
+        params.context,
+        params.question,
+        images.map((image) => ({ data: image.data, mimeType: image.mimeType })),
+        params.knowledge,
+        params.examples ?? [],
+      );
+      const answer = await askForDiagnosis(
+        (structured) =>
+          ctx.modelRegistry.complete(
+            model,
+            {
+              systemPrompt: SYSTEM_PROMPT,
+              messages: [
+                {
+                  role: "user",
+                  content: content.map((block) =>
+                    block.type === "text"
+                      ? { type: "text" as const, text: block.text }
+                      : { type: "image" as const, data: block.data, mimeType: block.mimeType },
+                  ),
+                  timestamp: Date.now(),
+                },
+              ],
+            },
+            {
+              signal,
+              maxTokens: outputBudget(model),
+              ...(structured ? { samplingParams: { response_format: RESPONSE_FORMAT } } : {}),
+            },
+          ),
+        canConstrain(model),
+      );
+      if ("error" in answer) return failure(spec, answer.error);
+      const { text, diagnosis, missing } = answer;
+      const complete = diagnosis !== undefined && missing.length === 0;
+      return {
+        content: [{ type: "text", text: diagnosis ? JSON.stringify(diagnosis, null, 2) : text }],
+        details: {
+          model: spec,
+          ...(fallback ? { fallback: true } : {}),
+          diagnosis: diagnosis ?? null,
+          ...(complete ? {} : { raw: text }),
+          ...(diagnosis && !complete ? { missing_keys: missing } : {}),
+          attempts: answer.attempts,
+          structured: answer.structured,
+          stop_reason: answer.stopReason,
+          prompt,
+          system_prompt: SYSTEM_PROMPT,
+          ...(examplesUsed ? { examples_used: examplesUsed } : {}),
+        },
+        ...(answer.usage ? { usage: answer.usage } : {}),
+      };
+    },
+  });
+}

package/package.json

@@ -1,6 +1,52 @@
 {
   "name": "@orangekame3/pi-qcaleval",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "0.0.2-beta.10",
+  "description": "pi extension that sends calibration plots to a calibration-evaluation VLM such as NVIDIA Ising-Calibration and returns a structured diagnosis",
+  "license": "Apache-2.0",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/orangekame3/pi-qcaleval.git"
+  },
+  "homepage": "https://github.com/orangekame3/pi-qcaleval#readme",
+  "keywords": [
+    "pi-package",
+    "pi-extension",
+    "calibration",
+    "qdash"
+  ],
+  "publishConfig": {
+    "access": "public"
+  },
+  "engines": {
+    "node": ">=20.0.0"
+  },
+  "type": "module",
+  "files": [
+    "extensions",
+    "skills"
+  ],
+  "pi": {
+    "extensions": [
+      "./extensions/qcaleval.ts"
+    ],
+    "skills": [
+      "./skills"
+    ]
+  },
+  "scripts": {
+    "check": "tsc --noEmit",
+    "test": "rm -rf .test-dist && tsc --noEmit false --outDir .test-dist && node --test tests/*.test.mjs; status=$?; rm -rf .test-dist; exit $status"
+  },
+  "peerDependencies": {
+    "@earendil-works/pi-ai": "*",
+    "@earendil-works/pi-coding-agent": "*",
+    "typebox": "*"
+  },
+  "devDependencies": {
+    "@earendil-works/pi-ai": "1.0.3",
+    "@earendil-works/pi-coding-agent": "1.0.3",
+    "@types/node": "^22.0.0",
+    "typebox": "^1.0.55",
+    "typescript": "^5.8.0"
+  }
+}

package/skills/qcaleval/SKILL.md

@@ -0,0 +1,47 @@
+---
+name: qcaleval
+description: Evaluate superconducting qubit calibration plots with the dedicated calibration VLM through the pi-qcaleval extension. Use when a calibration figure (Rabi, Ramsey, chevron, T1/T2, readout, spectroscopy, two-qubit tune-up) needs a fit-quality judgement, parameter extraction, or a success verdict.
+---
+
+<!-- Whether a calibration loop continues after an evaluation is the host's policy. This skill describes the tool, not when to stop. -->
+
+# Calibration plot evaluation
+
+`qcal_evaluate` sends a plot to a calibration-evaluation model (NVIDIA
+Ising-Calibration by default) and returns a JSON diagnosis with
+`technical_description`, `experimental_conclusion`, `experimental_significance`,
+`fit_quality`, `parameter_extraction`, and `success`.
+
+## Preferred tools
+
+- `qcal_evaluate`: judge one or more calibration plots. Always give `context`:
+  the task name, the target (qubit or coupling), the swept parameter, and what a
+  successful result looks like. Add `question` when a specific point matters
+  (for example "is the chevron centred on the current frequency?").
+- Pass `task_name` whenever you know the task (for example `CheckRabi`): the
+  host then supplies its own reference knowledge for that experiment, which is
+  more reliable than a summary you write yourself. Keep `context` for what the
+  reference cannot know: the target, the swept range, and previous values.
+- When the conversation holds an expected reference figure followed by the
+  measured result (a task review opens this way), pass `max_images: 2` and say
+  in `context` that the first figure is the reference and the last the
+  measurement. The host then supplies both; the default is the newest figure only.
+
+## Labeled examples (in-context)
+
+- When the user asks to judge a plot against the labeled reference examples
+  (in-context examples, few-shot), pass `use_examples: true` with `task_name`.
+  The host supplies its expert-labeled example plots for that task, and the
+  model sees each example figure followed by its diagnosis before the figure
+  to judge, as in the QCalEval papers. Leave it off otherwise.
+
+## How to use the result
+
+- Treat the diagnosis as the evaluation of record; reason on top of it instead
+  of re-judging the plot yourself.
+- Report `fit_quality` and `success` verbatim, then the extracted parameters.
+- When `success` is false, say what the diagnosis identified as the cause before
+  proposing a next step.
+- The model only sees the plot and the context you pass. If a conclusion needs
+  data that is not in the plot (previous values, device limits), fetch it and
+  include it in `context`.