create-quorum-router 0.1.8 → 0.1.10

package/README.md

@@ -17,7 +17,7 @@ deno task intake
 deno task supabase:status
 ```
 
-Current package version: `create-quorum-router@0.1.8`. Releases are published
+Current package version: `create-quorum-router@0.1.10`. Releases are published
 from an immutable Git tag through GitHub Actions OIDC Trusted Publishing.
 
 ## What the generated project supports
@@ -27,7 +27,9 @@ API.
 
 `deno task calibration:demo` runs the bundled calibration-by-task API against
 deterministic local observations. The report is advisory-only and is not
-connected to routing weights, provider eligibility, or execution.
+connected to routing weights, provider eligibility, or execution. The command
+does not call provider APIs; on a new Deno installation, its first run resolves
+the pinned Zod dependency before execution.
 
 `deno task intake` is the first real setup command. It detects local provider
 wrappers, checks OAuth/session status, runs safe list-only model inventory where
@@ -54,9 +56,15 @@ RUN_EXTERNAL_MODEL_DOGFOOD=1 deno task route:once --prompt "https://github.com/s
 # GitHub URL prompts fetch bounded repository context before invoking the selected provider.
 # Only use this with repositories you are allowed to send to that provider.
 RUN_EXTERNAL_MODEL_DOGFOOD=1 deno task best-route --prompt "Choose the safest launch copy."
-RUN_EXTERNAL_MODEL_DOGFOOD=1 RUN_EXPERIMENTAL_AGENT_CHAT=1 deno task agent-chat --prompt "Review this launch plan."
+RUN_EXTERNAL_MODEL_DOGFOOD=1 RUN_AGENT_CHAT=1 deno task agent-chat --prompt "Review this launch plan."
 ```
 
+`route:once` and `best-route` also accept
+`--calibration-evidence ./calibration-evidence.json`. The generated CLI
+caller-attested outcomes before invoking providers and stores the advisory
+metrics with hashed task/source identifiers in the route trace without changing
+routing authority.
+
 `auth:login` does not ask for API keys as the primary path. If OAuth/browser
 login is not wired in the generated scaffold, it fails closed and tells the user
 to use an installed provider CLI login, then rerun `deno task intake`.

package/bin/create-quorum-router.js

@@ -3,7 +3,7 @@ const fs = require("fs");
 const path = require("path");
 const { spawnSync } = require("child_process");
 
-const VERSION = "0.1.8";
+const VERSION = "0.1.10";
 const SUPPORTED_TEMPLATES = new Set(["basic"]);
 
 function usage() {
@@ -150,7 +150,7 @@ function main() {
     console.log("Then verify with: deno --version");
   }
   console.log(
-    "  # Optional one-shot real provider dogfood after intake reports a usable OAuth/session/wrapper provider:",
+    "  # Optional one-shot real provider dogfood after live probe verifies authentication:",
   );
   console.log(
     '  RUN_EXTERNAL_MODEL_DOGFOOD=1 deno task route:once --prompt "Review this README for risky claims."',
@@ -163,7 +163,7 @@ function main() {
     "  # Optional conversation-only Agent Chat stays explicit opt-in:",
   );
   console.log(
-    '  RUN_EXTERNAL_MODEL_DOGFOOD=1 RUN_EXPERIMENTAL_AGENT_CHAT=1 deno task agent-chat --prompt "Review this launch plan."',
+    '  RUN_EXTERNAL_MODEL_DOGFOOD=1 RUN_AGENT_CHAT=1 deno task agent-chat --prompt "Review this launch plan."',
   );
   console.log("");
   console.log(

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "create-quorum-router",
-  "version": "0.1.8",
+  "version": "0.1.10",
   "description": "Create a local QuorumRouter project.",
   "license": "MIT",
   "bin": {

package/templates/basic/README.md

@@ -1,7 +1,7 @@
 # QuorumRouter generated workspace
 
 This generated workspace contains the MIT-licensed QuorumRouter current release.
-npm latest targets v0.1.8.
+npm latest targets v0.1.10.
 
 QuorumRouter is **MIT**. It is **open source**. Commercial and production use
 are permitted under the MIT License.
@@ -51,7 +51,8 @@ deno task supabase:status
 `calibration:demo` exercises the bundled calibration-by-task API with local
 fixture observations. Calibration reports are advisory-only: the scaffold does
 not use them to change routing weights, ranks, provider eligibility, quorum, or
-execution.
+execution. The command does not call provider APIs; on a new Deno installation,
+its first run resolves the pinned Zod dependency before execution.
 
 `intake` detects local provider wrappers, checks OAuth/session status, runs safe
 model inventory/list-only probes where possible, writes local health traces
@@ -93,20 +94,24 @@ enabled by this audit integration.
 
 ## Real provider dogfood commands
 
-Run only after `intake` reports a usable OAuth/session/wrapper provider:
+Run only after a live probe verifies provider authentication:
 
 ```bash
 RUN_EXTERNAL_MODEL_DOGFOOD=1 deno task route:once --prompt "Review this README for risky claims."
 RUN_EXTERNAL_MODEL_DOGFOOD=1 deno task best-route --prompt "Choose the safest launch copy."
-RUN_EXTERNAL_MODEL_DOGFOOD=1 RUN_EXPERIMENTAL_AGENT_CHAT=1 deno task agent-chat --prompt "Review this launch plan."
+RUN_EXTERNAL_MODEL_DOGFOOD=1 RUN_AGENT_CHAT=1 deno task agent-chat --prompt "Review this launch plan."
 ```
 
 Behavior:
 
 - `route:once` requires `RUN_EXTERNAL_MODEL_DOGFOOD=1`.
 - `best-route` requires `RUN_EXTERNAL_MODEL_DOGFOOD=1`.
+- `route:once` and `best-route` accept
+  `--calibration-evidence ./calibration-evidence.json`; evidence is validated
+  before provider invocation. The trace stores advisory metrics with hashed task
+  and source identifiers, not raw labels.
 - `agent-chat` requires both `RUN_EXTERNAL_MODEL_DOGFOOD=1` and
-  `RUN_EXPERIMENTAL_AGENT_CHAT=1`.
+  `RUN_AGENT_CHAT=1`.
 - Live Agent Chat requires at least two distinct working provider/model
   identities. It passes the bounded transcript to alternating models, prints
   each response and `replying to` lineage as it arrives, and stores turns in
@@ -168,10 +173,9 @@ RUN_EXTERNAL_MODEL_DOGFOOD=1 \
 ```
 
 Supported provider aliases include `grok-cli`, `grok`, `xai`, `xAI`, `OpenAI`,
-`codex-cli`, `claude-code`, `gemini-cli`, `devin-cli`, and `qwen-cli`. Wrapper
-invocations use argv arrays, closed stdin, timeout guards, and sanitized
-stdout/stderr; CLI banners or auth/runtime errors are not accepted as valid
-model answers.
+`codex-cli`, `claude-code`, `gemini-cli`, and `devin-cli`. Wrapper invocations
+use argv arrays, closed stdin, timeout guards, and sanitized stdout/stderr; CLI
+banners or auth/runtime errors are not accepted as valid model answers.
 
 ## Auth and inventory
 

package/templates/basic/router.config.example.json

@@ -18,8 +18,8 @@
   },
   "runtime_boundaries": {
     "best_route_direct": "production-ready best-answer routing",
-    "agent_chat": "experimental explicit opt-in only",
-    "production_autonomous_runtime": false,
+    "agent_chat": "explicit opt-in conversation mode; no mutation authority",
+    "safeloop_execution": "production-capable only with explicit execution authority and approval",
     "live_supabase_agent_bus_writes": false,
     "service_role_runtime": false
   }

package/templates/basic/src/auth.ts

@@ -18,9 +18,11 @@ export async function runAuthStatus(): Promise<void> {
   console.log("Credential values printed: false");
   console.log(`Env fallback configured: ${envFallbackConfigured()}`);
   printInventoryTable(inventory);
-  const usableEntries = invokableEntries(inventory);
-  if (usableEntries.length === 0) {
-    console.log("Status: missing usable OAuth/session/wrapper provider");
+  const invokable = invokableEntries(inventory);
+  if (invokable.length === 0) {
+    console.log(
+      "Status: no invokable OAuth/session/wrapper provider discovered",
+    );
     console.log("Next action: deno task auth:login");
   } else if (authMode === "env") {
     console.log("Status: explicit private env fallback is configured");
@@ -29,7 +31,7 @@ export async function runAuthStatus(): Promise<void> {
     );
   } else {
     console.log(
-      "Status: at least one OAuth/session/wrapper provider appears usable",
+      "Status: at least one provider command is invokable; live authentication is not verified by auth:status",
     );
     console.log(
       'Next action: RUN_EXTERNAL_MODEL_DOGFOOD=1 deno task route:once --prompt "Review this README for risky claims."',

package/templates/basic/src/best_route.ts

@@ -22,6 +22,7 @@ import { buildTrace, score, writeTrace } from "./trace.ts";
 import { callWrapper } from "./wrapper_client.ts";
 import { preparePromptWithContext } from "./context.ts";
 import { selectCandidatesWithinBudget } from "./cost_aware.ts";
+import type { TaskCalibrationReport } from "./calibration.ts";
 
 function hasExplicitSelection(request: ProviderSelectionRequest): boolean {
   return Boolean(
@@ -180,6 +181,7 @@ async function discoverCandidates(authMode: ReturnType<typeof parseAuthMode>) {
 
 export async function invokeSelected(
   prompt: string,
+  calibration?: TaskCalibrationReport,
 ): Promise<
   { results: ProviderResult[]; tracePath: string; trace: DogfoodTrace }
 > {
@@ -188,7 +190,7 @@ export async function invokeSelected(
   const { request, candidates } = await discoverCandidates(authMode);
   if (candidates.length === 0) {
     throw new Error(
-      "OAuth/session-first provider unavailable. No usable OAuth/session/wrapper provider is available yet. Next: deno task auth:login",
+      "OAuth/session-first provider unavailable. No discovered provider has passed live authentication yet. Next: deno task auth:login",
     );
   }
   const prepared = await preparePromptWithContext(prompt);
@@ -250,6 +252,7 @@ export async function invokeSelected(
     requestedModel: request.model,
     providerSelectionHonored: selectionHonored(request, result),
     fallbackUsed: selectedIndex > 0 || selected.source === "env_fallback",
+    calibration,
   });
   const tracePath = await writeTrace("route-once-trace", trace);
   return { results: [result], tracePath, trace };
@@ -257,6 +260,7 @@ export async function invokeSelected(
 
 export async function runBestRoute(
   prompt: string,
+  calibration?: TaskCalibrationReport,
 ): Promise<
   { results: ProviderResult[]; tracePath: string; trace: DogfoodTrace }
 > {
@@ -270,7 +274,7 @@ export async function runBestRoute(
   const candidates = costAwareDecision.candidates;
   if (candidates.length === 0) {
     throw new Error(
-      "OAuth/session-first provider unavailable. best-route has no usable OAuth/session/wrapper provider yet. Next: deno task auth:login",
+      "OAuth/session-first provider unavailable. best-route has no provider with verified live authentication yet. Next: deno task auth:login",
     );
   }
   const prepared = await preparePromptWithContext(prompt);
@@ -325,6 +329,7 @@ export async function runBestRoute(
     costAware: costAwareDecision.cost.enabled
       ? costAwareDecision.cost
       : undefined,
+    calibration,
   });
   const tracePath = await writeTrace("best-route-trace", trace);
   return { results, tracePath, trace };

package/templates/basic/src/calibration_route.ts

@@ -0,0 +1,31 @@
+import {
+  aggregateTaskCalibration,
+  type TaskCalibrationOptions,
+  type TaskCalibrationReport,
+} from "./calibration.ts";
+
+export async function loadCalibrationEvidence(
+  path: string | undefined,
+): Promise<TaskCalibrationReport | undefined> {
+  if (path === undefined) return undefined;
+  const normalizedPath = path.trim();
+  if (!normalizedPath) {
+    throw new Error("--calibration-evidence requires a local JSON file path");
+  }
+  const parsed: unknown = JSON.parse(
+    await Deno.readTextFile(normalizedPath),
+  );
+  if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
+    throw new Error(
+      "calibration evidence must be an object with observations and optional options",
+    );
+  }
+  const record = parsed as Record<string, unknown>;
+  if (!Array.isArray(record.observations)) {
+    throw new Error("calibration evidence observations must be an array");
+  }
+  return aggregateTaskCalibration(
+    record.observations,
+    record.options as TaskCalibrationOptions | undefined,
+  );
+}

package/templates/basic/src/cli.ts

@@ -17,6 +17,7 @@ import {
   runSupabaseStatus,
 } from "./supabase.ts";
 import { buildTrace, writeTrace } from "./trace.ts";
+import { loadCalibrationEvidence } from "./calibration_route.ts";
 
 const DEFAULT_PROMPT = "Review this README for risky claims.";
 
@@ -61,6 +62,21 @@ function promptFromArgs(): string {
   return DEFAULT_PROMPT;
 }
 
+async function calibrationFromArgs() {
+  const indices = Deno.args.flatMap((arg, index) =>
+    arg === "--calibration-evidence" ? [index] : []
+  );
+  if (indices.length === 0) return await loadCalibrationEvidence(undefined);
+  if (indices.length !== 1) {
+    throw new Error("--calibration-evidence may be specified only once");
+  }
+  const path = Deno.args[indices[0] + 1];
+  if (path === undefined || path.startsWith("--")) {
+    throw new Error("--calibration-evidence requires a local JSON file path");
+  }
+  return await loadCalibrationEvidence(path);
+}
+
 async function runModelsList(): Promise<void> {
   const inventory = await inventoryCommand();
   console.log("QuorumRouter model inventory");
@@ -92,7 +108,8 @@ async function runHealth(): Promise<void> {
       )}`,
   );
   console.log(`Model inventory entries: ${inventory.entries.length}`);
-  console.log(`Usable providers: ${invokable.length}`);
+  console.log(`Discovered invokable providers: ${invokable.length}`);
+  console.log("Live provider authentication verified: false");
   console.log("Provider request sent: false");
   console.log(`Trace: ${tracePath}`);
   console.log(
@@ -110,9 +127,11 @@ try {
   else if (command === "health") await runHealth();
   else if (command === "supabase:status") await runSupabaseStatus();
   else if (command === "route:once") {
+    const calibration = await calibrationFromArgs();
     await preflightRequiredSupabaseAudit();
     const { results, tracePath, trace } = await invokeSelected(
       promptFromArgs(),
+      calibration,
     );
     await auditRouteOutcome(trace);
     console.log("QuorumRouter route:once");
@@ -126,8 +145,12 @@ try {
     console.log(`final: ${summarize(results[0].response_summary, 500)}`);
     console.log(`trace: ${tracePath}`);
   } else if (command === "best-route") {
+    const calibration = await calibrationFromArgs();
     await preflightRequiredSupabaseAudit();
-    const { results, tracePath, trace } = await runBestRoute(promptFromArgs());
+    const { results, tracePath, trace } = await runBestRoute(
+      promptFromArgs(),
+      calibration,
+    );
     await auditRouteOutcome(trace);
     console.log("QuorumRouter best-route");
     console.log(`models_called: ${results.length}`);

package/templates/basic/src/intake.ts

@@ -8,7 +8,7 @@ import { buildTrace, writeTrace } from "./trace.ts";
 export async function runIntake(): Promise<void> {
   const inventory = await discoverInventoryWithModelListing();
   await writeInventory(inventory);
-  const usable = invokableEntries(inventory);
+  const invokable = invokableEntries(inventory);
   console.log("QuorumRouter intake");
   console.log("");
   console.log("Step 1 — Detect local providers");
@@ -16,7 +16,10 @@ export async function runIntake(): Promise<void> {
   printInventoryTable(inventory);
   console.log("");
   console.log("Step 2 — OAuth/session status");
-  console.log(`Usable OAuth/session/wrapper providers: ${usable.length}`);
+  console.log(
+    `Discovered invokable OAuth/session/wrapper providers: ${invokable.length}`,
+  );
+  console.log("Live provider authentication verified: false");
   console.log("Provider request sent: false");
   console.log("Credential values printed: false");
   console.log("");
@@ -35,8 +38,8 @@ export async function runIntake(): Promise<void> {
     command: "intake-health",
     mode: "health",
     authMode: inventory.auth_mode,
-    errors: usable.length === 0
-      ? ["no usable OAuth/session/wrapper provider is available yet"]
+    errors: invokable.length === 0
+      ? ["no invokable OAuth/session/wrapper provider was discovered"]
       : [],
   });
   const tracePath = await writeTrace("intake-health-trace", trace);
@@ -44,10 +47,10 @@ export async function runIntake(): Promise<void> {
   console.log("");
   console.log("Step 6 — Recommended next action");
   console.log("");
-  if (usable.length === 0) {
+  if (invokable.length === 0) {
     console.log("QuorumRouter intake blocked");
     console.log("");
-    console.log("No usable OAuth/session/wrapper provider is available yet.");
+    console.log("No invokable OAuth/session/wrapper provider was discovered.");
     console.log("");
     console.log("Next:");
     console.log("  deno task auth:login");

package/templates/basic/src/model_inventory.ts

@@ -46,7 +46,7 @@ export async function inventoryCommand(): Promise<ModelInventory> {
 export function printInventoryTable(inventory: ModelInventory): void {
   console.log("Provider        Status       Auth        Models");
   for (const entry of inventory.entries) {
-    const status = entry.available ? "ready" : "blocked";
+    const status = entry.available ? "discovered" : "blocked";
     const models = entry.listed_models?.length
       ? entry.listed_models.join(", ")
       : entry.model;
@@ -56,4 +56,7 @@ export function printInventoryTable(inventory: ModelInventory): void {
       } ${models}`,
     );
   }
+  console.log(
+    "Status note: discovered means command/config present; live authentication is not verified.",
+  );
 }

package/templates/basic/src/provider_registry.ts

@@ -61,7 +61,6 @@ export function providerAliasesForCommand(
     claude: ["anthropic", "claude", "claude-code"],
     gemini: ["google", "gemini", "gemini-cli"],
     devin: ["cognition", "devin", "devin-cli"],
-    qwen: ["alibaba", "qwen", "qwen-cli"],
   };
   const providerAliases: Record<string, string[]> = {
     xai: ["grok", "grok-cli", "xai", "xai-cli"],
@@ -69,7 +68,6 @@ export function providerAliasesForCommand(
     anthropic: ["anthropic", "claude", "claude-code"],
     google: ["google", "gemini", "gemini-cli"],
     cognition: ["cognition", "devin", "devin-cli"],
-    alibaba: ["alibaba", "qwen", "qwen-cli"],
   };
   const normalizedProvider = normalizeSelectionValue(provider);
   return aliasSet([
@@ -220,19 +218,6 @@ export const LOCAL_PROVIDER_SPECS: ProviderSpec[] = [
       "Devin CLI does not expose a models subcommand in this environment.",
     notes: ["Uses existing Devin CLI session."],
   },
-  {
-    provider: "Alibaba",
-    model: "qwen-cli",
-    model_id: "alibaba/qwen-cli",
-    auth_mode: "session",
-    source: "local_cli",
-    command: "qwen",
-    args_template: ["-p", ""],
-    prompt_transport: "stdin",
-    list_blocked_reason:
-      "Qwen CLI model listing path is stdin/API-key dependent in this environment.",
-    notes: ["Uses existing local Qwen CLI session."],
-  },
 ];
 
 export function envFallbackEntry(configured: boolean): ModelInventoryEntry {

package/templates/basic/src/schema.ts

@@ -168,6 +168,23 @@ export type CostAwareTrace = {
   }>;
 };
 
+export type CalibrationTraceSummary = {
+  schema_version: "quorum-router.calibration-trace.v1";
+  advisory_only: true;
+  minimum_sample_count: number;
+  group_count: number;
+  groups: Array<{
+    task_type_sha256: string;
+    source_sha256: string;
+    sample_count: number;
+    accuracy: number;
+    mean_confidence: number;
+    brier_score: number;
+    mean_calibration_bias: number;
+    sample_status: "insufficient" | "sufficient";
+  }>;
+};
+
 export type DogfoodTrace = {
   run_id: string;
   timestamp: string;
@@ -186,6 +203,7 @@ export type DogfoodTrace = {
   prompt_summary?: string;
   prompt_context?: PromptContextTrace;
   cost_aware?: CostAwareTrace;
+  calibration?: CalibrationTraceSummary;
   response_summary?: string;
   schema_valid: boolean;
   redaction_ok: boolean;
@@ -217,9 +235,12 @@ export function assertOptIn(): void {
 }
 
 export function assertAgentChatOptIn(): void {
-  if (Deno.env.get("RUN_EXPERIMENTAL_AGENT_CHAT") !== "1") {
+  if (
+    Deno.env.get("RUN_AGENT_CHAT") !== "1" &&
+    Deno.env.get("RUN_EXPERIMENTAL_AGENT_CHAT") !== "1"
+  ) {
     throw new Error(
-      "agent_chat blocked: set RUN_EXPERIMENTAL_AGENT_CHAT=1; live multi-model dialogue is explicit opt-in",
+      "agent_chat blocked: set RUN_AGENT_CHAT=1; live multi-model dialogue is explicit opt-in and has no mutation authority",
     );
   }
 }

package/templates/basic/src/trace.ts

@@ -1,12 +1,14 @@
 import type {
   AgentChatTurn,
   AuthMode,
+  CalibrationTraceSummary,
   CostAwareTrace,
   DogfoodTrace,
   PromptContextTrace,
   ProviderResult,
   ScoreRow,
 } from "./schema.ts";
+import type { TaskCalibrationReport } from "./calibration.ts";
 import { redactionOk, summarize } from "./redact.ts";
 
 export const OUT_DIR = "out";
@@ -35,6 +37,29 @@ export async function promptHash(prompt: string): Promise<string> {
   ).join("");
 }
 
+async function calibrationTraceSummary(
+  report: TaskCalibrationReport,
+): Promise<CalibrationTraceSummary> {
+  return {
+    schema_version: "quorum-router.calibration-trace.v1",
+    advisory_only: true,
+    minimum_sample_count: report.minimum_sample_count,
+    group_count: report.groups.length,
+    groups: await Promise.all(report.groups.map(async (group) => ({
+      task_type_sha256: await promptHash(group.task_type),
+      source_sha256: await promptHash(
+        `${group.source.provider}\u0000${group.source.model}`,
+      ),
+      sample_count: group.sample_count,
+      accuracy: group.accuracy,
+      mean_confidence: group.mean_confidence,
+      brier_score: group.brier_score,
+      mean_calibration_bias: group.mean_calibration_bias,
+      sample_status: group.sample_status,
+    }))),
+  };
+}
+
 export function score(result: ProviderResult): ScoreRow {
   const text = result.response_summary;
   const clarity = text.length > 20 ? 4 : 2;
@@ -77,7 +102,11 @@ export async function buildTrace(args: {
   fallbackUsed?: boolean;
   costAware?: CostAwareTrace;
   agentChatTurns?: AgentChatTurn[];
+  calibration?: TaskCalibrationReport;
 }): Promise<DogfoodTrace> {
+  const calibration = args.calibration
+    ? await calibrationTraceSummary(args.calibration)
+    : undefined;
   const responseSummary = args.results?.map((r) =>
     `[${r.provider}/${r.model}] ${r.response_summary}`
   ).join("\n");
@@ -96,6 +125,7 @@ export async function buildTrace(args: {
     provider_selection_honored: args.providerSelectionHonored ?? true,
     fallback_used: args.fallbackUsed ?? false,
     cost_aware: args.costAware,
+    ...(calibration ? { calibration } : {}),
     prompt_hash: args.prompt ? await promptHash(args.prompt) : undefined,
     prompt_summary: args.prompt ? summarize(args.prompt, 160) : undefined,
     prompt_context: args.promptContext,