@klhapp/skillmux 1.1.0 → 1.2.0

package/CHANGELOG.md

@@ -5,6 +5,16 @@ All notable changes to this project are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.2.0](https://github.com/klhq/skillmux/compare/v1.1.0...v1.2.0) (2026-07-28)
+
+
+### Added
+
+* **calibration:** finish remote contract and docs ([f696da2](https://github.com/klhq/skillmux/commit/f696da25094c19537b1291f694114f8e4052ea69))
+* **calibration:** improve dataset quality ([#88](https://github.com/klhq/skillmux/issues/88)) ([e99b764](https://github.com/klhq/skillmux/commit/e99b764ce90b7432fca0b01c22560ba73b401dd2))
+* **calibration:** require labelled audit feedback ([16d9ec8](https://github.com/klhq/skillmux/commit/16d9ec81acbdca0e42a401b9e28f9e3814b3e0ab))
+* **calibration:** require labelled audit feedback ([#89](https://github.com/klhq/skillmux/issues/89)) ([a7a99e2](https://github.com/klhq/skillmux/commit/a7a99e2d82e619255135b1e941c69c8d6653012c))
+
 ## [1.1.0](https://github.com/klhq/skillmux/compare/v1.0.1...v1.1.0) (2026-07-28)
 
 

package/README.md

@@ -14,6 +14,7 @@ Built for agents that lack native skill triggering (Goose recipe workers, openco
 - [Docker Usage](#docker-usage)
 - [Configuration](#configuration) — inference modes, security scanning, installing skills, env vars
 - [CLI & Automation](docs/cli.md) — context management, remote target resolution, policy calibration, JSON envelopes
+- [Policy Calibration](docs/calibration.md) — labelled datasets, certification, apply lifecycle, and reference profile
 - [Benchmarks & Evaluation](#benchmarks--evaluation)
 - [FAQ & Troubleshooting](#faq--troubleshooting)
 - [Guarantees](#guarantees)
@@ -546,7 +547,11 @@ bun run src/cli.ts eval
 # hybrid recall@5:  1.000
 ```
 
-Custom policy calibration can also be performed against domain-specific query logs using `skillmux calibrate` (see [`docs/cli.md`](docs/cli.md#policy-calibration-skillmux-calibrate)).
+Custom policy calibration can also be performed against reviewed,
+domain-specific query datasets using `skillmux calibrate`. See the
+[calibration guide](docs/calibration.md) for the full operator lifecycle and
+the [CLI reference](docs/cli.md#policy-calibration-skillmux-calibrate) for
+command syntax.
 
 ## FAQ & Troubleshooting
 

package/config.remote.example.toml

@@ -18,7 +18,9 @@ endpoint = "https://reranker.example.com/v1/rerank"
 model = "your-reranker-model"
 api_key_env = "RERANKER_API_KEY"
 
-# Required when a reranker is configured. Calibrate these for your model.
+# Optional. Without these values, reranked results remain ambiguous.
+# Calibrate locally against your corpus before enabling automatic matches.
+# See docs/calibration.md; these example values are not universal defaults.
 [inference.thresholds]
 match_score = 0.90
 match_margin = 0.30

package/docs/calibration.md

@@ -0,0 +1,106 @@
+# Policy calibration
+
+Calibration selects the three reranker-score thresholds that turn an ordered
+shortlist into `matched`, `ambiguous`, or `no_match`. It is an operator action,
+not background learning, and it currently runs only against a local Skillmux
+target.
+
+## Lifecycle
+
+The complete workflow is:
+
+```text
+install CLI → configure vault/index/embedding/reranker → obtain labelled dataset
+    → calibrate run → review calibrate show RUN_ID → calibrate apply RUN_ID
+    → live-reloaded policy handles subsequent requests
+```
+
+First configure and index the same vault, embedding model, and reranker that
+will serve requests. Supply a reviewed dataset, or generate a starting point
+and review every label:
+
+```sh
+skillmux calibrate generate-dataset --out ./eval/queries.json
+skillmux calibrate run --dataset ./eval/queries.json
+skillmux calibrate show RUN_ID
+skillmux calibrate apply RUN_ID
+```
+
+Skillmux retrieves candidates and reranks exactly once for each evaluation
+query. It caches those observations, searches thresholds on the `tune` split,
+then certifies the selected policy on the frozen `test` split. Calibration
+starts only when an operator invokes `calibrate run`.
+
+The operator owns the labels: supply or review the cases, start the run,
+inspect its evidence, and explicitly apply an acceptable result. A successful
+run never changes live thresholds by itself.
+
+## Reading a run
+
+A `run_id` identifies one immutable calibration attempt and its evidence.
+`calibrate show RUN_ID` is read-only. It reports:
+
+- selected thresholds and tune/test metrics;
+- auto-match precision confidence and sample counts;
+- retrieval and delivered-shortlist recall;
+- a closed failure reason when certification fails;
+- reranker, embedding, corpus, and dataset fingerprints;
+- dataset provenance and the number of human-labelled cases; and
+- the attempt count for the dataset hash.
+
+`calibrate apply RUN_ID` accepts only a completed, test-certified run. It
+rechecks the reranker fingerprint, rejects thresholds masked by environment
+variables, atomically updates the TOML file, and lets the config watcher
+activate the new snapshot.
+
+## Dataset responsibilities
+
+Each case needs a query, expected outcome, relevant skill ids, and a fixed
+`tune` or `test` split. Unknown skill ids are rejected. Keep a skill entirely
+within one split so the test set measures generalization rather than memorized
+skill wording.
+
+Generated datasets are scaffolding, not ground truth. Review paraphrases,
+near-miss negatives, and ambiguous cases before using them for certification.
+Audit-derived cases require an explicit human label and provenance. Raw audit
+queries are excluded unless the importer is deliberately configured to retain
+them.
+
+## When to recalibrate
+
+Re-run calibration after a material change to the corpus, embedding or
+retrieval behavior, reranker adapter or model, or after collecting enough new
+human-labelled feedback. Do not recalibrate per user request. Every rerun gets
+a new `run_id`; the active policy remains unchanged until one is applied.
+
+## Local and remote targets
+
+Calibration is local-only in this release. Local commands operate on the
+configured local vault, index, inference endpoints, dataset path, evidence
+database, and TOML file. Human output always prints `Target: local`; JSON output
+uses `"target": "local"`.
+
+Remote servers advertise `"calibration": false`. Every
+`/admin/v1/calibrations` route returns HTTP `501` with
+`error: "not_implemented"`, and the CLI rejects remote calibration before
+uploading or claiming to execute a local dataset path. This also prevents raw
+evaluation queries from being exposed through the admin API.
+
+## Reference starting profile
+
+Reranker scores are not portable across models, adapters, model revisions, or
+corpora. The profile below is published only to make the checked-in BGE example
+concrete; it is not a certified substitute for calibration.
+
+| Model | Adapter | `match_score` | `match_margin` | `candidate_floor` |
+|---|---|---:|---:|---:|
+| `BAAI/bge-reranker-v2-m3` | `jina-v1` | `0.90` | `0.20` | `0.40` |
+
+Provenance: the small synthetic corpus and labelled decision cases in
+[`tests/router-core.spec.test.ts`](../tests/router-core.spec.test.ts), with the
+wire contract captured by
+[`tests/fixtures/reranker/jina-v1-request.json`](../tests/fixtures/reranker/jina-v1-request.json).
+That fixture is below the default 30-auto-match certification minimum, so the
+values are a smoke-test/reference profile, not a completed calibration run.
+Run the lifecycle above against the deployment's real corpus before enabling
+automatic matches in production.

package/docs/configuration.md

@@ -2,7 +2,10 @@
 
 Skillmux defaults to FTS5 plus local GTE-small semantic retrieval. Most users need no config file.
 
-For detailed CLI command reference, target resolution, policy calibration, and automation envelopes, see [`docs/cli.md`](cli.md).
+For detailed CLI command reference, target resolution, and automation
+envelopes, see [`docs/cli.md`](cli.md). For labelled datasets, threshold
+certification, reference values, and the apply lifecycle, see
+[`docs/calibration.md`](calibration.md).
 
 ## Machine config bootstrap
 
@@ -105,7 +108,13 @@ candidate list returned to the calling LLM after retrieval, reranking, and
 threshold filtering. It does not change retrieval depth or the matched,
 ambiguous, or no-match classification.
 
-Reranker thresholds have no universal default because score distributions are model-specific. When configuring a reranker, provide calibrated `inference.thresholds.match_score`, `inference.thresholds.match_margin`, and `inference.thresholds.candidate_floor`; otherwise configuration is rejected rather than silently applying unsuitable values.
+Reranker thresholds have no universal default because score distributions are
+model-specific. Without `inference.thresholds`, Skillmux still uses the
+reranker to order candidates but keeps outcomes ambiguous rather than
+auto-matching. Use `skillmux calibrate run` to select
+`match_score`, `match_margin`, and `candidate_floor`, then explicitly apply the
+certified run. The [calibration guide](calibration.md) also publishes a
+clearly-scoped BGE reference profile for smoke tests.
 
 ## HTTP server
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@klhapp/skillmux",
-  "version": "1.1.0",
+  "version": "1.2.0",
   "description": "Local read-only MCP server routing natural-language task queries to skills in a SKILL.md vault, with zero-loss delivery",
   "type": "module",
   "private": false,
@@ -24,6 +24,7 @@
     "src",
     "docs/schema.json",
     "docs/configuration.md",
+    "docs/calibration.md",
     "docs/releasing.md",
     "README.md",
     "LICENSE",

package/src/adapters.ts

@@ -1,6 +1,6 @@
 import { createHash } from "node:crypto";
 import { join } from "node:path";
-import { applyCalibrationRun, getCalibrationRun, insertCalibrationRun, listCalibrationRuns, loadDecisionCasesFromFile, openCalibrateDb, runCalibration, type CalibrationResult } from "./calibrate";
+import { applyCalibrationRun, getCalibrationRun, insertCalibrationRun, listCalibrationRuns, loadDecisionCasesFromFile, openCalibrateDb, runCalibration, summarizeDatasetProvenance, type CalibrationResult } from "./calibrate";
 import { createClients } from "./clients";
 import { DEFAULT_CONFIG_PATH, embeddingFingerprint, expandHome, loadConfig, rerankerFingerprint } from "./config";
 import { openIndex } from "./db";
@@ -139,7 +139,19 @@ export class LocalAdapter implements TargetAdapter {
   }): Promise<{ run_id?: string; result?: CalibrationResult }> {
     const config = await loadConfig(this.configPath);
     const datasetFile = opts?.datasetPath ?? join(expandHome(config.state_dir), "queries.json");
-    const cases = loadDecisionCasesFromFile(datasetFile);
+    const indexDb = openIndex(expandHome(config.state_dir));
+    let indexedSkills: Array<{ skill_id: string; content_sha256: string }>;
+    try {
+      indexedSkills = indexDb
+        .query("SELECT skill_id, content_sha256 FROM skills ORDER BY skill_id")
+        .all() as Array<{ skill_id: string; content_sha256: string }>;
+    } finally {
+      indexDb.close();
+    }
+    const cases = loadDecisionCasesFromFile(
+      datasetFile,
+      indexedSkills.map((skill) => skill.skill_id),
+    );
     const clients = createClients(config);
     configure({ config, clients });
     const result = await runCalibration({
@@ -168,18 +180,9 @@ export class LocalAdapter implements TargetAdapter {
       throw new Error("A configured remote reranker is required to record calibration.");
     }
     const datasetText = await Bun.file(datasetFile).text();
-    const indexDb = openIndex(expandHome(config.state_dir));
-    let corpusFingerprint: string;
-    try {
-      const rows = indexDb
-        .query("SELECT skill_id, content_sha256 FROM skills ORDER BY skill_id")
-        .all();
-      corpusFingerprint =
-        "vault:" +
-        createHash("sha256").update(JSON.stringify(rows)).digest("hex");
-    } finally {
-      indexDb.close();
-    }
+    const corpusFingerprint =
+      "vault:" +
+      createHash("sha256").update(JSON.stringify(indexedSkills)).digest("hex");
     const runId = `run_${crypto.randomUUID()}`;
     const db = openCalibrateDb(expandHome(config.state_dir));
     try {
@@ -191,6 +194,7 @@ export class LocalAdapter implements TargetAdapter {
         embedding_fingerprint: embeddingFingerprint(config),
         corpus_fingerprint: corpusFingerprint,
         dataset_hash: createHash("sha256").update(datasetText).digest("hex"),
+        dataset_provenance: summarizeDatasetProvenance(cases),
         candidate_limit: config.thresholds.candidate_limit,
         min_auto_match_precision: opts?.minAutoMatchPrecision ?? 0.99,
         min_auto_match_count: opts?.minAutoMatchCount ?? 30,
@@ -420,41 +424,30 @@ export class RemoteAdapter implements TargetAdapter {
     minDeliveredShortlistRecallAtK?: number;
     minAutoMatchCount?: number;
   }): Promise<{ run_id?: string; result?: CalibrationResult }> {
-    const { status, data } = await this.fetchJson("/admin/v1/calibrations", {
-      method: "POST",
-      headers: { "Content-Type": "application/json" },
-      body: JSON.stringify({
-        dataset_path: opts?.datasetPath,
-        min_auto_match_precision: opts?.minAutoMatchPrecision,
-        min_retrieval_recall_at_k: opts?.minRetrievalRecallAtK,
-        min_delivered_shortlist_recall_at_k: opts?.minDeliveredShortlistRecallAtK,
-        min_auto_match_count: opts?.minAutoMatchCount,
-      }),
-    });
-    if (status !== 202) {
-      throw new Error(`Remote calibration start failed (${status}): ${data?.message || data}`);
-    }
-    return data;
+    void opts;
+    throw this.remoteCalibrationNotImplemented();
   }
 
   async calibrateList(): Promise<any[]> {
-    const { status, data } = await this.fetchJson("/admin/v1/calibrations");
-    if (status !== 200) throw new Error(`Remote calibration list failed (${status}): ${data?.message || data}`);
-    return data;
+    throw this.remoteCalibrationNotImplemented();
   }
 
   async calibrateShow(runId: string): Promise<any> {
-    const { status, data } = await this.fetchJson(`/admin/v1/calibrations/${runId}`);
-    if (status !== 200) throw new Error(`Remote calibration show failed (${status}): ${data?.message || data}`);
-    return data;
+    void runId;
+    throw this.remoteCalibrationNotImplemented();
   }
 
   async calibrateApply(runId: string): Promise<any> {
-    const { status, data } = await this.fetchJson(`/admin/v1/calibrations/${runId}/apply`, {
-      method: "POST",
-    });
-    if (status !== 200) throw new Error(`Remote calibration apply failed (${status}): ${data?.message || data}`);
-    return data;
+    void runId;
+    throw this.remoteCalibrationNotImplemented();
+  }
+
+  private remoteCalibrationNotImplemented(): CliError {
+    return new CliError(
+      `Remote calibration is not implemented for target ${this.serverUrl}; ` +
+        "run `skillmux calibrate` against a local target.",
+      2,
+    );
   }
 }
 

package/src/calibrate.ts

@@ -3,7 +3,7 @@ import { join } from "node:path";
 import { Database } from "bun:sqlite";
 import { z } from "zod";
 import { decideResolveOutcome } from "./decision";
-import type { RankedCandidate } from "./types";
+import type { AuditRow, RankedCandidate } from "./types";
 
 export { generateDataset, type GenerateDatasetOptions } from "./dataset-generator";
 
@@ -15,11 +15,21 @@ export { generateDataset, type GenerateDatasetOptions } from "./dataset-generato
 export type DecisionSplit = "tune" | "test";
 export type DecisionOutcome = "matched" | "ambiguous" | "no_match";
 
+export interface DecisionCaseProvenance {
+  version: 1;
+  source: "authored" | "audit_import";
+  review_status: "human_labelled" | "unreviewed";
+  query_storage: "raw" | "redacted";
+  audit_id?: number;
+  labelled_at?: string;
+}
+
 export interface DecisionCase {
   query: string;
   split: DecisionSplit;
   expected_outcome: DecisionOutcome;
   relevant_skill_ids: string[];
+  provenance?: DecisionCaseProvenance;
 }
 
 // ---------------------------------------------------------------------------
@@ -31,6 +41,14 @@ const rawCaseSchema = z.object({
   split: z.enum(["tune", "test"]),
   expected_outcome: z.enum(["matched", "ambiguous", "no_match"]),
   relevant_skill_ids: z.array(z.string()),
+  provenance: z.object({
+    version: z.literal(1),
+    source: z.enum(["authored", "audit_import"]),
+    review_status: z.enum(["human_labelled", "unreviewed"]),
+    query_storage: z.enum(["raw", "redacted"]),
+    audit_id: z.number().int().positive().optional(),
+    labelled_at: z.string().datetime().optional(),
+  }).strict().optional(),
 }).strict();
 
 type RawCase = z.infer<typeof rawCaseSchema>;
@@ -45,6 +63,25 @@ function validateCase(raw: RawCase, idx: number): DecisionCase {
   }
 
   const { expected_outcome, relevant_skill_ids } = raw;
+  const provenance: DecisionCaseProvenance = raw.provenance ?? {
+    version: 1,
+    source: "authored",
+    review_status: "human_labelled",
+    query_storage: "raw",
+  };
+
+  if (provenance.source === "audit_import") {
+    if (provenance.audit_id === undefined) {
+      throw new Error(
+        `Validation error at case ${idx}: imported field "provenance.audit_id" is required`,
+      );
+    }
+    if (provenance.review_status !== "human_labelled" || !provenance.labelled_at) {
+      throw new Error(
+        `Validation error at case ${idx}: imported audit case is unreviewed; human label and "provenance.labelled_at" are required for certification`,
+      );
+    }
+  }
 
   if (expected_outcome === "matched") {
     if (relevant_skill_ids.length !== 1) {
@@ -67,7 +104,85 @@ function validateCase(raw: RawCase, idx: number): DecisionCase {
     }
   }
 
-  return raw as DecisionCase;
+  return { ...raw, provenance };
+}
+
+export interface AuditFeedbackLabel {
+  split: DecisionSplit;
+  expected_outcome: DecisionOutcome;
+  relevant_skill_ids: string[];
+  labelled_at: string;
+}
+
+export type AuditQueryPrivacy =
+  | { include_raw_query: true }
+  | { include_raw_query: false; redacted_query: string };
+
+/** Import an audit outcome only after a separate human label is supplied. */
+export function importLabelledAuditCase(
+  audit: AuditRow,
+  label: AuditFeedbackLabel,
+  privacy: AuditQueryPrivacy,
+): DecisionCase {
+  const query = privacy.include_raw_query ? audit.query : privacy.redacted_query.trim();
+  if (!query) {
+    throw new Error("A non-empty redacted_query is required when raw audit queries are excluded");
+  }
+  const parsed = rawCaseSchema.parse({
+    query,
+    split: label.split,
+    expected_outcome: label.expected_outcome,
+    relevant_skill_ids: label.relevant_skill_ids,
+    provenance: {
+      version: 1,
+      source: "audit_import",
+      review_status: "human_labelled",
+      query_storage: privacy.include_raw_query ? "raw" : "redacted",
+      audit_id: audit.id,
+      labelled_at: label.labelled_at,
+    },
+  });
+  return validateCase(parsed, audit.id);
+}
+
+export interface DatasetProvenanceSummary {
+  version: 1;
+  human_labelled_case_count: number;
+  imported_labelled_case_count: number;
+  imported_unreviewed_case_count: number;
+  raw_query_case_count: number;
+  redacted_query_case_count: number;
+}
+
+export function summarizeDatasetProvenance(
+  cases: DecisionCase[],
+): DatasetProvenanceSummary {
+  const provenance = (item: DecisionCase): DecisionCaseProvenance =>
+    item.provenance ?? {
+      version: 1,
+      source: "authored",
+      review_status: "human_labelled",
+      query_storage: "raw",
+    };
+  return {
+    version: 1,
+    human_labelled_case_count:
+      cases.filter((item) => provenance(item).review_status === "human_labelled").length,
+    imported_labelled_case_count:
+      cases.filter((item) =>
+        provenance(item).source === "audit_import" &&
+        provenance(item).review_status === "human_labelled"
+      ).length,
+    imported_unreviewed_case_count:
+      cases.filter((item) =>
+        provenance(item).source === "audit_import" &&
+        provenance(item).review_status === "unreviewed"
+      ).length,
+    raw_query_case_count:
+      cases.filter((item) => provenance(item).query_storage === "raw").length,
+    redacted_query_case_count:
+      cases.filter((item) => provenance(item).query_storage === "redacted").length,
+  };
 }
 
 // ---------------------------------------------------------------------------
@@ -113,8 +228,12 @@ function validateDatasetCompleteness(cases: DecisionCase[]): void {
  *     no_match → 0)
  *   - Dataset completeness (both splits, all outcome types in each split)
  */
-export function loadDecisionCases(raw: unknown[]): DecisionCase[] {
+export function loadDecisionCases(
+  raw: unknown[],
+  validSkillIds?: Iterable<string>,
+): DecisionCase[] {
   const parsed: DecisionCase[] = [];
+  const validIds = validSkillIds ? new Set(validSkillIds) : undefined;
 
   for (let i = 0; i < raw.length; i++) {
     const item = raw[i];
@@ -128,7 +247,17 @@ export function loadDecisionCases(raw: unknown[]): DecisionCase[] {
       );
     }
 
-    parsed.push(validateCase(result.data, i));
+    const parsedCase = validateCase(result.data, i);
+    if (validIds) {
+      for (const skillId of parsedCase.relevant_skill_ids) {
+        if (!validIds.has(skillId)) {
+          throw new Error(
+            `Validation error at case ${i}: field "relevant_skill_ids" references unknown vault skill "${skillId}"`,
+          );
+        }
+      }
+    }
+    parsed.push(parsedCase);
   }
 
   validateDatasetCompleteness(parsed);
@@ -139,9 +268,12 @@ export function loadDecisionCases(raw: unknown[]): DecisionCase[] {
  * Read a JSON file from disk and validate it as a decision-policy dataset.
  * Throws if the file cannot be read or the contents fail validation.
  */
-export function loadDecisionCasesFromFile(path: string): DecisionCase[] {
+export function loadDecisionCasesFromFile(
+  path: string,
+  validSkillIds?: Iterable<string>,
+): DecisionCase[] {
   const raw = JSON.parse(readFileSync(path, "utf8")) as unknown[];
-  return loadDecisionCases(raw);
+  return loadDecisionCases(raw, validSkillIds);
 }
 
 // ---------------------------------------------------------------------------
@@ -734,6 +866,7 @@ export interface CalibrationRunRecord {
   embedding_fingerprint: string;
   corpus_fingerprint: string;
   dataset_hash: string;
+  dataset_provenance?: DatasetProvenanceSummary;
   candidate_limit: number;
   attempt_count?: number;
   min_auto_match_precision: number;
@@ -756,6 +889,8 @@ export interface CalibrationRunSummary {
   embedding_fingerprint: string;
   corpus_fingerprint: string;
   dataset_hash: string;
+  human_labelled_case_count: number;
+  imported_labelled_case_count: number;
   candidate_limit: number;
   attempt_count: number;
   min_auto_match_precision: number;
@@ -806,6 +941,15 @@ export function openCalibrateDb(stateDir: string): Database {
   if (!columns.some((column) => column.name === "failed_reason")) {
     db.run("ALTER TABLE calibration_runs ADD COLUMN failed_reason TEXT");
   }
+  if (!columns.some((column) => column.name === "dataset_provenance")) {
+    db.run("ALTER TABLE calibration_runs ADD COLUMN dataset_provenance TEXT NOT NULL DEFAULT '{}'");
+  }
+  if (!columns.some((column) => column.name === "human_labelled_case_count")) {
+    db.run("ALTER TABLE calibration_runs ADD COLUMN human_labelled_case_count INTEGER NOT NULL DEFAULT 0");
+  }
+  if (!columns.some((column) => column.name === "imported_labelled_case_count")) {
+    db.run("ALTER TABLE calibration_runs ADD COLUMN imported_labelled_case_count INTEGER NOT NULL DEFAULT 0");
+  }
   return db;
 }
 
@@ -822,8 +966,9 @@ export function insertCalibrationRun(db: Database, run: CalibrationRunRecord): v
       candidate_limit,
       attempt_count, min_auto_match_precision, min_auto_match_count,
       min_delivered_shortlist_recall_at_k, min_shortlist_recall_at_5, failed_reason,
-      selected_thresholds, tune_metrics, test_metrics, observations
-    ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
+      selected_thresholds, tune_metrics, test_metrics, observations,
+      dataset_provenance, human_labelled_case_count, imported_labelled_case_count
+    ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
     [
       run.run_id,
       run.created_at,
@@ -843,6 +988,9 @@ export function insertCalibrationRun(db: Database, run: CalibrationRunRecord): v
       run.tune_metrics != null ? JSON.stringify(run.tune_metrics) : null,
       run.test_metrics != null ? JSON.stringify(run.test_metrics) : null,
       JSON.stringify(run.observations),
+      JSON.stringify(run.dataset_provenance ?? {}),
+      run.dataset_provenance?.human_labelled_case_count ?? 0,
+      run.dataset_provenance?.imported_labelled_case_count ?? 0,
     ],
   );
 }
@@ -866,6 +1014,9 @@ interface RawCalibrationRow {
   tune_metrics: string | null;
   test_metrics: string | null;
   observations: string;
+  dataset_provenance: string;
+  human_labelled_case_count: number;
+  imported_labelled_case_count: number;
 }
 
 function parseMetrics(json: string): CalibrationMetrics {
@@ -900,6 +1051,10 @@ function rowToRecord(row: RawCalibrationRow): CalibrationRunRecord {
     embedding_fingerprint: row.embedding_fingerprint,
     corpus_fingerprint: row.corpus_fingerprint,
     dataset_hash: row.dataset_hash,
+    dataset_provenance:
+      Object.keys(JSON.parse(row.dataset_provenance) as object).length > 0
+        ? JSON.parse(row.dataset_provenance) as DatasetProvenanceSummary
+        : undefined,
     candidate_limit: row.candidate_limit,
     attempt_count: row.attempt_count,
     min_auto_match_precision: row.min_auto_match_precision,
@@ -939,7 +1094,8 @@ export function listCalibrationRuns(db: Database): CalibrationRunSummary[] {
         reranker_fingerprint, embedding_fingerprint, corpus_fingerprint, dataset_hash,
         candidate_limit,
         attempt_count, min_auto_match_precision, min_auto_match_count,
-        min_delivered_shortlist_recall_at_k, min_shortlist_recall_at_5, failed_reason
+        min_delivered_shortlist_recall_at_k, min_shortlist_recall_at_5, failed_reason,
+        human_labelled_case_count, imported_labelled_case_count
        FROM calibration_runs ORDER BY created_at DESC`,
     )
     .all() as CalibrationRunSummary[];

package/src/cli.ts

@@ -446,7 +446,7 @@ async function handleCalibrateCommand(
       minAutoMatchCount,
     });
     emitSuccess({ isJson: ctx.isJson, target: ctx.target }, res, () => {
-      renderTargetBanner(ctx.target);
+      renderCalibrationTarget(ctx.target);
       console.log(`Calibration run complete.`);
       if (res.result) console.log(JSON.stringify(res.result, null, 2));
     });
@@ -456,7 +456,7 @@ async function handleCalibrateCommand(
   if (sub === "list") {
     const res = await adapter.calibrateList();
     emitSuccess({ isJson: ctx.isJson, target: ctx.target }, res, () => {
-      renderTargetBanner(ctx.target);
+      renderCalibrationTarget(ctx.target);
       renderTable(
         [
           { key: "run_id", header: "RUN_ID" },
@@ -474,7 +474,7 @@ async function handleCalibrateCommand(
     if (!runId) throw new Error("usage: skillmux calibrate show <run_id>");
     const res = await adapter.calibrateShow(runId);
     emitSuccess({ isJson: ctx.isJson, target: ctx.target }, res, () => {
-      renderTargetBanner(ctx.target);
+      renderCalibrationTarget(ctx.target);
       console.log(JSON.stringify(res, null, 2));
     });
     return;
@@ -485,7 +485,7 @@ async function handleCalibrateCommand(
     if (!runId) throw new Error("usage: skillmux calibrate apply <run_id>");
     const res = await adapter.calibrateApply(runId);
     emitSuccess({ isJson: ctx.isJson, target: ctx.target }, res, () => {
-      renderTargetBanner(ctx.target);
+      renderCalibrationTarget(ctx.target);
       console.log(`Applied calibration run "${runId}"`);
     });
     return;
@@ -501,6 +501,14 @@ async function handleCalibrateCommand(
   );
 }
 
+function renderCalibrationTarget(target: ResolvedTarget): void {
+  if (target.type === "local") {
+    console.log("Target: local");
+  } else {
+    console.log(`Target: remote (${target.name} -> ${target.server})`);
+  }
+}
+
 async function handleCompletionsCommand(shell: string) {
   if (shell !== "bash" && shell !== "zsh" && shell !== "fish") {
     throw new Error("usage: skillmux completions <bash|zsh|fish>");

package/src/dataset-generator.ts

@@ -13,14 +13,45 @@ export interface GenerateDatasetOptions {
   queriesPerSplit?: number;
 }
 
-const GENERIC_NO_MATCH_QUERIES = [
-  "what is the weather in Paris today",
-  "recipe for baking sourdough bread at home",
-  "what is the distance from Earth to Mars",
-  "explain quantum entanglement simply",
-  "how do I solve a quadratic equation",
-  "who won the 1998 World Cup",
-];
+const STOP_WORDS = new Set([
+  "a", "an", "and", "are", "as", "at", "be", "by", "for", "from", "in", "is",
+  "it", "of", "on", "or", "the", "this", "to", "use", "with",
+]);
+
+function words(value: string): string[] {
+  return value
+    .toLowerCase()
+    .match(/[a-z0-9]+/g)
+    ?.filter((word) => word.length > 2 && !STOP_WORDS.has(word)) ?? [];
+}
+
+function anchors(skill: VaultSkill): string[] {
+  const preferred = [...skill.aliases.flatMap(words), ...words(skill.title)];
+  const fallback = words(skill.description);
+  return [...new Set([...preferred, ...fallback])].slice(0, 2);
+}
+
+function matchedQuery(skill: VaultSkill, variant: number): string {
+  const [first = "specialized", second = "workflow"] = anchors(skill);
+  const templates = [
+    `I need practical guidance completing an unfamiliar ${first} ${second} task safely`,
+    `Which available workflow can handle my unusual ${first} ${second} problem end to end`,
+    `Please guide me through a difficult unfamiliar ${first} ${second} operation safely`,
+  ];
+  return templates[variant % templates.length]!;
+}
+
+function ambiguousQuery(first: VaultSkill, second: VaultSkill): string {
+  const [firstAnchor = "first"] = anchors(first);
+  const [secondAnchor = "second"] = anchors(second);
+  return `Help with a workflow spanning both ${firstAnchor} and ${secondAnchor} responsibilities`;
+}
+
+function nearMissQuery(first: VaultSkill, second: VaultSkill): string {
+  const [firstAnchor = "one"] = anchors(first);
+  const [secondAnchor = "another"] = anchors(second);
+  return `Explain the theory comparing ${firstAnchor} and ${secondAnchor} without performing either workflow`;
+}
 
 /**
  * Automatically generate a synthetic decision-policy calibration dataset
@@ -30,110 +61,58 @@ export function generateDataset(
   skills: VaultSkill[],
   options: GenerateDatasetOptions = {},
 ): RawDecisionCase[] {
-  const cases: RawDecisionCase[] = [];
+  if (skills.length < 4) {
+    throw new Error(
+      "Dataset generation requires at least 4 vault skills so tune and test can each contain matched and ambiguous cases without skill leakage",
+    );
+  }
 
-  // --- 1. Matched Cases ---
-  for (const skill of skills) {
-    // Primary query from title + description
-    cases.push({
-      query: `how do I ${skill.title.toLowerCase()}: ${skill.description.toLowerCase()}`,
-      split: "tune",
-      expected_outcome: "matched",
-      relevant_skill_ids: [skill.skill_id],
-    });
+  const cases: RawDecisionCase[] = [];
+  const sorted = [...skills].sort((a, b) => a.skill_id.localeCompare(b.skill_id));
+  const splitAt = Math.ceil(sorted.length / 2);
+  const bySplit: Record<DecisionSplit, VaultSkill[]> = {
+    tune: sorted.slice(0, splitAt),
+    test: sorted.slice(splitAt),
+  };
+  const targetPerSplit = Math.max(3, options.queriesPerSplit ?? 10);
 
-    // Secondary query from aliases
-    if (skill.aliases.length > 0) {
-      cases.push({
-        query: `help me with ${skill.aliases[0]}`,
-        split: "test",
-        expected_outcome: "matched",
-        relevant_skill_ids: [skill.skill_id],
-      });
-    } else {
+  for (const split of ["tune", "test"] as const) {
+    const splitSkills = bySplit[split];
+    for (let i = 0; i < splitSkills.length; i++) {
+      const skill = splitSkills[i]!;
       cases.push({
-        query: `execute task related to ${skill.title}`,
-        split: "test",
+        query: matchedQuery(skill, i),
+        split,
         expected_outcome: "matched",
         relevant_skill_ids: [skill.skill_id],
       });
     }
-  }
 
-  // --- 2. Ambiguous Cases ---
-  if (skills.length >= 2) {
-    // Pair skills for ambiguous multi-match
-    for (let i = 0; i < skills.length - 1; i += 2) {
-      const s1 = skills[i]!;
-      const s2 = skills[i + 1]!;
-      const split: DecisionSplit = i % 4 === 0 ? "tune" : "test";
-      cases.push({
-        query: `automated task using ${s1.title} and ${s2.title}`,
-        split,
-        expected_outcome: "ambiguous",
-        relevant_skill_ids: [s1.skill_id, s2.skill_id],
-      });
-    }
-  } else {
-    // Fallback ambiguous cases if fewer than 2 skills
-    cases.push({
-      query: "automate browser workflow testing",
-      split: "tune",
-      expected_outcome: "ambiguous",
-      relevant_skill_ids: ["mock-e2e", "mock-browser"],
-    });
-    cases.push({
-      query: "extract and fetch clean web text",
-      split: "test",
-      expected_outcome: "ambiguous",
-      relevant_skill_ids: ["mock-fetch", "mock-extract"],
-    });
-  }
-
-  // Ensure both tune and test have ambiguous cases
-  if (!cases.some((c) => c.split === "tune" && c.expected_outcome === "ambiguous")) {
-    const sIds = skills.length >= 2 ? [skills[0]!.skill_id, skills[1]!.skill_id] : ["mock-a", "mock-b"];
+    const first = splitSkills[0]!;
+    const second = splitSkills[1]!;
     cases.push({
-      query: "integrated workflow multi skill query",
-      split: "tune",
+      query: ambiguousQuery(first, second),
+      split,
       expected_outcome: "ambiguous",
-      relevant_skill_ids: sIds,
+      relevant_skill_ids: [first.skill_id, second.skill_id],
     });
-  }
-  if (!cases.some((c) => c.split === "test" && c.expected_outcome === "ambiguous")) {
-    const sIds = skills.length >= 2 ? [skills[0]!.skill_id, skills[1]!.skill_id] : ["mock-a", "mock-b"];
     cases.push({
-      query: "combined operations multi skill query",
-      split: "test",
-      expected_outcome: "ambiguous",
-      relevant_skill_ids: sIds,
-    });
-  }
-
-  // --- 3. No Match Cases ---
-  GENERIC_NO_MATCH_QUERIES.forEach((q, idx) => {
-    cases.push({
-      query: q,
-      split: idx % 2 === 0 ? "tune" : "test",
+      query: nearMissQuery(first, second),
+      split,
       expected_outcome: "no_match",
       relevant_skill_ids: [],
     });
-  });
 
-  // Ensure both splits have at least 1 matched case if skills were empty
-  if (skills.length === 0) {
-    cases.push({
-      query: "run mock container action",
-      split: "tune",
-      expected_outcome: "matched",
-      relevant_skill_ids: ["mock-container"],
-    });
-    cases.push({
-      query: "search mock API docs",
-      split: "test",
-      expected_outcome: "matched",
-      relevant_skill_ids: ["mock-docs"],
-    });
+    for (let i = cases.filter((item) => item.split === split).length; i < targetPerSplit; i++) {
+      const left = splitSkills[i % splitSkills.length]!;
+      const right = splitSkills[(i + 1) % splitSkills.length]!;
+      cases.push({
+        query: i % 2 === 0 ? matchedQuery(left, i) : nearMissQuery(left, right),
+        split,
+        expected_outcome: i % 2 === 0 ? "matched" : "no_match",
+        relevant_skill_ids: i % 2 === 0 ? [left.skill_id] : [],
+      });
+    }
   }
 
   return cases;

package/src/server.ts

@@ -4,7 +4,7 @@ import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
 import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
 import { z } from "zod";
 import { createClients } from "./clients";
-import { expandHome, loadConfig, rerankerFingerprint, resolveConfigPath } from "./config";
+import { loadConfig, resolveConfigPath } from "./config";
 import { ConfigWatcher, type ReloadStatus } from "./config-watcher";
 import { RuntimeSnapshotManager } from "./snapshot";
 import {
@@ -28,11 +28,6 @@ import {
   RELOADABLE_KEYS,
   RESTART_REQUIRED_KEYS,
 } from "./config-service";
-import {
-  applyCalibrationRun,
-  getCalibrationRun,
-  listCalibrationRuns,
-} from "./calibrate";
 
 export const metricsRegistry = new MetricsRegistry();
 export const readinessState = new ReadinessState();
@@ -381,7 +376,7 @@ export async function startServer(opts?: {
               JSON.stringify({
                 config_read: true,
                 config_write: !isExternallyManaged,
-                calibration: true,
+                calibration: false,
                 persistence: isExternallyManaged
                   ? "externally_managed"
                   : "writable",
@@ -465,70 +460,15 @@ export async function startServer(opts?: {
           }
 
           if (url.pathname.startsWith("/admin/v1/calibrations")) {
-            const { db } = await getRuntime();
-            if (
-              req.method === "GET" &&
-              url.pathname === "/admin/v1/calibrations"
-            ) {
-              const runs = listCalibrationRuns(db);
-              return new Response(JSON.stringify(runs), {
-                status: 200,
-                headers,
-              });
-            }
-            const runIdMatch = url.pathname.match(
-              /^\/admin\/v1\/calibrations\/([^\/]+)$/,
-            );
-            if (req.method === "GET" && runIdMatch && runIdMatch[1]) {
-              const runId = runIdMatch[1];
-              const run = getCalibrationRun(db, runId);
-              if (!run)
-                return new Response(
-                  JSON.stringify({ error: "Calibration run not found" }),
-                  { status: 404, headers },
-                );
-              return new Response(JSON.stringify(run), {
-                status: 200,
-                headers,
-              });
-            }
-            if (
-              req.method === "POST" &&
-              url.pathname === "/admin/v1/calibrations"
-            ) {
-              const runId = "run_" + Math.random().toString(36).slice(2, 10);
-              return new Response(
-                JSON.stringify({ run_id: runId, status: "running" }),
-                { status: 202, headers },
-              );
-            }
-            const applyMatch = url.pathname.match(
-              /^\/admin\/v1\/calibrations\/([^\/]+)\/apply$/,
+            return new Response(
+              JSON.stringify({
+                error: "not_implemented",
+                message:
+                  "Remote calibration is not implemented in this release. " +
+                  "Run `skillmux calibrate` against a local target.",
+              }),
+              { status: 501, headers },
             );
-            if (req.method === "POST" && applyMatch && applyMatch[1]) {
-              const runId = applyMatch[1];
-              const run = getCalibrationRun(db, runId);
-              if (!run)
-                return new Response(
-                  JSON.stringify({ error: "Calibration run not found" }),
-                  { status: 404, headers },
-                );
-              const active = snapshots.acquire();
-              const currentRerankerFingerprint = rerankerFingerprint(
-                active.snapshot.config,
-              );
-              active.release();
-              await applyCalibrationRun(
-                db,
-                runId,
-                expandHome(configPath),
-                { currentRerankerFingerprint },
-              );
-              return new Response(JSON.stringify({ ok: true, run_id: runId }), {
-                status: 200,
-                headers,
-              });
-            }
           }
 
           return new Response("Not Found", { status: 404, headers });