@klhapp/skillmux 1.5.2 → 1.6.0

package/CHANGELOG.md

@@ -5,6 +5,13 @@ All notable changes to this project are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.6.0](https://github.com/klhq/skillmux/compare/v1.5.2...v1.6.0) (2026-08-18)
+
+
+### Added
+
+* **calibrate:** add tune safety buffers ([#120](https://github.com/klhq/skillmux/issues/120)) ([66d501c](https://github.com/klhq/skillmux/commit/66d501c4ac16956e8e887622e5178caef743bec7))
+
 ## [1.5.2](https://github.com/klhq/skillmux/compare/v1.5.1...v1.5.2) (2026-08-17)
 
 

package/docs/calibration.md

@@ -89,6 +89,42 @@ The operator owns the labels: supply or review the cases, start the run,
 inspect its evidence, and explicitly apply an acceptable result. A successful
 run never changes live thresholds by itself.
 
+## Certification gates and preflight feasibility
+
+Calibration certifies threshold policies against statistical confidence gates before allowing them to be applied:
+
+| Flag | Default | Description |
+|---|---|---|
+| `--min-auto-match-precision` | `0.75` | Minimum 95% Wilson score lower confidence bound on auto-match precision |
+| `--min-auto-match-count` | `15` | Minimum number of auto-matches required in evaluation |
+| `--min-retrieval-recall-at-k` | `0.95` | Minimum top-k retrieval recall on matchable queries |
+| `--min-delivered-shortlist-recall-at-k` | `0.95` | Minimum delivered shortlist recall on matchable queries |
+| `--tune-auto-match-precision-buffer` | `0.03` | Tune-only selection buffer for Wilson auto-match precision lower bound |
+| `--tune-auto-match-count-buffer` | `3` | Tune-only selection buffer for minimum auto-match count |
+| `--tune-delivered-shortlist-recall-buffer` | `0.02` | Tune-only selection buffer for delivered shortlist recall |
+
+### Tune selection buffers vs production gates
+
+Tune selection buffers ensure that threshold optimization selects policies with sufficient headroom beyond production gates. A candidate policy during tune search must satisfy the production gates plus their respective tune selection buffers. Test-split certification evaluates selected policies against the original production gates without selection buffers.
+
+### Wilson lower confidence bound and evidence size
+
+`min-auto-match-precision` is evaluated not as raw sample accuracy, but as a **95% Wilson score lower confidence bound** ($z \approx 1.960$). This accounts for statistical uncertainty in small datasets.
+
+Because the Wilson lower bound penalizes small sample sizes:
+- A gate of **0.75** lower bound requires at least **15** flawless (15/15) auto-matches ($\text{Wilson}(15, 15) \approx 0.7961$).
+- A 20-case tune matched split can achieve at most $\text{Wilson}(20, 20) \approx 0.8389$.
+- A gate of **0.99** lower bound is statistically impossible on small datasets; it requires at least **381** flawless auto-matches ($\text{Wilson}(381, 381) \approx 0.9900$).
+
+### Preflight feasibility check
+
+To avoid running expensive remote embeddings and rerankings on gates that can never pass, Skillmux executes a **preflight feasibility calculation** immediately after loading the dataset and before creating a running calibration record:
+
+$$\text{effective\_trials} = \max(N_{\text{tune\_matched}}, \text{minAutoMatchCount})$$
+$$\text{max\_attainable\_precision} = \text{WilsonLowerBound}(N_{\text{tune\_matched}}, \text{effective\_trials})$$
+
+If $\text{max\_attainable\_precision} < \text{minAutoMatchPrecision}$, calibration fails immediately with an actionable error indicating the requested precision, requested count, available tune matched cases, and maximum attainable lower bound.
+
 ## Reading a run
 
 A `run_id` identifies one immutable calibration attempt and its evidence.
@@ -156,7 +192,7 @@ Provenance: the small synthetic corpus and labelled decision cases in
 [`tests/router-core.spec.test.ts`](../tests/router-core.spec.test.ts), with the
 wire contract captured by
 [`tests/fixtures/reranker/jina-v1-request.json`](../tests/fixtures/reranker/jina-v1-request.json).
-That fixture is below the default 30-auto-match certification minimum, so the
+That fixture is below the default 15-auto-match certification minimum, so the
 values are a smoke-test/reference profile, not a completed calibration run.
 Run the lifecycle above against the deployment's real corpus before enabling
 automatic matches in production.

package/docs/cli.md

@@ -298,9 +298,21 @@ certification gates, run evidence, reference values, and the complete operator
 lifecycle.
 
 ```sh
-# Run calibration on a dataset with the default four workers
+# Run calibration on a dataset with default certification gates (min precision 0.75, min count 15)
 skillmux calibrate run --dataset ./eval/queries.json
 
+# Specify explicit certification gates and tune selection buffers
+# Note: --min-auto-match-precision is interpreted as a 95% Wilson lower confidence bound.
+# Calibration preflight validates that the requested gate is mathematically attainable on the tune split.
+skillmux calibrate run --dataset ./eval/queries.json \
+  --min-auto-match-precision 0.75 \
+  --min-auto-match-count 15 \
+  --min-retrieval-recall-at-k 0.95 \
+  --min-delivered-shortlist-recall-at-k 0.95 \
+  --tune-auto-match-precision-buffer 0.03 \
+  --tune-auto-match-count-buffer 3 \
+  --tune-delivered-shortlist-recall-buffer 0.02
+
 # Set the bounded worker count
 skillmux calibrate run --dataset ./eval/queries.json --concurrency 6
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@klhapp/skillmux",
-  "version": "1.5.2",
+  "version": "1.6.0",
   "description": "Skill management and retrieval for AI agents: sync native skills across clients and route the long tail over MCP",
   "type": "module",
   "private": false,

package/src/adapters.ts

@@ -2,6 +2,7 @@ import { createHash } from "node:crypto";
 import { join } from "node:path";
 import {
   applyCalibrationRun,
+  assertCalibrationFeasibility,
   computeCorpusFingerprint,
   createInitialCalibrationRun,
   finalizeCalibrationRun,
@@ -36,6 +37,7 @@ import {
 } from "./config-service";
 import type { ResolvedTarget } from "./context";
 import {
+  classifyInferenceError,
   configure,
   retrieveAndRerankSnapshot,
   syncVaultIfNeeded,
@@ -43,6 +45,25 @@ import {
 } from "./router-core";
 import type { Clients, Config } from "./types";
 
+/**
+ * Maximum retry attempts for transient reranker availability errors during calibration.
+ *
+ * Single-capacity remote reranker endpoints can suffer isolated blips (e.g. transient
+ * connection reset or momentary 503) during long multi-case calibration runs.
+ * A small bounded budget of 2 retries (3 attempts total) allows calibration to ride out
+ * transient blips without stalling on hard failures.
+ */
+export const CALIBRATION_RERANK_MAX_RETRIES = 2;
+
+/**
+ * Base backoff delay (ms) between reranker retries during calibration.
+ *
+ * A conservative nonzero delay gives recovering remote endpoints time to settle.
+ * Because reranker admission is strictly FIFO serialized, waiting workers remain
+ * queued rather than creating synchronized retry storms.
+ */
+export const CALIBRATION_RERANK_RETRY_BACKOFF_MS = 250;
+
 export interface Capabilities {
   config_read: boolean;
   config_write: boolean;
@@ -109,6 +130,9 @@ export interface TargetAdapter {
     minRetrievalRecallAtK?: number;
     minDeliveredShortlistRecallAtK?: number;
     minAutoMatchCount?: number;
+    tuneAutoMatchPrecisionBuffer?: number;
+    tuneAutoMatchCountBuffer?: number;
+    tuneDeliveredShortlistRecallBuffer?: number;
     concurrency?: number;
     resumeRunId?: string;
     onProgress?: (completed: number, total: number) => void;
@@ -205,6 +229,9 @@ export class LocalAdapter implements TargetAdapter {
     minRetrievalRecallAtK?: number;
     minDeliveredShortlistRecallAtK?: number;
     minAutoMatchCount?: number;
+    tuneAutoMatchPrecisionBuffer?: number;
+    tuneAutoMatchCountBuffer?: number;
+    tuneDeliveredShortlistRecallBuffer?: number;
     concurrency?: number;
     resumeRunId?: string;
     onProgress?: (completed: number, total: number) => void;
@@ -215,7 +242,47 @@ export class LocalAdapter implements TargetAdapter {
     const wallStart = collectTiming ? performance.now() : 0;
 
     const config = await loadConfig(this.configPath);
-    const clients = this.clients ?? createClients(config);
+    const baseClients = this.clients ?? createClients(config);
+
+    // Bounded admission at the calibration reranker boundary: serialize rerank calls
+    // so concurrent case retrieval (embedding, lexical, vector) does not overwhelm
+    // a single-capacity remote reranker endpoint.
+    //
+    // If an admitted request encounters a transient reranker_unavailable error,
+    // retry with conservative backoff up to CALIBRATION_RERANK_MAX_RETRIES.
+    // Non-availability errors (protocol, timeouts) and permanent failures fail closed.
+    let rerankQueue = Promise.resolve() as Promise<any>;
+    const serializedRerank: typeof baseClients.rerank = baseClients.rerank
+      ? (query, docs) => {
+          const run = async () => {
+            let attempt = 0;
+            while (true) {
+              try {
+                return await baseClients.rerank!(query, docs);
+              } catch (err) {
+                attempt++;
+                const degradationReason = classifyInferenceError("reranker", err);
+                if (
+                  degradationReason === "reranker_unavailable" &&
+                  attempt <= CALIBRATION_RERANK_MAX_RETRIES
+                ) {
+                  await Bun.sleep(CALIBRATION_RERANK_RETRY_BACKOFF_MS * attempt);
+                  continue;
+                }
+                throw err;
+              }
+            }
+          };
+          const next = rerankQueue.then(run, run);
+          rerankQueue = next.catch(() => {});
+          return next;
+        }
+      : undefined;
+
+    const clients: Clients = {
+      ...baseClients,
+      rerank: serializedRerank,
+    };
     configure({ config, clients });
 
     // Measure vault synchronization
@@ -246,6 +313,7 @@ export class LocalAdapter implements TargetAdapter {
       datasetFile,
       indexedSkills.map((skill) => skill.skill_id),
     );
+
     const fingerprint = rerankerFingerprint(config);
     if (!fingerprint) {
       throw new Error("A configured remote reranker is required to record calibration.");
@@ -258,15 +326,27 @@ export class LocalAdapter implements TargetAdapter {
       k_vector: config.recall.k_vector,
       k_rerank: config.recall.k_rerank ?? Math.min(10, config.recall.k_lexical + config.recall.k_vector),
     };
-    const minAutoMatchPrecision = opts?.minAutoMatchPrecision ?? 0.99;
-    const minAutoMatchCount = opts?.minAutoMatchCount ?? 30;
+    const minAutoMatchPrecision = opts?.minAutoMatchPrecision ?? 0.75;
+    const minAutoMatchCount = opts?.minAutoMatchCount ?? 15;
     const minDeliveredShortlistRecallAtK =
       opts?.minDeliveredShortlistRecallAtK ??
       opts?.minRetrievalRecallAtK ??
       0.95;
     const minShortlistRecallAt5 = opts?.minRetrievalRecallAtK ?? 0.95;
+    const tuneAutoMatchPrecisionBuffer = opts?.tuneAutoMatchPrecisionBuffer ?? 0.03;
+    const tuneAutoMatchCountBuffer = opts?.tuneAutoMatchCountBuffer ?? 3;
+    const tuneDeliveredShortlistRecallBuffer = opts?.tuneDeliveredShortlistRecallBuffer ?? 0.02;
     const concurrency = opts?.concurrency ?? 4;
 
+    assertCalibrationFeasibility(cases, {
+      minAutoMatchPrecision,
+      minAutoMatchCount,
+      minDeliveredShortlistRecallAtK,
+      tuneAutoMatchPrecisionBuffer,
+      tuneAutoMatchCountBuffer,
+      tuneDeliveredShortlistRecallBuffer,
+    });
+
     const db = openCalibrateDb(expandHome(config.state_dir));
     let runId: string;
     let initialObservations: Map<number, QueryObservation> | undefined = undefined;
@@ -326,6 +406,30 @@ export class LocalAdapter implements TargetAdapter {
         if (existingRun.min_shortlist_recall_at_5 !== minShortlistRecallAt5) {
           throw new Error("Certification gate mismatch: min_retrieval_recall_at_k differs from original run");
         }
+        if (
+          existingRun.tune_auto_match_precision_buffer !== undefined &&
+          existingRun.tune_auto_match_precision_buffer !== tuneAutoMatchPrecisionBuffer
+        ) {
+          throw new Error(
+            "Certification gate mismatch: tune_auto_match_precision_buffer differs from original run",
+          );
+        }
+        if (
+          existingRun.tune_auto_match_count_buffer !== undefined &&
+          existingRun.tune_auto_match_count_buffer !== tuneAutoMatchCountBuffer
+        ) {
+          throw new Error(
+            "Certification gate mismatch: tune_auto_match_count_buffer differs from original run",
+          );
+        }
+        if (
+          existingRun.tune_delivered_shortlist_recall_buffer !== undefined &&
+          existingRun.tune_delivered_shortlist_recall_buffer !== tuneDeliveredShortlistRecallBuffer
+        ) {
+          throw new Error(
+            "Certification gate mismatch: tune_delivered_shortlist_recall_buffer differs from original run",
+          );
+        }
 
         initialObservations = getCalibrationObservations(db, runId);
         if (initialObservations.size === 0 && existingRun.observations && existingRun.observations.length > 0) {
@@ -348,6 +452,9 @@ export class LocalAdapter implements TargetAdapter {
           min_auto_match_count: minAutoMatchCount,
           min_delivered_shortlist_recall_at_k: minDeliveredShortlistRecallAtK,
           min_shortlist_recall_at_5: minShortlistRecallAt5,
+          tune_auto_match_precision_buffer: tuneAutoMatchPrecisionBuffer,
+          tune_auto_match_count_buffer: tuneAutoMatchCountBuffer,
+          tune_delivered_shortlist_recall_buffer: tuneDeliveredShortlistRecallBuffer,
         });
       }
 
@@ -395,6 +502,9 @@ export class LocalAdapter implements TargetAdapter {
         minRetrievalRecallAtK: minShortlistRecallAt5,
         minDeliveredShortlistRecallAtK,
         minAutoMatchCount,
+        tuneAutoMatchPrecisionBuffer,
+        tuneAutoMatchCountBuffer,
+        tuneDeliveredShortlistRecallBuffer,
         concurrency,
         initialObservations,
         onProgress,
@@ -425,6 +535,7 @@ export class LocalAdapter implements TargetAdapter {
         selected_thresholds: result.selected_thresholds,
         tune_metrics: result.tune_metrics,
         test_metrics: result.test_metrics,
+        tune_gate_slack: result.tune_gate_slack,
         observations: result.observations,
       });
 
@@ -670,6 +781,9 @@ export class RemoteAdapter implements TargetAdapter {
     minRetrievalRecallAtK?: number;
     minDeliveredShortlistRecallAtK?: number;
     minAutoMatchCount?: number;
+    tuneAutoMatchPrecisionBuffer?: number;
+    tuneAutoMatchCountBuffer?: number;
+    tuneDeliveredShortlistRecallBuffer?: number;
     concurrency?: number;
     resumeRunId?: string;
     onProgress?: (completed: number, total: number) => void;

package/src/calibrate.ts

@@ -330,6 +330,12 @@ export type CalibrationFailureReason =
   | "insufficient_sample"
   | "test_certification_failed";
 
+export interface TuneGateSlack {
+  auto_match_precision_lower_bound_slack: number;
+  auto_match_count_slack: number;
+  delivered_shortlist_recall_slack: number;
+}
+
 export interface CalibrationResult {
   status: CalibrationStatus;
   failed_reason?: CalibrationFailureReason;
@@ -337,6 +343,7 @@ export interface CalibrationResult {
   selected_thresholds?: SelectedThresholds;
   tune_metrics?: CalibrationMetrics;
   test_metrics?: CalibrationTestMetrics;
+  tune_gate_slack?: TuneGateSlack;
 }
 
 export interface RunCalibrationOptions {
@@ -357,6 +364,12 @@ export interface RunCalibrationOptions {
   minDeliveredShortlistRecallAtK?: number;
   /** Default: 30 */
   minAutoMatchCount?: number;
+  /** Tune-only selection buffer for Wilson auto-match precision lower bound. Default: 0.03 */
+  tuneAutoMatchPrecisionBuffer?: number;
+  /** Tune-only selection buffer for minimum auto-match count. Default: 3 */
+  tuneAutoMatchCountBuffer?: number;
+  /** Tune-only selection buffer for delivered shortlist recall. Default: 0.02 */
+  tuneDeliveredShortlistRecallBuffer?: number;
   candidateLimit: number;
   /** Default: 4 */
   concurrency?: number;
@@ -465,6 +478,97 @@ export function wilsonLowerBound(successes: number, total: number): number {
   return Math.max(0, (centre - adjustment) / denominator);
 }
 
+/**
+ * Computes the maximum attainable 95% Wilson lower bound on auto-match precision
+ * for the tune split subject to minAutoMatchCount and the number of matched tune cases.
+ * Under perfect ranking/classification (zero false positives), at most all true matched
+ * tune cases can be auto-matched.
+ */
+export function computeMaxAttainablePrecisionLowerBound(
+  cases: DecisionCase[],
+  minAutoMatchCount: number,
+): { tuneMatchedCount: number; maxAttainablePrecision: number } {
+  const tuneMatchedCount = cases.filter(
+    (c) => c.split === "tune" && c.expected_outcome === "matched",
+  ).length;
+  const effectiveTrials = Math.max(tuneMatchedCount, minAutoMatchCount);
+  const maxAttainablePrecision = wilsonLowerBound(tuneMatchedCount, effectiveTrials);
+  return { tuneMatchedCount, maxAttainablePrecision };
+}
+
+/**
+ * Asserts that the requested auto-match precision and count gates are mathematically
+ * attainable on the given dataset's tune split before inference begins.
+ */
+export function assertCalibrationFeasibility(
+  cases: DecisionCase[],
+  gates: {
+    minAutoMatchPrecision: number;
+    minAutoMatchCount: number;
+    minDeliveredShortlistRecallAtK?: number;
+    tuneAutoMatchPrecisionBuffer?: number;
+    tuneAutoMatchCountBuffer?: number;
+    tuneDeliveredShortlistRecallBuffer?: number;
+  },
+): void {
+  const precisionBuffer = gates.tuneAutoMatchPrecisionBuffer ?? 0.03;
+  const countBuffer = gates.tuneAutoMatchCountBuffer ?? 3;
+  const recallBuffer = gates.tuneDeliveredShortlistRecallBuffer ?? 0.02;
+
+  if (precisionBuffer < 0 || !Number.isFinite(precisionBuffer)) {
+    throw new Error("tune_auto_match_precision_buffer must be a non-negative number");
+  }
+  if (countBuffer < 0 || !Number.isInteger(countBuffer)) {
+    throw new Error("tune_auto_match_count_buffer must be a non-negative integer");
+  }
+  if (recallBuffer < 0 || !Number.isFinite(recallBuffer)) {
+    throw new Error("tune_delivered_shortlist_recall_buffer must be a non-negative number");
+  }
+
+  const effectivePrecision = gates.minAutoMatchPrecision + precisionBuffer;
+  if (effectivePrecision > 1.0) {
+    throw new Error(
+      `Requested calibration gates with tune buffers are mathematically impossible: ` +
+      `effective min_auto_match_precision (${effectivePrecision.toFixed(2)}) exceeds 1.0 ` +
+      `(min_auto_match_precision=${gates.minAutoMatchPrecision}, tune_buffer=${precisionBuffer}). ` +
+      `Lower --min-auto-match-precision or --tune-auto-match-precision-buffer.`,
+    );
+  }
+
+  const baseDeliveredRecall = gates.minDeliveredShortlistRecallAtK ?? 0.95;
+  const effectiveDeliveredRecall = baseDeliveredRecall + recallBuffer;
+  if (effectiveDeliveredRecall > 1.0) {
+    throw new Error(
+      `Requested calibration gates with tune buffers are mathematically impossible: ` +
+      `effective min_delivered_shortlist_recall_at_k (${effectiveDeliveredRecall.toFixed(2)}) exceeds 1.0 ` +
+      `(min_delivered_shortlist_recall_at_k=${baseDeliveredRecall}, tune_buffer=${recallBuffer}). ` +
+      `Lower --min-delivered-shortlist-recall-at-k or --tune-delivered-shortlist-recall-buffer.`,
+    );
+  }
+
+  const effectiveCount = gates.minAutoMatchCount + countBuffer;
+  const { tuneMatchedCount, maxAttainablePrecision } =
+    computeMaxAttainablePrecisionLowerBound(cases, effectiveCount);
+  if (effectiveCount > tuneMatchedCount) {
+    throw new Error(
+      `Requested calibration gates are mathematically unattainable on this dataset: ` +
+      `effective min_auto_match_count (${effectiveCount}) exceeds the number of matched cases in the ` +
+      `tune split (${tuneMatchedCount}) (min_auto_match_count=${gates.minAutoMatchCount}, tune_buffer=${countBuffer}). ` +
+      `Lower --min-auto-match-count or --tune-auto-match-count-buffer, or supply a dataset with more matched tune cases.`,
+    );
+  }
+  if (maxAttainablePrecision < effectivePrecision) {
+    throw new Error(
+      `Requested calibration gates are mathematically unattainable on this dataset: ` +
+      `min_auto_match_precision=${gates.minAutoMatchPrecision} requires more evidence than the ` +
+      `tune split provides (${tuneMatchedCount} matched cases, min_auto_match_count=${gates.minAutoMatchCount}, effective min_auto_match_count=${effectiveCount}). ` +
+      `The maximum attainable 95% Wilson lower bound under perfect classification is ` +
+      `${maxAttainablePrecision.toFixed(4)}. ` +
+      `Lower --min-auto-match-precision or supply a larger dataset.`,
+    );
+  }
+}
+
 function computeTestMetrics(
   observations: QueryObservation[],
   thresholds: SelectedThresholds,
@@ -799,10 +903,13 @@ export async function runCalibration(opts: RunCalibrationOptions): Promise<Calib
     getCandidates,
     getRankedCandidates,
     reranker,
-    minAutoMatchPrecision = 0.99,
+    minAutoMatchPrecision = 0.75,
     minRetrievalRecallAtK = 0.95,
     minDeliveredShortlistRecallAtK = minRetrievalRecallAtK,
-    minAutoMatchCount = 30,
+    minAutoMatchCount = 15,
+    tuneAutoMatchPrecisionBuffer = 0.03,
+    tuneAutoMatchCountBuffer = 3,
+    tuneDeliveredShortlistRecallBuffer = 0.02,
     candidateLimit,
     concurrency = 4,
     initialObservations,
@@ -904,9 +1011,16 @@ export async function runCalibration(opts: RunCalibrationOptions): Promise<Calib
     minDeliveredShortlistRecallAtK,
     minAutoMatchCount,
   };
+  const tuneGates = {
+    minAutoMatchPrecision: minAutoMatchPrecision + tuneAutoMatchPrecisionBuffer,
+    minRetrievalRecallAtK,
+    minDeliveredShortlistRecallAtK:
+      minDeliveredShortlistRecallAtK + tuneDeliveredShortlistRecallBuffer,
+    minAutoMatchCount: minAutoMatchCount + tuneAutoMatchCountBuffer,
+  };
   const selection = selectThresholds(
     tuneObs,
-    gates,
+    tuneGates,
     candidateLimit,
   );
 
@@ -919,8 +1033,16 @@ export async function runCalibration(opts: RunCalibrationOptions): Promise<Calib
   }
   const selected = selection.selected;
 
-  // --- Step 3: Report tune metrics ---
+  // --- Step 3: Report tune metrics & explicit tune gate slack ---
   const tune_metrics = computeMetrics(tuneObs, selected, candidateLimit);
+  const tune_gate_slack: TuneGateSlack = {
+    auto_match_precision_lower_bound_slack:
+      tune_metrics.auto_match_precision_lower_bound - minAutoMatchPrecision,
+    auto_match_count_slack:
+      tune_metrics.auto_match_count - minAutoMatchCount,
+    delivered_shortlist_recall_slack:
+      tune_metrics.delivered_shortlist_recall_at_k - minDeliveredShortlistRecallAtK,
+  };
 
   // --- Step 4: Evaluate untouched test split ---
   const test_metrics = computeTestMetrics(testObs, selected, candidateLimit);
@@ -932,10 +1054,18 @@ export async function runCalibration(opts: RunCalibrationOptions): Promise<Calib
       selected_thresholds: selected,
       tune_metrics,
       test_metrics,
+      tune_gate_slack,
     };
   }
 
-  return { status: "completed", observations, selected_thresholds: selected, tune_metrics, test_metrics };
+  return {
+    status: "completed",
+    observations,
+    selected_thresholds: selected,
+    tune_metrics,
+    test_metrics,
+    tune_gate_slack,
+  };
 }
 
 // ---------------------------------------------------------------------------
@@ -966,6 +1096,10 @@ export interface CalibrationRunRecord {
   min_auto_match_count?: number;
   min_delivered_shortlist_recall_at_k?: number;
   min_shortlist_recall_at_5: number;
+  tune_auto_match_precision_buffer?: number;
+  tune_auto_match_count_buffer?: number;
+  tune_delivered_shortlist_recall_buffer?: number;
+  tune_gate_slack?: TuneGateSlack;
   failed_reason?: CalibrationFailureReason;
   selected_thresholds?: SelectedThresholds;
   tune_metrics?: CalibrationMetrics;
@@ -990,6 +1124,9 @@ export interface CalibrationRunSummary {
   min_auto_match_count: number;
   min_delivered_shortlist_recall_at_k: number;
   min_shortlist_recall_at_5: number;
+  tune_auto_match_precision_buffer: number;
+  tune_auto_match_count_buffer: number;
+  tune_delivered_shortlist_recall_buffer: number;
   failed_reason?: CalibrationFailureReason;
 }
 
@@ -1053,7 +1190,11 @@ export function openCalibrateDb(stateDir: string): Database {
         dataset_provenance TEXT NOT NULL DEFAULT '{}',
         human_labelled_case_count INTEGER NOT NULL DEFAULT 0,
         imported_labelled_case_count INTEGER NOT NULL DEFAULT 0,
-        recall_settings TEXT NOT NULL DEFAULT '{}'
+        recall_settings TEXT NOT NULL DEFAULT '{}',
+        tune_auto_match_precision_buffer REAL NOT NULL DEFAULT 0.03,
+        tune_auto_match_count_buffer INTEGER NOT NULL DEFAULT 3,
+        tune_delivered_shortlist_recall_buffer REAL NOT NULL DEFAULT 0.02,
+        tune_gate_slack TEXT
       )`);
       db.run(
         `INSERT INTO calibration_runs_new (${selectCols}) SELECT ${selectCols} FROM calibration_runs`,
@@ -1106,6 +1247,18 @@ export function openCalibrateDb(stateDir: string): Database {
   if (!columns.some((column) => column.name === "recall_settings")) {
     db.run("ALTER TABLE calibration_runs ADD COLUMN recall_settings TEXT NOT NULL DEFAULT '{}'");
   }
+  if (!columns.some((column) => column.name === "tune_auto_match_precision_buffer")) {
+    db.run("ALTER TABLE calibration_runs ADD COLUMN tune_auto_match_precision_buffer REAL NOT NULL DEFAULT 0.03");
+  }
+  if (!columns.some((column) => column.name === "tune_auto_match_count_buffer")) {
+    db.run("ALTER TABLE calibration_runs ADD COLUMN tune_auto_match_count_buffer INTEGER NOT NULL DEFAULT 3");
+  }
+  if (!columns.some((column) => column.name === "tune_delivered_shortlist_recall_buffer")) {
+    db.run("ALTER TABLE calibration_runs ADD COLUMN tune_delivered_shortlist_recall_buffer REAL NOT NULL DEFAULT 0.02");
+  }
+  if (!columns.some((column) => column.name === "tune_gate_slack")) {
+    db.run("ALTER TABLE calibration_runs ADD COLUMN tune_gate_slack TEXT");
+  }
 
   db.run(`CREATE TABLE IF NOT EXISTS calibration_observations (
     run_id TEXT NOT NULL,
@@ -1135,6 +1288,9 @@ export interface CreateInitialCalibrationRunOptions {
   min_auto_match_count?: number;
   min_delivered_shortlist_recall_at_k?: number;
   min_shortlist_recall_at_5: number;
+  tune_auto_match_precision_buffer?: number;
+  tune_auto_match_count_buffer?: number;
+  tune_delivered_shortlist_recall_buffer?: number;
   dataset_provenance?: DatasetProvenanceSummary;
   recall_settings?: {
     k_lexical: number;
@@ -1161,8 +1317,10 @@ export function createInitialCalibrationRun(
       min_delivered_shortlist_recall_at_k, min_shortlist_recall_at_5, failed_reason,
       selected_thresholds, tune_metrics, test_metrics, observations,
       dataset_provenance, human_labelled_case_count, imported_labelled_case_count,
-      recall_settings
-    ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
+      recall_settings,
+      tune_auto_match_precision_buffer, tune_auto_match_count_buffer,
+      tune_delivered_shortlist_recall_buffer, tune_gate_slack
+    ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
     [
       run.run_id,
       run.created_at,
@@ -1186,6 +1344,10 @@ export function createInitialCalibrationRun(
       run.dataset_provenance?.human_labelled_case_count ?? 0,
       run.dataset_provenance?.imported_labelled_case_count ?? 0,
       JSON.stringify(run.recall_settings ?? {}),
+      run.tune_auto_match_precision_buffer ?? 0.03,
+      run.tune_auto_match_count_buffer ?? 3,
+      run.tune_delivered_shortlist_recall_buffer ?? 0.02,
+      null,
     ],
   );
 }
@@ -1251,6 +1413,7 @@ export interface FinalizeCalibrationRunOptions {
   selected_thresholds?: SelectedThresholds;
   tune_metrics?: CalibrationMetrics;
   test_metrics?: CalibrationTestMetrics;
+  tune_gate_slack?: TuneGateSlack;
   observations: QueryObservation[];
 }
 
@@ -1266,7 +1429,8 @@ export function finalizeCalibrationRun(
       selected_thresholds = ?,
       tune_metrics = ?,
       test_metrics = ?,
-      observations = ?
+      observations = ?,
+      tune_gate_slack = ?
      WHERE run_id = ?`,
     [
       run.status,
@@ -1275,6 +1439,7 @@ export function finalizeCalibrationRun(
       run.tune_metrics != null ? JSON.stringify(run.tune_metrics) : null,
       run.test_metrics != null ? JSON.stringify(run.test_metrics) : null,
       JSON.stringify(run.observations),
+      run.tune_gate_slack != null ? JSON.stringify(run.tune_gate_slack) : null,
       run.run_id,
     ],
   );
@@ -1296,8 +1461,10 @@ export function insertCalibrationRun(db: Database, run: CalibrationRunRecord): v
         min_delivered_shortlist_recall_at_k, min_shortlist_recall_at_5, failed_reason,
         selected_thresholds, tune_metrics, test_metrics, observations,
         dataset_provenance, human_labelled_case_count, imported_labelled_case_count,
-        recall_settings
-      ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
+        recall_settings,
+        tune_auto_match_precision_buffer, tune_auto_match_count_buffer,
+        tune_delivered_shortlist_recall_buffer, tune_gate_slack
+      ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
       [
         run.run_id,
         run.created_at,
@@ -1321,6 +1488,10 @@ export function insertCalibrationRun(db: Database, run: CalibrationRunRecord): v
         run.dataset_provenance?.human_labelled_case_count ?? 0,
         run.dataset_provenance?.imported_labelled_case_count ?? 0,
         JSON.stringify(run.recall_settings ?? {}),
+        run.tune_auto_match_precision_buffer ?? 0.03,
+        run.tune_auto_match_count_buffer ?? 3,
+        run.tune_delivered_shortlist_recall_buffer ?? 0.02,
+        run.tune_gate_slack != null ? JSON.stringify(run.tune_gate_slack) : null,
       ],
     );
     if (run.observations && run.observations.length > 0) {
@@ -1355,6 +1526,10 @@ interface RawCalibrationRow {
   human_labelled_case_count: number;
   imported_labelled_case_count: number;
   recall_settings: string;
+  tune_auto_match_precision_buffer: number | null;
+  tune_auto_match_count_buffer: number | null;
+  tune_delivered_shortlist_recall_buffer: number | null;
+  tune_gate_slack: string | null;
 }
 
 function parseMetrics(json: string): CalibrationMetrics {
@@ -1403,6 +1578,10 @@ function rowToRecord(row: RawCalibrationRow): CalibrationRunRecord {
     min_auto_match_count: row.min_auto_match_count,
     min_delivered_shortlist_recall_at_k: row.min_delivered_shortlist_recall_at_k,
     min_shortlist_recall_at_5: row.min_shortlist_recall_at_5,
+    tune_auto_match_precision_buffer: row.tune_auto_match_precision_buffer ?? 0.03,
+    tune_auto_match_count_buffer: row.tune_auto_match_count_buffer ?? 3,
+    tune_delivered_shortlist_recall_buffer: row.tune_delivered_shortlist_recall_buffer ?? 0.02,
+    tune_gate_slack: row.tune_gate_slack != null ? JSON.parse(row.tune_gate_slack) : undefined,
     failed_reason: row.failed_reason as CalibrationFailureReason | null ?? undefined,
     selected_thresholds: row.selected_thresholds != null
       ? (JSON.parse(row.selected_thresholds) as SelectedThresholds)
@@ -1445,7 +1624,9 @@ export function listCalibrationRuns(db: Database): CalibrationRunSummary[] {
         reranker_fingerprint, embedding_fingerprint, corpus_fingerprint, dataset_hash,
         candidate_limit,
         attempt_count, min_auto_match_precision, min_auto_match_count,
-        min_delivered_shortlist_recall_at_k, min_shortlist_recall_at_5, failed_reason,
+        min_delivered_shortlist_recall_at_k, min_shortlist_recall_at_5,
+        tune_auto_match_precision_buffer, tune_auto_match_count_buffer,
+        tune_delivered_shortlist_recall_buffer, failed_reason,
         human_labelled_case_count, imported_labelled_case_count
        FROM calibration_runs ORDER BY created_at DESC`,
     )

package/src/cli.ts

@@ -475,6 +475,9 @@ async function handleCalibrateCommand(
     let minRetrievalRecallAtK: number | undefined;
     let minDeliveredShortlistRecallAtK: number | undefined;
     let minAutoMatchCount: number | undefined;
+    let tuneAutoMatchPrecisionBuffer: number | undefined;
+    let tuneAutoMatchCountBuffer: number | undefined;
+    let tuneDeliveredShortlistRecallBuffer: number | undefined;
     let concurrency: number | undefined;
     let resumeRunId: string | undefined;
     let timing = false;
@@ -500,6 +503,15 @@ async function handleCalibrateCommand(
         if (!Number.isInteger(minAutoMatchCount) || minAutoMatchCount < 1) {
           throw new Error("--min-auto-match-count must be a positive integer");
         }
+      } else if (option === "--tune-auto-match-precision-buffer") {
+        tuneAutoMatchPrecisionBuffer = readNumber(option, args[++i]);
+      } else if (option === "--tune-auto-match-count-buffer") {
+        tuneAutoMatchCountBuffer = readNumber(option, args[++i]);
+        if (!Number.isInteger(tuneAutoMatchCountBuffer) || tuneAutoMatchCountBuffer < 0) {
+          throw new Error("--tune-auto-match-count-buffer must be a non-negative integer");
+        }
+      } else if (option === "--tune-delivered-shortlist-recall-buffer") {
+        tuneDeliveredShortlistRecallBuffer = readNumber(option, args[++i]);
       } else if (option === "--concurrency") {
         const raw = args[++i];
         if (raw === undefined) throw new Error("--concurrency requires a value");
@@ -513,6 +525,8 @@ async function handleCalibrateCommand(
         if (!resumeRunId) throw new Error("--resume requires a run_id value");
       } else if (option === "--timing") {
         timing = true;
+      } else if (option === "--json") {
+        // Global flag accepted in the documented subcommand position
       } else {
         throw new Error(`unknown calibrate run option: ${option}`);
       }
@@ -521,6 +535,8 @@ async function handleCalibrateCommand(
       ["--min-auto-match-precision", minAutoMatchPrecision],
       ["--min-retrieval-recall-at-k", minRetrievalRecallAtK],
       ["--min-delivered-shortlist-recall-at-k", minDeliveredShortlistRecallAtK],
+      ["--tune-auto-match-precision-buffer", tuneAutoMatchPrecisionBuffer],
+      ["--tune-delivered-shortlist-recall-buffer", tuneDeliveredShortlistRecallBuffer],
     ] as const) {
       if (value !== undefined && (value < 0 || value > 1)) {
         throw new Error(`${flag} must be between 0 and 1`);
@@ -532,6 +548,9 @@ async function handleCalibrateCommand(
       minRetrievalRecallAtK,
       minDeliveredShortlistRecallAtK,
       minAutoMatchCount,
+      tuneAutoMatchPrecisionBuffer,
+      tuneAutoMatchCountBuffer,
+      tuneDeliveredShortlistRecallBuffer,
       concurrency,
       resumeRunId,
       timing,
@@ -703,6 +722,12 @@ Setup:
 
 Calibration:
   skillmux calibrate run [--dataset <path>] [--concurrency <n>] [--resume <run_id>]
+                         [--min-auto-match-precision <0..1>] [--min-auto-match-count <n>]
+                         [--min-retrieval-recall-at-k <0..1>]
+                         [--min-delivered-shortlist-recall-at-k <0..1>]
+                         [--tune-auto-match-precision-buffer <0..1>]
+                         [--tune-auto-match-count-buffer <n>]
+                         [--tune-delivered-shortlist-recall-buffer <0..1>]
                          [--timing] [--json]
   skillmux calibrate <list|show|apply|generate-dataset>