@klhapp/skillmux 1.5.1 → 1.6.0

package/CHANGELOG.md

@@ -5,6 +5,23 @@ All notable changes to this project are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.6.0](https://github.com/klhq/skillmux/compare/v1.5.2...v1.6.0) (2026-08-18)
+
+
+### Added
+
+* **calibrate:** add tune safety buffers ([#120](https://github.com/klhq/skillmux/issues/120)) ([66d501c](https://github.com/klhq/skillmux/commit/66d501c4ac16956e8e887622e5178caef743bec7))
+
+## [1.5.2](https://github.com/klhq/skillmux/compare/v1.5.1...v1.5.2) (2026-08-17)
+
+
+### Changed
+
+* **calibration:** parallelize resumable evaluations ([#115](https://github.com/klhq/skillmux/issues/115)) ([49b3fba](https://github.com/klhq/skillmux/commit/49b3fba43961aa7a6648eda89c5c08d994c7433a))
+* **calibration:** report aggregate stage timings ([#119](https://github.com/klhq/skillmux/issues/119)) ([086128f](https://github.com/klhq/skillmux/commit/086128fdba8ded82ccf6a51d2e0117a18c2eea17))
+* **calibration:** reuse synchronized retrieval snapshot ([#117](https://github.com/klhq/skillmux/issues/117)) ([1fcf295](https://github.com/klhq/skillmux/commit/1fcf2959123b4f9ca6841a2978a3fa5552c23122))
+* **retrieval:** expose opt-in stage timings ([#118](https://github.com/klhq/skillmux/issues/118)) ([f459c90](https://github.com/klhq/skillmux/commit/f459c90f440834cea2ee008300bad3e15771949e))
+
 ## [1.5.1](https://github.com/klhq/skillmux/compare/v1.5.0...v1.5.1) (2026-08-17)
 
 

package/docs/calibration.md

@@ -30,14 +30,101 @@ skillmux calibrate apply RUN_ID
 ```
 
 Skillmux retrieves candidates and reranks exactly once for each evaluation
-query. It caches those observations, searches thresholds on the `tune` split,
-then certifies the selected policy on the frozen `test` split. Calibration
-starts only when an operator invokes `calibrate run`.
+query. It runs four queries at a time by default. Set a different positive
+worker limit with `--concurrency N`. The CLI writes completed-case progress to
+stderr without exposing query text.
+
+### Timing report
+
+Add `--timing` to any `calibrate run` invocation to write an aggregate
+performance report to **stderr** after the run finishes (with a completed or
+failed-gates result; a thrown error produces no report). Stdout remains valid
+JSON under `--json --timing`.
+
+```sh
+skillmux calibrate run --dataset ./eval/queries.json --timing
+```
+
+The report uses stable snake_case field names in milliseconds:
+
+| Field | Description |
+|---|---|
+| `cases_total` | Total dataset cases |
+| `cases_executed` | Cases retrieved in this invocation |
+| `cases_reused` | Cases loaded from a prior interrupted run (resume) |
+| `wall_ms` | Wall-clock duration of the full calibrateRun operation |
+| `vault_sync_ms` | One-time vault synchronization before retrieval |
+| `cumulative_embedding_ms` | Total worker time in embedding across all queries |
+| `cumulative_lexical_ms` | Total worker time in lexical search across all queries |
+| `cumulative_vector_ms` | Total worker time in vector search across all queries |
+| `cumulative_reranker_ms` | Total worker time in reranking across all queries |
+| `cumulative_checkpoint_ms` | Total worker time writing observation checkpoints |
+| `policy_evaluation_ms` | Threshold selection and test-split certification |
+
+**Cumulative vs wall time.** The cumulative fields (`cumulative_embedding_ms`,
+`cumulative_lexical_ms`, `cumulative_vector_ms`, `cumulative_reranker_ms`,
+`cumulative_checkpoint_ms`) are total _worker time_ summed across all concurrent
+query retrievals. Because multiple queries run at the same time, the sum of these
+fields typically exceeds `wall_ms`. They measure how much time each stage
+consumed across all workers, not how much wall-clock time each stage accounted
+for. `cases_executed + cases_reused = cases_total`.
+
+Timing collection is fully disabled when `--timing` is absent; it does not affect
+calibration results, resume behavior, checkpoint durability, or JSON schemas.
+Skillmux checkpoints each observation in the calibration evidence database.
+If inference fails or you interrupt the process, find the `running` run with
+`calibrate list` and resume it with the same dataset and certification flags:
+
+```sh
+skillmux calibrate run --dataset ./eval/queries.json --resume RUN_ID
+```
+
+Resume rejects changes to the dataset, corpus, inference models, recall
+settings, candidate limit, or certification gates. After all observations
+exist, Skillmux searches thresholds on the `tune` split and certifies the
+selected policy on the frozen `test` split. Calibration starts only when an
+operator invokes `calibrate run`.
 
 The operator owns the labels: supply or review the cases, start the run,
 inspect its evidence, and explicitly apply an acceptable result. A successful
 run never changes live thresholds by itself.
 
+## Certification gates and preflight feasibility
+
+Calibration certifies threshold policies against statistical confidence gates before allowing them to be applied:
+
+| Flag | Default | Description |
+|---|---|---|
+| `--min-auto-match-precision` | `0.75` | Minimum 95% Wilson score lower confidence bound on auto-match precision |
+| `--min-auto-match-count` | `15` | Minimum number of auto-matches required in evaluation |
+| `--min-retrieval-recall-at-k` | `0.95` | Minimum top-k retrieval recall on matchable queries |
+| `--min-delivered-shortlist-recall-at-k` | `0.95` | Minimum delivered shortlist recall on matchable queries |
+| `--tune-auto-match-precision-buffer` | `0.03` | Tune-only selection buffer for Wilson auto-match precision lower bound |
+| `--tune-auto-match-count-buffer` | `3` | Tune-only selection buffer for minimum auto-match count |
+| `--tune-delivered-shortlist-recall-buffer` | `0.02` | Tune-only selection buffer for delivered shortlist recall |
+
+### Tune selection buffers vs production gates
+
+Tune selection buffers ensure that threshold optimization selects policies with sufficient headroom beyond production gates. A candidate policy during tune search must satisfy the production gates plus their respective tune selection buffers. Test-split certification evaluates selected policies against the original production gates without selection buffers.
+
+### Wilson lower confidence bound and evidence size
+
+`min-auto-match-precision` is evaluated not as raw sample accuracy, but as a **95% Wilson score lower confidence bound** ($z \approx 1.960$). This accounts for statistical uncertainty in small datasets.
+
+Because the Wilson lower bound penalizes small sample sizes:
+- A gate of **0.75** lower bound requires at least **15** flawless (15/15) auto-matches ($\text{Wilson}(15, 15) \approx 0.7961$).
+- A 20-case tune matched split can achieve at most $\text{Wilson}(20, 20) \approx 0.8389$.
+- A gate of **0.99** lower bound is statistically impossible on small datasets; it requires at least **381** flawless auto-matches ($\text{Wilson}(381, 381) \approx 0.9900$).
+
+### Preflight feasibility check
+
+To avoid running expensive remote embeddings and rerankings on gates that can never pass, Skillmux executes a **preflight feasibility calculation** immediately after loading the dataset and before creating a running calibration record:
+
+$$\text{effective\_trials} = \max(N_{\text{tune\_matched}}, \text{minAutoMatchCount})$$
+$$\text{max\_attainable\_precision} = \text{WilsonLowerBound}(N_{\text{tune\_matched}}, \text{effective\_trials})$$
+
+If $\text{max\_attainable\_precision} < \text{minAutoMatchPrecision}$, calibration fails immediately with an actionable error indicating the requested precision, requested count, available tune matched cases, and maximum attainable lower bound.
+
 ## Reading a run
 
 A `run_id` identifies one immutable calibration attempt and its evidence.
@@ -105,7 +192,7 @@ Provenance: the small synthetic corpus and labelled decision cases in
 [`tests/router-core.spec.test.ts`](../tests/router-core.spec.test.ts), with the
 wire contract captured by
 [`tests/fixtures/reranker/jina-v1-request.json`](../tests/fixtures/reranker/jina-v1-request.json).
-That fixture is below the default 30-auto-match certification minimum, so the
+That fixture is below the default 15-auto-match certification minimum, so the
 values are a smoke-test/reference profile, not a completed calibration run.
 Run the lifecycle above against the deployment's real corpus before enabling
 automatic matches in production.

package/docs/cli.md

@@ -298,9 +298,30 @@ certification gates, run evidence, reference values, and the complete operator
 lifecycle.
 
 ```sh
-# Run calibration on a dataset
+# Run calibration on a dataset with default certification gates (min precision 0.75, min count 15)
 skillmux calibrate run --dataset ./eval/queries.json
 
+# Specify explicit certification gates and tune selection buffers
+# Note: --min-auto-match-precision is interpreted as a 95% Wilson lower confidence bound.
+# Calibration preflight validates that the requested gate is mathematically attainable on the tune split.
+skillmux calibrate run --dataset ./eval/queries.json \
+  --min-auto-match-precision 0.75 \
+  --min-auto-match-count 15 \
+  --min-retrieval-recall-at-k 0.95 \
+  --min-delivered-shortlist-recall-at-k 0.95 \
+  --tune-auto-match-precision-buffer 0.03 \
+  --tune-auto-match-count-buffer 3 \
+  --tune-delivered-shortlist-recall-buffer 0.02
+
+# Set the bounded worker count
+skillmux calibrate run --dataset ./eval/queries.json --concurrency 6
+
+# Print aggregate performance timing to stderr
+skillmux calibrate run --dataset ./eval/queries.json --timing
+
+# Resume a running or interrupted attempt with the same inputs and gates
+skillmux calibrate run --dataset ./eval/queries.json --resume <run_id>
+
 # List stored calibration runs in the evidence store
 skillmux calibrate list
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@klhapp/skillmux",
-  "version": "1.5.1",
+  "version": "1.6.0",
   "description": "Skill management and retrieval for AI agents: sync native skills across clients and route the long tail over MCP",
   "type": "module",
   "private": false,