@klhapp/skillmux 1.5.2 → 1.7.0

package/CHANGELOG.md

@@ -5,6 +5,27 @@ All notable changes to this project are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.7.0](https://github.com/klhq/skillmux/compare/v1.6.0...v1.7.0) (2026-08-21)
+
+
+### Added
+
+* **calibration:** remove obsolete threshold calibration ([#125](https://github.com/klhq/skillmux/issues/125)) ([1432948](https://github.com/klhq/skillmux/commit/143294848c333c35186d70742ef59c230dc850bb))
+* **eval:** rank Skillmux 2.0 evaluation ([#124](https://github.com/klhq/skillmux/issues/124)) ([2fab68d](https://github.com/klhq/skillmux/commit/2fab68d73702913fea5a6e4d8f892c2802a65a88))
+* **ranking:** ranked-only Skillmux 2.0 runtime contract ([#122](https://github.com/klhq/skillmux/issues/122)) ([00810df](https://github.com/klhq/skillmux/commit/00810df92b7de60289dad41fe1fcb5a982686493))
+
+
+### Changed
+
+* **audit:** remove legacy classifier outcomes ([#126](https://github.com/klhq/skillmux/issues/126)) ([953edc4](https://github.com/klhq/skillmux/commit/953edc40147fb2f87eec5a57d53d6fa022566b77))
+
+## [1.6.0](https://github.com/klhq/skillmux/compare/v1.5.2...v1.6.0) (2026-08-18)
+
+
+### Added
+
+* **calibrate:** add tune safety buffers ([#120](https://github.com/klhq/skillmux/issues/120)) ([66d501c](https://github.com/klhq/skillmux/commit/66d501c4ac16956e8e887622e5178caef743bec7))
+
 ## [1.5.2](https://github.com/klhq/skillmux/compare/v1.5.1...v1.5.2) (2026-08-17)
 
 

package/README.md

@@ -231,18 +231,18 @@ Skillmux exposes two tools:
 
 | Tool | Input | Result |
 | --- | --- | --- |
-| `resolve_skill` | Natural-language task description | A matched skill, an ordered shortlist, or no match |
+| `resolve_skill` | Natural-language task description | A ranked shortlist of candidates |
 | `fetch_skill` | Exact `skill_id` | The current `SKILL.md` body, SHA-256 digest, and supporting-file paths |
 
 Skillmux uses the best available capability:
 
 1. SQLite FTS5 provides lexical retrieval and offline fallback.
 2. Local or remote embeddings add semantic recall.
-3. An optional reranker orders candidates and can produce calibrated automatic matches.
+3. An optional reranker scores and reorders candidates.
 
-Without calibrated reranker thresholds, Skillmux returns an ordered shortlist and lets the calling model choose. Endpoint failures degrade to a healthy lower retrieval mode instead of taking the MCP server down.
+Skillmux returns a ranked shortlist and lets the calling model choose. Endpoint failures degrade to a healthy lower retrieval mode instead of taking the MCP server down.
 
-Read [MCP routing](docs/mcp-routing.md) for transports, outcomes, client instructions, retrieval modes, and the wire contract.
+Read [MCP routing](docs/mcp-routing.md) for transports, client instructions, retrieval modes, and the wire contract.
 
 ## Supported clients
 
@@ -264,7 +264,7 @@ Skillmux preserves existing instruction files and unmanaged target content. Run
 - **Managed ownership:** sync removes only entries recorded in the target's `.skillmux` marker.
 - **Current bytes:** MCP delivery hashes the file on disk and never serves a stale indexed body.
 - **Graceful retrieval:** embedding and reranker failures fall back without hiding the active capability.
-- **Auditable decisions:** each `resolve_skill` call records its outcome, candidates, scores, and latency in the state database.
+- **Auditable decisions:** each `resolve_skill` call records its query, retrieval capability, candidates, scores, and latency in the state database.
 
 ## Documentation
 
@@ -275,11 +275,11 @@ Start with the [documentation hub](docs/README.md).
 | [Getting started](docs/getting-started.md) | Native management, local MCP, and shared-service recipes |
 | [Concepts](docs/concepts.md) | Delivery tiers, deployment topologies, retrieval modes, and ownership |
 | [Managing skills](docs/skill-management.md) | Install, scan, pin, sync, report, overlays, and recovery |
-| [MCP routing](docs/mcp-routing.md) | Tools, outcomes, transports, retrieval, fallback, and integrity |
+| [MCP routing](docs/mcp-routing.md) | Tools, ranked candidate retrieval, transports, fallback, and integrity |
 | [Deployment](docs/deployment.md) | Docker, container command boundaries, HTTP surfaces and auth, CORS, rate limits, and comparable CLI, health, and metrics status |
 | [Configuration](docs/configuration.md) | Machine config, inference, HTTP surfaces, manifests, overlays, and container read-only configuration |
 | [CLI reference](docs/cli.md) | Host and container command surfaces, administrative contexts, automation, JSON output, and exit codes |
-| [Policy calibration](docs/calibration.md) | Labelled datasets, certification, and threshold application |
+| [Ranked-shortlist migration](docs/ranked-shortlist-migration.md) | Upgrade guide for the ranked-only contract |
 | [Troubleshooting](docs/troubleshooting.md) | `doctor`, deployment identity, common failures, and migration notes |
 | [MCP schema](docs/schema.json) | JSON Schema 2020-12 tool contract |
 

package/config.example.toml

@@ -15,3 +15,8 @@ vault_path = "~/skills"
 # vault_path for a given skill_id. Never holds skillmux.toml or the sync git
 # hook — those always stay in vault_path. See docs/configuration.md#local-vault-overlays.
 # local_vault_paths = ["~/skills-local"]
+
+# Optional: output configuration for resolve_skill ranked candidates.
+# [output]
+# top_k = 10
+# max_top_k = 50

package/config.remote.example.toml

@@ -28,10 +28,7 @@ endpoint = "https://reranker.example.com/v1/rerank"
 model = "your-reranker-model"
 api_key_env = "RERANKER_API_KEY"
 
-# Optional. Without these values, reranked results remain ambiguous.
-# Calibrate locally against your corpus before enabling automatic matches.
-# See docs/calibration.md; these example values are not universal defaults.
-[inference.thresholds]
-match_score = 0.90
-match_margin = 0.30
-candidate_floor = 0.50
+# Optional: output configuration for resolve_skill ranked candidates.
+# [output]
+# top_k = 10
+# max_top_k = 50

package/docs/README.md

@@ -49,8 +49,8 @@ deployed server only, never remote client skill directories. See
   retrieval capability.
 - [Managing skills](skill-management.md): install, scan, pin, sync, inspect,
   and recover skills.
-- [MCP routing](mcp-routing.md): register stdio or HTTP MCP and understand
-  retrieval outcomes.
+- [MCP routing](mcp-routing.md): register stdio or HTTP MCP and route queries
+  to ranked candidate shortlists.
 
 ## Operate Skillmux
 
@@ -62,8 +62,8 @@ deployed server only, never remote client skill directories. See
   server settings, local overlays, and container read-only configuration.
 - [CLI reference](cli.md): use host and container command surfaces,
   administrative contexts, JSON output, and exit codes.
-- [Policy calibration](calibration.md): create labelled datasets and certify
-  reranker thresholds.
+- [Ranked-shortlist migration](ranked-shortlist-migration.md): upgrade to the ranked-only
+  shortlist contract.
 - [Troubleshooting](troubleshooting.md): diagnose vault, sync, model, and server
   problems.
 

package/docs/assets/architecture.svg

@@ -149,6 +149,6 @@
 
   <rect x="420" y="608" width="740" height="82" rx="14" fill="#0f172a"/>
   <text x="444" y="633" class="chip" fill="#cbd5e1">MCP TOOL CONTRACT · STDIO + HTTP</text>
-  <text x="444" y="655" class="mono mono-light" style="font-size: 12px">resolve_skill → retrieve + fuse → optional rerank → matched / ambiguous / no_match</text>
+  <text x="444" y="655" class="mono mono-light" style="font-size: 12px">resolve_skill → lexical + vector → RRF → optional rerank → top_k → ranked candidates</text>
   <text x="444" y="677" class="mono mono-light" style="font-size: 12px">fetch_skill → exact current SKILL.md + SHA-256</text>
 </svg>

package/docs/cli.md

@@ -24,7 +24,7 @@ CLI. Its `skillmux --help` surface is intentionally limited to `serve`,
 `index`, `doctor`, `report`, `scan`, `skill which`, and read-only `config`
 inspection (`show`, `get`, `validate`, `diff`, and `status`). Run `init`,
 `install`, pinning, `sync`, project or target management, model downloads,
-contexts, calibration, evaluation, and configuration changes on the host.
+contexts, evaluation, and configuration changes on the host.
 
 When the image rejects one of those commands, it exits with code 2. JSON mode
 uses `CONTAINER_COMMAND_UNSUPPORTED` and includes `rejected_command`,
@@ -267,8 +267,8 @@ whole call fails and the manifest file is left untouched. To pin into a
 
 Config changes are categorized into live-reloadable and restart-required settings:
 
-- **Reloadable**: `vault_path`, `recall.*`, `thresholds.*`, `inference.embedding.*`, `server.rate_limit.*`
-- **Restart Required**: `server.hostname`, `server.auth_enabled`, `server.auth_token_env`, `server.admin.enabled`, `server.admin.token_env`, `inference.mode`, `state_dir`
+- **Reloadable**: `config.environment_overrides`, `vault_path`, `recall.k_lexical`, `recall.k_vector`, `recall.k_rerank`, `output.top_k`, `output.max_top_k`, `inference.embedding.endpoint`, `inference.embedding.api_key_env`, `inference.reranker.adapter`, `inference.reranker.endpoint`, `inference.reranker.model`, `inference.reranker.api_key_env`, `inference.timeout_ms`, `server.rate_limit.enabled`, `server.rate_limit.requests_per_minute`, `server.rate_limit.trust_proxy`
+- **Restart Required**: `server.hostname`, `server.auth_enabled`, `server.auth_token_env`, `server.admin.enabled`, `server.admin.token_env`, `inference.mode`, `inference.bundle`, `inference.models_dir`, `state_dir`, `inference.embedding.model`, `inference.embedding.dimension`, `inference.embedding.device`, `inference.embedding.dtype`
 
 The config file is optional. `skillmux serve` starts with defaults if its
 config path or parent directory is absent. In that case reload is inactive
@@ -287,44 +287,6 @@ skillmux skill which csv-formatter
 
 ---
 
-## Policy calibration (`skillmux calibrate`)
-
-Calibrate decision thresholds (`match_score`, `match_margin`, `candidate_floor`) against synthetic or labeled query datasets.
-Calibration is local-only in this release. Remote targets advertise the
-capability as unavailable and return `not_implemented`; a local dataset path is
-never uploaded or represented as remotely executed. See
-[`docs/calibration.md`](calibration.md) for dataset responsibilities,
-certification gates, run evidence, reference values, and the complete operator
-lifecycle.
-
-```sh
-# Run calibration on a dataset with the default four workers
-skillmux calibrate run --dataset ./eval/queries.json
-
-# Set the bounded worker count
-skillmux calibrate run --dataset ./eval/queries.json --concurrency 6
-
-# Print aggregate performance timing to stderr
-skillmux calibrate run --dataset ./eval/queries.json --timing
-
-# Resume a running or interrupted attempt with the same inputs and gates
-skillmux calibrate run --dataset ./eval/queries.json --resume <run_id>
-
-# List stored calibration runs in the evidence store
-skillmux calibrate list
-
-# Inspect detailed metrics and confusion matrix for a run
-skillmux calibrate show <run_id>
-
-# Apply calibrated thresholds to configuration (with fingerprint validation)
-skillmux calibrate apply <run_id>
-
-# Generate a synthetic decision dataset from vault skills
-skillmux calibrate generate-dataset --out ./eval/queries.json
-```
-
----
-
 ## Administrative HTTP API (`/admin/v1/*`)
 
 The HTTP server has two separate surfaces:
@@ -351,12 +313,9 @@ Requests require `Authorization: Bearer <token>` where `<token>` matches the env
 
 | Endpoint | Method | Description |
 |----------|--------|-------------|
-| `/admin/v1/capabilities` | `GET` | Advertises server features (`config_read`, `config_write`, `calibration`, `persistence`) |
+| `/admin/v1/capabilities` | `GET` | Advertises server features (`config_read`, `config_write`, `persistence`) |
 | `/admin/v1/config` | `GET` | Returns desired/effective config, sources, and `ETag` revision hash |
 | `/admin/v1/config` | `PATCH` | Applies dotted-key updates; requires matching `If-Match` header |
-| `/admin/v1/calibrations` | `GET`, `POST` | Returns `501 not_implemented` (calibration is local-only) |
-| `/admin/v1/calibrations/{run_id}` | `GET` | Returns `501 not_implemented`; raw evaluation queries are not exposed |
-| `/admin/v1/calibrations/{run_id}/apply` | `POST` | Returns `501 not_implemented` |
 
 ---
 

package/docs/concepts.md

@@ -168,21 +168,19 @@ Skillmux reports the active retrieval capability:
 | `reranked` | A configured reranker reorders the fused candidates |
 | `exact` | An exact skill ID resolves directly |
 
-A reranker does not enable automatic matches by itself. Skillmux needs
-calibrated `match_score`, `match_margin`, and `candidate_floor` thresholds
-before it returns a semantic result as `matched`.
+A reranker scores and reorders the fused candidates. Reranking does not classify
+outcomes or filter by score thresholds; Skillmux delivers the top-ranked candidates
+directly to the calling model.
 
-## Retrieval outcomes
+## Ranked shortlist retrieval
 
-`resolve_skill` returns one of three outcomes:
+`resolve_skill` always returns a ranked candidate list:
 
-- `matched`: one skill passed the calibrated policy, so Skillmux delivers its
-  `SKILL.md` body inline;
-- `ambiguous`: Skillmux returns an ordered candidate list and the calling model
-  chooses one with `fetch_skill`;
-- `no_match`: no candidate passed the policy and the agent continues without a
-  skill.
+- each candidate contains a `rank`, `skill_id`, `description`, and a nullable `score`;
+- `top_k` controls the maximum candidate count returned, bounded by `output.max_top_k`;
+- if no candidates match the query, `resolve_skill` returns an empty list (`candidates: []`);
+- the calling model reviews the candidate shortlist and calls `fetch_skill` with the chosen `skill_id` to retrieve complete instructions.
 
-Embedding or reranker failures reduce the active capability. Vault and index
-failures make the server unready because Skillmux can no longer guarantee
-valid retrieval.
+Embedding or reranker failures reduce the active capability and return
+structured degradation metadata. Vault and index failures make the server
+unready because Skillmux can no longer guarantee valid retrieval.

package/docs/configuration.md

@@ -27,9 +27,8 @@ commands, target resolution, and automation envelopes, see
 [CLI reference](cli.md). For Linux CLI installation, see the
 [pinned checksum-verified download](getting-started.md#install-the-cli) or the
 [attested GitHub CLI path](getting-started.md#install-with-github-cli-attestation).
-For labelled datasets, threshold certification,
-reference values, and the apply lifecycle, see
-[Policy calibration](calibration.md).
+For ranking evaluation and upgrading from earlier releases, see
+[Ranked-shortlist migration](ranked-shortlist-migration.md).
 
 ## Machine config bootstrap
 
@@ -70,7 +69,7 @@ loads it and cache it in `~/.cache/skillmux/models`. The full image
 already contains the model.
 
 Skillmux combines FTS5 and cosine result lists with reciprocal-rank fusion.
-Without a reranker, the calling model selects from the ordered shortlist.
+Skillmux returns the fused candidates as a ranked shortlist.
 
 Advanced local overrides:
 
@@ -120,9 +119,8 @@ Remote embedding `dimension` is required. `endpoint`, `api_key_env`, and the
 shared timeout reload live; model, dimension, device, and dtype require a
 restart. Changing only endpoint does not invalidate stored vectors.
 
-Reranker adapter and model form the calibration identity. Moving an unchanged
-deployment to another endpoint does not invalidate calibration; changing the
-adapter or model does.
+Reranker configuration pairs an adapter (such as `jina-v1` or `bifrost-v1`) with a
+model name and a complete endpoint URL.
 
 ## Configuration authority
 
@@ -142,7 +140,7 @@ Inspect provenance and active policy with `skillmux config show --sources` or `s
 
 ## Advanced retrieval
 
-Candidate-generation depth, reranking candidate budgets, and agent-context delivery are separate controls:
+Candidate-generation depth, reranking candidate budgets, and output candidate shortlist size are separate controls:
 
 ```toml
 [recall]
@@ -151,12 +149,14 @@ k_vector = 20
 k_rerank = 10
 
 [output]
-ambiguous_candidate_limit = 5
+top_k = 10
+max_top_k = 50
 ```
 
 - `recall.k_lexical` and `recall.k_vector` control initial candidate generation depth.
 - `recall.k_rerank` bounds the candidate shortlist sent to the reranker adapter (defaults to `10`, cannot exceed `k_lexical + k_vector`).
-- `output.ambiguous_candidate_limit` controls agent context: it caps the ambiguous candidate list returned to the calling LLM after retrieval, reranking, and threshold filtering (`thresholds.candidate_limit` is deprecated in 1.x).
+- `output.top_k` sets the default maximum number of candidates returned by `resolve_skill` (defaults to `10`).
+- `output.max_top_k` sets the upper bound for per-request `top_k` overrides (defaults to `50`).
 
 ### Failure visibility and degraded retrieval
 
@@ -164,7 +164,6 @@ When remote embedding or reranking fails or times out, Skillmux gracefully falls
 
 ```json
 {
-  "outcome": "ambiguous",
   "retrieval": "hybrid",
   "degraded_from": "reranked",
   "degradation_reason": "reranker_timeout",

package/docs/deployment.md

@@ -113,7 +113,7 @@ The supported container commands are `serve`, `index`, `doctor`, `report`,
 
 The image rejects host-management commands, including `init`, `sync`,
 `install`, `project`, `target`, `core`, `local-vault`, `models download`,
-context management, calibration, evaluation, and configuration initialization
+context management, evaluation, and configuration initialization
 or mutation. Install the Skillmux CLI on the host when a command needs to
 manage a local vault or agent directory.
 
@@ -141,8 +141,8 @@ docker run -d \
   ghcr.io/klhq/skillmux:latest-slim
 ```
 
-Set `SKILLMUX_CONFIG` and mount a TOML file when you need reranking, calibrated
-thresholds, server policy, or API-key environment names:
+Set `SKILLMUX_CONFIG` and mount a TOML file when you need reranking, custom
+recall/output settings, server policy, or API-key environment names:
 
 ```sh
 docker run -d \
@@ -242,15 +242,18 @@ The HTTP server provides:
 | `GET /health/ready` | Vault, index, inference, active capability, version, runtime, and image variant |
 | `GET /health` | Compatibility alias for liveness |
 | `GET /metrics` | Prometheus text exposition |
-| `GET /stats` | Aggregated routing outcomes for `skillmux report` |
+| `GET /stats` | Aggregated routing statistics for `skillmux report` |
 | `POST /mcp` | Streamable HTTP MCP transport |
 | `/admin/v1/*` | Administrative configuration API (when enabled) |
 
 The Docker health check calls `/health/ready`.
 
-Prometheus metrics cover request totals, resolve outcomes, latency, errors,
-rate-limit rejections, and a `skill_router_deployment_info` gauge labelled with
-`version`, `runtime`, and `image_variant`. The values match `skillmux doctor`,
+`GET /stats` returns JSON aggregated from audit rows over a query window (using `?since=`, defaulting to 24 hours). The response contains `since`, `until`, `total_requests`, `empty_shortlist_count`, `empty_shortlist_rate`, `retrieval_totals` (counts for `exact`, `reranked`, `hybrid`, and `lexical`), `degraded_count`, `average_latency_ms`, per-skill candidate counts in `skills` (`skill_id`, `candidate_count`), and `top_empty_shortlist_queries` (`query`, `count`). See [Managing skills](skill-management.md#use-routing-data-to-tune-tiers) for using these statistics to tune delivery tiers.
+
+Prometheus metrics cover request totals, resolve latency histograms, errors,
+rate-limit rejections, degraded retrieval totals by stage and reason, readiness
+and active retrieval capability gauges, and a `skill_router_deployment_info` gauge
+labelled with `version`, `runtime`, and `image_variant`. The values match `skillmux doctor`,
 `skillmux config status`, and `/health/ready`; use `image_variant="none"` for
 a host runtime. These operational outputs never include credentials, API keys,
 or token values. Health and metrics do not require bearer authentication, but
@@ -286,7 +289,7 @@ their mounted vault checkout and do not manage host agent directories.
 
 ## Persistent data and backups
 
-Persist `state_dir` to retain the index, audit log, and calibration evidence.
+Persist `state_dir` to retain the index and audit log.
 Skill content remains in the server's vault checkout and should use its own
 backup or Git workflow.
 

package/docs/getting-started.md

@@ -212,7 +212,7 @@ Register it with your MCP client:
 ```
 
 The client launches the process and closes it with the MCP session. Continue
-with [MCP routing](mcp-routing.md) for client behavior and retrieval outcomes.
+with [MCP routing](mcp-routing.md) for client behavior and ranked candidate retrieval.
 
 ## Run a shared MCP service
 

package/docs/mcp-routing.md

@@ -70,16 +70,30 @@ Input:
 
 ```json
 {
-  "query": "convert this spreadsheet to a Markdown table"
+  "query": "convert this spreadsheet to a Markdown table",
+  "top_k": 5
 }
 ```
 
-Skillmux returns one outcome:
+Skillmux returns a ranked candidates response:
 
-- `matched`: `structuredContent` contains match metadata and the text content
-  contains the `SKILL.md` body once;
-- `ambiguous`: `structuredContent` contains up to `candidate_limit` candidates;
-- `no_match`: the agent continues with its normal workflow.
+```json
+{
+  "retrieval": "reranked",
+  "candidates": [
+    {
+      "rank": 1,
+      "skill_id": "csv-formatter",
+      "description": "Convert CSV and spreadsheets into formatted Markdown tables.",
+      "score": 0.92
+    }
+  ]
+}
+```
+
+- `retrieval`: the effective retrieval capability (`reranked`, `hybrid`, or `lexical`).
+- `candidates`: zero through effective `top_k` candidates ordered by descending score with contiguous 1-based ranks.
+- If reranking or embedding fails, degradation metadata (`degraded_from`, `degradation_reason`) is included.
 
 ### `fetch_skill`
 
@@ -102,13 +116,9 @@ The complete wire contract lives in [schema.json](schema.json).
 Give the calling client these rules:
 
 1. Call `resolve_skill` when a task may benefit from a specialized workflow.
-2. Follow the delivered skill on `matched`.
-3. On `ambiguous`, choose the best candidate and call `fetch_skill`.
-4. On `no_match`, continue without loading a skill.
-
-Do not treat the first ambiguous candidate as an automatic match. Skillmux
-uses ambiguity to keep the final choice with the calling model when it lacks
-enough confidence.
+2. Review the returned ranked candidates shortlist.
+3. If a relevant candidate exists, call `fetch_skill` with its `skill_id` to retrieve complete instructions.
+4. If no candidate is relevant (or `candidates` is empty), continue under your normal workflow.
 
 ## Retrieval pipeline
 
@@ -118,7 +128,7 @@ Skillmux builds candidates in stages:
 2. Local or remote embeddings rank semantic similarity.
 3. Reciprocal-rank fusion combines both lists.
 4. An optional reranker scores the fused candidates.
-5. Calibrated thresholds select `matched`, `ambiguous`, or `no_match`.
+5. Returns up to effective `top_k` ranked candidates.
 
 The default local inference configuration uses FTS5 and quantized
 `Xenova/gte-small` embeddings. Skillmux CLI installations cache the
@@ -128,13 +138,8 @@ OpenAI-compatible embedding endpoint.
 
 Remote inference also supports `jina-v1` or `bifrost-v1` reranker adapters.
 
-Without a reranker, Skillmux returns a shortlist and does not auto-match.
-Without calibrated thresholds, a configured reranker orders the shortlist but
-still does not auto-match.
-
 Read [Configuration](configuration.md#local-inference) for local and remote
-inference settings. Read [Policy calibration](calibration.md) before enabling
-automatic matches.
+inference settings.
 
 ## Fallback and readiness
 
@@ -169,10 +174,10 @@ access.
 Each resolve request records:
 
 - timestamp and query;
-- retrieval capability and outcome;
+- retrieval capability;
+- degradation metadata (`degraded_from`, `degradation_reason`) when degraded;
 - candidates with scores;
-- selected skill ID, when present;
-- latency.
+- latency in milliseconds.
 
 Skillmux stores audit rows in the SQLite database under `state_dir`. Use
 `skillmux report` to summarize activity. Treat raw queries as private user

package/docs/ranked-shortlist-migration.md

@@ -0,0 +1,160 @@
+# Ranked-shortlist migration
+
+The ranked-shortlist release replaces threshold-based classification with ranked candidate retrieval. Instead of classifying a query as `matched`, `ambiguous`, or `no_match` and returning inline skill text on match, `resolve_skill` always returns a ranked shortlist of candidate summaries. The calling client reviews the shortlist and invokes `fetch_skill` to load the full instructions for the chosen skill.
+
+## Upgrading configuration
+
+Legacy threshold configuration tables and candidate limits are obsolete. Threshold values do not map semantically to candidate list lengths; replace threshold tables with retrieval depth and output bounds.
+
+In `~/.config/skillmux/config.toml`, configure `[recall]` to control candidate generation and `[output]` to set candidate shortlist limits:
+
+```toml
+vault_path = "~/skills"
+state_dir = "~/.local/state/skillmux"
+
+[recall]
+k_lexical = 20
+k_vector = 20
+k_rerank = 10
+
+[output]
+top_k = 10
+max_top_k = 50
+```
+
+`recall.k_lexical` and `recall.k_vector` set the number of candidates retrieved from FTS5 lexical search and vector similarity before reciprocal rank fusion. `recall.k_rerank` limits the candidate shortlist sent to an optional remote reranker adapter and cannot exceed `k_lexical + k_vector`.
+
+`output.top_k` sets the default maximum number of candidates returned by `resolve_skill`. `output.max_top_k` sets the upper bound for per-request `top_k` overrides. When a remote reranker is configured, `output.max_top_k` cannot exceed `recall.k_rerank`.
+
+Existing `calibrate.sqlite3` databases in `state_dir` remain untouched on disk. Skillmux does not delete or modify historical calibration data.
+
+## Obsolete configuration keys
+
+Startup fails with an explicit error when obsolete configuration keys are present:
+
+| Obsolete configuration key | Replacement | Migration guidance |
+| --- | --- | --- |
+| `[thresholds]` | `[output].top_k` | The `[thresholds]` table is obsolete. Threshold calibration was removed; use `[output]` with `top_k`. |
+| `output.ambiguous_candidate_limit` | `output.top_k` | `output.ambiguous_candidate_limit` is obsolete. Use `output.top_k` instead. |
+| `inference.thresholds` | Removed | `inference.thresholds` is obsolete. Threshold calibration was removed. |
+| `inference.calibration` | `skillmux eval` | `inference.calibration` is obsolete and should be deleted. Threshold calibration was removed; use `skillmux eval` for ranking evaluation. |
+| `skillmux calibrate` | `skillmux eval` | `skillmux calibrate` was removed. Threshold calibration was removed; use `skillmux eval` for ranking evaluation. |
+
+## Wire contract changes
+
+In the legacy classifier contract, `resolve_skill` returned an `outcome` classification and delivered the complete `SKILL.md` body directly when a single match met threshold criteria:
+
+```json
+{
+  "outcome": "matched",
+  "retrieval": "reranked",
+  "skill_id": "csv-formatter",
+  "title": "CSV Formatter",
+  "content_sha256": "8a3f...",
+  "body": "# CSV Formatter\n...",
+  "files": [],
+  "candidates": [
+    {
+      "rank": 1,
+      "skill_id": "csv-formatter",
+      "score": 0.95
+    }
+  ]
+}
+```
+
+In the current contract, `resolve_skill` returns ranked candidate summaries with descriptions and scores. The client reviews candidate descriptions and calls `fetch_skill` with the desired `skill_id` to retrieve instructions and supporting files:
+
+```json
+{
+  "retrieval": "reranked",
+  "candidates": [
+    {
+      "rank": 1,
+      "skill_id": "csv-formatter",
+      "description": "Convert CSV and spreadsheet data into clean Markdown tables.",
+      "score": 0.92
+    },
+    {
+      "rank": 2,
+      "skill_id": "table-helper",
+      "description": "Format ASCII and Markdown tables.",
+      "score": 0.74
+    }
+  ]
+}
+```
+
+When retrieval finds no candidates, `resolve_skill` returns an empty candidate list:
+
+```json
+{
+  "retrieval": "hybrid",
+  "candidates": []
+}
+```
+
+When upstream embedding or reranking endpoints fail or time out, Skillmux falls back to the next available retrieval lane and reports structured degradation metadata:
+
+```json
+{
+  "retrieval": "hybrid",
+  "degraded_from": "reranked",
+  "degradation_reason": "reranker_timeout",
+  "candidates": [
+    {
+      "rank": 1,
+      "skill_id": "csv-formatter",
+      "description": "Convert CSV and spreadsheet data into clean Markdown tables.",
+      "score": 0.033
+    }
+  ]
+}
+```
+
+## Audit database migration
+
+On startup, Skillmux inspects the `audit` table in `state_dir/index.sqlite3`. If legacy classifier columns (`outcome`, `selected_skill_id`, `degraded`) are present, it migrates the table inside an atomic SQLite transaction:
+
+```sql
+CREATE TABLE audit (
+  id INTEGER PRIMARY KEY AUTOINCREMENT,
+  ts TEXT NOT NULL,
+  query TEXT NOT NULL,
+  retrieval TEXT NOT NULL DEFAULT 'lexical',
+  degraded_from TEXT,
+  degradation_reason TEXT,
+  candidates TEXT NOT NULL,
+  latency_ms INTEGER NOT NULL
+);
+```
+
+Historical rows retain their `id`, `ts`, `query`, `retrieval`, `candidates`, and `latency_ms` values. Classifier-only columns are removed.
+
+## Ranking evaluation
+
+Use `skillmux eval` to evaluate retrieval and ranking quality across lexical and hybrid pipelines:
+
+```sh
+skillmux eval
+```
+
+The `skillmux eval` command reads the default `eval/queries.json` dataset from the repository because the CLI currently has no `--dataset` flag.
+
+Evaluation cases specify `relevant_skill_ids` as an array of skill identifiers rather than legacy outcome fields:
+
+```json
+{
+  "query": "format a csv file",
+  "relevant_skill_ids": ["csv-formatter"],
+  "split": "test"
+}
+```
+
+Legacy fields `expected` and `expected_outcome` are rejected with validation errors. Cases with empty `relevant_skill_ids: []` represent unjudged queries and are excluded from aggregate ranking metrics without penalizing the evaluation score.
+
+The evaluation report computes four ranking metrics for lexical and hybrid retrieval:
+- Recall@5 measures the fraction of relevant skills found in the top 5 candidates.
+- Recall@10 measures the fraction of relevant skills found in the top 10 candidates.
+- Mean Reciprocal Rank (MRR) measures the reciprocal rank of the first relevant candidate.
+- Binary nDCG@10 discounts relevance logarithmically across the top 10 positions.