@klhapp/skillmux 1.1.0 → 1.3.0

package/config.example.toml

@@ -1,7 +1,8 @@
-# This entire file is optional. Skill Router indexes an externally managed
-# agentskills.io-compatible vault and uses local GTE-small semantic retrieval.
-# vault_path is a neutral path, never a harness's static-loading scan
-# surface (e.g. ~/.agents/skills) — see docs/sdd/skr-cli/think.md.
+# This file is optional. It configures the Skillmux CLI or the full Skillmux
+# server image with local inference. Skillmux manages and indexes an
+# agentskills.io-compatible canonical vault. Keep the vault separate from
+# client skill directories such as ~/.agents/skills; skillmux sync selects
+# which core and project skills appear in those directories.
 vault_path = "~/skills"
 
 # Optional, default []: machine-local override-only vaults, checked before

package/config.remote.example.toml

@@ -1,5 +1,8 @@
-# Connect stronger inference endpoints. The reranker section is optional.
-vault_path = "~/.agents/skills"
+# Configure remote inference endpoints. This describes where inference runs,
+# not where Skillmux runs; local stdio and shared HTTP deployments can both use
+# this file. The reranker section is optional.
+# Keep the canonical vault separate from client skill directories.
+vault_path = "~/skills"
 
 [inference]
 mode = "remote"
@@ -18,7 +21,9 @@ endpoint = "https://reranker.example.com/v1/rerank"
 model = "your-reranker-model"
 api_key_env = "RERANKER_API_KEY"
 
-# Required when a reranker is configured. Calibrate these for your model.
+# Optional. Without these values, reranked results remain ambiguous.
+# Calibrate locally against your corpus before enabling automatic matches.
+# See docs/calibration.md; these example values are not universal defaults.
 [inference.thresholds]
 match_score = 0.90
 match_margin = 0.30

package/docs/README.md

@@ -0,0 +1,52 @@
+# Skillmux documentation
+
+Skillmux manages one `SKILL.md` vault. Choose a workflow based on where skills
+need to appear and where Skillmux should run.
+
+## Choose a use case
+
+| Goal | Start with | Recommended installation |
+| --- | --- | --- |
+| Put a small skill set in native client directories | [Manage native skills](getting-started.md#manage-native-skills) | Skillmux CLI |
+| Let one client search the full vault through MCP | [Add local MCP retrieval](getting-started.md#add-local-mcp-retrieval) | Skillmux CLI |
+| Serve the vault to several MCP clients | [Run a shared MCP service](getting-started.md#run-a-shared-mcp-service) | Full Skillmux server image |
+
+You can combine native management and local MCP retrieval on one machine. The
+shared service uses the same MCP tools over HTTP.
+
+Install the CLI with the Bun package, or use the standalone Linux executable
+when Bun is unavailable. The full server image is the shared-service default;
+the slim image is an advanced choice for remote embeddings or lexical-only
+retrieval.
+
+## Learn the model
+
+- [Getting started](getting-started.md): choose an installation, prepare a
+  vault, and follow one of three setup recipes.
+- [Concepts](concepts.md): separate delivery tiers, deployment topology, and
+  retrieval capability.
+- [Managing skills](skill-management.md): install, scan, pin, sync, inspect,
+  and recover skills.
+- [MCP routing](mcp-routing.md): register stdio or HTTP MCP and understand
+  retrieval outcomes.
+
+## Operate Skillmux
+
+- [Deployment](deployment.md): deploy the shared server, choose slim only when
+  needed, expose HTTP, and operate the service.
+- [Configuration reference](configuration.md): configure inference, manifests,
+  server settings, and local overlays.
+- [CLI reference](cli.md): use commands, contexts, JSON output, and exit codes.
+- [Policy calibration](calibration.md): create labelled datasets and certify
+  reranker thresholds.
+- [Troubleshooting](troubleshooting.md): diagnose vault, sync, model, and server
+  problems.
+
+## Integrate and contribute
+
+- [MCP contract](schema.json): JSON Schema 2020-12 definitions for tool inputs
+  and results.
+- [Contributing](../CONTRIBUTING.md): development setup, tests, and pull request
+  conventions.
+- [Releasing](releasing.md): maintainer release procedure.
+- [Security](../SECURITY.md): vulnerability reporting.

package/docs/assets/architecture.svg

@@ -0,0 +1,156 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1200 720" role="img" aria-labelledby="title desc">
+  <title id="title">Three ways to use Skillmux</title>
+  <desc id="desc">One canonical skill vault supports native skill management and local MCP retrieval through the Skillmux CLI, plus a shared HTTP MCP service through the full or slim Skillmux server image.</desc>
+
+  <defs>
+    <filter id="shadow" x="-15%" y="-15%" width="130%" height="140%">
+      <feDropShadow dx="0" dy="5" stdDeviation="8" flood-color="#0f172a" flood-opacity="0.10"/>
+    </filter>
+    <marker id="arrow-teal" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto">
+      <path d="M0 0L10 5L0 10Z" fill="#0f766e"/>
+    </marker>
+    <marker id="arrow-blue" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto">
+      <path d="M0 0L10 5L0 10Z" fill="#2563eb"/>
+    </marker>
+    <marker id="arrow-violet" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto">
+      <path d="M0 0L10 5L0 10Z" fill="#7c3aed"/>
+    </marker>
+    <style>
+      text {
+        font-family: "DejaVu Sans", Arial, sans-serif;
+      }
+      .title {
+        font-size: 27px;
+        font-weight: 750;
+        fill: #0f172a;
+      }
+      .subtitle {
+        font-size: 16px;
+        font-weight: 400;
+        fill: #475569;
+      }
+      .card-title {
+        font-size: 21px;
+        font-weight: 750;
+        fill: #0f172a;
+      }
+      .label {
+        font-size: 15px;
+        font-weight: 650;
+        fill: #1e293b;
+      }
+      .body {
+        font-size: 14px;
+        font-weight: 400;
+        fill: #475569;
+      }
+      .mono {
+        font-family: "DejaVu Sans Mono", Consolas, monospace;
+        font-size: 14px;
+        font-weight: 650;
+        fill: #1e293b;
+      }
+      .mono-light {
+        fill: #ffffff;
+      }
+      .chip {
+        font-size: 12px;
+        font-weight: 750;
+        letter-spacing: .5px;
+      }
+    </style>
+  </defs>
+
+  <rect width="1200" height="720" rx="24" fill="#f8fafc"/>
+  <rect x="1" y="1" width="1198" height="718" rx="23" fill="none" stroke="#e2e8f0" stroke-width="2"/>
+
+  <text x="48" y="50" class="title">One vault, three ways to use Skillmux</text>
+  <text x="48" y="78" class="subtitle">Choose a local workflow or serve the same retrieval contract to several clients.</text>
+
+  <g filter="url(#shadow)">
+    <rect x="330" y="106" width="540" height="118" rx="18" fill="#ffffff" stroke="#cbd5e1" stroke-width="2"/>
+    <circle cx="374" cy="147" r="18" fill="#0f172a"/>
+    <path d="M364 141H377L383 147L377 153H364M370 135V159" fill="none" stroke="#ffffff" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"/>
+    <text x="408" y="144" class="card-title">Canonical skill vault</text>
+    <text x="408" y="172" class="mono">~/skills/&lt;skill-id&gt;/SKILL.md</text>
+    <text x="408" y="200" class="body">Install · scan · pin · index · inspect usage</text>
+    <rect x="701" y="139" width="137" height="34" rx="9" fill="#f1f5f9" stroke="#cbd5e1"/>
+    <text x="769.5" y="161" text-anchor="middle" class="mono">skillmux.toml</text>
+  </g>
+
+  <path d="M480 224V252H220V277" fill="none" stroke="#0f766e" stroke-width="3" marker-end="url(#arrow-teal)"/>
+  <path d="M600 224V277" fill="none" stroke="#2563eb" stroke-width="3" marker-end="url(#arrow-blue)"/>
+  <path d="M720 224V252H980V277" fill="none" stroke="#7c3aed" stroke-width="3" marker-end="url(#arrow-violet)"/>
+
+  <g filter="url(#shadow)">
+    <rect x="40" y="284" width="360" height="302" rx="18" fill="#ffffff" stroke="#99d5cc" stroke-width="2"/>
+    <rect x="62" y="306" width="91" height="26" rx="13" fill="#ccfbf1"/>
+    <text x="107.5" y="324" text-anchor="middle" class="chip" fill="#115e59">USE CASE 1</text>
+    <text x="62" y="365" class="card-title">Skillmux CLI</text>
+    <text x="62" y="390" class="body">Manage native skills on the client machine.</text>
+
+    <rect x="62" y="414" width="276" height="48" rx="10" fill="#f0fdfa" stroke="#99d5cc"/>
+    <text x="200" y="444" text-anchor="middle" class="mono">skillmux init · sync</text>
+    <path d="M200 462V487" fill="none" stroke="#0f766e" stroke-width="2.5" marker-end="url(#arrow-teal)"/>
+    <rect x="62" y="493" width="276" height="49" rx="10" fill="#f8fafc" stroke="#cbd5e1"/>
+    <text x="200" y="514" text-anchor="middle" class="label">Native skill directories</text>
+    <text x="200" y="533" text-anchor="middle" class="body">managed core and project links</text>
+
+    <text x="62" y="570" class="body">Install with</text>
+    <rect x="160" y="550" width="85" height="27" rx="8" fill="#0f172a"/>
+    <text x="202.5" y="568" text-anchor="middle" class="chip" style="font-size: 10px" fill="#ffffff">BUN PACKAGE</text>
+    <rect x="251" y="550" width="87" height="27" rx="8" fill="#f1f5f9" stroke="#cbd5e1"/>
+    <text x="294.5" y="561" text-anchor="middle" class="chip" style="font-size: 10px" fill="#334155"><tspan x="294.5">LINUX</tspan><tspan x="294.5" dy="11">EXECUTABLE</tspan></text>
+  </g>
+
+  <g filter="url(#shadow)">
+    <rect x="420" y="284" width="360" height="302" rx="18" fill="#ffffff" stroke="#93c5fd" stroke-width="2"/>
+    <rect x="442" y="306" width="91" height="26" rx="13" fill="#dbeafe"/>
+    <text x="487.5" y="324" text-anchor="middle" class="chip" fill="#1d4ed8">USE CASE 2</text>
+    <text x="442" y="365" class="card-title">Add local MCP retrieval</text>
+    <text x="442" y="390" class="body">Run Skillmux beside one MCP client.</text>
+
+    <rect x="442" y="414" width="102" height="48" rx="10" fill="#f8fafc" stroke="#cbd5e1"/>
+    <text x="493" y="444" text-anchor="middle" class="label">AI client</text>
+    <path d="M544 438H573" fill="none" stroke="#2563eb" stroke-width="2.5" marker-end="url(#arrow-blue)"/>
+    <text x="558" y="428" text-anchor="middle" class="body">stdio</text>
+    <rect x="579" y="414" width="179" height="48" rx="10" fill="#eff6ff" stroke="#93c5fd"/>
+    <text x="668.5" y="444" text-anchor="middle" class="label">Local Skillmux</text>
+
+    <rect x="442" y="484" width="316" height="58" rx="10" fill="#f8fafc" stroke="#cbd5e1"/>
+    <text x="600" y="507" text-anchor="middle" class="label">FTS5 + local GTE-small</text>
+    <text x="600" y="530" text-anchor="middle" class="body">downloaded and cached on this machine</text>
+
+    <text x="442" y="570" class="body">Skillmux CLI: Bun package or Linux executable</text>
+  </g>
+
+  <g filter="url(#shadow)">
+    <rect x="800" y="284" width="360" height="302" rx="18" fill="#ffffff" stroke="#c4b5fd" stroke-width="2"/>
+    <rect x="822" y="306" width="91" height="26" rx="13" fill="#ede9fe"/>
+    <text x="867.5" y="324" text-anchor="middle" class="chip" fill="#6d28d9">USE CASE 3</text>
+    <text x="822" y="365" class="card-title">Run a shared MCP service</text>
+    <text x="822" y="390" class="body">Serve several clients from one deployment.</text>
+
+    <rect x="822" y="414" width="102" height="48" rx="10" fill="#f8fafc" stroke="#cbd5e1"/>
+    <text x="873" y="435" text-anchor="middle" class="label">AI clients</text>
+    <text x="873" y="453" text-anchor="middle" class="body">one or more</text>
+    <path d="M924 438H953" fill="none" stroke="#7c3aed" stroke-width="2.5" marker-end="url(#arrow-violet)"/>
+    <text x="938" y="428" text-anchor="middle" class="body">HTTP</text>
+    <rect x="959" y="414" width="179" height="48" rx="10" fill="#f5f3ff" stroke="#c4b5fd"/>
+    <text x="1048.5" y="435" text-anchor="middle" class="label">Skillmux server</text>
+    <text x="1048.5" y="453" text-anchor="middle" class="mono">POST /mcp</text>
+
+    <rect x="822" y="484" width="148" height="58" rx="10" fill="#f5f3ff" stroke="#c4b5fd"/>
+    <text x="896" y="507" text-anchor="middle" class="label">Full image</text>
+    <text x="896" y="530" text-anchor="middle" class="body">GTE-small included</text>
+    <rect x="982" y="484" width="156" height="58" rx="10" fill="#f8fafc" stroke="#cbd5e1"/>
+    <text x="1060" y="507" text-anchor="middle" class="label">Slim image</text>
+    <text x="1060" y="530" text-anchor="middle" class="body">remote or lexical</text>
+
+    <text x="822" y="570" class="body">Recommended for a shared HTTP deployment</text>
+  </g>
+
+  <rect x="420" y="616" width="740" height="66" rx="14" fill="#0f172a"/>
+  <text x="444" y="643" class="chip" fill="#cbd5e1">SHARED MCP CONTRACT</text>
+  <text x="444" y="667" class="mono mono-light">retrieve → rerank → resolve_skill / fetch_skill → verified SKILL.md</text>
+</svg>

package/docs/calibration.md

@@ -0,0 +1,111 @@
+# Policy calibration
+
+Calibration selects the three reranker-score thresholds that turn an ordered
+shortlist into `matched`, `ambiguous`, or `no_match`. It is an operator action,
+not background learning, and it currently runs only against a local Skillmux
+target.
+
+Read [MCP routing](mcp-routing.md#retrieval-pipeline) before calibrating a new
+retrieval deployment.
+
+## Lifecycle
+
+The complete workflow is:
+
+```text
+install CLI → configure vault/index/embedding/reranker → obtain labelled dataset
+    → calibrate run → review calibrate show RUN_ID → calibrate apply RUN_ID
+    → live-reloaded policy handles subsequent requests
+```
+
+First configure and index the same vault, embedding model, and reranker that
+will serve requests. Supply a reviewed dataset, or generate a starting point
+and review every label:
+
+```sh
+skillmux calibrate generate-dataset --out ./eval/queries.json
+skillmux calibrate run --dataset ./eval/queries.json
+skillmux calibrate show RUN_ID
+skillmux calibrate apply RUN_ID
+```
+
+Skillmux retrieves candidates and reranks exactly once for each evaluation
+query. It caches those observations, searches thresholds on the `tune` split,
+then certifies the selected policy on the frozen `test` split. Calibration
+starts only when an operator invokes `calibrate run`.
+
+The operator owns the labels: supply or review the cases, start the run,
+inspect its evidence, and explicitly apply an acceptable result. A successful
+run never changes live thresholds by itself.
+
+## Reading a run
+
+A `run_id` identifies one immutable calibration attempt and its evidence.
+`calibrate show RUN_ID` is read-only. It reports:
+
+- selected thresholds and tune/test metrics;
+- auto-match precision confidence and sample counts;
+- retrieval and delivered-shortlist recall;
+- a closed failure reason when certification fails;
+- reranker, embedding, corpus, and dataset fingerprints;
+- dataset provenance and the number of human-labelled cases; and
+- the attempt count for the dataset hash.
+
+`calibrate apply RUN_ID` accepts only a completed, test-certified run. It
+rechecks the reranker fingerprint, rejects thresholds masked by environment
+variables, atomically updates the TOML file, and lets the config watcher
+activate the new snapshot.
+
+## Dataset responsibilities
+
+Each case needs a query, expected outcome, relevant skill ids, and a fixed
+`tune` or `test` split. Unknown skill ids are rejected. Keep a skill entirely
+within one split so the test set measures generalization rather than memorized
+skill wording.
+
+Generated datasets are scaffolding, not ground truth. Review paraphrases,
+near-miss negatives, and ambiguous cases before using them for certification.
+Audit-derived cases require an explicit human label and provenance. Raw audit
+queries are excluded unless the importer is deliberately configured to retain
+them.
+
+## When to recalibrate
+
+Re-run calibration after a material change to the corpus, embedding or
+retrieval behavior, reranker adapter or model, or after collecting enough new
+human-labelled feedback. Do not recalibrate per user request. Every rerun gets
+a new `run_id`; the active policy remains unchanged until one is applied.
+
+## Local and remote targets
+
+Here, `local` and `remote` name CLI administration targets, not inference
+locations or MCP transports. Calibration is local-target-only in this release.
+Local commands operate on the
+configured local vault, index, inference endpoints, dataset path, evidence
+database, and TOML file. Human output always prints `Target: local`; JSON output
+uses `"target": "local"`.
+
+Remote servers advertise `"calibration": false`. Every
+`/admin/v1/calibrations` route returns HTTP `501` with
+`error: "not_implemented"`, and the CLI rejects remote calibration before
+uploading or claiming to execute a local dataset path. This also prevents raw
+evaluation queries from being exposed through the admin API.
+
+## Reference starting profile
+
+Reranker scores are not portable across models, adapters, model revisions, or
+corpora. The profile below is published only to make the checked-in BGE example
+concrete; it is not a certified substitute for calibration.
+
+| Model | Adapter | `match_score` | `match_margin` | `candidate_floor` |
+|---|---|---:|---:|---:|
+| `BAAI/bge-reranker-v2-m3` | `jina-v1` | `0.90` | `0.20` | `0.40` |
+
+Provenance: the small synthetic corpus and labelled decision cases in
+[`tests/router-core.spec.test.ts`](../tests/router-core.spec.test.ts), with the
+wire contract captured by
+[`tests/fixtures/reranker/jina-v1-request.json`](../tests/fixtures/reranker/jina-v1-request.json).
+That fixture is below the default 30-auto-match certification minimum, so the
+values are a smoke-test/reference profile, not a completed calibration run.
+Run the lifecycle above against the deployment's real corpus before enabling
+automatic matches in production.