selftune 0.2.19 → 0.2.21

package/packages/ui/src/types.ts

@@ -31,6 +31,10 @@ export interface EvolutionEntry {
   action: string;
   details: string;
   eval_snapshot?: EvalSnapshot | null;
+  validation_mode?: "structural_guard" | "host_replay" | "llm_judge" | null;
+  validation_agent?: string | null;
+  validation_fixture_id?: string | null;
+  validation_evidence_ref?: string | null;
 }
 
 export interface UnmatchedQuery {

package/skill/Workflows/Composability.md

@@ -104,6 +104,19 @@ This is for packaging questions like:
 - "Are my sibling skills competing for the same user intent?"
 - "Should I stop evolving these independently and redesign the family?"
 
+When trusted telemetry is sparse, the same command also emits a
+`cold_start_suspicion` block. That is a weaker, earlier signal based on the
+installed skill surfaces:
+
+1. Frontmatter / top-level description similarity
+2. Overlap in `## When to Use` language
+3. Shared command surface (for example, siblings that both wrap `mentor search`)
+4. Synthetic sibling-confusion probes derived from those overlapping surfaces
+
+Treat `cold_start_suspicion.candidate` as architecture suspicion, not proof.
+It is meant to tell you "this family may want a parent skill" before enough
+real usage exists to confirm it through trusted positive-query overlap.
+
 ## Steps
 
 ### 1. Run Analysis
@@ -140,6 +153,7 @@ Interpretation:
 
 - `consolidation_candidate: false` means keep improving the sibling descriptions/workflows separately
 - `consolidation_candidate: true` means the problem is likely packaging, not just wording
+- `cold_start_suspicion.candidate: true` means installed skill surfaces already look suspicious even though trusted telemetry is still sparse
 - `refactor_proposal` is a draft for human review only; do not auto-deploy a family rewrite
 
 ## Subagent Escalation
@@ -173,4 +187,4 @@ resolution plan with trigger ownership recommendations.
 
 **"Should I consolidate this sibling skill family?"**
 
-> Run `selftune eval family-overlap` and look for `consolidation_candidate` plus the `refactor_proposal`.
+> Run `selftune eval family-overlap` and look for `consolidation_candidate` when you have live evidence, or `cold_start_suspicion` when you only have installed skill surfaces plus cold-start evals.

package/skill/Workflows/Evolve.md

@@ -76,6 +76,45 @@ The evolution process writes multiple audit entries:
 | `validated` | Proposal tested against eval set | `eval_snapshot` with before/after pass rates      |
 | `deployed`  | Updated SKILL.md written to disk | `eval_snapshot` with final rates                  |
 
+Routing/body validation may also carry provenance fields such as:
+
+- `validation_mode` — `llm_judge`, `host_replay`, or `structural_guard`
+- `validation_agent` — which host/agent performed the validation
+- `validation_fixture_id` — fixture identifier when replay-backed validation is used
+- `before_pass_rate` / `after_pass_rate` — only present when trigger validation actually ran; structural-guard exits do not emit synthetic pass rates
+
+Most evolve runs today still validate through `llm_judge`. Routing evolution now
+auto-builds a replay fixture from the target skill plus installed sibling
+skills in the same registry, so replay-backed validation is preferred whenever
+that local fixture can be constructed because it captures host-style routing
+behavior instead of model judgment.
+
+For Claude Code, the replay path now stages a temporary project-local
+`.claude/skills` registry, swaps in the candidate routing table, and runs a
+one-turn Claude print-mode session with project/local settings only. Validation
+records whether Claude actually invoked the target skill, invoked a competing
+skill, invoked an unrelated skill, or made no routing decision at all.
+Unrelated skill use is treated as a replay failure even on negative evals,
+because it still indicates the runtime routed somewhere unexpected. If that
+runtime path is unavailable or fails to reach a runtime decision, selftune
+falls back to the existing fixture-backed surface simulation and notes the
+fallback in the replay evidence instead of pretending it was a runtime result.
+
+For non-Claude platforms today, replay remains fixture-backed: it evaluates the
+target routing table against the installed target/competing skill surfaces in a
+controlled replay fixture and records per-entry evidence. That is still a
+stronger signal than a free-form judge prompt, but you should describe it as
+replay-backed validation, not as live operator telemetry.
+
+Replay parsing is intentionally conservative: unreadable skill files degrade to
+empty surfaces instead of throwing, and malformed routing rows with empty
+trigger cells are ignored rather than treated as valid triggers. Claude replay
+also normalizes observed `Read` paths against the staged workspace, so relative
+skill reads still count as read-only evidence for the target or competing
+skill. Reads outside the staged skill set are treated as replay failures rather
+than benign negatives, because they indicate the runtime left the controlled
+evaluation surface.
+
 ## Parsing Instructions
 
 ### Track Evolution Progress