@heytherevibin/skillforge 0.7.0 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +11 -0
- package/README.md +46 -4
- package/RELEASING.md +1 -1
- package/package.json +1 -1
- package/python/app/main.py +256 -19
- package/python/app/mcp_server.py +227 -5
- package/python/app/route_policies.py +133 -0
- package/python/app/routing_signals.py +95 -0
- package/python/requirements.txt +1 -0
- package/python/tests/test_route_policies.py +115 -0
- package/python/tests/test_routing_signals.py +77 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,16 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.8.0
|
|
4
|
+
|
|
5
|
+
- **Smarter routing:** optional **conversation-aware** shortlist query (`SKILLFORGE_ROUTER_CONV_*`), **hybrid** retrieval (`SKILLFORGE_ROUTER_HYBRID` = `keyword` or `bm25` + `SKILLFORGE_ROUTER_HYBRID_ALPHA`), optional **Haiku rerank** (`SKILLFORGE_HAIKU_RERANK`, `SKILLFORGE_HAIKU_RERANK_MAX`, `SKILLFORGE_HAIKU_RERANK_MODEL`).
|
|
6
|
+
- **Skill cards:** YAML **`triggers`** / **`anti_triggers`** on **`SKILL.md`** are parsed and folded into summary embeddings and router prompts via **`app/routing_signals.py`** (`skill_routing_card`). Chunk RAG still scores on the **current** user message.
|
|
7
|
+
- **Dependency:** **`rank-bm25`** in `python/requirements.txt` (BM25 hybrid; optional at runtime if you use `keyword` only).
|
|
8
|
+
|
|
9
|
+
## 0.7.1
|
|
10
|
+
|
|
11
|
+
- **MCP:** **`search_skills`** (embedding shortlist + snippets), **`explain_route`** (routing diagnostics, no DB writes), **`get_skill`** (fetch one **`SKILL.md`** by name).
|
|
12
|
+
- **Route policies:** optional **`SKILLFORGE_ROUTE_POLICIES`**, **`SKILLFORGE_ROUTE_POLICIES_FILE`**, or **`project_root`/** `.skillforge/policies.json` / **`skillforge-policies.json`** — regex rules append **`include`** skills after the router (capped by **`SKILLFORGE_MAX_ACTIVE`**). Audit stored on route events under **`policy`**.
|
|
13
|
+
|
|
3
14
|
## 0.7.0
|
|
4
15
|
|
|
5
16
|
- **Breaking:** Removed the optional **HTTP API** (`skillforge start`), **`skillforge chat`** harness, and **`skillforge auth`** (bearer tokens were only used by HTTP). MCP (`skillforge mcp`), **`skillforge route`**, **`skillforge events`**, and **`skillforge index`** are unchanged.
|
package/README.md
CHANGED
|
@@ -154,7 +154,10 @@ With **Haiku** routing (uses your Anthropic key in the MCP process):
|
|
|
154
154
|
|
|
155
155
|
| Tool | Purpose |
|
|
156
156
|
|------|---------|
|
|
157
|
-
| `route_skills` | Returns routed **`SKILL.md`** context (chunks or full body). Pass **`project_root`** for per-repo SQLite under **`.skillforge/orchestrator.db`**. Optional **`include_project_rag`** (after **`skillforge index --project-root=…`**), **`session_id`**, **`user_id`** / **`SKILLFORGE_MCP_USER_ID`**, or env **`SKILLFORGE_PROJECT_ROOT`**. |
|
|
157
|
+
| `route_skills` | Returns routed **`SKILL.md`** context (chunks or full body). Pass **`project_root`** for per-repo SQLite under **`.skillforge/orchestrator.db`**. Optional **`include_project_rag`** (after **`skillforge index --project-root=…`**), **`session_id`**, **`user_id`** / **`SKILLFORGE_MCP_USER_ID`**, or env **`SKILLFORGE_PROJECT_ROOT`**. Route **`event.policy`** in SQLite logs policy merge audit when rules apply. |
|
|
158
|
+
| `search_skills` | Embedding-only shortlist for a **`query`** (scores + description snippets); does not run Haiku or mutate sessions. Optional **`limit`** (max 50). |
|
|
159
|
+
| `explain_route` | Same routing signal as **`route_skills`** without writing SQLite (**`picked_before_policy`**, **`picked_after_policy`**, shortlist facets, policy audit). For debugging. |
|
|
160
|
+
| `get_skill` | Fetch one catalog skill by **`skill_name`**; **`format`**: **`full`** or **`summary`**; optional **`max_chars`**. |
|
|
158
161
|
| `list_skills` | Catalog overview; optional **`user_id`** scopes usage stats. |
|
|
159
162
|
| `skill_feedback` | Feedback for the learning loop; optional **`user_id`**, **`session_id`** (stored with events). |
|
|
160
163
|
| `skill_referenced` | Mark a routed skill as **used** in the reply (increments **`referenced`** + weight; optional **`user_id`**). |
|
|
@@ -196,10 +199,14 @@ skillforge skills list
|
|
|
196
199
|
---
|
|
197
200
|
name: my-skill
|
|
198
201
|
description: Clear trigger conditions—used by the router.
|
|
202
|
+
triggers: When the user asks about X or mentions Y.
|
|
203
|
+
anti_triggers: Not for production deploy checks.
|
|
199
204
|
---
|
|
200
205
|
# My Skill
|
|
201
206
|
```
|
|
202
207
|
|
|
208
|
+
Optional **`triggers`** / **`anti_triggers`** strings are embedded with the summary card and shown to the Haiku router (they do not change chunk RAG, which still keys off the current user message).
|
|
209
|
+
|
|
203
210
|
Register with `skillforge skills add ./my-skill` or copy the folder to **`~/.skillforge/skills/`**.
|
|
204
211
|
|
|
205
212
|
**Skill packs** are git repositories with a root **`skillforge.json`** manifest listing skill folder names. Install:
|
|
@@ -217,10 +224,11 @@ skillforge pack remove <name>
|
|
|
217
224
|
## Routing pipeline
|
|
218
225
|
|
|
219
226
|
```
|
|
220
|
-
User prompt
|
|
221
|
-
→ Local embeddings (sentence-transformers)
|
|
222
|
-
→ Cosine similarity + per-user weights
|
|
227
|
+
User prompt (+ optional recent conversation for the shortlist query)
|
|
228
|
+
→ Local embeddings (sentence-transformers) on skill **cards** (title, description, optional triggers)
|
|
229
|
+
→ Cosine similarity ± hybrid keyword/BM25 fusion + per-user weights
|
|
223
230
|
→ Top-K candidates
|
|
231
|
+
→ Optional Haiku **rerank** on the shortlist (`SKILLFORGE_HAIKU_RERANK`)
|
|
224
232
|
→ Router model (Haiku) selects final active skills — *or* embedding-only mode takes top-N from candidates
|
|
225
233
|
→ Skill bodies injected; response model answers (e.g. Opus)
|
|
226
234
|
→ Usage signals update weights (optional)
|
|
@@ -230,6 +238,27 @@ Re-route: when overlap between successive active sets falls below a configurable
|
|
|
230
238
|
|
|
231
239
|
---
|
|
232
240
|
|
|
241
|
+
## Route policies (optional)
|
|
242
|
+
|
|
243
|
+
Rules use **`if_text_matches`** as a Python **`re.search`** pattern (with **`re.DOTALL`**) on the user **`prompt`**. **`include`** is a skill name or list of names. Matched skills are **appended** after Haiku/embedding picks until **`SKILLFORGE_MAX_ACTIVE`**.
|
|
244
|
+
|
|
245
|
+
**Load order:** env **`SKILLFORGE_ROUTE_POLICIES`** (inline JSON) → **`SKILLFORGE_ROUTE_POLICIES_FILE`** → **`<project_root>/.skillforge/policies.json`** → **`<project_root>/skillforge-policies.json`**.
|
|
246
|
+
|
|
247
|
+
Example **`.skillforge/policies.json`**:
|
|
248
|
+
|
|
249
|
+
```json
|
|
250
|
+
{
|
|
251
|
+
"rules": [
|
|
252
|
+
{
|
|
253
|
+
"if_text_matches": "(?i)(auth|oauth|jwt|password|login)",
|
|
254
|
+
"include": ["security-review"]
|
|
255
|
+
}
|
|
256
|
+
]
|
|
257
|
+
}
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
---
|
|
261
|
+
|
|
233
262
|
## Configuration
|
|
234
263
|
|
|
235
264
|
Environment variables (see also inline help and server defaults):
|
|
@@ -243,6 +272,16 @@ Environment variables (see also inline help and server defaults):
|
|
|
243
272
|
| `SKILLFORGE_TOP_K` | `15` | Embedding shortlist size. |
|
|
244
273
|
| `SKILLFORGE_MAX_ACTIVE` | `7` | Maximum skills injected per turn. |
|
|
245
274
|
| `SKILLFORGE_REROUTE_THRESHOLD` | `0.4` | Re-route sensitivity (Jaccard distance). |
|
|
275
|
+
| `SKILLFORGE_ROUTER_CONV_MAX_TURNS` | `0` | Include this many recent **conversation** messages in the **embedding shortlist** query (`0` = current user message only, legacy). |
|
|
276
|
+
| `SKILLFORGE_ROUTER_CONV_MSG_CHARS` | `320` | Max characters per message when building the shortlist query. |
|
|
277
|
+
| `SKILLFORGE_ROUTER_HYBRID` | `off` | `off` = dense cosine only. `keyword` = fuse with token overlap on skill cards. `bm25` = fuse with **BM25** (requires **`rank-bm25`**; falls back to keyword if missing). |
|
|
278
|
+
| `SKILLFORGE_ROUTER_HYBRID_ALPHA` | `0.72` | Hybrid weight on **dense** similarity (`1` = dense only; `0` = sparse only). |
|
|
279
|
+
| `SKILLFORGE_ROUTER_PROMPT_HISTORY_MSGS` | `8` | Max conversation turns sent to the **Haiku** router and reranker. |
|
|
280
|
+
| `SKILLFORGE_ROUTER_PROMPT_HISTORY_CHARS` | `360` | Max characters per turn in router / rerank prompts. |
|
|
281
|
+
| `SKILLFORGE_ROUTER_CATALOG_PREVIEW_CHARS` | `280` | Max characters of each skill **routing card** in the Haiku pick prompt. |
|
|
282
|
+
| `SKILLFORGE_HAIKU_RERANK` | `0` | Set **`1`** / **`true`** to rerank the Top-K shortlist with Haiku before the final pick (extra API call). |
|
|
283
|
+
| `SKILLFORGE_HAIKU_RERANK_MAX` | `SKILLFORGE_TOP_K` | Max candidates passed to the reranker. |
|
|
284
|
+
| `SKILLFORGE_HAIKU_RERANK_MODEL` | *(same as router)* | Model id for reranking when set; otherwise **`SKILLFORGE_ROUTER_MODEL`**. |
|
|
246
285
|
| `SKILLFORGE_CONTEXT_MODE` | `chunks` | `chunks` = embed **line-bounded chunks** from each picked skill body (RAG) up to **`SKILLFORGE_ROUTE_MAX_CHARS`**. `full_body` = inject entire **SKILL.md** per pick (legacy). |
|
|
247
286
|
| `SKILLFORGE_CHUNK_MAX_CHARS` | `1200` | Max characters per chunk (before overlap split). |
|
|
248
287
|
| `SKILLFORGE_CHUNK_OVERLAP` | `200` | Character overlap when hard-splitting an oversized section. |
|
|
@@ -264,6 +303,8 @@ Environment variables (see also inline help and server defaults):
|
|
|
264
303
|
| `SKILLFORGE_SKILL_HOT_RELOAD` | `1` | When **`0`** / **`false`**, disable **SKILL.md** hot-reload; restart the MCP process to refresh the catalog. |
|
|
265
304
|
| `SKILLFORGE_WATCH_SKILLS_INTERVAL` | `30` | Seconds between background catalog checks when hot reload is on. **`0`**: no background polling and no MCP **`tools.listChanged`**; **`tools/list`** and **`tools/call`** still reload when files change. |
|
|
266
305
|
| `SKILLFORGE_MCP_LIST_CHANGED` | `1` | When **`0`** / **`false`**, never emit **`notifications/tools/list_changed`** (and **`listChanged`** is not advertised), even if a background interval is set. |
|
|
306
|
+
| `SKILLFORGE_ROUTE_POLICIES` | `""` | Optional inline JSON policies document (see [Route policies](#route-policies-optional)). |
|
|
307
|
+
| `SKILLFORGE_ROUTE_POLICIES_FILE` | `""` | Path to a policies JSON file. |
|
|
267
308
|
|
|
268
309
|
---
|
|
269
310
|
|
|
@@ -274,6 +315,7 @@ Optional **per-project** state (when **`project_root`** or **`SKILLFORGE_PROJECT
|
|
|
274
315
|
```
|
|
275
316
|
<workspace>/.skillforge/
|
|
276
317
|
├── orchestrator.db # SQLite: sessions, weights, events, **project_chunks** (after `skillforge index`)
|
|
318
|
+
├── policies.json # Optional route policies (see README)
|
|
277
319
|
└── last_route.json # Last route_skills snapshot (after a routed call)
|
|
278
320
|
```
|
|
279
321
|
|
package/RELEASING.md
CHANGED
|
@@ -77,7 +77,7 @@ npm test
|
|
|
77
77
|
Python (syntax only):
|
|
78
78
|
|
|
79
79
|
```bash
|
|
80
|
-
for f in python/app/main.py python/app/mcp_server.py python/app/events_cli.py python/app/materialize.py python/app/db_paths.py python/app/route_cli.py python/app/mcp_contract.py python/app/chunking.py python/app/project_index.py python/app/index_cli.py python/app/context_fusion.py python/app/redaction.py; do python3 -m py_compile "$f"; done
|
|
80
|
+
for f in python/app/main.py python/app/mcp_server.py python/app/events_cli.py python/app/materialize.py python/app/db_paths.py python/app/route_cli.py python/app/mcp_contract.py python/app/chunking.py python/app/project_index.py python/app/index_cli.py python/app/context_fusion.py python/app/redaction.py python/app/route_policies.py python/app/routing_signals.py; do python3 -m py_compile "$f"; done
|
|
81
81
|
```
|
|
82
82
|
|
|
83
83
|
## Troubleshooting: `EOTP` / one-time password in CI
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@heytherevibin/skillforge",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.8.0",
|
|
4
4
|
"description": "Skill orchestration for Claude: hybrid embedding and router-based routing, MCP stdio server, per-user learning, and a large bundled SKILL.md catalog.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"claude",
|
package/python/app/main.py
CHANGED
|
@@ -8,7 +8,6 @@ Live usage: `skillforge events --watch` (terminal).
|
|
|
8
8
|
"""
|
|
9
9
|
from __future__ import annotations
|
|
10
10
|
|
|
11
|
-
import asyncio
|
|
12
11
|
import json
|
|
13
12
|
import os
|
|
14
13
|
import sqlite3
|
|
@@ -33,6 +32,14 @@ from app.project_index import (
|
|
|
33
32
|
retrieve_project_context_items,
|
|
34
33
|
)
|
|
35
34
|
from app.redaction import redaction_enabled, redact_secret_patterns, sanitize_context_items
|
|
35
|
+
from app.route_policies import load_route_policies_config, merge_policy_includes
|
|
36
|
+
from app.routing_signals import (
|
|
37
|
+
build_route_query_text,
|
|
38
|
+
keyword_overlap_scores,
|
|
39
|
+
normalize_minmax,
|
|
40
|
+
skill_routing_card,
|
|
41
|
+
tokenize_skills_query,
|
|
42
|
+
)
|
|
36
43
|
|
|
37
44
|
# ---------- Config (env-driven so the Node wrapper controls paths) ----------
|
|
38
45
|
BUNDLED_SKILLS = Path(os.getenv("SKILLFORGE_BUNDLED_SKILLS", "./skills"))
|
|
@@ -60,6 +67,21 @@ FUSION_FULL_BODY_PREVIEW_CHARS = max(400, int(os.getenv("SKILLFORGE_FUSION_FULL_
|
|
|
60
67
|
CONTEXT_OVERHEAD_SKILL = 48
|
|
61
68
|
CONTEXT_OVERHEAD_FILE = 56
|
|
62
69
|
|
|
70
|
+
ROUTER_HYBRID_MODE = os.getenv("SKILLFORGE_ROUTER_HYBRID", "off").strip().lower()
|
|
71
|
+
ROUTER_HYBRID_ALPHA = max(0.0, min(1.0, float(os.getenv("SKILLFORGE_ROUTER_HYBRID_ALPHA", "0.72"))))
|
|
72
|
+
ROUTER_PROMPT_HISTORY_MSGS = max(1, int(os.getenv("SKILLFORGE_ROUTER_PROMPT_HISTORY_MSGS", "8")))
|
|
73
|
+
ROUTER_PROMPT_HISTORY_CHARS = max(80, int(os.getenv("SKILLFORGE_ROUTER_PROMPT_HISTORY_CHARS", "360")))
|
|
74
|
+
ROUTER_CATALOG_PREVIEW_CHARS = max(80, int(os.getenv("SKILLFORGE_ROUTER_CATALOG_PREVIEW_CHARS", "280")))
|
|
75
|
+
HAIKU_RERANK_MAX = max(3, int(os.getenv("SKILLFORGE_HAIKU_RERANK_MAX", str(TOP_K_CANDIDATES))))
|
|
76
|
+
|
|
77
|
+
|
|
78
|
+
def _hybrid_mode_active(mode: str) -> bool:
|
|
79
|
+
return mode not in ("", "off", "0", "false", "no")
|
|
80
|
+
|
|
81
|
+
|
|
82
|
+
def _env_truthy(name: str, default: str = "0") -> bool:
|
|
83
|
+
return os.getenv(name, default).strip().lower() not in ("0", "false", "no", "")
|
|
84
|
+
|
|
63
85
|
|
|
64
86
|
def _context_budget_unified() -> int:
|
|
65
87
|
raw = os.getenv("SKILLFORGE_CONTEXT_BUDGET_CHARS", "").strip()
|
|
@@ -123,6 +145,8 @@ class Skill:
|
|
|
123
145
|
source: str # "bundled" | "user"
|
|
124
146
|
disabled: bool = False
|
|
125
147
|
embedding: np.ndarray | None = None
|
|
148
|
+
triggers: str = ""
|
|
149
|
+
anti_triggers: str = ""
|
|
126
150
|
|
|
127
151
|
|
|
128
152
|
def parse_skill_md(path: Path, source: str) -> Skill | None:
|
|
@@ -138,6 +162,8 @@ def parse_skill_md(path: Path, source: str) -> Skill | None:
|
|
|
138
162
|
name = path.parent.name
|
|
139
163
|
title = name.replace("-", " ").title()
|
|
140
164
|
description = ""
|
|
165
|
+
triggers = ""
|
|
166
|
+
anti_triggers = ""
|
|
141
167
|
body = text
|
|
142
168
|
if text.startswith("---"):
|
|
143
169
|
end = text.find("---", 3)
|
|
@@ -167,6 +193,10 @@ def parse_skill_md(path: Path, source: str) -> Skill | None:
|
|
|
167
193
|
title = v
|
|
168
194
|
elif k == "description":
|
|
169
195
|
description = v
|
|
196
|
+
elif k in ("triggers", "trigger"):
|
|
197
|
+
triggers = v
|
|
198
|
+
elif k in ("anti_triggers", "anti-triggers"):
|
|
199
|
+
anti_triggers = v
|
|
170
200
|
i += 1
|
|
171
201
|
if not description:
|
|
172
202
|
for chunk in body.split("\n\n"):
|
|
@@ -174,7 +204,15 @@ def parse_skill_md(path: Path, source: str) -> Skill | None:
|
|
|
174
204
|
if chunk and not chunk.startswith("#"):
|
|
175
205
|
description = chunk[:500]
|
|
176
206
|
break
|
|
177
|
-
return Skill(
|
|
207
|
+
return Skill(
|
|
208
|
+
name=name,
|
|
209
|
+
title=title,
|
|
210
|
+
description=description,
|
|
211
|
+
body=body,
|
|
212
|
+
source=source,
|
|
213
|
+
triggers=triggers,
|
|
214
|
+
anti_triggers=anti_triggers,
|
|
215
|
+
)
|
|
178
216
|
|
|
179
217
|
|
|
180
218
|
def load_all_skills() -> list[Skill]:
|
|
@@ -325,8 +363,26 @@ class Router:
|
|
|
325
363
|
"full_body",
|
|
326
364
|
) else "chunks"
|
|
327
365
|
self._by_name: dict[str, Skill] = {s.name: s for s in skills}
|
|
328
|
-
|
|
329
|
-
|
|
366
|
+
self._hybrid_mode = ROUTER_HYBRID_MODE
|
|
367
|
+
self._hybrid_alpha = ROUTER_HYBRID_ALPHA
|
|
368
|
+
self._routing_cards = [skill_routing_card(s) for s in skills]
|
|
369
|
+
self._bm25 = None
|
|
370
|
+
if self._hybrid_mode == "bm25" and skills:
|
|
371
|
+
try:
|
|
372
|
+
from rank_bm25 import BM25Okapi
|
|
373
|
+
|
|
374
|
+
toks = [tokenize_skills_query(c) for c in self._routing_cards]
|
|
375
|
+
if any(toks):
|
|
376
|
+
self._bm25 = BM25Okapi(toks)
|
|
377
|
+
except ImportError:
|
|
378
|
+
print(
|
|
379
|
+
"[skillforge] SKILLFORGE_ROUTER_HYBRID=bm25 but rank-bm25 is not installed; "
|
|
380
|
+
"using keyword overlap for sparse signal.",
|
|
381
|
+
file=sys.stderr,
|
|
382
|
+
)
|
|
383
|
+
|
|
384
|
+
texts = self._routing_cards
|
|
385
|
+
print(f"[skillforge] Embedding {len(skills)} skills (summary cards)...", file=sys.stderr)
|
|
330
386
|
embeddings = embed_model.encode(texts, show_progress_bar=False, convert_to_numpy=True)
|
|
331
387
|
for s, e in zip(skills, embeddings):
|
|
332
388
|
s.embedding = e / np.linalg.norm(e)
|
|
@@ -355,23 +411,48 @@ class Router:
|
|
|
355
411
|
self._chunk_embeddings = ce
|
|
356
412
|
print(
|
|
357
413
|
f"[skillforge] Ready. {len(skills)} skills; chunk matrix {self._chunk_embeddings.shape}; "
|
|
358
|
-
f"context_mode={self.context_mode}",
|
|
414
|
+
f"context_mode={self.context_mode}; router_hybrid={self._hybrid_mode}",
|
|
359
415
|
file=sys.stderr,
|
|
360
416
|
)
|
|
361
417
|
else:
|
|
362
418
|
print(
|
|
363
419
|
f"[skillforge] Ready. {len(skills)} skills, matrix shape: {self.matrix.shape}; "
|
|
364
|
-
f"context_mode={self.context_mode}",
|
|
420
|
+
f"context_mode={self.context_mode}; router_hybrid={self._hybrid_mode}",
|
|
365
421
|
file=sys.stderr,
|
|
366
422
|
)
|
|
367
423
|
|
|
368
|
-
def
|
|
424
|
+
def _sparse_scores(self, route_query: str) -> np.ndarray:
|
|
425
|
+
if not _hybrid_mode_active(self._hybrid_mode):
|
|
426
|
+
return np.zeros(len(self.skills), dtype=np.float64)
|
|
427
|
+
if self._hybrid_mode == "keyword":
|
|
428
|
+
return keyword_overlap_scores(route_query, self._routing_cards)
|
|
429
|
+
if self._hybrid_mode == "bm25":
|
|
430
|
+
if self._bm25 is not None:
|
|
431
|
+
q = tokenize_skills_query(route_query)
|
|
432
|
+
if not q:
|
|
433
|
+
return np.zeros(len(self.skills), dtype=np.float64)
|
|
434
|
+
return np.asarray(self._bm25.get_scores(q), dtype=np.float64)
|
|
435
|
+
return keyword_overlap_scores(route_query, self._routing_cards)
|
|
436
|
+
return keyword_overlap_scores(route_query, self._routing_cards)
|
|
437
|
+
|
|
438
|
+
def _base_routing_scores(self, route_query: str, q: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
|
|
439
|
+
"""Dense cosine similarities and fused ranking scores (or dense-only if hybrid off)."""
|
|
440
|
+
sims = (self.matrix @ q).flatten()
|
|
441
|
+
if not _hybrid_mode_active(self._hybrid_mode):
|
|
442
|
+
return sims, sims
|
|
443
|
+
sparse = self._sparse_scores(route_query)
|
|
444
|
+
d_norm = normalize_minmax(sims)
|
|
445
|
+
s_norm = normalize_minmax(sparse)
|
|
446
|
+
fused = self._hybrid_alpha * d_norm + (1.0 - self._hybrid_alpha) * s_norm
|
|
447
|
+
return sims, fused
|
|
448
|
+
|
|
449
|
+
def shortlist(self, route_query, con, k=TOP_K_CANDIDATES, user_id=""):
|
|
369
450
|
if len(self.skills) == 0:
|
|
370
451
|
return []
|
|
371
|
-
q = self.embed_model.encode(
|
|
452
|
+
q = self.embed_model.encode(route_query, convert_to_numpy=True)
|
|
372
453
|
q = q / np.linalg.norm(q)
|
|
373
|
-
sims = self.
|
|
374
|
-
biased =
|
|
454
|
+
sims, rank_scores = self._base_routing_scores(route_query, q)
|
|
455
|
+
biased = rank_scores.copy()
|
|
375
456
|
for i, s in enumerate(self.skills):
|
|
376
457
|
w, disabled = get_skill_weight(con, s.name, user_id=user_id)
|
|
377
458
|
if disabled:
|
|
@@ -381,6 +462,53 @@ class Router:
|
|
|
381
462
|
top_idx = np.argsort(-biased)[:k]
|
|
382
463
|
return [(self.skills[i], float(sims[i])) for i in top_idx if biased[i] > -100]
|
|
383
464
|
|
|
465
|
+
def shortlist_with_facets(
|
|
466
|
+
self,
|
|
467
|
+
route_query: str,
|
|
468
|
+
con: sqlite3.Connection,
|
|
469
|
+
*,
|
|
470
|
+
k: int | None = None,
|
|
471
|
+
user_id: str = "",
|
|
472
|
+
) -> list[dict[str, Any]]:
|
|
473
|
+
"""Embedding shortlist with cosine sim, learned weight, and routing score (no LLM)."""
|
|
474
|
+
limit = k if k is not None else TOP_K_CANDIDATES
|
|
475
|
+
if len(self.skills) == 0:
|
|
476
|
+
return []
|
|
477
|
+
q = self.embed_model.encode(route_query, convert_to_numpy=True)
|
|
478
|
+
q = q / np.linalg.norm(q)
|
|
479
|
+
sims, rank_scores = self._base_routing_scores(route_query, q)
|
|
480
|
+
sparse_full = (
|
|
481
|
+
self._sparse_scores(route_query) if _hybrid_mode_active(self._hybrid_mode) else np.zeros(
|
|
482
|
+
len(self.skills), dtype=np.float64
|
|
483
|
+
)
|
|
484
|
+
)
|
|
485
|
+
biased = rank_scores.copy()
|
|
486
|
+
for i, s in enumerate(self.skills):
|
|
487
|
+
w, disabled = get_skill_weight(con, s.name, user_id=user_id)
|
|
488
|
+
if disabled:
|
|
489
|
+
biased[i] = -999.0
|
|
490
|
+
else:
|
|
491
|
+
biased[i] += w
|
|
492
|
+
top_idx = np.argsort(-biased)[:limit]
|
|
493
|
+
out: list[dict[str, Any]] = []
|
|
494
|
+
for i in top_idx:
|
|
495
|
+
if biased[i] <= -100:
|
|
496
|
+
continue
|
|
497
|
+
s = self.skills[i]
|
|
498
|
+
w, _dis = get_skill_weight(con, s.name, user_id=user_id)
|
|
499
|
+
out.append({
|
|
500
|
+
"name": s.name,
|
|
501
|
+
"title": s.title,
|
|
502
|
+
"description_preview": (s.description or "")[:280],
|
|
503
|
+
"cosine_similarity": round(float(sims[i]), 6),
|
|
504
|
+
"sparse_signal": round(float(sparse_full[i]), 6),
|
|
505
|
+
"learned_weight": round(float(w), 4),
|
|
506
|
+
"routing_score": round(float(biased[i]), 6),
|
|
507
|
+
"source": s.source,
|
|
508
|
+
"router_hybrid": self._hybrid_mode,
|
|
509
|
+
})
|
|
510
|
+
return out
|
|
511
|
+
|
|
384
512
|
def build_context_items(
|
|
385
513
|
self,
|
|
386
514
|
prompt: str,
|
|
@@ -551,6 +679,77 @@ class Router:
|
|
|
551
679
|
rel_out.append(float(rel[i]))
|
|
552
680
|
return items, np.stack(em_rows), np.asarray(rel_out, dtype=np.float32)
|
|
553
681
|
|
|
682
|
+
async def rerank_candidates_haiku(
|
|
683
|
+
self,
|
|
684
|
+
route_query: str,
|
|
685
|
+
conversation: list | None,
|
|
686
|
+
candidates: list[tuple[Skill, float]],
|
|
687
|
+
) -> list[tuple[Skill, float]]:
|
|
688
|
+
if (
|
|
689
|
+
not candidates
|
|
690
|
+
or self.anthropic is None
|
|
691
|
+
or not _env_truthy("SKILLFORGE_HAIKU_RERANK", "0")
|
|
692
|
+
):
|
|
693
|
+
return candidates
|
|
694
|
+
cap = max(3, min(HAIKU_RERANK_MAX, len(candidates)))
|
|
695
|
+
head = candidates[:cap]
|
|
696
|
+
tail = candidates[cap:]
|
|
697
|
+
by_name = {s.name: (s, sc) for s, sc in head}
|
|
698
|
+
lines: list[str] = []
|
|
699
|
+
for idx, (s, _sc) in enumerate(head, start=1):
|
|
700
|
+
card = skill_routing_card(s)
|
|
701
|
+
preview = card[:220].replace("\n", " ")
|
|
702
|
+
lines.append(f"{idx}. {s.name} — {preview}")
|
|
703
|
+
hist = ""
|
|
704
|
+
if conversation:
|
|
705
|
+
msgs = conversation[-ROUTER_PROMPT_HISTORY_MSGS:]
|
|
706
|
+
parts: list[str] = []
|
|
707
|
+
for m in msgs:
|
|
708
|
+
if not isinstance(m, dict):
|
|
709
|
+
continue
|
|
710
|
+
role = str(m.get("role") or "user")
|
|
711
|
+
c = str(m.get("content") or "").strip()
|
|
712
|
+
if not c:
|
|
713
|
+
continue
|
|
714
|
+
parts.append(f"{role}: {c[:ROUTER_PROMPT_HISTORY_CHARS]}")
|
|
715
|
+
if parts:
|
|
716
|
+
hist = "\n\nConversation (recent):\n" + "\n".join(parts)
|
|
717
|
+
sys = (
|
|
718
|
+
"You reorder skill candidates by relevance to the user's task. "
|
|
719
|
+
"Output ONLY JSON: {\"order\": [\"skill_name\", ...]} with each candidate "
|
|
720
|
+
"skill name appearing exactly once, best match first. No extra keys."
|
|
721
|
+
)
|
|
722
|
+
user = (
|
|
723
|
+
f"Routing focus:\n{route_query}{hist}\n\nCandidates:\n" + "\n".join(lines)
|
|
724
|
+
)
|
|
725
|
+
try:
|
|
726
|
+
rerank_model = os.getenv("SKILLFORGE_HAIKU_RERANK_MODEL", "").strip() or ROUTER_MODEL
|
|
727
|
+
resp = await self.anthropic.messages.create(
|
|
728
|
+
model=rerank_model,
|
|
729
|
+
max_tokens=500,
|
|
730
|
+
system=sys,
|
|
731
|
+
messages=[{"role": "user", "content": user}],
|
|
732
|
+
)
|
|
733
|
+
text = resp.content[0].text.strip()
|
|
734
|
+
if text.startswith("```"):
|
|
735
|
+
text = text.split("```")[1]
|
|
736
|
+
if text.startswith("json"):
|
|
737
|
+
text = text[4:]
|
|
738
|
+
data = json.loads(text.strip())
|
|
739
|
+
order = data.get("order") or []
|
|
740
|
+
ordered: list[tuple[Skill, float]] = []
|
|
741
|
+
seen: set[str] = set()
|
|
742
|
+
for n in order:
|
|
743
|
+
if isinstance(n, str) and n in by_name and n not in seen:
|
|
744
|
+
ordered.append(by_name[n])
|
|
745
|
+
seen.add(n)
|
|
746
|
+
for s, sc in head:
|
|
747
|
+
if s.name not in seen:
|
|
748
|
+
ordered.append((s, sc))
|
|
749
|
+
return ordered + tail
|
|
750
|
+
except Exception:
|
|
751
|
+
return candidates
|
|
752
|
+
|
|
554
753
|
def pick_final_embedding_only(self, candidates):
|
|
555
754
|
"""Pick up to MAX_ACTIVE_SKILLS from the shortlist order (similarity + weights). No LLM call."""
|
|
556
755
|
if not candidates:
|
|
@@ -560,26 +759,46 @@ class Router:
|
|
|
560
759
|
"embedding-only: top candidates by similarity and learned weights"
|
|
561
760
|
)
|
|
562
761
|
|
|
563
|
-
async def pick_final(
|
|
762
|
+
async def pick_final(
|
|
763
|
+
self,
|
|
764
|
+
prompt,
|
|
765
|
+
conversation,
|
|
766
|
+
candidates,
|
|
767
|
+
route_query: str | None = None,
|
|
768
|
+
):
|
|
769
|
+
rq = (route_query if route_query is not None else prompt) or ""
|
|
564
770
|
if self.anthropic is None:
|
|
565
771
|
return self.pick_final_embedding_only(candidates)
|
|
566
772
|
if not candidates:
|
|
567
773
|
return [], "no candidates available"
|
|
568
774
|
catalog = "\n".join(
|
|
569
|
-
f"- {s.name}: {s
|
|
775
|
+
f"- {s.name}: {skill_routing_card(s)[:ROUTER_CATALOG_PREVIEW_CHARS]}"
|
|
776
|
+
for s, _ in candidates
|
|
570
777
|
)
|
|
571
778
|
recent = ""
|
|
572
779
|
if conversation:
|
|
573
|
-
|
|
574
|
-
|
|
575
|
-
|
|
780
|
+
msgs = conversation[-ROUTER_PROMPT_HISTORY_MSGS:]
|
|
781
|
+
parts: list[str] = []
|
|
782
|
+
for m in msgs:
|
|
783
|
+
if not isinstance(m, dict):
|
|
784
|
+
continue
|
|
785
|
+
role = str(m.get("role") or "user")
|
|
786
|
+
c = str(m.get("content") or "").strip()
|
|
787
|
+
if not c:
|
|
788
|
+
continue
|
|
789
|
+
parts.append(f"{role}: {c[:ROUTER_PROMPT_HISTORY_CHARS]}")
|
|
790
|
+
if parts:
|
|
791
|
+
recent = "\n\nRecent conversation:\n" + "\n".join(parts)
|
|
576
792
|
sys = (
|
|
577
793
|
"You are a skill router. Given a user prompt and a candidate list of skills, "
|
|
578
794
|
f"pick 0 to {MAX_ACTIVE_SKILLS} skills that would genuinely help answer this prompt. "
|
|
579
795
|
"Be ruthless — only include a skill if it directly applies. Empty list is valid. "
|
|
580
796
|
'Respond ONLY in JSON: {"skills": ["name1","name2"], "reasoning": "one sentence"}'
|
|
581
797
|
)
|
|
582
|
-
user =
|
|
798
|
+
user = (
|
|
799
|
+
f"User prompt:\n{prompt}\n\nRouting context (retrieval query):\n{rq}{recent}"
|
|
800
|
+
f"\n\nCandidate skills:\n{catalog}"
|
|
801
|
+
)
|
|
583
802
|
try:
|
|
584
803
|
resp = await self.anthropic.messages.create(
|
|
585
804
|
model=ROUTER_MODEL,
|
|
@@ -643,8 +862,23 @@ async def run_route_turn(
|
|
|
643
862
|
"""
|
|
644
863
|
sid = session_id or str(uuid.uuid4())
|
|
645
864
|
t0 = time.time()
|
|
646
|
-
|
|
647
|
-
|
|
865
|
+
route_query = build_route_query_text(prompt, conversation)
|
|
866
|
+
candidates = router.shortlist(route_query, con, user_id=user_id)
|
|
867
|
+
candidates = await router.rerank_candidates_haiku(route_query, conversation, candidates)
|
|
868
|
+
picked_names, reasoning = await router.pick_final(
|
|
869
|
+
prompt, conversation, candidates, route_query=route_query
|
|
870
|
+
)
|
|
871
|
+
pr = (project_root or "").strip()
|
|
872
|
+
policies_cfg = load_route_policies_config(pr or None)
|
|
873
|
+
picked_names, policy_audit = merge_policy_includes(
|
|
874
|
+
prompt,
|
|
875
|
+
picked_names,
|
|
876
|
+
policies_cfg,
|
|
877
|
+
router._by_name,
|
|
878
|
+
con,
|
|
879
|
+
user_id,
|
|
880
|
+
max_active=MAX_ACTIVE_SKILLS,
|
|
881
|
+
)
|
|
648
882
|
route_ms = (time.time() - t0) * 1000
|
|
649
883
|
|
|
650
884
|
prev_active: set[str] = set()
|
|
@@ -658,7 +892,6 @@ async def run_route_turn(
|
|
|
658
892
|
change = jaccard_change(prev_active, set(picked_names))
|
|
659
893
|
rerouted = change >= REROUTE_THRESHOLD and bool(prev_active)
|
|
660
894
|
|
|
661
|
-
pr = (project_root or "").strip()
|
|
662
895
|
want_fusion = CONTEXT_FUSION and include_project_rag and bool(pr)
|
|
663
896
|
context_fusion: dict[str, Any] | None = None
|
|
664
897
|
context_items: list[dict[str, Any]] = []
|
|
@@ -788,6 +1021,10 @@ async def run_route_turn(
|
|
|
788
1021
|
"include_project_rag": bool(include_project_rag and pr),
|
|
789
1022
|
"context_fusion": context_fusion,
|
|
790
1023
|
"context_redaction": context_redaction_stats,
|
|
1024
|
+
"policy": {
|
|
1025
|
+
"rules_loaded": len(policies_cfg.get("rules") or []) if isinstance(policies_cfg.get("rules"), list) else 0,
|
|
1026
|
+
"audit": policy_audit,
|
|
1027
|
+
},
|
|
791
1028
|
"chunk_sources_preview": [
|
|
792
1029
|
{
|
|
793
1030
|
"skill": c.get("skill"),
|
package/python/app/mcp_server.py
CHANGED
|
@@ -2,11 +2,11 @@
|
|
|
2
2
|
MCP server for skillforge.
|
|
3
3
|
|
|
4
4
|
Exposes skill routing as MCP tools so MCP-aware clients (Claude Desktop,
|
|
5
|
-
Claude Code, Cursor, etc.) can use the orchestrator
|
|
6
|
-
HTTP server.
|
|
5
|
+
Claude Code, Cursor, etc.) can use the orchestrator locally.
|
|
7
6
|
|
|
8
7
|
Tools exposed:
|
|
9
8
|
route_skills / skillforge_bootstrap — routing (+ optional project materialize).
|
|
9
|
+
search_skills / explain_route / get_skill — retrieval, debugging, deterministic fetch.
|
|
10
10
|
materialize_project — .cursor/rules, docs/SKILLFORGE-PRD.md, CLAUDE.md block.
|
|
11
11
|
list_skills, skill_feedback, skill_referenced, disable_skill.
|
|
12
12
|
|
|
@@ -25,6 +25,8 @@ from pathlib import Path
|
|
|
25
25
|
|
|
26
26
|
from app.db_paths import resolve_orchestrator_db
|
|
27
27
|
from app.main import (
|
|
28
|
+
TOP_K_CANDIDATES,
|
|
29
|
+
MAX_ACTIVE_SKILLS,
|
|
28
30
|
build_router_and_skills,
|
|
29
31
|
format_context_items_markdown,
|
|
30
32
|
init_db,
|
|
@@ -39,6 +41,8 @@ from app.main import (
|
|
|
39
41
|
from app.materialize import materialize_project_files
|
|
40
42
|
from app.mcp_contract import MCP_RESPONSE_SCHEMA_VERSION, build_route_skills_meta
|
|
41
43
|
from app.redaction import redaction_enabled, redact_display_path
|
|
44
|
+
from app.route_policies import load_route_policies_config, merge_policy_includes
|
|
45
|
+
from app.routing_signals import build_route_query_text
|
|
42
46
|
|
|
43
47
|
|
|
44
48
|
def _env_truthy(name: str, default: str = "1") -> bool:
|
|
@@ -85,7 +89,7 @@ class MCPServer:
|
|
|
85
89
|
self._db_cache: dict[str, sqlite3.Connection] = {}
|
|
86
90
|
|
|
87
91
|
def _mcp_user_id(self, args: dict) -> str:
|
|
88
|
-
"""Per-tool user namespace for weights/sessions/events
|
|
92
|
+
"""Per-tool user namespace for weights/sessions/events."""
|
|
89
93
|
raw = (
|
|
90
94
|
args.get("user_id")
|
|
91
95
|
or os.getenv("SKILLFORGE_MCP_USER_ID", "")
|
|
@@ -185,7 +189,7 @@ class MCPServer:
|
|
|
185
189
|
return {
|
|
186
190
|
"protocolVersion": "2024-11-05",
|
|
187
191
|
"capabilities": caps,
|
|
188
|
-
"serverInfo": {"name": "skillforge", "version": "0.7.
|
|
192
|
+
"serverInfo": {"name": "skillforge", "version": "0.7.1"},
|
|
189
193
|
}
|
|
190
194
|
|
|
191
195
|
def handle_tools_list(self, params):
|
|
@@ -231,12 +235,80 @@ class MCPServer:
|
|
|
231
235
|
},
|
|
232
236
|
"user_id": {
|
|
233
237
|
"type": "string",
|
|
234
|
-
"description": "Logical user id for weights/sessions/events
|
|
238
|
+
"description": "Logical user id for weights/sessions/events",
|
|
235
239
|
},
|
|
236
240
|
},
|
|
237
241
|
"required": ["prompt"],
|
|
238
242
|
},
|
|
239
243
|
},
|
|
244
|
+
{
|
|
245
|
+
"name": "search_skills",
|
|
246
|
+
"description": (
|
|
247
|
+
"Embedding-only retrieval: top skills for a query with similarity scores "
|
|
248
|
+
"and descriptions (no Haiku, no full route). Use to explore the catalog."
|
|
249
|
+
),
|
|
250
|
+
"inputSchema": {
|
|
251
|
+
"type": "object",
|
|
252
|
+
"properties": {
|
|
253
|
+
"query": {"type": "string", "description": "Search query or task text"},
|
|
254
|
+
"limit": {
|
|
255
|
+
"type": "integer",
|
|
256
|
+
"description": f"Max skills to return (default {TOP_K_CANDIDATES})",
|
|
257
|
+
},
|
|
258
|
+
"project_root": {"type": "string"},
|
|
259
|
+
"user_id": {"type": "string"},
|
|
260
|
+
},
|
|
261
|
+
"required": ["query"],
|
|
262
|
+
},
|
|
263
|
+
},
|
|
264
|
+
{
|
|
265
|
+
"name": "explain_route",
|
|
266
|
+
"description": (
|
|
267
|
+
"Debug routing: embedding facets for the shortlist (same query text as route_skills when "
|
|
268
|
+
"`conversation` is passed — conversation-aware when SKILLFORGE_ROUTER_CONV_MAX_TURNS > 0), "
|
|
269
|
+
"optional Haiku rerank, Haiku/embedding-only pick with reasoning, and policy merge audit. "
|
|
270
|
+
"Does not write sessions or increment uses."
|
|
271
|
+
),
|
|
272
|
+
"inputSchema": {
|
|
273
|
+
"type": "object",
|
|
274
|
+
"properties": {
|
|
275
|
+
"prompt": {"type": "string"},
|
|
276
|
+
"conversation": {"type": "array", "items": {"type": "object"}},
|
|
277
|
+
"limit": {
|
|
278
|
+
"type": "integer",
|
|
279
|
+
"description": "Max shortlist rows in facets (default TOP_K)",
|
|
280
|
+
},
|
|
281
|
+
"project_root": {"type": "string"},
|
|
282
|
+
"user_id": {"type": "string"},
|
|
283
|
+
},
|
|
284
|
+
"required": ["prompt"],
|
|
285
|
+
},
|
|
286
|
+
},
|
|
287
|
+
{
|
|
288
|
+
"name": "get_skill",
|
|
289
|
+
"description": (
|
|
290
|
+
"Load one skill by name: full SKILL.md body or a short summary. "
|
|
291
|
+
"Use for deterministic workflows when you already know the skill name."
|
|
292
|
+
),
|
|
293
|
+
"inputSchema": {
|
|
294
|
+
"type": "object",
|
|
295
|
+
"properties": {
|
|
296
|
+
"skill_name": {"type": "string"},
|
|
297
|
+
"format": {
|
|
298
|
+
"type": "string",
|
|
299
|
+
"enum": ["full", "summary"],
|
|
300
|
+
"description": "summary = description + first ~8k chars of body",
|
|
301
|
+
"default": "full",
|
|
302
|
+
},
|
|
303
|
+
"max_chars": {
|
|
304
|
+
"type": "integer",
|
|
305
|
+
"description": "If > 0, truncate body to this many characters",
|
|
306
|
+
"default": 0,
|
|
307
|
+
},
|
|
308
|
+
},
|
|
309
|
+
"required": ["skill_name"],
|
|
310
|
+
},
|
|
311
|
+
},
|
|
240
312
|
{
|
|
241
313
|
"name": "list_skills",
|
|
242
314
|
"description": (
|
|
@@ -358,6 +430,12 @@ class MCPServer:
|
|
|
358
430
|
|
|
359
431
|
if name == "route_skills":
|
|
360
432
|
return await self._tool_route_skills(args)
|
|
433
|
+
if name == "search_skills":
|
|
434
|
+
return self._tool_search_skills(args)
|
|
435
|
+
if name == "explain_route":
|
|
436
|
+
return await self._tool_explain_route(args)
|
|
437
|
+
if name == "get_skill":
|
|
438
|
+
return self._tool_get_skill(args)
|
|
361
439
|
if name == "list_skills":
|
|
362
440
|
return self._tool_list_skills(args)
|
|
363
441
|
if name == "skill_feedback":
|
|
@@ -458,6 +536,150 @@ class MCPServer:
|
|
|
458
536
|
"_meta": meta,
|
|
459
537
|
}
|
|
460
538
|
|
|
539
|
+
def _tool_search_skills(self, args):
|
|
540
|
+
query = (args.get("query") or "").strip()
|
|
541
|
+
user_id = self._mcp_user_id(args)
|
|
542
|
+
pr = self._project_root_from_args(args)
|
|
543
|
+
db_path = resolve_orchestrator_db(pr)
|
|
544
|
+
if not query:
|
|
545
|
+
return {
|
|
546
|
+
"content": [{"type": "text", "text": "query is required."}],
|
|
547
|
+
"isError": True,
|
|
548
|
+
}
|
|
549
|
+
try:
|
|
550
|
+
limit = int(args.get("limit") or TOP_K_CANDIDATES)
|
|
551
|
+
except (TypeError, ValueError):
|
|
552
|
+
limit = TOP_K_CANDIDATES
|
|
553
|
+
limit = max(1, min(limit, 50))
|
|
554
|
+
con = self._get_con(args)
|
|
555
|
+
facets = self.router.shortlist_with_facets(query, con, k=limit, user_id=user_id)
|
|
556
|
+
lines = ["# search_skills — embedding shortlist", ""]
|
|
557
|
+
for f in facets:
|
|
558
|
+
lines.append(
|
|
559
|
+
f"- **{f['name']}** (cos {f['cosine_similarity']}, score {f['routing_score']}): "
|
|
560
|
+
f"{(f.get('description_preview') or '')[:220]}"
|
|
561
|
+
)
|
|
562
|
+
text = "\n".join(lines)
|
|
563
|
+
return {
|
|
564
|
+
"content": [{"type": "text", "text": text}],
|
|
565
|
+
"_meta": {
|
|
566
|
+
"schema_version": MCP_RESPONSE_SCHEMA_VERSION,
|
|
567
|
+
"tool": "search_skills",
|
|
568
|
+
"orchestrator_db": redact_display_path(db_path) if redaction_enabled() else str(db_path),
|
|
569
|
+
"results": facets,
|
|
570
|
+
"count": len(facets),
|
|
571
|
+
},
|
|
572
|
+
}
|
|
573
|
+
|
|
574
|
+
async def _tool_explain_route(self, args):
|
|
575
|
+
prompt = (args.get("prompt") or "").strip()
|
|
576
|
+
conversation = args.get("conversation") or []
|
|
577
|
+
user_id = self._mcp_user_id(args)
|
|
578
|
+
pr = self._project_root_from_args(args)
|
|
579
|
+
db_path = resolve_orchestrator_db(pr)
|
|
580
|
+
if not prompt:
|
|
581
|
+
return {
|
|
582
|
+
"content": [{"type": "text", "text": "prompt is required."}],
|
|
583
|
+
"isError": True,
|
|
584
|
+
}
|
|
585
|
+
try:
|
|
586
|
+
limit = int(args.get("limit") or TOP_K_CANDIDATES)
|
|
587
|
+
except (TypeError, ValueError):
|
|
588
|
+
limit = TOP_K_CANDIDATES
|
|
589
|
+
limit = max(1, min(limit, 50))
|
|
590
|
+
con = self._get_con(args)
|
|
591
|
+
route_query = build_route_query_text(prompt, conversation)
|
|
592
|
+
facets = self.router.shortlist_with_facets(route_query, con, k=limit, user_id=user_id)
|
|
593
|
+
candidates = self.router.shortlist(route_query, con, user_id=user_id)
|
|
594
|
+
candidates = await self.router.rerank_candidates_haiku(route_query, conversation, candidates)
|
|
595
|
+
picked, reasoning = await self.router.pick_final(
|
|
596
|
+
prompt, conversation, candidates, route_query=route_query
|
|
597
|
+
)
|
|
598
|
+
policies_cfg = load_route_policies_config(pr)
|
|
599
|
+
merged, policy_audit = merge_policy_includes(
|
|
600
|
+
prompt,
|
|
601
|
+
list(picked),
|
|
602
|
+
policies_cfg,
|
|
603
|
+
self.router._by_name,
|
|
604
|
+
con,
|
|
605
|
+
user_id,
|
|
606
|
+
max_active=MAX_ACTIVE_SKILLS,
|
|
607
|
+
)
|
|
608
|
+
router_mode = "full" if self.router.anthropic else "embedding-only"
|
|
609
|
+
explain = {
|
|
610
|
+
"schema_version": MCP_RESPONSE_SCHEMA_VERSION,
|
|
611
|
+
"tool": "explain_route",
|
|
612
|
+
"orchestrator_db": redact_display_path(db_path) if redaction_enabled() else str(db_path),
|
|
613
|
+
"router_mode": router_mode,
|
|
614
|
+
"embedding_shortlist": facets,
|
|
615
|
+
"picked_before_policy": list(picked),
|
|
616
|
+
"picked_after_policy": merged,
|
|
617
|
+
"router_reasoning": reasoning,
|
|
618
|
+
"policy": {
|
|
619
|
+
"rules_loaded": len(policies_cfg.get("rules") or [])
|
|
620
|
+
if isinstance(policies_cfg.get("rules"), list)
|
|
621
|
+
else 0,
|
|
622
|
+
"audit": policy_audit,
|
|
623
|
+
},
|
|
624
|
+
}
|
|
625
|
+
lines = [
|
|
626
|
+
"# explain_route — routing diagnostics (no DB writes)",
|
|
627
|
+
"",
|
|
628
|
+
f"**Router:** {router_mode}",
|
|
629
|
+
f"**Picked (router):** {', '.join(picked) if picked else '_(none)_'}",
|
|
630
|
+
f"**After policies:** {', '.join(merged) if merged else '_(none)_'}",
|
|
631
|
+
f"**Reasoning:** {reasoning}" if reasoning else "**Reasoning:** _(n/a)_",
|
|
632
|
+
"",
|
|
633
|
+
"## Shortlist (embedding)",
|
|
634
|
+
]
|
|
635
|
+
for f in facets[:15]:
|
|
636
|
+
lines.append(
|
|
637
|
+
f"- `{f['name']}` cos={f['cosine_similarity']} weight={f['learned_weight']} "
|
|
638
|
+
f"score={f['routing_score']}"
|
|
639
|
+
)
|
|
640
|
+
if policy_audit:
|
|
641
|
+
lines.extend(["", "## Policy audit"])
|
|
642
|
+
for row in policy_audit[:30]:
|
|
643
|
+
lines.append(f"- {row}")
|
|
644
|
+
body = "\n".join(lines)
|
|
645
|
+
return {"content": [{"type": "text", "text": body}], "_meta": explain}
|
|
646
|
+
|
|
647
|
+
def _tool_get_skill(self, args):
|
|
648
|
+
name = (args.get("skill_name") or "").strip()
|
|
649
|
+
fmt = (args.get("format") or "full").strip().lower()
|
|
650
|
+
if fmt not in ("full", "summary"):
|
|
651
|
+
fmt = "full"
|
|
652
|
+
max_chars = args.get("max_chars")
|
|
653
|
+
try:
|
|
654
|
+
mc = int(max_chars) if max_chars is not None else 0
|
|
655
|
+
except (TypeError, ValueError):
|
|
656
|
+
mc = 0
|
|
657
|
+
if not name or name not in self.skills:
|
|
658
|
+
return {
|
|
659
|
+
"content": [{"type": "text", "text": f"Unknown skill: {name or '(empty)'}"}],
|
|
660
|
+
"isError": True,
|
|
661
|
+
}
|
|
662
|
+
s = self.skills[name]
|
|
663
|
+
if fmt == "summary":
|
|
664
|
+
body = f"{s.description}\n\n---\n\n{(s.body or '')[:8000]}"
|
|
665
|
+
else:
|
|
666
|
+
body = s.body or ""
|
|
667
|
+
if mc > 0:
|
|
668
|
+
body = body[:mc]
|
|
669
|
+
header = f"# get_skill: `{name}`\n**Source:** {s.source} · **format:** {fmt}\n\n"
|
|
670
|
+
text = header + body
|
|
671
|
+
return {
|
|
672
|
+
"content": [{"type": "text", "text": text}],
|
|
673
|
+
"_meta": {
|
|
674
|
+
"schema_version": MCP_RESPONSE_SCHEMA_VERSION,
|
|
675
|
+
"tool": "get_skill",
|
|
676
|
+
"skill_name": name,
|
|
677
|
+
"source": s.source,
|
|
678
|
+
"format": fmt,
|
|
679
|
+
"chars": len(body),
|
|
680
|
+
},
|
|
681
|
+
}
|
|
682
|
+
|
|
461
683
|
def _tool_list_skills(self, args):
|
|
462
684
|
user_id = self._mcp_user_id(args)
|
|
463
685
|
con = self._get_con(args)
|
|
@@ -0,0 +1,133 @@
|
|
|
1
|
+
"""Pluggable route policies: regex on prompt → force-include skill names.
|
|
2
|
+
|
|
3
|
+
Load order (first file that exists / first successful parse wins for env):
|
|
4
|
+
|
|
5
|
+
1. ``SKILLFORGE_ROUTE_POLICIES`` — JSON object inline (e.g. ``{\"rules\":[...]}``).
|
|
6
|
+
2. ``SKILLFORGE_ROUTE_POLICIES_FILE`` — path to a JSON file.
|
|
7
|
+
3. ``<project_root>/.skillforge/policies.json``
|
|
8
|
+
4. ``<project_root>/skillforge-policies.json``
|
|
9
|
+
|
|
10
|
+
Rule shape::
|
|
11
|
+
|
|
12
|
+
{
|
|
13
|
+
"rules": [
|
|
14
|
+
{
|
|
15
|
+
"if_text_matches": "(?i)(auth|oauth|jwt|password)",
|
|
16
|
+
"include": ["security-review"]
|
|
17
|
+
}
|
|
18
|
+
]
|
|
19
|
+
}
|
|
20
|
+
|
|
21
|
+
``if_text_matches`` is passed to ``re.search`` (``re.DOTALL``). ``include`` is a skill
|
|
22
|
+
name or list of names. Forced skills are appended after router picks until
|
|
23
|
+
``MAX_ACTIVE_SKILLS`` is reached.
|
|
24
|
+
"""
|
|
25
|
+
from __future__ import annotations
|
|
26
|
+
|
|
27
|
+
import json
|
|
28
|
+
import os
|
|
29
|
+
import re
|
|
30
|
+
import sqlite3
|
|
31
|
+
from pathlib import Path
|
|
32
|
+
from typing import Any
|
|
33
|
+
|
|
34
|
+
|
|
35
|
+
def load_route_policies_config(project_root: str | None) -> dict[str, Any]:
|
|
36
|
+
"""Return a dict with key ``rules`` (list). Empty rules if nothing configured."""
|
|
37
|
+
raw_env = os.getenv("SKILLFORGE_ROUTE_POLICIES", "").strip()
|
|
38
|
+
if raw_env:
|
|
39
|
+
try:
|
|
40
|
+
data = json.loads(raw_env)
|
|
41
|
+
return data if isinstance(data, dict) else {"rules": []}
|
|
42
|
+
except json.JSONDecodeError:
|
|
43
|
+
return {"rules": []}
|
|
44
|
+
|
|
45
|
+
paths: list[Path] = []
|
|
46
|
+
path_env = os.getenv("SKILLFORGE_ROUTE_POLICIES_FILE", "").strip()
|
|
47
|
+
if path_env:
|
|
48
|
+
paths.append(Path(path_env).expanduser())
|
|
49
|
+
if project_root:
|
|
50
|
+
pr = Path(project_root).expanduser().resolve()
|
|
51
|
+
paths.append(pr / ".skillforge" / "policies.json")
|
|
52
|
+
paths.append(pr / "skillforge-policies.json")
|
|
53
|
+
|
|
54
|
+
for p in paths:
|
|
55
|
+
if p.is_file():
|
|
56
|
+
try:
|
|
57
|
+
data = json.loads(p.read_text(encoding="utf-8"))
|
|
58
|
+
return data if isinstance(data, dict) else {"rules": []}
|
|
59
|
+
except (OSError, json.JSONDecodeError):
|
|
60
|
+
continue
|
|
61
|
+
return {"rules": []}
|
|
62
|
+
|
|
63
|
+
|
|
64
|
+
def merge_policy_includes(
|
|
65
|
+
prompt: str,
|
|
66
|
+
picked_names: list[str],
|
|
67
|
+
policies: dict[str, Any],
|
|
68
|
+
by_name: dict[str, Any],
|
|
69
|
+
con: sqlite3.Connection,
|
|
70
|
+
user_id: str,
|
|
71
|
+
*,
|
|
72
|
+
max_active: int,
|
|
73
|
+
) -> tuple[list[str], list[dict[str, Any]]]:
|
|
74
|
+
"""Append policy-driven skills after ``picked_names`` without duplicates.
|
|
75
|
+
|
|
76
|
+
Returns (merged_pick_list, audit_rows for events / explain_route).
|
|
77
|
+
"""
|
|
78
|
+
# Local import avoids circular import at module load time.
|
|
79
|
+
from app.main import get_skill_weight
|
|
80
|
+
|
|
81
|
+
rules = policies.get("rules") if isinstance(policies, dict) else None
|
|
82
|
+
if not isinstance(rules, list):
|
|
83
|
+
rules = []
|
|
84
|
+
|
|
85
|
+
audit: list[dict[str, Any]] = []
|
|
86
|
+
merged = list(picked_names)
|
|
87
|
+
extras: list[str] = []
|
|
88
|
+
|
|
89
|
+
for rule in rules:
|
|
90
|
+
if not isinstance(rule, dict):
|
|
91
|
+
continue
|
|
92
|
+
pat = rule.get("if_text_matches") or rule.get("pattern") or ""
|
|
93
|
+
if not isinstance(pat, str) or not pat.strip():
|
|
94
|
+
continue
|
|
95
|
+
try:
|
|
96
|
+
matched = bool(re.search(pat, prompt, flags=re.DOTALL))
|
|
97
|
+
except re.error:
|
|
98
|
+
audit.append({"pattern": pat, "effect": "invalid_regex"})
|
|
99
|
+
continue
|
|
100
|
+
if not matched:
|
|
101
|
+
continue
|
|
102
|
+
|
|
103
|
+
inc = rule.get("include")
|
|
104
|
+
if isinstance(inc, str):
|
|
105
|
+
inc = [inc]
|
|
106
|
+
if not isinstance(inc, list):
|
|
107
|
+
continue
|
|
108
|
+
|
|
109
|
+
for name in inc:
|
|
110
|
+
if not isinstance(name, str) or not name.strip():
|
|
111
|
+
continue
|
|
112
|
+
name = name.strip()
|
|
113
|
+
if name not in by_name:
|
|
114
|
+
audit.append({"pattern": pat, "skill": name, "effect": "unknown_skill"})
|
|
115
|
+
continue
|
|
116
|
+
_w, disabled = get_skill_weight(con, name, user_id=user_id)
|
|
117
|
+
if disabled:
|
|
118
|
+
audit.append({"pattern": pat, "skill": name, "effect": "disabled"})
|
|
119
|
+
continue
|
|
120
|
+
if name in merged or name in extras:
|
|
121
|
+
audit.append({"pattern": pat, "skill": name, "effect": "already_in_list"})
|
|
122
|
+
continue
|
|
123
|
+
extras.append(name)
|
|
124
|
+
audit.append({"pattern": pat, "skill": name, "effect": "added"})
|
|
125
|
+
|
|
126
|
+
for n in extras:
|
|
127
|
+
if len(merged) >= max_active:
|
|
128
|
+
audit.append({"skill": n, "effect": "skipped_max_active", "max": max_active})
|
|
129
|
+
break
|
|
130
|
+
if n not in merged:
|
|
131
|
+
merged.append(n)
|
|
132
|
+
|
|
133
|
+
return merged, audit
|
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
"""Conversation-aware routing text, skill routing cards, and sparse retrieval signals."""
|
|
2
|
+
from __future__ import annotations
|
|
3
|
+
|
|
4
|
+
import os
|
|
5
|
+
import re
|
|
6
|
+
from typing import Any, Protocol
|
|
7
|
+
|
|
8
|
+
import numpy as np
|
|
9
|
+
|
|
10
|
+
_TOKEN_RE = re.compile(r"[a-z0-9][a-z0-9_\-./]{2,}", re.I)
|
|
11
|
+
|
|
12
|
+
|
|
13
|
+
class _SkillCard(Protocol):
|
|
14
|
+
title: str
|
|
15
|
+
description: str
|
|
16
|
+
triggers: str
|
|
17
|
+
anti_triggers: str
|
|
18
|
+
|
|
19
|
+
|
|
20
|
+
def build_route_query_text(
|
|
21
|
+
prompt: str,
|
|
22
|
+
conversation: list[Any] | None,
|
|
23
|
+
*,
|
|
24
|
+
max_turns: int | None = None,
|
|
25
|
+
max_chars_per_msg: int | None = None,
|
|
26
|
+
) -> str:
|
|
27
|
+
"""Merge recent turns with the current user message for embedding shortlist / hybrid scores.
|
|
28
|
+
|
|
29
|
+
When ``SKILLFORGE_ROUTER_CONV_MAX_TURNS`` is 0 (default), returns ``prompt`` only (legacy behavior).
|
|
30
|
+
"""
|
|
31
|
+
conv = conversation or []
|
|
32
|
+
mt = max_turns
|
|
33
|
+
if mt is None:
|
|
34
|
+
mt = int(os.getenv("SKILLFORGE_ROUTER_CONV_MAX_TURNS", "0"))
|
|
35
|
+
mc = max_chars_per_msg
|
|
36
|
+
if mc is None:
|
|
37
|
+
mc = int(os.getenv("SKILLFORGE_ROUTER_CONV_MSG_CHARS", "320"))
|
|
38
|
+
prompt = (prompt or "").strip()
|
|
39
|
+
if mt <= 0 or not conv:
|
|
40
|
+
return prompt
|
|
41
|
+
tail = conv[-mt:]
|
|
42
|
+
parts: list[str] = []
|
|
43
|
+
for m in tail:
|
|
44
|
+
if not isinstance(m, dict):
|
|
45
|
+
continue
|
|
46
|
+
role = str(m.get("role") or "user")
|
|
47
|
+
content = str(m.get("content") or "").strip()
|
|
48
|
+
if not content:
|
|
49
|
+
continue
|
|
50
|
+
if len(content) > mc:
|
|
51
|
+
content = content[:mc] + "…"
|
|
52
|
+
parts.append(f"{role}: {content}")
|
|
53
|
+
if not parts:
|
|
54
|
+
return prompt
|
|
55
|
+
return "Conversation context:\n" + "\n".join(parts) + "\n\nCurrent user message:\n" + prompt
|
|
56
|
+
|
|
57
|
+
|
|
58
|
+
def skill_routing_card(s: _SkillCard) -> str:
|
|
59
|
+
"""Text embedded for each skill + used in hybrid / router prompts."""
|
|
60
|
+
title = (s.title or "").strip()
|
|
61
|
+
desc = (s.description or "").strip()
|
|
62
|
+
tr = (getattr(s, "triggers", None) or "").strip()
|
|
63
|
+
anti = (getattr(s, "anti_triggers", None) or "").strip()
|
|
64
|
+
parts = [f"{title}: {desc}"]
|
|
65
|
+
if tr:
|
|
66
|
+
parts.append(f"Triggers: {tr}")
|
|
67
|
+
if anti:
|
|
68
|
+
parts.append(f"Anti-triggers: {anti}")
|
|
69
|
+
return "\n".join(parts)
|
|
70
|
+
|
|
71
|
+
|
|
72
|
+
def tokenize_skills_query(text: str) -> list[str]:
|
|
73
|
+
return [t.lower() for t in _TOKEN_RE.findall(text or "")]
|
|
74
|
+
|
|
75
|
+
|
|
76
|
+
def normalize_minmax(arr: np.ndarray) -> np.ndarray:
|
|
77
|
+
a = np.asarray(arr, dtype=np.float64).reshape(-1)
|
|
78
|
+
if a.size == 0:
|
|
79
|
+
return a
|
|
80
|
+
lo, hi = float(a.min()), float(a.max())
|
|
81
|
+
if hi <= lo:
|
|
82
|
+
return np.zeros_like(a)
|
|
83
|
+
return (a - lo) / (hi - lo)
|
|
84
|
+
|
|
85
|
+
|
|
86
|
+
def keyword_overlap_scores(route_query: str, skill_cards: list[str]) -> np.ndarray:
|
|
87
|
+
"""Per-skill overlap counts (unnormalized); combine with dense via hybrid alpha."""
|
|
88
|
+
qt = set(tokenize_skills_query(route_query))
|
|
89
|
+
if not qt:
|
|
90
|
+
return np.zeros(len(skill_cards), dtype=np.float64)
|
|
91
|
+
out: list[float] = []
|
|
92
|
+
for card in skill_cards:
|
|
93
|
+
ct = set(tokenize_skills_query(card))
|
|
94
|
+
out.append(float(len(qt & ct)))
|
|
95
|
+
return np.array(out, dtype=np.float64)
|
package/python/requirements.txt
CHANGED
|
@@ -0,0 +1,115 @@
|
|
|
1
|
+
"""Tests for route policy loading and merge."""
|
|
2
|
+
from __future__ import annotations
|
|
3
|
+
|
|
4
|
+
import pytest
|
|
5
|
+
|
|
6
|
+
from app.main import Skill, init_db
|
|
7
|
+
from app.route_policies import load_route_policies_config, merge_policy_includes
|
|
8
|
+
|
|
9
|
+
|
|
10
|
+
@pytest.fixture
|
|
11
|
+
def skill_alpha() -> Skill:
|
|
12
|
+
return Skill(
|
|
13
|
+
name="alpha-skill",
|
|
14
|
+
title="Alpha",
|
|
15
|
+
description="test",
|
|
16
|
+
body="body",
|
|
17
|
+
source="bundled",
|
|
18
|
+
)
|
|
19
|
+
|
|
20
|
+
|
|
21
|
+
def test_merge_adds_on_regex_match(tmp_path, skill_alpha, monkeypatch) -> None:
|
|
22
|
+
monkeypatch.delenv("SKILLFORGE_ROUTE_POLICIES", raising=False)
|
|
23
|
+
monkeypatch.delenv("SKILLFORGE_ROUTE_POLICIES_FILE", raising=False)
|
|
24
|
+
con = init_db(tmp_path / "x.db")
|
|
25
|
+
policies = {"rules": [{"if_text_matches": r"(?i)oauth", "include": ["alpha-skill"]}]}
|
|
26
|
+
by_name = {skill_alpha.name: skill_alpha}
|
|
27
|
+
merged, audit = merge_policy_includes(
|
|
28
|
+
"Fix OAuth callback",
|
|
29
|
+
["other-skill"],
|
|
30
|
+
policies,
|
|
31
|
+
by_name,
|
|
32
|
+
con,
|
|
33
|
+
"",
|
|
34
|
+
max_active=7,
|
|
35
|
+
)
|
|
36
|
+
assert merged[0] == "other-skill"
|
|
37
|
+
assert "alpha-skill" in merged
|
|
38
|
+
assert any(r.get("effect") == "added" for r in audit)
|
|
39
|
+
|
|
40
|
+
|
|
41
|
+
def test_merge_unknown_skill_audited(tmp_path, skill_alpha, monkeypatch) -> None:
|
|
42
|
+
monkeypatch.delenv("SKILLFORGE_ROUTE_POLICIES", raising=False)
|
|
43
|
+
con = init_db(tmp_path / "y.db")
|
|
44
|
+
policies = {"rules": [{"if_text_matches": "auth", "include": ["missing"]}]}
|
|
45
|
+
by_name = {skill_alpha.name: skill_alpha}
|
|
46
|
+
merged, audit = merge_policy_includes(
|
|
47
|
+
"auth bug",
|
|
48
|
+
[],
|
|
49
|
+
policies,
|
|
50
|
+
by_name,
|
|
51
|
+
con,
|
|
52
|
+
"",
|
|
53
|
+
max_active=7,
|
|
54
|
+
)
|
|
55
|
+
assert merged == []
|
|
56
|
+
assert any(r.get("effect") == "unknown_skill" for r in audit)
|
|
57
|
+
|
|
58
|
+
|
|
59
|
+
def test_merge_respects_max_active(tmp_path, skill_alpha, monkeypatch) -> None:
|
|
60
|
+
monkeypatch.delenv("SKILLFORGE_ROUTE_POLICIES", raising=False)
|
|
61
|
+
con = init_db(tmp_path / "z.db")
|
|
62
|
+
policies = {"rules": [{"if_text_matches": "x", "include": ["alpha-skill"]}]}
|
|
63
|
+
by_name = {skill_alpha.name: skill_alpha}
|
|
64
|
+
picked = ["a", "b", "c", "d", "e", "f", "g"]
|
|
65
|
+
merged, audit = merge_policy_includes(
|
|
66
|
+
"x",
|
|
67
|
+
picked,
|
|
68
|
+
policies,
|
|
69
|
+
by_name,
|
|
70
|
+
con,
|
|
71
|
+
"",
|
|
72
|
+
max_active=7,
|
|
73
|
+
)
|
|
74
|
+
assert len(merged) == 7
|
|
75
|
+
assert "alpha-skill" not in merged
|
|
76
|
+
assert any(r.get("effect") == "skipped_max_active" for r in audit)
|
|
77
|
+
|
|
78
|
+
|
|
79
|
+
def test_load_from_project_file(tmp_path, monkeypatch) -> None:
|
|
80
|
+
monkeypatch.delenv("SKILLFORGE_ROUTE_POLICIES", raising=False)
|
|
81
|
+
monkeypatch.delenv("SKILLFORGE_ROUTE_POLICIES_FILE", raising=False)
|
|
82
|
+
p = tmp_path / "skillforge-policies.json"
|
|
83
|
+
p.write_text(
|
|
84
|
+
'{"rules": [{"if_text_matches": "hi", "include": ["z"]}]}',
|
|
85
|
+
encoding="utf-8",
|
|
86
|
+
)
|
|
87
|
+
root = str(tmp_path)
|
|
88
|
+
cfg = load_route_policies_config(root)
|
|
89
|
+
assert len(cfg.get("rules") or []) == 1
|
|
90
|
+
|
|
91
|
+
|
|
92
|
+
def test_load_inline_env_json(monkeypatch) -> None:
|
|
93
|
+
monkeypatch.setenv(
|
|
94
|
+
"SKILLFORGE_ROUTE_POLICIES",
|
|
95
|
+
'{"rules": [{"if_text_matches": "a", "include": ["b"]}]}',
|
|
96
|
+
)
|
|
97
|
+
cfg = load_route_policies_config(None)
|
|
98
|
+
assert cfg["rules"][0]["include"] == ["b"]
|
|
99
|
+
|
|
100
|
+
|
|
101
|
+
def test_invalid_regex_recorded(tmp_path, skill_alpha, monkeypatch) -> None:
|
|
102
|
+
monkeypatch.delenv("SKILLFORGE_ROUTE_POLICIES", raising=False)
|
|
103
|
+
con = init_db(tmp_path / "r.db")
|
|
104
|
+
policies = {"rules": [{"if_text_matches": "(bad[regex", "include": ["alpha-skill"]}]}
|
|
105
|
+
by_name = {skill_alpha.name: skill_alpha}
|
|
106
|
+
_m, audit = merge_policy_includes(
|
|
107
|
+
"x",
|
|
108
|
+
[],
|
|
109
|
+
policies,
|
|
110
|
+
by_name,
|
|
111
|
+
con,
|
|
112
|
+
"",
|
|
113
|
+
max_active=7,
|
|
114
|
+
)
|
|
115
|
+
assert any(r.get("effect") == "invalid_regex" for r in audit)
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
"""Tests for conversation-aware route text, skill cards, and hybrid helpers."""
|
|
2
|
+
from __future__ import annotations
|
|
3
|
+
|
|
4
|
+
import numpy as np
|
|
5
|
+
import pytest
|
|
6
|
+
|
|
7
|
+
from app.main import Skill, parse_skill_md
|
|
8
|
+
from app.routing_signals import (
|
|
9
|
+
build_route_query_text,
|
|
10
|
+
keyword_overlap_scores,
|
|
11
|
+
normalize_minmax,
|
|
12
|
+
skill_routing_card,
|
|
13
|
+
)
|
|
14
|
+
|
|
15
|
+
|
|
16
|
+
def test_build_route_query_legacy(monkeypatch) -> None:
|
|
17
|
+
monkeypatch.setenv("SKILLFORGE_ROUTER_CONV_MAX_TURNS", "0")
|
|
18
|
+
out = build_route_query_text("hello", [{"role": "user", "content": "prev"}])
|
|
19
|
+
assert out == "hello"
|
|
20
|
+
|
|
21
|
+
|
|
22
|
+
def test_build_route_query_merges_turns(monkeypatch) -> None:
|
|
23
|
+
monkeypatch.setenv("SKILLFORGE_ROUTER_CONV_MAX_TURNS", "2")
|
|
24
|
+
monkeypatch.setenv("SKILLFORGE_ROUTER_CONV_MSG_CHARS", "80")
|
|
25
|
+
conv = [
|
|
26
|
+
{"role": "user", "content": "first msg"},
|
|
27
|
+
{"role": "assistant", "content": "reply"},
|
|
28
|
+
]
|
|
29
|
+
out = build_route_query_text("current ask", conv)
|
|
30
|
+
assert "user: first msg" in out
|
|
31
|
+
assert "assistant: reply" in out
|
|
32
|
+
assert "Current user message:" in out
|
|
33
|
+
assert out.endswith("current ask")
|
|
34
|
+
|
|
35
|
+
|
|
36
|
+
def test_skill_routing_card_includes_triggers() -> None:
|
|
37
|
+
s = Skill(
|
|
38
|
+
name="x",
|
|
39
|
+
title="X Skill",
|
|
40
|
+
description="does things",
|
|
41
|
+
body="",
|
|
42
|
+
source="bundled",
|
|
43
|
+
triggers="when foo",
|
|
44
|
+
anti_triggers="not bar",
|
|
45
|
+
)
|
|
46
|
+
card = skill_routing_card(s)
|
|
47
|
+
assert "X Skill" in card
|
|
48
|
+
assert "Triggers: when foo" in card
|
|
49
|
+
assert "Anti-triggers: not bar" in card
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
def test_normalize_minmax() -> None:
|
|
53
|
+
a = np.array([1.0, 3.0, 5.0])
|
|
54
|
+
assert np.allclose(normalize_minmax(a), [0.0, 0.5, 1.0])
|
|
55
|
+
flat = np.array([2.0, 2.0, 2.0])
|
|
56
|
+
assert np.allclose(normalize_minmax(flat), [0.0, 0.0, 0.0])
|
|
57
|
+
|
|
58
|
+
|
|
59
|
+
def test_keyword_overlap_scores() -> None:
|
|
60
|
+
cards = ["alpha beta gamma", "foo bar"]
|
|
61
|
+
q = "beta search"
|
|
62
|
+
sc = keyword_overlap_scores(q, cards)
|
|
63
|
+
assert sc[0] > sc[1]
|
|
64
|
+
|
|
65
|
+
|
|
66
|
+
def test_parse_skill_triggers(tmp_path) -> None:
|
|
67
|
+
md = tmp_path / "my-skill" / "SKILL.md"
|
|
68
|
+
md.parent.mkdir(parents=True, exist_ok=True)
|
|
69
|
+
md.write_text(
|
|
70
|
+
"---\nname: Nice\ndescription: Desc\ntriggers: when testing\n"
|
|
71
|
+
"anti_triggers: never for prod\n---\n\n# Body\n",
|
|
72
|
+
encoding="utf-8",
|
|
73
|
+
)
|
|
74
|
+
s = parse_skill_md(md, "bundled")
|
|
75
|
+
assert s is not None
|
|
76
|
+
assert s.triggers == "when testing"
|
|
77
|
+
assert s.anti_triggers == "never for prod"
|