@heytherevibin/skillforge 0.2.1 → 0.8.0

package/CHANGELOG.md

@@ -1,5 +1,48 @@
 # Changelog
 
+## 0.8.0
+
+- **Smarter routing:** optional **conversation-aware** shortlist query (`SKILLFORGE_ROUTER_CONV_*`), **hybrid** retrieval (`SKILLFORGE_ROUTER_HYBRID` = `keyword` or `bm25` + `SKILLFORGE_ROUTER_HYBRID_ALPHA`), optional **Haiku rerank** (`SKILLFORGE_HAIKU_RERANK`, `SKILLFORGE_HAIKU_RERANK_MAX`, `SKILLFORGE_HAIKU_RERANK_MODEL`).
+- **Skill cards:** YAML **`triggers`** / **`anti_triggers`** on **`SKILL.md`** are parsed and folded into summary embeddings and router prompts via **`app/routing_signals.py`** (`skill_routing_card`). Chunk RAG still scores on the **current** user message.
+- **Dependency:** **`rank-bm25`** in `python/requirements.txt` (BM25 hybrid; optional at runtime if you use `keyword` only).
+
+## 0.7.1
+
+- **MCP:** **`search_skills`** (embedding shortlist + snippets), **`explain_route`** (routing diagnostics, no DB writes), **`get_skill`** (fetch one **`SKILL.md`** by name).
+- **Route policies:** optional **`SKILLFORGE_ROUTE_POLICIES`**, **`SKILLFORGE_ROUTE_POLICIES_FILE`**, or **`project_root`/** `.skillforge/policies.json` / **`skillforge-policies.json`** — regex rules append **`include`** skills after the router (capped by **`SKILLFORGE_MAX_ACTIVE`**). Audit stored on route events under **`policy`**.
+
+## 0.7.0
+
+- **Breaking:** Removed the optional **HTTP API** (`skillforge start`), **`skillforge chat`** harness, and **`skillforge auth`** (bearer tokens were only used by HTTP). MCP (`skillforge mcp`), **`skillforge route`**, **`skillforge events`**, and **`skillforge index`** are unchanged.
+- **Migration:** The CLI deletes a leftover **`~/.skillforge/auth.json`** on **every** invocation (including **`skillforge --help`**), once the file is gone the message stops.
+- Dropped **FastAPI**, **uvicorn**, and direct **pydantic** / **httpx** dependencies from `python/requirements.txt` (routing still uses libraries that may bundle their own deps).
+
+## 0.6.0
+
+- **Phase 4 context safety (MCP meta 1.4)**: Default **secret / credential pattern redaction** and optional **home-directory stripping** on injected chunk text, relative **`path`** fields, stored route **`prompt`** snippet, **`reasoning`**, and **`orchestrator_db`** in **`_meta`**. Disable with **`SKILLFORGE_REDACT_CONTEXT=0`**; path scrub with **`SKILLFORGE_REDACT_HOME_IN_PATHS=0`**. New **[`app/redaction.py`](python/app/redaction.py)**; route events include **`context_redaction`** hit counts.
+
+## 0.5.0
+
+- **Phase 3 context fusion (MCP meta 1.3)**: When **`include_project_rag`** is on and the project index is non-empty, skill + project chunk **pools** are merged with **greedy MMR** under a single **`SKILLFORGE_CONTEXT_BUDGET_CHARS`** (default: route max + project RAG max). Disable with **`SKILLFORGE_CONTEXT_FUSION=0`** to keep append-only behavior.
+- **Telemetry**: route events and **`_meta.fusion`** include MMR trace; each context item may carry **`mmr_rank`**, **`mmr_score`**, **`retrieval_relevance`**, **`max_sim_to_prior`**.
+- New **[`app/context_fusion.py`](python/app/context_fusion.py)**; **`load_project_fusion_pool`** in **`project_index`**.
+
+## 0.4.0
+
+- **Phase 2 project RAG (MCP 1.2)**: **`skillforge index --project-root=…`** walks the repo (bounded file sizes, ignore dirs), chunks text files, and stores **`project_chunks`** + embeddings in **`<project>/.skillforge/orchestrator.db`** (same DB as sessions/weights).
+- **MCP / CLI / HTTP**: optional **`include_project_rag`** (MCP + **`skillforge route --include-project-rag`**) appends top matching file chunks under **`SKILLFORGE_PROJECT_RAG_MAX_CHARS`**. **`_meta`**: schema **1.2**; **`sources`** may include **`kind: file`**; **`budget.chars_project_chunks`** / **`chars_context_items_total`**.
+
+## 0.3.0
+
+- **Phase 1 skill RAG (MCP 1.1)**: Default **`SKILLFORGE_CONTEXT_MODE=chunks`** — line-bounded chunks from each picked **`SKILL.md`** body, scored by query similarity, injected up to **`SKILLFORGE_ROUTE_MAX_CHARS`**. Set **`full_body`** for legacy whole-document injection per skill.
+- **`_meta`**: schema **1.1**; **`sources`** lists chunk-level rows with **`line_start` / `line_end`** and **`score`**; **`context_items_count`**.
+- New **[`app/chunking.py`](python/app/chunking.py)**; **`Router.build_context_items`**, **`format_context_items_markdown`**.
+
+## 0.2.2
+
+- **MCP Phase 0 contract**: **`route_skills`** success responses include versioned **`_meta`** (`schema_version` **1.0**, **`sources`**, **`budget`**, **`candidates_preview`**). Empty prompt returns **`isError`** + **`_meta.error`**. Shared builder in **`app.mcp_contract`**; **`skillforge route --json-meta`** matches.
+- **Docs**: README “MCP response contract” section.
+
 ## 0.2.1
 
 - Same code as **0.2.0**. **npm never allows reusing a version** after it has been published once—even if you **unpublish** it and only **0.1.0** remains visible. The registry still blocks **`0.2.0`**; ship **`0.2.1`** (or higher) instead.

package/README.md

@@ -10,7 +10,7 @@
 
 **Primary interface:** **stdio MCP** (`skillforge mcp`) — add it to Claude Desktop, Cursor, or Claude Code.
 
-**Optional:** **headless HTTP API** (`skillforge start`) for `/chat`, `/events`, and integrations. **Real-time usage:** run **`skillforge events --watch`** in a terminal (top skills, active sessions, and live **route** / **feedback** lines from SQLite).
+**Observability:** run **`skillforge events --watch`** in a terminal (top skills, active sessions, and live **route** / **feedback** lines from SQLite).
 
 ---
 
@@ -23,11 +23,10 @@
 - [Usage](#usage)
   - [Run modes](#run-modes)
   - [Model Context Protocol (MCP)](#model-context-protocol-mcp)
-  - [Multi-user authentication](#multi-user-authentication)
+    - [MCP response contract](#mcp-response-contract)
 - [Skills and packs](#skills-and-packs)
 - [Routing pipeline](#routing-pipeline)
 - [Configuration](#configuration)
-- [HTTP API](#http-api)
 - [Local data and operations](#local-data-and-operations)
 - [Security considerations](#security-considerations)
 - [Contributing and governance](#contributing-and-governance)
@@ -47,7 +46,7 @@
 | **Observability** | **`skillforge events`**: snapshots of **usage** + **active sessions**, **`--watch`** for realtime; **`--verbose`** for route detail. No browser UI. |
 | **Project bootstrap** | MCP tools **`materialize_project`** and **`skillforge_bootstrap`** write `.cursor/rules`, **`docs/SKILLFORGE-PRD.md`**, and a **`CLAUDE.md`** section (map **`/skillforge`** in rules to MCP tools). |
 | **Extensibility** | Custom skills, git-based **packs**, and overrides under a single user config directory. |
-| **Deployment flexibility** | **MCP stdio** (default story), optional **HTTP API**, dev **`skillforge chat`** harness. |
+| **Deployment flexibility** | **MCP stdio** as the default integration; **CLI** helpers (`route`, `events`, `index`). |
 
 Bundled content includes **200+** curated skills (coding, security, research, frontend/backend patterns, and more). Exact counts are validated in CI.
 
@@ -58,8 +57,8 @@ Bundled content includes **200+** curated skills (coding, security, research, fr
 | Dependency | Version | Notes |
 |------------|---------|--------|
 | **Node.js** | **>= 18** | Required for the CLI bootstrapper. Continuous integration runs on **Node 22**. |
-| **Python** | **>= 3.10** | Used on the host PATH for embeddings and the FastAPI orchestrator. |
-| **Anthropic API** | — | **`ANTHROPIC_API_KEY`** is required for **HTTP**, **CLI chat**, and the **full** (Haiku) router. **MCP** can run **without** it when using **embedding-only** routing (default when the key is omitted; see [MCP](#model-context-protocol-mcp)). |
+| **Python** | **>= 3.10** | Used on the host PATH for embeddings and routing (via the CLI-spawned **venv**). |
+| **Anthropic API** | — | **`ANTHROPIC_API_KEY`** enables the **full** (Haiku) router when you want it. **MCP** can run **without** it using **embedding-only** routing (default when the key is omitted; see [MCP](#model-context-protocol-mcp)). |
 
 **First run:** The CLI creates **`~/.skillforge/`**, a dedicated **Python venv**, installs Python dependencies, and caches the default embedding model (typically on the order of one to two minutes once; subsequent starts are fast).
 
@@ -73,12 +72,6 @@ npx --yes @heytherevibin/skillforge --help
 
 Add Skillforge to your MCP config (see [MCP](#model-context-protocol-mcp)). No `ANTHROPIC_API_KEY` is required for **embedding-only** routing.
 
-Optional HTTP API (e.g. for `skillforge chat`): set **`ANTHROPIC_API_KEY`**, then:
-
-```bash
-skillforge start
-```
-
 Live log (usage + routes): **`skillforge events --watch`**.
 
 ---
@@ -111,11 +104,9 @@ Source and issues: [github.com/heytherevibin/skillforge](https://github.com/heyt
 |---------|---------|
 | `skillforge --help` | Recommended first step; **MCP** is the main integration. |
 | `skillforge mcp` | **stdio** MCP server (Claude, Cursor, …). |
-| `skillforge start [--port=8000]` | Optional **HTTP API** (no HTML or WebSocket UI). |
 | `skillforge events [--watch]` | **Terminal** log: usage snapshot + routes; see **`skillforge events --help`**. |
 | `skillforge route […]` | **Terminal** routing — same pipeline as MCP **`route_skills`** (loads embed model); see **`skillforge route --help`**. |
 | `skillforge mcp config [--local] [--with-anthropic]` | **stdout**: JSON snippet for **`mcp.json`** (merge manually). |
-| `skillforge chat` | Dev harness: HTTP client to **`POST /chat`** (needs **`start`** + API key). |
 
 ### Model Context Protocol (MCP)
 
@@ -163,29 +154,34 @@ With **Haiku** routing (uses your Anthropic key in the MCP process):
 
 | Tool | Purpose |
 |------|---------|
-| `route_skills` | Returns routed **`SKILL.md`** bodies. Pass **`project_root`** (workspace path) for per-repo SQLite under **`.skillforge/orchestrator.db`** and learning; or set env **`SKILLFORGE_PROJECT_ROOT`**. Optional **`session_id`**, **`user_id`** / **`SKILLFORGE_MCP_USER_ID`**. |
+| `route_skills` | Returns routed **`SKILL.md`** context (chunks or full body). Pass **`project_root`** for per-repo SQLite under **`.skillforge/orchestrator.db`**. Optional **`include_project_rag`** (after **`skillforge index --project-root=…`**), **`session_id`**, **`user_id`** / **`SKILLFORGE_MCP_USER_ID`**, or env **`SKILLFORGE_PROJECT_ROOT`**. Route **`event.policy`** in SQLite logs policy merge audit when rules apply. |
+| `search_skills` | Embedding-only shortlist for a **`query`** (scores + description snippets); does not run Haiku or mutate sessions. Optional **`limit`** (max 50). |
+| `explain_route` | Same routing signal as **`route_skills`** without writing SQLite (**`picked_before_policy`**, **`picked_after_policy`**, shortlist facets, policy audit). For debugging. |
+| `get_skill` | Fetch one catalog skill by **`skill_name`**; **`format`**: **`full`** or **`summary`**; optional **`max_chars`**. |
 | `list_skills` | Catalog overview; optional **`user_id`** scopes usage stats. |
-| `skill_feedback` | Feedback for the learning loop; optional **`user_id`**, **`session_id`** (for `/events`). |
+| `skill_feedback` | Feedback for the learning loop; optional **`user_id`**, **`session_id`** (stored with events). |
 | `skill_referenced` | Mark a routed skill as **used** in the reply (increments **`referenced`** + weight; optional **`user_id`**). |
 | `disable_skill` | Toggle skills; optional **`user_id`**. |
 | `materialize_project` | Writes **`.cursor/rules/skillforge.mdc`**, **`docs/SKILLFORGE-PRD.md`**, updates **`CLAUDE.md`** (Skillforge block). Args: **`project_root`**, **`skill_names`** from **`route_skills`**. |
 | `skillforge_bootstrap` | **`route_skills`** + **`materialize_project`** in one call (needs **`project_root`**). |
 
-`/skillforge` is not registered by npm installs; add a **Cursor rule** or **CLAUDE.md** instruction so the agent calls these tools when the user asks.
+### MCP response contract
 
-Route events go to **`~/.skillforge/data/orchestrator.db`**; use **`skillforge events`** or **`GET /events`** when HTTP is running.
+Structured diagnostics for tool **`route_skills`** live in **`result._meta`** (hosts may ignore or log them):
 
-### Multi-user authentication
+- **`schema_version`**: **`1.4`** — same as **1.3** plus optional **`context_redaction`** (`enabled`, `secret_hits`, `path_hits`) on **`route_skills`** success.
+- **`sources`**: citations; each item has **`kind`** (`skill` or **`file`**), **`ref`**, **`line_start`** / **`line_end`**, **`score`**, optional **`mmr_rank`** after fusion.
+- **`fusion`**: present when MMR fusion ran (**`enabled`: true**): **`lambda`**, **`budget_chars`**, **`pool_skill`**, **`pool_project`**, **`mmr_trace`**, …
+- **`context_redaction`**: optional; when scrubbing is enabled, reports **`enabled`**, **`secret_hits`**, **`path_hits`** for exported context.
+- **`budget`**: **`chars_skill_bodies`**, **`chars_project_chunks`**, **`chars_context_items_total`**, **`chars_response_total`**, **`est_tokens_approx`** (rough `chars/4`).
+- **`candidates_preview`**: up to 15 shortlist entries **`{ name, score }`** for debugging.
+- **`picked`**, **`reasoning`**, **`session_id`**, **`user_id`**, **`rerouted`**, **`change_pct`**, **`route_ms`**, **`orchestrator_db`**.
 
-Bearer tokens isolate sessions, weights, and events when enabled:
+On validation errors (e.g. empty **`prompt`**), the tool returns **`isError`: true** and **`_meta.error`** (e.g. **`empty_prompt`**), still with **`schema_version`** and **`sources`: `[]`**.
 
-```bash
-skillforge auth add <user-id>
-skillforge auth list
-skillforge auth remove <user-id>
-```
+`/skillforge` is not registered by npm installs; add a **Cursor rule** or **CLAUDE.md** instruction so the agent calls these tools when the user asks.
 
-When any token exists, protected **HTTP API** routes require **`Authorization: Bearer <token>`**. Single-user mode applies when no tokens are configured.
+Route events go to **`~/.skillforge/data/orchestrator.db`** (or per-repo **`.skillforge/orchestrator.db`** when **`project_root`** is set); use **`skillforge events`** to inspect them.
 
 ---
 
@@ -203,10 +199,14 @@ skillforge skills list
 ---
 name: my-skill
 description: Clear trigger conditions—used by the router.
+triggers: When the user asks about X or mentions Y.
+anti_triggers: Not for production deploy checks.
 ---
 # My Skill
 ```
 
+Optional **`triggers`** / **`anti_triggers`** strings are embedded with the summary card and shown to the Haiku router (they do not change chunk RAG, which still keys off the current user message).
+
 Register with `skillforge skills add ./my-skill` or copy the folder to **`~/.skillforge/skills/`**.
 
 **Skill packs** are git repositories with a root **`skillforge.json`** manifest listing skill folder names. Install:
@@ -224,16 +224,38 @@ skillforge pack remove <name>
 ## Routing pipeline
 
 ```
-User prompt
-    → Local embeddings (sentence-transformers)
-    → Cosine similarity + per-user weights
+User prompt (+ optional recent conversation for the shortlist query)
+    → Local embeddings (sentence-transformers) on skill **cards** (title, description, optional triggers)
+    → Cosine similarity ± hybrid keyword/BM25 fusion + per-user weights
     → Top-K candidates
+    → Optional Haiku **rerank** on the shortlist (`SKILLFORGE_HAIKU_RERANK`)
     → Router model (Haiku) selects final active skills — *or* embedding-only mode takes top-N from candidates
     → Skill bodies injected; response model answers (e.g. Opus)
     → Usage signals update weights (optional)
 ```
 
-Re-route: when overlap between successive active sets falls below a configurable threshold, the pipeline selects a new set for the next turn. Events are stored in SQLite; stream them with **`skillforge events --watch`** (or **`GET /events`** when HTTP is running).
+Re-route: when overlap between successive active sets falls below a configurable threshold, the pipeline selects a new set for the next turn. Events are stored in SQLite; stream them with **`skillforge events --watch`**.
+
+---
+
+## Route policies (optional)
+
+Rules use **`if_text_matches`** as a Python **`re.search`** pattern (with **`re.DOTALL`**) on the user **`prompt`**. **`include`** is a skill name or list of names. Matched skills are **appended** after Haiku/embedding picks until **`SKILLFORGE_MAX_ACTIVE`**.
+
+**Load order:** env **`SKILLFORGE_ROUTE_POLICIES`** (inline JSON) → **`SKILLFORGE_ROUTE_POLICIES_FILE`** → **`<project_root>/.skillforge/policies.json`** → **`<project_root>/skillforge-policies.json`**.
+
+Example **`.skillforge/policies.json`**:
+
+```json
+{
+  "rules": [
+    {
+      "if_text_matches": "(?i)(auth|oauth|jwt|password|login)",
+      "include": ["security-review"]
+    }
+  ]
+}
+```
 
 ---
 
@@ -243,36 +265,46 @@ Environment variables (see also inline help and server defaults):
 
 | Variable | Default | Role |
 |----------|---------|------|
-| `ANTHROPIC_API_KEY` | — | **Required** for HTTP/CLI chat and answer streaming; **optional** for MCP if you use embedding-only routing (default when unset). |
-| `SKILLFORGE_ROUTER_MODE` | *(auto)* | `full` = always use Haiku for final pick (MCP: requires key for routing). `embedding` = skip Haiku; top `SKILLFORGE_MAX_ACTIVE` from shortlist. Unset = **auto**: MCP uses embedding-only when `ANTHROPIC_API_KEY` is absent, else full. HTTP: unset or `full` uses Haiku when key is present; set `embedding` to skip Haiku on the server (answer model still needs a key). |
-| `SKILLFORGE_PORT` | `8000` | HTTP listen port. |
+| `ANTHROPIC_API_KEY` | — | **Optional** for MCP: omit for embedding-only routing (default when unset); set for **full** (Haiku) routing. |
+| `SKILLFORGE_ROUTER_MODE` | *(auto)* | `full` = always use Haiku for final pick (requires key). `embedding` = skip Haiku; top `SKILLFORGE_MAX_ACTIVE` from shortlist. Unset = **auto**: embedding-only when `ANTHROPIC_API_KEY` is absent, else full. |
 | `SKILLFORGE_EMBED_MODEL` | `all-MiniLM-L6-v2` | Embedding model id. |
-| `SKILLFORGE_ROUTER_MODEL` | `claude-haiku-4-5-20251001` | Routing model. |
-| `SKILLFORGE_ANSWER_MODEL` | `claude-opus-4-7` | Main response model. |
+| `SKILLFORGE_ROUTER_MODEL` | `claude-haiku-4-5-20251001` | Routing model (Haiku). |
 | `SKILLFORGE_TOP_K` | `15` | Embedding shortlist size. |
 | `SKILLFORGE_MAX_ACTIVE` | `7` | Maximum skills injected per turn. |
 | `SKILLFORGE_REROUTE_THRESHOLD` | `0.4` | Re-route sensitivity (Jaccard distance). |
-| `SKILLFORGE_MCP_USER_ID` | `""` | Default logical **user id** for MCP tool calls when arguments omit `user_id` (weights, sessions, events—same SQLite namespace as HTTP `resolve_user`). |
+| `SKILLFORGE_ROUTER_CONV_MAX_TURNS` | `0` | Include this many recent **conversation** messages in the **embedding shortlist** query (`0` = current user message only, legacy). |
+| `SKILLFORGE_ROUTER_CONV_MSG_CHARS` | `320` | Max characters per message when building the shortlist query. |
+| `SKILLFORGE_ROUTER_HYBRID` | `off` | `off` = dense cosine only. `keyword` = fuse with token overlap on skill cards. `bm25` = fuse with **BM25** (requires **`rank-bm25`**; falls back to keyword if missing). |
+| `SKILLFORGE_ROUTER_HYBRID_ALPHA` | `0.72` | Hybrid weight on **dense** similarity (`1` = dense only; `0` = sparse only). |
+| `SKILLFORGE_ROUTER_PROMPT_HISTORY_MSGS` | `8` | Max conversation turns sent to the **Haiku** router and reranker. |
+| `SKILLFORGE_ROUTER_PROMPT_HISTORY_CHARS` | `360` | Max characters per turn in router / rerank prompts. |
+| `SKILLFORGE_ROUTER_CATALOG_PREVIEW_CHARS` | `280` | Max characters of each skill **routing card** in the Haiku pick prompt. |
+| `SKILLFORGE_HAIKU_RERANK` | `0` | Set **`1`** / **`true`** to rerank the Top-K shortlist with Haiku before the final pick (extra API call). |
+| `SKILLFORGE_HAIKU_RERANK_MAX` | `SKILLFORGE_TOP_K` | Max candidates passed to the reranker. |
+| `SKILLFORGE_HAIKU_RERANK_MODEL` | *(same as router)* | Model id for reranking when set; otherwise **`SKILLFORGE_ROUTER_MODEL`**. |
+| `SKILLFORGE_CONTEXT_MODE` | `chunks` | `chunks` = embed **line-bounded chunks** from each picked skill body (RAG) up to **`SKILLFORGE_ROUTE_MAX_CHARS`**. `full_body` = inject entire **SKILL.md** per pick (legacy). |
+| `SKILLFORGE_CHUNK_MAX_CHARS` | `1200` | Max characters per chunk (before overlap split). |
+| `SKILLFORGE_CHUNK_OVERLAP` | `200` | Character overlap when hard-splitting an oversized section. |
+| `SKILLFORGE_ROUTE_MAX_CHARS` | `60000` | Skill chunk char cap when **`SKILLFORGE_CONTEXT_FUSION`** is off (append path); also part of default unified budget sum when fusion is on. |
+| `SKILLFORGE_PROJECT_RAG_MAX_CHARS` | `24000` | Project chunk char cap when fusion is off (append path); part of default unified budget when fusion is on. |
+| `SKILLFORGE_CONTEXT_BUDGET_CHARS` | *(route + project RAG defaults)* | Single cap for **MMR-fused** skill + project context. |
+| `SKILLFORGE_CONTEXT_FUSION` | `1` | **`0`** / **`false`**: disable MMR fusion; append project chunks after skills. |
+| `SKILLFORGE_CONTEXT_MMR_LAMBDA` | `0.7` | MMR tradeoff: higher ⇒ query relevance; lower ⇒ diversity vs. already-selected chunks. |
+| `SKILLFORGE_FUSION_POOL_SKILL` | `96` | Max skill chunks in the fusion candidate pool. |
+| `SKILLFORGE_FUSION_POOL_PROJECT` | `96` | Max project chunks in the fusion candidate pool. |
+| `SKILLFORGE_FUSION_FULL_BODY_PREVIEW_CHARS` | `4000` | **`SKILL.md`** prefix length for embedding full-body / fallback fusion rows. |
+| `SKILLFORGE_PROJECT_RAG_MAX_CHUNKS` | `20000` | Max **project_chunks** rows loaded for one retrieval. |
+| `SKILLFORGE_INDEX_MAX_FILE_BYTES` | `524288` | Skip indexing files larger than this (bytes). |
+| `SKILLFORGE_INDEX_IGNORE_DIRS` | `""` | Extra comma-separated directory **basename** ignores (e.g. `out,tmp`). |
+| `SKILLFORGE_REDACT_CONTEXT` | `1` | When **`1`** (default), scrub common secret shapes and (with home path scrub) exported context before MCP/CLI output and route events. |
+| `SKILLFORGE_REDACT_HOME_IN_PATHS` | `1` | Replace resolved home-directory prefixes in chunk paths / DB path hints with **`[HOME]`**. |
+| `SKILLFORGE_MCP_USER_ID` | `""` | Default logical **user id** for MCP tool calls when arguments omit `user_id` (weights, sessions, events). |
 | `SKILLFORGE_PROJECT_ROOT` | `""` | Default workspace root when MCP **`project_root`** is omitted: events/weights/sessions live in **`<root>/.skillforge/orchestrator.db`**. Prefer passing **`project_root`** on each tool call from the host. |
 | `SKILLFORGE_SKILL_HOT_RELOAD` | `1` | When **`0`** / **`false`**, disable **SKILL.md** hot-reload; restart the MCP process to refresh the catalog. |
 | `SKILLFORGE_WATCH_SKILLS_INTERVAL` | `30` | Seconds between background catalog checks when hot reload is on. **`0`**: no background polling and no MCP **`tools.listChanged`**; **`tools/list`** and **`tools/call`** still reload when files change. |
 | `SKILLFORGE_MCP_LIST_CHANGED` | `1` | When **`0`** / **`false`**, never emit **`notifications/tools/list_changed`** (and **`listChanged`** is not advertised), even if a background interval is set. |
-
----
-
-## HTTP API
-
-| Method | Path | Description |
-|--------|------|-------------|
-| `POST` | `/chat` | Primary chat; SSE stream. |
-| `POST` | `/feedback` | Skill feedback for learning. |
-| `POST` | `/skills/disable` | Enable/disable a skill flag. |
-| `GET` | `/skills` | Catalog with stats and weights. |
-| `GET` | `/events` | Recent routing events (`?limit=`). |
-| `GET` | `/` | JSON service hint (use **`skillforge events --watch`** for a live terminal log). |
-| `GET` | `/healthz` | Health metadata (`skills_loaded`, **`live_log`** hint). |
-
-Authenticated mode applies **`Bearer`** tokens as described above. Do not expose unauthenticated instances beyond trusted networks.
+| `SKILLFORGE_ROUTE_POLICIES` | `""` | Optional inline JSON policies document (see [Route policies](#route-policies-optional)). |
+| `SKILLFORGE_ROUTE_POLICIES_FILE` | `""` | Path to a policies JSON file. |
 
 ---
 
@@ -282,7 +314,8 @@ Optional **per-project** state (when **`project_root`** or **`SKILLFORGE_PROJECT
 
 ```
 <workspace>/.skillforge/
-├── orchestrator.db   # SQLite for this repo (sessions, weights, events)
+├── orchestrator.db   # SQLite: sessions, weights, events, **project_chunks** (after `skillforge index`)
+├── policies.json     # Optional route policies (see README)
 └── last_route.json   # Last route_skills snapshot (after a routed call)
 ```
 
@@ -295,12 +328,12 @@ Global default when no project root:
 ├── skills/               # User-added skills
 ├── packs/<hash>/         # Cloned pack repositories
 ├── packs.json            # Pack registry
-└── auth.json             # Tokens (POSIX mode 0600 when used)
 ```
 
 | Command | Effect |
 |---------|--------|
 | `skillforge events` | Prints a **usage** snapshot and recent **`route`** / **`feedback`** rows; **`--watch`**, **`--project-root`** (per-repo DB), **`--user`**, **`--verbose`** (see **`--help`**). |
+| `skillforge index` | Chunk/embed text files under **`--project-root`** into **`project_chunks`**. **`--reset`**, **`--stats-only`**, **`--quiet`** (see **`--help`**). |
 | `skillforge reset` | Clears learning state and event history in the database. |
 | `skillforge install` | Re-runs bootstrap (venv and dependencies). |
 | `rm -rf ~/.skillforge` | Full removal of local state and venv. |
@@ -309,8 +342,8 @@ Global default when no project root:
 
 ## Security considerations
 
-- Treat **`ANTHROPIC_API_KEY`** and bearer tokens as **secrets**. Prefer environment injection or secret stores, not committed files.
-- For **internet-facing** deployments, use **TLS**, **reverse proxies**, and **mandatory** bearer authentication; assume an open HTTP port is reachable by untrusted clients.
+- **Redaction is best-effort regex scrubbing**, not a guarantee. Do not paste production secrets into prompts; treat routed context like **untrusted text** until reviewed.
+- Treat **`ANTHROPIC_API_KEY`** as a **secret** when you set it (e.g. for Haiku routing in MCP). Prefer environment injection or secret stores, not committed files.
 - Vulnerability disclosure: see **[SECURITY.md](SECURITY.md)**.
 
 ---

package/RELEASING.md

@@ -77,7 +77,7 @@ npm test
 Python (syntax only):
 
 ```bash
-for f in python/app/main.py python/app/cli.py python/app/mcp_server.py python/app/auth.py python/app/events_cli.py python/app/materialize.py python/app/db_paths.py python/app/route_cli.py; do python3 -m py_compile "$f"; done
+for f in python/app/main.py python/app/mcp_server.py python/app/events_cli.py python/app/materialize.py python/app/db_paths.py python/app/route_cli.py python/app/mcp_contract.py python/app/chunking.py python/app/project_index.py python/app/index_cli.py python/app/context_fusion.py python/app/redaction.py python/app/route_policies.py python/app/routing_signals.py; do python3 -m py_compile "$f"; done
 ```
 
 ## Troubleshooting: `EOTP` / one-time password in CI

package/SECURITY.md

@@ -26,6 +26,6 @@ We aim to acknowledge valid reports within a few business days.
 - Keep **2FA** enabled on the npm account that owns the `@heytherevibin` scope.
 - Prefer pinning action versions or reviewing Dependabot PRs before merge.
 
-## Runtime hardening
+## Runtime security
 
-If you expose the HTTP server beyond **localhost**, treat it as internet-facing: use **bearer-token auth** (`skillforge auth …`), terminate TLS at a reverse proxy, and restrict network access appropriately. The default single-user setup assumes a trusted local environment.
+Skillforge’s published surface is **local**: **stdio MCP** and **CLI** commands. Keep your **MCP host** and **`ANTHROPIC_API_KEY`** (if used) within a trusted environment. Do not commit secrets.

package/STRATEGY.md

@@ -7,8 +7,7 @@ Skillforge is an **npm-packaged skill orchestrator**: it routes tasks to a small
 ## Surfaces (today)
 
 - **MCP** (`skillforge mcp`): primary — `route_skills`, `list_skills`, feedback tools, `materialize_project`, `skillforge_bootstrap`.
-- **Terminal**: `skillforge events --watch` with optional **`--project-root`**.
-- **HTTP** (`skillforge start`): optional; still uses the global DB in **app_state** (not per-request project root unless extended later).
+- **Terminal**: `skillforge events --watch` with optional **`--project-root`**; `skillforge route`, `skillforge index`.
 
 ## Cursor reality
 
@@ -17,7 +16,6 @@ Native **`/skillforge`** in editor chat is **not** registered by this npm packag
 ## Near-term backlog
 
 - Shared **`orchestrate()`** API for MCP + CLI parity.
-- HTTP: optional **`project_root`** header or body for `/chat` if needed.
 - Tests: MCP handshake + `resolve_orchestrator_db` behavior.
 
 ## Non-goals (v1)