loopctl-mcp-server 2.39.0 → 2.41.0

package/README.md

@@ -2,7 +2,7 @@
 
 MCP (Model Context Protocol) server for [loopctl](https://loopctl.com) -- structural trust for AI development loops.
 
-Wraps the loopctl REST API into 73 typed MCP tools so AI coding agents (Claude Code, etc.) can interact with loopctl without writing curl commands.
+Wraps the loopctl REST API into 76 typed MCP tools so AI coding agents (Claude Code, etc.) can interact with loopctl without writing curl commands.
 
 ## Installation
 
@@ -123,6 +123,10 @@ autonomous agent) can self-remediate without a human. Full agent-tenant lifecycl
 
 ## Tools (84)
 
+> Plus **per-tenant generated Context Retriever tools** (`cr_*`) appended
+> dynamically at runtime — see [Dynamic per-tenant Context Retriever
+> tools](#dynamic-per-tenant-context-retriever-tools-epic-30) below.
+
 ### Project Tools
 
 | Tool | Description |
@@ -194,6 +198,8 @@ autonomous agent) can self-remediate without a human. Full agent-tenant lifecycl
 | How many articles? (counts only) | `knowledge_stats` |
 | Browse the catalog (lightweight id/title/category) | `knowledge_index` |
 | Find by topic / relevance (ranked, published-only, lags writes) | `knowledge_search` |
+| One trustworthy answer + provenance (curated-first, retrieval fallback) | `knowledge_hybrid_search` |
+| Survey a topic cheaply (capped stubs) then open only what you need | `knowledge_progressive_index` + `knowledge_progressive_drill` |
 | Enumerate / dedup / repair, or "does X exist?" (full fields, lag-free, all-status) | `knowledge_list` |
 
 | Tool | Description |
@@ -203,6 +209,9 @@ autonomous agent) can self-remediate without a human. Full agent-tenant lifecycl
 | `knowledge_count` | Count articles matching filters **without returning rows** within agent's visible set. Agent callers see only their own and `shared` articles. Same filters as `knowledge_list` (`category`, `status`, `tags`, `match`, `source_type`, `source_id`, `idempotency_key`, `project_id`). With `tags`+`match: all` (+`status`) → "how many published articles tagged both X and Y (that I can see)". Returns `{ count }`. |
 | `knowledge_facets` | Count articles grouped by **distinct tag** within agent's visible set, no rows. Agent callers see only their own and `shared` articles. `tag_prefix` (e.g. `book-`) gives the distinct count of a tag family plus per-member totals. Returns `{ data: { tag: count }, meta: { distinct_count } }`. Optional: `category`, `status`, `tags`, `match`, `project_id`, `limit`. |
 | `knowledge_search` | Search the knowledge wiki by topic (keyword, semantic, or combined). Returns snippets. **Ranked, published-only, and LAGS writes by minutes while embeddings index — do NOT use for existence/idempotency/dedup checks (a fresh write false-negatives); use `knowledge_list` for that.** `q` is optional when `tags`/`category` are supplied — that **list mode** returns the complete filtered set paginated via `offset`/`limit` over `meta.total_count`. `meta.total_count` is mode-dependent — read `meta.total_count_scope` (`keyword_matches`/`ranked_corpus`/`merged_candidates`/`filtered_set`) and don't use a relevance-mode count to size the wiki (use `knowledge_list` or `knowledge_stats`). Optional: `project_id`, `story_id` for attribution. |
+| `knowledge_hybrid_search` | Resolve a topic to a **single best answer with provenance** (US-31.4). Runs combined keyword+semantic over the full ranked pool, then decides whether a governed **curated** source actually answers. `meta.provenance` is `curated` (trust it — the canonical article is first in `data`, `meta.curated_article_id` points at it) or `retrieved` (best fuzzy match, `curated_article_id` null); `meta.confidence` is the winner's absolute score. Prefer over `knowledge_search` when you want one trustworthy answer, not a list to triage. Degrades to keyword-only like `knowledge_search` when embeddings are unavailable. Required: `query`. Optional: `project_id`, `category`, `tags`, `match`, `limit`, `offset`. |
+| `knowledge_progressive_index` | Progressive disclosure — a **cheap, capped index** of what's relevant to a topic (compact stubs: `id`/`title`/`category`/`summary`, **no bodies**), curated-preferred and hub-enriched, capped at top-K (`meta.truncated` when the pool exceeded it). Survey a topic without flooding context, then open only what you need via `knowledge_progressive_drill`. Required: `topic`. Optional: `category`, `limit`. |
+| `knowledge_progressive_drill` | Open one stub from `knowledge_progressive_index` — returns the **full article body** for the given id, scope-enforced. Resolves both tenant-owned articles and published system canonicals (the same set the index surfaces). Required: `article_id`. |
 | `knowledge_list` | List articles (`id`, `title`, `category`, `status`, `tags`, `source_type`, `source_id`, `idempotency_key`, timestamps), filtered + paginated. **Body-less summary by default** (safe to page up to `limit=1000`); pass `include_body: true` to also return `body`, in which case the page is bounded by a ~5 MB byte budget — continue via `meta.next_offset` while `meta.has_more`. **Lag-free, all-status** read of the DB of record — unlike `knowledge_search` (ranked, published-only, lags writes) and `knowledge_index` (id/title/category only). The right tool to enumerate/dedup/repair and for idempotency/existence checks: filter by `tags`, `source_type`+`source_id`, or `idempotency_key` and read `meta.total_count` (exact). Single full body → `knowledge_get`; relevant bodies → `knowledge_context`; bulk dump → `knowledge_export`. Optional: `project_id`, `category`, `status`, `tags`, `source_type`, `source_id`, `idempotency_key`, `offset`, `limit`, `include_body`. |
 | `knowledge_get` | Get full article content by ID. Use after search to read an article in detail. Optional: `project_id`, `story_id` for attribution. |
 | `knowledge_context` | Get relevance-and-recency-ranked full articles for a task query. Best knowledge for your current context. **Agent-memory scoping**: `memory_types` (comma-separated, OR — observation/finding/summary/decision/question/task), `agents` (comma-separated agent_ids, OR), `conversation_id` (exact) filter on article `metadata` (JSONB `@>`). Optional: `project_id`, `story_id` for attribution, `limit`, `recency_weight`. |
@@ -280,6 +289,26 @@ it is enforced server-side and a no-op for a non-superadmin key — see below.)
 | `list_routes` | List all available API routes on the loopctl server. |
 | `get_system_articles` | List or fetch system-scoped (global, cross-tenant) wiki articles. Public — no auth required. Optional: `slug` (fetch one), `category`. |
 
+### Dynamic per-tenant Context Retriever tools (Epic 30)
+
+Beyond the static tools above, the server appends **per-tenant generated tools** to
+`ListTools` at runtime. When your tenant declares an **entity** (`POST
+/api/v1/entities`), loopctl auto-generates governed query tools over that entity's
+allowlisted columns, and this MCP server fetches them from `GET
+/api/v1/retrieve/tools` and lists them alongside the static tools:
+
+| Generated tool | What it does |
+|---|---|
+| `cr_filter_<entity>_by_<field>` | Filter that entity's records where `<field>` equals a value. Params: the field value + `limit`/`offset`. |
+| `cr_search_<entity>` | Full-text search across that entity's searchable text fields. Params: `query` + `limit`/`offset`. |
+
+These are **tenant-scoped**: the listing reflects only the tenant of the process
+key (`LOOPCTL_AGENT_KEY`), resolved server-side — you never pass a tenant. A
+`cr_`-prefixed call is dispatched generically to `POST /api/v1/retrieve/:entity`
+through the same authenticated + witness/STH path as every static read tool. If the
+`/retrieve/tools` fetch fails, listing degrades to the static tools (never errors).
+The generated-tool count per tenant is bounded by the per-tenant entity cap.
+
 ### Dispatch & Chain of Custody (v2) Tools
 
 Key distribution for the dispatch pattern (Epic 26): per-dispatch ephemeral keys and capability-token recovery. See `docs/chain-of-custody-v2.md`.

package/index.js

@@ -25,6 +25,10 @@ import {
   createWitnessClient,
   resolveSthStatePath,
 } from "./lib/witness-sth.js";
+import {
+  createGeneratedToolsRuntime,
+  GENERATED_TOOL_PREFIX,
+} from "./lib/generated-tools.js";
 
 // Single source of truth for the server version: the package.json this file
 // ships with (npm always includes package.json in the published tarball).
@@ -61,6 +65,11 @@ const SERVER_VERSION = JSON.parse(
 // transparent retry above). The write is atomic + symlink-safe (temp + rename).
 const WITNESS_FS = { readFileSync, writeFileSync, renameSync, lstatSync, unlinkSync };
 
+// Default per-request HTTP timeout. Individual calls may pass a SHORTER budget via
+// `apiCall(..., { timeoutMs })` — e.g. the init-time generated-tools listing fetch,
+// which must degrade to static tools quickly rather than block connection setup.
+const DEFAULT_REQUEST_TIMEOUT_MS = 30_000;
+
 // One witness client PER API KEY (#298 review HIGH-2): the STH is per-tenant and
 // each key resolves to a tenant server-side, so distinct keys must NOT share an
 // in-memory cache or a state file (a collision causes spurious 409s + false
@@ -77,7 +86,7 @@ function witnessClientFor(apiKey) {
       fs: WITNESS_FS,
       getuid: typeof process.getuid === "function" ? () => process.getuid() : undefined,
       pid: process.pid,
-      timeoutMs: 30_000,
+      timeoutMs: DEFAULT_REQUEST_TIMEOUT_MS,
     });
     witnessClients.set(apiKey, client);
   }
@@ -104,7 +113,7 @@ function resolveKey(keyOverride) {
   );
 }
 
-async function apiCall(method, path, body, keyOverride, { exactKey = false } = {}) {
+async function apiCall(method, path, body, keyOverride, { exactKey = false, timeoutMs } = {}) {
   const url = `${getBaseUrl()}${path}`;
   // Secret-managing tools pass exactKey:true so the request uses the EXACT
   // role-pinned key (LOOPCTL_USER_KEY) and does NOT fall back to the global
@@ -137,10 +146,11 @@ async function apiCall(method, path, body, keyOverride, { exactKey = false } = {
   // attempt's Response; we keep body parsing / error shaping here.
   let response;
   try {
-    response = await witnessClientFor(key).send({ url, method, headers, serializedBody });
+    response = await witnessClientFor(key).send({ url, method, headers, serializedBody, timeoutMs });
   } catch (err) {
     if (err.name === "TimeoutError") {
-      return { error: true, status: 0, body: "Request timed out after 30s" };
+      const secs = Math.max(1, Math.round((timeoutMs ?? DEFAULT_REQUEST_TIMEOUT_MS) / 1000));
+      return { error: true, status: 0, body: `Request timed out after ${secs}s` };
     }
     const cause = err.cause?.message ? ` (${err.cause.message})` : "";
     return { error: true, status: 0, body: `Network error: ${err.message}${cause}` };
@@ -952,6 +962,52 @@ async function knowledgeSearch({ q, project_id, story_id, category, tags, match,
   return withRemediationNotice(result);
 }
 
+async function knowledgeHybridSearch({ query, project_id, category, tags, match, limit, offset }) {
+  const bodyPayload = { query };
+  if (project_id) bodyPayload.project_id = project_id;
+  if (category) bodyPayload.category = category;
+  if (tags) bodyPayload.tags = tags;
+  if (match) bodyPayload.match = match;
+  if (limit != null) bodyPayload.limit = limit;
+  if (offset != null) bodyPayload.offset = offset;
+
+  const result = await apiCall(
+    "POST",
+    "/api/v1/knowledge/hybrid_search",
+    bodyPayload,
+    process.env.LOOPCTL_AGENT_KEY,
+  );
+  // meta.provenance ("curated" | "retrieved") is the load-bearing field and rides
+  // through toContent verbatim. Same missing-embedding-key remediation surfacing as
+  // knowledge_search, since hybrid runs the combined pool underneath.
+  return withRemediationNotice(result);
+}
+
+async function knowledgeProgressiveIndex({ topic, category, limit }) {
+  const params = new URLSearchParams();
+  if (topic != null) params.set("topic", topic);
+  if (category) params.set("category", category);
+  if (limit != null) params.set("limit", String(limit));
+
+  const result = await apiCall(
+    "GET",
+    `/api/v1/knowledge/progressive_index?${params}`,
+    null,
+    process.env.LOOPCTL_AGENT_KEY,
+  );
+  return toContent(result);
+}
+
+async function knowledgeProgressiveDrill({ article_id }) {
+  const result = await apiCall(
+    "GET",
+    `/api/v1/knowledge/progressive/${article_id}`,
+    null,
+    process.env.LOOPCTL_AGENT_KEY,
+  );
+  return toContent(result);
+}
+
 async function knowledgeList({
   project_id,
   category,
@@ -2995,6 +3051,109 @@ const TOOLS = [
       required: [],
     },
   },
+  {
+    name: "knowledge_hybrid_search",
+    description:
+      "Resolve a topic to a SINGLE best answer WITH provenance — the hybrid " +
+      "retrieval entrypoint. Runs the combined keyword+semantic search over the full " +
+      "ranked pool, then decides whether a governed CURATED source actually answers. " +
+      "The verdict is meta.provenance: 'curated' means a curated, canonical article " +
+      "answers — TRUST IT (it is guaranteed first in the results and meta.curated_article_id " +
+      "points at it); 'retrieved' means no curated source cleared the bar, so the answer " +
+      "is the best semantic/keyword match (a fuzzy fallback — verify before relying on it, " +
+      "meta.curated_article_id is null). meta.confidence is the winning candidate's absolute " +
+      "score for its provenance class. Prefer this over knowledge_search when you want ONE " +
+      "trustworthy answer plus its provenance rather than a ranked list to triage yourself; " +
+      "use knowledge_search when you want to browse/enumerate matches. Additive — existing " +
+      "knowledge tools are unchanged. If semantic ranking is unavailable it degrades to " +
+      "keyword-only (meta.fallback/fallback_reason), same as knowledge_search.",
+    inputSchema: {
+      type: "object",
+      properties: {
+        query: {
+          type: "string",
+          description: "The topic/question to resolve (max 500 characters). Required.",
+        },
+        project_id: {
+          type: "string",
+          format: "uuid",
+          description: "Optional: scope to a project UUID.",
+        },
+        category: {
+          type: "string",
+          description: "Optional: filter by category.",
+        },
+        tags: {
+          type: "string",
+          description: "Optional: comma-separated tags to filter by.",
+        },
+        match: {
+          type: "string",
+          enum: ["any", "all"],
+          description: "Optional: tag match mode — 'any' (default, OR) or 'all' (AND, every tag).",
+        },
+        limit: {
+          type: "integer",
+          description: "Optional: max results to return (default 10).",
+        },
+        offset: {
+          type: "integer",
+          description: "Optional: results to skip for pagination (default 0).",
+        },
+      },
+      required: ["query"],
+    },
+  },
+  {
+    name: "knowledge_progressive_index",
+    description:
+      "Progressive disclosure — get a CHEAP, capped index of what's relevant to a topic " +
+      "(compact stubs: id/title/category/summary, NO bodies), then open only what you need " +
+      "with knowledge_progressive_drill. Curated sources are preferred and hub-linked " +
+      "neighbors are enriched in; results are capped at a configured top-K (meta.truncated " +
+      "is true when the candidate pool exceeded it). Use this to survey a topic without " +
+      "flooding your context with full article bodies — it's the index half of the hybrid " +
+      "interface. Follow up on a stub's id with knowledge_progressive_drill to read the body.",
+    inputSchema: {
+      type: "object",
+      properties: {
+        topic: {
+          type: "string",
+          description: "The topic to index (max 500 characters). Required.",
+        },
+        category: {
+          type: "string",
+          description: "Optional: filter by category.",
+        },
+        limit: {
+          type: "integer",
+          description: "Optional: top-K override (clamped to the configured cap).",
+        },
+      },
+      required: ["topic"],
+    },
+  },
+  {
+    name: "knowledge_progressive_drill",
+    description:
+      "Drill into one stub from knowledge_progressive_index — returns the FULL article " +
+      "body for the given id, scope-enforced. Resolves both tenant-owned articles and " +
+      "published system canonicals (the same set the index surfaces). This is the drill " +
+      "half of progressive disclosure: index cheaply, then open only the article(s) you " +
+      "need. (knowledge_get works for tenant articles too; use this to also reach the " +
+      "system canonicals the progressive index can surface.)",
+    inputSchema: {
+      type: "object",
+      properties: {
+        article_id: {
+          type: "string",
+          format: "uuid",
+          description: "The UUID of the article to open (from a progressive index stub).",
+        },
+      },
+      required: ["article_id"],
+    },
+  },
   {
     name: "knowledge_get",
     description:
@@ -4467,6 +4626,41 @@ const TOOLS = [
   },
 ];
 
+// ---------------------------------------------------------------------------
+// Context Retriever — dynamic per-tenant generated tools (US-30.5)
+// ---------------------------------------------------------------------------
+//
+// Epic 30 lets a tenant declare ENTITIES whose schema auto-generates agent tools
+// (`cr_filter_<entity>_by_<field>`, `cr_search_<entity>`) over governed loopctl
+// data. Those specs are generated SERVER-SIDE per tenant, so the MCP server can't
+// hard-code them in the static TOOLS array — it must fetch them at ListTools time
+// and append them, then dispatch any unknown `cr_`-prefixed CallTool to one
+// generic executor endpoint.
+//
+// Trust model: the stdio MCP process is ONE-TENANT-PER-PROCESS — the process key
+// (LOOPCTL_AGENT_KEY) resolves to a tenant server-side, and GET /retrieve/tools
+// returns ONLY that tenant's generated specs. The client never picks a tenant, so
+// cross-tenant listing/calling is impossible by construction (AC-30.5.3). Both the
+// listing fetch and the generic call ride the SAME `apiCall` path as static read
+// tools, so they carry identical auth + witness/STH headers (AC-30.5.4).
+
+// The generated-tools runtime (fetch + short-timeout/negative-TTL cache + generic
+// dispatch) lives in lib/generated-tools.js so its caching/TTL/negative-cache and
+// error-classification edge cases are exercised DIRECTLY by the test suite rather
+// than a hand-copied mirror. We wire the SHIPPED `apiCall` + static tool names +
+// `toContent` into it here. The process key (LOOPCTL_AGENT_KEY) is read per-call via
+// a getter so it rides the SAME auth + witness/STH path as every static read tool
+// (AC-30.5.4), and the static-name set powers the defense-in-depth drop of any spec
+// that isn't `cr_`-prefixed or that collides with a built-in tool name.
+const generatedToolsRuntime = createGeneratedToolsRuntime({
+  apiCall,
+  agentKey: () => process.env.LOOPCTL_AGENT_KEY,
+  staticToolNames: new Set(TOOLS.map((t) => t.name)),
+  toContent,
+});
+
+const { fetchGeneratedTools, callGeneratedTool } = generatedToolsRuntime;
+
 // ---------------------------------------------------------------------------
 // MCP Server
 // ---------------------------------------------------------------------------
@@ -4481,9 +4675,13 @@ const server = new Server(
   }
 );
 
-server.setRequestHandler(ListToolsRequestSchema, async () => ({
-  tools: TOOLS,
-}));
+server.setRequestHandler(ListToolsRequestSchema, async () => {
+  // Static hand-maintained tools PLUS the calling tenant's per-tenant generated
+  // Context Retriever tools (US-30.5). fetchGeneratedTools degrades to the static
+  // tools (returns []/cache) on any fetch failure, so listing never errors.
+  const generated = await fetchGeneratedTools();
+  return { tools: [...TOOLS, ...generated] };
+});
 
 server.setRequestHandler(CallToolRequestSchema, async (request) => {
   const { name, arguments: args } = request.params;
@@ -4605,6 +4803,15 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
     case "knowledge_search":
       return await knowledgeSearch(args);
 
+    case "knowledge_hybrid_search":
+      return await knowledgeHybridSearch(args);
+
+    case "knowledge_progressive_index":
+      return await knowledgeProgressiveIndex(args);
+
+    case "knowledge_progressive_drill":
+      return await knowledgeProgressiveDrill(args);
+
     case "knowledge_list":
       return await knowledgeList(args);
 
@@ -4753,6 +4960,13 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
       return await getAcceptanceCriteria(args);
 
     default:
+      // Per-tenant generated Context Retriever tools (US-30.5) are not in the
+      // static switch — dispatch any unknown `cr_`-prefixed name generically to
+      // the /retrieve/:entity executor. Static tools are handled above, so this
+      // never regresses them (AC-30.5.2).
+      if (typeof name === "string" && name.startsWith(GENERATED_TOOL_PREFIX)) {
+        return await callGeneratedTool(name, args);
+      }
       throw new Error(`Unknown tool: ${name}`);
   }
 });

package/lib/generated-tools.js

@@ -0,0 +1,209 @@
+// Per-tenant generated Context Retriever tool runtime (US-30.5).
+//
+// index.js is a stdio entry point with top-level await (importing it boots the MCP
+// server), so its handlers can't be imported by tests. This module extracts the
+// generated-tools fetch / cache / dispatch runtime — the logic with real caching,
+// TTL, negative-cache and error-classification edge cases — into a side-effect-free
+// factory (like lib/http-helpers.js). index.js wires the SHIPPED `apiCall`, static
+// tool names, and `toContent` into `createGeneratedToolsRuntime`, and the test suite
+// exercises the SAME code, so a regression fails CI instead of silently passing
+// against a hand-copied mirror.
+
+import {
+  retrieveToolsPath,
+  retrieveEntityPath,
+  specToMcpTool,
+  buildRetrieveBody,
+} from "./http-helpers.js";
+
+export const GENERATED_TOOL_PREFIX = "cr_";
+
+// ListTools is an init-time, agent-blocking call (Claude Code blocks on it during
+// connection setup). So the generated-tools fetch uses BOTH a SHORT timeout and a
+// NEGATIVE TTL: during any Context-Retriever backend slowness/outage the listing
+// degrades to the static tool surface quickly, and a failed fetch is cached for a
+// short window so subsequent ListTools calls do NOT each re-issue a blocking fetch
+// before degrading (us_30.5.json technical_notes: "keep it cached/short-timeout and
+// degrade to static on failure").
+export const GENERATED_TOOLS_CACHE_TTL_MS = 30_000; // positive: last fetch succeeded
+export const GENERATED_TOOLS_NEGATIVE_TTL_MS = 5_000; // negative: last fetch failed
+export const GENERATED_TOOLS_FETCH_TIMEOUT_MS = 5_000; // short per-fetch AbortSignal budget
+
+/**
+ * Build a generated-tools runtime bound to injected dependencies. State (cache +
+ * name->metadata map) is private to the returned closure.
+ *
+ * @param {{
+ *   apiCall: (method: string, path: string, body: unknown, key: (string|undefined), opts?: object) => Promise<any>,
+ *   agentKey: () => (string|undefined),
+ *   staticToolNames?: Set<string>,
+ *   toContent: (result: any) => object,
+ *   now?: () => number,
+ *   log?: (msg: string) => void,
+ *   cacheTtlMs?: number,
+ *   negativeTtlMs?: number,
+ *   fetchTimeoutMs?: number,
+ *   prefix?: string,
+ * }} deps
+ */
+export function createGeneratedToolsRuntime({
+  apiCall,
+  agentKey,
+  staticToolNames = new Set(),
+  toContent,
+  now = () => Date.now(),
+  log = (msg) => console.error(msg),
+  cacheTtlMs = GENERATED_TOOLS_CACHE_TTL_MS,
+  negativeTtlMs = GENERATED_TOOLS_NEGATIVE_TTL_MS,
+  fetchTimeoutMs = GENERATED_TOOLS_FETCH_TIMEOUT_MS,
+  prefix = GENERATED_TOOL_PREFIX,
+}) {
+  // `ok` records the LAST fetch outcome so (a) the TTL guard applies the negative
+  // TTL after a failure and (b) an unresolved CallTool can tell "temporarily
+  // unavailable" from "genuinely unknown to this tenant". `tools`/`metadata` retain
+  // the last KNOWN-GOOD listing so a failure degrades to it, never poisons it.
+  let cache = { tools: [], fetchedAt: 0, ok: true };
+  let metadataByName = new Map();
+
+  function cacheFailure(at, message) {
+    log(message);
+    // Negative caching: stamp fetchedAt on failure (ok:false) WITHOUT touching the
+    // retained tools/metadata, so subsequent calls within the negative-TTL window
+    // return the degraded static surface instead of re-issuing a blocking fetch.
+    cache = { tools: cache.tools, fetchedAt: at, ok: false };
+    return cache.tools;
+  }
+
+  /**
+   * Fetch the calling tenant's generated tool specs from GET /retrieve/tools and
+   * map them to the MCP `{ name, description, inputSchema }` shape, (re)populating
+   * the name->metadata map. On ANY failure this logs, negative-caches, and returns
+   * the last-known (possibly empty) tools so ListTools degrades to static rather
+   * than erroring (AC-30.5.1). Honors the positive/negative TTL unless `force`.
+   *
+   * @param {{ force?: boolean }} [opts]
+   * @returns {Promise<Array<{ name: string, description: string, inputSchema: object }>>}
+   */
+  async function fetchGeneratedTools({ force = false } = {}) {
+    const at = now();
+    if (!force && cache.fetchedAt !== 0) {
+      const ttl = cache.ok ? cacheTtlMs : negativeTtlMs;
+      if (at - cache.fetchedAt < ttl) return cache.tools;
+    }
+
+    let result;
+    try {
+      result = await apiCall("GET", retrieveToolsPath(), null, agentKey(), {
+        timeoutMs: fetchTimeoutMs,
+      });
+    } catch (err) {
+      return cacheFailure(
+        at,
+        `loopctl: failed to fetch generated tools, degrading to static tools: ${err.message}`,
+      );
+    }
+
+    if (result && result.error === true) {
+      const detail = typeof result.body === "string" ? result.body : JSON.stringify(result.body);
+      return cacheFailure(
+        at,
+        `loopctl: failed to fetch generated tools (HTTP ${result.status}), degrading to static tools: ${detail}`,
+      );
+    }
+
+    const specs = result && Array.isArray(result.data) ? result.data : [];
+    const tools = [];
+    const metadata = new Map();
+    for (const spec of specs) {
+      const tool = specToMcpTool(spec);
+      if (!tool) continue;
+      // Defense-in-depth (US-30.5 review): the `cr_` prefix is a security-relevant
+      // property the CallTool dispatcher relies on and the server is trusted to
+      // enforce. Re-check it HERE so a non-cr_ spec (which would be listed but
+      // UNCALLABLE) or one colliding with a STATIC tool name (which would shadow a
+      // trusted built-in's description/inputSchema under tenant-controlled text —
+      // a tool-confusion/description-spoofing vector) can never be listed or
+      // registered under a trusted identity.
+      if (!tool.name.startsWith(prefix) || staticToolNames.has(tool.name)) {
+        log(
+          `loopctl: dropping generated tool spec with invalid name "${tool.name}" ` +
+            `(must be ${prefix}-prefixed and not collide with a static tool)`,
+        );
+        continue;
+      }
+      tools.push(tool);
+      if (spec.metadata && typeof spec.metadata === "object") {
+        metadata.set(tool.name, spec.metadata);
+      }
+    }
+
+    metadataByName = metadata;
+    cache = { tools, fetchedAt: at, ok: true };
+    return tools;
+  }
+
+  /**
+   * Resolve a generated tool name to its STRUCTURED dispatch metadata. On a miss
+   * (name not in the current map) force ONE fresh fetch — bypassing the warm-TTL
+   * short-circuit — so a tool created server-side since the last listing resolves
+   * instead of being reported as authoritatively unknown.
+   *
+   * @param {string} name
+   * @returns {Promise<object|null>}
+   */
+  async function resolveGeneratedToolMetadata(name) {
+    if (metadataByName.has(name)) return metadataByName.get(name);
+    await fetchGeneratedTools({ force: true });
+    return metadataByName.get(name) || null;
+  }
+
+  /**
+   * Generic dispatcher for a per-tenant generated tool. Resolves (entity, field,
+   * operation) from STRUCTURED metadata (never by name-splitting — AC-30.5.2),
+   * builds the US-30.4 body, and POSTs to /retrieve/:entity via the SAME `apiCall`
+   * path as static reads (same auth + witness/STH — AC-30.5.4). An unresolvable
+   * name after a forced refetch distinguishes a TRANSIENT backend failure (503,
+   * retryable) from a name genuinely UNKNOWN to this tenant (404).
+   *
+   * @param {string} name
+   * @param {object} [args]
+   */
+  async function callGeneratedTool(name, args = {}) {
+    const metadata = await resolveGeneratedToolMetadata(name);
+    if (metadata && typeof metadata.entity === "string" && typeof metadata.operation === "string") {
+      const result = await apiCall(
+        "POST",
+        retrieveEntityPath(metadata.entity),
+        buildRetrieveBody(metadata, args),
+        agentKey(),
+      );
+      return toContent(result);
+    }
+
+    // Unresolved. If the forced refetch above FAILED (cache.ok === false), this is a
+    // transient backend blip, not a definitive "unknown to this tenant" — say so, so
+    // an agent retries instead of abandoning a valid tool.
+    if (!metadata && !cache.ok) {
+      return toContent({
+        error: true,
+        status: 503,
+        body:
+          `Generated tool "${name}" could not be resolved: the Context Retriever ` +
+          `tool listing is temporarily unavailable (the tools fetch failed). Retry shortly.`,
+      });
+    }
+    return toContent({
+      error: true,
+      status: 404,
+      body: `Unknown tool: ${name}. No generated tool by that name is available to this tenant.`,
+    });
+  }
+
+  return {
+    fetchGeneratedTools,
+    resolveGeneratedToolMetadata,
+    callGeneratedTool,
+    // Introspection seam for tests (not used by the server).
+    cacheState: () => ({ ...cache }),
+  };
+}

package/lib/http-helpers.js

@@ -86,6 +86,98 @@ export function memoryPath({ limit, offset, include_superseded, all_subjects } =
   ])}`;
 }
 
+/**
+ * Path for the Context Retriever generated-tool specs (US-30.5, US-30.4).
+ * The literal `/retrieve/tools` route is fixed and carries no query params — the
+ * calling tenant is resolved SERVER-SIDE from the API key, never from the client,
+ * so there is nothing for the client to parameterize here (AC-30.5.3/AC-30.5.4).
+ *
+ * @returns {string}
+ */
+export function retrieveToolsPath() {
+  return "/api/v1/retrieve/tools";
+}
+
+/**
+ * Path for executing a generated tool against a single entity (US-30.5, US-30.4).
+ * `entity` comes from the STRUCTURED metadata on a generated tool spec (never by
+ * splitting the tool name — entity/field names contain underscores), so it is
+ * encoded defensively.
+ *
+ * @param {string} entity
+ * @returns {string}
+ */
+export function retrieveEntityPath(entity) {
+  return `/api/v1/retrieve/${encodeURIComponent(entity)}`;
+}
+
+/**
+ * Map a raw generated-tool spec (as returned by GET /api/v1/retrieve/tools —
+ * `{ name, description, input_schema, metadata }`) into the MCP tool shape the
+ * ListTools handler must return (`{ name, description, inputSchema }`). The server
+ * emits snake_case `input_schema`; MCP expects camelCase `inputSchema`.
+ *
+ * Returns null for a malformed spec (no string `name`) so a single bad entry is
+ * skipped rather than corrupting the whole listing.
+ *
+ * @param {{ name?: unknown, description?: unknown, input_schema?: unknown }} spec
+ * @returns {{ name: string, description: string, inputSchema: object } | null}
+ */
+export function specToMcpTool(spec) {
+  if (!spec || typeof spec.name !== "string") return null;
+  return {
+    name: spec.name,
+    description: typeof spec.description === "string" ? spec.description : "",
+    inputSchema:
+      spec.input_schema && typeof spec.input_schema === "object"
+        ? spec.input_schema
+        : { type: "object", properties: {} },
+  };
+}
+
+/**
+ * Build the POST /api/v1/retrieve/:entity request body for a generated-tool call
+ * from the tool's STRUCTURED dispatch metadata (US-30.2 — `{ entity, field,
+ * operation }`) plus the caller's runtime `args`. The (entity, field, operation)
+ * come from metadata, NOT from splitting the tool name (AC-30.5.2), so entity and
+ * field names containing underscores dispatch unambiguously.
+ *
+ * The body shape matches the US-30.4 controller (`RetrieveRequest`): `op` selects
+ * the operation; `filter` carries `field` + `value`, `search` carries `query`;
+ * `limit`/`offset` pagination pass through when present. Nullish args are omitted
+ * so unset params never override server defaults.
+ *
+ * Filter-value sourcing (US-30.5 fix): the US-30.2 ToolGenerator emits the filter
+ * tool's input_schema with the value argument under the FIELD-NAME key (e.g.
+ * `{status}`, `required: ["status"]`) — there is NO `value` property. A
+ * schema-compliant agent therefore calls `cr_filter_project_by_status({status:
+ * "active"})`. So read the value from the field-named arg FIRST, falling back to a
+ * literal `value` arg (the shape the controller also accepts) for tolerance.
+ * Reading only `args.value` would drop every real agent-supplied filter value and
+ * dispatch an empty filter that silently returns zero rows.
+ *
+ * @param {{ entity: string, field?: string|null, operation: string }} metadata
+ * @param {Record<string, unknown>} [args]
+ * @returns {object}
+ */
+export function buildRetrieveBody(metadata, args = {}) {
+  const body = { op: metadata.operation };
+
+  if (metadata.operation === "filter") {
+    body.field = metadata.field;
+    const fieldArg = metadata.field != null ? args[metadata.field] : undefined;
+    const filterValue = fieldArg !== undefined ? fieldArg : args.value;
+    if (filterValue !== undefined) body.value = filterValue;
+  } else if (metadata.operation === "search") {
+    if (args.query !== undefined) body.query = args.query;
+  }
+
+  if (args.limit != null) body.limit = args.limit;
+  if (args.offset != null) body.offset = args.offset;
+
+  return body;
+}
+
 /**
  * Defensively parse the raw text body of a JSON-content-type HTTP response
  * (#249, mcp-03).

package/lib/witness-sth.js

@@ -186,6 +186,7 @@ export function bootstrapRetrySth(status, currentSthHeader) {
  *   getuid?: () => number,
  *   pid?: number,
  *   timeoutMs?: number,
+ *   makeTimeoutSignal?: (ms: number) => (AbortSignal|undefined),
  * }} [deps]
  */
 export function createWitnessClient({
@@ -195,6 +196,10 @@ export function createWitnessClient({
   getuid,
   pid,
   timeoutMs = 30_000,
+  // Injectable so tests can observe the EFFECTIVE per-attempt timeout (an
+  // AbortSignal from AbortSignal.timeout does not expose its ms). Production uses
+  // the real timer.
+  makeTimeoutSignal = (ms) => AbortSignal.timeout(ms),
 } = {}) {
   // Load any persisted STH so a FRESH process sends a real header on request #1
   // and skips the bootstrap-grace 412 entirely.
@@ -213,10 +218,13 @@ export function createWitnessClient({
     if (statePath && fs) persistSth(statePath, sth, { fs, pid });
   }
 
-  async function attempt({ url, method, headers, serializedBody }) {
+  async function attempt({ url, method, headers, serializedBody, timeoutMs: reqTimeoutMs }) {
     // Witness protocol: echo the cached STH, or (never seen one) request a
     // one-time bootstrap. A fresh AbortSignal per attempt so the retry gets its
-    // own timeout budget.
+    // own timeout budget. A per-request `timeoutMs` (e.g. the SHORT budget the
+    // init-time ListTools generated-tools fetch uses so a backend blip degrades to
+    // static quickly instead of blocking connection setup for the full 30s)
+    // overrides the client default.
     const withWitness = { ...headers };
     if (lastKnownSTH) withWitness["X-Loopctl-Last-Known-STH"] = lastKnownSTH;
     else withWitness["X-Loopctl-STH-Bootstrap"] = "true";
@@ -224,7 +232,7 @@ export function createWitnessClient({
     const options = {
       method,
       headers: withWitness,
-      signal: AbortSignal.timeout(timeoutMs),
+      signal: makeTimeoutSignal(reqTimeoutMs ?? timeoutMs),
     };
     if (serializedBody !== undefined) options.body = serializedBody;
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "loopctl-mcp-server",
-  "version": "2.39.0",
+  "version": "2.41.0",
   "description": "MCP server for loopctl — structural trust for AI development loops",
   "type": "module",
   "main": "index.js",