@prismer/runtime 2.0.8 → 2.2.63

package/built-in-skills/liteparse/SKILL.md

@@ -1,156 +1,176 @@
 ---
 name: liteparse
-description: Parse documents into LLM-ready content entirely on the local machine — PDF / DOCX / XLSX / PPTX / images → Markdown, structured JSON (with bounding boxes), or page screenshots, via the `lit` CLI. No cloud, no LLM, works offline. Use whenever the user attaches or points to a document that must be read before reasoning, or asks to extract text / tables / page images from a file.
+scope: common
+description: "Parse local document files into LLM-ready content on the daemon itself — PDFs → Markdown / structured JSON (with bounding boxes) / page screenshots via the bundled `lit` CLI. PDF works out of the box, offline, zero external deps (Tesseract + PDFium are bundled): digital PDFs use a near-instant native text path (`--no-ocr`), scanned PDFs fall back to bundled OCR. Image files (PNG/JPG) additionally need ImageMagick, and Office files (DOCX/XLSX/PPTX) need LibreOffice — install those on demand only when required. Use whenever the user attaches or points to a local file that must be read before reasoning, or asks to extract text / tables / page images from a file on disk. For web URLs and search, use the `ingest` skill instead."
 ---
 
 # LiteParse (local-first document parsing)
 
-LiteParse is a standalone OSS parser (Rust core, PDFium + Tesseract +
-LibreOffice) that turns documents into text the model can read — **fully
-local, zero cloud dependency, works with no network**. It is the
-daemon's local-first document-parsing path.
+LiteParse turns document files into text the model can read — **on the
+daemon machine, offline, zero cloud dependency**. It ships as the
+`@llamaindex/liteparse` npm package (CLI: `lit`) with **PDFium and
+Tesseract bundled inside the package** — the digital-PDF and scanned-OCR
+paths need no external system dependencies.
 
 > Upstream: [`run-llama/liteparse`](https://github.com/run-llama/liteparse)
 > + skill [`run-llama/llamaparse-agent-skills`](https://github.com/run-llama/llamaparse-agent-skills/blob/main/skills/liteparse/SKILL.md).
 > Apache-2.0 (core) / MIT (skill), © LlamaIndex. Vendored as a Prismer
-> built-in; the CLI (`lit`) self-installs on first use (see below).
+> built-in.
 
-## Scope — liteparse vs `ingest`
+## Capability tiers — what works where
+
+| Tier | Inputs | External dep | Status |
+| --- | --- | --- | --- |
+| **1 — PDF** | digital + scanned PDF | **none** (Tesseract + PDFium bundled) | **Default, out of the box** — `lit` is baked into the daemon image |
+| **2 — Images** | PNG / JPG / TIFF / … | **ImageMagick** | Install on demand |
+| **3 — Office** | DOCX / XLSX / PPTX / ODT | **LibreOffice** (~1GB) | Install on demand |
+
+Only Tier 1 is guaranteed present. Tiers 2 and 3 pull large system
+dependencies that are **deliberately not baked into the daemon image** —
+install them yourself, only when the actual input requires it (see
+below). Do not assume they are already installed.
 
-Two document paths coexist; pick by network + fidelity needs:
-
-- **`liteparse` (this skill) — local-first.** Local document files →
-  Markdown / JSON / screenshots, fully offline, zero cost, zero cloud.
-  Covers PDF, Word, PowerPoint, spreadsheets, images. Built-in
-  Tesseract OCR for clean scans; LibreOffice for Office formats.
-  **Default for**: files already on disk, offline / cloud-unreachable
-  situations, digital PDFs, quick extraction, bounding boxes, page
-  screenshots.
-- **`ingest` (sibling skill) — cloud-backed.** `cloud load` / `cloud
-  search` for **web URLs** (HQCC compression) and `cloud parse` for
-  **hi-res OCR** of scans / handwriting / dense tables. **Use when**:
-  the source is a web page (liteparse can't fetch URLs), or local
-  parsing quality is insufficient (dense tables, multi-column,
-  handwriting, low-quality scans).
-
-Decision rule: **local file → try `liteparse` first** (free, offline,
-fast). Escalate to `ingest`'s `cloud parse -m hires` only when local
-fidelity falls short. **Web URL / search → `ingest`** (liteparse parses
-local files and stdin, not remote pages — if you only have a URL,
-`curl -sL <url> -o file` then parse the local copy, or route to
-`ingest`).
-
-When local parsing is low-confidence (empty pages, garbled OCR),
-**flag the region** and consider escalating to `ingest`; never invent
-content.
-
-## First-use install (self-bootstrapping)
-
-The `lit` CLI ships via npm/pip/cargo (same CLI). Check, then install if
-missing. Do this once per environment, quietly.
+## Tier 1 — PDF (default, ready to use)
+
+`lit` is available on the daemon. Sanity-check once, quietly:
 
 ```bash
-# 1. Is it already available? (sandbox image may bake it)
-command -v lit && lit --version && echo "lit ready" || {
-  # 2. Install the CLI
-  npm i -g @llamaindex/liteparse && lit --version
-}
+command -v lit && lit --version    # baked into the image
 ```
 
-System dependencies for Office conversion + image handling (only needed
-for non-PDF inputs):
+If (and only if) it is somehow missing, install the package locally — it
+is ~38MB with the OCR/render binaries bundled and installs in a few
+seconds:
 
 ```bash
-# LibreOffice — required for DOCX/XLSX/PPTX/ODT/RTF conversion
-brew install --cask libreoffice        # macOS
-apt-get install -y libreoffice         # Debian/Ubuntu (daemon image)
-
-# ImageMagick — required for some image formats
-brew install imagemagick               # macOS
-apt-get install -y imagemagick         # Debian/Ubuntu
+npm i @llamaindex/liteparse && npx lit --version
 ```
 
-If install fails (no network, no package manager), do **not** fabricate
-parsed content — report `lit unavailable (<reason>)` and stop.
+> Do **not** rely on the CLI "self-installing on first use" — that path
+> is unreliable (npm permission / cache conflicts). Either it is baked
+> in, or you install the package explicitly as above.
 
-## CLI reference
+### Native text path vs OCR — pick deliberately, cost differs ~10×
 
-### Parse a file
+- **Digital PDF (has an embedded text layer): use `--no-ocr`.** PDFium
+  extracts the text directly — **near-instant (~0.35s/page)**, no OCR.
+  This is the fast, cheap path; prefer it whenever the PDF is
+  computer-generated.
+- **Scanned / image-only PDF: leave OCR on (default).** The bundled
+  Tesseract OCRs each page — **~5s/page**, an order of magnitude slower.
+  Only pay this when there is no real text layer.
+
+If unsure which kind you have, run `--no-ocr` first: empty/garbled output
+means it's a scan → re-run without `--no-ocr` to OCR it.
 
 ```bash
-lit parse document.pdf                                  # → text on stdout
-lit parse document.pdf --format json -o out.json        # structured JSON + bounding boxes
-lit parse document.pdf --target-pages "1-5,10,15-20"    # page subset
-lit parse document.pdf --no-ocr                          # skip OCR (digital PDFs only)
-lit parse document.pdf --dpi 300                         # higher render DPI
-lit parse scan.pdf --ocr-language eng+chi_sim            # Tesseract lang codes
-lit parse secured.pdf --password '****'                  # encrypted docs
-curl -sL https://example.com/report.pdf | lit parse -    # stdin (download-then-parse)
+lit parse report.pdf --no-ocr                       # digital PDF → fast native text
+lit parse scan.pdf                                   # scanned PDF → bundled OCR
+lit parse report.pdf --format markdown -o out.md     # Markdown output to a file
+lit parse report.pdf --format json -o out.json       # structured JSON + bounding boxes
+lit parse report.pdf --target-pages "1-5,10,15-20"   # page subset (huge speedup on OCR path)
 ```
 
-Key `lit parse` options: `-o/--output`, `--format json|text`
-(default text), `--no-ocr`, `--ocr-language <lang>`, `--ocr-server-url
-<url>` (plug an external OCR HTTP server), `--tessdata-path`,
-`--target-pages "1-5,10"`, `--max-pages <n>` (default 1000), `--dpi`
-(default 150), `--preserve-small-text`, `--password`, `--num-workers`,
-`-q/--quiet`.
+`--format` accepts `markdown` (default) or `json` (adds per-block
+bounding boxes). Other useful flags: `-o/--output`, `--target-pages
+"1-5,10"`, `--dpi <n>` (render DPI, default 150 — bump to 300 for small
+scanned text), `--ocr-language eng+chi_sim` (Tesseract lang codes for the
+OCR path), `--password '****'` (encrypted PDFs), `-q/--quiet`.
 
-### Batch a directory
+### Page screenshots (for figures/charts text can't capture)
 
 ```bash
-lit batch-parse ./input-dir ./output-dir
-lit batch-parse ./input-dir ./output-dir --recursive --extension .pdf --format json
+lit screenshot report.pdf -o ./shots                   # all pages → PNG
+lit screenshot report.pdf --target-pages "1,3,5" -o ./shots
+lit screenshot report.pdf --dpi 300 -o ./shots         # high-res
 ```
 
-### Page screenshots (for visual content text can't capture)
+## Tier 2 — Image files (install ImageMagick on demand)
+
+Parsing standalone image files (PNG/JPG/TIFF/…) needs **ImageMagick**,
+which is not in the daemon image. Install it the first time you actually
+have an image input:
 
 ```bash
-lit screenshot document.pdf -o ./screenshots                  # all pages → PNG
-lit screenshot document.pdf --target-pages "1,3,5" -o ./shots
-lit screenshot document.pdf --dpi 300 -o ./shots              # high-res
+apt-get update && apt-get install -y imagemagick   # Debian/Ubuntu (daemon image)
+brew install imagemagick                            # macOS (local dev)
+lit parse diagram.png --format markdown
 ```
 
-Supported formats: PDF · Word (DOC/DOCX/DOCM/ODT/RTF) · PowerPoint
-(PPT/PPTX/PPTM/ODP) · Spreadsheets (XLS/XLSX/ODS/CSV/TSV) · Images
-(JPG/PNG/GIF/BMP/TIFF/WEBP/SVG).
+If ImageMagick can't be installed (no network / no apt), report
+`ImageMagick unavailable` and stop — don't fabricate the image's contents.
+
+## Tier 3 — Office documents (install LibreOffice on demand)
+
+DOCX / XLSX / PPTX / ODT conversion needs **LibreOffice (~1GB)**, which is
+**intentionally not baked into the daemon image**. Install it only when
+you genuinely need to parse an Office file (it's a big download — don't do
+it speculatively):
+
+```bash
+apt-get update && apt-get install -y libreoffice   # Debian/Ubuntu (daemon image)
+brew install --cask libreoffice                     # macOS (local dev)
+lit parse deck.pptx --format markdown
+```
+
+If LibreOffice can't be installed, say so and ask the user for a PDF
+export of the document instead of guessing.
+
+> A shared **cloud parse service for heavy formats** (offloading Office /
+> hi-res OCR to a hosted backend so agents don't install 1GB deps) is a
+> future TODO — it is **not** running today. Don't route to a cloud OCR
+> endpoint.
+
+## Scope — liteparse vs `ingest`
+
+Two skills, clean split by **source location**:
+
+- **`liteparse` (this skill)** — **local files on disk.** PDF / image /
+  Office files already on the machine → Markdown / JSON / screenshots,
+  fully offline. This is the only local document-parsing path.
+- **`ingest` (sibling skill)** — **web URLs and search.** `cloud load` /
+  `cloud search` fetch and compress remote pages. `liteparse` cannot
+  fetch URLs; if you only have a URL, either `curl -sL <url> -o file`
+  then parse the local copy with `liteparse`, or route to `ingest`.
+
+Decision rule: **local file → `liteparse`. Web page / search → `ingest`.**
 
 ## Workflow
 
-1. **Locate the file.** If it's a workspace asset, resolve its local
-   path (see `assets` skill). If it's a URL, `curl -sL <url> -o
-   <name>` first — `lit` parses local files (and stdin), not remote
-   pages.
-2. **Pick mode.**
-   - Digital PDF / clean doc → `lit parse <f>` (add `--no-ocr` to skip
-     OCR and go faster when the PDF has a real text layer).
-   - Scan / image / handwriting → leave OCR on; set `--ocr-language`
-     and bump `--dpi 300` if text is small.
-   - Need spatial structure / tables with coordinates → `--format
-     json` (gives bounding boxes).
-   - Need to *see* a chart/figure/signature → `lit screenshot`, then
-     read the PNG.
-3. **Read the result as source of truth.** Base extraction/summary
-   only on what `lit` returned. If a page is empty or low-confidence,
-   say so; don't guess.
-4. **Deliver products, if the parse output itself is the deliverable.**
-   Write final files into `$PRISMER_ARTIFACTS_DIR` (the dispatch's
-   `artifacts/` dir — auto-archived as IMAssets by the daemon's
-   artifacts-watcher and attached to your reply). See `office-artifacts`
-   SKILL.md §Delivery contract for the full delivery rules.
+1. **Locate the file.** Workspace asset → resolve its local path (see the
+   `assets` skill). URL-only → `curl -sL <url> -o <name>` first, then
+   parse the local copy (or use `ingest`).
+2. **Pick the path by input type + kind.**
+   - PDF, computer-generated → `lit parse <f> --no-ocr` (fast native).
+   - PDF, scanned/image-only → `lit parse <f>` (OCR); set
+     `--ocr-language`, bump `--dpi 300` for small text, and use
+     `--target-pages` to avoid OCR'ing pages you don't need.
+   - Need spatial structure / table coordinates → `--format json`.
+   - Need to *see* a chart/figure/signature → `lit screenshot`, then read
+     the PNG.
+   - Image file → install ImageMagick (Tier 2), then parse.
+   - Office file → install LibreOffice (Tier 3), then parse.
+3. **Read the result as source of truth.** Base extraction/summary only on
+   what `lit` returned. Empty or low-confidence page → say so; never guess.
+4. **Deliver products** only when the user requests the parse output as a
+   file. Write the requested final file into `$PRISMER_ARTIFACTS_DIR`, then
+   explicitly run `cloud deliver <abs-path>` for the current reply or
+   `cloud task attach <abs-path>` for a task. Auto-scan is OFF; writing the
+   file alone is not delivery. See `office-artifacts` SKILL.md §Delivery contract.
 
 ## Output reporting
 
-- After a parse: `Parsed <filename>: <N> pages, format=<text|json>,
-  ocr=<on|off>` — then proceed with the user's actual question.
-- Don't dump the entire parsed body into chat unless asked; summarize
-  and cite page numbers.
-- After screenshots: `Rendered <N> page screenshot(s) → <dir>` and read
+- After a parse: `Parsed <filename>: <N> pages, format=<markdown|json>,
+  path=<native|ocr>` — then answer the user's actual question, citing page
+  numbers. Don't dump the whole parsed body into chat unless asked.
+- After screenshots: `Rendered <N> page screenshot(s) → <dir>`, then read
   the relevant ones.
 
 ## HARD RULE — never claim a parse you didn't run
 
 Forbidden unless `lit` actually ran with exit 0 and produced output:
-"parsed the document", "the PDF says…", "extracted the table". If `lit`
-is unavailable or the parse failed, completion text must start with
-`无法解析文档 (reason)` / `Cannot parse document (reason)` — do not
-substitute guessed content.
+"parsed the document", "the PDF says…", "extracted the table". If `lit` is
+unavailable, a required dependency (ImageMagick / LibreOffice) can't be
+installed, or the parse failed, completion text must start with `无法解析
+文档 (reason)` / `Cannot parse document (reason)` — do not substitute
+guessed content.

package/built-in-skills/mcp-builder/SKILL.md

@@ -1,5 +1,6 @@
 ---
 name: mcp-builder
+scope: coding
 description: Guide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK).
 license: Complete terms in LICENSE.txt
 ---

package/built-in-skills/mcp-builder/reference/evaluation.md

@@ -389,7 +389,7 @@ After creating your evaluation file, you can use the provided evaluation harness
 
    Or install manually:
    ```bash
-   pip install anthropic mcp
+   pip install 'anthropic>=0.39,<2' 'mcp>=1.9,<2'
    ```
 
 2. **Set API Key**
@@ -400,6 +400,15 @@ After creating your evaluation file, you can use the provided evaluation harness
 
 ## Evaluation File Format
 
+Every invocation below must additionally supply `--model "$ANTHROPIC_MODEL"`
+and one or more `--allow-tool <reviewed-read-only-tool-name>` options. The
+harness refuses implicit all-tool access. Review the stdio command before
+starting it; a tool's `readOnlyHint` is not an authorization boundary. Use a
+read-only test principal, never a production write-capable credential. Each
+task has a 600s limit, at most 20 tool turns, and 60s per tool/model request.
+Do not place secrets in `-H` or `-e` arguments; inherit stdio credentials from
+the environment or use a local, non-secret test endpoint.
+
 Evaluation files use XML format with `<qa_pair>` elements:
 
 ```xml
@@ -599,4 +608,4 @@ If tasks are timing out:
 - Use a more capable model (e.g., `claude-3-7-sonnet-20250219`)
 - Check if tools are returning too much data
 - Verify pagination is working correctly
-- Consider simplifying complex questions
+- Consider simplifying complex questions

package/built-in-skills/mcp-builder/reference/node_mcp_server.md

@@ -544,11 +544,13 @@ async function getUser(id: string): Promise<any> {
     "node": ">=18"
   },
   "dependencies": {
-    "@modelcontextprotocol/sdk": "^1.6.1",
+    "@modelcontextprotocol/sdk": "^1.12.0",
+    "express": "^4.21.2",
     "axios": "^1.7.9",
     "zod": "^3.23.8"
   },
   "devDependencies": {
+    "@types/express": "^4.17.21",
     "@types/node": "^22.10.0",
     "tsx": "^4.19.2",
     "typescript": "^5.7.2"
@@ -596,6 +598,8 @@ import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
 import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
 import { z } from "zod";
 import axios, { AxiosError } from "axios";
+import { StreamableHTTPServerTransport } from "@modelcontextprotocol/sdk/server/streamableHttp.js";
+import express from "express";
 
 // Constants
 const API_BASE_URL = "https://api.example.com/v1";
@@ -697,7 +701,14 @@ server.registerTool(
     }
   },
   async (params: UserSearchInput) => {
-    // Implementation as shown above
+    try {
+      const result = await makeApiRequest<unknown>('users/search', 'GET', undefined, {
+        query: params.query, limit: params.limit, offset: params.offset,
+      });
+      return { content: [{ type: 'text' as const, text: JSON.stringify(result) }] };
+    } catch (error) {
+      return { isError: true, content: [{ type: 'text' as const, text: handleApiError(error) }] };
+    }
   }
 );
 
@@ -722,6 +733,15 @@ async function runHTTP() {
   }
 
   const app = express();
+  const accessToken = process.env.MCP_ACCESS_TOKEN;
+  if (!accessToken) throw new Error('MCP_ACCESS_TOKEN is required for HTTP transport');
+  app.use((req, res, next) => {
+    if (!['localhost', '127.0.0.1'].includes(req.hostname) || req.headers.origin || req.headers.authorization !== `Bearer ${accessToken}`) {
+      res.sendStatus(403);
+      return;
+    }
+    next();
+  });
   app.use(express.json());
 
   app.post('/mcp', async (req, res) => {
@@ -735,7 +755,7 @@ async function runHTTP() {
   });
 
   const port = parseInt(process.env.PORT || '3000');
-  app.listen(port, () => {
+  app.listen(port, '127.0.0.1', () => {
     console.error(`MCP server running on http://localhost:${port}/mcp`);
   });
 }
@@ -825,6 +845,15 @@ import { StreamableHTTPServerTransport } from "@modelcontextprotocol/sdk/server/
 import express from "express";
 
 const app = express();
+const accessToken = process.env.MCP_ACCESS_TOKEN;
+if (!accessToken) throw new Error('MCP_ACCESS_TOKEN is required');
+app.use((req, res, next) => {
+  if (!['localhost', '127.0.0.1'].includes(req.hostname) || req.headers.origin || req.headers.authorization !== `Bearer ${accessToken}`) {
+    res.sendStatus(403);
+    return;
+  }
+  next();
+});
 app.use(express.json());
 
 app.post('/mcp', async (req, res) => {
@@ -840,7 +869,7 @@ app.post('/mcp', async (req, res) => {
   await transport.handleRequest(req, res, req.body);
 });
 
-app.listen(3000);
+app.listen(3000, '127.0.0.1');
 ```
 
 #### stdio (For Local Integrations)
@@ -967,4 +996,4 @@ Before finalizing your Node/TypeScript MCP server implementation, ensure:
 - [ ] dist/index.js created and executable
 - [ ] Server runs: `node dist/index.js --help`
 - [ ] All imports resolve correctly
-- [ ] Sample tool calls work as expected
+- [ ] Sample tool calls work as expected

package/built-in-skills/mcp-builder/reference/python_mcp_server.md

@@ -536,7 +536,11 @@ async def get_document(name: str) -> str:
     Resources are useful for static or semi-static data that doesn't
     require complex parameters. They use URI templates for flexible access.
     '''
-    document_path = f"./docs/{name}"
+    from pathlib import Path
+    root = Path('./docs').resolve()
+    document_path = (root / name).resolve()
+    if not document_path.is_relative_to(root) or not document_path.is_file():
+        raise ValueError('Document must be a file under the docs root')
     with open(document_path, "r") as f:
         return f.read()
 
@@ -716,4 +720,4 @@ Before finalizing your Python MCP server implementation, ensure:
 - [ ] Server runs successfully: `python your_server.py --help`
 - [ ] All imports resolve correctly
 - [ ] Sample tool calls work as expected
-- [ ] Error scenarios handled gracefully
+- [ ] Error scenarios handled gracefully

package/built-in-skills/mcp-builder/scripts/connections.py

@@ -67,7 +67,7 @@ class MCPConnection(ABC):
     async def call_tool(self, tool_name: str, arguments: dict[str, Any]) -> Any:
         """Call a tool on the MCP server with provided arguments."""
         result = await self.session.call_tool(tool_name, arguments=arguments)
-        return result.content
+        return result.model_dump(mode='json', exclude_none=True)
 
 
 class MCPConnectionStdio(MCPConnection):

package/built-in-skills/mcp-builder/scripts/evaluation.py

@@ -6,6 +6,7 @@ This script evaluates MCP servers by running test questions against them using C
 import argparse
 import asyncio
 import json
+import os
 import re
 import sys
 import time
@@ -70,10 +71,11 @@ def parse_evaluation_file(file_path: Path) -> list[dict[str, Any]]:
                     "answer": (answer_elem.text or "").strip(),
                 })
 
+        if not evaluations or any(not qa['question'] or not qa['answer'] for qa in evaluations):
+            raise ValueError('Evaluation must contain non-empty question/answer pairs')
         return evaluations
     except Exception as e:
-        print(f"Error parsing evaluation file {file_path}: {e}")
-        return []
+        raise ValueError(f"Invalid evaluation file: {file_path}") from e
 
 
 def extract_xml_content(text: str, tag: str) -> str | None:
@@ -89,6 +91,7 @@ async def agent_loop(
     question: str,
     tools: list[dict[str, Any]],
     connection: Any,
+    max_turns: int = 20,
 ) -> tuple[str, dict[str, Any]]:
     """Run the agent loop with MCP tools."""
     messages = [{"role": "user", "content": question}]
@@ -105,34 +108,39 @@ async def agent_loop(
     messages.append({"role": "assistant", "content": response.content})
 
     tool_metrics = {}
+    allowed_names = {tool['name'] for tool in tools}
+    turns = 0
 
     while response.stop_reason == "tool_use":
-        tool_use = next(block for block in response.content if block.type == "tool_use")
-        tool_name = tool_use.name
-        tool_input = tool_use.input
-
-        tool_start_ts = time.time()
-        try:
-            tool_result = await connection.call_tool(tool_name, tool_input)
-            tool_response = json.dumps(tool_result) if isinstance(tool_result, (dict, list)) else str(tool_result)
-        except Exception as e:
-            tool_response = f"Error executing tool {tool_name}: {str(e)}\n"
-            tool_response += traceback.format_exc()
-        tool_duration = time.time() - tool_start_ts
-
-        if tool_name not in tool_metrics:
-            tool_metrics[tool_name] = {"count": 0, "durations": []}
-        tool_metrics[tool_name]["count"] += 1
-        tool_metrics[tool_name]["durations"].append(tool_duration)
-
-        messages.append({
-            "role": "user",
-            "content": [{
-                "type": "tool_result",
-                "tool_use_id": tool_use.id,
-                "content": tool_response,
-            }]
-        })
+        turns += 1
+        if turns > max_turns:
+            raise RuntimeError('Tool turn budget exceeded')
+        results = []
+        for tool_use in response.content:
+            if tool_use.type != 'tool_use':
+                continue
+            tool_name = tool_use.name
+            tool_start_ts = time.monotonic()
+            is_error = False
+            try:
+                if tool_name not in allowed_names:
+                    raise PermissionError('Tool not allowlisted')
+                tool_result = await asyncio.wait_for(connection.call_tool(tool_name, tool_use.input), timeout=60)
+                if hasattr(tool_result, 'model_dump'):
+                    tool_result = tool_result.model_dump(mode='json', exclude_none=True)
+                is_error = isinstance(tool_result, dict) and bool(tool_result.get('isError'))
+                tool_response = json.dumps(tool_result, default=lambda v: v.model_dump(mode='json'))
+            except Exception as e:
+                is_error = True
+                tool_response = 'Tool failed: ' + type(e).__name__
+            metric = tool_metrics.setdefault(tool_name, {'count': 0, 'durations': []})
+            metric['count'] += 1
+            metric['durations'].append(time.monotonic() - tool_start_ts)
+            results.append({'type': 'tool_result', 'tool_use_id': tool_use.id,
+                            'content': tool_response, 'is_error': is_error})
+        if not results:
+            raise RuntimeError('tool_use response contained no tool calls')
+        messages.append({'role': 'user', 'content': results})
 
         response = await asyncio.to_thread(
             client.messages.create,
@@ -146,7 +154,7 @@ async def agent_loop(
 
     response_text = next(
         (block.text for block in response.content if hasattr(block, "text")),
-        None,
+        '',
     )
     return response_text, tool_metrics
 
@@ -220,14 +228,21 @@ TASK_TEMPLATE = """
 async def run_evaluation(
     eval_path: Path,
     connection: Any,
-    model: str = "claude-3-7-sonnet-20250219",
+    model: str,
+    allowed_tools: list[str] | None = None,
 ) -> str:
     """Run evaluation with MCP server tools."""
     print("🚀 Starting Evaluation")
 
-    client = Anthropic()
+    if not allowed_tools:
+        raise ValueError('Explicit --allow-tool is required; use reviewed read-only test tools')
+    client = Anthropic(timeout=60, max_retries=0)
 
     tools = await connection.list_tools()
+    missing = set(allowed_tools) - {tool['name'] for tool in tools}
+    if missing:
+        raise ValueError('Allowlisted tools not advertised: ' + ', '.join(sorted(missing)))
+    tools = [tool for tool in tools if tool['name'] in allowed_tools]
     print(f"📋 Loaded {len(tools)} tools from MCP server")
 
     qa_pairs = parse_evaluation_file(eval_path)
@@ -236,7 +251,7 @@ async def run_evaluation(
     results = []
     for i, qa_pair in enumerate(qa_pairs):
         print(f"Processing task {i + 1}/{len(qa_pairs)}")
-        result = await evaluate_single_task(client, model, qa_pair, tools, connection, i)
+        result = await asyncio.wait_for(evaluate_single_task(client, model, qa_pair, tools, connection, i), timeout=600)
         results.append(result)
 
     correct = sum(r["score"] for r in results)
@@ -321,7 +336,8 @@ Examples:
 
     parser.add_argument("eval_file", type=Path, help="Path to evaluation XML file")
     parser.add_argument("-t", "--transport", choices=["stdio", "sse", "http"], default="stdio", help="Transport type (default: stdio)")
-    parser.add_argument("-m", "--model", default="claude-3-7-sonnet-20250219", help="Claude model to use (default: claude-3-7-sonnet-20250219)")
+    parser.add_argument('-m', '--model', default=os.environ.get('ANTHROPIC_MODEL'), help='Authorized model ID, or ANTHROPIC_MODEL')
+    parser.add_argument('--allow-tool', action='append', required=True, help='Reviewed tool name; repeat for each permitted tool')
 
     stdio_group = parser.add_argument_group("stdio options")
     stdio_group.add_argument("-c", "--command", help="Command to run MCP server (stdio only)")
@@ -335,6 +351,8 @@ Examples:
     parser.add_argument("-o", "--output", type=Path, help="Output file for evaluation report (default: stdout)")
 
     args = parser.parse_args()
+    if not args.model:
+        parser.error('--model or ANTHROPIC_MODEL is required')
 
     if not args.eval_file.exists():
         print(f"Error: Evaluation file not found: {args.eval_file}")
@@ -360,7 +378,7 @@ Examples:
 
     async with connection:
         print("✅ Connected successfully")
-        report = await run_evaluation(args.eval_file, connection, args.model)
+        report = await run_evaluation(args.eval_file, connection, args.model, args.allow_tool)
 
         if args.output:
             args.output.write_text(report)

package/built-in-skills/mcp-builder/scripts/requirements.txt

@@ -1,2 +1,2 @@
-anthropic>=0.39.0
-mcp>=1.1.0
+anthropic>=0.39.0,<2
+mcp>=1.9.0,<2